2024-04-22T18:29:39.201 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T18:29:39.202 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/dedup-io-mixed} --name yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668454 --verbose --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668454/orig.config.yaml 2024-04-22T18:29:39.244 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T18:29:39.339 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668454 branch: wip-yuri2-testing-2024-04-17-0823-reef description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/dedup-io-mixed} email: null first_in_suite: false job_id: '7668454' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri2-testing-2024-04-17-0823-reef ceph: conf: global: mon election default strategy: 1 ms bind msgr2: false ms type: async osd debug inject dispatch delay duration: 0.1 osd debug inject dispatch delay probability: 0.1 osd_async_recovery_min_cost: 1 osd_object_clean_region_max_num_intervals: 1000 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd inject bad map crc probability: 0.1 osd max backfills: 3 osd max markdown count: 1000 osd max scrubs: 2 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - failed to encode map - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - \(POOL_APP_NOT_ENABLED\) mon_bind_msgr2: false sha1: 6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri2-testing-2024-04-17-0823-reef sha1: 6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 400 sha1: 6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri2-testing-2024-04-17-0823-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0 targets: smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP4lU1xAhS0ccTpXvPUiIQBiIvOZx7/qs2e0xFwav+/oEDogmpAH9By3o8WiGxiJPSJisGhDlpgCqj+bTR6lmyU= smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCX9iUvqlkVq+d0ykFVtV91xt8m5w0jIuQvMgYTGKr+JF3evHL9ahz5UEq79zp/6AiMEMow/+h7/GSURVTXTt+U= tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - exec: client.0: - sudo ceph osd pool create low_tier 4 - rados: clients: - client.0 dedup_chunk_algo: fastcdc dedup_chunk_size: '131072' enable_dedup: true low_tier_pool: low_tier objects: 50 op_weights: read: 100 set_chunk: 30 tier_evict: 10 tier_flush: 5 tier_promote: 10 write: 50 ops: 1500 set_chunk: true teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_18:19:58 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T18:29:39.340 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0/qa; will attempt to use it 2024-04-22T18:29:39.340 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0/qa/tasks 2024-04-22T18:29:39.340 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T18:29:39.342 INFO:teuthology.task.internal:Checking packages... 2024-04-22T18:29:39.369 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0' 2024-04-22T18:29:39.369 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T18:29:39.369 INFO:teuthology.packaging:ref: None 2024-04-22T18:29:39.369 INFO:teuthology.packaging:tag: None 2024-04-22T18:29:39.369 INFO:teuthology.packaging:branch: wip-yuri2-testing-2024-04-17-0823-reef 2024-04-22T18:29:39.369 INFO:teuthology.packaging:sha1: 6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0 2024-04-22T18:29:39.369 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri2-testing-2024-04-17-0823-reef 2024-04-22T18:29:39.515 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1240.g6a7528e4 2024-04-22T18:29:39.517 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T18:29:39.522 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T18:29:39.523 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T18:29:39.528 INFO:teuthology.task.internal:Saving configuration 2024-04-22T18:29:39.548 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T18:29:39.554 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T18:29:39.579 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668454', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 18:22:09.845630', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP4lU1xAhS0ccTpXvPUiIQBiIvOZx7/qs2e0xFwav+/oEDogmpAH9By3o8WiGxiJPSJisGhDlpgCqj+bTR6lmyU='} 2024-04-22T18:29:39.600 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668454', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 18:22:09.848160', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCX9iUvqlkVq+d0ykFVtV91xt8m5w0jIuQvMgYTGKr+JF3evHL9ahz5UEq79zp/6AiMEMow/+h7/GSURVTXTt+U='} 2024-04-22T18:29:39.601 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T18:29:39.609 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-04-22T18:29:39.610 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-22T18:29:39.610 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T18:29:39.677 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f63e77f2f70>, signals=[15]) 2024-04-22T18:29:39.678 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T18:29:39.684 INFO:teuthology.task.internal:Opening connections... 2024-04-22T18:29:39.684 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2024-04-22T18:29:39.686 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T18:29:39.765 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-04-22T18:29:39.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T18:29:39.841 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T18:29:39.848 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2024-04-22T18:29:39.866 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2024-04-22T18:29:39.866 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2024-04-22T18:29:39.923 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="CentOS Stream" 2024-04-22T18:29:39.923 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="9" 2024-04-22T18:29:39.923 INFO:teuthology.orchestra.run.smithi089.stdout:ID="centos" 2024-04-22T18:29:39.923 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="rhel fedora" 2024-04-22T18:29:39.923 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="9" 2024-04-22T18:29:39.924 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el9" 2024-04-22T18:29:39.924 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T18:29:39.924 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2024-04-22T18:29:39.924 INFO:teuthology.orchestra.run.smithi089.stdout:LOGO="fedora-logo-icon" 2024-04-22T18:29:39.924 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T18:29:39.924 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://centos.org/" 2024-04-22T18:29:39.924 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T18:29:39.924 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T18:29:39.924 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T18:29:39.925 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2024-04-22T18:29:39.944 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-04-22T18:29:39.962 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-04-22T18:29:39.962 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-04-22T18:29:40.026 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-04-22T18:29:40.026 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2024-04-22T18:29:40.026 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-04-22T18:29:40.027 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-04-22T18:29:40.027 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2024-04-22T18:29:40.027 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2024-04-22T18:29:40.027 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T18:29:40.027 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-04-22T18:29:40.027 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2024-04-22T18:29:40.027 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T18:29:40.027 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-04-22T18:29:40.027 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T18:29:40.027 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T18:29:40.027 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T18:29:40.028 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-04-22T18:29:40.060 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T18:29:40.069 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T18:29:40.074 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T18:29:40.074 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2024-04-22T18:29:40.076 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-04-22T18:29:40.092 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T18:29:40.098 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T18:29:40.098 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2024-04-22T18:29:40.133 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-04-22T18:29:40.421 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T18:29:40.478 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T18:29:40.494 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-22T18:29:40.494 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-22T18:29:40.495 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T18:29:40.495 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T18:29:40.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T18:29:40.512 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-04-22T18:29:40.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T18:29:40.513 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-04-22T18:29:40.568 INFO:teuthology.orchestra.run.smithi089.stdout:5.14.0-437.el9.x86_64 2024-04-22T18:29:40.569 INFO:teuthology.task.kernel:Running kernel on smithi089: 5.14.0-437.el9.x86_64 2024-04-22T18:29:40.569 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum install -y kernel 2024-04-22T18:29:40.571 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-437.el9.x86_64 2024-04-22T18:29:40.571 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.14.0-437.el9.x86_64 2024-04-22T18:29:40.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2024-04-22T18:29:42.892 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-22T18:29:44.684 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-22T18:29:51.127 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 918 kB/s | 8.0 MB 00:08 2024-04-22T18:29:52.126 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.2 MB 00:03 2024-04-22T18:29:53.294 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-04-22T18:29:53.777 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 16 kB 00:00 2024-04-22T18:29:55.898 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-04-22T18:29:57.586 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 8.2 MB/s | 6.2 MB 00:00 2024-04-22T18:29:59.212 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-04-22T18:30:00.632 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-22T18:30:01.013 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-22T18:30:02.834 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T18:30:02.834 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T18:30:02.919 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:30:02.920 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-04-22T18:30:02.920 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:30:03.052 DEBUG:teuthology.orchestra.run.smithi089:> echo no | sudo yum reinstall kernel || true 2024-04-22T18:30:03.551 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 06:30:00 PM UTC. 2024-04-22T18:30:03.680 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:30:03.691 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:30:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-04-22T18:30:03.691 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:30:03.691 INFO:teuthology.orchestra.run.smithi089.stdout:Reinstalling: 2024-04-22T18:30:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T18:30:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:30:03.691 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:30:03.691 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:30:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:30:03.692 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 255 k 2024-04-22T18:30:03.692 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 0 2024-04-22T18:30:03.692 INFO:teuthology.orchestra.run.smithi089.stderr:Operation aborted. 2024-04-22T18:30:03.751 INFO:teuthology.orchestra.run.smithi089.stdout:Is this ok [y/N]: 2024-04-22T18:30:03.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum reinstall -y kernel || true 2024-04-22T18:30:04.242 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:04 ago on Mon 22 Apr 2024 06:30:00 PM UTC. 2024-04-22T18:30:04.358 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:30:04.358 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:30:04.358 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-04-22T18:30:04.359 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:30:04.359 INFO:teuthology.orchestra.run.smithi089.stdout:Reinstalling: 2024-04-22T18:30:04.359 INFO:teuthology.orchestra.run.smithi089.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T18:30:04.359 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:30:04.359 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:30:04.359 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:30:04.359 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:30:04.359 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 255 k 2024-04-22T18:30:04.359 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 0 2024-04-22T18:30:04.359 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:30:05.072 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-437.el9.x86_64.rpm 467 kB/s | 255 kB 00:00 2024-04-22T18:30:05.072 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:30:05.072 INFO:teuthology.orchestra.run.smithi089.stdout:Total 357 kB/s | 255 kB 00:00 2024-04-22T18:30:05.090 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:30:05.280 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:30:05.280 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:30:05.285 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:30:05.285 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:30:05.433 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:30:05.507 INFO:teuthology.orchestra.run.smithi089.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T18:30:05.507 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T18:30:05.646 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-22T18:30:06.163 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T18:30:06.164 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T18:30:06.897 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T18:30:06.898 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:30:06.898 INFO:teuthology.orchestra.run.smithi089.stdout:Reinstalled: 2024-04-22T18:30:06.898 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T18:30:06.898 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:30:06.898 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:30:07.059 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T18:30:07.144 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T18:30:07.145 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T18:30:07.146 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T18:30:07.146 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-22T18:30:07.146 DEBUG:teuthology.orchestra.run.smithi089:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T18:30:07.177 INFO:teuthology.orchestra.run.smithi089.stdout:ttyS1 2024-04-22T18:30:07.194 DEBUG:teuthology.parallel:result is None 2024-04-22T18:30:07.560 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T18:30:07.560 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T18:30:07.656 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:30:07.657 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-22T18:30:07.657 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:30:07.766 DEBUG:teuthology.orchestra.run.smithi121:> echo no | sudo yum reinstall kernel || true 2024-04-22T18:30:08.269 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 06:30:05 PM UTC. 2024-04-22T18:30:08.388 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:30:08.405 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:30:08.405 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-22T18:30:08.405 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:30:08.406 INFO:teuthology.orchestra.run.smithi121.stdout:Reinstalling: 2024-04-22T18:30:08.406 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T18:30:08.406 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:30:08.406 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:30:08.406 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:30:08.406 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:30:08.406 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 255 k 2024-04-22T18:30:08.406 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 0 2024-04-22T18:30:08.406 INFO:teuthology.orchestra.run.smithi121.stderr:Operation aborted. 2024-04-22T18:30:08.466 INFO:teuthology.orchestra.run.smithi121.stdout:Is this ok [y/N]: 2024-04-22T18:30:08.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum reinstall -y kernel || true 2024-04-22T18:30:08.953 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 06:30:05 PM UTC. 2024-04-22T18:30:09.070 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:30:09.070 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:30:09.070 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-22T18:30:09.070 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:30:09.070 INFO:teuthology.orchestra.run.smithi121.stdout:Reinstalling: 2024-04-22T18:30:09.070 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T18:30:09.070 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:30:09.070 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:30:09.070 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:30:09.070 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:30:09.071 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 255 k 2024-04-22T18:30:09.071 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 0 2024-04-22T18:30:09.071 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:30:09.298 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.5 MB/s | 255 kB 00:00 2024-04-22T18:30:09.298 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:30:09.298 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.1 MB/s | 255 kB 00:00 2024-04-22T18:30:09.314 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:30:09.493 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:30:09.493 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:30:09.498 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:30:09.498 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:30:09.634 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:30:09.699 INFO:teuthology.orchestra.run.smithi121.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T18:30:09.699 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T18:30:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T18:30:10.391 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T18:30:11.120 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T18:30:11.120 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:30:11.120 INFO:teuthology.orchestra.run.smithi121.stdout:Reinstalled: 2024-04-22T18:30:11.120 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T18:30:11.120 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:30:11.120 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:30:11.310 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T18:30:11.357 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T18:30:11.357 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T18:30:11.357 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T18:30:11.357 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-22T18:30:11.358 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T18:30:11.391 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-04-22T18:30:11.419 DEBUG:teuthology.parallel:result is None 2024-04-22T18:30:11.420 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T18:30:11.429 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T18:30:11.430 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T18:30:11.433 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T18:30:11.475 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T18:30:11.481 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T18:30:11.493 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T18:30:11.493 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T18:30:11.496 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T18:30:11.616 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T18:30:11.623 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T18:30:11.624 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T18:30:11.626 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T18:30:11.659 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T18:30:11.670 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T18:30:11.677 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T18:30:11.686 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T18:30:11.687 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T18:30:11.695 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T18:30:11.695 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T18:30:11.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T18:30:11.738 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T18:30:11.747 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T18:30:11.747 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T18:30:11.781 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T18:30:11.796 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T18:30:11.881 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T18:30:11.927 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T18:30:12.011 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T18:30:12.052 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:30:12.052 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T18:30:12.118 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T18:30:12.159 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T18:30:12.209 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T18:30:12.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T18:30:12.330 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:30:12.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T18:30:12.393 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2024-04-22T18:30:12.396 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-04-22T18:30:12.431 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T18:30:12.483 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T18:30:12.940 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T18:30:12.948 INFO:teuthology.task.internal:Starting timer... 2024-04-22T18:30:12.948 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T18:30:12.961 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T18:30:12.970 DEBUG:teuthology.orchestra.run.smithi089:> sudo service auditd rotate 2024-04-22T18:30:13.023 INFO:teuthology.orchestra.run.smithi089.stdout:Rotating logs: 2024-04-22T18:30:13.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-04-22T18:30:13.078 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-04-22T18:30:13.080 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T18:30:13.080 DEBUG:teuthology.orchestra.run.smithi089:> /usr/sbin/getenforce 2024-04-22T18:30:13.113 INFO:teuthology.orchestra.run.smithi089.stdout:Permissive 2024-04-22T18:30:13.113 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-04-22T18:30:13.132 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-04-22T18:30:13.132 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi089.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive'} 2024-04-22T18:30:13.133 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T18:30:13.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T18:30:13.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T18:30:13.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T18:30:13.203 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T18:30:13.204 DEBUG:teuthology.orchestra.run.smithi089:> sudo /usr/sbin/setenforce permissive 2024-04-22T18:30:13.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-04-22T18:30:13.285 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T18:30:13.296 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T18:30:13.318 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T18:30:13.320 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi089.front.sepia.ceph.com,smithi121.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T18:34:03.749 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi089.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com')] 2024-04-22T18:34:03.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-04-22T18:34:03.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T18:34:03.831 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-04-22T18:34:03.908 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-04-22T18:34:03.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-22T18:34:03.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T18:34:03.984 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-04-22T18:34:04.058 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-22T18:34:04.059 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T18:34:04.069 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T18:34:04.070 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T18:34:04.070 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T18:34:04.074 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T18:34:04.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T18:34:04.101 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T18:34:04.113 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T18:34:04.129 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T18:34:04.136 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: ntpd: command not found 2024-04-22T18:34:04.142 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T18:34:04.146 INFO:teuthology.orchestra.run.smithi089.stdout:506 Cannot talk to daemon 2024-04-22T18:34:04.159 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T18:34:04.166 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-04-22T18:34:04.171 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T18:34:04.176 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-04-22T18:34:04.189 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T18:34:04.202 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T18:34:04.233 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-04-22T18:34:04.236 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T18:34:04.236 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-04-22T18:34:04.236 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T18:34:04.236 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T18:34:04.236 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T18:34:04.236 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T18:34:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-04-22T18:34:04.258 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T18:34:04.258 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-04-22T18:34:04.258 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T18:34:04.258 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T18:34:04.258 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T18:34:04.258 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T18:34:04.259 INFO:teuthology.run_tasks:Running task install... 2024-04-22T18:34:04.266 DEBUG:teuthology.task.install:project ceph 2024-04-22T18:34:04.267 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0'}} 2024-04-22T18:34:04.267 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0'} 2024-04-22T18:34:04.267 INFO:teuthology.task.install:Using flavor: default 2024-04-22T18:34:04.272 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T18:34:04.273 INFO:teuthology.task.install:extra packages: [] 2024-04-22T18:34:04.273 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0', 'tag': None, 'wait_for_package': False} 2024-04-22T18:34:04.274 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0 2024-04-22T18:34:04.276 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0', 'tag': None, 'wait_for_package': False} 2024-04-22T18:34:04.276 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0 2024-04-22T18:34:04.415 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-17-0823-reef/6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0/centos/9/flavors/default/ 2024-04-22T18:34:04.415 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1240.g6a7528e4 2024-04-22T18:34:04.424 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-17-0823-reef/6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0/centos/9/flavors/default/ 2024-04-22T18:34:04.424 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1240.g6a7528e4 2024-04-22T18:34:04.750 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-17-0823-reef/6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-17-0823-reef/6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-17-0823-reef/6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T18:34:04.750 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:34:04.751 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T18:34:04.776 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T18:34:04.776 DEBUG:teuthology.orchestra.run.smithi089:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T18:34:04.840 DEBUG:teuthology.orchestra.run.smithi089:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T18:34:04.913 DEBUG:teuthology.orchestra.run.smithi089:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T18:34:04.976 INFO:teuthology.orchestra.run.smithi089.stdout:check_obsoletes = 1 2024-04-22T18:34:04.978 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum clean all 2024-04-22T18:34:05.199 INFO:teuthology.orchestra.run.smithi089.stdout:48 files removed 2024-04-22T18:34:05.226 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-radosgw 2024-04-22T18:34:05.675 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-17-0823-reef/6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-17-0823-reef/6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-17-0823-reef/6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T18:34:05.675 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:34:05.675 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T18:34:05.705 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T18:34:05.705 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T18:34:05.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T18:34:05.849 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T18:34:05.914 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-04-22T18:34:05.915 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-04-22T18:34:06.142 INFO:teuthology.orchestra.run.smithi121.stdout:48 files removed 2024-04-22T18:34:06.169 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw 2024-04-22T18:34:06.212 INFO:teuthology.orchestra.run.smithi089.stdout:ceph packages for x86_64 106 kB/s | 77 kB 00:00 2024-04-22T18:34:06.938 INFO:teuthology.orchestra.run.smithi089.stdout:ceph noarch packages 17 kB/s | 11 kB 00:00 2024-04-22T18:34:07.588 INFO:teuthology.orchestra.run.smithi089.stdout:ceph source packages 3.0 kB/s | 1.9 kB 00:00 2024-04-22T18:34:07.996 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 49 kB/s | 77 kB 00:01 2024-04-22T18:34:08.747 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 16 kB/s | 11 kB 00:00 2024-04-22T18:34:09.523 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-04-22T18:34:10.080 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-22T18:34:11.902 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-22T18:34:17.057 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.2 MB 00:01 2024-04-22T18:34:17.104 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 869 kB/s | 8.0 MB 00:09 2024-04-22T18:34:18.599 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-04-22T18:34:43.360 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 869 kB/s | 21 MB 00:24 2024-04-22T18:34:44.744 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 733 kB/s | 19 MB 00:26 2024-04-22T18:34:48.446 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-22T18:34:49.028 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.2 MB 00:00 2024-04-22T18:34:50.586 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:34:50.588 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:34:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-22T18:34:50.588 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:34:50.589 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:34:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 7.7 M 2024-04-22T18:34:50.589 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-04-22T18:34:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 3.3 M 2024-04-22T18:34:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 3.0 M 2024-04-22T18:34:50.589 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-22T18:34:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T18:34:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 5.1 M 2024-04-22T18:34:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 18 M 2024-04-22T18:34:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 25 k 2024-04-22T18:34:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-22T18:34:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T18:34:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T18:34:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T18:34:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 708 k 2024-04-22T18:34:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T18:34:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T18:34:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T18:34:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 474 k 2024-04-22T18:34:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T18:34:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 4.5 M 2024-04-22T18:34:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T18:34:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T18:34:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T18:34:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 45 k 2024-04-22T18:34:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 129 k 2024-04-22T18:34:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 162 k 2024-04-22T18:34:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T18:34:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 321 k 2024-04-22T18:34:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 297 k 2024-04-22T18:34:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 99 k 2024-04-22T18:34:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T18:34:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T18:34:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:34:50.591 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:34:50.591 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:34:50.591 INFO:teuthology.orchestra.run.smithi121.stdout:Install 28 Packages 2024-04-22T18:34:50.591 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 2 Packages 2024-04-22T18:34:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:34:50.592 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 53 M 2024-04-22T18:34:50.592 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:34:50.670 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-22T18:34:52.664 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-22T18:34:54.889 INFO:teuthology.orchestra.run.smithi121.stdout:(1/30): ceph-radosgw-18.2.2-1240.g6a7528e4.el9. 2.0 MB/s | 7.7 MB 00:03 2024-04-22T18:34:55.064 INFO:teuthology.orchestra.run.smithi121.stdout:(2/30): ceph-selinux-18.2.2-1240.g6a7528e4.el9. 144 kB/s | 25 kB 00:00 2024-04-22T18:34:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:(3/30): ceph-base-18.2.2-1240.g6a7528e4.el9.x86 1.1 MB/s | 5.1 MB 00:04 2024-04-22T18:34:55.931 INFO:teuthology.orchestra.run.smithi121.stdout:(4/30): libcephfs2-18.2.2-1240.g6a7528e4.el9.x8 817 kB/s | 708 kB 00:00 2024-04-22T18:34:56.689 INFO:teuthology.orchestra.run.smithi121.stdout:(5/30): libradosstriper1-18.2.2-1240.g6a7528e4. 395 kB/s | 474 kB 00:01 2024-04-22T18:34:56.797 INFO:teuthology.orchestra.run.smithi121.stdout:(6/30): python3-ceph-argparse-18.2.2-1240.g6a75 416 kB/s | 45 kB 00:00 2024-04-22T18:34:57.390 INFO:teuthology.orchestra.run.smithi121.stdout:(7/30): python3-ceph-common-18.2.2-1240.g6a7528 218 kB/s | 129 kB 00:00 2024-04-22T18:34:57.789 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-22T18:34:58.206 INFO:teuthology.orchestra.run.smithi121.stdout:(8/30): python3-cephfs-18.2.2-1240.g6a7528e4.el 198 kB/s | 162 kB 00:00 2024-04-22T18:34:58.890 INFO:teuthology.orchestra.run.smithi121.stdout:(9/30): python3-rados-18.2.2-1240.g6a7528e4.el9 470 kB/s | 321 kB 00:00 2024-04-22T18:34:59.515 INFO:teuthology.orchestra.run.smithi121.stdout:(10/30): python3-rbd-18.2.2-1240.g6a7528e4.el9. 476 kB/s | 297 kB 00:00 2024-04-22T18:34:59.723 INFO:teuthology.orchestra.run.smithi121.stdout:(11/30): python3-rgw-18.2.2-1240.g6a7528e4.el9. 477 kB/s | 99 kB 00:00 2024-04-22T18:34:59.921 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:34:59.923 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:34:59.923 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-04-22T18:34:59.923 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:34:59.923 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:34:59.923 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 7.7 M 2024-04-22T18:34:59.923 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrading: 2024-04-22T18:34:59.923 INFO:teuthology.orchestra.run.smithi089.stdout: librados2 x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 3.3 M 2024-04-22T18:34:59.923 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1 x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 3.0 M 2024-04-22T18:34:59.923 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-04-22T18:34:59.923 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T18:34:59.924 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 5.1 M 2024-04-22T18:34:59.924 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 18 M 2024-04-22T18:34:59.924 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 25 k 2024-04-22T18:34:59.924 INFO:teuthology.orchestra.run.smithi089.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-04-22T18:34:59.924 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T18:34:59.924 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T18:34:59.924 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T18:34:59.924 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2 x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 708 k 2024-04-22T18:34:59.924 INFO:teuthology.orchestra.run.smithi089.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T18:34:59.924 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T18:34:59.924 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T18:34:59.924 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 474 k 2024-04-22T18:34:59.924 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T18:34:59.925 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2 x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 4.5 M 2024-04-22T18:34:59.925 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T18:34:59.925 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T18:34:59.925 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T18:34:59.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 45 k 2024-04-22T18:34:59.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 129 k 2024-04-22T18:34:59.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 162 k 2024-04-22T18:34:59.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T18:34:59.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 321 k 2024-04-22T18:34:59.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 297 k 2024-04-22T18:34:59.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 99 k 2024-04-22T18:34:59.925 INFO:teuthology.orchestra.run.smithi089.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T18:34:59.925 INFO:teuthology.orchestra.run.smithi089.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T18:34:59.925 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:34:59.926 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:34:59.926 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:34:59.926 INFO:teuthology.orchestra.run.smithi089.stdout:Install 28 Packages 2024-04-22T18:34:59.926 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrade 2 Packages 2024-04-22T18:34:59.926 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:34:59.926 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 53 M 2024-04-22T18:34:59.927 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:34:59.965 INFO:teuthology.orchestra.run.smithi121.stdout:(12/30): boost-program-options-1.75.0-8.el9.x86 442 kB/s | 107 kB 00:00 2024-04-22T18:35:00.099 INFO:teuthology.orchestra.run.smithi121.stdout:(13/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-04-22T18:35:00.166 INFO:teuthology.orchestra.run.smithi121.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 677 kB/s | 45 kB 00:00 2024-04-22T18:35:00.400 INFO:teuthology.orchestra.run.smithi121.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.8 MB/s | 662 kB 00:00 2024-04-22T18:35:00.533 INFO:teuthology.orchestra.run.smithi121.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-04-22T18:35:00.592 INFO:teuthology.orchestra.run.smithi121.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 724 kB/s | 42 kB 00:00 2024-04-22T18:35:00.817 INFO:teuthology.orchestra.run.smithi121.stdout:(18/30): fmt-8.1.1-5.el9.x86_64.rpm 492 kB/s | 111 kB 00:00 2024-04-22T18:35:00.884 INFO:teuthology.orchestra.run.smithi121.stdout:(19/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.5 MB/s | 309 kB 00:00 2024-04-22T18:35:01.034 INFO:teuthology.orchestra.run.smithi121.stdout:(20/30): libarrow-9.0.0-10.el9.x86_64.rpm 29 MB/s | 4.4 MB 00:00 2024-04-22T18:35:01.077 INFO:teuthology.orchestra.run.smithi121.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 619 kB/s | 26 kB 00:00 2024-04-22T18:35:01.118 INFO:teuthology.orchestra.run.smithi121.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-22T18:35:01.161 INFO:teuthology.orchestra.run.smithi121.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-22T18:35:01.211 INFO:teuthology.orchestra.run.smithi121.stdout:(24/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 16 MB/s | 838 kB 00:00 2024-04-22T18:35:01.261 INFO:teuthology.orchestra.run.smithi121.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-22T18:35:01.328 INFO:teuthology.orchestra.run.smithi121.stdout:(26/30): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-04-22T18:35:02.345 INFO:teuthology.orchestra.run.smithi121.stdout:(27/30): librgw2-18.2.2-1240.g6a7528e4.el9.x86_ 712 kB/s | 4.5 MB 00:06 2024-04-22T18:35:09.811 INFO:teuthology.orchestra.run.smithi121.stdout:(28/30): librbd1-18.2.2-1240.g6a7528e4.el9.x86_ 412 kB/s | 3.0 MB 00:07 2024-04-22T18:35:13.455 INFO:teuthology.orchestra.run.smithi089.stdout:(1/30): ceph-base-18.2.2-1240.g6a7528e4.el9.x86 395 kB/s | 5.1 MB 00:13 2024-04-22T18:35:13.556 INFO:teuthology.orchestra.run.smithi089.stdout:(2/30): ceph-selinux-18.2.2-1240.g6a7528e4.el9. 253 kB/s | 25 kB 00:00 2024-04-22T18:35:13.851 INFO:teuthology.orchestra.run.smithi121.stdout:(29/30): librados2-18.2.2-1240.g6a7528e4.el9.x8 269 kB/s | 3.3 MB 00:12 2024-04-22T18:35:15.439 INFO:teuthology.orchestra.run.smithi089.stdout:(3/30): libcephfs2-18.2.2-1240.g6a7528e4.el9.x8 376 kB/s | 708 kB 00:01 2024-04-22T18:35:16.839 INFO:teuthology.orchestra.run.smithi089.stdout:(4/30): libradosstriper1-18.2.2-1240.g6a7528e4. 339 kB/s | 474 kB 00:01 2024-04-22T18:35:20.014 INFO:teuthology.orchestra.run.smithi089.stdout:(5/30): ceph-radosgw-18.2.2-1240.g6a7528e4.el9. 401 kB/s | 7.7 MB 00:19 2024-04-22T18:35:20.239 INFO:teuthology.orchestra.run.smithi089.stdout:(6/30): python3-ceph-argparse-18.2.2-1240.g6a75 199 kB/s | 45 kB 00:00 2024-04-22T18:35:20.556 INFO:teuthology.orchestra.run.smithi089.stdout:(7/30): python3-ceph-common-18.2.2-1240.g6a7528 408 kB/s | 129 kB 00:00 2024-04-22T18:35:21.048 INFO:teuthology.orchestra.run.smithi089.stdout:(8/30): python3-cephfs-18.2.2-1240.g6a7528e4.el 330 kB/s | 162 kB 00:00 2024-04-22T18:35:22.023 INFO:teuthology.orchestra.run.smithi089.stdout:(9/30): python3-rados-18.2.2-1240.g6a7528e4.el9 330 kB/s | 321 kB 00:00 2024-04-22T18:35:23.515 INFO:teuthology.orchestra.run.smithi089.stdout:(10/30): python3-rbd-18.2.2-1240.g6a7528e4.el9. 199 kB/s | 297 kB 00:01 2024-04-22T18:35:24.065 INFO:teuthology.orchestra.run.smithi089.stdout:(11/30): librgw2-18.2.2-1240.g6a7528e4.el9.x86_ 632 kB/s | 4.5 MB 00:07 2024-04-22T18:35:24.132 INFO:teuthology.orchestra.run.smithi089.stdout:(12/30): boost-program-options-1.75.0-8.el9.x86 1.6 MB/s | 107 kB 00:00 2024-04-22T18:35:24.182 INFO:teuthology.orchestra.run.smithi089.stdout:(13/30): python3-rgw-18.2.2-1240.g6a7528e4.el9. 149 kB/s | 99 kB 00:00 2024-04-22T18:35:24.208 INFO:teuthology.orchestra.run.smithi089.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-04-22T18:35:24.350 INFO:teuthology.orchestra.run.smithi089.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 270 kB/s | 45 kB 00:00 2024-04-22T18:35:24.519 INFO:teuthology.orchestra.run.smithi089.stdout:(16/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-04-22T18:35:24.544 INFO:teuthology.orchestra.run.smithi089.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.5 MB/s | 292 kB 00:00 2024-04-22T18:35:24.594 INFO:teuthology.orchestra.run.smithi089.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 562 kB/s | 42 kB 00:00 2024-04-22T18:35:24.770 INFO:teuthology.orchestra.run.smithi089.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 491 kB/s | 111 kB 00:00 2024-04-22T18:35:24.845 INFO:teuthology.orchestra.run.smithi089.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-04-22T18:35:24.912 INFO:teuthology.orchestra.run.smithi089.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 390 kB/s | 26 kB 00:00 2024-04-22T18:35:24.979 INFO:teuthology.orchestra.run.smithi089.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 729 kB/s | 49 kB 00:00 2024-04-22T18:35:25.054 INFO:teuthology.orchestra.run.smithi089.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 900 kB/s | 67 kB 00:00 2024-04-22T18:35:25.254 INFO:teuthology.orchestra.run.smithi089.stdout:(24/30): libarrow-9.0.0-10.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-04-22T18:35:25.288 INFO:teuthology.orchestra.run.smithi089.stdout:(25/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 838 kB 00:00 2024-04-22T18:35:25.321 INFO:teuthology.orchestra.run.smithi089.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-04-22T18:35:25.572 INFO:teuthology.orchestra.run.smithi089.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-04-22T18:35:30.538 INFO:teuthology.orchestra.run.smithi089.stdout:(28/30): librados2-18.2.2-1240.g6a7528e4.el9.x8 647 kB/s | 3.3 MB 00:05 2024-04-22T18:35:33.273 INFO:teuthology.orchestra.run.smithi121.stdout:(30/30): ceph-common-18.2.2-1240.g6a7528e4.el9. 444 kB/s | 18 MB 00:42 2024-04-22T18:35:33.278 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:35:33.278 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.2 MB/s | 53 MB 00:42 2024-04-22T18:35:33.345 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:35:33.380 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:35:33.381 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:35:33.799 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:35:33.799 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:35:34.688 INFO:teuthology.orchestra.run.smithi089.stdout:(29/30): librbd1-18.2.2-1240.g6a7528e4.el9.x86_ 337 kB/s | 3.0 MB 00:09 2024-04-22T18:35:35.165 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:35:35.248 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-22T18:35:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-22T18:35:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-22T18:35:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-22T18:35:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:18.2.2-1240.g6a7528e4.el9.x86_64 5/32 2024-04-22T18:35:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:18.2.2-1240.g6a7528e4.el9.x86_64 5/32 2024-04-22T18:35:35.936 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:18.2.2-1240.g6a7528e4.el9.x86_64 6/32 2024-04-22T18:35:35.962 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:18.2.2-1240.g6a7528e4.el9.x86_64 7/32 2024-04-22T18:35:36.024 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:18.2.2-1240.g6a7528e4.el9.x86_64 7/32 2024-04-22T18:35:36.053 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-22T18:35:36.097 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:18.2.2-1240.g6a7528e4.el9. 9/32 2024-04-22T18:35:36.162 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:18.2.2-1240.g6a7528e4.el9.x86_64 10/32 2024-04-22T18:35:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:18.2.2-1240.g6a7528e4.el9.x86_6 11/32 2024-04-22T18:35:36.254 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1240.g6a7528e4.el9.x86_6 11/32 2024-04-22T18:35:36.361 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-22T18:35:36.404 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-22T18:35:36.444 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-22T18:35:36.469 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-22T18:35:36.497 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-22T18:35:36.533 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-22T18:35:36.565 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-22T18:35:36.863 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-22T18:35:36.912 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:18.2.2-1240.g6a7528e4.el9.x86_64 20/32 2024-04-22T18:35:37.001 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:18.2.2-1240.g6a7528e4.el9.x86_64 20/32 2024-04-22T18:35:37.033 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 21/32 2024-04-22T18:35:37.068 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-22T18:35:37.468 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-22T18:35:37.497 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:18.2.2-1240.g6a7528e4.el9.x86_64 24/32 2024-04-22T18:35:37.568 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:18.2.2-1240.g6a7528e4.el9.x86_64 24/32 2024-04-22T18:35:37.637 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 25/32 2024-04-22T18:35:37.666 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:18.2.2-1240.g6a7528e4.el9.x8 26/32 2024-04-22T18:35:39.724 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:18.2.2-1240.g6a7528e4.el9.x86_64 27/32 2024-04-22T18:35:39.765 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:18.2.2-1240.g6a7528e4.el9.x86_64 27/32 2024-04-22T18:35:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:18.2.2-1240.g6a7528e4.el9.x86_64 27/32 2024-04-22T18:35:40.326 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:18.2.2-1240.g6a7528e4.el9.x86_64 28/32 2024-04-22T18:35:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:18.2.2-1240.g6a7528e4.el9.x86_64 28/32 2024-04-22T18:35:40.422 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T18:35:40.422 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T18:35:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:35:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:18.2.2-1240.g6a7528e4.el9.x86_64 29/32 2024-04-22T18:35:41.196 INFO:teuthology.orchestra.run.smithi089.stdout:(30/30): ceph-common-18.2.2-1240.g6a7528e4.el9. 459 kB/s | 18 MB 00:40 2024-04-22T18:35:41.199 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:35:41.199 INFO:teuthology.orchestra.run.smithi089.stdout:Total 1.3 MB/s | 53 MB 00:41 2024-04-22T18:35:41.274 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:35:41.311 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:35:41.311 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:35:41.734 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:35:41.734 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:35:43.121 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:35:43.196 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-04-22T18:35:43.234 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-04-22T18:35:43.284 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-04-22T18:35:43.604 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-04-22T18:35:43.716 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librados2-2:18.2.2-1240.g6a7528e4.el9.x86_64 5/32 2024-04-22T18:35:43.808 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:18.2.2-1240.g6a7528e4.el9.x86_64 5/32 2024-04-22T18:35:43.892 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rados-2:18.2.2-1240.g6a7528e4.el9.x86_64 6/32 2024-04-22T18:35:43.918 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs2-2:18.2.2-1240.g6a7528e4.el9.x86_64 7/32 2024-04-22T18:35:43.980 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs2-2:18.2.2-1240.g6a7528e4.el9.x86_64 7/32 2024-04-22T18:35:44.009 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-04-22T18:35:44.052 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-argparse-2:18.2.2-1240.g6a7528e4.el9. 9/32 2024-04-22T18:35:44.118 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cephfs-2:18.2.2-1240.g6a7528e4.el9.x86_64 10/32 2024-04-22T18:35:44.145 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libradosstriper1-2:18.2.2-1240.g6a7528e4.el9.x86_6 11/32 2024-04-22T18:35:44.193 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1240.g6a7528e4.el9.x86_6 11/32 2024-04-22T18:35:44.311 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-04-22T18:35:44.345 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-04-22T18:35:44.383 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-04-22T18:35:44.409 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-04-22T18:35:44.436 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-04-22T18:35:44.472 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-04-22T18:35:44.505 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-04-22T18:35:44.803 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-04-22T18:35:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librbd1-2:18.2.2-1240.g6a7528e4.el9.x86_64 20/32 2024-04-22T18:35:44.932 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:18.2.2-1240.g6a7528e4.el9.x86_64 20/32 2024-04-22T18:35:44.964 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 21/32 2024-04-22T18:35:45.000 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-04-22T18:35:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-04-22T18:35:45.437 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librgw2-2:18.2.2-1240.g6a7528e4.el9.x86_64 24/32 2024-04-22T18:35:45.491 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librgw2-2:18.2.2-1240.g6a7528e4.el9.x86_64 24/32 2024-04-22T18:35:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 25/32 2024-04-22T18:35:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-common-2:18.2.2-1240.g6a7528e4.el9.x8 26/32 2024-04-22T18:35:47.681 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:18.2.2-1240.g6a7528e4.el9.x86_64 27/32 2024-04-22T18:35:47.719 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-common-2:18.2.2-1240.g6a7528e4.el9.x86_64 27/32 2024-04-22T18:35:48.239 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:18.2.2-1240.g6a7528e4.el9.x86_64 27/32 2024-04-22T18:35:48.281 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-base-2:18.2.2-1240.g6a7528e4.el9.x86_64 28/32 2024-04-22T18:35:48.383 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-base-2:18.2.2-1240.g6a7528e4.el9.x86_64 28/32 2024-04-22T18:35:48.383 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T18:35:48.383 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T18:35:48.384 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:35:48.416 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-selinux-2:18.2.2-1240.g6a7528e4.el9.x86_64 29/32 2024-04-22T18:35:49.708 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1240.g6a7528e4.el9.x86_64 29/32 2024-04-22T18:35:49.708 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-04-22T18:35:49.708 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-04-22T18:35:49.708 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-04-22T18:35:49.709 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-04-22T18:35:49.709 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-04-22T18:35:49.709 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /root 2024-04-22T18:35:49.709 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-04-22T18:35:49.709 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:35:50.468 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 30/32 2024-04-22T18:35:50.491 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 30/32 2024-04-22T18:35:50.491 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:35:50.491 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T18:35:50.491 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T18:35:50.491 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T18:35:50.491 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:35:50.493 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T18:35:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T18:35:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T18:35:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T18:35:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/32 2024-04-22T18:35:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:18.2.2-1240.g6a7528e4.el9.x86_64 2/32 2024-04-22T18:35:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 3/32 2024-04-22T18:35:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:18.2.2-1240.g6a7528e4.el9.x86_64 4/32 2024-04-22T18:35:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:18.2.2-1240.g6a7528e4.el9.x86_64 5/32 2024-04-22T18:35:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:18.2.2-1240.g6a7528e4.el9.x86_6 6/32 2024-04-22T18:35:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:18.2.2-1240.g6a7528e4.el9.x86_64 7/32 2024-04-22T18:35:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1240.g6a7528e4.el9. 8/32 2024-04-22T18:35:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:18.2.2-1240.g6a7528e4.el9.x8 9/32 2024-04-22T18:35:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:18.2.2-1240.g6a7528e4.el9.x86_64 10/32 2024-04-22T18:35:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:18.2.2-1240.g6a7528e4.el9.x86_64 11/32 2024-04-22T18:35:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 12/32 2024-04-22T18:35:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 13/32 2024-04-22T18:35:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-22T18:35:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-22T18:35:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-22T18:35:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-22T18:35:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-22T18:35:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-22T18:35:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-22T18:35:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-22T18:35:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-22T18:35:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-22T18:35:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-22T18:35:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-22T18:35:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-22T18:35:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-22T18:35:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-22T18:35:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:18.2.2-1240.g6a7528e4.el9.x86_64 29/32 2024-04-22T18:35:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-22T18:35:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:18.2.2-1240.g6a7528e4.el9.x86_64 31/32 2024-04-22T18:35:52.394 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T18:35:52.394 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:35:52.394 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-04-22T18:35:52.394 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:35:52.394 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:35:52.395 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:35:52.395 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T18:35:52.395 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:35:52.395 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:35:52.395 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:35:52.395 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:35:52.395 INFO:teuthology.orchestra.run.smithi121.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-22T18:35:52.395 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T18:35:52.395 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T18:35:52.395 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T18:35:52.395 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:35:52.395 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T18:35:52.396 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T18:35:52.396 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T18:35:52.396 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:35:52.396 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T18:35:52.396 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:35:52.396 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T18:35:52.396 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T18:35:52.396 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T18:35:52.396 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:35:52.396 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:35:52.396 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:35:52.396 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T18:35:52.396 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:35:52.397 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:35:52.397 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:35:52.397 INFO:teuthology.orchestra.run.smithi121.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T18:35:52.397 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T18:35:52.397 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:35:52.397 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:35:52.679 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-test 2024-04-22T18:35:53.290 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:05 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:35:53.414 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:35:53.415 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:35:53.415 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-22T18:35:53.415 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:35:53.415 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:35:53.415 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 40 M 2024-04-22T18:35:53.415 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-22T18:35:53.415 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 166 k 2024-04-22T18:35:53.416 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T18:35:53.416 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T18:35:53.416 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:35:53.416 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:35:53.416 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:35:53.416 INFO:teuthology.orchestra.run.smithi121.stdout:Install 4 Packages 2024-04-22T18:35:53.416 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:35:53.417 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 40 M 2024-04-22T18:35:53.417 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 163 M 2024-04-22T18:35:53.417 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:35:54.023 INFO:teuthology.orchestra.run.smithi121.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 814 kB/s | 305 kB 00:00 2024-04-22T18:35:54.116 INFO:teuthology.orchestra.run.smithi121.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 691 kB/s | 64 kB 00:00 2024-04-22T18:35:54.351 INFO:teuthology.orchestra.run.smithi121.stdout:(3/4): libcephsqlite-18.2.2-1240.g6a7528e4.el9. 236 kB/s | 166 kB 00:00 2024-04-22T18:35:57.372 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1240.g6a7528e4.el9.x86_64 29/32 2024-04-22T18:35:57.372 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /sys 2024-04-22T18:35:57.372 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /proc 2024-04-22T18:35:57.372 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /mnt 2024-04-22T18:35:57.372 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /var/tmp 2024-04-22T18:35:57.372 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /home 2024-04-22T18:35:57.372 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /root 2024-04-22T18:35:57.372 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /tmp 2024-04-22T18:35:57.372 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:35:58.089 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-radosgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 30/32 2024-04-22T18:35:58.114 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 30/32 2024-04-22T18:35:58.114 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:35:58.114 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T18:35:58.114 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T18:35:58.114 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T18:35:58.114 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:35:58.116 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T18:35:58.200 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-04-22T18:35:58.200 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T18:35:59.399 INFO:teuthology.orchestra.run.smithi121.stdout:(4/4): ceph-test-18.2.2-1240.g6a7528e4.el9.x86_ 6.9 MB/s | 40 MB 00:05 2024-04-22T18:35:59.403 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:35:59.403 INFO:teuthology.orchestra.run.smithi121.stdout:Total 6.7 MB/s | 40 MB 00:05 2024-04-22T18:35:59.429 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:35:59.436 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T18:35:59.436 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-base-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/32 2024-04-22T18:35:59.436 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-common-2:18.2.2-1240.g6a7528e4.el9.x86_64 2/32 2024-04-22T18:35:59.436 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-radosgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 3/32 2024-04-22T18:35:59.436 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-selinux-2:18.2.2-1240.g6a7528e4.el9.x86_64 4/32 2024-04-22T18:35:59.437 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs2-2:18.2.2-1240.g6a7528e4.el9.x86_64 5/32 2024-04-22T18:35:59.437 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libradosstriper1-2:18.2.2-1240.g6a7528e4.el9.x86_6 6/32 2024-04-22T18:35:59.437 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librgw2-2:18.2.2-1240.g6a7528e4.el9.x86_64 7/32 2024-04-22T18:35:59.437 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1240.g6a7528e4.el9. 8/32 2024-04-22T18:35:59.437 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-common-2:18.2.2-1240.g6a7528e4.el9.x8 9/32 2024-04-22T18:35:59.437 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cephfs-2:18.2.2-1240.g6a7528e4.el9.x86_64 10/32 2024-04-22T18:35:59.437 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rados-2:18.2.2-1240.g6a7528e4.el9.x86_64 11/32 2024-04-22T18:35:59.437 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 12/32 2024-04-22T18:35:59.437 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 13/32 2024-04-22T18:35:59.437 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-04-22T18:35:59.437 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-04-22T18:35:59.437 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-04-22T18:35:59.437 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-04-22T18:35:59.437 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-04-22T18:35:59.437 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-04-22T18:35:59.438 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-04-22T18:35:59.438 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-04-22T18:35:59.438 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-04-22T18:35:59.438 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-04-22T18:35:59.438 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-04-22T18:35:59.438 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-04-22T18:35:59.438 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-04-22T18:35:59.438 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-04-22T18:35:59.438 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-04-22T18:35:59.438 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:18.2.2-1240.g6a7528e4.el9.x86_64 29/32 2024-04-22T18:35:59.438 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-04-22T18:35:59.439 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:18.2.2-1240.g6a7528e4.el9.x86_64 31/32 2024-04-22T18:35:59.449 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:35:59.449 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:35:59.762 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:35:59.763 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:36:00.192 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-04-22T18:36:00.192 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:00.192 INFO:teuthology.orchestra.run.smithi089.stdout:Upgraded: 2024-04-22T18:36:00.192 INFO:teuthology.orchestra.run.smithi089.stdout: librados2-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:00.193 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:00.193 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:36:00.193 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T18:36:00.193 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:00.193 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:00.193 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:00.193 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:00.193 INFO:teuthology.orchestra.run.smithi089.stdout: fmt-8.1.1-5.el9.x86_64 2024-04-22T18:36:00.193 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T18:36:00.193 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T18:36:00.193 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T18:36:00.193 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:00.193 INFO:teuthology.orchestra.run.smithi089.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T18:36:00.194 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T18:36:00.194 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T18:36:00.194 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:00.194 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T18:36:00.194 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:00.194 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T18:36:00.194 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T18:36:00.194 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T18:36:00.194 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:00.194 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:00.194 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:00.194 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T18:36:00.194 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:00.194 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:00.195 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:00.195 INFO:teuthology.orchestra.run.smithi089.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T18:36:00.195 INFO:teuthology.orchestra.run.smithi089.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T18:36:00.195 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:00.195 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:36:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:36:00.308 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-22T18:36:00.357 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-22T18:36:00.384 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:18.2.2-1240.g6a7528e4.el9.x86_64 3/4 2024-04-22T18:36:00.487 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-test 2024-04-22T18:36:01.034 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:04 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:36:01.156 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:36:01.156 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:36:01.156 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-04-22T18:36:01.156 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:36:01.157 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:36:01.157 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 40 M 2024-04-22T18:36:01.157 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-04-22T18:36:01.157 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 166 k 2024-04-22T18:36:01.157 INFO:teuthology.orchestra.run.smithi089.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T18:36:01.157 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T18:36:01.157 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:01.157 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:36:01.157 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:36:01.157 INFO:teuthology.orchestra.run.smithi089.stdout:Install 4 Packages 2024-04-22T18:36:01.157 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:01.158 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 40 M 2024-04-22T18:36:01.158 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 163 M 2024-04-22T18:36:01.158 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:36:02.041 INFO:teuthology.orchestra.run.smithi089.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 417 kB/s | 305 kB 00:00 2024-04-22T18:36:02.150 INFO:teuthology.orchestra.run.smithi089.stdout:(2/4): libcephsqlite-18.2.2-1240.g6a7528e4.el9. 197 kB/s | 166 kB 00:00 2024-04-22T18:36:02.183 INFO:teuthology.orchestra.run.smithi089.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 446 kB/s | 64 kB 00:00 2024-04-22T18:36:04.044 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1240.g6a7528e4.el9.x86_64 3/4 2024-04-22T18:36:04.131 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:18.2.2-1240.g6a7528e4.el9.x86_64 4/4 2024-04-22T18:36:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-test-2:18.2.2-1240.g6a7528e4.el9.x86_64 4/4 2024-04-22T18:36:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/4 2024-04-22T18:36:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:18.2.2-1240.g6a7528e4.el9.x86_64 2/4 2024-04-22T18:36:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-22T18:36:06.234 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-22T18:36:06.235 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:06.235 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:36:06.235 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:06.235 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:06.235 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T18:36:06.235 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T18:36:06.235 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:06.235 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:36:06.511 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph 2024-04-22T18:36:07.012 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:19 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:36:07.135 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:36:07.137 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:36:07.137 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-22T18:36:07.137 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:36:07.137 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:36:07.137 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 6.5 k 2024-04-22T18:36:07.137 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-22T18:36:07.137 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 2.1 M 2024-04-22T18:36:07.138 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 1.5 M 2024-04-22T18:36:07.138 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 246 k 2024-04-22T18:36:07.138 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 4.7 M 2024-04-22T18:36:07.138 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 17 M 2024-04-22T18:36:07.138 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T18:36:07.138 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T18:36:07.138 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T18:36:07.138 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T18:36:07.138 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T18:36:07.138 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T18:36:07.138 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T18:36:07.138 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections 2024-04-22T18:36:07.138 INFO:teuthology.orchestra.run.smithi121.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-22T18:36:07.139 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T18:36:07.139 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T18:36:07.139 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T18:36:07.139 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T18:36:07.139 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T18:36:07.139 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T18:36:07.139 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T18:36:07.139 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T18:36:07.139 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T18:36:07.139 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T18:36:07.139 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T18:36:07.139 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T18:36:07.139 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T18:36:07.139 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T18:36:07.139 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T18:36:07.140 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T18:36:07.140 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T18:36:07.140 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T18:36:07.140 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T18:36:07.140 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:07.140 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:36:07.140 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:36:07.140 INFO:teuthology.orchestra.run.smithi121.stdout:Install 33 Packages 2024-04-22T18:36:07.140 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:07.141 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 29 M 2024-04-22T18:36:07.141 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 104 M 2024-04-22T18:36:07.141 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:36:08.166 INFO:teuthology.orchestra.run.smithi121.stdout:(1/33): ceph-18.2.2-1240.g6a7528e4.el9.x86_64.r 14 kB/s | 6.5 kB 00:00 2024-04-22T18:36:10.241 INFO:teuthology.orchestra.run.smithi121.stdout:(2/33): ceph-mds-18.2.2-1240.g6a7528e4.el9.x86_ 844 kB/s | 2.1 MB 00:02 2024-04-22T18:36:10.766 INFO:teuthology.orchestra.run.smithi121.stdout:(3/33): ceph-mgr-18.2.2-1240.g6a7528e4.el9.x86_ 486 kB/s | 1.5 MB 00:03 2024-04-22T18:36:11.058 INFO:teuthology.orchestra.run.smithi121.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1240.g6a75 843 kB/s | 246 kB 00:00 2024-04-22T18:36:11.158 INFO:teuthology.orchestra.run.smithi121.stdout:(5/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 253 kB 00:00 2024-04-22T18:36:11.258 INFO:teuthology.orchestra.run.smithi121.stdout:(6/33): python3-cryptography-36.0.1-4.el9.x86_6 12 MB/s | 1.2 MB 00:00 2024-04-22T18:36:11.325 INFO:teuthology.orchestra.run.smithi121.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-22T18:36:11.392 INFO:teuthology.orchestra.run.smithi121.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-04-22T18:36:11.459 INFO:teuthology.orchestra.run.smithi121.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 1.8 MB/s | 125 kB 00:00 2024-04-22T18:36:11.551 INFO:teuthology.orchestra.run.smithi121.stdout:(10/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-04-22T18:36:11.743 INFO:teuthology.orchestra.run.smithi121.stdout:(11/33): python3-mako-1.1.4-6.el9.noarch.rpm 898 kB/s | 172 kB 00:00 2024-04-22T18:36:11.793 INFO:teuthology.orchestra.run.smithi121.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 698 kB/s | 35 kB 00:00 2024-04-22T18:36:11.843 INFO:teuthology.orchestra.run.smithi121.stdout:(13/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-22T18:36:11.894 INFO:teuthology.orchestra.run.smithi121.stdout:(14/33): python3-toml-0.10.2-6.el9.noarch.rpm 835 kB/s | 42 kB 00:00 2024-04-22T18:36:12.077 INFO:teuthology.orchestra.run.smithi121.stdout:(15/33): ceph-mon-18.2.2-1240.g6a7528e4.el9.x86 1.2 MB/s | 4.7 MB 00:03 2024-04-22T18:36:12.236 INFO:teuthology.orchestra.run.smithi121.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 127 kB/s | 43 kB 00:00 2024-04-22T18:36:12.461 INFO:teuthology.orchestra.run.smithi121.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 449 kB/s | 172 kB 00:00 2024-04-22T18:36:12.486 INFO:teuthology.orchestra.run.smithi121.stdout:(18/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-04-22T18:36:12.537 INFO:teuthology.orchestra.run.smithi121.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-04-22T18:36:12.562 INFO:teuthology.orchestra.run.smithi121.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-04-22T18:36:12.612 INFO:teuthology.orchestra.run.smithi121.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2024-04-22T18:36:12.637 INFO:teuthology.orchestra.run.smithi121.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-04-22T18:36:12.688 INFO:teuthology.orchestra.run.smithi121.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2024-04-22T18:36:12.713 INFO:teuthology.orchestra.run.smithi121.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 616 kB/s | 46 kB 00:00 2024-04-22T18:36:12.771 INFO:teuthology.orchestra.run.smithi121.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 946 kB/s | 79 kB 00:00 2024-04-22T18:36:12.822 INFO:teuthology.orchestra.run.smithi121.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-22T18:36:12.856 INFO:teuthology.orchestra.run.smithi121.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-04-22T18:36:12.906 INFO:teuthology.orchestra.run.smithi121.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-04-22T18:36:12.931 INFO:teuthology.orchestra.run.smithi121.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2024-04-22T18:36:12.998 INFO:teuthology.orchestra.run.smithi121.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-22T18:36:13.090 INFO:teuthology.orchestra.run.smithi121.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-04-22T18:36:13.115 INFO:teuthology.orchestra.run.smithi121.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.3 MB/s | 427 kB 00:00 2024-04-22T18:36:23.887 INFO:teuthology.orchestra.run.smithi089.stdout:(4/4): ceph-test-18.2.2-1240.g6a7528e4.el9.x86_ 1.8 MB/s | 40 MB 00:22 2024-04-22T18:36:23.889 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:36:23.889 INFO:teuthology.orchestra.run.smithi089.stdout:Total 1.8 MB/s | 40 MB 00:22 2024-04-22T18:36:23.915 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:36:23.935 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:36:23.935 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:36:24.248 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:36:24.249 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:36:24.737 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:36:24.820 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-22T18:36:24.860 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-22T18:36:24.889 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephsqlite-2:18.2.2-1240.g6a7528e4.el9.x86_64 3/4 2024-04-22T18:36:28.543 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1240.g6a7528e4.el9.x86_64 3/4 2024-04-22T18:36:28.634 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-test-2:18.2.2-1240.g6a7528e4.el9.x86_64 4/4 2024-04-22T18:36:30.287 INFO:teuthology.orchestra.run.smithi121.stdout:(33/33): ceph-osd-18.2.2-1240.g6a7528e4.el9.x86 844 kB/s | 17 MB 00:20 2024-04-22T18:36:30.292 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:36:30.292 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.3 MB/s | 29 MB 00:23 2024-04-22T18:36:30.407 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:36:30.445 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:36:30.445 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:36:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-test-2:18.2.2-1240.g6a7528e4.el9.x86_64 4/4 2024-04-22T18:36:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-test-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/4 2024-04-22T18:36:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephsqlite-2:18.2.2-1240.g6a7528e4.el9.x86_64 2/4 2024-04-22T18:36:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-22T18:36:30.725 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:36:30.725 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:36:30.791 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-22T18:36:30.791 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:30.791 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:36:30.791 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:30.791 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:30.791 INFO:teuthology.orchestra.run.smithi089.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T18:36:30.792 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T18:36:30.792 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:30.792 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:36:31.070 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph 2024-04-22T18:36:31.573 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:34 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:36:31.697 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:36:31.699 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:36:31.699 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-04-22T18:36:31.699 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:36:31.699 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:36:31.699 INFO:teuthology.orchestra.run.smithi089.stdout: ceph x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 6.5 k 2024-04-22T18:36:31.700 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-04-22T18:36:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 2.1 M 2024-04-22T18:36:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 1.5 M 2024-04-22T18:36:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 246 k 2024-04-22T18:36:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 4.7 M 2024-04-22T18:36:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 17 M 2024-04-22T18:36:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T18:36:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T18:36:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T18:36:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T18:36:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T18:36:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T18:36:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T18:36:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections 2024-04-22T18:36:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-22T18:36:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T18:36:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T18:36:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T18:36:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T18:36:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T18:36:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T18:36:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T18:36:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T18:36:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T18:36:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T18:36:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T18:36:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T18:36:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T18:36:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T18:36:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T18:36:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T18:36:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T18:36:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T18:36:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T18:36:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:31.702 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:36:31.702 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:36:31.702 INFO:teuthology.orchestra.run.smithi089.stdout:Install 33 Packages 2024-04-22T18:36:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:31.703 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 29 M 2024-04-22T18:36:31.703 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 104 M 2024-04-22T18:36:31.703 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:36:31.891 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:36:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-22T18:36:31.989 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-22T18:36:32.021 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-22T18:36:32.113 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-22T18:36:32.200 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-22T18:36:32.240 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-22T18:36:32.284 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-22T18:36:32.314 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-22T18:36:32.348 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-22T18:36:32.382 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-22T18:36:32.422 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-22T18:36:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-22T18:36:32.492 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-22T18:36:32.530 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-22T18:36:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-22T18:36:32.604 INFO:teuthology.orchestra.run.smithi089.stdout:(1/33): ceph-18.2.2-1240.g6a7528e4.el9.x86_64.r 20 kB/s | 6.5 kB 00:00 2024-04-22T18:36:32.749 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-22T18:36:32.880 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-22T18:36:32.933 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-22T18:36:32.972 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-22T18:36:32.988 INFO:teuthology.orchestra.run.smithi089.stdout:(2/33): ceph-mgr-18.2.2-1240.g6a7528e4.el9.x86_ 2.0 MB/s | 1.5 MB 00:00 2024-04-22T18:36:33.029 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-22T18:36:33.084 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-22T18:36:33.146 INFO:teuthology.orchestra.run.smithi089.stdout:(3/33): ceph-mds-18.2.2-1240.g6a7528e4.el9.x86_ 2.4 MB/s | 2.1 MB 00:00 2024-04-22T18:36:33.230 INFO:teuthology.orchestra.run.smithi089.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1240.g6a75 2.9 MB/s | 246 kB 00:00 2024-04-22T18:36:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-22T18:36:33.426 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-22T18:36:33.504 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-22T18:36:33.555 INFO:teuthology.orchestra.run.smithi089.stdout:(5/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 779 kB/s | 253 kB 00:00 2024-04-22T18:36:33.592 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-22T18:36:33.633 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-22T18:36:33.672 INFO:teuthology.orchestra.run.smithi089.stdout:(6/33): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-04-22T18:36:33.739 INFO:teuthology.orchestra.run.smithi089.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-22T18:36:33.749 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-22T18:36:33.797 INFO:teuthology.orchestra.run.smithi089.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-04-22T18:36:33.864 INFO:teuthology.orchestra.run.smithi089.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 1.8 MB/s | 125 kB 00:00 2024-04-22T18:36:33.900 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1240.g6a7528e4.el9. 28/33 2024-04-22T18:36:33.927 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:18.2.2-1240.g6a7528e4.el9.x86_64 29/33 2024-04-22T18:36:33.931 INFO:teuthology.orchestra.run.smithi089.stdout:(10/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-04-22T18:36:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1240.g6a7528e4.el9.x86_64 29/33 2024-04-22T18:36:33.952 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:36:33.952 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T18:36:33.952 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T18:36:33.952 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T18:36:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:33.990 INFO:teuthology.orchestra.run.smithi089.stdout:(11/33): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-22T18:36:34.049 INFO:teuthology.orchestra.run.smithi089.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 594 kB/s | 35 kB 00:00 2024-04-22T18:36:34.107 INFO:teuthology.orchestra.run.smithi089.stdout:(13/33): python3-pytz-2021.1-5.el9.noarch.rpm 875 kB/s | 51 kB 00:00 2024-04-22T18:36:34.174 INFO:teuthology.orchestra.run.smithi089.stdout:(14/33): python3-toml-0.10.2-6.el9.noarch.rpm 624 kB/s | 42 kB 00:00 2024-04-22T18:36:34.316 INFO:teuthology.orchestra.run.smithi089.stdout:(15/33): ceph-mon-18.2.2-1240.g6a7528e4.el9.x86 2.7 MB/s | 4.7 MB 00:01 2024-04-22T18:36:34.341 INFO:teuthology.orchestra.run.smithi089.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 260 kB/s | 43 kB 00:00 2024-04-22T18:36:34.417 INFO:teuthology.orchestra.run.smithi089.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-04-22T18:36:34.458 INFO:teuthology.orchestra.run.smithi089.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-04-22T18:36:34.509 INFO:teuthology.orchestra.run.smithi089.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-04-22T18:36:34.534 INFO:teuthology.orchestra.run.smithi089.stdout:(20/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-04-22T18:36:34.551 INFO:teuthology.orchestra.run.smithi089.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 554 kB/s | 23 kB 00:00 2024-04-22T18:36:34.577 INFO:teuthology.orchestra.run.smithi089.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 463 kB/s | 19 kB 00:00 2024-04-22T18:36:34.602 INFO:teuthology.orchestra.run.smithi089.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-04-22T18:36:34.627 INFO:teuthology.orchestra.run.smithi089.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-04-22T18:36:34.661 INFO:teuthology.orchestra.run.smithi089.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-04-22T18:36:34.694 INFO:teuthology.orchestra.run.smithi089.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-04-22T18:36:34.720 INFO:teuthology.orchestra.run.smithi089.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 279 kB/s | 16 kB 00:00 2024-04-22T18:36:34.745 INFO:teuthology.orchestra.run.smithi089.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-04-22T18:36:34.762 INFO:teuthology.orchestra.run.smithi089.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 853 kB/s | 36 kB 00:00 2024-04-22T18:36:34.796 INFO:teuthology.orchestra.run.smithi089.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-22T18:36:34.846 INFO:teuthology.orchestra.run.smithi089.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-04-22T18:36:34.896 INFO:teuthology.orchestra.run.smithi089.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-04-22T18:36:35.404 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:18.2.2-1240.g6a7528e4.el9.x86_64 30/33 2024-04-22T18:36:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:18.2.2-1240.g6a7528e4.el9.x86_64 30/33 2024-04-22T18:36:35.438 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:36:35.438 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T18:36:35.439 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T18:36:35.439 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T18:36:35.439 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:35.571 INFO:teuthology.orchestra.run.smithi089.stdout:(33/33): ceph-osd-18.2.2-1240.g6a7528e4.el9.x86 6.4 MB/s | 17 MB 00:02 2024-04-22T18:36:35.575 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:36:35.575 INFO:teuthology.orchestra.run.smithi089.stdout:Total 7.5 MB/s | 29 MB 00:03 2024-04-22T18:36:35.693 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:36:35.731 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:36:35.731 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:36:35.888 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:18.2.2-1240.g6a7528e4.el9.x86_64 31/33 2024-04-22T18:36:35.911 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:18.2.2-1240.g6a7528e4.el9.x86_64 31/33 2024-04-22T18:36:35.911 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:36:35.911 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T18:36:35.911 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T18:36:35.911 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T18:36:35.911 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:36.009 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:36:36.010 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:36:36.105 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:18.2.2-1240.g6a7528e4.el9.x86_64 32/33 2024-04-22T18:36:36.127 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:18.2.2-1240.g6a7528e4.el9.x86_64 32/33 2024-04-22T18:36:36.127 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:36:36.127 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T18:36:36.127 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T18:36:36.127 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T18:36:36.128 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:18.2.2-1240.g6a7528e4.el9.x86_64 33/33 2024-04-22T18:36:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:36:37.241 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-22T18:36:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-22T18:36:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-22T18:36:37.399 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-22T18:36:37.476 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-22T18:36:37.517 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-22T18:36:37.552 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-22T18:36:37.582 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-04-22T18:36:37.616 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-04-22T18:36:37.649 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-04-22T18:36:37.690 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-04-22T18:36:37.726 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-04-22T18:36:37.760 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-04-22T18:36:37.798 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-04-22T18:36:37.822 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-2:18.2.2-1240.g6a7528e4.el9.x86_64 33/33 2024-04-22T18:36:37.822 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/33 2024-04-22T18:36:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:18.2.2-1240.g6a7528e4.el9.x86_64 2/33 2024-04-22T18:36:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:18.2.2-1240.g6a7528e4.el9.x86_64 3/33 2024-04-22T18:36:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:18.2.2-1240.g6a7528e4.el9.x86_64 4/33 2024-04-22T18:36:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:18.2.2-1240.g6a7528e4.el9.x86_64 5/33 2024-04-22T18:36:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1240.g6a7528e4.el9. 6/33 2024-04-22T18:36:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-22T18:36:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-22T18:36:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-22T18:36:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-22T18:36:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-22T18:36:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-22T18:36:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-22T18:36:37.823 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-22T18:36:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-22T18:36:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-22T18:36:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-22T18:36:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-22T18:36:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-22T18:36:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-22T18:36:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-22T18:36:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-22T18:36:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-22T18:36:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-22T18:36:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-22T18:36:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-22T18:36:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-22T18:36:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-22T18:36:37.824 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-22T18:36:37.825 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-22T18:36:37.825 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-22T18:36:37.825 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-22T18:36:37.870 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-04-22T18:36:38.025 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-04-22T18:36:38.105 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-04-22T18:36:38.159 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-04-22T18:36:38.197 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-04-22T18:36:38.254 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-04-22T18:36:38.310 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-04-22T18:36:38.497 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-04-22T18:36:38.543 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-04-22T18:36:38.630 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-04-22T18:36:38.727 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-04-22T18:36:38.767 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-04-22T18:36:38.812 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-22T18:36:38.812 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:38.812 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:36:38.812 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:38.812 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:38.812 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:38.813 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:36:38.813 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:38.813 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:38.813 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T18:36:38.813 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T18:36:38.813 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T18:36:38.813 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T18:36:38.813 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T18:36:38.813 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T18:36:38.813 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T18:36:38.813 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T18:36:38.813 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T18:36:38.813 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T18:36:38.813 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T18:36:38.813 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T18:36:38.814 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T18:36:38.814 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T18:36:38.814 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T18:36:38.814 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T18:36:38.814 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T18:36:38.814 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T18:36:38.814 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T18:36:38.814 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T18:36:38.814 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T18:36:38.814 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T18:36:38.814 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T18:36:38.814 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T18:36:38.814 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T18:36:38.815 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T18:36:38.815 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T18:36:38.815 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:38.815 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:36:38.882 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-04-22T18:36:39.034 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1240.g6a7528e4.el9. 28/33 2024-04-22T18:36:39.061 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-2:18.2.2-1240.g6a7528e4.el9.x86_64 29/33 2024-04-22T18:36:39.086 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1240.g6a7528e4.el9.x86_64 29/33 2024-04-22T18:36:39.086 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:36:39.086 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T18:36:39.086 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T18:36:39.086 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T18:36:39.086 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:39.207 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base 2024-04-22T18:36:39.721 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:51 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:36:39.800 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-base-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:36:39.840 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:36:39.841 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-22T18:36:39.841 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:36:39.912 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install cephadm 2024-04-22T18:36:40.419 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:52 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:36:40.504 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-osd-2:18.2.2-1240.g6a7528e4.el9.x86_64 30/33 2024-04-22T18:36:40.531 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-osd-2:18.2.2-1240.g6a7528e4.el9.x86_64 30/33 2024-04-22T18:36:40.531 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:36:40.531 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T18:36:40.531 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T18:36:40.531 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T18:36:40.531 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:40.538 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:36:40.539 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:36:40.539 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-22T18:36:40.539 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:36:40.539 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:36:40.539 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 224 k 2024-04-22T18:36:40.539 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:40.539 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:36:40.539 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:36:40.539 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-22T18:36:40.539 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:40.540 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 224 k 2024-04-22T18:36:40.540 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 216 k 2024-04-22T18:36:40.540 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:36:40.845 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm-18.2.2-1240.g6a7528e4.el9.noarch.rpm 735 kB/s | 224 kB 00:00 2024-04-22T18:36:40.845 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:36:40.846 INFO:teuthology.orchestra.run.smithi121.stdout:Total 731 kB/s | 224 kB 00:00 2024-04-22T18:36:40.846 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:36:40.851 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:36:40.851 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:36:40.854 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:36:40.854 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:36:40.922 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:36:40.971 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mon-2:18.2.2-1240.g6a7528e4.el9.x86_64 31/33 2024-04-22T18:36:40.994 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mon-2:18.2.2-1240.g6a7528e4.el9.x86_64 31/33 2024-04-22T18:36:40.995 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:36:40.995 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T18:36:40.995 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T18:36:40.995 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T18:36:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mds-2:18.2.2-1240.g6a7528e4.el9.x86_64 32/33 2024-04-22T18:36:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mds-2:18.2.2-1240.g6a7528e4.el9.x86_64 32/33 2024-04-22T18:36:41.220 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:36:41.220 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T18:36:41.220 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T18:36:41.220 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T18:36:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:41.284 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-2:18.2.2-1240.g6a7528e4.el9.x86_64 33/33 2024-04-22T18:36:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 1/1 2024-04-22T18:36:41.398 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 1/1 2024-04-22T18:36:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 1/1 2024-04-22T18:36:41.968 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 1/1 2024-04-22T18:36:41.968 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:41.968 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:36:41.968 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:36:41.968 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:41.968 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:36:42.184 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T18:36:42.698 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:54 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:36:42.818 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:36:42.819 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:36:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-04-22T18:36:42.819 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:36:42.819 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:36:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 143 k 2024-04-22T18:36:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:42.819 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:36:42.819 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:36:42.819 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-22T18:36:42.819 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:42.819 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 143 k 2024-04-22T18:36:42.820 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 395 k 2024-04-22T18:36:42.820 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:36:42.828 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-2:18.2.2-1240.g6a7528e4.el9.x86_64 33/33 2024-04-22T18:36:42.828 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/33 2024-04-22T18:36:42.828 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mds-2:18.2.2-1240.g6a7528e4.el9.x86_64 2/33 2024-04-22T18:36:42.828 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-2:18.2.2-1240.g6a7528e4.el9.x86_64 3/33 2024-04-22T18:36:42.828 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mon-2:18.2.2-1240.g6a7528e4.el9.x86_64 4/33 2024-04-22T18:36:42.828 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-osd-2:18.2.2-1240.g6a7528e4.el9.x86_64 5/33 2024-04-22T18:36:42.828 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1240.g6a7528e4.el9. 6/33 2024-04-22T18:36:42.828 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-04-22T18:36:42.828 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-04-22T18:36:42.828 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-04-22T18:36:42.828 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-04-22T18:36:42.829 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-04-22T18:36:42.829 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-04-22T18:36:42.829 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-04-22T18:36:42.829 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-04-22T18:36:42.829 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-04-22T18:36:42.829 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-22T18:36:42.829 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-22T18:36:42.829 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-22T18:36:42.829 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-22T18:36:42.829 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-22T18:36:42.829 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-22T18:36:42.829 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-22T18:36:42.829 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-22T18:36:42.830 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-22T18:36:42.830 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-22T18:36:42.830 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-22T18:36:42.830 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-22T18:36:42.830 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-22T18:36:42.830 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-22T18:36:42.830 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-22T18:36:42.830 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-22T18:36:42.830 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-22T18:36:43.618 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-immutable-object-cache-18.2.2-1240.g6a7528 179 kB/s | 143 kB 00:00 2024-04-22T18:36:43.618 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:36:43.619 INFO:teuthology.orchestra.run.smithi121.stdout:Total 179 kB/s | 143 kB 00:00 2024-04-22T18:36:43.619 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:36:43.626 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:36:43.626 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:36:43.672 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-22T18:36:43.672 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:43.672 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:36:43.672 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:43.672 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:43.672 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:43.672 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:36:43.672 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:43.673 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:43.673 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T18:36:43.673 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T18:36:43.673 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T18:36:43.673 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T18:36:43.673 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T18:36:43.673 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T18:36:43.673 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T18:36:43.673 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T18:36:43.673 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T18:36:43.673 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T18:36:43.673 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T18:36:43.673 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T18:36:43.674 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T18:36:43.674 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T18:36:43.674 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T18:36:43.674 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T18:36:43.674 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T18:36:43.674 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T18:36:43.674 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T18:36:43.674 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T18:36:43.674 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T18:36:43.674 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T18:36:43.674 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T18:36:43.675 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T18:36:43.675 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T18:36:43.675 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T18:36:43.675 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T18:36:43.675 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:43.675 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:36:43.682 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:36:43.683 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:36:43.840 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:36:43.927 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1240.g6a7528e4. 1/1 2024-04-22T18:36:43.950 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1240.g6a7528e4. 1/1 2024-04-22T18:36:43.950 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:36:43.950 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T18:36:43.950 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:44.013 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-base 2024-04-22T18:36:44.531 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:47 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:36:44.609 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-base-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:36:44.649 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:36:44.650 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-04-22T18:36:44.650 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:36:44.724 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install cephadm 2024-04-22T18:36:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1240.g6a7528e4. 1/1 2024-04-22T18:36:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:44.803 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:36:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:44.803 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:36:45.003 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr 2024-04-22T18:36:45.230 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:48 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:36:45.349 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:36:45.350 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:36:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-04-22T18:36:45.350 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:36:45.350 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:36:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 224 k 2024-04-22T18:36:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:45.351 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:36:45.351 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:36:45.351 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-04-22T18:36:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:45.351 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 224 k 2024-04-22T18:36:45.351 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 216 k 2024-04-22T18:36:45.351 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:36:45.523 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:57 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:36:45.605 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-mgr-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:36:45.650 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:36:45.651 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-22T18:36:45.651 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:36:45.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T18:36:45.904 INFO:teuthology.orchestra.run.smithi089.stdout:cephadm-18.2.2-1240.g6a7528e4.el9.noarch.rpm 405 kB/s | 224 kB 00:00 2024-04-22T18:36:45.905 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:36:45.905 INFO:teuthology.orchestra.run.smithi089.stdout:Total 404 kB/s | 224 kB 00:00 2024-04-22T18:36:45.905 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:36:45.910 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:36:45.910 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:36:45.913 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:36:45.913 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:36:45.983 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:36:46.247 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:58 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:36:46.367 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:36:46.368 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:36:46.368 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-22T18:36:46.368 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:36:46.368 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:36:46.368 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 3.5 M 2024-04-22T18:36:46.368 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-22T18:36:46.368 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 24 k 2024-04-22T18:36:46.368 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 15 k 2024-04-22T18:36:46.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T18:36:46.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T18:36:46.368 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:46.369 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:36:46.369 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:36:46.369 INFO:teuthology.orchestra.run.smithi121.stdout:Install 5 Packages 2024-04-22T18:36:46.369 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:46.369 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.8 M 2024-04-22T18:36:46.370 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 87 M 2024-04-22T18:36:46.370 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:36:46.382 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 1/1 2024-04-22T18:36:46.465 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 1/1 2024-04-22T18:36:46.824 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 1/1 2024-04-22T18:36:46.846 INFO:teuthology.orchestra.run.smithi121.stdout:(1/5): ceph-prometheus-alerts-18.2.2-1240.g6a75 57 kB/s | 15 kB 00:00 2024-04-22T18:36:46.996 INFO:teuthology.orchestra.run.smithi121.stdout:(2/5): ceph-grafana-dashboards-18.2.2-1240.g6a7 58 kB/s | 24 kB 00:00 2024-04-22T18:36:47.045 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 1/1 2024-04-22T18:36:47.045 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:47.045 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:36:47.045 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:36:47.045 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:47.045 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:36:47.122 INFO:teuthology.orchestra.run.smithi121.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 112 kB/s | 31 kB 00:00 2024-04-22T18:36:47.249 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T18:36:47.380 INFO:teuthology.orchestra.run.smithi121.stdout:(4/5): ceph-mgr-dashboard-18.2.2-1240.g6a7528e4 4.4 MB/s | 3.5 MB 00:00 2024-04-22T18:36:47.414 INFO:teuthology.orchestra.run.smithi121.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 452 kB/s | 188 kB 00:00 2024-04-22T18:36:47.416 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:36:47.416 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.6 MB/s | 3.8 MB 00:01 2024-04-22T18:36:47.417 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:36:47.424 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:36:47.424 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:36:47.483 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:36:47.483 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:36:47.729 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:36:47.764 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:50 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:36:47.803 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-04-22T18:36:47.835 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-04-22T18:36:47.864 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1240.g6a7528e4.el9.n 3/5 2024-04-22T18:36:47.885 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:36:47.886 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:36:47.887 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-04-22T18:36:47.887 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:36:47.887 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:36:47.887 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 143 k 2024-04-22T18:36:47.887 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:47.887 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:36:47.887 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:36:47.887 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-04-22T18:36:47.887 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:47.887 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 143 k 2024-04-22T18:36:47.887 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 395 k 2024-04-22T18:36:47.887 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:36:48.110 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-immutable-object-cache-18.2.2-1240.g6a7528 641 kB/s | 143 kB 00:00 2024-04-22T18:36:48.111 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:36:48.111 INFO:teuthology.orchestra.run.smithi089.stdout:Total 637 kB/s | 143 kB 00:00 2024-04-22T18:36:48.111 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:36:48.119 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:36:48.119 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:36:48.173 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:36:48.174 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:36:48.320 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:36:48.403 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1240.g6a7528e4. 1/1 2024-04-22T18:36:48.427 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1240.g6a7528e4. 1/1 2024-04-22T18:36:48.427 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:36:48.427 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T18:36:48.427 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:48.909 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1240.g6a7528e4.el9. 4/5 2024-04-22T18:36:48.951 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1240.g6a7528e4.el9.noarc 5/5 2024-04-22T18:36:49.280 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1240.g6a7528e4. 1/1 2024-04-22T18:36:49.280 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:49.280 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:36:49.280 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:36:49.281 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:49.281 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:36:49.471 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr 2024-04-22T18:36:49.983 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:52 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:36:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1240.g6a7528e4.el9.noarc 5/5 2024-04-22T18:36:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1240.g6a7528e4.el9. 1/5 2024-04-22T18:36:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1240.g6a7528e4.el9.noarc 2/5 2024-04-22T18:36:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1240.g6a7528e4.el9.n 3/5 2024-04-22T18:36:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-04-22T18:36:50.063 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-mgr-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:36:50.107 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:36:50.108 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-04-22T18:36:50.108 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:36:50.211 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T18:36:50.282 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-04-22T18:36:50.282 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:50.282 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:36:50.282 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:36:50.283 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:36:50.283 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:36:50.283 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T18:36:50.283 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T18:36:50.283 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:50.283 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:36:50.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T18:36:50.720 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:53 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:36:50.842 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:36:50.842 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:36:50.842 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-04-22T18:36:50.842 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:36:50.842 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:36:50.842 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 3.5 M 2024-04-22T18:36:50.843 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-04-22T18:36:50.843 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 24 k 2024-04-22T18:36:50.843 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 15 k 2024-04-22T18:36:50.843 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T18:36:50.843 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T18:36:50.843 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:50.843 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:36:50.843 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:36:50.843 INFO:teuthology.orchestra.run.smithi089.stdout:Install 5 Packages 2024-04-22T18:36:50.843 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:50.844 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 3.8 M 2024-04-22T18:36:50.844 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 87 M 2024-04-22T18:36:50.844 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:36:50.996 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:02 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:36:51.118 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:36:51.119 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-04-22T18:36:51.119 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-22T18:36:51.119 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-04-22T18:36:51.119 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:36:51.119 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 7.4 M 2024-04-22T18:36:51.119 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-22T18:36:51.119 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T18:36:51.120 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T18:36:51.120 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T18:36:51.120 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T18:36:51.120 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T18:36:51.120 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T18:36:51.120 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T18:36:51.120 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T18:36:51.120 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T18:36:51.120 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T18:36:51.120 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T18:36:51.120 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:51.120 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:36:51.121 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-04-22T18:36:51.121 INFO:teuthology.orchestra.run.smithi121.stdout:Install 12 Packages 2024-04-22T18:36:51.121 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:36:51.121 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 38 M 2024-04-22T18:36:51.121 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 209 M 2024-04-22T18:36:51.121 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:36:51.206 INFO:teuthology.orchestra.run.smithi089.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1240.g6a7 87 kB/s | 24 kB 00:00 2024-04-22T18:36:51.432 INFO:teuthology.orchestra.run.smithi089.stdout:(2/5): ceph-mgr-dashboard-18.2.2-1240.g6a7528e4 7.0 MB/s | 3.5 MB 00:00 2024-04-22T18:36:51.473 INFO:teuthology.orchestra.run.smithi089.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 115 kB/s | 31 kB 00:00 2024-04-22T18:36:51.532 INFO:teuthology.orchestra.run.smithi089.stdout:(4/5): ceph-prometheus-alerts-18.2.2-1240.g6a75 25 kB/s | 15 kB 00:00 2024-04-22T18:36:51.807 INFO:teuthology.orchestra.run.smithi089.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 502 kB/s | 188 kB 00:00 2024-04-22T18:36:51.811 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:36:51.811 INFO:teuthology.orchestra.run.smithi089.stdout:Total 3.9 MB/s | 3.8 MB 00:00 2024-04-22T18:36:51.811 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:36:51.818 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:36:51.818 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:36:51.877 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:36:51.877 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:36:51.879 INFO:teuthology.orchestra.run.smithi121.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-04-22T18:36:51.912 INFO:teuthology.orchestra.run.smithi121.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.4 MB/s | 803 kB 00:00 2024-04-22T18:36:52.116 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:36:52.121 INFO:teuthology.orchestra.run.smithi121.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 135 kB/s | 33 kB 00:00 2024-04-22T18:36:52.205 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-04-22T18:36:52.237 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-04-22T18:36:52.265 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1240.g6a7528e4.el9.n 3/5 2024-04-22T18:36:52.321 INFO:teuthology.orchestra.run.smithi121.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 91 kB/s | 18 kB 00:00 2024-04-22T18:36:52.563 INFO:teuthology.orchestra.run.smithi121.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 146 kB/s | 35 kB 00:00 2024-04-22T18:36:52.813 INFO:teuthology.orchestra.run.smithi121.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.3 MB/s | 3.0 MB 00:00 2024-04-22T18:36:52.964 INFO:teuthology.orchestra.run.smithi121.stdout:(7/12): ceph-mgr-diskprediction-local-18.2.2-12 5.8 MB/s | 7.4 MB 00:01 2024-04-22T18:36:53.189 INFO:teuthology.orchestra.run.smithi121.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 651 kB/s | 244 kB 00:00 2024-04-22T18:36:53.324 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1240.g6a7528e4.el9. 4/5 2024-04-22T18:36:53.361 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1240.g6a7528e4.el9.noarc 5/5 2024-04-22T18:36:53.614 INFO:teuthology.orchestra.run.smithi121.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 590 kB/s | 250 kB 00:00 2024-04-22T18:36:53.764 INFO:teuthology.orchestra.run.smithi121.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 3.9 MB/s | 4.7 MB 00:01 2024-04-22T18:36:54.139 INFO:teuthology.orchestra.run.smithi121.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.3 MB/s | 5.1 MB 00:01 2024-04-22T18:36:54.495 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1240.g6a7528e4.el9.noarc 5/5 2024-04-22T18:36:54.495 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1240.g6a7528e4.el9. 1/5 2024-04-22T18:36:54.496 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1240.g6a7528e4.el9.noarc 2/5 2024-04-22T18:36:54.496 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1240.g6a7528e4.el9.n 3/5 2024-04-22T18:36:54.496 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-04-22T18:36:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-04-22T18:36:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:54.730 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:36:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:36:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:36:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:36:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T18:36:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T18:36:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:54.730 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:36:54.953 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T18:36:55.454 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:58 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:36:55.575 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:36:55.576 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================= 2024-04-22T18:36:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-04-22T18:36:55.576 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================= 2024-04-22T18:36:55.576 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:36:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 7.4 M 2024-04-22T18:36:55.577 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-04-22T18:36:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T18:36:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T18:36:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T18:36:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T18:36:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T18:36:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T18:36:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T18:36:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T18:36:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T18:36:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T18:36:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T18:36:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:36:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================= 2024-04-22T18:36:55.578 INFO:teuthology.orchestra.run.smithi089.stdout:Install 12 Packages 2024-04-22T18:36:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:36:55.579 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 38 M 2024-04-22T18:36:55.579 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 209 M 2024-04-22T18:36:55.579 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:36:55.889 INFO:teuthology.orchestra.run.smithi121.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.1 MB/s | 16 MB 00:02 2024-04-22T18:36:55.891 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:36:55.891 INFO:teuthology.orchestra.run.smithi121.stdout:Total 8.0 MB/s | 38 MB 00:04 2024-04-22T18:36:56.198 INFO:teuthology.orchestra.run.smithi089.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 739 kB/s | 195 kB 00:00 2024-04-22T18:36:56.239 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:36:56.239 INFO:teuthology.orchestra.run.smithi089.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-04-22T18:36:56.266 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:36:56.266 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:36:56.323 INFO:teuthology.orchestra.run.smithi089.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-04-22T18:36:56.373 INFO:teuthology.orchestra.run.smithi089.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 366 kB/s | 18 kB 00:00 2024-04-22T18:36:56.473 INFO:teuthology.orchestra.run.smithi089.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 352 kB/s | 35 kB 00:00 2024-04-22T18:36:56.672 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:36:56.672 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:36:57.141 INFO:teuthology.orchestra.run.smithi089.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.3 MB/s | 3.0 MB 00:00 2024-04-22T18:36:57.233 INFO:teuthology.orchestra.run.smithi089.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.6 MB/s | 244 kB 00:00 2024-04-22T18:36:57.425 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:36:57.494 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T18:36:57.538 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T18:36:57.638 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T18:36:57.680 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T18:36:57.858 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T18:36:57.887 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T18:36:57.958 INFO:teuthology.orchestra.run.smithi089.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 3.2 MB/s | 4.7 MB 00:01 2024-04-22T18:36:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T18:36:58.066 INFO:teuthology.orchestra.run.smithi089.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-04-22T18:36:58.517 INFO:teuthology.orchestra.run.smithi089.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.0 MB/s | 5.1 MB 00:01 2024-04-22T18:36:58.613 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T18:36:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T18:36:58.925 INFO:teuthology.orchestra.run.smithi089.stdout:(11/12): ceph-mgr-diskprediction-local-18.2.2-1 2.5 MB/s | 7.4 MB 00:02 2024-04-22T18:36:59.967 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T18:37:01.014 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T18:37:01.045 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1240.g6a752 12/12 2024-04-22T18:37:02.242 INFO:teuthology.orchestra.run.smithi089.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.9 MB/s | 16 MB 00:04 2024-04-22T18:37:02.245 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:02.245 INFO:teuthology.orchestra.run.smithi089.stdout:Total 5.7 MB/s | 38 MB 00:06 2024-04-22T18:37:02.600 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:37:02.626 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:37:02.626 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:37:03.031 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:37:03.032 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:37:03.470 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1240.g6a752 12/12 2024-04-22T18:37:03.470 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1240.g6a752 1/12 2024-04-22T18:37:03.470 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T18:37:03.470 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T18:37:03.470 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T18:37:03.470 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T18:37:03.470 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T18:37:03.470 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T18:37:03.471 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T18:37:03.471 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T18:37:03.471 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T18:37:03.471 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T18:37:03.794 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:37:03.854 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T18:37:03.876 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T18:37:03.876 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:03.877 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:37:03.877 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:37:03.877 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T18:37:03.877 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T18:37:03.877 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T18:37:03.877 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T18:37:03.877 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T18:37:03.877 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T18:37:03.877 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T18:37:03.877 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T18:37:03.877 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T18:37:03.877 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T18:37:03.877 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T18:37:03.877 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:03.878 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:03.898 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T18:37:03.999 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T18:37:04.041 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T18:37:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T18:37:04.240 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T18:37:04.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-rook 2024-04-22T18:37:04.308 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T18:37:04.766 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:16 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:04.890 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:04.891 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:04.891 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-22T18:37:04.891 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:04.891 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:37:04.891 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 49 k 2024-04-22T18:37:04.891 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-22T18:37:04.891 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T18:37:04.891 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T18:37:04.891 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T18:37:04.891 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T18:37:04.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T18:37:04.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T18:37:04.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T18:37:04.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T18:37:04.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T18:37:04.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T18:37:04.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T18:37:04.892 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T18:37:04.892 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:04.892 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:37:04.892 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:04.892 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2024-04-22T18:37:04.892 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:04.893 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 2.2 M 2024-04-22T18:37:04.893 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 26 M 2024-04-22T18:37:04.893 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:37:04.950 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T18:37:05.087 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T18:37:05.162 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 267 kB/s | 19 kB 00:00 2024-04-22T18:37:05.212 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 210 kB/s | 26 kB 00:00 2024-04-22T18:37:05.296 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-04-22T18:37:05.321 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-04-22T18:37:05.354 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.7 MB/s | 279 kB 00:00 2024-04-22T18:37:05.380 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 916 kB/s | 54 kB 00:00 2024-04-22T18:37:05.522 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 106 kB/s | 15 kB 00:00 2024-04-22T18:37:05.547 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): ceph-mgr-rook-18.2.2-1240.g6a7528e4.el9 108 kB/s | 49 kB 00:00 2024-04-22T18:37:05.572 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 148 kB/s | 32 kB 00:00 2024-04-22T18:37:05.664 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.4 MB/s | 203 kB 00:00 2024-04-22T18:37:05.723 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-04-22T18:37:05.748 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 337 kB/s | 59 kB 00:00 2024-04-22T18:37:05.773 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 829 kB/s | 90 kB 00:00 2024-04-22T18:37:05.776 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:05.776 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-04-22T18:37:05.837 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:37:05.852 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:37:05.852 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:37:05.935 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:37:05.935 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:37:06.303 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T18:37:06.507 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:37:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T18:37:06.733 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T18:37:06.788 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T18:37:06.851 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T18:37:06.892 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T18:37:06.984 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T18:37:07.094 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T18:37:07.143 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T18:37:07.332 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T18:37:07.364 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1240.g6a752 12/12 2024-04-22T18:37:07.852 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T18:37:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T18:37:07.933 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T18:37:07.981 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T18:37:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:18.2.2-1240.g6a7528e4.el9.noarch 13/13 2024-04-22T18:37:08.708 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1240.g6a7528e4.el9.noarch 13/13 2024-04-22T18:37:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1240.g6a7528e4.el9.noarch 1/13 2024-04-22T18:37:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T18:37:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T18:37:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T18:37:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T18:37:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T18:37:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T18:37:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T18:37:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T18:37:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T18:37:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T18:37:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T18:37:09.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T18:37:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:09.014 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:37:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:37:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T18:37:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T18:37:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T18:37:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T18:37:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T18:37:09.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T18:37:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T18:37:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T18:37:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T18:37:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T18:37:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T18:37:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T18:37:09.015 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:09.015 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:09.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T18:37:09.733 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:21 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:09.851 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:09.852 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:09.852 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-22T18:37:09.852 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:09.852 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:37:09.852 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 138 k 2024-04-22T18:37:09.852 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-22T18:37:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T18:37:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T18:37:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T18:37:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T18:37:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T18:37:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:09.853 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:37:09.853 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:09.853 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-04-22T18:37:09.853 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:09.854 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 7.0 M 2024-04-22T18:37:09.854 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 33 M 2024-04-22T18:37:09.854 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:37:09.872 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1240.g6a752 12/12 2024-04-22T18:37:09.872 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1240.g6a752 1/12 2024-04-22T18:37:09.872 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T18:37:09.872 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T18:37:09.872 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T18:37:09.872 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T18:37:09.872 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T18:37:09.872 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T18:37:09.872 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T18:37:09.872 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T18:37:09.872 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T18:37:09.872 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T18:37:10.238 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T18:37:10.238 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:10.238 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:37:10.238 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:37:10.238 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T18:37:10.239 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T18:37:10.239 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T18:37:10.239 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T18:37:10.239 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T18:37:10.239 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T18:37:10.239 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T18:37:10.239 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T18:37:10.239 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T18:37:10.239 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T18:37:10.239 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T18:37:10.239 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:10.239 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:10.558 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 950 kB/s | 248 kB 00:00 2024-04-22T18:37:10.620 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-rook 2024-04-22T18:37:10.742 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-04-22T18:37:10.884 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): ceph-mgr-cephadm-18.2.2-1240.g6a7528e4.e 236 kB/s | 138 kB 00:00 2024-04-22T18:37:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 192 kB/s | 58 kB 00:00 2024-04-22T18:37:11.092 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-04-22T18:37:11.145 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:14 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:11.192 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 165 kB/s | 51 kB 00:00 2024-04-22T18:37:11.195 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:11.195 INFO:teuthology.orchestra.run.smithi121.stdout:Total 5.2 MB/s | 7.0 MB 00:01 2024-04-22T18:37:11.266 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:37:11.267 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:11.269 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:11.269 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-04-22T18:37:11.269 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:11.269 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:37:11.269 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 49 k 2024-04-22T18:37:11.269 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-04-22T18:37:11.269 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T18:37:11.269 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T18:37:11.269 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T18:37:11.270 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T18:37:11.270 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T18:37:11.270 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T18:37:11.270 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T18:37:11.270 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T18:37:11.270 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T18:37:11.270 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T18:37:11.270 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T18:37:11.270 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T18:37:11.270 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:11.270 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:37:11.271 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:11.271 INFO:teuthology.orchestra.run.smithi089.stdout:Install 13 Packages 2024-04-22T18:37:11.271 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:11.271 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 2.2 M 2024-04-22T18:37:11.271 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 26 M 2024-04-22T18:37:11.271 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:37:11.277 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:37:11.277 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:37:11.372 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:37:11.373 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:37:11.676 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:37:11.844 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T18:37:11.866 INFO:teuthology.orchestra.run.smithi089.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 218 kB/s | 19 kB 00:00 2024-04-22T18:37:11.894 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T18:37:11.916 INFO:teuthology.orchestra.run.smithi089.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 186 kB/s | 26 kB 00:00 2024-04-22T18:37:12.008 INFO:teuthology.orchestra.run.smithi089.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.8 MB/s | 222 kB 00:00 2024-04-22T18:37:12.033 INFO:teuthology.orchestra.run.smithi089.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-04-22T18:37:12.075 INFO:teuthology.orchestra.run.smithi089.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 279 kB 00:00 2024-04-22T18:37:12.100 INFO:teuthology.orchestra.run.smithi089.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 800 kB/s | 54 kB 00:00 2024-04-22T18:37:12.151 INFO:teuthology.orchestra.run.smithi089.stdout:(7/13): ceph-mgr-rook-18.2.2-1240.g6a7528e4.el9 132 kB/s | 49 kB 00:00 2024-04-22T18:37:12.176 INFO:teuthology.orchestra.run.smithi089.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 198 kB/s | 15 kB 00:00 2024-04-22T18:37:12.201 INFO:teuthology.orchestra.run.smithi089.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 255 kB/s | 32 kB 00:00 2024-04-22T18:37:12.268 INFO:teuthology.orchestra.run.smithi089.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.7 MB/s | 203 kB 00:00 2024-04-22T18:37:12.294 INFO:teuthology.orchestra.run.smithi089.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 643 kB/s | 59 kB 00:00 2024-04-22T18:37:12.344 INFO:teuthology.orchestra.run.smithi089.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-04-22T18:37:12.426 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T18:37:12.462 INFO:teuthology.orchestra.run.smithi089.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2024-04-22T18:37:12.462 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:12.462 INFO:teuthology.orchestra.run.smithi089.stdout:Total 1.8 MB/s | 2.2 MB 00:01 2024-04-22T18:37:12.515 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T18:37:12.526 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:37:12.540 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:37:12.540 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:37:12.576 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T18:37:12.598 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 6/6 2024-04-22T18:37:12.624 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:37:12.624 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:37:13.217 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 6/6 2024-04-22T18:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 1/6 2024-04-22T18:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T18:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T18:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T18:37:13.318 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T18:37:13.384 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T18:37:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T18:37:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T18:37:13.538 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T18:37:13.560 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T18:37:13.560 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:13.560 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:37:13.561 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:37:13.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T18:37:13.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T18:37:13.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T18:37:13.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T18:37:13.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T18:37:13.561 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T18:37:13.675 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T18:37:13.785 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T18:37:13.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-fuse 2024-04-22T18:37:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T18:37:14.311 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:26 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:14.431 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:14.432 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:14.432 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-22T18:37:14.432 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:14.432 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:37:14.432 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 848 k 2024-04-22T18:37:14.432 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-22T18:37:14.432 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T18:37:14.433 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:14.433 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:37:14.433 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:14.433 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-04-22T18:37:14.433 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:14.433 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 929 k 2024-04-22T18:37:14.433 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 2.7 M 2024-04-22T18:37:14.433 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:37:14.551 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T18:37:14.604 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T18:37:14.638 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T18:37:14.686 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T18:37:14.708 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-rook-2:18.2.2-1240.g6a7528e4.el9.noarch 13/13 2024-04-22T18:37:14.852 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 510 kB/s | 80 kB 00:00 2024-04-22T18:37:15.501 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1240.g6a7528e4.el9.noarch 13/13 2024-04-22T18:37:15.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1240.g6a7528e4.el9.noarch 1/13 2024-04-22T18:37:15.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T18:37:15.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T18:37:15.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T18:37:15.501 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T18:37:15.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T18:37:15.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T18:37:15.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T18:37:15.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T18:37:15.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T18:37:15.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T18:37:15.502 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T18:37:15.502 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-fuse-18.2.2-1240.g6a7528e4.el9.x86_ 1.0 MB/s | 848 kB 00:00 2024-04-22T18:37:15.503 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:15.504 INFO:teuthology.orchestra.run.smithi121.stdout:Total 867 kB/s | 929 kB 00:01 2024-04-22T18:37:15.515 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:37:15.536 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:37:15.536 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:37:15.612 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:37:15.612 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:37:15.794 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:37:15.802 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T18:37:15.802 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:15.802 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:37:15.802 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:37:15.802 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T18:37:15.803 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T18:37:15.803 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T18:37:15.803 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T18:37:15.803 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T18:37:15.803 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T18:37:15.803 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T18:37:15.803 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T18:37:15.803 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T18:37:15.803 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T18:37:15.803 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T18:37:15.803 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T18:37:15.803 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:15.803 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:15.935 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T18:37:15.999 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 2/2 2024-04-22T18:37:16.053 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T18:37:16.560 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:19 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:16.673 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 2/2 2024-04-22T18:37:16.674 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/2 2024-04-22T18:37:16.679 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:16.679 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:16.680 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-04-22T18:37:16.680 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:16.680 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:37:16.680 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 138 k 2024-04-22T18:37:16.680 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-04-22T18:37:16.680 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T18:37:16.680 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T18:37:16.680 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T18:37:16.680 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T18:37:16.680 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T18:37:16.680 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:16.680 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:37:16.680 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:16.680 INFO:teuthology.orchestra.run.smithi089.stdout:Install 6 Packages 2024-04-22T18:37:16.681 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:16.681 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 7.0 M 2024-04-22T18:37:16.681 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 33 M 2024-04-22T18:37:16.681 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:37:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T18:37:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:16.920 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:37:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T18:37:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:16.921 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:17.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-volume 2024-04-22T18:37:17.306 INFO:teuthology.orchestra.run.smithi089.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-04-22T18:37:17.473 INFO:teuthology.orchestra.run.smithi089.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-04-22T18:37:17.656 INFO:teuthology.orchestra.run.smithi089.stdout:(3/6): ceph-mgr-cephadm-18.2.2-1240.g6a7528e4.e 272 kB/s | 138 kB 00:00 2024-04-22T18:37:17.713 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:29 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:17.756 INFO:teuthology.orchestra.run.smithi089.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 204 kB/s | 58 kB 00:00 2024-04-22T18:37:17.823 INFO:teuthology.orchestra.run.smithi089.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-04-22T18:37:17.837 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:17.838 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:17.838 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-22T18:37:17.838 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:17.838 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:37:17.838 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 264 k 2024-04-22T18:37:17.838 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-22T18:37:17.839 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T18:37:17.839 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:17.839 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:37:17.839 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:17.839 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-04-22T18:37:17.839 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:17.839 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 341 k 2024-04-22T18:37:17.840 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 1.5 M 2024-04-22T18:37:17.840 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:37:17.940 INFO:teuthology.orchestra.run.smithi089.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 180 kB/s | 51 kB 00:00 2024-04-22T18:37:17.943 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:17.943 INFO:teuthology.orchestra.run.smithi089.stdout:Total 5.5 MB/s | 7.0 MB 00:01 2024-04-22T18:37:18.013 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:37:18.024 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:37:18.025 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:37:18.119 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:37:18.119 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:37:18.266 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 555 kB/s | 77 kB 00:00 2024-04-22T18:37:18.414 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:37:18.483 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-volume-18.2.2-1240.g6a7528e4.el9.no 741 kB/s | 264 kB 00:00 2024-04-22T18:37:18.483 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:18.484 INFO:teuthology.orchestra.run.smithi121.stdout:Total 529 kB/s | 341 kB 00:00 2024-04-22T18:37:18.495 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:37:18.505 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:37:18.505 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:37:18.530 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:37:18.530 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:37:18.597 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T18:37:18.659 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T18:37:18.672 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:37:18.851 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-22T18:37:18.885 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:18.2.2-1240.g6a7528e4.el9.noarch 2/2 2024-04-22T18:37:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:18.2.2-1240.g6a7528e4.el9.noarch 2/2 2024-04-22T18:37:18.901 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:37:18.901 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T18:37:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:19.193 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T18:37:19.279 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T18:37:19.348 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T18:37:19.371 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 6/6 2024-04-22T18:37:19.528 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:18.2.2-1240.g6a7528e4.el9.noarch 1/2 2024-04-22T18:37:19.828 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-22T18:37:19.828 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:19.828 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:37:19.828 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:37:19.828 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T18:37:19.829 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:19.829 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 6/6 2024-04-22T18:37:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 1/6 2024-04-22T18:37:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T18:37:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T18:37:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T18:37:20.117 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T18:37:20.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados-devel 2024-04-22T18:37:20.351 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T18:37:20.352 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:20.352 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:37:20.352 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:37:20.352 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T18:37:20.352 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T18:37:20.352 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T18:37:20.353 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T18:37:20.353 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T18:37:20.353 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:20.353 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:20.598 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-fuse 2024-04-22T18:37:20.638 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:32 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:20.760 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:20.761 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:20.761 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-04-22T18:37:20.761 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:20.761 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:37:20.761 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 127 k 2024-04-22T18:37:20.761 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:20.761 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:37:20.761 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:20.761 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-22T18:37:20.761 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:20.762 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 127 k 2024-04-22T18:37:20.762 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 456 k 2024-04-22T18:37:20.762 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:37:21.065 INFO:teuthology.orchestra.run.smithi121.stdout:librados-devel-18.2.2-1240.g6a7528e4.el9.x86_64 417 kB/s | 127 kB 00:00 2024-04-22T18:37:21.066 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:21.066 INFO:teuthology.orchestra.run.smithi121.stdout:Total 415 kB/s | 127 kB 00:00 2024-04-22T18:37:21.066 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:37:21.072 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:37:21.072 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:37:21.110 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:24 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:21.133 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:37:21.134 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:37:21.229 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:21.230 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-04-22T18:37:21.230 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:21.230 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:37:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 848 k 2024-04-22T18:37:21.230 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-04-22T18:37:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T18:37:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:21.231 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:37:21.231 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:21.231 INFO:teuthology.orchestra.run.smithi089.stdout:Install 2 Packages 2024-04-22T18:37:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:21.231 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 929 k 2024-04-22T18:37:21.231 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 2.7 M 2024-04-22T18:37:21.231 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:37:21.281 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:37:21.394 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:21.686 INFO:teuthology.orchestra.run.smithi089.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 464 kB/s | 80 kB 00:00 2024-04-22T18:37:21.788 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:22.011 INFO:teuthology.orchestra.run.smithi089.stdout:(2/2): ceph-fuse-18.2.2-1240.g6a7528e4.el9.x86_ 1.7 MB/s | 848 kB 00:00 2024-04-22T18:37:22.012 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:22.012 INFO:teuthology.orchestra.run.smithi089.stdout:Total 1.2 MB/s | 929 kB 00:00 2024-04-22T18:37:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:22.020 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:37:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:37:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:22.020 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:22.024 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:37:22.046 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:37:22.046 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:37:22.122 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:37:22.123 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:37:22.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs2 2024-04-22T18:37:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:37:22.442 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T18:37:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 2/2 2024-04-22T18:37:22.787 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:34 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:22.867 INFO:teuthology.orchestra.run.smithi121.stdout:Package libcephfs2-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:37:22.909 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:22.909 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-22T18:37:22.910 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:22.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs-devel 2024-04-22T18:37:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 2/2 2024-04-22T18:37:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/2 2024-04-22T18:37:23.344 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T18:37:23.344 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:23.344 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:37:23.344 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T18:37:23.344 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:23.344 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:23.506 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:35 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:23.594 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-volume 2024-04-22T18:37:23.624 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:23.625 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-04-22T18:37:23.625 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:23.625 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:37:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 31 k 2024-04-22T18:37:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:23.625 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:37:23.625 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:23.625 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-22T18:37:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:23.626 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 31 k 2024-04-22T18:37:23.626 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 139 k 2024-04-22T18:37:23.626 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:37:23.844 INFO:teuthology.orchestra.run.smithi121.stdout:libcephfs-devel-18.2.2-1240.g6a7528e4.el9.x86_6 143 kB/s | 31 kB 00:00 2024-04-22T18:37:23.844 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stdout:Total 142 kB/s | 31 kB 00:00 2024-04-22T18:37:23.845 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:37:23.848 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:37:23.850 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:37:23.851 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:37:23.935 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:37:24.056 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:24.117 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:27 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:24.240 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:24.241 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:24.241 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-04-22T18:37:24.241 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:24.241 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:37:24.241 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume noarch 2:18.2.2-1240.g6a7528e4.el9 ceph-noarch 264 k 2024-04-22T18:37:24.241 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-04-22T18:37:24.241 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T18:37:24.241 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:24.241 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:37:24.241 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:24.241 INFO:teuthology.orchestra.run.smithi089.stdout:Install 2 Packages 2024-04-22T18:37:24.241 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:24.242 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 341 k 2024-04-22T18:37:24.242 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 1.5 M 2024-04-22T18:37:24.242 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:37:24.366 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:24.507 INFO:teuthology.orchestra.run.smithi089.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 392 kB/s | 77 kB 00:00 2024-04-22T18:37:24.567 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:24.567 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:24.567 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:37:24.567 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:37:24.567 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:24.567 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:24.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados2 2024-04-22T18:37:24.799 INFO:teuthology.orchestra.run.smithi089.stdout:(2/2): ceph-volume-18.2.2-1240.g6a7528e4.el9.no 539 kB/s | 264 kB 00:00 2024-04-22T18:37:24.801 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:24.801 INFO:teuthology.orchestra.run.smithi089.stdout:Total 609 kB/s | 341 kB 00:00 2024-04-22T18:37:24.813 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:37:24.822 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:37:24.822 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:37:24.847 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:37:24.847 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:37:24.993 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:37:25.174 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-22T18:37:25.216 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:37 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:25.218 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-volume-2:18.2.2-1240.g6a7528e4.el9.noarch 2/2 2024-04-22T18:37:25.233 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-volume-2:18.2.2-1240.g6a7528e4.el9.noarch 2/2 2024-04-22T18:37:25.233 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:37:25.234 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T18:37:25.234 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:25.294 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:37:25.335 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:25.336 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-22T18:37:25.336 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:25.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librbd1 2024-04-22T18:37:25.853 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-volume-2:18.2.2-1240.g6a7528e4.el9.noarch 1/2 2024-04-22T18:37:25.904 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:37 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:25.982 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:37:26.023 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-22T18:37:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:26.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rados 2024-04-22T18:37:26.142 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-22T18:37:26.142 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:26.142 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:37:26.142 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume-2:18.2.2-1240.g6a7528e4.el9.noarch 2024-04-22T18:37:26.143 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T18:37:26.143 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:26.143 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:26.462 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados-devel 2024-04-22T18:37:26.594 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:38 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:26.672 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rados-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:37:26.712 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:26.713 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-22T18:37:26.713 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:26.774 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rgw 2024-04-22T18:37:26.983 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:29 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:27.108 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:27.109 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:27.109 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-04-22T18:37:27.109 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:27.109 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:37:27.110 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 127 k 2024-04-22T18:37:27.110 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:27.110 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:37:27.110 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:27.110 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-04-22T18:37:27.110 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:27.110 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 127 k 2024-04-22T18:37:27.110 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 456 k 2024-04-22T18:37:27.110 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:37:27.289 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:39 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:27.330 INFO:teuthology.orchestra.run.smithi089.stdout:librados-devel-18.2.2-1240.g6a7528e4.el9.x86_64 577 kB/s | 127 kB 00:00 2024-04-22T18:37:27.330 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:27.331 INFO:teuthology.orchestra.run.smithi089.stdout:Total 572 kB/s | 127 kB 00:00 2024-04-22T18:37:27.331 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:37:27.336 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:37:27.336 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:37:27.368 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:37:27.399 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:37:27.399 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:37:27.408 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:27.409 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-22T18:37:27.410 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:27.464 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-cephfs 2024-04-22T18:37:27.553 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:37:27.652 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librados-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:27.969 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:39 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:28.047 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-cephfs-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:37:28.087 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-22T18:37:28.088 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:28.101 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:28.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rbd 2024-04-22T18:37:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:28.325 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:37:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:37:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:28.326 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:28.571 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs2 2024-04-22T18:37:28.660 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:40 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:28.739 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:37:28.780 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-22T18:37:28.781 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:28.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-fuse 2024-04-22T18:37:29.088 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:32 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:29.167 INFO:teuthology.orchestra.run.smithi089.stdout:Package libcephfs2-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:37:29.208 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:29.209 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-04-22T18:37:29.209 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:29.285 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs-devel 2024-04-22T18:37:29.354 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:41 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:29.473 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:29.474 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:29.474 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-22T18:37:29.474 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:29.474 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:37:29.474 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 87 k 2024-04-22T18:37:29.474 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:29.474 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:37:29.474 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:29.474 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-22T18:37:29.474 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:29.475 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 87 k 2024-04-22T18:37:29.475 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 227 k 2024-04-22T18:37:29.475 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:37:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-fuse-18.2.2-1240.g6a7528e4.el9.x86_64.rpm 391 kB/s | 87 kB 00:00 2024-04-22T18:37:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:29.697 INFO:teuthology.orchestra.run.smithi121.stdout:Total 389 kB/s | 87 kB 00:00 2024-04-22T18:37:29.698 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:37:29.704 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:37:29.704 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:37:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:37:29.766 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:37:29.802 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:32 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:37:29.922 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:29.923 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:29.923 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-04-22T18:37:29.923 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:29.923 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:37:29.923 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 31 k 2024-04-22T18:37:29.923 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:29.923 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:37:29.924 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:29.924 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-04-22T18:37:29.924 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:29.924 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 31 k 2024-04-22T18:37:29.924 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 139 k 2024-04-22T18:37:29.924 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:37:30.009 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:30.308 INFO:teuthology.orchestra.run.smithi089.stdout:libcephfs-devel-18.2.2-1240.g6a7528e4.el9.x86_6 81 kB/s | 31 kB 00:00 2024-04-22T18:37:30.309 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:30.309 INFO:teuthology.orchestra.run.smithi089.stdout:Total 81 kB/s | 31 kB 00:00 2024-04-22T18:37:30.309 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:37:30.313 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:37:30.313 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:37:30.315 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:37:30.315 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:37:30.363 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:30.399 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:37:30.511 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:30.625 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:30.625 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:30.625 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:37:30.625 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:37:30.625 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:30.625 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:30.872 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:30.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-mirror 2024-04-22T18:37:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:31.073 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:37:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:37:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:31.074 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:31.269 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados2 2024-04-22T18:37:31.421 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:43 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:31.543 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:31.544 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-04-22T18:37:31.544 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:31.544 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:37:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 3.0 M 2024-04-22T18:37:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:31.544 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:37:31.544 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:31.545 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-22T18:37:31.545 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:31.545 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.0 M 2024-04-22T18:37:31.545 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 12 M 2024-04-22T18:37:31.545 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:37:31.776 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:34 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:31.854 INFO:teuthology.orchestra.run.smithi089.stdout:Package librados2-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:37:31.894 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:31.896 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-04-22T18:37:31.896 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:31.970 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librbd1 2024-04-22T18:37:32.477 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:35 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:32.533 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-mirror-18.2.2-1240.g6a7528e4.el9.x86_64.rpm 3.0 MB/s | 3.0 MB 00:00 2024-04-22T18:37:32.533 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:32.534 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.0 MB/s | 3.0 MB 00:00 2024-04-22T18:37:32.534 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:37:32.543 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:37:32.543 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:37:32.556 INFO:teuthology.orchestra.run.smithi089.stdout:Package librbd1-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:37:32.597 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:32.598 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-04-22T18:37:32.598 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:32.622 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:37:32.622 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:37:32.668 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rados 2024-04-22T18:37:33.036 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:37:33.108 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:33.132 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:33.132 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:37:33.132 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T18:37:33.133 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T18:37:33.133 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T18:37:33.133 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:33.179 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:36 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:33.258 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rados-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:37:33.298 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:33.299 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-04-22T18:37:33.299 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:33.367 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rgw 2024-04-22T18:37:33.884 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:36 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:33.963 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rgw-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:37:34.004 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:34.005 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-04-22T18:37:34.005 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:34.077 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-cephfs 2024-04-22T18:37:34.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:34.117 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:34.117 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:37:34.117 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:37:34.118 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:34.118 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:34.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-nbd 2024-04-22T18:37:34.587 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:37 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:34.665 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-cephfs-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:37:34.705 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:34.706 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-04-22T18:37:34.706 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:34.779 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rbd 2024-04-22T18:37:34.862 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:46 ago on Mon 22 Apr 2024 06:34:48 PM UTC. 2024-04-22T18:37:34.984 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-22T18:37:34.984 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:34.985 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-22T18:37:34.985 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:34.985 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-22T18:37:34.985 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 172 k 2024-04-22T18:37:34.985 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:34.985 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-22T18:37:34.985 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-22T18:37:34.985 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-22T18:37:34.985 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:34.985 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 172 k 2024-04-22T18:37:34.985 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 494 k 2024-04-22T18:37:34.985 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-22T18:37:35.294 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:38 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:35.373 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 is already installed. 2024-04-22T18:37:35.413 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:35.414 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-04-22T18:37:35.414 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:35.469 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-nbd-18.2.2-1240.g6a7528e4.el9.x86_64.rpm 356 kB/s | 172 kB 00:00 2024-04-22T18:37:35.470 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:35.470 INFO:teuthology.orchestra.run.smithi121.stdout:Total 355 kB/s | 172 kB 00:00 2024-04-22T18:37:35.470 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-22T18:37:35.478 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-22T18:37:35.478 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-22T18:37:35.483 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-fuse 2024-04-22T18:37:35.547 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-22T18:37:35.548 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-22T18:37:35.705 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-22T18:37:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:35.986 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:38 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:36.106 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:36.107 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-04-22T18:37:36.107 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:36.107 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:37:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 87 k 2024-04-22T18:37:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:36.107 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:37:36.107 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:36.107 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-04-22T18:37:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:36.108 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 87 k 2024-04-22T18:37:36.108 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 227 k 2024-04-22T18:37:36.108 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:37:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:36.426 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:36.426 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:36.426 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-22T18:37:36.426 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:37:36.426 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:36.426 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-22T18:37:36.673 DEBUG:teuthology.parallel:result is None 2024-04-22T18:37:38.495 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-fuse-18.2.2-1240.g6a7528e4.el9.x86_64.rpm 36 kB/s | 87 kB 00:02 2024-04-22T18:37:38.495 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:38.495 INFO:teuthology.orchestra.run.smithi089.stdout:Total 36 kB/s | 87 kB 00:02 2024-04-22T18:37:38.496 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:37:38.504 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:37:38.504 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:37:38.570 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:37:38.571 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:37:38.714 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:37:38.848 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:39.157 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:39.401 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:37:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:37:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:39.401 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:39.648 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-mirror 2024-04-22T18:37:40.182 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:43 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:40.305 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:40.306 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:40.306 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-04-22T18:37:40.306 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:40.306 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:37:40.306 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 3.0 M 2024-04-22T18:37:40.306 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:40.306 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:37:40.306 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:40.307 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-04-22T18:37:40.307 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:40.307 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 3.0 M 2024-04-22T18:37:40.307 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 12 M 2024-04-22T18:37:40.307 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:37:41.282 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-mirror-18.2.2-1240.g6a7528e4.el9.x86_64.rpm 3.1 MB/s | 3.0 MB 00:00 2024-04-22T18:37:41.282 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:41.283 INFO:teuthology.orchestra.run.smithi089.stdout:Total 3.1 MB/s | 3.0 MB 00:00 2024-04-22T18:37:41.283 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:37:41.292 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:37:41.292 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:37:41.372 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:37:41.372 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:37:41.794 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:37:41.857 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-mirror-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:41.882 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:41.882 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T18:37:41.882 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T18:37:41.883 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T18:37:41.883 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T18:37:41.883 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:42.999 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-mirror-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:42.999 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:42.999 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:37:42.999 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:37:42.999 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:42.999 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:43.225 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-nbd 2024-04-22T18:37:43.742 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:46 ago on Mon 22 Apr 2024 06:34:57 PM UTC. 2024-04-22T18:37:43.864 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-04-22T18:37:43.865 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:43.865 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-04-22T18:37:43.865 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:43.865 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-04-22T18:37:43.865 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd x86_64 2:18.2.2-1240.g6a7528e4.el9 ceph 172 k 2024-04-22T18:37:43.865 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:43.865 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-04-22T18:37:43.865 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-04-22T18:37:43.866 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-04-22T18:37:43.866 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:43.866 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 172 k 2024-04-22T18:37:43.866 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 494 k 2024-04-22T18:37:43.866 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-04-22T18:37:44.685 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-nbd-18.2.2-1240.g6a7528e4.el9.x86_64.rpm 210 kB/s | 172 kB 00:00 2024-04-22T18:37:44.686 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-04-22T18:37:44.686 INFO:teuthology.orchestra.run.smithi089.stdout:Total 210 kB/s | 172 kB 00:00 2024-04-22T18:37:44.686 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-04-22T18:37:44.694 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-04-22T18:37:44.694 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-04-22T18:37:44.763 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-04-22T18:37:44.763 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-04-22T18:37:44.914 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-04-22T18:37:45.023 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-nbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:45.407 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:45.644 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-nbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 1/1 2024-04-22T18:37:45.644 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:45.644 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-04-22T18:37:45.644 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd-2:18.2.2-1240.g6a7528e4.el9.x86_64 2024-04-22T18:37:45.644 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:45.645 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-04-22T18:37:45.888 DEBUG:teuthology.parallel:result is None 2024-04-22T18:37:45.889 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0 2024-04-22T18:37:46.129 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T18:37:46.149 INFO:teuthology.orchestra.run.smithi089.stdout:18.2.2-1240.g6a7528e4.el9 2024-04-22T18:37:46.149 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1240.g6a7528e4.el9 2024-04-22T18:37:46.150 INFO:teuthology.task.install:The correct ceph version 18.2.2-1240.g6a7528e4 is installed. 2024-04-22T18:37:46.151 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0 2024-04-22T18:37:46.293 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T18:37:46.317 INFO:teuthology.orchestra.run.smithi121.stdout:18.2.2-1240.g6a7528e4.el9 2024-04-22T18:37:46.318 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1240.g6a7528e4.el9 2024-04-22T18:37:46.318 INFO:teuthology.task.install:The correct ceph version 18.2.2-1240.g6a7528e4 is installed. 2024-04-22T18:37:46.319 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T18:37:46.319 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:37:46.319 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T18:37:46.343 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:37:46.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T18:37:46.386 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T18:37:46.386 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:37:46.386 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T18:37:46.409 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T18:37:46.472 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:37:46.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T18:37:46.500 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T18:37:46.568 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T18:37:46.568 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:37:46.568 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T18:37:46.605 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T18:37:46.668 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:37:46.669 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T18:37:46.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T18:37:46.758 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T18:37:46.759 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:37:46.759 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T18:37:46.781 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T18:37:46.845 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:37:46.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T18:37:46.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T18:37:46.935 INFO:teuthology.run_tasks:Running task ceph... 2024-04-22T18:37:47.019 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-22T18:37:47.020 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /var/log/ceph 2024-04-22T18:37:47.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2024-04-22T18:37:47.046 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-22T18:37:47.047 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-22T18:37:47.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-22T18:37:47.112 INFO:tasks.ceph:Creating extra log directories... 2024-04-22T18:37:47.112 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-22T18:37:47.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-22T18:37:47.186 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-22T18:37:47.186 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-04-22T18:37:47.186 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668454', 'branch': 'wip-yuri2-testing-2024-04-17-0823-reef', 'description': 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/dedup-io-mixed}', 'email': None, 'first_in_suite': False, 'job_id': '7668454', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri2-testing-2024-04-17-0823-reef'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': '6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri2-testing-2024-04-17-0823-reef', 'sha1': '6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 400, 'sha1': '6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri2-testing-2024-04-17-0823-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0', 'targets': {'smithi089.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP4lU1xAhS0ccTpXvPUiIQBiIvOZx7/qs2e0xFwav+/oEDogmpAH9By3o8WiGxiJPSJisGhDlpgCqj+bTR6lmyU=', 'smithi121.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCX9iUvqlkVq+d0ykFVtV91xt8m5w0jIuQvMgYTGKr+JF3evHL9ahz5UEq79zp/6AiMEMow/+h7/GSURVTXTt+U='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': '6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0', 'cluster': 'ceph'}}, {'exec': {'mon.a': ['while ! ceph balancer status ; do sleep 1 ; done', 'ceph balancer mode crush-compat', 'ceph balancer on']}}, {'thrashosds': {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'exec': {'client.0': ['sudo ceph osd pool create low_tier 4']}}, {'rados': {'clients': ['client.0'], 'dedup_chunk_algo': 'fastcdc', 'dedup_chunk_size': '131072', 'enable_dedup': True, 'low_tier_pool': 'low_tier', 'objects': 50, 'op_weights': {'read': 100, 'set_chunk': 30, 'tier_evict': 10, 'tier_flush': 5, 'tier_promote': 10, 'write': 50}, 'ops': 1500, 'set_chunk': True}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-22_18:19:58', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-22T18:37:47.187 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-22T18:37:47.212 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-22T18:37:47.230 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/run/ceph 2024-04-22T18:37:47.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2024-04-22T18:37:47.293 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:37:47.293 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T18:37:47.342 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T18:37:47.343 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_1 2024-04-22T18:37:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T18:37:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T18:37:47.397 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-22T18:37:47.398 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T18:37:47.398 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T18:37:47.398 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-04-22 18:37:42.126583545 +0000 2024-04-22T18:37:47.398 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-04-22 18:33:40.618263437 +0000 2024-04-22T18:37:47.398 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-04-22 18:33:40.618263437 +0000 2024-04-22T18:37:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-04-22 18:33:40.618263437 +0000 2024-04-22T18:37:47.398 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T18:37:47.459 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-04-22T18:37:47.459 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-04-22T18:37:47.459 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000206919 s, 2.5 MB/s 2024-04-22T18:37:47.460 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T18:37:47.516 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_2 2024-04-22T18:37:47.570 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T18:37:47.570 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T18:37:47.571 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-22T18:37:47.571 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T18:37:47.571 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T18:37:47.571 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-04-22 18:37:42.127583523 +0000 2024-04-22T18:37:47.571 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-04-22 18:33:40.968255635 +0000 2024-04-22T18:37:47.571 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-04-22 18:33:40.968255635 +0000 2024-04-22T18:37:47.571 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-04-22 18:33:40.968255635 +0000 2024-04-22T18:37:47.571 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T18:37:47.632 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-04-22T18:37:47.632 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-04-22T18:37:47.632 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000206311 s, 2.5 MB/s 2024-04-22T18:37:47.633 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T18:37:47.689 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_3 2024-04-22T18:37:47.743 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T18:37:47.743 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T18:37:47.743 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T18:37:47.743 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T18:37:47.743 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T18:37:47.743 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-04-22 18:37:42.127583523 +0000 2024-04-22T18:37:47.744 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-04-22 18:33:41.317247855 +0000 2024-04-22T18:37:47.744 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-04-22 18:33:41.317247855 +0000 2024-04-22T18:37:47.744 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-04-22 18:33:41.317247855 +0000 2024-04-22T18:37:47.744 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T18:37:47.805 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-04-22T18:37:47.805 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-04-22T18:37:47.805 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000185729 s, 2.8 MB/s 2024-04-22T18:37:47.806 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T18:37:47.862 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_4 2024-04-22T18:37:47.916 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T18:37:47.916 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T18:37:47.916 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-22T18:37:47.916 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T18:37:47.917 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T18:37:47.917 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-04-22 18:37:42.128583501 +0000 2024-04-22T18:37:47.917 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-04-22 18:33:41.658240253 +0000 2024-04-22T18:37:47.917 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-04-22 18:33:41.658240253 +0000 2024-04-22T18:37:47.917 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-04-22 18:33:41.658240253 +0000 2024-04-22T18:37:47.917 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T18:37:47.978 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-04-22T18:37:47.979 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-04-22T18:37:47.979 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.0001979 s, 2.6 MB/s 2024-04-22T18:37:47.980 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T18:37:48.035 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-22T18:37:48.037 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:37:48.037 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T18:37:48.053 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T18:37:48.053 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-04-22T18:37:48.110 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T18:37:48.110 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T18:37:48.110 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-22T18:37:48.110 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T18:37:48.110 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T18:37:48.110 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-22 18:37:33.380100277 +0000 2024-04-22T18:37:48.110 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-22 18:33:45.186933679 +0000 2024-04-22T18:37:48.110 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-22 18:33:45.186933679 +0000 2024-04-22T18:37:48.110 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-04-22 18:33:45.186933679 +0000 2024-04-22T18:37:48.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T18:37:48.173 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-22T18:37:48.173 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-22T18:37:48.173 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000268343 s, 1.9 MB/s 2024-04-22T18:37:48.175 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T18:37:48.233 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-04-22T18:37:48.288 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T18:37:48.289 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T18:37:48.289 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-22T18:37:48.289 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T18:37:48.289 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T18:37:48.289 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-22 18:37:33.380100277 +0000 2024-04-22T18:37:48.289 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-22 18:33:45.526929122 +0000 2024-04-22T18:37:48.289 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-22 18:33:45.526929122 +0000 2024-04-22T18:37:48.289 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-04-22 18:33:45.526929122 +0000 2024-04-22T18:37:48.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T18:37:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-22T18:37:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-22T18:37:48.352 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000214757 s, 2.4 MB/s 2024-04-22T18:37:48.353 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T18:37:48.409 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-04-22T18:37:48.464 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T18:37:48.464 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T18:37:48.464 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-22T18:37:48.464 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T18:37:48.464 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T18:37:48.464 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-22 18:37:33.380100277 +0000 2024-04-22T18:37:48.465 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-22 18:33:45.841924900 +0000 2024-04-22T18:37:48.465 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-22 18:33:45.841924900 +0000 2024-04-22T18:37:48.465 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-04-22 18:33:45.841924900 +0000 2024-04-22T18:37:48.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T18:37:48.526 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-22T18:37:48.526 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-22T18:37:48.526 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000206897 s, 2.5 MB/s 2024-04-22T18:37:48.527 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T18:37:48.582 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-04-22T18:37:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T18:37:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T18:37:48.637 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-22T18:37:48.637 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T18:37:48.637 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T18:37:48.637 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-22 18:37:33.381100264 +0000 2024-04-22T18:37:48.637 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-22 18:33:46.186920276 +0000 2024-04-22T18:37:48.637 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-22 18:33:46.186920276 +0000 2024-04-22T18:37:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-04-22 18:33:46.186920276 +0000 2024-04-22T18:37:48.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T18:37:48.699 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-22T18:37:48.699 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-22T18:37:48.699 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000217072 s, 2.4 MB/s 2024-04-22T18:37:48.700 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T18:37:48.755 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-22T18:37:48.755 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-22T18:37:48.755 INFO:tasks.ceph:Generating config... 2024-04-22T18:37:48.756 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-22T18:37:48.757 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-22T18:37:48.757 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-22T18:37:48.757 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-22T18:37:48.757 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-22T18:37:48.757 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-22T18:37:48.757 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-22T18:37:48.757 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-22T18:37:48.757 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-22T18:37:48.757 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-22T18:37:48.757 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-22T18:37:48.757 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-22T18:37:48.758 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-22T18:37:48.758 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-22T18:37:48.758 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-22T18:37:48.758 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-04-22T18:37:48.758 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-22T18:37:48.758 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-22T18:37:48.758 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-22T18:37:48.758 INFO:tasks.ceph:[osd] osd inject bad map crc probability = 0.1 2024-04-22T18:37:48.758 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-04-22T18:37:48.758 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-22T18:37:48.758 INFO:tasks.ceph:[osd] osd max scrubs = 2 2024-04-22T18:37:48.758 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-22T18:37:48.758 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-22T18:37:48.758 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-22T18:37:48.758 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-22T18:37:48.759 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-22T18:37:48.759 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-04-22T18:37:48.759 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-04-22T18:37:48.759 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-22T18:37:48.759 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-04-22T18:37:48.759 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-04-22T18:37:48.759 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-04-22T18:37:48.759 INFO:tasks.ceph:[global] ms type = async 2024-04-22T18:37:48.759 INFO:tasks.ceph:[global] osd debug inject dispatch delay duration = 0.1 2024-04-22T18:37:48.759 INFO:tasks.ceph:[global] osd debug inject dispatch delay probability = 0.1 2024-04-22T18:37:48.759 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2024-04-22T18:37:48.759 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-04-22T18:37:48.759 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-04-22T18:37:48.759 INFO:tasks.ceph:[global] osd_pool_default_size = 2 2024-04-22T18:37:48.759 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-22T18:37:48.760 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-22T18:37:48.760 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-22T18:37:48.760 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-22T18:37:48.760 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-22T18:37:48.760 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-22T18:37:48.760 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-22T18:37:48.760 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-22T18:37:48.760 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-22T18:37:48.760 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-22T18:37:48.760 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-22T18:37:48.760 INFO:tasks.ceph:Setting up mon.a... 2024-04-22T18:37:48.760 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-22T18:37:48.819 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/ceph.keyring 2024-04-22T18:37:48.822 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-22T18:37:48.866 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T18:37:48.891 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.89:6789]'), ('mon.c', '[v1:172.21.15.89:6790]'), ('mon.b', '[v1:172.21.15.121:6789]')] 2024-04-22T18:37:48.891 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.89:6789],[v1:172.21.15.89:6790],[v1:172.21.15.121:6789]', 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-22T18:37:48.892 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:37:48.892 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T18:37:48.946 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.89:6789]' --addv c '[v1:172.21.15.89:6790]' --addv b '[v1:172.21.15.121:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-22T18:37:49.032 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-22T18:37:49.032 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: generated fsid 9e25744c-eff1-4a4a-8d47-0dda406b2c35 2024-04-22T18:37:49.032 INFO:teuthology.orchestra.run.smithi089.stdout:setting min_mon_release = pacific 2024-04-22T18:37:49.032 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 0 2024-04-22T18:37:49.032 INFO:teuthology.orchestra.run.smithi089.stdout:fsid 9e25744c-eff1-4a4a-8d47-0dda406b2c35 2024-04-22T18:37:49.032 INFO:teuthology.orchestra.run.smithi089.stdout:last_changed 2024-04-22T18:37:49.033835+0000 2024-04-22T18:37:49.032 INFO:teuthology.orchestra.run.smithi089.stdout:created 2024-04-22T18:37:49.033835+0000 2024-04-22T18:37:49.032 INFO:teuthology.orchestra.run.smithi089.stdout:min_mon_release 16 (pacific) 2024-04-22T18:37:49.032 INFO:teuthology.orchestra.run.smithi089.stdout:election_strategy: 1 2024-04-22T18:37:49.033 INFO:teuthology.orchestra.run.smithi089.stdout:0: v1:172.21.15.89:6789/0 mon.a 2024-04-22T18:37:49.033 INFO:teuthology.orchestra.run.smithi089.stdout:1: v1:172.21.15.121:6789/0 mon.b 2024-04-22T18:37:49.033 INFO:teuthology.orchestra.run.smithi089.stdout:2: v1:172.21.15.89:6790/0 mon.c 2024-04-22T18:37:49.033 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-22T18:37:49.037 DEBUG:teuthology.orchestra.run.smithi089:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T18:37:49.052 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9e25744c-eff1-4a4a-8d47-0dda406b2c35... 2024-04-22T18:37:49.053 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-22T18:37:49.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-22T18:37:49.132 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2024-04-22T18:37:49.132 INFO:teuthology.orchestra.run.smithi089.stdout: chdir = "" 2024-04-22T18:37:49.132 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-22T18:37:49.132 INFO:teuthology.orchestra.run.smithi089.stdout: auth supported = cephx 2024-04-22T18:37:49.132 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.132 INFO:teuthology.orchestra.run.smithi089.stdout: filestore xattr use omap = true 2024-04-22T18:37:49.132 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.133 INFO:teuthology.orchestra.run.smithi089.stdout: mon clock drift allowed = 1.000 2024-04-22T18:37:49.133 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.133 INFO:teuthology.orchestra.run.smithi089.stdout: osd crush chooseleaf type = 0 2024-04-22T18:37:49.133 INFO:teuthology.orchestra.run.smithi089.stdout: auth debug = true 2024-04-22T18:37:49.133 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.133 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on old message = true 2024-04-22T18:37:49.133 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on bug = true 2024-04-22T18:37:49.133 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.133 INFO:teuthology.orchestra.run.smithi089.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-22T18:37:49.133 INFO:teuthology.orchestra.run.smithi089.stdout: mon pg warn max object skew = 0 2024-04-22T18:37:49.133 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.133 INFO:teuthology.orchestra.run.smithi089.stdout: # disable pg_autoscaler by default for new pools 2024-04-22T18:37:49.133 INFO:teuthology.orchestra.run.smithi089.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-22T18:37:49.134 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.134 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default size = 2 2024-04-22T18:37:49.134 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.134 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow primary affinity = true 2024-04-22T18:37:49.134 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow pg remap = true 2024-04-22T18:37:49.134 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on legacy crush tunables = false 2024-04-22T18:37:49.134 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on crush straw calc version zero = false 2024-04-22T18:37:49.134 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on no sortbitwise = false 2024-04-22T18:37:49.134 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on osd down out interval zero = false 2024-04-22T18:37:49.134 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on too few osds = false 2024-04-22T18:37:49.134 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-22T18:37:49.134 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-22T18:37:49.134 INFO:teuthology.orchestra.run.smithi089.stdout: mon_allow_pool_size_one = true 2024-04-22T18:37:49.135 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.135 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-22T18:37:49.135 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.135 INFO:teuthology.orchestra.run.smithi089.stdout: osd default data pool replay window = 5 2024-04-22T18:37:49.135 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.135 INFO:teuthology.orchestra.run.smithi089.stdout: mon allow pool delete = true 2024-04-22T18:37:49.135 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.135 INFO:teuthology.orchestra.run.smithi089.stdout: mon cluster log file level = debug 2024-04-22T18:37:49.135 INFO:teuthology.orchestra.run.smithi089.stdout: debug asserts on shutdown = true 2024-04-22T18:37:49.135 INFO:teuthology.orchestra.run.smithi089.stdout: mon health detail to clog = false 2024-04-22T18:37:49.135 INFO:teuthology.orchestra.run.smithi089.stdout: mon host = "[v1:172.21.15.89:6789],[v1:172.21.15.89:6790],[v1:172.21.15.121:6789]" 2024-04-22T18:37:49.135 INFO:teuthology.orchestra.run.smithi089.stdout: mon election default strategy = 1 2024-04-22T18:37:49.136 INFO:teuthology.orchestra.run.smithi089.stdout: ms bind msgr2 = False 2024-04-22T18:37:49.136 INFO:teuthology.orchestra.run.smithi089.stdout: ms type = async 2024-04-22T18:37:49.136 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug inject dispatch delay duration = 0.1 2024-04-22T18:37:49.136 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug inject dispatch delay probability = 0.1 2024-04-22T18:37:49.136 INFO:teuthology.orchestra.run.smithi089.stdout: osd_async_recovery_min_cost = 1 2024-04-22T18:37:49.136 INFO:teuthology.orchestra.run.smithi089.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-22T18:37:49.136 INFO:teuthology.orchestra.run.smithi089.stdout: osd_pool_default_min_size = 2 2024-04-22T18:37:49.136 INFO:teuthology.orchestra.run.smithi089.stdout: osd_pool_default_size = 2 2024-04-22T18:37:49.136 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = 9e25744c-eff1-4a4a-8d47-0dda406b2c35 2024-04-22T18:37:49.136 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.136 INFO:teuthology.orchestra.run.smithi089.stdout:[osd] 2024-04-22T18:37:49.136 INFO:teuthology.orchestra.run.smithi089.stdout: osd journal size = 100 2024-04-22T18:37:49.136 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.137 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub load threshold = 5.0 2024-04-22T18:37:49.137 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub max interval = 120 2024-04-22T18:37:49.137 INFO:teuthology.orchestra.run.smithi089.stdout: osd mclock profile = high_recovery_ops 2024-04-22T18:37:49.137 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.137 INFO:teuthology.orchestra.run.smithi089.stdout: osd recover clone overlap = true 2024-04-22T18:37:49.137 INFO:teuthology.orchestra.run.smithi089.stdout: osd recovery max chunk = 1048576 2024-04-22T18:37:49.137 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.137 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug shutdown = true 2024-04-22T18:37:49.137 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug op order = true 2024-04-22T18:37:49.137 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify stray on activate = true 2024-04-22T18:37:49.137 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.137 INFO:teuthology.orchestra.run.smithi089.stdout: osd open classes on start = true 2024-04-22T18:37:49.137 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug pg log writeout = true 2024-04-22T18:37:49.137 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.138 INFO:teuthology.orchestra.run.smithi089.stdout: osd deep scrub update digest min age = 30 2024-04-22T18:37:49.138 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.138 INFO:teuthology.orchestra.run.smithi089.stdout: osd map max advance = 10 2024-04-22T18:37:49.138 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.138 INFO:teuthology.orchestra.run.smithi089.stdout: journal zero on create = true 2024-04-22T18:37:49.138 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.138 INFO:teuthology.orchestra.run.smithi089.stdout: filestore ondisk finisher threads = 3 2024-04-22T18:37:49.138 INFO:teuthology.orchestra.run.smithi089.stdout: filestore apply finisher threads = 3 2024-04-22T18:37:49.138 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.138 INFO:teuthology.orchestra.run.smithi089.stdout: bdev debug aio = true 2024-04-22T18:37:49.138 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug misdirected ops = true 2024-04-22T18:37:49.138 INFO:teuthology.orchestra.run.smithi089.stdout: debug monc = 20 2024-04-22T18:37:49.138 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore block size = 96636764160 2024-04-22T18:37:49.139 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore compression algorithm = snappy 2024-04-22T18:37:49.139 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore compression mode = aggressive 2024-04-22T18:37:49.139 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore fsck on mount = True 2024-04-22T18:37:49.139 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore zero block detection = True 2024-04-22T18:37:49.139 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluefs = 20 2024-04-22T18:37:49.139 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluestore = 20 2024-04-22T18:37:49.139 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-04-22T18:37:49.139 INFO:teuthology.orchestra.run.smithi089.stdout: debug osd = 20 2024-04-22T18:37:49.139 INFO:teuthology.orchestra.run.smithi089.stdout: debug rocksdb = 10 2024-04-22T18:37:49.139 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd backfillfull_ratio = 0.85 2024-04-22T18:37:49.139 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd full ratio = 0.9 2024-04-22T18:37:49.139 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd nearfull ratio = 0.8 2024-04-22T18:37:49.139 INFO:teuthology.orchestra.run.smithi089.stdout: osd blocked scrub grace period = 3600 2024-04-22T18:37:49.139 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug reject backfill probability = 0.3 2024-04-22T18:37:49.140 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify cached snaps = True 2024-04-22T18:37:49.140 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify missing on start = True 2024-04-22T18:37:49.140 INFO:teuthology.orchestra.run.smithi089.stdout: osd failsafe full ratio = 0.95 2024-04-22T18:37:49.140 INFO:teuthology.orchestra.run.smithi089.stdout: osd inject bad map crc probability = 0.1 2024-04-22T18:37:49.140 INFO:teuthology.orchestra.run.smithi089.stdout: osd max backfills = 3 2024-04-22T18:37:49.140 INFO:teuthology.orchestra.run.smithi089.stdout: osd max markdown count = 1000 2024-04-22T18:37:49.140 INFO:teuthology.orchestra.run.smithi089.stdout: osd max scrubs = 2 2024-04-22T18:37:49.140 INFO:teuthology.orchestra.run.smithi089.stdout: osd mclock override recovery settings = True 2024-04-22T18:37:49.140 INFO:teuthology.orchestra.run.smithi089.stdout: osd objectstore = bluestore 2024-04-22T18:37:49.140 INFO:teuthology.orchestra.run.smithi089.stdout: osd op queue = debug_random 2024-04-22T18:37:49.140 INFO:teuthology.orchestra.run.smithi089.stdout: osd op queue cut off = debug_random 2024-04-22T18:37:49.140 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub min interval = 60 2024-04-22T18:37:49.140 INFO:teuthology.orchestra.run.smithi089.stdout: osd shutdown pgref assert = True 2024-04-22T18:37:49.140 INFO:teuthology.orchestra.run.smithi089.stdout: osd snap trim sleep = 2 2024-04-22T18:37:49.141 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.141 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr] 2024-04-22T18:37:49.141 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-04-22T18:37:49.141 INFO:teuthology.orchestra.run.smithi089.stdout: debug mgr = 20 2024-04-22T18:37:49.141 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-04-22T18:37:49.141 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-04-22T18:37:49.141 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-04-22T18:37:49.141 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-04-22T18:37:49.141 INFO:teuthology.orchestra.run.smithi089.stdout: mgr/telemetry/nag = false 2024-04-22T18:37:49.141 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.141 INFO:teuthology.orchestra.run.smithi089.stdout:[mon] 2024-04-22T18:37:49.141 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-04-22T18:37:49.141 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-04-22T18:37:49.141 INFO:teuthology.orchestra.run.smithi089.stdout: debug paxos = 20 2024-04-22T18:37:49.142 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-04-22T18:37:49.142 INFO:teuthology.orchestra.run.smithi089.stdout: mon data avail warn = 5 2024-04-22T18:37:49.142 INFO:teuthology.orchestra.run.smithi089.stdout: mon mgr mkfs grace = 240 2024-04-22T18:37:49.142 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-04-22T18:37:49.142 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd reporter subtree level = osd 2024-04-22T18:37:49.142 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd prime pg temp = true 2024-04-22T18:37:49.142 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-04-22T18:37:49.142 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.142 INFO:teuthology.orchestra.run.smithi089.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-22T18:37:49.142 INFO:teuthology.orchestra.run.smithi089.stdout: auth mon ticket ttl = 660 # 11m 2024-04-22T18:37:49.142 INFO:teuthology.orchestra.run.smithi089.stdout: auth service ticket ttl = 240 # 4m 2024-04-22T18:37:49.142 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.142 INFO:teuthology.orchestra.run.smithi089.stdout: # don't complain about insecure global_id in the test suite 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: # 1m isn't quite enough 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: mon_down_mkfs_grace = 2m 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_filestore_osds = false 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: mon min osdmap epochs = 50 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: mon osdmap full prune interval = 2 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: mon osdmap full prune min = 15 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: mon osdmap full prune txsize = 2 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: mon scrub interval = 300 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: paxos service trim min = 10 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout:[client] 2024-04-22T18:37:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: rgw cache enabled = true 2024-04-22T18:37:49.144 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable ops log = true 2024-04-22T18:37:49.144 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable usage log = true 2024-04-22T18:37:49.144 INFO:teuthology.orchestra.run.smithi089.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-22T18:37:49.144 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-22T18:37:49.144 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.a] 2024-04-22T18:37:49.144 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.c] 2024-04-22T18:37:49.144 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.b] 2024-04-22T18:37:49.145 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-04-22T18:37:49.145 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2024-04-22T18:37:49.145 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-22T18:37:49.145 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2024-04-22T18:37:49.145 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.145 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2024-04-22T18:37:49.145 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.145 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2024-04-22T18:37:49.145 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.145 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2024-04-22T18:37:49.145 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2024-04-22T18:37:49.145 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.146 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2024-04-22T18:37:49.146 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2024-04-22T18:37:49.146 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.146 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-22T18:37:49.146 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2024-04-22T18:37:49.146 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.146 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2024-04-22T18:37:49.146 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-22T18:37:49.146 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.146 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2024-04-22T18:37:49.146 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.146 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2024-04-22T18:37:49.146 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2024-04-22T18:37:49.146 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2024-04-22T18:37:49.146 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2024-04-22T18:37:49.147 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2024-04-22T18:37:49.147 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2024-04-22T18:37:49.147 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2024-04-22T18:37:49.147 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-22T18:37:49.147 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-22T18:37:49.147 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2024-04-22T18:37:49.147 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.147 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-22T18:37:49.147 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.147 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2024-04-22T18:37:49.147 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.147 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2024-04-22T18:37:49.147 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.147 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2024-04-22T18:37:49.147 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2024-04-22T18:37:49.148 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2024-04-22T18:37:49.148 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "[v1:172.21.15.89:6789],[v1:172.21.15.89:6790],[v1:172.21.15.121:6789]" 2024-04-22T18:37:49.148 INFO:teuthology.orchestra.run.smithi121.stdout: mon election default strategy = 1 2024-04-22T18:37:49.148 INFO:teuthology.orchestra.run.smithi121.stdout: ms bind msgr2 = False 2024-04-22T18:37:49.148 INFO:teuthology.orchestra.run.smithi121.stdout: ms type = async 2024-04-22T18:37:49.148 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug inject dispatch delay duration = 0.1 2024-04-22T18:37:49.148 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug inject dispatch delay probability = 0.1 2024-04-22T18:37:49.148 INFO:teuthology.orchestra.run.smithi121.stdout: osd_async_recovery_min_cost = 1 2024-04-22T18:37:49.148 INFO:teuthology.orchestra.run.smithi121.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-22T18:37:49.148 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_min_size = 2 2024-04-22T18:37:49.148 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_size = 2 2024-04-22T18:37:49.148 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 9e25744c-eff1-4a4a-8d47-0dda406b2c35 2024-04-22T18:37:49.148 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.148 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2024-04-22T18:37:49.149 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2024-04-22T18:37:49.149 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.149 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2024-04-22T18:37:49.149 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 120 2024-04-22T18:37:49.149 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2024-04-22T18:37:49.149 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.149 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2024-04-22T18:37:49.149 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2024-04-22T18:37:49.149 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.149 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2024-04-22T18:37:49.149 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2024-04-22T18:37:49.149 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2024-04-22T18:37:49.149 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.150 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2024-04-22T18:37:49.150 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2024-04-22T18:37:49.150 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.150 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2024-04-22T18:37:49.150 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.150 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2024-04-22T18:37:49.150 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.150 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2024-04-22T18:37:49.150 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.150 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2024-04-22T18:37:49.150 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2024-04-22T18:37:49.150 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.150 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2024-04-22T18:37:49.150 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2024-04-22T18:37:49.150 INFO:teuthology.orchestra.run.smithi121.stdout: debug monc = 20 2024-04-22T18:37:49.151 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2024-04-22T18:37:49.151 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore compression algorithm = snappy 2024-04-22T18:37:49.151 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore compression mode = aggressive 2024-04-22T18:37:49.151 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2024-04-22T18:37:49.151 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore zero block detection = True 2024-04-22T18:37:49.151 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 20 2024-04-22T18:37:49.151 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 20 2024-04-22T18:37:49.151 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-22T18:37:49.151 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2024-04-22T18:37:49.151 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 10 2024-04-22T18:37:49.151 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2024-04-22T18:37:49.151 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2024-04-22T18:37:49.151 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2024-04-22T18:37:49.151 INFO:teuthology.orchestra.run.smithi121.stdout: osd blocked scrub grace period = 3600 2024-04-22T18:37:49.151 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug reject backfill probability = 0.3 2024-04-22T18:37:49.152 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify cached snaps = True 2024-04-22T18:37:49.152 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify missing on start = True 2024-04-22T18:37:49.152 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2024-04-22T18:37:49.152 INFO:teuthology.orchestra.run.smithi121.stdout: osd inject bad map crc probability = 0.1 2024-04-22T18:37:49.152 INFO:teuthology.orchestra.run.smithi121.stdout: osd max backfills = 3 2024-04-22T18:37:49.152 INFO:teuthology.orchestra.run.smithi121.stdout: osd max markdown count = 1000 2024-04-22T18:37:49.152 INFO:teuthology.orchestra.run.smithi121.stdout: osd max scrubs = 2 2024-04-22T18:37:49.152 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock override recovery settings = True 2024-04-22T18:37:49.152 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2024-04-22T18:37:49.152 INFO:teuthology.orchestra.run.smithi121.stdout: osd op queue = debug_random 2024-04-22T18:37:49.152 INFO:teuthology.orchestra.run.smithi121.stdout: osd op queue cut off = debug_random 2024-04-22T18:37:49.152 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub min interval = 60 2024-04-22T18:37:49.152 INFO:teuthology.orchestra.run.smithi121.stdout: osd shutdown pgref assert = True 2024-04-22T18:37:49.152 INFO:teuthology.orchestra.run.smithi121.stdout: osd snap trim sleep = 2 2024-04-22T18:37:49.152 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.153 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2024-04-22T18:37:49.153 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-22T18:37:49.153 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2024-04-22T18:37:49.153 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-04-22T18:37:49.153 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-04-22T18:37:49.153 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-04-22T18:37:49.153 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-04-22T18:37:49.153 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2024-04-22T18:37:49.153 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.153 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2024-04-22T18:37:49.153 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-22T18:37:49.153 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-04-22T18:37:49.153 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2024-04-22T18:37:49.153 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-04-22T18:37:49.153 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2024-04-22T18:37:49.154 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2024-04-22T18:37:49.155 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.155 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2024-04-22T18:37:49.155 INFO:teuthology.orchestra.run.smithi121.stdout: mon min osdmap epochs = 50 2024-04-22T18:37:49.155 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune interval = 2 2024-04-22T18:37:49.155 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune min = 15 2024-04-22T18:37:49.155 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune txsize = 2 2024-04-22T18:37:49.155 INFO:teuthology.orchestra.run.smithi121.stdout: mon scrub interval = 300 2024-04-22T18:37:49.155 INFO:teuthology.orchestra.run.smithi121.stdout: paxos service trim min = 10 2024-04-22T18:37:49.155 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-22T18:37:49.155 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2024-04-22T18:37:49.155 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2024-04-22T18:37:49.155 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2024-04-22T18:37:49.155 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2024-04-22T18:37:49.155 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-22T18:37:49.155 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-22T18:37:49.156 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2024-04-22T18:37:49.156 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2024-04-22T18:37:49.156 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2024-04-22T18:37:49.157 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-22T18:37:49.157 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-22T18:37:49.248 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-22T18:37:49.249 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:37:49.249 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-22T18:37:49.263 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:37:49.263 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-22T18:37:49.318 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi089.front.sepia.ceph.com 2024-04-22T18:37:49.318 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:37:49.318 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-22T18:37:49.318 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T18:37:49.390 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:37:49.391 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-22T18:37:49.445 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2024-04-22T18:37:49.445 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:37:49.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-22T18:37:49.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T18:37:49.478 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:37:49.478 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-22T18:37:49.533 INFO:tasks.ceph:Setting up mon nodes... 2024-04-22T18:37:49.533 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-22T18:37:49.533 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-22T18:37:49.581 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-22T18:37:49.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T18:37:49.643 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T18:37:49.646 INFO:tasks.ceph:Setting up mds nodes... 2024-04-22T18:37:49.646 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-22T18:37:49.649 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T18:37:49.688 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-22T18:37:49.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-22T18:37:49.737 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-22T18:37:49.749 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-22T18:37:49.749 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-22T18:37:49.749 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-22T18:37:49.773 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-22T18:37:49.773 INFO:tasks.ceph:role: osd.0 2024-04-22T18:37:49.773 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-22T18:37:49.773 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-22T18:37:49.843 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T18:37:49.843 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T18:37:49.843 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T18:37:49.843 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T18:37:49.844 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T18:37:49.844 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-04-22T18:37:49.844 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T18:37:49.844 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T18:37:49.844 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T18:37:49.844 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T18:37:50.365 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-04-22T18:37:50.367 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-04-22T18:37:50.367 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-22T18:37:50.496 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-22T18:37:50.527 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-22T18:37:50.593 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-22T18:37:50.593 INFO:tasks.ceph:role: osd.1 2024-04-22T18:37:50.593 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-22T18:37:50.593 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-22T18:37:50.664 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T18:37:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T18:37:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T18:37:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T18:37:50.664 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T18:37:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-04-22T18:37:50.664 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T18:37:50.664 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T18:37:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T18:37:50.664 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T18:37:51.109 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-04-22T18:37:51.112 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-04-22T18:37:51.112 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-22T18:37:51.203 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-22T18:37:51.229 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-22T18:37:51.293 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-22T18:37:51.293 INFO:tasks.ceph:role: osd.2 2024-04-22T18:37:51.293 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-22T18:37:51.293 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-22T18:37:51.363 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T18:37:51.363 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T18:37:51.363 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T18:37:51.364 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T18:37:51.364 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T18:37:51.364 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-04-22T18:37:51.364 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T18:37:51.364 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T18:37:51.364 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T18:37:51.364 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T18:37:51.866 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-04-22T18:37:51.868 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-04-22T18:37:51.868 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-22T18:37:51.995 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-22T18:37:52.022 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-22T18:37:52.086 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-22T18:37:52.086 INFO:tasks.ceph:role: osd.3 2024-04-22T18:37:52.086 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-22T18:37:52.086 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-22T18:37:52.156 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T18:37:52.156 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T18:37:52.156 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T18:37:52.157 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T18:37:52.157 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T18:37:52.157 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-04-22T18:37:52.157 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T18:37:52.157 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T18:37:52.157 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T18:37:52.157 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T18:37:52.681 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-04-22T18:37:52.683 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-04-22T18:37:52.683 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-22T18:37:52.827 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-22T18:37:52.860 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T18:37:52.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:52.957+0000 7f4cc5e12640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-22T18:37:52.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:52.958+0000 7f4cc5e12640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-22T18:37:52.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:52.958+0000 7f4cc5e12640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-22T18:37:52.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:52.958+0000 7f4cc5e12640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-22T18:37:52.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:52.958+0000 7f4cc5e12640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-22T18:37:52.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:52.958+0000 7f4cc5e12640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-22T18:37:55.803 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T18:37:55.830 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T18:37:55.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:55.916+0000 7fee2a298640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-22T18:37:55.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:55.917+0000 7fee2a298640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-22T18:37:55.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:55.917+0000 7fee2a298640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-22T18:37:55.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:55.917+0000 7fee2a298640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-22T18:37:55.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:55.917+0000 7fee2a298640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-22T18:37:55.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:55.917+0000 7fee2a298640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-22T18:37:58.767 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T18:37:58.795 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T18:37:58.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:58.883+0000 7f3ff3f33640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-22T18:37:58.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:58.883+0000 7f3ff3f33640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-22T18:37:58.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:58.883+0000 7f3ff3f33640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-22T18:37:58.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:58.883+0000 7f3ff3f33640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-22T18:37:58.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:58.883+0000 7f3ff3f33640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-22T18:37:58.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:37:58.884+0000 7f3ff3f33640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-22T18:38:01.729 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T18:38:01.754 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T18:38:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:38:01.839+0000 7f6d80a9c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-22T18:38:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:38:01.839+0000 7f6d80a9c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-22T18:38:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:38:01.839+0000 7f6d80a9c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-22T18:38:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:38:01.840+0000 7f6d80a9c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-22T18:38:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:38:01.840+0000 7f6d80a9c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-22T18:38:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-04-22T18:38:01.840+0000 7f6d80a9c640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-22T18:38:04.699 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T18:38:04.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-22T18:38:04.755 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-22T18:38:04.755 INFO:tasks.ceph:role: osd.4 2024-04-22T18:38:04.755 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-22T18:38:04.755 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-22T18:38:04.829 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T18:38:04.829 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T18:38:04.829 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T18:38:04.829 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T18:38:04.829 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T18:38:04.829 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-04-22T18:38:04.829 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T18:38:04.829 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T18:38:04.830 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T18:38:04.830 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T18:38:05.332 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-04-22T18:38:05.335 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-04-22T18:38:05.335 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-22T18:38:05.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-22T18:38:05.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-22T18:38:05.556 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-22T18:38:05.556 INFO:tasks.ceph:role: osd.5 2024-04-22T18:38:05.556 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-22T18:38:05.557 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-22T18:38:05.628 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T18:38:05.628 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T18:38:05.628 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T18:38:05.628 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T18:38:05.628 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T18:38:05.628 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-04-22T18:38:05.629 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T18:38:05.629 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T18:38:05.629 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T18:38:05.629 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T18:38:06.093 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-04-22T18:38:06.095 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-04-22T18:38:06.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-22T18:38:06.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-22T18:38:06.220 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-22T18:38:06.286 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-22T18:38:06.286 INFO:tasks.ceph:role: osd.6 2024-04-22T18:38:06.287 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-22T18:38:06.287 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-22T18:38:06.358 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T18:38:06.359 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T18:38:06.359 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T18:38:06.359 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T18:38:06.359 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T18:38:06.359 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-04-22T18:38:06.359 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T18:38:06.359 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T18:38:06.359 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T18:38:06.359 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T18:38:06.860 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-04-22T18:38:06.863 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-04-22T18:38:06.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-22T18:38:06.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-22T18:38:07.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-22T18:38:07.089 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-22T18:38:07.089 INFO:tasks.ceph:role: osd.7 2024-04-22T18:38:07.090 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-22T18:38:07.090 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-22T18:38:07.162 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T18:38:07.162 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T18:38:07.162 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T18:38:07.162 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T18:38:07.162 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T18:38:07.162 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-04-22T18:38:07.163 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T18:38:07.163 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T18:38:07.163 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T18:38:07.163 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T18:38:07.635 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-04-22T18:38:07.637 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-04-22T18:38:07.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-22T18:38:07.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-22T18:38:07.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T18:38:07.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:07.892+0000 7f4c75a1a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-22T18:38:07.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:07.893+0000 7f4c75a1a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-22T18:38:07.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:07.893+0000 7f4c75a1a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-22T18:38:07.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:07.893+0000 7f4c75a1a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-22T18:38:07.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:07.893+0000 7f4c75a1a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-22T18:38:07.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:07.893+0000 7f4c75a1a640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-22T18:38:10.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-22T18:38:10.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T18:38:10.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:10.860+0000 7fcfa70b5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-22T18:38:10.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:10.860+0000 7fcfa70b5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-22T18:38:10.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:10.860+0000 7fcfa70b5640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-22T18:38:10.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:10.860+0000 7fcfa70b5640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-22T18:38:10.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:10.860+0000 7fcfa70b5640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-22T18:38:10.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:10.861+0000 7fcfa70b5640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-22T18:38:13.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-22T18:38:13.739 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T18:38:13.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:13.828+0000 7f050e807640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-22T18:38:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:13.828+0000 7f050e807640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-22T18:38:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:13.828+0000 7f050e807640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-22T18:38:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:13.828+0000 7f050e807640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-22T18:38:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:13.828+0000 7f050e807640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-22T18:38:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:13.829+0000 7f050e807640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-22T18:38:16.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-22T18:38:16.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T18:38:16.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:16.799+0000 7f340c803640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-22T18:38:16.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:16.799+0000 7f340c803640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-22T18:38:16.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:16.799+0000 7f340c803640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-22T18:38:16.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:16.799+0000 7f340c803640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-22T18:38:16.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:16.799+0000 7f340c803640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-22T18:38:16.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-22T18:38:16.800+0000 7f340c803640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-22T18:38:19.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-22T18:38:19.671 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-22T18:38:19.671 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:38:19.671 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-22T18:38:19.699 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:38:19.699 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-22T18:38:19.766 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:38:19.766 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-22T18:38:19.832 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:38:19.832 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-22T18:38:19.898 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:38:19.898 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-22T18:38:19.963 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:38:19.963 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-22T18:38:19.991 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:38:19.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-22T18:38:20.060 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:38:20.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-22T18:38:20.127 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:38:20.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-22T18:38:20.193 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:38:20.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-22T18:38:20.259 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:38:20.259 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-22T18:38:20.274 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:38:20.274 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-22T18:38:20.315 INFO:tasks.ceph:Adding keys to all mons... 2024-04-22T18:38:20.315 DEBUG:teuthology.orchestra.run.smithi089:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-22T18:38:20.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-22T18:38:20.358 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr.y] 2024-04-22T18:38:20.358 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQB9riZmG9jAIhAAp+zrLJfcVzyBMqAJxcABbA== 2024-04-22T18:38:20.358 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.0] 2024-04-22T18:38:20.358 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQCAriZmCx0ZORAAzuWzwdQtRvuzRzpr2ccZKw== 2024-04-22T18:38:20.358 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.1] 2024-04-22T18:38:20.358 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQCDriZmyoywNhAARuqLOvj/b+cMGkcGnShPYw== 2024-04-22T18:38:20.359 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.2] 2024-04-22T18:38:20.359 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQCGriZm3RivNBAADPBsmJTDo2S4jwC4MuZtRA== 2024-04-22T18:38:20.359 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.3] 2024-04-22T18:38:20.359 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQCJriZmrQkTMhAAHtq8h+RGnlNwZWRXyFtBDg== 2024-04-22T18:38:20.359 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr.x] 2024-04-22T18:38:20.359 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQB9riZmrt2AJhAAEtidmha98NA48hQLPkFvJQ== 2024-04-22T18:38:20.359 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.4] 2024-04-22T18:38:20.359 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQCPriZmj31INRAAZnbNHIai4YisVOS1cDxXjQ== 2024-04-22T18:38:20.359 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.5] 2024-04-22T18:38:20.359 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQCSriZmoZ5cMxAA8FnT0d/dcaOMaJstXXOITg== 2024-04-22T18:38:20.359 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.6] 2024-04-22T18:38:20.360 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQCVriZmQZdvMRAAlOSmUNWJoGGfE0zkq3pgTw== 2024-04-22T18:38:20.360 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.7] 2024-04-22T18:38:20.360 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQCYriZmNCu3LxAACeY8BSfd1iV+vx0BOS+REw== 2024-04-22T18:38:20.360 INFO:teuthology.orchestra.run.smithi089.stdout:[client.0] 2024-04-22T18:38:20.360 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQB9riZm9bQhKRAAb0p/6nuRnDrtTKp3vC7sLg== 2024-04-22T18:38:20.360 INFO:teuthology.orchestra.run.smithi089.stdout:[client.1] 2024-04-22T18:38:20.360 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQB9riZmeQwaLBAAH5r5KBZkuLekoKbRp/Plnw== 2024-04-22T18:38:20.378 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.y] 2024-04-22T18:38:20.378 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQB9riZmG9jAIhAAp+zrLJfcVzyBMqAJxcABbA== 2024-04-22T18:38:20.378 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.0] 2024-04-22T18:38:20.378 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCAriZmCx0ZORAAzuWzwdQtRvuzRzpr2ccZKw== 2024-04-22T18:38:20.379 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.1] 2024-04-22T18:38:20.379 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCDriZmyoywNhAARuqLOvj/b+cMGkcGnShPYw== 2024-04-22T18:38:20.379 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.2] 2024-04-22T18:38:20.379 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCGriZm3RivNBAADPBsmJTDo2S4jwC4MuZtRA== 2024-04-22T18:38:20.379 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.3] 2024-04-22T18:38:20.379 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCJriZmrQkTMhAAHtq8h+RGnlNwZWRXyFtBDg== 2024-04-22T18:38:20.379 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.x] 2024-04-22T18:38:20.379 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQB9riZmrt2AJhAAEtidmha98NA48hQLPkFvJQ== 2024-04-22T18:38:20.379 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.4] 2024-04-22T18:38:20.379 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCPriZmj31INRAAZnbNHIai4YisVOS1cDxXjQ== 2024-04-22T18:38:20.379 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.5] 2024-04-22T18:38:20.379 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCSriZmoZ5cMxAA8FnT0d/dcaOMaJstXXOITg== 2024-04-22T18:38:20.380 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.6] 2024-04-22T18:38:20.380 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCVriZmQZdvMRAAlOSmUNWJoGGfE0zkq3pgTw== 2024-04-22T18:38:20.380 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.7] 2024-04-22T18:38:20.380 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCYriZmNCu3LxAACeY8BSfd1iV+vx0BOS+REw== 2024-04-22T18:38:20.380 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2024-04-22T18:38:20.380 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQB9riZm9bQhKRAAb0p/6nuRnDrtTKp3vC7sLg== 2024-04-22T18:38:20.380 INFO:teuthology.orchestra.run.smithi121.stdout:[client.1] 2024-04-22T18:38:20.380 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQB9riZmeQwaLBAAH5r5KBZkuLekoKbRp/Plnw== 2024-04-22T18:38:20.381 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T18:38:20.404 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T18:38:20.471 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.521 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.571 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.574 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.621 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.671 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T18:38:20.673 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T18:38:20.721 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.771 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.821 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.871 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.873 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T18:38:20.921 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T18:38:20.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T18:38:20.974 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T18:38:20.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T18:38:21.026 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-22T18:38:21.027 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-22T18:38:21.049 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T18:38:21.142 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-22T18:38:21.165 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-22T18:38:21.228 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T18:38:21.321 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-22T18:38:21.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-22T18:38:21.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T18:38:21.462 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-22T18:38:21.485 DEBUG:teuthology.orchestra.run.smithi089:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-22T18:38:21.487 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-22T18:38:21.539 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-22T18:38:21.540 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-22T18:38:21.540 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-22T18:38:21.542 INFO:tasks.ceph.mon.a:Started 2024-04-22T18:38:21.542 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-22T18:38:21.542 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-22T18:38:21.544 INFO:tasks.ceph.mon.c:Started 2024-04-22T18:38:21.544 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-22T18:38:21.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-22T18:38:21.582 INFO:tasks.ceph.mon.b:Started 2024-04-22T18:38:21.582 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-22T18:38:21.583 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-22T18:38:21.583 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-22T18:38:21.585 INFO:tasks.ceph.mgr.y:Started 2024-04-22T18:38:21.585 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-22T18:38:21.585 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-22T18:38:21.587 INFO:tasks.ceph.mgr.x:Started 2024-04-22T18:38:21.587 INFO:tasks.ceph:Setting crush tunables to default 2024-04-22T18:38:21.587 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd crush tunables default 2024-04-22T18:38:26.937 INFO:teuthology.orchestra.run.smithi089.stderr:adjusted tunables profile to default 2024-04-22T18:38:26.949 INFO:tasks.ceph:check_enable_crimson: False 2024-04-22T18:38:26.949 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-22T18:38:26.949 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:38:26.949 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-22T18:38:26.973 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:26.973+0000 7f712761f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T18:38:26.973 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:38:26.973 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-22T18:38:26.979 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:26.980+0000 7ff0adff1040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T18:38:27.002 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:38:27.002 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-22T18:38:27.065 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-04-22T18:38:27.065 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-22T18:38:27.111 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:27.112+0000 7f712761f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T18:38:27.128 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:38:27.128 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-22T18:38:27.133 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:27.134+0000 7ff0adff1040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T18:38:27.151 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:38:27.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-22T18:38:27.165 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:27.166+0000 7f712761f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T18:38:27.186 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:27.187+0000 7ff0adff1040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T18:38:27.187 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:38:27.187 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-22T18:38:27.250 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-22T18:38:27.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-22T18:38:27.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new bbf33cfb-0f07-4df6-b90f-83e9edb039a6 0 2024-04-22T18:38:27.426 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:27.427+0000 7ff0adff1040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T18:38:27.450 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:27.451+0000 7f712761f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T18:38:27.471 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:27.472+0000 7ff0adff1040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T18:38:27.508 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:27.508+0000 7f712761f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T18:38:27.548 INFO:teuthology.orchestra.run.smithi121.stdout:0 2024-04-22T18:38:27.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 30415a8e-9f47-416f-9fd0-716c2cb3f562 1 2024-04-22T18:38:27.740 INFO:teuthology.orchestra.run.smithi121.stdout:1 2024-04-22T18:38:27.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new b8da6b88-8355-4225-930d-bfed881089b7 2 2024-04-22T18:38:27.931 INFO:teuthology.orchestra.run.smithi121.stdout:2 2024-04-22T18:38:27.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 3eaec913-9d7d-4b03-a28f-0c334e7c39f1 3 2024-04-22T18:38:28.121 INFO:teuthology.orchestra.run.smithi121.stdout:3 2024-04-22T18:38:28.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 887acc3b-6f6b-49eb-a039-13050a2c7d64 4 2024-04-22T18:38:28.188 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:28.189+0000 7ff0adff1040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T18:38:28.243 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:28.244+0000 7f712761f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T18:38:28.273 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:28.274+0000 7ff0adff1040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T18:38:28.310 INFO:teuthology.orchestra.run.smithi121.stdout:4 2024-04-22T18:38:28.315 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:28.316+0000 7ff0adff1040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T18:38:28.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new dcaddfd0-f8d1-4c28-bda2-d17eed606b60 5 2024-04-22T18:38:28.333 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:28.334+0000 7f712761f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T18:38:28.378 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:28.378+0000 7f712761f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T18:38:28.478 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:28.479+0000 7ff0adff1040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T18:38:28.521 INFO:teuthology.orchestra.run.smithi121.stdout:5 2024-04-22T18:38:28.532 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new acf0527b-3020-4b0a-b64c-5068d21a5781 6 2024-04-22T18:38:28.533 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:28.534+0000 7ff0adff1040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T18:38:28.571 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:28.571+0000 7f712761f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T18:38:28.585 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:28.586+0000 7ff0adff1040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T18:38:28.629 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:28.630+0000 7f712761f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T18:38:28.685 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:28.686+0000 7f712761f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T18:38:28.696 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:28.697+0000 7ff0adff1040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T18:38:28.751 INFO:teuthology.orchestra.run.smithi121.stdout:6 2024-04-22T18:38:28.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 305394bc-be3d-420b-88d7-d273670540b1 7 2024-04-22T18:38:28.803 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:28.803+0000 7f712761f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T18:38:28.945 INFO:teuthology.orchestra.run.smithi121.stdout:7 2024-04-22T18:38:28.956 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-22T18:38:28.956 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-22T18:38:28.959 INFO:tasks.ceph.osd.0:Started 2024-04-22T18:38:28.959 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-22T18:38:28.959 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-22T18:38:28.961 INFO:tasks.ceph.osd.1:Started 2024-04-22T18:38:28.961 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-22T18:38:28.961 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-22T18:38:28.963 INFO:tasks.ceph.osd.2:Started 2024-04-22T18:38:28.963 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-22T18:38:28.963 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-22T18:38:28.965 INFO:tasks.ceph.osd.3:Started 2024-04-22T18:38:28.965 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-22T18:38:28.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-22T18:38:28.998 INFO:tasks.ceph.osd.4:Started 2024-04-22T18:38:28.998 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-22T18:38:28.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-22T18:38:29.000 INFO:tasks.ceph.osd.5:Started 2024-04-22T18:38:29.000 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-22T18:38:29.001 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-22T18:38:29.002 INFO:tasks.ceph.osd.6:Started 2024-04-22T18:38:29.003 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-22T18:38:29.003 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-22T18:38:29.005 INFO:tasks.ceph.osd.7:Started 2024-04-22T18:38:29.005 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T18:38:29.133 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:29.134+0000 7ff0adff1040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T18:38:29.216 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:29.217+0000 7f712761f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T18:38:29.241 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:38:29.241 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":10,"fsid":"9e25744c-eff1-4a4a-8d47-0dda406b2c35","created":"2024-04-22T18:38:26.841687+0000","modified":"2024-04-22T18:38:28.942710+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T18:38:28.942710+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bbf33cfb-0f07-4df6-b90f-83e9edb039a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"30415a8e-9f47-416f-9fd0-716c2cb3f562","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8da6b88-8355-4225-930d-bfed881089b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3eaec913-9d7d-4b03-a28f-0c334e7c39f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"887acc3b-6f6b-49eb-a039-13050a2c7d64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcaddfd0-f8d1-4c28-bda2-d17eed606b60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"acf0527b-3020-4b0a-b64c-5068d21a5781","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"305394bc-be3d-420b-88d7-d273670540b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T18:38:29.252 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-22T18:38:29.252 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-22T18:38:29.414 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:29.416+0000 7ff0adff1040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T18:38:29.466 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:29.467+0000 7ff0adff1040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T18:38:29.469 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:29.470+0000 7f712761f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T18:38:29.521 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:29.521+0000 7f712761f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T18:38:29.562 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:29.563+0000 7f3e8ca1f640 -1 Falling back to public interface 2024-04-22T18:38:29.565 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:29.566+0000 7f30a369a640 -1 Falling back to public interface 2024-04-22T18:38:29.572 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:29.573+0000 7f8618c0f640 -1 Falling back to public interface 2024-04-22T18:38:29.574 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:29.575+0000 7f1e3c387640 -1 Falling back to public interface 2024-04-22T18:38:29.582 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:29.583+0000 7ff0adff1040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T18:38:29.593 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:29.593+0000 7f3d0da1d640 -1 Falling back to public interface 2024-04-22T18:38:29.600 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:29.601+0000 7fa775a0d640 -1 Falling back to public interface 2024-04-22T18:38:29.618 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:29.619+0000 7f7f5180f640 -1 Falling back to public interface 2024-04-22T18:38:29.630 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:29.631+0000 7fec6881b640 -1 Falling back to public interface 2024-04-22T18:38:29.636 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:29.637+0000 7f712761f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T18:38:29.854 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:29.855+0000 7ff0adff1040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T18:38:29.902 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:29.903+0000 7ff0adff1040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T18:38:29.908 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:29.909+0000 7f712761f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T18:38:29.956 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:29.957+0000 7f712761f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T18:38:30.057 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:30.058+0000 7ff0adff1040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T18:38:30.059 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T18:38:30.109 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:30.110+0000 7f712761f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T18:38:30.200 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:30.201+0000 7ff0adff1040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T18:38:30.219 INFO:teuthology.misc.health.smithi089.stdout: 2024-04-22T18:38:30.219 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":10,"fsid":"9e25744c-eff1-4a4a-8d47-0dda406b2c35","created":"2024-04-22T18:38:26.841687+0000","modified":"2024-04-22T18:38:28.942710+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T18:38:28.942710+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bbf33cfb-0f07-4df6-b90f-83e9edb039a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"30415a8e-9f47-416f-9fd0-716c2cb3f562","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b8da6b88-8355-4225-930d-bfed881089b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3eaec913-9d7d-4b03-a28f-0c334e7c39f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"887acc3b-6f6b-49eb-a039-13050a2c7d64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcaddfd0-f8d1-4c28-bda2-d17eed606b60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"acf0527b-3020-4b0a-b64c-5068d21a5781","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"305394bc-be3d-420b-88d7-d273670540b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T18:38:30.229 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-22T18:38:30.246 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:30.247+0000 7f712761f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T18:38:30.252 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:30.253+0000 7ff0adff1040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T18:38:30.295 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:30.296+0000 7f712761f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T18:38:30.321 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:30.322+0000 7ff0adff1040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T18:38:30.363 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:30.364+0000 7f712761f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T18:38:30.366 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:30.367+0000 7ff0adff1040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T18:38:30.408 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:30.409+0000 7f712761f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T18:38:30.836 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:30.837+0000 7ff0adff1040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T18:38:30.877 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:30.878+0000 7f712761f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T18:38:30.952 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:30.954+0000 7ff0adff1040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T18:38:30.995 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-22T18:38:30.996+0000 7f712761f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T18:38:31.965 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:31.966+0000 7ff04f514640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-22T18:38:31.966 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-04-22T18:38:31.967+0000 7ff04f514640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-22T18:38:33.028 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:33.029+0000 7f3e8ca1f640 -1 osd.0 0 log_to_monitors true 2024-04-22T18:38:33.033 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:33.033+0000 7f3d0da1d640 -1 osd.6 0 log_to_monitors true 2024-04-22T18:38:33.072 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:33.074+0000 7f30a369a640 -1 osd.1 0 log_to_monitors true 2024-04-22T18:38:33.078 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:33.078+0000 7fa775a0d640 -1 osd.5 0 log_to_monitors true 2024-04-22T18:38:33.103 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:33.104+0000 7f8618c0f640 -1 osd.3 0 log_to_monitors true 2024-04-22T18:38:33.107 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:33.109+0000 7f1e3c387640 -1 osd.2 0 log_to_monitors true 2024-04-22T18:38:33.133 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:33.134+0000 7f7f5180f640 -1 osd.4 0 log_to_monitors true 2024-04-22T18:38:33.135 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:33.136+0000 7fec6881b640 -1 osd.7 0 log_to_monitors true 2024-04-22T18:38:34.976 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:34.977+0000 7f3d0947a640 -1 osd.6 0 waiting for initial osdmap 2024-04-22T18:38:34.977 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:34.977+0000 7fa77136f640 -1 osd.5 0 waiting for initial osdmap 2024-04-22T18:38:34.977 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:34.978+0000 7f7f4d1a1640 -1 osd.4 0 waiting for initial osdmap 2024-04-22T18:38:34.977 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:34.978+0000 7fa768d81640 -1 osd.5 0 handle_osd_map injecting map crc failure 2024-04-22T18:38:34.977 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:34.978+0000 7f7f44bb3640 -1 osd.4 0 handle_osd_map injecting map crc failure 2024-04-22T18:38:34.983 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:34.983+0000 7fa76d198640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T18:38:34.983 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:34.984+0000 7f7f48fca640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T18:38:34.986 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:34.987+0000 7f1e38314640 -1 osd.2 0 waiting for initial osdmap 2024-04-22T18:38:34.986 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:34.987+0000 7f1e2fd26640 -1 osd.2 0 handle_osd_map injecting map crc failure 2024-04-22T18:38:34.992 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:34.993+0000 7f1e3413d640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T18:38:35.083 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:35.084+0000 7f3d052a3640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T18:38:35.650 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:35.650+0000 7fec641e4640 -1 osd.7 0 waiting for initial osdmap 2024-04-22T18:38:35.654 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:35.655+0000 7fec5c005640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T18:38:35.815 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:35.816+0000 7f3e8847e640 -1 osd.0 0 waiting for initial osdmap 2024-04-22T18:38:35.816 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:35.818+0000 7f3e7be88640 -1 osd.0 0 handle_osd_map injecting map crc failure 2024-04-22T18:38:35.828 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:35.830+0000 7f309f627640 -1 osd.1 0 waiting for initial osdmap 2024-04-22T18:38:35.829 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:35.830+0000 7f3e8029f640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T18:38:35.830 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:35.831+0000 7f3093031640 -1 osd.1 0 handle_osd_map injecting map crc failure 2024-04-22T18:38:35.836 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:35.837+0000 7f3097448640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T18:38:35.898 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:35.900+0000 7f861452c640 -1 osd.3 0 waiting for initial osdmap 2024-04-22T18:38:35.902 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:35.903+0000 7f860c34d640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T18:38:37.036 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T18:38:37.317 INFO:teuthology.misc.health.smithi089.stdout: 2024-04-22T18:38:37.317 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":14,"fsid":"9e25744c-eff1-4a4a-8d47-0dda406b2c35","created":"2024-04-22T18:38:26.841687+0000","modified":"2024-04-22T18:38:36.991764+0000","last_up_change":"2024-04-22T18:38:35.971820+0000","last_in_change":"2024-04-22T18:38:28.942710+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T18:38:36.991610+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bbf33cfb-0f07-4df6-b90f-83e9edb039a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6800","nonce":31890}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6801","nonce":31890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6803","nonce":31890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6802","nonce":31890}]},"public_addr":"172.21.15.89:6800/31890","cluster_addr":"172.21.15.89:6801/31890","heartbeat_back_addr":"172.21.15.89:6803/31890","heartbeat_front_addr":"172.21.15.89:6802/31890","state":["exists","up"]},{"osd":1,"uuid":"30415a8e-9f47-416f-9fd0-716c2cb3f562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6804","nonce":31899}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6805","nonce":31899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6807","nonce":31899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6806","nonce":31899}]},"public_addr":"172.21.15.89:6804/31899","cluster_addr":"172.21.15.89:6805/31899","heartbeat_back_addr":"172.21.15.89:6807/31899","heartbeat_front_addr":"172.21.15.89:6806/31899","state":["exists","up"]},{"osd":2,"uuid":"b8da6b88-8355-4225-930d-bfed881089b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6812","nonce":31903}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6813","nonce":31903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6815","nonce":31903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6814","nonce":31903}]},"public_addr":"172.21.15.89:6812/31903","cluster_addr":"172.21.15.89:6813/31903","heartbeat_back_addr":"172.21.15.89:6815/31903","heartbeat_front_addr":"172.21.15.89:6814/31903","state":["exists","up"]},{"osd":3,"uuid":"3eaec913-9d7d-4b03-a28f-0c334e7c39f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6808","nonce":31913}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6809","nonce":31913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6811","nonce":31913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6810","nonce":31913}]},"public_addr":"172.21.15.89:6808/31913","cluster_addr":"172.21.15.89:6809/31913","heartbeat_back_addr":"172.21.15.89:6811/31913","heartbeat_front_addr":"172.21.15.89:6810/31913","state":["exists","up"]},{"osd":4,"uuid":"887acc3b-6f6b-49eb-a039-13050a2c7d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":31922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":31922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":31922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":31922}]},"public_addr":"172.21.15.121:6808/31922","cluster_addr":"172.21.15.121:6809/31922","heartbeat_back_addr":"172.21.15.121:6811/31922","heartbeat_front_addr":"172.21.15.121:6810/31922","state":["exists","up"]},{"osd":5,"uuid":"dcaddfd0-f8d1-4c28-bda2-d17eed606b60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":31921}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":31921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":31921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":31921}]},"public_addr":"172.21.15.121:6804/31921","cluster_addr":"172.21.15.121:6805/31921","heartbeat_back_addr":"172.21.15.121:6807/31921","heartbeat_front_addr":"172.21.15.121:6806/31921","state":["exists","up"]},{"osd":6,"uuid":"acf0527b-3020-4b0a-b64c-5068d21a5781","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":31920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":31920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":31920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":31920}]},"public_addr":"172.21.15.121:6800/31920","cluster_addr":"172.21.15.121:6801/31920","heartbeat_back_addr":"172.21.15.121:6803/31920","heartbeat_front_addr":"172.21.15.121:6802/31920","state":["exists","up"]},{"osd":7,"uuid":"305394bc-be3d-420b-88d7-d273670540b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":31939}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":31939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":31939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":31939}]},"public_addr":"172.21.15.121:6812/31939","cluster_addr":"172.21.15.121:6813/31939","heartbeat_back_addr":"172.21.15.121:6815/31939","heartbeat_front_addr":"172.21.15.121:6814/31939","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.023620+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.099729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.074483+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.122275+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.116180+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.108716+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.005302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.108554+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T18:38:37.328 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-22T18:38:37.328 INFO:tasks.ceph:Creating RBD pool 2024-04-22T18:38:37.328 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-22T18:38:38.003 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'rbd' created 2024-04-22T18:38:38.016 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-22T18:38:39.005 INFO:teuthology.orchestra.run.smithi089.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-22T18:38:39.021 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-22T18:38:39.021 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T18:38:39.021 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-22T18:38:39.022 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-22T18:38:39.022 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-22T18:38:39.359 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:38:39.370 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":4,"flags":0,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6816","nonce":31679}]},"active_addr":"172.21.15.89:6816/31679","active_change":"2024-04-22T18:38:30.955790+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4104,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":4168002553}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":715792134}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":1723933404}]}]} 2024-04-22T18:38:39.375 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-22T18:38:39.375 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-22T18:38:39.376 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T18:38:39.656 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:38:39.656 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":16,"fsid":"9e25744c-eff1-4a4a-8d47-0dda406b2c35","created":"2024-04-22T18:38:26.841687+0000","modified":"2024-04-22T18:38:39.001543+0000","last_up_change":"2024-04-22T18:38:35.971820+0000","last_in_change":"2024-04-22T18:38:28.942710+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T18:38:36.991610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T18:38:37.622620+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bbf33cfb-0f07-4df6-b90f-83e9edb039a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6800","nonce":31890}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6801","nonce":31890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6803","nonce":31890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6802","nonce":31890}]},"public_addr":"172.21.15.89:6800/31890","cluster_addr":"172.21.15.89:6801/31890","heartbeat_back_addr":"172.21.15.89:6803/31890","heartbeat_front_addr":"172.21.15.89:6802/31890","state":["exists","up"]},{"osd":1,"uuid":"30415a8e-9f47-416f-9fd0-716c2cb3f562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6804","nonce":31899}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6805","nonce":31899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6807","nonce":31899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6806","nonce":31899}]},"public_addr":"172.21.15.89:6804/31899","cluster_addr":"172.21.15.89:6805/31899","heartbeat_back_addr":"172.21.15.89:6807/31899","heartbeat_front_addr":"172.21.15.89:6806/31899","state":["exists","up"]},{"osd":2,"uuid":"b8da6b88-8355-4225-930d-bfed881089b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6812","nonce":31903}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6813","nonce":31903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6815","nonce":31903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6814","nonce":31903}]},"public_addr":"172.21.15.89:6812/31903","cluster_addr":"172.21.15.89:6813/31903","heartbeat_back_addr":"172.21.15.89:6815/31903","heartbeat_front_addr":"172.21.15.89:6814/31903","state":["exists","up"]},{"osd":3,"uuid":"3eaec913-9d7d-4b03-a28f-0c334e7c39f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6808","nonce":31913}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6809","nonce":31913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6811","nonce":31913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6810","nonce":31913}]},"public_addr":"172.21.15.89:6808/31913","cluster_addr":"172.21.15.89:6809/31913","heartbeat_back_addr":"172.21.15.89:6811/31913","heartbeat_front_addr":"172.21.15.89:6810/31913","state":["exists","up"]},{"osd":4,"uuid":"887acc3b-6f6b-49eb-a039-13050a2c7d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":31922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":31922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":31922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":31922}]},"public_addr":"172.21.15.121:6808/31922","cluster_addr":"172.21.15.121:6809/31922","heartbeat_back_addr":"172.21.15.121:6811/31922","heartbeat_front_addr":"172.21.15.121:6810/31922","state":["exists","up"]},{"osd":5,"uuid":"dcaddfd0-f8d1-4c28-bda2-d17eed606b60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":31921}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":31921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":31921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":31921}]},"public_addr":"172.21.15.121:6804/31921","cluster_addr":"172.21.15.121:6805/31921","heartbeat_back_addr":"172.21.15.121:6807/31921","heartbeat_front_addr":"172.21.15.121:6806/31921","state":["exists","up"]},{"osd":6,"uuid":"acf0527b-3020-4b0a-b64c-5068d21a5781","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":31920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":31920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":31920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":31920}]},"public_addr":"172.21.15.121:6800/31920","cluster_addr":"172.21.15.121:6801/31920","heartbeat_back_addr":"172.21.15.121:6803/31920","heartbeat_front_addr":"172.21.15.121:6802/31920","state":["exists","up"]},{"osd":7,"uuid":"305394bc-be3d-420b-88d7-d273670540b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":31939}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":31939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":31939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":31939}]},"public_addr":"172.21.15.121:6812/31939","cluster_addr":"172.21.15.121:6813/31939","heartbeat_back_addr":"172.21.15.121:6815/31939","heartbeat_front_addr":"172.21.15.121:6814/31939","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.023620+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.099729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.074483+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.122275+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.116180+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.108716+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.005302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.108554+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T18:38:39.668 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-22T18:38:39.669 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T18:38:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:38:39.949 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":16,"fsid":"9e25744c-eff1-4a4a-8d47-0dda406b2c35","created":"2024-04-22T18:38:26.841687+0000","modified":"2024-04-22T18:38:39.001543+0000","last_up_change":"2024-04-22T18:38:35.971820+0000","last_in_change":"2024-04-22T18:38:28.942710+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T18:38:36.991610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T18:38:37.622620+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bbf33cfb-0f07-4df6-b90f-83e9edb039a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6800","nonce":31890}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6801","nonce":31890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6803","nonce":31890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6802","nonce":31890}]},"public_addr":"172.21.15.89:6800/31890","cluster_addr":"172.21.15.89:6801/31890","heartbeat_back_addr":"172.21.15.89:6803/31890","heartbeat_front_addr":"172.21.15.89:6802/31890","state":["exists","up"]},{"osd":1,"uuid":"30415a8e-9f47-416f-9fd0-716c2cb3f562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6804","nonce":31899}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6805","nonce":31899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6807","nonce":31899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6806","nonce":31899}]},"public_addr":"172.21.15.89:6804/31899","cluster_addr":"172.21.15.89:6805/31899","heartbeat_back_addr":"172.21.15.89:6807/31899","heartbeat_front_addr":"172.21.15.89:6806/31899","state":["exists","up"]},{"osd":2,"uuid":"b8da6b88-8355-4225-930d-bfed881089b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6812","nonce":31903}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6813","nonce":31903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6815","nonce":31903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6814","nonce":31903}]},"public_addr":"172.21.15.89:6812/31903","cluster_addr":"172.21.15.89:6813/31903","heartbeat_back_addr":"172.21.15.89:6815/31903","heartbeat_front_addr":"172.21.15.89:6814/31903","state":["exists","up"]},{"osd":3,"uuid":"3eaec913-9d7d-4b03-a28f-0c334e7c39f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6808","nonce":31913}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6809","nonce":31913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6811","nonce":31913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6810","nonce":31913}]},"public_addr":"172.21.15.89:6808/31913","cluster_addr":"172.21.15.89:6809/31913","heartbeat_back_addr":"172.21.15.89:6811/31913","heartbeat_front_addr":"172.21.15.89:6810/31913","state":["exists","up"]},{"osd":4,"uuid":"887acc3b-6f6b-49eb-a039-13050a2c7d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":31922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":31922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":31922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":31922}]},"public_addr":"172.21.15.121:6808/31922","cluster_addr":"172.21.15.121:6809/31922","heartbeat_back_addr":"172.21.15.121:6811/31922","heartbeat_front_addr":"172.21.15.121:6810/31922","state":["exists","up"]},{"osd":5,"uuid":"dcaddfd0-f8d1-4c28-bda2-d17eed606b60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":31921}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":31921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":31921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":31921}]},"public_addr":"172.21.15.121:6804/31921","cluster_addr":"172.21.15.121:6805/31921","heartbeat_back_addr":"172.21.15.121:6807/31921","heartbeat_front_addr":"172.21.15.121:6806/31921","state":["exists","up"]},{"osd":6,"uuid":"acf0527b-3020-4b0a-b64c-5068d21a5781","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":31920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":31920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":31920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":31920}]},"public_addr":"172.21.15.121:6800/31920","cluster_addr":"172.21.15.121:6801/31920","heartbeat_back_addr":"172.21.15.121:6803/31920","heartbeat_front_addr":"172.21.15.121:6802/31920","state":["exists","up"]},{"osd":7,"uuid":"305394bc-be3d-420b-88d7-d273670540b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":31939}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":31939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":31939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":31939}]},"public_addr":"172.21.15.121:6812/31939","cluster_addr":"172.21.15.121:6813/31939","heartbeat_back_addr":"172.21.15.121:6815/31939","heartbeat_front_addr":"172.21.15.121:6814/31939","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.023620+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.099729+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.074483+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.122275+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.116180+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.108716+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.005302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:38:34.108554+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T18:38:39.962 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T18:38:39.962 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-22T18:38:39.963 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-22T18:38:39.963 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-22T18:38:39.964 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-22T18:38:39.964 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-22T18:38:39.964 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-22T18:38:39.965 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-22T18:38:40.008 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:40.009+0000 7fec57bee640 -1 osd.7 16 handle_osd_map injecting map crc failure 2024-04-22T18:38:40.190 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:40.190 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-22T18:38:40.197 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:40.198 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T18:38:40.200 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:40.201 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-22T18:38:40.202 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:40.202 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-22T18:38:40.204 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:40.204 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T18:38:40.205 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:40.205 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-22T18:38:40.211 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:40.211 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-22T18:38:40.214 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:40.214 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T18:38:40.778 INFO:teuthology.orchestra.run.smithi089.stdout:55834574849 2024-04-22T18:38:40.782 INFO:teuthology.orchestra.run.smithi089.stdout:55834574849 2024-04-22T18:38:40.784 INFO:teuthology.orchestra.run.smithi089.stdout:55834574849 2024-04-22T18:38:40.798 INFO:teuthology.orchestra.run.smithi089.stdout:55834574849 2024-04-22T18:38:40.800 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.6 2024-04-22T18:38:40.802 INFO:teuthology.orchestra.run.smithi089.stdout:55834574849 2024-04-22T18:38:40.806 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.5 2024-04-22T18:38:40.806 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.0 2024-04-22T18:38:40.806 INFO:teuthology.orchestra.run.smithi089.stdout:55834574849 2024-04-22T18:38:40.818 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.2 2024-04-22T18:38:40.820 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.4 2024-04-22T18:38:40.821 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.7 2024-04-22T18:38:40.824 INFO:teuthology.orchestra.run.smithi089.stdout:55834574849 2024-04-22T18:38:40.836 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.3 2024-04-22T18:38:40.837 INFO:teuthology.orchestra.run.smithi089.stdout:55834574849 2024-04-22T18:38:40.849 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.1 2024-04-22T18:38:41.801 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-22T18:38:41.806 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-22T18:38:41.807 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T18:38:41.819 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T18:38:41.821 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-22T18:38:41.821 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-22T18:38:41.837 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-22T18:38:41.850 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T18:38:42.378 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:42.397 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.6 2024-04-22T18:38:42.397 DEBUG:teuthology.parallel:result is None 2024-04-22T18:38:42.415 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:42.427 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.2 2024-04-22T18:38:42.427 DEBUG:teuthology.parallel:result is None 2024-04-22T18:38:42.441 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:42.447 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:42.457 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.5 2024-04-22T18:38:42.457 DEBUG:teuthology.parallel:result is None 2024-04-22T18:38:42.459 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:42.463 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.0 2024-04-22T18:38:42.463 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:42.463 DEBUG:teuthology.parallel:result is None 2024-04-22T18:38:42.463 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:42.477 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.7 2024-04-22T18:38:42.477 DEBUG:teuthology.parallel:result is None 2024-04-22T18:38:42.478 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.3 2024-04-22T18:38:42.478 DEBUG:teuthology.parallel:result is None 2024-04-22T18:38:42.480 INFO:teuthology.orchestra.run.smithi089.stdout:55834574850 2024-04-22T18:38:42.482 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.4 2024-04-22T18:38:42.482 DEBUG:teuthology.parallel:result is None 2024-04-22T18:38:42.492 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.1 2024-04-22T18:38:42.493 DEBUG:teuthology.parallel:result is None 2024-04-22T18:38:42.493 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T18:38:42.493 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:38:42.807 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:38:42.807 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:38:42.819 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-22T18:38:40.961215+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":45,"num_write_kb":263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23,"ondisk_log_size":23,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":217156,"kb_used_data":1000,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757564,"statfs":{"total":773094113280,"available":772871745536,"internally_reserved":0,"allocated":1024000,"data_stored":464176,"data_compressed":6340,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:39.026612+0000","last_change":"2024-04-22T18:38:39.026078+0000","last_active":"2024-04-22T18:38:39.026612+0000","last_peered":"2024-04-22T18:38:39.026612+0000","last_clean":"2024-04-22T18:38:39.026612+0000","last_became_active":"2024-04-22T18:38:39.025741+0000","last_became_peered":"2024-04-22T18:38:39.025741+0000","last_unstale":"2024-04-22T18:38:39.026612+0000","last_undegraded":"2024-04-22T18:38:39.026612+0000","last_fullsized":"2024-04-22T18:38:39.026612+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:01.125349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:39.110822+0000","last_change":"2024-04-22T18:38:39.110499+0000","last_active":"2024-04-22T18:38:39.110822+0000","last_peered":"2024-04-22T18:38:39.110822+0000","last_clean":"2024-04-22T18:38:39.110822+0000","last_became_active":"2024-04-22T18:38:39.010173+0000","last_became_peered":"2024-04-22T18:38:39.010173+0000","last_unstale":"2024-04-22T18:38:39.110822+0000","last_undegraded":"2024-04-22T18:38:39.110822+0000","last_fullsized":"2024-04-22T18:38:39.110822+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:58.222501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.118329+0000","last_change":"2024-04-22T18:38:39.210344+0000","last_active":"2024-04-22T18:38:40.118329+0000","last_peered":"2024-04-22T18:38:40.118329+0000","last_clean":"2024-04-22T18:38:40.118329+0000","last_became_active":"2024-04-22T18:38:39.210049+0000","last_became_peered":"2024-04-22T18:38:39.210049+0000","last_unstale":"2024-04-22T18:38:40.118329+0000","last_undegraded":"2024-04-22T18:38:40.118329+0000","last_fullsized":"2024-04-22T18:38:40.118329+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.878594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:39.308304+0000","last_change":"2024-04-22T18:38:39.307735+0000","last_active":"2024-04-22T18:38:39.308304+0000","last_peered":"2024-04-22T18:38:39.308304+0000","last_clean":"2024-04-22T18:38:39.308304+0000","last_became_active":"2024-04-22T18:38:39.307459+0000","last_became_peered":"2024-04-22T18:38:39.307459+0000","last_unstale":"2024-04-22T18:38:39.308304+0000","last_undegraded":"2024-04-22T18:38:39.308304+0000","last_fullsized":"2024-04-22T18:38:39.308304+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.680561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:39.113535+0000","last_change":"2024-04-22T18:38:39.111223+0000","last_active":"2024-04-22T18:38:39.113535+0000","last_peered":"2024-04-22T18:38:39.113535+0000","last_clean":"2024-04-22T18:38:39.113535+0000","last_became_active":"2024-04-22T18:38:39.110914+0000","last_became_peered":"2024-04-22T18:38:39.110914+0000","last_unstale":"2024-04-22T18:38:39.113535+0000","last_undegraded":"2024-04-22T18:38:39.113535+0000","last_fullsized":"2024-04-22T18:38:39.113535+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.570327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:39.164727+0000","last_change":"2024-04-22T18:38:39.164369+0000","last_active":"2024-04-22T18:38:39.164727+0000","last_peered":"2024-04-22T18:38:39.164727+0000","last_clean":"2024-04-22T18:38:39.164727+0000","last_became_active":"2024-04-22T18:38:39.164089+0000","last_became_peered":"2024-04-22T18:38:39.164089+0000","last_unstale":"2024-04-22T18:38:39.164727+0000","last_undegraded":"2024-04-22T18:38:39.164727+0000","last_fullsized":"2024-04-22T18:38:39.164727+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:44.825470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.018146+0000","last_change":"2024-04-22T18:38:39.125885+0000","last_active":"2024-04-22T18:38:40.018146+0000","last_peered":"2024-04-22T18:38:40.018146+0000","last_clean":"2024-04-22T18:38:40.018146+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:38:40.018146+0000","last_undegraded":"2024-04-22T18:38:40.018146+0000","last_fullsized":"2024-04-22T18:38:40.018146+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.321452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:39.010974+0000","last_change":"2024-04-22T18:38:39.010398+0000","last_active":"2024-04-22T18:38:39.010974+0000","last_peered":"2024-04-22T18:38:39.010974+0000","last_clean":"2024-04-22T18:38:39.010974+0000","last_became_active":"2024-04-22T18:38:39.010101+0000","last_became_peered":"2024-04-22T18:38:39.010101+0000","last_unstale":"2024-04-22T18:38:39.010974+0000","last_undegraded":"2024-04-22T18:38:39.010974+0000","last_fullsized":"2024-04-22T18:38:39.010974+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.708134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'23","reported_seq":43,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.147186+0000","last_change":"2024-04-22T18:38:38.109358+0000","last_active":"2024-04-22T18:38:40.147186+0000","last_peered":"2024-04-22T18:38:40.147186+0000","last_clean":"2024-04-22T18:38:40.147186+0000","last_became_active":"2024-04-22T18:38:38.108568+0000","last_became_peered":"2024-04-22T18:38:38.108568+0000","last_unstale":"2024-04-22T18:38:40.147186+0000","last_undegraded":"2024-04-22T18:38:40.147186+0000","last_fullsized":"2024-04-22T18:38:40.147186+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:38:36.991764+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:46.436569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":45,"num_write_kb":263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":45,"num_write_kb":263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":278552,"data_compressed":12680,"data_compressed_allocated":16384,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},"log_size":23,"ondisk_log_size":23,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574850,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":42078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":85.509}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":85.391000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":41566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]}]},{"osd":5,"up_from":13,"seq":55834574850,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27112,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":98304,"data_stored":37310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27124,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344716,"statfs":{"total":96636764160,"available":96608989184,"internally_reserved":0,"allocated":106496,"data_stored":41938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":41566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27112,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":98304,"data_stored":37310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574850,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":41566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":159744,"data_stored":180842,"data_compressed":6340,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":87.209999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6340,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6340,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:38:42.821 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:38:43.101 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:38:43.102 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:38:43.113 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-22T18:38:42.961526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":56,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30,"ondisk_log_size":30,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":217192,"kb_used_data":1028,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757528,"statfs":{"total":773094113280,"available":772871708672,"internally_reserved":0,"allocated":1052672,"data_stored":611614,"data_compressed":12680,"data_compressed_allocated":16384,"data_compressed_original":262144,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":11,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000310"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.185188+0000","last_change":"2024-04-22T18:38:39.026078+0000","last_active":"2024-04-22T18:38:40.185188+0000","last_peered":"2024-04-22T18:38:40.185188+0000","last_clean":"2024-04-22T18:38:40.185188+0000","last_became_active":"2024-04-22T18:38:39.025741+0000","last_became_peered":"2024-04-22T18:38:39.025741+0000","last_unstale":"2024-04-22T18:38:40.185188+0000","last_undegraded":"2024-04-22T18:38:40.185188+0000","last_fullsized":"2024-04-22T18:38:40.185188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:01.125349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:39.110822+0000","last_change":"2024-04-22T18:38:39.110499+0000","last_active":"2024-04-22T18:38:39.110822+0000","last_peered":"2024-04-22T18:38:39.110822+0000","last_clean":"2024-04-22T18:38:39.110822+0000","last_became_active":"2024-04-22T18:38:39.010173+0000","last_became_peered":"2024-04-22T18:38:39.010173+0000","last_unstale":"2024-04-22T18:38:39.110822+0000","last_undegraded":"2024-04-22T18:38:39.110822+0000","last_fullsized":"2024-04-22T18:38:39.110822+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:58.222501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.118329+0000","last_change":"2024-04-22T18:38:39.210344+0000","last_active":"2024-04-22T18:38:40.118329+0000","last_peered":"2024-04-22T18:38:40.118329+0000","last_clean":"2024-04-22T18:38:40.118329+0000","last_became_active":"2024-04-22T18:38:39.210049+0000","last_became_peered":"2024-04-22T18:38:39.210049+0000","last_unstale":"2024-04-22T18:38:40.118329+0000","last_undegraded":"2024-04-22T18:38:40.118329+0000","last_fullsized":"2024-04-22T18:38:40.118329+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.878594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:39.308304+0000","last_change":"2024-04-22T18:38:39.307735+0000","last_active":"2024-04-22T18:38:39.308304+0000","last_peered":"2024-04-22T18:38:39.308304+0000","last_clean":"2024-04-22T18:38:39.308304+0000","last_became_active":"2024-04-22T18:38:39.307459+0000","last_became_peered":"2024-04-22T18:38:39.307459+0000","last_unstale":"2024-04-22T18:38:39.308304+0000","last_undegraded":"2024-04-22T18:38:39.308304+0000","last_fullsized":"2024-04-22T18:38:39.308304+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.680561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.179144+0000","last_change":"2024-04-22T18:38:39.111223+0000","last_active":"2024-04-22T18:38:40.179144+0000","last_peered":"2024-04-22T18:38:40.179144+0000","last_clean":"2024-04-22T18:38:40.179144+0000","last_became_active":"2024-04-22T18:38:39.110914+0000","last_became_peered":"2024-04-22T18:38:39.110914+0000","last_unstale":"2024-04-22T18:38:40.179144+0000","last_undegraded":"2024-04-22T18:38:40.179144+0000","last_fullsized":"2024-04-22T18:38:40.179144+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.570327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:39.164727+0000","last_change":"2024-04-22T18:38:39.164369+0000","last_active":"2024-04-22T18:38:39.164727+0000","last_peered":"2024-04-22T18:38:39.164727+0000","last_clean":"2024-04-22T18:38:39.164727+0000","last_became_active":"2024-04-22T18:38:39.164089+0000","last_became_peered":"2024-04-22T18:38:39.164089+0000","last_unstale":"2024-04-22T18:38:39.164727+0000","last_undegraded":"2024-04-22T18:38:39.164727+0000","last_fullsized":"2024-04-22T18:38:39.164727+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:44.825470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.018146+0000","last_change":"2024-04-22T18:38:39.125885+0000","last_active":"2024-04-22T18:38:40.018146+0000","last_peered":"2024-04-22T18:38:40.018146+0000","last_clean":"2024-04-22T18:38:40.018146+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:38:40.018146+0000","last_undegraded":"2024-04-22T18:38:40.018146+0000","last_fullsized":"2024-04-22T18:38:40.018146+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.321452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.179177+0000","last_change":"2024-04-22T18:38:39.010398+0000","last_active":"2024-04-22T18:38:40.179177+0000","last_peered":"2024-04-22T18:38:40.179177+0000","last_clean":"2024-04-22T18:38:40.179177+0000","last_became_active":"2024-04-22T18:38:39.010101+0000","last_became_peered":"2024-04-22T18:38:39.010101+0000","last_unstale":"2024-04-22T18:38:40.179177+0000","last_undegraded":"2024-04-22T18:38:40.179177+0000","last_fullsized":"2024-04-22T18:38:40.179177+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.708134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'30","reported_seq":56,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:41.067411+0000","last_change":"2024-04-22T18:38:38.109358+0000","last_active":"2024-04-22T18:38:41.067411+0000","last_peered":"2024-04-22T18:38:41.067411+0000","last_clean":"2024-04-22T18:38:41.067411+0000","last_became_active":"2024-04-22T18:38:38.108568+0000","last_became_peered":"2024-04-22T18:38:38.108568+0000","last_unstale":"2024-04-22T18:38:41.067411+0000","last_undegraded":"2024-04-22T18:38:41.067411+0000","last_fullsized":"2024-04-22T18:38:41.067411+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:38:36.991764+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:46.436569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":56,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":56,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":278552,"data_compressed":12730,"data_compressed_allocated":16384,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},"log_size":30,"ondisk_log_size":30,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":184888,"data_compressed":6340,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":85.509}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":85.391000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":41566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27112,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":98304,"data_stored":37310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":74.019999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27124,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344716,"statfs":{"total":96636764160,"available":96608989184,"internally_reserved":0,"allocated":106496,"data_stored":41938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":46194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27112,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":98304,"data_stored":37310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":41566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":159744,"data_stored":180842,"data_compressed":6340,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":87.209999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6365,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6365,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:38:43.115 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T18:38:43.115 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T18:38:43.115 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-22T18:38:43.115 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-22T18:38:43.428 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:38:43.429 INFO:teuthology.orchestra.run.smithi089.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T18:38:43.441 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-22T18:38:43.441 INFO:teuthology.run_tasks:Running task exec... 2024-04-22T18:38:43.451 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T18:38:43.451 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi089.front.sepia.ceph.com 2024-04-22T18:38:43.451 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! ceph balancer status ; do sleep 1 ; done' 2024-04-22T18:38:43.727 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:43.727 INFO:teuthology.orchestra.run.smithi089.stdout: "active": true, 2024-04-22T18:38:43.728 INFO:teuthology.orchestra.run.smithi089.stdout: "last_optimize_duration": "0:00:00.000407", 2024-04-22T18:38:43.728 INFO:teuthology.orchestra.run.smithi089.stdout: "last_optimize_started": "Mon Apr 22 18:38:30 2024", 2024-04-22T18:38:43.728 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "upmap", 2024-04-22T18:38:43.728 INFO:teuthology.orchestra.run.smithi089.stdout: "no_optimization_needed": false, 2024-04-22T18:38:43.728 INFO:teuthology.orchestra.run.smithi089.stdout: "optimize_result": "No pools available", 2024-04-22T18:38:43.728 INFO:teuthology.orchestra.run.smithi089.stdout: "plans": [] 2024-04-22T18:38:43.728 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:43.740 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode crush-compat' 2024-04-22T18:38:44.064 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2024-04-22T18:38:44.354 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-22T18:38:44.367 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-22T18:38:44.367 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuri2-testing-2024-04-17-0823-reef'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 2, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': '6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri2-testing-2024-04-17-0823-reef', 'sha1': '6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0'}} 2024-04-22T18:38:44.367 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-04-22T18:38:44.367 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-22T18:38:44.367 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T18:38:44.368 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:38:44.644 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:38:44.644 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:38:44.656 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-04-22T18:38:44.038881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":56,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30,"ondisk_log_size":30,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":217192,"kb_used_data":1028,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757528,"statfs":{"total":773094113280,"available":772871708672,"internally_reserved":0,"allocated":1052672,"data_stored":611614,"data_compressed":12680,"data_compressed_allocated":16384,"data_compressed_original":262144,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":11,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.077665"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.185188+0000","last_change":"2024-04-22T18:38:39.026078+0000","last_active":"2024-04-22T18:38:40.185188+0000","last_peered":"2024-04-22T18:38:40.185188+0000","last_clean":"2024-04-22T18:38:40.185188+0000","last_became_active":"2024-04-22T18:38:39.025741+0000","last_became_peered":"2024-04-22T18:38:39.025741+0000","last_unstale":"2024-04-22T18:38:40.185188+0000","last_undegraded":"2024-04-22T18:38:40.185188+0000","last_fullsized":"2024-04-22T18:38:40.185188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:01.125349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:39.110822+0000","last_change":"2024-04-22T18:38:39.110499+0000","last_active":"2024-04-22T18:38:39.110822+0000","last_peered":"2024-04-22T18:38:39.110822+0000","last_clean":"2024-04-22T18:38:39.110822+0000","last_became_active":"2024-04-22T18:38:39.010173+0000","last_became_peered":"2024-04-22T18:38:39.010173+0000","last_unstale":"2024-04-22T18:38:39.110822+0000","last_undegraded":"2024-04-22T18:38:39.110822+0000","last_fullsized":"2024-04-22T18:38:39.110822+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:58.222501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.118329+0000","last_change":"2024-04-22T18:38:39.210344+0000","last_active":"2024-04-22T18:38:40.118329+0000","last_peered":"2024-04-22T18:38:40.118329+0000","last_clean":"2024-04-22T18:38:40.118329+0000","last_became_active":"2024-04-22T18:38:39.210049+0000","last_became_peered":"2024-04-22T18:38:39.210049+0000","last_unstale":"2024-04-22T18:38:40.118329+0000","last_undegraded":"2024-04-22T18:38:40.118329+0000","last_fullsized":"2024-04-22T18:38:40.118329+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.878594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:39.308304+0000","last_change":"2024-04-22T18:38:39.307735+0000","last_active":"2024-04-22T18:38:39.308304+0000","last_peered":"2024-04-22T18:38:39.308304+0000","last_clean":"2024-04-22T18:38:39.308304+0000","last_became_active":"2024-04-22T18:38:39.307459+0000","last_became_peered":"2024-04-22T18:38:39.307459+0000","last_unstale":"2024-04-22T18:38:39.308304+0000","last_undegraded":"2024-04-22T18:38:39.308304+0000","last_fullsized":"2024-04-22T18:38:39.308304+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.680561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.179144+0000","last_change":"2024-04-22T18:38:39.111223+0000","last_active":"2024-04-22T18:38:40.179144+0000","last_peered":"2024-04-22T18:38:40.179144+0000","last_clean":"2024-04-22T18:38:40.179144+0000","last_became_active":"2024-04-22T18:38:39.110914+0000","last_became_peered":"2024-04-22T18:38:39.110914+0000","last_unstale":"2024-04-22T18:38:40.179144+0000","last_undegraded":"2024-04-22T18:38:40.179144+0000","last_fullsized":"2024-04-22T18:38:40.179144+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.570327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:39.164727+0000","last_change":"2024-04-22T18:38:39.164369+0000","last_active":"2024-04-22T18:38:39.164727+0000","last_peered":"2024-04-22T18:38:39.164727+0000","last_clean":"2024-04-22T18:38:39.164727+0000","last_became_active":"2024-04-22T18:38:39.164089+0000","last_became_peered":"2024-04-22T18:38:39.164089+0000","last_unstale":"2024-04-22T18:38:39.164727+0000","last_undegraded":"2024-04-22T18:38:39.164727+0000","last_fullsized":"2024-04-22T18:38:39.164727+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:44.825470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.018146+0000","last_change":"2024-04-22T18:38:39.125885+0000","last_active":"2024-04-22T18:38:40.018146+0000","last_peered":"2024-04-22T18:38:40.018146+0000","last_clean":"2024-04-22T18:38:40.018146+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:38:40.018146+0000","last_undegraded":"2024-04-22T18:38:40.018146+0000","last_fullsized":"2024-04-22T18:38:40.018146+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.321452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.179177+0000","last_change":"2024-04-22T18:38:39.010398+0000","last_active":"2024-04-22T18:38:40.179177+0000","last_peered":"2024-04-22T18:38:40.179177+0000","last_clean":"2024-04-22T18:38:40.179177+0000","last_became_active":"2024-04-22T18:38:39.010101+0000","last_became_peered":"2024-04-22T18:38:39.010101+0000","last_unstale":"2024-04-22T18:38:40.179177+0000","last_undegraded":"2024-04-22T18:38:40.179177+0000","last_fullsized":"2024-04-22T18:38:40.179177+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.708134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'30","reported_seq":56,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:41.067411+0000","last_change":"2024-04-22T18:38:38.109358+0000","last_active":"2024-04-22T18:38:41.067411+0000","last_peered":"2024-04-22T18:38:41.067411+0000","last_clean":"2024-04-22T18:38:41.067411+0000","last_became_active":"2024-04-22T18:38:38.108568+0000","last_became_peered":"2024-04-22T18:38:38.108568+0000","last_unstale":"2024-04-22T18:38:41.067411+0000","last_undegraded":"2024-04-22T18:38:41.067411+0000","last_fullsized":"2024-04-22T18:38:41.067411+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:38:36.991764+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:46.436569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":56,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":56,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":278552,"data_compressed":12730,"data_compressed_allocated":16384,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},"log_size":30,"ondisk_log_size":30,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":184888,"data_compressed":6340,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":85.509}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":85.391000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":41566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27112,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":98304,"data_stored":37310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":74.019999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27124,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344716,"statfs":{"total":96636764160,"available":96608989184,"internally_reserved":0,"allocated":106496,"data_stored":41938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":46194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27112,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":98304,"data_stored":37310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":41566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":159744,"data_stored":180842,"data_compressed":6340,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":87.209999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6365,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6365,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:38:44.657 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:38:44.935 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:38:44.936 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:38:44.947 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-04-22T18:38:44.038881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":56,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30,"ondisk_log_size":30,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":217192,"kb_used_data":1028,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757528,"statfs":{"total":773094113280,"available":772871708672,"internally_reserved":0,"allocated":1052672,"data_stored":611614,"data_compressed":12680,"data_compressed_allocated":16384,"data_compressed_original":262144,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":11,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.077665"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.185188+0000","last_change":"2024-04-22T18:38:39.026078+0000","last_active":"2024-04-22T18:38:40.185188+0000","last_peered":"2024-04-22T18:38:40.185188+0000","last_clean":"2024-04-22T18:38:40.185188+0000","last_became_active":"2024-04-22T18:38:39.025741+0000","last_became_peered":"2024-04-22T18:38:39.025741+0000","last_unstale":"2024-04-22T18:38:40.185188+0000","last_undegraded":"2024-04-22T18:38:40.185188+0000","last_fullsized":"2024-04-22T18:38:40.185188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:01.125349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:39.110822+0000","last_change":"2024-04-22T18:38:39.110499+0000","last_active":"2024-04-22T18:38:39.110822+0000","last_peered":"2024-04-22T18:38:39.110822+0000","last_clean":"2024-04-22T18:38:39.110822+0000","last_became_active":"2024-04-22T18:38:39.010173+0000","last_became_peered":"2024-04-22T18:38:39.010173+0000","last_unstale":"2024-04-22T18:38:39.110822+0000","last_undegraded":"2024-04-22T18:38:39.110822+0000","last_fullsized":"2024-04-22T18:38:39.110822+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:58.222501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.118329+0000","last_change":"2024-04-22T18:38:39.210344+0000","last_active":"2024-04-22T18:38:40.118329+0000","last_peered":"2024-04-22T18:38:40.118329+0000","last_clean":"2024-04-22T18:38:40.118329+0000","last_became_active":"2024-04-22T18:38:39.210049+0000","last_became_peered":"2024-04-22T18:38:39.210049+0000","last_unstale":"2024-04-22T18:38:40.118329+0000","last_undegraded":"2024-04-22T18:38:40.118329+0000","last_fullsized":"2024-04-22T18:38:40.118329+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.878594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:39.308304+0000","last_change":"2024-04-22T18:38:39.307735+0000","last_active":"2024-04-22T18:38:39.308304+0000","last_peered":"2024-04-22T18:38:39.308304+0000","last_clean":"2024-04-22T18:38:39.308304+0000","last_became_active":"2024-04-22T18:38:39.307459+0000","last_became_peered":"2024-04-22T18:38:39.307459+0000","last_unstale":"2024-04-22T18:38:39.308304+0000","last_undegraded":"2024-04-22T18:38:39.308304+0000","last_fullsized":"2024-04-22T18:38:39.308304+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.680561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.179144+0000","last_change":"2024-04-22T18:38:39.111223+0000","last_active":"2024-04-22T18:38:40.179144+0000","last_peered":"2024-04-22T18:38:40.179144+0000","last_clean":"2024-04-22T18:38:40.179144+0000","last_became_active":"2024-04-22T18:38:39.110914+0000","last_became_peered":"2024-04-22T18:38:39.110914+0000","last_unstale":"2024-04-22T18:38:40.179144+0000","last_undegraded":"2024-04-22T18:38:40.179144+0000","last_fullsized":"2024-04-22T18:38:40.179144+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.570327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:39.164727+0000","last_change":"2024-04-22T18:38:39.164369+0000","last_active":"2024-04-22T18:38:39.164727+0000","last_peered":"2024-04-22T18:38:39.164727+0000","last_clean":"2024-04-22T18:38:39.164727+0000","last_became_active":"2024-04-22T18:38:39.164089+0000","last_became_peered":"2024-04-22T18:38:39.164089+0000","last_unstale":"2024-04-22T18:38:39.164727+0000","last_undegraded":"2024-04-22T18:38:39.164727+0000","last_fullsized":"2024-04-22T18:38:39.164727+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:44.825470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.018146+0000","last_change":"2024-04-22T18:38:39.125885+0000","last_active":"2024-04-22T18:38:40.018146+0000","last_peered":"2024-04-22T18:38:40.018146+0000","last_clean":"2024-04-22T18:38:40.018146+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:38:40.018146+0000","last_undegraded":"2024-04-22T18:38:40.018146+0000","last_fullsized":"2024-04-22T18:38:40.018146+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.321452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.179177+0000","last_change":"2024-04-22T18:38:39.010398+0000","last_active":"2024-04-22T18:38:40.179177+0000","last_peered":"2024-04-22T18:38:40.179177+0000","last_clean":"2024-04-22T18:38:40.179177+0000","last_became_active":"2024-04-22T18:38:39.010101+0000","last_became_peered":"2024-04-22T18:38:39.010101+0000","last_unstale":"2024-04-22T18:38:40.179177+0000","last_undegraded":"2024-04-22T18:38:40.179177+0000","last_fullsized":"2024-04-22T18:38:40.179177+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.708134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'30","reported_seq":56,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:41.067411+0000","last_change":"2024-04-22T18:38:38.109358+0000","last_active":"2024-04-22T18:38:41.067411+0000","last_peered":"2024-04-22T18:38:41.067411+0000","last_clean":"2024-04-22T18:38:41.067411+0000","last_became_active":"2024-04-22T18:38:38.108568+0000","last_became_peered":"2024-04-22T18:38:38.108568+0000","last_unstale":"2024-04-22T18:38:41.067411+0000","last_undegraded":"2024-04-22T18:38:41.067411+0000","last_fullsized":"2024-04-22T18:38:41.067411+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:38:36.991764+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:46.436569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":56,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":56,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":278552,"data_compressed":12730,"data_compressed_allocated":16384,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},"log_size":30,"ondisk_log_size":30,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":184888,"data_compressed":6340,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":85.509}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":85.391000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":139264,"data_stored":41566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27112,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":98304,"data_stored":37310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":74.019999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27124,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344716,"statfs":{"total":96636764160,"available":96608989184,"internally_reserved":0,"allocated":106496,"data_stored":41938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":46194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27112,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344728,"statfs":{"total":96636764160,"available":96609001472,"internally_reserved":0,"allocated":98304,"data_stored":37310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":41566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":159744,"data_stored":180842,"data_compressed":6340,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":87.209999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6365,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6365,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:38:44.949 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T18:38:44.949 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-22T18:38:45.237 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 18 2024-04-22T18:38:45.237 INFO:teuthology.orchestra.run.smithi089.stdout:fsid 9e25744c-eff1-4a4a-8d47-0dda406b2c35 2024-04-22T18:38:45.237 INFO:teuthology.orchestra.run.smithi089.stdout:created 2024-04-22T18:38:26.841687+0000 2024-04-22T18:38:45.237 INFO:teuthology.orchestra.run.smithi089.stdout:modified 2024-04-22T18:38:44.030293+0000 2024-04-22T18:38:45.237 INFO:teuthology.orchestra.run.smithi089.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-22T18:38:45.237 INFO:teuthology.orchestra.run.smithi089.stdout:crush_version 5 2024-04-22T18:38:45.237 INFO:teuthology.orchestra.run.smithi089.stdout:full_ratio 0.95 2024-04-22T18:38:45.237 INFO:teuthology.orchestra.run.smithi089.stdout:backfillfull_ratio 0.9 2024-04-22T18:38:45.238 INFO:teuthology.orchestra.run.smithi089.stdout:nearfull_ratio 0.85 2024-04-22T18:38:45.238 INFO:teuthology.orchestra.run.smithi089.stdout:require_min_compat_client luminous 2024-04-22T18:38:45.238 INFO:teuthology.orchestra.run.smithi089.stdout:min_compat_client jewel 2024-04-22T18:38:45.238 INFO:teuthology.orchestra.run.smithi089.stdout:require_osd_release reef 2024-04-22T18:38:45.238 INFO:teuthology.orchestra.run.smithi089.stdout:stretch_mode_enabled false 2024-04-22T18:38:45.238 INFO:teuthology.orchestra.run.smithi089.stdout:pool 1 '.mgr' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 16 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2024-04-22T18:38:45.238 INFO:teuthology.orchestra.run.smithi089.stdout:pool 2 'rbd' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 17 flags hashpspool stripe_width 0 application rbd read_balance_score 1.99 2024-04-22T18:38:45.238 INFO:teuthology.orchestra.run.smithi089.stdout:max_osd 8 2024-04-22T18:38:45.238 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.89:6800/31890 v1:172.21.15.89:6801/31890 exists,up bbf33cfb-0f07-4df6-b90f-83e9edb039a6 2024-04-22T18:38:45.238 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.89:6804/31899 v1:172.21.15.89:6805/31899 exists,up 30415a8e-9f47-416f-9fd0-716c2cb3f562 2024-04-22T18:38:45.238 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.89:6812/31903 v1:172.21.15.89:6813/31903 exists,up b8da6b88-8355-4225-930d-bfed881089b7 2024-04-22T18:38:45.238 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.89:6808/31913 v1:172.21.15.89:6809/31913 exists,up 3eaec913-9d7d-4b03-a28f-0c334e7c39f1 2024-04-22T18:38:45.239 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6808/31922 v1:172.21.15.121:6809/31922 exists,up 887acc3b-6f6b-49eb-a039-13050a2c7d64 2024-04-22T18:38:45.239 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6804/31921 v1:172.21.15.121:6805/31921 exists,up dcaddfd0-f8d1-4c28-bda2-d17eed606b60 2024-04-22T18:38:45.239 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6800/31920 v1:172.21.15.121:6801/31920 exists,up acf0527b-3020-4b0a-b64c-5068d21a5781 2024-04-22T18:38:45.239 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6812/31939 v1:172.21.15.121:6813/31939 exists,up 305394bc-be3d-420b-88d7-d273670540b1 2024-04-22T18:38:45.249 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.89:6800/31890 v1:172.21.15.89:6801/31890 exists,up bbf33cfb-0f07-4df6-b90f-83e9edb039a6', 'osd.1 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.89:6804/31899 v1:172.21.15.89:6805/31899 exists,up 30415a8e-9f47-416f-9fd0-716c2cb3f562', 'osd.2 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.89:6812/31903 v1:172.21.15.89:6813/31903 exists,up b8da6b88-8355-4225-930d-bfed881089b7', 'osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.89:6808/31913 v1:172.21.15.89:6809/31913 exists,up 3eaec913-9d7d-4b03-a28f-0c334e7c39f1', 'osd.4 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6808/31922 v1:172.21.15.121:6809/31922 exists,up 887acc3b-6f6b-49eb-a039-13050a2c7d64', 'osd.5 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6804/31921 v1:172.21.15.121:6805/31921 exists,up dcaddfd0-f8d1-4c28-bda2-d17eed606b60', 'osd.6 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6800/31920 v1:172.21.15.121:6801/31920 exists,up acf0527b-3020-4b0a-b64c-5068d21a5781', 'osd.7 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.121:6812/31939 v1:172.21.15.121:6813/31939 exists,up 305394bc-be3d-420b-88d7-d273670540b1'] 2024-04-22T18:38:45.250 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-22T18:38:45.346 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mon.a", 2024-04-22T18:38:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster": "ceph", 2024-04-22T18:38:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-22T18:38:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "admin_socket_mode": "", 2024-04-22T18:38:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-22T18:38:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_client_required": "cephx, none", 2024-04-22T18:38:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_cluster_required": "cephx", 2024-04-22T18:38:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_debug": "true", 2024-04-22T18:38:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-22T18:38:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-22T18:38:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_service_required": "cephx", 2024-04-22T18:38:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-22T18:38:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_supported": "cephx", 2024-04-22T18:38:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_aio": "true", 2024-04-22T18:38:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-22T18:38:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_aio_poll_ms": "250", 2024-04-22T18:38:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_aio_reap_max": "16", 2024-04-22T18:38:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_async_discard": "false", 2024-04-22T18:38:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_block_size": "4096", 2024-04-22T18:38:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_debug_aio": "false", 2024-04-22T18:38:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-22T18:38:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-22T18:38:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_debug_inflight_ios": "false", 2024-04-22T18:38:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_enable_discard": "false", 2024-04-22T18:38:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_flock_retry": "3", 2024-04-22T18:38:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-22T18:38:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_inject_crash": "0", 2024-04-22T18:38:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-22T18:38:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_ioring": "false", 2024-04-22T18:38:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_ioring_hipri": "false", 2024-04-22T18:38:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-22T18:38:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-22T18:38:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-22T18:38:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-22T18:38:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_type": "", 2024-04-22T18:38:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_alloc_size": "1048576", 2024-04-22T18:38:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_allocator": "hybrid", 2024-04-22T18:38:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_buffered_io": "true", 2024-04-22T18:38:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_check_for_zeros": "false", 2024-04-22T18:38:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-22T18:38:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-22T18:38:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_compact_log_sync": "false", 2024-04-22T18:38:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-22T18:38:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-22T18:38:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-22T18:38:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-22T18:38:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_max_log_runway": "4194304", 2024-04-22T18:38:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_max_prefetch": "1048576", 2024-04-22T18:38:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_min_flush_size": "524288", 2024-04-22T18:38:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_min_log_runway": "1048576", 2024-04-22T18:38:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_replay_recovery": "false", 2024-04-22T18:38:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-22T18:38:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-22T18:38:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_sync_write": "false", 2024-04-22T18:38:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-22T18:38:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-22T18:38:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-22T18:38:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_allocation_from_file": "true", 2024-04-22T18:38:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_allocator": "hybrid", 2024-04-22T18:38:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-22T18:38:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-22T18:38:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-22T18:38:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-22T18:38:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-22T18:38:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-22T18:38:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-22T18:38:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_create": "true", 2024-04-22T18:38:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_db_create": "false", 2024-04-22T18:38:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_db_path": "", 2024-04-22T18:38:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_db_size": "0", 2024-04-22T18:38:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_path": "", 2024-04-22T18:38:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_preallocate_file": "false", 2024-04-22T18:38:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_size": "107374182400", 2024-04-22T18:38:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_wal_create": "false", 2024-04-22T18:38:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_wal_path": "", 2024-04-22T18:38:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_wal_size": "100663296", 2024-04-22T18:38:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_bluefs": "true", 2024-04-22T18:38:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-22T18:38:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-22T18:38:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-22T18:38:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-22T18:38:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-22T18:38:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-22T18:38:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-22T18:38:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-22T18:38:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_autotune": "true", 2024-04-22T18:38:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-22T18:38:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-22T18:38:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-22T18:38:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-22T18:38:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_size": "0", 2024-04-22T18:38:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-22T18:38:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-22T18:38:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-22T18:38:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-22T18:38:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_type": "2q", 2024-04-22T18:38:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-22T18:38:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_clone_cow": "true", 2024-04-22T18:38:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-22T18:38:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-22T18:38:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-22T18:38:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-22T18:38:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-22T18:38:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-22T18:38:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-22T18:38:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_mode": "none", 2024-04-22T18:38:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-22T18:38:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_csum_type": "crc32c", 2024-04-22T18:38:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-22T18:38:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_freelist": "false", 2024-04-22T18:38:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-22T18:38:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-22T18:38:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_inject_bug21040": "false", 2024-04-22T18:38:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-22T18:38:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-22T18:38:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-22T18:38:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-22T18:38:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-22T18:38:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-22T18:38:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-22T18:38:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-22T18:38:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-22T18:38:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-22T18:38:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-22T18:38:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_small_allocations": "0", 2024-04-22T18:38:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-22T18:38:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_default_buffered_read": "true", 2024-04-22T18:38:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_default_buffered_write": "false", 2024-04-22T18:38:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-22T18:38:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-22T18:38:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-22T18:38:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-22T18:38:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-22T18:38:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-22T18:38:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-22T18:38:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-22T18:38:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fail_eio": "false", 2024-04-22T18:38:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-22T18:38:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-22T18:38:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-22T18:38:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-22T18:38:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-22T18:38:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-22T18:38:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_on_mount": "false", 2024-04-22T18:38:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-22T18:38:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_on_umount": "false", 2024-04-22T18:38:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-22T18:38:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-22T18:38:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-22T18:38:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-22T18:38:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-22T18:38:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-22T18:38:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-22T18:38:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-22T18:38:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_ignore_data_csum": "false", 2024-04-22T18:38:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-22T18:38:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-22T18:38:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-22T18:38:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-22T18:38:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_log_op_age": "5.000000", 2024-04-22T18:38:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_max_alloc_size": "0", 2024-04-22T18:38:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_max_blob_size": "0", 2024-04-22T18:38:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-22T18:38:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-22T18:38:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-22T18:38:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_max_deferred_txc": "32", 2024-04-22T18:38:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_min_alloc_size": "0", 2024-04-22T18:38:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-22T18:38:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-22T18:38:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_nid_prealloc": "1024", 2024-04-22T18:38:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-22T18:38:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-22T18:38:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-22T18:38:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-22T18:38:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_retry_disk_reads": "3", 2024-04-22T18:38:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_rocksdb_cf": "true", 2024-04-22T18:38:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-22T18:38:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-22T18:38:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-22T18:38:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-22T18:38:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-22T18:38:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-22T18:38:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_spdk_mem": "512", 2024-04-22T18:38:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-22T18:38:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-22T18:38:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-22T18:38:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-22T18:38:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-22T18:38:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-22T18:38:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-22T18:38:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_tracing": "false", 2024-04-22T18:38:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-22T18:38:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-22T18:38:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-22T18:38:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-22T18:38:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-22T18:38:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-22T18:38:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-22T18:38:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-22T18:38:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-22T18:38:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_zero_block_detection": "false", 2024-04-22T18:38:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-22T18:38:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-22T18:38:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-22T18:38:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-22T18:38:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-22T18:38:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-22T18:38:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-22T18:38:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-22T18:38:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-22T18:38:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-22T18:38:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-22T18:38:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-22T18:38:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-22T18:38:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-22T18:38:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "cephx_cluster_require_signatures": "false", 2024-04-22T18:38:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "cephx_cluster_require_version": "2", 2024-04-22T18:38:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "cephx_require_signatures": "false", 2024-04-22T18:38:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "cephx_require_version": "2", 2024-04-22T18:38:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "cephx_service_require_signatures": "false", 2024-04-22T18:38:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "cephx_service_require_version": "2", 2024-04-22T18:38:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "cephx_sign_messages": "true", 2024-04-22T18:38:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "chdir": "", 2024-04-22T18:38:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "client_acl_type": "", 2024-04-22T18:38:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "client_asio_thread_count": "2", 2024-04-22T18:38:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "client_cache_mid": "0.750000", 2024-04-22T18:38:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "client_cache_size": "16384", 2024-04-22T18:38:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "client_caps_release_delay": "5", 2024-04-22T18:38:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "client_check_pool_perm": "true", 2024-04-22T18:38:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-22T18:38:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "client_debug_force_sync_read": "false", 2024-04-22T18:38:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "client_debug_getattr_caps": "false", 2024-04-22T18:38:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "client_debug_inject_tick_delay": "0", 2024-04-22T18:38:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-22T18:38:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_die_on_failed_remount": "false", 2024-04-22T18:38:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_dirsize_rbytes": "true", 2024-04-22T18:38:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_force_lazyio": "false", 2024-04-22T18:38:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_fs": "", 2024-04-22T18:38:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-22T18:38:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_inject_release_failure": "false", 2024-04-22T18:38:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_max_inline_size": "4096", 2024-04-22T18:38:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-22T18:38:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_mds_namespace": "", 2024-04-22T18:38:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_metadata": "", 2024-04-22T18:38:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_mount_gid": "-1", 2024-04-22T18:38:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_mount_timeout": "300", 2024-04-22T18:38:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_mount_uid": "-1", 2024-04-22T18:38:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_mountpoint": "/", 2024-04-22T18:38:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_notify_timeout": "10", 2024-04-22T18:38:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "client_oc": "true", 2024-04-22T18:38:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "client_oc_max_dirty": "104857600", 2024-04-22T18:38:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-22T18:38:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "client_oc_max_objects": "1000", 2024-04-22T18:38:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "client_oc_size": "209715200", 2024-04-22T18:38:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "client_oc_target_dirty": "8388608", 2024-04-22T18:38:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "client_permissions": "true", 2024-04-22T18:38:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "client_quota": "true", 2024-04-22T18:38:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "client_quota_df": "true", 2024-04-22T18:38:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "client_readahead_max_bytes": "0", 2024-04-22T18:38:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "client_readahead_max_periods": "4", 2024-04-22T18:38:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "client_readahead_min": "131072", 2024-04-22T18:38:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "client_reconnect_stale": "false", 2024-04-22T18:38:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "client_shutdown_timeout": "30", 2024-04-22T18:38:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client_snapdir": ".snap", 2024-04-22T18:38:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client_tick_interval": "1", 2024-04-22T18:38:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client_trace": "", 2024-04-22T18:38:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client_try_dentry_invalidate": "false", 2024-04-22T18:38:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client_use_faked_inos": "false", 2024-04-22T18:38:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client_use_random_mds": "false", 2024-04-22T18:38:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "clog_to_graylog": "false", 2024-04-22T18:38:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-22T18:38:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "clog_to_graylog_port": "12201", 2024-04-22T18:38:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "clog_to_monitors": "default=true", 2024-04-22T18:38:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "clog_to_syslog": "false", 2024-04-22T18:38:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-22T18:38:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "clog_to_syslog_level": "info", 2024-04-22T18:38:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "-", 2024-04-22T18:38:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_network": "", 2024-04-22T18:38:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_network_interface": "", 2024-04-22T18:38:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "compressor_zlib_isal": "false", 2024-04-22T18:38:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "compressor_zlib_level": "5", 2024-04-22T18:38:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "compressor_zlib_winsize": "-15", 2024-04-22T18:38:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "compressor_zstd_level": "1", 2024-04-22T18:38:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-22T18:38:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-22T18:38:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "crimson_alien_op_num_threads": "6", 2024-04-22T18:38:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-22T18:38:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "crimson_osd_obc_lru_size": "10", 2024-04-22T18:38:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-22T18:38:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "crimson_seastar_smp": "1", 2024-04-22T18:38:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_location": "", 2024-04-22T18:38:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_location_hook": "", 2024-04-22T18:38:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_location_hook_timeout": "10", 2024-04-22T18:38:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "daemonize": "false", 2024-04-22T18:38:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "daos_pool": "tank", 2024-04-22T18:38:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-22T18:38:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-22T18:38:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-22T18:38:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_alienstore": "0/5", 2024-04-22T18:38:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_allow_any_pool_priority": "false", 2024-04-22T18:38:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_asok": "1/5", 2024-04-22T18:38:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_asok_assert_abort": "false", 2024-04-22T18:38:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_asserts_on_shutdown": "true", 2024-04-22T18:38:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_auth": "20/20", 2024-04-22T18:38:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_bdev": "1/3", 2024-04-22T18:38:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_bluefs": "1/5", 2024-04-22T18:38:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_bluestore": "1/5", 2024-04-22T18:38:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_buffer": "0/1", 2024-04-22T18:38:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_ceph_exporter": "1/5", 2024-04-22T18:38:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_cephfs_mirror": "0/5", 2024-04-22T18:38:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_cephsqlite": "0/5", 2024-04-22T18:38:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_client": "0/5", 2024-04-22T18:38:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_compressor": "1/5", 2024-04-22T18:38:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_context": "0/1", 2024-04-22T18:38:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_crush": "1/1", 2024-04-22T18:38:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_crypto": "1/5", 2024-04-22T18:38:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_cyanstore": "0/5", 2024-04-22T18:38:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_deliberately_leak_memory": "false", 2024-04-22T18:38:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_disable_randomized_ping": "false", 2024-04-22T18:38:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_dpdk": "1/5", 2024-04-22T18:38:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_eventtrace": "1/5", 2024-04-22T18:38:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_filer": "0/1", 2024-04-22T18:38:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_filestore": "1/3", 2024-04-22T18:38:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_finisher": "1/1", 2024-04-22T18:38:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_fuse": "1/5", 2024-04-22T18:38:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_heartbeat_testing_span": "0", 2024-04-22T18:38:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_heartbeatmap": "1/5", 2024-04-22T18:38:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-22T18:38:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_javaclient": "1/5", 2024-04-22T18:38:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_journal": "1/3", 2024-04-22T18:38:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_journaler": "0/5", 2024-04-22T18:38:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_kstore": "1/5", 2024-04-22T18:38:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_leveldb": "4/5", 2024-04-22T18:38:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_lockdep": "0/1", 2024-04-22T18:38:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mclock": "1/5", 2024-04-22T18:38:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mds": "1/5", 2024-04-22T18:38:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mds_balancer": "1/5", 2024-04-22T18:38:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mds_locker": "1/5", 2024-04-22T18:38:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mds_log": "1/5", 2024-04-22T18:38:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mds_log_expire": "1/5", 2024-04-22T18:38:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mds_migrator": "1/5", 2024-04-22T18:38:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_memstore": "1/5", 2024-04-22T18:38:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mgr": "2/5", 2024-04-22T18:38:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mgrc": "1/5", 2024-04-22T18:38:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mon": "20/20", 2024-04-22T18:38:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_monc": "0/10", 2024-04-22T18:38:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_ms": "1/1", 2024-04-22T18:38:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_none": "0/5", 2024-04-22T18:38:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_objclass": "0/5", 2024-04-22T18:38:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_objectcacher": "0/5", 2024-04-22T18:38:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_objecter": "0/1", 2024-04-22T18:38:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_optracker": "0/5", 2024-04-22T18:38:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_osd": "1/5", 2024-04-22T18:38:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_paxos": "20/20", 2024-04-22T18:38:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_perfcounter": "1/5", 2024-04-22T18:38:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_prioritycache": "1/5", 2024-04-22T18:38:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rados": "0/5", 2024-04-22T18:38:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rbd": "0/5", 2024-04-22T18:38:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rbd_mirror": "0/5", 2024-04-22T18:38:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rbd_pwl": "0/5", 2024-04-22T18:38:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rbd_replay": "0/5", 2024-04-22T18:38:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_refs": "0/0", 2024-04-22T18:38:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_reserver": "1/1", 2024-04-22T18:38:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rgw": "1/5", 2024-04-22T18:38:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rgw_access": "1/5", 2024-04-22T18:38:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rgw_datacache": "1/5", 2024-04-22T18:38:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rgw_dbstore": "1/5", 2024-04-22T18:38:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rgw_flight": "1/5", 2024-04-22T18:38:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rgw_sync": "1/5", 2024-04-22T18:38:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rocksdb": "4/5", 2024-04-22T18:38:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore": "0/5", 2024-04-22T18:38:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_backref": "0/5", 2024-04-22T18:38:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_cache": "0/5", 2024-04-22T18:38:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_cleaner": "0/5", 2024-04-22T18:38:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_device": "0/5", 2024-04-22T18:38:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_epm": "0/5", 2024-04-22T18:38:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-22T18:38:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_journal": "0/5", 2024-04-22T18:38:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_lba": "0/5", 2024-04-22T18:38:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_odata": "0/5", 2024-04-22T18:38:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_omap": "0/5", 2024-04-22T18:38:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_onode": "0/5", 2024-04-22T18:38:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_t": "0/5", 2024-04-22T18:38:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_tm": "0/5", 2024-04-22T18:38:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_striper": "0/1", 2024-04-22T18:38:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_test": "0/5", 2024-04-22T18:38:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_throttle": "1/1", 2024-04-22T18:38:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_timer": "0/1", 2024-04-22T18:38:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_tp": "0/5", 2024-04-22T18:38:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-22T18:38:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "device_failure_prediction_mode": "none", 2024-04-22T18:38:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-22T18:38:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-22T18:38:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "err_to_graylog": "false", 2024-04-22T18:38:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "err_to_journald": "false", 2024-04-22T18:38:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "err_to_stderr": "true", 2024-04-22T18:38:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "err_to_syslog": "false", 2024-04-22T18:38:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event_tracing": "false", 2024-04-22T18:38:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "exporter_addr": "0.0.0.0", 2024-04-22T18:38:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "exporter_http_port": "9926", 2024-04-22T18:38:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "exporter_prio_limit": "5", 2024-04-22T18:38:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-22T18:38:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "exporter_sort_metrics": "true", 2024-04-22T18:38:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "exporter_stats_period": "5", 2024-04-22T18:38:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "fake_statfs_for_testing": "0", 2024-04-22T18:38:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "fatal_signal_handlers": "true", 2024-04-22T18:38:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "filer_max_purge_ops": "10", 2024-04-22T18:38:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "filer_max_truncate_ops": "128", 2024-04-22T18:38:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_apply_finisher_threads": "1", 2024-04-22T18:38:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_blackhole": "false", 2024-04-22T18:38:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_btrfs_clone_range": "true", 2024-04-22T18:38:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_btrfs_snap": "true", 2024-04-22T18:38:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_caller_concurrency": "10", 2024-04-22T18:38:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_collect_device_partition_information": "true", 2024-04-22T18:38:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_commit_timeout": "600.000000", 2024-04-22T18:38:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_debug_inject_read_err": "false", 2024-04-22T18:38:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_debug_omap_check": "false", 2024-04-22T18:38:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_debug_random_read_err": "0.000000", 2024-04-22T18:38:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_debug_verify_split": "false", 2024-04-22T18:38:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_dump_file": "", 2024-04-22T18:38:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-22T18:38:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-22T18:38:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_fadvise": "true", 2024-04-22T18:38:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_fail_eio": "true", 2024-04-22T18:38:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_fd_cache_shards": "16", 2024-04-22T18:38:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_fd_cache_size": "128", 2024-04-22T18:38:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_fiemap": "false", 2024-04-22T18:38:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_fiemap_threshold": "4096", 2024-04-22T18:38:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-22T18:38:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-22T18:38:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_inject_stall": "0", 2024-04-22T18:38:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_journal_parallel": "false", 2024-04-22T18:38:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_journal_trailing": "false", 2024-04-22T18:38:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_journal_writeahead": "false", 2024-04-22T18:38:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_kill_at": "0", 2024-04-22T18:38:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-22T18:38:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-22T18:38:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-22T18:38:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-22T18:38:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-22T18:38:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattrs": "0", 2024-04-22T18:38:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-22T18:38:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-22T18:38:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-22T18:38:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-22T18:38:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_xattr_value_size": "0", 2024-04-22T18:38:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-22T18:38:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-22T18:38:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-22T18:38:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_merge_threshold": "-10", 2024-04-22T18:38:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-22T18:38:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_odsync_write": "false", 2024-04-22T18:38:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_omap_backend": "rocksdb", 2024-04-22T18:38:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_omap_backend_path": "", 2024-04-22T18:38:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-22T18:38:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-22T18:38:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-22T18:38:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_op_thread_timeout": "60", 2024-04-22T18:38:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_op_threads": "2", 2024-04-22T18:38:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_punch_hole": "false", 2024-04-22T18:38:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-22T18:38:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-22T18:38:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-22T18:38:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-22T18:38:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-22T18:38:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-22T18:38:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-22T18:38:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-22T18:38:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-22T18:38:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_max_ops": "50", 2024-04-22T18:38:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-22T18:38:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_seek_data_hole": "false", 2024-04-22T18:38:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_sloppy_crc": "false", 2024-04-22T18:38:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-22T18:38:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_splice": "false", 2024-04-22T18:38:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_split_multiple": "2", 2024-04-22T18:38:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_split_rand_factor": "20", 2024-04-22T18:38:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_update_to": "1000", 2024-04-22T18:38:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-22T18:38:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-22T18:38:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-22T18:38:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-22T18:38:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-22T18:38:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-22T18:38:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_enable": "true", 2024-04-22T18:38:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-22T18:38:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-22T18:38:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-22T18:38:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-22T18:38:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-22T18:38:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-22T18:38:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_xfs_extsize": "false", 2024-04-22T18:38:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_zfs_snap": "false", 2024-04-22T18:38:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "fio_dir": "/tmp/fio", 2024-04-22T18:38:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "fsid": "9e25744c-eff1-4a4a-8d47-0dda406b2c35", 2024-04-22T18:38:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_allow_other": "true", 2024-04-22T18:38:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_atomic_o_trunc": "true", 2024-04-22T18:38:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_big_writes": "true", 2024-04-22T18:38:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_debug": "false", 2024-04-22T18:38:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_default_permissions": "false", 2024-04-22T18:38:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_disable_pagecache": "false", 2024-04-22T18:38:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_max_write": "0", 2024-04-22T18:38:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_multithreaded": "true", 2024-04-22T18:38:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_require_active_mds": "true", 2024-04-22T18:38:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_set_user_groups": "true", 2024-04-22T18:38:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_splice_move": "true", 2024-04-22T18:38:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_splice_read": "true", 2024-04-22T18:38:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_splice_write": "true", 2024-04-22T18:38:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-22T18:38:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_use_invalidate_cb": "true", 2024-04-22T18:38:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-22T18:38:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "gss_target_name": "ceph", 2024-04-22T18:38:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_file": "", 2024-04-22T18:38:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_inject_failure": "0", 2024-04-22T18:38:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_interval": "5", 2024-04-22T18:38:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "host": "smithi089", 2024-04-22T18:38:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-22T18:38:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-22T18:38:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-22T18:38:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-22T18:38:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-22T18:38:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-22T18:38:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-22T18:38:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-22T18:38:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-22T18:38:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-22T18:38:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-22T18:38:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-22T18:38:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-22T18:38:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "inject_early_sigterm": "false", 2024-04-22T18:38:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "jaeger_agent_port": "6799", 2024-04-22T18:38:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "jaeger_tracing_enable": "false", 2024-04-22T18:38:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_aio": "true", 2024-04-22T18:38:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_align_min_size": "65536", 2024-04-22T18:38:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_block_align": "true", 2024-04-22T18:38:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_block_size": "4096", 2024-04-22T18:38:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_dio": "true", 2024-04-22T18:38:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_discard": "false", 2024-04-22T18:38:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_force_aio": "false", 2024-04-22T18:38:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_ignore_corruption": "false", 2024-04-22T18:38:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_max_write_bytes": "10485760", 2024-04-22T18:38:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_max_write_entries": "100", 2024-04-22T18:38:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_replay_from": "0", 2024-04-22T18:38:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-22T18:38:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-22T18:38:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-22T18:38:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-22T18:38:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_write_header_frequency": "0", 2024-04-22T18:38:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_zero_on_create": "false", 2024-04-22T18:38:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "journaler_prefetch_periods": "10", 2024-04-22T18:38:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "journaler_prezero_periods": "5", 2024-04-22T18:38:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "journaler_write_head_interval": "15", 2024-04-22T18:38:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "", 2024-04-22T18:38:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "keyfile": "", 2024-04-22T18:38:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-22T18:38:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_backend": "rocksdb", 2024-04-22T18:38:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_default_stripe_size": "65536", 2024-04-22T18:38:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_fsck_on_mount": "false", 2024-04-22T18:38:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-22T18:38:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_max_bytes": "67108864", 2024-04-22T18:38:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_max_ops": "512", 2024-04-22T18:38:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_nid_prealloc": "1024", 2024-04-22T18:38:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_onode_map_size": "1024", 2024-04-22T18:38:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-22T18:38:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_sync_submit_transaction": "false", 2024-04-22T18:38:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_sync_transaction": "false", 2024-04-22T18:38:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_block_size": "65536", 2024-04-22T18:38:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_bloom_size": "0", 2024-04-22T18:38:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_cache_size": "536870912", 2024-04-22T18:38:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_compact_on_mount": "false", 2024-04-22T18:38:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_compression": "false", 2024-04-22T18:38:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_log": "", 2024-04-22T18:38:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_log_to_ceph_log": "true", 2024-04-22T18:38:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_max_open_files": "0", 2024-04-22T18:38:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_paranoid": "false", 2024-04-22T18:38:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_write_buffer_size": "33554432", 2024-04-22T18:38:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "librados_thread_count": "2", 2024-04-22T18:38:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "lockdep": "false", 2024-04-22T18:38:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "lockdep_force_backtrace": "false", 2024-04-22T18:38:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "log_coarse_timestamps": "true", 2024-04-22T18:38:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-22T18:38:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "log_flush_on_exit": "false", 2024-04-22T18:38:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "log_graylog_host": "127.0.0.1", 2024-04-22T18:38:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "log_graylog_port": "12201", 2024-04-22T18:38:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "log_max_new": "1000", 2024-04-22T18:38:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "log_max_recent": "10000", 2024-04-22T18:38:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "log_stderr_prefix": "", 2024-04-22T18:38:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "log_stop_at_utilization": "0.970000", 2024-04-22T18:38:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": "true", 2024-04-22T18:38:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_graylog": "false", 2024-04-22T18:38:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_journald": "false", 2024-04-22T18:38:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_stderr": "false", 2024-04-22T18:38:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_syslog": "false", 2024-04-22T18:38:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "max_rotating_auth_attempts": "10", 2024-04-22T18:38:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-22T18:38:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_action_on_write_error": "1", 2024-04-22T18:38:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_alternate_name_max": "8192", 2024-04-22T18:38:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_asio_thread_count": "2", 2024-04-22T18:38:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_export_pin": "true", 2024-04-22T18:38:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_fragment_dirs": "true", 2024-04-22T18:38:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-22T18:38:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_fragment_interval": "5", 2024-04-22T18:38:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-22T18:38:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-22T18:38:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_interval": "10", 2024-04-22T18:38:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_max": "-1", 2024-04-22T18:38:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_max_until": "-1", 2024-04-22T18:38:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_merge_size": "50", 2024-04-22T18:38:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_midchunk": "0.300000", 2024-04-22T18:38:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-22T18:38:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_min_start": "0.200000", 2024-04-22T18:38:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_minchunk": "0.001000", 2024-04-22T18:38:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_mode": "0", 2024-04-22T18:38:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_need_max": "1.200000", 2024-04-22T18:38:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_need_min": "0.800000", 2024-04-22T18:38:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-22T18:38:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-22T18:38:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_split_bits": "3", 2024-04-22T18:38:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-22T18:38:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_split_size": "10000", 2024-04-22T18:38:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-22T18:38:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_target_decay": "10.000000", 2024-04-22T18:38:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-22T18:38:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_beacon_grace": "15.000000", 2024-04-22T18:38:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_beacon_interval": "4.000000", 2024-04-22T18:38:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-22T18:38:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-22T18:38:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache_mid": "0.700000", 2024-04-22T18:38:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache_release_free_interval": "10", 2024-04-22T18:38:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache_reservation": "0.050000", 2024-04-22T18:38:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-22T18:38:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache_trim_interval": "1", 2024-04-22T18:38:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache_trim_threshold": "262144", 2024-04-22T18:38:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-22T18:38:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-22T18:38:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-22T18:38:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_client_prealloc_inos": "1000", 2024-04-22T18:38:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-22T18:38:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_connect_bootstrapping": "false", 2024-04-22T18:38:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_damage_table_max_entries": "10000", 2024-04-22T18:38:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-22T18:38:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_debug_auth_pins": "false", 2024-04-22T18:38:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_debug_frag": "false", 2024-04-22T18:38:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_debug_scatterstat": "false", 2024-04-22T18:38:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_debug_subtrees": "false", 2024-04-22T18:38:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_decay_halflife": "5.000000", 2024-04-22T18:38:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_default_dir_hash": "2", 2024-04-22T18:38:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_defer_session_stale": "true", 2024-04-22T18:38:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_deny_all_reconnect": "false", 2024-04-22T18:38:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dir_keys_per_op": "16384", 2024-04-22T18:38:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dir_max_commit_size": "10", 2024-04-22T18:38:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dir_max_entries": "0", 2024-04-22T18:38:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dir_prefetch": "true", 2024-04-22T18:38:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-22T18:38:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-22T18:38:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dump_cache_on_map": "false", 2024-04-22T18:38:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-22T18:38:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-22T18:38:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_early_reply": "true", 2024-04-22T18:38:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_enable_op_tracker": "true", 2024-04-22T18:38:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_enforce_unique_name": "true", 2024-04-22T18:38:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-22T18:38:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-22T18:38:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_export_ephemeral_random": "true", 2024-04-22T18:38:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-22T18:38:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-22T18:38:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-22T18:38:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-22T18:38:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-22T18:38:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-22T18:38:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-22T18:38:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-22T18:38:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_health_summarize_threshold": "10", 2024-04-22T18:38:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-22T18:38:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-22T18:38:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_inject_health_dummy": "false", 2024-04-22T18:38:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-22T18:38:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_inject_migrator_session_race": "false", 2024-04-22T18:38:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-22T18:38:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-22T18:38:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-22T18:38:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_join_fs": "", 2024-04-22T18:38:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_journal_format": "1", 2024-04-22T18:38:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_create_at": "0", 2024-04-22T18:38:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_export_at": "0", 2024-04-22T18:38:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_import_at": "0", 2024-04-22T18:38:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_journal_at": "0", 2024-04-22T18:38:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_journal_expire_at": "0", 2024-04-22T18:38:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_journal_replay_at": "0", 2024-04-22T18:38:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_link_at": "0", 2024-04-22T18:38:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_mdstable_at": "0", 2024-04-22T18:38:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_openc_at": "0", 2024-04-22T18:38:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_rename_at": "0", 2024-04-22T18:38:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-22T18:38:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log_events_per_segment": "1024", 2024-04-22T18:38:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log_max_events": "-1", 2024-04-22T18:38:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log_max_segments": "128", 2024-04-22T18:38:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log_pause": "false", 2024-04-22T18:38:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log_segment_size": "0", 2024-04-22T18:38:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-22T18:38:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log_warn_factor": "2.000000", 2024-04-22T18:38:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_caps_per_client": "1048576", 2024-04-22T18:38:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_completed_flushes": "100000", 2024-04-22T18:38:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_completed_requests": "100000", 2024-04-22T18:38:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_export_size": "20971520", 2024-04-22T18:38:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_file_recover": "32", 2024-04-22T18:38:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_purge_files": "64", 2024-04-22T18:38:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_purge_ops": "8192", 2024-04-22T18:38:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-22T18:38:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-22T18:38:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_snaps_per_dir": "100", 2024-04-22T18:38:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_metrics_update_interval": "2", 2024-04-22T18:38:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_min_caps_per_client": "100", 2024-04-22T18:38:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_min_caps_working_set": "10000", 2024-04-22T18:38:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-22T18:38:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_numa_node": "-1", 2024-04-22T18:38:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-22T18:38:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_op_complaint_time": "30.000000", 2024-04-22T18:38:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_op_history_duration": "600", 2024-04-22T18:38:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_op_history_size": "20", 2024-04-22T18:38:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_op_history_slow_op_size": "20", 2024-04-22T18:38:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-22T18:38:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_op_log_threshold": "5", 2024-04-22T18:38:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_ping_grace": "15", 2024-04-22T18:38:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_ping_interval": "5", 2024-04-22T18:38:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-22T18:38:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-22T18:38:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_recall_max_caps": "30000", 2024-04-22T18:38:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-22T18:38:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-22T18:38:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-22T18:38:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_recall_warning_threshold": "262144", 2024-04-22T18:38:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-22T18:38:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_replay_interval": "1.000000", 2024-04-22T18:38:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-22T18:38:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-22T18:38:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_root_ino_gid": "0", 2024-04-22T18:38:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_root_ino_uid": "0", 2024-04-22T18:38:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-22T18:38:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-22T18:38:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-22T18:38:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-22T18:38:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-22T18:38:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-22T18:38:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-22T18:38:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-22T18:38:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-22T18:38:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-22T18:38:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_shutdown_check": "0", 2024-04-22T18:38:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_skip_ino": "0", 2024-04-22T18:38:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-22T18:38:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_snap_max_uid": "4294967294", 2024-04-22T18:38:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_snap_min_uid": "0", 2024-04-22T18:38:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_snap_rstat": "false", 2024-04-22T18:38:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_standby_replay_damaged": "false", 2024-04-22T18:38:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_symlink_recovery": "true", 2024-04-22T18:38:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-22T18:38:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_thrash_exports": "0", 2024-04-22T18:38:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_thrash_fragments": "0", 2024-04-22T18:38:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_tick_interval": "5.000000", 2024-04-22T18:38:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_valgrind_exit": "false", 2024-04-22T18:38:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_verify_backtrace": "1", 2024-04-22T18:38:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_verify_scatter": "false", 2024-04-22T18:38:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-22T18:38:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_wipe_sessions": "false", 2024-04-22T18:38:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool_debug": "false", 2024-04-22T18:38:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-22T18:38:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "memstore_device_bytes": "1073741824", 2024-04-22T18:38:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "memstore_page_set": "false", 2024-04-22T18:38:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "memstore_page_size": "65536", 2024-04-22T18:38:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_client_bytes": "134217728", 2024-04-22T18:38:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_client_messages": "512", 2024-04-22T18:38:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-22T18:38:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-22T18:38:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-22T18:38:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-22T18:38:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_disabled_modules": "", 2024-04-22T18:38:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-22T18:38:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_max_pg_num_change": "128", 2024-04-22T18:38:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_mds_bytes": "134217728", 2024-04-22T18:38:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_mds_messages": "128", 2024-04-22T18:38:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-22T18:38:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_mon_bytes": "134217728", 2024-04-22T18:38:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_mon_messages": "128", 2024-04-22T18:38:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_osd_bytes": "536870912", 2024-04-22T18:38:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_osd_messages": "8192", 2024-04-22T18:38:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_pool": "true", 2024-04-22T18:38:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-22T18:38:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_standby_modules": "true", 2024-04-22T18:38:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_stats_period": "5", 2024-04-22T18:38:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_stats_threshold": "5", 2024-04-22T18:38:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_tick_period": "2", 2024-04-22T18:38:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-22T18:38:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-22T18:38:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_allow_pool_delete": "true", 2024-04-22T18:38:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_allow_pool_size_one": "true", 2024-04-22T18:38:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_auth_validate_all_caps": "true", 2024-04-22T18:38:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-22T18:38:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-22T18:38:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_bytes": "104857600", 2024-04-22T18:38:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_directed_command_retry": "2", 2024-04-22T18:38:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-22T18:38:45.405 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-22T18:38:45.405 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-22T18:38:45.405 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-22T18:38:45.405 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_hunt_parallel": "3", 2024-04-22T18:38:45.405 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_log_interval": "1.000000", 2024-04-22T18:38:45.405 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-22T18:38:45.405 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_ping_interval": "10.000000", 2024-04-22T18:38:45.405 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-22T18:38:45.405 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-22T18:38:45.405 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-22T18:38:45.405 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-22T18:38:45.405 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_file_level": "debug", 2024-04-22T18:38:45.405 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_file": "true", 2024-04-22T18:38:45.406 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-22T18:38:45.406 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-22T18:38:45.406 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-22T18:38:45.406 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_journald": "false", 2024-04-22T18:38:45.406 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-22T18:38:45.406 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-22T18:38:45.406 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-22T18:38:45.406 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-22T18:38:45.406 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_compact_on_bootstrap": "false", 2024-04-22T18:38:45.406 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_compact_on_start": "false", 2024-04-22T18:38:45.406 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_compact_on_trim": "true", 2024-04-22T18:38:45.406 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-22T18:38:45.406 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-22T18:38:45.406 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-22T18:38:45.407 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cpu_threads": "4", 2024-04-22T18:38:45.407 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_crush_min_required_version": "hammer", 2024-04-22T18:38:45.407 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_daemon_bytes": "419430400", 2024-04-22T18:38:45.407 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-22T18:38:45.407 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_data_avail_crit": "5", 2024-04-22T18:38:45.407 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_data_avail_warn": "5", 2024-04-22T18:38:45.407 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_data_size_warn": "16106127360", 2024-04-22T18:38:45.407 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-22T18:38:45.407 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-22T18:38:45.407 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_dump_json": "false", 2024-04-22T18:38:45.407 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-22T18:38:45.407 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_dump_transactions": "false", 2024-04-22T18:38:45.407 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_extra_checks": "false", 2024-04-22T18:38:45.408 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-22T18:38:45.408 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-22T18:38:45.408 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_no_require_quincy": "false", 2024-04-22T18:38:45.408 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_no_require_reef": "false", 2024-04-22T18:38:45.408 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-22T18:38:45.408 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-22T18:38:45.408 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-22T18:38:45.408 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_down_mkfs_grace": "120", 2024-04-22T18:38:45.408 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_election_default_strategy": "1", 2024-04-22T18:38:45.408 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_election_timeout": "5.000000", 2024-04-22T18:38:45.408 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-22T18:38:45.408 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_elector_ping_divisor": "2", 2024-04-22T18:38:45.409 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-22T18:38:45.409 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_enable_op_tracker": "true", 2024-04-22T18:38:45.409 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_fake_pool_delete": "false", 2024-04-22T18:38:45.409 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_force_quorum_join": "false", 2024-04-22T18:38:45.409 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_globalid_prealloc": "10000", 2024-04-22T18:38:45.409 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_health_detail_to_clog": "false", 2024-04-22T18:38:45.409 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_health_log_update_period": "5", 2024-04-22T18:38:45.409 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_health_max_detail": "50", 2024-04-22T18:38:45.409 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_health_to_clog": "true", 2024-04-22T18:38:45.409 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_health_to_clog_interval": "600", 2024-04-22T18:38:45.409 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-22T18:38:45.409 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_host": "[v1:172.21.15.89:6789],[v1:172.21.15.89:6790],[v1:172.21.15.121:6789]", 2024-04-22T18:38:45.409 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_host_override": "", 2024-04-22T18:38:45.410 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_initial_members": "", 2024-04-22T18:38:45.410 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-22T18:38:45.410 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-22T18:38:45.410 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-22T18:38:45.410 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-22T18:38:45.410 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-22T18:38:45.410 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_lease": "5.000000", 2024-04-22T18:38:45.410 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-22T18:38:45.410 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-22T18:38:45.410 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_log_full_interval": "50", 2024-04-22T18:38:45.410 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_log_max": "10000", 2024-04-22T18:38:45.410 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_log_max_summary": "50", 2024-04-22T18:38:45.410 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-22T18:38:45.411 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_log_epochs": "500", 2024-04-22T18:38:45.411 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-22T18:38:45.411 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-22T18:38:45.411 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_osd": "10000", 2024-04-22T18:38:45.411 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_pg_per_osd": "10000", 2024-04-22T18:38:45.411 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_pool_pg_num": "65536", 2024-04-22T18:38:45.411 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-22T18:38:45.411 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-22T18:38:45.411 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mds_force_trim_to": "0", 2024-04-22T18:38:45.411 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mds_skip_sanity": "false", 2024-04-22T18:38:45.411 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_memory_autotune": "true", 2024-04-22T18:38:45.412 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_memory_target": "2147483648", 2024-04-22T18:38:45.412 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mgr_beacon_grace": "30", 2024-04-22T18:38:45.412 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-22T18:38:45.412 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mgr_digest_period": "5", 2024-04-22T18:38:45.412 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mgr_inactive_grace": "60", 2024-04-22T18:38:45.412 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-22T18:38:45.412 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-22T18:38:45.412 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_min_osdmap_epochs": "50", 2024-04-22T18:38:45.412 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_op_complaint_time": "30", 2024-04-22T18:38:45.412 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_op_history_duration": "600", 2024-04-22T18:38:45.412 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_op_history_size": "20", 2024-04-22T18:38:45.412 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_op_history_slow_op_size": "20", 2024-04-22T18:38:45.412 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-22T18:38:45.412 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_op_log_threshold": "5", 2024-04-22T18:38:45.413 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-22T18:38:45.413 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-22T18:38:45.413 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-22T18:38:45.413 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_auto_mark_in": "false", 2024-04-22T18:38:45.413 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-22T18:38:45.413 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-22T18:38:45.413 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-22T18:38:45.413 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_cache_size": "500", 2024-04-22T18:38:45.413 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-22T18:38:45.413 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-22T18:38:45.413 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-22T18:38:45.413 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_down_out_interval": "600", 2024-04-22T18:38:45.413 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-22T18:38:45.414 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-22T18:38:45.414 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_force_trim_to": "0", 2024-04-22T18:38:45.414 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-22T18:38:45.414 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-22T18:38:45.414 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-22T18:38:45.414 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-22T18:38:45.414 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-22T18:38:45.414 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-22T18:38:45.414 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_max_creating_pgs": "1024", 2024-04-22T18:38:45.414 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-22T18:38:45.414 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_min_down_reporters": "2", 2024-04-22T18:38:45.414 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-22T18:38:45.414 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-22T18:38:45.415 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-22T18:38:45.415 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-22T18:38:45.415 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-22T18:38:45.415 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-22T18:38:45.415 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_report_timeout": "900", 2024-04-22T18:38:45.415 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-22T18:38:45.415 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-22T18:38:45.415 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-22T18:38:45.415 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-22T18:38:45.415 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-22T18:38:45.415 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-22T18:38:45.415 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-22T18:38:45.415 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-22T18:38:45.415 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-22T18:38:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pg_stuck_threshold": "60", 2024-04-22T18:38:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-22T18:38:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-22T18:38:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-22T18:38:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-22T18:38:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-22T18:38:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-22T18:38:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_probe_timeout": "2.000000", 2024-04-22T18:38:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_reweight_max_change": "0.050000", 2024-04-22T18:38:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_reweight_max_osds": "4", 2024-04-22T18:38:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-22T18:38:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-22T18:38:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-22T18:38:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-22T18:38:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-22T18:38:45.417 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_scrub_interval": "300", 2024-04-22T18:38:45.417 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_scrub_max_keys": "100", 2024-04-22T18:38:45.417 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_scrub_timeout": "300", 2024-04-22T18:38:45.417 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_session_timeout": "300", 2024-04-22T18:38:45.417 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_smart_report_timeout": "5", 2024-04-22T18:38:45.417 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_stat_smooth_intervals": "6", 2024-04-22T18:38:45.417 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-22T18:38:45.417 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_stretch_pool_min_size": "2", 2024-04-22T18:38:45.417 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_stretch_pool_size": "4", 2024-04-22T18:38:45.417 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-22T18:38:45.417 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-22T18:38:45.417 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_sync_debug": "false", 2024-04-22T18:38:45.417 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-22T18:38:45.417 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-22T18:38:45.417 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_sync_provider_kill_at": "0", 2024-04-22T18:38:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_sync_requester_kill_at": "0", 2024-04-22T18:38:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_sync_timeout": "60.000000", 2024-04-22T18:38:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_target_pg_per_osd": "100", 2024-04-22T18:38:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_tick_interval": "5", 2024-04-22T18:38:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_timecheck_interval": "300.000000", 2024-04-22T18:38:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-22T18:38:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_use_min_delay_socket": "false", 2024-04-22T18:38:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_older_version_delay": "604800", 2024-04-22T18:38:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-22T18:38:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-22T18:38:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-22T18:38:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-22T18:38:45.419 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-22T18:38:45.419 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-22T18:38:45.419 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-22T18:38:45.419 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_misplaced": "false", 2024-04-22T18:38:45.419 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-22T18:38:45.419 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_older_version": "true", 2024-04-22T18:38:45.419 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-22T18:38:45.419 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-22T18:38:45.419 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-22T18:38:45.419 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-22T18:38:45.419 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-22T18:38:45.419 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-22T18:38:45.419 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-22T18:38:45.419 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-22T18:38:45.420 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-22T18:38:45.420 INFO:teuthology.orchestra.run.smithi089.stdout: "monmap": "", 2024-04-22T18:38:45.420 INFO:teuthology.orchestra.run.smithi089.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-22T18:38:45.420 INFO:teuthology.orchestra.run.smithi089.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-22T18:38:45.420 INFO:teuthology.orchestra.run.smithi089.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-22T18:38:45.420 INFO:teuthology.orchestra.run.smithi089.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-22T18:38:45.420 INFO:teuthology.orchestra.run.smithi089.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-22T18:38:45.420 INFO:teuthology.orchestra.run.smithi089.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-22T18:38:45.420 INFO:teuthology.orchestra.run.smithi089.stdout: "motr_tracing_enabled": "false", 2024-04-22T18:38:45.420 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_op_threads": "3", 2024-04-22T18:38:45.420 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-22T18:38:45.420 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_cm": "false", 2024-04-22T18:38:45.420 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_device_name": "", 2024-04-22T18:38:45.420 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_dscp": "96", 2024-04-22T18:38:45.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-22T18:38:45.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-22T18:38:45.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_local_gid": "", 2024-04-22T18:38:45.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-22T18:38:45.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_port_num": "1", 2024-04-22T18:38:45.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-22T18:38:45.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-22T18:38:45.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-22T18:38:45.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-22T18:38:45.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_sl": "3", 2024-04-22T18:38:45.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_support_srq": "true", 2024-04-22T18:38:45.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_type": "ib", 2024-04-22T18:38:45.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_reap_threshold": "5", 2024-04-22T18:38:45.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_before_connect": "false", 2024-04-22T18:38:45.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_ipv4": "true", 2024-04-22T18:38:45.422 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_ipv6": "false", 2024-04-22T18:38:45.422 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_msgr1": "true", 2024-04-22T18:38:45.422 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_msgr2": "false", 2024-04-22T18:38:45.422 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_port_max": "7568", 2024-04-22T18:38:45.422 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_port_min": "6800", 2024-04-22T18:38:45.422 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-22T18:38:45.422 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_retry_count": "3", 2024-04-22T18:38:45.422 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_retry_delay": "5", 2024-04-22T18:38:45.422 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_blackhole_client": "false", 2024-04-22T18:38:45.422 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_blackhole_mds": "false", 2024-04-22T18:38:45.422 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_blackhole_mgr": "false", 2024-04-22T18:38:45.422 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_blackhole_mon": "false", 2024-04-22T18:38:45.422 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_blackhole_osd": "false", 2024-04-22T18:38:45.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_client_mode": "crc secure", 2024-04-22T18:38:45.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_cluster_mode": "crc secure", 2024-04-22T18:38:45.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_cluster_type": "", 2024-04-22T18:38:45.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_compress_secure": "false", 2024-04-22T18:38:45.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_connection_idle_timeout": "900", 2024-04-22T18:38:45.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_connection_ready_timeout": "10", 2024-04-22T18:38:45.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_crc_data": "true", 2024-04-22T18:38:45.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_crc_header": "true", 2024-04-22T18:38:45.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_die_on_bad_msg": "false", 2024-04-22T18:38:45.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_die_on_bug": "true", 2024-04-22T18:38:45.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_die_on_old_message": "true", 2024-04-22T18:38:45.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_die_on_skipped_message": "false", 2024-04-22T18:38:45.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-22T18:38:45.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-22T18:38:45.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_coremask": "0xF", 2024-04-22T18:38:45.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-22T18:38:45.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-22T18:38:45.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_enable_tso": "true", 2024-04-22T18:38:45.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-22T18:38:45.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-22T18:38:45.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_hugepages": "", 2024-04-22T18:38:45.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-22T18:38:45.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-22T18:38:45.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_lro": "true", 2024-04-22T18:38:45.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_memory_channel": "4", 2024-04-22T18:38:45.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-22T18:38:45.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_pmd": "", 2024-04-22T18:38:45.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_port_id": "0", 2024-04-22T18:38:45.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-22T18:38:45.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-22T18:38:45.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dump_on_send": "false", 2024-04-22T18:38:45.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_initial_backoff": "0.200000", 2024-04-22T18:38:45.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_inject_delay_max": "1.000000", 2024-04-22T18:38:45.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-22T18:38:45.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_inject_delay_type": "", 2024-04-22T18:38:45.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-22T18:38:45.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_inject_network_congestion": "0", 2024-04-22T18:38:45.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_inject_socket_failures": "0", 2024-04-22T18:38:45.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_learn_addr_from_peer": "true", 2024-04-22T18:38:45.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_max_accept_failures": "4", 2024-04-22T18:38:45.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_max_backoff": "15.000000", 2024-04-22T18:38:45.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_mon_client_mode": "secure crc", 2024-04-22T18:38:45.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-22T18:38:45.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_mon_service_mode": "secure crc", 2024-04-22T18:38:45.426 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_osd_compress_min_size": "1024", 2024-04-22T18:38:45.426 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_osd_compress_mode": "none", 2024-04-22T18:38:45.426 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-22T18:38:45.426 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-22T18:38:45.426 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_pq_min_cost": "65536", 2024-04-22T18:38:45.426 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_public_type": "", 2024-04-22T18:38:45.426 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_service_mode": "crc secure", 2024-04-22T18:38:45.426 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_tcp_listen_backlog": "512", 2024-04-22T18:38:45.426 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_tcp_nodelay": "true", 2024-04-22T18:38:45.426 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-22T18:38:45.426 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_tcp_rcvbuf": "0", 2024-04-22T18:38:45.426 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_type": "async", 2024-04-22T18:38:45.426 INFO:teuthology.orchestra.run.smithi089.stdout: "no_config_file": "false", 2024-04-22T18:38:45.427 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_completion_locks_per_session": "32", 2024-04-22T18:38:45.427 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-22T18:38:45.427 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-22T18:38:45.427 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_inflight_ops": "1024", 2024-04-22T18:38:45.427 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-22T18:38:45.427 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-22T18:38:45.427 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_tick_interval": "5.000000", 2024-04-22T18:38:45.427 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_timeout": "10.000000", 2024-04-22T18:38:45.427 INFO:teuthology.orchestra.run.smithi089.stdout: "objectstore_blackhole": "false", 2024-04-22T18:38:45.427 INFO:teuthology.orchestra.run.smithi089.stdout: "openssl_engine_opts": "", 2024-04-22T18:38:45.427 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_agent_delay_time": "5.000000", 2024-04-22T18:38:45.427 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_agent_hist_halflife": "1000", 2024-04-22T18:38:45.427 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_agent_max_low_ops": "2", 2024-04-22T18:38:45.427 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_agent_max_ops": "4", 2024-04-22T18:38:45.428 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-22T18:38:45.428 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-22T18:38:45.428 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_agent_slop": "0.020000", 2024-04-22T18:38:45.428 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-22T18:38:45.428 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-22T18:38:45.428 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_asio_thread_count": "2", 2024-04-22T18:38:45.428 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_async_recovery_min_cost": "1", 2024-04-22T18:38:45.428 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-22T18:38:45.428 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-22T18:38:45.428 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_backfill_scan_max": "512", 2024-04-22T18:38:45.428 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_backfill_scan_min": "64", 2024-04-22T18:38:45.428 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_backoff_on_degraded": "false", 2024-04-22T18:38:45.428 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_backoff_on_peering": "false", 2024-04-22T18:38:45.428 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_backoff_on_unfound": "true", 2024-04-22T18:38:45.429 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_beacon_report_interval": "300", 2024-04-22T18:38:45.429 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bench_duration": "30", 2024-04-22T18:38:45.429 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-22T18:38:45.429 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bench_max_block_size": "67108864", 2024-04-22T18:38:45.429 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-22T18:38:45.429 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_blkin_trace_all": "false", 2024-04-22T18:38:45.429 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-22T18:38:45.429 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-22T18:38:45.429 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-22T18:38:45.429 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-22T18:38:45.429 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_check_for_log_corruption": "false", 2024-04-22T18:38:45.429 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-22T18:38:45.429 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-22T18:38:45.429 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-22T18:38:45.429 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-22T18:38:45.430 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_class_update_on_start": "true", 2024-04-22T18:38:45.430 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_client_message_cap": "256", 2024-04-22T18:38:45.430 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_client_message_size_cap": "524288000", 2024-04-22T18:38:45.430 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_client_op_priority": "63", 2024-04-22T18:38:45.430 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_client_watch_timeout": "30", 2024-04-22T18:38:45.430 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_command_max_records": "256", 2024-04-22T18:38:45.430 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_compact_on_start": "false", 2024-04-22T18:38:45.430 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-22T18:38:45.430 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-22T18:38:45.430 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-22T18:38:45.430 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_crush_update_on_start": "true", 2024-04-22T18:38:45.430 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_crush_update_weight_set": "true", 2024-04-22T18:38:45.430 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-22T18:38:45.431 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-22T18:38:45.431 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-22T18:38:45.431 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-22T18:38:45.431 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-22T18:38:45.431 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_feed_pullee": "-1", 2024-04-22T18:38:45.431 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-22T18:38:45.431 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-22T18:38:45.431 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_inject_dispatch_delay_probability": "0.100000", 2024-04-22T18:38:45.431 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_misdirected_ops": "false", 2024-04-22T18:38:45.431 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_no_acting_change": "false", 2024-04-22T18:38:45.431 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_no_purge_strays": "false", 2024-04-22T18:38:45.431 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_op_order": "false", 2024-04-22T18:38:45.431 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-22T18:38:45.431 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-22T18:38:45.432 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-22T18:38:45.432 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-22T18:38:45.432 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_shutdown": "false", 2024-04-22T18:38:45.432 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-22T18:38:45.432 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-22T18:38:45.432 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-22T18:38:45.432 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-22T18:38:45.432 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_verify_snaps": "false", 2024-04-22T18:38:45.432 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-22T18:38:45.432 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-22T18:38:45.432 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_deep_scrub_keys": "1024", 2024-04-22T18:38:45.432 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-22T18:38:45.432 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-22T18:38:45.432 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-22T18:38:45.432 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_deep_scrub_stride": "524288", 2024-04-22T18:38:45.433 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-22T18:38:45.433 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-22T18:38:45.433 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_default_notify_timeout": "30", 2024-04-22T18:38:45.433 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_delete_sleep": "0.000000", 2024-04-22T18:38:45.433 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-22T18:38:45.433 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-22T18:38:45.433 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-22T18:38:45.433 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_discard_disconnected_ops": "true", 2024-04-22T18:38:45.433 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_enable_op_tracker": "true", 2024-04-22T18:38:45.433 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-22T18:38:45.433 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-22T18:38:45.433 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-22T18:38:45.433 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-22T18:38:45.434 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_fast_info": "true", 2024-04-22T18:38:45.434 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_fast_shutdown": "true", 2024-04-22T18:38:45.434 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-22T18:38:45.434 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-22T18:38:45.434 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-22T18:38:45.434 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-22T18:38:45.434 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-22T18:38:45.434 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_function_tracing": "false", 2024-04-22T18:38:45.434 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_heartbeat_grace": "20", 2024-04-22T18:38:45.434 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_heartbeat_interval": "6", 2024-04-22T18:38:45.434 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-22T18:38:45.434 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_heartbeat_min_peers": "10", 2024-04-22T18:38:45.434 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_heartbeat_min_size": "2000", 2024-04-22T18:38:45.434 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_heartbeat_stale": "600", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_hit_set_max_size": "100000", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_hit_set_min_size": "1000", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_journal_size": "5120", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_kill_backfill_at": "0", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_map_cache_size": "50", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_map_dedup": "true", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_map_message_max": "40", 2024-04-22T18:38:45.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-22T18:38:45.436 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_map_share_max_epochs": "40", 2024-04-22T18:38:45.436 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_attr_name_len": "100", 2024-04-22T18:38:45.436 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_attr_size": "0", 2024-04-22T18:38:45.436 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_backfills": "1", 2024-04-22T18:38:45.436 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_markdown_count": "5", 2024-04-22T18:38:45.436 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_markdown_period": "600", 2024-04-22T18:38:45.436 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_object_name_len": "2048", 2024-04-22T18:38:45.436 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_object_namespace_len": "256", 2024-04-22T18:38:45.436 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_object_size": "134217728", 2024-04-22T18:38:45.436 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-22T18:38:45.436 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-22T18:38:45.436 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_pg_blocked_by": "16", 2024-04-22T18:38:45.436 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_pg_log_entries": "10000", 2024-04-22T18:38:45.436 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-22T18:38:45.437 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_pgls": "1024", 2024-04-22T18:38:45.437 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_push_cost": "8388608", 2024-04-22T18:38:45.437 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_push_objects": "10", 2024-04-22T18:38:45.437 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_scrubs": "3", 2024-04-22T18:38:45.437 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-22T18:38:45.437 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_trimming_pgs": "2", 2024-04-22T18:38:45.437 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_write_op_reply_len": "64", 2024-04-22T18:38:45.437 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_write_size": "90", 2024-04-22T18:38:45.437 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-22T18:38:45.437 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-22T18:38:45.437 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-22T18:38:45.437 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-22T18:38:45.437 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-22T18:38:45.438 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-22T18:38:45.438 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-22T18:38:45.438 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-22T18:38:45.438 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_profile": "balanced", 2024-04-22T18:38:45.438 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-22T18:38:45.438 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-22T18:38:45.438 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-22T18:38:45.438 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-22T18:38:45.438 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-22T18:38:45.438 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-22T18:38:45.438 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-22T18:38:45.438 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-22T18:38:45.438 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-22T18:38:45.438 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-22T18:38:45.439 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-22T18:38:45.439 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_memory_base": "805306368", 2024-04-22T18:38:45.439 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_memory_cache_min": "134217728", 2024-04-22T18:38:45.439 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-22T18:38:45.439 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-22T18:38:45.439 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_memory_target": "4294967296", 2024-04-22T18:38:45.439 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_memory_target_autotune": "false", 2024-04-22T18:38:45.439 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-22T18:38:45.439 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_min_pg_log_entries": "250", 2024-04-22T18:38:45.439 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_min_recovery_priority": "0", 2024-04-22T18:38:45.439 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-22T18:38:45.439 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-22T18:38:45.439 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mon_report_interval": "5", 2024-04-22T18:38:45.439 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-22T18:38:45.440 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-22T18:38:45.440 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_num_cache_shards": "32", 2024-04-22T18:38:45.440 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_num_op_tracker_shard": "32", 2024-04-22T18:38:45.440 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_numa_auto_affinity": "true", 2024-04-22T18:38:45.440 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_numa_node": "-1", 2024-04-22T18:38:45.440 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_numa_prefer_iface": "true", 2024-04-22T18:38:45.440 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-04-22T18:38:45.440 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_objecter_finishers": "1", 2024-04-22T18:38:45.440 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_objectstore": "bluestore", 2024-04-22T18:38:45.440 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_objectstore_fuse": "false", 2024-04-22T18:38:45.440 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_objectstore_tracing": "false", 2024-04-22T18:38:45.440 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_complaint_time": "30.000000", 2024-04-22T18:38:45.440 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_history_duration": "600", 2024-04-22T18:38:45.440 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_history_size": "20", 2024-04-22T18:38:45.441 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_history_slow_op_size": "20", 2024-04-22T18:38:45.441 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-22T18:38:45.441 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_log_threshold": "5", 2024-04-22T18:38:45.441 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_num_shards": "0", 2024-04-22T18:38:45.441 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_num_shards_hdd": "5", 2024-04-22T18:38:45.441 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_num_shards_ssd": "8", 2024-04-22T18:38:45.441 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-22T18:38:45.441 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-22T18:38:45.441 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-22T18:38:45.441 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-22T18:38:45.441 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_pq_min_cost": "65536", 2024-04-22T18:38:45.441 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-22T18:38:45.441 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_queue_cut_off": "high", 2024-04-22T18:38:45.441 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-22T18:38:45.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_thread_timeout": "15", 2024-04-22T18:38:45.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_open_classes_on_start": "true", 2024-04-22T18:38:45.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_os_flags": "0", 2024-04-22T18:38:45.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_peering_op_priority": "255", 2024-04-22T18:38:45.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_delete_cost": "1048576", 2024-04-22T18:38:45.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_delete_priority": "5", 2024-04-22T18:38:45.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-22T18:38:45.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-22T18:38:45.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-22T18:38:45.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_log_trim_max": "10000", 2024-04-22T18:38:45.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_log_trim_min": "100", 2024-04-22T18:38:45.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-22T18:38:45.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-22T18:38:45.442 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_stat_report_interval_max": "500", 2024-04-22T18:38:45.443 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-22T18:38:45.443 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-22T18:38:45.443 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-22T18:38:45.443 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-22T18:38:45.443 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-22T18:38:45.443 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-22T18:38:45.443 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_crimson": "false", 2024-04-22T18:38:45.443 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-22T18:38:45.443 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-22T18:38:45.443 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-22T18:38:45.443 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-22T18:38:45.443 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-22T18:38:45.443 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-22T18:38:45.443 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-22T18:38:45.444 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-22T18:38:45.444 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_flags": "0", 2024-04-22T18:38:45.444 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-22T18:38:45.444 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_min_size": "2", 2024-04-22T18:38:45.444 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-22T18:38:45.444 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_pg_num": "32", 2024-04-22T18:38:45.444 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_pgp_num": "0", 2024-04-22T18:38:45.444 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-22T18:38:45.444 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_size": "2", 2024-04-22T18:38:45.444 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_type": "replicated", 2024-04-22T18:38:45.444 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-22T18:38:45.444 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-22T18:38:45.444 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_push_per_object_cost": "1000", 2024-04-22T18:38:45.444 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-22T18:38:45.445 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recover_clone_overlap": "true", 2024-04-22T18:38:45.445 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-22T18:38:45.445 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_cost": "20971520", 2024-04-22T18:38:45.445 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-22T18:38:45.445 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_max_active": "0", 2024-04-22T18:38:45.445 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-22T18:38:45.445 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-22T18:38:45.445 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-22T18:38:45.445 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-22T18:38:45.445 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_max_single_start": "1", 2024-04-22T18:38:45.445 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_op_priority": "3", 2024-04-22T18:38:45.445 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-22T18:38:45.445 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_priority": "5", 2024-04-22T18:38:45.445 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-22T18:38:45.445 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_sleep": "0.000000", 2024-04-22T18:38:45.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-22T18:38:45.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-22T18:38:45.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-22T18:38:45.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_repair_during_recovery": "false", 2024-04-22T18:38:45.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_requested_scrub_priority": "120", 2024-04-22T18:38:45.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-22T18:38:45.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-22T18:38:45.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_auto_repair": "false", 2024-04-22T18:38:45.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-22T18:38:45.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-22T18:38:45.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_begin_hour": "0", 2024-04-22T18:38:45.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_begin_week_day": "0", 2024-04-22T18:38:45.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_chunk_max": "25", 2024-04-22T18:38:45.446 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_chunk_min": "5", 2024-04-22T18:38:45.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_cost": "52428800", 2024-04-22T18:38:45.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_during_recovery": "false", 2024-04-22T18:38:45.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_end_hour": "0", 2024-04-22T18:38:45.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_end_week_day": "0", 2024-04-22T18:38:45.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_event_cost": "4096", 2024-04-22T18:38:45.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-22T18:38:45.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-22T18:38:45.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_invalid_stats": "true", 2024-04-22T18:38:45.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-22T18:38:45.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-22T18:38:45.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_max_preemptions": "5", 2024-04-22T18:38:45.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-22T18:38:45.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_priority": "5", 2024-04-22T18:38:45.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_reservation_timeout": "5000", 2024-04-22T18:38:45.447 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_sleep": "0.000000", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_slow_reservation_response": "2200", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_skip_data_digest": "false", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_smart_report_timeout": "5", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_snap_trim_cost": "1048576", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_snap_trim_priority": "5", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-22T18:38:45.448 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_target_transaction_size": "30", 2024-04-22T18:38:45.449 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-22T18:38:45.449 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-22T18:38:45.449 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-22T18:38:45.449 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-22T18:38:45.449 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-22T18:38:45.449 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-22T18:38:45.449 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-22T18:38:45.449 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-22T18:38:45.449 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-22T18:38:45.449 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-22T18:38:45.449 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tracing": "false", 2024-04-22T18:38:45.449 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_use_stale_snap": "false", 2024-04-22T18:38:45.449 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-22T18:38:45.449 INFO:teuthology.orchestra.run.smithi089.stdout: "osdc_blkin_trace_all": "false", 2024-04-22T18:38:45.449 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_kill_at": "0", 2024-04-22T18:38:45.450 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_max_join_drift": "10", 2024-04-22T18:38:45.450 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_min": "500", 2024-04-22T18:38:45.450 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_min_wait": "0.050000", 2024-04-22T18:38:45.450 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_propose_interval": "1.000000", 2024-04-22T18:38:45.450 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_service_trim_max": "500", 2024-04-22T18:38:45.450 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-22T18:38:45.450 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_service_trim_min": "10", 2024-04-22T18:38:45.450 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_stash_full_interval": "25", 2024-04-22T18:38:45.450 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_trim_max": "500", 2024-04-22T18:38:45.450 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_trim_min": "250", 2024-04-22T18:38:45.450 INFO:teuthology.orchestra.run.smithi089.stdout: "perf": "true", 2024-04-22T18:38:45.450 INFO:teuthology.orchestra.run.smithi089.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-22T18:38:45.450 INFO:teuthology.orchestra.run.smithi089.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-22T18:38:45.451 INFO:teuthology.orchestra.run.smithi089.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-22T18:38:45.451 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "-", 2024-04-22T18:38:45.451 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrv": "", 2024-04-22T18:38:45.451 INFO:teuthology.orchestra.run.smithi089.stdout: "public_bind_addr": "-", 2024-04-22T18:38:45.451 INFO:teuthology.orchestra.run.smithi089.stdout: "public_network": "", 2024-04-22T18:38:45.451 INFO:teuthology.orchestra.run.smithi089.stdout: "public_network_interface": "", 2024-04-22T18:38:45.451 INFO:teuthology.orchestra.run.smithi089.stdout: "qat_compressor_enabled": "false", 2024-04-22T18:38:45.451 INFO:teuthology.orchestra.run.smithi089.stdout: "qat_compressor_session_max_number": "256", 2024-04-22T18:38:45.451 INFO:teuthology.orchestra.run.smithi089.stdout: "rados_mon_op_timeout": "0", 2024-04-22T18:38:45.451 INFO:teuthology.orchestra.run.smithi089.stdout: "rados_osd_op_timeout": "0", 2024-04-22T18:38:45.451 INFO:teuthology.orchestra.run.smithi089.stdout: "rados_tracing": "false", 2024-04-22T18:38:45.451 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_atime_update_interval": "60", 2024-04-22T18:38:45.451 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-22T18:38:45.451 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_balance_parent_reads": "false", 2024-04-22T18:38:45.451 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_balance_snap_reads": "false", 2024-04-22T18:38:45.452 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_blkin_trace_all": "false", 2024-04-22T18:38:45.452 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-22T18:38:45.452 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-22T18:38:45.452 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache": "true", 2024-04-22T18:38:45.452 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-22T18:38:45.452 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-22T18:38:45.452 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-22T18:38:45.452 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-22T18:38:45.452 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_policy": "writearound", 2024-04-22T18:38:45.452 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_size": "33554432", 2024-04-22T18:38:45.452 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-22T18:38:45.452 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-22T18:38:45.452 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_clone_copy_on_read": "false", 2024-04-22T18:38:45.453 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_compression_hint": "none", 2024-04-22T18:38:45.453 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_concurrent_management_ops": "10", 2024-04-22T18:38:45.453 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-22T18:38:45.453 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_clone_format": "auto", 2024-04-22T18:38:45.453 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_data_pool": "", 2024-04-22T18:38:45.453 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_features": "61", 2024-04-22T18:38:45.453 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_format": "2", 2024-04-22T18:38:45.453 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_map_options": "", 2024-04-22T18:38:45.453 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_order": "22", 2024-04-22T18:38:45.453 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_pool": "rbd", 2024-04-22T18:38:45.453 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-22T18:38:45.453 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_stripe_count": "0", 2024-04-22T18:38:45.453 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_stripe_unit": "0", 2024-04-22T18:38:45.453 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-22T18:38:45.453 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-22T18:38:45.454 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-22T18:38:45.454 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_enable_alloc_hint": "true", 2024-04-22T18:38:45.454 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-22T18:38:45.454 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_io_scheduler": "simple", 2024-04-22T18:38:45.454 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-22T18:38:45.454 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-22T18:38:45.454 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-22T18:38:45.454 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-22T18:38:45.454 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-22T18:38:45.454 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-22T18:38:45.454 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-22T18:38:45.454 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-22T18:38:45.454 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-22T18:38:45.455 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_order": "24", 2024-04-22T18:38:45.455 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_pool": "", 2024-04-22T18:38:45.455 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_splay_width": "4", 2024-04-22T18:38:45.455 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_localize_parent_reads": "false", 2024-04-22T18:38:45.455 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_localize_snap_reads": "false", 2024-04-22T18:38:45.455 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-22T18:38:45.455 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-22T18:38:45.455 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-22T18:38:45.455 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-22T18:38:45.455 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-22T18:38:45.455 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-22T18:38:45.455 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-22T18:38:45.455 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-22T18:38:45.456 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-22T18:38:45.456 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-22T18:38:45.456 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-22T18:38:45.456 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-22T18:38:45.456 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-22T18:38:45.456 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-22T18:38:45.456 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-22T18:38:45.456 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-22T18:38:45.456 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-22T18:38:45.456 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-22T18:38:45.456 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-22T18:38:45.456 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-22T18:38:45.456 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-22T18:38:45.456 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-22T18:38:45.457 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-22T18:38:45.457 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-22T18:38:45.457 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-22T18:38:45.457 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-22T18:38:45.457 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-22T18:38:45.457 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-22T18:38:45.457 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-22T18:38:45.457 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-22T18:38:45.457 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-22T18:38:45.457 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mtime_update_interval": "60", 2024-04-22T18:38:45.457 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_non_blocking_aio": "true", 2024-04-22T18:38:45.457 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_op_thread_timeout": "60", 2024-04-22T18:38:45.457 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_op_threads": "1", 2024-04-22T18:38:45.457 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_parent_cache_enabled": "false", 2024-04-22T18:38:45.458 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-22T18:38:45.458 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-22T18:38:45.458 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-22T18:38:45.458 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_plugins": "", 2024-04-22T18:38:45.458 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_bps_burst": "0", 2024-04-22T18:38:45.458 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-22T18:38:45.458 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_bps_limit": "0", 2024-04-22T18:38:45.458 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_exclude_ops": "0", 2024-04-22T18:38:45.458 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_iops_burst": "0", 2024-04-22T18:38:45.458 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-22T18:38:45.458 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_iops_limit": "0", 2024-04-22T18:38:45.458 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-22T18:38:45.458 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-22T18:38:45.458 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-22T18:38:45.458 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-22T18:38:45.459 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-22T18:38:45.459 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-22T18:38:45.459 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-22T18:38:45.459 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-22T18:38:45.459 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-22T18:38:45.459 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-22T18:38:45.459 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-22T18:38:45.459 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-22T18:38:45.459 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-22T18:38:45.459 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-22T18:38:45.459 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_read_from_replica_policy": "default", 2024-04-22T18:38:45.459 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-22T18:38:45.459 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-22T18:38:45.459 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-22T18:38:45.460 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-22T18:38:45.460 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_skip_partial_discard": "true", 2024-04-22T18:38:45.460 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-22T18:38:45.460 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_tracing": "false", 2024-04-22T18:38:45.460 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_validate_names": "true", 2024-04-22T18:38:45.460 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_validate_pool": "true", 2024-04-22T18:38:45.460 INFO:teuthology.orchestra.run.smithi089.stdout: "restapi_base_url": "", 2024-04-22T18:38:45.460 INFO:teuthology.orchestra.run.smithi089.stdout: "restapi_log_level": "", 2024-04-22T18:38:45.460 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_acl_grants_max_num": "100", 2024-04-22T18:38:45.460 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_admin_entry": "admin", 2024-04-22T18:38:45.460 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-22T18:38:45.460 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_backend_store": "rados", 2024-04-22T18:38:45.460 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_barbican_url": "", 2024-04-22T18:38:45.460 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_beast_enable_async": "true", 2024-04-22T18:38:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-22T18:38:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-22T18:38:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-22T18:38:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-22T18:38:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-22T18:38:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-22T18:38:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-22T18:38:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_cache_enabled": "true", 2024-04-22T18:38:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_cache_expiry_interval": "900", 2024-04-22T18:38:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_cache_lru_size": "10000", 2024-04-22T18:38:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_config_store": "rados", 2024-04-22T18:38:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_content_length_compat": "false", 2024-04-22T18:38:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_copy_obj_progress": "true", 2024-04-22T18:38:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-22T18:38:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_cors_rules_max_num": "100", 2024-04-22T18:38:45.462 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_cross_domain_policy": "", 2024-04-22T18:38:45.462 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-22T18:38:45.462 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_addr": "", 2024-04-22T18:38:45.462 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-22T18:38:45.462 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-22T18:38:45.462 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-22T18:38:45.462 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-22T18:38:45.462 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_password": "", 2024-04-22T18:38:45.462 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-22T18:38:45.462 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_username": "", 2024-04-22T18:38:45.462 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_require_ssl": "true", 2024-04-22T18:38:45.462 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-22T18:38:45.462 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-22T18:38:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-22T18:38:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-22T18:38:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-22T18:38:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-22T18:38:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-22T18:38:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-22T18:38:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-22T18:38:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-22T18:38:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-22T18:38:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-22T18:38:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-22T18:38:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-22T18:38:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-22T18:38:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_addr": "", 2024-04-22T18:38:45.463 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_auth": "token", 2024-04-22T18:38:45.464 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_namespace": "", 2024-04-22T18:38:45.464 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_prefix": "", 2024-04-22T18:38:45.464 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-22T18:38:45.464 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-22T18:38:45.464 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-22T18:38:45.464 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-22T18:38:45.464 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_token_file": "", 2024-04-22T18:38:45.464 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-22T18:38:45.464 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_curl_buffersize": "524288", 2024-04-22T18:38:45.464 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-22T18:38:45.464 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_curl_low_speed_time": "300", 2024-04-22T18:38:45.464 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-22T18:38:45.464 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-22T18:38:45.464 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-22T18:38:45.464 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-22T18:38:45.465 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-22T18:38:45.465 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-22T18:38:45.465 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-22T18:38:45.465 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-22T18:38:45.465 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-22T18:38:45.465 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-22T18:38:45.465 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-22T18:38:45.465 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data_log_changes_size": "1000", 2024-04-22T18:38:45.465 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data_log_num_shards": "128", 2024-04-22T18:38:45.465 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-22T18:38:45.465 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data_log_window": "30", 2024-04-22T18:38:45.465 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-22T18:38:45.465 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-22T18:38:45.466 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-22T18:38:45.466 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-22T18:38:45.466 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-22T18:38:45.466 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-22T18:38:45.466 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-22T18:38:45.466 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-22T18:38:45.466 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-22T18:38:45.466 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-22T18:38:45.466 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-22T18:38:45.466 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-22T18:38:45.466 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-22T18:38:45.466 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-22T18:38:45.466 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-22T18:38:45.467 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-22T18:38:45.467 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-22T18:38:45.467 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-22T18:38:45.467 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-22T18:38:45.467 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-22T18:38:45.467 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-22T18:38:45.467 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-22T18:38:45.467 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-22T18:38:45.467 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-22T18:38:45.467 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dns_name": "", 2024-04-22T18:38:45.467 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dns_s3website_name": "", 2024-04-22T18:38:45.467 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dynamic_resharding": "true", 2024-04-22T18:38:45.467 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-22T18:38:45.467 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enable_gc_threads": "true", 2024-04-22T18:38:45.467 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enable_lc_threads": "true", 2024-04-22T18:38:45.468 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enable_ops_log": "false", 2024-04-22T18:38:45.468 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enable_quota_threads": "true", 2024-04-22T18:38:45.468 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enable_static_website": "false", 2024-04-22T18:38:45.468 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enable_usage_log": "false", 2024-04-22T18:38:45.468 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enforce_swift_acls": "true", 2024-04-22T18:38:45.468 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_exit_timeout_secs": "120", 2024-04-22T18:38:45.468 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_expose_bucket": "false", 2024-04-22T18:38:45.468 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_extended_http_attrs": "", 2024-04-22T18:38:45.468 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_filter": "none", 2024-04-22T18:38:45.468 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-22T18:38:45.468 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_frontends": "beast port=7480", 2024-04-22T18:38:45.468 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-22T18:38:45.468 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_max_deferred": "50", 2024-04-22T18:38:45.468 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-22T18:38:45.469 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_max_objs": "32", 2024-04-22T18:38:45.469 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-22T18:38:45.469 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-22T18:38:45.469 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-22T18:38:45.469 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-22T18:38:45.469 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_processor_period": "3600", 2024-04-22T18:38:45.469 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-22T18:38:45.469 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-22T18:38:45.469 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-22T18:38:45.469 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-22T18:38:45.469 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_init_timeout": "300", 2024-04-22T18:38:45.469 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-22T18:38:45.469 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-22T18:38:45.470 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-22T18:38:45.470 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-22T18:38:45.470 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_domain": "", 2024-04-22T18:38:45.470 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_password": "", 2024-04-22T18:38:45.470 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_password_path": "", 2024-04-22T18:38:45.470 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_project": "", 2024-04-22T18:38:45.470 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_tenant": "", 2024-04-22T18:38:45.470 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_token": "", 2024-04-22T18:38:45.470 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_token_path": "", 2024-04-22T18:38:45.470 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_user": "", 2024-04-22T18:38:45.470 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_api_version": "2", 2024-04-22T18:38:45.470 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_barbican_domain": "", 2024-04-22T18:38:45.471 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_barbican_password": "", 2024-04-22T18:38:45.471 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_barbican_project": "", 2024-04-22T18:38:45.471 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-22T18:38:45.471 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_barbican_user": "", 2024-04-22T18:38:45.471 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-22T18:38:45.471 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-22T18:38:45.471 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-22T18:38:45.471 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-22T18:38:45.471 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-22T18:38:45.471 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_url": "", 2024-04-22T18:38:45.471 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-22T18:38:45.471 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lc_debug_interval": "-1", 2024-04-22T18:38:45.471 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lc_lock_max_time": "90", 2024-04-22T18:38:45.471 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lc_max_objs": "32", 2024-04-22T18:38:45.472 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lc_max_rules": "1000", 2024-04-22T18:38:45.472 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lc_max_worker": "3", 2024-04-22T18:38:45.472 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-22T18:38:45.472 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lc_thread_delay": "0", 2024-04-22T18:38:45.472 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-22T18:38:45.472 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ldap_dnattr": "uid", 2024-04-22T18:38:45.472 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-22T18:38:45.472 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ldap_searchfilter": "", 2024-04-22T18:38:45.472 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-22T18:38:45.472 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ldap_uri": "ldaps://", 2024-04-22T18:38:45.472 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-22T18:38:45.472 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-22T18:38:45.473 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-22T18:38:45.473 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_log_http_headers": "", 2024-04-22T18:38:45.473 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-22T18:38:45.473 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-22T18:38:45.473 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_log_object_name_utc": "false", 2024-04-22T18:38:45.473 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-04-22T18:38:45.473 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_attr_name_len": "0", 2024-04-22T18:38:45.473 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_attr_size": "0", 2024-04-22T18:38:45.473 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-22T18:38:45.473 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_chunk_size": "4194304", 2024-04-22T18:38:45.473 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-22T18:38:45.473 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-22T18:38:45.473 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-22T18:38:45.474 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_listing_results": "1000", 2024-04-22T18:38:45.474 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_notify_retries": "10", 2024-04-22T18:38:45.474 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-22T18:38:45.474 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_put_param_size": "1048576", 2024-04-22T18:38:45.474 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_put_size": "5368709120", 2024-04-22T18:38:45.474 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_slo_entries": "1000", 2024-04-22T18:38:45.474 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_md_log_max_shards": "64", 2024-04-22T18:38:45.474 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-22T18:38:45.474 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-22T18:38:45.474 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-22T18:38:45.474 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-22T18:38:45.474 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_mp_lock_max_time": "600", 2024-04-22T18:38:45.474 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-22T18:38:45.474 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-22T18:38:45.475 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-22T18:38:45.475 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-22T18:38:45.475 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-22T18:38:45.475 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-22T18:38:45.475 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-22T18:38:45.475 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-22T18:38:45.475 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_max_gc": "300", 2024-04-22T18:38:45.475 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-22T18:38:45.475 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-22T18:38:45.475 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-22T18:38:45.475 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-22T18:38:45.475 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-22T18:38:45.475 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-22T18:38:45.476 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-22T18:38:45.476 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_num_async_rados_threads": "32", 2024-04-22T18:38:45.476 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_num_control_oids": "8", 2024-04-22T18:38:45.476 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_numa_node": "-1", 2024-04-22T18:38:45.476 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-22T18:38:45.476 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-22T18:38:45.476 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_objexp_chunk_size": "100", 2024-04-22T18:38:45.476 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_objexp_gc_interval": "600", 2024-04-22T18:38:45.476 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-22T18:38:45.476 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-22T18:38:45.476 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-22T18:38:45.476 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_op_thread_timeout": "600", 2024-04-22T18:38:45.476 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_op_tracing": "false", 2024-04-22T18:38:45.476 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_opa_token": "", 2024-04-22T18:38:45.477 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_opa_url": "", 2024-04-22T18:38:45.477 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_opa_verify_ssl": "true", 2024-04-22T18:38:45.477 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-22T18:38:45.477 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-22T18:38:45.477 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ops_log_rados": "false", 2024-04-22T18:38:45.477 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ops_log_socket_path": "", 2024-04-22T18:38:45.477 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-22T18:38:45.477 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-22T18:38:45.477 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-22T18:38:45.477 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_period_push_interval": "2.000000", 2024-04-22T18:38:45.477 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-22T18:38:45.477 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-22T18:38:45.477 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-22T18:38:45.477 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_print_continue": "true", 2024-04-22T18:38:45.477 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-22T18:38:45.478 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-22T18:38:45.478 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-22T18:38:45.478 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-22T18:38:45.478 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-22T18:38:45.478 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_rados_tracing": "false", 2024-04-22T18:38:45.478 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_realm": "", 2024-04-22T18:38:45.478 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_realm_id": "", 2024-04-22T18:38:45.478 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-22T18:38:45.478 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_region": "", 2024-04-22T18:38:45.478 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-22T18:38:45.478 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-22T18:38:45.478 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-22T18:38:45.478 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-22T18:38:45.479 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_request_uri": "", 2024-04-22T18:38:45.479 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_reshard_batch_size": "64", 2024-04-22T18:38:45.479 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-22T18:38:45.479 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_reshard_max_aio": "128", 2024-04-22T18:38:45.479 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_reshard_num_logs": "16", 2024-04-22T18:38:45.479 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_reshard_thread_interval": "600", 2024-04-22T18:38:45.479 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_resolve_cname": "false", 2024-04-22T18:38:45.479 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-22T18:38:45.479 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_run_sync_thread": "true", 2024-04-22T18:38:45.479 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-22T18:38:45.479 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-22T18:38:45.479 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-22T18:38:45.479 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-22T18:38:45.480 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-22T18:38:45.480 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-22T18:38:45.480 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-22T18:38:45.480 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-22T18:38:45.480 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_scheduler_type": "throttler", 2024-04-22T18:38:45.480 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_script_uri": "", 2024-04-22T18:38:45.480 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_service_provider_name": "", 2024-04-22T18:38:45.480 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-22T18:38:45.480 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sts_client_id": "", 2024-04-22T18:38:45.480 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sts_client_secret": "", 2024-04-22T18:38:45.480 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sts_entry": "sts", 2024-04-22T18:38:45.480 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sts_key": "sts", 2024-04-22T18:38:45.480 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-22T18:38:45.481 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sts_min_session_duration": "900", 2024-04-22T18:38:45.481 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sts_token_introspection_url": "", 2024-04-22T18:38:45.481 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_account_in_url": "false", 2024-04-22T18:38:45.481 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_auth_entry": "auth", 2024-04-22T18:38:45.481 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_auth_url": "", 2024-04-22T18:38:45.481 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_custom_header": "", 2024-04-22T18:38:45.481 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-22T18:38:45.481 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_need_stats": "true", 2024-04-22T18:38:45.481 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_tenant_name": "", 2024-04-22T18:38:45.481 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_token_expiration": "86400", 2024-04-22T18:38:45.481 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_url": "", 2024-04-22T18:38:45.481 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_url_prefix": "swift", 2024-04-22T18:38:45.481 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-22T18:38:45.481 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-22T18:38:45.481 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-22T18:38:45.482 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_lease_period": "120", 2024-04-22T18:38:45.482 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-22T18:38:45.482 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-22T18:38:45.482 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-22T18:38:45.482 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-22T18:38:45.482 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-22T18:38:45.482 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-22T18:38:45.482 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-22T18:38:45.482 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-22T18:38:45.482 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-22T18:38:45.482 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_thread_pool_size": "512", 2024-04-22T18:38:45.482 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_torrent_comment": "", 2024-04-22T18:38:45.482 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_torrent_createby": "", 2024-04-22T18:38:45.482 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_torrent_encoding": "", 2024-04-22T18:38:45.483 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_torrent_flag": "false", 2024-04-22T18:38:45.483 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_torrent_origin": "", 2024-04-22T18:38:45.483 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-22T18:38:45.483 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_torrent_tracker": "", 2024-04-22T18:38:45.483 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_trust_forwarded_https": "false", 2024-04-22T18:38:45.483 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-22T18:38:45.483 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-22T18:38:45.483 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_usage_max_shards": "32", 2024-04-22T18:38:45.483 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_usage_max_user_shards": "1", 2024-04-22T18:38:45.483 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_use_opa_authz": "false", 2024-04-22T18:38:45.483 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-22T18:38:45.483 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-22T18:38:45.483 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_max_buckets": "1000", 2024-04-22T18:38:45.483 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_policies_max_num": "100", 2024-04-22T18:38:45.483 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-22T18:38:45.484 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-22T18:38:45.484 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-22T18:38:45.484 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-22T18:38:45.484 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_unique_email": "true", 2024-04-22T18:38:45.484 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_verify_ssl": "true", 2024-04-22T18:38:45.484 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-22T18:38:45.484 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_zone": "", 2024-04-22T18:38:45.484 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_zone_id": "", 2024-04-22T18:38:45.484 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-22T18:38:45.484 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_zonegroup": "", 2024-04-22T18:38:45.484 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_zonegroup_id": "", 2024-04-22T18:38:45.484 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-22T18:38:45.484 INFO:teuthology.orchestra.run.smithi089.stdout: "rgwlc_auto_session_clear": "true", 2024-04-22T18:38:45.484 INFO:teuthology.orchestra.run.smithi089.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-22T18:38:45.484 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_block_size": "4096", 2024-04-22T18:38:45.485 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-22T18:38:45.485 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-22T18:38:45.485 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-22T18:38:45.485 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-22T18:38:45.485 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-22T18:38:45.485 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cache_size": "536870912", 2024-04-22T18:38:45.485 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-22T18:38:45.485 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-22T18:38:45.485 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-22T18:38:45.485 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-22T18:38:45.485 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-22T18:38:45.485 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-22T18:38:45.485 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-22T18:38:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-22T18:38:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_index_type": "binary_search", 2024-04-22T18:38:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-22T18:38:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-22T18:38:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_partition_filters": "false", 2024-04-22T18:38:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_perf": "false", 2024-04-22T18:38:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-22T18:38:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-22T18:38:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-22T18:38:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: "run_dir": "/var/run/ceph", 2024-04-22T18:38:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_block_create": "true", 2024-04-22T18:38:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_cache_lru_size": "67108864", 2024-04-22T18:38:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-22T18:38:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_default_max_object_size": "16777216", 2024-04-22T18:38:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-22T18:38:45.487 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_device_size": "53687091200", 2024-04-22T18:38:45.487 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_journal_batch_capacity": "16", 2024-04-22T18:38:45.487 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-22T18:38:45.487 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-22T18:38:45.487 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-22T18:38:45.487 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_main_device_type": "SSD", 2024-04-22T18:38:45.487 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-22T18:38:45.487 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-22T18:38:45.487 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-22T18:38:45.487 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-22T18:38:45.487 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-22T18:38:45.487 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_segment_size": "67108864", 2024-04-22T18:38:45.487 INFO:teuthology.orchestra.run.smithi089.stdout: "set_keepcaps": "false", 2024-04-22T18:38:45.487 INFO:teuthology.orchestra.run.smithi089.stdout: "setgroup": "", 2024-04-22T18:38:45.488 INFO:teuthology.orchestra.run.smithi089.stdout: "setuser": "", 2024-04-22T18:38:45.488 INFO:teuthology.orchestra.run.smithi089.stdout: "setuser_match_path": "", 2024-04-22T18:38:45.488 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-22T18:38:45.488 INFO:teuthology.orchestra.run.smithi089.stdout: "thp": "false", 2024-04-22T18:38:45.488 INFO:teuthology.orchestra.run.smithi089.stdout: "threadpool_default_timeout": "60", 2024-04-22T18:38:45.488 INFO:teuthology.orchestra.run.smithi089.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-22T18:38:45.488 INFO:teuthology.orchestra.run.smithi089.stdout: "throttler_perf_counter": "true" 2024-04-22T18:38:45.488 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:45.490 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-22T18:38:45.599 INFO:teuthology.orchestra.run.smithi089.stdout:mon.a: {} 2024-04-22T18:38:45.599 INFO:teuthology.orchestra.run.smithi089.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-22T18:38:45.609 INFO:teuthology.orchestra.run.smithi089.stdout:mon.b: {} 2024-04-22T18:38:45.609 INFO:teuthology.orchestra.run.smithi089.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-22T18:38:45.617 INFO:teuthology.orchestra.run.smithi089.stdout:mon.c: {} 2024-04-22T18:38:45.618 INFO:teuthology.orchestra.run.smithi089.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-22T18:38:45.628 DEBUG:teuthology.orchestra.run.smithi121:> type ceph-objectstore-tool 2024-04-22T18:38:45.645 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-22T18:38:45.645 DEBUG:teuthology.orchestra.run.smithi089:> type ceph-objectstore-tool 2024-04-22T18:38:45.659 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-22T18:38:45.659 INFO:teuthology.run_tasks:Running task exec... 2024-04-22T18:38:45.667 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T18:38:45.668 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi089.front.sepia.ceph.com 2024-04-22T18:38:45.668 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create low_tier 4' 2024-04-22T18:38:45.668 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-22T18:38:45.668 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:38:45.668 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:38:45.668 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-22T18:38:45.669 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-22T18:38:45.669 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-22T18:38:45.669 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-22T18:38:45.669 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:38:45.670 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-22T18:38:45.670 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:38:45.670 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-22T18:38:45.670 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:38:45.701 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:45.702+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:45.771 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:45.772+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:45.853 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:45.853 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:38:45.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:45.854 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:119 1.0 e18/14)", 2024-04-22T18:38:45.854 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:45.664792+0000", 2024-04-22T18:38:45.854 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.19059547700000001, 2024-04-22T18:38:45.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.190616487, 2024-04-22T18:38:45.854 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:45.854 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:38:45.854 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:45.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:45.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:45.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.664792+0000", 2024-04-22T18:38:45.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:45.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:45.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:45.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:45.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.664792+0000", 2024-04-22T18:38:45.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:45.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:45.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:45.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:45.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.664792+0000", 2024-04-22T18:38:45.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:45.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:45.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:45.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:45.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.664861+0000", 2024-04-22T18:38:45.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8948999999999993e-05 2024-04-22T18:38:45.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:45.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:45.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:45.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.664867+0000", 2024-04-22T18:38:45.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8150000000000002e-06 2024-04-22T18:38:45.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:45.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:45.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:45.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.664872+0000", 2024-04-22T18:38:45.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6250000000000004e-06 2024-04-22T18:38:45.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:45.856 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:45.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:45.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:45.857 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:38:45.857 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 1 2024-04-22T18:38:45.857 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:45.866 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:38:45.878 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:38:45.878 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:38:45.878 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:45.879+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:45.912 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:38:45.912 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:38:45.948 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:38:45.948 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:38:45.972 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:45.973+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:45.983 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:45.983 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:45.984 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:38:45.984 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:38:45.992 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:38:46.018 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:38:46.019 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:38:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:38:46.051 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 18, 2024-04-22T18:38:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: "fsid": "9e25744c-eff1-4a4a-8d47-0dda406b2c35", 2024-04-22T18:38:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: "created": "2024-04-22T18:38:26.841687+0000", 2024-04-22T18:38:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: "modified": "2024-04-22T18:38:44.030293+0000", 2024-04-22T18:38:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: "last_up_change": "2024-04-22T18:38:35.971820+0000", 2024-04-22T18:38:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: "last_in_change": "2024-04-22T18:38:28.942710+0000", 2024-04-22T18:38:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-22T18:38:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_num": 5799936, 2024-04-22T18:38:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_set": [ 2024-04-22T18:38:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "pglog_hardlimit", 2024-04-22T18:38:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "purged_snapdirs", 2024-04-22T18:38:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_deletes", 2024-04-22T18:38:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "sortbitwise" 2024-04-22T18:38:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:38:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_version": 5, 2024-04-22T18:38:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "full_ratio": 0.94999998807907104, 2024-04-22T18:38:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-22T18:38:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-22T18:38:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_snapshot": "", 2024-04-22T18:38:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_max": 2, 2024-04-22T18:38:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "max_osd": 8, 2024-04-22T18:38:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "require_min_compat_client": "luminous", 2024-04-22T18:38:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "min_compat_client": "jewel", 2024-04-22T18:38:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "require_osd_release": "reef", 2024-04-22T18:38:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_crimson": false, 2024-04-22T18:38:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "pools": [ 2024-04-22T18:38:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 1, 2024-04-22T18:38:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": ".mgr", 2024-04-22T18:38:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:36.991610+0000", 2024-04-22T18:38:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:38:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:38:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:38:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:38:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:38:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:38:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:38:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:38:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:38:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:38:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:38:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:38:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 1, 2024-04-22T18:38:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 1, 2024-04-22T18:38:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 1, 2024-04-22T18:38:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 1, 2024-04-22T18:38:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 1, 2024-04-22T18:38:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:38:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:38:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:38:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:38:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:38:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:38:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:38:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "16", 2024-04-22T18:38:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:38:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:38:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-04-22T18:38:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:38:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:38:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:38:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:38:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:38:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:38:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:38:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:38:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:38:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:38:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:38:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:38:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:38:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:38:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:38:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:38:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:38:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:38:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:38:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:38:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:38:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:38:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:38:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:38:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:38:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:38:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:38:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:38:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:38:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:38:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:38:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:38:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:38:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:38:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "options": { 2024-04-22T18:38:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_max": 32, 2024-04-22T18:38:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_min": 1 2024-04-22T18:38:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-04-22T18:38:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": {} 2024-04-22T18:38:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:38:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 8, 2024-04-22T18:38:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 8, 2024-04-22T18:38:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.25, 2024-04-22T18:38:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2, 2024-04-22T18:38:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2, 2024-04-22T18:38:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 1, 2024-04-22T18:38:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 1, 2024-04-22T18:38:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 1 2024-04-22T18:38:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 2, 2024-04-22T18:38:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "rbd", 2024-04-22T18:38:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:37.622620+0000", 2024-04-22T18:38:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:38:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:38:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:38:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:38:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:38:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:38:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:38:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:38:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:38:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:38:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:38:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:38:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 8, 2024-04-22T18:38:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 8, 2024-04-22T18:38:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 8, 2024-04-22T18:38:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 8, 2024-04-22T18:38:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 8, 2024-04-22T18:38:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:38:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:38:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:38:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:38:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:38:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:38:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:38:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "17", 2024-04-22T18:38:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:38:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:38:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-04-22T18:38:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:38:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:38:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:38:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:38:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:38:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:38:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:38:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:38:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:38:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:38:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:38:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:38:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:38:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:38:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:38:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:38:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:38:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:38:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:38:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:38:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:38:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:38:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:38:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:38:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:38:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:38:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:38:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:38:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:38:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:38:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:38:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:38:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:38:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:38:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "options": {}, 2024-04-22T18:38:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-04-22T18:38:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd": {} 2024-04-22T18:38:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:38:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 1.9900000095367432, 2024-04-22T18:38:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 1.9900000095367432, 2024-04-22T18:38:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.87999999523162842, 2024-04-22T18:38:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 1.75, 2024-04-22T18:38:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 1.75, 2024-04-22T18:38:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 1, 2024-04-22T18:38:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 1, 2024-04-22T18:38:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 1 2024-04-22T18:38:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:38:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "osds": [ 2024-04-22T18:38:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-04-22T18:38:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "bbf33cfb-0f07-4df6-b90f-83e9edb039a6", 2024-04-22T18:38:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:38:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:38:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:38:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:38:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:38:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:38:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:38:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 0, 2024-04-22T18:38:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:38:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:38:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:38:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6800", 2024-04-22T18:38:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:38:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:38:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6801", 2024-04-22T18:38:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:38:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:38:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6803", 2024-04-22T18:38:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:38:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:38:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6802", 2024-04-22T18:38:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:38:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6800/31890", 2024-04-22T18:38:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6801/31890", 2024-04-22T18:38:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6803/31890", 2024-04-22T18:38:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6802/31890", 2024-04-22T18:38:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:38:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:38:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:38:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-04-22T18:38:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "30415a8e-9f47-416f-9fd0-716c2cb3f562", 2024-04-22T18:38:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:38:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:38:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:38:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:38:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:38:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:38:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:38:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 15, 2024-04-22T18:38:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:38:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:38:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:38:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6804", 2024-04-22T18:38:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:38:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:38:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6805", 2024-04-22T18:38:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:38:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:38:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6807", 2024-04-22T18:38:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:38:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:38:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6806", 2024-04-22T18:38:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:38:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6804/31899", 2024-04-22T18:38:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6805/31899", 2024-04-22T18:38:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6807/31899", 2024-04-22T18:38:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6806/31899", 2024-04-22T18:38:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:38:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:38:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:38:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-04-22T18:38:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "b8da6b88-8355-4225-930d-bfed881089b7", 2024-04-22T18:38:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:38:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:38:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:38:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:38:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:38:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:38:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:38:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 15, 2024-04-22T18:38:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:38:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:38:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:38:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6812", 2024-04-22T18:38:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:38:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:38:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6813", 2024-04-22T18:38:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:38:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:38:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6815", 2024-04-22T18:38:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:38:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:38:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6814", 2024-04-22T18:38:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:38:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6812/31903", 2024-04-22T18:38:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6813/31903", 2024-04-22T18:38:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6815/31903", 2024-04-22T18:38:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6814/31903", 2024-04-22T18:38:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:38:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:38:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:38:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-04-22T18:38:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "3eaec913-9d7d-4b03-a28f-0c334e7c39f1", 2024-04-22T18:38:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:38:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:38:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:38:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:38:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:38:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:38:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:38:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 0, 2024-04-22T18:38:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:38:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:38:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:38:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6808", 2024-04-22T18:38:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:38:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:38:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6809", 2024-04-22T18:38:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:38:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:38:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6811", 2024-04-22T18:38:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:38:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:38:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6810", 2024-04-22T18:38:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:38:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6808/31913", 2024-04-22T18:38:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6809/31913", 2024-04-22T18:38:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6811/31913", 2024-04-22T18:38:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6810/31913", 2024-04-22T18:38:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:38:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:38:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:38:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-04-22T18:38:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "887acc3b-6f6b-49eb-a039-13050a2c7d64", 2024-04-22T18:38:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:38:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:38:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:38:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:38:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:38:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:38:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:38:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 0, 2024-04-22T18:38:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:38:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:38:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:38:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6808", 2024-04-22T18:38:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:38:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:38:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6809", 2024-04-22T18:38:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:38:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:38:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6811", 2024-04-22T18:38:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:38:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:38:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6810", 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6808/31922", 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6809/31922", 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6811/31922", 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6810/31922", 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-04-22T18:38:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "dcaddfd0-f8d1-4c28-bda2-d17eed606b60", 2024-04-22T18:38:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:38:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:38:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:38:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:38:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:38:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:38:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:38:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 15, 2024-04-22T18:38:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:38:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:38:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:38:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6804", 2024-04-22T18:38:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:38:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:38:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6805", 2024-04-22T18:38:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:38:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:38:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6807", 2024-04-22T18:38:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:38:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:38:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6806", 2024-04-22T18:38:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:38:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6804/31921", 2024-04-22T18:38:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6805/31921", 2024-04-22T18:38:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6807/31921", 2024-04-22T18:38:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6806/31921", 2024-04-22T18:38:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:38:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:38:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:38:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-04-22T18:38:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "acf0527b-3020-4b0a-b64c-5068d21a5781", 2024-04-22T18:38:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:38:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:38:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:38:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:38:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:38:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:38:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:38:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 15, 2024-04-22T18:38:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:38:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:38:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:38:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6800", 2024-04-22T18:38:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:38:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:38:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6801", 2024-04-22T18:38:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:38:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:38:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6803", 2024-04-22T18:38:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:38:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:38:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6802", 2024-04-22T18:38:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:38:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6800/31920", 2024-04-22T18:38:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6801/31920", 2024-04-22T18:38:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6803/31920", 2024-04-22T18:38:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6802/31920", 2024-04-22T18:38:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:38:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:38:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:38:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-04-22T18:38:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "305394bc-be3d-420b-88d7-d273670540b1", 2024-04-22T18:38:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:38:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:38:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:38:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:38:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:38:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:38:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:38:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 15, 2024-04-22T18:38:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:38:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:38:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:38:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6812", 2024-04-22T18:38:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:38:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:38:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6813", 2024-04-22T18:38:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:38:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:38:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6815", 2024-04-22T18:38:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:38:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:38:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:38:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:38:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6814", 2024-04-22T18:38:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:38:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6812/31939", 2024-04-22T18:38:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6813/31939", 2024-04-22T18:38:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6815/31939", 2024-04-22T18:38:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6814/31939", 2024-04-22T18:38:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:38:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:38:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:38:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:38:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_xinfo": [ 2024-04-22T18:38:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-04-22T18:38:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:38:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:38:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:38:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:38:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:38:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.023620+0000", 2024-04-22T18:38:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:38:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-04-22T18:38:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:38:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:38:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:38:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:38:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:38:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.099729+0000", 2024-04-22T18:38:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:38:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-04-22T18:38:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:38:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:38:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:38:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:38:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:38:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.074483+0000", 2024-04-22T18:38:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:38:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-04-22T18:38:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:38:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:38:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:38:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:38:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:38:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.122275+0000", 2024-04-22T18:38:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:38:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-04-22T18:38:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:38:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:38:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:38:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:38:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:38:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.116180+0000", 2024-04-22T18:38:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:38:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-04-22T18:38:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:38:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:38:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:38:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:38:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:38:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.108716+0000", 2024-04-22T18:38:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:38:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-04-22T18:38:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:38:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:38:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:38:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:38:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:38:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.005302+0000", 2024-04-22T18:38:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:38:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-04-22T18:38:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:38:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:38:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:38:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:38:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:38:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.108554+0000", 2024-04-22T18:38:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:38:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:38:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap": [], 2024-04-22T18:38:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_items": [], 2024-04-22T18:38:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_primaries": [], 2024-04-22T18:38:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_temp": [], 2024-04-22T18:38:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_temp": [], 2024-04-22T18:38:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "blocklist": {}, 2024-04-22T18:38:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "range_blocklist": {}, 2024-04-22T18:38:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profiles": { 2024-04-22T18:38:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "default": { 2024-04-22T18:38:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "crush-failure-domain": "osd", 2024-04-22T18:38:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "k": "2", 2024-04-22T18:38:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "m": "1", 2024-04-22T18:38:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "plugin": "jerasure", 2024-04-22T18:38:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "technique": "reed_sol_van" 2024-04-22T18:38:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps_queue": [], 2024-04-22T18:38:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "new_removed_snaps": [], 2024-04-22T18:38:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "new_purged_snaps": [], 2024-04-22T18:38:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_node_flags": {}, 2024-04-22T18:38:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "device_class_flags": {}, 2024-04-22T18:38:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode": { 2024-04-22T18:38:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_enabled": false, 2024-04-22T18:38:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_bucket_count": 0, 2024-04-22T18:38:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "degraded_stretch_mode": 0, 2024-04-22T18:38:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "recovering_stretch_mode": 0, 2024-04-22T18:38:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_bucket": 0 2024-04-22T18:38:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:46.106 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:46.107 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:38:46.107 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:38:46.107 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:46.067+0000 7f3e79e84640 -1 osd.0 18 handle_osd_map injecting map crc failure 2024-04-22T18:38:46.108 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'low_tier' created 2024-04-22T18:38:46.109 INFO:tasks.thrashosds.thrasher:j is {'epoch': 18, 'fsid': '9e25744c-eff1-4a4a-8d47-0dda406b2c35', 'created': '2024-04-22T18:38:26.841687+0000', 'modified': '2024-04-22T18:38:44.030293+0000', 'last_up_change': '2024-04-22T18:38:35.971820+0000', 'last_in_change': '2024-04-22T18:38:28.942710+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T18:38:36.991610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-22T18:38:37.622620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'bbf33cfb-0f07-4df6-b90f-83e9edb039a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6800', 'nonce': 31890}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6801', 'nonce': 31890}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6803', 'nonce': 31890}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6802', 'nonce': 31890}]}, 'public_addr': '172.21.15.89:6800/31890', 'cluster_addr': '172.21.15.89:6801/31890', 'heartbeat_back_addr': '172.21.15.89:6803/31890', 'heartbeat_front_addr': '172.21.15.89:6802/31890', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '30415a8e-9f47-416f-9fd0-716c2cb3f562', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6804', 'nonce': 31899}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6805', 'nonce': 31899}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6807', 'nonce': 31899}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6806', 'nonce': 31899}]}, 'public_addr': '172.21.15.89:6804/31899', 'cluster_addr': '172.21.15.89:6805/31899', 'heartbeat_back_addr': '172.21.15.89:6807/31899', 'heartbeat_front_addr': '172.21.15.89:6806/31899', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b8da6b88-8355-4225-930d-bfed881089b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6812', 'nonce': 31903}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6813', 'nonce': 31903}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6815', 'nonce': 31903}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6814', 'nonce': 31903}]}, 'public_addr': '172.21.15.89:6812/31903', 'cluster_addr': '172.21.15.89:6813/31903', 'heartbeat_back_addr': '172.21.15.89:6815/31903', 'heartbeat_front_addr': '172.21.15.89:6814/31903', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3eaec913-9d7d-4b03-a28f-0c334e7c39f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6808', 'nonce': 31913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6809', 'nonce': 31913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6811', 'nonce': 31913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6810', 'nonce': 31913}]}, 'public_addr': '172.21.15.89:6808/31913', 'cluster_addr': '172.21.15.89:6809/31913', 'heartbeat_back_addr': '172.21.15.89:6811/31913', 'heartbeat_front_addr': '172.21.15.89:6810/31913', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '887acc3b-6f6b-49eb-a039-13050a2c7d64', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 31922}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 31922}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 31922}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 31922}]}, 'public_addr': '172.21.15.121:6808/31922', 'cluster_addr': '172.21.15.121:6809/31922', 'heartbeat_back_addr': '172.21.15.121:6811/31922', 'heartbeat_front_addr': '172.21.15.121:6810/31922', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'dcaddfd0-f8d1-4c28-bda2-d17eed606b60', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 31921}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 31921}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 31921}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 31921}]}, 'public_addr': '172.21.15.121:6804/31921', 'cluster_addr': '172.21.15.121:6805/31921', 'heartbeat_back_addr': '172.21.15.121:6807/31921', 'heartbeat_front_addr': '172.21.15.121:6806/31921', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'acf0527b-3020-4b0a-b64c-5068d21a5781', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 31920}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 31920}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 31920}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 31920}]}, 'public_addr': '172.21.15.121:6800/31920', 'cluster_addr': '172.21.15.121:6801/31920', 'heartbeat_back_addr': '172.21.15.121:6803/31920', 'heartbeat_front_addr': '172.21.15.121:6802/31920', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '305394bc-be3d-420b-88d7-d273670540b1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 31939}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 31939}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 31939}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 31939}]}, 'public_addr': '172.21.15.121:6812/31939', 'cluster_addr': '172.21.15.121:6813/31939', 'heartbeat_back_addr': '172.21.15.121:6815/31939', 'heartbeat_front_addr': '172.21.15.121:6814/31939', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.023620+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.099729+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.074483+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.122275+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.116180+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.108716+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.005302+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.108554+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-22T18:38:46.109 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:38:46.109 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:38:46.110 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:38:46.110 INFO:teuthology.run_tasks:Running task rados... 2024-04-22T18:38:46.121 INFO:tasks.rados:Beginning rados... 2024-04-22T18:38:46.121 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-04-22T18:38:46.129 INFO:tasks.rados:joining rados 2024-04-22T18:38:46.129 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:46.108+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:46.130 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2024-04-22T18:38:46.130 INFO:tasks.rados:starting run 0 out of 1 2024-04-22T18:38:46.130 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-22T18:38:46.130 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-04-22T18:38:46.133 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:46.133 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:46.142 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:38:46.184 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:38:46.184 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:38:46.238 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:46.239+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:46.295 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:46.295 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:46.304 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:38:46.308 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:46.309+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:46.409 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:46.410+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:46.439 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:46.440+0000 7f3cfee88640 -1 osd.6 18 handle_osd_map injecting map crc failure 2024-04-22T18:38:46.449 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:46.449 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:46.458 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:38:46.468 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:38:46.468 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:38:46.482 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-04-22T18:38:46.065254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":56,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30,"ondisk_log_size":30,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":217336,"kb_used_data":1152,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757384,"statfs":{"total":773094113280,"available":772871561216,"internally_reserved":0,"allocated":1179648,"data_stored":865502,"data_compressed":22212,"data_compressed_allocated":28672,"data_compressed_original":458752,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":11,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.104038"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.639034+0000","last_change":"2024-04-22T18:38:39.026078+0000","last_active":"2024-04-22T18:38:40.639034+0000","last_peered":"2024-04-22T18:38:40.639034+0000","last_clean":"2024-04-22T18:38:40.639034+0000","last_became_active":"2024-04-22T18:38:39.025741+0000","last_became_peered":"2024-04-22T18:38:39.025741+0000","last_unstale":"2024-04-22T18:38:40.639034+0000","last_undegraded":"2024-04-22T18:38:40.639034+0000","last_fullsized":"2024-04-22T18:38:40.639034+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:01.125349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:41.709283+0000","last_change":"2024-04-22T18:38:39.110499+0000","last_active":"2024-04-22T18:38:41.709283+0000","last_peered":"2024-04-22T18:38:41.709283+0000","last_clean":"2024-04-22T18:38:41.709283+0000","last_became_active":"2024-04-22T18:38:39.010173+0000","last_became_peered":"2024-04-22T18:38:39.010173+0000","last_unstale":"2024-04-22T18:38:41.709283+0000","last_undegraded":"2024-04-22T18:38:41.709283+0000","last_fullsized":"2024-04-22T18:38:41.709283+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:58.222501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.118329+0000","last_change":"2024-04-22T18:38:39.210344+0000","last_active":"2024-04-22T18:38:40.118329+0000","last_peered":"2024-04-22T18:38:40.118329+0000","last_clean":"2024-04-22T18:38:40.118329+0000","last_became_active":"2024-04-22T18:38:39.210049+0000","last_became_peered":"2024-04-22T18:38:39.210049+0000","last_unstale":"2024-04-22T18:38:40.118329+0000","last_undegraded":"2024-04-22T18:38:40.118329+0000","last_fullsized":"2024-04-22T18:38:40.118329+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.878594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:41.709260+0000","last_change":"2024-04-22T18:38:39.307735+0000","last_active":"2024-04-22T18:38:41.709260+0000","last_peered":"2024-04-22T18:38:41.709260+0000","last_clean":"2024-04-22T18:38:41.709260+0000","last_became_active":"2024-04-22T18:38:39.307459+0000","last_became_peered":"2024-04-22T18:38:39.307459+0000","last_unstale":"2024-04-22T18:38:41.709260+0000","last_undegraded":"2024-04-22T18:38:41.709260+0000","last_fullsized":"2024-04-22T18:38:41.709260+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.680561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-22T18:38:46.054802+0000","last_change":"2024-04-22T18:38:46.054802+0000","last_active":"2024-04-22T18:38:46.054802+0000","last_peered":"2024-04-22T18:38:46.054802+0000","last_clean":"2024-04-22T18:38:46.054802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T18:38:46.054802+0000","last_undegraded":"2024-04-22T18:38:46.054802+0000","last_fullsized":"2024-04-22T18:38:46.054802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.484236+0000","last_change":"2024-04-22T18:38:39.111223+0000","last_active":"2024-04-22T18:38:40.484236+0000","last_peered":"2024-04-22T18:38:40.484236+0000","last_clean":"2024-04-22T18:38:40.484236+0000","last_became_active":"2024-04-22T18:38:39.110914+0000","last_became_peered":"2024-04-22T18:38:39.110914+0000","last_unstale":"2024-04-22T18:38:40.484236+0000","last_undegraded":"2024-04-22T18:38:40.484236+0000","last_fullsized":"2024-04-22T18:38:40.484236+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.570327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-22T18:38:46.054802+0000","last_change":"2024-04-22T18:38:46.054802+0000","last_active":"2024-04-22T18:38:46.054802+0000","last_peered":"2024-04-22T18:38:46.054802+0000","last_clean":"2024-04-22T18:38:46.054802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T18:38:46.054802+0000","last_undegraded":"2024-04-22T18:38:46.054802+0000","last_fullsized":"2024-04-22T18:38:46.054802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:41.709256+0000","last_change":"2024-04-22T18:38:39.164369+0000","last_active":"2024-04-22T18:38:41.709256+0000","last_peered":"2024-04-22T18:38:41.709256+0000","last_clean":"2024-04-22T18:38:41.709256+0000","last_became_active":"2024-04-22T18:38:39.164089+0000","last_became_peered":"2024-04-22T18:38:39.164089+0000","last_unstale":"2024-04-22T18:38:41.709256+0000","last_undegraded":"2024-04-22T18:38:41.709256+0000","last_fullsized":"2024-04-22T18:38:41.709256+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:44.825470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-22T18:38:46.054802+0000","last_change":"2024-04-22T18:38:46.054802+0000","last_active":"2024-04-22T18:38:46.054802+0000","last_peered":"2024-04-22T18:38:46.054802+0000","last_clean":"2024-04-22T18:38:46.054802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T18:38:46.054802+0000","last_undegraded":"2024-04-22T18:38:46.054802+0000","last_fullsized":"2024-04-22T18:38:46.054802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.018146+0000","last_change":"2024-04-22T18:38:39.125885+0000","last_active":"2024-04-22T18:38:40.018146+0000","last_peered":"2024-04-22T18:38:40.018146+0000","last_clean":"2024-04-22T18:38:40.018146+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:38:40.018146+0000","last_undegraded":"2024-04-22T18:38:40.018146+0000","last_fullsized":"2024-04-22T18:38:40.018146+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.321452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-22T18:38:46.054802+0000","last_change":"2024-04-22T18:38:46.054802+0000","last_active":"2024-04-22T18:38:46.054802+0000","last_peered":"2024-04-22T18:38:46.054802+0000","last_clean":"2024-04-22T18:38:46.054802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T18:38:46.054802+0000","last_undegraded":"2024-04-22T18:38:46.054802+0000","last_fullsized":"2024-04-22T18:38:46.054802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:40.584345+0000","last_change":"2024-04-22T18:38:39.010398+0000","last_active":"2024-04-22T18:38:40.584345+0000","last_peered":"2024-04-22T18:38:40.584345+0000","last_clean":"2024-04-22T18:38:40.584345+0000","last_became_active":"2024-04-22T18:38:39.010101+0000","last_became_peered":"2024-04-22T18:38:39.010101+0000","last_unstale":"2024-04-22T18:38:40.584345+0000","last_undegraded":"2024-04-22T18:38:40.584345+0000","last_fullsized":"2024-04-22T18:38:40.584345+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.708134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'30","reported_seq":56,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-22T18:38:41.067411+0000","last_change":"2024-04-22T18:38:38.109358+0000","last_active":"2024-04-22T18:38:41.067411+0000","last_peered":"2024-04-22T18:38:41.067411+0000","last_clean":"2024-04-22T18:38:41.067411+0000","last_became_active":"2024-04-22T18:38:38.108568+0000","last_became_peered":"2024-04-22T18:38:38.108568+0000","last_unstale":"2024-04-22T18:38:41.067411+0000","last_undegraded":"2024-04-22T18:38:41.067411+0000","last_fullsized":"2024-04-22T18:38:41.067411+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:38:36.991764+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:46.436569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":56,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":56,"num_write_kb":392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":544792,"data_compressed":27575,"data_compressed_allocated":36864,"data_compressed_original":520192,"omap_allocated":0,"internal_metadata":0},"log_size":30,"ondisk_log_size":30,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":184888,"data_compressed":6340,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":85.509}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":85.391000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":159744,"data_stored":51798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":118784,"data_stored":47542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27136,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":118784,"data_stored":47542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]}]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":46194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":118784,"data_stored":47542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":159744,"data_stored":51798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":192512,"data_stored":388198,"data_compressed":15872,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":405516,"data_compressed":21210,"data_compressed_allocated":28672,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6365,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:38:46.483 INFO:tasks.thrashosds.thrasher:No pools; doing nothing 2024-04-22T18:38:46.510 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:46.511+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:46.560 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:46.561 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:46.569 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:38:46.610 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:46.611+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:46.671 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:46.671 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:46.679 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:38:46.711 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:46.712+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:46.780 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:46.780 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:46.790 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:38:46.834 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:46.835+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:46.891 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:46.891 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:46.900 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:38:46.941 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:46.942+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:47.014 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:47.015+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:38:47.042 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:47.042 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:47.051 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:38:47.065 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:47.066+0000 7fa766d7d640 -1 osd.5 19 handle_osd_map injecting map crc failure 2024-04-22T18:38:47.067 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:38:47.067 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'unique_pool_0' created 2024-04-22T18:38:47.068 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:47.069+0000 7fec55bea640 -1 osd.7 19 handle_osd_map injecting map crc failure 2024-04-22T18:38:47.087 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-22T18:38:47.128 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:47.129+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:38:47.175 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:47.175 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:47.185 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:38:47.216 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:47.217+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:47.312 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:47.312 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:47.317 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:47.318+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:47.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:38:47.418 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:47.419+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:47.423 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:47.423 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:47.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:38:47.519 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:47.521+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:38:47.573 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:47.573 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:47.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:38:47.626 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:47.627+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:47.687 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:47.687 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:47.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:38:47.721 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:47.722+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:47.822 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:47.822+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:47.841 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:47.841 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:47.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:38:47.922 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:47.924+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:47.995 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:47.995 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:48.004 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:38:48.047 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:48.048+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:48.068 INFO:teuthology.orchestra.run.smithi089.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-22T18:38:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:48.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:38:48.158 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:48.159+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:48.196 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:38:48.241 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:48.242+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:48.261 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:48.261 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:48.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:38:48.315 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:38:48.315 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:38:48.326 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:48.327+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:38:48.331 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:38:48.331 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:38:48.347 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:38:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:38:48.363 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:38:48.363 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:38:48.371 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:48.371 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:48.379 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:38:48.380 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:38:48.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:38:48.396 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:38:48.396 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:38:48.412 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:38:48.412 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:38:48.427 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:48.427+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:48.428 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:38:48.429 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:38:48.483 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:48.484 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:38:48.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:38:48.484 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [] 2024-04-22T18:38:48.484 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:48.492 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:38:48.534 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:48.535+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:48.629 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:48.630+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:48.634 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:48.634 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:38:48.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.634 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:160 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:38:48.634 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454851+0000", 2024-04-22T18:38:48.634 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.18108263699999999, 2024-04-22T18:38:48.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18111386199999999, 2024-04-22T18:38:48.634 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:48.635 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:38:48.635 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:48.635 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:48.635 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:48.635 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 160 2024-04-22T18:38:48.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.635 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:48.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:48.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454851+0000", 2024-04-22T18:38:48.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:48.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454851+0000", 2024-04-22T18:38:48.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:48.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454849+0000", 2024-04-22T18:38:48.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:48.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:48.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454917+0000", 2024-04-22T18:38:48.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8095000000000004e-05 2024-04-22T18:38:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454927+0000", 2024-04-22T18:38:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5519999999999993e-06 2024-04-22T18:38:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454932+0000", 2024-04-22T18:38:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2490000000000001e-06 2024-04-22T18:38:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:48.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555049+0000", 2024-04-22T18:38:48.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100116606 2024-04-22T18:38:48.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:48.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555132+0000", 2024-04-22T18:38:48.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3358999999999999e-05 2024-04-22T18:38:48.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:48.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555230+0000", 2024-04-22T18:38:48.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7547999999999995e-05 2024-04-22T18:38:48.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:48.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556139+0000", 2024-04-22T18:38:48.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090974299999999997 2024-04-22T18:38:48.641 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.642 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:48.642 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.643 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:161 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:38:48.644 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:38:48.644 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.18099421900000001, 2024-04-22T18:38:48.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18107548000000001, 2024-04-22T18:38:48.644 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:48.644 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:38:48.644 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:48.644 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:48.644 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:48.644 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 161 2024-04-22T18:38:48.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.644 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:48.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:48.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:38:48.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:48.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:38:48.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:48.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454938+0000", 2024-04-22T18:38:48.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:48.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:48.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454982+0000", 2024-04-22T18:38:48.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3470999999999998e-05 2024-04-22T18:38:48.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:48.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454988+0000", 2024-04-22T18:38:48.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7640000000000002e-06 2024-04-22T18:38:48.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:48.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555054+0000", 2024-04-22T18:38:48.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006607300000001 2024-04-22T18:38:48.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:48.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555607+0000", 2024-04-22T18:38:48.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055319500000000003 2024-04-22T18:38:48.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:48.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555689+0000", 2024-04-22T18:38:48.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1817999999999995e-05 2024-04-22T18:38:48.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:48.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555804+0000", 2024-04-22T18:38:48.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115346 2024-04-22T18:38:48.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:48.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556554+0000", 2024-04-22T18:38:48.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000750102 2024-04-22T18:38:48.648 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.648 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:48.648 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.648 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.648 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:38:48.648 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 2 2024-04-22T18:38:48.648 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:48.649 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:38:48.733 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:48.734+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:48.791 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:48.791 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:38:48.791 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:38:48.791 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:38:48.791 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:48.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:38:48.841 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:48.842+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:48.931 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:48.933+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:48.944 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:48.944 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:38:48.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:38:48.944 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:38:48.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.944 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:38:48.944 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:38:48.944 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.2018352379999993, 2024-04-22T18:38:48.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30520528099999999, 2024-04-22T18:38:48.944 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:48.944 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:48.945 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:48.945 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:48.945 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:48.945 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 35 2024-04-22T18:38:48.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.945 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:48.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:48.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:38:48.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:48.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:38:48.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:48.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741852+0000", 2024-04-22T18:38:48.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:48.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:48.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741927+0000", 2024-04-22T18:38:48.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5123000000000006e-05 2024-04-22T18:38:48.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:48.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741953+0000", 2024-04-22T18:38:48.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5378999999999999e-05 2024-04-22T18:38:48.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:48.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741977+0000", 2024-04-22T18:38:48.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4431999999999999e-05 2024-04-22T18:38:48.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:48.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844243+0000", 2024-04-22T18:38:48.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102265525 2024-04-22T18:38:48.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844330+0000", 2024-04-22T18:38:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7417999999999995e-05 2024-04-22T18:38:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844470+0000", 2024-04-22T18:38:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000140197 2024-04-22T18:38:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944767+0000", 2024-04-22T18:38:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100296409 2024-04-22T18:38:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:48.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.046968+0000", 2024-04-22T18:38:48.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10220148499999999 2024-04-22T18:38:48.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:48.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047018+0000", 2024-04-22T18:38:48.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9206000000000003e-05 2024-04-22T18:38:48.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:48.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047059+0000", 2024-04-22T18:38:48.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1496999999999999e-05 2024-04-22T18:38:48.949 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.949 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:48.950 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.950 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:38:48.950 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:38:48.950 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.2017047139999999, 2024-04-22T18:38:48.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:38:48.950 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:48.950 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:48.950 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:48.950 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:48.950 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:48.950 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:38:48.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.951 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:48.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:48.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:38:48.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:48.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:38:48.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:48.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:38:48.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:48.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:48.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:38:48.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:38:48.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:48.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:38:48.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:38:48.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:48.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:38:48.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:38:48.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:48.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:38:48.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:38:48.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:48.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:38:48.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:38:48.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:48.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:38:48.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:38:48.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:48.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:38:48.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:38:48.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:38:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:38:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:38:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:38:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:38:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:38:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:48.955 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.956 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:38:48.956 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:38:48.956 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.2016650230000003, 2024-04-22T18:38:48.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:38:48.956 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:48.956 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:48.956 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:48.956 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:48.956 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:48.956 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:38:48.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.956 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:48.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:48.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:38:48.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:48.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:38:48.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:48.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:38:48.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:48.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:48.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:38:48.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:38:48.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:48.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:38:48.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:38:48.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:48.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:38:48.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:38:48.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:48.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:38:48.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:38:48.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:48.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:38:48.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:38:48.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:48.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:38:48.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:38:48.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:48.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:38:48.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:38:48.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:48.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:38:48.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:38:48.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:48.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:38:48.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:38:48.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:48.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:38:48.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:38:48.961 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.961 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:48.961 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.961 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:38:48.962 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:38:48.962 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.9574839759999998, 2024-04-22T18:38:48.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402419109, 2024-04-22T18:38:48.962 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:48.962 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:48.962 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:48.962 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:48.962 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:48.962 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 68 2024-04-22T18:38:48.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.962 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:48.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:48.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:38:48.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:48.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:38:48.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:48.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986204+0000", 2024-04-22T18:38:48.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:48.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:48.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986208+0000", 2024-04-22T18:38:48.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8149999999999999e-06 2024-04-22T18:38:48.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:48.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986214+0000", 2024-04-22T18:38:48.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.604e-06 2024-04-22T18:38:48.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:48.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986218+0000", 2024-04-22T18:38:48.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6689999999999999e-06 2024-04-22T18:38:48.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:48.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986833+0000", 2024-04-22T18:38:48.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000615648 2024-04-22T18:38:48.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:48.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986891+0000", 2024-04-22T18:38:48.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7915999999999999e-05 2024-04-22T18:38:48.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:48.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986981+0000", 2024-04-22T18:38:48.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759999999999994e-05 2024-04-22T18:38:48.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:48.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.987584+0000", 2024-04-22T18:38:48.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060270199999999999 2024-04-22T18:38:48.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:48.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388527+0000", 2024-04-22T18:38:48.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40094366999999997 2024-04-22T18:38:48.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:48.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388583+0000", 2024-04-22T18:38:48.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5383000000000002e-05 2024-04-22T18:38:48.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388624+0000", 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501999999999997e-05 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.9574681869999999, 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:48.967 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:38:48.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:48.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:48.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:38:48.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:38:48.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:48.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:38:48.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:38:48.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:48.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:38:48.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:38:48.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:48.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:38:48.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:38:48.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:48.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:38:48.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:38:48.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:48.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:38:48.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:38:48.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:48.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:38:48.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:38:48.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:48.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:38:48.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:38:48.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:48.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:38:48.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:38:48.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:48.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:38:48.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:38:48.972 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.972 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:48.972 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.973 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:38:48.973 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:38:48.973 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.8410751779999996, 2024-04-22T18:38:48.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305563733, 2024-04-22T18:38:48.973 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:48.973 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:48.973 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:48.973 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:48.973 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:48.973 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 87 2024-04-22T18:38:48.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.973 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:48.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:48.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:38:48.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:48.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:38:48.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:48.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102613+0000", 2024-04-22T18:38:48.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:48.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:48.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102668+0000", 2024-04-22T18:38:48.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4669000000000001e-05 2024-04-22T18:38:48.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:48.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102680+0000", 2024-04-22T18:38:48.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2026e-05 2024-04-22T18:38:48.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:48.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102687+0000", 2024-04-22T18:38:48.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8339999999999999e-06 2024-04-22T18:38:48.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:48.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405644+0000", 2024-04-22T18:38:48.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30295623199999999 2024-04-22T18:38:48.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:48.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405743+0000", 2024-04-22T18:38:48.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.98e-05 2024-04-22T18:38:48.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:48.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405868+0000", 2024-04-22T18:38:48.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012457899999999999 2024-04-22T18:38:48.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:48.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407047+0000", 2024-04-22T18:38:48.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011791449999999999 2024-04-22T18:38:48.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:48.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408088+0000", 2024-04-22T18:38:48.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010411680000000001 2024-04-22T18:38:48.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:48.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408135+0000", 2024-04-22T18:38:48.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6255999999999999e-05 2024-04-22T18:38:48.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:48.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408178+0000", 2024-04-22T18:38:48.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2877000000000003e-05 2024-04-22T18:38:48.978 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.978 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:48.978 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.978 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:38:48.978 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:38:48.978 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.8409952489999997, 2024-04-22T18:38:48.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30560089000000001, 2024-04-22T18:38:48.978 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:48.978 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:48.979 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:48.979 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:48.979 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:48.979 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 88 2024-04-22T18:38:48.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.979 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:48.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:48.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:38:48.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:48.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:38:48.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:48.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102693+0000", 2024-04-22T18:38:48.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:48.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:48.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102698+0000", 2024-04-22T18:38:48.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9699999999999998e-06 2024-04-22T18:38:48.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:48.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102708+0000", 2024-04-22T18:38:48.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.006e-05 2024-04-22T18:38:48.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:48.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102715+0000", 2024-04-22T18:38:48.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6599999999999995e-06 2024-04-22T18:38:48.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:48.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406637+0000", 2024-04-22T18:38:48.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392130099999998 2024-04-22T18:38:48.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:48.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406696+0000", 2024-04-22T18:38:48.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9066e-05 2024-04-22T18:38:48.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:48.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406784+0000", 2024-04-22T18:38:48.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8194000000000003e-05 2024-04-22T18:38:48.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:48.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407447+0000", 2024-04-22T18:38:48.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066295700000000004 2024-04-22T18:38:48.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408237+0000", 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079032600000000005 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408270+0000", 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3157000000000001e-05 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408295+0000", 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4151999999999998e-05 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.984 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:48.984 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:38:48.984 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.0897631250000002, 2024-04-22T18:38:48.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30533819600000001, 2024-04-22T18:38:48.984 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:48.984 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:48.984 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:48.984 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:48.984 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:48.984 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 106 2024-04-22T18:38:48.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.984 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:48.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:48.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:38:48.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:48.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:38:48.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:48.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853925+0000", 2024-04-22T18:38:48.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:48.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:48.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853958+0000", 2024-04-22T18:38:48.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2536000000000002e-05 2024-04-22T18:38:48.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:48.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853964+0000", 2024-04-22T18:38:48.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6239999999999999e-06 2024-04-22T18:38:48.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:48.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853967+0000", 2024-04-22T18:38:48.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9700000000000001e-06 2024-04-22T18:38:48.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:48.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855144+0000", 2024-04-22T18:38:48.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001176774 2024-04-22T18:38:48.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:48.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855225+0000", 2024-04-22T18:38:48.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0697000000000003e-05 2024-04-22T18:38:48.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:48.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855353+0000", 2024-04-22T18:38:48.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012788 2024-04-22T18:38:48.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:48.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058003+0000", 2024-04-22T18:38:48.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202649939 2024-04-22T18:38:48.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:48.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159184+0000", 2024-04-22T18:38:48.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101181058 2024-04-22T18:38:48.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:48.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159238+0000", 2024-04-22T18:38:48.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3967e-05 2024-04-22T18:38:48.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:48.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159264+0000", 2024-04-22T18:38:48.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6237999999999998e-05 2024-04-22T18:38:48.989 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.989 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:48.989 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.990 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:48.990 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:38:48.990 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.0897162749999998, 2024-04-22T18:38:48.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30544294500000002, 2024-04-22T18:38:48.990 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:48.990 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:48.990 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:48.990 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:48.990 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:48.990 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 107 2024-04-22T18:38:48.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.991 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:48.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:48.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:38:48.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:48.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:38:48.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:48.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853972+0000", 2024-04-22T18:38:48.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:48.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:48.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853975+0000", 2024-04-22T18:38:48.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0939999999999999e-06 2024-04-22T18:38:48.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:48.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853978+0000", 2024-04-22T18:38:48.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:38:48.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:48.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853981+0000", 2024-04-22T18:38:48.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.858e-06 2024-04-22T18:38:48.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:48.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855831+0000", 2024-04-22T18:38:48.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018503689999999999 2024-04-22T18:38:48.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:48.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855932+0000", 2024-04-22T18:38:48.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010071 2024-04-22T18:38:48.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:48.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856029+0000", 2024-04-22T18:38:48.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6751999999999998e-05 2024-04-22T18:38:48.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:48.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058013+0000", 2024-04-22T18:38:48.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20198412600000001 2024-04-22T18:38:48.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:48.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159326+0000", 2024-04-22T18:38:48.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101312969 2024-04-22T18:38:48.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:48.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159376+0000", 2024-04-22T18:38:48.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9968e-05 2024-04-22T18:38:48.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:48.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159416+0000", 2024-04-22T18:38:48.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9641999999999998e-05 2024-04-22T18:38:48.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.995 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:48.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:48.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.995 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:48.996 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:38:48.996 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.089627879, 2024-04-22T18:38:48.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30548257499999998, 2024-04-22T18:38:48.996 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:48.996 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:48.996 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:48.996 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:48.996 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:48.996 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 108 2024-04-22T18:38:48.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.996 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:48.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:48.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:38:48.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:38:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:38:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854064+0000", 2024-04-22T18:38:48.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5820000000000002e-06 2024-04-22T18:38:48.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:48.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854080+0000", 2024-04-22T18:38:48.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5379999999999998e-05 2024-04-22T18:38:48.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:48.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854084+0000", 2024-04-22T18:38:48.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.963e-06 2024-04-22T18:38:48.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:48.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856413+0000", 2024-04-22T18:38:48.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023298070000000001 2024-04-22T18:38:48.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:48.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856494+0000", 2024-04-22T18:38:48.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0863000000000002e-05 2024-04-22T18:38:48.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:48.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:48.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:48.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856624+0000", 2024-04-22T18:38:48.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129799 2024-04-22T18:38:48.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:49.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058947+0000", 2024-04-22T18:38:49.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20232328899999999 2024-04-22T18:38:49.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:49.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159468+0000", 2024-04-22T18:38:49.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100520597 2024-04-22T18:38:49.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:49.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159508+0000", 2024-04-22T18:38:49.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9830000000000003e-05 2024-04-22T18:38:49.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:49.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159544+0000", 2024-04-22T18:38:49.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5944e-05 2024-04-22T18:38:49.001 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.001 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:49.001 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.001 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:49.001 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:38:49.001 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.0895833870000002, 2024-04-22T18:38:49.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30556502499999999, 2024-04-22T18:38:49.001 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:49.002 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.002 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:49.002 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:49.002 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:49.002 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:38:49.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.002 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:49.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:49.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:38:49.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:49.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:38:49.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:49.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854105+0000", 2024-04-22T18:38:49.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:49.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:49.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854175+0000", 2024-04-22T18:38:49.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0178000000000005e-05 2024-04-22T18:38:49.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:49.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854191+0000", 2024-04-22T18:38:49.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5965999999999999e-05 2024-04-22T18:38:49.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854196+0000", 2024-04-22T18:38:49.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.42e-06 2024-04-22T18:38:49.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:49.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956526+0000", 2024-04-22T18:38:49.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10233094299999999 2024-04-22T18:38:49.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:49.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956634+0000", 2024-04-22T18:38:49.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107398 2024-04-22T18:38:49.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:49.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956730+0000", 2024-04-22T18:38:49.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6217000000000004e-05 2024-04-22T18:38:49.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:49.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058955+0000", 2024-04-22T18:38:49.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102224749 2024-04-22T18:38:49.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:49.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159596+0000", 2024-04-22T18:38:49.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100641297 2024-04-22T18:38:49.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:49.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159637+0000", 2024-04-22T18:38:49.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874000000000002e-05 2024-04-22T18:38:49.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:49.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159671+0000", 2024-04-22T18:38:49.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3603999999999999e-05 2024-04-22T18:38:49.007 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.007 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:49.007 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.007 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:49.007 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:38:49.007 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.0894895399999998, 2024-04-22T18:38:49.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30683048800000001, 2024-04-22T18:38:49.007 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:49.007 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.007 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:49.007 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:49.007 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:49.007 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110 2024-04-22T18:38:49.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.008 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:49.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:49.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:38:49.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:49.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:38:49.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:49.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:38:49.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:49.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:49.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854202+0000", 2024-04-22T18:38:49.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:38:49.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:49.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854206+0000", 2024-04-22T18:38:49.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5870000000000001e-06 2024-04-22T18:38:49.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854208+0000", 2024-04-22T18:38:49.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7199999999999998e-06 2024-04-22T18:38:49.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:49.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057285+0000", 2024-04-22T18:38:49.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20307620800000001 2024-04-22T18:38:49.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:49.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057414+0000", 2024-04-22T18:38:49.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012896899999999999 2024-04-22T18:38:49.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:49.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057588+0000", 2024-04-22T18:38:49.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000174802 2024-04-22T18:38:49.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:49.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159429+0000", 2024-04-22T18:38:49.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101840783 2024-04-22T18:38:49.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:49.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160947+0000", 2024-04-22T18:38:49.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001518152 2024-04-22T18:38:49.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:49.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160992+0000", 2024-04-22T18:38:49.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4725000000000003e-05 2024-04-22T18:38:49.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:49.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161030+0000", 2024-04-22T18:38:49.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7789e-05 2024-04-22T18:38:49.012 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.012 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:49.012 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.012 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:49.013 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:38:49.013 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.0894783590000001, 2024-04-22T18:38:49.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:38:49.013 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:49.013 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.013 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:49.013 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:49.013 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:49.013 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:38:49.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.013 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:49.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:49.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:38:49.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:49.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:38:49.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:49.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:38:49.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:49.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:49.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:38:49.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:38:49.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:49.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:38:49.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:38:49.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:38:49.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:38:49.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:49.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:38:49.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:38:49.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:49.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:38:49.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:38:49.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:49.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:38:49.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:38:49.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:49.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:38:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:38:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:38:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:38:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:38:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:38:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:49.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:38:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:38:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:38:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.0894627010000004, 2024-04-22T18:38:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:38:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:38:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:38:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:38:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:49.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:38:49.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:49.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:49.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:38:49.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:38:49.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:49.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:38:49.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:38:49.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:38:49.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:38:49.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:49.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:38:49.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:38:49.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:49.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:38:49.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:38:49.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:49.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:38:49.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:38:49.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:49.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:38:49.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:38:49.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.023 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:49.024 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.024 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:49.024 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:38:49.024 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.0894115390000003, 2024-04-22T18:38:49.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:38:49.024 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:49.024 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.024 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:49.024 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:49.024 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:49.024 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:38:49.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.024 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:49.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:49.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:38:49.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:49.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:38:49.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:49.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:38:49.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:49.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:49.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:38:49.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:38:49.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:49.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:38:49.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:38:49.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:38:49.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:38:49.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:49.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:38:49.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:38:49.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:49.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:38:49.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:38:49.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:49.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:38:49.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:38:49.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:49.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:38:49.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:38:49.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:49.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:38:49.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:38:49.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:49.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:38:49.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:38:49.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:49.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:38:49.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:38:49.029 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.029 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:49.029 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.029 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:49.029 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:38:49.030 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.0893963150000001, 2024-04-22T18:38:49.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:38:49.030 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:49.030 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.030 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:49.030 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:49.030 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:49.030 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:38:49.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.030 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:49.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:49.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:38:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:38:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:38:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:38:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:38:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:38:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:38:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:38:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:38:49.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:49.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:38:49.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:38:49.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:49.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:38:49.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:38:49.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:38:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:38:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:38:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:38:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:38:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:38:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:49.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:38:49.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:38:49.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:49.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:38:49.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:38:49.035 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.035 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:49.035 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.0893367380000001, 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:38:49.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:38:49.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:49.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:38:49.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:38:49.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:38:49.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:38:49.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:49.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:38:49.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:38:49.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:49.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:38:49.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:38:49.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:49.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:38:49.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:38:49.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:49.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:38:49.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:38:49.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:49.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:38:49.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:38:49.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:49.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:38:49.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:38:49.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.0892544219999998, 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:38:49.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:38:49.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:49.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:38:49.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:38:49.046 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.046 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:49.046 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.046 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:49.046 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:38:49.046 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.0892442659999997, 2024-04-22T18:38:49.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:38:49.047 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:49.047 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.047 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:49.047 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:49.047 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:49.047 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:38:49.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.047 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:49.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:49.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:38:49.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:49.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:38:49.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:49.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:38:49.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:49.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:49.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:38:49.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:38:49.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:49.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:38:49.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:38:49.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:38:49.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:38:49.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:49.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:38:49.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:38:49.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:49.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:38:49.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:38:49.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:49.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:38:49.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:38:49.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:49.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:38:49.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:38:49.052 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.052 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:49.052 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.052 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:161 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:38:49.052 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:38:49.052 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.488749975, 2024-04-22T18:38:49.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40239169000000002, 2024-04-22T18:38:49.052 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:49.052 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.052 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:49.052 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:49.052 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:49.053 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 161 2024-04-22T18:38:49.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.053 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:49.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:49.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:38:49.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:49.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:38:49.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:49.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454938+0000", 2024-04-22T18:38:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454982+0000", 2024-04-22T18:38:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3470999999999998e-05 2024-04-22T18:38:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454988+0000", 2024-04-22T18:38:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7640000000000002e-06 2024-04-22T18:38:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555054+0000", 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006607300000001 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555607+0000", 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055319500000000003 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555689+0000", 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1817999999999995e-05 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555804+0000", 2024-04-22T18:38:49.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115346 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556554+0000", 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000750102 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857265+0000", 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30071122700000003 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857302+0000", 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7069e-05 2024-04-22T18:38:49.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:49.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:49.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:49.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857331+0000", 2024-04-22T18:38:49.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8291999999999999e-05 2024-04-22T18:38:49.057 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.057 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:49.057 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.057 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:49.057 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:49.057 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:49.058 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:49.060+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:49.059 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:38:49.080 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:49.082+0000 7f3e7be88640 -1 osd.0 21 handle_osd_map injecting map crc failure 2024-04-22T18:38:49.084 DEBUG:teuthology.orchestra.run.smithi089:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0 2024-04-22T18:38:49.087 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:38:49.111 INFO:tasks.rados.rados.0.smithi089.stdout:adding op weight read -> 100 2024-04-22T18:38:49.111 INFO:tasks.rados.rados.0.smithi089.stdout:adding op weight write -> 25 2024-04-22T18:38:49.112 INFO:tasks.rados.rados.0.smithi089.stdout:adding op weight delete -> 10 2024-04-22T18:38:49.112 INFO:tasks.rados.rados.0.smithi089.stdout:adding op weight set_chunk -> 30 2024-04-22T18:38:49.112 INFO:tasks.rados.rados.0.smithi089.stdout:adding op weight tier_promote -> 10 2024-04-22T18:38:49.112 INFO:tasks.rados.rados.0.smithi089.stdout:adding op weight tier_evict -> 10 2024-04-22T18:38:49.112 INFO:tasks.rados.rados.0.smithi089.stdout:adding op weight tier_flush -> 5 2024-04-22T18:38:49.112 INFO:tasks.rados.rados.0.smithi089.stdout:adding op weight write_excl -> 25 2024-04-22T18:38:49.112 INFO:tasks.rados.rados.0.smithi089.stdout:ceph version 18.2.2-1240-g6a7528e4 (6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0) reef (stable) 2024-04-22T18:38:49.112 INFO:tasks.rados.rados.0.smithi089.stdout:Configuration: 2024-04-22T18:38:49.112 INFO:tasks.rados.rados.0.smithi089.stdout: Number of operations: 1500 2024-04-22T18:38:49.112 INFO:tasks.rados.rados.0.smithi089.stdout: Number of objects: 50 2024-04-22T18:38:49.112 INFO:tasks.rados.rados.0.smithi089.stdout: Max in flight operations: 16 2024-04-22T18:38:49.112 INFO:tasks.rados.rados.0.smithi089.stdout: Object size (in bytes): 4000000 2024-04-22T18:38:49.112 INFO:tasks.rados.rados.0.smithi089.stdout: Write stride min: 400000 2024-04-22T18:38:49.112 INFO:tasks.rados.rados.0.smithi089.stdout: Write stride max: 800000 2024-04-22T18:38:49.159 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:49.160+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:38:49.172 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:49.172 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:38:49.173 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:38:49.173 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:49.182 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:38:49.265 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:49.267+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:49.323 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:49.324 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:38:49.324 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:38:49.324 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:38:49.324 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:49.332 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:38:49.360 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:49.361+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:49.461 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:49.462+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:38:49.475 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:38:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:38:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:38:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:34 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'20, mlcod=16'14)", 2024-04-22T18:38:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:39.743670+0000", 2024-04-22T18:38:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.7321641000000003, 2024-04-22T18:38:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20229561200000001, 2024-04-22T18:38:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.743670+0000", 2024-04-22T18:38:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.743670+0000", 2024-04-22T18:38:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.743670+0000", 2024-04-22T18:38:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.743675+0000", 2024-04-22T18:38:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0819999999999998e-06 2024-04-22T18:38:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.743681+0000", 2024-04-22T18:38:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9309999999999996e-06 2024-04-22T18:38:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.843780+0000", 2024-04-22T18:38:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100098778 2024-04-22T18:38:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.844770+0000", 2024-04-22T18:38:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00099004900000000001 2024-04-22T18:38:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.844789+0000", 2024-04-22T18:38:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9143e-05 2024-04-22T18:38:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.945937+0000", 2024-04-22T18:38:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101148259 2024-04-22T18:38:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.945966+0000", 2024-04-22T18:38:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8929999999999999e-05 2024-04-22T18:38:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:35 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'21, mlcod=16'16)", 2024-04-22T18:38:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:39.844672+0000", 2024-04-22T18:38:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.6311626960000005, 2024-04-22T18:38:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20185124900000001, 2024-04-22T18:38:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.844672+0000", 2024-04-22T18:38:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.844672+0000", 2024-04-22T18:38:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.844670+0000", 2024-04-22T18:38:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.844713+0000", 2024-04-22T18:38:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3049000000000001e-05 2024-04-22T18:38:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.844724+0000", 2024-04-22T18:38:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1218e-05 2024-04-22T18:38:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.844732+0000", 2024-04-22T18:38:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8609999999999998e-06 2024-04-22T18:38:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.945458+0000", 2024-04-22T18:38:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100725341 2024-04-22T18:38:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.945479+0000", 2024-04-22T18:38:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1070999999999999e-05 2024-04-22T18:38:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.046432+0000", 2024-04-22T18:38:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100953652 2024-04-22T18:38:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.046523+0000", 2024-04-22T18:38:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0696e-05 2024-04-22T18:38:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:36 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'22, mlcod=16'16)", 2024-04-22T18:38:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:39.845303+0000", 2024-04-22T18:38:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.6305313790000007, 2024-04-22T18:38:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.201257186, 2024-04-22T18:38:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.845303+0000", 2024-04-22T18:38:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.845303+0000", 2024-04-22T18:38:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.845302+0000", 2024-04-22T18:38:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.845312+0000", 2024-04-22T18:38:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1519999999999998e-06 2024-04-22T18:38:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.845318+0000", 2024-04-22T18:38:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.161e-06 2024-04-22T18:38:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.845323+0000", 2024-04-22T18:38:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1150000000000002e-06 2024-04-22T18:38:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.945775+0000", 2024-04-22T18:38:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100451924 2024-04-22T18:38:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.945788+0000", 2024-04-22T18:38:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3074999999999999e-05 2024-04-22T18:38:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.046528+0000", 2024-04-22T18:38:49.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100739705 2024-04-22T18:38:49.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.046560+0000", 2024-04-22T18:38:49.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2704999999999999e-05 2024-04-22T18:38:49.489 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.489 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.489 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.490 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:37 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'23, mlcod=16'16)", 2024-04-22T18:38:49.490 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:39.944608+0000", 2024-04-22T18:38:49.490 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.5312262830000005, 2024-04-22T18:38:49.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10220341300000001, 2024-04-22T18:38:49.490 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.490 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.490 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.944608+0000", 2024-04-22T18:38:49.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.944608+0000", 2024-04-22T18:38:49.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.944606+0000", 2024-04-22T18:38:49.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:38:49.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.944620+0000", 2024-04-22T18:38:49.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3594000000000001e-05 2024-04-22T18:38:49.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.944632+0000", 2024-04-22T18:38:49.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2824e-05 2024-04-22T18:38:49.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:39.944648+0000", 2024-04-22T18:38:49.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5438e-05 2024-04-22T18:38:49.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.046080+0000", 2024-04-22T18:38:49.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10143237500000001 2024-04-22T18:38:49.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.046111+0000", 2024-04-22T18:38:49.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1219e-05 2024-04-22T18:38:49.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.046770+0000", 2024-04-22T18:38:49.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065852599999999999 2024-04-22T18:38:49.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.046812+0000", 2024-04-22T18:38:49.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1732e-05 2024-04-22T18:38:49.494 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.494 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.494 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.494 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:38 1.0 e17/14 1:30306672:devicehealth::main.db.0000000000000000:head v 17'24, mlcod=16'22)", 2024-04-22T18:38:49.494 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:40.248519+0000", 2024-04-22T18:38:49.494 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.2273156020000009, 2024-04-22T18:38:49.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103094375, 2024-04-22T18:38:49.494 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.495 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.495 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.248519+0000", 2024-04-22T18:38:49.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.248519+0000", 2024-04-22T18:38:49.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.248515+0000", 2024-04-22T18:38:49.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999962 2024-04-22T18:38:49.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.248591+0000", 2024-04-22T18:38:49.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5471999999999998e-05 2024-04-22T18:38:49.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.248617+0000", 2024-04-22T18:38:49.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6350999999999999e-05 2024-04-22T18:38:49.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.248632+0000", 2024-04-22T18:38:49.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4731e-05 2024-04-22T18:38:49.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.248730+0000", 2024-04-22T18:38:49.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7856999999999995e-05 2024-04-22T18:38:49.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.248759+0000", 2024-04-22T18:38:49.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9825999999999999e-05 2024-04-22T18:38:49.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.351569+0000", 2024-04-22T18:38:49.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102809442 2024-04-22T18:38:49.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.351613+0000", 2024-04-22T18:38:49.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4416000000000003e-05 2024-04-22T18:38:49.498 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.498 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.498 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.499 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:39 1.0 e17/14 1:30306672:devicehealth::main.db.0000000000000000:head v 17'25, mlcod=16'22)", 2024-04-22T18:38:49.499 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:40.249251+0000", 2024-04-22T18:38:49.499 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.2265831400000007, 2024-04-22T18:38:49.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102930717, 2024-04-22T18:38:49.499 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.499 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.499 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.249251+0000", 2024-04-22T18:38:49.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.249251+0000", 2024-04-22T18:38:49.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.249250+0000", 2024-04-22T18:38:49.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:49.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.249306+0000", 2024-04-22T18:38:49.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5503e-05 2024-04-22T18:38:49.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.249318+0000", 2024-04-22T18:38:49.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2517000000000001e-05 2024-04-22T18:38:49.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.249325+0000", 2024-04-22T18:38:49.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8179999999999998e-06 2024-04-22T18:38:49.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.351131+0000", 2024-04-22T18:38:49.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10180637200000001 2024-04-22T18:38:49.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.351170+0000", 2024-04-22T18:38:49.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8645e-05 2024-04-22T18:38:49.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.352145+0000", 2024-04-22T18:38:49.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000974679 2024-04-22T18:38:49.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:40.352182+0000", 2024-04-22T18:38:49.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7497999999999999e-05 2024-04-22T18:38:49.503 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.503 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.503 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.504 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:62 1.0 e17/14 1:30306672:devicehealth::main.db.0000000000000000:head v 17'40, mlcod=17'38)", 2024-04-22T18:38:49.504 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:41.780972+0000", 2024-04-22T18:38:49.504 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.694862562, 2024-04-22T18:38:49.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101546389, 2024-04-22T18:38:49.504 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.504 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.504 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.780972+0000", 2024-04-22T18:38:49.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.780972+0000", 2024-04-22T18:38:49.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.780969+0000", 2024-04-22T18:38:49.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:38:49.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.780987+0000", 2024-04-22T18:38:49.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7643e-05 2024-04-22T18:38:49.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.781012+0000", 2024-04-22T18:38:49.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5775999999999999e-05 2024-04-22T18:38:49.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.881110+0000", 2024-04-22T18:38:49.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100097638 2024-04-22T18:38:49.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.881215+0000", 2024-04-22T18:38:49.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104788 2024-04-22T18:38:49.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.881244+0000", 2024-04-22T18:38:49.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8670000000000002e-05 2024-04-22T18:38:49.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.882472+0000", 2024-04-22T18:38:49.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012286350000000001 2024-04-22T18:38:49.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.882518+0000", 2024-04-22T18:38:49.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6174999999999998e-05 2024-04-22T18:38:49.508 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.508 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.508 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.508 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:68 1.0 e17/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'42, mlcod=17'39)", 2024-04-22T18:38:49.508 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:41.987068+0000", 2024-04-22T18:38:49.509 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.4887668390000002, 2024-04-22T18:38:49.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30106360999999998, 2024-04-22T18:38:49.509 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.509 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.509 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987068+0000", 2024-04-22T18:38:49.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987068+0000", 2024-04-22T18:38:49.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987067+0000", 2024-04-22T18:38:49.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:38:49.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987077+0000", 2024-04-22T18:38:49.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8279999999999998e-06 2024-04-22T18:38:49.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987084+0000", 2024-04-22T18:38:49.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8099999999999998e-06 2024-04-22T18:38:49.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987091+0000", 2024-04-22T18:38:49.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6189999999999996e-06 2024-04-22T18:38:49.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087218+0000", 2024-04-22T18:38:49.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10012695100000001 2024-04-22T18:38:49.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087246+0000", 2024-04-22T18:38:49.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8255e-05 2024-04-22T18:38:49.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.288092+0000", 2024-04-22T18:38:49.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20084555000000001 2024-04-22T18:38:49.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.288131+0000", 2024-04-22T18:38:49.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9455000000000001e-05 2024-04-22T18:38:49.513 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.513 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.513 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.513 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:69 1.0 e17/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'43, mlcod=17'39)", 2024-04-22T18:38:49.513 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:42.087047+0000", 2024-04-22T18:38:49.513 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.3887872459999997, 2024-04-22T18:38:49.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.201104544, 2024-04-22T18:38:49.514 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.514 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.514 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087047+0000", 2024-04-22T18:38:49.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087047+0000", 2024-04-22T18:38:49.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087044+0000", 2024-04-22T18:38:49.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:38:49.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087113+0000", 2024-04-22T18:38:49.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9497000000000002e-05 2024-04-22T18:38:49.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087137+0000", 2024-04-22T18:38:49.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.393e-05 2024-04-22T18:38:49.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087150+0000", 2024-04-22T18:38:49.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2869e-05 2024-04-22T18:38:49.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087659+0000", 2024-04-22T18:38:49.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050907800000000005 2024-04-22T18:38:49.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087680+0000", 2024-04-22T18:38:49.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0724e-05 2024-04-22T18:38:49.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.288134+0000", 2024-04-22T18:38:49.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20045370000000001 2024-04-22T18:38:49.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.288152+0000", 2024-04-22T18:38:49.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8233e-05 2024-04-22T18:38:49.517 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.517 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.518 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.518 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:70 1.0 e17/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'44, mlcod=17'39)", 2024-04-22T18:38:49.518 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:42.087569+0000", 2024-04-22T18:38:49.518 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.3882652269999998, 2024-04-22T18:38:49.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20137501599999999, 2024-04-22T18:38:49.518 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.518 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.518 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087569+0000", 2024-04-22T18:38:49.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087569+0000", 2024-04-22T18:38:49.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087568+0000", 2024-04-22T18:38:49.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:38:49.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087576+0000", 2024-04-22T18:38:49.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.481e-06 2024-04-22T18:38:49.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087585+0000", 2024-04-22T18:38:49.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9830000000000002e-06 2024-04-22T18:38:49.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087593+0000", 2024-04-22T18:38:49.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5490000000000006e-06 2024-04-22T18:38:49.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.287757+0000", 2024-04-22T18:38:49.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20016344699999999 2024-04-22T18:38:49.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.287787+0000", 2024-04-22T18:38:49.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0091999999999999e-05 2024-04-22T18:38:49.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.288915+0000", 2024-04-22T18:38:49.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011286130000000001 2024-04-22T18:38:49.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.288944+0000", 2024-04-22T18:38:49.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8991000000000001e-05 2024-04-22T18:38:49.522 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.522 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.522 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.522 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:72 1.0 e17/14 1:30306672:devicehealth::main.db.0000000000000000:head v 17'46, mlcod=17'43)", 2024-04-22T18:38:49.522 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:42.490715+0000", 2024-04-22T18:38:49.522 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.9851196230000001, 2024-04-22T18:38:49.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10128431, 2024-04-22T18:38:49.523 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.523 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.523 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.490715+0000", 2024-04-22T18:38:49.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.490715+0000", 2024-04-22T18:38:49.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.490714+0000", 2024-04-22T18:38:49.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:38:49.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.490783+0000", 2024-04-22T18:38:49.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9783000000000006e-05 2024-04-22T18:38:49.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.490798+0000", 2024-04-22T18:38:49.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4990999999999999e-05 2024-04-22T18:38:49.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.490804+0000", 2024-04-22T18:38:49.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4979999999999997e-06 2024-04-22T18:38:49.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.490834+0000", 2024-04-22T18:38:49.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0457e-05 2024-04-22T18:38:49.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.490848+0000", 2024-04-22T18:38:49.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3577999999999999e-05 2024-04-22T18:38:49.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.591972+0000", 2024-04-22T18:38:49.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101124064 2024-04-22T18:38:49.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.591999+0000", 2024-04-22T18:38:49.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7121999999999999e-05 2024-04-22T18:38:49.527 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.527 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.527 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.527 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:104 1.0 e18/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'66, mlcod=17'64)", 2024-04-22T18:38:49.527 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:44.854278+0000", 2024-04-22T18:38:49.527 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.6215562940000003, 2024-04-22T18:38:49.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102213205, 2024-04-22T18:38:49.527 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.527 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.527 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.854278+0000", 2024-04-22T18:38:49.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.854278+0000", 2024-04-22T18:38:49.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.854275+0000", 2024-04-22T18:38:49.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:38:49.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.854300+0000", 2024-04-22T18:38:49.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5057000000000001e-05 2024-04-22T18:38:49.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.854321+0000", 2024-04-22T18:38:49.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1129999999999999e-05 2024-04-22T18:38:49.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.854333+0000", 2024-04-22T18:38:49.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1956e-05 2024-04-22T18:38:49.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.854919+0000", 2024-04-22T18:38:49.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058543799999999997 2024-04-22T18:38:49.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.854948+0000", 2024-04-22T18:38:49.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9204999999999999e-05 2024-04-22T18:38:49.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.956462+0000", 2024-04-22T18:38:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101514008 2024-04-22T18:38:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.956491+0000", 2024-04-22T18:38:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9679000000000002e-05 2024-04-22T18:38:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:105 1.0 e18/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'67, mlcod=17'64)", 2024-04-22T18:38:49.531 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:44.854852+0000", 2024-04-22T18:38:49.532 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.6209822169999999, 2024-04-22T18:38:49.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101905653, 2024-04-22T18:38:49.532 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.532 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.532 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.854852+0000", 2024-04-22T18:38:49.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.854852+0000", 2024-04-22T18:38:49.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.854851+0000", 2024-04-22T18:38:49.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:38:49.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.854863+0000", 2024-04-22T18:38:49.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1369000000000001e-05 2024-04-22T18:38:49.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.854870+0000", 2024-04-22T18:38:49.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6460000000000004e-06 2024-04-22T18:38:49.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.854873+0000", 2024-04-22T18:38:49.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.704e-06 2024-04-22T18:38:49.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.955787+0000", 2024-04-22T18:38:49.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100913895 2024-04-22T18:38:49.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.955808+0000", 2024-04-22T18:38:49.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1251e-05 2024-04-22T18:38:49.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.956734+0000", 2024-04-22T18:38:49.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00092587799999999999 2024-04-22T18:38:49.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.956758+0000", 2024-04-22T18:38:49.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3697999999999999e-05 2024-04-22T18:38:49.536 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.536 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.536 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.536 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:106 1.0 e18/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'68, mlcod=17'64)", 2024-04-22T18:38:49.536 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:44.855453+0000", 2024-04-22T18:38:49.536 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.6203812470000001, 2024-04-22T18:38:49.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103718301, 2024-04-22T18:38:49.536 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.536 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.536 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.855453+0000", 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.855453+0000", 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.855453+0000", 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.855666+0000", 2024-04-22T18:38:49.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021382000000000001 2024-04-22T18:38:49.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.855675+0000", 2024-04-22T18:38:49.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9190000000000005e-06 2024-04-22T18:38:49.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.855680+0000", 2024-04-22T18:38:49.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:38:49.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.956057+0000", 2024-04-22T18:38:49.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10037662 2024-04-22T18:38:49.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.956067+0000", 2024-04-22T18:38:49.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0291e-05 2024-04-22T18:38:49.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.959137+0000", 2024-04-22T18:38:49.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030697390000000001 2024-04-22T18:38:49.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.959172+0000", 2024-04-22T18:38:49.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4433e-05 2024-04-22T18:38:49.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.540 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.540 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:107 1.0 e18/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'69, mlcod=17'64)", 2024-04-22T18:38:49.540 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:44.856121+0000", 2024-04-22T18:38:49.540 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.6197130240000002, 2024-04-22T18:38:49.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103071581, 2024-04-22T18:38:49.540 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.541 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.541 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.856121+0000", 2024-04-22T18:38:49.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.856121+0000", 2024-04-22T18:38:49.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.856120+0000", 2024-04-22T18:38:49.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:38:49.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.856131+0000", 2024-04-22T18:38:49.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0209e-05 2024-04-22T18:38:49.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.856140+0000", 2024-04-22T18:38:49.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3300000000000005e-06 2024-04-22T18:38:49.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.856146+0000", 2024-04-22T18:38:49.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8170000000000004e-06 2024-04-22T18:38:49.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.956265+0000", 2024-04-22T18:38:49.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100119319 2024-04-22T18:38:49.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.956279+0000", 2024-04-22T18:38:49.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3602e-05 2024-04-22T18:38:49.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.959174+0000", 2024-04-22T18:38:49.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002895351 2024-04-22T18:38:49.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.959193+0000", 2024-04-22T18:38:49.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9048000000000001e-05 2024-04-22T18:38:49.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.544 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.545 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:108 1.0 e18/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'70, mlcod=17'64)", 2024-04-22T18:38:49.545 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:44.856734+0000", 2024-04-22T18:38:49.545 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.6191009659999995, 2024-04-22T18:38:49.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10731078099999999, 2024-04-22T18:38:49.545 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.545 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.545 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.856734+0000", 2024-04-22T18:38:49.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.856734+0000", 2024-04-22T18:38:49.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.856732+0000", 2024-04-22T18:38:49.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:38:49.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.856742+0000", 2024-04-22T18:38:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3330000000000003e-06 2024-04-22T18:38:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.856751+0000", 2024-04-22T18:38:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2150000000000007e-06 2024-04-22T18:38:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.856757+0000", 2024-04-22T18:38:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6550000000000001e-06 2024-04-22T18:38:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.956518+0000", 2024-04-22T18:38:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099761483999999997 2024-04-22T18:38:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.956538+0000", 2024-04-22T18:38:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9993e-05 2024-04-22T18:38:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.964011+0000", 2024-04-22T18:38:49.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.007473375 2024-04-22T18:38:49.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:44.964044+0000", 2024-04-22T18:38:49.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2814000000000001e-05 2024-04-22T18:38:49.549 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.549 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.549 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.549 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:111 1.0 e18/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 18'73, mlcod=17'64)", 2024-04-22T18:38:49.549 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:45.058736+0000", 2024-04-22T18:38:49.549 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.4170982839999997, 2024-04-22T18:38:49.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10166581, 2024-04-22T18:38:49.550 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.550 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.550 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.058736+0000", 2024-04-22T18:38:49.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.058736+0000", 2024-04-22T18:38:49.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.058736+0000", 2024-04-22T18:38:49.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:49.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.058747+0000", 2024-04-22T18:38:49.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0798000000000001e-05 2024-04-22T18:38:49.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.058752+0000", 2024-04-22T18:38:49.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.558e-06 2024-04-22T18:38:49.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.058756+0000", 2024-04-22T18:38:49.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7869999999999999e-06 2024-04-22T18:38:49.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.159275+0000", 2024-04-22T18:38:49.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10051899 2024-04-22T18:38:49.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.159287+0000", 2024-04-22T18:38:49.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2458e-05 2024-04-22T18:38:49.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.160372+0000", 2024-04-22T18:38:49.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010848679999999999 2024-04-22T18:38:49.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.160402+0000", 2024-04-22T18:38:49.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9720999999999999e-05 2024-04-22T18:38:49.553 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.554 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.554 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.554 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:119 1.0 e18/14)", 2024-04-22T18:38:49.554 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:45.664792+0000", 2024-04-22T18:38:49.554 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.811042058, 2024-04-22T18:38:49.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20257809199999999, 2024-04-22T18:38:49.554 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.554 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.554 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.664792+0000", 2024-04-22T18:38:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.664792+0000", 2024-04-22T18:38:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.664792+0000", 2024-04-22T18:38:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.664861+0000", 2024-04-22T18:38:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8948999999999993e-05 2024-04-22T18:38:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.664867+0000", 2024-04-22T18:38:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8150000000000002e-06 2024-04-22T18:38:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.664872+0000", 2024-04-22T18:38:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6250000000000004e-06 2024-04-22T18:38:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.864852+0000", 2024-04-22T18:38:49.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.199980146 2024-04-22T18:38:49.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.864907+0000", 2024-04-22T18:38:49.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4775999999999999e-05 2024-04-22T18:38:49.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.867332+0000", 2024-04-22T18:38:49.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0024247660000000001 2024-04-22T18:38:49.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:45.867371+0000", 2024-04-22T18:38:49.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8627999999999997e-05 2024-04-22T18:38:49.558 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.558 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.558 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.558 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:160 1.0 e21/14 1:30306672:devicehealth::main.db.0000000000000000:head v 21'114, mlcod=21'112)", 2024-04-22T18:38:49.558 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:48.555364+0000", 2024-04-22T18:38:49.558 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.92047076299999997, 2024-04-22T18:38:49.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20131558799999999, 2024-04-22T18:38:49.558 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.558 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.558 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.555364+0000", 2024-04-22T18:38:49.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.555364+0000", 2024-04-22T18:38:49.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.555363+0000", 2024-04-22T18:38:49.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:38:49.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.555397+0000", 2024-04-22T18:38:49.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4159999999999998e-05 2024-04-22T18:38:49.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.555407+0000", 2024-04-22T18:38:49.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.023e-05 2024-04-22T18:38:49.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.555413+0000", 2024-04-22T18:38:49.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1709999999999999e-06 2024-04-22T18:38:49.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.655582+0000", 2024-04-22T18:38:49.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100169307 2024-04-22T18:38:49.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.655596+0000", 2024-04-22T18:38:49.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3108999999999999e-05 2024-04-22T18:38:49.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.756658+0000", 2024-04-22T18:38:49.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101062391 2024-04-22T18:38:49.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.756679+0000", 2024-04-22T18:38:49.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1331999999999998e-05 2024-04-22T18:38:49.562 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.562 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.562 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.563 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:161 1.0 e21/14 1:30306672:devicehealth::main.db.0000000000000000:head v 21'115, mlcod=21'112)", 2024-04-22T18:38:49.563 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:48.555982+0000", 2024-04-22T18:38:49.563 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.91985289800000003, 2024-04-22T18:38:49.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20110583400000001, 2024-04-22T18:38:49.563 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:49.563 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:49.563 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:49.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:49.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.555982+0000", 2024-04-22T18:38:49.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:49.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.555982+0000", 2024-04-22T18:38:49.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:49.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:49.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.555981+0000", 2024-04-22T18:38:49.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:38:49.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:49.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.556024+0000", 2024-04-22T18:38:49.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3918000000000002e-05 2024-04-22T18:38:49.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:49.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.556034+0000", 2024-04-22T18:38:49.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7839999999999998e-06 2024-04-22T18:38:49.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:49.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.556039+0000", 2024-04-22T18:38:49.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9389999999999996e-06 2024-04-22T18:38:49.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:49.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.756215+0000", 2024-04-22T18:38:49.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20017604899999999 2024-04-22T18:38:49.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:49.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.756246+0000", 2024-04-22T18:38:49.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1183000000000003e-05 2024-04-22T18:38:49.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:49.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.757061+0000", 2024-04-22T18:38:49.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00081483599999999999 2024-04-22T18:38:49.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:49.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:49.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:49.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:48.757087+0000", 2024-04-22T18:38:49.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6177e-05 2024-04-22T18:38:49.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.567 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:49.567 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:49.567 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:49.568 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:38:49.569 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:49.570+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:49.669 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:49.671+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:49.710 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:49.711 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:38:49.711 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:38:49.711 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:49.719 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:38:49.769 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:49.771+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:38:49.820 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:49.820 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:38:49.820 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:38:49.820 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:38:49.820 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:49.828 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:38:49.870 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:49.872+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:49.970 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:38:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:38:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [] 2024-04-22T18:38:49.971 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:49.971 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:49.972+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:49.979 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:38:50.072 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:50.073+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:50.081 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:38:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:38:50.081 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:50.084 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:38:50.093 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:38:50.172 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:50.174+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:50.194 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:38:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:38:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:38:50.195 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:50.203 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:38:50.288 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:50.289+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:50.345 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:38:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:38:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [] 2024-04-22T18:38:50.346 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:50.354 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:38:50.396 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:50.397+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:50.440 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:38:50.484 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:50.485+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:50.498 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:50.498 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:38:50.499 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:38:50.499 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:50.508 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:38:50.559 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:38:50.559 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:38:50.576 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:38:50.576 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:38:50.576 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:50.578+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:50.591 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:38:50.592 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:38:50.608 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:38:50.608 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:38:50.625 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:38:50.625 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:38:50.641 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:38:50.642 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:38:50.651 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:50.651 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:50.658 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:38:50.658 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:38:50.661 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:38:50.675 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:38:50.676 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:38:50.677 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:50.678+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:50.776 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:50.776 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:50.777 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:50.779+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:50.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:38:50.878 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:50.879+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:50.898 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:50.898 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:50.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:38:50.908 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:50.908+0000 7f309102d640 -1 osd.1 22 handle_osd_map injecting map crc failure 2024-04-22T18:38:50.978 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:50.980+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:51.049 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:51.049 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:51.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:38:51.102 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:51.104+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:51.161 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:51.161 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:51.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:38:51.180 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:51.182+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:51.281 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:51.283+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:38:51.313 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:51.313 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:51.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:38:51.382 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:51.384+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:51.422 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:51.422 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:51.431 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:38:51.482 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:51.483+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:51.484 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:38:51.484 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:38:51.484 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-04-22T18:38:51.484 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-22T18:38:51.484 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-22T18:38:51.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2024-04-22T18:38:51.536 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:51.536 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:51.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:38:51.588 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:51.589+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:51.588 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-22T18:38:51.589 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:51.648 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:51.648 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:51.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:38:51.684 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:51.686+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:51.714 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:51.712+0000 7f7f42baf640 -1 osd.4 23 handle_osd_map injecting map crc failure 2024-04-22T18:38:51.714 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:51.712+0000 7f309102d640 -1 osd.1 23 handle_osd_map injecting map crc failure 2024-04-22T18:38:51.757 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:51.757 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:51.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:38:51.785 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:51.786+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:38:51.868 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:51.869 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:51.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:38:51.922 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:51.924+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:51.980 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:51.980 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:51.986 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:51.987+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:38:51.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:38:52.087 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:52.088+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:52.099 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:38:52.130 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:52.130 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:52.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:38:52.223 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:52.224+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:52.280 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:52.280 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:52.289 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:52.289+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:52.289 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:38:52.390 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:52.391+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:52.392 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:52.392 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:52.401 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:38:52.491 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:52.493+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:52.542 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:52.542 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:52.551 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:38:52.595 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:52.596+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:52.652 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:52.652 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:52.661 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:38:52.687 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:38:52.693 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:52.694+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:52.794 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:52.795+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:38:52.806 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:52.806 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:52.816 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:38:52.821 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:38:52.821 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:38:52.840 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:38:52.840 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:38:52.856 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:38:52.856 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:38:52.872 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:38:52.872 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:38:52.889 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:38:52.889 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:38:52.894 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:52.896+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:52.906 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:38:52.906 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:38:52.923 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:38:52.923 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:38:52.925 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:52.925 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:38:52.925 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:38:52.925 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:38:52.925 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:52.934 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:38:52.940 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:38:52.940 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:38:52.996 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:52.997+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:38:53.057 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:53.057 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:38:53.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:38:53.057 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [] 2024-04-22T18:38:53.057 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:53.058 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:38:53.083 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:53.084+0000 7f8607f36640 -1 osd.3 25 handle_osd_map injecting map crc failure 2024-04-22T18:38:53.085 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:38:53.097 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:53.099+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:53.185 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:38:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:38:53.185 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:53.194 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:38:53.238 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:53.239+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:53.295 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:53.295 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:38:53.295 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:38:53.295 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:38:53.295 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:53.299 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:53.300+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:53.304 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:38:53.400 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:53.401+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:53.445 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:53.445 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:38:53.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:38:53.445 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [] 2024-04-22T18:38:53.445 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:53.454 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:38:53.500 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:53.502+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:53.555 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:53.555 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:38:53.555 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:38:53.556 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:53.564 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:38:53.601 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:53.603+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:53.666 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:38:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:38:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:38:53.667 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:53.675 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:38:53.701 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:53.703+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:53.778 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:53.778 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:38:53.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:38:53.778 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [] 2024-04-22T18:38:53.778 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:53.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:38:53.831 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:53.833+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:53.899 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:53.899 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:38:53.899 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:38:53.899 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:53.903 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:53.905+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:38:53.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:38:54.005 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:54.006+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:54.053 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:54.053 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:38:54.053 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:38:54.053 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:38:54.053 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:54.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:38:54.086 INFO:tasks.rados.rados.0.smithi089.stdout:1: write initial oid 1 2024-04-22T18:38:54.086 INFO:tasks.rados.rados.0.smithi089.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-04-22T18:38:54.091 INFO:tasks.rados.rados.0.smithi089.stdout:1: writing smithi08937929-1 from 416807 to 892056 tid 1 2024-04-22T18:38:54.097 INFO:tasks.rados.rados.0.smithi089.stdout:1: writing smithi08937929-1 from 1542129 to 2016807 tid 2 2024-04-22T18:38:54.098 INFO:tasks.rados.rados.0.smithi089.stdout:2: write initial oid 2 2024-04-22T18:38:54.098 INFO:tasks.rados.rados.0.smithi089.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-04-22T18:38:54.102 INFO:tasks.rados.rados.0.smithi089.stdout:2: writing smithi08937929-2 from 416807 to 892056 tid 1 2024-04-22T18:38:54.109 INFO:tasks.rados.rados.0.smithi089.stdout:2: writing smithi08937929-2 from 1542129 to 2016807 tid 2 2024-04-22T18:38:54.109 INFO:tasks.rados.rados.0.smithi089.stdout:3: write initial oid 3 2024-04-22T18:38:54.109 INFO:tasks.rados.rados.0.smithi089.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-04-22T18:38:54.114 INFO:tasks.rados.rados.0.smithi089.stdout:3: writing smithi08937929-3 from 433614 to 984112 tid 1 2024-04-22T18:38:54.120 INFO:tasks.rados.rados.0.smithi089.stdout:3: writing smithi08937929-3 from 1600611 to 2033614 tid 2 2024-04-22T18:38:54.120 INFO:tasks.rados.rados.0.smithi089.stdout:4: write initial oid 4 2024-04-22T18:38:54.121 INFO:tasks.rados.rados.0.smithi089.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-04-22T18:38:54.126 INFO:tasks.rados.rados.0.smithi089.stdout:4: writing smithi08937929-4 from 450421 to 1076168 tid 1 2024-04-22T18:38:54.132 INFO:tasks.rados.rados.0.smithi089.stdout:4: writing smithi08937929-4 from 1659093 to 2050421 tid 2 2024-04-22T18:38:54.132 INFO:tasks.rados.rados.0.smithi089.stdout:5: write initial oid 5 2024-04-22T18:38:54.132 INFO:tasks.rados.rados.0.smithi089.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-04-22T18:38:54.138 INFO:tasks.rados.rados.0.smithi089.stdout:5: writing smithi08937929-5 from 467228 to 1168224 tid 1 2024-04-22T18:38:54.143 INFO:tasks.rados.rados.0.smithi089.stdout:5: writing smithi08937929-5 from 1717575 to 2067228 tid 2 2024-04-22T18:38:54.143 INFO:tasks.rados.rados.0.smithi089.stdout:6: write initial oid 6 2024-04-22T18:38:54.143 INFO:tasks.rados.rados.0.smithi089.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-04-22T18:38:54.146 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:54.148+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:54.150 INFO:tasks.rados.rados.0.smithi089.stdout:6: writing smithi08937929-6 from 484035 to 1260280 tid 1 2024-04-22T18:38:54.155 INFO:tasks.rados.rados.0.smithi089.stdout:6: writing smithi08937929-6 from 2059704 to 2084035 tid 2 2024-04-22T18:38:54.155 INFO:tasks.rados.rados.0.smithi089.stdout:7: write initial oid 7 2024-04-22T18:38:54.155 INFO:tasks.rados.rados.0.smithi089.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-04-22T18:38:54.160 INFO:tasks.rados.rados.0.smithi089.stdout:7: writing smithi08937929-7 from 500842 to 952336 tid 1 2024-04-22T18:38:54.166 INFO:tasks.rados.rados.0.smithi089.stdout:7: writing smithi08937929-7 from 1718186 to 2100842 tid 2 2024-04-22T18:38:54.166 INFO:tasks.rados.rados.0.smithi089.stdout:8: write initial oid 8 2024-04-22T18:38:54.166 INFO:tasks.rados.rados.0.smithi089.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-04-22T18:38:54.172 INFO:tasks.rados.rados.0.smithi089.stdout:8: writing smithi08937929-8 from 517649 to 1044392 tid 1 2024-04-22T18:38:54.177 INFO:tasks.rados.rados.0.smithi089.stdout:8: writing smithi08937929-8 from 1776668 to 2117649 tid 2 2024-04-22T18:38:54.178 INFO:tasks.rados.rados.0.smithi089.stdout:9: write initial oid 9 2024-04-22T18:38:54.178 INFO:tasks.rados.rados.0.smithi089.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-04-22T18:38:54.184 INFO:tasks.rados.rados.0.smithi089.stdout:9: writing smithi08937929-9 from 534456 to 1252801 tid 1 2024-04-22T18:38:54.189 INFO:tasks.rados.rados.0.smithi089.stdout:9: writing smithi08937929-9 from 1951503 to 2134456 tid 2 2024-04-22T18:38:54.189 INFO:tasks.rados.rados.0.smithi089.stdout:10: write initial oid 10 2024-04-22T18:38:54.189 INFO:tasks.rados.rados.0.smithi089.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-04-22T18:38:54.196 INFO:tasks.rados.rados.0.smithi089.stdout:10: writing smithi08937929-10 from 551263 to 1344857 tid 1 2024-04-22T18:38:54.201 INFO:tasks.rados.rados.0.smithi089.stdout:10: writing smithi08937929-10 from 1893632 to 2151263 tid 2 2024-04-22T18:38:54.201 INFO:tasks.rados.rados.0.smithi089.stdout:11: write initial oid 11 2024-04-22T18:38:54.201 INFO:tasks.rados.rados.0.smithi089.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-04-22T18:38:54.206 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:54.206 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:38:54.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:38:54.206 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [] 2024-04-22T18:38:54.206 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:54.207 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:54.207+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:38:54.207 INFO:tasks.rados.rados.0.smithi089.stdout:11: writing smithi08937929-11 from 568070 to 1036913 tid 1 2024-04-22T18:38:54.213 INFO:tasks.rados.rados.0.smithi089.stdout:11: writing smithi08937929-11 from 1552114 to 2168070 tid 2 2024-04-22T18:38:54.213 INFO:tasks.rados.rados.0.smithi089.stdout:12: write initial oid 12 2024-04-22T18:38:54.213 INFO:tasks.rados.rados.0.smithi089.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-04-22T18:38:54.214 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:38:54.219 INFO:tasks.rados.rados.0.smithi089.stdout:12: writing smithi08937929-12 from 584877 to 1128969 tid 1 2024-04-22T18:38:54.225 INFO:tasks.rados.rados.0.smithi089.stdout:12: writing smithi08937929-12 from 1610596 to 2172599 tid 2 2024-04-22T18:38:54.225 INFO:tasks.rados.rados.0.smithi089.stdout:13: write initial oid 13 2024-04-22T18:38:54.225 INFO:tasks.rados.rados.0.smithi089.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-04-22T18:38:54.231 INFO:tasks.rados.rados.0.smithi089.stdout:13: writing smithi08937929-13 from 601684 to 1221025 tid 1 2024-04-22T18:38:54.237 INFO:tasks.rados.rados.0.smithi089.stdout:13: writing smithi08937929-13 from 1669078 to 2201684 tid 2 2024-04-22T18:38:54.237 INFO:tasks.rados.rados.0.smithi089.stdout:14: write initial oid 14 2024-04-22T18:38:54.237 INFO:tasks.rados.rados.0.smithi089.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-04-22T18:38:54.244 INFO:tasks.rados.rados.0.smithi089.stdout:14: writing smithi08937929-14 from 618491 to 1313081 tid 1 2024-04-22T18:38:54.249 INFO:tasks.rados.rados.0.smithi089.stdout:14: writing smithi08937929-14 from 2011207 to 2218491 tid 2 2024-04-22T18:38:54.249 INFO:tasks.rados.rados.0.smithi089.stdout:15: write initial oid 15 2024-04-22T18:38:54.249 INFO:tasks.rados.rados.0.smithi089.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-04-22T18:38:54.257 INFO:tasks.rados.rados.0.smithi089.stdout:15: writing smithi08937929-15 from 635298 to 1405137 tid 1 2024-04-22T18:38:54.262 INFO:tasks.rados.rados.0.smithi089.stdout:15: writing smithi08937929-15 from 2069689 to 2235298 tid 2 2024-04-22T18:38:54.262 INFO:tasks.rados.rados.0.smithi089.stdout:16: write initial oid 16 2024-04-22T18:38:54.262 INFO:tasks.rados.rados.0.smithi089.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-04-22T18:38:54.268 INFO:tasks.rados.rados.0.smithi089.stdout:16: writing smithi08937929-16 from 652105 to 1097193 tid 1 2024-04-22T18:38:54.274 INFO:tasks.rados.rados.0.smithi089.stdout:16: writing smithi08937929-16 from 1728171 to 2181159 tid 2 2024-04-22T18:38:54.274 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.274 INFO:tasks.rados.rados.0.smithi089.stdout:2: finishing write tid 1 to smithi08937929-2 2024-04-22T18:38:54.274 INFO:tasks.rados.rados.0.smithi089.stdout:4: finishing write tid 1 to smithi08937929-4 2024-04-22T18:38:54.274 INFO:tasks.rados.rados.0.smithi089.stdout:4: finishing write tid 2 to smithi08937929-4 2024-04-22T18:38:54.307 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:54.309+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:54.346 INFO:tasks.rados.rados.0.smithi089.stdout:5: finishing write tid 1 to smithi08937929-5 2024-04-22T18:38:54.346 INFO:tasks.rados.rados.0.smithi089.stdout:4: finishing write tid 3 to smithi08937929-4 2024-04-22T18:38:54.346 INFO:tasks.rados.rados.0.smithi089.stdout:5: finishing write tid 2 to smithi08937929-5 2024-04-22T18:38:54.346 INFO:tasks.rados.rados.0.smithi089.stdout:5: finishing write tid 3 to smithi08937929-5 2024-04-22T18:38:54.346 INFO:tasks.rados.rados.0.smithi089.stdout:5: finishing write tid 4 to smithi08937929-5 2024-04-22T18:38:54.346 INFO:tasks.rados.rados.0.smithi089.stdout:5: oid 5 updating version 0 to 1 2024-04-22T18:38:54.347 INFO:tasks.rados.rados.0.smithi089.stdout:5: oid 5 updating version 1 to 2 2024-04-22T18:38:54.347 INFO:tasks.rados.rados.0.smithi089.stdout:5: oid 5 updating version 2 to 3 2024-04-22T18:38:54.347 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-22T18:38:54.347 INFO:tasks.rados.rados.0.smithi089.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-04-22T18:38:54.347 INFO:tasks.rados.rados.0.smithi089.stdout:5: done (15 left) 2024-04-22T18:38:54.347 INFO:tasks.rados.rados.0.smithi089.stdout:17: write initial oid 17 2024-04-22T18:38:54.347 INFO:tasks.rados.rados.0.smithi089.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-04-22T18:38:54.352 INFO:tasks.rados.rados.0.smithi089.stdout:17: writing smithi08937929-17 from 668912 to 1305602 tid 1 2024-04-22T18:38:54.357 INFO:tasks.rados.rados.0.smithi089.stdout:17: writing smithi08937929-17 from 1903006 to 2268912 tid 2 2024-04-22T18:38:54.358 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.26914919900000001, 2024-04-22T18:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26916846999999999, 2024-04-22T18:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:38:54.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:38:54.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:38:54.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:38:54.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:38:54.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:38:54.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:38:54.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:38:54.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:38:54.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:38:54.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:38:54.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:38:54.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:38:54.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:54.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:38:54.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:38:54.367 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.367 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.367 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.368 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2 4.8 4:12d9f826:::smithi08937929-1:head [write 1542129~474678 in=474678b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:38:54.368 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:38:54.368 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.16843792899999999, 2024-04-22T18:38:54.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.168541775, 2024-04-22T18:38:54.368 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.368 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:38:54.368 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:54.368 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:54.368 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:54.368 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2 2024-04-22T18:38:54.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.369 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:38:54.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:38:54.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194568+0000", 2024-04-22T18:38:54.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:38:54.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194869+0000", 2024-04-22T18:38:54.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030115200000000001 2024-04-22T18:38:54.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194879+0000", 2024-04-22T18:38:54.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8479999999999996e-06 2024-04-22T18:38:54.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294954+0000", 2024-04-22T18:38:54.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100074702 2024-04-22T18:38:54.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296019+0000", 2024-04-22T18:38:54.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010649189999999999 2024-04-22T18:38:54.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296086+0000", 2024-04-22T18:38:54.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7261000000000005e-05 2024-04-22T18:38:54.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:38:54.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296173+0000", 2024-04-22T18:38:54.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6935999999999996e-05 2024-04-22T18:38:54.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:54.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297196+0000", 2024-04-22T18:38:54.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023415 2024-04-22T18:38:54.373 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.373 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.373 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.373 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:53 4.3 e27/20 4:ca46fde9:::smithi08937929-14:head v 27'1, mlcod=0'0)", 2024-04-22T18:38:54.373 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:38:54.373 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.11519072599999999, 2024-04-22T18:38:54.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.115373743, 2024-04-22T18:38:54.374 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.374 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:54.374 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:38:54.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:38:54.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:38:54.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967296 2024-04-22T18:38:54.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248504+0000", 2024-04-22T18:38:54.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000686091 2024-04-22T18:38:54.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248512+0000", 2024-04-22T18:38:54.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1850000000000004e-06 2024-04-22T18:38:54.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248517+0000", 2024-04-22T18:38:54.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5469999999999998e-06 2024-04-22T18:38:54.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.348632+0000", 2024-04-22T18:38:54.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100115069 2024-04-22T18:38:54.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.348644+0000", 2024-04-22T18:38:54.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2001000000000001e-05 2024-04-22T18:38:54.377 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.377 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.377 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.378 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:54 4.3 e27/20 4:ca46fde9:::smithi08937929-14:head v 27'2, mlcod=0'0)", 2024-04-22T18:38:54.378 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.252046+0000", 2024-04-22T18:38:54.378 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.11096318300000001, 2024-04-22T18:38:54.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.111203081, 2024-04-22T18:38:54.378 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.378 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:54.378 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252046+0000", 2024-04-22T18:38:54.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252046+0000", 2024-04-22T18:38:54.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252045+0000", 2024-04-22T18:38:54.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:54.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252215+0000", 2024-04-22T18:38:54.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017004499999999999 2024-04-22T18:38:54.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252221+0000", 2024-04-22T18:38:54.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.677e-06 2024-04-22T18:38:54.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252225+0000", 2024-04-22T18:38:54.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7730000000000001e-06 2024-04-22T18:38:54.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.349338+0000", 2024-04-22T18:38:54.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.097113853999999999 2024-04-22T18:38:54.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.349350+0000", 2024-04-22T18:38:54.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1268e-05 2024-04-22T18:38:54.381 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.381 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.382 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.382 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:55 4.3 e27/20)", 2024-04-22T18:38:54.382 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:38:54.382 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.11039553100000001, 2024-04-22T18:38:54.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.11069207, 2024-04-22T18:38:54.382 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.382 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-22T18:38:54.382 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:38:54.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:38:54.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:38:54.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:54.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252616+0000", 2024-04-22T18:38:54.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1889999999999999e-06 2024-04-22T18:38:54.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252622+0000", 2024-04-22T18:38:54.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0589999999999999e-06 2024-04-22T18:38:54.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252625+0000", 2024-04-22T18:38:54.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3239999999999999e-06 2024-04-22T18:38:54.385 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.385 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.385 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.385 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=62b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:38:54.385 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:38:54.385 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.068036975, 2024-04-22T18:38:54.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.068386399, 2024-04-22T18:38:54.385 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.386 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:38:54.386 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:54.386 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:54.386 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:54.386 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3 2024-04-22T18:38:54.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.386 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:38:54.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:38:54.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294970+0000", 2024-04-22T18:38:54.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:38:54.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294983+0000", 2024-04-22T18:38:54.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3429e-05 2024-04-22T18:38:54.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294993+0000", 2024-04-22T18:38:54.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0416e-05 2024-04-22T18:38:54.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295007+0000", 2024-04-22T18:38:54.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3742e-05 2024-04-22T18:38:54.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296793+0000", 2024-04-22T18:38:54.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017860420000000001 2024-04-22T18:38:54.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296841+0000", 2024-04-22T18:38:54.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7706e-05 2024-04-22T18:38:54.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:38:54.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296913+0000", 2024-04-22T18:38:54.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2578999999999997e-05 2024-04-22T18:38:54.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:54.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297946+0000", 2024-04-22T18:38:54.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001032802 2024-04-22T18:38:54.390 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.390 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.390 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.391 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:38:54.391 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:38:54.391 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.067996547000000004, 2024-04-22T18:38:54.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.068426491000000006, 2024-04-22T18:38:54.391 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.391 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for rw locks", 2024-04-22T18:38:54.391 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:54.391 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:54.391 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:54.391 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4 2024-04-22T18:38:54.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.392 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:38:54.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:38:54.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295011+0000", 2024-04-22T18:38:54.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:54.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295016+0000", 2024-04-22T18:38:54.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2400000000000001e-06 2024-04-22T18:38:54.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295022+0000", 2024-04-22T18:38:54.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8519999999999998e-06 2024-04-22T18:38:54.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295027+0000", 2024-04-22T18:38:54.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9749999999999998e-06 2024-04-22T18:38:54.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297126+0000", 2024-04-22T18:38:54.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020981070000000001 2024-04-22T18:38:54.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:38:54.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297167+0000", 2024-04-22T18:38:54.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1842999999999999e-05 2024-04-22T18:38:54.395 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.395 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.395 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.395 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.395 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:38:54.395 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 7 2024-04-22T18:38:54.395 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:54.396 INFO:tasks.rados.rados.0.smithi089.stdout:2: finishing write tid 2 to smithi08937929-2 2024-04-22T18:38:54.396 INFO:tasks.rados.rados.0.smithi089.stdout:10: finishing write tid 1 to smithi08937929-10 2024-04-22T18:38:54.397 INFO:tasks.rados.rados.0.smithi089.stdout:1: finishing write tid 1 to smithi08937929-1 2024-04-22T18:38:54.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:38:54.398 INFO:tasks.rados.rados.0.smithi089.stdout:1: finishing write tid 2 to smithi08937929-1 2024-04-22T18:38:54.406 INFO:tasks.rados.rados.0.smithi089.stdout:12: finishing write tid 1 to smithi08937929-12 2024-04-22T18:38:54.407 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:54.409+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:38:54.409 INFO:tasks.rados.rados.0.smithi089.stdout:12: finishing write tid 2 to smithi08937929-12 2024-04-22T18:38:54.412 INFO:tasks.rados.rados.0.smithi089.stdout:2: finishing write tid 3 to smithi08937929-2 2024-04-22T18:38:54.412 INFO:tasks.rados.rados.0.smithi089.stdout:2: finishing write tid 4 to smithi08937929-2 2024-04-22T18:38:54.413 INFO:tasks.rados.rados.0.smithi089.stdout:2: oid 2 updating version 0 to 1 2024-04-22T18:38:54.413 INFO:tasks.rados.rados.0.smithi089.stdout:2: oid 2 updating version 1 to 2 2024-04-22T18:38:54.413 INFO:tasks.rados.rados.0.smithi089.stdout:2: oid 2 updating version 2 to 3 2024-04-22T18:38:54.413 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-22T18:38:54.413 INFO:tasks.rados.rados.0.smithi089.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-04-22T18:38:54.413 INFO:tasks.rados.rados.0.smithi089.stdout:2: done (15 left) 2024-04-22T18:38:54.413 INFO:tasks.rados.rados.0.smithi089.stdout:18: write initial oid 18 2024-04-22T18:38:54.413 INFO:tasks.rados.rados.0.smithi089.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-04-22T18:38:54.420 INFO:tasks.rados.rados.0.smithi089.stdout:18: writing smithi08937929-18 from 685719 to 1397658 tid 1 2024-04-22T18:38:54.426 INFO:tasks.rados.rados.0.smithi089.stdout:18: writing smithi08937929-18 from 1845135 to 2285719 tid 2 2024-04-22T18:38:54.426 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.446 INFO:tasks.rados.rados.0.smithi089.stdout:4: finishing write tid 4 to smithi08937929-4 2024-04-22T18:38:54.446 INFO:tasks.rados.rados.0.smithi089.stdout:4: oid 4 updating version 0 to 1 2024-04-22T18:38:54.446 INFO:tasks.rados.rados.0.smithi089.stdout:4: oid 4 updating version 1 to 2 2024-04-22T18:38:54.446 INFO:tasks.rados.rados.0.smithi089.stdout:4: oid 4 updating version 2 to 3 2024-04-22T18:38:54.446 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-22T18:38:54.446 INFO:tasks.rados.rados.0.smithi089.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-04-22T18:38:54.446 INFO:tasks.rados.rados.0.smithi089.stdout:4: done (15 left) 2024-04-22T18:38:54.446 INFO:tasks.rados.rados.0.smithi089.stdout:19: write initial oid 19 2024-04-22T18:38:54.447 INFO:tasks.rados.rados.0.smithi089.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-04-22T18:38:54.454 INFO:tasks.rados.rados.0.smithi089.stdout:19: writing smithi08937929-19 from 702526 to 1489714 tid 1 2024-04-22T18:38:54.458 INFO:tasks.rados.rados.0.smithi089.stdout:19: writing smithi08937929-19 from 1903617 to 2302526 tid 2 2024-04-22T18:38:54.458 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.497 INFO:tasks.rados.rados.0.smithi089.stdout:1: finishing write tid 3 to smithi08937929-1 2024-04-22T18:38:54.497 INFO:tasks.rados.rados.0.smithi089.stdout:1: finishing write tid 4 to smithi08937929-1 2024-04-22T18:38:54.497 INFO:tasks.rados.rados.0.smithi089.stdout:1: oid 1 updating version 0 to 1 2024-04-22T18:38:54.498 INFO:tasks.rados.rados.0.smithi089.stdout:1: oid 1 updating version 1 to 2 2024-04-22T18:38:54.498 INFO:tasks.rados.rados.0.smithi089.stdout:1: oid 1 updating version 2 to 3 2024-04-22T18:38:54.498 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-22T18:38:54.498 INFO:tasks.rados.rados.0.smithi089.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-04-22T18:38:54.498 INFO:tasks.rados.rados.0.smithi089.stdout:1: done (15 left) 2024-04-22T18:38:54.498 INFO:tasks.rados.rados.0.smithi089.stdout:20: write initial oid 20 2024-04-22T18:38:54.498 INFO:tasks.rados.rados.0.smithi089.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-04-22T18:38:54.503 INFO:tasks.rados.rados.0.smithi089.stdout:20: writing smithi08937929-20 from 719333 to 1181770 tid 1 2024-04-22T18:38:54.508 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:54.509+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:54.509 INFO:tasks.rados.rados.0.smithi089.stdout:20: writing smithi08937929-20 from 1962099 to 2319333 tid 2 2024-04-22T18:38:54.510 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.540 INFO:tasks.rados.rados.0.smithi089.stdout:10: finishing write tid 2 to smithi08937929-10 2024-04-22T18:38:54.541 INFO:tasks.rados.rados.0.smithi089.stdout:10: finishing write tid 3 to smithi08937929-10 2024-04-22T18:38:54.541 INFO:tasks.rados.rados.0.smithi089.stdout:11: finishing write tid 1 to smithi08937929-11 2024-04-22T18:38:54.541 INFO:tasks.rados.rados.0.smithi089.stdout:11: finishing write tid 2 to smithi08937929-11 2024-04-22T18:38:54.541 INFO:tasks.rados.rados.0.smithi089.stdout:11: finishing write tid 3 to smithi08937929-11 2024-04-22T18:38:54.541 INFO:tasks.rados.rados.0.smithi089.stdout:11: finishing write tid 4 to smithi08937929-11 2024-04-22T18:38:54.541 INFO:tasks.rados.rados.0.smithi089.stdout:11: oid 11 updating version 0 to 4 2024-04-22T18:38:54.541 INFO:tasks.rados.rados.0.smithi089.stdout:11: oid 11 updating version 4 to 6 2024-04-22T18:38:54.541 INFO:tasks.rados.rados.0.smithi089.stdout:11: oid 11 version 6 is already newer than 5 2024-04-22T18:38:54.541 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 6 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-22T18:38:54.541 INFO:tasks.rados.rados.0.smithi089.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-04-22T18:38:54.541 INFO:tasks.rados.rados.0.smithi089.stdout:11: done (15 left) 2024-04-22T18:38:54.541 INFO:tasks.rados.rados.0.smithi089.stdout:21: write initial oid 21 2024-04-22T18:38:54.541 INFO:tasks.rados.rados.0.smithi089.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-04-22T18:38:54.542 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:38:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:38:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:38:54.542 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:54.547 INFO:tasks.rados.rados.0.smithi089.stdout:21: writing smithi08937929-21 from 736140 to 1273826 tid 1 2024-04-22T18:38:54.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:38:54.554 INFO:tasks.rados.rados.0.smithi089.stdout:21: writing smithi08937929-21 from 2020581 to 2336140 tid 2 2024-04-22T18:38:54.554 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.554 INFO:tasks.rados.rados.0.smithi089.stdout:10: finishing write tid 4 to smithi08937929-10 2024-04-22T18:38:54.554 INFO:tasks.rados.rados.0.smithi089.stdout:10: oid 10 updating version 0 to 1 2024-04-22T18:38:54.554 INFO:tasks.rados.rados.0.smithi089.stdout:10: oid 10 updating version 1 to 3 2024-04-22T18:38:54.554 INFO:tasks.rados.rados.0.smithi089.stdout:10: oid 10 version 3 is already newer than 2 2024-04-22T18:38:54.554 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-22T18:38:54.554 INFO:tasks.rados.rados.0.smithi089.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-04-22T18:38:54.554 INFO:tasks.rados.rados.0.smithi089.stdout:10: done (15 left) 2024-04-22T18:38:54.555 INFO:tasks.rados.rados.0.smithi089.stdout:22: write initial oid 22 2024-04-22T18:38:54.555 INFO:tasks.rados.rados.0.smithi089.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-04-22T18:38:54.560 INFO:tasks.rados.rados.0.smithi089.stdout:22: writing smithi08937929-22 from 752947 to 1365882 tid 1 2024-04-22T18:38:54.566 INFO:tasks.rados.rados.0.smithi089.stdout:22: writing smithi08937929-22 from 1962710 to 2352947 tid 2 2024-04-22T18:38:54.566 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.567 INFO:tasks.rados.rados.0.smithi089.stdout:6: finishing write tid 1 to smithi08937929-6 2024-04-22T18:38:54.567 INFO:tasks.rados.rados.0.smithi089.stdout:6: finishing write tid 2 to smithi08937929-6 2024-04-22T18:38:54.567 INFO:tasks.rados.rados.0.smithi089.stdout:6: finishing write tid 3 to smithi08937929-6 2024-04-22T18:38:54.567 INFO:tasks.rados.rados.0.smithi089.stdout:6: finishing write tid 4 to smithi08937929-6 2024-04-22T18:38:54.567 INFO:tasks.rados.rados.0.smithi089.stdout:6: oid 6 updating version 0 to 2 2024-04-22T18:38:54.567 INFO:tasks.rados.rados.0.smithi089.stdout:6: oid 6 version 2 is already newer than 1 2024-04-22T18:38:54.567 INFO:tasks.rados.rados.0.smithi089.stdout:6: oid 6 updating version 2 to 3 2024-04-22T18:38:54.567 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-22T18:38:54.567 INFO:tasks.rados.rados.0.smithi089.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-04-22T18:38:54.567 INFO:tasks.rados.rados.0.smithi089.stdout:13: finishing write tid 1 to smithi08937929-13 2024-04-22T18:38:54.567 INFO:tasks.rados.rados.0.smithi089.stdout:6: done (15 left) 2024-04-22T18:38:54.567 INFO:tasks.rados.rados.0.smithi089.stdout:23: write initial oid 23 2024-04-22T18:38:54.567 INFO:tasks.rados.rados.0.smithi089.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-04-22T18:38:54.573 INFO:tasks.rados.rados.0.smithi089.stdout:23: writing smithi08937929-23 from 769754 to 1457938 tid 1 2024-04-22T18:38:54.579 INFO:tasks.rados.rados.0.smithi089.stdout:23: writing smithi08937929-23 from 2021192 to 2369754 tid 2 2024-04-22T18:38:54.579 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.579 INFO:tasks.rados.rados.0.smithi089.stdout:14: finishing write tid 1 to smithi08937929-14 2024-04-22T18:38:54.579 INFO:tasks.rados.rados.0.smithi089.stdout:14: finishing write tid 2 to smithi08937929-14 2024-04-22T18:38:54.579 INFO:tasks.rados.rados.0.smithi089.stdout:14: finishing write tid 3 to smithi08937929-14 2024-04-22T18:38:54.579 INFO:tasks.rados.rados.0.smithi089.stdout:14: finishing write tid 4 to smithi08937929-14 2024-04-22T18:38:54.579 INFO:tasks.rados.rados.0.smithi089.stdout:14: oid 14 updating version 0 to 2 2024-04-22T18:38:54.579 INFO:tasks.rados.rados.0.smithi089.stdout:14: oid 14 version 2 is already newer than 1 2024-04-22T18:38:54.579 INFO:tasks.rados.rados.0.smithi089.stdout:14: oid 14 updating version 2 to 3 2024-04-22T18:38:54.580 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-22T18:38:54.580 INFO:tasks.rados.rados.0.smithi089.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-04-22T18:38:54.580 INFO:tasks.rados.rados.0.smithi089.stdout:12: finishing write tid 3 to smithi08937929-12 2024-04-22T18:38:54.580 INFO:tasks.rados.rados.0.smithi089.stdout:12: finishing write tid 4 to smithi08937929-12 2024-04-22T18:38:54.580 INFO:tasks.rados.rados.0.smithi089.stdout:12: oid 12 updating version 0 to 1 2024-04-22T18:38:54.580 INFO:tasks.rados.rados.0.smithi089.stdout:12: oid 12 updating version 1 to 2 2024-04-22T18:38:54.580 INFO:tasks.rados.rados.0.smithi089.stdout:12: oid 12 updating version 2 to 3 2024-04-22T18:38:54.580 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-22T18:38:54.580 INFO:tasks.rados.rados.0.smithi089.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-04-22T18:38:54.580 INFO:tasks.rados.rados.0.smithi089.stdout:8: finishing write tid 1 to smithi08937929-8 2024-04-22T18:38:54.580 INFO:tasks.rados.rados.0.smithi089.stdout:12: done (15 left) 2024-04-22T18:38:54.580 INFO:tasks.rados.rados.0.smithi089.stdout:14: done (14 left) 2024-04-22T18:38:54.580 INFO:tasks.rados.rados.0.smithi089.stdout:24: write initial oid 24 2024-04-22T18:38:54.581 INFO:tasks.rados.rados.0.smithi089.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-04-22T18:38:54.585 INFO:tasks.rados.rados.0.smithi089.stdout:24: writing smithi08937929-24 from 786561 to 1266347 tid 1 2024-04-22T18:38:54.590 INFO:tasks.rados.rados.0.smithi089.stdout:24: writing smithi08937929-24 from 1796027 to 2263691 tid 2 2024-04-22T18:38:54.591 INFO:tasks.rados.rados.0.smithi089.stdout:25: write initial oid 25 2024-04-22T18:38:54.591 INFO:tasks.rados.rados.0.smithi089.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-04-22T18:38:54.596 INFO:tasks.rados.rados.0.smithi089.stdout:25: writing smithi08937929-25 from 403368 to 958403 tid 1 2024-04-22T18:38:54.602 INFO:tasks.rados.rados.0.smithi089.stdout:25: writing smithi08937929-25 from 1454509 to 2182184 tid 2 2024-04-22T18:38:54.602 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.602 INFO:tasks.rados.rados.0.smithi089.stdout:8: finishing write tid 2 to smithi08937929-8 2024-04-22T18:38:54.607 INFO:tasks.rados.rados.0.smithi089.stdout:20: finishing write tid 1 to smithi08937929-20 2024-04-22T18:38:54.607 INFO:tasks.rados.rados.0.smithi089.stdout:3: finishing write tid 1 to smithi08937929-3 2024-04-22T18:38:54.608 INFO:tasks.rados.rados.0.smithi089.stdout:3: finishing write tid 2 to smithi08937929-3 2024-04-22T18:38:54.608 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:54.610+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:54.616 INFO:tasks.rados.rados.0.smithi089.stdout:3: finishing write tid 3 to smithi08937929-3 2024-04-22T18:38:54.616 INFO:tasks.rados.rados.0.smithi089.stdout:3: finishing write tid 4 to smithi08937929-3 2024-04-22T18:38:54.616 INFO:tasks.rados.rados.0.smithi089.stdout:3: oid 3 updating version 0 to 4 2024-04-22T18:38:54.616 INFO:tasks.rados.rados.0.smithi089.stdout:3: oid 3 updating version 4 to 5 2024-04-22T18:38:54.616 INFO:tasks.rados.rados.0.smithi089.stdout:3: oid 3 updating version 5 to 6 2024-04-22T18:38:54.616 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 6 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-22T18:38:54.616 INFO:tasks.rados.rados.0.smithi089.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-04-22T18:38:54.616 INFO:tasks.rados.rados.0.smithi089.stdout:3: done (15 left) 2024-04-22T18:38:54.616 INFO:tasks.rados.rados.0.smithi089.stdout:26: write initial oid 26 2024-04-22T18:38:54.616 INFO:tasks.rados.rados.0.smithi089.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-04-22T18:38:54.621 INFO:tasks.rados.rados.0.smithi089.stdout:26: writing smithi08937929-26 from 420175 to 1050459 tid 1 2024-04-22T18:38:54.627 INFO:tasks.rados.rados.0.smithi089.stdout:26: writing smithi08937929-26 from 1796638 to 2267971 tid 2 2024-04-22T18:38:54.627 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.654 INFO:tasks.rados.rados.0.smithi089.stdout:16: finishing write tid 1 to smithi08937929-16 2024-04-22T18:38:54.655 INFO:tasks.rados.rados.0.smithi089.stdout:16: finishing write tid 2 to smithi08937929-16 2024-04-22T18:38:54.655 INFO:tasks.rados.rados.0.smithi089.stdout:17: finishing write tid 1 to smithi08937929-17 2024-04-22T18:38:54.655 INFO:tasks.rados.rados.0.smithi089.stdout:16: finishing write tid 3 to smithi08937929-16 2024-04-22T18:38:54.655 INFO:tasks.rados.rados.0.smithi089.stdout:16: finishing write tid 4 to smithi08937929-16 2024-04-22T18:38:54.655 INFO:tasks.rados.rados.0.smithi089.stdout:16: oid 16 updating version 0 to 3 2024-04-22T18:38:54.655 INFO:tasks.rados.rados.0.smithi089.stdout:16: oid 16 version 3 is already newer than 1 2024-04-22T18:38:54.655 INFO:tasks.rados.rados.0.smithi089.stdout:16: oid 16 version 3 is already newer than 2 2024-04-22T18:38:54.655 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-22T18:38:54.655 INFO:tasks.rados.rados.0.smithi089.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-04-22T18:38:54.655 INFO:tasks.rados.rados.0.smithi089.stdout:16: done (15 left) 2024-04-22T18:38:54.655 INFO:tasks.rados.rados.0.smithi089.stdout:27: write initial oid 27 2024-04-22T18:38:54.655 INFO:tasks.rados.rados.0.smithi089.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-04-22T18:38:54.656 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:54.656 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:38:54.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:38:54.657 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:38:54.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.657 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:38:54.657 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:38:54.657 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.56153275499999999, 2024-04-22T18:38:54.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:38:54.657 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.657 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.657 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:54.657 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:54.657 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:54.657 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:38:54.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.658 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:38:54.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:38:54.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:38:54.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:38:54.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:38:54.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:38:54.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:38:54.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:38:54.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:38:54.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:38:54.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:38:54.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:38:54.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:38:54.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:38:54.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:38:54.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:38:54.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:38:54.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:54.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:38:54.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:38:54.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:54.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:38:54.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:38:54.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:38:54.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:38:54.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:38:54.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:38:54.662 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.662 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.662 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.663 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:17 4.1 e27/20 4:809dfd21:::smithi08937929-5:head v 27'1, mlcod=0'0)", 2024-04-22T18:38:54.663 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.141796+0000", 2024-04-22T18:38:54.663 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.51359686800000004, 2024-04-22T18:38:54.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105540862, 2024-04-22T18:38:54.663 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.663 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.663 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.141796+0000", 2024-04-22T18:38:54.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.141796+0000", 2024-04-22T18:38:54.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.141795+0000", 2024-04-22T18:38:54.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:54.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.142224+0000", 2024-04-22T18:38:54.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000429264 2024-04-22T18:38:54.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.142236+0000", 2024-04-22T18:38:54.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1958e-05 2024-04-22T18:38:54.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.142245+0000", 2024-04-22T18:38:54.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6850000000000007e-06 2024-04-22T18:38:54.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.142298+0000", 2024-04-22T18:38:54.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2747000000000002e-05 2024-04-22T18:38:54.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.142329+0000", 2024-04-22T18:38:54.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1287e-05 2024-04-22T18:38:54.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247306+0000", 2024-04-22T18:38:54.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104976713 2024-04-22T18:38:54.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247336+0000", 2024-04-22T18:38:54.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0676999999999998e-05 2024-04-22T18:38:54.666 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.667 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.667 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.667 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:18 4.1 e27/20 4:809dfd21:::smithi08937929-5:head v 27'2, mlcod=0'0)", 2024-04-22T18:38:54.667 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.146391+0000", 2024-04-22T18:38:54.667 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.50900189799999995, 2024-04-22T18:38:54.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101366119, 2024-04-22T18:38:54.667 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.667 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.667 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.146391+0000", 2024-04-22T18:38:54.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.146391+0000", 2024-04-22T18:38:54.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.146389+0000", 2024-04-22T18:38:54.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:54.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.146551+0000", 2024-04-22T18:38:54.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016207400000000001 2024-04-22T18:38:54.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.146561+0000", 2024-04-22T18:38:54.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0249e-05 2024-04-22T18:38:54.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.146574+0000", 2024-04-22T18:38:54.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2418999999999999e-05 2024-04-22T18:38:54.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.246683+0000", 2024-04-22T18:38:54.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100109215 2024-04-22T18:38:54.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.246704+0000", 2024-04-22T18:38:54.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1141999999999998e-05 2024-04-22T18:38:54.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247737+0000", 2024-04-22T18:38:54.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010323109999999999 2024-04-22T18:38:54.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247757+0000", 2024-04-22T18:38:54.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0126e-05 2024-04-22T18:38:54.671 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.671 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.671 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.671 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2 4.8 4:12d9f826:::smithi08937929-1:head [write 1542129~474678 in=474678b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:38:54.671 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:38:54.671 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.46082148499999998, 2024-04-22T18:38:54.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20435667099999999, 2024-04-22T18:38:54.671 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.671 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194568+0000", 2024-04-22T18:38:54.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:38:54.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194869+0000", 2024-04-22T18:38:54.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030115200000000001 2024-04-22T18:38:54.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194879+0000", 2024-04-22T18:38:54.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8479999999999996e-06 2024-04-22T18:38:54.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294954+0000", 2024-04-22T18:38:54.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100074702 2024-04-22T18:38:54.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296019+0000", 2024-04-22T18:38:54.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010649189999999999 2024-04-22T18:38:54.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296086+0000", 2024-04-22T18:38:54.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7261000000000005e-05 2024-04-22T18:38:54.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:38:54.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296173+0000", 2024-04-22T18:38:54.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6935999999999996e-05 2024-04-22T18:38:54.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:54.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297196+0000", 2024-04-22T18:38:54.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023415 2024-04-22T18:38:54.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:54.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398879+0000", 2024-04-22T18:38:54.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101682563 2024-04-22T18:38:54.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398906+0000", 2024-04-22T18:38:54.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6526999999999999e-05 2024-04-22T18:38:54.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398928+0000", 2024-04-22T18:38:54.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2045e-05 2024-04-22T18:38:54.677 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.677 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.677 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.677 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:37 4.2 e27/20 4:4fb26dd8:::smithi08937929-10:head v 27'1, mlcod=0'0)", 2024-04-22T18:38:54.677 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.235982+0000", 2024-04-22T18:38:54.677 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.41941023399999999, 2024-04-22T18:38:54.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0051164540000000003, 2024-04-22T18:38:54.677 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.677 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.677 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.235982+0000", 2024-04-22T18:38:54.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.235982+0000", 2024-04-22T18:38:54.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.235982+0000", 2024-04-22T18:38:54.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:54.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.236974+0000", 2024-04-22T18:38:54.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00099184399999999997 2024-04-22T18:38:54.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.236984+0000", 2024-04-22T18:38:54.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8239999999999995e-06 2024-04-22T18:38:54.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.236991+0000", 2024-04-22T18:38:54.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4869999999999996e-06 2024-04-22T18:38:54.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.237067+0000", 2024-04-22T18:38:54.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.64e-05 2024-04-22T18:38:54.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.237092+0000", 2024-04-22T18:38:54.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4559e-05 2024-04-22T18:38:54.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.241038+0000", 2024-04-22T18:38:54.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0039461729999999999 2024-04-22T18:38:54.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.241099+0000", 2024-04-22T18:38:54.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0436999999999998e-05 2024-04-22T18:38:54.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.681 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.681 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:38 4.2 e27/20 4:4fb26dd8:::smithi08937929-10:head v 27'2, mlcod=0'0)", 2024-04-22T18:38:54.681 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.237322+0000", 2024-04-22T18:38:54.681 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.41807074700000002, 2024-04-22T18:38:54.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104155552, 2024-04-22T18:38:54.682 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.682 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.682 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.237322+0000", 2024-04-22T18:38:54.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.237322+0000", 2024-04-22T18:38:54.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.237321+0000", 2024-04-22T18:38:54.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:54.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.237602+0000", 2024-04-22T18:38:54.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028083900000000001 2024-04-22T18:38:54.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.237609+0000", 2024-04-22T18:38:54.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6190000000000002e-06 2024-04-22T18:38:54.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.237615+0000", 2024-04-22T18:38:54.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2720000000000003e-06 2024-04-22T18:38:54.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.238015+0000", 2024-04-22T18:38:54.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039917899999999999 2024-04-22T18:38:54.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.238034+0000", 2024-04-22T18:38:54.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8949000000000001e-05 2024-04-22T18:38:54.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.341439+0000", 2024-04-22T18:38:54.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103404884 2024-04-22T18:38:54.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.341477+0000", 2024-04-22T18:38:54.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8729000000000001e-05 2024-04-22T18:38:54.686 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.686 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.686 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.686 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:39 4.2 e27/20 4:4fb26dd8:::smithi08937929-10:head v 27'3, mlcod=0'0)", 2024-04-22T18:38:54.686 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239412+0000", 2024-04-22T18:38:54.686 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.41598017100000001, 2024-04-22T18:38:54.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103469451, 2024-04-22T18:38:54.686 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.686 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.686 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239412+0000", 2024-04-22T18:38:54.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239412+0000", 2024-04-22T18:38:54.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239412+0000", 2024-04-22T18:38:54.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:54.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239416+0000", 2024-04-22T18:38:54.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9740000000000004e-06 2024-04-22T18:38:54.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239422+0000", 2024-04-22T18:38:54.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.993e-06 2024-04-22T18:38:54.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239426+0000", 2024-04-22T18:38:54.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.797e-06 2024-04-22T18:38:54.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.341214+0000", 2024-04-22T18:38:54.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101787893 2024-04-22T18:38:54.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.341240+0000", 2024-04-22T18:38:54.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5715e-05 2024-04-22T18:38:54.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.342834+0000", 2024-04-22T18:38:54.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001594158 2024-04-22T18:38:54.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.342882+0000", 2024-04-22T18:38:54.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7450999999999997e-05 2024-04-22T18:38:54.690 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.690 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.690 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.691 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:19 4.1 e27/20 4:809dfd21:::smithi08937929-5:head v 27'3, mlcod=0'0)", 2024-04-22T18:38:54.691 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.246389+0000", 2024-04-22T18:38:54.691 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.40900331400000001, 2024-04-22T18:38:54.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.007067177, 2024-04-22T18:38:54.691 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.691 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.691 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.246389+0000", 2024-04-22T18:38:54.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.246389+0000", 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.246388+0000", 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.246396+0000", 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.525e-06 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.246406+0000", 2024-04-22T18:38:54.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0581000000000001e-05 2024-04-22T18:38:54.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.246412+0000", 2024-04-22T18:38:54.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2249999999999997e-06 2024-04-22T18:38:54.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247391+0000", 2024-04-22T18:38:54.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00097831700000000003 2024-04-22T18:38:54.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247410+0000", 2024-04-22T18:38:54.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9576e-05 2024-04-22T18:38:54.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.253436+0000", 2024-04-22T18:38:54.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0060257059999999996 2024-04-22T18:38:54.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.253456+0000", 2024-04-22T18:38:54.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0367999999999999e-05 2024-04-22T18:38:54.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:53 4.3 e27/20 4:ca46fde9:::smithi08937929-14:head v 27'1, mlcod=0'0)", 2024-04-22T18:38:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:38:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.40757428200000001, 2024-04-22T18:38:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20214333400000001, 2024-04-22T18:38:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:38:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:38:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:38:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967296 2024-04-22T18:38:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248504+0000", 2024-04-22T18:38:54.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000686091 2024-04-22T18:38:54.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248512+0000", 2024-04-22T18:38:54.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1850000000000004e-06 2024-04-22T18:38:54.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248517+0000", 2024-04-22T18:38:54.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5469999999999998e-06 2024-04-22T18:38:54.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.348632+0000", 2024-04-22T18:38:54.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100115069 2024-04-22T18:38:54.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.348644+0000", 2024-04-22T18:38:54.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2001000000000001e-05 2024-04-22T18:38:54.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449930+0000", 2024-04-22T18:38:54.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101286574 2024-04-22T18:38:54.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449961+0000", 2024-04-22T18:38:54.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1046000000000001e-05 2024-04-22T18:38:54.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.699 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.699 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:54 4.3 e27/20 4:ca46fde9:::smithi08937929-14:head v 27'2, mlcod=0'0)", 2024-04-22T18:38:54.699 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.252046+0000", 2024-04-22T18:38:54.699 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.40334673900000001, 2024-04-22T18:38:54.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19793179499999999, 2024-04-22T18:38:54.699 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.699 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.700 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252046+0000", 2024-04-22T18:38:54.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252046+0000", 2024-04-22T18:38:54.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252045+0000", 2024-04-22T18:38:54.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:54.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252215+0000", 2024-04-22T18:38:54.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017004499999999999 2024-04-22T18:38:54.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252221+0000", 2024-04-22T18:38:54.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.677e-06 2024-04-22T18:38:54.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252225+0000", 2024-04-22T18:38:54.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7730000000000001e-06 2024-04-22T18:38:54.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.349338+0000", 2024-04-22T18:38:54.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.097113853999999999 2024-04-22T18:38:54.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.349350+0000", 2024-04-22T18:38:54.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1268e-05 2024-04-22T18:38:54.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449967+0000", 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100616754 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449977+0000", 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0978999999999999e-05 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:55 4.3 e27/20)", 2024-04-22T18:38:54.703 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:38:54.704 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.40277908699999998, 2024-04-22T18:38:54.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19766915500000001, 2024-04-22T18:38:54.704 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.704 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.704 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:38:54.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:38:54.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252616+0000", 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1889999999999999e-06 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252622+0000", 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0589999999999999e-06 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252625+0000", 2024-04-22T18:38:54.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3239999999999999e-06 2024-04-22T18:38:54.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449805+0000", 2024-04-22T18:38:54.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19717992500000001 2024-04-22T18:38:54.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449819+0000", 2024-04-22T18:38:54.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.367e-05 2024-04-22T18:38:54.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.450269+0000", 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045026300000000001 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.450282+0000", 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3229e-05 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=62b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:38:54.707 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:38:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.36042053099999999, 2024-04-22T18:38:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20408478299999999, 2024-04-22T18:38:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3 2024-04-22T18:38:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:38:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:38:54.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294970+0000", 2024-04-22T18:38:54.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:38:54.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294983+0000", 2024-04-22T18:38:54.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3429e-05 2024-04-22T18:38:54.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294993+0000", 2024-04-22T18:38:54.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0416e-05 2024-04-22T18:38:54.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295007+0000", 2024-04-22T18:38:54.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3742e-05 2024-04-22T18:38:54.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296793+0000", 2024-04-22T18:38:54.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017860420000000001 2024-04-22T18:38:54.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296841+0000", 2024-04-22T18:38:54.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7706e-05 2024-04-22T18:38:54.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:38:54.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296913+0000", 2024-04-22T18:38:54.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2578999999999997e-05 2024-04-22T18:38:54.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:54.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297946+0000", 2024-04-22T18:38:54.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001032802 2024-04-22T18:38:54.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:54.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498963+0000", 2024-04-22T18:38:54.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20101640400000001 2024-04-22T18:38:54.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498998+0000", 2024-04-22T18:38:54.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5759000000000001e-05 2024-04-22T18:38:54.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499057+0000", 2024-04-22T18:38:54.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8270999999999998e-05 2024-04-22T18:38:54.713 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.713 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.713 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.713 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:38:54.713 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:38:54.713 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.36038010300000001, 2024-04-22T18:38:54.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20418823899999999, 2024-04-22T18:38:54.713 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.713 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:54.714 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:54.714 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:54.714 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:54.714 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4 2024-04-22T18:38:54.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.714 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:38:54.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:38:54.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295011+0000", 2024-04-22T18:38:54.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:54.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295016+0000", 2024-04-22T18:38:54.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2400000000000001e-06 2024-04-22T18:38:54.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295022+0000", 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8519999999999998e-06 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295027+0000", 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9749999999999998e-06 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297126+0000", 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020981070000000001 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:38:54.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297167+0000", 2024-04-22T18:38:54.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1842999999999999e-05 2024-04-22T18:38:54.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499096+0000", 2024-04-22T18:38:54.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201928846 2024-04-22T18:38:54.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499142+0000", 2024-04-22T18:38:54.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6152000000000001e-05 2024-04-22T18:38:54.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499201+0000", 2024-04-22T18:38:54.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8190999999999999e-05 2024-04-22T18:38:54.718 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.718 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.718 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.718 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:41 4.2 e27/20 4:409e6456:::smithi08937929-11:head v 27'4, mlcod=0'0)", 2024-04-22T18:38:54.718 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.339934+0000", 2024-04-22T18:38:54.718 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.31545823899999997, 2024-04-22T18:38:54.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029725649999999999, 2024-04-22T18:38:54.718 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.718 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.718 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.339934+0000", 2024-04-22T18:38:54.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.339934+0000", 2024-04-22T18:38:54.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.339932+0000", 2024-04-22T18:38:54.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:38:54.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.340293+0000", 2024-04-22T18:38:54.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036051900000000001 2024-04-22T18:38:54.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.340307+0000", 2024-04-22T18:38:54.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3834e-05 2024-04-22T18:38:54.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.340316+0000", 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9649999999999997e-06 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.341513+0000", 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001197753 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.341531+0000", 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.802e-05 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.342884+0000", 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001353038 2024-04-22T18:38:54.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.342907+0000", 2024-04-22T18:38:54.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2435000000000001e-05 2024-04-22T18:38:54.722 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.722 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.722 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.722 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:42 4.2 e27/20 4:409e6456:::smithi08937929-11:head v 27'5, mlcod=0'0)", 2024-04-22T18:38:54.722 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.340642+0000", 2024-04-22T18:38:54.722 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.31474992800000001, 2024-04-22T18:38:54.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037873960000000002, 2024-04-22T18:38:54.722 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.340642+0000", 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.340642+0000", 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.340642+0000", 2024-04-22T18:38:54.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:54.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.341217+0000", 2024-04-22T18:38:54.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057521500000000001 2024-04-22T18:38:54.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.341222+0000", 2024-04-22T18:38:54.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:38:54.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.341226+0000", 2024-04-22T18:38:54.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0419999999999996e-06 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.342253+0000", 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001026833 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.342266+0000", 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2768e-05 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.344405+0000", 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002138718 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.344430+0000", 2024-04-22T18:38:54.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5157999999999999e-05 2024-04-22T18:38:54.726 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.726 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.726 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.726 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.4502.0:3 4.8 e27/20 ondisk, result = 0)", 2024-04-22T18:38:54.726 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.398835+0000", 2024-04-22T18:38:54.726 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.25655784700000001, 2024-04-22T18:38:54.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10022919499999999, 2024-04-22T18:38:54.726 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.726 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:54.727 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398835+0000", 2024-04-22T18:38:54.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398835+0000", 2024-04-22T18:38:54.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398834+0000", 2024-04-22T18:38:54.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:54.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398838+0000", 2024-04-22T18:38:54.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3330000000000001e-06 2024-04-22T18:38:54.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398841+0000", 2024-04-22T18:38:54.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2509999999999999e-06 2024-04-22T18:38:54.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398844+0000", 2024-04-22T18:38:54.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.802e-06 2024-04-22T18:38:54.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498951+0000", 2024-04-22T18:38:54.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100107713 2024-04-22T18:38:54.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498959+0000", 2024-04-22T18:38:54.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4429999999999997e-06 2024-04-22T18:38:54.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499064+0000", 2024-04-22T18:38:54.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010502599999999999 2024-04-22T18:38:54.730 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.730 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.730 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.730 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:43 4.2 e27/20 4:409e6456:::smithi08937929-11:head v 27'6, mlcod=0'0)", 2024-04-22T18:38:54.730 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.441876+0000", 2024-04-22T18:38:54.730 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.213516652, 2024-04-22T18:38:54.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000484623, 2024-04-22T18:38:54.731 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.731 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.731 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.441876+0000", 2024-04-22T18:38:54.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.441876+0000", 2024-04-22T18:38:54.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.441874+0000", 2024-04-22T18:38:54.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:54.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.441885+0000", 2024-04-22T18:38:54.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0729e-05 2024-04-22T18:38:54.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.441895+0000", 2024-04-22T18:38:54.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0373e-05 2024-04-22T18:38:54.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.441903+0000", 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1340000000000004e-06 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.441936+0000", 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2434000000000002e-05 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.441949+0000", 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3311e-05 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.442339+0000", 2024-04-22T18:38:54.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000389563 2024-04-22T18:38:54.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.442360+0000", 2024-04-22T18:38:54.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1665999999999999e-05 2024-04-22T18:38:54.734 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.734 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.734 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.734 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:73 4.1 e27/20 4:8db271f9:::smithi08937929-19:head v 27'4, mlcod=27'2)", 2024-04-22T18:38:54.734 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.457462+0000", 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.197930881, 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10421660100000001, 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.457462+0000", 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.457462+0000", 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.457460+0000", 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.457640+0000", 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018060299999999999 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.457653+0000", 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2615e-05 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.457660+0000", 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4640000000000003e-06 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.557752+0000", 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100092245 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.557786+0000", 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4180000000000001e-05 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.561650+0000", 2024-04-22T18:38:54.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038637060000000002 2024-04-22T18:38:54.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.561678+0000", 2024-04-22T18:38:54.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8493999999999999e-05 2024-04-22T18:38:54.738 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.738 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.738 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.738 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:74 4.1 e27/20 4:8db271f9:::smithi08937929-19:head v 27'5, mlcod=27'2)", 2024-04-22T18:38:54.738 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.461139+0000", 2024-04-22T18:38:54.738 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.19425353300000001, 2024-04-22T18:38:54.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100760604, 2024-04-22T18:38:54.739 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.739 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.739 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.461139+0000", 2024-04-22T18:38:54.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.461139+0000", 2024-04-22T18:38:54.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.461137+0000", 2024-04-22T18:38:54.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:38:54.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.461234+0000", 2024-04-22T18:38:54.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7008000000000004e-05 2024-04-22T18:38:54.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.461248+0000", 2024-04-22T18:38:54.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3767999999999999e-05 2024-04-22T18:38:54.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.461255+0000", 2024-04-22T18:38:54.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8290000000000002e-06 2024-04-22T18:38:54.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.558517+0000", 2024-04-22T18:38:54.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.097261873999999998 2024-04-22T18:38:54.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.558537+0000", 2024-04-22T18:38:54.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0638000000000001e-05 2024-04-22T18:38:54.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.561872+0000", 2024-04-22T18:38:54.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0033349090000000001 2024-04-22T18:38:54.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.561899+0000", 2024-04-22T18:38:54.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7268e-05 2024-04-22T18:38:54.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.742 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:65 4.3 e27/20 4:c0d4d677:::smithi08937929-17:head v 27'4, mlcod=27'2)", 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.556259+0000", 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.099133108999999997, 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0019251940000000001, 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.556259+0000", 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.556259+0000", 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.556258+0000", 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.556842+0000", 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058429599999999999 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:54.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.556851+0000", 2024-04-22T18:38:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7860000000000002e-06 2024-04-22T18:38:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.556857+0000", 2024-04-22T18:38:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2770000000000002e-06 2024-04-22T18:38:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.556891+0000", 2024-04-22T18:38:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3677000000000003e-05 2024-04-22T18:38:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:54.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.556911+0000", 2024-04-22T18:38:54.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9919e-05 2024-04-22T18:38:54.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:54.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.558147+0000", 2024-04-22T18:38:54.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001235863 2024-04-22T18:38:54.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:54.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:54.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:54.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.558185+0000", 2024-04-22T18:38:54.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7558999999999997e-05 2024-04-22T18:38:54.746 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.746 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.746 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.747 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:54.747 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:54.747 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:54.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:38:54.750 INFO:tasks.rados.rados.0.smithi089.stdout:27: writing smithi08937929-27 from 436982 to 1142515 tid 1 2024-04-22T18:38:54.750 INFO:tasks.rados.rados.0.smithi089.stdout:27: writing smithi08937929-27 from 1855120 to 2436982 tid 2 2024-04-22T18:38:54.750 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.750 INFO:tasks.rados.rados.0.smithi089.stdout:7: finishing write tid 1 to smithi08937929-7 2024-04-22T18:38:54.751 INFO:tasks.rados.rados.0.smithi089.stdout:7: finishing write tid 2 to smithi08937929-7 2024-04-22T18:38:54.751 INFO:tasks.rados.rados.0.smithi089.stdout:7: finishing write tid 3 to smithi08937929-7 2024-04-22T18:38:54.751 INFO:tasks.rados.rados.0.smithi089.stdout:19: finishing write tid 1 to smithi08937929-19 2024-04-22T18:38:54.751 INFO:tasks.rados.rados.0.smithi089.stdout:7: finishing write tid 4 to smithi08937929-7 2024-04-22T18:38:54.751 INFO:tasks.rados.rados.0.smithi089.stdout:7: oid 7 updating version 0 to 2 2024-04-22T18:38:54.751 INFO:tasks.rados.rados.0.smithi089.stdout:7: oid 7 version 2 is already newer than 1 2024-04-22T18:38:54.751 INFO:tasks.rados.rados.0.smithi089.stdout:7: oid 7 updating version 2 to 3 2024-04-22T18:38:54.751 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-22T18:38:54.751 INFO:tasks.rados.rados.0.smithi089.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-04-22T18:38:54.751 INFO:tasks.rados.rados.0.smithi089.stdout:19: finishing write tid 2 to smithi08937929-19 2024-04-22T18:38:54.751 INFO:tasks.rados.rados.0.smithi089.stdout:19: finishing write tid 3 to smithi08937929-19 2024-04-22T18:38:54.751 INFO:tasks.rados.rados.0.smithi089.stdout:7: done (15 left) 2024-04-22T18:38:54.751 INFO:tasks.rados.rados.0.smithi089.stdout:28: write initial oid 28 2024-04-22T18:38:54.752 INFO:tasks.rados.rados.0.smithi089.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-04-22T18:38:54.752 INFO:tasks.rados.rados.0.smithi089.stdout:28: writing smithi08937929-28 from 453789 to 1234571 tid 1 2024-04-22T18:38:54.752 INFO:tasks.rados.rados.0.smithi089.stdout:28: writing smithi08937929-28 from 1913602 to 2388604 tid 2 2024-04-22T18:38:54.752 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.752 INFO:tasks.rados.rados.0.smithi089.stdout:19: finishing write tid 4 to smithi08937929-19 2024-04-22T18:38:54.752 INFO:tasks.rados.rados.0.smithi089.stdout:19: oid 19 updating version 0 to 5 2024-04-22T18:38:54.752 INFO:tasks.rados.rados.0.smithi089.stdout:19: oid 19 updating version 5 to 6 2024-04-22T18:38:54.752 INFO:tasks.rados.rados.0.smithi089.stdout:19: oid 19 version 6 is already newer than 4 2024-04-22T18:38:54.752 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-22T18:38:54.752 INFO:tasks.rados.rados.0.smithi089.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-04-22T18:38:54.752 INFO:tasks.rados.rados.0.smithi089.stdout:19: done (15 left) 2024-04-22T18:38:54.752 INFO:tasks.rados.rados.0.smithi089.stdout:29: write initial oid 29 2024-04-22T18:38:54.752 INFO:tasks.rados.rados.0.smithi089.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-04-22T18:38:54.752 INFO:tasks.rados.rados.0.smithi089.stdout:29: writing smithi08937929-29 from 470596 to 926627 tid 1 2024-04-22T18:38:54.753 INFO:tasks.rados.rados.0.smithi089.stdout:29: writing smithi08937929-29 from 1572084 to 2190744 tid 2 2024-04-22T18:38:54.753 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.753 INFO:tasks.rados.rados.0.smithi089.stdout:13: finishing write tid 2 to smithi08937929-13 2024-04-22T18:38:54.753 INFO:tasks.rados.rados.0.smithi089.stdout:13: finishing write tid 3 to smithi08937929-13 2024-04-22T18:38:54.753 INFO:tasks.rados.rados.0.smithi089.stdout:13: finishing write tid 4 to smithi08937929-13 2024-04-22T18:38:54.753 INFO:tasks.rados.rados.0.smithi089.stdout:13: oid 13 updating version 0 to 4 2024-04-22T18:38:54.753 INFO:tasks.rados.rados.0.smithi089.stdout:13: oid 13 updating version 4 to 6 2024-04-22T18:38:54.753 INFO:tasks.rados.rados.0.smithi089.stdout:13: oid 13 version 6 is already newer than 5 2024-04-22T18:38:54.753 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-22T18:38:54.753 INFO:tasks.rados.rados.0.smithi089.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-04-22T18:38:54.753 INFO:tasks.rados.rados.0.smithi089.stdout:13: done (15 left) 2024-04-22T18:38:54.753 INFO:tasks.rados.rados.0.smithi089.stdout:30: write initial oid 30 2024-04-22T18:38:54.753 INFO:tasks.rados.rados.0.smithi089.stdout:15: finishing write tid 1 to smithi08937929-15 2024-04-22T18:38:54.753 INFO:tasks.rados.rados.0.smithi089.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-04-22T18:38:54.754 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:54.748+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:54.754 INFO:tasks.rados.rados.0.smithi089.stdout:30: writing smithi08937929-30 from 487403 to 1018683 tid 1 2024-04-22T18:38:54.758 INFO:tasks.rados.rados.0.smithi089.stdout:30: writing smithi08937929-30 from 1514213 to 1992884 tid 2 2024-04-22T18:38:54.759 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.759 INFO:tasks.rados.rados.0.smithi089.stdout:21: finishing write tid 1 to smithi08937929-21 2024-04-22T18:38:54.759 INFO:tasks.rados.rados.0.smithi089.stdout:21: finishing write tid 2 to smithi08937929-21 2024-04-22T18:38:54.759 INFO:tasks.rados.rados.0.smithi089.stdout:21: finishing write tid 3 to smithi08937929-21 2024-04-22T18:38:54.759 INFO:tasks.rados.rados.0.smithi089.stdout:21: finishing write tid 4 to smithi08937929-21 2024-04-22T18:38:54.759 INFO:tasks.rados.rados.0.smithi089.stdout:21: oid 21 updating version 0 to 6 2024-04-22T18:38:54.759 INFO:tasks.rados.rados.0.smithi089.stdout:21: oid 21 version 6 is already newer than 4 2024-04-22T18:38:54.759 INFO:tasks.rados.rados.0.smithi089.stdout:21: oid 21 version 6 is already newer than 5 2024-04-22T18:38:54.759 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-22T18:38:54.759 INFO:tasks.rados.rados.0.smithi089.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-04-22T18:38:54.759 INFO:tasks.rados.rados.0.smithi089.stdout:21: done (15 left) 2024-04-22T18:38:54.759 INFO:tasks.rados.rados.0.smithi089.stdout:31: write initial oid 31 2024-04-22T18:38:54.759 INFO:tasks.rados.rados.0.smithi089.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-04-22T18:38:54.764 INFO:tasks.rados.rados.0.smithi089.stdout:31: writing smithi08937929-31 from 504210 to 1110739 tid 1 2024-04-22T18:38:54.770 INFO:tasks.rados.rados.0.smithi089.stdout:31: writing smithi08937929-31 from 1572695 to 2195024 tid 2 2024-04-22T18:38:54.770 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.770 INFO:tasks.rados.rados.0.smithi089.stdout:8: finishing write tid 3 to smithi08937929-8 2024-04-22T18:38:54.770 INFO:tasks.rados.rados.0.smithi089.stdout:8: finishing write tid 4 to smithi08937929-8 2024-04-22T18:38:54.770 INFO:tasks.rados.rados.0.smithi089.stdout:8: oid 8 updating version 0 to 1 2024-04-22T18:38:54.771 INFO:tasks.rados.rados.0.smithi089.stdout:8: oid 8 updating version 1 to 2 2024-04-22T18:38:54.771 INFO:tasks.rados.rados.0.smithi089.stdout:8: oid 8 updating version 2 to 3 2024-04-22T18:38:54.771 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-22T18:38:54.771 INFO:tasks.rados.rados.0.smithi089.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-04-22T18:38:54.771 INFO:tasks.rados.rados.0.smithi089.stdout:22: finishing write tid 1 to smithi08937929-22 2024-04-22T18:38:54.771 INFO:tasks.rados.rados.0.smithi089.stdout:22: finishing write tid 2 to smithi08937929-22 2024-04-22T18:38:54.771 INFO:tasks.rados.rados.0.smithi089.stdout:8: done (15 left) 2024-04-22T18:38:54.771 INFO:tasks.rados.rados.0.smithi089.stdout:32: write initial oid 32 2024-04-22T18:38:54.771 INFO:tasks.rados.rados.0.smithi089.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-04-22T18:38:54.777 INFO:tasks.rados.rados.0.smithi089.stdout:32: writing smithi08937929-32 from 521017 to 1319148 tid 1 2024-04-22T18:38:54.782 INFO:tasks.rados.rados.0.smithi089.stdout:32: writing smithi08937929-32 from 1747530 to 2229870 tid 2 2024-04-22T18:38:54.782 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.808 INFO:tasks.rados.rados.0.smithi089.stdout:9: finishing write tid 1 to smithi08937929-9 2024-04-22T18:38:54.808 INFO:tasks.rados.rados.0.smithi089.stdout:9: finishing write tid 2 to smithi08937929-9 2024-04-22T18:38:54.847 INFO:tasks.rados.rados.0.smithi089.stdout:15: finishing write tid 2 to smithi08937929-15 2024-04-22T18:38:54.847 INFO:tasks.rados.rados.0.smithi089.stdout:15: finishing write tid 3 to smithi08937929-15 2024-04-22T18:38:54.847 INFO:tasks.rados.rados.0.smithi089.stdout:15: finishing write tid 4 to smithi08937929-15 2024-04-22T18:38:54.847 INFO:tasks.rados.rados.0.smithi089.stdout:15: oid 15 updating version 0 to 8 2024-04-22T18:38:54.847 INFO:tasks.rados.rados.0.smithi089.stdout:15: oid 15 version 8 is already newer than 7 2024-04-22T18:38:54.847 INFO:tasks.rados.rados.0.smithi089.stdout:15: oid 15 updating version 8 to 9 2024-04-22T18:38:54.848 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 9 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-22T18:38:54.848 INFO:tasks.rados.rados.0.smithi089.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-04-22T18:38:54.848 INFO:tasks.rados.rados.0.smithi089.stdout:15: done (15 left) 2024-04-22T18:38:54.848 INFO:tasks.rados.rados.0.smithi089.stdout:33: write initial oid 33 2024-04-22T18:38:54.848 INFO:tasks.rados.rados.0.smithi089.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-04-22T18:38:54.849 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:54.851+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:54.852 INFO:tasks.rados.rados.0.smithi089.stdout:33: writing smithi08937929-33 from 537824 to 1011204 tid 1 2024-04-22T18:38:54.860 INFO:tasks.rados.rados.0.smithi089.stdout:33: writing smithi08937929-33 from 1806012 to 2432010 tid 2 2024-04-22T18:38:54.860 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.865 INFO:tasks.rados.rados.0.smithi089.stdout:22: finishing write tid 3 to smithi08937929-22 2024-04-22T18:38:54.865 INFO:tasks.rados.rados.0.smithi089.stdout:22: finishing write tid 4 to smithi08937929-22 2024-04-22T18:38:54.865 INFO:tasks.rados.rados.0.smithi089.stdout:22: oid 22 updating version 0 to 6 2024-04-22T18:38:54.865 INFO:tasks.rados.rados.0.smithi089.stdout:22: oid 22 version 6 is already newer than 5 2024-04-22T18:38:54.865 INFO:tasks.rados.rados.0.smithi089.stdout:22: oid 22 version 6 is already newer than 4 2024-04-22T18:38:54.865 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-22T18:38:54.865 INFO:tasks.rados.rados.0.smithi089.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-04-22T18:38:54.865 INFO:tasks.rados.rados.0.smithi089.stdout:22: done (15 left) 2024-04-22T18:38:54.865 INFO:tasks.rados.rados.0.smithi089.stdout:34: write initial oid 34 2024-04-22T18:38:54.865 INFO:tasks.rados.rados.0.smithi089.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-04-22T18:38:54.870 INFO:tasks.rados.rados.0.smithi089.stdout:34: writing smithi08937929-34 from 554631 to 1103260 tid 1 2024-04-22T18:38:54.877 INFO:tasks.rados.rados.0.smithi089.stdout:34: writing smithi08937929-34 from 1748141 to 2234150 tid 2 2024-04-22T18:38:54.877 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.892 INFO:tasks.rados.rados.0.smithi089.stdout:32: finishing write tid 1 to smithi08937929-32 2024-04-22T18:38:54.892 INFO:tasks.rados.rados.0.smithi089.stdout:32: finishing write tid 2 to smithi08937929-32 2024-04-22T18:38:54.892 INFO:tasks.rados.rados.0.smithi089.stdout:32: finishing write tid 3 to smithi08937929-32 2024-04-22T18:38:54.893 INFO:tasks.rados.rados.0.smithi089.stdout:32: finishing write tid 4 to smithi08937929-32 2024-04-22T18:38:54.893 INFO:tasks.rados.rados.0.smithi089.stdout:32: oid 32 updating version 0 to 3 2024-04-22T18:38:54.893 INFO:tasks.rados.rados.0.smithi089.stdout:32: oid 32 version 3 is already newer than 1 2024-04-22T18:38:54.893 INFO:tasks.rados.rados.0.smithi089.stdout:32: oid 32 version 3 is already newer than 2 2024-04-22T18:38:54.893 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 3 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-22T18:38:54.893 INFO:tasks.rados.rados.0.smithi089.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-04-22T18:38:54.893 INFO:tasks.rados.rados.0.smithi089.stdout:32: done (15 left) 2024-04-22T18:38:54.893 INFO:tasks.rados.rados.0.smithi089.stdout:35: write initial oid 35 2024-04-22T18:38:54.893 INFO:tasks.rados.rados.0.smithi089.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-04-22T18:38:54.894 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:54.894 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:38:54.894 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:38:54.894 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:54.898 INFO:tasks.rados.rados.0.smithi089.stdout:35: writing smithi08937929-35 from 571438 to 1195316 tid 1 2024-04-22T18:38:54.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:38:54.905 INFO:tasks.rados.rados.0.smithi089.stdout:35: writing smithi08937929-35 from 1806623 to 2436290 tid 2 2024-04-22T18:38:54.906 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.906 INFO:tasks.rados.rados.0.smithi089.stdout:29: finishing write tid 1 to smithi08937929-29 2024-04-22T18:38:54.906 INFO:tasks.rados.rados.0.smithi089.stdout:29: finishing write tid 2 to smithi08937929-29 2024-04-22T18:38:54.906 INFO:tasks.rados.rados.0.smithi089.stdout:29: finishing write tid 3 to smithi08937929-29 2024-04-22T18:38:54.906 INFO:tasks.rados.rados.0.smithi089.stdout:29: finishing write tid 4 to smithi08937929-29 2024-04-22T18:38:54.906 INFO:tasks.rados.rados.0.smithi089.stdout:29: oid 29 updating version 0 to 5 2024-04-22T18:38:54.906 INFO:tasks.rados.rados.0.smithi089.stdout:29: oid 29 updating version 5 to 6 2024-04-22T18:38:54.906 INFO:tasks.rados.rados.0.smithi089.stdout:29: oid 29 version 6 is already newer than 4 2024-04-22T18:38:54.906 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 6 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-22T18:38:54.906 INFO:tasks.rados.rados.0.smithi089.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-04-22T18:38:54.906 INFO:tasks.rados.rados.0.smithi089.stdout:29: done (15 left) 2024-04-22T18:38:54.906 INFO:tasks.rados.rados.0.smithi089.stdout:36: write initial oid 36 2024-04-22T18:38:54.907 INFO:tasks.rados.rados.0.smithi089.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-04-22T18:38:54.912 INFO:tasks.rados.rados.0.smithi089.stdout:36: writing smithi08937929-36 from 588245 to 1287372 tid 1 2024-04-22T18:38:54.919 INFO:tasks.rados.rados.0.smithi089.stdout:36: writing smithi08937929-36 from 1865105 to 2354783 tid 2 2024-04-22T18:38:54.919 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.919 INFO:tasks.rados.rados.0.smithi089.stdout:20: finishing write tid 2 to smithi08937929-20 2024-04-22T18:38:54.919 INFO:tasks.rados.rados.0.smithi089.stdout:20: finishing write tid 3 to smithi08937929-20 2024-04-22T18:38:54.919 INFO:tasks.rados.rados.0.smithi089.stdout:20: finishing write tid 4 to smithi08937929-20 2024-04-22T18:38:54.919 INFO:tasks.rados.rados.0.smithi089.stdout:20: oid 20 updating version 0 to 2 2024-04-22T18:38:54.919 INFO:tasks.rados.rados.0.smithi089.stdout:20: oid 20 version 2 is already newer than 1 2024-04-22T18:38:54.919 INFO:tasks.rados.rados.0.smithi089.stdout:20: oid 20 updating version 2 to 3 2024-04-22T18:38:54.919 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-22T18:38:54.920 INFO:tasks.rados.rados.0.smithi089.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-04-22T18:38:54.920 INFO:tasks.rados.rados.0.smithi089.stdout:20: done (15 left) 2024-04-22T18:38:54.920 INFO:tasks.rados.rados.0.smithi089.stdout:37: write initial oid 37 2024-04-22T18:38:54.920 INFO:tasks.rados.rados.0.smithi089.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-04-22T18:38:54.926 INFO:tasks.rados.rados.0.smithi089.stdout:37: writing smithi08937929-37 from 605052 to 1379428 tid 1 2024-04-22T18:38:54.932 INFO:tasks.rados.rados.0.smithi089.stdout:37: writing smithi08937929-37 from 1923587 to 2556923 tid 2 2024-04-22T18:38:54.932 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:54.932 INFO:tasks.rados.rados.0.smithi089.stdout:37: finishing write tid 1 to smithi08937929-37 2024-04-22T18:38:54.936 INFO:tasks.rados.rados.0.smithi089.stdout:37: finishing write tid 2 to smithi08937929-37 2024-04-22T18:38:54.951 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:38:54.957 INFO:tasks.rados.rados.0.smithi089.stdout:17: finishing write tid 2 to smithi08937929-17 2024-04-22T18:38:54.987 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:54.989+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:55.009 INFO:tasks.rados.rados.0.smithi089.stdout:9: finishing write tid 3 to smithi08937929-9 2024-04-22T18:38:55.010 INFO:tasks.rados.rados.0.smithi089.stdout:9: finishing write tid 4 to smithi08937929-9 2024-04-22T18:38:55.010 INFO:tasks.rados.rados.0.smithi089.stdout:9: oid 9 updating version 0 to 8 2024-04-22T18:38:55.010 INFO:tasks.rados.rados.0.smithi089.stdout:9: oid 9 version 8 is already newer than 7 2024-04-22T18:38:55.010 INFO:tasks.rados.rados.0.smithi089.stdout:9: oid 9 updating version 8 to 9 2024-04-22T18:38:55.010 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 9 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-22T18:38:55.010 INFO:tasks.rados.rados.0.smithi089.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-04-22T18:38:55.010 INFO:tasks.rados.rados.0.smithi089.stdout:9: done (15 left) 2024-04-22T18:38:55.010 INFO:tasks.rados.rados.0.smithi089.stdout:38: write initial oid 38 2024-04-22T18:38:55.010 INFO:tasks.rados.rados.0.smithi089.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-04-22T18:38:55.015 INFO:tasks.rados.rados.0.smithi089.stdout:38: writing smithi08937929-38 from 621859 to 1071484 tid 1 2024-04-22T18:38:55.024 INFO:tasks.rados.rados.0.smithi089.stdout:38: writing smithi08937929-38 from 1865716 to 2621859 tid 2 2024-04-22T18:38:55.024 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.024 INFO:tasks.rados.rados.0.smithi089.stdout:36: finishing write tid 1 to smithi08937929-36 2024-04-22T18:38:55.025 INFO:tasks.rados.rados.0.smithi089.stdout:36: finishing write tid 2 to smithi08937929-36 2024-04-22T18:38:55.025 INFO:tasks.rados.rados.0.smithi089.stdout:36: finishing write tid 3 to smithi08937929-36 2024-04-22T18:38:55.025 INFO:tasks.rados.rados.0.smithi089.stdout:36: finishing write tid 4 to smithi08937929-36 2024-04-22T18:38:55.025 INFO:tasks.rados.rados.0.smithi089.stdout:36: oid 36 updating version 0 to 8 2024-04-22T18:38:55.025 INFO:tasks.rados.rados.0.smithi089.stdout:36: oid 36 updating version 8 to 9 2024-04-22T18:38:55.025 INFO:tasks.rados.rados.0.smithi089.stdout:36: oid 36 version 9 is already newer than 7 2024-04-22T18:38:55.025 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-22T18:38:55.025 INFO:tasks.rados.rados.0.smithi089.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-04-22T18:38:55.025 INFO:tasks.rados.rados.0.smithi089.stdout:36: done (15 left) 2024-04-22T18:38:55.025 INFO:tasks.rados.rados.0.smithi089.stdout:39: write initial oid 39 2024-04-22T18:38:55.025 INFO:tasks.rados.rados.0.smithi089.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-04-22T18:38:55.030 INFO:tasks.rados.rados.0.smithi089.stdout:39: writing smithi08937929-39 from 638666 to 1163540 tid 1 2024-04-22T18:38:55.038 INFO:tasks.rados.rados.0.smithi089.stdout:39: writing smithi08937929-39 from 1924198 to 2561203 tid 2 2024-04-22T18:38:55.038 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.038 INFO:tasks.rados.rados.0.smithi089.stdout:37: finishing write tid 3 to smithi08937929-37 2024-04-22T18:38:55.038 INFO:tasks.rados.rados.0.smithi089.stdout:37: finishing write tid 4 to smithi08937929-37 2024-04-22T18:38:55.038 INFO:tasks.rados.rados.0.smithi089.stdout:37: oid 37 updating version 0 to 9 2024-04-22T18:38:55.038 INFO:tasks.rados.rados.0.smithi089.stdout:37: oid 37 version 9 is already newer than 8 2024-04-22T18:38:55.039 INFO:tasks.rados.rados.0.smithi089.stdout:37: oid 37 version 9 is already newer than 7 2024-04-22T18:38:55.039 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 9 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-22T18:38:55.039 INFO:tasks.rados.rados.0.smithi089.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-04-22T18:38:55.039 INFO:tasks.rados.rados.0.smithi089.stdout:37: done (15 left) 2024-04-22T18:38:55.039 INFO:tasks.rados.rados.0.smithi089.stdout:40: write initial oid 40 2024-04-22T18:38:55.039 INFO:tasks.rados.rados.0.smithi089.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-04-22T18:38:55.045 INFO:tasks.rados.rados.0.smithi089.stdout:40: writing smithi08937929-40 from 655473 to 1371949 tid 1 2024-04-22T18:38:55.047 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:55.047 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:38:55.047 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:38:55.048 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:38:55.048 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:55.051 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:55.051+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:55.053 INFO:tasks.rados.rados.0.smithi089.stdout:40: writing smithi08937929-40 from 2099033 to 2655473 tid 2 2024-04-22T18:38:55.053 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.056 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:38:55.066 INFO:tasks.rados.rados.0.smithi089.stdout:26: finishing write tid 1 to smithi08937929-26 2024-04-22T18:38:55.067 INFO:tasks.rados.rados.0.smithi089.stdout:26: finishing write tid 2 to smithi08937929-26 2024-04-22T18:38:55.067 INFO:tasks.rados.rados.0.smithi089.stdout:26: finishing write tid 3 to smithi08937929-26 2024-04-22T18:38:55.071 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:38:55.071 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:38:55.088 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:38:55.088 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:38:55.101 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:38:55.107 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:38:55.107 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:38:55.123 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:38:55.123 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:38:55.151 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:55.152+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:38:55.160 INFO:tasks.rados.rados.0.smithi089.stdout:17: finishing write tid 3 to smithi08937929-17 2024-04-22T18:38:55.161 INFO:tasks.rados.rados.0.smithi089.stdout:31: finishing write tid 1 to smithi08937929-31 2024-04-22T18:38:55.161 INFO:tasks.rados.rados.0.smithi089.stdout:31: finishing write tid 2 to smithi08937929-31 2024-04-22T18:38:55.161 INFO:tasks.rados.rados.0.smithi089.stdout:17: finishing write tid 4 to smithi08937929-17 2024-04-22T18:38:55.161 INFO:tasks.rados.rados.0.smithi089.stdout:17: oid 17 updating version 0 to 6 2024-04-22T18:38:55.161 INFO:tasks.rados.rados.0.smithi089.stdout:17: oid 17 version 6 is already newer than 4 2024-04-22T18:38:55.161 INFO:tasks.rados.rados.0.smithi089.stdout:17: oid 17 version 6 is already newer than 5 2024-04-22T18:38:55.161 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-22T18:38:55.161 INFO:tasks.rados.rados.0.smithi089.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-04-22T18:38:55.161 INFO:tasks.rados.rados.0.smithi089.stdout:17: done (15 left) 2024-04-22T18:38:55.161 INFO:tasks.rados.rados.0.smithi089.stdout:41: write initial oid 41 2024-04-22T18:38:55.161 INFO:tasks.rados.rados.0.smithi089.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-04-22T18:38:55.168 INFO:tasks.rados.rados.0.smithi089.stdout:41: writing smithi08937929-41 from 672280 to 1464005 tid 1 2024-04-22T18:38:55.175 INFO:tasks.rados.rados.0.smithi089.stdout:41: writing smithi08937929-41 from 2157515 to 2672280 tid 2 2024-04-22T18:38:55.175 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.175 INFO:tasks.rados.rados.0.smithi089.stdout:18: finishing write tid 1 to smithi08937929-18 2024-04-22T18:38:55.176 INFO:tasks.rados.rados.0.smithi089.stdout:31: finishing write tid 3 to smithi08937929-31 2024-04-22T18:38:55.176 INFO:tasks.rados.rados.0.smithi089.stdout:31: finishing write tid 4 to smithi08937929-31 2024-04-22T18:38:55.176 INFO:tasks.rados.rados.0.smithi089.stdout:31: oid 31 updating version 0 to 1 2024-04-22T18:38:55.176 INFO:tasks.rados.rados.0.smithi089.stdout:31: oid 31 updating version 1 to 2 2024-04-22T18:38:55.176 INFO:tasks.rados.rados.0.smithi089.stdout:31: oid 31 updating version 2 to 3 2024-04-22T18:38:55.176 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 3 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-22T18:38:55.176 INFO:tasks.rados.rados.0.smithi089.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-04-22T18:38:55.176 INFO:tasks.rados.rados.0.smithi089.stdout:31: done (15 left) 2024-04-22T18:38:55.176 INFO:tasks.rados.rados.0.smithi089.stdout:42: write initial oid 42 2024-04-22T18:38:55.176 INFO:tasks.rados.rados.0.smithi089.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-04-22T18:38:55.181 INFO:tasks.rados.rados.0.smithi089.stdout:42: writing smithi08937929-42 from 689087 to 1156061 tid 1 2024-04-22T18:38:55.189 INFO:tasks.rados.rados.0.smithi089.stdout:42: writing smithi08937929-42 from 1699644 to 2483976 tid 2 2024-04-22T18:38:55.189 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.189 INFO:tasks.rados.rados.0.smithi089.stdout:33: finishing write tid 1 to smithi08937929-33 2024-04-22T18:38:55.189 INFO:tasks.rados.rados.0.smithi089.stdout:33: finishing write tid 2 to smithi08937929-33 2024-04-22T18:38:55.189 INFO:tasks.rados.rados.0.smithi089.stdout:26: finishing write tid 4 to smithi08937929-26 2024-04-22T18:38:55.189 INFO:tasks.rados.rados.0.smithi089.stdout:26: oid 26 updating version 0 to 5 2024-04-22T18:38:55.189 INFO:tasks.rados.rados.0.smithi089.stdout:26: oid 26 updating version 5 to 6 2024-04-22T18:38:55.190 INFO:tasks.rados.rados.0.smithi089.stdout:26: oid 26 version 6 is already newer than 4 2024-04-22T18:38:55.190 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-22T18:38:55.190 INFO:tasks.rados.rados.0.smithi089.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-04-22T18:38:55.190 INFO:tasks.rados.rados.0.smithi089.stdout:27: finishing write tid 1 to smithi08937929-27 2024-04-22T18:38:55.190 INFO:tasks.rados.rados.0.smithi089.stdout:27: finishing write tid 2 to smithi08937929-27 2024-04-22T18:38:55.190 INFO:tasks.rados.rados.0.smithi089.stdout:26: done (15 left) 2024-04-22T18:38:55.190 INFO:tasks.rados.rados.0.smithi089.stdout:43: write initial oid 43 2024-04-22T18:38:55.190 INFO:tasks.rados.rados.0.smithi089.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-04-22T18:38:55.195 INFO:tasks.rados.rados.0.smithi089.stdout:43: writing smithi08937929-43 from 705894 to 1248117 tid 1 2024-04-22T18:38:55.202 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:55.202 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:38:55.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:38:55.202 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:38:55.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.202 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:21 4.a 4:5aa97f94:::smithi08937929-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:55.202 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:38:55.202 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.049152782, 2024-04-22T18:38:55.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21044934000000001, 2024-04-22T18:38:55.203 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.203 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:55.203 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:55.203 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:55.203 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:55.203 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 21 2024-04-22T18:38:55.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.203 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:38:55.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:38:55.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153183+0000", 2024-04-22T18:38:55.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:38:55.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154022+0000", 2024-04-22T18:38:55.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00083893300000000004 2024-04-22T18:38:55.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154032+0000", 2024-04-22T18:38:55.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0433e-05 2024-04-22T18:38:55.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154051+0000", 2024-04-22T18:38:55.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8227e-05 2024-04-22T18:38:55.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154103+0000", 2024-04-22T18:38:55.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2491999999999998e-05 2024-04-22T18:38:55.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:38:55.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154111+0000", 2024-04-22T18:38:55.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3470000000000001e-06 2024-04-22T18:38:55.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257848+0000", 2024-04-22T18:38:55.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103737703 2024-04-22T18:38:55.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257981+0000", 2024-04-22T18:38:55.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013290399999999999 2024-04-22T18:38:55.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:55.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.258138+0000", 2024-04-22T18:38:55.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015701199999999999 2024-04-22T18:38:55.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:55.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.261944+0000", 2024-04-22T18:38:55.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038054650000000001 2024-04-22T18:38:55.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:55.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363523+0000", 2024-04-22T18:38:55.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101579239 2024-04-22T18:38:55.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:55.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363577+0000", 2024-04-22T18:38:55.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4095000000000003e-05 2024-04-22T18:38:55.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363635+0000", 2024-04-22T18:38:55.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8362999999999997e-05 2024-04-22T18:38:55.209 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.209 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.209 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.209 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:22 4.a 4:5aa97f94:::smithi08937929-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:55.209 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:38:55.209 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.045078806, 2024-04-22T18:38:55.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30948398900000001, 2024-04-22T18:38:55.210 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.210 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:55.210 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:55.210 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:55.210 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:55.210 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 22 2024-04-22T18:38:55.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.210 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:38:55.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:38:55.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157259+0000", 2024-04-22T18:38:55.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:55.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157318+0000", 2024-04-22T18:38:55.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8906000000000001e-05 2024-04-22T18:38:55.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157327+0000", 2024-04-22T18:38:55.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:38:55.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157332+0000", 2024-04-22T18:38:55.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4689999999999996e-06 2024-04-22T18:38:55.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257479+0000", 2024-04-22T18:38:55.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014700999999999 2024-04-22T18:38:55.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:38:55.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257492+0000", 2024-04-22T18:38:55.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2574e-05 2024-04-22T18:38:55.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259582+0000", 2024-04-22T18:38:55.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020904019999999999 2024-04-22T18:38:55.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259668+0000", 2024-04-22T18:38:55.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5826000000000001e-05 2024-04-22T18:38:55.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:55.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259772+0000", 2024-04-22T18:38:55.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104048 2024-04-22T18:38:55.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:55.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363453+0000", 2024-04-22T18:38:55.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103680183 2024-04-22T18:38:55.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:55.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466692+0000", 2024-04-22T18:38:55.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10323948199999999 2024-04-22T18:38:55.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:55.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466720+0000", 2024-04-22T18:38:55.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7886e-05 2024-04-22T18:38:55.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466744+0000", 2024-04-22T18:38:55.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4049e-05 2024-04-22T18:38:55.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.215 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.216 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:23 4.a 4:5aa97f94:::smithi08937929-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:55.216 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:38:55.216 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.0450011189999999, 2024-04-22T18:38:55.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39003820300000003, 2024-04-22T18:38:55.216 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.216 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:55.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:55.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:55.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:55.216 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 23 2024-04-22T18:38:55.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.216 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:38:55.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:38:55.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157337+0000", 2024-04-22T18:38:55.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:55.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157340+0000", 2024-04-22T18:38:55.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8609999999999998e-06 2024-04-22T18:38:55.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157343+0000", 2024-04-22T18:38:55.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9859999999999999e-06 2024-04-22T18:38:55.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157346+0000", 2024-04-22T18:38:55.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8119999999999999e-06 2024-04-22T18:38:55.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257563+0000", 2024-04-22T18:38:55.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100217497 2024-04-22T18:38:55.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:38:55.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257574+0000", 2024-04-22T18:38:55.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0777e-05 2024-04-22T18:38:55.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260112+0000", 2024-04-22T18:38:55.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002537681 2024-04-22T18:38:55.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260191+0000", 2024-04-22T18:38:55.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9320999999999995e-05 2024-04-22T18:38:55.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:55.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260333+0000", 2024-04-22T18:38:55.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014177699999999999 2024-04-22T18:38:55.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:55.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363792+0000", 2024-04-22T18:38:55.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103458862 2024-04-22T18:38:55.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:55.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547303+0000", 2024-04-22T18:38:55.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18351162100000001 2024-04-22T18:38:55.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:55.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547337+0000", 2024-04-22T18:38:55.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3640999999999999e-05 2024-04-22T18:38:55.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547376+0000", 2024-04-22T18:38:55.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8915000000000003e-05 2024-04-22T18:38:55.222 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.222 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.222 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.223 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:24 4.a 4:5aa97f94:::smithi08937929-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:55.223 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:38:55.223 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.0449746879999999, 2024-04-22T18:38:55.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39017823499999998, 2024-04-22T18:38:55.223 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.223 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:55.223 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:55.223 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:55.223 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:55.223 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 24 2024-04-22T18:38:55.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.224 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:38:55.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:38:55.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:38:55.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:55.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157366+0000", 2024-04-22T18:38:55.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3120000000000001e-06 2024-04-22T18:38:55.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157370+0000", 2024-04-22T18:38:55.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1300000000000003e-06 2024-04-22T18:38:55.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157373+0000", 2024-04-22T18:38:55.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-04-22T18:38:55.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260553+0000", 2024-04-22T18:38:55.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103179996 2024-04-22T18:38:55.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:38:55.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260625+0000", 2024-04-22T18:38:55.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2236000000000004e-05 2024-04-22T18:38:55.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547414+0000", 2024-04-22T18:38:55.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.286789181 2024-04-22T18:38:55.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547471+0000", 2024-04-22T18:38:55.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6192e-05 2024-04-22T18:38:55.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547542+0000", 2024-04-22T18:38:55.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1622e-05 2024-04-22T18:38:55.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.228 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.228 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:49 4.a 4:5d2aee44:::smithi08937929-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:55.228 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:38:55.229 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.96817602999999997, 2024-04-22T18:38:55.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31344442500000003, 2024-04-22T18:38:55.229 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.229 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:55.229 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:55.229 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:55.229 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:55.229 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 49 2024-04-22T18:38:55.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.229 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:38:55.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:38:55.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234158+0000", 2024-04-22T18:38:55.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999957 2024-04-22T18:38:55.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234901+0000", 2024-04-22T18:38:55.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074250800000000001 2024-04-22T18:38:55.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234914+0000", 2024-04-22T18:38:55.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3244999999999999e-05 2024-04-22T18:38:55.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234922+0000", 2024-04-22T18:38:55.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4590000000000001e-06 2024-04-22T18:38:55.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260665+0000", 2024-04-22T18:38:55.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025743423000000001 2024-04-22T18:38:55.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260779+0000", 2024-04-22T18:38:55.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114063 2024-04-22T18:38:55.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:55.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260913+0000", 2024-04-22T18:38:55.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013355499999999999 2024-04-22T18:38:55.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:55.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363798+0000", 2024-04-22T18:38:55.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102885003 2024-04-22T18:38:55.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:55.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547567+0000", 2024-04-22T18:38:55.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.183768971 2024-04-22T18:38:55.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:55.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547589+0000", 2024-04-22T18:38:55.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2507999999999999e-05 2024-04-22T18:38:55.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547607+0000", 2024-04-22T18:38:55.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8009999999999999e-05 2024-04-22T18:38:55.234 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.234 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.234 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.235 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:50 4.a 4:5d2aee44:::smithi08937929-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:55.235 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:38:55.235 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.96272985099999997, 2024-04-22T18:38:55.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40856295399999998, 2024-04-22T18:38:55.235 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.235 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:55.235 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:55.235 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:55.235 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:55.235 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 50 2024-04-22T18:38:55.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.235 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:38:55.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:38:55.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239607+0000", 2024-04-22T18:38:55.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:55.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240309+0000", 2024-04-22T18:38:55.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070201599999999997 2024-04-22T18:38:55.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240320+0000", 2024-04-22T18:38:55.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1024e-05 2024-04-22T18:38:55.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240327+0000", 2024-04-22T18:38:55.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0049999999999998e-06 2024-04-22T18:38:55.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362065+0000", 2024-04-22T18:38:55.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.121737812 2024-04-22T18:38:55.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362121+0000", 2024-04-22T18:38:55.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5868000000000001e-05 2024-04-22T18:38:55.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:55.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362210+0000", 2024-04-22T18:38:55.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8533000000000001e-05 2024-04-22T18:38:55.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:55.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365322+0000", 2024-04-22T18:38:55.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031121220000000002 2024-04-22T18:38:55.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:55.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648054+0000", 2024-04-22T18:38:55.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.282732549 2024-04-22T18:38:55.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648105+0000", 2024-04-22T18:38:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1276999999999997e-05 2024-04-22T18:38:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648172+0000", 2024-04-22T18:38:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6398999999999999e-05 2024-04-22T18:38:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:51 4.a 4:5d2aee44:::smithi08937929-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:38:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.96200731699999997, 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40793057199999999, 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 51 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:38:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:38:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240336+0000", 2024-04-22T18:38:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:38:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240339+0000", 2024-04-22T18:38:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9009999999999998e-06 2024-04-22T18:38:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240342+0000", 2024-04-22T18:38:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:38:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363080+0000", 2024-04-22T18:38:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122738075 2024-04-22T18:38:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363164+0000", 2024-04-22T18:38:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3722999999999998e-05 2024-04-22T18:38:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363287+0000", 2024-04-22T18:38:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012301099999999999 2024-04-22T18:38:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466753+0000", 2024-04-22T18:38:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103466217 2024-04-22T18:38:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648205+0000", 2024-04-22T18:38:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.181451788 2024-04-22T18:38:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648231+0000", 2024-04-22T18:38:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6597999999999998e-05 2024-04-22T18:38:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648262+0000", 2024-04-22T18:38:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0871000000000003e-05 2024-04-22T18:38:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:52 4.a 4:5d2aee44:::smithi08937929-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:38:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.96199485799999995, 2024-04-22T18:38:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40812731200000002, 2024-04-22T18:38:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 52 2024-04-22T18:38:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:38:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:38:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:38:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240345+0000", 2024-04-22T18:38:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4300000000000001e-06 2024-04-22T18:38:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240349+0000", 2024-04-22T18:38:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2570000000000001e-06 2024-04-22T18:38:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240358+0000", 2024-04-22T18:38:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-22T18:38:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363697+0000", 2024-04-22T18:38:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12333965299999999 2024-04-22T18:38:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:38:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363777+0000", 2024-04-22T18:38:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9883999999999996e-05 2024-04-22T18:38:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648296+0000", 2024-04-22T18:38:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451861299999998 2024-04-22T18:38:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648379+0000", 2024-04-22T18:38:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3302999999999998e-05 2024-04-22T18:38:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648471+0000", 2024-04-22T18:38:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2029999999999998e-05 2024-04-22T18:38:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:38:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.94269152300000003, 2024-04-22T18:38:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:38:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:38:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:38:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:38:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:38:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:38:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:38:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:38:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:38:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:38:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:38:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:38:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:38:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:38:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:38:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:38:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:38:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:38:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:38:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:38:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:38:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:38:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:38:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:38:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:38:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:38:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:29 4.e e27/20 4:744777ef:::smithi08937929-8:head v 27'1, mlcod=0'0)", 2024-04-22T18:38:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264023+0000", 2024-04-22T18:38:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.93831529599999997, 2024-04-22T18:38:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10131754, 2024-04-22T18:38:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264023+0000", 2024-04-22T18:38:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264023+0000", 2024-04-22T18:38:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264023+0000", 2024-04-22T18:38:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264383+0000", 2024-04-22T18:38:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000360016 2024-04-22T18:38:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264394+0000", 2024-04-22T18:38:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0786e-05 2024-04-22T18:38:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264401+0000", 2024-04-22T18:38:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8120000000000002e-06 2024-04-22T18:38:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265687+0000", 2024-04-22T18:38:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012858660000000001 2024-04-22T18:38:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265712+0000", 2024-04-22T18:38:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5375e-05 2024-04-22T18:38:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365303+0000", 2024-04-22T18:38:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099591381000000007 2024-04-22T18:38:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365341+0000", 2024-04-22T18:38:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7703000000000001e-05 2024-04-22T18:38:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:38:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.93790909300000003, 2024-04-22T18:38:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:38:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:38:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:38:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:38:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:38:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:38:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:38:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:38:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:38:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:38:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:38:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:38:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:38:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:38:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:38:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:38:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:38:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:38:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:38:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:38:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:38:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.93781488999999996, 2024-04-22T18:38:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:38:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:38:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:38:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:38:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:38:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:38:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:38:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:38:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:38:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:38:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:38:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:38:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:38:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:38:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:38:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:38:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:38:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:38:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:38:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:38:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:38:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:38:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:38:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:38:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:38:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:38:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.93779400199999996, 2024-04-22T18:38:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:38:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:38:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:38:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:38:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:38:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:38:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:38:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:38:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:38:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:38:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:38:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:38:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:38:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:38:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:38:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:38:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:38:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:38:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:38:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:38:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:38:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:38:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:30 4.e e27/20 4:744777ef:::smithi08937929-8:head v 27'2, mlcod=0'0)", 2024-04-22T18:38:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.265427+0000", 2024-04-22T18:38:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.93691139300000004, 2024-04-22T18:38:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102124983, 2024-04-22T18:38:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265427+0000", 2024-04-22T18:38:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265427+0000", 2024-04-22T18:38:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265426+0000", 2024-04-22T18:38:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265617+0000", 2024-04-22T18:38:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019075900000000001 2024-04-22T18:38:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265628+0000", 2024-04-22T18:38:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1336e-05 2024-04-22T18:38:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265636+0000", 2024-04-22T18:38:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0940000000000008e-06 2024-04-22T18:38:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364506+0000", 2024-04-22T18:38:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098869159999999998 2024-04-22T18:38:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364527+0000", 2024-04-22T18:38:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1107999999999998e-05 2024-04-22T18:38:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367522+0000", 2024-04-22T18:38:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029958680000000001 2024-04-22T18:38:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367552+0000", 2024-04-22T18:38:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9858000000000001e-05 2024-04-22T18:38:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.4502.0:22 4.a e27/20 ondisk, result = 0)", 2024-04-22T18:38:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.346987+0000", 2024-04-22T18:38:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.85535200700000003, 2024-04-22T18:38:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.119862203, 2024-04-22T18:38:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.346987+0000", 2024-04-22T18:38:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.346987+0000", 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.346985+0000", 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.346991+0000", 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1329999999999998e-06 2024-04-22T18:38:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.347002+0000", 2024-04-22T18:38:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1404e-05 2024-04-22T18:38:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.447116+0000", 2024-04-22T18:38:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10011421199999999 2024-04-22T18:38:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466683+0000", 2024-04-22T18:38:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019566887000000002 2024-04-22T18:38:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466688+0000", 2024-04-22T18:38:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3249999999999998e-06 2024-04-22T18:38:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466849+0000", 2024-04-22T18:38:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000160483 2024-04-22T18:38:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.4502.0:23 4.a e27/20 ondisk, result = 0)", 2024-04-22T18:38:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.447125+0000", 2024-04-22T18:38:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.75521415000000003, 2024-04-22T18:38:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100259183, 2024-04-22T18:38:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.447125+0000", 2024-04-22T18:38:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.447125+0000", 2024-04-22T18:38:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.447124+0000", 2024-04-22T18:38:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.447130+0000", 2024-04-22T18:38:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3999999999999997e-06 2024-04-22T18:38:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.447136+0000", 2024-04-22T18:38:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0190000000000003e-06 2024-04-22T18:38:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547220+0000", 2024-04-22T18:38:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100083939 2024-04-22T18:38:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547291+0000", 2024-04-22T18:38:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1253000000000004e-05 2024-04-22T18:38:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547299+0000", 2024-04-22T18:38:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5700000000000004e-06 2024-04-22T18:38:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547384+0000", 2024-04-22T18:38:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5074999999999994e-05 2024-04-22T18:38:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:31 4.e e27/20 4:744777ef:::smithi08937929-8:head v 27'3, mlcod=0'0)", 2024-04-22T18:38:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.547737+0000", 2024-04-22T18:38:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.65460153799999998, 2024-04-22T18:38:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100796317, 2024-04-22T18:38:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547737+0000", 2024-04-22T18:38:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547737+0000", 2024-04-22T18:38:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547736+0000", 2024-04-22T18:38:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547745+0000", 2024-04-22T18:38:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0350000000000007e-06 2024-04-22T18:38:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547756+0000", 2024-04-22T18:38:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1065e-05 2024-04-22T18:38:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.647847+0000", 2024-04-22T18:38:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100091241 2024-04-22T18:38:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.647923+0000", 2024-04-22T18:38:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5968000000000002e-05 2024-04-22T18:38:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.647954+0000", 2024-04-22T18:38:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0491e-05 2024-04-22T18:38:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648503+0000", 2024-04-22T18:38:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054940299999999996 2024-04-22T18:38:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648534+0000", 2024-04-22T18:38:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0307e-05 2024-04-22T18:38:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.4502.0:57 4.a e27/20 ondisk, result = 0)", 2024-04-22T18:38:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.648032+0000", 2024-04-22T18:38:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.55430642399999996, 2024-04-22T18:38:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100455379, 2024-04-22T18:38:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648032+0000", 2024-04-22T18:38:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648032+0000", 2024-04-22T18:38:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648032+0000", 2024-04-22T18:38:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648034+0000", 2024-04-22T18:38:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.345e-06 2024-04-22T18:38:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648038+0000", 2024-04-22T18:38:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1400000000000002e-06 2024-04-22T18:38:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748164+0000", 2024-04-22T18:38:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10012520699999999 2024-04-22T18:38:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748263+0000", 2024-04-22T18:38:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9358000000000007e-05 2024-04-22T18:38:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748285+0000", 2024-04-22T18:38:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1713e-05 2024-04-22T18:38:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748488+0000", 2024-04-22T18:38:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00020302800000000001 2024-04-22T18:38:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.4502.0:58 4.a e27/20 ondisk, result = 0)", 2024-04-22T18:38:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.748185+0000", 2024-04-22T18:38:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.45415333800000002, 2024-04-22T18:38:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10056585899999999, 2024-04-22T18:38:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748185+0000", 2024-04-22T18:38:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748185+0000", 2024-04-22T18:38:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748183+0000", 2024-04-22T18:38:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:38:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748201+0000", 2024-04-22T18:38:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7708999999999999e-05 2024-04-22T18:38:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748223+0000", 2024-04-22T18:38:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.266e-05 2024-04-22T18:38:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748237+0000", 2024-04-22T18:38:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3777e-05 2024-04-22T18:38:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848640+0000", 2024-04-22T18:38:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100403407 2024-04-22T18:38:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848649+0000", 2024-04-22T18:38:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2649999999999996e-06 2024-04-22T18:38:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848751+0000", 2024-04-22T18:38:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102644 2024-04-22T18:38:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.4502.0:59 4.a e27/20 ondisk, result = 0)", 2024-04-22T18:38:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.748247+0000", 2024-04-22T18:38:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.45409202799999998, 2024-04-22T18:38:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10063032600000001, 2024-04-22T18:38:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748247+0000", 2024-04-22T18:38:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748247+0000", 2024-04-22T18:38:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748246+0000", 2024-04-22T18:38:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748253+0000", 2024-04-22T18:38:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3289999999999996e-06 2024-04-22T18:38:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748266+0000", 2024-04-22T18:38:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2653000000000001e-05 2024-04-22T18:38:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748277+0000", 2024-04-22T18:38:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1717e-05 2024-04-22T18:38:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848801+0000", 2024-04-22T18:38:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10052420400000001 2024-04-22T18:38:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848807+0000", 2024-04-22T18:38:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1699999999999996e-06 2024-04-22T18:38:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848877+0000", 2024-04-22T18:38:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0443000000000003e-05 2024-04-22T18:38:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:55.308 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:55.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:38:55.312 INFO:tasks.rados.rados.0.smithi089.stdout:43: writing smithi08937929-43 from 1758126 to 2402469 tid 2 2024-04-22T18:38:55.312 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.312 INFO:tasks.rados.rados.0.smithi089.stdout:42: finishing write tid 1 to smithi08937929-42 2024-04-22T18:38:55.312 INFO:tasks.rados.rados.0.smithi089.stdout:18: finishing write tid 2 to smithi08937929-18 2024-04-22T18:38:55.312 INFO:tasks.rados.rados.0.smithi089.stdout:18: finishing write tid 3 to smithi08937929-18 2024-04-22T18:38:55.312 INFO:tasks.rados.rados.0.smithi089.stdout:18: finishing write tid 4 to smithi08937929-18 2024-04-22T18:38:55.312 INFO:tasks.rados.rados.0.smithi089.stdout:18: oid 18 updating version 0 to 8 2024-04-22T18:38:55.313 INFO:tasks.rados.rados.0.smithi089.stdout:18: oid 18 updating version 8 to 9 2024-04-22T18:38:55.313 INFO:tasks.rados.rados.0.smithi089.stdout:18: oid 18 version 9 is already newer than 7 2024-04-22T18:38:55.313 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 9 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-22T18:38:55.313 INFO:tasks.rados.rados.0.smithi089.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-04-22T18:38:55.313 INFO:tasks.rados.rados.0.smithi089.stdout:18: done (15 left) 2024-04-22T18:38:55.313 INFO:tasks.rados.rados.0.smithi089.stdout:44: write initial oid 44 2024-04-22T18:38:55.313 INFO:tasks.rados.rados.0.smithi089.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-04-22T18:38:55.313 INFO:tasks.rados.rados.0.smithi089.stdout:44: writing smithi08937929-44 from 722701 to 1340173 tid 1 2024-04-22T18:38:55.313 INFO:tasks.rados.rados.0.smithi089.stdout:44: writing smithi08937929-44 from 1816608 to 2604609 tid 2 2024-04-22T18:38:55.313 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.313 INFO:tasks.rados.rados.0.smithi089.stdout:25: finishing write tid 1 to smithi08937929-25 2024-04-22T18:38:55.313 INFO:tasks.rados.rados.0.smithi089.stdout:25: finishing write tid 2 to smithi08937929-25 2024-04-22T18:38:55.313 INFO:tasks.rados.rados.0.smithi089.stdout:33: finishing write tid 3 to smithi08937929-33 2024-04-22T18:38:55.314 INFO:tasks.rados.rados.0.smithi089.stdout:33: finishing write tid 4 to smithi08937929-33 2024-04-22T18:38:55.314 INFO:tasks.rados.rados.0.smithi089.stdout:33: oid 33 updating version 0 to 5 2024-04-22T18:38:55.314 INFO:tasks.rados.rados.0.smithi089.stdout:33: oid 33 version 5 is already newer than 4 2024-04-22T18:38:55.314 INFO:tasks.rados.rados.0.smithi089.stdout:33: oid 33 updating version 5 to 6 2024-04-22T18:38:55.314 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-22T18:38:55.314 INFO:tasks.rados.rados.0.smithi089.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-04-22T18:38:55.314 INFO:tasks.rados.rados.0.smithi089.stdout:33: done (15 left) 2024-04-22T18:38:55.314 INFO:tasks.rados.rados.0.smithi089.stdout:45: write initial oid 45 2024-04-22T18:38:55.314 INFO:tasks.rados.rados.0.smithi089.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-04-22T18:38:55.314 INFO:tasks.rados.rados.0.smithi089.stdout:45: writing smithi08937929-45 from 739508 to 1432229 tid 1 2024-04-22T18:38:55.314 INFO:tasks.rados.rados.0.smithi089.stdout:45: writing smithi08937929-45 from 1875090 to 2523102 tid 2 2024-04-22T18:38:55.314 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.314 INFO:tasks.rados.rados.0.smithi089.stdout:42: finishing write tid 2 to smithi08937929-42 2024-04-22T18:38:55.314 INFO:tasks.rados.rados.0.smithi089.stdout:45: finishing write tid 1 to smithi08937929-45 2024-04-22T18:38:55.315 INFO:tasks.rados.rados.0.smithi089.stdout:45: finishing write tid 2 to smithi08937929-45 2024-04-22T18:38:55.315 INFO:tasks.rados.rados.0.smithi089.stdout:45: finishing write tid 3 to smithi08937929-45 2024-04-22T18:38:55.315 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:55.309+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:38:55.372 INFO:tasks.rados.rados.0.smithi089.stdout:27: finishing write tid 3 to smithi08937929-27 2024-04-22T18:38:55.372 INFO:tasks.rados.rados.0.smithi089.stdout:27: finishing write tid 4 to smithi08937929-27 2024-04-22T18:38:55.372 INFO:tasks.rados.rados.0.smithi089.stdout:27: oid 27 updating version 0 to 9 2024-04-22T18:38:55.372 INFO:tasks.rados.rados.0.smithi089.stdout:27: oid 27 version 9 is already newer than 8 2024-04-22T18:38:55.372 INFO:tasks.rados.rados.0.smithi089.stdout:27: oid 27 version 9 is already newer than 7 2024-04-22T18:38:55.372 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 9 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-22T18:38:55.372 INFO:tasks.rados.rados.0.smithi089.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-04-22T18:38:55.372 INFO:tasks.rados.rados.0.smithi089.stdout:23: finishing write tid 1 to smithi08937929-23 2024-04-22T18:38:55.372 INFO:tasks.rados.rados.0.smithi089.stdout:35: finishing write tid 1 to smithi08937929-35 2024-04-22T18:38:55.372 INFO:tasks.rados.rados.0.smithi089.stdout:35: finishing write tid 2 to smithi08937929-35 2024-04-22T18:38:55.372 INFO:tasks.rados.rados.0.smithi089.stdout:23: finishing write tid 2 to smithi08937929-23 2024-04-22T18:38:55.373 INFO:tasks.rados.rados.0.smithi089.stdout:23: finishing write tid 3 to smithi08937929-23 2024-04-22T18:38:55.373 INFO:tasks.rados.rados.0.smithi089.stdout:23: finishing write tid 4 to smithi08937929-23 2024-04-22T18:38:55.373 INFO:tasks.rados.rados.0.smithi089.stdout:23: oid 23 updating version 0 to 7 2024-04-22T18:38:55.373 INFO:tasks.rados.rados.0.smithi089.stdout:23: oid 23 updating version 7 to 8 2024-04-22T18:38:55.373 INFO:tasks.rados.rados.0.smithi089.stdout:23: oid 23 updating version 8 to 9 2024-04-22T18:38:55.373 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-22T18:38:55.373 INFO:tasks.rados.rados.0.smithi089.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-04-22T18:38:55.373 INFO:tasks.rados.rados.0.smithi089.stdout:24: finishing write tid 1 to smithi08937929-24 2024-04-22T18:38:55.373 INFO:tasks.rados.rados.0.smithi089.stdout:23: done (15 left) 2024-04-22T18:38:55.373 INFO:tasks.rados.rados.0.smithi089.stdout:27: done (14 left) 2024-04-22T18:38:55.373 INFO:tasks.rados.rados.0.smithi089.stdout:46: write initial oid 46 2024-04-22T18:38:55.373 INFO:tasks.rados.rados.0.smithi089.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-04-22T18:38:55.380 INFO:tasks.rados.rados.0.smithi089.stdout:46: writing smithi08937929-46 from 756315 to 1524285 tid 1 2024-04-22T18:38:55.386 INFO:tasks.rados.rados.0.smithi089.stdout:46: writing smithi08937929-46 from 1933572 to 2725242 tid 2 2024-04-22T18:38:55.386 INFO:tasks.rados.rados.0.smithi089.stdout:47: write initial oid 47 2024-04-22T18:38:55.386 INFO:tasks.rados.rados.0.smithi089.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-04-22T18:38:55.393 INFO:tasks.rados.rados.0.smithi089.stdout:47: writing smithi08937929-47 from 773122 to 1332694 tid 1 2024-04-22T18:38:55.401 INFO:tasks.rados.rados.0.smithi089.stdout:47: writing smithi08937929-47 from 1992054 to 2643735 tid 2 2024-04-22T18:38:55.401 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.401 INFO:tasks.rados.rados.0.smithi089.stdout:34: finishing write tid 1 to smithi08937929-34 2024-04-22T18:38:55.401 INFO:tasks.rados.rados.0.smithi089.stdout:24: finishing write tid 2 to smithi08937929-24 2024-04-22T18:38:55.401 INFO:tasks.rados.rados.0.smithi089.stdout:24: finishing write tid 3 to smithi08937929-24 2024-04-22T18:38:55.401 INFO:tasks.rados.rados.0.smithi089.stdout:24: finishing write tid 4 to smithi08937929-24 2024-04-22T18:38:55.402 INFO:tasks.rados.rados.0.smithi089.stdout:24: oid 24 updating version 0 to 10 2024-04-22T18:38:55.402 INFO:tasks.rados.rados.0.smithi089.stdout:24: oid 24 updating version 10 to 12 2024-04-22T18:38:55.402 INFO:tasks.rados.rados.0.smithi089.stdout:24: oid 24 version 12 is already newer than 11 2024-04-22T18:38:55.402 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 12 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-22T18:38:55.402 INFO:tasks.rados.rados.0.smithi089.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-04-22T18:38:55.402 INFO:tasks.rados.rados.0.smithi089.stdout:35: finishing write tid 3 to smithi08937929-35 2024-04-22T18:38:55.402 INFO:tasks.rados.rados.0.smithi089.stdout:35: finishing write tid 4 to smithi08937929-35 2024-04-22T18:38:55.402 INFO:tasks.rados.rados.0.smithi089.stdout:35: oid 35 updating version 0 to 9 2024-04-22T18:38:55.402 INFO:tasks.rados.rados.0.smithi089.stdout:35: oid 35 version 9 is already newer than 7 2024-04-22T18:38:55.402 INFO:tasks.rados.rados.0.smithi089.stdout:35: oid 35 version 9 is already newer than 8 2024-04-22T18:38:55.402 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 9 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-22T18:38:55.402 INFO:tasks.rados.rados.0.smithi089.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-04-22T18:38:55.402 INFO:tasks.rados.rados.0.smithi089.stdout:44: finishing write tid 1 to smithi08937929-44 2024-04-22T18:38:55.402 INFO:tasks.rados.rados.0.smithi089.stdout:44: finishing write tid 2 to smithi08937929-44 2024-04-22T18:38:55.403 INFO:tasks.rados.rados.0.smithi089.stdout:24: done (15 left) 2024-04-22T18:38:55.403 INFO:tasks.rados.rados.0.smithi089.stdout:35: done (14 left) 2024-04-22T18:38:55.403 INFO:tasks.rados.rados.0.smithi089.stdout:48: write initial oid 48 2024-04-22T18:38:55.403 INFO:tasks.rados.rados.0.smithi089.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-04-22T18:38:55.408 INFO:tasks.rados.rados.0.smithi089.stdout:48: writing smithi08937929-48 from 789929 to 1424750 tid 1 2024-04-22T18:38:55.410 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:55.412+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:55.415 INFO:tasks.rados.rados.0.smithi089.stdout:48: writing smithi08937929-48 from 2050536 to 2789929 tid 2 2024-04-22T18:38:55.416 INFO:tasks.rados.rados.0.smithi089.stdout:49: write initial oid 49 2024-04-22T18:38:55.416 INFO:tasks.rados.rados.0.smithi089.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-04-22T18:38:55.421 INFO:tasks.rados.rados.0.smithi089.stdout:49: writing smithi08937929-49 from 406736 to 1116806 tid 1 2024-04-22T18:38:55.422 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:55.422 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:38:55.423 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:38:55.423 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:55.429 INFO:tasks.rados.rados.0.smithi089.stdout:49: writing smithi08937929-49 from 1709018 to 2364368 tid 2 2024-04-22T18:38:55.429 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.429 INFO:tasks.rados.rados.0.smithi089.stdout:44: finishing write tid 3 to smithi08937929-44 2024-04-22T18:38:55.429 INFO:tasks.rados.rados.0.smithi089.stdout:42: finishing write tid 3 to smithi08937929-42 2024-04-22T18:38:55.429 INFO:tasks.rados.rados.0.smithi089.stdout:45: finishing write tid 4 to smithi08937929-45 2024-04-22T18:38:55.429 INFO:tasks.rados.rados.0.smithi089.stdout:45: oid 45 updating version 0 to 6 2024-04-22T18:38:55.429 INFO:tasks.rados.rados.0.smithi089.stdout:45: oid 45 version 6 is already newer than 4 2024-04-22T18:38:55.429 INFO:tasks.rados.rados.0.smithi089.stdout:45: oid 45 version 6 is already newer than 5 2024-04-22T18:38:55.429 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-22T18:38:55.429 INFO:tasks.rados.rados.0.smithi089.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-04-22T18:38:55.429 INFO:tasks.rados.rados.0.smithi089.stdout:45: done (15 left) 2024-04-22T18:38:55.429 INFO:tasks.rados.rados.0.smithi089.stdout:50: write initial oid 50 2024-04-22T18:38:55.430 INFO:tasks.rados.rados.0.smithi089.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-04-22T18:38:55.431 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:38:55.434 INFO:tasks.rados.rados.0.smithi089.stdout:50: writing smithi08937929-50 from 423543 to 1208862 tid 1 2024-04-22T18:38:55.441 INFO:tasks.rados.rados.0.smithi089.stdout:50: writing smithi08937929-50 from 1767500 to 2566508 tid 2 2024-04-22T18:38:55.441 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.489 INFO:tasks.rados.rados.0.smithi089.stdout:28: finishing write tid 1 to smithi08937929-28 2024-04-22T18:38:55.489 INFO:tasks.rados.rados.0.smithi089.stdout:28: finishing write tid 2 to smithi08937929-28 2024-04-22T18:38:55.489 INFO:tasks.rados.rados.0.smithi089.stdout:28: finishing write tid 3 to smithi08937929-28 2024-04-22T18:38:55.489 INFO:tasks.rados.rados.0.smithi089.stdout:34: finishing write tid 2 to smithi08937929-34 2024-04-22T18:38:55.489 INFO:tasks.rados.rados.0.smithi089.stdout:28: finishing write tid 4 to smithi08937929-28 2024-04-22T18:38:55.489 INFO:tasks.rados.rados.0.smithi089.stdout:28: oid 28 updating version 0 to 15 2024-04-22T18:38:55.489 INFO:tasks.rados.rados.0.smithi089.stdout:28: oid 28 version 15 is already newer than 14 2024-04-22T18:38:55.489 INFO:tasks.rados.rados.0.smithi089.stdout:28: oid 28 version 15 is already newer than 13 2024-04-22T18:38:55.490 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 15 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-22T18:38:55.490 INFO:tasks.rados.rados.0.smithi089.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-04-22T18:38:55.490 INFO:tasks.rados.rados.0.smithi089.stdout:34: finishing write tid 3 to smithi08937929-34 2024-04-22T18:38:55.490 INFO:tasks.rados.rados.0.smithi089.stdout:34: finishing write tid 4 to smithi08937929-34 2024-04-22T18:38:55.490 INFO:tasks.rados.rados.0.smithi089.stdout:34: oid 34 updating version 0 to 12 2024-04-22T18:38:55.490 INFO:tasks.rados.rados.0.smithi089.stdout:34: oid 34 version 12 is already newer than 10 2024-04-22T18:38:55.490 INFO:tasks.rados.rados.0.smithi089.stdout:34: oid 34 version 12 is already newer than 11 2024-04-22T18:38:55.490 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-22T18:38:55.490 INFO:tasks.rados.rados.0.smithi089.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-04-22T18:38:55.490 INFO:tasks.rados.rados.0.smithi089.stdout:30: finishing write tid 1 to smithi08937929-30 2024-04-22T18:38:55.490 INFO:tasks.rados.rados.0.smithi089.stdout:30: finishing write tid 2 to smithi08937929-30 2024-04-22T18:38:55.490 INFO:tasks.rados.rados.0.smithi089.stdout:28: done (15 left) 2024-04-22T18:38:55.490 INFO:tasks.rados.rados.0.smithi089.stdout:34: done (14 left) 2024-04-22T18:38:55.491 INFO:tasks.rados.rados.0.smithi089.stdout:51: copy oid 2 target oid 2 low_tier 2024-04-22T18:38:55.491 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 2 to 2 low_tier tgt_pool_name low_tier 2024-04-22T18:38:55.491 INFO:tasks.rados.rados.0.smithi089.stdout:52: copy oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.491 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 3 to 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:55.491 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.491 INFO:tasks.rados.rados.0.smithi089.stdout:38: finishing write tid 1 to smithi08937929-38 2024-04-22T18:38:55.491 INFO:tasks.rados.rados.0.smithi089.stdout:30: finishing write tid 3 to smithi08937929-30 2024-04-22T18:38:55.491 INFO:tasks.rados.rados.0.smithi089.stdout:30: finishing write tid 4 to smithi08937929-30 2024-04-22T18:38:55.491 INFO:tasks.rados.rados.0.smithi089.stdout:30: oid 30 updating version 0 to 4 2024-04-22T18:38:55.491 INFO:tasks.rados.rados.0.smithi089.stdout:30: oid 30 updating version 4 to 5 2024-04-22T18:38:55.491 INFO:tasks.rados.rados.0.smithi089.stdout:30: oid 30 updating version 5 to 6 2024-04-22T18:38:55.491 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-22T18:38:55.491 INFO:tasks.rados.rados.0.smithi089.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-04-22T18:38:55.492 INFO:tasks.rados.rados.0.smithi089.stdout:38: finishing write tid 2 to smithi08937929-38 2024-04-22T18:38:55.492 INFO:tasks.rados.rados.0.smithi089.stdout:38: finishing write tid 3 to smithi08937929-38 2024-04-22T18:38:55.492 INFO:tasks.rados.rados.0.smithi089.stdout:30: done (15 left) 2024-04-22T18:38:55.492 INFO:tasks.rados.rados.0.smithi089.stdout:53: copy oid 4 target oid 4 low_tier 2024-04-22T18:38:55.492 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 4 to 4 low_tier tgt_pool_name low_tier 2024-04-22T18:38:55.492 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.492 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:38:55.493 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:38:55.506 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:38:55.507 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:38:55.515 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:55.517+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:55.573 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:55.573 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:38:55.573 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:38:55.573 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:38:55.573 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:55.576 INFO:tasks.rados.rados.0.smithi089.stdout:38: finishing write tid 4 to smithi08937929-38 2024-04-22T18:38:55.576 INFO:tasks.rados.rados.0.smithi089.stdout:38: oid 38 updating version 0 to 12 2024-04-22T18:38:55.576 INFO:tasks.rados.rados.0.smithi089.stdout:38: oid 38 version 12 is already newer than 11 2024-04-22T18:38:55.576 INFO:tasks.rados.rados.0.smithi089.stdout:38: oid 38 version 12 is already newer than 10 2024-04-22T18:38:55.576 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-22T18:38:55.576 INFO:tasks.rados.rados.0.smithi089.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-04-22T18:38:55.576 INFO:tasks.rados.rados.0.smithi089.stdout:38: done (15 left) 2024-04-22T18:38:55.576 INFO:tasks.rados.rados.0.smithi089.stdout:54: copy oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.576 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 5 to 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:55.576 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:38:55.590 INFO:tasks.rados.rados.0.smithi089.stdout:42: finishing write tid 4 to smithi08937929-42 2024-04-22T18:38:55.590 INFO:tasks.rados.rados.0.smithi089.stdout:42: oid 42 updating version 0 to 2 2024-04-22T18:38:55.590 INFO:tasks.rados.rados.0.smithi089.stdout:42: oid 42 updating version 2 to 3 2024-04-22T18:38:55.590 INFO:tasks.rados.rados.0.smithi089.stdout:42: oid 42 version 3 is already newer than 1 2024-04-22T18:38:55.591 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 3 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-22T18:38:55.591 INFO:tasks.rados.rados.0.smithi089.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-04-22T18:38:55.591 INFO:tasks.rados.rados.0.smithi089.stdout:42: done (15 left) 2024-04-22T18:38:55.591 INFO:tasks.rados.rados.0.smithi089.stdout:55: copy oid 6 target oid 6 low_tier 2024-04-22T18:38:55.591 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 6 to 6 low_tier tgt_pool_name low_tier 2024-04-22T18:38:55.591 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.596 INFO:tasks.rados.rados.0.smithi089.stdout:51: finishing copy op to oid 2 low_tier 2024-04-22T18:38:55.596 INFO:tasks.rados.rados.0.smithi089.stdout:51: done (15 left) 2024-04-22T18:38:55.597 INFO:tasks.rados.rados.0.smithi089.stdout:56: copy oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.597 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 7 to 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:55.597 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.604 INFO:tasks.rados.rados.0.smithi089.stdout:47: finishing write tid 1 to smithi08937929-47 2024-04-22T18:38:55.604 INFO:tasks.rados.rados.0.smithi089.stdout:47: finishing write tid 2 to smithi08937929-47 2024-04-22T18:38:55.605 INFO:tasks.rados.rados.0.smithi089.stdout:40: finishing write tid 1 to smithi08937929-40 2024-04-22T18:38:55.605 INFO:tasks.rados.rados.0.smithi089.stdout:40: finishing write tid 2 to smithi08937929-40 2024-04-22T18:38:55.605 INFO:tasks.rados.rados.0.smithi089.stdout:47: finishing write tid 3 to smithi08937929-47 2024-04-22T18:38:55.608 INFO:tasks.rados.rados.0.smithi089.stdout:46: finishing write tid 1 to smithi08937929-46 2024-04-22T18:38:55.608 INFO:tasks.rados.rados.0.smithi089.stdout:46: finishing write tid 2 to smithi08937929-46 2024-04-22T18:38:55.611 INFO:tasks.rados.rados.0.smithi089.stdout:25: finishing write tid 3 to smithi08937929-25 2024-04-22T18:38:55.611 INFO:tasks.rados.rados.0.smithi089.stdout:25: finishing write tid 4 to smithi08937929-25 2024-04-22T18:38:55.611 INFO:tasks.rados.rados.0.smithi089.stdout:25: oid 25 updating version 0 to 10 2024-04-22T18:38:55.611 INFO:tasks.rados.rados.0.smithi089.stdout:25: oid 25 updating version 10 to 12 2024-04-22T18:38:55.611 INFO:tasks.rados.rados.0.smithi089.stdout:25: oid 25 version 12 is already newer than 11 2024-04-22T18:38:55.611 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 12 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-22T18:38:55.611 INFO:tasks.rados.rados.0.smithi089.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-04-22T18:38:55.611 INFO:tasks.rados.rados.0.smithi089.stdout:44: finishing write tid 4 to smithi08937929-44 2024-04-22T18:38:55.611 INFO:tasks.rados.rados.0.smithi089.stdout:44: oid 44 updating version 0 to 10 2024-04-22T18:38:55.611 INFO:tasks.rados.rados.0.smithi089.stdout:44: oid 44 updating version 10 to 12 2024-04-22T18:38:55.611 INFO:tasks.rados.rados.0.smithi089.stdout:44: oid 44 version 12 is already newer than 11 2024-04-22T18:38:55.611 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-22T18:38:55.611 INFO:tasks.rados.rados.0.smithi089.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-04-22T18:38:55.612 INFO:tasks.rados.rados.0.smithi089.stdout:48: finishing write tid 1 to smithi08937929-48 2024-04-22T18:38:55.612 INFO:tasks.rados.rados.0.smithi089.stdout:25: done (15 left) 2024-04-22T18:38:55.612 INFO:tasks.rados.rados.0.smithi089.stdout:44: done (14 left) 2024-04-22T18:38:55.612 INFO:tasks.rados.rados.0.smithi089.stdout:57: copy oid 8 target oid 8 low_tier 2024-04-22T18:38:55.612 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 8 to 8 low_tier tgt_pool_name low_tier 2024-04-22T18:38:55.612 INFO:tasks.rados.rados.0.smithi089.stdout:58: copy oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.613 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 9 to 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:55.613 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.613 INFO:tasks.rados.rados.0.smithi089.stdout:49: finishing write tid 1 to smithi08937929-49 2024-04-22T18:38:55.613 INFO:tasks.rados.rados.0.smithi089.stdout:48: finishing write tid 2 to smithi08937929-48 2024-04-22T18:38:55.613 INFO:tasks.rados.rados.0.smithi089.stdout:48: finishing write tid 3 to smithi08937929-48 2024-04-22T18:38:55.618 INFO:tasks.rados.rados.0.smithi089.stdout:39: finishing write tid 1 to smithi08937929-39 2024-04-22T18:38:55.618 INFO:tasks.rados.rados.0.smithi089.stdout:39: finishing write tid 2 to smithi08937929-39 2024-04-22T18:38:55.618 INFO:tasks.rados.rados.0.smithi089.stdout:39: finishing write tid 3 to smithi08937929-39 2024-04-22T18:38:55.618 INFO:tasks.rados.rados.0.smithi089.stdout:41: finishing write tid 1 to smithi08937929-41 2024-04-22T18:38:55.619 INFO:tasks.rados.rados.0.smithi089.stdout:39: finishing write tid 4 to smithi08937929-39 2024-04-22T18:38:55.619 INFO:tasks.rados.rados.0.smithi089.stdout:39: oid 39 updating version 0 to 14 2024-04-22T18:38:55.619 INFO:tasks.rados.rados.0.smithi089.stdout:39: oid 39 updating version 14 to 15 2024-04-22T18:38:55.620 INFO:tasks.rados.rados.0.smithi089.stdout:39: oid 39 version 15 is already newer than 13 2024-04-22T18:38:55.620 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 15 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-22T18:38:55.620 INFO:tasks.rados.rados.0.smithi089.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-04-22T18:38:55.620 INFO:tasks.rados.rados.0.smithi089.stdout:39: done (15 left) 2024-04-22T18:38:55.620 INFO:tasks.rados.rados.0.smithi089.stdout:59: copy oid 10 target oid 10 low_tier 2024-04-22T18:38:55.620 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 10 to 10 low_tier tgt_pool_name low_tier 2024-04-22T18:38:55.620 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.620 INFO:tasks.rados.rados.0.smithi089.stdout:52: finishing copy op to oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.620 INFO:tasks.rados.rados.0.smithi089.stdout:52: done (15 left) 2024-04-22T18:38:55.620 INFO:tasks.rados.rados.0.smithi089.stdout:60: copy oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.620 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 11 to 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:55.620 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.622 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:55.624+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:55.624 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:38:55.624 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:38:55.641 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:38:55.642 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:38:55.698 INFO:tasks.rados.rados.0.smithi089.stdout:48: finishing write tid 4 to smithi08937929-48 2024-04-22T18:38:55.698 INFO:tasks.rados.rados.0.smithi089.stdout:48: oid 48 updating version 0 to 8 2024-04-22T18:38:55.699 INFO:tasks.rados.rados.0.smithi089.stdout:48: oid 48 version 8 is already newer than 7 2024-04-22T18:38:55.699 INFO:tasks.rados.rados.0.smithi089.stdout:48: oid 48 updating version 8 to 9 2024-04-22T18:38:55.699 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-22T18:38:55.699 INFO:tasks.rados.rados.0.smithi089.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-04-22T18:38:55.699 INFO:tasks.rados.rados.0.smithi089.stdout:48: done (15 left) 2024-04-22T18:38:55.699 INFO:tasks.rados.rados.0.smithi089.stdout:61: copy oid 12 target oid 12 low_tier 2024-04-22T18:38:55.699 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 12 to 12 low_tier tgt_pool_name low_tier 2024-04-22T18:38:55.699 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.699 INFO:tasks.rados.rados.0.smithi089.stdout:58: finishing copy op to oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.699 INFO:tasks.rados.rados.0.smithi089.stdout:58: done (15 left) 2024-04-22T18:38:55.699 INFO:tasks.rados.rados.0.smithi089.stdout:62: copy oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.700 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 13 to 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:55.700 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.706 INFO:tasks.rados.rados.0.smithi089.stdout:40: finishing write tid 3 to smithi08937929-40 2024-04-22T18:38:55.706 INFO:tasks.rados.rados.0.smithi089.stdout:40: finishing write tid 4 to smithi08937929-40 2024-04-22T18:38:55.706 INFO:tasks.rados.rados.0.smithi089.stdout:40: oid 40 updating version 0 to 2 2024-04-22T18:38:55.706 INFO:tasks.rados.rados.0.smithi089.stdout:40: oid 40 updating version 2 to 3 2024-04-22T18:38:55.707 INFO:tasks.rados.rados.0.smithi089.stdout:40: oid 40 version 3 is already newer than 1 2024-04-22T18:38:55.707 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 3 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-22T18:38:55.707 INFO:tasks.rados.rados.0.smithi089.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-04-22T18:38:55.707 INFO:tasks.rados.rados.0.smithi089.stdout:40: done (15 left) 2024-04-22T18:38:55.707 INFO:tasks.rados.rados.0.smithi089.stdout:63: copy oid 14 target oid 14 low_tier 2024-04-22T18:38:55.707 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 14 to 14 low_tier tgt_pool_name low_tier 2024-04-22T18:38:55.707 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.707 INFO:tasks.rados.rados.0.smithi089.stdout:43: finishing write tid 1 to smithi08937929-43 2024-04-22T18:38:55.707 INFO:tasks.rados.rados.0.smithi089.stdout:43: finishing write tid 2 to smithi08937929-43 2024-04-22T18:38:55.707 INFO:tasks.rados.rados.0.smithi089.stdout:43: finishing write tid 3 to smithi08937929-43 2024-04-22T18:38:55.707 INFO:tasks.rados.rados.0.smithi089.stdout:43: finishing write tid 4 to smithi08937929-43 2024-04-22T18:38:55.707 INFO:tasks.rados.rados.0.smithi089.stdout:43: oid 43 updating version 0 to 6 2024-04-22T18:38:55.707 INFO:tasks.rados.rados.0.smithi089.stdout:43: oid 43 version 6 is already newer than 5 2024-04-22T18:38:55.708 INFO:tasks.rados.rados.0.smithi089.stdout:43: oid 43 version 6 is already newer than 4 2024-04-22T18:38:55.708 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-22T18:38:55.708 INFO:tasks.rados.rados.0.smithi089.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-04-22T18:38:55.708 INFO:tasks.rados.rados.0.smithi089.stdout:43: done (15 left) 2024-04-22T18:38:55.708 INFO:tasks.rados.rados.0.smithi089.stdout:64: copy oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.708 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 15 to 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:55.708 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.708 INFO:tasks.rados.rados.0.smithi089.stdout:47: finishing write tid 4 to smithi08937929-47 2024-04-22T18:38:55.708 INFO:tasks.rados.rados.0.smithi089.stdout:47: oid 47 updating version 0 to 12 2024-04-22T18:38:55.708 INFO:tasks.rados.rados.0.smithi089.stdout:47: oid 47 version 12 is already newer than 11 2024-04-22T18:38:55.708 INFO:tasks.rados.rados.0.smithi089.stdout:47: oid 47 version 12 is already newer than 10 2024-04-22T18:38:55.708 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-22T18:38:55.708 INFO:tasks.rados.rados.0.smithi089.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-04-22T18:38:55.708 INFO:tasks.rados.rados.0.smithi089.stdout:47: done (15 left) 2024-04-22T18:38:55.709 INFO:tasks.rados.rados.0.smithi089.stdout:65: copy oid 16 target oid 16 low_tier 2024-04-22T18:38:55.709 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 16 to 16 low_tier tgt_pool_name low_tier 2024-04-22T18:38:55.709 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.715 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:55.715+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:38:55.716 INFO:tasks.rados.rados.0.smithi089.stdout:55: finishing copy op to oid 6 low_tier 2024-04-22T18:38:55.716 INFO:tasks.rados.rados.0.smithi089.stdout:55: done (15 left) 2024-04-22T18:38:55.716 INFO:tasks.rados.rados.0.smithi089.stdout:66: copy oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.716 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 17 to 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:55.716 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.725 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:55.725 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:55.734 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:38:55.799 INFO:tasks.rados.rados.0.smithi089.stdout:46: finishing write tid 3 to smithi08937929-46 2024-04-22T18:38:55.800 INFO:tasks.rados.rados.0.smithi089.stdout:53: finishing copy op to oid 4 low_tier 2024-04-22T18:38:55.800 INFO:tasks.rados.rados.0.smithi089.stdout:59: finishing copy op to oid 10 low_tier 2024-04-22T18:38:55.800 INFO:tasks.rados.rados.0.smithi089.stdout:53: done (15 left) 2024-04-22T18:38:55.800 INFO:tasks.rados.rados.0.smithi089.stdout:59: done (14 left) 2024-04-22T18:38:55.800 INFO:tasks.rados.rados.0.smithi089.stdout:67: copy oid 18 target oid 18 low_tier 2024-04-22T18:38:55.800 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 18 to 18 low_tier tgt_pool_name low_tier 2024-04-22T18:38:55.800 INFO:tasks.rados.rados.0.smithi089.stdout:68: copy oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.800 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 19 to 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:55.800 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.800 INFO:tasks.rados.rados.0.smithi089.stdout:54: finishing copy op to oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.800 INFO:tasks.rados.rados.0.smithi089.stdout:54: done (15 left) 2024-04-22T18:38:55.800 INFO:tasks.rados.rados.0.smithi089.stdout:69: copy oid 20 target oid 20 low_tier 2024-04-22T18:38:55.800 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 20 to 20 low_tier tgt_pool_name low_tier 2024-04-22T18:38:55.800 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.801 INFO:tasks.rados.rados.0.smithi089.stdout:46: finishing write tid 4 to smithi08937929-46 2024-04-22T18:38:55.801 INFO:tasks.rados.rados.0.smithi089.stdout:46: oid 46 updating version 0 to 4 2024-04-22T18:38:55.801 INFO:tasks.rados.rados.0.smithi089.stdout:46: oid 46 updating version 4 to 6 2024-04-22T18:38:55.801 INFO:tasks.rados.rados.0.smithi089.stdout:46: oid 46 version 6 is already newer than 5 2024-04-22T18:38:55.801 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-22T18:38:55.801 INFO:tasks.rados.rados.0.smithi089.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-04-22T18:38:55.801 INFO:tasks.rados.rados.0.smithi089.stdout:46: done (15 left) 2024-04-22T18:38:55.801 INFO:tasks.rados.rados.0.smithi089.stdout:70: copy oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.801 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 21 to 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:55.801 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.805 INFO:tasks.rados.rados.0.smithi089.stdout:49: finishing write tid 2 to smithi08937929-49 2024-04-22T18:38:55.805 INFO:tasks.rados.rados.0.smithi089.stdout:49: finishing write tid 3 to smithi08937929-49 2024-04-22T18:38:55.816 INFO:tasks.rados.rados.0.smithi089.stdout:49: finishing write tid 4 to smithi08937929-49 2024-04-22T18:38:55.816 INFO:tasks.rados.rados.0.smithi089.stdout:49: oid 49 updating version 0 to 15 2024-04-22T18:38:55.816 INFO:tasks.rados.rados.0.smithi089.stdout:49: oid 49 version 15 is already newer than 13 2024-04-22T18:38:55.816 INFO:tasks.rados.rados.0.smithi089.stdout:49: oid 49 version 15 is already newer than 14 2024-04-22T18:38:55.816 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:38:55.816 INFO:tasks.rados.rados.0.smithi089.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-04-22T18:38:55.816 INFO:tasks.rados.rados.0.smithi089.stdout:49: done (15 left) 2024-04-22T18:38:55.816 INFO:tasks.rados.rados.0.smithi089.stdout:71: copy oid 22 target oid 22 low_tier 2024-04-22T18:38:55.816 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 22 to 22 low_tier tgt_pool_name low_tier 2024-04-22T18:38:55.816 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.817 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:55.818+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:55.842 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:55.842 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:55.851 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:38:55.916 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:55.917+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:55.923 INFO:tasks.rados.rados.0.smithi089.stdout:61: finishing copy op to oid 12 low_tier 2024-04-22T18:38:55.923 INFO:tasks.rados.rados.0.smithi089.stdout:68: finishing copy op to oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.923 INFO:tasks.rados.rados.0.smithi089.stdout:61: done (15 left) 2024-04-22T18:38:55.923 INFO:tasks.rados.rados.0.smithi089.stdout:68: done (14 left) 2024-04-22T18:38:55.923 INFO:tasks.rados.rados.0.smithi089.stdout:72: copy oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.923 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 23 to 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:55.923 INFO:tasks.rados.rados.0.smithi089.stdout:73: copy oid 24 target oid 24 low_tier 2024-04-22T18:38:55.923 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 24 to 24 low_tier tgt_pool_name low_tier 2024-04-22T18:38:55.923 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.923 INFO:tasks.rados.rados.0.smithi089.stdout:70: finishing copy op to oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.923 INFO:tasks.rados.rados.0.smithi089.stdout:70: done (15 left) 2024-04-22T18:38:55.923 INFO:tasks.rados.rados.0.smithi089.stdout:74: copy oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.924 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 25 to 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:55.924 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.924 INFO:tasks.rados.rados.0.smithi089.stdout:41: finishing write tid 2 to smithi08937929-41 2024-04-22T18:38:55.925 INFO:tasks.rados.rados.0.smithi089.stdout:56: finishing copy op to oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.925 INFO:tasks.rados.rados.0.smithi089.stdout:56: done (15 left) 2024-04-22T18:38:55.925 INFO:tasks.rados.rados.0.smithi089.stdout:75: copy oid 26 target oid 26 low_tier 2024-04-22T18:38:55.925 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 26 to 26 low_tier tgt_pool_name low_tier 2024-04-22T18:38:55.925 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.925 INFO:tasks.rados.rados.0.smithi089.stdout:57: finishing copy op to oid 8 low_tier 2024-04-22T18:38:55.926 INFO:tasks.rados.rados.0.smithi089.stdout:57: done (15 left) 2024-04-22T18:38:55.926 INFO:tasks.rados.rados.0.smithi089.stdout:76: copy oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:55.926 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 27 to 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:55.926 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:55.957 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:55.957 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:55.966 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:38:56.008 INFO:tasks.rados.rados.0.smithi089.stdout:69: finishing copy op to oid 20 low_tier 2024-04-22T18:38:56.008 INFO:tasks.rados.rados.0.smithi089.stdout:69: done (15 left) 2024-04-22T18:38:56.008 INFO:tasks.rados.rados.0.smithi089.stdout:77: copy oid 28 target oid 28 low_tier 2024-04-22T18:38:56.009 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 28 to 28 low_tier tgt_pool_name low_tier 2024-04-22T18:38:56.009 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.014 INFO:tasks.rados.rados.0.smithi089.stdout:66: finishing copy op to oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.015 INFO:tasks.rados.rados.0.smithi089.stdout:66: done (15 left) 2024-04-22T18:38:56.015 INFO:tasks.rados.rados.0.smithi089.stdout:78: copy oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.015 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 29 to 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:56.015 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.017 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:56.017+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:56.025 INFO:tasks.rados.rados.0.smithi089.stdout:60: finishing copy op to oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.025 INFO:tasks.rados.rados.0.smithi089.stdout:60: done (15 left) 2024-04-22T18:38:56.025 INFO:tasks.rados.rados.0.smithi089.stdout:79: copy oid 30 target oid 30 low_tier 2024-04-22T18:38:56.025 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 30 to 30 low_tier tgt_pool_name low_tier 2024-04-22T18:38:56.025 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.030 INFO:tasks.rados.rados.0.smithi089.stdout:64: finishing copy op to oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.030 INFO:tasks.rados.rados.0.smithi089.stdout:64: done (15 left) 2024-04-22T18:38:56.030 INFO:tasks.rados.rados.0.smithi089.stdout:80: copy oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.030 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 31 to 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:56.030 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.030 INFO:tasks.rados.rados.0.smithi089.stdout:65: finishing copy op to oid 16 low_tier 2024-04-22T18:38:56.030 INFO:tasks.rados.rados.0.smithi089.stdout:65: done (15 left) 2024-04-22T18:38:56.030 INFO:tasks.rados.rados.0.smithi089.stdout:81: copy oid 32 target oid 32 low_tier 2024-04-22T18:38:56.031 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 32 to 32 low_tier tgt_pool_name low_tier 2024-04-22T18:38:56.031 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.110 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:56.110 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:56.112 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:38:56.118 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:56.119+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:56.120 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:38:56.127 INFO:tasks.rados.rados.0.smithi089.stdout:41: finishing write tid 3 to smithi08937929-41 2024-04-22T18:38:56.127 INFO:tasks.rados.rados.0.smithi089.stdout:41: finishing write tid 4 to smithi08937929-41 2024-04-22T18:38:56.128 INFO:tasks.rados.rados.0.smithi089.stdout:41: oid 41 updating version 0 to 17 2024-04-22T18:38:56.128 INFO:tasks.rados.rados.0.smithi089.stdout:41: oid 41 version 17 is already newer than 16 2024-04-22T18:38:56.128 INFO:tasks.rados.rados.0.smithi089.stdout:41: oid 41 updating version 17 to 18 2024-04-22T18:38:56.128 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 18 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-22T18:38:56.128 INFO:tasks.rados.rados.0.smithi089.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-04-22T18:38:56.128 INFO:tasks.rados.rados.0.smithi089.stdout:50: finishing write tid 1 to smithi08937929-50 2024-04-22T18:38:56.128 INFO:tasks.rados.rados.0.smithi089.stdout:50: finishing write tid 2 to smithi08937929-50 2024-04-22T18:38:56.128 INFO:tasks.rados.rados.0.smithi089.stdout:41: done (15 left) 2024-04-22T18:38:56.128 INFO:tasks.rados.rados.0.smithi089.stdout:82: copy oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.128 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 33 to 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:56.128 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.128 INFO:tasks.rados.rados.0.smithi089.stdout:50: finishing write tid 3 to smithi08937929-50 2024-04-22T18:38:56.128 INFO:tasks.rados.rados.0.smithi089.stdout:50: finishing write tid 4 to smithi08937929-50 2024-04-22T18:38:56.128 INFO:tasks.rados.rados.0.smithi089.stdout:50: oid 50 updating version 0 to 19 2024-04-22T18:38:56.128 INFO:tasks.rados.rados.0.smithi089.stdout:50: oid 50 updating version 19 to 21 2024-04-22T18:38:56.129 INFO:tasks.rados.rados.0.smithi089.stdout:50: oid 50 version 21 is already newer than 20 2024-04-22T18:38:56.129 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 21 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-22T18:38:56.129 INFO:tasks.rados.rados.0.smithi089.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-04-22T18:38:56.129 INFO:tasks.rados.rados.0.smithi089.stdout:50: done (15 left) 2024-04-22T18:38:56.129 INFO:tasks.rados.rados.0.smithi089.stdout:83: copy oid 34 target oid 34 low_tier 2024-04-22T18:38:56.129 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 34 to 34 low_tier tgt_pool_name low_tier 2024-04-22T18:38:56.129 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.130 INFO:tasks.rados.rados.0.smithi089.stdout:67: finishing copy op to oid 18 low_tier 2024-04-22T18:38:56.130 INFO:tasks.rados.rados.0.smithi089.stdout:74: finishing copy op to oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.131 INFO:tasks.rados.rados.0.smithi089.stdout:67: done (15 left) 2024-04-22T18:38:56.131 INFO:tasks.rados.rados.0.smithi089.stdout:74: done (14 left) 2024-04-22T18:38:56.131 INFO:tasks.rados.rados.0.smithi089.stdout:84: copy oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.131 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 35 to 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:56.131 INFO:tasks.rados.rados.0.smithi089.stdout:85: copy oid 36 target oid 36 low_tier 2024-04-22T18:38:56.131 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 36 to 36 low_tier tgt_pool_name low_tier 2024-04-22T18:38:56.131 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.135 INFO:tasks.rados.rados.0.smithi089.stdout:78: finishing copy op to oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.135 INFO:tasks.rados.rados.0.smithi089.stdout:78: done (15 left) 2024-04-22T18:38:56.136 INFO:tasks.rados.rados.0.smithi089.stdout:86: copy oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.136 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 37 to 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:56.136 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.136 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:56.137+0000 7f8605f32640 -1 osd.3 27 handle_osd_map injecting map crc failure 2024-04-22T18:38:56.138 INFO:tasks.rados.rados.0.smithi089.stdout:81: finishing copy op to oid 32 low_tier 2024-04-22T18:38:56.139 INFO:tasks.rados.rados.0.smithi089.stdout:81: done (15 left) 2024-04-22T18:38:56.139 INFO:tasks.rados.rados.0.smithi089.stdout:87: copy oid 38 target oid 38 low_tier 2024-04-22T18:38:56.139 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 38 to 38 low_tier tgt_pool_name low_tier 2024-04-22T18:38:56.139 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.219 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:56.219+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:38:56.227 INFO:tasks.rados.rados.0.smithi089.stdout:71: finishing copy op to oid 22 low_tier 2024-04-22T18:38:56.227 INFO:tasks.rados.rados.0.smithi089.stdout:62: finishing copy op to oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.227 INFO:tasks.rados.rados.0.smithi089.stdout:63: finishing copy op to oid 14 low_tier 2024-04-22T18:38:56.228 INFO:tasks.rados.rados.0.smithi089.stdout:62: done (15 left) 2024-04-22T18:38:56.228 INFO:tasks.rados.rados.0.smithi089.stdout:63: done (14 left) 2024-04-22T18:38:56.228 INFO:tasks.rados.rados.0.smithi089.stdout:71: done (13 left) 2024-04-22T18:38:56.228 INFO:tasks.rados.rados.0.smithi089.stdout:88: copy oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.228 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 39 to 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:56.228 INFO:tasks.rados.rados.0.smithi089.stdout:89: copy oid 40 target oid 40 low_tier 2024-04-22T18:38:56.228 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 40 to 40 low_tier tgt_pool_name low_tier 2024-04-22T18:38:56.228 INFO:tasks.rados.rados.0.smithi089.stdout:90: copy oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.228 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 41 to 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:56.228 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.232 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:56.232+0000 7f1e2dd22640 -1 osd.2 27 handle_osd_map injecting map crc failure 2024-04-22T18:38:56.235 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:56.235 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:56.240 INFO:tasks.rados.rados.0.smithi089.stdout:73: finishing copy op to oid 24 low_tier 2024-04-22T18:38:56.240 INFO:tasks.rados.rados.0.smithi089.stdout:73: done (15 left) 2024-04-22T18:38:56.240 INFO:tasks.rados.rados.0.smithi089.stdout:91: copy oid 42 target oid 42 low_tier 2024-04-22T18:38:56.240 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 42 to 42 low_tier tgt_pool_name low_tier 2024-04-22T18:38:56.240 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.243 INFO:tasks.rados.rados.0.smithi089.stdout:75: finishing copy op to oid 26 low_tier 2024-04-22T18:38:56.243 INFO:tasks.rados.rados.0.smithi089.stdout:75: done (15 left) 2024-04-22T18:38:56.243 INFO:tasks.rados.rados.0.smithi089.stdout:92: copy oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.243 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 43 to 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:56.243 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.246 INFO:tasks.rados.rados.0.smithi089.stdout:85: finishing copy op to oid 36 low_tier 2024-04-22T18:38:56.246 INFO:tasks.rados.rados.0.smithi089.stdout:85: done (15 left) 2024-04-22T18:38:56.246 INFO:tasks.rados.rados.0.smithi089.stdout:93: copy oid 44 target oid 44 low_tier 2024-04-22T18:38:56.246 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 44 to 44 low_tier tgt_pool_name low_tier 2024-04-22T18:38:56.246 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.246 INFO:tasks.rados.rados.0.smithi089.stdout:79: finishing copy op to oid 30 low_tier 2024-04-22T18:38:56.246 INFO:tasks.rados.rados.0.smithi089.stdout:79: done (15 left) 2024-04-22T18:38:56.246 INFO:tasks.rados.rados.0.smithi089.stdout:94: copy oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.246 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 45 to 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:56.246 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.247 INFO:tasks.rados.rados.0.smithi089.stdout:77: finishing copy op to oid 28 low_tier 2024-04-22T18:38:56.247 INFO:tasks.rados.rados.0.smithi089.stdout:77: done (15 left) 2024-04-22T18:38:56.247 INFO:tasks.rados.rados.0.smithi089.stdout:95: copy oid 46 target oid 46 low_tier 2024-04-22T18:38:56.247 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 46 to 46 low_tier tgt_pool_name low_tier 2024-04-22T18:38:56.247 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.248 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:38:56.320 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:56.321+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:56.335 INFO:tasks.rados.rados.0.smithi089.stdout:84: finishing copy op to oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.335 INFO:tasks.rados.rados.0.smithi089.stdout:84: done (15 left) 2024-04-22T18:38:56.335 INFO:tasks.rados.rados.0.smithi089.stdout:96: copy oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.336 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 47 to 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:56.336 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.340 INFO:tasks.rados.rados.0.smithi089.stdout:94: finishing copy op to oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.340 INFO:tasks.rados.rados.0.smithi089.stdout:94: done (15 left) 2024-04-22T18:38:56.341 INFO:tasks.rados.rados.0.smithi089.stdout:97: copy oid 48 target oid 48 low_tier 2024-04-22T18:38:56.341 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 48 to 48 low_tier tgt_pool_name low_tier 2024-04-22T18:38:56.341 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.347 INFO:tasks.rados.rados.0.smithi089.stdout:92: finishing copy op to oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.347 INFO:tasks.rados.rados.0.smithi089.stdout:92: done (15 left) 2024-04-22T18:38:56.347 INFO:tasks.rados.rados.0.smithi089.stdout:98: copy oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.347 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 49 to 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:56.347 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.351 INFO:tasks.rados.rados.0.smithi089.stdout:96: finishing copy op to oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.351 INFO:tasks.rados.rados.0.smithi089.stdout:96: done (15 left) 2024-04-22T18:38:56.351 INFO:tasks.rados.rados.0.smithi089.stdout:99: copy oid 50 target oid 50 low_tier 2024-04-22T18:38:56.351 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 50 to 50 low_tier tgt_pool_name low_tier 2024-04-22T18:38:56.351 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.364 INFO:tasks.rados.rados.0.smithi089.stdout:93: finishing copy op to oid 44 low_tier 2024-04-22T18:38:56.364 INFO:tasks.rados.rados.0.smithi089.stdout:93: done (15 left) 2024-04-22T18:38:56.364 INFO:tasks.rados.rados.0.smithi089.stdout:100: copy oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.364 INFO:tasks.rados.rados.0.smithi089.stdout:copy op oid 1 to 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-22T18:38:56.364 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.366 INFO:tasks.rados.rados.0.smithi089.stdout:72: finishing copy op to oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.366 INFO:tasks.rados.rados.0.smithi089.stdout:72: done (15 left) 2024-04-22T18:38:56.366 INFO:tasks.rados.rados.0.smithi089.stdout:101: set_chunk oid 2 target oid 2 low_tier 2024-04-22T18:38:56.366 INFO:tasks.rados.rados.0.smithi089.stdout:101: set_chunk oid 2 offset: 350720 length: 1444864 target oid smithi08937929-2 low_tier offset: 350720 snap -1 2024-04-22T18:38:56.366 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.366 INFO:tasks.rados.rados.0.smithi089.stdout:76: finishing copy op to oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.366 INFO:tasks.rados.rados.0.smithi089.stdout:76: done (15 left) 2024-04-22T18:38:56.366 INFO:tasks.rados.rados.0.smithi089.stdout:102: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.366 INFO:tasks.rados.rados.0.smithi089.stdout:102: set_chunk oid 3 offset: 260608 length: 169984 target oid smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 260608 snap -1 2024-04-22T18:38:56.366 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.398 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:56.398 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:56.407 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:38:56.452 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:56.452+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:38:56.466 INFO:tasks.rados.rados.0.smithi089.stdout:88: finishing copy op to oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.466 INFO:tasks.rados.rados.0.smithi089.stdout:88: done (15 left) 2024-04-22T18:38:56.466 INFO:tasks.rados.rados.0.smithi089.stdout:103: set_chunk oid 4 target oid 4 low_tier 2024-04-22T18:38:56.466 INFO:tasks.rados.rados.0.smithi089.stdout:103: set_chunk oid 4 offset: 517120 length: 626688 target oid smithi08937929-4 low_tier offset: 517120 snap -1 2024-04-22T18:38:56.466 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.467 INFO:tasks.rados.rados.0.smithi089.stdout:91: finishing copy op to oid 42 low_tier 2024-04-22T18:38:56.467 INFO:tasks.rados.rados.0.smithi089.stdout:91: done (15 left) 2024-04-22T18:38:56.467 INFO:tasks.rados.rados.0.smithi089.stdout:104: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.467 INFO:tasks.rados.rados.0.smithi089.stdout:104: set_chunk oid 5 offset: 1156096 length: 820736 target oid smithi08937929-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1156096 snap -1 2024-04-22T18:38:56.467 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.471 INFO:tasks.rados.rados.0.smithi089.stdout:90: finishing copy op to oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.471 INFO:tasks.rados.rados.0.smithi089.stdout:90: done (15 left) 2024-04-22T18:38:56.471 INFO:tasks.rados.rados.0.smithi089.stdout:105: set_chunk oid 6 target oid 6 low_tier 2024-04-22T18:38:56.471 INFO:tasks.rados.rados.0.smithi089.stdout:105: set_chunk oid 6 offset: 314880 length: 1545728 target oid smithi08937929-6 low_tier offset: 314880 snap -1 2024-04-22T18:38:56.471 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.471 INFO:tasks.rados.rados.0.smithi089.stdout:100: finishing copy op to oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.472 INFO:tasks.rados.rados.0.smithi089.stdout:100: done (15 left) 2024-04-22T18:38:56.472 INFO:tasks.rados.rados.0.smithi089.stdout:106: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.472 INFO:tasks.rados.rados.0.smithi089.stdout:106: set_chunk oid 7 offset: 214016 length: 215040 target oid smithi08937929-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 214016 snap -1 2024-04-22T18:38:56.472 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.521 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:56.521 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:56.521 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:56.523+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:56.529 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:38:56.543 INFO:tasks.rados.rados.0.smithi089.stdout:87: finishing copy op to oid 38 low_tier 2024-04-22T18:38:56.543 INFO:tasks.rados.rados.0.smithi089.stdout:87: done (15 left) 2024-04-22T18:38:56.543 INFO:tasks.rados.rados.0.smithi089.stdout:107: set_chunk oid 8 target oid 8 low_tier 2024-04-22T18:38:56.543 INFO:tasks.rados.rados.0.smithi089.stdout:107: set_chunk oid 8 offset: 428032 length: 1131520 target oid smithi08937929-8 low_tier offset: 428032 snap -1 2024-04-22T18:38:56.543 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.543 INFO:tasks.rados.rados.0.smithi089.stdout:95: finishing copy op to oid 46 low_tier 2024-04-22T18:38:56.543 INFO:tasks.rados.rados.0.smithi089.stdout:95: done (15 left) 2024-04-22T18:38:56.543 INFO:tasks.rados.rados.0.smithi089.stdout:108: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.544 INFO:tasks.rados.rados.0.smithi089.stdout:108: set_chunk oid 9 offset: 644608 length: 139776 target oid smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 644608 snap -1 2024-04-22T18:38:56.544 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.568 INFO:tasks.rados.rados.0.smithi089.stdout:83: finishing copy op to oid 34 low_tier 2024-04-22T18:38:56.568 INFO:tasks.rados.rados.0.smithi089.stdout:83: done (15 left) 2024-04-22T18:38:56.568 INFO:tasks.rados.rados.0.smithi089.stdout:109: set_chunk oid 10 target oid 10 low_tier 2024-04-22T18:38:56.568 INFO:tasks.rados.rados.0.smithi089.stdout:109: set_chunk oid 10 offset: 1418240 length: 153088 target oid smithi08937929-10 low_tier offset: 1418240 snap -1 2024-04-22T18:38:56.568 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.568 INFO:tasks.rados.rados.0.smithi089.stdout:86: finishing copy op to oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.568 INFO:tasks.rados.rados.0.smithi089.stdout:86: done (15 left) 2024-04-22T18:38:56.568 INFO:tasks.rados.rados.0.smithi089.stdout:110: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.568 INFO:tasks.rados.rados.0.smithi089.stdout:110: set_chunk oid 11 offset: 771072 length: 1076736 target oid smithi08937929-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 771072 snap -1 2024-04-22T18:38:56.568 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.572 INFO:tasks.rados.rados.0.smithi089.stdout:99: finishing copy op to oid 50 low_tier 2024-04-22T18:38:56.572 INFO:tasks.rados.rados.0.smithi089.stdout:99: done (15 left) 2024-04-22T18:38:56.572 INFO:tasks.rados.rados.0.smithi089.stdout:111: set_chunk oid 12 target oid 12 low_tier 2024-04-22T18:38:56.572 INFO:tasks.rados.rados.0.smithi089.stdout:111: set_chunk oid 12 offset: 48128 length: 1675776 target oid smithi08937929-12 low_tier offset: 48128 snap -1 2024-04-22T18:38:56.572 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.599 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:38:56.599 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:38:56.599 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-22T18:38:56.599 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.848003 2024-04-22T18:38:56.599 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0.8480028985254894 2024-04-22T18:38:56.642 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:56.643+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:56.644 INFO:tasks.rados.rados.0.smithi089.stdout:101: finishing set_chunk to oid 2 2024-04-22T18:38:56.644 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-22T18:38:56.644 INFO:tasks.rados.rados.0.smithi089.stdout:101: done (15 left) 2024-04-22T18:38:56.644 INFO:tasks.rados.rados.0.smithi089.stdout:112: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.644 INFO:tasks.rados.rados.0.smithi089.stdout:112: set_chunk oid 13 offset: 222208 length: 1040384 target oid smithi08937929-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 222208 snap -1 2024-04-22T18:38:56.644 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.645 INFO:tasks.rados.rados.0.smithi089.stdout:80: finishing copy op to oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.645 INFO:tasks.rados.rados.0.smithi089.stdout:80: done (15 left) 2024-04-22T18:38:56.645 INFO:tasks.rados.rados.0.smithi089.stdout:113: set_chunk oid 14 target oid 14 low_tier 2024-04-22T18:38:56.645 INFO:tasks.rados.rados.0.smithi089.stdout:113: set_chunk oid 14 offset: 780288 length: 1311744 target oid smithi08937929-14 low_tier offset: 780288 snap -1 2024-04-22T18:38:56.645 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.646 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:56.646 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:56.655 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:38:56.656 INFO:tasks.rados.rados.0.smithi089.stdout:102: finishing set_chunk to oid 3 2024-04-22T18:38:56.656 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 6 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-22T18:38:56.656 INFO:tasks.rados.rados.0.smithi089.stdout:102: done (15 left) 2024-04-22T18:38:56.656 INFO:tasks.rados.rados.0.smithi089.stdout:114: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.656 INFO:tasks.rados.rados.0.smithi089.stdout:114: set_chunk oid 15 offset: 994304 length: 508416 target oid smithi08937929-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 994304 snap -1 2024-04-22T18:38:56.656 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:98: finishing copy op to oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:97: finishing copy op to oid 48 low_tier 2024-04-22T18:38:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:97: done (15 left) 2024-04-22T18:38:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:98: done (14 left) 2024-04-22T18:38:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:115: set_chunk oid 16 target oid 16 low_tier 2024-04-22T18:38:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:115: set_chunk oid 16 offset: 1036800 length: 194560 target oid smithi08937929-16 low_tier offset: 1036800 snap -1 2024-04-22T18:38:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:116: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:116: set_chunk oid 17 offset: 1652224 length: 227840 target oid smithi08937929-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1652224 snap -1 2024-04-22T18:38:56.679 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:82: finishing copy op to oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:89: finishing copy op to oid 40 low_tier 2024-04-22T18:38:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:82: done (15 left) 2024-04-22T18:38:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:89: done (14 left) 2024-04-22T18:38:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:117: set_chunk oid 18 target oid 18 low_tier 2024-04-22T18:38:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:117: set_chunk oid 18 offset: 1943040 length: 120320 target oid smithi08937929-18 low_tier offset: 1943040 snap -1 2024-04-22T18:38:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:118: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:118: set_chunk oid 19 offset: 1267200 length: 498176 target oid smithi08937929-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1267200 snap -1 2024-04-22T18:38:56.680 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.723 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:56.723+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:38:56.779 INFO:tasks.rados.rados.0.smithi089.stdout:103: finishing set_chunk to oid 4 2024-04-22T18:38:56.779 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-22T18:38:56.779 INFO:tasks.rados.rados.0.smithi089.stdout:103: done (15 left) 2024-04-22T18:38:56.779 INFO:tasks.rados.rados.0.smithi089.stdout:119: set_chunk oid 20 target oid 20 low_tier 2024-04-22T18:38:56.779 INFO:tasks.rados.rados.0.smithi089.stdout:119: set_chunk oid 20 offset: 58368 length: 1395200 target oid smithi08937929-20 low_tier offset: 58368 snap -1 2024-04-22T18:38:56.779 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.783 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:56.784 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:56.793 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:38:56.834 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:56.834+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:56.857 INFO:tasks.rados.rados.0.smithi089.stdout:108: finishing set_chunk to oid 9 2024-04-22T18:38:56.857 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 9 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-22T18:38:56.857 INFO:tasks.rados.rados.0.smithi089.stdout:108: done (15 left) 2024-04-22T18:38:56.857 INFO:tasks.rados.rados.0.smithi089.stdout:120: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.857 INFO:tasks.rados.rados.0.smithi089.stdout:120: set_chunk oid 21 offset: 146944 length: 826880 target oid smithi08937929-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 146944 snap -1 2024-04-22T18:38:56.857 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.875 INFO:tasks.rados.rados.0.smithi089.stdout:105: finishing set_chunk to oid 6 2024-04-22T18:38:56.875 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-22T18:38:56.875 INFO:tasks.rados.rados.0.smithi089.stdout:105: done (15 left) 2024-04-22T18:38:56.875 INFO:tasks.rados.rados.0.smithi089.stdout:121: set_chunk oid 22 target oid 22 low_tier 2024-04-22T18:38:56.876 INFO:tasks.rados.rados.0.smithi089.stdout:121: set_chunk oid 22 offset: 998400 length: 843776 target oid smithi08937929-22 low_tier offset: 998400 snap -1 2024-04-22T18:38:56.876 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.882 INFO:tasks.rados.rados.0.smithi089.stdout:106: finishing set_chunk to oid 7 2024-04-22T18:38:56.882 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-22T18:38:56.882 INFO:tasks.rados.rados.0.smithi089.stdout:106: done (15 left) 2024-04-22T18:38:56.883 INFO:tasks.rados.rados.0.smithi089.stdout:122: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.883 INFO:tasks.rados.rados.0.smithi089.stdout:122: set_chunk oid 23 offset: 292864 length: 1954816 target oid smithi08937929-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 292864 snap -1 2024-04-22T18:38:56.883 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.883 INFO:tasks.rados.rados.0.smithi089.stdout:117: finishing set_chunk to oid 18 2024-04-22T18:38:56.884 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 9 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-22T18:38:56.884 INFO:tasks.rados.rados.0.smithi089.stdout:117: done (15 left) 2024-04-22T18:38:56.884 INFO:tasks.rados.rados.0.smithi089.stdout:123: set_chunk oid 24 target oid 24 low_tier 2024-04-22T18:38:56.884 INFO:tasks.rados.rados.0.smithi089.stdout:123: set_chunk oid 24 offset: 1569792 length: 599552 target oid smithi08937929-24 low_tier offset: 1569792 snap -1 2024-04-22T18:38:56.884 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.925 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:56.927+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:56.936 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:56.936 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:56.944 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:38:56.980 INFO:tasks.rados.rados.0.smithi089.stdout:114: finishing set_chunk to oid 15 2024-04-22T18:38:56.980 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 9 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-22T18:38:56.980 INFO:tasks.rados.rados.0.smithi089.stdout:114: done (15 left) 2024-04-22T18:38:56.981 INFO:tasks.rados.rados.0.smithi089.stdout:124: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:56.981 INFO:tasks.rados.rados.0.smithi089.stdout:124: set_chunk oid 25 offset: 479744 length: 376320 target oid smithi08937929-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 479744 snap -1 2024-04-22T18:38:56.981 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:56.983 INFO:tasks.rados.rados.0.smithi089.stdout:119: finishing set_chunk to oid 20 2024-04-22T18:38:56.983 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-22T18:38:56.983 INFO:tasks.rados.rados.0.smithi089.stdout:119: done (15 left) 2024-04-22T18:38:56.983 INFO:tasks.rados.rados.0.smithi089.stdout:125: set_chunk oid 26 target oid 26 low_tier 2024-04-22T18:38:56.983 INFO:tasks.rados.rados.0.smithi089.stdout:125: set_chunk oid 26 offset: 1580544 length: 787968 target oid smithi08937929-26 low_tier offset: 1580544 snap -1 2024-04-22T18:38:56.983 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.026 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:57.028+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:57.081 INFO:tasks.rados.rados.0.smithi089.stdout:110: finishing set_chunk to oid 11 2024-04-22T18:38:57.081 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 6 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-22T18:38:57.082 INFO:tasks.rados.rados.0.smithi089.stdout:112: finishing set_chunk to oid 13 2024-04-22T18:38:57.082 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-22T18:38:57.082 INFO:tasks.rados.rados.0.smithi089.stdout:110: done (15 left) 2024-04-22T18:38:57.082 INFO:tasks.rados.rados.0.smithi089.stdout:112: done (14 left) 2024-04-22T18:38:57.082 INFO:tasks.rados.rados.0.smithi089.stdout:126: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.082 INFO:tasks.rados.rados.0.smithi089.stdout:126: set_chunk oid 27 offset: 525824 length: 1633280 target oid smithi08937929-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 525824 snap -1 2024-04-22T18:38:57.082 INFO:tasks.rados.rados.0.smithi089.stdout:127: set_chunk oid 28 target oid 28 low_tier 2024-04-22T18:38:57.082 INFO:tasks.rados.rados.0.smithi089.stdout:127: set_chunk oid 28 offset: 157696 length: 555008 target oid smithi08937929-28 low_tier offset: 157696 snap -1 2024-04-22T18:38:57.082 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.082 INFO:tasks.rados.rados.0.smithi089.stdout:115: finishing set_chunk to oid 16 2024-04-22T18:38:57.082 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-22T18:38:57.083 INFO:tasks.rados.rados.0.smithi089.stdout:115: done (15 left) 2024-04-22T18:38:57.083 INFO:tasks.rados.rados.0.smithi089.stdout:128: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.083 INFO:tasks.rados.rados.0.smithi089.stdout:128: set_chunk oid 29 offset: 408064 length: 1645568 target oid smithi08937929-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 408064 snap -1 2024-04-22T18:38:57.083 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.083 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:57.083 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:57.085 INFO:tasks.rados.rados.0.smithi089.stdout:109: finishing set_chunk to oid 10 2024-04-22T18:38:57.086 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-22T18:38:57.086 INFO:tasks.rados.rados.0.smithi089.stdout:109: done (15 left) 2024-04-22T18:38:57.086 INFO:tasks.rados.rados.0.smithi089.stdout:129: set_chunk oid 30 target oid 30 low_tier 2024-04-22T18:38:57.086 INFO:tasks.rados.rados.0.smithi089.stdout:129: set_chunk oid 30 offset: 504832 length: 179712 target oid smithi08937929-30 low_tier offset: 504832 snap -1 2024-04-22T18:38:57.086 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.086 INFO:tasks.rados.rados.0.smithi089.stdout:120: finishing set_chunk to oid 21 2024-04-22T18:38:57.086 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-22T18:38:57.086 INFO:tasks.rados.rados.0.smithi089.stdout:120: done (15 left) 2024-04-22T18:38:57.086 INFO:tasks.rados.rados.0.smithi089.stdout:130: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.086 INFO:tasks.rados.rados.0.smithi089.stdout:130: set_chunk oid 31 offset: 410624 length: 1716224 target oid smithi08937929-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 410624 snap -1 2024-04-22T18:38:57.086 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.086 INFO:tasks.rados.rados.0.smithi089.stdout:107: finishing set_chunk to oid 8 2024-04-22T18:38:57.086 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-22T18:38:57.087 INFO:tasks.rados.rados.0.smithi089.stdout:111: finishing set_chunk to oid 12 2024-04-22T18:38:57.087 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-22T18:38:57.087 INFO:tasks.rados.rados.0.smithi089.stdout:104: finishing set_chunk to oid 5 2024-04-22T18:38:57.087 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-22T18:38:57.087 INFO:tasks.rados.rados.0.smithi089.stdout:118: finishing set_chunk to oid 19 2024-04-22T18:38:57.087 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-22T18:38:57.087 INFO:tasks.rados.rados.0.smithi089.stdout:104: done (15 left) 2024-04-22T18:38:57.087 INFO:tasks.rados.rados.0.smithi089.stdout:107: done (14 left) 2024-04-22T18:38:57.087 INFO:tasks.rados.rados.0.smithi089.stdout:111: done (13 left) 2024-04-22T18:38:57.087 INFO:tasks.rados.rados.0.smithi089.stdout:118: done (12 left) 2024-04-22T18:38:57.087 INFO:tasks.rados.rados.0.smithi089.stdout:131: set_chunk oid 32 target oid 32 low_tier 2024-04-22T18:38:57.087 INFO:tasks.rados.rados.0.smithi089.stdout:131: set_chunk oid 32 offset: 836096 length: 133120 target oid smithi08937929-32 low_tier offset: 836096 snap -1 2024-04-22T18:38:57.087 INFO:tasks.rados.rados.0.smithi089.stdout:132: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.088 INFO:tasks.rados.rados.0.smithi089.stdout:132: set_chunk oid 33 offset: 347648 length: 927744 target oid smithi08937929-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 347648 snap -1 2024-04-22T18:38:57.088 INFO:tasks.rados.rados.0.smithi089.stdout:133: set_chunk oid 34 target oid 34 low_tier 2024-04-22T18:38:57.088 INFO:tasks.rados.rados.0.smithi089.stdout:133: set_chunk oid 34 offset: 1950208 length: 301056 target oid smithi08937929-34 low_tier offset: 1950208 snap -1 2024-04-22T18:38:57.088 INFO:tasks.rados.rados.0.smithi089.stdout:134: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.088 INFO:tasks.rados.rados.0.smithi089.stdout:134: set_chunk oid 35 offset: 749056 length: 372736 target oid smithi08937929-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 749056 snap -1 2024-04-22T18:38:57.088 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.088 INFO:tasks.rados.rados.0.smithi089.stdout:122: finishing set_chunk to oid 23 2024-04-22T18:38:57.088 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-22T18:38:57.088 INFO:tasks.rados.rados.0.smithi089.stdout:122: done (15 left) 2024-04-22T18:38:57.088 INFO:tasks.rados.rados.0.smithi089.stdout:135: set_chunk oid 36 target oid 36 low_tier 2024-04-22T18:38:57.088 INFO:tasks.rados.rados.0.smithi089.stdout:135: set_chunk oid 36 offset: 30208 length: 1171968 target oid smithi08937929-36 low_tier offset: 30208 snap -1 2024-04-22T18:38:57.088 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.091 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:38:57.111 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:57.113+0000 7f7f44bb3640 -1 osd.4 28 handle_osd_map injecting map crc failure 2024-04-22T18:38:57.112 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.3 primary-affinity to 0.848003 (8555742) 2024-04-22T18:38:57.127 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:57.129+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:57.183 INFO:tasks.rados.rados.0.smithi089.stdout:116: finishing set_chunk to oid 17 2024-04-22T18:38:57.183 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-22T18:38:57.183 INFO:tasks.rados.rados.0.smithi089.stdout:113: finishing set_chunk to oid 14 2024-04-22T18:38:57.183 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-22T18:38:57.183 INFO:tasks.rados.rados.0.smithi089.stdout:124: finishing set_chunk to oid 25 2024-04-22T18:38:57.183 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 12 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-22T18:38:57.183 INFO:tasks.rados.rados.0.smithi089.stdout:113: done (15 left) 2024-04-22T18:38:57.183 INFO:tasks.rados.rados.0.smithi089.stdout:116: done (14 left) 2024-04-22T18:38:57.183 INFO:tasks.rados.rados.0.smithi089.stdout:124: done (13 left) 2024-04-22T18:38:57.183 INFO:tasks.rados.rados.0.smithi089.stdout:136: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.183 INFO:tasks.rados.rados.0.smithi089.stdout:136: set_chunk oid 37 offset: 55808 length: 1153536 target oid smithi08937929-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 55808 snap -1 2024-04-22T18:38:57.183 INFO:tasks.rados.rados.0.smithi089.stdout:137: set_chunk oid 38 target oid 38 low_tier 2024-04-22T18:38:57.184 INFO:tasks.rados.rados.0.smithi089.stdout:137: set_chunk oid 38 offset: 1810944 length: 46592 target oid smithi08937929-38 low_tier offset: 1810944 snap -1 2024-04-22T18:38:57.184 INFO:tasks.rados.rados.0.smithi089.stdout:138: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.184 INFO:tasks.rados.rados.0.smithi089.stdout:138: set_chunk oid 39 offset: 1501184 length: 653312 target oid smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1501184 snap -1 2024-04-22T18:38:57.184 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.184 INFO:tasks.rados.rados.0.smithi089.stdout:121: finishing set_chunk to oid 22 2024-04-22T18:38:57.184 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-22T18:38:57.184 INFO:tasks.rados.rados.0.smithi089.stdout:121: done (15 left) 2024-04-22T18:38:57.184 INFO:tasks.rados.rados.0.smithi089.stdout:139: set_chunk oid 40 target oid 40 low_tier 2024-04-22T18:38:57.184 INFO:tasks.rados.rados.0.smithi089.stdout:139: set_chunk oid 40 offset: 936960 length: 571904 target oid smithi08937929-40 low_tier offset: 936960 snap -1 2024-04-22T18:38:57.184 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.197 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:57.197 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:57.206 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:38:57.228 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:57.230+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:57.286 INFO:tasks.rados.rados.0.smithi089.stdout:130: finishing set_chunk to oid 31 2024-04-22T18:38:57.286 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 3 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-22T18:38:57.286 INFO:tasks.rados.rados.0.smithi089.stdout:130: done (15 left) 2024-04-22T18:38:57.286 INFO:tasks.rados.rados.0.smithi089.stdout:140: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.286 INFO:tasks.rados.rados.0.smithi089.stdout:140: set_chunk oid 41 offset: 233984 length: 1971200 target oid smithi08937929-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 233984 snap -1 2024-04-22T18:38:57.286 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.307 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:57.307 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:57.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:38:57.360 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:57.362+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:57.386 INFO:tasks.rados.rados.0.smithi089.stdout:126: finishing set_chunk to oid 27 2024-04-22T18:38:57.387 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 9 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-22T18:38:57.387 INFO:tasks.rados.rados.0.smithi089.stdout:133: finishing set_chunk to oid 34 2024-04-22T18:38:57.387 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-22T18:38:57.387 INFO:tasks.rados.rados.0.smithi089.stdout:126: done (15 left) 2024-04-22T18:38:57.387 INFO:tasks.rados.rados.0.smithi089.stdout:133: done (14 left) 2024-04-22T18:38:57.387 INFO:tasks.rados.rados.0.smithi089.stdout:141: set_chunk oid 42 target oid 42 low_tier 2024-04-22T18:38:57.387 INFO:tasks.rados.rados.0.smithi089.stdout:141: set_chunk oid 42 offset: 231424 length: 1875456 target oid smithi08937929-42 low_tier offset: 231424 snap -1 2024-04-22T18:38:57.387 INFO:tasks.rados.rados.0.smithi089.stdout:142: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.387 INFO:tasks.rados.rados.0.smithi089.stdout:142: set_chunk oid 43 offset: 92672 length: 1287168 target oid smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 92672 snap -1 2024-04-22T18:38:57.387 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.388 INFO:tasks.rados.rados.0.smithi089.stdout:138: finishing set_chunk to oid 39 2024-04-22T18:38:57.389 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 15 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-22T18:38:57.389 INFO:tasks.rados.rados.0.smithi089.stdout:138: done (15 left) 2024-04-22T18:38:57.389 INFO:tasks.rados.rados.0.smithi089.stdout:143: set_chunk oid 44 target oid 44 low_tier 2024-04-22T18:38:57.389 INFO:tasks.rados.rados.0.smithi089.stdout:143: set_chunk oid 44 offset: 884224 length: 1102848 target oid smithi08937929-44 low_tier offset: 884224 snap -1 2024-04-22T18:38:57.389 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.389 INFO:tasks.rados.rados.0.smithi089.stdout:125: finishing set_chunk to oid 26 2024-04-22T18:38:57.389 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-22T18:38:57.389 INFO:tasks.rados.rados.0.smithi089.stdout:125: done (15 left) 2024-04-22T18:38:57.389 INFO:tasks.rados.rados.0.smithi089.stdout:144: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.389 INFO:tasks.rados.rados.0.smithi089.stdout:144: set_chunk oid 45 offset: 886272 length: 1440768 target oid smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 886272 snap -1 2024-04-22T18:38:57.389 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.419 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:57.420 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:57.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:38:57.430 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:57.431+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:57.491 INFO:tasks.rados.rados.0.smithi089.stdout:136: finishing set_chunk to oid 37 2024-04-22T18:38:57.491 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 9 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-22T18:38:57.491 INFO:tasks.rados.rados.0.smithi089.stdout:136: done (15 left) 2024-04-22T18:38:57.491 INFO:tasks.rados.rados.0.smithi089.stdout:145: set_chunk oid 46 target oid 46 low_tier 2024-04-22T18:38:57.491 INFO:tasks.rados.rados.0.smithi089.stdout:145: set_chunk oid 46 offset: 1321472 length: 1080832 target oid smithi08937929-46 low_tier offset: 1321472 snap -1 2024-04-22T18:38:57.491 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.491 INFO:tasks.rados.rados.0.smithi089.stdout:129: finishing set_chunk to oid 30 2024-04-22T18:38:57.491 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-22T18:38:57.491 INFO:tasks.rados.rados.0.smithi089.stdout:129: done (15 left) 2024-04-22T18:38:57.491 INFO:tasks.rados.rados.0.smithi089.stdout:146: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.492 INFO:tasks.rados.rados.0.smithi089.stdout:146: set_chunk oid 47 offset: 313344 length: 2379776 target oid smithi08937929-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 313344 snap -1 2024-04-22T18:38:57.492 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.492 INFO:tasks.rados.rados.0.smithi089.stdout:123: finishing set_chunk to oid 24 2024-04-22T18:38:57.492 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 12 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-22T18:38:57.492 INFO:tasks.rados.rados.0.smithi089.stdout:123: done (15 left) 2024-04-22T18:38:57.492 INFO:tasks.rados.rados.0.smithi089.stdout:147: set_chunk oid 48 target oid 48 low_tier 2024-04-22T18:38:57.492 INFO:tasks.rados.rados.0.smithi089.stdout:147: set_chunk oid 48 offset: 232448 length: 2506752 target oid smithi08937929-48 low_tier offset: 232448 snap -1 2024-04-22T18:38:57.492 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.492 INFO:tasks.rados.rados.0.smithi089.stdout:143: finishing set_chunk to oid 44 2024-04-22T18:38:57.492 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-22T18:38:57.492 INFO:tasks.rados.rados.0.smithi089.stdout:143: done (15 left) 2024-04-22T18:38:57.492 INFO:tasks.rados.rados.0.smithi089.stdout:148: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.492 INFO:tasks.rados.rados.0.smithi089.stdout:148: set_chunk oid 49 offset: 1273344 length: 1325568 target oid smithi08937929-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1273344 snap -1 2024-04-22T18:38:57.492 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.530 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:57.532+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:57.573 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:57.573 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:57.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:38:57.587 INFO:tasks.rados.rados.0.smithi089.stdout:132: finishing set_chunk to oid 33 2024-04-22T18:38:57.587 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-22T18:38:57.587 INFO:tasks.rados.rados.0.smithi089.stdout:132: done (15 left) 2024-04-22T18:38:57.587 INFO:tasks.rados.rados.0.smithi089.stdout:149: set_chunk oid 50 target oid 50 low_tier 2024-04-22T18:38:57.588 INFO:tasks.rados.rados.0.smithi089.stdout:149: set_chunk oid 50 offset: 516608 length: 2073600 target oid smithi08937929-50 low_tier offset: 516608 snap -1 2024-04-22T18:38:57.588 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.588 INFO:tasks.rados.rados.0.smithi089.stdout:139: finishing set_chunk to oid 40 2024-04-22T18:38:57.588 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 3 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-22T18:38:57.588 INFO:tasks.rados.rados.0.smithi089.stdout:139: done (15 left) 2024-04-22T18:38:57.588 INFO:tasks.rados.rados.0.smithi089.stdout:150: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.588 INFO:tasks.rados.rados.0.smithi089.stdout:150: set_chunk oid 1 offset: 488960 length: 488960 target oid smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 488960 snap -1 2024-04-22T18:38:57.588 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.591 INFO:tasks.rados.rados.0.smithi089.stdout:147: finishing set_chunk to oid 48 2024-04-22T18:38:57.591 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-22T18:38:57.591 INFO:tasks.rados.rados.0.smithi089.stdout:147: done (15 left) 2024-04-22T18:38:57.592 INFO:tasks.rados.rados.0.smithi089.stdout:151: set_chunk oid 2 target oid 2 low_tier 2024-04-22T18:38:57.592 INFO:tasks.rados.rados.0.smithi089.stdout:151: set_chunk oid 2 offset: 1441792 length: 455680 target oid smithi08937929-2 low_tier offset: 1441792 snap -1 2024-04-22T18:38:57.592 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.592 INFO:tasks.rados.rados.0.smithi089.stdout:151: finishing set_chunk to oid 2 2024-04-22T18:38:57.592 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 1441792 length: 455680 2024-04-22T18:38:57.592 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-22T18:38:57.592 INFO:tasks.rados.rados.0.smithi089.stdout:151: done (15 left) 2024-04-22T18:38:57.592 INFO:tasks.rados.rados.0.smithi089.stdout:152: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.592 INFO:tasks.rados.rados.0.smithi089.stdout:152: set_chunk oid 3 offset: 1303552 length: 309760 target oid smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1303552 snap -1 2024-04-22T18:38:57.592 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.652 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:38:57.666 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:57.668+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:57.688 INFO:tasks.rados.rados.0.smithi089.stdout:127: finishing set_chunk to oid 28 2024-04-22T18:38:57.688 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 15 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-22T18:38:57.688 INFO:tasks.rados.rados.0.smithi089.stdout:127: done (15 left) 2024-04-22T18:38:57.688 INFO:tasks.rados.rados.0.smithi089.stdout:153: set_chunk oid 4 target oid 4 low_tier 2024-04-22T18:38:57.688 INFO:tasks.rados.rados.0.smithi089.stdout:153: set_chunk oid 4 offset: 351232 length: 98304 target oid smithi08937929-4 low_tier offset: 351232 snap -1 2024-04-22T18:38:57.688 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.689 INFO:tasks.rados.rados.0.smithi089.stdout:141: finishing set_chunk to oid 42 2024-04-22T18:38:57.689 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 3 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-22T18:38:57.689 INFO:tasks.rados.rados.0.smithi089.stdout:141: done (15 left) 2024-04-22T18:38:57.689 INFO:tasks.rados.rados.0.smithi089.stdout:154: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.689 INFO:tasks.rados.rados.0.smithi089.stdout:154: set_chunk oid 5 offset: 230912 length: 95232 target oid smithi08937929-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 230912 snap -1 2024-04-22T18:38:57.689 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.690 INFO:tasks.rados.rados.0.smithi089.stdout:134: finishing set_chunk to oid 35 2024-04-22T18:38:57.690 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 9 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-22T18:38:57.690 INFO:tasks.rados.rados.0.smithi089.stdout:134: done (15 left) 2024-04-22T18:38:57.690 INFO:tasks.rados.rados.0.smithi089.stdout:155: set_chunk oid 6 target oid 6 low_tier 2024-04-22T18:38:57.690 INFO:tasks.rados.rados.0.smithi089.stdout:155: set_chunk oid 6 offset: 199680 length: 29184 target oid smithi08937929-6 low_tier offset: 199680 snap -1 2024-04-22T18:38:57.691 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.692 INFO:tasks.rados.rados.0.smithi089.stdout:148: finishing set_chunk to oid 49 2024-04-22T18:38:57.692 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:38:57.692 INFO:tasks.rados.rados.0.smithi089.stdout:148: done (15 left) 2024-04-22T18:38:57.692 INFO:tasks.rados.rados.0.smithi089.stdout:156: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.692 INFO:tasks.rados.rados.0.smithi089.stdout:156: set_chunk oid 7 offset: 1037312 length: 801280 target oid smithi08937929-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1037312 snap -1 2024-04-22T18:38:57.693 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.724 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:38:57.724 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:38:57.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:38:57.733 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:57.733+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:38:57.769 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:38:57.770 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:38:57.786 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:38:57.786 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:38:57.789 INFO:tasks.rados.rados.0.smithi089.stdout:140: finishing set_chunk to oid 41 2024-04-22T18:38:57.789 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 18 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-22T18:38:57.789 INFO:tasks.rados.rados.0.smithi089.stdout:140: done (15 left) 2024-04-22T18:38:57.789 INFO:tasks.rados.rados.0.smithi089.stdout:157: set_chunk oid 8 target oid 8 low_tier 2024-04-22T18:38:57.789 INFO:tasks.rados.rados.0.smithi089.stdout:157: set_chunk oid 8 offset: 936448 length: 605696 target oid smithi08937929-8 low_tier offset: 936448 snap -1 2024-04-22T18:38:57.789 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.791 INFO:tasks.rados.rados.0.smithi089.stdout:144: finishing set_chunk to oid 45 2024-04-22T18:38:57.791 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-22T18:38:57.791 INFO:tasks.rados.rados.0.smithi089.stdout:144: done (15 left) 2024-04-22T18:38:57.791 INFO:tasks.rados.rados.0.smithi089.stdout:158: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.791 INFO:tasks.rados.rados.0.smithi089.stdout:158: set_chunk oid 9 offset: 792064 length: 1281536 target oid smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 792064 snap -1 2024-04-22T18:38:57.791 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.803 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:38:57.803 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:38:57.818 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:38:57.818 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:38:57.834 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:57.836+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:57.835 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:38:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:38:57.851 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:38:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:38:57.867 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:38:57.867 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:38:57.878 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:57.878 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:38:57.878 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:38:57.878 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:57.883 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:38:57.883 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:38:57.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:38:57.891 INFO:tasks.rados.rados.0.smithi089.stdout:145: finishing set_chunk to oid 46 2024-04-22T18:38:57.891 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-22T18:38:57.891 INFO:tasks.rados.rados.0.smithi089.stdout:154: finishing set_chunk to oid 5 2024-04-22T18:38:57.891 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-22T18:38:57.892 INFO:tasks.rados.rados.0.smithi089.stdout:153: finishing set_chunk to oid 4 2024-04-22T18:38:57.892 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-22T18:38:57.892 INFO:tasks.rados.rados.0.smithi089.stdout:145: done (15 left) 2024-04-22T18:38:57.892 INFO:tasks.rados.rados.0.smithi089.stdout:153: done (14 left) 2024-04-22T18:38:57.892 INFO:tasks.rados.rados.0.smithi089.stdout:154: done (13 left) 2024-04-22T18:38:57.892 INFO:tasks.rados.rados.0.smithi089.stdout:159: set_chunk oid 10 target oid 10 low_tier 2024-04-22T18:38:57.892 INFO:tasks.rados.rados.0.smithi089.stdout:159: set_chunk oid 10 offset: 679424 length: 1135104 target oid smithi08937929-10 low_tier offset: 679424 snap -1 2024-04-22T18:38:57.892 INFO:tasks.rados.rados.0.smithi089.stdout:160: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.892 INFO:tasks.rados.rados.0.smithi089.stdout:160: set_chunk oid 11 offset: 209408 length: 1241600 target oid smithi08937929-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 209408 snap -1 2024-04-22T18:38:57.892 INFO:tasks.rados.rados.0.smithi089.stdout:161: set_chunk oid 12 target oid 12 low_tier 2024-04-22T18:38:57.892 INFO:tasks.rados.rados.0.smithi089.stdout:161: set_chunk oid 12 offset: 140800 length: 160768 target oid smithi08937929-12 low_tier offset: 140800 snap -1 2024-04-22T18:38:57.892 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.892 INFO:tasks.rados.rados.0.smithi089.stdout:157: finishing set_chunk to oid 8 2024-04-22T18:38:57.892 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 936448 length: 605696 2024-04-22T18:38:57.892 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout:157: done (15 left) 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout:162: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout:162: set_chunk oid 13 offset: 549376 length: 1020416 target oid smithi08937929-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 549376 snap -1 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout:161: finishing set_chunk to oid 12 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 140800 length: 160768 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout:161: done (15 left) 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout:163: set_chunk oid 14 target oid 14 low_tier 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout:163: set_chunk oid 14 offset: 1708032 length: 227328 target oid smithi08937929-14 low_tier offset: 1708032 snap -1 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout:146: finishing set_chunk to oid 47 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout:146: done (15 left) 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout:164: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.893 INFO:tasks.rados.rados.0.smithi089.stdout:164: set_chunk oid 15 offset: 744960 length: 670720 target oid smithi08937929-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 744960 snap -1 2024-04-22T18:38:57.894 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.894 INFO:tasks.rados.rados.0.smithi089.stdout:162: finishing set_chunk to oid 13 2024-04-22T18:38:57.894 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 549376 length: 1020416 2024-04-22T18:38:57.894 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-22T18:38:57.894 INFO:tasks.rados.rados.0.smithi089.stdout:162: done (15 left) 2024-04-22T18:38:57.894 INFO:tasks.rados.rados.0.smithi089.stdout:165: set_chunk oid 16 target oid 16 low_tier 2024-04-22T18:38:57.894 INFO:tasks.rados.rados.0.smithi089.stdout:165: set_chunk oid 16 offset: 1347072 length: 205824 target oid smithi08937929-16 low_tier offset: 1347072 snap -1 2024-04-22T18:38:57.894 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.894 INFO:tasks.rados.rados.0.smithi089.stdout:150: finishing set_chunk to oid 1 2024-04-22T18:38:57.894 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-22T18:38:57.894 INFO:tasks.rados.rados.0.smithi089.stdout:150: done (15 left) 2024-04-22T18:38:57.895 INFO:tasks.rados.rados.0.smithi089.stdout:166: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.895 INFO:tasks.rados.rados.0.smithi089.stdout:166: set_chunk oid 17 offset: 630784 length: 1114112 target oid smithi08937929-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 630784 snap -1 2024-04-22T18:38:57.895 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.895 INFO:tasks.rados.rados.0.smithi089.stdout:164: finishing set_chunk to oid 15 2024-04-22T18:38:57.895 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 744960 length: 670720 2024-04-22T18:38:57.895 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-22T18:38:57.895 INFO:tasks.rados.rados.0.smithi089.stdout:164: done (15 left) 2024-04-22T18:38:57.895 INFO:tasks.rados.rados.0.smithi089.stdout:167: set_chunk oid 18 target oid 18 low_tier 2024-04-22T18:38:57.895 INFO:tasks.rados.rados.0.smithi089.stdout:167: set_chunk oid 18 offset: 1237504 length: 858624 target oid smithi08937929-18 low_tier offset: 1237504 snap -1 2024-04-22T18:38:57.895 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.937 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:57.938+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:57.993 INFO:tasks.rados.rados.0.smithi089.stdout:159: finishing set_chunk to oid 10 2024-04-22T18:38:57.993 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 679424 length: 1135104 2024-04-22T18:38:57.993 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-22T18:38:57.993 INFO:tasks.rados.rados.0.smithi089.stdout:155: finishing set_chunk to oid 6 2024-04-22T18:38:57.993 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-22T18:38:57.993 INFO:tasks.rados.rados.0.smithi089.stdout:155: done (15 left) 2024-04-22T18:38:57.993 INFO:tasks.rados.rados.0.smithi089.stdout:159: done (14 left) 2024-04-22T18:38:57.994 INFO:tasks.rados.rados.0.smithi089.stdout:168: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.994 INFO:tasks.rados.rados.0.smithi089.stdout:168: set_chunk oid 19 offset: 716288 length: 474624 target oid smithi08937929-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 716288 snap -1 2024-04-22T18:38:57.994 INFO:tasks.rados.rados.0.smithi089.stdout:169: set_chunk oid 20 target oid 20 low_tier 2024-04-22T18:38:57.994 INFO:tasks.rados.rados.0.smithi089.stdout:169: set_chunk oid 20 offset: 1180160 length: 737792 target oid smithi08937929-20 low_tier offset: 1180160 snap -1 2024-04-22T18:38:57.994 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.994 INFO:tasks.rados.rados.0.smithi089.stdout:160: finishing set_chunk to oid 11 2024-04-22T18:38:57.994 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 209408 length: 1241600 2024-04-22T18:38:57.994 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-22T18:38:57.994 INFO:tasks.rados.rados.0.smithi089.stdout:160: done (15 left) 2024-04-22T18:38:57.994 INFO:tasks.rados.rados.0.smithi089.stdout:170: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.994 INFO:tasks.rados.rados.0.smithi089.stdout:170: set_chunk oid 21 offset: 40448 length: 1603584 target oid smithi08937929-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 40448 snap -1 2024-04-22T18:38:57.994 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.994 INFO:tasks.rados.rados.0.smithi089.stdout:169: finishing set_chunk to oid 20 2024-04-22T18:38:57.995 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 1180160 length: 737792 2024-04-22T18:38:57.995 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-22T18:38:57.995 INFO:tasks.rados.rados.0.smithi089.stdout:169: done (15 left) 2024-04-22T18:38:57.995 INFO:tasks.rados.rados.0.smithi089.stdout:171: set_chunk oid 22 target oid 22 low_tier 2024-04-22T18:38:57.995 INFO:tasks.rados.rados.0.smithi089.stdout:171: set_chunk oid 22 offset: 228864 length: 1055232 target oid smithi08937929-22 low_tier offset: 228864 snap -1 2024-04-22T18:38:57.995 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.995 INFO:tasks.rados.rados.0.smithi089.stdout:170: finishing set_chunk to oid 21 2024-04-22T18:38:57.995 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 40448 length: 1603584 2024-04-22T18:38:57.995 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-22T18:38:57.995 INFO:tasks.rados.rados.0.smithi089.stdout:170: done (15 left) 2024-04-22T18:38:57.995 INFO:tasks.rados.rados.0.smithi089.stdout:172: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:57.995 INFO:tasks.rados.rados.0.smithi089.stdout:172: set_chunk oid 23 offset: 369152 length: 826880 target oid smithi08937929-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 369152 snap -1 2024-04-22T18:38:57.995 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:57.995 INFO:tasks.rados.rados.0.smithi089.stdout:171: finishing set_chunk to oid 22 2024-04-22T18:38:57.996 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 228864 length: 1055232 2024-04-22T18:38:57.996 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-22T18:38:57.996 INFO:tasks.rados.rados.0.smithi089.stdout:171: done (15 left) 2024-04-22T18:38:57.996 INFO:tasks.rados.rados.0.smithi089.stdout:173: set_chunk oid 24 target oid 24 low_tier 2024-04-22T18:38:57.996 INFO:tasks.rados.rados.0.smithi089.stdout:173: set_chunk oid 24 offset: 845824 length: 704512 target oid smithi08937929-24 low_tier offset: 845824 snap -1 2024-04-22T18:38:57.996 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.029 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:58.029 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:38:58.029 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:38:58.029 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:38:58.029 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:58.035 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:58.037+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:58.038 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:38:58.092 INFO:tasks.rados.rados.0.smithi089.stdout:163: finishing set_chunk to oid 14 2024-04-22T18:38:58.092 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 1708032 length: 227328 2024-04-22T18:38:58.092 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-22T18:38:58.092 INFO:tasks.rados.rados.0.smithi089.stdout:166: finishing set_chunk to oid 17 2024-04-22T18:38:58.093 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 630784 length: 1114112 2024-04-22T18:38:58.093 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-22T18:38:58.093 INFO:tasks.rados.rados.0.smithi089.stdout:163: done (15 left) 2024-04-22T18:38:58.093 INFO:tasks.rados.rados.0.smithi089.stdout:166: done (14 left) 2024-04-22T18:38:58.093 INFO:tasks.rados.rados.0.smithi089.stdout:174: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.093 INFO:tasks.rados.rados.0.smithi089.stdout:174: set_chunk oid 25 offset: 123904 length: 2000896 target oid smithi08937929-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 123904 snap -1 2024-04-22T18:38:58.093 INFO:tasks.rados.rados.0.smithi089.stdout:175: set_chunk oid 26 target oid 26 low_tier 2024-04-22T18:38:58.093 INFO:tasks.rados.rados.0.smithi089.stdout:175: set_chunk oid 26 offset: 321536 length: 1422848 target oid smithi08937929-26 low_tier offset: 321536 snap -1 2024-04-22T18:38:58.093 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.093 INFO:tasks.rados.rados.0.smithi089.stdout:167: finishing set_chunk to oid 18 2024-04-22T18:38:58.093 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1237504 length: 858624 2024-04-22T18:38:58.093 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-22T18:38:58.093 INFO:tasks.rados.rados.0.smithi089.stdout:167: done (15 left) 2024-04-22T18:38:58.094 INFO:tasks.rados.rados.0.smithi089.stdout:176: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.094 INFO:tasks.rados.rados.0.smithi089.stdout:176: set_chunk oid 27 offset: 2230784 length: 141312 target oid smithi08937929-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2230784 snap -1 2024-04-22T18:38:58.094 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.094 INFO:tasks.rados.rados.0.smithi089.stdout:149: finishing set_chunk to oid 50 2024-04-22T18:38:58.094 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 21 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-22T18:38:58.094 INFO:tasks.rados.rados.0.smithi089.stdout:149: done (15 left) 2024-04-22T18:38:58.094 INFO:tasks.rados.rados.0.smithi089.stdout:177: set_chunk oid 28 target oid 28 low_tier 2024-04-22T18:38:58.094 INFO:tasks.rados.rados.0.smithi089.stdout:177: set_chunk oid 28 offset: 642048 length: 192512 target oid smithi08937929-28 low_tier offset: 642048 snap -1 2024-04-22T18:38:58.094 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.094 INFO:tasks.rados.rados.0.smithi089.stdout:174: finishing set_chunk to oid 25 2024-04-22T18:38:58.094 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 123904 length: 2000896 2024-04-22T18:38:58.094 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-22T18:38:58.094 INFO:tasks.rados.rados.0.smithi089.stdout:174: done (15 left) 2024-04-22T18:38:58.094 INFO:tasks.rados.rados.0.smithi089.stdout:177 retry set_chunk ! 2024-04-22T18:38:58.095 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 15 2024-04-22T18:38:58.095 INFO:tasks.rados.rados.0.smithi089.stdout:172: finishing set_chunk to oid 23 2024-04-22T18:38:58.095 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 369152 length: 826880 2024-04-22T18:38:58.095 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-22T18:38:58.095 INFO:tasks.rados.rados.0.smithi089.stdout:172: done (14 left) 2024-04-22T18:38:58.095 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 14 2024-04-22T18:38:58.095 INFO:tasks.rados.rados.0.smithi089.stdout:175: finishing set_chunk to oid 26 2024-04-22T18:38:58.095 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 321536 length: 1422848 2024-04-22T18:38:58.095 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-22T18:38:58.095 INFO:tasks.rados.rados.0.smithi089.stdout:175: done (13 left) 2024-04-22T18:38:58.095 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 13 2024-04-22T18:38:58.095 INFO:tasks.rados.rados.0.smithi089.stdout:177: finishing set_chunk to oid 28 2024-04-22T18:38:58.095 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 642048 length: 192512 2024-04-22T18:38:58.095 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-22T18:38:58.096 INFO:tasks.rados.rados.0.smithi089.stdout:177: done (12 left) 2024-04-22T18:38:58.096 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 12 2024-04-22T18:38:58.096 INFO:tasks.rados.rados.0.smithi089.stdout:156: finishing set_chunk to oid 7 2024-04-22T18:38:58.096 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-22T18:38:58.096 INFO:tasks.rados.rados.0.smithi089.stdout:156: done (11 left) 2024-04-22T18:38:58.096 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 11 2024-04-22T18:38:58.121 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:58.121+0000 7f8605f32640 -1 osd.3 29 handle_osd_map injecting map crc failure 2024-04-22T18:38:58.128 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:38:58.136 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:58.138+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:58.182 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:58.182 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:38:58.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:38:58.182 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:38:58.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.183 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:38:58.183 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:38:58.183 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.0890222080000003, 2024-04-22T18:38:58.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:38:58.183 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.183 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.183 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.183 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.183 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.183 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:38:58.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.183 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:38:58.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:38:58.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:38:58.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:38:58.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:38:58.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:38:58.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:38:58.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:38:58.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:38:58.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:38:58.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:38:58.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:38:58.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:38:58.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:38:58.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:38:58.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:38:58.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:38:58.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:38:58.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:38:58.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:38:58.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:38:58.188 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.188 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.188 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.188 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:17 4.1 e27/20 4:809dfd21:::smithi08937929-5:head v 27'1, mlcod=0'0)", 2024-04-22T18:38:58.188 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.141796+0000", 2024-04-22T18:38:58.188 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.0410863209999999, 2024-04-22T18:38:58.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105540862, 2024-04-22T18:38:58.188 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.189 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.189 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.141796+0000", 2024-04-22T18:38:58.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.141796+0000", 2024-04-22T18:38:58.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.141795+0000", 2024-04-22T18:38:58.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:58.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.142224+0000", 2024-04-22T18:38:58.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000429264 2024-04-22T18:38:58.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.142236+0000", 2024-04-22T18:38:58.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1958e-05 2024-04-22T18:38:58.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.142245+0000", 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6850000000000007e-06 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.142298+0000", 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2747000000000002e-05 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.142329+0000", 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1287e-05 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247306+0000", 2024-04-22T18:38:58.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104976713 2024-04-22T18:38:58.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247336+0000", 2024-04-22T18:38:58.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0676999999999998e-05 2024-04-22T18:38:58.192 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.192 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.192 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.192 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2 4.8 4:12d9f826:::smithi08937929-1:head [write 1542129~474678 in=474678b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:38:58.192 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:38:58.192 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.9883109380000001, 2024-04-22T18:38:58.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20435667099999999, 2024-04-22T18:38:58.193 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.193 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.193 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.193 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.193 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.193 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2 2024-04-22T18:38:58.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.193 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:38:58.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:38:58.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194568+0000", 2024-04-22T18:38:58.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:38:58.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194869+0000", 2024-04-22T18:38:58.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030115200000000001 2024-04-22T18:38:58.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194879+0000", 2024-04-22T18:38:58.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8479999999999996e-06 2024-04-22T18:38:58.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294954+0000", 2024-04-22T18:38:58.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100074702 2024-04-22T18:38:58.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296019+0000", 2024-04-22T18:38:58.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010649189999999999 2024-04-22T18:38:58.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296086+0000", 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7261000000000005e-05 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296173+0000", 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6935999999999996e-05 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297196+0000", 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023415 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398879+0000", 2024-04-22T18:38:58.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101682563 2024-04-22T18:38:58.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398906+0000", 2024-04-22T18:38:58.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6526999999999999e-05 2024-04-22T18:38:58.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398928+0000", 2024-04-22T18:38:58.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2045e-05 2024-04-22T18:38:58.197 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.197 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.197 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.198 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:53 4.3 e27/20 4:ca46fde9:::smithi08937929-14:head v 27'1, mlcod=0'0)", 2024-04-22T18:38:58.198 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:38:58.198 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.935063735, 2024-04-22T18:38:58.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20214333400000001, 2024-04-22T18:38:58.198 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.198 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.198 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:38:58.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967296 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248504+0000", 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000686091 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248512+0000", 2024-04-22T18:38:58.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1850000000000004e-06 2024-04-22T18:38:58.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248517+0000", 2024-04-22T18:38:58.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5469999999999998e-06 2024-04-22T18:38:58.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.348632+0000", 2024-04-22T18:38:58.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100115069 2024-04-22T18:38:58.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.348644+0000", 2024-04-22T18:38:58.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2001000000000001e-05 2024-04-22T18:38:58.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449930+0000", 2024-04-22T18:38:58.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101286574 2024-04-22T18:38:58.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449961+0000", 2024-04-22T18:38:58.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1046000000000001e-05 2024-04-22T18:38:58.202 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.202 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.202 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.202 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:54 4.3 e27/20 4:ca46fde9:::smithi08937929-14:head v 27'2, mlcod=0'0)", 2024-04-22T18:38:58.202 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.252046+0000", 2024-04-22T18:38:58.202 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.9308361920000001, 2024-04-22T18:38:58.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19793179499999999, 2024-04-22T18:38:58.202 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.202 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.202 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252046+0000", 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252046+0000", 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252045+0000", 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252215+0000", 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017004499999999999 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252221+0000", 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.677e-06 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252225+0000", 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7730000000000001e-06 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.349338+0000", 2024-04-22T18:38:58.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.097113853999999999 2024-04-22T18:38:58.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.349350+0000", 2024-04-22T18:38:58.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1268e-05 2024-04-22T18:38:58.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449967+0000", 2024-04-22T18:38:58.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100616754 2024-04-22T18:38:58.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449977+0000", 2024-04-22T18:38:58.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0978999999999999e-05 2024-04-22T18:38:58.206 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.206 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.206 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.206 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:55 4.3 e27/20)", 2024-04-22T18:38:58.206 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:38:58.206 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.9302685400000001, 2024-04-22T18:38:58.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19766915500000001, 2024-04-22T18:38:58.206 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.206 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.206 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:38:58.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:38:58.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:38:58.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:58.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252616+0000", 2024-04-22T18:38:58.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1889999999999999e-06 2024-04-22T18:38:58.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252622+0000", 2024-04-22T18:38:58.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0589999999999999e-06 2024-04-22T18:38:58.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252625+0000", 2024-04-22T18:38:58.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3239999999999999e-06 2024-04-22T18:38:58.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449805+0000", 2024-04-22T18:38:58.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19717992500000001 2024-04-22T18:38:58.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449819+0000", 2024-04-22T18:38:58.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.367e-05 2024-04-22T18:38:58.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.450269+0000", 2024-04-22T18:38:58.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045026300000000001 2024-04-22T18:38:58.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.450282+0000", 2024-04-22T18:38:58.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3229e-05 2024-04-22T18:38:58.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.210 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.210 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=62b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:38:58.210 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:38:58.210 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.8879099840000002, 2024-04-22T18:38:58.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20408478299999999, 2024-04-22T18:38:58.210 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.210 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294970+0000", 2024-04-22T18:38:58.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:38:58.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294983+0000", 2024-04-22T18:38:58.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3429e-05 2024-04-22T18:38:58.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294993+0000", 2024-04-22T18:38:58.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0416e-05 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295007+0000", 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3742e-05 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296793+0000", 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017860420000000001 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296841+0000", 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7706e-05 2024-04-22T18:38:58.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:38:58.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296913+0000", 2024-04-22T18:38:58.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2578999999999997e-05 2024-04-22T18:38:58.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297946+0000", 2024-04-22T18:38:58.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001032802 2024-04-22T18:38:58.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498963+0000", 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20101640400000001 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498998+0000", 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5759000000000001e-05 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499057+0000", 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8270999999999998e-05 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.215 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:38:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:38:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.8878695560000001, 2024-04-22T18:38:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20418823899999999, 2024-04-22T18:38:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4 2024-04-22T18:38:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:38:58.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295011+0000", 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295016+0000", 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2400000000000001e-06 2024-04-22T18:38:58.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295022+0000", 2024-04-22T18:38:58.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8519999999999998e-06 2024-04-22T18:38:58.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295027+0000", 2024-04-22T18:38:58.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9749999999999998e-06 2024-04-22T18:38:58.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297126+0000", 2024-04-22T18:38:58.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020981070000000001 2024-04-22T18:38:58.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:38:58.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297167+0000", 2024-04-22T18:38:58.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1842999999999999e-05 2024-04-22T18:38:58.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499096+0000", 2024-04-22T18:38:58.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201928846 2024-04-22T18:38:58.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499142+0000", 2024-04-22T18:38:58.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6152000000000001e-05 2024-04-22T18:38:58.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499201+0000", 2024-04-22T18:38:58.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8190999999999999e-05 2024-04-22T18:38:58.220 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.220 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.220 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.220 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:73 4.1 e27/20 4:8db271f9:::smithi08937929-19:head v 27'4, mlcod=27'2)", 2024-04-22T18:38:58.220 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.457462+0000", 2024-04-22T18:38:58.220 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.7254203339999998, 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10421660100000001, 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.457462+0000", 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.457462+0000", 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.457460+0000", 2024-04-22T18:38:58.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:38:58.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.457640+0000", 2024-04-22T18:38:58.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018060299999999999 2024-04-22T18:38:58.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.457653+0000", 2024-04-22T18:38:58.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2615e-05 2024-04-22T18:38:58.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.457660+0000", 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4640000000000003e-06 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.557752+0000", 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100092245 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.557786+0000", 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4180000000000001e-05 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.561650+0000", 2024-04-22T18:38:58.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038637060000000002 2024-04-22T18:38:58.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.561678+0000", 2024-04-22T18:38:58.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8493999999999999e-05 2024-04-22T18:38:58.224 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.224 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.224 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.224 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:81 4.8 4:1bc4668a:::smithi08937929-21:head [write 736140~537686,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.224 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.549839+0000", 2024-04-22T18:38:58.224 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.6330425589999997, 2024-04-22T18:38:58.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20180494099999999, 2024-04-22T18:38:58.225 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.225 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.225 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 81 2024-04-22T18:38:58.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.225 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.549839+0000", 2024-04-22T18:38:58.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.549839+0000", 2024-04-22T18:38:58.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.549837+0000", 2024-04-22T18:38:58.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:38:58.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.550355+0000", 2024-04-22T18:38:58.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051818999999999997 2024-04-22T18:38:58.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.550370+0000", 2024-04-22T18:38:58.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5073e-05 2024-04-22T18:38:58.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.550378+0000", 2024-04-22T18:38:58.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9580000000000007e-06 2024-04-22T18:38:58.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.650511+0000", 2024-04-22T18:38:58.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100132675 2024-04-22T18:38:58.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.650590+0000", 2024-04-22T18:38:58.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8949999999999995e-05 2024-04-22T18:38:58.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:38:58.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.650684+0000", 2024-04-22T18:38:58.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4846999999999995e-05 2024-04-22T18:38:58.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.652918+0000", 2024-04-22T18:38:58.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022340049999999998 2024-04-22T18:38:58.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.751590+0000", 2024-04-22T18:38:58.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098671777000000002 2024-04-22T18:38:58.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.751620+0000", 2024-04-22T18:38:58.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0190999999999999e-05 2024-04-22T18:38:58.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.751644+0000", 2024-04-22T18:38:58.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3841000000000001e-05 2024-04-22T18:38:58.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.230 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.230 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:82 4.8 4:1bc4668a:::smithi08937929-21:head [write 2020581~315559] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.230 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.555572+0000", 2024-04-22T18:38:58.230 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.6273100660000002, 2024-04-22T18:38:58.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19891041200000001, 2024-04-22T18:38:58.230 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.230 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.230 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.231 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.231 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.231 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 82 2024-04-22T18:38:58.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.231 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555572+0000", 2024-04-22T18:38:58.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555572+0000", 2024-04-22T18:38:58.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555570+0000", 2024-04-22T18:38:58.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:58.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555825+0000", 2024-04-22T18:38:58.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00025443699999999999 2024-04-22T18:38:58.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555836+0000", 2024-04-22T18:38:58.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1693e-05 2024-04-22T18:38:58.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555843+0000", 2024-04-22T18:38:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.861e-06 2024-04-22T18:38:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.651510+0000", 2024-04-22T18:38:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.095666449000000001 2024-04-22T18:38:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.651598+0000", 2024-04-22T18:38:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7989000000000001e-05 2024-04-22T18:38:58.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:38:58.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.651726+0000", 2024-04-22T18:38:58.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012811100000000001 2024-04-22T18:38:58.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.653977+0000", 2024-04-22T18:38:58.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002251074 2024-04-22T18:38:58.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754316+0000", 2024-04-22T18:38:58.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10033908599999999 2024-04-22T18:38:58.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754449+0000", 2024-04-22T18:38:58.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000133497 2024-04-22T18:38:58.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754482+0000", 2024-04-22T18:38:58.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2830000000000002e-05 2024-04-22T18:38:58.235 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.235 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.235 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.236 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:83 4.8 4:1bc4668a:::smithi08937929-21:head [setxattr _header (56) in=7b,truncate 2336140] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.236 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.555848+0000", 2024-04-22T18:38:58.236 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.6270335760000001, 2024-04-22T18:38:58.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19872846999999999, 2024-04-22T18:38:58.236 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.236 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.236 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.236 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.236 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.236 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 83 2024-04-22T18:38:58.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.236 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555848+0000", 2024-04-22T18:38:58.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555848+0000", 2024-04-22T18:38:58.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555848+0000", 2024-04-22T18:38:58.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:58.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555852+0000", 2024-04-22T18:38:58.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.69e-06 2024-04-22T18:38:58.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555856+0000", 2024-04-22T18:38:58.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.055e-06 2024-04-22T18:38:58.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555859+0000", 2024-04-22T18:38:58.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1269999999999999e-06 2024-04-22T18:38:58.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.652495+0000", 2024-04-22T18:38:58.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.096636298999999995 2024-04-22T18:38:58.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.652599+0000", 2024-04-22T18:38:58.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010438300000000001 2024-04-22T18:38:58.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:38:58.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.652736+0000", 2024-04-22T18:38:58.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013642600000000001 2024-04-22T18:38:58.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.654000+0000", 2024-04-22T18:38:58.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012638370000000001 2024-04-22T18:38:58.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754516+0000", 2024-04-22T18:38:58.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100516297 2024-04-22T18:38:58.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754547+0000", 2024-04-22T18:38:58.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0673000000000003e-05 2024-04-22T18:38:58.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754577+0000", 2024-04-22T18:38:58.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0207999999999999e-05 2024-04-22T18:38:58.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.241 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.241 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:84 4.8 4:1bc4668a:::smithi08937929-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.241 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.555861+0000", 2024-04-22T18:38:58.241 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.6270209709999999, 2024-04-22T18:38:58.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.198839506, 2024-04-22T18:38:58.242 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.242 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:58.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.242 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 84 2024-04-22T18:38:58.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.242 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555861+0000", 2024-04-22T18:38:58.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555861+0000", 2024-04-22T18:38:58.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555861+0000", 2024-04-22T18:38:58.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:58.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555862+0000", 2024-04-22T18:38:58.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.494e-06 2024-04-22T18:38:58.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555865+0000", 2024-04-22T18:38:58.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.954e-06 2024-04-22T18:38:58.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555870+0000", 2024-04-22T18:38:58.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5070000000000002e-06 2024-04-22T18:38:58.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.652949+0000", 2024-04-22T18:38:58.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.097078988000000005 2024-04-22T18:38:58.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:38:58.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.653026+0000", 2024-04-22T18:38:58.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7313999999999999e-05 2024-04-22T18:38:58.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754603+0000", 2024-04-22T18:38:58.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101577218 2024-04-22T18:38:58.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754647+0000", 2024-04-22T18:38:58.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3850999999999997e-05 2024-04-22T18:38:58.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754700+0000", 2024-04-22T18:38:58.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3489000000000002e-05 2024-04-22T18:38:58.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.246 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.246 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:141 4.8 4:187adac1:::smithi08937929-36:head [write 588245~699127,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.247 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.915008+0000", 2024-04-22T18:38:58.247 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.267873749, 2024-04-22T18:38:58.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10420373600000001, 2024-04-22T18:38:58.247 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.247 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.247 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 141 2024-04-22T18:38:58.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.247 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.915008+0000", 2024-04-22T18:38:58.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.915008+0000", 2024-04-22T18:38:58.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.915003+0000", 2024-04-22T18:38:58.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999952 2024-04-22T18:38:58.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.915821+0000", 2024-04-22T18:38:58.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081771399999999998 2024-04-22T18:38:58.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.915841+0000", 2024-04-22T18:38:58.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0069e-05 2024-04-22T18:38:58.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.915853+0000", 2024-04-22T18:38:58.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2082000000000001e-05 2024-04-22T18:38:58.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.015970+0000", 2024-04-22T18:38:58.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100117412 2024-04-22T18:38:58.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.016077+0000", 2024-04-22T18:38:58.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106156 2024-04-22T18:38:58.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:38:58.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.016190+0000", 2024-04-22T18:38:58.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113108 2024-04-22T18:38:58.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.018510+0000", 2024-04-22T18:38:58.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023202560000000001 2024-04-22T18:38:58.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.019128+0000", 2024-04-22T18:38:58.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00061837600000000002 2024-04-22T18:38:58.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.019174+0000", 2024-04-22T18:38:58.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6112999999999997e-05 2024-04-22T18:38:58.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.019212+0000", 2024-04-22T18:38:58.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7389999999999999e-05 2024-04-22T18:38:58.252 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.252 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.252 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.252 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:69 4.3 e27/20 4:c6fa81cf:::smithi08937929-18:head v 27'7, mlcod=27'3)", 2024-04-22T18:38:58.252 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.957201+0000", 2024-04-22T18:38:58.252 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.2256805659999999, 2024-04-22T18:38:58.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104402907, 2024-04-22T18:38:58.252 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.253 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.253 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957201+0000", 2024-04-22T18:38:58.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957201+0000", 2024-04-22T18:38:58.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957200+0000", 2024-04-22T18:38:58.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:58.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957475+0000", 2024-04-22T18:38:58.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027468399999999997 2024-04-22T18:38:58.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957483+0000", 2024-04-22T18:38:58.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0430000000000008e-06 2024-04-22T18:38:58.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957487+0000", 2024-04-22T18:38:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3490000000000002e-06 2024-04-22T18:38:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957521+0000", 2024-04-22T18:38:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3359999999999999e-05 2024-04-22T18:38:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957541+0000", 2024-04-22T18:38:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0971e-05 2024-04-22T18:38:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.061579+0000", 2024-04-22T18:38:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104037861 2024-04-22T18:38:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.061604+0000", 2024-04-22T18:38:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4888999999999998e-05 2024-04-22T18:38:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:70 4.3 e27/20 4:c6fa81cf:::smithi08937929-18:head v 27'8, mlcod=27'3)", 2024-04-22T18:38:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:38:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.2253896399999999, 2024-04-22T18:38:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20440983300000001, 2024-04-22T18:38:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:38:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:38:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:38:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957839+0000", 2024-04-22T18:38:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000346981 2024-04-22T18:38:58.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957849+0000", 2024-04-22T18:38:58.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8060000000000006e-06 2024-04-22T18:38:58.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957856+0000", 2024-04-22T18:38:58.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0319999999999998e-06 2024-04-22T18:38:58.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958600+0000", 2024-04-22T18:38:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074474700000000003 2024-04-22T18:38:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958615+0000", 2024-04-22T18:38:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4961e-05 2024-04-22T18:38:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161867+0000", 2024-04-22T18:38:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20325188399999999 2024-04-22T18:38:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161902+0000", 2024-04-22T18:38:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4884e-05 2024-04-22T18:38:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:71 4.3 e27/20 4:c6fa81cf:::smithi08937929-18:head v 27'9, mlcod=27'3)", 2024-04-22T18:38:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:38:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.2250196180000001, 2024-04-22T18:38:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20407488600000001, 2024-04-22T18:38:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:38:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:38:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:38:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957867+0000", 2024-04-22T18:38:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8329999999999999e-06 2024-04-22T18:38:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957872+0000", 2024-04-22T18:38:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4530000000000004e-06 2024-04-22T18:38:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957877+0000", 2024-04-22T18:38:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8969999999999999e-06 2024-04-22T18:38:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959214+0000", 2024-04-22T18:38:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001337443 2024-04-22T18:38:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959236+0000", 2024-04-22T18:38:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1889e-05 2024-04-22T18:38:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161910+0000", 2024-04-22T18:38:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202673879 2024-04-22T18:38:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161937+0000", 2024-04-22T18:38:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7202000000000001e-05 2024-04-22T18:38:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:97 4.3 e27/20 4:c93c842e:::smithi08937929-25:head v 27'10, mlcod=27'3)", 2024-04-22T18:38:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:38:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.2250015579999998, 2024-04-22T18:38:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.204078805, 2024-04-22T18:38:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:38:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:38:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:38:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958337+0000", 2024-04-22T18:38:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045682399999999999 2024-04-22T18:38:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958346+0000", 2024-04-22T18:38:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6840000000000008e-06 2024-04-22T18:38:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958353+0000", 2024-04-22T18:38:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0330000000000003e-06 2024-04-22T18:38:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959462+0000", 2024-04-22T18:38:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011084440000000001 2024-04-22T18:38:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959479+0000", 2024-04-22T18:38:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6654e-05 2024-04-22T18:38:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161940+0000", 2024-04-22T18:38:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202461112 2024-04-22T18:38:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161959+0000", 2024-04-22T18:38:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9477999999999999e-05 2024-04-22T18:38:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:98 4.3 e27/20 4:c93c842e:::smithi08937929-25:head v 27'11, mlcod=27'3)", 2024-04-22T18:38:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:38:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.2245234530000002, 2024-04-22T18:38:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203627161, 2024-04-22T18:38:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:38:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:38:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:38:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958880+0000", 2024-04-22T18:38:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052192199999999997 2024-04-22T18:38:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958887+0000", 2024-04-22T18:38:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:38:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958891+0000", 2024-04-22T18:38:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2899999999999996e-06 2024-04-22T18:38:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.060417+0000", 2024-04-22T18:38:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10152593 2024-04-22T18:38:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.060433+0000", 2024-04-22T18:38:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6232e-05 2024-04-22T18:38:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161964+0000", 2024-04-22T18:38:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101531234 2024-04-22T18:38:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161986+0000", 2024-04-22T18:38:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1090999999999998e-05 2024-04-22T18:38:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:99 4.3 e27/20 4:c93c842e:::smithi08937929-25:head v 27'12, mlcod=27'3)", 2024-04-22T18:38:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:38:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.2239875379999998, 2024-04-22T18:38:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20320666600000001, 2024-04-22T18:38:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:38:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:38:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:38:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958897+0000", 2024-04-22T18:38:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4429999999999999e-06 2024-04-22T18:38:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958900+0000", 2024-04-22T18:38:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4490000000000002e-06 2024-04-22T18:38:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958902+0000", 2024-04-22T18:38:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.322e-06 2024-04-22T18:38:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161689+0000", 2024-04-22T18:38:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20278693 2024-04-22T18:38:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161712+0000", 2024-04-22T18:38:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2938999999999998e-05 2024-04-22T18:38:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.162077+0000", 2024-04-22T18:38:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036527800000000002 2024-04-22T18:38:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.162101+0000", 2024-04-22T18:38:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3785e-05 2024-04-22T18:38:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.278 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:58.280 INFO:tasks.rados.rados.0.smithi089.stdout:135: finishing set_chunk to oid 36 2024-04-22T18:38:58.280 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-22T18:38:58.280 INFO:tasks.rados.rados.0.smithi089.stdout:135: done (10 left) 2024-04-22T18:38:58.280 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 10 2024-04-22T18:38:58.280 INFO:tasks.rados.rados.0.smithi089.stdout:128: finishing set_chunk to oid 29 2024-04-22T18:38:58.280 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 6 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-22T18:38:58.280 INFO:tasks.rados.rados.0.smithi089.stdout:128: done (9 left) 2024-04-22T18:38:58.280 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 9 2024-04-22T18:38:58.281 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:38:58.281 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:58.280+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:58.293 INFO:tasks.rados.rados.0.smithi089.stdout:168: finishing set_chunk to oid 19 2024-04-22T18:38:58.293 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-22T18:38:58.293 INFO:tasks.rados.rados.0.smithi089.stdout:168: done (8 left) 2024-04-22T18:38:58.293 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 8 2024-04-22T18:38:58.327 INFO:tasks.rados.rados.0.smithi089.stdout:152: finishing set_chunk to oid 3 2024-04-22T18:38:58.327 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 6 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-22T18:38:58.327 INFO:tasks.rados.rados.0.smithi089.stdout:152: done (7 left) 2024-04-22T18:38:58.327 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 7 2024-04-22T18:38:58.380 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:58.380+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:58.393 INFO:tasks.rados.rados.0.smithi089.stdout:165: finishing set_chunk to oid 16 2024-04-22T18:38:58.393 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-22T18:38:58.393 INFO:tasks.rados.rados.0.smithi089.stdout:165: done (6 left) 2024-04-22T18:38:58.393 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 6 2024-04-22T18:38:58.393 INFO:tasks.rados.rados.0.smithi089.stdout:173: finishing set_chunk to oid 24 2024-04-22T18:38:58.393 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 12 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-22T18:38:58.393 INFO:tasks.rados.rados.0.smithi089.stdout:173: done (5 left) 2024-04-22T18:38:58.393 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 5 2024-04-22T18:38:58.424 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:38:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:38:58.424 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:58.427 INFO:tasks.rados.rados.0.smithi089.stdout:137: finishing set_chunk to oid 38 2024-04-22T18:38:58.427 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-22T18:38:58.427 INFO:tasks.rados.rados.0.smithi089.stdout:137: done (4 left) 2024-04-22T18:38:58.427 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 4 2024-04-22T18:38:58.428 INFO:tasks.rados.rados.0.smithi089.stdout:131: finishing set_chunk to oid 32 2024-04-22T18:38:58.428 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 3 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-22T18:38:58.428 INFO:tasks.rados.rados.0.smithi089.stdout:131: done (3 left) 2024-04-22T18:38:58.428 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 3 2024-04-22T18:38:58.428 INFO:tasks.rados.rados.0.smithi089.stdout:158: finishing set_chunk to oid 9 2024-04-22T18:38:58.428 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 9 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-22T18:38:58.428 INFO:tasks.rados.rados.0.smithi089.stdout:158: done (2 left) 2024-04-22T18:38:58.429 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 2 2024-04-22T18:38:58.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:38:58.481 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:58.482+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:38:58.535 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:58.535 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:38:58.535 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:38:58.535 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:38:58.535 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:58.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:38:58.581 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:58.582+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:38:58.602 INFO:tasks.rados.rados.0.smithi089.stdout:176: finishing set_chunk to oid 27 2024-04-22T18:38:58.602 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 9 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-22T18:38:58.603 INFO:tasks.rados.rados.0.smithi089.stdout:176: done (1 left) 2024-04-22T18:38:58.603 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 1 2024-04-22T18:38:58.603 INFO:tasks.rados.rados.0.smithi089.stdout:142: finishing set_chunk to oid 43 2024-04-22T18:38:58.603 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-22T18:38:58.603 INFO:tasks.rados.rados.0.smithi089.stdout:142: done (0 left) 2024-04-22T18:38:58.603 INFO:tasks.rados.rados.0.smithi089.stdout:178: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.603 INFO:tasks.rados.rados.0.smithi089.stdout:178: set_chunk oid 29 offset: 551424 length: 1443328 target oid smithi08937929-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 551424 snap -1 2024-04-22T18:38:58.603 INFO:tasks.rados.rados.0.smithi089.stdout:179: set_chunk oid 30 target oid 30 low_tier 2024-04-22T18:38:58.603 INFO:tasks.rados.rados.0.smithi089.stdout:179: set_chunk oid 30 offset: 680448 length: 1135104 target oid smithi08937929-30 low_tier offset: 680448 snap -1 2024-04-22T18:38:58.603 INFO:tasks.rados.rados.0.smithi089.stdout:180: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.603 INFO:tasks.rados.rados.0.smithi089.stdout:180: set_chunk oid 31 offset: 930816 length: 1506304 target oid smithi08937929-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 930816 snap -1 2024-04-22T18:38:58.603 INFO:tasks.rados.rados.0.smithi089.stdout:181: set_chunk oid 32 target oid 32 low_tier 2024-04-22T18:38:58.603 INFO:tasks.rados.rados.0.smithi089.stdout:181: set_chunk oid 32 offset: 1292800 length: 897024 target oid smithi08937929-32 low_tier offset: 1292800 snap -1 2024-04-22T18:38:58.603 INFO:tasks.rados.rados.0.smithi089.stdout:182: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.604 INFO:tasks.rados.rados.0.smithi089.stdout:182: set_chunk oid 33 offset: 1534464 length: 244736 target oid smithi08937929-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1534464 snap -1 2024-04-22T18:38:58.604 INFO:tasks.rados.rados.0.smithi089.stdout:183: set_chunk oid 34 target oid 34 low_tier 2024-04-22T18:38:58.604 INFO:tasks.rados.rados.0.smithi089.stdout:183: set_chunk oid 34 offset: 769024 length: 984576 target oid smithi08937929-34 low_tier offset: 769024 snap -1 2024-04-22T18:38:58.604 INFO:tasks.rados.rados.0.smithi089.stdout:184: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.604 INFO:tasks.rados.rados.0.smithi089.stdout:184: set_chunk oid 35 offset: 1213952 length: 868864 target oid smithi08937929-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1213952 snap -1 2024-04-22T18:38:58.604 INFO:tasks.rados.rados.0.smithi089.stdout:185: set_chunk oid 36 target oid 36 low_tier 2024-04-22T18:38:58.604 INFO:tasks.rados.rados.0.smithi089.stdout:185: set_chunk oid 36 offset: 965120 length: 773632 target oid smithi08937929-36 low_tier offset: 965120 snap -1 2024-04-22T18:38:58.604 INFO:tasks.rados.rados.0.smithi089.stdout:186: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.604 INFO:tasks.rados.rados.0.smithi089.stdout:186: set_chunk oid 37 offset: 1193984 length: 294400 target oid smithi08937929-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1193984 snap -1 2024-04-22T18:38:58.604 INFO:tasks.rados.rados.0.smithi089.stdout:187: set_chunk oid 38 target oid 38 low_tier 2024-04-22T18:38:58.604 INFO:tasks.rados.rados.0.smithi089.stdout:187: set_chunk oid 38 offset: 757248 length: 268288 target oid smithi08937929-38 low_tier offset: 757248 snap -1 2024-04-22T18:38:58.604 INFO:tasks.rados.rados.0.smithi089.stdout:188: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.604 INFO:tasks.rados.rados.0.smithi089.stdout:188: set_chunk oid 39 offset: 30208 length: 2283520 target oid smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 30208 snap -1 2024-04-22T18:38:58.605 INFO:tasks.rados.rados.0.smithi089.stdout:189: set_chunk oid 40 target oid 40 low_tier 2024-04-22T18:38:58.605 INFO:tasks.rados.rados.0.smithi089.stdout:189: set_chunk oid 40 offset: 908800 length: 1651200 target oid smithi08937929-40 low_tier offset: 908800 snap -1 2024-04-22T18:38:58.605 INFO:tasks.rados.rados.0.smithi089.stdout:190: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.605 INFO:tasks.rados.rados.0.smithi089.stdout:190: set_chunk oid 41 offset: 122368 length: 2205696 target oid smithi08937929-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 122368 snap -1 2024-04-22T18:38:58.605 INFO:tasks.rados.rados.0.smithi089.stdout:191: set_chunk oid 42 target oid 42 low_tier 2024-04-22T18:38:58.605 INFO:tasks.rados.rados.0.smithi089.stdout:191: set_chunk oid 42 offset: 669696 length: 220160 target oid smithi08937929-42 low_tier offset: 669696 snap -1 2024-04-22T18:38:58.605 INFO:tasks.rados.rados.0.smithi089.stdout:192: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.605 INFO:tasks.rados.rados.0.smithi089.stdout:192: set_chunk oid 43 offset: 1128448 length: 1395712 target oid smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1128448 snap -1 2024-04-22T18:38:58.605 INFO:tasks.rados.rados.0.smithi089.stdout:193: set_chunk oid 44 target oid 44 low_tier 2024-04-22T18:38:58.605 INFO:tasks.rados.rados.0.smithi089.stdout:193: set_chunk oid 44 offset: 524800 length: 953856 target oid smithi08937929-44 low_tier offset: 524800 snap -1 2024-04-22T18:38:58.605 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.605 INFO:tasks.rados.rados.0.smithi089.stdout:178: finishing set_chunk to oid 29 2024-04-22T18:38:58.605 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 551424 length: 1443328 2024-04-22T18:38:58.605 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-22T18:38:58.606 INFO:tasks.rados.rados.0.smithi089.stdout:179: finishing set_chunk to oid 30 2024-04-22T18:38:58.606 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 680448 length: 1135104 2024-04-22T18:38:58.606 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-22T18:38:58.606 INFO:tasks.rados.rados.0.smithi089.stdout:185: finishing set_chunk to oid 36 2024-04-22T18:38:58.606 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 965120 length: 773632 2024-04-22T18:38:58.606 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-22T18:38:58.606 INFO:tasks.rados.rados.0.smithi089.stdout:178: done (15 left) 2024-04-22T18:38:58.606 INFO:tasks.rados.rados.0.smithi089.stdout:179: done (14 left) 2024-04-22T18:38:58.606 INFO:tasks.rados.rados.0.smithi089.stdout:185: done (13 left) 2024-04-22T18:38:58.606 INFO:tasks.rados.rados.0.smithi089.stdout:194: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.606 INFO:tasks.rados.rados.0.smithi089.stdout:194: set_chunk oid 45 offset: 752640 length: 118784 target oid smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 752640 snap -1 2024-04-22T18:38:58.606 INFO:tasks.rados.rados.0.smithi089.stdout:195: set_chunk oid 46 target oid 46 low_tier 2024-04-22T18:38:58.606 INFO:tasks.rados.rados.0.smithi089.stdout:195: set_chunk oid 46 offset: 367104 length: 399872 target oid smithi08937929-46 low_tier offset: 367104 snap -1 2024-04-22T18:38:58.607 INFO:tasks.rados.rados.0.smithi089.stdout:196: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.607 INFO:tasks.rados.rados.0.smithi089.stdout:196: set_chunk oid 47 offset: 964608 length: 670720 target oid smithi08937929-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 964608 snap -1 2024-04-22T18:38:58.607 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.607 INFO:tasks.rados.rados.0.smithi089.stdout:186: finishing set_chunk to oid 37 2024-04-22T18:38:58.607 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1193984 length: 294400 2024-04-22T18:38:58.607 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-22T18:38:58.607 INFO:tasks.rados.rados.0.smithi089.stdout:186: done (15 left) 2024-04-22T18:38:58.607 INFO:tasks.rados.rados.0.smithi089.stdout:197: set_chunk oid 48 target oid 48 low_tier 2024-04-22T18:38:58.607 INFO:tasks.rados.rados.0.smithi089.stdout:197: set_chunk oid 48 offset: 1471488 length: 197120 target oid smithi08937929-48 low_tier offset: 1471488 snap -1 2024-04-22T18:38:58.607 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.607 INFO:tasks.rados.rados.0.smithi089.stdout:189: finishing set_chunk to oid 40 2024-04-22T18:38:58.607 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 908800 length: 1651200 2024-04-22T18:38:58.607 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-22T18:38:58.607 INFO:tasks.rados.rados.0.smithi089.stdout:193: finishing set_chunk to oid 44 2024-04-22T18:38:58.607 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 524800 length: 953856 2024-04-22T18:38:58.608 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-22T18:38:58.608 INFO:tasks.rados.rados.0.smithi089.stdout:189: done (15 left) 2024-04-22T18:38:58.608 INFO:tasks.rados.rados.0.smithi089.stdout:193: done (14 left) 2024-04-22T18:38:58.608 INFO:tasks.rados.rados.0.smithi089.stdout:198: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.608 INFO:tasks.rados.rados.0.smithi089.stdout:198: set_chunk oid 49 offset: 1097216 length: 1292288 target oid smithi08937929-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1097216 snap -1 2024-04-22T18:38:58.608 INFO:tasks.rados.rados.0.smithi089.stdout:199: set_chunk oid 50 target oid 50 low_tier 2024-04-22T18:38:58.608 INFO:tasks.rados.rados.0.smithi089.stdout:199: set_chunk oid 50 offset: 2612224 length: 78336 target oid smithi08937929-50 low_tier offset: 2612224 snap -1 2024-04-22T18:38:58.608 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.608 INFO:tasks.rados.rados.0.smithi089.stdout:190: finishing set_chunk to oid 41 2024-04-22T18:38:58.608 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 122368 length: 2205696 2024-04-22T18:38:58.608 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-22T18:38:58.608 INFO:tasks.rados.rados.0.smithi089.stdout:190: done (15 left) 2024-04-22T18:38:58.608 INFO:tasks.rados.rados.0.smithi089.stdout:200: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.608 INFO:tasks.rados.rados.0.smithi089.stdout:200: set_chunk oid 1 offset: 1794560 length: 34304 target oid smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1794560 snap -1 2024-04-22T18:38:58.609 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.609 INFO:tasks.rados.rados.0.smithi089.stdout:188: finishing set_chunk to oid 39 2024-04-22T18:38:58.609 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 30208 length: 2283520 2024-04-22T18:38:58.609 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-22T18:38:58.609 INFO:tasks.rados.rados.0.smithi089.stdout:192: finishing set_chunk to oid 43 2024-04-22T18:38:58.609 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1128448 length: 1395712 2024-04-22T18:38:58.609 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-22T18:38:58.609 INFO:tasks.rados.rados.0.smithi089.stdout:188: done (15 left) 2024-04-22T18:38:58.609 INFO:tasks.rados.rados.0.smithi089.stdout:192: done (14 left) 2024-04-22T18:38:58.609 INFO:tasks.rados.rados.0.smithi089.stdout:201: set_chunk oid 2 target oid 2 low_tier 2024-04-22T18:38:58.609 INFO:tasks.rados.rados.0.smithi089.stdout:201: set_chunk oid 2 offset: 666112 length: 190464 target oid smithi08937929-2 low_tier offset: 666112 snap -1 2024-04-22T18:38:58.609 INFO:tasks.rados.rados.0.smithi089.stdout:202: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.609 INFO:tasks.rados.rados.0.smithi089.stdout:202: set_chunk oid 3 offset: 1014272 length: 447488 target oid smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1014272 snap -1 2024-04-22T18:38:58.609 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.610 INFO:tasks.rados.rados.0.smithi089.stdout:196: finishing set_chunk to oid 47 2024-04-22T18:38:58.610 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 964608 length: 670720 2024-04-22T18:38:58.610 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-22T18:38:58.610 INFO:tasks.rados.rados.0.smithi089.stdout:196: done (15 left) 2024-04-22T18:38:58.610 INFO:tasks.rados.rados.0.smithi089.stdout:203: set_chunk oid 4 target oid 4 low_tier 2024-04-22T18:38:58.610 INFO:tasks.rados.rados.0.smithi089.stdout:203: set_chunk oid 4 offset: 75264 length: 714240 target oid smithi08937929-4 low_tier offset: 75264 snap -1 2024-04-22T18:38:58.610 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.610 INFO:tasks.rados.rados.0.smithi089.stdout:197: finishing set_chunk to oid 48 2024-04-22T18:38:58.610 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1471488 length: 197120 2024-04-22T18:38:58.610 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-22T18:38:58.610 INFO:tasks.rados.rados.0.smithi089.stdout:197: done (15 left) 2024-04-22T18:38:58.610 INFO:tasks.rados.rados.0.smithi089.stdout:204: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.610 INFO:tasks.rados.rados.0.smithi089.stdout:204: set_chunk oid 5 offset: 148992 length: 1375744 target oid smithi08937929-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 148992 snap -1 2024-04-22T18:38:58.611 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.611 INFO:tasks.rados.rados.0.smithi089.stdout:203: finishing set_chunk to oid 4 2024-04-22T18:38:58.611 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 75264 length: 714240 2024-04-22T18:38:58.611 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-22T18:38:58.611 INFO:tasks.rados.rados.0.smithi089.stdout:203: done (15 left) 2024-04-22T18:38:58.611 INFO:tasks.rados.rados.0.smithi089.stdout:205: set_chunk oid 6 target oid 6 low_tier 2024-04-22T18:38:58.611 INFO:tasks.rados.rados.0.smithi089.stdout:205: set_chunk oid 6 offset: 1520640 length: 161280 target oid smithi08937929-6 low_tier offset: 1520640 snap -1 2024-04-22T18:38:58.611 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.611 INFO:tasks.rados.rados.0.smithi089.stdout:204: finishing set_chunk to oid 5 2024-04-22T18:38:58.611 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 148992 length: 1375744 2024-04-22T18:38:58.611 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-22T18:38:58.611 INFO:tasks.rados.rados.0.smithi089.stdout:204: done (15 left) 2024-04-22T18:38:58.611 INFO:tasks.rados.rados.0.smithi089.stdout:206: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.611 INFO:tasks.rados.rados.0.smithi089.stdout:206: set_chunk oid 7 offset: 905728 length: 416256 target oid smithi08937929-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 905728 snap -1 2024-04-22T18:38:58.611 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.612 INFO:tasks.rados.rados.0.smithi089.stdout:205: finishing set_chunk to oid 6 2024-04-22T18:38:58.612 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1520640 length: 161280 2024-04-22T18:38:58.612 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-22T18:38:58.612 INFO:tasks.rados.rados.0.smithi089.stdout:205: done (15 left) 2024-04-22T18:38:58.612 INFO:tasks.rados.rados.0.smithi089.stdout:207: set_chunk oid 8 target oid 8 low_tier 2024-04-22T18:38:58.612 INFO:tasks.rados.rados.0.smithi089.stdout:207: set_chunk oid 8 offset: 1041920 length: 22016 target oid smithi08937929-8 low_tier offset: 1041920 snap -1 2024-04-22T18:38:58.612 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.612 INFO:tasks.rados.rados.0.smithi089.stdout:206: finishing set_chunk to oid 7 2024-04-22T18:38:58.612 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 905728 length: 416256 2024-04-22T18:38:58.612 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-22T18:38:58.612 INFO:tasks.rados.rados.0.smithi089.stdout:206: done (15 left) 2024-04-22T18:38:58.612 INFO:tasks.rados.rados.0.smithi089.stdout:208: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.612 INFO:tasks.rados.rados.0.smithi089.stdout:208: set_chunk oid 9 offset: 1153024 length: 531968 target oid smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1153024 snap -1 2024-04-22T18:38:58.612 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.647 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:58.648 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:38:58.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:38:58.648 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:38:58.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.648 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:21 4.a 4:5aa97f94:::smithi08937929-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.648 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:38:58.648 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.4946573729999999, 2024-04-22T18:38:58.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21044934000000001, 2024-04-22T18:38:58.648 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.648 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.648 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.648 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.648 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 21 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153183+0000", 2024-04-22T18:38:58.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:38:58.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154022+0000", 2024-04-22T18:38:58.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00083893300000000004 2024-04-22T18:38:58.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154032+0000", 2024-04-22T18:38:58.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0433e-05 2024-04-22T18:38:58.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154051+0000", 2024-04-22T18:38:58.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8227e-05 2024-04-22T18:38:58.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154103+0000", 2024-04-22T18:38:58.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2491999999999998e-05 2024-04-22T18:38:58.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:38:58.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154111+0000", 2024-04-22T18:38:58.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3470000000000001e-06 2024-04-22T18:38:58.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257848+0000", 2024-04-22T18:38:58.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103737703 2024-04-22T18:38:58.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257981+0000", 2024-04-22T18:38:58.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013290399999999999 2024-04-22T18:38:58.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:58.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.258138+0000", 2024-04-22T18:38:58.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015701199999999999 2024-04-22T18:38:58.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.261944+0000", 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038054650000000001 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363523+0000", 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101579239 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363577+0000", 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4095000000000003e-05 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363635+0000", 2024-04-22T18:38:58.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8362999999999997e-05 2024-04-22T18:38:58.654 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.654 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.654 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.654 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:22 4.a 4:5aa97f94:::smithi08937929-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.654 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:38:58.654 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.490583397, 2024-04-22T18:38:58.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30948398900000001, 2024-04-22T18:38:58.654 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.654 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.654 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.654 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.654 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.654 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 22 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157259+0000", 2024-04-22T18:38:58.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157318+0000", 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8906000000000001e-05 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157327+0000", 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157332+0000", 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4689999999999996e-06 2024-04-22T18:38:58.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257479+0000", 2024-04-22T18:38:58.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014700999999999 2024-04-22T18:38:58.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:38:58.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257492+0000", 2024-04-22T18:38:58.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2574e-05 2024-04-22T18:38:58.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259582+0000", 2024-04-22T18:38:58.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020904019999999999 2024-04-22T18:38:58.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259668+0000", 2024-04-22T18:38:58.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5826000000000001e-05 2024-04-22T18:38:58.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:58.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259772+0000", 2024-04-22T18:38:58.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104048 2024-04-22T18:38:58.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363453+0000", 2024-04-22T18:38:58.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103680183 2024-04-22T18:38:58.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466692+0000", 2024-04-22T18:38:58.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10323948199999999 2024-04-22T18:38:58.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466720+0000", 2024-04-22T18:38:58.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7886e-05 2024-04-22T18:38:58.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466744+0000", 2024-04-22T18:38:58.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4049e-05 2024-04-22T18:38:58.660 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.660 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.660 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.660 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:23 4.a 4:5aa97f94:::smithi08937929-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.660 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:38:58.660 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.4905057099999999, 2024-04-22T18:38:58.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39003820300000003, 2024-04-22T18:38:58.660 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.660 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.661 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 23 2024-04-22T18:38:58.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.661 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:38:58.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:38:58.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157337+0000", 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157340+0000", 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8609999999999998e-06 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157343+0000", 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9859999999999999e-06 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157346+0000", 2024-04-22T18:38:58.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8119999999999999e-06 2024-04-22T18:38:58.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257563+0000", 2024-04-22T18:38:58.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100217497 2024-04-22T18:38:58.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:38:58.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257574+0000", 2024-04-22T18:38:58.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0777e-05 2024-04-22T18:38:58.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260112+0000", 2024-04-22T18:38:58.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002537681 2024-04-22T18:38:58.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260191+0000", 2024-04-22T18:38:58.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9320999999999995e-05 2024-04-22T18:38:58.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:58.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260333+0000", 2024-04-22T18:38:58.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014177699999999999 2024-04-22T18:38:58.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363792+0000", 2024-04-22T18:38:58.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103458862 2024-04-22T18:38:58.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547303+0000", 2024-04-22T18:38:58.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18351162100000001 2024-04-22T18:38:58.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547337+0000", 2024-04-22T18:38:58.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3640999999999999e-05 2024-04-22T18:38:58.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547376+0000", 2024-04-22T18:38:58.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8915000000000003e-05 2024-04-22T18:38:58.666 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.666 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.666 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.666 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:24 4.a 4:5aa97f94:::smithi08937929-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.666 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:38:58.666 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.4904792789999997, 2024-04-22T18:38:58.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39017823499999998, 2024-04-22T18:38:58.666 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 24 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:38:58.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157366+0000", 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3120000000000001e-06 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157370+0000", 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1300000000000003e-06 2024-04-22T18:38:58.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157373+0000", 2024-04-22T18:38:58.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-04-22T18:38:58.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260553+0000", 2024-04-22T18:38:58.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103179996 2024-04-22T18:38:58.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:38:58.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260625+0000", 2024-04-22T18:38:58.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2236000000000004e-05 2024-04-22T18:38:58.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547414+0000", 2024-04-22T18:38:58.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.286789181 2024-04-22T18:38:58.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547471+0000", 2024-04-22T18:38:58.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6192e-05 2024-04-22T18:38:58.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547542+0000", 2024-04-22T18:38:58.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1622e-05 2024-04-22T18:38:58.671 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.671 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.671 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.671 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:49 4.a 4:5d2aee44:::smithi08937929-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.671 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:38:58.671 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.4136806210000001, 2024-04-22T18:38:58.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31344442500000003, 2024-04-22T18:38:58.671 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.671 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.671 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.671 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.671 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.671 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 49 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234158+0000", 2024-04-22T18:38:58.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999957 2024-04-22T18:38:58.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234901+0000", 2024-04-22T18:38:58.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074250800000000001 2024-04-22T18:38:58.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234914+0000", 2024-04-22T18:38:58.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3244999999999999e-05 2024-04-22T18:38:58.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234922+0000", 2024-04-22T18:38:58.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4590000000000001e-06 2024-04-22T18:38:58.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260665+0000", 2024-04-22T18:38:58.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025743423000000001 2024-04-22T18:38:58.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260779+0000", 2024-04-22T18:38:58.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114063 2024-04-22T18:38:58.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:58.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260913+0000", 2024-04-22T18:38:58.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013355499999999999 2024-04-22T18:38:58.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363798+0000", 2024-04-22T18:38:58.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102885003 2024-04-22T18:38:58.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547567+0000", 2024-04-22T18:38:58.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.183768971 2024-04-22T18:38:58.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547589+0000", 2024-04-22T18:38:58.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2507999999999999e-05 2024-04-22T18:38:58.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547607+0000", 2024-04-22T18:38:58.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8009999999999999e-05 2024-04-22T18:38:58.676 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.676 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.676 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.676 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:50 4.a 4:5d2aee44:::smithi08937929-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.676 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:38:58.676 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.4082344420000004, 2024-04-22T18:38:58.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40856295399999998, 2024-04-22T18:38:58.677 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.677 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.677 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.677 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.677 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.677 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 50 2024-04-22T18:38:58.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.677 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:38:58.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239607+0000", 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240309+0000", 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070201599999999997 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240320+0000", 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1024e-05 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240327+0000", 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0049999999999998e-06 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362065+0000", 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.121737812 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362121+0000", 2024-04-22T18:38:58.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5868000000000001e-05 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362210+0000", 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8533000000000001e-05 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365322+0000", 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031121220000000002 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648054+0000", 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.282732549 2024-04-22T18:38:58.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648105+0000", 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1276999999999997e-05 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648172+0000", 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6398999999999999e-05 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:51 4.a 4:5d2aee44:::smithi08937929-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.681 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:38:58.682 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.407511908, 2024-04-22T18:38:58.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40793057199999999, 2024-04-22T18:38:58.682 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.682 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.682 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 51 2024-04-22T18:38:58.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.682 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:38:58.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:38:58.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:38:58.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:58.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240336+0000", 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240339+0000", 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9009999999999998e-06 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240342+0000", 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363080+0000", 2024-04-22T18:38:58.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122738075 2024-04-22T18:38:58.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363164+0000", 2024-04-22T18:38:58.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3722999999999998e-05 2024-04-22T18:38:58.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:58.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363287+0000", 2024-04-22T18:38:58.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012301099999999999 2024-04-22T18:38:58.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466753+0000", 2024-04-22T18:38:58.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103466217 2024-04-22T18:38:58.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648205+0000", 2024-04-22T18:38:58.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.181451788 2024-04-22T18:38:58.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648231+0000", 2024-04-22T18:38:58.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6597999999999998e-05 2024-04-22T18:38:58.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648262+0000", 2024-04-22T18:38:58.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0871000000000003e-05 2024-04-22T18:38:58.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.687 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:52 4.a 4:5d2aee44:::smithi08937929-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.687 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:38:58.687 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.4074994490000003, 2024-04-22T18:38:58.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40812731200000002, 2024-04-22T18:38:58.687 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.688 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:58.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.688 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 52 2024-04-22T18:38:58.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.688 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:38:58.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:38:58.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:38:58.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:58.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240345+0000", 2024-04-22T18:38:58.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4300000000000001e-06 2024-04-22T18:38:58.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240349+0000", 2024-04-22T18:38:58.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2570000000000001e-06 2024-04-22T18:38:58.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240358+0000", 2024-04-22T18:38:58.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-22T18:38:58.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363697+0000", 2024-04-22T18:38:58.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12333965299999999 2024-04-22T18:38:58.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:38:58.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363777+0000", 2024-04-22T18:38:58.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9883999999999996e-05 2024-04-22T18:38:58.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648296+0000", 2024-04-22T18:38:58.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451861299999998 2024-04-22T18:38:58.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648379+0000", 2024-04-22T18:38:58.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3302999999999998e-05 2024-04-22T18:38:58.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648471+0000", 2024-04-22T18:38:58.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2029999999999998e-05 2024-04-22T18:38:58.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.692 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.692 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.692 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:38:58.692 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.3881961140000003, 2024-04-22T18:38:58.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:38:58.692 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.693 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.693 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:38:58.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.693 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:38:58.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:38:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:38:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:38:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:38:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:38:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:38:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:38:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:38:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:38:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:38:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:38:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:38:58.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:38:58.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:58.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:38:58.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:38:58.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:38:58.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:38:58.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.698 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:29 4.e e27/20 4:744777ef:::smithi08937929-8:head v 27'1, mlcod=0'0)", 2024-04-22T18:38:58.698 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264023+0000", 2024-04-22T18:38:58.698 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.3838198869999996, 2024-04-22T18:38:58.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10131754, 2024-04-22T18:38:58.698 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.698 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.698 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264023+0000", 2024-04-22T18:38:58.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264023+0000", 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264023+0000", 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264383+0000", 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000360016 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264394+0000", 2024-04-22T18:38:58.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0786e-05 2024-04-22T18:38:58.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264401+0000", 2024-04-22T18:38:58.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8120000000000002e-06 2024-04-22T18:38:58.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265687+0000", 2024-04-22T18:38:58.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012858660000000001 2024-04-22T18:38:58.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265712+0000", 2024-04-22T18:38:58.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5375e-05 2024-04-22T18:38:58.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365303+0000", 2024-04-22T18:38:58.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099591381000000007 2024-04-22T18:38:58.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365341+0000", 2024-04-22T18:38:58.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7703000000000001e-05 2024-04-22T18:38:58.701 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.701 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.701 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.702 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.702 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:38:58.702 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.3834136839999998, 2024-04-22T18:38:58.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:38:58.702 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.702 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.702 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.702 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.702 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.702 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:38:58.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.702 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:38:58.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:38:58.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:38:58.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:38:58.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:38:58.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:38:58.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:38:58.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:38:58.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:38:58.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:38:58.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:58.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:38:58.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:38:58.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:38:58.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:38:58.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:38:58.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:38:58.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:38:58.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:38:58.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:38:58.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:38:58.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.707 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.707 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.707 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:38:58.707 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.383319481, 2024-04-22T18:38:58.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:38:58.707 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.708 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.708 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.708 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.708 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.708 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:38:58.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.708 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:38:58.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:38:58.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:38:58.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:58.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:38:58.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:38:58.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:38:58.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:38:58.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:38:58.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:38:58.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:38:58.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:38:58.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:38:58.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:38:58.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:58.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:38:58.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:38:58.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:38:58.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:38:58.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.712 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.3832985930000001, 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:38:58.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:38:58.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:38:58.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:58.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:38:58.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:38:58.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:38:58.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:38:58.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:38:58.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:38:58.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:38:58.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:38:58.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:38:58.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:38:58.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:38:58.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:38:58.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:38:58.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:38:58.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:38:58.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:38:58.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:38:58.717 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.717 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.717 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.718 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:30 4.e e27/20 4:744777ef:::smithi08937929-8:head v 27'2, mlcod=0'0)", 2024-04-22T18:38:58.718 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.265427+0000", 2024-04-22T18:38:58.718 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.3824159839999997, 2024-04-22T18:38:58.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102124983, 2024-04-22T18:38:58.718 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.718 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.718 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265427+0000", 2024-04-22T18:38:58.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265427+0000", 2024-04-22T18:38:58.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265426+0000", 2024-04-22T18:38:58.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:58.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265617+0000", 2024-04-22T18:38:58.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019075900000000001 2024-04-22T18:38:58.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265628+0000", 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1336e-05 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.265636+0000", 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0940000000000008e-06 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364506+0000", 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098869159999999998 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364527+0000", 2024-04-22T18:38:58.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1107999999999998e-05 2024-04-22T18:38:58.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367522+0000", 2024-04-22T18:38:58.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029958680000000001 2024-04-22T18:38:58.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367552+0000", 2024-04-22T18:38:58.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9858000000000001e-05 2024-04-22T18:38:58.721 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.721 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.722 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.722 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.4502.0:22 4.a e27/20 ondisk, result = 0)", 2024-04-22T18:38:58.722 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.346987+0000", 2024-04-22T18:38:58.722 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.3008565980000002, 2024-04-22T18:38:58.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.119862203, 2024-04-22T18:38:58.722 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.722 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:38:58.722 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.346987+0000", 2024-04-22T18:38:58.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.346987+0000", 2024-04-22T18:38:58.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.346985+0000", 2024-04-22T18:38:58.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:58.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.346991+0000", 2024-04-22T18:38:58.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1329999999999998e-06 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.347002+0000", 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1404e-05 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.447116+0000", 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10011421199999999 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466683+0000", 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.019566887000000002 2024-04-22T18:38:58.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466688+0000", 2024-04-22T18:38:58.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3249999999999998e-06 2024-04-22T18:38:58.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466849+0000", 2024-04-22T18:38:58.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000160483 2024-04-22T18:38:58.725 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.725 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.725 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:38:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.6002909299999999, 2024-04-22T18:38:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:38:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:38:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:38:58.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:38:58.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:38:58.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:38:58.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:38:58.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:38:58.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:38:58.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:38:58.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:38:58.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:38:58.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:38:58.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:38:58.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:38:58.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:38:58.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:38:58.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:38:58.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:38:58.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.5924595240000001, 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:38:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:38:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:38:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:38:58.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:38:58.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:38:58.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:38:58.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:38:58.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:38:58.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:38:58.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:38:58.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:38:58.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:38:58.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:38:58.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:38:58.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:38:58.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:185 4.a 4:537c89f8:::smithi08937929-47:head [write 773122~559572,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.395912+0000", 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.2519315670000002, 2024-04-22T18:38:58.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10987006000000001, 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 185 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.395912+0000", 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.395912+0000", 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.395909+0000", 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.396414+0000", 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050523300000000001 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.396429+0000", 2024-04-22T18:38:58.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4826e-05 2024-04-22T18:38:58.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.396442+0000", 2024-04-22T18:38:58.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2824e-05 2024-04-22T18:38:58.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403539+0000", 2024-04-22T18:38:58.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0070974879999999999 2024-04-22T18:38:58.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403646+0000", 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106845 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403748+0000", 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010211400000000001 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405912+0000", 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002163243 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.505723+0000", 2024-04-22T18:38:58.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099811752000000004 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.505756+0000", 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2279e-05 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.505782+0000", 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.741 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:186 4.a 4:537c89f8:::smithi08937929-47:head [write 1992054~651681] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.742 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:38:58.742 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.2449539889999999, 2024-04-22T18:38:58.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20325226800000001, 2024-04-22T18:38:58.742 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.742 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.742 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.742 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.742 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.742 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 186 2024-04-22T18:38:58.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.742 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:38:58.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402888+0000", 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403454+0000", 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056661000000000005 2024-04-22T18:38:58.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403465+0000", 2024-04-22T18:38:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0727000000000001e-05 2024-04-22T18:38:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403473+0000", 2024-04-22T18:38:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7039999999999994e-06 2024-04-22T18:38:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404592+0000", 2024-04-22T18:38:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001118827 2024-04-22T18:38:58.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404674+0000", 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2793999999999994e-05 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404779+0000", 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104465 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.406080+0000", 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013006789999999999 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606037+0000", 2024-04-22T18:38:58.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19995755600000001 2024-04-22T18:38:58.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606098+0000", 2024-04-22T18:38:58.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1075999999999996e-05 2024-04-22T18:38:58.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606142+0000", 2024-04-22T18:38:58.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3411999999999997e-05 2024-04-22T18:38:58.746 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:187 4.a 4:537c89f8:::smithi08937929-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.244363849, 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203643623, 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 187 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.747 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:58.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403501+0000", 2024-04-22T18:38:58.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2146000000000001e-05 2024-04-22T18:38:58.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:58.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403506+0000", 2024-04-22T18:38:58.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3040000000000004e-06 2024-04-22T18:38:58.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:58.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403511+0000", 2024-04-22T18:38:58.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9599999999999999e-06 2024-04-22T18:38:58.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:58.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405587+0000", 2024-04-22T18:38:58.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002076296 2024-04-22T18:38:58.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:58.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405664+0000", 2024-04-22T18:38:58.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6316000000000006e-05 2024-04-22T18:38:58.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:58.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405785+0000", 2024-04-22T18:38:58.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121276 2024-04-22T18:38:58.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:58.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.408030+0000", 2024-04-22T18:38:58.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022452169999999999 2024-04-22T18:38:58.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:58.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607059+0000", 2024-04-22T18:38:58.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199029135 2024-04-22T18:38:58.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:58.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607094+0000", 2024-04-22T18:38:58.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4686e-05 2024-04-22T18:38:58.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:58.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:58.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:58.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607123+0000", 2024-04-22T18:38:58.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9113000000000001e-05 2024-04-22T18:38:58.752 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.752 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.752 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.752 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:58.752 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:58.752 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:58.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:38:58.754 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:58.754+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:58.756 INFO:tasks.rados.rados.0.smithi089.stdout:198: finishing set_chunk to oid 49 2024-04-22T18:38:58.756 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1097216 length: 1292288 2024-04-22T18:38:58.756 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:38:58.756 INFO:tasks.rados.rados.0.smithi089.stdout:198: done (15 left) 2024-04-22T18:38:58.756 INFO:tasks.rados.rados.0.smithi089.stdout:209: set_chunk oid 10 target oid 10 low_tier 2024-04-22T18:38:58.756 INFO:tasks.rados.rados.0.smithi089.stdout:209: set_chunk oid 10 offset: 279552 length: 1377792 target oid smithi08937929-10 low_tier offset: 279552 snap -1 2024-04-22T18:38:58.756 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.756 INFO:tasks.rados.rados.0.smithi089.stdout:187: finishing set_chunk to oid 38 2024-04-22T18:38:58.757 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-22T18:38:58.757 INFO:tasks.rados.rados.0.smithi089.stdout:187: done (15 left) 2024-04-22T18:38:58.757 INFO:tasks.rados.rados.0.smithi089.stdout:210: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.757 INFO:tasks.rados.rados.0.smithi089.stdout:210: set_chunk oid 11 offset: 861696 length: 1105408 target oid smithi08937929-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 861696 snap -1 2024-04-22T18:38:58.757 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.757 INFO:tasks.rados.rados.0.smithi089.stdout:200: finishing set_chunk to oid 1 2024-04-22T18:38:58.757 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-22T18:38:58.757 INFO:tasks.rados.rados.0.smithi089.stdout:200: done (15 left) 2024-04-22T18:38:58.757 INFO:tasks.rados.rados.0.smithi089.stdout:211: set_chunk oid 12 target oid 12 low_tier 2024-04-22T18:38:58.757 INFO:tasks.rados.rados.0.smithi089.stdout:211: set_chunk oid 12 offset: 1215488 length: 842240 target oid smithi08937929-12 low_tier offset: 1215488 snap -1 2024-04-22T18:38:58.757 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.757 INFO:tasks.rados.rados.0.smithi089.stdout:211: finishing set_chunk to oid 12 2024-04-22T18:38:58.757 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1215488 length: 842240 2024-04-22T18:38:58.757 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-22T18:38:58.757 INFO:tasks.rados.rados.0.smithi089.stdout:211: done (15 left) 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout:212: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout:212: set_chunk oid 13 offset: 36352 length: 1275392 target oid smithi08937929-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 36352 snap -1 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout:212: finishing set_chunk to oid 13 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 36352 length: 1275392 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout:212: done (15 left) 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout:213: set_chunk oid 14 target oid 14 low_tier 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout:213: set_chunk oid 14 offset: 755712 length: 522240 target oid smithi08937929-14 low_tier offset: 755712 snap -1 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout:213: finishing set_chunk to oid 14 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 755712 length: 522240 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout:213: done (15 left) 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout:214: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.758 INFO:tasks.rados.rados.0.smithi089.stdout:214: set_chunk oid 15 offset: 429056 length: 1155072 target oid smithi08937929-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 429056 snap -1 2024-04-22T18:38:58.759 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.797 INFO:tasks.rados.rados.0.smithi089.stdout:180: finishing set_chunk to oid 31 2024-04-22T18:38:58.797 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 930816 length: 1506304 2024-04-22T18:38:58.797 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-22T18:38:58.797 INFO:tasks.rados.rados.0.smithi089.stdout:180: done (15 left) 2024-04-22T18:38:58.797 INFO:tasks.rados.rados.0.smithi089.stdout:215: set_chunk oid 16 target oid 16 low_tier 2024-04-22T18:38:58.798 INFO:tasks.rados.rados.0.smithi089.stdout:215: set_chunk oid 16 offset: 822272 length: 517632 target oid smithi08937929-16 low_tier offset: 822272 snap -1 2024-04-22T18:38:58.798 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.798 INFO:tasks.rados.rados.0.smithi089.stdout:191: finishing set_chunk to oid 42 2024-04-22T18:38:58.798 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 669696 length: 220160 2024-04-22T18:38:58.798 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-22T18:38:58.798 INFO:tasks.rados.rados.0.smithi089.stdout:209: finishing set_chunk to oid 10 2024-04-22T18:38:58.798 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 279552 length: 1377792 2024-04-22T18:38:58.798 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-22T18:38:58.798 INFO:tasks.rados.rados.0.smithi089.stdout:201: finishing set_chunk to oid 2 2024-04-22T18:38:58.798 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 666112 length: 190464 2024-04-22T18:38:58.798 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-22T18:38:58.798 INFO:tasks.rados.rados.0.smithi089.stdout:210: finishing set_chunk to oid 11 2024-04-22T18:38:58.798 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 861696 length: 1105408 2024-04-22T18:38:58.799 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-22T18:38:58.799 INFO:tasks.rados.rados.0.smithi089.stdout:191: done (15 left) 2024-04-22T18:38:58.799 INFO:tasks.rados.rados.0.smithi089.stdout:201: done (14 left) 2024-04-22T18:38:58.799 INFO:tasks.rados.rados.0.smithi089.stdout:209: done (13 left) 2024-04-22T18:38:58.799 INFO:tasks.rados.rados.0.smithi089.stdout:210: done (12 left) 2024-04-22T18:38:58.799 INFO:tasks.rados.rados.0.smithi089.stdout:216: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.799 INFO:tasks.rados.rados.0.smithi089.stdout:216: set_chunk oid 17 offset: 1278976 length: 799232 target oid smithi08937929-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1278976 snap -1 2024-04-22T18:38:58.799 INFO:tasks.rados.rados.0.smithi089.stdout:217: set_chunk oid 18 target oid 18 low_tier 2024-04-22T18:38:58.799 INFO:tasks.rados.rados.0.smithi089.stdout:217: set_chunk oid 18 offset: 628736 length: 687104 target oid smithi08937929-18 low_tier offset: 628736 snap -1 2024-04-22T18:38:58.799 INFO:tasks.rados.rados.0.smithi089.stdout:218: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.799 INFO:tasks.rados.rados.0.smithi089.stdout:218: set_chunk oid 19 offset: 1532928 length: 664064 target oid smithi08937929-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1532928 snap -1 2024-04-22T18:38:58.799 INFO:tasks.rados.rados.0.smithi089.stdout:219: set_chunk oid 20 target oid 20 low_tier 2024-04-22T18:38:58.799 INFO:tasks.rados.rados.0.smithi089.stdout:219: set_chunk oid 20 offset: 845824 length: 930304 target oid smithi08937929-20 low_tier offset: 845824 snap -1 2024-04-22T18:38:58.799 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.799 INFO:tasks.rados.rados.0.smithi089.stdout:202: finishing set_chunk to oid 3 2024-04-22T18:38:58.800 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1014272 length: 447488 2024-04-22T18:38:58.800 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-22T18:38:58.800 INFO:tasks.rados.rados.0.smithi089.stdout:202: done (15 left) 2024-04-22T18:38:58.800 INFO:tasks.rados.rados.0.smithi089.stdout:220: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.800 INFO:tasks.rados.rados.0.smithi089.stdout:220: set_chunk oid 21 offset: 206848 length: 1879040 target oid smithi08937929-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 206848 snap -1 2024-04-22T18:38:58.800 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.800 INFO:tasks.rados.rados.0.smithi089.stdout:215: finishing set_chunk to oid 16 2024-04-22T18:38:58.800 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 822272 length: 517632 2024-04-22T18:38:58.800 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-22T18:38:58.800 INFO:tasks.rados.rados.0.smithi089.stdout:215: done (15 left) 2024-04-22T18:38:58.800 INFO:tasks.rados.rados.0.smithi089.stdout:221: set_chunk oid 22 target oid 22 low_tier 2024-04-22T18:38:58.800 INFO:tasks.rados.rados.0.smithi089.stdout:221: set_chunk oid 22 offset: 1859072 length: 493568 target oid smithi08937929-22 low_tier offset: 1859072 snap -1 2024-04-22T18:38:58.800 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.800 INFO:tasks.rados.rados.0.smithi089.stdout:219: finishing set_chunk to oid 20 2024-04-22T18:38:58.800 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 845824 length: 930304 2024-04-22T18:38:58.801 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-22T18:38:58.801 INFO:tasks.rados.rados.0.smithi089.stdout:219: done (15 left) 2024-04-22T18:38:58.801 INFO:tasks.rados.rados.0.smithi089.stdout:222: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.801 INFO:tasks.rados.rados.0.smithi089.stdout:222: set_chunk oid 23 offset: 164352 length: 1216000 target oid smithi08937929-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 164352 snap -1 2024-04-22T18:38:58.801 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.801 INFO:tasks.rados.rados.0.smithi089.stdout:216: finishing set_chunk to oid 17 2024-04-22T18:38:58.801 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1278976 length: 799232 2024-04-22T18:38:58.801 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-22T18:38:58.801 INFO:tasks.rados.rados.0.smithi089.stdout:216: done (15 left) 2024-04-22T18:38:58.801 INFO:tasks.rados.rados.0.smithi089.stdout:223: set_chunk oid 24 target oid 24 low_tier 2024-04-22T18:38:58.801 INFO:tasks.rados.rados.0.smithi089.stdout:223: set_chunk oid 24 offset: 12800 length: 1000448 target oid smithi08937929-24 low_tier offset: 12800 snap -1 2024-04-22T18:38:58.801 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.801 INFO:tasks.rados.rados.0.smithi089.stdout:207: finishing set_chunk to oid 8 2024-04-22T18:38:58.801 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 1041920 length: 22016 2024-04-22T18:38:58.801 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-22T18:38:58.802 INFO:tasks.rados.rados.0.smithi089.stdout:207: done (15 left) 2024-04-22T18:38:58.802 INFO:tasks.rados.rados.0.smithi089.stdout:224: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.802 INFO:tasks.rados.rados.0.smithi089.stdout:224: set_chunk oid 25 offset: 1110528 length: 1140736 target oid smithi08937929-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1110528 snap -1 2024-04-22T18:38:58.802 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.806 INFO:tasks.rados.rados.0.smithi089.stdout:194: finishing set_chunk to oid 45 2024-04-22T18:38:58.806 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-22T18:38:58.806 INFO:tasks.rados.rados.0.smithi089.stdout:184: finishing set_chunk to oid 35 2024-04-22T18:38:58.806 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 9 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-22T18:38:58.806 INFO:tasks.rados.rados.0.smithi089.stdout:184: done (15 left) 2024-04-22T18:38:58.806 INFO:tasks.rados.rados.0.smithi089.stdout:194: done (14 left) 2024-04-22T18:38:58.806 INFO:tasks.rados.rados.0.smithi089.stdout:225: set_chunk oid 26 target oid 26 low_tier 2024-04-22T18:38:58.806 INFO:tasks.rados.rados.0.smithi089.stdout:225: set_chunk oid 26 offset: 1139200 length: 239104 target oid smithi08937929-26 low_tier offset: 1139200 snap -1 2024-04-22T18:38:58.806 INFO:tasks.rados.rados.0.smithi089.stdout:226: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.806 INFO:tasks.rados.rados.0.smithi089.stdout:226: set_chunk oid 27 offset: 634368 length: 427008 target oid smithi08937929-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 634368 snap -1 2024-04-22T18:38:58.806 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.806 INFO:tasks.rados.rados.0.smithi089.stdout:199: finishing set_chunk to oid 50 2024-04-22T18:38:58.807 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 21 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-22T18:38:58.807 INFO:tasks.rados.rados.0.smithi089.stdout:199: done (15 left) 2024-04-22T18:38:58.807 INFO:tasks.rados.rados.0.smithi089.stdout:227: set_chunk oid 28 target oid 28 low_tier 2024-04-22T18:38:58.807 INFO:tasks.rados.rados.0.smithi089.stdout:227: set_chunk oid 28 offset: 1132544 length: 211456 target oid smithi08937929-28 low_tier offset: 1132544 snap -1 2024-04-22T18:38:58.807 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.807 INFO:tasks.rados.rados.0.smithi089.stdout:218: finishing set_chunk to oid 19 2024-04-22T18:38:58.807 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1532928 length: 664064 2024-04-22T18:38:58.807 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-22T18:38:58.807 INFO:tasks.rados.rados.0.smithi089.stdout:218: done (15 left) 2024-04-22T18:38:58.807 INFO:tasks.rados.rados.0.smithi089.stdout:228: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.807 INFO:tasks.rados.rados.0.smithi089.stdout:228: set_chunk oid 29 offset: 367104 length: 160768 target oid smithi08937929-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 367104 snap -1 2024-04-22T18:38:58.807 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.807 INFO:tasks.rados.rados.0.smithi089.stdout:222: finishing set_chunk to oid 23 2024-04-22T18:38:58.807 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 164352 length: 1216000 2024-04-22T18:38:58.807 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout:222: done (15 left) 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout:229: set_chunk oid 30 target oid 30 low_tier 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout:229: set_chunk oid 30 offset: 5120 length: 1209344 target oid smithi08937929-30 low_tier offset: 5120 snap -1 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout:223: finishing set_chunk to oid 24 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 12800 length: 1000448 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout:223: done (15 left) 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout:230: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout:230: set_chunk oid 31 offset: 2098176 length: 53248 target oid smithi08937929-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2098176 snap -1 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout:226: finishing set_chunk to oid 27 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 634368 length: 427008 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout:226: done (15 left) 2024-04-22T18:38:58.808 INFO:tasks.rados.rados.0.smithi089.stdout:230 retry set_chunk ! 2024-04-22T18:38:58.809 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 15 2024-04-22T18:38:58.809 INFO:tasks.rados.rados.0.smithi089.stdout:195: finishing set_chunk to oid 46 2024-04-22T18:38:58.809 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-22T18:38:58.809 INFO:tasks.rados.rados.0.smithi089.stdout:195: done (14 left) 2024-04-22T18:38:58.809 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 14 2024-04-22T18:38:58.809 INFO:tasks.rados.rados.0.smithi089.stdout:214: finishing set_chunk to oid 15 2024-04-22T18:38:58.809 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 429056 length: 1155072 2024-04-22T18:38:58.809 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-22T18:38:58.809 INFO:tasks.rados.rados.0.smithi089.stdout:214: done (13 left) 2024-04-22T18:38:58.809 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 13 2024-04-22T18:38:58.854 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:38:58.855+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:38:58.896 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:58.896 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:38:58.896 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:38:58.896 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:58.897 INFO:tasks.rados.rados.0.smithi089.stdout:228: finishing set_chunk to oid 29 2024-04-22T18:38:58.897 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 367104 length: 160768 2024-04-22T18:38:58.897 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-22T18:38:58.897 INFO:tasks.rados.rados.0.smithi089.stdout:220: finishing set_chunk to oid 21 2024-04-22T18:38:58.897 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 206848 length: 1879040 2024-04-22T18:38:58.897 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-22T18:38:58.897 INFO:tasks.rados.rados.0.smithi089.stdout:220: done (12 left) 2024-04-22T18:38:58.897 INFO:tasks.rados.rados.0.smithi089.stdout:228: done (11 left) 2024-04-22T18:38:58.897 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 11 2024-04-22T18:38:58.900 INFO:tasks.rados.rados.0.smithi089.stdout:208: finishing set_chunk to oid 9 2024-04-22T18:38:58.900 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1153024 length: 531968 2024-04-22T18:38:58.901 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-22T18:38:58.901 INFO:tasks.rados.rados.0.smithi089.stdout:208: done (10 left) 2024-04-22T18:38:58.901 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 10 2024-04-22T18:38:58.901 INFO:tasks.rados.rados.0.smithi089.stdout:229: finishing set_chunk to oid 30 2024-04-22T18:38:58.901 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 5120 length: 1209344 2024-04-22T18:38:58.901 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-22T18:38:58.901 INFO:tasks.rados.rados.0.smithi089.stdout:229: done (9 left) 2024-04-22T18:38:58.901 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 9 2024-04-22T18:38:58.901 INFO:tasks.rados.rados.0.smithi089.stdout:230: finishing set_chunk to oid 31 2024-04-22T18:38:58.901 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2098176 length: 53248 2024-04-22T18:38:58.901 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-22T18:38:58.901 INFO:tasks.rados.rados.0.smithi089.stdout:230: done (8 left) 2024-04-22T18:38:58.901 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 8 2024-04-22T18:38:58.901 INFO:tasks.rados.rados.0.smithi089.stdout:181: finishing set_chunk to oid 32 2024-04-22T18:38:58.902 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 3 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-22T18:38:58.902 INFO:tasks.rados.rados.0.smithi089.stdout:181: done (7 left) 2024-04-22T18:38:58.902 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 7 2024-04-22T18:38:58.902 INFO:tasks.rados.rados.0.smithi089.stdout:182: finishing set_chunk to oid 33 2024-04-22T18:38:58.902 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-22T18:38:58.902 INFO:tasks.rados.rados.0.smithi089.stdout:182: done (6 left) 2024-04-22T18:38:58.902 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 6 2024-04-22T18:38:58.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:38:58.955 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:38:58.956+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:38:58.998 INFO:tasks.rados.rados.0.smithi089.stdout:183: finishing set_chunk to oid 34 2024-04-22T18:38:58.998 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-22T18:38:58.998 INFO:tasks.rados.rados.0.smithi089.stdout:183: done (5 left) 2024-04-22T18:38:58.998 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 5 2024-04-22T18:38:58.999 INFO:tasks.rados.rados.0.smithi089.stdout:227: finishing set_chunk to oid 28 2024-04-22T18:38:58.999 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 15 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-22T18:38:58.999 INFO:tasks.rados.rados.0.smithi089.stdout:227: done (4 left) 2024-04-22T18:38:58.999 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 4 2024-04-22T18:38:59.001 INFO:tasks.rados.rados.0.smithi089.stdout:225: finishing set_chunk to oid 26 2024-04-22T18:38:59.001 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-22T18:38:59.001 INFO:tasks.rados.rados.0.smithi089.stdout:225: done (3 left) 2024-04-22T18:38:59.001 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 3 2024-04-22T18:38:59.008 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:59.008 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:38:59.008 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:38:59.008 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:38:59.008 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:59.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:38:59.058 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:38:59.059+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:38:59.103 INFO:tasks.rados.rados.0.smithi089.stdout:221: finishing set_chunk to oid 22 2024-04-22T18:38:59.103 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-22T18:38:59.103 INFO:tasks.rados.rados.0.smithi089.stdout:221: done (2 left) 2024-04-22T18:38:59.103 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 2 2024-04-22T18:38:59.103 INFO:tasks.rados.rados.0.smithi089.stdout:217: finishing set_chunk to oid 18 2024-04-22T18:38:59.104 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 9 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-22T18:38:59.104 INFO:tasks.rados.rados.0.smithi089.stdout:217: done (1 left) 2024-04-22T18:38:59.104 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 1 2024-04-22T18:38:59.104 INFO:tasks.rados.rados.0.smithi089.stdout:224: finishing set_chunk to oid 25 2024-04-22T18:38:59.104 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 12 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-22T18:38:59.104 INFO:tasks.rados.rados.0.smithi089.stdout:224: done (0 left) 2024-04-22T18:38:59.104 INFO:tasks.rados.rados.0.smithi089.stdout:231: set_chunk oid 32 target oid 32 low_tier 2024-04-22T18:38:59.104 INFO:tasks.rados.rados.0.smithi089.stdout:231: set_chunk oid 32 offset: 762368 length: 469504 target oid smithi08937929-32 low_tier offset: 762368 snap -1 2024-04-22T18:38:59.104 INFO:tasks.rados.rados.0.smithi089.stdout:232: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.104 INFO:tasks.rados.rados.0.smithi089.stdout:232: set_chunk oid 33 offset: 291328 length: 2046976 target oid smithi08937929-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 291328 snap -1 2024-04-22T18:38:59.104 INFO:tasks.rados.rados.0.smithi089.stdout:233: set_chunk oid 34 target oid 34 low_tier 2024-04-22T18:38:59.104 INFO:tasks.rados.rados.0.smithi089.stdout:233: set_chunk oid 34 offset: 1148928 length: 961024 target oid smithi08937929-34 low_tier offset: 1148928 snap -1 2024-04-22T18:38:59.104 INFO:tasks.rados.rados.0.smithi089.stdout:234: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.104 INFO:tasks.rados.rados.0.smithi089.stdout:234: set_chunk oid 35 offset: 591872 length: 2048 target oid smithi08937929-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 591872 snap -1 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:235: set_chunk oid 36 target oid 36 low_tier 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:235: set_chunk oid 36 offset: 606720 length: 1724416 target oid smithi08937929-36 low_tier offset: 606720 snap -1 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:236: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:236: set_chunk oid 37 offset: 733184 length: 1754624 target oid smithi08937929-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 733184 snap -1 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:237: set_chunk oid 38 target oid 38 low_tier 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:237: set_chunk oid 38 offset: 180736 length: 1139712 target oid smithi08937929-38 low_tier offset: 180736 snap -1 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:238: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:238: set_chunk oid 39 offset: 899584 length: 114688 target oid smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 899584 snap -1 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:239: set_chunk oid 40 target oid 40 low_tier 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:239: set_chunk oid 40 offset: 557568 length: 1289728 target oid smithi08937929-40 low_tier offset: 557568 snap -1 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:240: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:240: set_chunk oid 41 offset: 1128448 length: 882176 target oid smithi08937929-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1128448 snap -1 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:241: set_chunk oid 42 target oid 42 low_tier 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:241: set_chunk oid 42 offset: 720384 length: 210944 target oid smithi08937929-42 low_tier offset: 720384 snap -1 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:242: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.105 INFO:tasks.rados.rados.0.smithi089.stdout:242: set_chunk oid 43 offset: 158720 length: 1909760 target oid smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 158720 snap -1 2024-04-22T18:38:59.106 INFO:tasks.rados.rados.0.smithi089.stdout:243: set_chunk oid 44 target oid 44 low_tier 2024-04-22T18:38:59.106 INFO:tasks.rados.rados.0.smithi089.stdout:243: set_chunk oid 44 offset: 2236928 length: 10240 target oid smithi08937929-44 low_tier offset: 2236928 snap -1 2024-04-22T18:38:59.106 INFO:tasks.rados.rados.0.smithi089.stdout:244: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.106 INFO:tasks.rados.rados.0.smithi089.stdout:244: set_chunk oid 45 offset: 1619456 length: 555008 target oid smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1619456 snap -1 2024-04-22T18:38:59.106 INFO:tasks.rados.rados.0.smithi089.stdout:245: set_chunk oid 46 target oid 46 low_tier 2024-04-22T18:38:59.106 INFO:tasks.rados.rados.0.smithi089.stdout:245: set_chunk oid 46 offset: 908800 length: 587264 target oid smithi08937929-46 low_tier offset: 908800 snap -1 2024-04-22T18:38:59.106 INFO:tasks.rados.rados.0.smithi089.stdout:246: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.106 INFO:tasks.rados.rados.0.smithi089.stdout:246: set_chunk oid 47 offset: 162816 length: 1676288 target oid smithi08937929-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 162816 snap -1 2024-04-22T18:38:59.106 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.106 INFO:tasks.rados.rados.0.smithi089.stdout:236: finishing set_chunk to oid 37 2024-04-22T18:38:59.106 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 733184 length: 1754624 2024-04-22T18:38:59.106 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-22T18:38:59.106 INFO:tasks.rados.rados.0.smithi089.stdout:232: finishing set_chunk to oid 33 2024-04-22T18:38:59.106 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 291328 length: 2046976 2024-04-22T18:38:59.106 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-22T18:38:59.107 INFO:tasks.rados.rados.0.smithi089.stdout:232: done (15 left) 2024-04-22T18:38:59.107 INFO:tasks.rados.rados.0.smithi089.stdout:236: done (14 left) 2024-04-22T18:38:59.107 INFO:tasks.rados.rados.0.smithi089.stdout:247: set_chunk oid 48 target oid 48 low_tier 2024-04-22T18:38:59.107 INFO:tasks.rados.rados.0.smithi089.stdout:247: set_chunk oid 48 offset: 1363456 length: 1326592 target oid smithi08937929-48 low_tier offset: 1363456 snap -1 2024-04-22T18:38:59.107 INFO:tasks.rados.rados.0.smithi089.stdout:248: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.107 INFO:tasks.rados.rados.0.smithi089.stdout:248: set_chunk oid 49 offset: 2009600 length: 228352 target oid smithi08937929-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2009600 snap -1 2024-04-22T18:38:59.107 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.107 INFO:tasks.rados.rados.0.smithi089.stdout:241: finishing set_chunk to oid 42 2024-04-22T18:38:59.107 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 720384 length: 210944 2024-04-22T18:38:59.107 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-22T18:38:59.107 INFO:tasks.rados.rados.0.smithi089.stdout:239: finishing set_chunk to oid 40 2024-04-22T18:38:59.107 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 557568 length: 1289728 2024-04-22T18:38:59.107 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-22T18:38:59.107 INFO:tasks.rados.rados.0.smithi089.stdout:239: done (15 left) 2024-04-22T18:38:59.108 INFO:tasks.rados.rados.0.smithi089.stdout:241: done (14 left) 2024-04-22T18:38:59.108 INFO:tasks.rados.rados.0.smithi089.stdout:249: set_chunk oid 50 target oid 50 low_tier 2024-04-22T18:38:59.108 INFO:tasks.rados.rados.0.smithi089.stdout:249: set_chunk oid 50 offset: 871936 length: 201728 target oid smithi08937929-50 low_tier offset: 871936 snap -1 2024-04-22T18:38:59.108 INFO:tasks.rados.rados.0.smithi089.stdout:250: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.108 INFO:tasks.rados.rados.0.smithi089.stdout:250: set_chunk oid 1 offset: 1529344 length: 112128 target oid smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1529344 snap -1 2024-04-22T18:38:59.108 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.108 INFO:tasks.rados.rados.0.smithi089.stdout:246: finishing set_chunk to oid 47 2024-04-22T18:38:59.108 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 162816 length: 1676288 2024-04-22T18:38:59.108 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-22T18:38:59.108 INFO:tasks.rados.rados.0.smithi089.stdout:246: done (15 left) 2024-04-22T18:38:59.108 INFO:tasks.rados.rados.0.smithi089.stdout:251: set_chunk oid 2 target oid 2 low_tier 2024-04-22T18:38:59.108 INFO:tasks.rados.rados.0.smithi089.stdout:251: set_chunk oid 2 offset: 404992 length: 229376 target oid smithi08937929-2 low_tier offset: 404992 snap -1 2024-04-22T18:38:59.108 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.108 INFO:tasks.rados.rados.0.smithi089.stdout:242: finishing set_chunk to oid 43 2024-04-22T18:38:59.108 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 158720 length: 1909760 2024-04-22T18:38:59.109 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-22T18:38:59.109 INFO:tasks.rados.rados.0.smithi089.stdout:242: done (15 left) 2024-04-22T18:38:59.109 INFO:tasks.rados.rados.0.smithi089.stdout:252: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.109 INFO:tasks.rados.rados.0.smithi089.stdout:252: set_chunk oid 3 offset: 8704 length: 446976 target oid smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 8704 snap -1 2024-04-22T18:38:59.109 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.109 INFO:tasks.rados.rados.0.smithi089.stdout:248: finishing set_chunk to oid 49 2024-04-22T18:38:59.109 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2009600 length: 228352 2024-04-22T18:38:59.109 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:38:59.109 INFO:tasks.rados.rados.0.smithi089.stdout:248: done (15 left) 2024-04-22T18:38:59.109 INFO:tasks.rados.rados.0.smithi089.stdout:253: set_chunk oid 4 target oid 4 low_tier 2024-04-22T18:38:59.109 INFO:tasks.rados.rados.0.smithi089.stdout:253: set_chunk oid 4 offset: 528896 length: 1132032 target oid smithi08937929-4 low_tier offset: 528896 snap -1 2024-04-22T18:38:59.109 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.124 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:59.125+0000 7f3e7be88640 -1 osd.0 30 handle_osd_map injecting map crc failure 2024-04-22T18:38:59.126 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:38:59.156 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:59.157+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:59.162 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:38:59.162 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:38:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:38:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:38:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:38:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:38:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.420795163000001, 2024-04-22T18:38:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30520528099999999, 2024-04-22T18:38:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 35 2024-04-22T18:38:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741852+0000", 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741927+0000", 2024-04-22T18:38:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5123000000000006e-05 2024-04-22T18:38:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741953+0000", 2024-04-22T18:38:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5378999999999999e-05 2024-04-22T18:38:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741977+0000", 2024-04-22T18:38:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4431999999999999e-05 2024-04-22T18:38:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844243+0000", 2024-04-22T18:38:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102265525 2024-04-22T18:38:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844330+0000", 2024-04-22T18:38:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7417999999999995e-05 2024-04-22T18:38:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844470+0000", 2024-04-22T18:38:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000140197 2024-04-22T18:38:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944767+0000", 2024-04-22T18:38:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100296409 2024-04-22T18:38:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.046968+0000", 2024-04-22T18:38:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10220148499999999 2024-04-22T18:38:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047018+0000", 2024-04-22T18:38:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9206000000000003e-05 2024-04-22T18:38:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047059+0000", 2024-04-22T18:38:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1496999999999999e-05 2024-04-22T18:38:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:38:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:38:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.420664639000002, 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:38:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:38:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:38:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:38:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:38:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:38:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:38:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:38:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:38:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:38:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:38:59.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:38:59.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:38:59.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:38:59.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:38:59.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:38:59.173 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:38:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:38:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.420624948, 2024-04-22T18:38:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:38:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:38:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:38:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:38:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:38:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:38:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:38:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:38:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:38:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:38:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:38:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:38:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:38:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:38:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:38:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:38:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:38:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:38:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:38:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:38:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:38:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:38:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:38:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:38:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:38:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:38:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:38:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.176443900999999, 2024-04-22T18:38:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402419109, 2024-04-22T18:38:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 68 2024-04-22T18:38:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:38:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:38:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986204+0000", 2024-04-22T18:38:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986208+0000", 2024-04-22T18:38:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8149999999999999e-06 2024-04-22T18:38:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986214+0000", 2024-04-22T18:38:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.604e-06 2024-04-22T18:38:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986218+0000", 2024-04-22T18:38:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6689999999999999e-06 2024-04-22T18:38:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986833+0000", 2024-04-22T18:38:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000615648 2024-04-22T18:38:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986891+0000", 2024-04-22T18:38:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7915999999999999e-05 2024-04-22T18:38:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986981+0000", 2024-04-22T18:38:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759999999999994e-05 2024-04-22T18:38:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.987584+0000", 2024-04-22T18:38:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060270199999999999 2024-04-22T18:38:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388527+0000", 2024-04-22T18:38:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40094366999999997 2024-04-22T18:38:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388583+0000", 2024-04-22T18:38:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5383000000000002e-05 2024-04-22T18:38:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388624+0000", 2024-04-22T18:38:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501999999999997e-05 2024-04-22T18:38:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:38:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:38:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.176428112, 2024-04-22T18:38:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:38:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:38:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:38:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:38:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:38:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:38:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:38:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:38:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:38:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:38:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:38:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:38:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:38:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:38:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:38:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:38:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:38:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:38:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:38:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:38:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:38:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:38:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:38:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:38:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:38:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:38:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:38:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 16.060035103000001, 2024-04-22T18:38:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305563733, 2024-04-22T18:38:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 87 2024-04-22T18:38:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:38:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:38:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102613+0000", 2024-04-22T18:38:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102668+0000", 2024-04-22T18:38:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4669000000000001e-05 2024-04-22T18:38:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102680+0000", 2024-04-22T18:38:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2026e-05 2024-04-22T18:38:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102687+0000", 2024-04-22T18:38:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8339999999999999e-06 2024-04-22T18:38:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405644+0000", 2024-04-22T18:38:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30295623199999999 2024-04-22T18:38:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405743+0000", 2024-04-22T18:38:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.98e-05 2024-04-22T18:38:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405868+0000", 2024-04-22T18:38:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012457899999999999 2024-04-22T18:38:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407047+0000", 2024-04-22T18:38:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011791449999999999 2024-04-22T18:38:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408088+0000", 2024-04-22T18:38:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010411680000000001 2024-04-22T18:38:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408135+0000", 2024-04-22T18:38:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6255999999999999e-05 2024-04-22T18:38:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408178+0000", 2024-04-22T18:38:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2877000000000003e-05 2024-04-22T18:38:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:38:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:38:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 16.059955173999999, 2024-04-22T18:38:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30560089000000001, 2024-04-22T18:38:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 88 2024-04-22T18:38:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:38:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:38:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102693+0000", 2024-04-22T18:38:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:38:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102698+0000", 2024-04-22T18:38:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9699999999999998e-06 2024-04-22T18:38:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102708+0000", 2024-04-22T18:38:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.006e-05 2024-04-22T18:38:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102715+0000", 2024-04-22T18:38:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6599999999999995e-06 2024-04-22T18:38:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406637+0000", 2024-04-22T18:38:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392130099999998 2024-04-22T18:38:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406696+0000", 2024-04-22T18:38:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9066e-05 2024-04-22T18:38:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406784+0000", 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8194000000000003e-05 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407447+0000", 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066295700000000004 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408237+0000", 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079032600000000005 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408270+0000", 2024-04-22T18:38:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3157000000000001e-05 2024-04-22T18:38:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408295+0000", 2024-04-22T18:38:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4151999999999998e-05 2024-04-22T18:38:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:38:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.308723049999999, 2024-04-22T18:38:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30533819600000001, 2024-04-22T18:38:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 106 2024-04-22T18:38:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:38:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:38:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853925+0000", 2024-04-22T18:38:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853958+0000", 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2536000000000002e-05 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853964+0000", 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6239999999999999e-06 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853967+0000", 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9700000000000001e-06 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855144+0000", 2024-04-22T18:38:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001176774 2024-04-22T18:38:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855225+0000", 2024-04-22T18:38:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0697000000000003e-05 2024-04-22T18:38:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855353+0000", 2024-04-22T18:38:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012788 2024-04-22T18:38:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058003+0000", 2024-04-22T18:38:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202649939 2024-04-22T18:38:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159184+0000", 2024-04-22T18:38:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101181058 2024-04-22T18:38:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159238+0000", 2024-04-22T18:38:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3967e-05 2024-04-22T18:38:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159264+0000", 2024-04-22T18:38:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6237999999999998e-05 2024-04-22T18:38:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:38:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.308676200000001, 2024-04-22T18:38:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30544294500000002, 2024-04-22T18:38:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 107 2024-04-22T18:38:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:38:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:38:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853972+0000", 2024-04-22T18:38:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853975+0000", 2024-04-22T18:38:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0939999999999999e-06 2024-04-22T18:38:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853978+0000", 2024-04-22T18:38:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:38:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853981+0000", 2024-04-22T18:38:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.858e-06 2024-04-22T18:38:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855831+0000", 2024-04-22T18:38:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018503689999999999 2024-04-22T18:38:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855932+0000", 2024-04-22T18:38:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010071 2024-04-22T18:38:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856029+0000", 2024-04-22T18:38:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6751999999999998e-05 2024-04-22T18:38:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058013+0000", 2024-04-22T18:38:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20198412600000001 2024-04-22T18:38:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159326+0000", 2024-04-22T18:38:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101312969 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159376+0000", 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9968e-05 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159416+0000", 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9641999999999998e-05 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:38:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.308587804, 2024-04-22T18:38:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30548257499999998, 2024-04-22T18:38:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 108 2024-04-22T18:38:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:38:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:38:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:38:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854064+0000", 2024-04-22T18:38:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5820000000000002e-06 2024-04-22T18:38:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854080+0000", 2024-04-22T18:38:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5379999999999998e-05 2024-04-22T18:38:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854084+0000", 2024-04-22T18:38:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.963e-06 2024-04-22T18:38:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856413+0000", 2024-04-22T18:38:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023298070000000001 2024-04-22T18:38:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856494+0000", 2024-04-22T18:38:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0863000000000002e-05 2024-04-22T18:38:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856624+0000", 2024-04-22T18:38:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129799 2024-04-22T18:38:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058947+0000", 2024-04-22T18:38:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20232328899999999 2024-04-22T18:38:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159468+0000", 2024-04-22T18:38:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100520597 2024-04-22T18:38:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159508+0000", 2024-04-22T18:38:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9830000000000003e-05 2024-04-22T18:38:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159544+0000", 2024-04-22T18:38:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5944e-05 2024-04-22T18:38:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:38:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.308543311999999, 2024-04-22T18:38:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30556502499999999, 2024-04-22T18:38:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854105+0000", 2024-04-22T18:38:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854175+0000", 2024-04-22T18:38:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0178000000000005e-05 2024-04-22T18:38:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854191+0000", 2024-04-22T18:38:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5965999999999999e-05 2024-04-22T18:38:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854196+0000", 2024-04-22T18:38:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.42e-06 2024-04-22T18:38:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956526+0000", 2024-04-22T18:38:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10233094299999999 2024-04-22T18:38:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956634+0000", 2024-04-22T18:38:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107398 2024-04-22T18:38:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956730+0000", 2024-04-22T18:38:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6217000000000004e-05 2024-04-22T18:38:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058955+0000", 2024-04-22T18:38:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102224749 2024-04-22T18:38:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159596+0000", 2024-04-22T18:38:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100641297 2024-04-22T18:38:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159637+0000", 2024-04-22T18:38:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874000000000002e-05 2024-04-22T18:38:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159671+0000", 2024-04-22T18:38:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3603999999999999e-05 2024-04-22T18:38:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:38:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.308449465000001, 2024-04-22T18:38:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30683048800000001, 2024-04-22T18:38:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110 2024-04-22T18:38:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:38:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:38:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:38:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854202+0000", 2024-04-22T18:38:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:38:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854206+0000", 2024-04-22T18:38:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5870000000000001e-06 2024-04-22T18:38:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854208+0000", 2024-04-22T18:38:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7199999999999998e-06 2024-04-22T18:38:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057285+0000", 2024-04-22T18:38:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20307620800000001 2024-04-22T18:38:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057414+0000", 2024-04-22T18:38:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012896899999999999 2024-04-22T18:38:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057588+0000", 2024-04-22T18:38:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000174802 2024-04-22T18:38:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159429+0000", 2024-04-22T18:38:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101840783 2024-04-22T18:38:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160947+0000", 2024-04-22T18:38:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001518152 2024-04-22T18:38:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160992+0000", 2024-04-22T18:38:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4725000000000003e-05 2024-04-22T18:38:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161030+0000", 2024-04-22T18:38:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7789e-05 2024-04-22T18:38:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:38:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.308438283999999, 2024-04-22T18:38:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:38:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:38:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:38:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:38:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:38:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:38:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:38:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:38:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:38:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:38:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:38:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:38:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:38:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:38:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:38:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:38:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:38:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:38:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:38:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:38:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:38:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:38:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:38:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:38:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:38:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:38:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.308422626, 2024-04-22T18:38:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:38:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:38:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:38:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:38:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:38:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:38:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:38:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:38:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:38:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:38:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:38:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:38:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:38:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:38:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:38:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:38:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:38:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:38:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:38:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:38:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:38:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:38:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:38:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:38:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:38:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:38:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.308371464, 2024-04-22T18:38:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:38:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:38:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:38:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:38:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:38:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:38:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:38:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:38:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:38:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:38:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:38:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:38:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:38:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:38:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:38:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:38:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:38:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:38:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:38:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:38:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:38:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:38:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:38:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:38:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:38:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:38:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.30835624, 2024-04-22T18:38:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:38:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:38:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:38:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:38:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:38:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:38:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:38:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:38:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:38:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:38:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:38:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:38:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:38:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:38:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:38:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:38:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:38:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:38:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:38:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:38:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:38:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:38:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:38:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:38:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:38:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:38:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.308296663, 2024-04-22T18:38:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:38:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:38:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:38:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:38:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:38:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:38:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:38:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:38:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:38:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:38:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:38:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:38:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:38:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:38:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:38:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:38:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:38:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:38:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:38:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:38:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:38:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:38:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:38:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:38:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:38:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:38:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.308214347, 2024-04-22T18:38:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:38:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:38:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:38:59.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:38:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:38:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:38:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:38:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:38:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:38:59.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:38:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:38:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:38:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:38:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:38:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:38:59.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:38:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:38:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:38:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:38:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:38:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:38:59.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:38:59.264 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.264 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.264 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.264 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:38:59.264 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:38:59.264 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.308204191, 2024-04-22T18:38:59.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:38:59.264 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.264 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.264 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.264 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.264 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:38:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:38:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:38:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:38:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:38:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:38:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:38:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:38:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:38:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:38:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:38:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:38:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:38:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:38:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:38:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:38:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:38:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:38:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:38:59.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:38:59.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:38:59.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:38:59.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:38:59.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:38:59.269 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.269 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.269 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.270 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:161 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:38:59.270 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:38:59.270 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.707709899999999, 2024-04-22T18:38:59.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40239169000000002, 2024-04-22T18:38:59.270 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:38:59.270 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.270 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:38:59.270 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:38:59.270 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:38:59.270 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 161 2024-04-22T18:38:59.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.270 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:38:59.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:38:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:38:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:38:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:38:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:38:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:38:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454938+0000", 2024-04-22T18:38:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454982+0000", 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3470999999999998e-05 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454988+0000", 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7640000000000002e-06 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555054+0000", 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006607300000001 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:38:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555607+0000", 2024-04-22T18:38:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055319500000000003 2024-04-22T18:38:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:38:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555689+0000", 2024-04-22T18:38:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1817999999999995e-05 2024-04-22T18:38:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:38:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555804+0000", 2024-04-22T18:38:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115346 2024-04-22T18:38:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:38:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556554+0000", 2024-04-22T18:38:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000750102 2024-04-22T18:38:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857265+0000", 2024-04-22T18:38:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30071122700000003 2024-04-22T18:38:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:38:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857302+0000", 2024-04-22T18:38:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7069e-05 2024-04-22T18:38:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:38:59.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:38:59.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:38:59.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857331+0000", 2024-04-22T18:38:59.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8291999999999999e-05 2024-04-22T18:38:59.275 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.275 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.275 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.275 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:38:59.275 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:38:59.275 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:38:59.277 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:38:59.279 INFO:tasks.rados.rados.0.smithi089.stdout:231: finishing set_chunk to oid 32 2024-04-22T18:38:59.279 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 762368 length: 469504 2024-04-22T18:38:59.279 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-22T18:38:59.280 INFO:tasks.rados.rados.0.smithi089.stdout:244: finishing set_chunk to oid 45 2024-04-22T18:38:59.280 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1619456 length: 555008 2024-04-22T18:38:59.280 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-22T18:38:59.280 INFO:tasks.rados.rados.0.smithi089.stdout:231: done (15 left) 2024-04-22T18:38:59.280 INFO:tasks.rados.rados.0.smithi089.stdout:244: done (14 left) 2024-04-22T18:38:59.280 INFO:tasks.rados.rados.0.smithi089.stdout:254: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.280 INFO:tasks.rados.rados.0.smithi089.stdout:254: set_chunk oid 5 offset: 490496 length: 542720 target oid smithi08937929-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 490496 snap -1 2024-04-22T18:38:59.280 INFO:tasks.rados.rados.0.smithi089.stdout:255: set_chunk oid 6 target oid 6 low_tier 2024-04-22T18:38:59.280 INFO:tasks.rados.rados.0.smithi089.stdout:235: finishing set_chunk to oid 36 2024-04-22T18:38:59.280 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 606720 length: 1724416 2024-04-22T18:38:59.280 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-22T18:38:59.280 INFO:tasks.rados.rados.0.smithi089.stdout:255: set_chunk oid 6 offset: 1052672 length: 261632 target oid smithi08937929-6 low_tier offset: 1052672 snap -1 2024-04-22T18:38:59.280 INFO:tasks.rados.rados.0.smithi089.stdout:235: done (15 left) 2024-04-22T18:38:59.280 INFO:tasks.rados.rados.0.smithi089.stdout:256: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.281 INFO:tasks.rados.rados.0.smithi089.stdout:256: set_chunk oid 7 offset: 0 length: 494592 target oid smithi08937929-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 0 snap -1 2024-04-22T18:38:59.281 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.281 INFO:tasks.rados.rados.0.smithi089.stdout:247: finishing set_chunk to oid 48 2024-04-22T18:38:59.281 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1363456 length: 1326592 2024-04-22T18:38:59.281 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-22T18:38:59.281 INFO:tasks.rados.rados.0.smithi089.stdout:247: done (15 left) 2024-04-22T18:38:59.281 INFO:tasks.rados.rados.0.smithi089.stdout:257: set_chunk oid 8 target oid 8 low_tier 2024-04-22T18:38:59.281 INFO:tasks.rados.rados.0.smithi089.stdout:257: set_chunk oid 8 offset: 1081344 length: 60416 target oid smithi08937929-8 low_tier offset: 1081344 snap -1 2024-04-22T18:38:59.281 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.281 INFO:tasks.rados.rados.0.smithi089.stdout:255: finishing set_chunk to oid 6 2024-04-22T18:38:59.281 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1052672 length: 261632 2024-04-22T18:38:59.281 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-22T18:38:59.281 INFO:tasks.rados.rados.0.smithi089.stdout:255: done (15 left) 2024-04-22T18:38:59.281 INFO:tasks.rados.rados.0.smithi089.stdout:258: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.282 INFO:tasks.rados.rados.0.smithi089.stdout:258: set_chunk oid 9 offset: 1063424 length: 167936 target oid smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1063424 snap -1 2024-04-22T18:38:59.282 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.282 INFO:tasks.rados.rados.0.smithi089.stdout:251: finishing set_chunk to oid 2 2024-04-22T18:38:59.282 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 404992 length: 229376 2024-04-22T18:38:59.282 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-22T18:38:59.282 INFO:tasks.rados.rados.0.smithi089.stdout:251: done (15 left) 2024-04-22T18:38:59.282 INFO:tasks.rados.rados.0.smithi089.stdout:259: set_chunk oid 10 target oid 10 low_tier 2024-04-22T18:38:59.282 INFO:tasks.rados.rados.0.smithi089.stdout:259: set_chunk oid 10 offset: 295424 length: 78848 target oid smithi08937929-10 low_tier offset: 295424 snap -1 2024-04-22T18:38:59.282 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.282 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:59.277+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:38:59.300 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:38:59.300+0000 7f8605f32640 -1 osd.3 30 handle_osd_map injecting map crc failure 2024-04-22T18:38:59.301 INFO:tasks.rados.rados.0.smithi089.stdout:252: finishing set_chunk to oid 3 2024-04-22T18:38:59.301 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 8704 length: 446976 2024-04-22T18:38:59.301 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-22T18:38:59.301 INFO:tasks.rados.rados.0.smithi089.stdout:252: done (15 left) 2024-04-22T18:38:59.301 INFO:tasks.rados.rados.0.smithi089.stdout:260: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.301 INFO:tasks.rados.rados.0.smithi089.stdout:260: set_chunk oid 11 offset: 1017856 length: 380416 target oid smithi08937929-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1017856 snap -1 2024-04-22T18:38:59.301 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.301 INFO:tasks.rados.rados.0.smithi089.stdout:258: finishing set_chunk to oid 9 2024-04-22T18:38:59.301 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1063424 length: 167936 2024-04-22T18:38:59.301 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-22T18:38:59.302 INFO:tasks.rados.rados.0.smithi089.stdout:260: finishing set_chunk to oid 11 2024-04-22T18:38:59.302 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1017856 length: 380416 2024-04-22T18:38:59.302 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-22T18:38:59.302 INFO:tasks.rados.rados.0.smithi089.stdout:258: done (15 left) 2024-04-22T18:38:59.302 INFO:tasks.rados.rados.0.smithi089.stdout:260: done (14 left) 2024-04-22T18:38:59.302 INFO:tasks.rados.rados.0.smithi089.stdout:261: set_chunk oid 12 target oid 12 low_tier 2024-04-22T18:38:59.302 INFO:tasks.rados.rados.0.smithi089.stdout:261: set_chunk oid 12 offset: 776704 length: 1313280 target oid smithi08937929-12 low_tier offset: 776704 snap -1 2024-04-22T18:38:59.302 INFO:tasks.rados.rados.0.smithi089.stdout:262: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.302 INFO:tasks.rados.rados.0.smithi089.stdout:262: set_chunk oid 13 offset: 219136 length: 1700352 target oid smithi08937929-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 219136 snap -1 2024-04-22T18:38:59.302 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.377 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:38:59.378+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:38:59.389 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:59.389 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:38:59.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.389 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:393 4.b 4:defe92fb:::smithi08937929-44:head [assert-version v12,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:38:59.389 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:38:59.389 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.28998046, 2024-04-22T18:38:59.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29000685799999998, 2024-04-22T18:38:59.390 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.390 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:38:59.390 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.390 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.390 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.390 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 393 2024-04-22T18:38:59.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.390 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:38:59.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:38:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101194+0000", 2024-04-22T18:38:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101201+0000", 2024-04-22T18:38:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1269999999999997e-06 2024-04-22T18:38:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101211+0000", 2024-04-22T18:38:59.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9380000000000004e-06 2024-04-22T18:38:59.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101220+0000", 2024-04-22T18:38:59.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3459999999999994e-06 2024-04-22T18:38:59.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101324+0000", 2024-04-22T18:38:59.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104366 2024-04-22T18:38:59.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101379+0000", 2024-04-22T18:38:59.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5127999999999998e-05 2024-04-22T18:38:59.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:59.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.330649+0000", 2024-04-22T18:38:59.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.22927022699999999 2024-04-22T18:38:59.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.331182+0000", 2024-04-22T18:38:59.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053250900000000002 2024-04-22T18:38:59.394 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.394 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.394 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.394 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.394 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:38:59.394 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 1 2024-04-22T18:38:59.394 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:59.398 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:38:59.403 INFO:tasks.rados.rados.0.smithi089.stdout:233: finishing set_chunk to oid 34 2024-04-22T18:38:59.403 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 1148928 length: 961024 2024-04-22T18:38:59.403 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-22T18:38:59.403 INFO:tasks.rados.rados.0.smithi089.stdout:237: finishing set_chunk to oid 38 2024-04-22T18:38:59.403 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 180736 length: 1139712 2024-04-22T18:38:59.403 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-22T18:38:59.403 INFO:tasks.rados.rados.0.smithi089.stdout:245: finishing set_chunk to oid 46 2024-04-22T18:38:59.403 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 908800 length: 587264 2024-04-22T18:38:59.403 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-22T18:38:59.403 INFO:tasks.rados.rados.0.smithi089.stdout:240: finishing set_chunk to oid 41 2024-04-22T18:38:59.403 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1128448 length: 882176 2024-04-22T18:38:59.404 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-22T18:38:59.404 INFO:tasks.rados.rados.0.smithi089.stdout:253: finishing set_chunk to oid 4 2024-04-22T18:38:59.404 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 528896 length: 1132032 2024-04-22T18:38:59.404 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-22T18:38:59.404 INFO:tasks.rados.rados.0.smithi089.stdout:256: finishing set_chunk to oid 7 2024-04-22T18:38:59.404 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 0 length: 494592 2024-04-22T18:38:59.404 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-22T18:38:59.404 INFO:tasks.rados.rados.0.smithi089.stdout:257: finishing set_chunk to oid 8 2024-04-22T18:38:59.404 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 1081344 length: 60416 2024-04-22T18:38:59.404 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-22T18:38:59.404 INFO:tasks.rados.rados.0.smithi089.stdout:233: done (15 left) 2024-04-22T18:38:59.404 INFO:tasks.rados.rados.0.smithi089.stdout:237: done (14 left) 2024-04-22T18:38:59.404 INFO:tasks.rados.rados.0.smithi089.stdout:240: done (13 left) 2024-04-22T18:38:59.405 INFO:tasks.rados.rados.0.smithi089.stdout:245: done (12 left) 2024-04-22T18:38:59.405 INFO:tasks.rados.rados.0.smithi089.stdout:253: done (11 left) 2024-04-22T18:38:59.405 INFO:tasks.rados.rados.0.smithi089.stdout:256: done (10 left) 2024-04-22T18:38:59.405 INFO:tasks.rados.rados.0.smithi089.stdout:257: done (9 left) 2024-04-22T18:38:59.405 INFO:tasks.rados.rados.0.smithi089.stdout:263: set_chunk oid 14 target oid 14 low_tier 2024-04-22T18:38:59.405 INFO:tasks.rados.rados.0.smithi089.stdout:263: set_chunk oid 14 offset: 41984 length: 1281024 target oid smithi08937929-14 low_tier offset: 41984 snap -1 2024-04-22T18:38:59.405 INFO:tasks.rados.rados.0.smithi089.stdout:264: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.405 INFO:tasks.rados.rados.0.smithi089.stdout:264: set_chunk oid 15 offset: 496640 length: 1432576 target oid smithi08937929-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 496640 snap -1 2024-04-22T18:38:59.405 INFO:tasks.rados.rados.0.smithi089.stdout:265: set_chunk oid 16 target oid 16 low_tier 2024-04-22T18:38:59.405 INFO:tasks.rados.rados.0.smithi089.stdout:265: set_chunk oid 16 offset: 1840640 length: 331264 target oid smithi08937929-16 low_tier offset: 1840640 snap -1 2024-04-22T18:38:59.405 INFO:tasks.rados.rados.0.smithi089.stdout:266: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.405 INFO:tasks.rados.rados.0.smithi089.stdout:266: set_chunk oid 17 offset: 644608 length: 420352 target oid smithi08937929-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 644608 snap -1 2024-04-22T18:38:59.405 INFO:tasks.rados.rados.0.smithi089.stdout:267: set_chunk oid 18 target oid 18 low_tier 2024-04-22T18:38:59.405 INFO:tasks.rados.rados.0.smithi089.stdout:267: set_chunk oid 18 offset: 403968 length: 512512 target oid smithi08937929-18 low_tier offset: 403968 snap -1 2024-04-22T18:38:59.405 INFO:tasks.rados.rados.0.smithi089.stdout:268: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.406 INFO:tasks.rados.rados.0.smithi089.stdout:268: set_chunk oid 19 offset: 883712 length: 1372672 target oid smithi08937929-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 883712 snap -1 2024-04-22T18:38:59.406 INFO:tasks.rados.rados.0.smithi089.stdout:269: set_chunk oid 20 target oid 20 low_tier 2024-04-22T18:38:59.406 INFO:tasks.rados.rados.0.smithi089.stdout:269: set_chunk oid 20 offset: 2146816 length: 141824 target oid smithi08937929-20 low_tier offset: 2146816 snap -1 2024-04-22T18:38:59.406 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.406 INFO:tasks.rados.rados.0.smithi089.stdout:249: finishing set_chunk to oid 50 2024-04-22T18:38:59.406 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 871936 length: 201728 2024-04-22T18:38:59.406 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-22T18:38:59.406 INFO:tasks.rados.rados.0.smithi089.stdout:249: done (15 left) 2024-04-22T18:38:59.406 INFO:tasks.rados.rados.0.smithi089.stdout:270: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.406 INFO:tasks.rados.rados.0.smithi089.stdout:270: set_chunk oid 21 offset: 182272 length: 1471488 target oid smithi08937929-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 182272 snap -1 2024-04-22T18:38:59.406 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.406 INFO:tasks.rados.rados.0.smithi089.stdout:263: finishing set_chunk to oid 14 2024-04-22T18:38:59.406 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 41984 length: 1281024 2024-04-22T18:38:59.407 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-22T18:38:59.407 INFO:tasks.rados.rados.0.smithi089.stdout:263: done (15 left) 2024-04-22T18:38:59.407 INFO:tasks.rados.rados.0.smithi089.stdout:271: set_chunk oid 22 target oid 22 low_tier 2024-04-22T18:38:59.407 INFO:tasks.rados.rados.0.smithi089.stdout:271: set_chunk oid 22 offset: 1683456 length: 667136 target oid smithi08937929-22 low_tier offset: 1683456 snap -1 2024-04-22T18:38:59.407 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.407 INFO:tasks.rados.rados.0.smithi089.stdout:250: finishing set_chunk to oid 1 2024-04-22T18:38:59.407 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-22T18:38:59.407 INFO:tasks.rados.rados.0.smithi089.stdout:250: done (15 left) 2024-04-22T18:38:59.407 INFO:tasks.rados.rados.0.smithi089.stdout:272: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.407 INFO:tasks.rados.rados.0.smithi089.stdout:272: set_chunk oid 23 offset: 243712 length: 873472 target oid smithi08937929-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 243712 snap -1 2024-04-22T18:38:59.407 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.414 INFO:tasks.rados.rados.0.smithi089.stdout:270: finishing set_chunk to oid 21 2024-04-22T18:38:59.415 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 182272 length: 1471488 2024-04-22T18:38:59.415 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-22T18:38:59.415 INFO:tasks.rados.rados.0.smithi089.stdout:270: done (15 left) 2024-04-22T18:38:59.415 INFO:tasks.rados.rados.0.smithi089.stdout:273: set_chunk oid 24 target oid 24 low_tier 2024-04-22T18:38:59.415 INFO:tasks.rados.rados.0.smithi089.stdout:273: set_chunk oid 24 offset: 212480 length: 1927168 target oid smithi08937929-24 low_tier offset: 212480 snap -1 2024-04-22T18:38:59.415 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.415 INFO:tasks.rados.rados.0.smithi089.stdout:259: finishing set_chunk to oid 10 2024-04-22T18:38:59.415 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-22T18:38:59.415 INFO:tasks.rados.rados.0.smithi089.stdout:259: done (15 left) 2024-04-22T18:38:59.415 INFO:tasks.rados.rados.0.smithi089.stdout:274: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.415 INFO:tasks.rados.rados.0.smithi089.stdout:274: set_chunk oid 25 offset: 1014784 length: 136704 target oid smithi08937929-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1014784 snap -1 2024-04-22T18:38:59.415 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.415 INFO:tasks.rados.rados.0.smithi089.stdout:267: finishing set_chunk to oid 18 2024-04-22T18:38:59.415 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 403968 length: 512512 2024-04-22T18:38:59.416 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-22T18:38:59.416 INFO:tasks.rados.rados.0.smithi089.stdout:267: done (15 left) 2024-04-22T18:38:59.416 INFO:tasks.rados.rados.0.smithi089.stdout:275: set_chunk oid 26 target oid 26 low_tier 2024-04-22T18:38:59.416 INFO:tasks.rados.rados.0.smithi089.stdout:275: set_chunk oid 26 offset: 951296 length: 410624 target oid smithi08937929-26 low_tier offset: 951296 snap -1 2024-04-22T18:38:59.416 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.430 INFO:tasks.rados.rados.0.smithi089.stdout:243: finishing set_chunk to oid 44 2024-04-22T18:38:59.430 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-22T18:38:59.430 INFO:tasks.rados.rados.0.smithi089.stdout:243: done (15 left) 2024-04-22T18:38:59.430 INFO:tasks.rados.rados.0.smithi089.stdout:276: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.430 INFO:tasks.rados.rados.0.smithi089.stdout:276: set_chunk oid 27 offset: 658944 length: 529408 target oid smithi08937929-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 658944 snap -1 2024-04-22T18:38:59.430 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.478 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:59.478+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:38:59.501 INFO:tasks.rados.rados.0.smithi089.stdout:274: finishing set_chunk to oid 25 2024-04-22T18:38:59.501 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1014784 length: 136704 2024-04-22T18:38:59.501 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-22T18:38:59.501 INFO:tasks.rados.rados.0.smithi089.stdout:274: done (15 left) 2024-04-22T18:38:59.501 INFO:tasks.rados.rados.0.smithi089.stdout:277: set_chunk oid 28 target oid 28 low_tier 2024-04-22T18:38:59.501 INFO:tasks.rados.rados.0.smithi089.stdout:277: set_chunk oid 28 offset: 422400 length: 386560 target oid smithi08937929-28 low_tier offset: 422400 snap -1 2024-04-22T18:38:59.501 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.502 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:59.502 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:38:59.502 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:38:59.502 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:38:59.502 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:59.502 INFO:tasks.rados.rados.0.smithi089.stdout:262: finishing set_chunk to oid 13 2024-04-22T18:38:59.502 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 219136 length: 1700352 2024-04-22T18:38:59.503 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-22T18:38:59.503 INFO:tasks.rados.rados.0.smithi089.stdout:262: done (15 left) 2024-04-22T18:38:59.503 INFO:tasks.rados.rados.0.smithi089.stdout:278: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.503 INFO:tasks.rados.rados.0.smithi089.stdout:278: set_chunk oid 29 offset: 247296 length: 448000 target oid smithi08937929-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 247296 snap -1 2024-04-22T18:38:59.503 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.510 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:38:59.517 INFO:tasks.rados.rados.0.smithi089.stdout:264: finishing set_chunk to oid 15 2024-04-22T18:38:59.517 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 496640 length: 1432576 2024-04-22T18:38:59.517 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-22T18:38:59.517 INFO:tasks.rados.rados.0.smithi089.stdout:264: done (15 left) 2024-04-22T18:38:59.517 INFO:tasks.rados.rados.0.smithi089.stdout:279: set_chunk oid 30 target oid 30 low_tier 2024-04-22T18:38:59.517 INFO:tasks.rados.rados.0.smithi089.stdout:279: set_chunk oid 30 offset: 658944 length: 696832 target oid smithi08937929-30 low_tier offset: 658944 snap -1 2024-04-22T18:38:59.517 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.579 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:59.580+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:59.603 INFO:tasks.rados.rados.0.smithi089.stdout:234: finishing set_chunk to oid 35 2024-04-22T18:38:59.603 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 9 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-22T18:38:59.603 INFO:tasks.rados.rados.0.smithi089.stdout:261: finishing set_chunk to oid 12 2024-04-22T18:38:59.603 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 776704 length: 1313280 2024-04-22T18:38:59.603 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-22T18:38:59.603 INFO:tasks.rados.rados.0.smithi089.stdout:268: finishing set_chunk to oid 19 2024-04-22T18:38:59.603 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 883712 length: 1372672 2024-04-22T18:38:59.603 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-22T18:38:59.603 INFO:tasks.rados.rados.0.smithi089.stdout:272: finishing set_chunk to oid 23 2024-04-22T18:38:59.603 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 243712 length: 873472 2024-04-22T18:38:59.603 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-22T18:38:59.603 INFO:tasks.rados.rados.0.smithi089.stdout:271: finishing set_chunk to oid 22 2024-04-22T18:38:59.604 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 1683456 length: 667136 2024-04-22T18:38:59.604 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-22T18:38:59.604 INFO:tasks.rados.rados.0.smithi089.stdout:275: finishing set_chunk to oid 26 2024-04-22T18:38:59.604 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 951296 length: 410624 2024-04-22T18:38:59.604 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-22T18:38:59.604 INFO:tasks.rados.rados.0.smithi089.stdout:273: finishing set_chunk to oid 24 2024-04-22T18:38:59.604 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 212480 length: 1927168 2024-04-22T18:38:59.604 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-22T18:38:59.604 INFO:tasks.rados.rados.0.smithi089.stdout:279: finishing set_chunk to oid 30 2024-04-22T18:38:59.604 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 658944 length: 696832 2024-04-22T18:38:59.604 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-22T18:38:59.604 INFO:tasks.rados.rados.0.smithi089.stdout:234: done (15 left) 2024-04-22T18:38:59.604 INFO:tasks.rados.rados.0.smithi089.stdout:261: done (14 left) 2024-04-22T18:38:59.604 INFO:tasks.rados.rados.0.smithi089.stdout:268: done (13 left) 2024-04-22T18:38:59.605 INFO:tasks.rados.rados.0.smithi089.stdout:271: done (12 left) 2024-04-22T18:38:59.605 INFO:tasks.rados.rados.0.smithi089.stdout:272: done (11 left) 2024-04-22T18:38:59.605 INFO:tasks.rados.rados.0.smithi089.stdout:273: done (10 left) 2024-04-22T18:38:59.605 INFO:tasks.rados.rados.0.smithi089.stdout:275: done (9 left) 2024-04-22T18:38:59.605 INFO:tasks.rados.rados.0.smithi089.stdout:279: done (8 left) 2024-04-22T18:38:59.605 INFO:tasks.rados.rados.0.smithi089.stdout:280: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.605 INFO:tasks.rados.rados.0.smithi089.stdout:280: set_chunk oid 31 offset: 198144 length: 1040896 target oid smithi08937929-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 198144 snap -1 2024-04-22T18:38:59.605 INFO:tasks.rados.rados.0.smithi089.stdout:281: set_chunk oid 32 target oid 32 low_tier 2024-04-22T18:38:59.605 INFO:tasks.rados.rados.0.smithi089.stdout:281: set_chunk oid 32 offset: 406016 length: 1667584 target oid smithi08937929-32 low_tier offset: 406016 snap -1 2024-04-22T18:38:59.605 INFO:tasks.rados.rados.0.smithi089.stdout:282: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.605 INFO:tasks.rados.rados.0.smithi089.stdout:282: set_chunk oid 33 offset: 2028032 length: 244736 target oid smithi08937929-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2028032 snap -1 2024-04-22T18:38:59.605 INFO:tasks.rados.rados.0.smithi089.stdout:283: set_chunk oid 34 target oid 34 low_tier 2024-04-22T18:38:59.605 INFO:tasks.rados.rados.0.smithi089.stdout:283: set_chunk oid 34 offset: 2057216 length: 433152 target oid smithi08937929-34 low_tier offset: 2057216 snap -1 2024-04-22T18:38:59.605 INFO:tasks.rados.rados.0.smithi089.stdout:284: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.606 INFO:tasks.rados.rados.0.smithi089.stdout:284: set_chunk oid 35 offset: 679424 length: 1018880 target oid smithi08937929-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 679424 snap -1 2024-04-22T18:38:59.606 INFO:tasks.rados.rados.0.smithi089.stdout:285: set_chunk oid 36 target oid 36 low_tier 2024-04-22T18:38:59.606 INFO:tasks.rados.rados.0.smithi089.stdout:285: set_chunk oid 36 offset: 375296 length: 896000 target oid smithi08937929-36 low_tier offset: 375296 snap -1 2024-04-22T18:38:59.606 INFO:tasks.rados.rados.0.smithi089.stdout:286: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.606 INFO:tasks.rados.rados.0.smithi089.stdout:286: set_chunk oid 37 offset: 86528 length: 1526272 target oid smithi08937929-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 86528 snap -1 2024-04-22T18:38:59.606 INFO:tasks.rados.rados.0.smithi089.stdout:287: set_chunk oid 38 target oid 38 low_tier 2024-04-22T18:38:59.606 INFO:tasks.rados.rados.0.smithi089.stdout:287: set_chunk oid 38 offset: 346112 length: 1770496 target oid smithi08937929-38 low_tier offset: 346112 snap -1 2024-04-22T18:38:59.606 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.606 INFO:tasks.rados.rados.0.smithi089.stdout:277: finishing set_chunk to oid 28 2024-04-22T18:38:59.606 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 422400 length: 386560 2024-04-22T18:38:59.606 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-22T18:38:59.606 INFO:tasks.rados.rados.0.smithi089.stdout:277: done (15 left) 2024-04-22T18:38:59.606 INFO:tasks.rados.rados.0.smithi089.stdout:287 retry set_chunk ! 2024-04-22T18:38:59.607 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 15 2024-04-22T18:38:59.607 INFO:tasks.rados.rados.0.smithi089.stdout:280: finishing set_chunk to oid 31 2024-04-22T18:38:59.607 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 198144 length: 1040896 2024-04-22T18:38:59.607 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-22T18:38:59.607 INFO:tasks.rados.rados.0.smithi089.stdout:280: done (14 left) 2024-04-22T18:38:59.607 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 14 2024-04-22T18:38:59.607 INFO:tasks.rados.rados.0.smithi089.stdout:278: finishing set_chunk to oid 29 2024-04-22T18:38:59.607 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 247296 length: 448000 2024-04-22T18:38:59.607 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-22T18:38:59.607 INFO:tasks.rados.rados.0.smithi089.stdout:278: done (13 left) 2024-04-22T18:38:59.607 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 13 2024-04-22T18:38:59.621 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:38:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:38:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:38:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:68 1.0 e17/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'42, mlcod=17'39)", 2024-04-22T18:38:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:41.987068+0000", 2024-04-22T18:38:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.633939635000001, 2024-04-22T18:38:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30106360999999998, 2024-04-22T18:38:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987068+0000", 2024-04-22T18:38:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987068+0000", 2024-04-22T18:38:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987067+0000", 2024-04-22T18:38:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:38:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987077+0000", 2024-04-22T18:38:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8279999999999998e-06 2024-04-22T18:38:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987084+0000", 2024-04-22T18:38:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8099999999999998e-06 2024-04-22T18:38:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987091+0000", 2024-04-22T18:38:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6189999999999996e-06 2024-04-22T18:38:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087218+0000", 2024-04-22T18:38:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10012695100000001 2024-04-22T18:38:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087246+0000", 2024-04-22T18:38:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8255e-05 2024-04-22T18:38:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.288092+0000", 2024-04-22T18:38:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20084555000000001 2024-04-22T18:38:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.288131+0000", 2024-04-22T18:38:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9455000000000001e-05 2024-04-22T18:38:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:53 4.3 4:ca46fde9:::smithi08937929-14:head [write 618491~694590,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.247073+0000", 2024-04-22T18:38:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.3739347080000002, 2024-04-22T18:38:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20306998500000001, 2024-04-22T18:38:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 53 2024-04-22T18:38:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.247073+0000", 2024-04-22T18:38:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.247073+0000", 2024-04-22T18:38:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.247071+0000", 2024-04-22T18:38:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:38:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.247434+0000", 2024-04-22T18:38:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036311600000000002 2024-04-22T18:38:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.247443+0000", 2024-04-22T18:38:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9209999999999998e-06 2024-04-22T18:38:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.247451+0000", 2024-04-22T18:38:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0479999999999999e-06 2024-04-22T18:38:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.247489+0000", 2024-04-22T18:38:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7919000000000001e-05 2024-04-22T18:38:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.247576+0000", 2024-04-22T18:38:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6951000000000001e-05 2024-04-22T18:38:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:38:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.247679+0000", 2024-04-22T18:38:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010351199999999999 2024-04-22T18:38:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251330+0000", 2024-04-22T18:38:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003650398 2024-04-22T18:38:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.450078+0000", 2024-04-22T18:38:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19874872299999999 2024-04-22T18:38:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.450116+0000", 2024-04-22T18:38:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7379999999999998e-05 2024-04-22T18:38:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.450143+0000", 2024-04-22T18:38:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6925999999999999e-05 2024-04-22T18:38:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:55 4.3 4:ca46fde9:::smithi08937929-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.251793+0000", 2024-04-22T18:38:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.369214618, 2024-04-22T18:38:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20367179699999999, 2024-04-22T18:38:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 55 2024-04-22T18:38:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251793+0000", 2024-04-22T18:38:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251793+0000", 2024-04-22T18:38:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251792+0000", 2024-04-22T18:38:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251796+0000", 2024-04-22T18:38:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2390000000000004e-06 2024-04-22T18:38:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251802+0000", 2024-04-22T18:38:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5500000000000002e-06 2024-04-22T18:38:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251807+0000", 2024-04-22T18:38:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3970000000000002e-06 2024-04-22T18:38:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.252415+0000", 2024-04-22T18:38:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060764300000000005 2024-04-22T18:38:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.252462+0000", 2024-04-22T18:38:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7196e-05 2024-04-22T18:38:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:38:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.252536+0000", 2024-04-22T18:38:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3806000000000005e-05 2024-04-22T18:38:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.255369+0000", 2024-04-22T18:38:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002833113 2024-04-22T18:38:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455412+0000", 2024-04-22T18:38:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.200042628 2024-04-22T18:38:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455437+0000", 2024-04-22T18:38:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5211e-05 2024-04-22T18:38:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455464+0000", 2024-04-22T18:38:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7724000000000001e-05 2024-04-22T18:38:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:56 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:38:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.3691949489999997, 2024-04-22T18:38:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30396841899999999, 2024-04-22T18:38:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:38:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 56 2024-04-22T18:38:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:38:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:38:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:38:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251815+0000", 2024-04-22T18:38:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7800000000000001e-06 2024-04-22T18:38:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251820+0000", 2024-04-22T18:38:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5829999999999997e-06 2024-04-22T18:38:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251825+0000", 2024-04-22T18:38:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7879999999999997e-06 2024-04-22T18:38:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.252721+0000", 2024-04-22T18:38:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089552200000000005 2024-04-22T18:38:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:38:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.252794+0000", 2024-04-22T18:38:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3440000000000002e-05 2024-04-22T18:38:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555574+0000", 2024-04-22T18:38:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30278038099999999 2024-04-22T18:38:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555687+0000", 2024-04-22T18:38:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112804 2024-04-22T18:38:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555781+0000", 2024-04-22T18:38:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3624999999999999e-05 2024-04-22T18:38:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:61 4.b 4:d923883e:::smithi08937929-16:head [write 652105~445088,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:38:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.350812328, 2024-04-22T18:38:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.38578178600000002, 2024-04-22T18:38:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 61 2024-04-22T18:38:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:38:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:38:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270192+0000", 2024-04-22T18:38:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:38:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270401+0000", 2024-04-22T18:38:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020921199999999999 2024-04-22T18:38:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270415+0000", 2024-04-22T18:38:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4191e-05 2024-04-22T18:38:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270422+0000", 2024-04-22T18:38:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7329999999999997e-06 2024-04-22T18:38:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270461+0000", 2024-04-22T18:38:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9530999999999998e-05 2024-04-22T18:38:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270550+0000", 2024-04-22T18:38:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8508e-05 2024-04-22T18:38:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270648+0000", 2024-04-22T18:38:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8377999999999999e-05 2024-04-22T18:38:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.271907+0000", 2024-04-22T18:38:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012589599999999999 2024-04-22T18:38:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655904+0000", 2024-04-22T18:38:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.383997115 2024-04-22T18:38:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655933+0000", 2024-04-22T18:38:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9011999999999999e-05 2024-04-22T18:38:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655977+0000", 2024-04-22T18:38:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3578000000000002e-05 2024-04-22T18:38:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:62 4.b 4:d923883e:::smithi08937929-16:head [write 1728171~452988] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:38:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.3448536799999999, 2024-04-22T18:38:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.379937407, 2024-04-22T18:38:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62 2024-04-22T18:38:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:38:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:38:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276151+0000", 2024-04-22T18:38:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:38:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276370+0000", 2024-04-22T18:38:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000218951 2024-04-22T18:38:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276382+0000", 2024-04-22T18:38:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1517e-05 2024-04-22T18:38:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276389+0000", 2024-04-22T18:38:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8939999999999993e-06 2024-04-22T18:38:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276430+0000", 2024-04-22T18:38:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0958999999999999e-05 2024-04-22T18:38:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276485+0000", 2024-04-22T18:38:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4208e-05 2024-04-22T18:38:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276567+0000", 2024-04-22T18:38:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2006000000000001e-05 2024-04-22T18:38:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277685+0000", 2024-04-22T18:38:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001118237 2024-04-22T18:38:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656024+0000", 2024-04-22T18:38:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.378339588 2024-04-22T18:38:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656054+0000", 2024-04-22T18:38:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9373000000000002e-05 2024-04-22T18:38:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656091+0000", 2024-04-22T18:38:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7168000000000001e-05 2024-04-22T18:38:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:63 4.b 4:d923883e:::smithi08937929-16:head [setxattr _header (56) in=7b,truncate 2252105] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:38:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.344611424, 2024-04-22T18:38:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37982964200000002, 2024-04-22T18:38:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 63 2024-04-22T18:38:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:38:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:38:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276395+0000", 2024-04-22T18:38:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276400+0000", 2024-04-22T18:38:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6090000000000003e-06 2024-04-22T18:38:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276407+0000", 2024-04-22T18:38:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5850000000000002e-06 2024-04-22T18:38:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276411+0000", 2024-04-22T18:38:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2569999999999999e-06 2024-04-22T18:38:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277254+0000", 2024-04-22T18:38:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084384300000000005 2024-04-22T18:38:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277312+0000", 2024-04-22T18:38:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7395999999999997e-05 2024-04-22T18:38:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277388+0000", 2024-04-22T18:38:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6604e-05 2024-04-22T18:38:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277870+0000", 2024-04-22T18:38:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000481545 2024-04-22T18:38:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656138+0000", 2024-04-22T18:38:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37826752499999999 2024-04-22T18:38:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656177+0000", 2024-04-22T18:38:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9419999999999999e-05 2024-04-22T18:38:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656226+0000", 2024-04-22T18:38:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8562e-05 2024-04-22T18:38:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:64 4.b 4:d923883e:::smithi08937929-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:38:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.3445808030000004, 2024-04-22T18:38:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37999559999999999, 2024-04-22T18:38:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:38:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 64 2024-04-22T18:38:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:38:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:38:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:38:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276429+0000", 2024-04-22T18:38:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7329999999999999e-06 2024-04-22T18:38:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276435+0000", 2024-04-22T18:38:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.02e-06 2024-04-22T18:38:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276439+0000", 2024-04-22T18:38:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7659999999999998e-06 2024-04-22T18:38:59.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277530+0000", 2024-04-22T18:38:59.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010909470000000001 2024-04-22T18:38:59.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:38:59.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277603+0000", 2024-04-22T18:38:59.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2638999999999994e-05 2024-04-22T18:38:59.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656259+0000", 2024-04-22T18:38:59.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37865579300000002 2024-04-22T18:38:59.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656330+0000", 2024-04-22T18:38:59.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0899e-05 2024-04-22T18:38:59.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656422+0000", 2024-04-22T18:38:59.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2397000000000003e-05 2024-04-22T18:38:59.664 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.664 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.664 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.665 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:65 4.3 4:c0d4d677:::smithi08937929-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:59.665 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.354893+0000", 2024-04-22T18:38:59.665 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.2661140150000003, 2024-04-22T18:38:59.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30130554999999998, 2024-04-22T18:38:59.665 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.665 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.665 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.665 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.665 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.665 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65 2024-04-22T18:38:59.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.665 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.354893+0000", 2024-04-22T18:38:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.354893+0000", 2024-04-22T18:38:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.354890+0000", 2024-04-22T18:38:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999962 2024-04-22T18:38:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.355220+0000", 2024-04-22T18:38:59.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00033014200000000001 2024-04-22T18:38:59.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.355235+0000", 2024-04-22T18:38:59.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5413e-05 2024-04-22T18:38:59.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455327+0000", 2024-04-22T18:38:59.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009154000000001 2024-04-22T18:38:59.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555819+0000", 2024-04-22T18:38:59.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100492812 2024-04-22T18:38:59.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555964+0000", 2024-04-22T18:38:59.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000144907 2024-04-22T18:38:59.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:38:59.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.556141+0000", 2024-04-22T18:38:59.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017619599999999999 2024-04-22T18:38:59.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656140+0000", 2024-04-22T18:38:59.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099999085000000001 2024-04-22T18:38:59.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656144+0000", 2024-04-22T18:38:59.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9689999999999996e-06 2024-04-22T18:38:59.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656167+0000", 2024-04-22T18:38:59.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3363e-05 2024-04-22T18:38:59.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656199+0000", 2024-04-22T18:38:59.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1828000000000001e-05 2024-04-22T18:38:59.670 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.670 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.670 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.670 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:59.670 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:38:59.670 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1656507270000001, 2024-04-22T18:38:59.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:38:59.671 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.671 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.671 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.671 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.671 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.671 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:38:59.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.671 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:38:59.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:38:59.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:38:59.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:38:59.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:38:59.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:38:59.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:38:59.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:38:59.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:38:59.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:38:59.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:38:59.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:38:59.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:38:59.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:38:59.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:38:59.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:38:59.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:38:59.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:38:59.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:38:59.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:38:59.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:38:59.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:38:59.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:38:59.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:38:59.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:38:59.676 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.676 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.676 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.676 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:113 4.b 4:d3068ee3:::smithi08937929-29:head [write 470596~456031,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:38:59.676 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.687365+0000", 2024-04-22T18:38:59.676 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.933642291, 2024-04-22T18:38:59.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20804932300000001, 2024-04-22T18:38:59.676 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.676 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.676 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.676 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.677 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.677 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 113 2024-04-22T18:38:59.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.677 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.687365+0000", 2024-04-22T18:38:59.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.687365+0000", 2024-04-22T18:38:59.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.687363+0000", 2024-04-22T18:38:59.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:38:59.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.687459+0000", 2024-04-22T18:38:59.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6688000000000007e-05 2024-04-22T18:38:59.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.687474+0000", 2024-04-22T18:38:59.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4348000000000001e-05 2024-04-22T18:38:59.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.787609+0000", 2024-04-22T18:38:59.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100135261 2024-04-22T18:38:59.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.887793+0000", 2024-04-22T18:38:59.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100184252 2024-04-22T18:38:59.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.887874+0000", 2024-04-22T18:38:59.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0533999999999998e-05 2024-04-22T18:38:59.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:59.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.887960+0000", 2024-04-22T18:38:59.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6222000000000002e-05 2024-04-22T18:38:59.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.889554+0000", 2024-04-22T18:38:59.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001594339 2024-04-22T18:38:59.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.895348+0000", 2024-04-22T18:38:59.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057940029999999998 2024-04-22T18:38:59.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.895376+0000", 2024-04-22T18:38:59.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7569e-05 2024-04-22T18:38:59.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.895414+0000", 2024-04-22T18:38:59.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8559999999999997e-05 2024-04-22T18:38:59.681 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.681 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.682 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.682 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:40 3.1 3:be9a15cb:::smithi08937929-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:38:59.682 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.794661+0000", 2024-04-22T18:38:59.682 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.8263466310000001, 2024-04-22T18:38:59.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.298299751, 2024-04-22T18:38:59.682 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.682 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.682 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.794661+0000", 2024-04-22T18:38:59.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.794661+0000", 2024-04-22T18:38:59.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.794659+0000", 2024-04-22T18:38:59.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:38:59.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.794665+0000", 2024-04-22T18:38:59.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5799999999999999e-06 2024-04-22T18:38:59.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.794675+0000", 2024-04-22T18:38:59.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4429999999999992e-06 2024-04-22T18:38:59.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.794680+0000", 2024-04-22T18:38:59.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1660000000000002e-06 2024-04-22T18:38:59.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.894788+0000", 2024-04-22T18:38:59.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010804 2024-04-22T18:38:59.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.894864+0000", 2024-04-22T18:38:59.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6389000000000003e-05 2024-04-22T18:38:59.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:38:59.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.894999+0000", 2024-04-22T18:38:59.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013485 2024-04-22T18:38:59.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.895628+0000", 2024-04-22T18:38:59.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00062914199999999996 2024-04-22T18:38:59.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.092902+0000", 2024-04-22T18:38:59.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19727352300000001 2024-04-22T18:38:59.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.092931+0000", 2024-04-22T18:38:59.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8980999999999999e-05 2024-04-22T18:38:59.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.092960+0000", 2024-04-22T18:38:59.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9807000000000001e-05 2024-04-22T18:38:59.687 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.687 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.687 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.687 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:315 4.b 4:d923883e:::smithi08937929-16:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:38:59.687 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:38:59.687 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.6291879470000001, 2024-04-22T18:38:59.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40309447300000001, 2024-04-22T18:38:59.688 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.688 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.688 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 315 2024-04-22T18:38:59.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.688 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:38:59.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:38:59.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991817+0000", 2024-04-22T18:38:59.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:38:59.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991825+0000", 2024-04-22T18:38:59.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7279999999999995e-06 2024-04-22T18:38:59.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991834+0000", 2024-04-22T18:38:59.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1980000000000007e-06 2024-04-22T18:38:59.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991841+0000", 2024-04-22T18:38:59.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0740000000000004e-06 2024-04-22T18:38:59.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991881+0000", 2024-04-22T18:38:59.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-04-22T18:38:59.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991940+0000", 2024-04-22T18:38:59.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9382000000000002e-05 2024-04-22T18:38:59.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:59.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394250+0000", 2024-04-22T18:38:59.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40230947700000003 2024-04-22T18:38:59.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394752+0000", 2024-04-22T18:38:59.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050269699999999997 2024-04-22T18:38:59.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394865+0000", 2024-04-22T18:38:59.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011267700000000001 2024-04-22T18:38:59.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394892+0000", 2024-04-22T18:38:59.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7535999999999999e-05 2024-04-22T18:38:59.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394914+0000", 2024-04-22T18:38:59.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1398000000000001e-05 2024-04-22T18:38:59.693 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.693 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.693 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.693 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:42 3.1 3:a1535cdd:::smithi08937929-24 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:38:59.693 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.093002+0000", 2024-04-22T18:38:59.693 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.528005249, 2024-04-22T18:38:59.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30111063900000001, 2024-04-22T18:38:59.693 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.694 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.694 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093002+0000", 2024-04-22T18:38:59.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093002+0000", 2024-04-22T18:38:59.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093001+0000", 2024-04-22T18:38:59.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:38:59.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093007+0000", 2024-04-22T18:38:59.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6509999999999998e-06 2024-04-22T18:38:59.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093018+0000", 2024-04-22T18:38:59.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1119e-05 2024-04-22T18:38:59.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093026+0000", 2024-04-22T18:38:59.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9049999999999997e-06 2024-04-22T18:38:59.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093063+0000", 2024-04-22T18:38:59.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6987999999999999e-05 2024-04-22T18:38:59.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093142+0000", 2024-04-22T18:38:59.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9877999999999997e-05 2024-04-22T18:38:59.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:38:59.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093306+0000", 2024-04-22T18:38:59.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016320500000000001 2024-04-22T18:38:59.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093803+0000", 2024-04-22T18:38:59.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049711999999999998 2024-04-22T18:38:59.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394037+0000", 2024-04-22T18:38:59.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30023460000000002 2024-04-22T18:38:59.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394073+0000", 2024-04-22T18:38:59.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5227e-05 2024-04-22T18:38:59.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394113+0000", 2024-04-22T18:38:59.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0037999999999999e-05 2024-04-22T18:38:59.698 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.698 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.698 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.699 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:331 4.6 4:6700eab5:::smithi08937929-32:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:38:59.699 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:38:59.699 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.024450812, 2024-04-22T18:38:59.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.305691029, 2024-04-22T18:38:59.699 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.699 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.699 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.699 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.699 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.699 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 331 2024-04-22T18:38:59.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.699 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:38:59.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:38:59.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:38:59.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:38:59.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596560+0000", 2024-04-22T18:38:59.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8149999999999996e-06 2024-04-22T18:38:59.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596568+0000", 2024-04-22T18:38:59.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1489999999999997e-06 2024-04-22T18:38:59.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596576+0000", 2024-04-22T18:38:59.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.191e-06 2024-04-22T18:38:59.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596619+0000", 2024-04-22T18:38:59.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3507000000000002e-05 2024-04-22T18:38:59.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596709+0000", 2024-04-22T18:38:59.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0043000000000005e-05 2024-04-22T18:38:59.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:59.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.801630+0000", 2024-04-22T18:38:59.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20492061 2024-04-22T18:38:59.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.802094+0000", 2024-04-22T18:38:59.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046431899999999998 2024-04-22T18:38:59.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.902165+0000", 2024-04-22T18:38:59.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10007037000000001 2024-04-22T18:38:59.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.902218+0000", 2024-04-22T18:38:59.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3897999999999998e-05 2024-04-22T18:38:59.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.902248+0000", 2024-04-22T18:38:59.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9023999999999999e-05 2024-04-22T18:38:59.704 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.704 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.704 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.704 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:344 4.6 4:630bd8e5:::smithi08937929-45:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:38:59.704 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.597052+0000", 2024-04-22T18:38:59.704 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.0239551280000001, 2024-04-22T18:38:59.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20405029499999999, 2024-04-22T18:38:59.705 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.705 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.705 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.705 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.705 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.705 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 344 2024-04-22T18:38:59.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.705 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.597052+0000", 2024-04-22T18:38:59.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.597052+0000", 2024-04-22T18:38:59.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.597052+0000", 2024-04-22T18:38:59.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:38:59.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.597056+0000", 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.053e-06 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.597061+0000", 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6860000000000003e-06 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.597067+0000", 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8250000000000001e-06 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.597109+0000", 2024-04-22T18:38:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2243999999999997e-05 2024-04-22T18:38:59.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.597179+0000", 2024-04-22T18:38:59.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9696999999999994e-05 2024-04-22T18:38:59.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:59.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798678+0000", 2024-04-22T18:38:59.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20149941699999999 2024-04-22T18:38:59.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.801004+0000", 2024-04-22T18:38:59.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0023253940000000002 2024-04-22T18:38:59.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.801020+0000", 2024-04-22T18:38:59.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5849999999999999e-05 2024-04-22T18:38:59.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.801066+0000", 2024-04-22T18:38:59.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6229000000000001e-05 2024-04-22T18:38:59.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.801102+0000", 2024-04-22T18:38:59.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6545000000000003e-05 2024-04-22T18:38:59.710 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.710 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.710 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.710 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:367 4.3 4:c6fa81cf:::smithi08937929-18:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:38:59.710 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:38:59.710 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.82329383099999998, 2024-04-22T18:38:59.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30292533399999999, 2024-04-22T18:38:59.710 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.710 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.711 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.711 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.711 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.711 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 367 2024-04-22T18:38:59.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.711 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:38:59.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:38:59.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797719+0000", 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0669999999999997e-06 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797725+0000", 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3049999999999998e-06 2024-04-22T18:38:59.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797731+0000", 2024-04-22T18:38:59.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7649999999999999e-06 2024-04-22T18:38:59.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.897957+0000", 2024-04-22T18:38:59.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100226045 2024-04-22T18:38:59.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.898010+0000", 2024-04-22T18:38:59.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2679999999999997e-05 2024-04-22T18:38:59.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:38:59.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.999633+0000", 2024-04-22T18:38:59.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101623639 2024-04-22T18:38:59.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.000232+0000", 2024-04-22T18:38:59.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059861099999999998 2024-04-22T18:38:59.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100531+0000", 2024-04-22T18:38:59.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100299112 2024-04-22T18:38:59.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100599+0000", 2024-04-22T18:38:59.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7989000000000002e-05 2024-04-22T18:38:59.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100639+0000", 2024-04-22T18:38:59.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9919000000000002e-05 2024-04-22T18:38:59.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.715 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.716 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:374 4.3 4:c93c842e:::smithi08937929-25:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:38:59.716 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.798795+0000", 2024-04-22T18:38:59.716 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.82221266100000001, 2024-04-22T18:38:59.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30197590899999999, 2024-04-22T18:38:59.716 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.716 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.716 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 374 2024-04-22T18:38:59.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.716 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798795+0000", 2024-04-22T18:38:59.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798795+0000", 2024-04-22T18:38:59.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798793+0000", 2024-04-22T18:38:59.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:59.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798801+0000", 2024-04-22T18:38:59.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5299999999999999e-06 2024-04-22T18:38:59.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798811+0000", 2024-04-22T18:38:59.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0047999999999999e-05 2024-04-22T18:38:59.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798818+0000", 2024-04-22T18:38:59.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4240000000000004e-06 2024-04-22T18:38:59.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.898121+0000", 2024-04-22T18:38:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099302611999999998 2024-04-22T18:38:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.898189+0000", 2024-04-22T18:38:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8032000000000002e-05 2024-04-22T18:38:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:38:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.999945+0000", 2024-04-22T18:38:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101755686 2024-04-22T18:38:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100695+0000", 2024-04-22T18:38:59.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100750622 2024-04-22T18:38:59.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100703+0000", 2024-04-22T18:38:59.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3690000000000001e-06 2024-04-22T18:38:59.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100731+0000", 2024-04-22T18:38:59.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8427000000000001e-05 2024-04-22T18:38:59.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100771+0000", 2024-04-22T18:38:59.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9579000000000002e-05 2024-04-22T18:38:59.721 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.721 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.721 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.721 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:393 4.b 4:defe92fb:::smithi08937929-44:head [assert-version v12,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:38:59.721 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:38:59.721 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.51981176500000004, 2024-04-22T18:38:59.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33050822299999999, 2024-04-22T18:38:59.721 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.721 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.722 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:38:59.722 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:38:59.722 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:38:59.722 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 393 2024-04-22T18:38:59.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.722 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:38:59.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:38:59.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101194+0000", 2024-04-22T18:38:59.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:38:59.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101201+0000", 2024-04-22T18:38:59.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1269999999999997e-06 2024-04-22T18:38:59.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101211+0000", 2024-04-22T18:38:59.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9380000000000004e-06 2024-04-22T18:38:59.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101220+0000", 2024-04-22T18:38:59.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3459999999999994e-06 2024-04-22T18:38:59.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101324+0000", 2024-04-22T18:38:59.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104366 2024-04-22T18:38:59.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101379+0000", 2024-04-22T18:38:59.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5127999999999998e-05 2024-04-22T18:38:59.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:38:59.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.330649+0000", 2024-04-22T18:38:59.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.22927022699999999 2024-04-22T18:38:59.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.331182+0000", 2024-04-22T18:38:59.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053250900000000002 2024-04-22T18:38:59.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.431635+0000", 2024-04-22T18:38:59.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10045290599999999 2024-04-22T18:38:59.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.431677+0000", 2024-04-22T18:38:59.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1816000000000001e-05 2024-04-22T18:38:59.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.431704+0000", 2024-04-22T18:38:59.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7132e-05 2024-04-22T18:38:59.726 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.727 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.727 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.727 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:31 3.1 3:b7b88fa0:::smithi08937929-44 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:38:59.727 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.101557+0000", 2024-04-22T18:38:59.727 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.51945020900000005, 2024-04-22T18:38:59.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.22893401799999999, 2024-04-22T18:38:59.727 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:38:59.727 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:38:59.727 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:38:59.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:38:59.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101557+0000", 2024-04-22T18:38:59.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:38:59.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101557+0000", 2024-04-22T18:38:59.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:38:59.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:38:59.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101555+0000", 2024-04-22T18:38:59.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:38:59.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:38:59.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101563+0000", 2024-04-22T18:38:59.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0600000000000008e-06 2024-04-22T18:38:59.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:38:59.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101574+0000", 2024-04-22T18:38:59.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0977e-05 2024-04-22T18:38:59.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:38:59.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101587+0000", 2024-04-22T18:38:59.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2884999999999999e-05 2024-04-22T18:38:59.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:38:59.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.102112+0000", 2024-04-22T18:38:59.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00052499299999999997 2024-04-22T18:38:59.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:38:59.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.102162+0000", 2024-04-22T18:38:59.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0009000000000002e-05 2024-04-22T18:38:59.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:38:59.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.102263+0000", 2024-04-22T18:38:59.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010105199999999999 2024-04-22T18:38:59.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:38:59.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.102636+0000", 2024-04-22T18:38:59.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037250199999999998 2024-04-22T18:38:59.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:38:59.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.330415+0000", 2024-04-22T18:38:59.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.22777872900000001 2024-04-22T18:38:59.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:38:59.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.330461+0000", 2024-04-22T18:38:59.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6397999999999999e-05 2024-04-22T18:38:59.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:38:59.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:38:59.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:38:59.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.330491+0000", 2024-04-22T18:38:59.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0108999999999999e-05 2024-04-22T18:38:59.732 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.732 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.732 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.732 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:38:59.732 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:38:59.732 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:59.734 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:38:59.736 INFO:tasks.rados.rados.0.smithi089.stdout:276: finishing set_chunk to oid 27 2024-04-22T18:38:59.736 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 658944 length: 529408 2024-04-22T18:38:59.736 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-22T18:38:59.736 INFO:tasks.rados.rados.0.smithi089.stdout:276: done (12 left) 2024-04-22T18:38:59.736 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 12 2024-04-22T18:38:59.736 INFO:tasks.rados.rados.0.smithi089.stdout:285: finishing set_chunk to oid 36 2024-04-22T18:38:59.736 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 375296 length: 896000 2024-04-22T18:38:59.736 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-22T18:38:59.737 INFO:tasks.rados.rados.0.smithi089.stdout:285: done (11 left) 2024-04-22T18:38:59.737 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 11 2024-04-22T18:38:59.737 INFO:tasks.rados.rados.0.smithi089.stdout:284: finishing set_chunk to oid 35 2024-04-22T18:38:59.737 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 679424 length: 1018880 2024-04-22T18:38:59.737 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-22T18:38:59.737 INFO:tasks.rados.rados.0.smithi089.stdout:284: done (10 left) 2024-04-22T18:38:59.737 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 10 2024-04-22T18:38:59.737 INFO:tasks.rados.rados.0.smithi089.stdout:281: finishing set_chunk to oid 32 2024-04-22T18:38:59.737 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 406016 length: 1667584 2024-04-22T18:38:59.737 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-22T18:38:59.737 INFO:tasks.rados.rados.0.smithi089.stdout:281: done (9 left) 2024-04-22T18:38:59.737 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 9 2024-04-22T18:38:59.737 INFO:tasks.rados.rados.0.smithi089.stdout:287: finishing set_chunk to oid 38 2024-04-22T18:38:59.737 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 346112 length: 1770496 2024-04-22T18:38:59.737 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-22T18:38:59.738 INFO:tasks.rados.rados.0.smithi089.stdout:287: done (8 left) 2024-04-22T18:38:59.738 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 8 2024-04-22T18:38:59.738 INFO:tasks.rados.rados.0.smithi089.stdout:265: finishing set_chunk to oid 16 2024-04-22T18:38:59.738 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-22T18:38:59.738 INFO:tasks.rados.rados.0.smithi089.stdout:265: done (7 left) 2024-04-22T18:38:59.738 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 7 2024-04-22T18:38:59.738 INFO:tasks.rados.rados.0.smithi089.stdout:266: finishing set_chunk to oid 17 2024-04-22T18:38:59.738 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-22T18:38:59.738 INFO:tasks.rados.rados.0.smithi089.stdout:266: done (6 left) 2024-04-22T18:38:59.738 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 6 2024-04-22T18:38:59.738 INFO:tasks.rados.rados.0.smithi089.stdout:286: finishing set_chunk to oid 37 2024-04-22T18:38:59.738 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 86528 length: 1526272 2024-04-22T18:38:59.738 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-22T18:38:59.738 INFO:tasks.rados.rados.0.smithi089.stdout:286: done (5 left) 2024-04-22T18:38:59.738 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 5 2024-04-22T18:38:59.739 INFO:tasks.rados.rados.0.smithi089.stdout:269: finishing set_chunk to oid 20 2024-04-22T18:38:59.739 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-22T18:38:59.739 INFO:tasks.rados.rados.0.smithi089.stdout:269: done (4 left) 2024-04-22T18:38:59.739 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 4 2024-04-22T18:38:59.739 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:38:59.734+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:38:59.802 INFO:tasks.rados.rados.0.smithi089.stdout:283: finishing set_chunk to oid 34 2024-04-22T18:38:59.802 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 2057216 length: 433152 2024-04-22T18:38:59.802 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-22T18:38:59.802 INFO:tasks.rados.rados.0.smithi089.stdout:254: finishing set_chunk to oid 5 2024-04-22T18:38:59.802 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-22T18:38:59.802 INFO:tasks.rados.rados.0.smithi089.stdout:254: done (3 left) 2024-04-22T18:38:59.802 INFO:tasks.rados.rados.0.smithi089.stdout:283: done (2 left) 2024-04-22T18:38:59.802 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 2 2024-04-22T18:38:59.803 INFO:tasks.rados.rados.0.smithi089.stdout:282: finishing set_chunk to oid 33 2024-04-22T18:38:59.803 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-22T18:38:59.803 INFO:tasks.rados.rados.0.smithi089.stdout:282: done (1 left) 2024-04-22T18:38:59.803 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 1 2024-04-22T18:38:59.834 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:38:59.834+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:38:59.877 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:38:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:38:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:38:59.877 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:38:59.886 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:38:59.896 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:38:59.910 INFO:tasks.rados.rados.0.smithi089.stdout:238: finishing set_chunk to oid 39 2024-04-22T18:38:59.910 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 15 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-22T18:38:59.910 INFO:tasks.rados.rados.0.smithi089.stdout:238: done (0 left) 2024-04-22T18:38:59.910 INFO:tasks.rados.rados.0.smithi089.stdout:288: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.910 INFO:tasks.rados.rados.0.smithi089.stdout:288: set_chunk oid 39 offset: 176640 length: 2063360 target oid smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 176640 snap -1 2024-04-22T18:38:59.910 INFO:tasks.rados.rados.0.smithi089.stdout:289: set_chunk oid 40 target oid 40 low_tier 2024-04-22T18:38:59.910 INFO:tasks.rados.rados.0.smithi089.stdout:289: set_chunk oid 40 offset: 18944 length: 1477120 target oid smithi08937929-40 low_tier offset: 18944 snap -1 2024-04-22T18:38:59.910 INFO:tasks.rados.rados.0.smithi089.stdout:290: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.910 INFO:tasks.rados.rados.0.smithi089.stdout:290: set_chunk oid 41 offset: 225792 length: 262144 target oid smithi08937929-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 225792 snap -1 2024-04-22T18:38:59.911 INFO:tasks.rados.rados.0.smithi089.stdout:291: set_chunk oid 42 target oid 42 low_tier 2024-04-22T18:38:59.911 INFO:tasks.rados.rados.0.smithi089.stdout:291: set_chunk oid 42 offset: 1402880 length: 561664 target oid smithi08937929-42 low_tier offset: 1402880 snap -1 2024-04-22T18:38:59.911 INFO:tasks.rados.rados.0.smithi089.stdout:292: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.911 INFO:tasks.rados.rados.0.smithi089.stdout:292: set_chunk oid 43 offset: 1700864 length: 51200 target oid smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1700864 snap -1 2024-04-22T18:38:59.911 INFO:tasks.rados.rados.0.smithi089.stdout:293: set_chunk oid 44 target oid 44 low_tier 2024-04-22T18:38:59.911 INFO:tasks.rados.rados.0.smithi089.stdout:293: set_chunk oid 44 offset: 30720 length: 1741824 target oid smithi08937929-44 low_tier offset: 30720 snap -1 2024-04-22T18:38:59.911 INFO:tasks.rados.rados.0.smithi089.stdout:294: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.911 INFO:tasks.rados.rados.0.smithi089.stdout:294: set_chunk oid 45 offset: 338432 length: 1764352 target oid smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 338432 snap -1 2024-04-22T18:38:59.911 INFO:tasks.rados.rados.0.smithi089.stdout:295: set_chunk oid 46 target oid 46 low_tier 2024-04-22T18:38:59.911 INFO:tasks.rados.rados.0.smithi089.stdout:295: set_chunk oid 46 offset: 1522688 length: 924672 target oid smithi08937929-46 low_tier offset: 1522688 snap -1 2024-04-22T18:38:59.911 INFO:tasks.rados.rados.0.smithi089.stdout:296: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.911 INFO:tasks.rados.rados.0.smithi089.stdout:296: set_chunk oid 47 offset: 97280 length: 2065920 target oid smithi08937929-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 97280 snap -1 2024-04-22T18:38:59.911 INFO:tasks.rados.rados.0.smithi089.stdout:297: set_chunk oid 48 target oid 48 low_tier 2024-04-22T18:38:59.912 INFO:tasks.rados.rados.0.smithi089.stdout:297: set_chunk oid 48 offset: 614912 length: 1162240 target oid smithi08937929-48 low_tier offset: 614912 snap -1 2024-04-22T18:38:59.912 INFO:tasks.rados.rados.0.smithi089.stdout:298: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.912 INFO:tasks.rados.rados.0.smithi089.stdout:298: set_chunk oid 49 offset: 464896 length: 706560 target oid smithi08937929-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 464896 snap -1 2024-04-22T18:38:59.912 INFO:tasks.rados.rados.0.smithi089.stdout:299: set_chunk oid 50 target oid 50 low_tier 2024-04-22T18:38:59.912 INFO:tasks.rados.rados.0.smithi089.stdout:299: set_chunk oid 50 offset: 1143808 length: 1275904 target oid smithi08937929-50 low_tier offset: 1143808 snap -1 2024-04-22T18:38:59.912 INFO:tasks.rados.rados.0.smithi089.stdout:300: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.912 INFO:tasks.rados.rados.0.smithi089.stdout:300: set_chunk oid 1 offset: 875520 length: 210432 target oid smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 875520 snap -1 2024-04-22T18:38:59.912 INFO:tasks.rados.rados.0.smithi089.stdout:301: set_chunk oid 2 target oid 2 low_tier 2024-04-22T18:38:59.912 INFO:tasks.rados.rados.0.smithi089.stdout:301: set_chunk oid 2 offset: 255488 length: 662016 target oid smithi08937929-2 low_tier offset: 255488 snap -1 2024-04-22T18:38:59.912 INFO:tasks.rados.rados.0.smithi089.stdout:302: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.912 INFO:tasks.rados.rados.0.smithi089.stdout:302: set_chunk oid 3 offset: 892416 length: 364544 target oid smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 892416 snap -1 2024-04-22T18:38:59.912 INFO:tasks.rados.rados.0.smithi089.stdout:303: set_chunk oid 4 target oid 4 low_tier 2024-04-22T18:38:59.912 INFO:tasks.rados.rados.0.smithi089.stdout:303: set_chunk oid 4 offset: 1030656 length: 828928 target oid smithi08937929-4 low_tier offset: 1030656 snap -1 2024-04-22T18:38:59.912 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.913 INFO:tasks.rados.rados.0.smithi089.stdout:290: finishing set_chunk to oid 41 2024-04-22T18:38:59.913 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 225792 length: 262144 2024-04-22T18:38:59.913 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-22T18:38:59.913 INFO:tasks.rados.rados.0.smithi089.stdout:290: done (15 left) 2024-04-22T18:38:59.913 INFO:tasks.rados.rados.0.smithi089.stdout:304: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.913 INFO:tasks.rados.rados.0.smithi089.stdout:304: set_chunk oid 5 offset: 714752 length: 94208 target oid smithi08937929-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 714752 snap -1 2024-04-22T18:38:59.913 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.913 INFO:tasks.rados.rados.0.smithi089.stdout:293: finishing set_chunk to oid 44 2024-04-22T18:38:59.913 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 30720 length: 1741824 2024-04-22T18:38:59.913 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-22T18:38:59.913 INFO:tasks.rados.rados.0.smithi089.stdout:295: finishing set_chunk to oid 46 2024-04-22T18:38:59.913 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1522688 length: 924672 2024-04-22T18:38:59.913 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-22T18:38:59.914 INFO:tasks.rados.rados.0.smithi089.stdout:299: finishing set_chunk to oid 50 2024-04-22T18:38:59.914 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1143808 length: 1275904 2024-04-22T18:38:59.914 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-22T18:38:59.914 INFO:tasks.rados.rados.0.smithi089.stdout:291: finishing set_chunk to oid 42 2024-04-22T18:38:59.914 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 1402880 length: 561664 2024-04-22T18:38:59.914 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-22T18:38:59.914 INFO:tasks.rados.rados.0.smithi089.stdout:294: finishing set_chunk to oid 45 2024-04-22T18:38:59.914 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 338432 length: 1764352 2024-04-22T18:38:59.914 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-22T18:38:59.914 INFO:tasks.rados.rados.0.smithi089.stdout:300: finishing set_chunk to oid 1 2024-04-22T18:38:59.914 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 875520 length: 210432 2024-04-22T18:38:59.914 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-22T18:38:59.914 INFO:tasks.rados.rados.0.smithi089.stdout:291: done (15 left) 2024-04-22T18:38:59.915 INFO:tasks.rados.rados.0.smithi089.stdout:293: done (14 left) 2024-04-22T18:38:59.915 INFO:tasks.rados.rados.0.smithi089.stdout:294: done (13 left) 2024-04-22T18:38:59.915 INFO:tasks.rados.rados.0.smithi089.stdout:295: done (12 left) 2024-04-22T18:38:59.915 INFO:tasks.rados.rados.0.smithi089.stdout:299: done (11 left) 2024-04-22T18:38:59.915 INFO:tasks.rados.rados.0.smithi089.stdout:300: done (10 left) 2024-04-22T18:38:59.915 INFO:tasks.rados.rados.0.smithi089.stdout:305: set_chunk oid 6 target oid 6 low_tier 2024-04-22T18:38:59.915 INFO:tasks.rados.rados.0.smithi089.stdout:305: set_chunk oid 6 offset: 465408 length: 276480 target oid smithi08937929-6 low_tier offset: 465408 snap -1 2024-04-22T18:38:59.915 INFO:tasks.rados.rados.0.smithi089.stdout:306: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.915 INFO:tasks.rados.rados.0.smithi089.stdout:306: set_chunk oid 7 offset: 121344 length: 310272 target oid smithi08937929-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 121344 snap -1 2024-04-22T18:38:59.915 INFO:tasks.rados.rados.0.smithi089.stdout:307: set_chunk oid 8 target oid 8 low_tier 2024-04-22T18:38:59.915 INFO:tasks.rados.rados.0.smithi089.stdout:307: set_chunk oid 8 offset: 643584 length: 557056 target oid smithi08937929-8 low_tier offset: 643584 snap -1 2024-04-22T18:38:59.915 INFO:tasks.rados.rados.0.smithi089.stdout:308: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.915 INFO:tasks.rados.rados.0.smithi089.stdout:308: set_chunk oid 9 offset: 275456 length: 1419264 target oid smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 275456 snap -1 2024-04-22T18:38:59.915 INFO:tasks.rados.rados.0.smithi089.stdout:309: set_chunk oid 10 target oid 10 low_tier 2024-04-22T18:38:59.916 INFO:tasks.rados.rados.0.smithi089.stdout:309: set_chunk oid 10 offset: 470528 length: 516608 target oid smithi08937929-10 low_tier offset: 470528 snap -1 2024-04-22T18:38:59.916 INFO:tasks.rados.rados.0.smithi089.stdout:310: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.916 INFO:tasks.rados.rados.0.smithi089.stdout:288: finishing set_chunk to oid 39 2024-04-22T18:38:59.916 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 176640 length: 2063360 2024-04-22T18:38:59.916 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-22T18:38:59.916 INFO:tasks.rados.rados.0.smithi089.stdout:297: finishing set_chunk to oid 48 2024-04-22T18:38:59.916 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 614912 length: 1162240 2024-04-22T18:38:59.916 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-22T18:38:59.916 INFO:tasks.rados.rados.0.smithi089.stdout:310: set_chunk oid 11 offset: 741376 length: 629248 target oid smithi08937929-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 741376 snap -1 2024-04-22T18:38:59.916 INFO:tasks.rados.rados.0.smithi089.stdout:288: done (15 left) 2024-04-22T18:38:59.916 INFO:tasks.rados.rados.0.smithi089.stdout:297: done (14 left) 2024-04-22T18:38:59.916 INFO:tasks.rados.rados.0.smithi089.stdout:311: set_chunk oid 12 target oid 12 low_tier 2024-04-22T18:38:59.916 INFO:tasks.rados.rados.0.smithi089.stdout:311: set_chunk oid 12 offset: 263168 length: 1913344 target oid smithi08937929-12 low_tier offset: 263168 snap -1 2024-04-22T18:38:59.916 INFO:tasks.rados.rados.0.smithi089.stdout:312: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.917 INFO:tasks.rados.rados.0.smithi089.stdout:312: set_chunk oid 13 offset: 601600 length: 1342976 target oid smithi08937929-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 601600 snap -1 2024-04-22T18:38:59.917 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.917 INFO:tasks.rados.rados.0.smithi089.stdout:307: finishing set_chunk to oid 8 2024-04-22T18:38:59.917 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 643584 length: 557056 2024-04-22T18:38:59.917 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-22T18:38:59.917 INFO:tasks.rados.rados.0.smithi089.stdout:306: finishing set_chunk to oid 7 2024-04-22T18:38:59.917 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 121344 length: 310272 2024-04-22T18:38:59.917 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-22T18:38:59.917 INFO:tasks.rados.rados.0.smithi089.stdout:306: done (15 left) 2024-04-22T18:38:59.917 INFO:tasks.rados.rados.0.smithi089.stdout:307: done (14 left) 2024-04-22T18:38:59.917 INFO:tasks.rados.rados.0.smithi089.stdout:313: set_chunk oid 14 target oid 14 low_tier 2024-04-22T18:38:59.917 INFO:tasks.rados.rados.0.smithi089.stdout:313: set_chunk oid 14 offset: 635392 length: 260096 target oid smithi08937929-14 low_tier offset: 635392 snap -1 2024-04-22T18:38:59.917 INFO:tasks.rados.rados.0.smithi089.stdout:314: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.917 INFO:tasks.rados.rados.0.smithi089.stdout:314: set_chunk oid 15 offset: 51200 length: 1626112 target oid smithi08937929-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 51200 snap -1 2024-04-22T18:38:59.918 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.918 INFO:tasks.rados.rados.0.smithi089.stdout:311: finishing set_chunk to oid 12 2024-04-22T18:38:59.918 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 263168 length: 1913344 2024-04-22T18:38:59.918 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-22T18:38:59.918 INFO:tasks.rados.rados.0.smithi089.stdout:311: done (15 left) 2024-04-22T18:38:59.918 INFO:tasks.rados.rados.0.smithi089.stdout:315: set_chunk oid 16 target oid 16 low_tier 2024-04-22T18:38:59.918 INFO:tasks.rados.rados.0.smithi089.stdout:315: set_chunk oid 16 offset: 155136 length: 398336 target oid smithi08937929-16 low_tier offset: 155136 snap -1 2024-04-22T18:38:59.918 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.918 INFO:tasks.rados.rados.0.smithi089.stdout:310: finishing set_chunk to oid 11 2024-04-22T18:38:59.918 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 741376 length: 629248 2024-04-22T18:38:59.918 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-22T18:38:59.918 INFO:tasks.rados.rados.0.smithi089.stdout:313: finishing set_chunk to oid 14 2024-04-22T18:38:59.918 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 635392 length: 260096 2024-04-22T18:38:59.919 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-22T18:38:59.919 INFO:tasks.rados.rados.0.smithi089.stdout:310: done (15 left) 2024-04-22T18:38:59.919 INFO:tasks.rados.rados.0.smithi089.stdout:313: done (14 left) 2024-04-22T18:38:59.919 INFO:tasks.rados.rados.0.smithi089.stdout:316: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.919 INFO:tasks.rados.rados.0.smithi089.stdout:316: set_chunk oid 17 offset: 249344 length: 1667584 target oid smithi08937929-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 249344 snap -1 2024-04-22T18:38:59.919 INFO:tasks.rados.rados.0.smithi089.stdout:317: set_chunk oid 18 target oid 18 low_tier 2024-04-22T18:38:59.919 INFO:tasks.rados.rados.0.smithi089.stdout:317: set_chunk oid 18 offset: 1033216 length: 641024 target oid smithi08937929-18 low_tier offset: 1033216 snap -1 2024-04-22T18:38:59.919 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.919 INFO:tasks.rados.rados.0.smithi089.stdout:316: finishing set_chunk to oid 17 2024-04-22T18:38:59.919 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 249344 length: 1667584 2024-04-22T18:38:59.919 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-22T18:38:59.919 INFO:tasks.rados.rados.0.smithi089.stdout:316: done (15 left) 2024-04-22T18:38:59.919 INFO:tasks.rados.rados.0.smithi089.stdout:318: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.919 INFO:tasks.rados.rados.0.smithi089.stdout:318: set_chunk oid 19 offset: 198656 length: 1055232 target oid smithi08937929-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 198656 snap -1 2024-04-22T18:38:59.920 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.920 INFO:tasks.rados.rados.0.smithi089.stdout:317: finishing set_chunk to oid 18 2024-04-22T18:38:59.920 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1033216 length: 641024 2024-04-22T18:38:59.920 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-22T18:38:59.920 INFO:tasks.rados.rados.0.smithi089.stdout:317: done (15 left) 2024-04-22T18:38:59.920 INFO:tasks.rados.rados.0.smithi089.stdout:319: set_chunk oid 20 target oid 20 low_tier 2024-04-22T18:38:59.920 INFO:tasks.rados.rados.0.smithi089.stdout:319: set_chunk oid 20 offset: 1017856 length: 861184 target oid smithi08937929-20 low_tier offset: 1017856 snap -1 2024-04-22T18:38:59.920 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.920 INFO:tasks.rados.rados.0.smithi089.stdout:319: finishing set_chunk to oid 20 2024-04-22T18:38:59.920 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 1017856 length: 861184 2024-04-22T18:38:59.920 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-22T18:38:59.920 INFO:tasks.rados.rados.0.smithi089.stdout:319: done (15 left) 2024-04-22T18:38:59.920 INFO:tasks.rados.rados.0.smithi089.stdout:320: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:38:59.921 INFO:tasks.rados.rados.0.smithi089.stdout:320: set_chunk oid 21 offset: 606720 length: 1516544 target oid smithi08937929-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 606720 snap -1 2024-04-22T18:38:59.921 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:38:59.951 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:38:59.951+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:00.005 INFO:tasks.rados.rados.0.smithi089.stdout:301: finishing set_chunk to oid 2 2024-04-22T18:39:00.005 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 255488 length: 662016 2024-04-22T18:39:00.005 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-22T18:39:00.005 INFO:tasks.rados.rados.0.smithi089.stdout:301: done (15 left) 2024-04-22T18:39:00.005 INFO:tasks.rados.rados.0.smithi089.stdout:321: set_chunk oid 22 target oid 22 low_tier 2024-04-22T18:39:00.005 INFO:tasks.rados.rados.0.smithi089.stdout:321: set_chunk oid 22 offset: 183808 length: 1866240 target oid smithi08937929-22 low_tier offset: 183808 snap -1 2024-04-22T18:39:00.005 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.005 INFO:tasks.rados.rados.0.smithi089.stdout:304: finishing set_chunk to oid 5 2024-04-22T18:39:00.005 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 714752 length: 94208 2024-04-22T18:39:00.005 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-22T18:39:00.006 INFO:tasks.rados.rados.0.smithi089.stdout:304: done (15 left) 2024-04-22T18:39:00.006 INFO:tasks.rados.rados.0.smithi089.stdout:322: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.006 INFO:tasks.rados.rados.0.smithi089.stdout:322: set_chunk oid 23 offset: 151040 length: 759808 target oid smithi08937929-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 151040 snap -1 2024-04-22T18:39:00.006 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.006 INFO:tasks.rados.rados.0.smithi089.stdout:318: finishing set_chunk to oid 19 2024-04-22T18:39:00.006 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 198656 length: 1055232 2024-04-22T18:39:00.006 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-22T18:39:00.006 INFO:tasks.rados.rados.0.smithi089.stdout:318: done (15 left) 2024-04-22T18:39:00.006 INFO:tasks.rados.rados.0.smithi089.stdout:323: set_chunk oid 24 target oid 24 low_tier 2024-04-22T18:39:00.006 INFO:tasks.rados.rados.0.smithi089.stdout:323: set_chunk oid 24 offset: 1321472 length: 261632 target oid smithi08937929-24 low_tier offset: 1321472 snap -1 2024-04-22T18:39:00.006 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.008 INFO:tasks.rados.rados.0.smithi089.stdout:320: finishing set_chunk to oid 21 2024-04-22T18:39:00.008 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 606720 length: 1516544 2024-04-22T18:39:00.008 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-22T18:39:00.008 INFO:tasks.rados.rados.0.smithi089.stdout:320: done (15 left) 2024-04-22T18:39:00.008 INFO:tasks.rados.rados.0.smithi089.stdout:324: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.008 INFO:tasks.rados.rados.0.smithi089.stdout:324: set_chunk oid 25 offset: 1203712 length: 755712 target oid smithi08937929-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1203712 snap -1 2024-04-22T18:39:00.008 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.008 INFO:tasks.rados.rados.0.smithi089.stdout:324: finishing set_chunk to oid 25 2024-04-22T18:39:00.008 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1203712 length: 755712 2024-04-22T18:39:00.008 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-22T18:39:00.009 INFO:tasks.rados.rados.0.smithi089.stdout:324: done (15 left) 2024-04-22T18:39:00.009 INFO:tasks.rados.rados.0.smithi089.stdout:325: set_chunk oid 26 target oid 26 low_tier 2024-04-22T18:39:00.009 INFO:tasks.rados.rados.0.smithi089.stdout:325: set_chunk oid 26 offset: 1318912 length: 770560 target oid smithi08937929-26 low_tier offset: 1318912 snap -1 2024-04-22T18:39:00.009 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.009 INFO:tasks.rados.rados.0.smithi089.stdout:298: finishing set_chunk to oid 49 2024-04-22T18:39:00.009 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:39:00.009 INFO:tasks.rados.rados.0.smithi089.stdout:298: done (15 left) 2024-04-22T18:39:00.009 INFO:tasks.rados.rados.0.smithi089.stdout:326: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.009 INFO:tasks.rados.rados.0.smithi089.stdout:326: set_chunk oid 27 offset: 1008640 length: 710656 target oid smithi08937929-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1008640 snap -1 2024-04-22T18:39:00.009 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.010 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:00.010 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:00.010 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:00.010 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:00.010 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:00.019 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:00.023 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:00.024 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:00.035 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:00.036+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:00.040 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:00.041 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:00.057 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:00.057 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:00.089 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:00.089 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:00.107 INFO:tasks.rados.rados.0.smithi089.stdout:296: finishing set_chunk to oid 47 2024-04-22T18:39:00.107 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 97280 length: 2065920 2024-04-22T18:39:00.107 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-22T18:39:00.107 INFO:tasks.rados.rados.0.smithi089.stdout:305: finishing set_chunk to oid 6 2024-04-22T18:39:00.107 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 465408 length: 276480 2024-04-22T18:39:00.108 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-22T18:39:00.108 INFO:tasks.rados.rados.0.smithi089.stdout:289: finishing set_chunk to oid 40 2024-04-22T18:39:00.108 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 18944 length: 1477120 2024-04-22T18:39:00.108 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-22T18:39:00.108 INFO:tasks.rados.rados.0.smithi089.stdout:289: done (15 left) 2024-04-22T18:39:00.108 INFO:tasks.rados.rados.0.smithi089.stdout:296: done (14 left) 2024-04-22T18:39:00.108 INFO:tasks.rados.rados.0.smithi089.stdout:305: done (13 left) 2024-04-22T18:39:00.108 INFO:tasks.rados.rados.0.smithi089.stdout:327: set_chunk oid 28 target oid 28 low_tier 2024-04-22T18:39:00.108 INFO:tasks.rados.rados.0.smithi089.stdout:327: set_chunk oid 28 offset: 1260544 length: 469504 target oid smithi08937929-28 low_tier offset: 1260544 snap -1 2024-04-22T18:39:00.108 INFO:tasks.rados.rados.0.smithi089.stdout:328: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.108 INFO:tasks.rados.rados.0.smithi089.stdout:328: set_chunk oid 29 offset: 1590272 length: 118272 target oid smithi08937929-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1590272 snap -1 2024-04-22T18:39:00.108 INFO:tasks.rados.rados.0.smithi089.stdout:329: set_chunk oid 30 target oid 30 low_tier 2024-04-22T18:39:00.108 INFO:tasks.rados.rados.0.smithi089.stdout:329: set_chunk oid 30 offset: 2225152 length: 101376 target oid smithi08937929-30 low_tier offset: 2225152 snap -1 2024-04-22T18:39:00.108 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.109 INFO:tasks.rados.rados.0.smithi089.stdout:312: finishing set_chunk to oid 13 2024-04-22T18:39:00.109 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 601600 length: 1342976 2024-04-22T18:39:00.109 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-22T18:39:00.109 INFO:tasks.rados.rados.0.smithi089.stdout:312: done (15 left) 2024-04-22T18:39:00.109 INFO:tasks.rados.rados.0.smithi089.stdout:330: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.109 INFO:tasks.rados.rados.0.smithi089.stdout:330: set_chunk oid 31 offset: 62976 length: 301056 target oid smithi08937929-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 62976 snap -1 2024-04-22T18:39:00.109 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.109 INFO:tasks.rados.rados.0.smithi089.stdout:314: finishing set_chunk to oid 15 2024-04-22T18:39:00.109 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 51200 length: 1626112 2024-04-22T18:39:00.109 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-22T18:39:00.109 INFO:tasks.rados.rados.0.smithi089.stdout:314: done (15 left) 2024-04-22T18:39:00.109 INFO:tasks.rados.rados.0.smithi089.stdout:331: set_chunk oid 32 target oid 32 low_tier 2024-04-22T18:39:00.109 INFO:tasks.rados.rados.0.smithi089.stdout:331: set_chunk oid 32 offset: 1324544 length: 980480 target oid smithi08937929-32 low_tier offset: 1324544 snap -1 2024-04-22T18:39:00.109 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.110 INFO:tasks.rados.rados.0.smithi089.stdout:331: finishing set_chunk to oid 32 2024-04-22T18:39:00.110 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 1324544 length: 980480 2024-04-22T18:39:00.110 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-22T18:39:00.110 INFO:tasks.rados.rados.0.smithi089.stdout:331: done (15 left) 2024-04-22T18:39:00.110 INFO:tasks.rados.rados.0.smithi089.stdout:332: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.110 INFO:tasks.rados.rados.0.smithi089.stdout:332: set_chunk oid 33 offset: 179712 length: 2018816 target oid smithi08937929-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 179712 snap -1 2024-04-22T18:39:00.110 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.110 INFO:tasks.rados.rados.0.smithi089.stdout:321: finishing set_chunk to oid 22 2024-04-22T18:39:00.110 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 183808 length: 1866240 2024-04-22T18:39:00.110 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-22T18:39:00.110 INFO:tasks.rados.rados.0.smithi089.stdout:303: finishing set_chunk to oid 4 2024-04-22T18:39:00.110 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 1030656 length: 828928 2024-04-22T18:39:00.110 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-22T18:39:00.110 INFO:tasks.rados.rados.0.smithi089.stdout:308: finishing set_chunk to oid 9 2024-04-22T18:39:00.111 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 275456 length: 1419264 2024-04-22T18:39:00.111 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-22T18:39:00.111 INFO:tasks.rados.rados.0.smithi089.stdout:303: done (15 left) 2024-04-22T18:39:00.111 INFO:tasks.rados.rados.0.smithi089.stdout:308: done (14 left) 2024-04-22T18:39:00.111 INFO:tasks.rados.rados.0.smithi089.stdout:321: done (13 left) 2024-04-22T18:39:00.111 INFO:tasks.rados.rados.0.smithi089.stdout:333: set_chunk oid 34 target oid 34 low_tier 2024-04-22T18:39:00.111 INFO:tasks.rados.rados.0.smithi089.stdout:333: set_chunk oid 34 offset: 249856 length: 1118720 target oid smithi08937929-34 low_tier offset: 249856 snap -1 2024-04-22T18:39:00.111 INFO:tasks.rados.rados.0.smithi089.stdout:322: finishing set_chunk to oid 23 2024-04-22T18:39:00.111 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 151040 length: 759808 2024-04-22T18:39:00.111 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-22T18:39:00.111 INFO:tasks.rados.rados.0.smithi089.stdout:322: done (13 left) 2024-04-22T18:39:00.111 INFO:tasks.rados.rados.0.smithi089.stdout:334: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.111 INFO:tasks.rados.rados.0.smithi089.stdout:334: set_chunk oid 35 offset: 1508864 length: 734720 target oid smithi08937929-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1508864 snap -1 2024-04-22T18:39:00.111 INFO:tasks.rados.rados.0.smithi089.stdout:335: set_chunk oid 36 target oid 36 low_tier 2024-04-22T18:39:00.111 INFO:tasks.rados.rados.0.smithi089.stdout:335: set_chunk oid 36 offset: 679936 length: 671232 target oid smithi08937929-36 low_tier offset: 679936 snap -1 2024-04-22T18:39:00.112 INFO:tasks.rados.rados.0.smithi089.stdout:336: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.112 INFO:tasks.rados.rados.0.smithi089.stdout:336: set_chunk oid 37 offset: 482816 length: 1530880 target oid smithi08937929-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 482816 snap -1 2024-04-22T18:39:00.112 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.112 INFO:tasks.rados.rados.0.smithi089.stdout:325: finishing set_chunk to oid 26 2024-04-22T18:39:00.112 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1318912 length: 770560 2024-04-22T18:39:00.112 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-22T18:39:00.112 INFO:tasks.rados.rados.0.smithi089.stdout:323: finishing set_chunk to oid 24 2024-04-22T18:39:00.112 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 1321472 length: 261632 2024-04-22T18:39:00.112 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-22T18:39:00.112 INFO:tasks.rados.rados.0.smithi089.stdout:332: finishing set_chunk to oid 33 2024-04-22T18:39:00.112 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 179712 length: 2018816 2024-04-22T18:39:00.112 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-22T18:39:00.112 INFO:tasks.rados.rados.0.smithi089.stdout:323: done (15 left) 2024-04-22T18:39:00.112 INFO:tasks.rados.rados.0.smithi089.stdout:325: done (14 left) 2024-04-22T18:39:00.112 INFO:tasks.rados.rados.0.smithi089.stdout:332: done (13 left) 2024-04-22T18:39:00.113 INFO:tasks.rados.rados.0.smithi089.stdout:337: set_chunk oid 38 target oid 38 low_tier 2024-04-22T18:39:00.113 INFO:tasks.rados.rados.0.smithi089.stdout:337: set_chunk oid 38 offset: 982016 length: 746496 target oid smithi08937929-38 low_tier offset: 982016 snap -1 2024-04-22T18:39:00.113 INFO:tasks.rados.rados.0.smithi089.stdout:338: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.113 INFO:tasks.rados.rados.0.smithi089.stdout:338: set_chunk oid 39 offset: 673792 length: 933888 target oid smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 673792 snap -1 2024-04-22T18:39:00.113 INFO:tasks.rados.rados.0.smithi089.stdout:339: set_chunk oid 40 target oid 40 low_tier 2024-04-22T18:39:00.113 INFO:tasks.rados.rados.0.smithi089.stdout:339: set_chunk oid 40 offset: 1848320 length: 577536 target oid smithi08937929-40 low_tier offset: 1848320 snap -1 2024-04-22T18:39:00.113 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.113 INFO:tasks.rados.rados.0.smithi089.stdout:335: finishing set_chunk to oid 36 2024-04-22T18:39:00.113 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 679936 length: 671232 2024-04-22T18:39:00.113 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-22T18:39:00.113 INFO:tasks.rados.rados.0.smithi089.stdout:335: done (15 left) 2024-04-22T18:39:00.113 INFO:tasks.rados.rados.0.smithi089.stdout:340: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.113 INFO:tasks.rados.rados.0.smithi089.stdout:340: set_chunk oid 41 offset: 2275840 length: 378880 target oid smithi08937929-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2275840 snap -1 2024-04-22T18:39:00.114 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.114 INFO:tasks.rados.rados.0.smithi089.stdout:326: finishing set_chunk to oid 27 2024-04-22T18:39:00.114 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1008640 length: 710656 2024-04-22T18:39:00.114 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-22T18:39:00.114 INFO:tasks.rados.rados.0.smithi089.stdout:334: finishing set_chunk to oid 35 2024-04-22T18:39:00.114 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1508864 length: 734720 2024-04-22T18:39:00.114 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-22T18:39:00.114 INFO:tasks.rados.rados.0.smithi089.stdout:327: finishing set_chunk to oid 28 2024-04-22T18:39:00.114 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 1260544 length: 469504 2024-04-22T18:39:00.114 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-22T18:39:00.114 INFO:tasks.rados.rados.0.smithi089.stdout:333: finishing set_chunk to oid 34 2024-04-22T18:39:00.115 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 249856 length: 1118720 2024-04-22T18:39:00.115 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-22T18:39:00.115 INFO:tasks.rados.rados.0.smithi089.stdout:326: done (15 left) 2024-04-22T18:39:00.115 INFO:tasks.rados.rados.0.smithi089.stdout:327: done (14 left) 2024-04-22T18:39:00.115 INFO:tasks.rados.rados.0.smithi089.stdout:333: done (13 left) 2024-04-22T18:39:00.115 INFO:tasks.rados.rados.0.smithi089.stdout:334: done (12 left) 2024-04-22T18:39:00.115 INFO:tasks.rados.rados.0.smithi089.stdout:341: set_chunk oid 42 target oid 42 low_tier 2024-04-22T18:39:00.115 INFO:tasks.rados.rados.0.smithi089.stdout:341: set_chunk oid 42 offset: 1362944 length: 1152512 target oid smithi08937929-42 low_tier offset: 1362944 snap -1 2024-04-22T18:39:00.115 INFO:tasks.rados.rados.0.smithi089.stdout:341 retry set_chunk ! 2024-04-22T18:39:00.115 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 13 2024-04-22T18:39:00.115 INFO:tasks.rados.rados.0.smithi089.stdout:338: finishing set_chunk to oid 39 2024-04-22T18:39:00.115 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 673792 length: 933888 2024-04-22T18:39:00.115 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-22T18:39:00.115 INFO:tasks.rados.rados.0.smithi089.stdout:338: done (12 left) 2024-04-22T18:39:00.116 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 12 2024-04-22T18:39:00.116 INFO:tasks.rados.rados.0.smithi089.stdout:341: finishing set_chunk to oid 42 2024-04-22T18:39:00.116 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 1362944 length: 1152512 2024-04-22T18:39:00.116 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-22T18:39:00.116 INFO:tasks.rados.rados.0.smithi089.stdout:341: done (11 left) 2024-04-22T18:39:00.116 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 11 2024-04-22T18:39:00.116 INFO:tasks.rados.rados.0.smithi089.stdout:309: finishing set_chunk to oid 10 2024-04-22T18:39:00.116 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-22T18:39:00.116 INFO:tasks.rados.rados.0.smithi089.stdout:309: done (10 left) 2024-04-22T18:39:00.116 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 10 2024-04-22T18:39:00.120 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:00.120 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:00.136 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:00.137+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:00.137 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:00.137 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:00.149 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:00.149 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:00.154 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:00.154 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:00.158 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:00.172 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:00.172 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:00.204 INFO:tasks.rados.rados.0.smithi089.stdout:328: finishing set_chunk to oid 29 2024-04-22T18:39:00.204 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1590272 length: 118272 2024-04-22T18:39:00.204 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-22T18:39:00.204 INFO:tasks.rados.rados.0.smithi089.stdout:328: done (9 left) 2024-04-22T18:39:00.204 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 9 2024-04-22T18:39:00.205 INFO:tasks.rados.rados.0.smithi089.stdout:336: finishing set_chunk to oid 37 2024-04-22T18:39:00.205 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 482816 length: 1530880 2024-04-22T18:39:00.205 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-22T18:39:00.205 INFO:tasks.rados.rados.0.smithi089.stdout:336: done (8 left) 2024-04-22T18:39:00.205 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 8 2024-04-22T18:39:00.205 INFO:tasks.rados.rados.0.smithi089.stdout:337: finishing set_chunk to oid 38 2024-04-22T18:39:00.205 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 982016 length: 746496 2024-04-22T18:39:00.206 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-22T18:39:00.206 INFO:tasks.rados.rados.0.smithi089.stdout:337: done (7 left) 2024-04-22T18:39:00.206 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 7 2024-04-22T18:39:00.209 INFO:tasks.rados.rados.0.smithi089.stdout:329: finishing set_chunk to oid 30 2024-04-22T18:39:00.209 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-22T18:39:00.209 INFO:tasks.rados.rados.0.smithi089.stdout:329: done (6 left) 2024-04-22T18:39:00.210 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 6 2024-04-22T18:39:00.237 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:00.238+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:00.274 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:00.274 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:00.283 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:00.305 INFO:tasks.rados.rados.0.smithi089.stdout:315: finishing set_chunk to oid 16 2024-04-22T18:39:00.306 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-22T18:39:00.306 INFO:tasks.rados.rados.0.smithi089.stdout:315: done (5 left) 2024-04-22T18:39:00.306 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 5 2024-04-22T18:39:00.306 INFO:tasks.rados.rados.0.smithi089.stdout:330: finishing set_chunk to oid 31 2024-04-22T18:39:00.307 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 3 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-22T18:39:00.307 INFO:tasks.rados.rados.0.smithi089.stdout:330: done (4 left) 2024-04-22T18:39:00.307 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 4 2024-04-22T18:39:00.308 INFO:tasks.rados.rados.0.smithi089.stdout:340: finishing set_chunk to oid 41 2024-04-22T18:39:00.308 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 18 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-22T18:39:00.309 INFO:tasks.rados.rados.0.smithi089.stdout:340: done (3 left) 2024-04-22T18:39:00.309 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 3 2024-04-22T18:39:00.338 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:00.338+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:00.405 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:00.406 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:00.407 INFO:tasks.rados.rados.0.smithi089.stdout:339: finishing set_chunk to oid 40 2024-04-22T18:39:00.407 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 3 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-22T18:39:00.407 INFO:tasks.rados.rados.0.smithi089.stdout:339: done (2 left) 2024-04-22T18:39:00.407 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 2 2024-04-22T18:39:00.407 INFO:tasks.rados.rados.0.smithi089.stdout:292: finishing set_chunk to oid 43 2024-04-22T18:39:00.407 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-22T18:39:00.407 INFO:tasks.rados.rados.0.smithi089.stdout:292: done (1 left) 2024-04-22T18:39:00.407 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 1 2024-04-22T18:39:00.415 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:00.438 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:00.439+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:00.510 INFO:tasks.rados.rados.0.smithi089.stdout:302: finishing set_chunk to oid 3 2024-04-22T18:39:00.510 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 6 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-22T18:39:00.510 INFO:tasks.rados.rados.0.smithi089.stdout:302: done (0 left) 2024-04-22T18:39:00.510 INFO:tasks.rados.rados.0.smithi089.stdout:342: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.510 INFO:tasks.rados.rados.0.smithi089.stdout:342: set_chunk oid 43 offset: 1920000 length: 186368 target oid smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1920000 snap -1 2024-04-22T18:39:00.511 INFO:tasks.rados.rados.0.smithi089.stdout:343: set_chunk oid 44 target oid 44 low_tier 2024-04-22T18:39:00.511 INFO:tasks.rados.rados.0.smithi089.stdout:343: set_chunk oid 44 offset: 59392 length: 1303040 target oid smithi08937929-44 low_tier offset: 59392 snap -1 2024-04-22T18:39:00.511 INFO:tasks.rados.rados.0.smithi089.stdout:344: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.511 INFO:tasks.rados.rados.0.smithi089.stdout:344: set_chunk oid 45 offset: 201728 length: 484864 target oid smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 201728 snap -1 2024-04-22T18:39:00.511 INFO:tasks.rados.rados.0.smithi089.stdout:345: set_chunk oid 46 target oid 46 low_tier 2024-04-22T18:39:00.511 INFO:tasks.rados.rados.0.smithi089.stdout:345: set_chunk oid 46 offset: 1533440 length: 731136 target oid smithi08937929-46 low_tier offset: 1533440 snap -1 2024-04-22T18:39:00.511 INFO:tasks.rados.rados.0.smithi089.stdout:346: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.511 INFO:tasks.rados.rados.0.smithi089.stdout:346: set_chunk oid 47 offset: 545792 length: 1414656 target oid smithi08937929-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 545792 snap -1 2024-04-22T18:39:00.511 INFO:tasks.rados.rados.0.smithi089.stdout:347: set_chunk oid 48 target oid 48 low_tier 2024-04-22T18:39:00.511 INFO:tasks.rados.rados.0.smithi089.stdout:347: set_chunk oid 48 offset: 196608 length: 48128 target oid smithi08937929-48 low_tier offset: 196608 snap -1 2024-04-22T18:39:00.511 INFO:tasks.rados.rados.0.smithi089.stdout:348: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.511 INFO:tasks.rados.rados.0.smithi089.stdout:348: set_chunk oid 49 offset: 176640 length: 344576 target oid smithi08937929-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 176640 snap -1 2024-04-22T18:39:00.511 INFO:tasks.rados.rados.0.smithi089.stdout:349: set_chunk oid 50 target oid 50 low_tier 2024-04-22T18:39:00.511 INFO:tasks.rados.rados.0.smithi089.stdout:349: set_chunk oid 50 offset: 150016 length: 661504 target oid smithi08937929-50 low_tier offset: 150016 snap -1 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:350: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:350: set_chunk oid 1 offset: 643072 length: 257536 target oid smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 643072 snap -1 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:351: set_chunk oid 2 target oid 2 low_tier 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:351: set_chunk oid 2 offset: 661504 length: 1045504 target oid smithi08937929-2 low_tier offset: 661504 snap -1 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:352: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:352: set_chunk oid 3 offset: 1009152 length: 596992 target oid smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1009152 snap -1 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:353: set_chunk oid 4 target oid 4 low_tier 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:353: set_chunk oid 4 offset: 1240576 length: 100352 target oid smithi08937929-4 low_tier offset: 1240576 snap -1 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:354: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:354: set_chunk oid 5 offset: 934400 length: 825344 target oid smithi08937929-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 934400 snap -1 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:355: set_chunk oid 6 target oid 6 low_tier 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:355: set_chunk oid 6 offset: 154624 length: 557056 target oid smithi08937929-6 low_tier offset: 154624 snap -1 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:356: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:356: set_chunk oid 7 offset: 32256 length: 1777152 target oid smithi08937929-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 32256 snap -1 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:357: set_chunk oid 8 target oid 8 low_tier 2024-04-22T18:39:00.512 INFO:tasks.rados.rados.0.smithi089.stdout:357: set_chunk oid 8 offset: 122368 length: 512000 target oid smithi08937929-8 low_tier offset: 122368 snap -1 2024-04-22T18:39:00.513 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.513 INFO:tasks.rados.rados.0.smithi089.stdout:351: finishing set_chunk to oid 2 2024-04-22T18:39:00.513 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 661504 length: 1045504 2024-04-22T18:39:00.513 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-22T18:39:00.513 INFO:tasks.rados.rados.0.smithi089.stdout:351: done (15 left) 2024-04-22T18:39:00.513 INFO:tasks.rados.rados.0.smithi089.stdout:358: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.513 INFO:tasks.rados.rados.0.smithi089.stdout:358: set_chunk oid 9 offset: 172544 length: 336384 target oid smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 172544 snap -1 2024-04-22T18:39:00.513 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.513 INFO:tasks.rados.rados.0.smithi089.stdout:352: finishing set_chunk to oid 3 2024-04-22T18:39:00.513 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1009152 length: 596992 2024-04-22T18:39:00.513 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-22T18:39:00.513 INFO:tasks.rados.rados.0.smithi089.stdout:352: done (15 left) 2024-04-22T18:39:00.513 INFO:tasks.rados.rados.0.smithi089.stdout:359: set_chunk oid 10 target oid 10 low_tier 2024-04-22T18:39:00.513 INFO:tasks.rados.rados.0.smithi089.stdout:359: set_chunk oid 10 offset: 1726976 length: 81408 target oid smithi08937929-10 low_tier offset: 1726976 snap -1 2024-04-22T18:39:00.514 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.514 INFO:tasks.rados.rados.0.smithi089.stdout:350: finishing set_chunk to oid 1 2024-04-22T18:39:00.514 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 643072 length: 257536 2024-04-22T18:39:00.514 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-22T18:39:00.514 INFO:tasks.rados.rados.0.smithi089.stdout:346: finishing set_chunk to oid 47 2024-04-22T18:39:00.514 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 545792 length: 1414656 2024-04-22T18:39:00.514 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-22T18:39:00.514 INFO:tasks.rados.rados.0.smithi089.stdout:346: done (15 left) 2024-04-22T18:39:00.514 INFO:tasks.rados.rados.0.smithi089.stdout:350: done (14 left) 2024-04-22T18:39:00.514 INFO:tasks.rados.rados.0.smithi089.stdout:360: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.514 INFO:tasks.rados.rados.0.smithi089.stdout:360: set_chunk oid 11 offset: 208384 length: 1937920 target oid smithi08937929-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 208384 snap -1 2024-04-22T18:39:00.514 INFO:tasks.rados.rados.0.smithi089.stdout:361: set_chunk oid 12 target oid 12 low_tier 2024-04-22T18:39:00.515 INFO:tasks.rados.rados.0.smithi089.stdout:361: set_chunk oid 12 offset: 1108992 length: 74240 target oid smithi08937929-12 low_tier offset: 1108992 snap -1 2024-04-22T18:39:00.515 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.515 INFO:tasks.rados.rados.0.smithi089.stdout:360: finishing set_chunk to oid 11 2024-04-22T18:39:00.515 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 208384 length: 1937920 2024-04-22T18:39:00.515 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-22T18:39:00.515 INFO:tasks.rados.rados.0.smithi089.stdout:360: done (15 left) 2024-04-22T18:39:00.515 INFO:tasks.rados.rados.0.smithi089.stdout:362: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.515 INFO:tasks.rados.rados.0.smithi089.stdout:362: set_chunk oid 13 offset: 1516544 length: 529408 target oid smithi08937929-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1516544 snap -1 2024-04-22T18:39:00.515 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.539 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:00.540+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:00.557 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:00.557 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:00.566 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:00.614 INFO:tasks.rados.rados.0.smithi089.stdout:343: finishing set_chunk to oid 44 2024-04-22T18:39:00.614 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 59392 length: 1303040 2024-04-22T18:39:00.614 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-22T18:39:00.614 INFO:tasks.rados.rados.0.smithi089.stdout:343: done (15 left) 2024-04-22T18:39:00.614 INFO:tasks.rados.rados.0.smithi089.stdout:363: set_chunk oid 14 target oid 14 low_tier 2024-04-22T18:39:00.614 INFO:tasks.rados.rados.0.smithi089.stdout:363: set_chunk oid 14 offset: 708608 length: 716800 target oid smithi08937929-14 low_tier offset: 708608 snap -1 2024-04-22T18:39:00.614 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.615 INFO:tasks.rados.rados.0.smithi089.stdout:348: finishing set_chunk to oid 49 2024-04-22T18:39:00.615 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 176640 length: 344576 2024-04-22T18:39:00.615 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:39:00.615 INFO:tasks.rados.rados.0.smithi089.stdout:348: done (15 left) 2024-04-22T18:39:00.615 INFO:tasks.rados.rados.0.smithi089.stdout:364: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.615 INFO:tasks.rados.rados.0.smithi089.stdout:364: set_chunk oid 15 offset: 216576 length: 1318400 target oid smithi08937929-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 216576 snap -1 2024-04-22T18:39:00.615 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.615 INFO:tasks.rados.rados.0.smithi089.stdout:347: finishing set_chunk to oid 48 2024-04-22T18:39:00.615 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 196608 length: 48128 2024-04-22T18:39:00.615 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-22T18:39:00.615 INFO:tasks.rados.rados.0.smithi089.stdout:347: done (15 left) 2024-04-22T18:39:00.615 INFO:tasks.rados.rados.0.smithi089.stdout:365: set_chunk oid 16 target oid 16 low_tier 2024-04-22T18:39:00.615 INFO:tasks.rados.rados.0.smithi089.stdout:365: set_chunk oid 16 offset: 2099712 length: 118784 target oid smithi08937929-16 low_tier offset: 2099712 snap -1 2024-04-22T18:39:00.615 INFO:tasks.rados.rados.0.smithi089.stdout:345: finishing set_chunk to oid 46 2024-04-22T18:39:00.616 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1533440 length: 731136 2024-04-22T18:39:00.616 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-22T18:39:00.616 INFO:tasks.rados.rados.0.smithi089.stdout:356: finishing set_chunk to oid 7 2024-04-22T18:39:00.616 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 32256 length: 1777152 2024-04-22T18:39:00.616 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-22T18:39:00.616 INFO:tasks.rados.rados.0.smithi089.stdout:349: finishing set_chunk to oid 50 2024-04-22T18:39:00.616 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 150016 length: 661504 2024-04-22T18:39:00.616 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-22T18:39:00.616 INFO:tasks.rados.rados.0.smithi089.stdout:353: finishing set_chunk to oid 4 2024-04-22T18:39:00.616 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-22T18:39:00.616 INFO:tasks.rados.rados.0.smithi089.stdout:354: finishing set_chunk to oid 5 2024-04-22T18:39:00.616 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 934400 length: 825344 2024-04-22T18:39:00.616 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-22T18:39:00.617 INFO:tasks.rados.rados.0.smithi089.stdout:355: finishing set_chunk to oid 6 2024-04-22T18:39:00.617 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 154624 length: 557056 2024-04-22T18:39:00.617 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-22T18:39:00.617 INFO:tasks.rados.rados.0.smithi089.stdout:357: finishing set_chunk to oid 8 2024-04-22T18:39:00.617 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 122368 length: 512000 2024-04-22T18:39:00.617 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-22T18:39:00.617 INFO:tasks.rados.rados.0.smithi089.stdout:363: finishing set_chunk to oid 14 2024-04-22T18:39:00.617 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 708608 length: 716800 2024-04-22T18:39:00.617 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-22T18:39:00.617 INFO:tasks.rados.rados.0.smithi089.stdout:345: done (15 left) 2024-04-22T18:39:00.617 INFO:tasks.rados.rados.0.smithi089.stdout:349: done (14 left) 2024-04-22T18:39:00.617 INFO:tasks.rados.rados.0.smithi089.stdout:353: done (13 left) 2024-04-22T18:39:00.617 INFO:tasks.rados.rados.0.smithi089.stdout:354: done (12 left) 2024-04-22T18:39:00.617 INFO:tasks.rados.rados.0.smithi089.stdout:355: done (11 left) 2024-04-22T18:39:00.618 INFO:tasks.rados.rados.0.smithi089.stdout:356: done (10 left) 2024-04-22T18:39:00.618 INFO:tasks.rados.rados.0.smithi089.stdout:357: done (9 left) 2024-04-22T18:39:00.618 INFO:tasks.rados.rados.0.smithi089.stdout:363: done (8 left) 2024-04-22T18:39:00.618 INFO:tasks.rados.rados.0.smithi089.stdout:366: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.618 INFO:tasks.rados.rados.0.smithi089.stdout:366: set_chunk oid 17 offset: 185344 length: 1230336 target oid smithi08937929-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 185344 snap -1 2024-04-22T18:39:00.618 INFO:tasks.rados.rados.0.smithi089.stdout:367: set_chunk oid 18 target oid 18 low_tier 2024-04-22T18:39:00.618 INFO:tasks.rados.rados.0.smithi089.stdout:367: set_chunk oid 18 offset: 1350656 length: 475648 target oid smithi08937929-18 low_tier offset: 1350656 snap -1 2024-04-22T18:39:00.618 INFO:tasks.rados.rados.0.smithi089.stdout:368: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.618 INFO:tasks.rados.rados.0.smithi089.stdout:368: set_chunk oid 19 offset: 655872 length: 897536 target oid smithi08937929-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 655872 snap -1 2024-04-22T18:39:00.618 INFO:tasks.rados.rados.0.smithi089.stdout:369: set_chunk oid 20 target oid 20 low_tier 2024-04-22T18:39:00.618 INFO:tasks.rados.rados.0.smithi089.stdout:369: set_chunk oid 20 offset: 105472 length: 574976 target oid smithi08937929-20 low_tier offset: 105472 snap -1 2024-04-22T18:39:00.618 INFO:tasks.rados.rados.0.smithi089.stdout:370: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.618 INFO:tasks.rados.rados.0.smithi089.stdout:370: set_chunk oid 21 offset: 823296 length: 632320 target oid smithi08937929-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 823296 snap -1 2024-04-22T18:39:00.618 INFO:tasks.rados.rados.0.smithi089.stdout:371: set_chunk oid 22 target oid 22 low_tier 2024-04-22T18:39:00.618 INFO:tasks.rados.rados.0.smithi089.stdout:371: set_chunk oid 22 offset: 593408 length: 801792 target oid smithi08937929-22 low_tier offset: 593408 snap -1 2024-04-22T18:39:00.619 INFO:tasks.rados.rados.0.smithi089.stdout:372: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.619 INFO:tasks.rados.rados.0.smithi089.stdout:372: set_chunk oid 23 offset: 979456 length: 562688 target oid smithi08937929-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 979456 snap -1 2024-04-22T18:39:00.619 INFO:tasks.rados.rados.0.smithi089.stdout:373: set_chunk oid 24 target oid 24 low_tier 2024-04-22T18:39:00.619 INFO:tasks.rados.rados.0.smithi089.stdout:373: set_chunk oid 24 offset: 2076160 length: 134656 target oid smithi08937929-24 low_tier offset: 2076160 snap -1 2024-04-22T18:39:00.619 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.619 INFO:tasks.rados.rados.0.smithi089.stdout:361: finishing set_chunk to oid 12 2024-04-22T18:39:00.619 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1108992 length: 74240 2024-04-22T18:39:00.619 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-22T18:39:00.619 INFO:tasks.rados.rados.0.smithi089.stdout:361: done (15 left) 2024-04-22T18:39:00.619 INFO:tasks.rados.rados.0.smithi089.stdout:374: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.619 INFO:tasks.rados.rados.0.smithi089.stdout:374: set_chunk oid 25 offset: 1509376 length: 324096 target oid smithi08937929-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1509376 snap -1 2024-04-22T18:39:00.619 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.619 INFO:tasks.rados.rados.0.smithi089.stdout:364: finishing set_chunk to oid 15 2024-04-22T18:39:00.619 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 216576 length: 1318400 2024-04-22T18:39:00.620 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-22T18:39:00.620 INFO:tasks.rados.rados.0.smithi089.stdout:365: finishing set_chunk to oid 16 2024-04-22T18:39:00.620 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 2099712 length: 118784 2024-04-22T18:39:00.620 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-22T18:39:00.620 INFO:tasks.rados.rados.0.smithi089.stdout:364: done (15 left) 2024-04-22T18:39:00.620 INFO:tasks.rados.rados.0.smithi089.stdout:365: done (14 left) 2024-04-22T18:39:00.620 INFO:tasks.rados.rados.0.smithi089.stdout:375: set_chunk oid 26 target oid 26 low_tier 2024-04-22T18:39:00.620 INFO:tasks.rados.rados.0.smithi089.stdout:375: set_chunk oid 26 offset: 500736 length: 906752 target oid smithi08937929-26 low_tier offset: 500736 snap -1 2024-04-22T18:39:00.620 INFO:tasks.rados.rados.0.smithi089.stdout:376: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.620 INFO:tasks.rados.rados.0.smithi089.stdout:376: set_chunk oid 27 offset: 635904 length: 17920 target oid smithi08937929-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 635904 snap -1 2024-04-22T18:39:00.620 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.620 INFO:tasks.rados.rados.0.smithi089.stdout:369: finishing set_chunk to oid 20 2024-04-22T18:39:00.620 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 105472 length: 574976 2024-04-22T18:39:00.620 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-22T18:39:00.620 INFO:tasks.rados.rados.0.smithi089.stdout:369: done (15 left) 2024-04-22T18:39:00.621 INFO:tasks.rados.rados.0.smithi089.stdout:377: set_chunk oid 28 target oid 28 low_tier 2024-04-22T18:39:00.621 INFO:tasks.rados.rados.0.smithi089.stdout:377: set_chunk oid 28 offset: 3072 length: 135680 target oid smithi08937929-28 low_tier offset: 3072 snap -1 2024-04-22T18:39:00.621 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.621 INFO:tasks.rados.rados.0.smithi089.stdout:366: finishing set_chunk to oid 17 2024-04-22T18:39:00.621 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 185344 length: 1230336 2024-04-22T18:39:00.621 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-22T18:39:00.621 INFO:tasks.rados.rados.0.smithi089.stdout:366: done (15 left) 2024-04-22T18:39:00.621 INFO:tasks.rados.rados.0.smithi089.stdout:378: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.621 INFO:tasks.rados.rados.0.smithi089.stdout:378: set_chunk oid 29 offset: 326656 length: 1812480 target oid smithi08937929-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 326656 snap -1 2024-04-22T18:39:00.621 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.621 INFO:tasks.rados.rados.0.smithi089.stdout:378: finishing set_chunk to oid 29 2024-04-22T18:39:00.621 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 326656 length: 1812480 2024-04-22T18:39:00.621 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-22T18:39:00.621 INFO:tasks.rados.rados.0.smithi089.stdout:374: finishing set_chunk to oid 25 2024-04-22T18:39:00.622 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1509376 length: 324096 2024-04-22T18:39:00.622 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-22T18:39:00.622 INFO:tasks.rados.rados.0.smithi089.stdout:374: done (15 left) 2024-04-22T18:39:00.622 INFO:tasks.rados.rados.0.smithi089.stdout:378: done (14 left) 2024-04-22T18:39:00.622 INFO:tasks.rados.rados.0.smithi089.stdout:379: set_chunk oid 30 target oid 30 low_tier 2024-04-22T18:39:00.622 INFO:tasks.rados.rados.0.smithi089.stdout:379: set_chunk oid 30 offset: 505856 length: 1895936 target oid smithi08937929-30 low_tier offset: 505856 snap -1 2024-04-22T18:39:00.622 INFO:tasks.rados.rados.0.smithi089.stdout:380: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.622 INFO:tasks.rados.rados.0.smithi089.stdout:380: set_chunk oid 31 offset: 1474048 length: 174592 target oid smithi08937929-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1474048 snap -1 2024-04-22T18:39:00.622 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.622 INFO:tasks.rados.rados.0.smithi089.stdout:380: finishing set_chunk to oid 31 2024-04-22T18:39:00.622 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1474048 length: 174592 2024-04-22T18:39:00.622 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-22T18:39:00.622 INFO:tasks.rados.rados.0.smithi089.stdout:380: done (15 left) 2024-04-22T18:39:00.622 INFO:tasks.rados.rados.0.smithi089.stdout:381: set_chunk oid 32 target oid 32 low_tier 2024-04-22T18:39:00.623 INFO:tasks.rados.rados.0.smithi089.stdout:381: set_chunk oid 32 offset: 1372160 length: 336384 target oid smithi08937929-32 low_tier offset: 1372160 snap -1 2024-04-22T18:39:00.623 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.640 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:00.641+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:00.707 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:00.707 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:00.712 INFO:tasks.rados.rados.0.smithi089.stdout:370: finishing set_chunk to oid 21 2024-04-22T18:39:00.712 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 823296 length: 632320 2024-04-22T18:39:00.712 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-22T18:39:00.713 INFO:tasks.rados.rados.0.smithi089.stdout:370: done (15 left) 2024-04-22T18:39:00.713 INFO:tasks.rados.rados.0.smithi089.stdout:382: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.713 INFO:tasks.rados.rados.0.smithi089.stdout:382: set_chunk oid 33 offset: 696320 length: 527872 target oid smithi08937929-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 696320 snap -1 2024-04-22T18:39:00.713 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.713 INFO:tasks.rados.rados.0.smithi089.stdout:368: finishing set_chunk to oid 19 2024-04-22T18:39:00.713 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 655872 length: 897536 2024-04-22T18:39:00.713 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-22T18:39:00.713 INFO:tasks.rados.rados.0.smithi089.stdout:371: finishing set_chunk to oid 22 2024-04-22T18:39:00.713 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 593408 length: 801792 2024-04-22T18:39:00.713 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-22T18:39:00.713 INFO:tasks.rados.rados.0.smithi089.stdout:379: finishing set_chunk to oid 30 2024-04-22T18:39:00.713 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 505856 length: 1895936 2024-04-22T18:39:00.713 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-22T18:39:00.713 INFO:tasks.rados.rados.0.smithi089.stdout:368: done (15 left) 2024-04-22T18:39:00.714 INFO:tasks.rados.rados.0.smithi089.stdout:371: done (14 left) 2024-04-22T18:39:00.714 INFO:tasks.rados.rados.0.smithi089.stdout:379: done (13 left) 2024-04-22T18:39:00.714 INFO:tasks.rados.rados.0.smithi089.stdout:383: set_chunk oid 34 target oid 34 low_tier 2024-04-22T18:39:00.714 INFO:tasks.rados.rados.0.smithi089.stdout:383: set_chunk oid 34 offset: 467456 length: 1874944 target oid smithi08937929-34 low_tier offset: 467456 snap -1 2024-04-22T18:39:00.714 INFO:tasks.rados.rados.0.smithi089.stdout:384: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.714 INFO:tasks.rados.rados.0.smithi089.stdout:384: set_chunk oid 35 offset: 393216 length: 296960 target oid smithi08937929-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 393216 snap -1 2024-04-22T18:39:00.714 INFO:tasks.rados.rados.0.smithi089.stdout:385: set_chunk oid 36 target oid 36 low_tier 2024-04-22T18:39:00.714 INFO:tasks.rados.rados.0.smithi089.stdout:385: set_chunk oid 36 offset: 385024 length: 697856 target oid smithi08937929-36 low_tier offset: 385024 snap -1 2024-04-22T18:39:00.714 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.714 INFO:tasks.rados.rados.0.smithi089.stdout:372: finishing set_chunk to oid 23 2024-04-22T18:39:00.714 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 979456 length: 562688 2024-04-22T18:39:00.714 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-22T18:39:00.714 INFO:tasks.rados.rados.0.smithi089.stdout:372: done (15 left) 2024-04-22T18:39:00.714 INFO:tasks.rados.rados.0.smithi089.stdout:386: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.715 INFO:tasks.rados.rados.0.smithi089.stdout:386: set_chunk oid 37 offset: 576000 length: 124928 target oid smithi08937929-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 576000 snap -1 2024-04-22T18:39:00.715 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.715 INFO:tasks.rados.rados.0.smithi089.stdout:382: finishing set_chunk to oid 33 2024-04-22T18:39:00.715 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 696320 length: 527872 2024-04-22T18:39:00.715 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-22T18:39:00.715 INFO:tasks.rados.rados.0.smithi089.stdout:373: finishing set_chunk to oid 24 2024-04-22T18:39:00.715 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 2076160 length: 134656 2024-04-22T18:39:00.715 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-22T18:39:00.715 INFO:tasks.rados.rados.0.smithi089.stdout:373: done (15 left) 2024-04-22T18:39:00.715 INFO:tasks.rados.rados.0.smithi089.stdout:382: done (14 left) 2024-04-22T18:39:00.715 INFO:tasks.rados.rados.0.smithi089.stdout:387: set_chunk oid 38 target oid 38 low_tier 2024-04-22T18:39:00.715 INFO:tasks.rados.rados.0.smithi089.stdout:387: set_chunk oid 38 offset: 232960 length: 1852928 target oid smithi08937929-38 low_tier offset: 232960 snap -1 2024-04-22T18:39:00.715 INFO:tasks.rados.rados.0.smithi089.stdout:388: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.715 INFO:tasks.rados.rados.0.smithi089.stdout:388: set_chunk oid 39 offset: 230400 length: 1939968 target oid smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 230400 snap -1 2024-04-22T18:39:00.716 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.716 INFO:tasks.rados.rados.0.smithi089.stdout:386: finishing set_chunk to oid 37 2024-04-22T18:39:00.716 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 576000 length: 124928 2024-04-22T18:39:00.716 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-22T18:39:00.716 INFO:tasks.rados.rados.0.smithi089.stdout:384: finishing set_chunk to oid 35 2024-04-22T18:39:00.716 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 393216 length: 296960 2024-04-22T18:39:00.716 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-22T18:39:00.716 INFO:tasks.rados.rados.0.smithi089.stdout:384: done (15 left) 2024-04-22T18:39:00.716 INFO:tasks.rados.rados.0.smithi089.stdout:386: done (14 left) 2024-04-22T18:39:00.716 INFO:tasks.rados.rados.0.smithi089.stdout:389: set_chunk oid 40 target oid 40 low_tier 2024-04-22T18:39:00.716 INFO:tasks.rados.rados.0.smithi089.stdout:389: set_chunk oid 40 offset: 129536 length: 27136 target oid smithi08937929-40 low_tier offset: 129536 snap -1 2024-04-22T18:39:00.716 INFO:tasks.rados.rados.0.smithi089.stdout:390: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:00.716 INFO:tasks.rados.rados.0.smithi089.stdout:390: set_chunk oid 41 offset: 2404352 length: 150016 target oid smithi08937929-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2404352 snap -1 2024-04-22T18:39:00.716 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout:387: finishing set_chunk to oid 38 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 232960 length: 1852928 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout:387: done (15 left) 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout:391: set_chunk oid 42 target oid 42 low_tier 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout:391: set_chunk oid 42 offset: 850944 length: 1150976 target oid smithi08937929-42 low_tier offset: 850944 snap -1 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout:381: finishing set_chunk to oid 32 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 1372160 length: 336384 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout:381: done (15 left) 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout:391 retry set_chunk ! 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 15 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout:358: finishing set_chunk to oid 9 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 9 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-22T18:39:00.717 INFO:tasks.rados.rados.0.smithi089.stdout:358: done (14 left) 2024-04-22T18:39:00.718 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 14 2024-04-22T18:39:00.718 INFO:tasks.rados.rados.0.smithi089.stdout:391: finishing set_chunk to oid 42 2024-04-22T18:39:00.718 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 850944 length: 1150976 2024-04-22T18:39:00.718 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-22T18:39:00.718 INFO:tasks.rados.rados.0.smithi089.stdout:391: done (13 left) 2024-04-22T18:39:00.718 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 13 2024-04-22T18:39:00.718 INFO:tasks.rados.rados.0.smithi089.stdout:344: finishing set_chunk to oid 45 2024-04-22T18:39:00.718 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-22T18:39:00.719 INFO:tasks.rados.rados.0.smithi089.stdout:344: done (12 left) 2024-04-22T18:39:00.719 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 12 2024-04-22T18:39:00.719 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:00.741 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:00.742+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:00.811 INFO:tasks.rados.rados.0.smithi089.stdout:375: finishing set_chunk to oid 26 2024-04-22T18:39:00.811 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 500736 length: 906752 2024-04-22T18:39:00.811 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-22T18:39:00.811 INFO:tasks.rados.rados.0.smithi089.stdout:385: finishing set_chunk to oid 36 2024-04-22T18:39:00.811 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 385024 length: 697856 2024-04-22T18:39:00.811 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-22T18:39:00.811 INFO:tasks.rados.rados.0.smithi089.stdout:375: done (11 left) 2024-04-22T18:39:00.811 INFO:tasks.rados.rados.0.smithi089.stdout:385: done (10 left) 2024-04-22T18:39:00.812 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 10 2024-04-22T18:39:00.812 INFO:tasks.rados.rados.0.smithi089.stdout:376: finishing set_chunk to oid 27 2024-04-22T18:39:00.812 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 635904 length: 17920 2024-04-22T18:39:00.812 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-22T18:39:00.812 INFO:tasks.rados.rados.0.smithi089.stdout:376: done (9 left) 2024-04-22T18:39:00.812 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 9 2024-04-22T18:39:00.812 INFO:tasks.rados.rados.0.smithi089.stdout:342: finishing set_chunk to oid 43 2024-04-22T18:39:00.812 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-22T18:39:00.812 INFO:tasks.rados.rados.0.smithi089.stdout:342: done (8 left) 2024-04-22T18:39:00.812 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 8 2024-04-22T18:39:00.813 INFO:tasks.rados.rados.0.smithi089.stdout:359: finishing set_chunk to oid 10 2024-04-22T18:39:00.813 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-22T18:39:00.813 INFO:tasks.rados.rados.0.smithi089.stdout:359: done (7 left) 2024-04-22T18:39:00.814 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 7 2024-04-22T18:39:00.814 INFO:tasks.rados.rados.0.smithi089.stdout:367: finishing set_chunk to oid 18 2024-04-22T18:39:00.814 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 9 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-22T18:39:00.814 INFO:tasks.rados.rados.0.smithi089.stdout:367: done (6 left) 2024-04-22T18:39:00.814 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 6 2024-04-22T18:39:00.814 INFO:tasks.rados.rados.0.smithi089.stdout:389: finishing set_chunk to oid 40 2024-04-22T18:39:00.814 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 3 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-22T18:39:00.814 INFO:tasks.rados.rados.0.smithi089.stdout:389: done (5 left) 2024-04-22T18:39:00.814 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 5 2024-04-22T18:39:00.815 INFO:tasks.rados.rados.0.smithi089.stdout:362: finishing set_chunk to oid 13 2024-04-22T18:39:00.815 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-22T18:39:00.815 INFO:tasks.rados.rados.0.smithi089.stdout:362: done (4 left) 2024-04-22T18:39:00.815 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 4 2024-04-22T18:39:00.842 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:00.843+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:00.862 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:00.862 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:00.871 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:00.911 INFO:tasks.rados.rados.0.smithi089.stdout:383: finishing set_chunk to oid 34 2024-04-22T18:39:00.911 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 467456 length: 1874944 2024-04-22T18:39:00.911 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-22T18:39:00.911 INFO:tasks.rados.rados.0.smithi089.stdout:383: done (3 left) 2024-04-22T18:39:00.911 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 3 2024-04-22T18:39:00.911 INFO:tasks.rados.rados.0.smithi089.stdout:390: finishing set_chunk to oid 41 2024-04-22T18:39:00.911 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2404352 length: 150016 2024-04-22T18:39:00.911 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-22T18:39:00.911 INFO:tasks.rados.rados.0.smithi089.stdout:388: finishing set_chunk to oid 39 2024-04-22T18:39:00.912 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 230400 length: 1939968 2024-04-22T18:39:00.912 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-22T18:39:00.912 INFO:tasks.rados.rados.0.smithi089.stdout:388: done (2 left) 2024-04-22T18:39:00.912 INFO:tasks.rados.rados.0.smithi089.stdout:390: done (1 left) 2024-04-22T18:39:00.912 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 1 2024-04-22T18:39:00.943 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:00.944+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:00.972 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:00.973 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:00.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:01.043 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:01.044+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:01.096 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:01.096 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:01.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:01.120 INFO:tasks.rados.rados.0.smithi089.stdout:377: finishing set_chunk to oid 28 2024-04-22T18:39:01.120 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 15 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-22T18:39:01.120 INFO:tasks.rados.rados.0.smithi089.stdout:377: done (0 left) 2024-04-22T18:39:01.120 INFO:tasks.rados.rados.0.smithi089.stdout:392: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.120 INFO:tasks.rados.rados.0.smithi089.stdout:392: set_chunk oid 43 offset: 1084928 length: 946176 target oid smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1084928 snap -1 2024-04-22T18:39:01.120 INFO:tasks.rados.rados.0.smithi089.stdout:393: set_chunk oid 44 target oid 44 low_tier 2024-04-22T18:39:01.120 INFO:tasks.rados.rados.0.smithi089.stdout:393: set_chunk oid 44 offset: 1202176 length: 1363968 target oid smithi08937929-44 low_tier offset: 1202176 snap -1 2024-04-22T18:39:01.120 INFO:tasks.rados.rados.0.smithi089.stdout:394: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.120 INFO:tasks.rados.rados.0.smithi089.stdout:394: set_chunk oid 45 offset: 975872 length: 606208 target oid smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 975872 snap -1 2024-04-22T18:39:01.120 INFO:tasks.rados.rados.0.smithi089.stdout:395: set_chunk oid 46 target oid 46 low_tier 2024-04-22T18:39:01.121 INFO:tasks.rados.rados.0.smithi089.stdout:395: set_chunk oid 46 offset: 467456 length: 2064384 target oid smithi08937929-46 low_tier offset: 467456 snap -1 2024-04-22T18:39:01.121 INFO:tasks.rados.rados.0.smithi089.stdout:396: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.121 INFO:tasks.rados.rados.0.smithi089.stdout:396: set_chunk oid 47 offset: 2350080 length: 86016 target oid smithi08937929-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2350080 snap -1 2024-04-22T18:39:01.121 INFO:tasks.rados.rados.0.smithi089.stdout:397: set_chunk oid 48 target oid 48 low_tier 2024-04-22T18:39:01.121 INFO:tasks.rados.rados.0.smithi089.stdout:397: set_chunk oid 48 offset: 992256 length: 1429504 target oid smithi08937929-48 low_tier offset: 992256 snap -1 2024-04-22T18:39:01.121 INFO:tasks.rados.rados.0.smithi089.stdout:398: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.121 INFO:tasks.rados.rados.0.smithi089.stdout:398: set_chunk oid 49 offset: 883712 length: 884224 target oid smithi08937929-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 883712 snap -1 2024-04-22T18:39:01.121 INFO:tasks.rados.rados.0.smithi089.stdout:399: set_chunk oid 50 target oid 50 low_tier 2024-04-22T18:39:01.121 INFO:tasks.rados.rados.0.smithi089.stdout:399: set_chunk oid 50 offset: 346112 length: 1980928 target oid smithi08937929-50 low_tier offset: 346112 snap -1 2024-04-22T18:39:01.121 INFO:tasks.rados.rados.0.smithi089.stdout:400: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.121 INFO:tasks.rados.rados.0.smithi089.stdout:400: set_chunk oid 1 offset: 466944 length: 947200 target oid smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 466944 snap -1 2024-04-22T18:39:01.121 INFO:tasks.rados.rados.0.smithi089.stdout:401: set_chunk oid 2 target oid 2 low_tier 2024-04-22T18:39:01.121 INFO:tasks.rados.rados.0.smithi089.stdout:401: set_chunk oid 2 offset: 537088 length: 1444864 target oid smithi08937929-2 low_tier offset: 537088 snap -1 2024-04-22T18:39:01.121 INFO:tasks.rados.rados.0.smithi089.stdout:402: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.121 INFO:tasks.rados.rados.0.smithi089.stdout:402: set_chunk oid 3 offset: 516096 length: 913920 target oid smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 516096 snap -1 2024-04-22T18:39:01.122 INFO:tasks.rados.rados.0.smithi089.stdout:403: set_chunk oid 4 target oid 4 low_tier 2024-04-22T18:39:01.122 INFO:tasks.rados.rados.0.smithi089.stdout:403: set_chunk oid 4 offset: 908800 length: 437248 target oid smithi08937929-4 low_tier offset: 908800 snap -1 2024-04-22T18:39:01.122 INFO:tasks.rados.rados.0.smithi089.stdout:404: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.122 INFO:tasks.rados.rados.0.smithi089.stdout:404: set_chunk oid 5 offset: 890368 length: 857600 target oid smithi08937929-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 890368 snap -1 2024-04-22T18:39:01.122 INFO:tasks.rados.rados.0.smithi089.stdout:405: set_chunk oid 6 target oid 6 low_tier 2024-04-22T18:39:01.122 INFO:tasks.rados.rados.0.smithi089.stdout:405: set_chunk oid 6 offset: 511488 length: 1139712 target oid smithi08937929-6 low_tier offset: 511488 snap -1 2024-04-22T18:39:01.122 INFO:tasks.rados.rados.0.smithi089.stdout:406: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.122 INFO:tasks.rados.rados.0.smithi089.stdout:406: set_chunk oid 7 offset: 1522688 length: 484352 target oid smithi08937929-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1522688 snap -1 2024-04-22T18:39:01.122 INFO:tasks.rados.rados.0.smithi089.stdout:407: set_chunk oid 8 target oid 8 low_tier 2024-04-22T18:39:01.122 INFO:tasks.rados.rados.0.smithi089.stdout:407: set_chunk oid 8 offset: 701952 length: 1253888 target oid smithi08937929-8 low_tier offset: 701952 snap -1 2024-04-22T18:39:01.122 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.122 INFO:tasks.rados.rados.0.smithi089.stdout:393: finishing set_chunk to oid 44 2024-04-22T18:39:01.122 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 1202176 length: 1363968 2024-04-22T18:39:01.122 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-22T18:39:01.123 INFO:tasks.rados.rados.0.smithi089.stdout:394: finishing set_chunk to oid 45 2024-04-22T18:39:01.123 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 975872 length: 606208 2024-04-22T18:39:01.123 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-22T18:39:01.123 INFO:tasks.rados.rados.0.smithi089.stdout:393: done (15 left) 2024-04-22T18:39:01.123 INFO:tasks.rados.rados.0.smithi089.stdout:394: done (14 left) 2024-04-22T18:39:01.123 INFO:tasks.rados.rados.0.smithi089.stdout:408: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.123 INFO:tasks.rados.rados.0.smithi089.stdout:408: set_chunk oid 9 offset: 93696 length: 264192 target oid smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 93696 snap -1 2024-04-22T18:39:01.123 INFO:tasks.rados.rados.0.smithi089.stdout:409: set_chunk oid 10 target oid 10 low_tier 2024-04-22T18:39:01.123 INFO:tasks.rados.rados.0.smithi089.stdout:409: set_chunk oid 10 offset: 524288 length: 1492480 target oid smithi08937929-10 low_tier offset: 524288 snap -1 2024-04-22T18:39:01.123 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.123 INFO:tasks.rados.rados.0.smithi089.stdout:392: finishing set_chunk to oid 43 2024-04-22T18:39:01.123 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1084928 length: 946176 2024-04-22T18:39:01.123 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-22T18:39:01.123 INFO:tasks.rados.rados.0.smithi089.stdout:396: finishing set_chunk to oid 47 2024-04-22T18:39:01.123 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2350080 length: 86016 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout:392: done (15 left) 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout:396: done (14 left) 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout:410: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout:410: set_chunk oid 11 offset: 1144320 length: 35840 target oid smithi08937929-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1144320 snap -1 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout:411: set_chunk oid 12 target oid 12 low_tier 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout:411: set_chunk oid 12 offset: 1188864 length: 264192 target oid smithi08937929-12 low_tier offset: 1188864 snap -1 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout:398: finishing set_chunk to oid 49 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 883712 length: 884224 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout:400: finishing set_chunk to oid 1 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 466944 length: 947200 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout:397: finishing set_chunk to oid 48 2024-04-22T18:39:01.124 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 992256 length: 1429504 2024-04-22T18:39:01.125 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-22T18:39:01.125 INFO:tasks.rados.rados.0.smithi089.stdout:405: finishing set_chunk to oid 6 2024-04-22T18:39:01.125 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 511488 length: 1139712 2024-04-22T18:39:01.125 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-22T18:39:01.125 INFO:tasks.rados.rados.0.smithi089.stdout:397: done (15 left) 2024-04-22T18:39:01.125 INFO:tasks.rados.rados.0.smithi089.stdout:398: done (14 left) 2024-04-22T18:39:01.125 INFO:tasks.rados.rados.0.smithi089.stdout:400: done (13 left) 2024-04-22T18:39:01.125 INFO:tasks.rados.rados.0.smithi089.stdout:405: done (12 left) 2024-04-22T18:39:01.125 INFO:tasks.rados.rados.0.smithi089.stdout:412: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.125 INFO:tasks.rados.rados.0.smithi089.stdout:412: set_chunk oid 13 offset: 501248 length: 490496 target oid smithi08937929-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 501248 snap -1 2024-04-22T18:39:01.125 INFO:tasks.rados.rados.0.smithi089.stdout:413: set_chunk oid 14 target oid 14 low_tier 2024-04-22T18:39:01.125 INFO:tasks.rados.rados.0.smithi089.stdout:413: set_chunk oid 14 offset: 626688 length: 1476608 target oid smithi08937929-14 low_tier offset: 626688 snap -1 2024-04-22T18:39:01.125 INFO:tasks.rados.rados.0.smithi089.stdout:414: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.125 INFO:tasks.rados.rados.0.smithi089.stdout:414: set_chunk oid 15 offset: 722432 length: 1417216 target oid smithi08937929-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 722432 snap -1 2024-04-22T18:39:01.126 INFO:tasks.rados.rados.0.smithi089.stdout:415: set_chunk oid 16 target oid 16 low_tier 2024-04-22T18:39:01.126 INFO:tasks.rados.rados.0.smithi089.stdout:415: set_chunk oid 16 offset: 580608 length: 504832 target oid smithi08937929-16 low_tier offset: 580608 snap -1 2024-04-22T18:39:01.126 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.126 INFO:tasks.rados.rados.0.smithi089.stdout:409: finishing set_chunk to oid 10 2024-04-22T18:39:01.126 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 524288 length: 1492480 2024-04-22T18:39:01.126 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-22T18:39:01.126 INFO:tasks.rados.rados.0.smithi089.stdout:409: done (15 left) 2024-04-22T18:39:01.126 INFO:tasks.rados.rados.0.smithi089.stdout:416: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.126 INFO:tasks.rados.rados.0.smithi089.stdout:416: set_chunk oid 17 offset: 1095168 length: 98816 target oid smithi08937929-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1095168 snap -1 2024-04-22T18:39:01.126 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.126 INFO:tasks.rados.rados.0.smithi089.stdout:413: finishing set_chunk to oid 14 2024-04-22T18:39:01.126 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 626688 length: 1476608 2024-04-22T18:39:01.126 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-22T18:39:01.126 INFO:tasks.rados.rados.0.smithi089.stdout:413: done (15 left) 2024-04-22T18:39:01.126 INFO:tasks.rados.rados.0.smithi089.stdout:417: set_chunk oid 18 target oid 18 low_tier 2024-04-22T18:39:01.127 INFO:tasks.rados.rados.0.smithi089.stdout:417: set_chunk oid 18 offset: 585216 length: 28672 target oid smithi08937929-18 low_tier offset: 585216 snap -1 2024-04-22T18:39:01.127 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.127 INFO:tasks.rados.rados.0.smithi089.stdout:412: finishing set_chunk to oid 13 2024-04-22T18:39:01.127 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 501248 length: 490496 2024-04-22T18:39:01.127 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-22T18:39:01.127 INFO:tasks.rados.rados.0.smithi089.stdout:410: finishing set_chunk to oid 11 2024-04-22T18:39:01.127 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1144320 length: 35840 2024-04-22T18:39:01.127 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-22T18:39:01.127 INFO:tasks.rados.rados.0.smithi089.stdout:410: done (15 left) 2024-04-22T18:39:01.127 INFO:tasks.rados.rados.0.smithi089.stdout:412: done (14 left) 2024-04-22T18:39:01.127 INFO:tasks.rados.rados.0.smithi089.stdout:418: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.127 INFO:tasks.rados.rados.0.smithi089.stdout:418: set_chunk oid 19 offset: 26624 length: 2191360 target oid smithi08937929-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 26624 snap -1 2024-04-22T18:39:01.127 INFO:tasks.rados.rados.0.smithi089.stdout:419: set_chunk oid 20 target oid 20 low_tier 2024-04-22T18:39:01.128 INFO:tasks.rados.rados.0.smithi089.stdout:419: set_chunk oid 20 offset: 358400 length: 1290752 target oid smithi08937929-20 low_tier offset: 358400 snap -1 2024-04-22T18:39:01.128 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.128 INFO:tasks.rados.rados.0.smithi089.stdout:414: finishing set_chunk to oid 15 2024-04-22T18:39:01.128 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 722432 length: 1417216 2024-04-22T18:39:01.128 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-22T18:39:01.128 INFO:tasks.rados.rados.0.smithi089.stdout:414: done (15 left) 2024-04-22T18:39:01.128 INFO:tasks.rados.rados.0.smithi089.stdout:420: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.128 INFO:tasks.rados.rados.0.smithi089.stdout:420: set_chunk oid 21 offset: 1296896 length: 666624 target oid smithi08937929-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1296896 snap -1 2024-04-22T18:39:01.128 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.128 INFO:tasks.rados.rados.0.smithi089.stdout:419: finishing set_chunk to oid 20 2024-04-22T18:39:01.128 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 358400 length: 1290752 2024-04-22T18:39:01.128 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-22T18:39:01.128 INFO:tasks.rados.rados.0.smithi089.stdout:419: done (15 left) 2024-04-22T18:39:01.128 INFO:tasks.rados.rados.0.smithi089.stdout:421: set_chunk oid 22 target oid 22 low_tier 2024-04-22T18:39:01.128 INFO:tasks.rados.rados.0.smithi089.stdout:421: set_chunk oid 22 offset: 508928 length: 280576 target oid smithi08937929-22 low_tier offset: 508928 snap -1 2024-04-22T18:39:01.129 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.129 INFO:tasks.rados.rados.0.smithi089.stdout:416: finishing set_chunk to oid 17 2024-04-22T18:39:01.129 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-22T18:39:01.129 INFO:tasks.rados.rados.0.smithi089.stdout:416: done (15 left) 2024-04-22T18:39:01.129 INFO:tasks.rados.rados.0.smithi089.stdout:422: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.129 INFO:tasks.rados.rados.0.smithi089.stdout:422: set_chunk oid 23 offset: 885248 length: 96768 target oid smithi08937929-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 885248 snap -1 2024-04-22T18:39:01.129 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.141 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:39:01.148 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:01.149+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:01.207 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:01.207 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:01.218 INFO:tasks.rados.rados.0.smithi089.stdout:401: finishing set_chunk to oid 2 2024-04-22T18:39:01.218 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 537088 length: 1444864 2024-04-22T18:39:01.218 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-22T18:39:01.218 INFO:tasks.rados.rados.0.smithi089.stdout:401: done (15 left) 2024-04-22T18:39:01.218 INFO:tasks.rados.rados.0.smithi089.stdout:423: set_chunk oid 24 target oid 24 low_tier 2024-04-22T18:39:01.218 INFO:tasks.rados.rados.0.smithi089.stdout:423: set_chunk oid 24 offset: 897024 length: 543744 target oid smithi08937929-24 low_tier offset: 897024 snap -1 2024-04-22T18:39:01.218 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.218 INFO:tasks.rados.rados.0.smithi089.stdout:402: finishing set_chunk to oid 3 2024-04-22T18:39:01.218 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 516096 length: 913920 2024-04-22T18:39:01.218 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-22T18:39:01.219 INFO:tasks.rados.rados.0.smithi089.stdout:395: finishing set_chunk to oid 46 2024-04-22T18:39:01.219 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 467456 length: 2064384 2024-04-22T18:39:01.219 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-22T18:39:01.219 INFO:tasks.rados.rados.0.smithi089.stdout:399: finishing set_chunk to oid 50 2024-04-22T18:39:01.219 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 346112 length: 1980928 2024-04-22T18:39:01.219 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-22T18:39:01.219 INFO:tasks.rados.rados.0.smithi089.stdout:403: finishing set_chunk to oid 4 2024-04-22T18:39:01.219 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 908800 length: 437248 2024-04-22T18:39:01.219 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-22T18:39:01.219 INFO:tasks.rados.rados.0.smithi089.stdout:404: finishing set_chunk to oid 5 2024-04-22T18:39:01.219 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 890368 length: 857600 2024-04-22T18:39:01.219 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-22T18:39:01.219 INFO:tasks.rados.rados.0.smithi089.stdout:407: finishing set_chunk to oid 8 2024-04-22T18:39:01.219 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 701952 length: 1253888 2024-04-22T18:39:01.219 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-22T18:39:01.220 INFO:tasks.rados.rados.0.smithi089.stdout:395: done (15 left) 2024-04-22T18:39:01.220 INFO:tasks.rados.rados.0.smithi089.stdout:399: done (14 left) 2024-04-22T18:39:01.220 INFO:tasks.rados.rados.0.smithi089.stdout:402: done (13 left) 2024-04-22T18:39:01.220 INFO:tasks.rados.rados.0.smithi089.stdout:403: done (12 left) 2024-04-22T18:39:01.220 INFO:tasks.rados.rados.0.smithi089.stdout:404: done (11 left) 2024-04-22T18:39:01.220 INFO:tasks.rados.rados.0.smithi089.stdout:407: done (10 left) 2024-04-22T18:39:01.220 INFO:tasks.rados.rados.0.smithi089.stdout:424: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.220 INFO:tasks.rados.rados.0.smithi089.stdout:424: set_chunk oid 25 offset: 504832 length: 204800 target oid smithi08937929-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 504832 snap -1 2024-04-22T18:39:01.220 INFO:tasks.rados.rados.0.smithi089.stdout:425: set_chunk oid 26 target oid 26 low_tier 2024-04-22T18:39:01.220 INFO:tasks.rados.rados.0.smithi089.stdout:425: set_chunk oid 26 offset: 9216 length: 1729536 target oid smithi08937929-26 low_tier offset: 9216 snap -1 2024-04-22T18:39:01.220 INFO:tasks.rados.rados.0.smithi089.stdout:426: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.220 INFO:tasks.rados.rados.0.smithi089.stdout:426: set_chunk oid 27 offset: 320000 length: 23040 target oid smithi08937929-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 320000 snap -1 2024-04-22T18:39:01.220 INFO:tasks.rados.rados.0.smithi089.stdout:427: set_chunk oid 28 target oid 28 low_tier 2024-04-22T18:39:01.220 INFO:tasks.rados.rados.0.smithi089.stdout:427: set_chunk oid 28 offset: 918528 length: 58368 target oid smithi08937929-28 low_tier offset: 918528 snap -1 2024-04-22T18:39:01.221 INFO:tasks.rados.rados.0.smithi089.stdout:428: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.221 INFO:tasks.rados.rados.0.smithi089.stdout:428: set_chunk oid 29 offset: 981504 length: 1398784 target oid smithi08937929-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 981504 snap -1 2024-04-22T18:39:01.221 INFO:tasks.rados.rados.0.smithi089.stdout:429: set_chunk oid 30 target oid 30 low_tier 2024-04-22T18:39:01.221 INFO:tasks.rados.rados.0.smithi089.stdout:429: set_chunk oid 30 offset: 96768 length: 1764864 target oid smithi08937929-30 low_tier offset: 96768 snap -1 2024-04-22T18:39:01.221 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.221 INFO:tasks.rados.rados.0.smithi089.stdout:411: finishing set_chunk to oid 12 2024-04-22T18:39:01.221 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1188864 length: 264192 2024-04-22T18:39:01.221 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-22T18:39:01.221 INFO:tasks.rados.rados.0.smithi089.stdout:418: finishing set_chunk to oid 19 2024-04-22T18:39:01.221 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 26624 length: 2191360 2024-04-22T18:39:01.221 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-22T18:39:01.221 INFO:tasks.rados.rados.0.smithi089.stdout:408: finishing set_chunk to oid 9 2024-04-22T18:39:01.221 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 93696 length: 264192 2024-04-22T18:39:01.221 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-22T18:39:01.221 INFO:tasks.rados.rados.0.smithi089.stdout:408: done (15 left) 2024-04-22T18:39:01.222 INFO:tasks.rados.rados.0.smithi089.stdout:411: done (14 left) 2024-04-22T18:39:01.222 INFO:tasks.rados.rados.0.smithi089.stdout:418: done (13 left) 2024-04-22T18:39:01.222 INFO:tasks.rados.rados.0.smithi089.stdout:430: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.222 INFO:tasks.rados.rados.0.smithi089.stdout:430: set_chunk oid 31 offset: 114176 length: 555008 target oid smithi08937929-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 114176 snap -1 2024-04-22T18:39:01.222 INFO:tasks.rados.rados.0.smithi089.stdout:431: set_chunk oid 32 target oid 32 low_tier 2024-04-22T18:39:01.222 INFO:tasks.rados.rados.0.smithi089.stdout:431: set_chunk oid 32 offset: 124928 length: 2165248 target oid smithi08937929-32 low_tier offset: 124928 snap -1 2024-04-22T18:39:01.222 INFO:tasks.rados.rados.0.smithi089.stdout:432: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.222 INFO:tasks.rados.rados.0.smithi089.stdout:432: set_chunk oid 33 offset: 494592 length: 915456 target oid smithi08937929-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 494592 snap -1 2024-04-22T18:39:01.222 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.222 INFO:tasks.rados.rados.0.smithi089.stdout:415: finishing set_chunk to oid 16 2024-04-22T18:39:01.222 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 580608 length: 504832 2024-04-22T18:39:01.222 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-22T18:39:01.222 INFO:tasks.rados.rados.0.smithi089.stdout:415: done (15 left) 2024-04-22T18:39:01.222 INFO:tasks.rados.rados.0.smithi089.stdout:433: set_chunk oid 34 target oid 34 low_tier 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout:433: set_chunk oid 34 offset: 619520 length: 708608 target oid smithi08937929-34 low_tier offset: 619520 snap -1 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout:430: finishing set_chunk to oid 31 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 114176 length: 555008 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout:430: done (15 left) 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout:434: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout:434: set_chunk oid 35 offset: 1147392 length: 1330176 target oid smithi08937929-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1147392 snap -1 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout:432: finishing set_chunk to oid 33 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 494592 length: 915456 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout:432: done (15 left) 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout:435: set_chunk oid 36 target oid 36 low_tier 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout:435: set_chunk oid 36 offset: 101376 length: 2413568 target oid smithi08937929-36 low_tier offset: 101376 snap -1 2024-04-22T18:39:01.223 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.224 INFO:tasks.rados.rados.0.smithi089.stdout:435: finishing set_chunk to oid 36 2024-04-22T18:39:01.224 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 101376 length: 2413568 2024-04-22T18:39:01.224 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-22T18:39:01.224 INFO:tasks.rados.rados.0.smithi089.stdout:435: done (15 left) 2024-04-22T18:39:01.224 INFO:tasks.rados.rados.0.smithi089.stdout:436: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.224 INFO:tasks.rados.rados.0.smithi089.stdout:436: set_chunk oid 37 offset: 2405888 length: 182784 target oid smithi08937929-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2405888 snap -1 2024-04-22T18:39:01.224 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:01.266 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:01.267+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:01.317 INFO:tasks.rados.rados.0.smithi089.stdout:422: finishing set_chunk to oid 23 2024-04-22T18:39:01.318 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 885248 length: 96768 2024-04-22T18:39:01.318 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-22T18:39:01.318 INFO:tasks.rados.rados.0.smithi089.stdout:423: finishing set_chunk to oid 24 2024-04-22T18:39:01.318 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 897024 length: 543744 2024-04-22T18:39:01.318 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-22T18:39:01.318 INFO:tasks.rados.rados.0.smithi089.stdout:425: finishing set_chunk to oid 26 2024-04-22T18:39:01.318 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 9216 length: 1729536 2024-04-22T18:39:01.318 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-22T18:39:01.318 INFO:tasks.rados.rados.0.smithi089.stdout:406: finishing set_chunk to oid 7 2024-04-22T18:39:01.318 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1522688 length: 484352 2024-04-22T18:39:01.318 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-22T18:39:01.318 INFO:tasks.rados.rados.0.smithi089.stdout:434: finishing set_chunk to oid 35 2024-04-22T18:39:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1147392 length: 1330176 2024-04-22T18:39:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-22T18:39:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:429: finishing set_chunk to oid 30 2024-04-22T18:39:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 96768 length: 1764864 2024-04-22T18:39:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-22T18:39:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:406: done (15 left) 2024-04-22T18:39:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:422: done (14 left) 2024-04-22T18:39:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:423: done (13 left) 2024-04-22T18:39:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:425: done (12 left) 2024-04-22T18:39:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:429: done (11 left) 2024-04-22T18:39:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:434: done (10 left) 2024-04-22T18:39:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:437: set_chunk oid 38 target oid 38 low_tier 2024-04-22T18:39:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:428: finishing set_chunk to oid 29 2024-04-22T18:39:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 981504 length: 1398784 2024-04-22T18:39:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-22T18:39:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:424: finishing set_chunk to oid 25 2024-04-22T18:39:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 504832 length: 204800 2024-04-22T18:39:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-22T18:39:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:437: set_chunk oid 38 offset: 2305536 length: 238592 target oid smithi08937929-38 low_tier offset: 2305536 snap -1 2024-04-22T18:39:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:424: done (10 left) 2024-04-22T18:39:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:428: done (9 left) 2024-04-22T18:39:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:438: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:438: set_chunk oid 39 offset: 708608 length: 1688064 target oid smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 708608 snap -1 2024-04-22T18:39:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:439: set_chunk oid 40 target oid 40 low_tier 2024-04-22T18:39:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:439: set_chunk oid 40 offset: 1896448 length: 653824 target oid smithi08937929-40 low_tier offset: 1896448 snap -1 2024-04-22T18:39:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:440: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:440: set_chunk oid 41 offset: 367616 length: 1289728 target oid smithi08937929-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 367616 snap -1 2024-04-22T18:39:01.321 INFO:tasks.rados.rados.0.smithi089.stdout:441: set_chunk oid 42 target oid 42 low_tier 2024-04-22T18:39:01.321 INFO:tasks.rados.rados.0.smithi089.stdout:441: set_chunk oid 42 offset: 254464 length: 1257472 target oid smithi08937929-42 low_tier offset: 254464 snap -1 2024-04-22T18:39:01.321 INFO:tasks.rados.rados.0.smithi089.stdout:442: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.321 INFO:tasks.rados.rados.0.smithi089.stdout:442: set_chunk oid 43 offset: 315392 length: 1780224 target oid smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 315392 snap -1 2024-04-22T18:39:01.321 INFO:tasks.rados.rados.0.smithi089.stdout:443: set_chunk oid 44 target oid 44 low_tier 2024-04-22T18:39:01.321 INFO:tasks.rados.rados.0.smithi089.stdout:443: set_chunk oid 44 offset: 314368 length: 2057216 target oid smithi08937929-44 low_tier offset: 314368 snap -1 2024-04-22T18:39:01.321 INFO:tasks.rados.rados.0.smithi089.stdout:444: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.321 INFO:tasks.rados.rados.0.smithi089.stdout:444: set_chunk oid 45 offset: 479744 length: 1887232 target oid smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 479744 snap -1 2024-04-22T18:39:01.321 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.321 INFO:tasks.rados.rados.0.smithi089.stdout:439: finishing set_chunk to oid 40 2024-04-22T18:39:01.321 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 1896448 length: 653824 2024-04-22T18:39:01.321 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-22T18:39:01.321 INFO:tasks.rados.rados.0.smithi089.stdout:441: finishing set_chunk to oid 42 2024-04-22T18:39:01.321 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 254464 length: 1257472 2024-04-22T18:39:01.322 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-22T18:39:01.322 INFO:tasks.rados.rados.0.smithi089.stdout:439: done (15 left) 2024-04-22T18:39:01.322 INFO:tasks.rados.rados.0.smithi089.stdout:441: done (14 left) 2024-04-22T18:39:01.322 INFO:tasks.rados.rados.0.smithi089.stdout:445: set_chunk oid 46 target oid 46 low_tier 2024-04-22T18:39:01.322 INFO:tasks.rados.rados.0.smithi089.stdout:445: set_chunk oid 46 offset: 104448 length: 1995776 target oid smithi08937929-46 low_tier offset: 104448 snap -1 2024-04-22T18:39:01.322 INFO:tasks.rados.rados.0.smithi089.stdout:446: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.322 INFO:tasks.rados.rados.0.smithi089.stdout:446: set_chunk oid 47 offset: 663040 length: 1270784 target oid smithi08937929-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 663040 snap -1 2024-04-22T18:39:01.322 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.322 INFO:tasks.rados.rados.0.smithi089.stdout:431: finishing set_chunk to oid 32 2024-04-22T18:39:01.322 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 124928 length: 2165248 2024-04-22T18:39:01.322 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-22T18:39:01.322 INFO:tasks.rados.rados.0.smithi089.stdout:431: done (15 left) 2024-04-22T18:39:01.322 INFO:tasks.rados.rados.0.smithi089.stdout:447: set_chunk oid 48 target oid 48 low_tier 2024-04-22T18:39:01.323 INFO:tasks.rados.rados.0.smithi089.stdout:447: set_chunk oid 48 offset: 985600 length: 763392 target oid smithi08937929-48 low_tier offset: 985600 snap -1 2024-04-22T18:39:01.323 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.323 INFO:tasks.rados.rados.0.smithi089.stdout:446: finishing set_chunk to oid 47 2024-04-22T18:39:01.323 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 663040 length: 1270784 2024-04-22T18:39:01.323 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-22T18:39:01.323 INFO:tasks.rados.rados.0.smithi089.stdout:446: done (15 left) 2024-04-22T18:39:01.323 INFO:tasks.rados.rados.0.smithi089.stdout:448: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.323 INFO:tasks.rados.rados.0.smithi089.stdout:448: set_chunk oid 49 offset: 689152 length: 860672 target oid smithi08937929-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 689152 snap -1 2024-04-22T18:39:01.323 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.323 INFO:tasks.rados.rados.0.smithi089.stdout:420: finishing set_chunk to oid 21 2024-04-22T18:39:01.323 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-22T18:39:01.323 INFO:tasks.rados.rados.0.smithi089.stdout:420: done (15 left) 2024-04-22T18:39:01.323 INFO:tasks.rados.rados.0.smithi089.stdout:449: set_chunk oid 50 target oid 50 low_tier 2024-04-22T18:39:01.323 INFO:tasks.rados.rados.0.smithi089.stdout:449: set_chunk oid 50 offset: 233984 length: 22016 target oid smithi08937929-50 low_tier offset: 233984 snap -1 2024-04-22T18:39:01.323 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.346 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:01.346+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:01.367 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:01.367 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:01.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:01.420 INFO:tasks.rados.rados.0.smithi089.stdout:433: finishing set_chunk to oid 34 2024-04-22T18:39:01.420 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 619520 length: 708608 2024-04-22T18:39:01.420 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-22T18:39:01.420 INFO:tasks.rados.rados.0.smithi089.stdout:440: finishing set_chunk to oid 41 2024-04-22T18:39:01.420 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 367616 length: 1289728 2024-04-22T18:39:01.421 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-22T18:39:01.421 INFO:tasks.rados.rados.0.smithi089.stdout:438: finishing set_chunk to oid 39 2024-04-22T18:39:01.421 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 708608 length: 1688064 2024-04-22T18:39:01.421 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-22T18:39:01.421 INFO:tasks.rados.rados.0.smithi089.stdout:433: done (15 left) 2024-04-22T18:39:01.421 INFO:tasks.rados.rados.0.smithi089.stdout:438: done (14 left) 2024-04-22T18:39:01.421 INFO:tasks.rados.rados.0.smithi089.stdout:440: done (13 left) 2024-04-22T18:39:01.421 INFO:tasks.rados.rados.0.smithi089.stdout:450: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.421 INFO:tasks.rados.rados.0.smithi089.stdout:450: set_chunk oid 1 offset: 1253376 length: 628224 target oid smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1253376 snap -1 2024-04-22T18:39:01.421 INFO:tasks.rados.rados.0.smithi089.stdout:451: set_chunk oid 2 target oid 2 low_tier 2024-04-22T18:39:01.421 INFO:tasks.rados.rados.0.smithi089.stdout:451: set_chunk oid 2 offset: 1024 length: 1159680 target oid smithi08937929-2 low_tier offset: 1024 snap -1 2024-04-22T18:39:01.421 INFO:tasks.rados.rados.0.smithi089.stdout:452: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.421 INFO:tasks.rados.rados.0.smithi089.stdout:442: finishing set_chunk to oid 43 2024-04-22T18:39:01.422 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 315392 length: 1780224 2024-04-22T18:39:01.422 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-22T18:39:01.422 INFO:tasks.rados.rados.0.smithi089.stdout:452: set_chunk oid 3 offset: 276992 length: 639488 target oid smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 276992 snap -1 2024-04-22T18:39:01.422 INFO:tasks.rados.rados.0.smithi089.stdout:442: done (15 left) 2024-04-22T18:39:01.422 INFO:tasks.rados.rados.0.smithi089.stdout:453: set_chunk oid 4 target oid 4 low_tier 2024-04-22T18:39:01.422 INFO:tasks.rados.rados.0.smithi089.stdout:453: set_chunk oid 4 offset: 1127936 length: 419840 target oid smithi08937929-4 low_tier offset: 1127936 snap -1 2024-04-22T18:39:01.422 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.422 INFO:tasks.rados.rados.0.smithi089.stdout:443: finishing set_chunk to oid 44 2024-04-22T18:39:01.422 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 314368 length: 2057216 2024-04-22T18:39:01.422 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-22T18:39:01.422 INFO:tasks.rados.rados.0.smithi089.stdout:444: finishing set_chunk to oid 45 2024-04-22T18:39:01.422 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 479744 length: 1887232 2024-04-22T18:39:01.422 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-22T18:39:01.422 INFO:tasks.rados.rados.0.smithi089.stdout:445: finishing set_chunk to oid 46 2024-04-22T18:39:01.422 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 104448 length: 1995776 2024-04-22T18:39:01.423 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-22T18:39:01.423 INFO:tasks.rados.rados.0.smithi089.stdout:443: done (15 left) 2024-04-22T18:39:01.423 INFO:tasks.rados.rados.0.smithi089.stdout:444: done (14 left) 2024-04-22T18:39:01.423 INFO:tasks.rados.rados.0.smithi089.stdout:445: done (13 left) 2024-04-22T18:39:01.423 INFO:tasks.rados.rados.0.smithi089.stdout:454: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.423 INFO:tasks.rados.rados.0.smithi089.stdout:454: set_chunk oid 5 offset: 1125376 length: 932352 target oid smithi08937929-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1125376 snap -1 2024-04-22T18:39:01.423 INFO:tasks.rados.rados.0.smithi089.stdout:455: set_chunk oid 6 target oid 6 low_tier 2024-04-22T18:39:01.423 INFO:tasks.rados.rados.0.smithi089.stdout:455: set_chunk oid 6 offset: 782848 length: 654848 target oid smithi08937929-6 low_tier offset: 782848 snap -1 2024-04-22T18:39:01.423 INFO:tasks.rados.rados.0.smithi089.stdout:456: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.423 INFO:tasks.rados.rados.0.smithi089.stdout:456: set_chunk oid 7 offset: 309760 length: 1470976 target oid smithi08937929-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 309760 snap -1 2024-04-22T18:39:01.423 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.423 INFO:tasks.rados.rados.0.smithi089.stdout:450: finishing set_chunk to oid 1 2024-04-22T18:39:01.423 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1253376 length: 628224 2024-04-22T18:39:01.423 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-22T18:39:01.424 INFO:tasks.rados.rados.0.smithi089.stdout:450: done (15 left) 2024-04-22T18:39:01.424 INFO:tasks.rados.rados.0.smithi089.stdout:457: set_chunk oid 8 target oid 8 low_tier 2024-04-22T18:39:01.424 INFO:tasks.rados.rados.0.smithi089.stdout:457: set_chunk oid 8 offset: 43008 length: 1449472 target oid smithi08937929-8 low_tier offset: 43008 snap -1 2024-04-22T18:39:01.424 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.424 INFO:tasks.rados.rados.0.smithi089.stdout:453: finishing set_chunk to oid 4 2024-04-22T18:39:01.424 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 1127936 length: 419840 2024-04-22T18:39:01.424 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-22T18:39:01.424 INFO:tasks.rados.rados.0.smithi089.stdout:453: done (15 left) 2024-04-22T18:39:01.424 INFO:tasks.rados.rados.0.smithi089.stdout:458: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.424 INFO:tasks.rados.rados.0.smithi089.stdout:458: set_chunk oid 9 offset: 495104 length: 1247232 target oid smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 495104 snap -1 2024-04-22T18:39:01.424 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.424 INFO:tasks.rados.rados.0.smithi089.stdout:456: finishing set_chunk to oid 7 2024-04-22T18:39:01.424 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 309760 length: 1470976 2024-04-22T18:39:01.424 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-22T18:39:01.424 INFO:tasks.rados.rados.0.smithi089.stdout:454: finishing set_chunk to oid 5 2024-04-22T18:39:01.425 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1125376 length: 932352 2024-04-22T18:39:01.425 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-22T18:39:01.425 INFO:tasks.rados.rados.0.smithi089.stdout:421: finishing set_chunk to oid 22 2024-04-22T18:39:01.425 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-22T18:39:01.425 INFO:tasks.rados.rados.0.smithi089.stdout:421: done (15 left) 2024-04-22T18:39:01.425 INFO:tasks.rados.rados.0.smithi089.stdout:454: done (14 left) 2024-04-22T18:39:01.425 INFO:tasks.rados.rados.0.smithi089.stdout:456: done (13 left) 2024-04-22T18:39:01.425 INFO:tasks.rados.rados.0.smithi089.stdout:459: set_chunk oid 10 target oid 10 low_tier 2024-04-22T18:39:01.425 INFO:tasks.rados.rados.0.smithi089.stdout:459: set_chunk oid 10 offset: 676864 length: 800768 target oid smithi08937929-10 low_tier offset: 676864 snap -1 2024-04-22T18:39:01.425 INFO:tasks.rados.rados.0.smithi089.stdout:460: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.425 INFO:tasks.rados.rados.0.smithi089.stdout:460: set_chunk oid 11 offset: 133632 length: 856064 target oid smithi08937929-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 133632 snap -1 2024-04-22T18:39:01.425 INFO:tasks.rados.rados.0.smithi089.stdout:461: set_chunk oid 12 target oid 12 low_tier 2024-04-22T18:39:01.425 INFO:tasks.rados.rados.0.smithi089.stdout:461: set_chunk oid 12 offset: 1144832 length: 732672 target oid smithi08937929-12 low_tier offset: 1144832 snap -1 2024-04-22T18:39:01.425 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.426 INFO:tasks.rados.rados.0.smithi089.stdout:457: finishing set_chunk to oid 8 2024-04-22T18:39:01.426 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 43008 length: 1449472 2024-04-22T18:39:01.426 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-22T18:39:01.426 INFO:tasks.rados.rados.0.smithi089.stdout:457: done (15 left) 2024-04-22T18:39:01.426 INFO:tasks.rados.rados.0.smithi089.stdout:462: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.426 INFO:tasks.rados.rados.0.smithi089.stdout:462: set_chunk oid 13 offset: 1054208 length: 593920 target oid smithi08937929-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1054208 snap -1 2024-04-22T18:39:01.426 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.426 INFO:tasks.rados.rados.0.smithi089.stdout:461: finishing set_chunk to oid 12 2024-04-22T18:39:01.426 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1144832 length: 732672 2024-04-22T18:39:01.426 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-22T18:39:01.426 INFO:tasks.rados.rados.0.smithi089.stdout:459: finishing set_chunk to oid 10 2024-04-22T18:39:01.426 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 676864 length: 800768 2024-04-22T18:39:01.426 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-22T18:39:01.426 INFO:tasks.rados.rados.0.smithi089.stdout:459: done (15 left) 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout:461: done (14 left) 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout:463: set_chunk oid 14 target oid 14 low_tier 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout:463: set_chunk oid 14 offset: 884224 length: 1264128 target oid smithi08937929-14 low_tier offset: 884224 snap -1 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout:464: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout:464: set_chunk oid 15 offset: 366592 length: 847360 target oid smithi08937929-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 366592 snap -1 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout:460: finishing set_chunk to oid 11 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 133632 length: 856064 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout:460: done (15 left) 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout:465: set_chunk oid 16 target oid 16 low_tier 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout:465: set_chunk oid 16 offset: 494592 length: 296960 target oid smithi08937929-16 low_tier offset: 494592 snap -1 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout:463: finishing set_chunk to oid 14 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 884224 length: 1264128 2024-04-22T18:39:01.427 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-22T18:39:01.428 INFO:tasks.rados.rados.0.smithi089.stdout:463: done (15 left) 2024-04-22T18:39:01.428 INFO:tasks.rados.rados.0.smithi089.stdout:466: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.428 INFO:tasks.rados.rados.0.smithi089.stdout:466: set_chunk oid 17 offset: 1106944 length: 987648 target oid smithi08937929-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1106944 snap -1 2024-04-22T18:39:01.428 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.447 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:01.448+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:01.478 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:01.478 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:01.488 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:01.519 INFO:tasks.rados.rados.0.smithi089.stdout:451: finishing set_chunk to oid 2 2024-04-22T18:39:01.519 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 1024 length: 1159680 2024-04-22T18:39:01.519 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-22T18:39:01.519 INFO:tasks.rados.rados.0.smithi089.stdout:451: done (15 left) 2024-04-22T18:39:01.519 INFO:tasks.rados.rados.0.smithi089.stdout:466 retry set_chunk ! 2024-04-22T18:39:01.519 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 15 2024-04-22T18:39:01.519 INFO:tasks.rados.rados.0.smithi089.stdout:447: finishing set_chunk to oid 48 2024-04-22T18:39:01.519 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 985600 length: 763392 2024-04-22T18:39:01.519 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-22T18:39:01.519 INFO:tasks.rados.rados.0.smithi089.stdout:448: finishing set_chunk to oid 49 2024-04-22T18:39:01.519 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 689152 length: 860672 2024-04-22T18:39:01.519 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:39:01.519 INFO:tasks.rados.rados.0.smithi089.stdout:447: done (14 left) 2024-04-22T18:39:01.520 INFO:tasks.rados.rados.0.smithi089.stdout:448: done (13 left) 2024-04-22T18:39:01.520 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 13 2024-04-22T18:39:01.520 INFO:tasks.rados.rados.0.smithi089.stdout:452: finishing set_chunk to oid 3 2024-04-22T18:39:01.520 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 276992 length: 639488 2024-04-22T18:39:01.520 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-22T18:39:01.520 INFO:tasks.rados.rados.0.smithi089.stdout:452: done (12 left) 2024-04-22T18:39:01.520 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 12 2024-04-22T18:39:01.520 INFO:tasks.rados.rados.0.smithi089.stdout:465: finishing set_chunk to oid 16 2024-04-22T18:39:01.520 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 494592 length: 296960 2024-04-22T18:39:01.520 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-22T18:39:01.520 INFO:tasks.rados.rados.0.smithi089.stdout:465: done (11 left) 2024-04-22T18:39:01.520 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 11 2024-04-22T18:39:01.520 INFO:tasks.rados.rados.0.smithi089.stdout:458: finishing set_chunk to oid 9 2024-04-22T18:39:01.520 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 495104 length: 1247232 2024-04-22T18:39:01.521 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-22T18:39:01.521 INFO:tasks.rados.rados.0.smithi089.stdout:417: finishing set_chunk to oid 18 2024-04-22T18:39:01.521 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 9 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-22T18:39:01.521 INFO:tasks.rados.rados.0.smithi089.stdout:417: done (10 left) 2024-04-22T18:39:01.521 INFO:tasks.rados.rados.0.smithi089.stdout:458: done (9 left) 2024-04-22T18:39:01.521 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 9 2024-04-22T18:39:01.521 INFO:tasks.rados.rados.0.smithi089.stdout:426: finishing set_chunk to oid 27 2024-04-22T18:39:01.521 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 9 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-22T18:39:01.521 INFO:tasks.rados.rados.0.smithi089.stdout:426: done (8 left) 2024-04-22T18:39:01.521 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 8 2024-04-22T18:39:01.521 INFO:tasks.rados.rados.0.smithi089.stdout:455: finishing set_chunk to oid 6 2024-04-22T18:39:01.521 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 782848 length: 654848 2024-04-22T18:39:01.521 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-22T18:39:01.521 INFO:tasks.rados.rados.0.smithi089.stdout:462: finishing set_chunk to oid 13 2024-04-22T18:39:01.521 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1054208 length: 593920 2024-04-22T18:39:01.522 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-22T18:39:01.522 INFO:tasks.rados.rados.0.smithi089.stdout:455: done (7 left) 2024-04-22T18:39:01.522 INFO:tasks.rados.rados.0.smithi089.stdout:462: done (6 left) 2024-04-22T18:39:01.522 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 6 2024-04-22T18:39:01.522 INFO:tasks.rados.rados.0.smithi089.stdout:464: finishing set_chunk to oid 15 2024-04-22T18:39:01.522 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 366592 length: 847360 2024-04-22T18:39:01.522 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-22T18:39:01.522 INFO:tasks.rados.rados.0.smithi089.stdout:464: done (5 left) 2024-04-22T18:39:01.522 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 5 2024-04-22T18:39:01.522 INFO:tasks.rados.rados.0.smithi089.stdout:466: finishing set_chunk to oid 17 2024-04-22T18:39:01.522 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1106944 length: 987648 2024-04-22T18:39:01.522 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-22T18:39:01.522 INFO:tasks.rados.rados.0.smithi089.stdout:466: done (4 left) 2024-04-22T18:39:01.522 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 4 2024-04-22T18:39:01.522 INFO:tasks.rados.rados.0.smithi089.stdout:436: finishing set_chunk to oid 37 2024-04-22T18:39:01.523 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 9 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-22T18:39:01.523 INFO:tasks.rados.rados.0.smithi089.stdout:436: done (3 left) 2024-04-22T18:39:01.523 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 3 2024-04-22T18:39:01.523 INFO:tasks.rados.rados.0.smithi089.stdout:427: finishing set_chunk to oid 28 2024-04-22T18:39:01.523 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 15 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-22T18:39:01.523 INFO:tasks.rados.rados.0.smithi089.stdout:427: done (2 left) 2024-04-22T18:39:01.523 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 2 2024-04-22T18:39:01.523 INFO:tasks.rados.rados.0.smithi089.stdout:437: finishing set_chunk to oid 38 2024-04-22T18:39:01.523 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-22T18:39:01.523 INFO:tasks.rados.rados.0.smithi089.stdout:437: done (1 left) 2024-04-22T18:39:01.523 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 1 2024-04-22T18:39:01.572 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:01.573+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:01.629 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:01.629 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:01.638 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:01.649 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:01.650+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:01.721 INFO:tasks.rados.rados.0.smithi089.stdout:449: finishing set_chunk to oid 50 2024-04-22T18:39:01.721 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 21 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-22T18:39:01.721 INFO:tasks.rados.rados.0.smithi089.stdout:449: done (0 left) 2024-04-22T18:39:01.721 INFO:tasks.rados.rados.0.smithi089.stdout:467: set_chunk oid 18 target oid 18 low_tier 2024-04-22T18:39:01.721 INFO:tasks.rados.rados.0.smithi089.stdout:467: set_chunk oid 18 offset: 227840 length: 1840128 target oid smithi08937929-18 low_tier offset: 227840 snap -1 2024-04-22T18:39:01.721 INFO:tasks.rados.rados.0.smithi089.stdout:468: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.722 INFO:tasks.rados.rados.0.smithi089.stdout:468: set_chunk oid 19 offset: 929792 length: 1360896 target oid smithi08937929-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 929792 snap -1 2024-04-22T18:39:01.722 INFO:tasks.rados.rados.0.smithi089.stdout:469: set_chunk oid 20 target oid 20 low_tier 2024-04-22T18:39:01.722 INFO:tasks.rados.rados.0.smithi089.stdout:469: set_chunk oid 20 offset: 75264 length: 1689088 target oid smithi08937929-20 low_tier offset: 75264 snap -1 2024-04-22T18:39:01.722 INFO:tasks.rados.rados.0.smithi089.stdout:470: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.722 INFO:tasks.rados.rados.0.smithi089.stdout:470: set_chunk oid 21 offset: 417280 length: 339968 target oid smithi08937929-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 417280 snap -1 2024-04-22T18:39:01.722 INFO:tasks.rados.rados.0.smithi089.stdout:471: set_chunk oid 22 target oid 22 low_tier 2024-04-22T18:39:01.722 INFO:tasks.rados.rados.0.smithi089.stdout:471: set_chunk oid 22 offset: 550912 length: 1386496 target oid smithi08937929-22 low_tier offset: 550912 snap -1 2024-04-22T18:39:01.722 INFO:tasks.rados.rados.0.smithi089.stdout:472: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.722 INFO:tasks.rados.rados.0.smithi089.stdout:472: set_chunk oid 23 offset: 221184 length: 1485312 target oid smithi08937929-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 221184 snap -1 2024-04-22T18:39:01.722 INFO:tasks.rados.rados.0.smithi089.stdout:473: set_chunk oid 24 target oid 24 low_tier 2024-04-22T18:39:01.722 INFO:tasks.rados.rados.0.smithi089.stdout:473: set_chunk oid 24 offset: 1111040 length: 813568 target oid smithi08937929-24 low_tier offset: 1111040 snap -1 2024-04-22T18:39:01.722 INFO:tasks.rados.rados.0.smithi089.stdout:474: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:474: set_chunk oid 25 offset: 1590784 length: 452096 target oid smithi08937929-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1590784 snap -1 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:475: set_chunk oid 26 target oid 26 low_tier 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:475: set_chunk oid 26 offset: 1226240 length: 720896 target oid smithi08937929-26 low_tier offset: 1226240 snap -1 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:476: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:476: set_chunk oid 27 offset: 732672 length: 1688576 target oid smithi08937929-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 732672 snap -1 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:477: set_chunk oid 28 target oid 28 low_tier 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:477: set_chunk oid 28 offset: 1397248 length: 124928 target oid smithi08937929-28 low_tier offset: 1397248 snap -1 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:478: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:478: set_chunk oid 29 offset: 1078272 length: 214528 target oid smithi08937929-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1078272 snap -1 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:479: set_chunk oid 30 target oid 30 low_tier 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:479: set_chunk oid 30 offset: 885248 length: 740864 target oid smithi08937929-30 low_tier offset: 885248 snap -1 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:480: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:480: set_chunk oid 31 offset: 647680 length: 1817088 target oid smithi08937929-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 647680 snap -1 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:481: set_chunk oid 32 target oid 32 low_tier 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:481: set_chunk oid 32 offset: 1072640 length: 806400 target oid smithi08937929-32 low_tier offset: 1072640 snap -1 2024-04-22T18:39:01.723 INFO:tasks.rados.rados.0.smithi089.stdout:482: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.724 INFO:tasks.rados.rados.0.smithi089.stdout:482: set_chunk oid 33 offset: 182784 length: 269824 target oid smithi08937929-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 182784 snap -1 2024-04-22T18:39:01.724 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.724 INFO:tasks.rados.rados.0.smithi089.stdout:469: finishing set_chunk to oid 20 2024-04-22T18:39:01.724 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 75264 length: 1689088 2024-04-22T18:39:01.724 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-22T18:39:01.724 INFO:tasks.rados.rados.0.smithi089.stdout:469: done (15 left) 2024-04-22T18:39:01.724 INFO:tasks.rados.rados.0.smithi089.stdout:483: set_chunk oid 34 target oid 34 low_tier 2024-04-22T18:39:01.724 INFO:tasks.rados.rados.0.smithi089.stdout:470: finishing set_chunk to oid 21 2024-04-22T18:39:01.724 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 417280 length: 339968 2024-04-22T18:39:01.724 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-22T18:39:01.724 INFO:tasks.rados.rados.0.smithi089.stdout:483: set_chunk oid 34 offset: 9216 length: 592384 target oid smithi08937929-34 low_tier offset: 9216 snap -1 2024-04-22T18:39:01.724 INFO:tasks.rados.rados.0.smithi089.stdout:470: done (15 left) 2024-04-22T18:39:01.724 INFO:tasks.rados.rados.0.smithi089.stdout:484: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.724 INFO:tasks.rados.rados.0.smithi089.stdout:484: set_chunk oid 35 offset: 913408 length: 1375232 target oid smithi08937929-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 913408 snap -1 2024-04-22T18:39:01.725 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.750 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:01.751+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:01.780 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:01.781 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:01.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:01.820 INFO:tasks.rados.rados.0.smithi089.stdout:480: finishing set_chunk to oid 31 2024-04-22T18:39:01.820 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 647680 length: 1817088 2024-04-22T18:39:01.821 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-22T18:39:01.821 INFO:tasks.rados.rados.0.smithi089.stdout:480: done (15 left) 2024-04-22T18:39:01.821 INFO:tasks.rados.rados.0.smithi089.stdout:485: set_chunk oid 36 target oid 36 low_tier 2024-04-22T18:39:01.821 INFO:tasks.rados.rados.0.smithi089.stdout:485: set_chunk oid 36 offset: 1021440 length: 118784 target oid smithi08937929-36 low_tier offset: 1021440 snap -1 2024-04-22T18:39:01.821 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.821 INFO:tasks.rados.rados.0.smithi089.stdout:482: finishing set_chunk to oid 33 2024-04-22T18:39:01.821 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 182784 length: 269824 2024-04-22T18:39:01.821 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-22T18:39:01.821 INFO:tasks.rados.rados.0.smithi089.stdout:482: done (15 left) 2024-04-22T18:39:01.821 INFO:tasks.rados.rados.0.smithi089.stdout:486: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.821 INFO:tasks.rados.rados.0.smithi089.stdout:486: set_chunk oid 37 offset: 899584 length: 517120 target oid smithi08937929-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 899584 snap -1 2024-04-22T18:39:01.821 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.821 INFO:tasks.rados.rados.0.smithi089.stdout:485: finishing set_chunk to oid 36 2024-04-22T18:39:01.821 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 1021440 length: 118784 2024-04-22T18:39:01.822 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-22T18:39:01.822 INFO:tasks.rados.rados.0.smithi089.stdout:485: done (15 left) 2024-04-22T18:39:01.822 INFO:tasks.rados.rados.0.smithi089.stdout:487: set_chunk oid 38 target oid 38 low_tier 2024-04-22T18:39:01.822 INFO:tasks.rados.rados.0.smithi089.stdout:487: set_chunk oid 38 offset: 832512 length: 120832 target oid smithi08937929-38 low_tier offset: 832512 snap -1 2024-04-22T18:39:01.822 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.850 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:01.851+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:01.924 INFO:tasks.rados.rados.0.smithi089.stdout:467: finishing set_chunk to oid 18 2024-04-22T18:39:01.924 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 227840 length: 1840128 2024-04-22T18:39:01.924 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-22T18:39:01.924 INFO:tasks.rados.rados.0.smithi089.stdout:467: done (15 left) 2024-04-22T18:39:01.925 INFO:tasks.rados.rados.0.smithi089.stdout:488: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.925 INFO:tasks.rados.rados.0.smithi089.stdout:488: set_chunk oid 39 offset: 698880 length: 1675776 target oid smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 698880 snap -1 2024-04-22T18:39:01.925 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.925 INFO:tasks.rados.rados.0.smithi089.stdout:471: finishing set_chunk to oid 22 2024-04-22T18:39:01.925 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 550912 length: 1386496 2024-04-22T18:39:01.925 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-22T18:39:01.925 INFO:tasks.rados.rados.0.smithi089.stdout:468: finishing set_chunk to oid 19 2024-04-22T18:39:01.925 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 929792 length: 1360896 2024-04-22T18:39:01.925 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-22T18:39:01.925 INFO:tasks.rados.rados.0.smithi089.stdout:468: done (15 left) 2024-04-22T18:39:01.925 INFO:tasks.rados.rados.0.smithi089.stdout:471: done (14 left) 2024-04-22T18:39:01.925 INFO:tasks.rados.rados.0.smithi089.stdout:489: set_chunk oid 40 target oid 40 low_tier 2024-04-22T18:39:01.925 INFO:tasks.rados.rados.0.smithi089.stdout:489: set_chunk oid 40 offset: 615936 length: 634368 target oid smithi08937929-40 low_tier offset: 615936 snap -1 2024-04-22T18:39:01.925 INFO:tasks.rados.rados.0.smithi089.stdout:490: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.926 INFO:tasks.rados.rados.0.smithi089.stdout:490: set_chunk oid 41 offset: 761856 length: 716800 target oid smithi08937929-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 761856 snap -1 2024-04-22T18:39:01.926 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.926 INFO:tasks.rados.rados.0.smithi089.stdout:472: finishing set_chunk to oid 23 2024-04-22T18:39:01.926 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 221184 length: 1485312 2024-04-22T18:39:01.926 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-22T18:39:01.926 INFO:tasks.rados.rados.0.smithi089.stdout:473: finishing set_chunk to oid 24 2024-04-22T18:39:01.926 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 1111040 length: 813568 2024-04-22T18:39:01.926 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-22T18:39:01.926 INFO:tasks.rados.rados.0.smithi089.stdout:475: finishing set_chunk to oid 26 2024-04-22T18:39:01.926 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1226240 length: 720896 2024-04-22T18:39:01.926 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-22T18:39:01.926 INFO:tasks.rados.rados.0.smithi089.stdout:476: finishing set_chunk to oid 27 2024-04-22T18:39:01.926 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 732672 length: 1688576 2024-04-22T18:39:01.926 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-22T18:39:01.927 INFO:tasks.rados.rados.0.smithi089.stdout:472: done (15 left) 2024-04-22T18:39:01.927 INFO:tasks.rados.rados.0.smithi089.stdout:473: done (14 left) 2024-04-22T18:39:01.927 INFO:tasks.rados.rados.0.smithi089.stdout:475: done (13 left) 2024-04-22T18:39:01.927 INFO:tasks.rados.rados.0.smithi089.stdout:476: done (12 left) 2024-04-22T18:39:01.927 INFO:tasks.rados.rados.0.smithi089.stdout:491: set_chunk oid 42 target oid 42 low_tier 2024-04-22T18:39:01.927 INFO:tasks.rados.rados.0.smithi089.stdout:491: set_chunk oid 42 offset: 40960 length: 2460160 target oid smithi08937929-42 low_tier offset: 40960 snap -1 2024-04-22T18:39:01.927 INFO:tasks.rados.rados.0.smithi089.stdout:492: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.927 INFO:tasks.rados.rados.0.smithi089.stdout:492: set_chunk oid 43 offset: 169984 length: 613376 target oid smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 169984 snap -1 2024-04-22T18:39:01.927 INFO:tasks.rados.rados.0.smithi089.stdout:493: set_chunk oid 44 target oid 44 low_tier 2024-04-22T18:39:01.927 INFO:tasks.rados.rados.0.smithi089.stdout:493: set_chunk oid 44 offset: 38400 length: 244736 target oid smithi08937929-44 low_tier offset: 38400 snap -1 2024-04-22T18:39:01.927 INFO:tasks.rados.rados.0.smithi089.stdout:494: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.927 INFO:tasks.rados.rados.0.smithi089.stdout:494: set_chunk oid 45 offset: 937472 length: 226304 target oid smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 937472 snap -1 2024-04-22T18:39:01.927 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.927 INFO:tasks.rados.rados.0.smithi089.stdout:484: finishing set_chunk to oid 35 2024-04-22T18:39:01.928 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 913408 length: 1375232 2024-04-22T18:39:01.928 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-22T18:39:01.928 INFO:tasks.rados.rados.0.smithi089.stdout:484: done (15 left) 2024-04-22T18:39:01.928 INFO:tasks.rados.rados.0.smithi089.stdout:495: set_chunk oid 46 target oid 46 low_tier 2024-04-22T18:39:01.928 INFO:tasks.rados.rados.0.smithi089.stdout:495: set_chunk oid 46 offset: 1845248 length: 308224 target oid smithi08937929-46 low_tier offset: 1845248 snap -1 2024-04-22T18:39:01.928 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.928 INFO:tasks.rados.rados.0.smithi089.stdout:487: finishing set_chunk to oid 38 2024-04-22T18:39:01.928 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 832512 length: 120832 2024-04-22T18:39:01.928 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-22T18:39:01.928 INFO:tasks.rados.rados.0.smithi089.stdout:487: done (15 left) 2024-04-22T18:39:01.928 INFO:tasks.rados.rados.0.smithi089.stdout:496: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.928 INFO:tasks.rados.rados.0.smithi089.stdout:496: set_chunk oid 47 offset: 1754112 length: 68608 target oid smithi08937929-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1754112 snap -1 2024-04-22T18:39:01.928 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.928 INFO:tasks.rados.rados.0.smithi089.stdout:491: finishing set_chunk to oid 42 2024-04-22T18:39:01.929 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 40960 length: 2460160 2024-04-22T18:39:01.929 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-22T18:39:01.929 INFO:tasks.rados.rados.0.smithi089.stdout:491: done (15 left) 2024-04-22T18:39:01.929 INFO:tasks.rados.rados.0.smithi089.stdout:497: set_chunk oid 48 target oid 48 low_tier 2024-04-22T18:39:01.929 INFO:tasks.rados.rados.0.smithi089.stdout:497: set_chunk oid 48 offset: 353280 length: 2054656 target oid smithi08937929-48 low_tier offset: 353280 snap -1 2024-04-22T18:39:01.929 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.929 INFO:tasks.rados.rados.0.smithi089.stdout:489: finishing set_chunk to oid 40 2024-04-22T18:39:01.929 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 615936 length: 634368 2024-04-22T18:39:01.929 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-22T18:39:01.929 INFO:tasks.rados.rados.0.smithi089.stdout:489: done (15 left) 2024-04-22T18:39:01.929 INFO:tasks.rados.rados.0.smithi089.stdout:498: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.929 INFO:tasks.rados.rados.0.smithi089.stdout:498: set_chunk oid 49 offset: 365568 length: 1596416 target oid smithi08937929-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 365568 snap -1 2024-04-22T18:39:01.929 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.929 INFO:tasks.rados.rados.0.smithi089.stdout:492: finishing set_chunk to oid 43 2024-04-22T18:39:01.930 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 169984 length: 613376 2024-04-22T18:39:01.930 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-22T18:39:01.930 INFO:tasks.rados.rados.0.smithi089.stdout:492: done (15 left) 2024-04-22T18:39:01.930 INFO:tasks.rados.rados.0.smithi089.stdout:499: set_chunk oid 50 target oid 50 low_tier 2024-04-22T18:39:01.930 INFO:tasks.rados.rados.0.smithi089.stdout:499: set_chunk oid 50 offset: 6656 length: 1547776 target oid smithi08937929-50 low_tier offset: 6656 snap -1 2024-04-22T18:39:01.930 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.930 INFO:tasks.rados.rados.0.smithi089.stdout:490: finishing set_chunk to oid 41 2024-04-22T18:39:01.930 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 761856 length: 716800 2024-04-22T18:39:01.930 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-22T18:39:01.930 INFO:tasks.rados.rados.0.smithi089.stdout:496: finishing set_chunk to oid 47 2024-04-22T18:39:01.930 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1754112 length: 68608 2024-04-22T18:39:01.930 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-22T18:39:01.930 INFO:tasks.rados.rados.0.smithi089.stdout:495: finishing set_chunk to oid 46 2024-04-22T18:39:01.930 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1845248 length: 308224 2024-04-22T18:39:01.930 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-22T18:39:01.931 INFO:tasks.rados.rados.0.smithi089.stdout:488: finishing set_chunk to oid 39 2024-04-22T18:39:01.931 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 698880 length: 1675776 2024-04-22T18:39:01.931 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-22T18:39:01.931 INFO:tasks.rados.rados.0.smithi089.stdout:488: done (15 left) 2024-04-22T18:39:01.931 INFO:tasks.rados.rados.0.smithi089.stdout:490: done (14 left) 2024-04-22T18:39:01.931 INFO:tasks.rados.rados.0.smithi089.stdout:495: done (13 left) 2024-04-22T18:39:01.931 INFO:tasks.rados.rados.0.smithi089.stdout:496: done (12 left) 2024-04-22T18:39:01.931 INFO:tasks.rados.rados.0.smithi089.stdout:500: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.931 INFO:tasks.rados.rados.0.smithi089.stdout:500: set_chunk oid 1 offset: 860160 length: 639488 target oid smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 860160 snap -1 2024-04-22T18:39:01.931 INFO:tasks.rados.rados.0.smithi089.stdout:501: set_chunk oid 2 target oid 2 low_tier 2024-04-22T18:39:01.931 INFO:tasks.rados.rados.0.smithi089.stdout:501: set_chunk oid 2 offset: 108544 length: 1862656 target oid smithi08937929-2 low_tier offset: 108544 snap -1 2024-04-22T18:39:01.931 INFO:tasks.rados.rados.0.smithi089.stdout:502: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.931 INFO:tasks.rados.rados.0.smithi089.stdout:502: set_chunk oid 3 offset: 290304 length: 388096 target oid smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 290304 snap -1 2024-04-22T18:39:01.931 INFO:tasks.rados.rados.0.smithi089.stdout:503: set_chunk oid 4 target oid 4 low_tier 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout:503: set_chunk oid 4 offset: 1450496 length: 322560 target oid smithi08937929-4 low_tier offset: 1450496 snap -1 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout:500: finishing set_chunk to oid 1 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 860160 length: 639488 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout:500: done (15 left) 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout:504: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout:504: set_chunk oid 5 offset: 451072 length: 1498624 target oid smithi08937929-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 451072 snap -1 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout:499: finishing set_chunk to oid 50 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 6656 length: 1547776 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout:499: done (15 left) 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout:505: set_chunk oid 6 target oid 6 low_tier 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout:505: set_chunk oid 6 offset: 184832 length: 515072 target oid smithi08937929-6 low_tier offset: 184832 snap -1 2024-04-22T18:39:01.932 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.933 INFO:tasks.rados.rados.0.smithi089.stdout:505: finishing set_chunk to oid 6 2024-04-22T18:39:01.933 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 184832 length: 515072 2024-04-22T18:39:01.933 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-22T18:39:01.933 INFO:tasks.rados.rados.0.smithi089.stdout:505: done (15 left) 2024-04-22T18:39:01.933 INFO:tasks.rados.rados.0.smithi089.stdout:506: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:01.933 INFO:tasks.rados.rados.0.smithi089.stdout:506: set_chunk oid 7 offset: 692736 length: 289792 target oid smithi08937929-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 692736 snap -1 2024-04-22T18:39:01.933 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:01.933 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:01.933 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:01.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:01.985 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:01.986+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:02.025 INFO:tasks.rados.rados.0.smithi089.stdout:474: finishing set_chunk to oid 25 2024-04-22T18:39:02.025 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1590784 length: 452096 2024-04-22T18:39:02.025 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-22T18:39:02.025 INFO:tasks.rados.rados.0.smithi089.stdout:478: finishing set_chunk to oid 29 2024-04-22T18:39:02.025 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1078272 length: 214528 2024-04-22T18:39:02.025 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-22T18:39:02.025 INFO:tasks.rados.rados.0.smithi089.stdout:474: done (15 left) 2024-04-22T18:39:02.025 INFO:tasks.rados.rados.0.smithi089.stdout:478: done (14 left) 2024-04-22T18:39:02.025 INFO:tasks.rados.rados.0.smithi089.stdout:507: set_chunk oid 8 target oid 8 low_tier 2024-04-22T18:39:02.025 INFO:tasks.rados.rados.0.smithi089.stdout:507: set_chunk oid 8 offset: 648704 length: 204800 target oid smithi08937929-8 low_tier offset: 648704 snap -1 2024-04-22T18:39:02.025 INFO:tasks.rados.rados.0.smithi089.stdout:508: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.026 INFO:tasks.rados.rados.0.smithi089.stdout:508: set_chunk oid 9 offset: 200192 length: 339968 target oid smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 200192 snap -1 2024-04-22T18:39:02.026 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.026 INFO:tasks.rados.rados.0.smithi089.stdout:481: finishing set_chunk to oid 32 2024-04-22T18:39:02.026 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 1072640 length: 806400 2024-04-22T18:39:02.026 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-22T18:39:02.026 INFO:tasks.rados.rados.0.smithi089.stdout:481: done (15 left) 2024-04-22T18:39:02.026 INFO:tasks.rados.rados.0.smithi089.stdout:509: set_chunk oid 10 target oid 10 low_tier 2024-04-22T18:39:02.026 INFO:tasks.rados.rados.0.smithi089.stdout:509: set_chunk oid 10 offset: 47616 length: 360960 target oid smithi08937929-10 low_tier offset: 47616 snap -1 2024-04-22T18:39:02.026 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.026 INFO:tasks.rados.rados.0.smithi089.stdout:486: finishing set_chunk to oid 37 2024-04-22T18:39:02.026 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 899584 length: 517120 2024-04-22T18:39:02.026 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-22T18:39:02.026 INFO:tasks.rados.rados.0.smithi089.stdout:494: finishing set_chunk to oid 45 2024-04-22T18:39:02.026 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 937472 length: 226304 2024-04-22T18:39:02.027 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-22T18:39:02.027 INFO:tasks.rados.rados.0.smithi089.stdout:486: done (15 left) 2024-04-22T18:39:02.027 INFO:tasks.rados.rados.0.smithi089.stdout:494: done (14 left) 2024-04-22T18:39:02.027 INFO:tasks.rados.rados.0.smithi089.stdout:510: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.027 INFO:tasks.rados.rados.0.smithi089.stdout:510: set_chunk oid 11 offset: 403456 length: 43008 target oid smithi08937929-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 403456 snap -1 2024-04-22T18:39:02.027 INFO:tasks.rados.rados.0.smithi089.stdout:511: set_chunk oid 12 target oid 12 low_tier 2024-04-22T18:39:02.027 INFO:tasks.rados.rados.0.smithi089.stdout:511: set_chunk oid 12 offset: 11776 length: 1006592 target oid smithi08937929-12 low_tier offset: 11776 snap -1 2024-04-22T18:39:02.027 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.027 INFO:tasks.rados.rados.0.smithi089.stdout:507: finishing set_chunk to oid 8 2024-04-22T18:39:02.027 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 648704 length: 204800 2024-04-22T18:39:02.027 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-22T18:39:02.027 INFO:tasks.rados.rados.0.smithi089.stdout:507: done (15 left) 2024-04-22T18:39:02.027 INFO:tasks.rados.rados.0.smithi089.stdout:512: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.027 INFO:tasks.rados.rados.0.smithi089.stdout:512: set_chunk oid 13 offset: 440832 length: 1727488 target oid smithi08937929-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 440832 snap -1 2024-04-22T18:39:02.027 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.028 INFO:tasks.rados.rados.0.smithi089.stdout:509: finishing set_chunk to oid 10 2024-04-22T18:39:02.028 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 47616 length: 360960 2024-04-22T18:39:02.028 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-22T18:39:02.028 INFO:tasks.rados.rados.0.smithi089.stdout:509: done (15 left) 2024-04-22T18:39:02.028 INFO:tasks.rados.rados.0.smithi089.stdout:513: set_chunk oid 14 target oid 14 low_tier 2024-04-22T18:39:02.028 INFO:tasks.rados.rados.0.smithi089.stdout:513: set_chunk oid 14 offset: 1428992 length: 707584 target oid smithi08937929-14 low_tier offset: 1428992 snap -1 2024-04-22T18:39:02.028 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.028 INFO:tasks.rados.rados.0.smithi089.stdout:511: finishing set_chunk to oid 12 2024-04-22T18:39:02.028 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 11776 length: 1006592 2024-04-22T18:39:02.028 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-22T18:39:02.028 INFO:tasks.rados.rados.0.smithi089.stdout:511: done (15 left) 2024-04-22T18:39:02.028 INFO:tasks.rados.rados.0.smithi089.stdout:514: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.028 INFO:tasks.rados.rados.0.smithi089.stdout:514: set_chunk oid 15 offset: 1354752 length: 268800 target oid smithi08937929-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1354752 snap -1 2024-04-22T18:39:02.028 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.028 INFO:tasks.rados.rados.0.smithi089.stdout:498: finishing set_chunk to oid 49 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 365568 length: 1596416 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout:498: done (15 left) 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout:515: set_chunk oid 16 target oid 16 low_tier 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout:515: set_chunk oid 16 offset: 396800 length: 1521664 target oid smithi08937929-16 low_tier offset: 396800 snap -1 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout:512: finishing set_chunk to oid 13 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 440832 length: 1727488 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout:512: done (15 left) 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout:516: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout:516: set_chunk oid 17 offset: 645120 length: 290304 target oid smithi08937929-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 645120 snap -1 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout:514: finishing set_chunk to oid 15 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1354752 length: 268800 2024-04-22T18:39:02.029 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-22T18:39:02.030 INFO:tasks.rados.rados.0.smithi089.stdout:514: done (15 left) 2024-04-22T18:39:02.030 INFO:tasks.rados.rados.0.smithi089.stdout:517: set_chunk oid 18 target oid 18 low_tier 2024-04-22T18:39:02.030 INFO:tasks.rados.rados.0.smithi089.stdout:517: set_chunk oid 18 offset: 57856 length: 1333248 target oid smithi08937929-18 low_tier offset: 57856 snap -1 2024-04-22T18:39:02.030 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.030 INFO:tasks.rados.rados.0.smithi089.stdout:515: finishing set_chunk to oid 16 2024-04-22T18:39:02.030 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 396800 length: 1521664 2024-04-22T18:39:02.030 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-22T18:39:02.030 INFO:tasks.rados.rados.0.smithi089.stdout:515: done (15 left) 2024-04-22T18:39:02.030 INFO:tasks.rados.rados.0.smithi089.stdout:518: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.030 INFO:tasks.rados.rados.0.smithi089.stdout:518: set_chunk oid 19 offset: 918016 length: 485376 target oid smithi08937929-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 918016 snap -1 2024-04-22T18:39:02.030 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.030 INFO:tasks.rados.rados.0.smithi089.stdout:504: finishing set_chunk to oid 5 2024-04-22T18:39:02.030 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 451072 length: 1498624 2024-04-22T18:39:02.030 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-22T18:39:02.030 INFO:tasks.rados.rados.0.smithi089.stdout:504: done (15 left) 2024-04-22T18:39:02.031 INFO:tasks.rados.rados.0.smithi089.stdout:519: set_chunk oid 20 target oid 20 low_tier 2024-04-22T18:39:02.031 INFO:tasks.rados.rados.0.smithi089.stdout:519: set_chunk oid 20 offset: 1026048 length: 1106944 target oid smithi08937929-20 low_tier offset: 1026048 snap -1 2024-04-22T18:39:02.031 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.031 INFO:tasks.rados.rados.0.smithi089.stdout:501: finishing set_chunk to oid 2 2024-04-22T18:39:02.031 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 108544 length: 1862656 2024-04-22T18:39:02.031 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-22T18:39:02.031 INFO:tasks.rados.rados.0.smithi089.stdout:501: done (15 left) 2024-04-22T18:39:02.031 INFO:tasks.rados.rados.0.smithi089.stdout:520: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.031 INFO:tasks.rados.rados.0.smithi089.stdout:520: set_chunk oid 21 offset: 210432 length: 1100800 target oid smithi08937929-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 210432 snap -1 2024-04-22T18:39:02.031 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.031 INFO:tasks.rados.rados.0.smithi089.stdout:502: finishing set_chunk to oid 3 2024-04-22T18:39:02.031 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 290304 length: 388096 2024-04-22T18:39:02.031 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-22T18:39:02.031 INFO:tasks.rados.rados.0.smithi089.stdout:502: done (15 left) 2024-04-22T18:39:02.031 INFO:tasks.rados.rados.0.smithi089.stdout:521: set_chunk oid 22 target oid 22 low_tier 2024-04-22T18:39:02.032 INFO:tasks.rados.rados.0.smithi089.stdout:521: set_chunk oid 22 offset: 349696 length: 1127424 target oid smithi08937929-22 low_tier offset: 349696 snap -1 2024-04-22T18:39:02.032 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.032 INFO:tasks.rados.rados.0.smithi089.stdout:508: finishing set_chunk to oid 9 2024-04-22T18:39:02.032 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 200192 length: 339968 2024-04-22T18:39:02.032 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-22T18:39:02.032 INFO:tasks.rados.rados.0.smithi089.stdout:508: done (15 left) 2024-04-22T18:39:02.032 INFO:tasks.rados.rados.0.smithi089.stdout:522: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.032 INFO:tasks.rados.rados.0.smithi089.stdout:522: set_chunk oid 23 offset: 572416 length: 1593344 target oid smithi08937929-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 572416 snap -1 2024-04-22T18:39:02.032 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.043 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:02.043 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:02.052 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:02.053 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:02.053+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:02.120 INFO:tasks.rados.rados.0.smithi089.stdout:497: finishing set_chunk to oid 48 2024-04-22T18:39:02.120 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 353280 length: 2054656 2024-04-22T18:39:02.120 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-22T18:39:02.121 INFO:tasks.rados.rados.0.smithi089.stdout:497: done (15 left) 2024-04-22T18:39:02.121 INFO:tasks.rados.rados.0.smithi089.stdout:523: set_chunk oid 24 target oid 24 low_tier 2024-04-22T18:39:02.121 INFO:tasks.rados.rados.0.smithi089.stdout:523: set_chunk oid 24 offset: 178176 length: 1040384 target oid smithi08937929-24 low_tier offset: 178176 snap -1 2024-04-22T18:39:02.121 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.121 INFO:tasks.rados.rados.0.smithi089.stdout:513: finishing set_chunk to oid 14 2024-04-22T18:39:02.121 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 1428992 length: 707584 2024-04-22T18:39:02.121 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-22T18:39:02.121 INFO:tasks.rados.rados.0.smithi089.stdout:513: done (15 left) 2024-04-22T18:39:02.121 INFO:tasks.rados.rados.0.smithi089.stdout:524: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.121 INFO:tasks.rados.rados.0.smithi089.stdout:524: set_chunk oid 25 offset: 867328 length: 566784 target oid smithi08937929-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 867328 snap -1 2024-04-22T18:39:02.121 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.122 INFO:tasks.rados.rados.0.smithi089.stdout:518: finishing set_chunk to oid 19 2024-04-22T18:39:02.122 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 918016 length: 485376 2024-04-22T18:39:02.122 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-22T18:39:02.123 INFO:tasks.rados.rados.0.smithi089.stdout:521: finishing set_chunk to oid 22 2024-04-22T18:39:02.123 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 349696 length: 1127424 2024-04-22T18:39:02.123 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-22T18:39:02.123 INFO:tasks.rados.rados.0.smithi089.stdout:519: finishing set_chunk to oid 20 2024-04-22T18:39:02.123 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 1026048 length: 1106944 2024-04-22T18:39:02.123 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-22T18:39:02.123 INFO:tasks.rados.rados.0.smithi089.stdout:518: done (15 left) 2024-04-22T18:39:02.123 INFO:tasks.rados.rados.0.smithi089.stdout:519: done (14 left) 2024-04-22T18:39:02.123 INFO:tasks.rados.rados.0.smithi089.stdout:521: done (13 left) 2024-04-22T18:39:02.123 INFO:tasks.rados.rados.0.smithi089.stdout:525: set_chunk oid 26 target oid 26 low_tier 2024-04-22T18:39:02.123 INFO:tasks.rados.rados.0.smithi089.stdout:525: set_chunk oid 26 offset: 1281024 length: 299008 target oid smithi08937929-26 low_tier offset: 1281024 snap -1 2024-04-22T18:39:02.123 INFO:tasks.rados.rados.0.smithi089.stdout:526: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.123 INFO:tasks.rados.rados.0.smithi089.stdout:526: set_chunk oid 27 offset: 1096704 length: 1070080 target oid smithi08937929-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1096704 snap -1 2024-04-22T18:39:02.124 INFO:tasks.rados.rados.0.smithi089.stdout:526 retry set_chunk ! 2024-04-22T18:39:02.124 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 15 2024-04-22T18:39:02.124 INFO:tasks.rados.rados.0.smithi089.stdout:520: finishing set_chunk to oid 21 2024-04-22T18:39:02.124 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 210432 length: 1100800 2024-04-22T18:39:02.124 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-22T18:39:02.124 INFO:tasks.rados.rados.0.smithi089.stdout:522: finishing set_chunk to oid 23 2024-04-22T18:39:02.124 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 572416 length: 1593344 2024-04-22T18:39:02.124 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-22T18:39:02.124 INFO:tasks.rados.rados.0.smithi089.stdout:520: done (14 left) 2024-04-22T18:39:02.124 INFO:tasks.rados.rados.0.smithi089.stdout:522: done (13 left) 2024-04-22T18:39:02.124 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 13 2024-04-22T18:39:02.125 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:02.122+0000 7f8607f36640 -1 osd.3 31 handle_osd_map injecting map crc failure 2024-04-22T18:39:02.125 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:39:02.127 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:39:02.127 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-22T18:39:02.153 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:02.154+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:02.183 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:39:02.195 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:02.195 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:02.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:02.221 INFO:tasks.rados.rados.0.smithi089.stdout:516: finishing set_chunk to oid 17 2024-04-22T18:39:02.221 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 645120 length: 290304 2024-04-22T18:39:02.221 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-22T18:39:02.221 INFO:tasks.rados.rados.0.smithi089.stdout:516: done (12 left) 2024-04-22T18:39:02.221 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 12 2024-04-22T18:39:02.221 INFO:tasks.rados.rados.0.smithi089.stdout:517: finishing set_chunk to oid 18 2024-04-22T18:39:02.221 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 57856 length: 1333248 2024-04-22T18:39:02.221 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-22T18:39:02.221 INFO:tasks.rados.rados.0.smithi089.stdout:517: done (11 left) 2024-04-22T18:39:02.221 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 11 2024-04-22T18:39:02.222 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:02.223+0000 7f7f44bb3640 -1 osd.4 31 handle_osd_map injecting map crc failure 2024-04-22T18:39:02.223 INFO:tasks.rados.rados.0.smithi089.stdout:523: finishing set_chunk to oid 24 2024-04-22T18:39:02.223 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 178176 length: 1040384 2024-04-22T18:39:02.223 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-22T18:39:02.223 INFO:tasks.rados.rados.0.smithi089.stdout:525: finishing set_chunk to oid 26 2024-04-22T18:39:02.223 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1281024 length: 299008 2024-04-22T18:39:02.223 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout:477: finishing set_chunk to oid 28 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 15 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout:479: finishing set_chunk to oid 30 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout:477: done (10 left) 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout:479: done (9 left) 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout:523: done (8 left) 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout:525: done (7 left) 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 7 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout:483: finishing set_chunk to oid 34 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout:483: done (6 left) 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 6 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout:526: finishing set_chunk to oid 27 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1096704 length: 1070080 2024-04-22T18:39:02.224 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-22T18:39:02.225 INFO:tasks.rados.rados.0.smithi089.stdout:526: done (5 left) 2024-04-22T18:39:02.225 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 5 2024-04-22T18:39:02.253 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:02.253+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:02.306 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:02.306 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:02.321 INFO:tasks.rados.rados.0.smithi089.stdout:524: finishing set_chunk to oid 25 2024-04-22T18:39:02.321 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 867328 length: 566784 2024-04-22T18:39:02.321 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-22T18:39:02.322 INFO:tasks.rados.rados.0.smithi089.stdout:524: done (4 left) 2024-04-22T18:39:02.322 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 4 2024-04-22T18:39:02.323 INFO:tasks.rados.rados.0.smithi089.stdout:503: finishing set_chunk to oid 4 2024-04-22T18:39:02.323 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-22T18:39:02.323 INFO:tasks.rados.rados.0.smithi089.stdout:503: done (3 left) 2024-04-22T18:39:02.323 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 3 2024-04-22T18:39:02.323 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:02.324 INFO:tasks.rados.rados.0.smithi089.stdout:493: finishing set_chunk to oid 44 2024-04-22T18:39:02.324 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-22T18:39:02.324 INFO:tasks.rados.rados.0.smithi089.stdout:493: done (2 left) 2024-04-22T18:39:02.324 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 2 2024-04-22T18:39:02.325 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:02.325 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:02.328 INFO:tasks.rados.rados.0.smithi089.stdout:506: finishing set_chunk to oid 7 2024-04-22T18:39:02.328 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-22T18:39:02.328 INFO:tasks.rados.rados.0.smithi089.stdout:506: done (1 left) 2024-04-22T18:39:02.328 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 1 2024-04-22T18:39:02.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:02.340 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:02.340 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:02.354 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:02.355+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:02.356 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:02.356 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:02.372 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:02.373 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:02.407 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:02.407 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:02.435 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:02.435 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:02.440 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:02.440 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:02.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:02.455 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:02.456+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:02.456 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:02.457 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:02.530 INFO:tasks.rados.rados.0.smithi089.stdout:510: finishing set_chunk to oid 11 2024-04-22T18:39:02.530 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 6 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-22T18:39:02.530 INFO:tasks.rados.rados.0.smithi089.stdout:510: done (0 left) 2024-04-22T18:39:02.531 INFO:tasks.rados.rados.0.smithi089.stdout:527: set_chunk oid 28 target oid 28 low_tier 2024-04-22T18:39:02.531 INFO:tasks.rados.rados.0.smithi089.stdout:527: set_chunk oid 28 offset: 1843200 length: 45568 target oid smithi08937929-28 low_tier offset: 1843200 snap -1 2024-04-22T18:39:02.531 INFO:tasks.rados.rados.0.smithi089.stdout:528: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.531 INFO:tasks.rados.rados.0.smithi089.stdout:528: set_chunk oid 29 offset: 1313792 length: 879616 target oid smithi08937929-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1313792 snap -1 2024-04-22T18:39:02.531 INFO:tasks.rados.rados.0.smithi089.stdout:529: set_chunk oid 30 target oid 30 low_tier 2024-04-22T18:39:02.531 INFO:tasks.rados.rados.0.smithi089.stdout:529: set_chunk oid 30 offset: 915968 length: 217600 target oid smithi08937929-30 low_tier offset: 915968 snap -1 2024-04-22T18:39:02.531 INFO:tasks.rados.rados.0.smithi089.stdout:530: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.531 INFO:tasks.rados.rados.0.smithi089.stdout:530: set_chunk oid 31 offset: 104960 length: 37376 target oid smithi08937929-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 104960 snap -1 2024-04-22T18:39:02.531 INFO:tasks.rados.rados.0.smithi089.stdout:531: set_chunk oid 32 target oid 32 low_tier 2024-04-22T18:39:02.531 INFO:tasks.rados.rados.0.smithi089.stdout:531: set_chunk oid 32 offset: 565760 length: 505856 target oid smithi08937929-32 low_tier offset: 565760 snap -1 2024-04-22T18:39:02.531 INFO:tasks.rados.rados.0.smithi089.stdout:532: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.531 INFO:tasks.rados.rados.0.smithi089.stdout:532: set_chunk oid 33 offset: 295936 length: 1655296 target oid smithi08937929-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 295936 snap -1 2024-04-22T18:39:02.531 INFO:tasks.rados.rados.0.smithi089.stdout:533: set_chunk oid 34 target oid 34 low_tier 2024-04-22T18:39:02.531 INFO:tasks.rados.rados.0.smithi089.stdout:533: set_chunk oid 34 offset: 1920000 length: 280576 target oid smithi08937929-34 low_tier offset: 1920000 snap -1 2024-04-22T18:39:02.531 INFO:tasks.rados.rados.0.smithi089.stdout:534: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:534: set_chunk oid 35 offset: 950272 length: 1204736 target oid smithi08937929-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 950272 snap -1 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:535: set_chunk oid 36 target oid 36 low_tier 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:535: set_chunk oid 36 offset: 394752 length: 1797632 target oid smithi08937929-36 low_tier offset: 394752 snap -1 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:536: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:536: set_chunk oid 37 offset: 771584 length: 1401856 target oid smithi08937929-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 771584 snap -1 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:537: set_chunk oid 38 target oid 38 low_tier 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:537: set_chunk oid 38 offset: 2024960 length: 155136 target oid smithi08937929-38 low_tier offset: 2024960 snap -1 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:538: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:538: set_chunk oid 39 offset: 1482240 length: 441344 target oid smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1482240 snap -1 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:539: set_chunk oid 40 target oid 40 low_tier 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:539: set_chunk oid 40 offset: 1825280 length: 768512 target oid smithi08937929-40 low_tier offset: 1825280 snap -1 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:540: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:540: set_chunk oid 41 offset: 249344 length: 262144 target oid smithi08937929-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 249344 snap -1 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:541: set_chunk oid 42 target oid 42 low_tier 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:541: set_chunk oid 42 offset: 1589248 length: 169472 target oid smithi08937929-42 low_tier offset: 1589248 snap -1 2024-04-22T18:39:02.532 INFO:tasks.rados.rados.0.smithi089.stdout:542: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.533 INFO:tasks.rados.rados.0.smithi089.stdout:542: set_chunk oid 43 offset: 818688 length: 1377280 target oid smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 818688 snap -1 2024-04-22T18:39:02.533 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.533 INFO:tasks.rados.rados.0.smithi089.stdout:528: finishing set_chunk to oid 29 2024-04-22T18:39:02.533 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1313792 length: 879616 2024-04-22T18:39:02.533 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-22T18:39:02.533 INFO:tasks.rados.rados.0.smithi089.stdout:531: finishing set_chunk to oid 32 2024-04-22T18:39:02.533 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 565760 length: 505856 2024-04-22T18:39:02.533 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-22T18:39:02.533 INFO:tasks.rados.rados.0.smithi089.stdout:528: done (15 left) 2024-04-22T18:39:02.533 INFO:tasks.rados.rados.0.smithi089.stdout:531: done (14 left) 2024-04-22T18:39:02.533 INFO:tasks.rados.rados.0.smithi089.stdout:543: set_chunk oid 44 target oid 44 low_tier 2024-04-22T18:39:02.533 INFO:tasks.rados.rados.0.smithi089.stdout:543: set_chunk oid 44 offset: 1445888 length: 751104 target oid smithi08937929-44 low_tier offset: 1445888 snap -1 2024-04-22T18:39:02.533 INFO:tasks.rados.rados.0.smithi089.stdout:544: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.534 INFO:tasks.rados.rados.0.smithi089.stdout:544: set_chunk oid 45 offset: 1615872 length: 383488 target oid smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1615872 snap -1 2024-04-22T18:39:02.534 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.534 INFO:tasks.rados.rados.0.smithi089.stdout:541: finishing set_chunk to oid 42 2024-04-22T18:39:02.534 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 1589248 length: 169472 2024-04-22T18:39:02.534 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-22T18:39:02.534 INFO:tasks.rados.rados.0.smithi089.stdout:536: finishing set_chunk to oid 37 2024-04-22T18:39:02.534 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 771584 length: 1401856 2024-04-22T18:39:02.534 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-22T18:39:02.534 INFO:tasks.rados.rados.0.smithi089.stdout:536: done (15 left) 2024-04-22T18:39:02.534 INFO:tasks.rados.rados.0.smithi089.stdout:541: done (14 left) 2024-04-22T18:39:02.534 INFO:tasks.rados.rados.0.smithi089.stdout:545: set_chunk oid 46 target oid 46 low_tier 2024-04-22T18:39:02.534 INFO:tasks.rados.rados.0.smithi089.stdout:545: set_chunk oid 46 offset: 186880 length: 881664 target oid smithi08937929-46 low_tier offset: 186880 snap -1 2024-04-22T18:39:02.534 INFO:tasks.rados.rados.0.smithi089.stdout:546: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.535 INFO:tasks.rados.rados.0.smithi089.stdout:546: set_chunk oid 47 offset: 50688 length: 520192 target oid smithi08937929-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 50688 snap -1 2024-04-22T18:39:02.535 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.535 INFO:tasks.rados.rados.0.smithi089.stdout:543: finishing set_chunk to oid 44 2024-04-22T18:39:02.535 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 1445888 length: 751104 2024-04-22T18:39:02.535 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-22T18:39:02.535 INFO:tasks.rados.rados.0.smithi089.stdout:543: done (15 left) 2024-04-22T18:39:02.535 INFO:tasks.rados.rados.0.smithi089.stdout:547: set_chunk oid 48 target oid 48 low_tier 2024-04-22T18:39:02.535 INFO:tasks.rados.rados.0.smithi089.stdout:547: set_chunk oid 48 offset: 1953280 length: 677888 target oid smithi08937929-48 low_tier offset: 1953280 snap -1 2024-04-22T18:39:02.535 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.535 INFO:tasks.rados.rados.0.smithi089.stdout:544: finishing set_chunk to oid 45 2024-04-22T18:39:02.535 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1615872 length: 383488 2024-04-22T18:39:02.535 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-22T18:39:02.535 INFO:tasks.rados.rados.0.smithi089.stdout:544: done (15 left) 2024-04-22T18:39:02.535 INFO:tasks.rados.rados.0.smithi089.stdout:548: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.536 INFO:tasks.rados.rados.0.smithi089.stdout:548: set_chunk oid 49 offset: 1988096 length: 24064 target oid smithi08937929-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1988096 snap -1 2024-04-22T18:39:02.536 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.536 INFO:tasks.rados.rados.0.smithi089.stdout:548: finishing set_chunk to oid 49 2024-04-22T18:39:02.536 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1988096 length: 24064 2024-04-22T18:39:02.536 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:39:02.536 INFO:tasks.rados.rados.0.smithi089.stdout:548: done (15 left) 2024-04-22T18:39:02.536 INFO:tasks.rados.rados.0.smithi089.stdout:549: set_chunk oid 50 target oid 50 low_tier 2024-04-22T18:39:02.536 INFO:tasks.rados.rados.0.smithi089.stdout:549: set_chunk oid 50 offset: 1613312 length: 498688 target oid smithi08937929-50 low_tier offset: 1613312 snap -1 2024-04-22T18:39:02.536 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.536 INFO:tasks.rados.rados.0.smithi089.stdout:534: finishing set_chunk to oid 35 2024-04-22T18:39:02.536 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 950272 length: 1204736 2024-04-22T18:39:02.536 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-22T18:39:02.536 INFO:tasks.rados.rados.0.smithi089.stdout:533: finishing set_chunk to oid 34 2024-04-22T18:39:02.536 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 1920000 length: 280576 2024-04-22T18:39:02.536 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-22T18:39:02.537 INFO:tasks.rados.rados.0.smithi089.stdout:529: finishing set_chunk to oid 30 2024-04-22T18:39:02.537 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 915968 length: 217600 2024-04-22T18:39:02.537 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-22T18:39:02.537 INFO:tasks.rados.rados.0.smithi089.stdout:538: finishing set_chunk to oid 39 2024-04-22T18:39:02.537 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1482240 length: 441344 2024-04-22T18:39:02.537 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-22T18:39:02.537 INFO:tasks.rados.rados.0.smithi089.stdout:529: done (15 left) 2024-04-22T18:39:02.537 INFO:tasks.rados.rados.0.smithi089.stdout:533: done (14 left) 2024-04-22T18:39:02.537 INFO:tasks.rados.rados.0.smithi089.stdout:534: done (13 left) 2024-04-22T18:39:02.537 INFO:tasks.rados.rados.0.smithi089.stdout:538: done (12 left) 2024-04-22T18:39:02.537 INFO:tasks.rados.rados.0.smithi089.stdout:550: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.537 INFO:tasks.rados.rados.0.smithi089.stdout:550: set_chunk oid 1 offset: 1046016 length: 729088 target oid smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1046016 snap -1 2024-04-22T18:39:02.537 INFO:tasks.rados.rados.0.smithi089.stdout:551: set_chunk oid 2 target oid 2 low_tier 2024-04-22T18:39:02.537 INFO:tasks.rados.rados.0.smithi089.stdout:551: set_chunk oid 2 offset: 977408 length: 82944 target oid smithi08937929-2 low_tier offset: 977408 snap -1 2024-04-22T18:39:02.537 INFO:tasks.rados.rados.0.smithi089.stdout:552: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.538 INFO:tasks.rados.rados.0.smithi089.stdout:552: set_chunk oid 3 offset: 1256448 length: 749056 target oid smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1256448 snap -1 2024-04-22T18:39:02.538 INFO:tasks.rados.rados.0.smithi089.stdout:553: set_chunk oid 4 target oid 4 low_tier 2024-04-22T18:39:02.538 INFO:tasks.rados.rados.0.smithi089.stdout:553: set_chunk oid 4 offset: 260608 length: 253440 target oid smithi08937929-4 low_tier offset: 260608 snap -1 2024-04-22T18:39:02.538 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.538 INFO:tasks.rados.rados.0.smithi089.stdout:540: finishing set_chunk to oid 41 2024-04-22T18:39:02.538 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 249344 length: 262144 2024-04-22T18:39:02.538 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-22T18:39:02.538 INFO:tasks.rados.rados.0.smithi089.stdout:545: finishing set_chunk to oid 46 2024-04-22T18:39:02.538 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 186880 length: 881664 2024-04-22T18:39:02.538 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-22T18:39:02.538 INFO:tasks.rados.rados.0.smithi089.stdout:540: done (15 left) 2024-04-22T18:39:02.538 INFO:tasks.rados.rados.0.smithi089.stdout:545: done (14 left) 2024-04-22T18:39:02.538 INFO:tasks.rados.rados.0.smithi089.stdout:554: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.538 INFO:tasks.rados.rados.0.smithi089.stdout:554: set_chunk oid 5 offset: 89600 length: 1742848 target oid smithi08937929-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 89600 snap -1 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout:555: set_chunk oid 6 target oid 6 low_tier 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout:555: set_chunk oid 6 offset: 1185280 length: 30208 target oid smithi08937929-6 low_tier offset: 1185280 snap -1 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout:549: finishing set_chunk to oid 50 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1613312 length: 498688 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout:553: finishing set_chunk to oid 4 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 260608 length: 253440 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout:549: done (15 left) 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout:553: done (14 left) 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout:556: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout:556: set_chunk oid 7 offset: 669184 length: 400384 target oid smithi08937929-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 669184 snap -1 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout:557: set_chunk oid 8 target oid 8 low_tier 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout:557: set_chunk oid 8 offset: 173568 length: 645632 target oid smithi08937929-8 low_tier offset: 173568 snap -1 2024-04-22T18:39:02.539 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.540 INFO:tasks.rados.rados.0.smithi089.stdout:554: finishing set_chunk to oid 5 2024-04-22T18:39:02.540 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 89600 length: 1742848 2024-04-22T18:39:02.540 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-22T18:39:02.540 INFO:tasks.rados.rados.0.smithi089.stdout:554: done (15 left) 2024-04-22T18:39:02.540 INFO:tasks.rados.rados.0.smithi089.stdout:558: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.540 INFO:tasks.rados.rados.0.smithi089.stdout:558: set_chunk oid 9 offset: 232960 length: 1597440 target oid smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 232960 snap -1 2024-04-22T18:39:02.540 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.540 INFO:tasks.rados.rados.0.smithi089.stdout:556: finishing set_chunk to oid 7 2024-04-22T18:39:02.540 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 669184 length: 400384 2024-04-22T18:39:02.540 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-22T18:39:02.540 INFO:tasks.rados.rados.0.smithi089.stdout:556: done (15 left) 2024-04-22T18:39:02.540 INFO:tasks.rados.rados.0.smithi089.stdout:559: set_chunk oid 10 target oid 10 low_tier 2024-04-22T18:39:02.540 INFO:tasks.rados.rados.0.smithi089.stdout:559: set_chunk oid 10 offset: 1931264 length: 68608 target oid smithi08937929-10 low_tier offset: 1931264 snap -1 2024-04-22T18:39:02.540 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.548 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:02.548 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:02.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:02.548 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:02.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.548 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:02.548 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:02.548 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.806055325999999, 2024-04-22T18:39:02.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30520528099999999, 2024-04-22T18:39:02.548 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.548 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.548 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.548 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 35 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741852+0000", 2024-04-22T18:39:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741927+0000", 2024-04-22T18:39:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5123000000000006e-05 2024-04-22T18:39:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741953+0000", 2024-04-22T18:39:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5378999999999999e-05 2024-04-22T18:39:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741977+0000", 2024-04-22T18:39:02.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4431999999999999e-05 2024-04-22T18:39:02.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844243+0000", 2024-04-22T18:39:02.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102265525 2024-04-22T18:39:02.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844330+0000", 2024-04-22T18:39:02.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7417999999999995e-05 2024-04-22T18:39:02.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844470+0000", 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000140197 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944767+0000", 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100296409 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.046968+0000", 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10220148499999999 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047018+0000", 2024-04-22T18:39:02.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9206000000000003e-05 2024-04-22T18:39:02.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047059+0000", 2024-04-22T18:39:02.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1496999999999999e-05 2024-04-22T18:39:02.553 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.553 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.553 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.553 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:02.553 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:02.553 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.805924802, 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:39:02.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:39:02.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:39:02.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:39:02.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:39:02.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:39:02.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:39:02.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:39:02.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:39:02.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:39:02.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:39:02.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:39:02.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:39:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:39:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:39:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:39:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:39:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:39:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.805885110999998, 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:02.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:02.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:39:02.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:02.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:39:02.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:39:02.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:39:02.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:39:02.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:39:02.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:39:02.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:39:02.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:39:02.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:39:02.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:39:02.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:39:02.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:39:02.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.561704064000001, 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402419109, 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.564 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 68 2024-04-22T18:39:02.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.565 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:02.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:02.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986204+0000", 2024-04-22T18:39:02.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:02.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986208+0000", 2024-04-22T18:39:02.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8149999999999999e-06 2024-04-22T18:39:02.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986214+0000", 2024-04-22T18:39:02.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.604e-06 2024-04-22T18:39:02.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986218+0000", 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6689999999999999e-06 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986833+0000", 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000615648 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986891+0000", 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7915999999999999e-05 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986981+0000", 2024-04-22T18:39:02.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759999999999994e-05 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.987584+0000", 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060270199999999999 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388527+0000", 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40094366999999997 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388583+0000", 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5383000000000002e-05 2024-04-22T18:39:02.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388624+0000", 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501999999999997e-05 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.561688275000002, 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.569 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.570 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.570 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.570 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:39:02.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.570 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:02.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:02.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:39:02.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:02.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:39:02.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:39:02.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:39:02.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:39:02.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:39:02.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:39:02.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:39:02.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:39:02.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:39:02.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:39:02.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:39:02.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:39:02.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:39:02.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:39:02.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:39:02.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:39:02.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:39:02.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:39:02.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:39:02.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:39:02.574 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.574 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.574 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.445295265999999, 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305563733, 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 87 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102613+0000", 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102668+0000", 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4669000000000001e-05 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102680+0000", 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2026e-05 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102687+0000", 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8339999999999999e-06 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405644+0000", 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30295623199999999 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405743+0000", 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.98e-05 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405868+0000", 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012457899999999999 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407047+0000", 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011791449999999999 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408088+0000", 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010411680000000001 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408135+0000", 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6255999999999999e-05 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408178+0000", 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2877000000000003e-05 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.445215337, 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30560089000000001, 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 88 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:02.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:02.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102693+0000", 2024-04-22T18:39:02.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:02.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102698+0000", 2024-04-22T18:39:02.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9699999999999998e-06 2024-04-22T18:39:02.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102708+0000", 2024-04-22T18:39:02.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.006e-05 2024-04-22T18:39:02.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102715+0000", 2024-04-22T18:39:02.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6599999999999995e-06 2024-04-22T18:39:02.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406637+0000", 2024-04-22T18:39:02.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392130099999998 2024-04-22T18:39:02.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406696+0000", 2024-04-22T18:39:02.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9066e-05 2024-04-22T18:39:02.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406784+0000", 2024-04-22T18:39:02.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8194000000000003e-05 2024-04-22T18:39:02.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407447+0000", 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066295700000000004 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408237+0000", 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079032600000000005 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408270+0000", 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3157000000000001e-05 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408295+0000", 2024-04-22T18:39:02.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4151999999999998e-05 2024-04-22T18:39:02.585 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.585 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.585 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.585 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:02.585 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:02.585 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.693983212999999, 2024-04-22T18:39:02.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30533819600000001, 2024-04-22T18:39:02.585 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.585 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.585 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.585 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.585 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.586 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 106 2024-04-22T18:39:02.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.586 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:02.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:02.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853925+0000", 2024-04-22T18:39:02.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:02.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853958+0000", 2024-04-22T18:39:02.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2536000000000002e-05 2024-04-22T18:39:02.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853964+0000", 2024-04-22T18:39:02.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6239999999999999e-06 2024-04-22T18:39:02.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853967+0000", 2024-04-22T18:39:02.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9700000000000001e-06 2024-04-22T18:39:02.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855144+0000", 2024-04-22T18:39:02.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001176774 2024-04-22T18:39:02.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855225+0000", 2024-04-22T18:39:02.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0697000000000003e-05 2024-04-22T18:39:02.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855353+0000", 2024-04-22T18:39:02.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012788 2024-04-22T18:39:02.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058003+0000", 2024-04-22T18:39:02.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202649939 2024-04-22T18:39:02.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159184+0000", 2024-04-22T18:39:02.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101181058 2024-04-22T18:39:02.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159238+0000", 2024-04-22T18:39:02.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3967e-05 2024-04-22T18:39:02.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159264+0000", 2024-04-22T18:39:02.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6237999999999998e-05 2024-04-22T18:39:02.590 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.590 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.590 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.591 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:02.591 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:02.591 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.693936362999999, 2024-04-22T18:39:02.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30544294500000002, 2024-04-22T18:39:02.591 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.591 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.591 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.591 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.591 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.591 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 107 2024-04-22T18:39:02.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.591 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:02.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:02.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853972+0000", 2024-04-22T18:39:02.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:02.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853975+0000", 2024-04-22T18:39:02.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0939999999999999e-06 2024-04-22T18:39:02.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853978+0000", 2024-04-22T18:39:02.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:39:02.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853981+0000", 2024-04-22T18:39:02.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.858e-06 2024-04-22T18:39:02.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855831+0000", 2024-04-22T18:39:02.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018503689999999999 2024-04-22T18:39:02.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855932+0000", 2024-04-22T18:39:02.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010071 2024-04-22T18:39:02.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856029+0000", 2024-04-22T18:39:02.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6751999999999998e-05 2024-04-22T18:39:02.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058013+0000", 2024-04-22T18:39:02.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20198412600000001 2024-04-22T18:39:02.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159326+0000", 2024-04-22T18:39:02.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101312969 2024-04-22T18:39:02.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159376+0000", 2024-04-22T18:39:02.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9968e-05 2024-04-22T18:39:02.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159416+0000", 2024-04-22T18:39:02.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9641999999999998e-05 2024-04-22T18:39:02.596 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.596 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.596 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.596 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:02.596 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:02.596 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.693847967, 2024-04-22T18:39:02.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30548257499999998, 2024-04-22T18:39:02.596 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.596 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.597 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.597 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.597 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.597 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 108 2024-04-22T18:39:02.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.597 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:02.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:02.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:02.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:02.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854064+0000", 2024-04-22T18:39:02.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5820000000000002e-06 2024-04-22T18:39:02.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854080+0000", 2024-04-22T18:39:02.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5379999999999998e-05 2024-04-22T18:39:02.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854084+0000", 2024-04-22T18:39:02.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.963e-06 2024-04-22T18:39:02.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856413+0000", 2024-04-22T18:39:02.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023298070000000001 2024-04-22T18:39:02.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856494+0000", 2024-04-22T18:39:02.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0863000000000002e-05 2024-04-22T18:39:02.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856624+0000", 2024-04-22T18:39:02.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129799 2024-04-22T18:39:02.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058947+0000", 2024-04-22T18:39:02.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20232328899999999 2024-04-22T18:39:02.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159468+0000", 2024-04-22T18:39:02.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100520597 2024-04-22T18:39:02.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159508+0000", 2024-04-22T18:39:02.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9830000000000003e-05 2024-04-22T18:39:02.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159544+0000", 2024-04-22T18:39:02.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5944e-05 2024-04-22T18:39:02.601 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.602 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.602 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.602 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:02.602 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:02.602 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.693803474999999, 2024-04-22T18:39:02.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30556502499999999, 2024-04-22T18:39:02.602 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.602 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.602 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.602 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.602 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.602 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:39:02.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.603 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:02.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:02.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854105+0000", 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854175+0000", 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0178000000000005e-05 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854191+0000", 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5965999999999999e-05 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854196+0000", 2024-04-22T18:39:02.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.42e-06 2024-04-22T18:39:02.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956526+0000", 2024-04-22T18:39:02.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10233094299999999 2024-04-22T18:39:02.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956634+0000", 2024-04-22T18:39:02.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107398 2024-04-22T18:39:02.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956730+0000", 2024-04-22T18:39:02.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6217000000000004e-05 2024-04-22T18:39:02.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058955+0000", 2024-04-22T18:39:02.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102224749 2024-04-22T18:39:02.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159596+0000", 2024-04-22T18:39:02.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100641297 2024-04-22T18:39:02.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159637+0000", 2024-04-22T18:39:02.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874000000000002e-05 2024-04-22T18:39:02.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159671+0000", 2024-04-22T18:39:02.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3603999999999999e-05 2024-04-22T18:39:02.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.607 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.607 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:02.607 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:02.607 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.693709628000001, 2024-04-22T18:39:02.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30683048800000001, 2024-04-22T18:39:02.608 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.608 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.608 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110 2024-04-22T18:39:02.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.608 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:02.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:02.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:02.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:02.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854202+0000", 2024-04-22T18:39:02.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:39:02.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854206+0000", 2024-04-22T18:39:02.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5870000000000001e-06 2024-04-22T18:39:02.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854208+0000", 2024-04-22T18:39:02.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7199999999999998e-06 2024-04-22T18:39:02.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057285+0000", 2024-04-22T18:39:02.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20307620800000001 2024-04-22T18:39:02.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057414+0000", 2024-04-22T18:39:02.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012896899999999999 2024-04-22T18:39:02.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057588+0000", 2024-04-22T18:39:02.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000174802 2024-04-22T18:39:02.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159429+0000", 2024-04-22T18:39:02.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101840783 2024-04-22T18:39:02.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160947+0000", 2024-04-22T18:39:02.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001518152 2024-04-22T18:39:02.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160992+0000", 2024-04-22T18:39:02.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4725000000000003e-05 2024-04-22T18:39:02.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161030+0000", 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7789e-05 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.693698446999999, 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.616 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:39:02.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.617 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:02.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:02.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:39:02.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:02.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:39:02.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:39:02.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:39:02.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:39:02.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:39:02.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:39:02.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:39:02.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:39:02.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:39:02.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:39:02.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:39:02.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:39:02.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:39:02.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:39:02.621 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.621 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.621 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.621 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:02.621 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:02.621 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.693682789, 2024-04-22T18:39:02.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:39:02.622 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.622 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.622 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.622 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.622 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.622 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:39:02.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.622 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:02.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:02.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:02.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:02.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:39:02.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:39:02.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:39:02.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:39:02.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:39:02.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:39:02.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:39:02.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:39:02.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:39:02.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:39:02.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:39:02.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:39:02.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:39:02.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:39:02.626 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.626 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.627 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.627 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:02.627 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:02.627 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.693631626999998, 2024-04-22T18:39:02.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:39:02.627 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.627 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.627 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.627 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.627 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.627 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:39:02.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.627 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:02.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:02.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:02.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:02.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:39:02.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:39:02.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:39:02.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:39:02.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:39:02.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:39:02.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:39:02.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:39:02.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:39:02.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:39:02.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:39:02.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:39:02.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:39:02.632 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.632 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.632 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.632 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:02.632 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:02.632 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.693616403, 2024-04-22T18:39:02.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:39:02.632 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:02.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:39:02.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:02.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:39:02.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:39:02.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:39:02.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:39:02.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:39:02.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:39:02.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:39:02.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:39:02.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:39:02.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:39:02.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:39:02.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:39:02.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:39:02.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:39:02.637 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.637 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.637 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.693556825999998, 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:39:02.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:39:02.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:39:02.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:39:02.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:39:02.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:39:02.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:39:02.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:39:02.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:39:02.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:39:02.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:39:02.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:39:02.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:39:02.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:39:02.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:39:02.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:39:02.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:39:02.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:39:02.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.693474510000001, 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.643 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.644 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:39:02.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.644 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:02.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:02.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:39:02.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:02.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:39:02.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:39:02.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:39:02.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:39:02.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:39:02.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:39:02.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:39:02.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:39:02.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:39:02.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:39:02.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:39:02.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:39:02.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:39:02.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:39:02.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:39:02.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:39:02.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:39:02.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:39:02.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:39:02.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:39:02.648 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.648 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.648 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.648 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:02.649 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:02.649 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.693464354, 2024-04-22T18:39:02.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:39:02.649 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.649 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.649 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.649 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.649 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.649 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:39:02.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.649 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:02.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:02.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:39:02.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:02.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:39:02.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:39:02.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:39:02.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:39:02.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:39:02.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:39:02.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:39:02.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:39:02.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:39:02.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:39:02.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:39:02.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:39:02.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:39:02.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:39:02.654 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.654 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.654 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.654 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:161 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:39:02.654 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:02.654 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.092970062999999, 2024-04-22T18:39:02.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40239169000000002, 2024-04-22T18:39:02.654 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:02.654 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:02.654 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:02.654 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:02.654 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 161 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:02.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454938+0000", 2024-04-22T18:39:02.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:02.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:02.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454982+0000", 2024-04-22T18:39:02.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3470999999999998e-05 2024-04-22T18:39:02.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:02.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454988+0000", 2024-04-22T18:39:02.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7640000000000002e-06 2024-04-22T18:39:02.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555054+0000", 2024-04-22T18:39:02.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006607300000001 2024-04-22T18:39:02.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:02.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555607+0000", 2024-04-22T18:39:02.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055319500000000003 2024-04-22T18:39:02.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:02.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555689+0000", 2024-04-22T18:39:02.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1817999999999995e-05 2024-04-22T18:39:02.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:02.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555804+0000", 2024-04-22T18:39:02.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115346 2024-04-22T18:39:02.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:02.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556554+0000", 2024-04-22T18:39:02.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000750102 2024-04-22T18:39:02.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:02.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857265+0000", 2024-04-22T18:39:02.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30071122700000003 2024-04-22T18:39:02.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:02.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857302+0000", 2024-04-22T18:39:02.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7069e-05 2024-04-22T18:39:02.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:02.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:02.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:02.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857331+0000", 2024-04-22T18:39:02.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8291999999999999e-05 2024-04-22T18:39:02.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.659 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:02.659 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:02.659 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:02.661 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:02.666 INFO:tasks.rados.rados.0.smithi089.stdout:530: finishing set_chunk to oid 31 2024-04-22T18:39:02.666 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 104960 length: 37376 2024-04-22T18:39:02.667 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-22T18:39:02.667 INFO:tasks.rados.rados.0.smithi089.stdout:530: done (15 left) 2024-04-22T18:39:02.667 INFO:tasks.rados.rados.0.smithi089.stdout:560: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.667 INFO:tasks.rados.rados.0.smithi089.stdout:560: set_chunk oid 11 offset: 796672 length: 494080 target oid smithi08937929-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 796672 snap -1 2024-04-22T18:39:02.667 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.667 INFO:tasks.rados.rados.0.smithi089.stdout:551: finishing set_chunk to oid 2 2024-04-22T18:39:02.667 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 977408 length: 82944 2024-04-22T18:39:02.667 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-22T18:39:02.667 INFO:tasks.rados.rados.0.smithi089.stdout:532: finishing set_chunk to oid 33 2024-04-22T18:39:02.667 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 295936 length: 1655296 2024-04-22T18:39:02.667 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-22T18:39:02.667 INFO:tasks.rados.rados.0.smithi089.stdout:532: done (15 left) 2024-04-22T18:39:02.667 INFO:tasks.rados.rados.0.smithi089.stdout:551: done (14 left) 2024-04-22T18:39:02.667 INFO:tasks.rados.rados.0.smithi089.stdout:561: set_chunk oid 12 target oid 12 low_tier 2024-04-22T18:39:02.668 INFO:tasks.rados.rados.0.smithi089.stdout:561: set_chunk oid 12 offset: 1060352 length: 392704 target oid smithi08937929-12 low_tier offset: 1060352 snap -1 2024-04-22T18:39:02.668 INFO:tasks.rados.rados.0.smithi089.stdout:562: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.668 INFO:tasks.rados.rados.0.smithi089.stdout:562: set_chunk oid 13 offset: 411136 length: 722944 target oid smithi08937929-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 411136 snap -1 2024-04-22T18:39:02.668 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.668 INFO:tasks.rados.rados.0.smithi089.stdout:547: finishing set_chunk to oid 48 2024-04-22T18:39:02.668 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1953280 length: 677888 2024-04-22T18:39:02.668 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-22T18:39:02.668 INFO:tasks.rados.rados.0.smithi089.stdout:547: done (15 left) 2024-04-22T18:39:02.668 INFO:tasks.rados.rados.0.smithi089.stdout:563: set_chunk oid 14 target oid 14 low_tier 2024-04-22T18:39:02.668 INFO:tasks.rados.rados.0.smithi089.stdout:563: set_chunk oid 14 offset: 541696 length: 131584 target oid smithi08937929-14 low_tier offset: 541696 snap -1 2024-04-22T18:39:02.668 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.668 INFO:tasks.rados.rados.0.smithi089.stdout:560: finishing set_chunk to oid 11 2024-04-22T18:39:02.668 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 796672 length: 494080 2024-04-22T18:39:02.668 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-22T18:39:02.668 INFO:tasks.rados.rados.0.smithi089.stdout:560: done (15 left) 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout:564: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout:564: set_chunk oid 15 offset: 960000 length: 64000 target oid smithi08937929-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 960000 snap -1 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout:539: finishing set_chunk to oid 40 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 1825280 length: 768512 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout:539: done (15 left) 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout:565: set_chunk oid 16 target oid 16 low_tier 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout:565: set_chunk oid 16 offset: 349184 length: 482816 target oid smithi08937929-16 low_tier offset: 349184 snap -1 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout:542: finishing set_chunk to oid 43 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 818688 length: 1377280 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout:542: done (15 left) 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout:566: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.669 INFO:tasks.rados.rados.0.smithi089.stdout:566: set_chunk oid 17 offset: 1953280 length: 57856 target oid smithi08937929-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1953280 snap -1 2024-04-22T18:39:02.670 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.670 INFO:tasks.rados.rados.0.smithi089.stdout:535: finishing set_chunk to oid 36 2024-04-22T18:39:02.670 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 394752 length: 1797632 2024-04-22T18:39:02.670 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-22T18:39:02.670 INFO:tasks.rados.rados.0.smithi089.stdout:535: done (15 left) 2024-04-22T18:39:02.670 INFO:tasks.rados.rados.0.smithi089.stdout:567: set_chunk oid 18 target oid 18 low_tier 2024-04-22T18:39:02.670 INFO:tasks.rados.rados.0.smithi089.stdout:567: set_chunk oid 18 offset: 392192 length: 890880 target oid smithi08937929-18 low_tier offset: 392192 snap -1 2024-04-22T18:39:02.670 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.670 INFO:tasks.rados.rados.0.smithi089.stdout:550: finishing set_chunk to oid 1 2024-04-22T18:39:02.670 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1046016 length: 729088 2024-04-22T18:39:02.670 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-22T18:39:02.670 INFO:tasks.rados.rados.0.smithi089.stdout:550: done (15 left) 2024-04-22T18:39:02.670 INFO:tasks.rados.rados.0.smithi089.stdout:568: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.670 INFO:tasks.rados.rados.0.smithi089.stdout:568: set_chunk oid 19 offset: 968192 length: 667136 target oid smithi08937929-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 968192 snap -1 2024-04-22T18:39:02.671 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.671 INFO:tasks.rados.rados.0.smithi089.stdout:567: finishing set_chunk to oid 18 2024-04-22T18:39:02.671 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 392192 length: 890880 2024-04-22T18:39:02.671 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-22T18:39:02.671 INFO:tasks.rados.rados.0.smithi089.stdout:567: done (15 left) 2024-04-22T18:39:02.671 INFO:tasks.rados.rados.0.smithi089.stdout:569: set_chunk oid 20 target oid 20 low_tier 2024-04-22T18:39:02.671 INFO:tasks.rados.rados.0.smithi089.stdout:569: set_chunk oid 20 offset: 309760 length: 54784 target oid smithi08937929-20 low_tier offset: 309760 snap -1 2024-04-22T18:39:02.671 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.671 INFO:tasks.rados.rados.0.smithi089.stdout:546: finishing set_chunk to oid 47 2024-04-22T18:39:02.671 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 50688 length: 520192 2024-04-22T18:39:02.671 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-22T18:39:02.671 INFO:tasks.rados.rados.0.smithi089.stdout:546: done (15 left) 2024-04-22T18:39:02.671 INFO:tasks.rados.rados.0.smithi089.stdout:570: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.671 INFO:tasks.rados.rados.0.smithi089.stdout:570: set_chunk oid 21 offset: 1265152 length: 1035264 target oid smithi08937929-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1265152 snap -1 2024-04-22T18:39:02.671 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.672 INFO:tasks.rados.rados.0.smithi089.stdout:555: finishing set_chunk to oid 6 2024-04-22T18:39:02.672 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1185280 length: 30208 2024-04-22T18:39:02.672 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-22T18:39:02.672 INFO:tasks.rados.rados.0.smithi089.stdout:555: done (15 left) 2024-04-22T18:39:02.672 INFO:tasks.rados.rados.0.smithi089.stdout:571: set_chunk oid 22 target oid 22 low_tier 2024-04-22T18:39:02.672 INFO:tasks.rados.rados.0.smithi089.stdout:571: set_chunk oid 22 offset: 726016 length: 947200 target oid smithi08937929-22 low_tier offset: 726016 snap -1 2024-04-22T18:39:02.672 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.672 INFO:tasks.rados.rados.0.smithi089.stdout:569: finishing set_chunk to oid 20 2024-04-22T18:39:02.672 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 309760 length: 54784 2024-04-22T18:39:02.672 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-22T18:39:02.672 INFO:tasks.rados.rados.0.smithi089.stdout:562: finishing set_chunk to oid 13 2024-04-22T18:39:02.672 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 411136 length: 722944 2024-04-22T18:39:02.672 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-22T18:39:02.672 INFO:tasks.rados.rados.0.smithi089.stdout:562: done (15 left) 2024-04-22T18:39:02.672 INFO:tasks.rados.rados.0.smithi089.stdout:569: done (14 left) 2024-04-22T18:39:02.673 INFO:tasks.rados.rados.0.smithi089.stdout:572: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.673 INFO:tasks.rados.rados.0.smithi089.stdout:572: set_chunk oid 23 offset: 818688 length: 879616 target oid smithi08937929-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 818688 snap -1 2024-04-22T18:39:02.673 INFO:tasks.rados.rados.0.smithi089.stdout:573: set_chunk oid 24 target oid 24 low_tier 2024-04-22T18:39:02.673 INFO:tasks.rados.rados.0.smithi089.stdout:573: set_chunk oid 24 offset: 1235968 length: 402944 target oid smithi08937929-24 low_tier offset: 1235968 snap -1 2024-04-22T18:39:02.673 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.673 INFO:tasks.rados.rados.0.smithi089.stdout:570: finishing set_chunk to oid 21 2024-04-22T18:39:02.673 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1265152 length: 1035264 2024-04-22T18:39:02.673 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-22T18:39:02.673 INFO:tasks.rados.rados.0.smithi089.stdout:570: done (15 left) 2024-04-22T18:39:02.673 INFO:tasks.rados.rados.0.smithi089.stdout:574: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.673 INFO:tasks.rados.rados.0.smithi089.stdout:574: set_chunk oid 25 offset: 186880 length: 1035264 target oid smithi08937929-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 186880 snap -1 2024-04-22T18:39:02.673 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.673 INFO:tasks.rados.rados.0.smithi089.stdout:564: finishing set_chunk to oid 15 2024-04-22T18:39:02.673 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 960000 length: 64000 2024-04-22T18:39:02.673 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-22T18:39:02.674 INFO:tasks.rados.rados.0.smithi089.stdout:564: done (15 left) 2024-04-22T18:39:02.674 INFO:tasks.rados.rados.0.smithi089.stdout:575: set_chunk oid 26 target oid 26 low_tier 2024-04-22T18:39:02.674 INFO:tasks.rados.rados.0.smithi089.stdout:575: set_chunk oid 26 offset: 287744 length: 1893888 target oid smithi08937929-26 low_tier offset: 287744 snap -1 2024-04-22T18:39:02.674 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.674 INFO:tasks.rados.rados.0.smithi089.stdout:574: finishing set_chunk to oid 25 2024-04-22T18:39:02.674 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 186880 length: 1035264 2024-04-22T18:39:02.674 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-22T18:39:02.674 INFO:tasks.rados.rados.0.smithi089.stdout:574: done (15 left) 2024-04-22T18:39:02.674 INFO:tasks.rados.rados.0.smithi089.stdout:576: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.674 INFO:tasks.rados.rados.0.smithi089.stdout:576: set_chunk oid 27 offset: 637440 length: 621056 target oid smithi08937929-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 637440 snap -1 2024-04-22T18:39:02.674 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.675 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:02.661+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:02.727 INFO:tasks.rados.rados.0.smithi089.stdout:552: finishing set_chunk to oid 3 2024-04-22T18:39:02.727 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1256448 length: 749056 2024-04-22T18:39:02.727 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-22T18:39:02.727 INFO:tasks.rados.rados.0.smithi089.stdout:552: done (15 left) 2024-04-22T18:39:02.727 INFO:tasks.rados.rados.0.smithi089.stdout:576 retry set_chunk ! 2024-04-22T18:39:02.727 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 15 2024-04-22T18:39:02.727 INFO:tasks.rados.rados.0.smithi089.stdout:558: finishing set_chunk to oid 9 2024-04-22T18:39:02.727 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 232960 length: 1597440 2024-04-22T18:39:02.727 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-22T18:39:02.727 INFO:tasks.rados.rados.0.smithi089.stdout:558: done (14 left) 2024-04-22T18:39:02.727 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 14 2024-04-22T18:39:02.727 INFO:tasks.rados.rados.0.smithi089.stdout:557: finishing set_chunk to oid 8 2024-04-22T18:39:02.728 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 173568 length: 645632 2024-04-22T18:39:02.728 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-22T18:39:02.728 INFO:tasks.rados.rados.0.smithi089.stdout:527: finishing set_chunk to oid 28 2024-04-22T18:39:02.728 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 15 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-22T18:39:02.728 INFO:tasks.rados.rados.0.smithi089.stdout:527: done (13 left) 2024-04-22T18:39:02.728 INFO:tasks.rados.rados.0.smithi089.stdout:557: done (12 left) 2024-04-22T18:39:02.728 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 12 2024-04-22T18:39:02.728 INFO:tasks.rados.rados.0.smithi089.stdout:561: finishing set_chunk to oid 12 2024-04-22T18:39:02.728 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1060352 length: 392704 2024-04-22T18:39:02.728 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-22T18:39:02.728 INFO:tasks.rados.rados.0.smithi089.stdout:568: finishing set_chunk to oid 19 2024-04-22T18:39:02.728 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 968192 length: 667136 2024-04-22T18:39:02.728 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-22T18:39:02.728 INFO:tasks.rados.rados.0.smithi089.stdout:561: done (11 left) 2024-04-22T18:39:02.729 INFO:tasks.rados.rados.0.smithi089.stdout:568: done (10 left) 2024-04-22T18:39:02.729 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 10 2024-04-22T18:39:02.729 INFO:tasks.rados.rados.0.smithi089.stdout:565: finishing set_chunk to oid 16 2024-04-22T18:39:02.729 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 349184 length: 482816 2024-04-22T18:39:02.729 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-22T18:39:02.729 INFO:tasks.rados.rados.0.smithi089.stdout:565: done (9 left) 2024-04-22T18:39:02.729 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 9 2024-04-22T18:39:02.729 INFO:tasks.rados.rados.0.smithi089.stdout:566: finishing set_chunk to oid 17 2024-04-22T18:39:02.729 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-22T18:39:02.729 INFO:tasks.rados.rados.0.smithi089.stdout:566: done (8 left) 2024-04-22T18:39:02.729 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 8 2024-04-22T18:39:02.729 INFO:tasks.rados.rados.0.smithi089.stdout:572: finishing set_chunk to oid 23 2024-04-22T18:39:02.729 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 818688 length: 879616 2024-04-22T18:39:02.729 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-22T18:39:02.730 INFO:tasks.rados.rados.0.smithi089.stdout:572: done (7 left) 2024-04-22T18:39:02.730 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 7 2024-04-22T18:39:02.730 INFO:tasks.rados.rados.0.smithi089.stdout:571: finishing set_chunk to oid 22 2024-04-22T18:39:02.730 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 726016 length: 947200 2024-04-22T18:39:02.730 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-22T18:39:02.730 INFO:tasks.rados.rados.0.smithi089.stdout:571: done (6 left) 2024-04-22T18:39:02.730 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 6 2024-04-22T18:39:02.730 INFO:tasks.rados.rados.0.smithi089.stdout:575: finishing set_chunk to oid 26 2024-04-22T18:39:02.730 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 287744 length: 1893888 2024-04-22T18:39:02.730 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-22T18:39:02.730 INFO:tasks.rados.rados.0.smithi089.stdout:575: done (5 left) 2024-04-22T18:39:02.730 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 5 2024-04-22T18:39:02.761 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:02.762+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:02.776 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:02.777 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:02.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:02.777 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:713 4.3 4:ca46fde9:::smithi08937929-14:head [assert-version v3,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:02.777 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.627086+0000", 2024-04-22T18:39:02.777 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.15137599400000001, 2024-04-22T18:39:02.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.15140255999999999, 2024-04-22T18:39:02.777 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:02.777 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:02.777 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:02.777 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:02.777 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:02.777 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 713 2024-04-22T18:39:02.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:02.777 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:02.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:02.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:02.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627086+0000", 2024-04-22T18:39:02.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:02.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:02.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:02.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:02.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627086+0000", 2024-04-22T18:39:02.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:02.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:02.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:02.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:02.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627085+0000", 2024-04-22T18:39:02.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:02.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627098+0000", 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2869e-05 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627111+0000", 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2978e-05 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627117+0000", 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7209999999999997e-06 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:02.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:02.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:02.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627152+0000", 2024-04-22T18:39:02.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5052000000000003e-05 2024-04-22T18:39:02.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:02.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:02.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:02.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627204+0000", 2024-04-22T18:39:02.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1214000000000001e-05 2024-04-22T18:39:02.780 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:02.780 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:02.780 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:02.780 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:02.780 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:39:02.780 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 1 2024-04-22T18:39:02.780 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:02.785 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:02.826 INFO:tasks.rados.rados.0.smithi089.stdout:573: finishing set_chunk to oid 24 2024-04-22T18:39:02.826 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 1235968 length: 402944 2024-04-22T18:39:02.826 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-22T18:39:02.826 INFO:tasks.rados.rados.0.smithi089.stdout:576: finishing set_chunk to oid 27 2024-04-22T18:39:02.826 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 637440 length: 621056 2024-04-22T18:39:02.826 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-22T18:39:02.826 INFO:tasks.rados.rados.0.smithi089.stdout:573: done (4 left) 2024-04-22T18:39:02.826 INFO:tasks.rados.rados.0.smithi089.stdout:576: done (3 left) 2024-04-22T18:39:02.827 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 3 2024-04-22T18:39:02.827 INFO:tasks.rados.rados.0.smithi089.stdout:559: finishing set_chunk to oid 10 2024-04-22T18:39:02.827 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-22T18:39:02.827 INFO:tasks.rados.rados.0.smithi089.stdout:559: done (2 left) 2024-04-22T18:39:02.827 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 2 2024-04-22T18:39:02.869 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:02.870+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:02.927 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:02.927 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:02.927 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:02.927 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:02.927 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:02.933 INFO:tasks.rados.rados.0.smithi089.stdout:537: finishing set_chunk to oid 38 2024-04-22T18:39:02.933 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-22T18:39:02.933 INFO:tasks.rados.rados.0.smithi089.stdout:537: done (1 left) 2024-04-22T18:39:02.933 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 1 2024-04-22T18:39:02.933 INFO:tasks.rados.rados.0.smithi089.stdout:563: finishing set_chunk to oid 14 2024-04-22T18:39:02.933 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-22T18:39:02.934 INFO:tasks.rados.rados.0.smithi089.stdout:563: done (0 left) 2024-04-22T18:39:02.934 INFO:tasks.rados.rados.0.smithi089.stdout:577: set_chunk oid 28 target oid 28 low_tier 2024-04-22T18:39:02.934 INFO:tasks.rados.rados.0.smithi089.stdout:577: set_chunk oid 28 offset: 878080 length: 951808 target oid smithi08937929-28 low_tier offset: 878080 snap -1 2024-04-22T18:39:02.934 INFO:tasks.rados.rados.0.smithi089.stdout:578: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.934 INFO:tasks.rados.rados.0.smithi089.stdout:578: set_chunk oid 29 offset: 461312 length: 1781248 target oid smithi08937929-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 461312 snap -1 2024-04-22T18:39:02.934 INFO:tasks.rados.rados.0.smithi089.stdout:579: set_chunk oid 30 target oid 30 low_tier 2024-04-22T18:39:02.934 INFO:tasks.rados.rados.0.smithi089.stdout:579: set_chunk oid 30 offset: 1070080 length: 748544 target oid smithi08937929-30 low_tier offset: 1070080 snap -1 2024-04-22T18:39:02.934 INFO:tasks.rados.rados.0.smithi089.stdout:580: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.934 INFO:tasks.rados.rados.0.smithi089.stdout:580: set_chunk oid 31 offset: 966656 length: 1062400 target oid smithi08937929-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 966656 snap -1 2024-04-22T18:39:02.934 INFO:tasks.rados.rados.0.smithi089.stdout:581: set_chunk oid 32 target oid 32 low_tier 2024-04-22T18:39:02.934 INFO:tasks.rados.rados.0.smithi089.stdout:581: set_chunk oid 32 offset: 331264 length: 1492480 target oid smithi08937929-32 low_tier offset: 331264 snap -1 2024-04-22T18:39:02.934 INFO:tasks.rados.rados.0.smithi089.stdout:582: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.934 INFO:tasks.rados.rados.0.smithi089.stdout:582: set_chunk oid 33 offset: 674816 length: 1653760 target oid smithi08937929-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 674816 snap -1 2024-04-22T18:39:02.935 INFO:tasks.rados.rados.0.smithi089.stdout:583: set_chunk oid 34 target oid 34 low_tier 2024-04-22T18:39:02.935 INFO:tasks.rados.rados.0.smithi089.stdout:583: set_chunk oid 34 offset: 405504 length: 1588224 target oid smithi08937929-34 low_tier offset: 405504 snap -1 2024-04-22T18:39:02.935 INFO:tasks.rados.rados.0.smithi089.stdout:584: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.935 INFO:tasks.rados.rados.0.smithi089.stdout:584: set_chunk oid 35 offset: 458240 length: 1418240 target oid smithi08937929-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 458240 snap -1 2024-04-22T18:39:02.935 INFO:tasks.rados.rados.0.smithi089.stdout:585: set_chunk oid 36 target oid 36 low_tier 2024-04-22T18:39:02.935 INFO:tasks.rados.rados.0.smithi089.stdout:585: set_chunk oid 36 offset: 164864 length: 506368 target oid smithi08937929-36 low_tier offset: 164864 snap -1 2024-04-22T18:39:02.935 INFO:tasks.rados.rados.0.smithi089.stdout:586: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.935 INFO:tasks.rados.rados.0.smithi089.stdout:586: set_chunk oid 37 offset: 973824 length: 950784 target oid smithi08937929-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 973824 snap -1 2024-04-22T18:39:02.935 INFO:tasks.rados.rados.0.smithi089.stdout:587: set_chunk oid 38 target oid 38 low_tier 2024-04-22T18:39:02.935 INFO:tasks.rados.rados.0.smithi089.stdout:587: set_chunk oid 38 offset: 556544 length: 994304 target oid smithi08937929-38 low_tier offset: 556544 snap -1 2024-04-22T18:39:02.935 INFO:tasks.rados.rados.0.smithi089.stdout:588: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.935 INFO:tasks.rados.rados.0.smithi089.stdout:588: set_chunk oid 39 offset: 676352 length: 1778688 target oid smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 676352 snap -1 2024-04-22T18:39:02.935 INFO:tasks.rados.rados.0.smithi089.stdout:589: set_chunk oid 40 target oid 40 low_tier 2024-04-22T18:39:02.935 INFO:tasks.rados.rados.0.smithi089.stdout:589: set_chunk oid 40 offset: 1066496 length: 970752 target oid smithi08937929-40 low_tier offset: 1066496 snap -1 2024-04-22T18:39:02.935 INFO:tasks.rados.rados.0.smithi089.stdout:590: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.936 INFO:tasks.rados.rados.0.smithi089.stdout:590: set_chunk oid 41 offset: 1321472 length: 73216 target oid smithi08937929-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1321472 snap -1 2024-04-22T18:39:02.936 INFO:tasks.rados.rados.0.smithi089.stdout:591: set_chunk oid 42 target oid 42 low_tier 2024-04-22T18:39:02.936 INFO:tasks.rados.rados.0.smithi089.stdout:591: set_chunk oid 42 offset: 18944 length: 1910272 target oid smithi08937929-42 low_tier offset: 18944 snap -1 2024-04-22T18:39:02.936 INFO:tasks.rados.rados.0.smithi089.stdout:592: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.936 INFO:tasks.rados.rados.0.smithi089.stdout:592: set_chunk oid 43 offset: 1650688 length: 804864 target oid smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1650688 snap -1 2024-04-22T18:39:02.936 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:02.936 INFO:tasks.rados.rados.0.smithi089.stdout:577: finishing set_chunk to oid 28 2024-04-22T18:39:02.936 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 878080 length: 951808 2024-04-22T18:39:02.936 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-22T18:39:02.936 INFO:tasks.rados.rados.0.smithi089.stdout:583: finishing set_chunk to oid 34 2024-04-22T18:39:02.936 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 405504 length: 1588224 2024-04-22T18:39:02.936 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-22T18:39:02.936 INFO:tasks.rados.rados.0.smithi089.stdout:578: finishing set_chunk to oid 29 2024-04-22T18:39:02.936 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 461312 length: 1781248 2024-04-22T18:39:02.936 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-22T18:39:02.937 INFO:tasks.rados.rados.0.smithi089.stdout:579: finishing set_chunk to oid 30 2024-04-22T18:39:02.937 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 1070080 length: 748544 2024-04-22T18:39:02.937 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-22T18:39:02.937 INFO:tasks.rados.rados.0.smithi089.stdout:581: finishing set_chunk to oid 32 2024-04-22T18:39:02.937 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 331264 length: 1492480 2024-04-22T18:39:02.937 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-22T18:39:02.937 INFO:tasks.rados.rados.0.smithi089.stdout:577: done (15 left) 2024-04-22T18:39:02.937 INFO:tasks.rados.rados.0.smithi089.stdout:578: done (14 left) 2024-04-22T18:39:02.937 INFO:tasks.rados.rados.0.smithi089.stdout:579: done (13 left) 2024-04-22T18:39:02.937 INFO:tasks.rados.rados.0.smithi089.stdout:581: done (12 left) 2024-04-22T18:39:02.937 INFO:tasks.rados.rados.0.smithi089.stdout:583: done (11 left) 2024-04-22T18:39:02.937 INFO:tasks.rados.rados.0.smithi089.stdout:593: set_chunk oid 44 target oid 44 low_tier 2024-04-22T18:39:02.937 INFO:tasks.rados.rados.0.smithi089.stdout:593: set_chunk oid 44 offset: 932864 length: 66560 target oid smithi08937929-44 low_tier offset: 932864 snap -1 2024-04-22T18:39:02.937 INFO:tasks.rados.rados.0.smithi089.stdout:594: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.937 INFO:tasks.rados.rados.0.smithi089.stdout:594: set_chunk oid 45 offset: 280576 length: 2447872 target oid smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 280576 snap -1 2024-04-22T18:39:02.938 INFO:tasks.rados.rados.0.smithi089.stdout:595: set_chunk oid 46 target oid 46 low_tier 2024-04-22T18:39:02.938 INFO:tasks.rados.rados.0.smithi089.stdout:595: set_chunk oid 46 offset: 737792 length: 932864 target oid smithi08937929-46 low_tier offset: 737792 snap -1 2024-04-22T18:39:02.938 INFO:tasks.rados.rados.0.smithi089.stdout:596: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.938 INFO:tasks.rados.rados.0.smithi089.stdout:596: set_chunk oid 47 offset: 1046528 length: 1121280 target oid smithi08937929-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1046528 snap -1 2024-04-22T18:39:02.938 INFO:tasks.rados.rados.0.smithi089.stdout:584: finishing set_chunk to oid 35 2024-04-22T18:39:02.938 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 458240 length: 1418240 2024-04-22T18:39:02.938 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-22T18:39:02.938 INFO:tasks.rados.rados.0.smithi089.stdout:589: finishing set_chunk to oid 40 2024-04-22T18:39:02.938 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 1066496 length: 970752 2024-04-22T18:39:02.939 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-22T18:39:02.939 INFO:tasks.rados.rados.0.smithi089.stdout:590: finishing set_chunk to oid 41 2024-04-22T18:39:02.939 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1321472 length: 73216 2024-04-22T18:39:02.939 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-22T18:39:02.939 INFO:tasks.rados.rados.0.smithi089.stdout:588: finishing set_chunk to oid 39 2024-04-22T18:39:02.939 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 676352 length: 1778688 2024-04-22T18:39:02.939 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-22T18:39:02.939 INFO:tasks.rados.rados.0.smithi089.stdout:584: done (14 left) 2024-04-22T18:39:02.939 INFO:tasks.rados.rados.0.smithi089.stdout:588: done (13 left) 2024-04-22T18:39:02.939 INFO:tasks.rados.rados.0.smithi089.stdout:589: done (12 left) 2024-04-22T18:39:02.939 INFO:tasks.rados.rados.0.smithi089.stdout:590: done (11 left) 2024-04-22T18:39:02.939 INFO:tasks.rados.rados.0.smithi089.stdout:597: set_chunk oid 48 target oid 48 low_tier 2024-04-22T18:39:02.939 INFO:tasks.rados.rados.0.smithi089.stdout:597: set_chunk oid 48 offset: 334336 length: 56832 target oid smithi08937929-48 low_tier offset: 334336 snap -1 2024-04-22T18:39:02.939 INFO:tasks.rados.rados.0.smithi089.stdout:598: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.940 INFO:tasks.rados.rados.0.smithi089.stdout:598: set_chunk oid 49 offset: 509952 length: 1938432 target oid smithi08937929-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 509952 snap -1 2024-04-22T18:39:02.940 INFO:tasks.rados.rados.0.smithi089.stdout:599: set_chunk oid 50 target oid 50 low_tier 2024-04-22T18:39:02.940 INFO:tasks.rados.rados.0.smithi089.stdout:599: set_chunk oid 50 offset: 59904 length: 961536 target oid smithi08937929-50 low_tier offset: 59904 snap -1 2024-04-22T18:39:02.940 INFO:tasks.rados.rados.0.smithi089.stdout:600: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:02.940 INFO:tasks.rados.rados.0.smithi089.stdout:600: set_chunk oid 1 offset: 180224 length: 1463808 target oid smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 180224 snap -1 2024-04-22T18:39:02.940 INFO:tasks.rados.rados.0.smithi089.stdout:601 oid_not_in_use 35 oid_redirect_not_in_use 0 2024-04-22T18:39:02.940 INFO:tasks.rados.rados.0.smithi089.stdout:600 wait for manifest initialization 2024-04-22T18:39:02.940 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 15 2024-04-22T18:39:02.940 INFO:tasks.rados.rados.0.smithi089.stdout:586: finishing set_chunk to oid 37 2024-04-22T18:39:02.940 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 973824 length: 950784 2024-04-22T18:39:02.940 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-22T18:39:02.940 INFO:tasks.rados.rados.0.smithi089.stdout:592: finishing set_chunk to oid 43 2024-04-22T18:39:02.940 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1650688 length: 804864 2024-04-22T18:39:02.940 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-22T18:39:02.941 INFO:tasks.rados.rados.0.smithi089.stdout:586: done (14 left) 2024-04-22T18:39:02.941 INFO:tasks.rados.rados.0.smithi089.stdout:592: done (13 left) 2024-04-22T18:39:02.941 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 13 2024-04-22T18:39:02.941 INFO:tasks.rados.rados.0.smithi089.stdout:595: finishing set_chunk to oid 46 2024-04-22T18:39:02.941 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 737792 length: 932864 2024-04-22T18:39:02.941 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-22T18:39:02.941 INFO:tasks.rados.rados.0.smithi089.stdout:593: finishing set_chunk to oid 44 2024-04-22T18:39:02.941 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 932864 length: 66560 2024-04-22T18:39:02.941 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-22T18:39:02.941 INFO:tasks.rados.rados.0.smithi089.stdout:593: done (12 left) 2024-04-22T18:39:02.941 INFO:tasks.rados.rados.0.smithi089.stdout:595: done (11 left) 2024-04-22T18:39:02.941 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 11 2024-04-22T18:39:02.941 INFO:tasks.rados.rados.0.smithi089.stdout:594: finishing set_chunk to oid 45 2024-04-22T18:39:02.941 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 280576 length: 2447872 2024-04-22T18:39:02.941 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-22T18:39:02.942 INFO:tasks.rados.rados.0.smithi089.stdout:594: done (10 left) 2024-04-22T18:39:02.942 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 10 2024-04-22T18:39:02.942 INFO:tasks.rados.rados.0.smithi089.stdout:599: finishing set_chunk to oid 50 2024-04-22T18:39:02.942 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 59904 length: 961536 2024-04-22T18:39:02.942 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-22T18:39:02.942 INFO:tasks.rados.rados.0.smithi089.stdout:599: done (9 left) 2024-04-22T18:39:02.942 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 9 2024-04-22T18:39:02.942 INFO:tasks.rados.rados.0.smithi089.stdout:597: finishing set_chunk to oid 48 2024-04-22T18:39:02.942 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 334336 length: 56832 2024-04-22T18:39:02.942 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-22T18:39:02.942 INFO:tasks.rados.rados.0.smithi089.stdout:598: finishing set_chunk to oid 49 2024-04-22T18:39:02.942 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 509952 length: 1938432 2024-04-22T18:39:02.942 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:39:02.943 INFO:tasks.rados.rados.0.smithi089.stdout:597: done (8 left) 2024-04-22T18:39:02.943 INFO:tasks.rados.rados.0.smithi089.stdout:598: done (7 left) 2024-04-22T18:39:02.943 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 7 2024-04-22T18:39:02.944 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:02.985 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:02.985+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:03.030 INFO:tasks.rados.rados.0.smithi089.stdout:580: finishing set_chunk to oid 31 2024-04-22T18:39:03.031 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 966656 length: 1062400 2024-04-22T18:39:03.031 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-22T18:39:03.031 INFO:tasks.rados.rados.0.smithi089.stdout:580: done (6 left) 2024-04-22T18:39:03.031 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 6 2024-04-22T18:39:03.031 INFO:tasks.rados.rados.0.smithi089.stdout:582: finishing set_chunk to oid 33 2024-04-22T18:39:03.031 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 674816 length: 1653760 2024-04-22T18:39:03.031 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-22T18:39:03.031 INFO:tasks.rados.rados.0.smithi089.stdout:591: finishing set_chunk to oid 42 2024-04-22T18:39:03.031 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 18944 length: 1910272 2024-04-22T18:39:03.031 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-22T18:39:03.031 INFO:tasks.rados.rados.0.smithi089.stdout:582: done (5 left) 2024-04-22T18:39:03.031 INFO:tasks.rados.rados.0.smithi089.stdout:591: done (4 left) 2024-04-22T18:39:03.031 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 4 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout:585: finishing set_chunk to oid 36 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 164864 length: 506368 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout:585: done (3 left) 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 3 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout:587: finishing set_chunk to oid 38 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 556544 length: 994304 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout:587: done (2 left) 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 2 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout:596: finishing set_chunk to oid 47 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1046528 length: 1121280 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout:596: done (1 left) 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 1 2024-04-22T18:39:03.032 INFO:tasks.rados.rados.0.smithi089.stdout:600: finishing set_chunk to oid 1 2024-04-22T18:39:03.033 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 180224 length: 1463808 2024-04-22T18:39:03.033 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-22T18:39:03.033 INFO:tasks.rados.rados.0.smithi089.stdout:600: done (0 left) 2024-04-22T18:39:03.033 INFO:tasks.rados.rados.0.smithi089.stdout:601 oid_not_in_use 50 oid_redirect_not_in_use 0 2024-04-22T18:39:03.033 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 51 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.033 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 52 low_tier 2024-04-22T18:39:03.033 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 53 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.033 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 54 low_tier 2024-04-22T18:39:03.033 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 55 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.033 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 56 low_tier 2024-04-22T18:39:03.033 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 57 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.033 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 58 low_tier 2024-04-22T18:39:03.033 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 59 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.033 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 60 low_tier 2024-04-22T18:39:03.033 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 61 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 62 low_tier 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 63 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 64 low_tier 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 65 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 66 low_tier 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 67 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 68 low_tier 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 69 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 70 low_tier 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 71 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 72 low_tier 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 73 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 74 low_tier 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 75 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 76 low_tier 2024-04-22T18:39:03.034 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 77 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.035 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 78 low_tier 2024-04-22T18:39:03.035 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 79 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.035 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 80 low_tier 2024-04-22T18:39:03.035 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 81 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.035 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 82 low_tier 2024-04-22T18:39:03.035 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 83 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.035 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 84 low_tier 2024-04-22T18:39:03.035 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 85 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.035 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 86 low_tier 2024-04-22T18:39:03.035 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 87 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.035 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 88 low_tier 2024-04-22T18:39:03.035 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 89 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.035 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 90 low_tier 2024-04-22T18:39:03.035 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 91 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.035 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 92 low_tier 2024-04-22T18:39:03.036 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 93 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.036 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 94 low_tier 2024-04-22T18:39:03.036 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 95 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.036 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 96 low_tier 2024-04-22T18:39:03.036 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 97 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.036 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 98 low_tier 2024-04-22T18:39:03.036 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 99 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-22T18:39:03.036 INFO:tasks.rados.rados.0.smithi089.stdout: redirect_not_in_use: 100 low_tier 2024-04-22T18:39:03.036 INFO:tasks.rados.rados.0.smithi089.stdout:601: read oid 13 snap -1 2024-04-22T18:39:03.036 INFO:tasks.rados.rados.0.smithi089.stdout:601: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-22T18:39:03.036 INFO:tasks.rados.rados.0.smithi089.stdout:602: read oid 23 snap -1 2024-04-22T18:39:03.036 INFO:tasks.rados.rados.0.smithi089.stdout:602: expect (ObjNum 22 snap 0 seq_num 22) 2024-04-22T18:39:03.036 INFO:tasks.rados.rados.0.smithi089.stdout:603: write (excl) oid 48 current snap is 0 2024-04-22T18:39:03.036 INFO:tasks.rados.rados.0.smithi089.stdout:603: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-04-22T18:39:03.037 INFO:tasks.rados.rados.0.smithi089.stdout:603: writing smithi08937929-48 from 440350 to 900918 tid 1 2024-04-22T18:39:03.039 INFO:tasks.rados.rados.0.smithi089.stdout:603: writing smithi08937929-48 from 1309629 to 1852295 tid 2 2024-04-22T18:39:03.045 INFO:tasks.rados.rados.0.smithi089.stdout:603: writing smithi08937929-48 from 2323973 to 2840350 tid 3 2024-04-22T18:39:03.045 INFO:tasks.rados.rados.0.smithi089.stdout:604: read oid 10 snap -1 2024-04-22T18:39:03.046 INFO:tasks.rados.rados.0.smithi089.stdout:604: expect (ObjNum 9 snap 0 seq_num 9) 2024-04-22T18:39:03.046 INFO:tasks.rados.rados.0.smithi089.stdout:605: read oid 32 snap -1 2024-04-22T18:39:03.046 INFO:tasks.rados.rados.0.smithi089.stdout:605: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-22T18:39:03.046 INFO:tasks.rados.rados.0.smithi089.stdout:606: write oid 43 current snap is 0 2024-04-22T18:39:03.046 INFO:tasks.rados.rados.0.smithi089.stdout:606: seq_num 51 ranges {457157=535817,1768111=402677} 2024-04-22T18:39:03.050 INFO:tasks.rados.rados.0.smithi089.stdout:606: writing smithi08937929-43 from 457157 to 992974 tid 1 2024-04-22T18:39:03.057 INFO:tasks.rados.rados.0.smithi089.stdout:606: writing smithi08937929-43 from 1768111 to 2170788 tid 2 2024-04-22T18:39:03.057 INFO:tasks.rados.rados.0.smithi089.stdout:607: write oid 44 current snap is 0 2024-04-22T18:39:03.057 INFO:tasks.rados.rados.0.smithi089.stdout:607: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2024-04-22T18:39:03.063 INFO:tasks.rados.rados.0.smithi089.stdout:607: writing smithi08937929-44 from 473964 to 1085030 tid 1 2024-04-22T18:39:03.063 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:03.064+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:03.069 INFO:tasks.rados.rados.0.smithi089.stdout:607: writing smithi08937929-44 from 1826593 to 2372928 tid 2 2024-04-22T18:39:03.072 INFO:tasks.rados.rados.0.smithi089.stdout:607: writing smithi08937929-44 from 2778819 to 2873964 tid 3 2024-04-22T18:39:03.073 INFO:tasks.rados.rados.0.smithi089.stdout:608: read oid 49 snap -1 2024-04-22T18:39:03.073 INFO:tasks.rados.rados.0.smithi089.stdout:608: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-22T18:39:03.073 INFO:tasks.rados.rados.0.smithi089.stdout:609: read oid 36 snap -1 2024-04-22T18:39:03.073 INFO:tasks.rados.rados.0.smithi089.stdout:609: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-22T18:39:03.073 INFO:tasks.rados.rados.0.smithi089.stdout:610: write (excl) oid 12 current snap is 0 2024-04-22T18:39:03.073 INFO:tasks.rados.rados.0.smithi089.stdout:601: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-22T18:39:03.095 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:03.095 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:03.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:03.095 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:03.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.095 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4199.0:68 1.0 e17/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'42, mlcod=17'39)", 2024-04-22T18:39:03.095 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:41.987068+0000", 2024-04-22T18:39:03.096 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.108143389999999, 2024-04-22T18:39:03.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30106360999999998, 2024-04-22T18:39:03.096 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.096 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.096 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987068+0000", 2024-04-22T18:39:03.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987068+0000", 2024-04-22T18:39:03.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987067+0000", 2024-04-22T18:39:03.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:03.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987077+0000", 2024-04-22T18:39:03.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8279999999999998e-06 2024-04-22T18:39:03.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987084+0000", 2024-04-22T18:39:03.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8099999999999998e-06 2024-04-22T18:39:03.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:41.987091+0000", 2024-04-22T18:39:03.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6189999999999996e-06 2024-04-22T18:39:03.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087218+0000", 2024-04-22T18:39:03.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10012695100000001 2024-04-22T18:39:03.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.087246+0000", 2024-04-22T18:39:03.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8255e-05 2024-04-22T18:39:03.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.288092+0000", 2024-04-22T18:39:03.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20084555000000001 2024-04-22T18:39:03.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:42.288131+0000", 2024-04-22T18:39:03.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9455000000000001e-05 2024-04-22T18:39:03.099 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.099 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.100 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.100 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:56 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.100 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:39:03.100 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8433987040000002, 2024-04-22T18:39:03.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30396841899999999, 2024-04-22T18:39:03.100 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.100 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:03.100 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.100 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.100 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.100 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 56 2024-04-22T18:39:03.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.101 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:39:03.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:39:03.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:39:03.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:03.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251815+0000", 2024-04-22T18:39:03.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7800000000000001e-06 2024-04-22T18:39:03.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251820+0000", 2024-04-22T18:39:03.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5829999999999997e-06 2024-04-22T18:39:03.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251825+0000", 2024-04-22T18:39:03.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7879999999999997e-06 2024-04-22T18:39:03.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.252721+0000", 2024-04-22T18:39:03.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089552200000000005 2024-04-22T18:39:03.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:03.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.252794+0000", 2024-04-22T18:39:03.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3440000000000002e-05 2024-04-22T18:39:03.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555574+0000", 2024-04-22T18:39:03.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30278038099999999 2024-04-22T18:39:03.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555687+0000", 2024-04-22T18:39:03.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112804 2024-04-22T18:39:03.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555781+0000", 2024-04-22T18:39:03.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3624999999999999e-05 2024-04-22T18:39:03.104 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.104 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.105 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.105 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:61 4.b 4:d923883e:::smithi08937929-16:head [write 652105~445088,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.105 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:03.105 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8250160829999995, 2024-04-22T18:39:03.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.38578178600000002, 2024-04-22T18:39:03.105 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.105 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.105 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.105 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.105 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.105 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 61 2024-04-22T18:39:03.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.106 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:03.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:03.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270192+0000", 2024-04-22T18:39:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270401+0000", 2024-04-22T18:39:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020921199999999999 2024-04-22T18:39:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270415+0000", 2024-04-22T18:39:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4191e-05 2024-04-22T18:39:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270422+0000", 2024-04-22T18:39:03.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7329999999999997e-06 2024-04-22T18:39:03.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270461+0000", 2024-04-22T18:39:03.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9530999999999998e-05 2024-04-22T18:39:03.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270550+0000", 2024-04-22T18:39:03.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8508e-05 2024-04-22T18:39:03.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:03.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270648+0000", 2024-04-22T18:39:03.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8377999999999999e-05 2024-04-22T18:39:03.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.271907+0000", 2024-04-22T18:39:03.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012589599999999999 2024-04-22T18:39:03.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655904+0000", 2024-04-22T18:39:03.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.383997115 2024-04-22T18:39:03.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655933+0000", 2024-04-22T18:39:03.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9011999999999999e-05 2024-04-22T18:39:03.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655977+0000", 2024-04-22T18:39:03.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3578000000000002e-05 2024-04-22T18:39:03.110 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.110 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.110 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:62 4.b 4:d923883e:::smithi08937929-16:head [write 1728171~452988] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8190574349999995, 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.379937407, 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:39:03.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:39:03.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276151+0000", 2024-04-22T18:39:03.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:03.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276370+0000", 2024-04-22T18:39:03.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000218951 2024-04-22T18:39:03.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276382+0000", 2024-04-22T18:39:03.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1517e-05 2024-04-22T18:39:03.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276389+0000", 2024-04-22T18:39:03.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8939999999999993e-06 2024-04-22T18:39:03.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276430+0000", 2024-04-22T18:39:03.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0958999999999999e-05 2024-04-22T18:39:03.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276485+0000", 2024-04-22T18:39:03.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4208e-05 2024-04-22T18:39:03.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:03.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276567+0000", 2024-04-22T18:39:03.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2006000000000001e-05 2024-04-22T18:39:03.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277685+0000", 2024-04-22T18:39:03.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001118237 2024-04-22T18:39:03.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656024+0000", 2024-04-22T18:39:03.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.378339588 2024-04-22T18:39:03.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656054+0000", 2024-04-22T18:39:03.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9373000000000002e-05 2024-04-22T18:39:03.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656091+0000", 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7168000000000001e-05 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:63 4.b 4:d923883e:::smithi08937929-16:head [setxattr _header (56) in=7b,truncate 2252105] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8188151789999996, 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37982964200000002, 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.116 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.117 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.117 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 63 2024-04-22T18:39:03.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.117 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:39:03.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:39:03.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276395+0000", 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276400+0000", 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6090000000000003e-06 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276407+0000", 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5850000000000002e-06 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276411+0000", 2024-04-22T18:39:03.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2569999999999999e-06 2024-04-22T18:39:03.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277254+0000", 2024-04-22T18:39:03.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084384300000000005 2024-04-22T18:39:03.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277312+0000", 2024-04-22T18:39:03.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7395999999999997e-05 2024-04-22T18:39:03.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:03.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277388+0000", 2024-04-22T18:39:03.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6604e-05 2024-04-22T18:39:03.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277870+0000", 2024-04-22T18:39:03.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000481545 2024-04-22T18:39:03.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656138+0000", 2024-04-22T18:39:03.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37826752499999999 2024-04-22T18:39:03.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656177+0000", 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9419999999999999e-05 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656226+0000", 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8562e-05 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:64 4.b 4:d923883e:::smithi08937929-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.121 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:03.122 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8187845580000008, 2024-04-22T18:39:03.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37999559999999999, 2024-04-22T18:39:03.122 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.122 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:03.122 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.122 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.122 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.122 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 64 2024-04-22T18:39:03.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.122 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:03.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:03.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:03.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:03.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276429+0000", 2024-04-22T18:39:03.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7329999999999999e-06 2024-04-22T18:39:03.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276435+0000", 2024-04-22T18:39:03.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.02e-06 2024-04-22T18:39:03.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276439+0000", 2024-04-22T18:39:03.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7659999999999998e-06 2024-04-22T18:39:03.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277530+0000", 2024-04-22T18:39:03.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010909470000000001 2024-04-22T18:39:03.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:03.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277603+0000", 2024-04-22T18:39:03.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2638999999999994e-05 2024-04-22T18:39:03.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656259+0000", 2024-04-22T18:39:03.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37865579300000002 2024-04-22T18:39:03.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656330+0000", 2024-04-22T18:39:03.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0899e-05 2024-04-22T18:39:03.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656422+0000", 2024-04-22T18:39:03.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2397000000000003e-05 2024-04-22T18:39:03.126 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.126 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.126 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.126 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:65 4.3 4:c0d4d677:::smithi08937929-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.354893+0000", 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.7403177700000008, 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30130554999999998, 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.354893+0000", 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.354893+0000", 2024-04-22T18:39:03.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.354890+0000", 2024-04-22T18:39:03.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999962 2024-04-22T18:39:03.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.355220+0000", 2024-04-22T18:39:03.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00033014200000000001 2024-04-22T18:39:03.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.355235+0000", 2024-04-22T18:39:03.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5413e-05 2024-04-22T18:39:03.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455327+0000", 2024-04-22T18:39:03.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009154000000001 2024-04-22T18:39:03.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555819+0000", 2024-04-22T18:39:03.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100492812 2024-04-22T18:39:03.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555964+0000", 2024-04-22T18:39:03.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000144907 2024-04-22T18:39:03.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:03.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.556141+0000", 2024-04-22T18:39:03.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017619599999999999 2024-04-22T18:39:03.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656140+0000", 2024-04-22T18:39:03.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099999085000000001 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656144+0000", 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9689999999999996e-06 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656167+0000", 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3363e-05 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656199+0000", 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1828000000000001e-05 2024-04-22T18:39:03.131 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.132 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.132 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.132 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.132 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:03.132 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.6398544820000005, 2024-04-22T18:39:03.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:39:03.132 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.132 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.132 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.132 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.132 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.132 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:39:03.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.133 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:03.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:03.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:39:03.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:03.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:39:03.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:39:03.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:39:03.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:39:03.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:39:03.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:39:03.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:39:03.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:39:03.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:39:03.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:39:03.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:03.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:39:03.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:39:03.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:39:03.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:39:03.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:39:03.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:39:03.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:39:03.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:39:03.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:39:03.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:39:03.137 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.137 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.137 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.137 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:113 4.b 4:d3068ee3:::smithi08937929-29:head [write 470596~456031,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.138 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.687365+0000", 2024-04-22T18:39:03.138 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.4078460459999995, 2024-04-22T18:39:03.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20804932300000001, 2024-04-22T18:39:03.138 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.138 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.138 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.138 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.138 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.138 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 113 2024-04-22T18:39:03.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.138 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.687365+0000", 2024-04-22T18:39:03.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.687365+0000", 2024-04-22T18:39:03.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.687363+0000", 2024-04-22T18:39:03.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:03.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.687459+0000", 2024-04-22T18:39:03.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6688000000000007e-05 2024-04-22T18:39:03.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.687474+0000", 2024-04-22T18:39:03.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4348000000000001e-05 2024-04-22T18:39:03.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.787609+0000", 2024-04-22T18:39:03.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100135261 2024-04-22T18:39:03.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.887793+0000", 2024-04-22T18:39:03.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100184252 2024-04-22T18:39:03.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.887874+0000", 2024-04-22T18:39:03.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0533999999999998e-05 2024-04-22T18:39:03.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:03.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.887960+0000", 2024-04-22T18:39:03.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6222000000000002e-05 2024-04-22T18:39:03.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.889554+0000", 2024-04-22T18:39:03.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001594339 2024-04-22T18:39:03.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.895348+0000", 2024-04-22T18:39:03.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057940029999999998 2024-04-22T18:39:03.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.895376+0000", 2024-04-22T18:39:03.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7569e-05 2024-04-22T18:39:03.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.895414+0000", 2024-04-22T18:39:03.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8559999999999997e-05 2024-04-22T18:39:03.143 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.143 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.143 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.143 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:40 3.1 3:be9a15cb:::smithi08937929-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:03.143 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.794661+0000", 2024-04-22T18:39:03.143 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.3005503860000003, 2024-04-22T18:39:03.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.298299751, 2024-04-22T18:39:03.143 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.143 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.143 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.794661+0000", 2024-04-22T18:39:03.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.794661+0000", 2024-04-22T18:39:03.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.794659+0000", 2024-04-22T18:39:03.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:03.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.794665+0000", 2024-04-22T18:39:03.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5799999999999999e-06 2024-04-22T18:39:03.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.794675+0000", 2024-04-22T18:39:03.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4429999999999992e-06 2024-04-22T18:39:03.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.794680+0000", 2024-04-22T18:39:03.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1660000000000002e-06 2024-04-22T18:39:03.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.894788+0000", 2024-04-22T18:39:03.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010804 2024-04-22T18:39:03.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.894864+0000", 2024-04-22T18:39:03.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6389000000000003e-05 2024-04-22T18:39:03.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:03.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.894999+0000", 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013485 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.895628+0000", 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00062914199999999996 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.092902+0000", 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19727352300000001 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.092931+0000", 2024-04-22T18:39:03.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8980999999999999e-05 2024-04-22T18:39:03.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.092960+0000", 2024-04-22T18:39:03.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9807000000000001e-05 2024-04-22T18:39:03.148 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.148 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.148 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.148 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:315 4.b 4:d923883e:::smithi08937929-16:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:03.148 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:03.148 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1033917019999997, 2024-04-22T18:39:03.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40309447300000001, 2024-04-22T18:39:03.148 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 315 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:03.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991817+0000", 2024-04-22T18:39:03.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:03.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991825+0000", 2024-04-22T18:39:03.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7279999999999995e-06 2024-04-22T18:39:03.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991834+0000", 2024-04-22T18:39:03.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1980000000000007e-06 2024-04-22T18:39:03.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991841+0000", 2024-04-22T18:39:03.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0740000000000004e-06 2024-04-22T18:39:03.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991881+0000", 2024-04-22T18:39:03.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-04-22T18:39:03.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991940+0000", 2024-04-22T18:39:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9382000000000002e-05 2024-04-22T18:39:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394250+0000", 2024-04-22T18:39:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40230947700000003 2024-04-22T18:39:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394752+0000", 2024-04-22T18:39:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050269699999999997 2024-04-22T18:39:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394865+0000", 2024-04-22T18:39:03.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011267700000000001 2024-04-22T18:39:03.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394892+0000", 2024-04-22T18:39:03.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7535999999999999e-05 2024-04-22T18:39:03.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394914+0000", 2024-04-22T18:39:03.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1398000000000001e-05 2024-04-22T18:39:03.153 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.153 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.153 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.154 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:42 3.1 3:a1535cdd:::smithi08937929-24 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:03.154 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.093002+0000", 2024-04-22T18:39:03.154 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.002209004, 2024-04-22T18:39:03.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30111063900000001, 2024-04-22T18:39:03.154 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.154 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.154 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093002+0000", 2024-04-22T18:39:03.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093002+0000", 2024-04-22T18:39:03.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093001+0000", 2024-04-22T18:39:03.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:03.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093007+0000", 2024-04-22T18:39:03.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6509999999999998e-06 2024-04-22T18:39:03.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093018+0000", 2024-04-22T18:39:03.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1119e-05 2024-04-22T18:39:03.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093026+0000", 2024-04-22T18:39:03.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9049999999999997e-06 2024-04-22T18:39:03.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093063+0000", 2024-04-22T18:39:03.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6987999999999999e-05 2024-04-22T18:39:03.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093142+0000", 2024-04-22T18:39:03.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9877999999999997e-05 2024-04-22T18:39:03.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:03.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093306+0000", 2024-04-22T18:39:03.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016320500000000001 2024-04-22T18:39:03.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093803+0000", 2024-04-22T18:39:03.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049711999999999998 2024-04-22T18:39:03.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394037+0000", 2024-04-22T18:39:03.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30023460000000002 2024-04-22T18:39:03.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394073+0000", 2024-04-22T18:39:03.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5227e-05 2024-04-22T18:39:03.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394113+0000", 2024-04-22T18:39:03.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0037999999999999e-05 2024-04-22T18:39:03.159 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.159 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.159 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.159 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:331 4.6 4:6700eab5:::smithi08937929-32:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:03.159 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:39:03.159 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.498654567, 2024-04-22T18:39:03.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.305691029, 2024-04-22T18:39:03.159 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.159 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.159 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.159 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.160 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.160 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 331 2024-04-22T18:39:03.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.160 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:39:03.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:39:03.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:39:03.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:03.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596560+0000", 2024-04-22T18:39:03.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8149999999999996e-06 2024-04-22T18:39:03.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596568+0000", 2024-04-22T18:39:03.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1489999999999997e-06 2024-04-22T18:39:03.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596576+0000", 2024-04-22T18:39:03.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.191e-06 2024-04-22T18:39:03.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596619+0000", 2024-04-22T18:39:03.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3507000000000002e-05 2024-04-22T18:39:03.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596709+0000", 2024-04-22T18:39:03.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0043000000000005e-05 2024-04-22T18:39:03.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.801630+0000", 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20492061 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.802094+0000", 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046431899999999998 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.902165+0000", 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10007037000000001 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.902218+0000", 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3897999999999998e-05 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.902248+0000", 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9023999999999999e-05 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:367 4.3 4:c6fa81cf:::smithi08937929-18:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:39:03.164 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.2974975860000004, 2024-04-22T18:39:03.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30292533399999999, 2024-04-22T18:39:03.165 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.165 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.165 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.165 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.165 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.165 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 367 2024-04-22T18:39:03.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.165 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:39:03.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:39:03.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:39:03.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:03.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797719+0000", 2024-04-22T18:39:03.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0669999999999997e-06 2024-04-22T18:39:03.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797725+0000", 2024-04-22T18:39:03.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3049999999999998e-06 2024-04-22T18:39:03.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797731+0000", 2024-04-22T18:39:03.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7649999999999999e-06 2024-04-22T18:39:03.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.897957+0000", 2024-04-22T18:39:03.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100226045 2024-04-22T18:39:03.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.898010+0000", 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2679999999999997e-05 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.999633+0000", 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101623639 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.000232+0000", 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059861099999999998 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100531+0000", 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100299112 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100599+0000", 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7989000000000002e-05 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100639+0000", 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9919000000000002e-05 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.170 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:374 4.3 4:c93c842e:::smithi08937929-25:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:03.170 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.798795+0000", 2024-04-22T18:39:03.170 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.2964164159999996, 2024-04-22T18:39:03.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30197590899999999, 2024-04-22T18:39:03.170 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.170 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.170 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.170 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.170 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.170 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 374 2024-04-22T18:39:03.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.170 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798795+0000", 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798795+0000", 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798793+0000", 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798801+0000", 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5299999999999999e-06 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798811+0000", 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0047999999999999e-05 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798818+0000", 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4240000000000004e-06 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.898121+0000", 2024-04-22T18:39:03.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099302611999999998 2024-04-22T18:39:03.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.898189+0000", 2024-04-22T18:39:03.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8032000000000002e-05 2024-04-22T18:39:03.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:03.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.999945+0000", 2024-04-22T18:39:03.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101755686 2024-04-22T18:39:03.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100695+0000", 2024-04-22T18:39:03.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100750622 2024-04-22T18:39:03.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100703+0000", 2024-04-22T18:39:03.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3690000000000001e-06 2024-04-22T18:39:03.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100731+0000", 2024-04-22T18:39:03.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8427000000000001e-05 2024-04-22T18:39:03.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100771+0000", 2024-04-22T18:39:03.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9579000000000002e-05 2024-04-22T18:39:03.175 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.175 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.175 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.175 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:393 4.b 4:defe92fb:::smithi08937929-44:head [assert-version v12,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:03.175 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:39:03.175 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.99401552, 2024-04-22T18:39:03.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33050822299999999, 2024-04-22T18:39:03.175 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.176 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.176 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.176 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.176 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.176 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 393 2024-04-22T18:39:03.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.176 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:39:03.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:39:03.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101194+0000", 2024-04-22T18:39:03.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:03.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101201+0000", 2024-04-22T18:39:03.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1269999999999997e-06 2024-04-22T18:39:03.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101211+0000", 2024-04-22T18:39:03.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9380000000000004e-06 2024-04-22T18:39:03.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101220+0000", 2024-04-22T18:39:03.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3459999999999994e-06 2024-04-22T18:39:03.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101324+0000", 2024-04-22T18:39:03.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104366 2024-04-22T18:39:03.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101379+0000", 2024-04-22T18:39:03.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5127999999999998e-05 2024-04-22T18:39:03.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:03.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.330649+0000", 2024-04-22T18:39:03.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.22927022699999999 2024-04-22T18:39:03.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.331182+0000", 2024-04-22T18:39:03.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053250900000000002 2024-04-22T18:39:03.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.431635+0000", 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10045290599999999 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.431677+0000", 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1816000000000001e-05 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.431704+0000", 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7132e-05 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.180 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:31 3.1 3:b7b88fa0:::smithi08937929-44 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:03.181 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.101557+0000", 2024-04-22T18:39:03.181 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.9936539639999999, 2024-04-22T18:39:03.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.22893401799999999, 2024-04-22T18:39:03.181 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.181 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.181 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101557+0000", 2024-04-22T18:39:03.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101557+0000", 2024-04-22T18:39:03.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101555+0000", 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101563+0000", 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0600000000000008e-06 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101574+0000", 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0977e-05 2024-04-22T18:39:03.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101587+0000", 2024-04-22T18:39:03.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2884999999999999e-05 2024-04-22T18:39:03.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.102112+0000", 2024-04-22T18:39:03.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00052499299999999997 2024-04-22T18:39:03.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.102162+0000", 2024-04-22T18:39:03.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0009000000000002e-05 2024-04-22T18:39:03.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.102263+0000", 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010105199999999999 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.102636+0000", 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037250199999999998 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.330415+0000", 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.22777872900000001 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.330461+0000", 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6397999999999999e-05 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.330491+0000", 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0108999999999999e-05 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:415 4.b 4:d923883e:::smithi08937929-16:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402145+0000", 2024-04-22T18:39:03.185 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.693066022, 2024-04-22T18:39:03.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30209665400000002, 2024-04-22T18:39:03.186 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.186 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.186 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.186 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.186 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.186 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 415 2024-04-22T18:39:03.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.186 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402145+0000", 2024-04-22T18:39:03.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402145+0000", 2024-04-22T18:39:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402144+0000", 2024-04-22T18:39:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402149+0000", 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0760000000000002e-06 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402159+0000", 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7499999999999998e-06 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402165+0000", 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0700000000000003e-06 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402195+0000", 2024-04-22T18:39:03.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0485e-05 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402293+0000", 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7153999999999998e-05 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503807+0000", 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101514666 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604684+0000", 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100876493 2024-04-22T18:39:03.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.704174+0000", 2024-04-22T18:39:03.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099490331000000001 2024-04-22T18:39:03.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.704200+0000", 2024-04-22T18:39:03.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5902e-05 2024-04-22T18:39:03.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.704242+0000", 2024-04-22T18:39:03.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1686e-05 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:416 4.3 4:c0d4d677:::smithi08937929-17:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402174+0000", 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6930370139999997, 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30212472000000001, 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 416 2024-04-22T18:39:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.192 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402174+0000", 2024-04-22T18:39:03.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402174+0000", 2024-04-22T18:39:03.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402173+0000", 2024-04-22T18:39:03.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402178+0000", 2024-04-22T18:39:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4259999999999996e-06 2024-04-22T18:39:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402184+0000", 2024-04-22T18:39:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.066e-06 2024-04-22T18:39:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402189+0000", 2024-04-22T18:39:03.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:39:03.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402426+0000", 2024-04-22T18:39:03.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023742300000000001 2024-04-22T18:39:03.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402489+0000", 2024-04-22T18:39:03.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2891999999999993e-05 2024-04-22T18:39:03.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:03.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.504423+0000", 2024-04-22T18:39:03.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101933444 2024-04-22T18:39:03.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.704136+0000", 2024-04-22T18:39:03.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.199712895 2024-04-22T18:39:03.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.704250+0000", 2024-04-22T18:39:03.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114645 2024-04-22T18:39:03.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.704273+0000", 2024-04-22T18:39:03.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2736999999999999e-05 2024-04-22T18:39:03.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.704299+0000", 2024-04-22T18:39:03.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5833000000000002e-05 2024-04-22T18:39:03.196 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.196 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.196 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.196 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:713 4.3 4:ca46fde9:::smithi08937929-14:head [assert-version v3,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:03.197 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.627086+0000", 2024-04-22T18:39:03.197 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.46812493100000002, 2024-04-22T18:39:03.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30289147900000002, 2024-04-22T18:39:03.197 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.197 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.197 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 713 2024-04-22T18:39:03.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.197 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627086+0000", 2024-04-22T18:39:03.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627086+0000", 2024-04-22T18:39:03.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627085+0000", 2024-04-22T18:39:03.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:03.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627098+0000", 2024-04-22T18:39:03.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2869e-05 2024-04-22T18:39:03.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627111+0000", 2024-04-22T18:39:03.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2978e-05 2024-04-22T18:39:03.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627117+0000", 2024-04-22T18:39:03.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7209999999999997e-06 2024-04-22T18:39:03.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627152+0000", 2024-04-22T18:39:03.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5052000000000003e-05 2024-04-22T18:39:03.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627204+0000", 2024-04-22T18:39:03.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1214000000000001e-05 2024-04-22T18:39:03.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:03.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.929177+0000", 2024-04-22T18:39:03.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30197355399999998 2024-04-22T18:39:03.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.929785+0000", 2024-04-22T18:39:03.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060751100000000003 2024-04-22T18:39:03.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.929922+0000", 2024-04-22T18:39:03.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013767599999999999 2024-04-22T18:39:03.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.929952+0000", 2024-04-22T18:39:03.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9708000000000001e-05 2024-04-22T18:39:03.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.929978+0000", 2024-04-22T18:39:03.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5497e-05 2024-04-22T18:39:03.202 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.202 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.202 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.202 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.202 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.202 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:03.204 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:03.204 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.0 to 1 (10000) 2024-04-22T18:39:03.204 INFO:tasks.rados.rados.0.smithi089.stdout:603: finishing write tid 1 to smithi08937929-48 2024-04-22T18:39:03.204 INFO:tasks.rados.rados.0.smithi089.stdout:602: expect (ObjNum 22 snap 0 seq_num 22) 2024-04-22T18:39:03.205 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-22T18:39:03.206 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:03.204+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:03.223 INFO:tasks.rados.rados.0.smithi089.stdout:603: finishing write tid 2 to smithi08937929-48 2024-04-22T18:39:03.223 INFO:tasks.rados.rados.0.smithi089.stdout:606: finishing write tid 1 to smithi08937929-43 2024-04-22T18:39:03.224 INFO:tasks.rados.rados.0.smithi089.stdout:606: finishing write tid 2 to smithi08937929-43 2024-04-22T18:39:03.224 INFO:tasks.rados.rados.0.smithi089.stdout:610: seq_num 53 ranges {490771=686315,1885075=406346} 2024-04-22T18:39:03.230 INFO:tasks.rados.rados.0.smithi089.stdout:610: writing smithi08937929-12 from 490771 to 1177086 tid 1 2024-04-22T18:39:03.237 INFO:tasks.rados.rados.0.smithi089.stdout:610: writing smithi08937929-12 from 1885075 to 2291421 tid 2 2024-04-22T18:39:03.237 INFO:tasks.rados.rados.0.smithi089.stdout:601: done (9 left) 2024-04-22T18:39:03.237 INFO:tasks.rados.rados.0.smithi089.stdout:602: done (8 left) 2024-04-22T18:39:03.237 INFO:tasks.rados.rados.0.smithi089.stdout:611: set_chunk oid 22 target oid 2024-04-22T18:39:03.304 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:03.305+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:03.330 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:03.331 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:03.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.331 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:774 4.b e32/20)", 2024-04-22T18:39:03.331 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.251757+0000", 2024-04-22T18:39:03.331 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.080302480999999995, 2024-04-22T18:39:03.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.080320570999999993, 2024-04-22T18:39:03.331 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.331 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:03.331 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251757+0000", 2024-04-22T18:39:03.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251757+0000", 2024-04-22T18:39:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251755+0000", 2024-04-22T18:39:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251825+0000", 2024-04-22T18:39:03.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9703000000000007e-05 2024-04-22T18:39:03.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251838+0000", 2024-04-22T18:39:03.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2788e-05 2024-04-22T18:39:03.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251846+0000", 2024-04-22T18:39:03.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6889999999999993e-06 2024-04-22T18:39:03.333 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.333 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.333 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.333 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.333 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:39:03.334 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 1 2024-04-22T18:39:03.334 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:03.342 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:03.405 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:03.406+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:03.486 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:03.486 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:03.495 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:03.537 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:03.538+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:03.598 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:7 4.f 4:f83a159b:::smithi08937929-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.111347+0000", 2024-04-22T18:39:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4864576169999992, 2024-04-22T18:39:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30271893700000002, 2024-04-22T18:39:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 7 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111347+0000", 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111347+0000", 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111346+0000", 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111350+0000", 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.473e-06 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111353+0000", 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9100000000000001e-06 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111356+0000", 2024-04-22T18:39:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2770000000000001e-06 2024-04-22T18:39:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.212813+0000", 2024-04-22T18:39:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10145739099999999 2024-04-22T18:39:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.212869+0000", 2024-04-22T18:39:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5630000000000001e-05 2024-04-22T18:39:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.212949+0000", 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9679999999999996e-05 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.313864+0000", 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100915174 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.413991+0000", 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100126703 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414021+0000", 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0698999999999999e-05 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414066+0000", 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4718000000000001e-05 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:8 4.f 4:f83a159b:::smithi08937929-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4864264180000006, 2024-04-22T18:39:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30292182499999998, 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 8 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111380+0000", 2024-04-22T18:39:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2419999999999999e-06 2024-04-22T18:39:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111384+0000", 2024-04-22T18:39:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.715e-06 2024-04-22T18:39:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111387+0000", 2024-04-22T18:39:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.982e-06 2024-04-22T18:39:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414113+0000", 2024-04-22T18:39:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30272622700000001 2024-04-22T18:39:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414203+0000", 2024-04-22T18:39:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0316999999999996e-05 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414300+0000", 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6760000000000002e-05 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:9 4.f 4:f2c48a0e:::smithi08937929-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4810308340000002, 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49290714600000002, 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 9 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116773+0000", 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117024+0000", 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025191299999999999 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117036+0000", 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1595e-05 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117046+0000", 2024-04-22T18:39:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0456e-05 2024-04-22T18:39:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414324+0000", 2024-04-22T18:39:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297277603 2024-04-22T18:39:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414404+0000", 2024-04-22T18:39:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9738000000000001e-05 2024-04-22T18:39:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414494+0000", 2024-04-22T18:39:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9733000000000003e-05 2024-04-22T18:39:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518194+0000", 2024-04-22T18:39:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10370061 2024-04-22T18:39:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609618+0000", 2024-04-22T18:39:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091423360999999995 2024-04-22T18:39:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609655+0000", 2024-04-22T18:39:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7580000000000003e-05 2024-04-22T18:39:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609681+0000", 2024-04-22T18:39:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5908e-05 2024-04-22T18:39:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:10 4.f 4:f2c48a0e:::smithi08937929-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4751785739999992, 2024-04-22T18:39:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48712681400000002, 2024-04-22T18:39:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 10 2024-04-22T18:39:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122625+0000", 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122828+0000", 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020359499999999999 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122840+0000", 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1239e-05 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122845+0000", 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2719999999999997e-06 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415105+0000", 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29225972099999997 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415169+0000", 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4170000000000004e-05 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415257+0000", 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8331999999999994e-05 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518199+0000", 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10294174 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609707+0000", 2024-04-22T18:39:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091508408999999999 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609729+0000", 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1917e-05 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609753+0000", 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3776000000000001e-05 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:11 4.f 4:f2c48a0e:::smithi08937929-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4749552010000002, 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49452214500000002, 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 11 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122849+0000", 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122853+0000", 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9029999999999997e-06 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122856+0000", 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.01e-06 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122859+0000", 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415755+0000", 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29289606800000001 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415808+0000", 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3415999999999998e-05 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415888+0000", 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9061999999999996e-05 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518203+0000", 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102315533 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617164+0000", 2024-04-22T18:39:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098960548999999995 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617241+0000", 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7292000000000005e-05 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617372+0000", 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013075399999999999 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:12 4.f 4:f2c48a0e:::smithi08937929-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4749433369999991, 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49473910999999998, 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 12 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122863+0000", 2024-04-22T18:39:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.863e-06 2024-04-22T18:39:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122866+0000", 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122869+0000", 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5950000000000001e-06 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416008+0000", 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29313975199999998 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416048+0000", 2024-04-22T18:39:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9888000000000002e-05 2024-04-22T18:39:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617447+0000", 2024-04-22T18:39:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20139881400000001 2024-04-22T18:39:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617537+0000", 2024-04-22T18:39:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9606999999999998e-05 2024-04-22T18:39:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617600+0000", 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3884e-05 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:33 4.f 4:f0ed3d49:::smithi08937929-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4110031359999997, 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62359692099999997, 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 33 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186799+0000", 2024-04-22T18:39:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187103+0000", 2024-04-22T18:39:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030354900000000002 2024-04-22T18:39:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187117+0000", 2024-04-22T18:39:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4578e-05 2024-04-22T18:39:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187129+0000", 2024-04-22T18:39:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1421e-05 2024-04-22T18:39:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516134+0000", 2024-04-22T18:39:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32900553900000001 2024-04-22T18:39:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516215+0000", 2024-04-22T18:39:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0537999999999994e-05 2024-04-22T18:39:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516302+0000", 2024-04-22T18:39:03.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6601000000000007e-05 2024-04-22T18:39:03.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518207+0000", 2024-04-22T18:39:03.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019058370000000001 2024-04-22T18:39:03.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810302+0000", 2024-04-22T18:39:03.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29209509700000003 2024-04-22T18:39:03.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810348+0000", 2024-04-22T18:39:03.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5992999999999998e-05 2024-04-22T18:39:03.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810398+0000", 2024-04-22T18:39:03.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0046000000000002e-05 2024-04-22T18:39:03.634 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.634 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.634 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.635 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:34 4.f 4:f0ed3d49:::smithi08937929-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.635 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:03.635 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4064011240000003, 2024-04-22T18:39:03.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.61954655800000002, 2024-04-22T18:39:03.635 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.635 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.635 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.635 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.635 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.635 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34 2024-04-22T18:39:03.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.635 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:03.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:03.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191401+0000", 2024-04-22T18:39:03.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:03.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191503+0000", 2024-04-22T18:39:03.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010190400000000001 2024-04-22T18:39:03.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191517+0000", 2024-04-22T18:39:03.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3267e-05 2024-04-22T18:39:03.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191523+0000", 2024-04-22T18:39:03.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-04-22T18:39:03.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517062+0000", 2024-04-22T18:39:03.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32553894300000003 2024-04-22T18:39:03.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517141+0000", 2024-04-22T18:39:03.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9667999999999997e-05 2024-04-22T18:39:03.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:03.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517267+0000", 2024-04-22T18:39:03.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000125494 2024-04-22T18:39:03.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518213+0000", 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000946275 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810901+0000", 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29268821699999997 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810929+0000", 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7645999999999999e-05 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810950+0000", 2024-04-22T18:39:03.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1333999999999999e-05 2024-04-22T18:39:03.640 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.640 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.640 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.640 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.640 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:03.640 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4062781560000008, 2024-04-22T18:39:03.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:39:03.640 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.640 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.641 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.641 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.641 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.641 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:39:03.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.641 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:03.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:03.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:39:03.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:03.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:39:03.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:39:03.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:39:03.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:39:03.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.645 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:03.646 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4062670809999993, 2024-04-22T18:39:03.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:39:03.646 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.646 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:03.646 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.646 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.646 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.646 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:39:03.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.646 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:03.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:39:03.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:03.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:39:03.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:39:03.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:39:03.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:39:03.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:39:03.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:39:03.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:39:03.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:39:03.650 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.650 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.650 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.650 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:38 4.2 4:4fb26dd8:::smithi08937929-10:head [write 1893632~257631] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.650 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:03.650 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.3945444770000002, 2024-04-22T18:39:03.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33863800199999999, 2024-04-22T18:39:03.650 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.650 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.651 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.651 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.651 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.651 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 38 2024-04-22T18:39:03.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.651 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:03.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:03.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203258+0000", 2024-04-22T18:39:03.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:03.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203396+0000", 2024-04-22T18:39:03.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013710799999999999 2024-04-22T18:39:03.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203406+0000", 2024-04-22T18:39:03.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0295000000000001e-05 2024-04-22T18:39:03.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203411+0000", 2024-04-22T18:39:03.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5079999999999996e-06 2024-04-22T18:39:03.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203450+0000", 2024-04-22T18:39:03.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8420999999999997e-05 2024-04-22T18:39:03.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:03.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203455+0000", 2024-04-22T18:39:03.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:39:03.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237071+0000", 2024-04-22T18:39:03.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.033615891000000002 2024-04-22T18:39:03.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237129+0000", 2024-04-22T18:39:03.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8134000000000003e-05 2024-04-22T18:39:03.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237221+0000", 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2053999999999997e-05 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.341419+0000", 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104197568 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541792+0000", 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.200373312 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541838+0000", 2024-04-22T18:39:03.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6004000000000002e-05 2024-04-22T18:39:03.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541898+0000", 2024-04-22T18:39:03.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0297000000000002e-05 2024-04-22T18:39:03.656 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.656 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.656 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.656 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:39 4.2 4:4fb26dd8:::smithi08937929-10:head [setxattr _header (56) in=7b,truncate 2151263] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.656 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:03.656 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.3943874829999992, 2024-04-22T18:39:03.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33862203000000002, 2024-04-22T18:39:03.657 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.657 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.657 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.657 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.657 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.657 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 39 2024-04-22T18:39:03.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.657 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:03.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:03.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:03.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:03.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203423+0000", 2024-04-22T18:39:03.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0589999999999999e-06 2024-04-22T18:39:03.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203429+0000", 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2559999999999999e-06 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203434+0000", 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8049999999999997e-06 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203475+0000", 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1205000000000003e-05 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203478+0000", 2024-04-22T18:39:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7140000000000002e-06 2024-04-22T18:39:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239180+0000", 2024-04-22T18:39:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035702195999999999 2024-04-22T18:39:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239225+0000", 2024-04-22T18:39:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5722e-05 2024-04-22T18:39:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239291+0000", 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5870000000000005e-05 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342114+0000", 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102822301 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541946+0000", 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.199832544 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541988+0000", 2024-04-22T18:39:03.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1499999999999999e-05 2024-04-22T18:39:03.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542039+0000", 2024-04-22T18:39:03.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1576000000000002e-05 2024-04-22T18:39:03.662 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.662 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.662 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.662 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:40 4.2 4:4fb26dd8:::smithi08937929-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.662 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:03.662 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.394350223, 2024-04-22T18:39:03.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33924657200000002, 2024-04-22T18:39:03.663 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.663 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:03.663 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.663 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.663 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.663 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 40 2024-04-22T18:39:03.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.663 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:03.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:03.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:03.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:03.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203457+0000", 2024-04-22T18:39:03.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.129e-06 2024-04-22T18:39:03.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203463+0000", 2024-04-22T18:39:03.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7440000000000004e-06 2024-04-22T18:39:03.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203468+0000", 2024-04-22T18:39:03.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2619999999999998e-06 2024-04-22T18:39:03.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203491+0000", 2024-04-22T18:39:03.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3260000000000001e-05 2024-04-22T18:39:03.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:03.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203493+0000", 2024-04-22T18:39:03.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9599999999999999e-06 2024-04-22T18:39:03.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239420+0000", 2024-04-22T18:39:03.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035927212 2024-04-22T18:39:03.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:03.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239462+0000", 2024-04-22T18:39:03.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.159e-05 2024-04-22T18:39:03.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542547+0000", 2024-04-22T18:39:03.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30308492599999998 2024-04-22T18:39:03.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542611+0000", 2024-04-22T18:39:03.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4622000000000006e-05 2024-04-22T18:39:03.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542701+0000", 2024-04-22T18:39:03.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9579000000000004e-05 2024-04-22T18:39:03.667 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.667 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.668 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.668 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:41 4.2 4:409e6456:::smithi08937929-11:head [write 568070~468843,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.668 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:03.668 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.3886339519999993, 2024-04-22T18:39:03.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33297193600000002, 2024-04-22T18:39:03.668 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.668 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.668 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.668 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.668 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.668 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 41 2024-04-22T18:39:03.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.668 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:03.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:03.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209168+0000", 2024-04-22T18:39:03.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:03.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209393+0000", 2024-04-22T18:39:03.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022485199999999999 2024-04-22T18:39:03.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209404+0000", 2024-04-22T18:39:03.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1209999999999999e-05 2024-04-22T18:39:03.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209411+0000", 2024-04-22T18:39:03.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8580000000000002e-06 2024-04-22T18:39:03.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209457+0000", 2024-04-22T18:39:03.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6351999999999999e-05 2024-04-22T18:39:03.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:03.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209463+0000", 2024-04-22T18:39:03.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.322e-06 2024-04-22T18:39:03.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339572+0000", 2024-04-22T18:39:03.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13010901999999999 2024-04-22T18:39:03.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339673+0000", 2024-04-22T18:39:03.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101332 2024-04-22T18:39:03.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:03.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339765+0000", 2024-04-22T18:39:03.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1780999999999994e-05 2024-04-22T18:39:03.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342117+0000", 2024-04-22T18:39:03.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002351709 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542074+0000", 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995749800000001 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542112+0000", 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7891e-05 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542143+0000", 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0756000000000002e-05 2024-04-22T18:39:03.673 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.674 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.674 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.674 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:42 4.2 4:409e6456:::smithi08937929-11:head [write 1552114~615956] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.674 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:03.674 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.3822956430000009, 2024-04-22T18:39:03.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326735257, 2024-04-22T18:39:03.674 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.674 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.674 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.674 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.674 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.674 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 42 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215508+0000", 2024-04-22T18:39:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215766+0000", 2024-04-22T18:39:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025766999999999999 2024-04-22T18:39:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215776+0000", 2024-04-22T18:39:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0772e-05 2024-04-22T18:39:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215785+0000", 2024-04-22T18:39:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6880000000000005e-06 2024-04-22T18:39:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215818+0000", 2024-04-22T18:39:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2960999999999998e-05 2024-04-22T18:39:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215821+0000", 2024-04-22T18:39:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8569999999999999e-06 2024-04-22T18:39:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340332+0000", 2024-04-22T18:39:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.124511409 2024-04-22T18:39:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340416+0000", 2024-04-22T18:39:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3793000000000002e-05 2024-04-22T18:39:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340543+0000", 2024-04-22T18:39:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012720299999999999 2024-04-22T18:39:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342361+0000", 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018174370000000001 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542176+0000", 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19981518400000001 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542211+0000", 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5128e-05 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542244+0000", 2024-04-22T18:39:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3256000000000001e-05 2024-04-22T18:39:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:43 4.2 4:409e6456:::smithi08937929-11:head [setxattr _header (56) in=7b,truncate 2168070] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.3820108700000002, 2024-04-22T18:39:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32656540499999998, 2024-04-22T18:39:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 43 2024-04-22T18:39:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215793+0000", 2024-04-22T18:39:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215797+0000", 2024-04-22T18:39:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4190000000000004e-06 2024-04-22T18:39:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215804+0000", 2024-04-22T18:39:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:39:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215809+0000", 2024-04-22T18:39:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3689999999999998e-06 2024-04-22T18:39:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215840+0000", 2024-04-22T18:39:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-05 2024-04-22T18:39:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215843+0000", 2024-04-22T18:39:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8849999999999999e-06 2024-04-22T18:39:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342483+0000", 2024-04-22T18:39:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126640172 2024-04-22T18:39:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342529+0000", 2024-04-22T18:39:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5668999999999999e-05 2024-04-22T18:39:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342597+0000", 2024-04-22T18:39:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7538000000000003e-05 2024-04-22T18:39:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.347102+0000", 2024-04-22T18:39:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045058040000000004 2024-04-22T18:39:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542278+0000", 2024-04-22T18:39:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.195175879 2024-04-22T18:39:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542314+0000", 2024-04-22T18:39:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6134999999999999e-05 2024-04-22T18:39:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542359+0000", 2024-04-22T18:39:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4848999999999997e-05 2024-04-22T18:39:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:44 4.2 4:409e6456:::smithi08937929-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.3819908620000003, 2024-04-22T18:39:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326706418, 2024-04-22T18:39:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 44 2024-04-22T18:39:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215813+0000", 2024-04-22T18:39:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215816+0000", 2024-04-22T18:39:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9799999999999998e-06 2024-04-22T18:39:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215822+0000", 2024-04-22T18:39:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5570000000000003e-06 2024-04-22T18:39:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215827+0000", 2024-04-22T18:39:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2739999999999998e-06 2024-04-22T18:39:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215855+0000", 2024-04-22T18:39:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7682999999999999e-05 2024-04-22T18:39:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215857+0000", 2024-04-22T18:39:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9250000000000002e-06 2024-04-22T18:39:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342717+0000", 2024-04-22T18:39:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126860696 2024-04-22T18:39:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342758+0000", 2024-04-22T18:39:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0295e-05 2024-04-22T18:39:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542408+0000", 2024-04-22T18:39:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19964990599999999 2024-04-22T18:39:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542457+0000", 2024-04-22T18:39:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9784999999999999e-05 2024-04-22T18:39:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542520+0000", 2024-04-22T18:39:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2960999999999996e-05 2024-04-22T18:39:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:49 4.a e27/20 4:5d2aee44:::smithi08937929-13:head v 27'4, mlcod=0'0)", 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.261026+0000", 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.3367785019999996, 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.281745405, 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.261026+0000", 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.261026+0000", 2024-04-22T18:39:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.261025+0000", 2024-04-22T18:39:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.261563+0000", 2024-04-22T18:39:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053753000000000004 2024-04-22T18:39:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.261572+0000", 2024-04-22T18:39:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4080000000000004e-06 2024-04-22T18:39:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.261579+0000", 2024-04-22T18:39:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8789999999999997e-06 2024-04-22T18:39:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541746+0000", 2024-04-22T18:39:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.280167153 2024-04-22T18:39:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541760+0000", 2024-04-22T18:39:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4079e-05 2024-04-22T18:39:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542740+0000", 2024-04-22T18:39:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00097993100000000007 2024-04-22T18:39:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542772+0000", 2024-04-22T18:39:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1424999999999998e-05 2024-04-22T18:39:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.8048975870000001, 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:39:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:39:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:39:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:39:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:39:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:39:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:39:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:39:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:39:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:39:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:39:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:39:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:39:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:39:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:39:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:709 4.2 4:4fb26dd8:::smithi08937929-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.528719+0000", 2024-04-22T18:39:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.069085949, 2024-04-22T18:39:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30036358099999999, 2024-04-22T18:39:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 709 2024-04-22T18:39:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528719+0000", 2024-04-22T18:39:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528719+0000", 2024-04-22T18:39:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528718+0000", 2024-04-22T18:39:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528723+0000", 2024-04-22T18:39:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8550000000000001e-06 2024-04-22T18:39:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528729+0000", 2024-04-22T18:39:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8680000000000004e-06 2024-04-22T18:39:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528735+0000", 2024-04-22T18:39:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8309999999999997e-06 2024-04-22T18:39:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528766+0000", 2024-04-22T18:39:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1970000000000001e-05 2024-04-22T18:39:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528823+0000", 2024-04-22T18:39:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6440000000000002e-05 2024-04-22T18:39:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728211+0000", 2024-04-22T18:39:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.199387975 2024-04-22T18:39:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728615+0000", 2024-04-22T18:39:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000404161 2024-04-22T18:39:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.829015+0000", 2024-04-22T18:39:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100400197 2024-04-22T18:39:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.829053+0000", 2024-04-22T18:39:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7373000000000003e-05 2024-04-22T18:39:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.829082+0000", 2024-04-22T18:39:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9717e-05 2024-04-22T18:39:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:03.707 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:03.708 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:03.708 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:03.709+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:03.753 INFO:tasks.rados.rados.0.smithi089.stdout:611: set_chunk oid 22 offset: 1965568 length: 242176 target oid 7685983c885dd6cc65a5e512f0c37fea60b78d0f92e43419763e351098043064 offset: 0 snap -1 2024-04-22T18:39:03.753 INFO:tasks.rados.rados.0.smithi089.stdout:612: read oid 31 snap -1 2024-04-22T18:39:03.753 INFO:tasks.rados.rados.0.smithi089.stdout:612: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-22T18:39:03.753 INFO:tasks.rados.rados.0.smithi089.stdout:613: read oid 37 snap -1 2024-04-22T18:39:03.753 INFO:tasks.rados.rados.0.smithi089.stdout:613: expect (ObjNum 36 snap 0 seq_num 36) 2024-04-22T18:39:03.753 INFO:tasks.rados.rados.0.smithi089.stdout:614: read oid 7 snap -1 2024-04-22T18:39:03.753 INFO:tasks.rados.rados.0.smithi089.stdout:614: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-22T18:39:03.753 INFO:tasks.rados.rados.0.smithi089.stdout:615: delete oid 8 current snap is 0 2024-04-22T18:39:03.753 INFO:tasks.rados.rados.0.smithi089.stdout:609: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-22T18:39:03.809 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:03.809+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:03.838 INFO:tasks.rados.rados.0.smithi089.stdout:606: finishing write tid 3 to smithi08937929-43 2024-04-22T18:39:03.838 INFO:tasks.rados.rados.0.smithi089.stdout:606: finishing write tid 4 to smithi08937929-43 2024-04-22T18:39:03.838 INFO:tasks.rados.rados.0.smithi089.stdout:606: oid 43 updating version 0 to 13 2024-04-22T18:39:03.838 INFO:tasks.rados.rados.0.smithi089.stdout:606: oid 43 updating version 13 to 15 2024-04-22T18:39:03.839 INFO:tasks.rados.rados.0.smithi089.stdout:606: oid 43 version 15 is already newer than 14 2024-04-22T18:39:03.839 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 15 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-04-22T18:39:03.839 INFO:tasks.rados.rados.0.smithi089.stdout:606: left oid 43 (ObjNum 51 snap 0 seq_num 51) 2024-04-22T18:39:03.839 INFO:tasks.rados.rados.0.smithi089.stdout:604: expect (ObjNum 9 snap 0 seq_num 9) 2024-04-22T18:39:03.851 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:03.851 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:03.851 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:03.851 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:03.860 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:03.908 INFO:tasks.rados.rados.0.smithi089.stdout:607: finishing write tid 1 to smithi08937929-44 2024-04-22T18:39:03.908 INFO:tasks.rados.rados.0.smithi089.stdout:608: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-22T18:39:03.909 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:03.910+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:610: finishing write tid 1 to smithi08937929-12 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:610: finishing write tid 2 to smithi08937929-12 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:603: finishing write tid 3 to smithi08937929-48 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:604: done (12 left) 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:606: done (11 left) 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:608: done (10 left) 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:609: done (9 left) 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:615: done (8 left) 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:616: read oid 41 snap -1 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:616: expect (ObjNum 40 snap 0 seq_num 40) 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:617: write (excl) oid 33 current snap is 0 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:603: finishing write tid 4 to smithi08937929-48 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:603: finishing write tid 5 to smithi08937929-48 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:603: oid 48 updating version 0 to 19 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:603: oid 48 version 19 is already newer than 16 2024-04-22T18:39:04.001 INFO:tasks.rados.rados.0.smithi089.stdout:603: oid 48 version 19 is already newer than 18 2024-04-22T18:39:04.002 INFO:tasks.rados.rados.0.smithi089.stdout:603: oid 48 version 19 is already newer than 17 2024-04-22T18:39:04.002 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 19 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-04-22T18:39:04.002 INFO:tasks.rados.rados.0.smithi089.stdout:603: left oid 48 (ObjNum 50 snap 0 seq_num 50) 2024-04-22T18:39:04.002 INFO:tasks.rados.rados.0.smithi089.stdout:610: finishing write tid 3 to smithi08937929-12 2024-04-22T18:39:04.002 INFO:tasks.rados.rados.0.smithi089.stdout:617: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-04-22T18:39:04.003 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:04.003 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:04.003 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:04.003 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:04.003 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:04.007 INFO:tasks.rados.rados.0.smithi089.stdout:617: writing smithi08937929-33 from 524385 to 1077551 tid 1 2024-04-22T18:39:04.009 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:04.010+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:04.013 INFO:tasks.rados.rados.0.smithi089.stdout:617: writing smithi08937929-33 from 1602039 to 2012054 tid 2 2024-04-22T18:39:04.016 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:04.019 INFO:tasks.rados.rados.0.smithi089.stdout:617: writing smithi08937929-33 from 2577441 to 2924385 tid 3 2024-04-22T18:39:04.019 INFO:tasks.rados.rados.0.smithi089.stdout:603: done (9 left) 2024-04-22T18:39:04.019 INFO:tasks.rados.rados.0.smithi089.stdout:618: read oid 43 snap -1 2024-04-22T18:39:04.019 INFO:tasks.rados.rados.0.smithi089.stdout:618: expect (ObjNum 51 snap 0 seq_num 51) 2024-04-22T18:39:04.019 INFO:tasks.rados.rados.0.smithi089.stdout:619: delete oid 40 current snap is 0 2024-04-22T18:39:04.019 INFO:tasks.rados.rados.0.smithi089.stdout:610: finishing write tid 4 to smithi08937929-12 2024-04-22T18:39:04.019 INFO:tasks.rados.rados.0.smithi089.stdout:610: oid 12 updating version 0 to 25 2024-04-22T18:39:04.019 INFO:tasks.rados.rados.0.smithi089.stdout:610: oid 12 version 25 is already newer than 24 2024-04-22T18:39:04.019 INFO:tasks.rados.rados.0.smithi089.stdout:610: oid 12 updating version 25 to 26 2024-04-22T18:39:04.019 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 26 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-04-22T18:39:04.019 INFO:tasks.rados.rados.0.smithi089.stdout:610: left oid 12 (ObjNum 53 snap 0 seq_num 53) 2024-04-22T18:39:04.019 INFO:tasks.rados.rados.0.smithi089.stdout:605: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-22T18:39:04.100 INFO:tasks.rados.rados.0.smithi089.stdout:607: finishing write tid 2 to smithi08937929-44 2024-04-22T18:39:04.100 INFO:tasks.rados.rados.0.smithi089.stdout:607: finishing write tid 3 to smithi08937929-44 2024-04-22T18:39:04.100 INFO:tasks.rados.rados.0.smithi089.stdout:607: finishing write tid 4 to smithi08937929-44 2024-04-22T18:39:04.100 INFO:tasks.rados.rados.0.smithi089.stdout:607: finishing write tid 5 to smithi08937929-44 2024-04-22T18:39:04.100 INFO:tasks.rados.rados.0.smithi089.stdout:607: oid 44 updating version 0 to 31 2024-04-22T18:39:04.101 INFO:tasks.rados.rados.0.smithi089.stdout:607: oid 44 version 31 is already newer than 29 2024-04-22T18:39:04.101 INFO:tasks.rados.rados.0.smithi089.stdout:607: oid 44 version 31 is already newer than 28 2024-04-22T18:39:04.101 INFO:tasks.rados.rados.0.smithi089.stdout:607: oid 44 version 31 is already newer than 30 2024-04-22T18:39:04.101 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 31 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-04-22T18:39:04.101 INFO:tasks.rados.rados.0.smithi089.stdout:607: left oid 44 (ObjNum 52 snap 0 seq_num 52) 2024-04-22T18:39:04.101 INFO:tasks.rados.rados.0.smithi089.stdout:611: finishing set_chunk to oid 22 2024-04-22T18:39:04.101 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 22 set_chunk returned error code -95 offset: 1965568 length: 242176 2024-04-22T18:39:04.101 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-22T18:39:04.101 INFO:tasks.rados.rados.0.smithi089.stdout:612: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-22T18:39:04.110 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:04.110+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:04.122 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:04.122 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:04.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:04.122 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:04.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.122 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:25 4.d e27/20 4:bee1184f:::smithi08937929-7:head v 27'1, mlcod=0'0)", 2024-04-22T18:39:04.122 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.363505+0000", 2024-04-22T18:39:04.122 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.7592568770000003, 2024-04-22T18:39:04.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002987511, 2024-04-22T18:39:04.122 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.122 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.122 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.363505+0000", 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.363505+0000", 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.363505+0000", 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.363985+0000", 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047988499999999999 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.363995+0000", 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0258000000000001e-05 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364015+0000", 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0662e-05 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364870+0000", 2024-04-22T18:39:04.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00085470000000000001 2024-04-22T18:39:04.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364888+0000", 2024-04-22T18:39:04.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8023999999999999e-05 2024-04-22T18:39:04.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.366462+0000", 2024-04-22T18:39:04.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015740159999999999 2024-04-22T18:39:04.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.366492+0000", 2024-04-22T18:39:04.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0198000000000001e-05 2024-04-22T18:39:04.126 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.126 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.126 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.126 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:26 4.d e27/20 4:bee1184f:::smithi08937929-7:head v 27'2, mlcod=0'0)", 2024-04-22T18:39:04.126 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.364412+0000", 2024-04-22T18:39:04.126 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.7583495899999999, 2024-04-22T18:39:04.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002242754, 2024-04-22T18:39:04.126 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.126 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.126 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364412+0000", 2024-04-22T18:39:04.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364412+0000", 2024-04-22T18:39:04.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364412+0000", 2024-04-22T18:39:04.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:04.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364814+0000", 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000402264 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364823+0000", 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8470000000000002e-06 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364830+0000", 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7389999999999999e-06 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365589+0000", 2024-04-22T18:39:04.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075836499999999999 2024-04-22T18:39:04.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365600+0000", 2024-04-22T18:39:04.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1308000000000001e-05 2024-04-22T18:39:04.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.366630+0000", 2024-04-22T18:39:04.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001030188 2024-04-22T18:39:04.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.366655+0000", 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4686e-05 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:27 4.d e27/20 4:bee1184f:::smithi08937929-7:head v 27'3, mlcod=0'0)", 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.7575747330000002, 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101135974, 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365186+0000", 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365194+0000", 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3289999999999994e-06 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365200+0000", 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6590000000000004e-06 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465352+0000", 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10015208 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465511+0000", 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015880000000000001 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465570+0000", 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8952000000000001e-05 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466292+0000", 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072212100000000005 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466323+0000", 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0899999999999999e-05 2024-04-22T18:39:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:306 4.d e29/20 4:bee1184f:::smithi08937929-7:head v 29'9, mlcod=28'7)", 2024-04-22T18:39:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.093254+0000", 2024-04-22T18:39:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.029507594, 2024-04-22T18:39:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066511800000000002, 2024-04-22T18:39:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093254+0000", 2024-04-22T18:39:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093254+0000", 2024-04-22T18:39:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093253+0000", 2024-04-22T18:39:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093262+0000", 2024-04-22T18:39:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4089999999999992e-06 2024-04-22T18:39:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093272+0000", 2024-04-22T18:39:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0064999999999999e-05 2024-04-22T18:39:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093279+0000", 2024-04-22T18:39:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3089999999999998e-06 2024-04-22T18:39:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093318+0000", 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9054999999999998e-05 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093333+0000", 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4423000000000001e-05 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093893+0000", 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056035799999999995 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093919+0000", 2024-04-22T18:39:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6012000000000001e-05 2024-04-22T18:39:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:47 3.0 e30/19 3:30f16451:::smithi08937929-50 low_tier:head v 30'20, mlcod=29'18)", 2024-04-22T18:39:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.5246587160000002, 2024-04-22T18:39:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100803438, 2024-04-22T18:39:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598108+0000", 2024-04-22T18:39:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9119999999999997e-06 2024-04-22T18:39:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598118+0000", 2024-04-22T18:39:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0138e-05 2024-04-22T18:39:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598125+0000", 2024-04-22T18:39:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3010000000000001e-06 2024-04-22T18:39:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698224+0000", 2024-04-22T18:39:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099221 2024-04-22T18:39:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698239+0000", 2024-04-22T18:39:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5053e-05 2024-04-22T18:39:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698875+0000", 2024-04-22T18:39:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063560699999999995 2024-04-22T18:39:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698906+0000", 2024-04-22T18:39:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1547999999999997e-05 2024-04-22T18:39:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.5.13:4 3.0 e30/19 3:1ab765eb:::smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'21, mlcod=29'18)", 2024-04-22T18:39:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.698033+0000", 2024-04-22T18:39:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.424729234, 2024-04-22T18:39:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032895060000000002, 2024-04-22T18:39:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698033+0000", 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698033+0000", 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698031+0000", 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698048+0000", 2024-04-22T18:39:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6460000000000002e-05 2024-04-22T18:39:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698059+0000", 2024-04-22T18:39:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-04-22T18:39:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698067+0000", 2024-04-22T18:39:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4699999999999996e-06 2024-04-22T18:39:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698443+0000", 2024-04-22T18:39:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000375795 2024-04-22T18:39:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698464+0000", 2024-04-22T18:39:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1180000000000001e-05 2024-04-22T18:39:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.701297+0000", 2024-04-22T18:39:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028325859999999998 2024-04-22T18:39:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.701322+0000", 2024-04-22T18:39:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5496000000000001e-05 2024-04-22T18:39:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:28 3.0 e30/19 3:05e6fa13:::smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'22, mlcod=30'20)", 2024-04-22T18:39:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.797813+0000", 2024-04-22T18:39:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.3249488459999998, 2024-04-22T18:39:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00055748400000000002, 2024-04-22T18:39:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797813+0000", 2024-04-22T18:39:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797813+0000", 2024-04-22T18:39:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797811+0000", 2024-04-22T18:39:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797819+0000", 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.763e-06 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797833+0000", 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3728e-05 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797842+0000", 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1349999999999998e-06 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797880+0000", 2024-04-22T18:39:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7670999999999999e-05 2024-04-22T18:39:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797900+0000", 2024-04-22T18:39:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9792999999999998e-05 2024-04-22T18:39:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798346+0000", 2024-04-22T18:39:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044668900000000001 2024-04-22T18:39:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798370+0000", 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4113000000000001e-05 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.5.13:5 3.0 e30/19 3:1ab765eb:::smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=30'21)", 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.9203405929999997, 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101014042, 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202428+0000", 2024-04-22T18:39:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5889999999999997e-06 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202446+0000", 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8176999999999999e-05 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202456+0000", 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302599+0000", 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100142759 2024-04-22T18:39:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302628+0000", 2024-04-22T18:39:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9507999999999999e-05 2024-04-22T18:39:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303350+0000", 2024-04-22T18:39:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072151599999999995 2024-04-22T18:39:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303435+0000", 2024-04-22T18:39:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5486000000000001e-05 2024-04-22T18:39:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:53 3.0 e31/19 3:25a201b0:::smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 31'24, mlcod=30'22)", 2024-04-22T18:39:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.7196982590000003, 2024-04-22T18:39:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100751671, 2024-04-22T18:39:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403062+0000", 2024-04-22T18:39:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403070+0000", 2024-04-22T18:39:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8499999999999994e-06 2024-04-22T18:39:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403079+0000", 2024-04-22T18:39:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4620000000000002e-06 2024-04-22T18:39:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403087+0000", 2024-04-22T18:39:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4789999999999999e-06 2024-04-22T18:39:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503186+0000", 2024-04-22T18:39:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099271 2024-04-22T18:39:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503207+0000", 2024-04-22T18:39:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0973999999999999e-05 2024-04-22T18:39:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503773+0000", 2024-04-22T18:39:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056557600000000003 2024-04-22T18:39:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503815+0000", 2024-04-22T18:39:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2441999999999998e-05 2024-04-22T18:39:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:33 3.0 e31/19 3:29860007:::smithi08937929-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 31'25, mlcod=30'22)", 2024-04-22T18:39:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.502323+0000", 2024-04-22T18:39:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.620438279, 2024-04-22T18:39:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017670019999999999, 2024-04-22T18:39:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502323+0000", 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502323+0000", 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502320+0000", 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502337+0000", 2024-04-22T18:39:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6767000000000001e-05 2024-04-22T18:39:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502354+0000", 2024-04-22T18:39:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7180000000000002e-05 2024-04-22T18:39:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502364+0000", 2024-04-22T18:39:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0179e-05 2024-04-22T18:39:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503405+0000", 2024-04-22T18:39:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010405740000000001 2024-04-22T18:39:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503425+0000", 2024-04-22T18:39:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0324999999999999e-05 2024-04-22T18:39:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.504061+0000", 2024-04-22T18:39:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063580199999999998 2024-04-22T18:39:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.504090+0000", 2024-04-22T18:39:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9535000000000001e-05 2024-04-22T18:39:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:43 3.0 e32/19 3:29860007:::smithi08937929-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 32'30, mlcod=31'28)", 2024-04-22T18:39:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.628028+0000", 2024-04-22T18:39:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.4947337199999999, 2024-04-22T18:39:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00067497399999999995, 2024-04-22T18:39:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628028+0000", 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628028+0000", 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628028+0000", 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628037+0000", 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0119999999999994e-06 2024-04-22T18:39:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628047+0000", 2024-04-22T18:39:04.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0531e-05 2024-04-22T18:39:04.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628055+0000", 2024-04-22T18:39:04.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1550000000000007e-06 2024-04-22T18:39:04.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628099+0000", 2024-04-22T18:39:04.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3832000000000003e-05 2024-04-22T18:39:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628117+0000", 2024-04-22T18:39:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7649e-05 2024-04-22T18:39:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628671+0000", 2024-04-22T18:39:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00055402700000000001 2024-04-22T18:39:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628703+0000", 2024-04-22T18:39:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2177e-05 2024-04-22T18:39:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.166 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:04.166 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:39:04.167 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:04.168 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.1 to 1 (10000) 2024-04-22T18:39:04.168 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-22T18:39:04.183 INFO:tasks.rados.rados.0.smithi089.stdout:613: expect (ObjNum 36 snap 0 seq_num 36) 2024-04-22T18:39:04.227 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:04.227+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:04.269 INFO:tasks.rados.rados.0.smithi089.stdout:614: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-22T18:39:04.292 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:04.292 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:04.302 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:04.311 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:04.312+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:04.351 INFO:tasks.rados.rados.0.smithi089.stdout:616: expect (ObjNum 40 snap 0 seq_num 40) 2024-04-22T18:39:04.412 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:04.413+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:04.441 INFO:tasks.rados.rados.0.smithi089.stdout:617: finishing write tid 1 to smithi08937929-33 2024-04-22T18:39:04.441 INFO:tasks.rados.rados.0.smithi089.stdout:617: finishing write tid 2 to smithi08937929-33 2024-04-22T18:39:04.441 INFO:tasks.rados.rados.0.smithi089.stdout:605: done (10 left) 2024-04-22T18:39:04.441 INFO:tasks.rados.rados.0.smithi089.stdout:607: done (9 left) 2024-04-22T18:39:04.441 INFO:tasks.rados.rados.0.smithi089.stdout:610: done (8 left) 2024-04-22T18:39:04.442 INFO:tasks.rados.rados.0.smithi089.stdout:611: done (7 left) 2024-04-22T18:39:04.442 INFO:tasks.rados.rados.0.smithi089.stdout:612: done (6 left) 2024-04-22T18:39:04.442 INFO:tasks.rados.rados.0.smithi089.stdout:613: done (5 left) 2024-04-22T18:39:04.442 INFO:tasks.rados.rados.0.smithi089.stdout:614: done (4 left) 2024-04-22T18:39:04.442 INFO:tasks.rados.rados.0.smithi089.stdout:616: done (3 left) 2024-04-22T18:39:04.442 INFO:tasks.rados.rados.0.smithi089.stdout:619: done (2 left) 2024-04-22T18:39:04.442 INFO:tasks.rados.rados.0.smithi089.stdout:620: read oid 39 snap -1 2024-04-22T18:39:04.442 INFO:tasks.rados.rados.0.smithi089.stdout:620: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-22T18:39:04.442 INFO:tasks.rados.rados.0.smithi089.stdout:621: read oid 30 snap -1 2024-04-22T18:39:04.442 INFO:tasks.rados.rados.0.smithi089.stdout:621: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-22T18:39:04.442 INFO:tasks.rados.rados.0.smithi089.stdout:622: tier_flush oid 26 2024-04-22T18:39:04.442 INFO:tasks.rados.rados.0.smithi089.stdout:622: tier_flush oid 26 snap -1 2024-04-22T18:39:04.442 INFO:tasks.rados.rados.0.smithi089.stdout:618: expect (ObjNum 51 snap 0 seq_num 51) 2024-04-22T18:39:04.448 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:04.448 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:04.448 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:04.448 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:04.448 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:04.459 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:04.468 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:39:04.544 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:04.544+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:04.581 INFO:tasks.rados.rados.0.smithi089.stdout:617: finishing write tid 3 to smithi08937929-33 2024-04-22T18:39:04.581 INFO:tasks.rados.rados.0.smithi089.stdout:617: finishing write tid 4 to smithi08937929-33 2024-04-22T18:39:04.581 INFO:tasks.rados.rados.0.smithi089.stdout:617: finishing write tid 5 to smithi08937929-33 2024-04-22T18:39:04.581 INFO:tasks.rados.rados.0.smithi089.stdout:617: oid 33 updating version 0 to 15 2024-04-22T18:39:04.581 INFO:tasks.rados.rados.0.smithi089.stdout:617: oid 33 version 15 is already newer than 12 2024-04-22T18:39:04.581 INFO:tasks.rados.rados.0.smithi089.stdout:617: oid 33 version 15 is already newer than 14 2024-04-22T18:39:04.581 INFO:tasks.rados.rados.0.smithi089.stdout:617: oid 33 version 15 is already newer than 13 2024-04-22T18:39:04.581 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 15 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-04-22T18:39:04.581 INFO:tasks.rados.rados.0.smithi089.stdout:617: left oid 33 (ObjNum 55 snap 0 seq_num 55) 2024-04-22T18:39:04.581 INFO:tasks.rados.rados.0.smithi089.stdout:617: done (4 left) 2024-04-22T18:39:04.581 INFO:tasks.rados.rados.0.smithi089.stdout:618: done (3 left) 2024-04-22T18:39:04.582 INFO:tasks.rados.rados.0.smithi089.stdout:623: write oid 1 current snap is 0 2024-04-22T18:39:04.582 INFO:tasks.rados.rados.0.smithi089.stdout:623: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2024-04-22T18:39:04.588 INFO:tasks.rados.rados.0.smithi089.stdout:623: writing smithi08937929-1 from 557999 to 1261663 tid 1 2024-04-22T18:39:04.592 INFO:tasks.rados.rados.0.smithi089.stdout:623: writing smithi08937929-1 from 1719003 to 2132687 tid 2 2024-04-22T18:39:04.597 INFO:tasks.rados.rados.0.smithi089.stdout:623: writing smithi08937929-1 from 2632287 to 2957999 tid 3 2024-04-22T18:39:04.597 INFO:tasks.rados.rados.0.smithi089.stdout:624: tier_evict oid 47 2024-04-22T18:39:04.597 INFO:tasks.rados.rados.0.smithi089.stdout:624: tier_evict oid 47 snap -1 2024-04-22T18:39:04.597 INFO:tasks.rados.rados.0.smithi089.stdout:620: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-22T18:39:04.608 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:6 4.f e24/20 4:f83a159b:::smithi08937929-2:head v 24'2, mlcod=0'0)", 2024-04-22T18:39:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.402952344999999, 2024-04-22T18:39:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10183012700000001, 2024-04-22T18:39:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.206027+0000", 2024-04-22T18:39:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023705500000000001 2024-04-22T18:39:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.206038+0000", 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.148e-05 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.206046+0000", 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8679999999999999e-06 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.306227+0000", 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100181223 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.306266+0000", 2024-04-22T18:39:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8906000000000003e-05 2024-04-22T18:39:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.307598+0000", 2024-04-22T18:39:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001331561 2024-04-22T18:39:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.307620+0000", 2024-04-22T18:39:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.226e-05 2024-04-22T18:39:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:45 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'1, mlcod=0'0)", 2024-04-22T18:39:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.343630659, 2024-04-22T18:39:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14345418900000001, 2024-04-22T18:39:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265727+0000", 2024-04-22T18:39:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061502799999999999 2024-04-22T18:39:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265737+0000", 2024-04-22T18:39:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0278999999999999e-05 2024-04-22T18:39:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265744+0000", 2024-04-22T18:39:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1740000000000003e-06 2024-04-22T18:39:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407227+0000", 2024-04-22T18:39:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14148266900000001 2024-04-22T18:39:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407258+0000", 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1619000000000003e-05 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408534+0000", 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001275829 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408566+0000", 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1813999999999997e-05 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:46 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'2, mlcod=0'0)", 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.341006256, 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14304645699999999, 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268301+0000", 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056585700000000001 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268309+0000", 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2830000000000004e-06 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268312+0000", 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7179999999999998e-06 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407999+0000", 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13968636300000001 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408011+0000", 2024-04-22T18:39:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2116999999999999e-05 2024-04-22T18:39:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410762+0000", 2024-04-22T18:39:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002751524 2024-04-22T18:39:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410783+0000", 2024-04-22T18:39:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0270000000000001e-05 2024-04-22T18:39:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:47 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'3, mlcod=0'0)", 2024-04-22T18:39:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.339273686, 2024-04-22T18:39:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14132698599999999, 2024-04-22T18:39:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269468+0000", 2024-04-22T18:39:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269473+0000", 2024-04-22T18:39:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7790000000000003e-06 2024-04-22T18:39:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269478+0000", 2024-04-22T18:39:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0080000000000002e-06 2024-04-22T18:39:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269489+0000", 2024-04-22T18:39:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.075e-05 2024-04-22T18:39:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408595+0000", 2024-04-22T18:39:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13910662300000001 2024-04-22T18:39:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408611+0000", 2024-04-22T18:39:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6124e-05 2024-04-22T18:39:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410786+0000", 2024-04-22T18:39:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021743819999999999 2024-04-22T18:39:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410796+0000", 2024-04-22T18:39:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.007e-05 2024-04-22T18:39:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:9 4.f e27/20 4:f2c48a0e:::smithi08937929-3:head v 27'4, mlcod=24'2)", 2024-04-22T18:39:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.441905+0000", 2024-04-22T18:39:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.166837897000001, 2024-04-22T18:39:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10138892200000001, 2024-04-22T18:39:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.441905+0000", 2024-04-22T18:39:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.441905+0000", 2024-04-22T18:39:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.441902+0000", 2024-04-22T18:39:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442053+0000", 2024-04-22T18:39:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015104800000000001 2024-04-22T18:39:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442067+0000", 2024-04-22T18:39:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3964e-05 2024-04-22T18:39:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442073+0000", 2024-04-22T18:39:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1249999999999998e-06 2024-04-22T18:39:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442111+0000", 2024-04-22T18:39:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7988000000000003e-05 2024-04-22T18:39:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442127+0000", 2024-04-22T18:39:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.662e-05 2024-04-22T18:39:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543270+0000", 2024-04-22T18:39:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101143 2024-04-22T18:39:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543293+0000", 2024-04-22T18:39:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3147e-05 2024-04-22T18:39:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:10 4.f e27/20 4:f2c48a0e:::smithi08937929-3:head v 27'5, mlcod=24'2)", 2024-04-22T18:39:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.166661116, 2024-04-22T18:39:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10161616800000001, 2024-04-22T18:39:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442159+0000", 2024-04-22T18:39:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7935000000000005e-05 2024-04-22T18:39:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442163+0000", 2024-04-22T18:39:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8430000000000003e-06 2024-04-22T18:39:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442166+0000", 2024-04-22T18:39:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0759999999999999e-06 2024-04-22T18:39:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542745+0000", 2024-04-22T18:39:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10057941200000001 2024-04-22T18:39:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542770+0000", 2024-04-22T18:39:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5134e-05 2024-04-22T18:39:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543679+0000", 2024-04-22T18:39:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00090833099999999996 2024-04-22T18:39:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543697+0000", 2024-04-22T18:39:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8876e-05 2024-04-22T18:39:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:11 4.f e27/20 4:f2c48a0e:::smithi08937929-3:head v 27'6, mlcod=24'2)", 2024-04-22T18:39:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.442170+0000", 2024-04-22T18:39:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.166572702, 2024-04-22T18:39:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101722358, 2024-04-22T18:39:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442170+0000", 2024-04-22T18:39:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442170+0000", 2024-04-22T18:39:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442169+0000", 2024-04-22T18:39:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442174+0000", 2024-04-22T18:39:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0350000000000003e-06 2024-04-22T18:39:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442177+0000", 2024-04-22T18:39:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0580000000000002e-06 2024-04-22T18:39:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442181+0000", 2024-04-22T18:39:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1649999999999999e-06 2024-04-22T18:39:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543319+0000", 2024-04-22T18:39:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101138408 2024-04-22T18:39:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543336+0000", 2024-04-22T18:39:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6509999999999999e-05 2024-04-22T18:39:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543880+0000", 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054429900000000004 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543892+0000", 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2138e-05 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:77 4.0 4:01baf5a9:::smithi08937929-20:head [write 719333~462437,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.505878+0000", 2024-04-22T18:39:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.102864901, 2024-04-22T18:39:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10366277, 2024-04-22T18:39:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 77 2024-04-22T18:39:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.505878+0000", 2024-04-22T18:39:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.505878+0000", 2024-04-22T18:39:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.505876+0000", 2024-04-22T18:39:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506122+0000", 2024-04-22T18:39:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024626900000000002 2024-04-22T18:39:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506132+0000", 2024-04-22T18:39:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0216e-05 2024-04-22T18:39:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506139+0000", 2024-04-22T18:39:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6030000000000001e-06 2024-04-22T18:39:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506191+0000", 2024-04-22T18:39:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.185e-05 2024-04-22T18:39:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506333+0000", 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014197299999999999 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506480+0000", 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014780199999999999 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.507794+0000", 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001313587 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609429+0000", 2024-04-22T18:39:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101634658 2024-04-22T18:39:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609493+0000", 2024-04-22T18:39:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4777999999999995e-05 2024-04-22T18:39:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609540+0000", 2024-04-22T18:39:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6820000000000002e-05 2024-04-22T18:39:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:78 4.0 4:01baf5a9:::smithi08937929-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.096807565000001, 2024-04-22T18:39:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.398522192, 2024-04-22T18:39:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 78 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511934+0000", 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512012+0000", 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8329000000000002e-05 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512023+0000", 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0755e-05 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512028+0000", 2024-04-22T18:39:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6470000000000003e-06 2024-04-22T18:39:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512068+0000", 2024-04-22T18:39:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9721999999999997e-05 2024-04-22T18:39:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512125+0000", 2024-04-22T18:39:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6564999999999998e-05 2024-04-22T18:39:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512200+0000", 2024-04-22T18:39:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5708000000000001e-05 2024-04-22T18:39:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513131+0000", 2024-04-22T18:39:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093063 2024-04-22T18:39:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910394+0000", 2024-04-22T18:39:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.397263388 2024-04-22T18:39:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910430+0000", 2024-04-22T18:39:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5485999999999999e-05 2024-04-22T18:39:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910457+0000", 2024-04-22T18:39:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7086000000000001e-05 2024-04-22T18:39:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:79 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.096706909, 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39855419399999997, 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 79 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512035+0000", 2024-04-22T18:39:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512045+0000", 2024-04-22T18:39:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0379e-05 2024-04-22T18:39:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512051+0000", 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6030000000000004e-06 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512057+0000", 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7270000000000004e-06 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512756+0000", 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069954000000000001 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512806+0000", 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9922000000000001e-05 2024-04-22T18:39:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512927+0000", 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012059 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.515883+0000", 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029563739999999999 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910510+0000", 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39462745199999999 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910542+0000", 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1225e-05 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910590+0000", 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8052e-05 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:80 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.096681808, 2024-04-22T18:39:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39866376999999997, 2024-04-22T18:39:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 80 2024-04-22T18:39:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512060+0000", 2024-04-22T18:39:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512063+0000", 2024-04-22T18:39:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.999e-06 2024-04-22T18:39:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512069+0000", 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.891e-06 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512073+0000", 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4429999999999996e-06 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513084+0000", 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010102609999999999 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513123+0000", 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9237999999999999e-05 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910618+0000", 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39749549299999998 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910670+0000", 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1221000000000003e-05 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910724+0000", 2024-04-22T18:39:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4777000000000001e-05 2024-04-22T18:39:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:33 4.f e27/20 4:f0ed3d49:::smithi08937929-9:head v 27'7, mlcod=24'2)", 2024-04-22T18:39:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.609449+0000", 2024-04-22T18:39:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.9992938739999992, 2024-04-22T18:39:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10183384600000001, 2024-04-22T18:39:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609449+0000", 2024-04-22T18:39:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609449+0000", 2024-04-22T18:39:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609448+0000", 2024-04-22T18:39:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609626+0000", 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017809600000000001 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609638+0000", 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2045000000000001e-05 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709771+0000", 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013347 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.710008+0000", 2024-04-22T18:39:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023696200000000001 2024-04-22T18:39:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.710045+0000", 2024-04-22T18:39:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6903000000000002e-05 2024-04-22T18:39:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.711263+0000", 2024-04-22T18:39:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012178110000000001 2024-04-22T18:39:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.711282+0000", 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9525e-05 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:34 4.f e27/20 4:f0ed3d49:::smithi08937929-9:head v 27'8, mlcod=24'2)", 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.709796+0000", 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.8989468729999999, 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10102066, 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709796+0000", 2024-04-22T18:39:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709796+0000", 2024-04-22T18:39:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709793+0000", 2024-04-22T18:39:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709914+0000", 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000120864 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709932+0000", 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8125999999999999e-05 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709949+0000", 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6480000000000001e-05 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.809972+0000", 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10002336000000001 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.809986+0000", 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3913e-05 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810789+0000", 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00080341899999999999 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810816+0000", 2024-04-22T18:39:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6880999999999999e-05 2024-04-22T18:39:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:35 4.f e27/20 4:f0ed3d49:::smithi08937929-9:head v 27'9, mlcod=24'2)", 2024-04-22T18:39:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.709964+0000", 2024-04-22T18:39:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.8987781930000001, 2024-04-22T18:39:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101040596, 2024-04-22T18:39:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709964+0000", 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709964+0000", 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709963+0000", 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709975+0000", 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.236e-05 2024-04-22T18:39:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709990+0000", 2024-04-22T18:39:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4296e-05 2024-04-22T18:39:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810061+0000", 2024-04-22T18:39:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10007135 2024-04-22T18:39:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810402+0000", 2024-04-22T18:39:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00034100299999999999 2024-04-22T18:39:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810422+0000", 2024-04-22T18:39:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9749e-05 2024-04-22T18:39:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810973+0000", 2024-04-22T18:39:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00055107799999999998 2024-04-22T18:39:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.811005+0000", 2024-04-22T18:39:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079999999999998e-05 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:5 3.2 e30/29 3:663b82a1:::smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=0'0)", 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.38124249, 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20090134500000001, 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227499+0000", 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227506+0000", 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6540000000000002e-06 2024-04-22T18:39:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227516+0000", 2024-04-22T18:39:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0054000000000001e-05 2024-04-22T18:39:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327607+0000", 2024-04-22T18:39:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009167099999999 2024-04-22T18:39:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327688+0000", 2024-04-22T18:39:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1000000000000004e-05 2024-04-22T18:39:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327719+0000", 2024-04-22T18:39:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0664000000000003e-05 2024-04-22T18:39:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428371+0000", 2024-04-22T18:39:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10065239099999999 2024-04-22T18:39:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428401+0000", 2024-04-22T18:39:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9839e-05 2024-04-22T18:39:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:19 3.2 e30/29 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'24, mlcod=28'19)", 2024-04-22T18:39:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.327656+0000", 2024-04-22T18:39:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.2810863100000001, 2024-04-22T18:39:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101011095, 2024-04-22T18:39:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327656+0000", 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327656+0000", 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327655+0000", 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327667+0000", 2024-04-22T18:39:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2480000000000001e-05 2024-04-22T18:39:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327679+0000", 2024-04-22T18:39:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.22e-05 2024-04-22T18:39:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327686+0000", 2024-04-22T18:39:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8580000000000002e-06 2024-04-22T18:39:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428037+0000", 2024-04-22T18:39:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10035092399999999 2024-04-22T18:39:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428057+0000", 2024-04-22T18:39:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0030999999999998e-05 2024-04-22T18:39:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428638+0000", 2024-04-22T18:39:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058120100000000005 2024-04-22T18:39:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428667+0000", 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8741000000000001e-05 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:334 4.c e30/20 4:3227431c:::smithi08937929-35:head v 30'18, mlcod=30'16)", 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.698741+0000", 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.9100013269999998, 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10216847599999999, 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698741+0000", 2024-04-22T18:39:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698741+0000", 2024-04-22T18:39:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698740+0000", 2024-04-22T18:39:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698747+0000", 2024-04-22T18:39:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9199999999999998e-06 2024-04-22T18:39:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698756+0000", 2024-04-22T18:39:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1249999999999999e-06 2024-04-22T18:39:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698762+0000", 2024-04-22T18:39:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798915+0000", 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10015304899999999 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798937+0000", 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1335000000000002e-05 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.800879+0000", 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001942088 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.800910+0000", 2024-04-22T18:39:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0892000000000002e-05 2024-04-22T18:39:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:419 4.0 4:01baf5a9:::smithi08937929-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:39:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.2065207879999997, 2024-04-22T18:39:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30324478199999999, 2024-04-22T18:39:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 419 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402219+0000", 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402229+0000", 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2550000000000003e-06 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402242+0000", 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2989999999999999e-05 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402250+0000", 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7900000000000005e-06 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404085+0000", 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001834397 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404092+0000", 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7889999999999997e-06 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502534+0000", 2024-04-22T18:39:04.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098442331999999994 2024-04-22T18:39:04.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502631+0000", 2024-04-22T18:39:04.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7617999999999999e-05 2024-04-22T18:39:04.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:04.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604414+0000", 2024-04-22T18:39:04.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101782847 2024-04-22T18:39:04.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604888+0000", 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047325999999999998 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705376+0000", 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100488623 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705424+0000", 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7964999999999999e-05 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705466+0000", 2024-04-22T18:39:04.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2274999999999998e-05 2024-04-22T18:39:04.688 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.688 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.688 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.688 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.3.13:13 3.2 e31/29 3:549b9960:::smithi08937929-20 low_tier:head v 31'29, mlcod=30'27)", 2024-04-22T18:39:04.688 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.503273+0000", 2024-04-22T18:39:04.688 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.105469931, 2024-04-22T18:39:04.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10079894, 2024-04-22T18:39:04.688 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.688 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.688 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503273+0000", 2024-04-22T18:39:04.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503273+0000", 2024-04-22T18:39:04.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503272+0000", 2024-04-22T18:39:04.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:04.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503278+0000", 2024-04-22T18:39:04.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3210000000000001e-06 2024-04-22T18:39:04.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503294+0000", 2024-04-22T18:39:04.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.522e-05 2024-04-22T18:39:04.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503301+0000", 2024-04-22T18:39:04.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6879999999999999e-06 2024-04-22T18:39:04.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.603425+0000", 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100124017 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.603451+0000", 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5510000000000001e-05 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604033+0000", 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058202199999999996 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604071+0000", 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8587000000000002e-05 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:70 3.2 e32/29 3:5f0623ce:::smithi08937929-38 low_tier:head v 32'37, mlcod=31'35)", 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 2.0803037569999998, 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20070027900000001, 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:04.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:04.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:04.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:04.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:04.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528438+0000", 2024-04-22T18:39:04.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:04.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:04.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528443+0000", 2024-04-22T18:39:04.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8640000000000004e-06 2024-04-22T18:39:04.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:04.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528453+0000", 2024-04-22T18:39:04.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3910000000000004e-06 2024-04-22T18:39:04.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:04.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528459+0000", 2024-04-22T18:39:04.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-04-22T18:39:04.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:04.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728627+0000", 2024-04-22T18:39:04.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20016778700000001 2024-04-22T18:39:04.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:04.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728660+0000", 2024-04-22T18:39:04.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3024000000000001e-05 2024-04-22T18:39:04.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:04.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729107+0000", 2024-04-22T18:39:04.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044719200000000001 2024-04-22T18:39:04.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:04.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:04.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:04.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729139+0000", 2024-04-22T18:39:04.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1625000000000003e-05 2024-04-22T18:39:04.696 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.696 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.696 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.696 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:04.696 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:04.696 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:04.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:04.698 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:04.698 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:04.698 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:04.698 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:04.699 INFO:tasks.rados.rados.0.smithi089.stdout:621: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-22T18:39:04.699 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:04.699 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:04.700 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:04.698+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:04.711 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:04.727 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:04.727 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:04.743 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:04.743 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:04.759 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:04.760 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:04.770 INFO:tasks.rados.rados.0.smithi089.stdout:622: got (0) Success 2024-04-22T18:39:04.771 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-22T18:39:04.771 INFO:tasks.rados.rados.0.smithi089.stdout:623: finishing write tid 1 to smithi08937929-1 2024-04-22T18:39:04.771 INFO:tasks.rados.rados.0.smithi089.stdout:623: finishing write tid 2 to smithi08937929-1 2024-04-22T18:39:04.771 INFO:tasks.rados.rados.0.smithi089.stdout:620: done (4 left) 2024-04-22T18:39:04.771 INFO:tasks.rados.rados.0.smithi089.stdout:621: done (3 left) 2024-04-22T18:39:04.771 INFO:tasks.rados.rados.0.smithi089.stdout:622: done (2 left) 2024-04-22T18:39:04.771 INFO:tasks.rados.rados.0.smithi089.stdout:625: tier_flush oid 16 2024-04-22T18:39:04.771 INFO:tasks.rados.rados.0.smithi089.stdout:625: tier_flush oid 16 snap -1 2024-04-22T18:39:04.771 INFO:tasks.rados.rados.0.smithi089.stdout:626: set_chunk oid 33 target oid 2024-04-22T18:39:04.777 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:04.778 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:04.797 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:04.798+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:04.812 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:04.812 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:04.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:04.906 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:04.906+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:04.965 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:04.965 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:04.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:04.988 INFO:tasks.rados.rados.0.smithi089.stdout:626: set_chunk oid 33 offset: 50688 length: 2123264 target oid e84dcf1525446de6471b1e6cb3fa1c1f8f547580ab0d342e976a4fd27976afa4 offset: 0 snap -1 2024-04-22T18:39:04.988 INFO:tasks.rados.rados.0.smithi089.stdout:627: read oid 37 snap -1 2024-04-22T18:39:04.988 INFO:tasks.rados.rados.0.smithi089.stdout:627: expect (ObjNum 36 snap 0 seq_num 36) 2024-04-22T18:39:04.988 INFO:tasks.rados.rados.0.smithi089.stdout:624: got (22) Invalid argument 2024-04-22T18:39:04.988 INFO:tasks.rados.rados.0.smithi089.stdout:628: read oid 38 snap -1 2024-04-22T18:39:04.988 INFO:tasks.rados.rados.0.smithi089.stdout:628: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-22T18:39:04.988 INFO:tasks.rados.rados.0.smithi089.stdout:624: done (5 left) 2024-04-22T18:39:04.988 INFO:tasks.rados.rados.0.smithi089.stdout:629: read oid 27 snap -1 2024-04-22T18:39:04.988 INFO:tasks.rados.rados.0.smithi089.stdout:629: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-22T18:39:04.988 INFO:tasks.rados.rados.0.smithi089.stdout:630: read oid 8 snap -1 2024-04-22T18:39:04.989 INFO:tasks.rados.rados.0.smithi089.stdout:630: expect deleted 2024-04-22T18:39:04.989 INFO:tasks.rados.rados.0.smithi089.stdout:631: write (excl) oid 40 current snap is 0 2024-04-22T18:39:04.989 INFO:tasks.rados.rados.0.smithi089.stdout:623: finishing write tid 3 to smithi08937929-1 2024-04-22T18:39:04.989 INFO:tasks.rados.rados.0.smithi089.stdout:623: finishing write tid 4 to smithi08937929-1 2024-04-22T18:39:04.989 INFO:tasks.rados.rados.0.smithi089.stdout:631: seq_num 58 ranges {574806=778913,2061132=557342} 2024-04-22T18:39:04.994 INFO:tasks.rados.rados.0.smithi089.stdout:631: writing smithi08937929-40 from 574806 to 1353719 tid 1 2024-04-22T18:39:05.001 INFO:tasks.rados.rados.0.smithi089.stdout:631: writing smithi08937929-40 from 2061132 to 2618474 tid 2 2024-04-22T18:39:05.001 INFO:tasks.rados.rados.0.smithi089.stdout:632: read oid 2 snap -1 2024-04-22T18:39:05.002 INFO:tasks.rados.rados.0.smithi089.stdout:632: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-22T18:39:05.002 INFO:tasks.rados.rados.0.smithi089.stdout:633: set_chunk oid 31 target oid 2024-04-22T18:39:05.014 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:05.015+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:05.100 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:05.101+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:05.115 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:05.115 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:05.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:05.151 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.2 to 1 (10000) 2024-04-22T18:39:05.151 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:39:05.165 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-22T18:39:05.209 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:05.210+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:05.269 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:05.270 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:05.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:05.319 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:05.320+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:05.402 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:05.403+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:05.419 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:05.419 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:05.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:05.452 INFO:tasks.rados.rados.0.smithi089.stdout:633: set_chunk oid 31 offset: 2283008 length: 30720 target oid 4c7eea521d2218c5965fd3666c694a967a939e29a6928d528b6ed1101176b8ac offset: 0 snap -1 2024-04-22T18:39:05.452 INFO:tasks.rados.rados.0.smithi089.stdout:634: read oid 3 snap -1 2024-04-22T18:39:05.452 INFO:tasks.rados.rados.0.smithi089.stdout:634: expect (ObjNum 2 snap 0 seq_num 2) 2024-04-22T18:39:05.452 INFO:tasks.rados.rados.0.smithi089.stdout:635: write oid 34 current snap is 0 2024-04-22T18:39:05.452 INFO:tasks.rados.rados.0.smithi089.stdout:623: finishing write tid 5 to smithi08937929-1 2024-04-22T18:39:05.452 INFO:tasks.rados.rados.0.smithi089.stdout:623: oid 1 updating version 0 to 19 2024-04-22T18:39:05.452 INFO:tasks.rados.rados.0.smithi089.stdout:623: oid 1 version 19 is already newer than 17 2024-04-22T18:39:05.453 INFO:tasks.rados.rados.0.smithi089.stdout:623: oid 1 version 19 is already newer than 18 2024-04-22T18:39:05.453 INFO:tasks.rados.rados.0.smithi089.stdout:623: oid 1 version 19 is already newer than 16 2024-04-22T18:39:05.453 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 19 (ObjNum 57 snap 0 seq_num 57) dirty exists 2024-04-22T18:39:05.453 INFO:tasks.rados.rados.0.smithi089.stdout:623: left oid 1 (ObjNum 57 snap 0 seq_num 57) 2024-04-22T18:39:05.453 INFO:tasks.rados.rados.0.smithi089.stdout:626: finishing set_chunk to oid 33 2024-04-22T18:39:05.453 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 33 set_chunk returned error code -95 offset: 50688 length: 2123264 2024-04-22T18:39:05.453 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 0 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-04-22T18:39:05.453 INFO:tasks.rados.rados.0.smithi089.stdout:627: expect (ObjNum 36 snap 0 seq_num 36) 2024-04-22T18:39:05.503 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:05.503+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:05.528 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:05.528 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:05.533 INFO:tasks.rados.rados.0.smithi089.stdout:625: got (0) Success 2024-04-22T18:39:05.533 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-22T18:39:05.533 INFO:tasks.rados.rados.0.smithi089.stdout:631: finishing write tid 1 to smithi08937929-40 2024-04-22T18:39:05.533 INFO:tasks.rados.rados.0.smithi089.stdout:628: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-22T18:39:05.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:05.604 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:05.604+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:05.617 INFO:tasks.rados.rados.0.smithi089.stdout:629: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-22T18:39:05.638 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:05.638 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:05.648 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:05.693 INFO:tasks.rados.rados.0.smithi089.stdout:632: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-22T18:39:05.705 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:05.706+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:05.748 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:05.748 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:05.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:05.757 INFO:tasks.rados.rados.0.smithi089.stdout:631: finishing write tid 2 to smithi08937929-40 2024-04-22T18:39:05.758 INFO:tasks.rados.rados.0.smithi089.stdout:631: finishing write tid 3 to smithi08937929-40 2024-04-22T18:39:05.758 INFO:tasks.rados.rados.0.smithi089.stdout:631: finishing write tid 4 to smithi08937929-40 2024-04-22T18:39:05.758 INFO:tasks.rados.rados.0.smithi089.stdout:631: oid 40 updating version 0 to 18 2024-04-22T18:39:05.758 INFO:tasks.rados.rados.0.smithi089.stdout:631: oid 40 version 18 is already newer than 17 2024-04-22T18:39:05.758 INFO:tasks.rados.rados.0.smithi089.stdout:631: oid 40 updating version 18 to 19 2024-04-22T18:39:05.758 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 19 (ObjNum 58 snap 0 seq_num 58) dirty exists 2024-04-22T18:39:05.758 INFO:tasks.rados.rados.0.smithi089.stdout:631: left oid 40 (ObjNum 58 snap 0 seq_num 58) 2024-04-22T18:39:05.758 INFO:tasks.rados.rados.0.smithi089.stdout:635: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-04-22T18:39:05.762 INFO:tasks.rados.rados.0.smithi089.stdout:635: writing smithi08937929-34 from 591613 to 1045775 tid 1 2024-04-22T18:39:05.768 INFO:tasks.rados.rados.0.smithi089.stdout:635: writing smithi08937929-34 from 1719614 to 2136967 tid 2 2024-04-22T18:39:05.773 INFO:tasks.rados.rados.0.smithi089.stdout:635: writing smithi08937929-34 from 2570780 to 2991613 tid 3 2024-04-22T18:39:05.773 INFO:tasks.rados.rados.0.smithi089.stdout:623: done (11 left) 2024-04-22T18:39:05.773 INFO:tasks.rados.rados.0.smithi089.stdout:625: done (10 left) 2024-04-22T18:39:05.773 INFO:tasks.rados.rados.0.smithi089.stdout:626: done (9 left) 2024-04-22T18:39:05.773 INFO:tasks.rados.rados.0.smithi089.stdout:627: done (8 left) 2024-04-22T18:39:05.773 INFO:tasks.rados.rados.0.smithi089.stdout:628: done (7 left) 2024-04-22T18:39:05.773 INFO:tasks.rados.rados.0.smithi089.stdout:629: done (6 left) 2024-04-22T18:39:05.773 INFO:tasks.rados.rados.0.smithi089.stdout:630: done (5 left) 2024-04-22T18:39:05.773 INFO:tasks.rados.rados.0.smithi089.stdout:631: done (4 left) 2024-04-22T18:39:05.773 INFO:tasks.rados.rados.0.smithi089.stdout:632: done (3 left) 2024-04-22T18:39:05.773 INFO:tasks.rados.rados.0.smithi089.stdout:636: write oid 44 current snap is 0 2024-04-22T18:39:05.774 INFO:tasks.rados.rados.0.smithi089.stdout:636: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-04-22T18:39:05.779 INFO:tasks.rados.rados.0.smithi089.stdout:636: writing smithi08937929-44 from 608420 to 1137831 tid 1 2024-04-22T18:39:05.785 INFO:tasks.rados.rados.0.smithi089.stdout:636: writing smithi08937929-44 from 1778096 to 2339107 tid 2 2024-04-22T18:39:05.789 INFO:tasks.rados.rados.0.smithi089.stdout:636: writing smithi08937929-44 from 2881850 to 3008420 tid 3 2024-04-22T18:39:05.789 INFO:tasks.rados.rados.0.smithi089.stdout:637: read oid 42 snap -1 2024-04-22T18:39:05.789 INFO:tasks.rados.rados.0.smithi089.stdout:637: expect (ObjNum 41 snap 0 seq_num 41) 2024-04-22T18:39:05.789 INFO:tasks.rados.rados.0.smithi089.stdout:638: set_chunk oid 38 target oid 2024-04-22T18:39:05.841 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:05.842+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:05.893 INFO:tasks.rados.rados.0.smithi089.stdout:638: set_chunk oid 38 offset: 677376 length: 1786368 target oid 1191d350d79a33806f503cd30b7083325edabffc109c4b1339e2919eef143552 offset: 0 snap -1 2024-04-22T18:39:05.893 INFO:tasks.rados.rados.0.smithi089.stdout:639: read oid 22 snap -1 2024-04-22T18:39:05.893 INFO:tasks.rados.rados.0.smithi089.stdout:639: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-22T18:39:05.893 INFO:tasks.rados.rados.0.smithi089.stdout:640: read oid 21 snap -1 2024-04-22T18:39:05.893 INFO:tasks.rados.rados.0.smithi089.stdout:640: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-22T18:39:05.894 INFO:tasks.rados.rados.0.smithi089.stdout:641: read oid 43 snap -1 2024-04-22T18:39:05.894 INFO:tasks.rados.rados.0.smithi089.stdout:641: expect (ObjNum 51 snap 0 seq_num 51) 2024-04-22T18:39:05.894 INFO:tasks.rados.rados.0.smithi089.stdout:642: delete oid 13 current snap is 0 2024-04-22T18:39:05.894 INFO:tasks.rados.rados.0.smithi089.stdout:634: expect (ObjNum 2 snap 0 seq_num 2) 2024-04-22T18:39:05.902 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:05.902 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:05.906 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:05.907+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:05.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:05.958 INFO:tasks.rados.rados.0.smithi089.stdout:633: finishing set_chunk to oid 31 2024-04-22T18:39:05.958 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 3 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-22T18:39:05.958 INFO:tasks.rados.rados.0.smithi089.stdout:636: finishing write tid 1 to smithi08937929-44 2024-04-22T18:39:05.958 INFO:tasks.rados.rados.0.smithi089.stdout:636: finishing write tid 2 to smithi08937929-44 2024-04-22T18:39:05.958 INFO:tasks.rados.rados.0.smithi089.stdout:638: finishing set_chunk to oid 38 2024-04-22T18:39:05.958 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 38 set_chunk returned error code -95 offset: 677376 length: 1786368 2024-04-22T18:39:05.958 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-22T18:39:05.958 INFO:tasks.rados.rados.0.smithi089.stdout:633: done (9 left) 2024-04-22T18:39:05.958 INFO:tasks.rados.rados.0.smithi089.stdout:634: done (8 left) 2024-04-22T18:39:05.959 INFO:tasks.rados.rados.0.smithi089.stdout:638: done (7 left) 2024-04-22T18:39:05.959 INFO:tasks.rados.rados.0.smithi089.stdout:642: done (6 left) 2024-04-22T18:39:05.959 INFO:tasks.rados.rados.0.smithi089.stdout:643: read oid 13 snap -1 2024-04-22T18:39:05.959 INFO:tasks.rados.rados.0.smithi089.stdout:643: expect deleted 2024-04-22T18:39:05.959 INFO:tasks.rados.rados.0.smithi089.stdout:644: read oid 45 snap -1 2024-04-22T18:39:05.959 INFO:tasks.rados.rados.0.smithi089.stdout:644: expect (ObjNum 44 snap 0 seq_num 44) 2024-04-22T18:39:05.959 INFO:tasks.rados.rados.0.smithi089.stdout:645: set_chunk oid 16 target oid 2024-04-22T18:39:06.007 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:06.008+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:06.053 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:06.054 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:06.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:06.146 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:06.147+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:06.150 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:06.150+0000 7f3e7be88640 -1 osd.0 35 handle_osd_map injecting map crc failure 2024-04-22T18:39:06.151 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.3 to 1 (10000) 2024-04-22T18:39:06.165 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-22T18:39:06.197 INFO:tasks.rados.rados.0.smithi089.stdout:645: set_chunk oid 16 offset: 158208 length: 476672 target oid 59bff1359e5fb6fca79c117a6cf8aa5f12349977bac48c48f13b0eba610d7b37 offset: 0 snap -1 2024-04-22T18:39:06.197 INFO:tasks.rados.rados.0.smithi089.stdout:646: read oid 26 snap -1 2024-04-22T18:39:06.197 INFO:tasks.rados.rados.0.smithi089.stdout:646: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-22T18:39:06.197 INFO:tasks.rados.rados.0.smithi089.stdout:647: read oid 39 snap -1 2024-04-22T18:39:06.197 INFO:tasks.rados.rados.0.smithi089.stdout:647: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-22T18:39:06.197 INFO:tasks.rados.rados.0.smithi089.stdout:648: read oid 35 snap -1 2024-04-22T18:39:06.197 INFO:tasks.rados.rados.0.smithi089.stdout:648: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-22T18:39:06.198 INFO:tasks.rados.rados.0.smithi089.stdout:649: set_chunk oid 2 target oid 2024-04-22T18:39:06.206 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:06.207 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:06.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:06.240 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:06.241+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:06.309 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:06.310+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:06.359 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:06.359 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:06.368 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:06.410 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:06.411+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:06.471 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:06.471 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:06.480 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:06.525 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:06.525+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:06.571 INFO:tasks.rados.rados.0.smithi089.stdout:649: set_chunk oid 2 offset: 887808 length: 89088 target oid 0466c7e53723cb91b9bf67b4dcb995abdfb7e46710e928c03ab311e627460ae2 offset: 0 snap -1 2024-04-22T18:39:06.571 INFO:tasks.rados.rados.0.smithi089.stdout:650: tier_evict oid 9 2024-04-22T18:39:06.571 INFO:tasks.rados.rados.0.smithi089.stdout:650: tier_evict oid 9 snap -1 2024-04-22T18:39:06.571 INFO:tasks.rados.rados.0.smithi089.stdout:636: finishing write tid 3 to smithi08937929-44 2024-04-22T18:39:06.571 INFO:tasks.rados.rados.0.smithi089.stdout:651: read oid 4 snap -1 2024-04-22T18:39:06.571 INFO:tasks.rados.rados.0.smithi089.stdout:651: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-22T18:39:06.571 INFO:tasks.rados.rados.0.smithi089.stdout:652: write (excl) oid 25 current snap is 0 2024-04-22T18:39:06.571 INFO:tasks.rados.rados.0.smithi089.stdout:652: seq_num 62 ranges {642034=796262,1895060=564680} 2024-04-22T18:39:06.578 INFO:tasks.rados.rados.0.smithi089.stdout:652: writing smithi08937929-25 from 642034 to 1438296 tid 1 2024-04-22T18:39:06.582 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:06.582 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:06.584 INFO:tasks.rados.rados.0.smithi089.stdout:652: writing smithi08937929-25 from 1895060 to 2459740 tid 2 2024-04-22T18:39:06.584 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:06.584 INFO:tasks.rados.rados.0.smithi089.stdout:637: expect (ObjNum 41 snap 0 seq_num 41) 2024-04-22T18:39:06.593 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:06.613 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:06.614+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:06.680 INFO:tasks.rados.rados.0.smithi089.stdout:641: expect (ObjNum 51 snap 0 seq_num 51) 2024-04-22T18:39:06.714 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:06.714+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:06.738 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:06.738 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:06.747 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:06.789 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:39:06.815 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:06.815+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:06.818 INFO:tasks.rados.rados.0.smithi089.stdout:639: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-22T18:39:06.894 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:06.894 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:06.902 INFO:tasks.rados.rados.0.smithi089.stdout:644: expect (ObjNum 44 snap 0 seq_num 44) 2024-04-22T18:39:06.906 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:06.914 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:06.914 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:06.915 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:06.915+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:06.931 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:06.932 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:06.948 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:06.948 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:06.963 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:06.964 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:06.980 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:06.981 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:06.993 INFO:tasks.rados.rados.0.smithi089.stdout:635: finishing write tid 1 to smithi08937929-34 2024-04-22T18:39:06.993 INFO:tasks.rados.rados.0.smithi089.stdout:636: finishing write tid 4 to smithi08937929-44 2024-04-22T18:39:06.993 INFO:tasks.rados.rados.0.smithi089.stdout:640: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-22T18:39:06.997 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:06.997 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:07.010 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:07.010 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:07.010 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:07.011 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:07.011 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:07.014 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:07.014 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:07.015 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:07.016+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:07.019 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:07.031 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:07.031 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:07.075 INFO:tasks.rados.rados.0.smithi089.stdout:635: finishing write tid 2 to smithi08937929-34 2024-04-22T18:39:07.075 INFO:tasks.rados.rados.0.smithi089.stdout:637: done (15 left) 2024-04-22T18:39:07.075 INFO:tasks.rados.rados.0.smithi089.stdout:639: done (14 left) 2024-04-22T18:39:07.075 INFO:tasks.rados.rados.0.smithi089.stdout:640: done (13 left) 2024-04-22T18:39:07.075 INFO:tasks.rados.rados.0.smithi089.stdout:641: done (12 left) 2024-04-22T18:39:07.075 INFO:tasks.rados.rados.0.smithi089.stdout:643: done (11 left) 2024-04-22T18:39:07.075 INFO:tasks.rados.rados.0.smithi089.stdout:644: done (10 left) 2024-04-22T18:39:07.075 INFO:tasks.rados.rados.0.smithi089.stdout:653: read oid 24 snap -1 2024-04-22T18:39:07.076 INFO:tasks.rados.rados.0.smithi089.stdout:653: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-22T18:39:07.076 INFO:tasks.rados.rados.0.smithi089.stdout:654: set_chunk oid 42 target oid 2024-04-22T18:39:07.116 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:07.117+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:07.135 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:07.135 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:07.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:07.135 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:07.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.135 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:7 4.f 4:f83a159b:::smithi08937929-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:07.135 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.111347+0000", 2024-04-22T18:39:07.135 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.023850908, 2024-04-22T18:39:07.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30271893700000002, 2024-04-22T18:39:07.135 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.136 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.136 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.136 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.136 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.136 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 7 2024-04-22T18:39:07.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.136 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111347+0000", 2024-04-22T18:39:07.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111347+0000", 2024-04-22T18:39:07.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111346+0000", 2024-04-22T18:39:07.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:07.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111350+0000", 2024-04-22T18:39:07.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.473e-06 2024-04-22T18:39:07.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111353+0000", 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9100000000000001e-06 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111356+0000", 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2770000000000001e-06 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.212813+0000", 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10145739099999999 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.212869+0000", 2024-04-22T18:39:07.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5630000000000001e-05 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.212949+0000", 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9679999999999996e-05 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.313864+0000", 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100915174 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.413991+0000", 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100126703 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414021+0000", 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0698999999999999e-05 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414066+0000", 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4718000000000001e-05 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:8 4.f 4:f83a159b:::smithi08937929-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.023819709, 2024-04-22T18:39:07.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30292182499999998, 2024-04-22T18:39:07.141 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.141 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:07.141 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.141 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.141 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.141 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 8 2024-04-22T18:39:07.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.141 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:07.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:07.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:07.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:07.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111380+0000", 2024-04-22T18:39:07.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2419999999999999e-06 2024-04-22T18:39:07.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111384+0000", 2024-04-22T18:39:07.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.715e-06 2024-04-22T18:39:07.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111387+0000", 2024-04-22T18:39:07.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.982e-06 2024-04-22T18:39:07.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414113+0000", 2024-04-22T18:39:07.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30272622700000001 2024-04-22T18:39:07.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414203+0000", 2024-04-22T18:39:07.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0316999999999996e-05 2024-04-22T18:39:07.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414300+0000", 2024-04-22T18:39:07.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6760000000000002e-05 2024-04-22T18:39:07.144 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.144 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.144 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.144 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:9 4.f 4:f2c48a0e:::smithi08937929-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.018424124999999, 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49290714600000002, 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 9 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:07.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116773+0000", 2024-04-22T18:39:07.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:07.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117024+0000", 2024-04-22T18:39:07.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025191299999999999 2024-04-22T18:39:07.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117036+0000", 2024-04-22T18:39:07.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1595e-05 2024-04-22T18:39:07.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117046+0000", 2024-04-22T18:39:07.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0456e-05 2024-04-22T18:39:07.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414324+0000", 2024-04-22T18:39:07.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297277603 2024-04-22T18:39:07.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414404+0000", 2024-04-22T18:39:07.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9738000000000001e-05 2024-04-22T18:39:07.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:07.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414494+0000", 2024-04-22T18:39:07.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9733000000000003e-05 2024-04-22T18:39:07.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:07.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518194+0000", 2024-04-22T18:39:07.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10370061 2024-04-22T18:39:07.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:07.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609618+0000", 2024-04-22T18:39:07.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091423360999999995 2024-04-22T18:39:07.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609655+0000", 2024-04-22T18:39:07.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7580000000000003e-05 2024-04-22T18:39:07.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609681+0000", 2024-04-22T18:39:07.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5908e-05 2024-04-22T18:39:07.150 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.150 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.150 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.150 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:10 4.f 4:f2c48a0e:::smithi08937929-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:07.150 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:07.150 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.012571865, 2024-04-22T18:39:07.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48712681400000002, 2024-04-22T18:39:07.150 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.150 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.150 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.151 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.151 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.151 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 10 2024-04-22T18:39:07.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.151 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:07.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:07.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122625+0000", 2024-04-22T18:39:07.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:07.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122828+0000", 2024-04-22T18:39:07.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020359499999999999 2024-04-22T18:39:07.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122840+0000", 2024-04-22T18:39:07.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1239e-05 2024-04-22T18:39:07.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122845+0000", 2024-04-22T18:39:07.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2719999999999997e-06 2024-04-22T18:39:07.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415105+0000", 2024-04-22T18:39:07.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29225972099999997 2024-04-22T18:39:07.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415169+0000", 2024-04-22T18:39:07.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4170000000000004e-05 2024-04-22T18:39:07.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:07.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415257+0000", 2024-04-22T18:39:07.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8331999999999994e-05 2024-04-22T18:39:07.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:07.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518199+0000", 2024-04-22T18:39:07.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10294174 2024-04-22T18:39:07.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:07.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609707+0000", 2024-04-22T18:39:07.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091508408999999999 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609729+0000", 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1917e-05 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609753+0000", 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3776000000000001e-05 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.155 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:11 4.f 4:f2c48a0e:::smithi08937929-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:07.156 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:07.156 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.012348491999999, 2024-04-22T18:39:07.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49452214500000002, 2024-04-22T18:39:07.156 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.156 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.156 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.156 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.156 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.156 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 11 2024-04-22T18:39:07.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.156 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:07.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:07.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122849+0000", 2024-04-22T18:39:07.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:07.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122853+0000", 2024-04-22T18:39:07.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9029999999999997e-06 2024-04-22T18:39:07.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122856+0000", 2024-04-22T18:39:07.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.01e-06 2024-04-22T18:39:07.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122859+0000", 2024-04-22T18:39:07.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:07.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415755+0000", 2024-04-22T18:39:07.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29289606800000001 2024-04-22T18:39:07.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415808+0000", 2024-04-22T18:39:07.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3415999999999998e-05 2024-04-22T18:39:07.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:07.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415888+0000", 2024-04-22T18:39:07.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9061999999999996e-05 2024-04-22T18:39:07.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518203+0000", 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102315533 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617164+0000", 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098960548999999995 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617241+0000", 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7292000000000005e-05 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617372+0000", 2024-04-22T18:39:07.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013075399999999999 2024-04-22T18:39:07.161 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.161 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.161 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.161 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:12 4.f 4:f2c48a0e:::smithi08937929-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:07.161 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:07.161 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.012336628, 2024-04-22T18:39:07.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49473910999999998, 2024-04-22T18:39:07.161 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.161 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:07.161 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.161 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.162 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.162 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 12 2024-04-22T18:39:07.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.162 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:07.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:07.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:07.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:07.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122863+0000", 2024-04-22T18:39:07.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.863e-06 2024-04-22T18:39:07.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122866+0000", 2024-04-22T18:39:07.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:39:07.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122869+0000", 2024-04-22T18:39:07.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5950000000000001e-06 2024-04-22T18:39:07.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416008+0000", 2024-04-22T18:39:07.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29313975199999998 2024-04-22T18:39:07.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:07.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416048+0000", 2024-04-22T18:39:07.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9888000000000002e-05 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617447+0000", 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20139881400000001 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617537+0000", 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9606999999999998e-05 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617600+0000", 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3884e-05 2024-04-22T18:39:07.165 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.166 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.166 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.166 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:33 4.f 4:f0ed3d49:::smithi08937929-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:07.166 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:07.166 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.948396427, 2024-04-22T18:39:07.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62359692099999997, 2024-04-22T18:39:07.166 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.166 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.166 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.166 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.166 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.167 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 33 2024-04-22T18:39:07.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.167 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:07.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:07.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186799+0000", 2024-04-22T18:39:07.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:07.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187103+0000", 2024-04-22T18:39:07.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030354900000000002 2024-04-22T18:39:07.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187117+0000", 2024-04-22T18:39:07.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4578e-05 2024-04-22T18:39:07.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187129+0000", 2024-04-22T18:39:07.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1421e-05 2024-04-22T18:39:07.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516134+0000", 2024-04-22T18:39:07.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32900553900000001 2024-04-22T18:39:07.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516215+0000", 2024-04-22T18:39:07.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0537999999999994e-05 2024-04-22T18:39:07.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:07.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516302+0000", 2024-04-22T18:39:07.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6601000000000007e-05 2024-04-22T18:39:07.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:07.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518207+0000", 2024-04-22T18:39:07.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019058370000000001 2024-04-22T18:39:07.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:07.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810302+0000", 2024-04-22T18:39:07.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29209509700000003 2024-04-22T18:39:07.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810348+0000", 2024-04-22T18:39:07.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5992999999999998e-05 2024-04-22T18:39:07.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810398+0000", 2024-04-22T18:39:07.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0046000000000002e-05 2024-04-22T18:39:07.171 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.171 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.171 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.171 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:34 4.f 4:f0ed3d49:::smithi08937929-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.943794414999999, 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.61954655800000002, 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:07.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191401+0000", 2024-04-22T18:39:07.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:07.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191503+0000", 2024-04-22T18:39:07.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010190400000000001 2024-04-22T18:39:07.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191517+0000", 2024-04-22T18:39:07.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3267e-05 2024-04-22T18:39:07.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191523+0000", 2024-04-22T18:39:07.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-04-22T18:39:07.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517062+0000", 2024-04-22T18:39:07.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32553894300000003 2024-04-22T18:39:07.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517141+0000", 2024-04-22T18:39:07.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9667999999999997e-05 2024-04-22T18:39:07.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:07.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517267+0000", 2024-04-22T18:39:07.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000125494 2024-04-22T18:39:07.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:07.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518213+0000", 2024-04-22T18:39:07.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000946275 2024-04-22T18:39:07.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:07.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810901+0000", 2024-04-22T18:39:07.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29268821699999997 2024-04-22T18:39:07.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810929+0000", 2024-04-22T18:39:07.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7645999999999999e-05 2024-04-22T18:39:07.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810950+0000", 2024-04-22T18:39:07.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1333999999999999e-05 2024-04-22T18:39:07.177 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.177 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.177 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.177 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:07.177 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:07.177 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.943671447, 2024-04-22T18:39:07.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:39:07.177 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.177 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.178 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.178 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.178 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.178 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:39:07.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.178 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:07.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:07.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:39:07.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:07.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:39:07.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:39:07.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:39:07.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:39:07.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:07.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:39:07.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:39:07.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:07.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:39:07.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:39:07.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:07.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:39:07.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:39:07.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:39:07.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:39:07.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:39:07.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:39:07.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.182 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.183 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:07.183 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:07.183 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.943660372, 2024-04-22T18:39:07.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:39:07.183 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.183 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:07.183 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.183 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.183 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.183 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:39:07.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:07.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:39:07.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:39:07.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:39:07.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:39:07.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:39:07.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:39:07.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:39:07.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:39:07.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:07.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:39:07.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:39:07.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:39:07.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:39:07.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:39:07.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:39:07.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:39:07.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:39:07.187 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.187 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.188 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.188 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:38 4.2 4:4fb26dd8:::smithi08937929-10:head [write 1893632~257631] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:07.188 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:07.188 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.931937768000001, 2024-04-22T18:39:07.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33863800199999999, 2024-04-22T18:39:07.188 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.188 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.188 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.188 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.188 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.188 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 38 2024-04-22T18:39:07.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.189 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:07.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:07.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203258+0000", 2024-04-22T18:39:07.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:07.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203396+0000", 2024-04-22T18:39:07.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013710799999999999 2024-04-22T18:39:07.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203406+0000", 2024-04-22T18:39:07.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0295000000000001e-05 2024-04-22T18:39:07.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203411+0000", 2024-04-22T18:39:07.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5079999999999996e-06 2024-04-22T18:39:07.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203450+0000", 2024-04-22T18:39:07.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8420999999999997e-05 2024-04-22T18:39:07.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:07.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203455+0000", 2024-04-22T18:39:07.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:39:07.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237071+0000", 2024-04-22T18:39:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.033615891000000002 2024-04-22T18:39:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237129+0000", 2024-04-22T18:39:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8134000000000003e-05 2024-04-22T18:39:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237221+0000", 2024-04-22T18:39:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2053999999999997e-05 2024-04-22T18:39:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:07.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.341419+0000", 2024-04-22T18:39:07.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104197568 2024-04-22T18:39:07.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:07.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541792+0000", 2024-04-22T18:39:07.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.200373312 2024-04-22T18:39:07.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541838+0000", 2024-04-22T18:39:07.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6004000000000002e-05 2024-04-22T18:39:07.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541898+0000", 2024-04-22T18:39:07.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0297000000000002e-05 2024-04-22T18:39:07.194 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.194 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.194 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.194 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:39 4.2 4:4fb26dd8:::smithi08937929-10:head [setxattr _header (56) in=7b,truncate 2151263] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:07.194 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:07.194 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.931780774, 2024-04-22T18:39:07.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33862203000000002, 2024-04-22T18:39:07.194 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.194 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.195 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.195 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.195 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.195 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 39 2024-04-22T18:39:07.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.195 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:07.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:07.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203423+0000", 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0589999999999999e-06 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203429+0000", 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2559999999999999e-06 2024-04-22T18:39:07.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203434+0000", 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8049999999999997e-06 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203475+0000", 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1205000000000003e-05 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203478+0000", 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7140000000000002e-06 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239180+0000", 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035702195999999999 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239225+0000", 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5722e-05 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239291+0000", 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5870000000000005e-05 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:07.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342114+0000", 2024-04-22T18:39:07.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102822301 2024-04-22T18:39:07.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:07.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541946+0000", 2024-04-22T18:39:07.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.199832544 2024-04-22T18:39:07.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541988+0000", 2024-04-22T18:39:07.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1499999999999999e-05 2024-04-22T18:39:07.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542039+0000", 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1576000000000002e-05 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:40 4.2 4:4fb26dd8:::smithi08937929-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.931743514000001, 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33924657200000002, 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.200 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.201 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.201 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 40 2024-04-22T18:39:07.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.201 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:07.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:07.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:07.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:07.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203457+0000", 2024-04-22T18:39:07.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.129e-06 2024-04-22T18:39:07.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203463+0000", 2024-04-22T18:39:07.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7440000000000004e-06 2024-04-22T18:39:07.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203468+0000", 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2619999999999998e-06 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203491+0000", 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3260000000000001e-05 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203493+0000", 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9599999999999999e-06 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239420+0000", 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035927212 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239462+0000", 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.159e-05 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542547+0000", 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30308492599999998 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542611+0000", 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4622000000000006e-05 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542701+0000", 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9579000000000004e-05 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:41 4.2 4:409e6456:::smithi08937929-11:head [write 568070~468843,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.926027243, 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33297193600000002, 2024-04-22T18:39:07.205 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 41 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:07.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209168+0000", 2024-04-22T18:39:07.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:07.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209393+0000", 2024-04-22T18:39:07.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022485199999999999 2024-04-22T18:39:07.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209404+0000", 2024-04-22T18:39:07.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1209999999999999e-05 2024-04-22T18:39:07.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209411+0000", 2024-04-22T18:39:07.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8580000000000002e-06 2024-04-22T18:39:07.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209457+0000", 2024-04-22T18:39:07.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6351999999999999e-05 2024-04-22T18:39:07.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:07.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209463+0000", 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.322e-06 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339572+0000", 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13010901999999999 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339673+0000", 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101332 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339765+0000", 2024-04-22T18:39:07.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1780999999999994e-05 2024-04-22T18:39:07.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:07.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342117+0000", 2024-04-22T18:39:07.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002351709 2024-04-22T18:39:07.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:07.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542074+0000", 2024-04-22T18:39:07.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995749800000001 2024-04-22T18:39:07.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542112+0000", 2024-04-22T18:39:07.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7891e-05 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542143+0000", 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0756000000000002e-05 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:42 4.2 4:409e6456:::smithi08937929-11:head [write 1552114~615956] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.919688934, 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326735257, 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.211 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 42 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215508+0000", 2024-04-22T18:39:07.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:07.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215766+0000", 2024-04-22T18:39:07.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025766999999999999 2024-04-22T18:39:07.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215776+0000", 2024-04-22T18:39:07.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0772e-05 2024-04-22T18:39:07.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215785+0000", 2024-04-22T18:39:07.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6880000000000005e-06 2024-04-22T18:39:07.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215818+0000", 2024-04-22T18:39:07.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2960999999999998e-05 2024-04-22T18:39:07.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:07.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215821+0000", 2024-04-22T18:39:07.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8569999999999999e-06 2024-04-22T18:39:07.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340332+0000", 2024-04-22T18:39:07.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.124511409 2024-04-22T18:39:07.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340416+0000", 2024-04-22T18:39:07.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3793000000000002e-05 2024-04-22T18:39:07.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:07.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340543+0000", 2024-04-22T18:39:07.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012720299999999999 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342361+0000", 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018174370000000001 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542176+0000", 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19981518400000001 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542211+0000", 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5128e-05 2024-04-22T18:39:07.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542244+0000", 2024-04-22T18:39:07.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3256000000000001e-05 2024-04-22T18:39:07.217 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.217 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.217 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.217 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:43 4.2 4:409e6456:::smithi08937929-11:head [setxattr _header (56) in=7b,truncate 2168070] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:07.217 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:07.217 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.919404160999999, 2024-04-22T18:39:07.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32656540499999998, 2024-04-22T18:39:07.217 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 43 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:07.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215793+0000", 2024-04-22T18:39:07.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:07.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215797+0000", 2024-04-22T18:39:07.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4190000000000004e-06 2024-04-22T18:39:07.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215804+0000", 2024-04-22T18:39:07.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:39:07.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215809+0000", 2024-04-22T18:39:07.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3689999999999998e-06 2024-04-22T18:39:07.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215840+0000", 2024-04-22T18:39:07.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-05 2024-04-22T18:39:07.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:07.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215843+0000", 2024-04-22T18:39:07.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8849999999999999e-06 2024-04-22T18:39:07.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342483+0000", 2024-04-22T18:39:07.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126640172 2024-04-22T18:39:07.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342529+0000", 2024-04-22T18:39:07.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5668999999999999e-05 2024-04-22T18:39:07.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:07.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342597+0000", 2024-04-22T18:39:07.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7538000000000003e-05 2024-04-22T18:39:07.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:07.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.347102+0000", 2024-04-22T18:39:07.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045058040000000004 2024-04-22T18:39:07.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:07.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542278+0000", 2024-04-22T18:39:07.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.195175879 2024-04-22T18:39:07.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542314+0000", 2024-04-22T18:39:07.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6134999999999999e-05 2024-04-22T18:39:07.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542359+0000", 2024-04-22T18:39:07.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4848999999999997e-05 2024-04-22T18:39:07.223 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.223 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.223 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.223 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:44 4.2 4:409e6456:::smithi08937929-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:07.223 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:07.223 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.919384152999999, 2024-04-22T18:39:07.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326706418, 2024-04-22T18:39:07.223 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 44 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:07.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215813+0000", 2024-04-22T18:39:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215816+0000", 2024-04-22T18:39:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9799999999999998e-06 2024-04-22T18:39:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215822+0000", 2024-04-22T18:39:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5570000000000003e-06 2024-04-22T18:39:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215827+0000", 2024-04-22T18:39:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2739999999999998e-06 2024-04-22T18:39:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215855+0000", 2024-04-22T18:39:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7682999999999999e-05 2024-04-22T18:39:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215857+0000", 2024-04-22T18:39:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9250000000000002e-06 2024-04-22T18:39:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342717+0000", 2024-04-22T18:39:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126860696 2024-04-22T18:39:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342758+0000", 2024-04-22T18:39:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0295e-05 2024-04-22T18:39:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542408+0000", 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19964990599999999 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542457+0000", 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9784999999999999e-05 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542520+0000", 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2960999999999996e-05 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:49 4.a e27/20 4:5d2aee44:::smithi08937929-13:head v 27'4, mlcod=0'0)", 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.261026+0000", 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.874171793, 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.281745405, 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.261026+0000", 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.261026+0000", 2024-04-22T18:39:07.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.261025+0000", 2024-04-22T18:39:07.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:07.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.261563+0000", 2024-04-22T18:39:07.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053753000000000004 2024-04-22T18:39:07.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.261572+0000", 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4080000000000004e-06 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.261579+0000", 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8789999999999997e-06 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541746+0000", 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.280167153 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541760+0000", 2024-04-22T18:39:07.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4079e-05 2024-04-22T18:39:07.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542740+0000", 2024-04-22T18:39:07.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00097993100000000007 2024-04-22T18:39:07.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542772+0000", 2024-04-22T18:39:07.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1424999999999998e-05 2024-04-22T18:39:07.232 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.232 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.232 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.233 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:07.233 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:07.233 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.342290878, 2024-04-22T18:39:07.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:39:07.233 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.233 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.233 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.233 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.233 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.233 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:39:07.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.233 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:07.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:07.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:39:07.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:07.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:39:07.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.237 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.238 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.238 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:709 4.2 4:4fb26dd8:::smithi08937929-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:07.238 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.528719+0000", 2024-04-22T18:39:07.238 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.6064792399999996, 2024-04-22T18:39:07.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30036358099999999, 2024-04-22T18:39:07.238 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.238 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.238 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:07.238 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:07.238 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:07.238 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 709 2024-04-22T18:39:07.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528719+0000", 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528719+0000", 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528718+0000", 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:07.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528723+0000", 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8550000000000001e-06 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528729+0000", 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8680000000000004e-06 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528735+0000", 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8309999999999997e-06 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528766+0000", 2024-04-22T18:39:07.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1970000000000001e-05 2024-04-22T18:39:07.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528823+0000", 2024-04-22T18:39:07.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6440000000000002e-05 2024-04-22T18:39:07.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:07.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728211+0000", 2024-04-22T18:39:07.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.199387975 2024-04-22T18:39:07.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:07.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728615+0000", 2024-04-22T18:39:07.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000404161 2024-04-22T18:39:07.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:07.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.829015+0000", 2024-04-22T18:39:07.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100400197 2024-04-22T18:39:07.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.829053+0000", 2024-04-22T18:39:07.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7373000000000003e-05 2024-04-22T18:39:07.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.829082+0000", 2024-04-22T18:39:07.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9717e-05 2024-04-22T18:39:07.243 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.243 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.243 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.243 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.243 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.243 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:07.243 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:39:07.245 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:07.245 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:07.158+0000 7f8605f32640 -1 osd.3 36 handle_osd_map injecting map crc failure 2024-04-22T18:39:07.245 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:07.161+0000 7f1e2fd26640 -1 osd.2 35 handle_osd_map injecting map crc failure 2024-04-22T18:39:07.246 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.4 to 1 (10000) 2024-04-22T18:39:07.246 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-22T18:39:07.247 INFO:tasks.rados.rados.0.smithi089.stdout:654: set_chunk oid 42 offset: 496640 length: 1232896 target oid 17b0c19f5e8f62dac97d39f389bb09b88afd816285b70c887278d67a5ab285d6 offset: 0 snap -1 2024-04-22T18:39:07.247 INFO:tasks.rados.rados.0.smithi089.stdout:655: write (excl) oid 29 current snap is 0 2024-04-22T18:39:07.247 INFO:tasks.rados.rados.0.smithi089.stdout:635: finishing write tid 3 to smithi08937929-34 2024-04-22T18:39:07.247 INFO:tasks.rados.rados.0.smithi089.stdout:635: finishing write tid 4 to smithi08937929-34 2024-04-22T18:39:07.247 INFO:tasks.rados.rados.0.smithi089.stdout:635: finishing write tid 5 to smithi08937929-34 2024-04-22T18:39:07.247 INFO:tasks.rados.rados.0.smithi089.stdout:635: oid 34 updating version 0 to 31 2024-04-22T18:39:07.247 INFO:tasks.rados.rados.0.smithi089.stdout:635: oid 34 updating version 31 to 32 2024-04-22T18:39:07.247 INFO:tasks.rados.rados.0.smithi089.stdout:635: oid 34 updating version 32 to 34 2024-04-22T18:39:07.248 INFO:tasks.rados.rados.0.smithi089.stdout:635: oid 34 version 34 is already newer than 33 2024-04-22T18:39:07.248 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 34 (ObjNum 59 snap 0 seq_num 59) dirty exists 2024-04-22T18:39:07.248 INFO:tasks.rados.rados.0.smithi089.stdout:635: left oid 34 (ObjNum 59 snap 0 seq_num 59) 2024-04-22T18:39:07.248 INFO:tasks.rados.rados.0.smithi089.stdout:636: finishing write tid 5 to smithi08937929-44 2024-04-22T18:39:07.248 INFO:tasks.rados.rados.0.smithi089.stdout:636: oid 44 updating version 0 to 33 2024-04-22T18:39:07.248 INFO:tasks.rados.rados.0.smithi089.stdout:636: oid 44 updating version 33 to 35 2024-04-22T18:39:07.248 INFO:tasks.rados.rados.0.smithi089.stdout:636: oid 44 version 35 is already newer than 34 2024-04-22T18:39:07.248 INFO:tasks.rados.rados.0.smithi089.stdout:636: oid 44 updating version 35 to 36 2024-04-22T18:39:07.248 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 36 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-04-22T18:39:07.248 INFO:tasks.rados.rados.0.smithi089.stdout:636: left oid 44 (ObjNum 60 snap 0 seq_num 60) 2024-04-22T18:39:07.248 INFO:tasks.rados.rados.0.smithi089.stdout:648: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-22T18:39:07.249 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:07.245+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:07.263 INFO:tasks.rados.rados.0.smithi089.stdout:646: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-22T18:39:07.344 INFO:tasks.rados.rados.0.smithi089.stdout:645: finishing set_chunk to oid 16 2024-04-22T18:39:07.344 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 16 set_chunk returned error code -95 offset: 158208 length: 476672 2024-04-22T18:39:07.345 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-22T18:39:07.345 INFO:tasks.rados.rados.0.smithi089.stdout:647: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-22T18:39:07.345 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:07.346+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:07.375 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:07.375 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:07.375 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:07.375 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:07.385 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:07.438 INFO:tasks.rados.rados.0.smithi089.stdout:651: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-22T18:39:07.446 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:07.446+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:07.505 INFO:tasks.rados.rados.0.smithi089.stdout:652: finishing write tid 1 to smithi08937929-25 2024-04-22T18:39:07.505 INFO:tasks.rados.rados.0.smithi089.stdout:649: finishing set_chunk to oid 2 2024-04-22T18:39:07.505 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk returned error code -95 offset: 887808 length: 89088 2024-04-22T18:39:07.505 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-22T18:39:07.505 INFO:tasks.rados.rados.0.smithi089.stdout:650: got (22) Invalid argument 2024-04-22T18:39:07.505 INFO:tasks.rados.rados.0.smithi089.stdout:652: finishing write tid 2 to smithi08937929-25 2024-04-22T18:39:07.505 INFO:tasks.rados.rados.0.smithi089.stdout:655: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-04-22T18:39:07.512 INFO:tasks.rados.rados.0.smithi089.stdout:655: writing smithi08937929-29 from 658841 to 1130352 tid 1 2024-04-22T18:39:07.519 INFO:tasks.rados.rados.0.smithi089.stdout:655: writing smithi08937929-29 from 1553542 to 2261880 tid 2 2024-04-22T18:39:07.525 INFO:tasks.rados.rados.0.smithi089.stdout:655: writing smithi08937929-29 from 2964119 to 3058841 tid 3 2024-04-22T18:39:07.525 INFO:tasks.rados.rados.0.smithi089.stdout:635: done (12 left) 2024-04-22T18:39:07.525 INFO:tasks.rados.rados.0.smithi089.stdout:636: done (11 left) 2024-04-22T18:39:07.525 INFO:tasks.rados.rados.0.smithi089.stdout:645: done (10 left) 2024-04-22T18:39:07.525 INFO:tasks.rados.rados.0.smithi089.stdout:646: done (9 left) 2024-04-22T18:39:07.525 INFO:tasks.rados.rados.0.smithi089.stdout:647: done (8 left) 2024-04-22T18:39:07.525 INFO:tasks.rados.rados.0.smithi089.stdout:648: done (7 left) 2024-04-22T18:39:07.525 INFO:tasks.rados.rados.0.smithi089.stdout:649: done (6 left) 2024-04-22T18:39:07.525 INFO:tasks.rados.rados.0.smithi089.stdout:650: done (5 left) 2024-04-22T18:39:07.526 INFO:tasks.rados.rados.0.smithi089.stdout:651: done (4 left) 2024-04-22T18:39:07.526 INFO:tasks.rados.rados.0.smithi089.stdout:656: delete oid 10 current snap is 0 2024-04-22T18:39:07.526 INFO:tasks.rados.rados.0.smithi089.stdout:652: finishing write tid 3 to smithi08937929-25 2024-04-22T18:39:07.526 INFO:tasks.rados.rados.0.smithi089.stdout:652: finishing write tid 4 to smithi08937929-25 2024-04-22T18:39:07.526 INFO:tasks.rados.rados.0.smithi089.stdout:652: oid 25 updating version 0 to 25 2024-04-22T18:39:07.526 INFO:tasks.rados.rados.0.smithi089.stdout:652: oid 25 updating version 25 to 27 2024-04-22T18:39:07.526 INFO:tasks.rados.rados.0.smithi089.stdout:652: oid 25 version 27 is already newer than 26 2024-04-22T18:39:07.526 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 27 (ObjNum 62 snap 0 seq_num 62) dirty exists 2024-04-22T18:39:07.526 INFO:tasks.rados.rados.0.smithi089.stdout:652: left oid 25 (ObjNum 62 snap 0 seq_num 62) 2024-04-22T18:39:07.526 INFO:tasks.rados.rados.0.smithi089.stdout:653: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-22T18:39:07.530 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:07.530 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:07.531 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:07.531 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:07.531 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:07.539 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:07.547 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:07.548+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:07.629 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:07.630+0000 7f7f44bb3640 -1 osd.4 36 handle_osd_map injecting map crc failure 2024-04-22T18:39:07.643 INFO:tasks.rados.rados.0.smithi089.stdout:654: finishing set_chunk to oid 42 2024-04-22T18:39:07.643 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 42 set_chunk returned error code -95 offset: 496640 length: 1232896 2024-04-22T18:39:07.643 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-22T18:39:07.643 INFO:tasks.rados.rados.0.smithi089.stdout:655: finishing write tid 1 to smithi08937929-29 2024-04-22T18:39:07.643 INFO:tasks.rados.rados.0.smithi089.stdout:655: finishing write tid 2 to smithi08937929-29 2024-04-22T18:39:07.643 INFO:tasks.rados.rados.0.smithi089.stdout:652: done (4 left) 2024-04-22T18:39:07.643 INFO:tasks.rados.rados.0.smithi089.stdout:653: done (3 left) 2024-04-22T18:39:07.643 INFO:tasks.rados.rados.0.smithi089.stdout:654: done (2 left) 2024-04-22T18:39:07.643 INFO:tasks.rados.rados.0.smithi089.stdout:656: done (1 left) 2024-04-22T18:39:07.643 INFO:tasks.rados.rados.0.smithi089.stdout:657: read oid 35 snap -1 2024-04-22T18:39:07.644 INFO:tasks.rados.rados.0.smithi089.stdout:657: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-22T18:39:07.644 INFO:tasks.rados.rados.0.smithi089.stdout:658: read oid 48 snap -1 2024-04-22T18:39:07.644 INFO:tasks.rados.rados.0.smithi089.stdout:658: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-22T18:39:07.644 INFO:tasks.rados.rados.0.smithi089.stdout:659: read oid 22 snap -1 2024-04-22T18:39:07.644 INFO:tasks.rados.rados.0.smithi089.stdout:659: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-22T18:39:07.644 INFO:tasks.rados.rados.0.smithi089.stdout:660: read oid 39 snap -1 2024-04-22T18:39:07.644 INFO:tasks.rados.rados.0.smithi089.stdout:660: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-22T18:39:07.644 INFO:tasks.rados.rados.0.smithi089.stdout:661: read oid 13 snap -1 2024-04-22T18:39:07.644 INFO:tasks.rados.rados.0.smithi089.stdout:661: expect deleted 2024-04-22T18:39:07.644 INFO:tasks.rados.rados.0.smithi089.stdout:662: set_chunk oid 49 target oid 2024-04-22T18:39:07.647 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:07.648+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:07.684 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:25 4.d e27/20 4:bee1184f:::smithi08937929-7:head v 27'1, mlcod=0'0)", 2024-04-22T18:39:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.363505+0000", 2024-04-22T18:39:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.321043032, 2024-04-22T18:39:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002987511, 2024-04-22T18:39:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.363505+0000", 2024-04-22T18:39:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.363505+0000", 2024-04-22T18:39:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.363505+0000", 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.363985+0000", 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047988499999999999 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.363995+0000", 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0258000000000001e-05 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364015+0000", 2024-04-22T18:39:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0662e-05 2024-04-22T18:39:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364870+0000", 2024-04-22T18:39:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00085470000000000001 2024-04-22T18:39:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364888+0000", 2024-04-22T18:39:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8023999999999999e-05 2024-04-22T18:39:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.366462+0000", 2024-04-22T18:39:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015740159999999999 2024-04-22T18:39:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.366492+0000", 2024-04-22T18:39:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0198000000000001e-05 2024-04-22T18:39:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:26 4.d e27/20 4:bee1184f:::smithi08937929-7:head v 27'2, mlcod=0'0)", 2024-04-22T18:39:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.364412+0000", 2024-04-22T18:39:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.320135745, 2024-04-22T18:39:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002242754, 2024-04-22T18:39:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364412+0000", 2024-04-22T18:39:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364412+0000", 2024-04-22T18:39:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364412+0000", 2024-04-22T18:39:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364814+0000", 2024-04-22T18:39:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000402264 2024-04-22T18:39:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364823+0000", 2024-04-22T18:39:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8470000000000002e-06 2024-04-22T18:39:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.364830+0000", 2024-04-22T18:39:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7389999999999999e-06 2024-04-22T18:39:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365589+0000", 2024-04-22T18:39:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075836499999999999 2024-04-22T18:39:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365600+0000", 2024-04-22T18:39:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1308000000000001e-05 2024-04-22T18:39:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.366630+0000", 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001030188 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.366655+0000", 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4686e-05 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:27 4.d e27/20 4:bee1184f:::smithi08937929-7:head v 27'3, mlcod=0'0)", 2024-04-22T18:39:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.319360888, 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101135974, 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365186+0000", 2024-04-22T18:39:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365194+0000", 2024-04-22T18:39:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3289999999999994e-06 2024-04-22T18:39:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365200+0000", 2024-04-22T18:39:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6590000000000004e-06 2024-04-22T18:39:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465352+0000", 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10015208 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465511+0000", 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015880000000000001 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465570+0000", 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8952000000000001e-05 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466292+0000", 2024-04-22T18:39:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072212100000000005 2024-04-22T18:39:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466323+0000", 2024-04-22T18:39:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0899999999999999e-05 2024-04-22T18:39:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:306 4.d e29/20 4:bee1184f:::smithi08937929-7:head v 29'9, mlcod=28'7)", 2024-04-22T18:39:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.093254+0000", 2024-04-22T18:39:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.5912937490000001, 2024-04-22T18:39:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066511800000000002, 2024-04-22T18:39:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093254+0000", 2024-04-22T18:39:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093254+0000", 2024-04-22T18:39:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093253+0000", 2024-04-22T18:39:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093262+0000", 2024-04-22T18:39:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4089999999999992e-06 2024-04-22T18:39:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093272+0000", 2024-04-22T18:39:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0064999999999999e-05 2024-04-22T18:39:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093279+0000", 2024-04-22T18:39:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3089999999999998e-06 2024-04-22T18:39:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093318+0000", 2024-04-22T18:39:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9054999999999998e-05 2024-04-22T18:39:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093333+0000", 2024-04-22T18:39:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4423000000000001e-05 2024-04-22T18:39:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093893+0000", 2024-04-22T18:39:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056035799999999995 2024-04-22T18:39:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.093919+0000", 2024-04-22T18:39:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6012000000000001e-05 2024-04-22T18:39:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:47 3.0 e30/19 3:30f16451:::smithi08937929-50 low_tier:head v 30'20, mlcod=29'18)", 2024-04-22T18:39:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.0864448709999994, 2024-04-22T18:39:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100803438, 2024-04-22T18:39:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598108+0000", 2024-04-22T18:39:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9119999999999997e-06 2024-04-22T18:39:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598118+0000", 2024-04-22T18:39:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0138e-05 2024-04-22T18:39:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598125+0000", 2024-04-22T18:39:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3010000000000001e-06 2024-04-22T18:39:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698224+0000", 2024-04-22T18:39:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099221 2024-04-22T18:39:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698239+0000", 2024-04-22T18:39:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5053e-05 2024-04-22T18:39:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698875+0000", 2024-04-22T18:39:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063560699999999995 2024-04-22T18:39:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698906+0000", 2024-04-22T18:39:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1547999999999997e-05 2024-04-22T18:39:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.5.13:4 3.0 e30/19 3:1ab765eb:::smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'21, mlcod=29'18)", 2024-04-22T18:39:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.698033+0000", 2024-04-22T18:39:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.9865153890000009, 2024-04-22T18:39:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032895060000000002, 2024-04-22T18:39:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698033+0000", 2024-04-22T18:39:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698033+0000", 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698031+0000", 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698048+0000", 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6460000000000002e-05 2024-04-22T18:39:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698059+0000", 2024-04-22T18:39:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-04-22T18:39:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698067+0000", 2024-04-22T18:39:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4699999999999996e-06 2024-04-22T18:39:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698443+0000", 2024-04-22T18:39:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000375795 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698464+0000", 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1180000000000001e-05 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.701297+0000", 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028325859999999998 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.701322+0000", 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5496000000000001e-05 2024-04-22T18:39:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:28 3.0 e30/19 3:05e6fa13:::smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'22, mlcod=30'20)", 2024-04-22T18:39:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.797813+0000", 2024-04-22T18:39:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8867350009999999, 2024-04-22T18:39:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00055748400000000002, 2024-04-22T18:39:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797813+0000", 2024-04-22T18:39:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797813+0000", 2024-04-22T18:39:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797811+0000", 2024-04-22T18:39:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797819+0000", 2024-04-22T18:39:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.763e-06 2024-04-22T18:39:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797833+0000", 2024-04-22T18:39:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3728e-05 2024-04-22T18:39:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797842+0000", 2024-04-22T18:39:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1349999999999998e-06 2024-04-22T18:39:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797880+0000", 2024-04-22T18:39:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7670999999999999e-05 2024-04-22T18:39:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797900+0000", 2024-04-22T18:39:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9792999999999998e-05 2024-04-22T18:39:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798346+0000", 2024-04-22T18:39:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044668900000000001 2024-04-22T18:39:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798370+0000", 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4113000000000001e-05 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.5.13:5 3.0 e30/19 3:1ab765eb:::smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=30'21)", 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.4821267480000007, 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101014042, 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202428+0000", 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5889999999999997e-06 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202446+0000", 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8176999999999999e-05 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202456+0000", 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-04-22T18:39:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302599+0000", 2024-04-22T18:39:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100142759 2024-04-22T18:39:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302628+0000", 2024-04-22T18:39:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9507999999999999e-05 2024-04-22T18:39:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303350+0000", 2024-04-22T18:39:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072151599999999995 2024-04-22T18:39:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303435+0000", 2024-04-22T18:39:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5486000000000001e-05 2024-04-22T18:39:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:53 3.0 e31/19 3:25a201b0:::smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 31'24, mlcod=30'22)", 2024-04-22T18:39:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.2814844139999995, 2024-04-22T18:39:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100751671, 2024-04-22T18:39:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403062+0000", 2024-04-22T18:39:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403070+0000", 2024-04-22T18:39:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8499999999999994e-06 2024-04-22T18:39:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403079+0000", 2024-04-22T18:39:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4620000000000002e-06 2024-04-22T18:39:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403087+0000", 2024-04-22T18:39:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4789999999999999e-06 2024-04-22T18:39:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503186+0000", 2024-04-22T18:39:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099271 2024-04-22T18:39:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503207+0000", 2024-04-22T18:39:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0973999999999999e-05 2024-04-22T18:39:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503773+0000", 2024-04-22T18:39:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056557600000000003 2024-04-22T18:39:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503815+0000", 2024-04-22T18:39:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2441999999999998e-05 2024-04-22T18:39:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:33 3.0 e31/19 3:29860007:::smithi08937929-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 31'25, mlcod=30'22)", 2024-04-22T18:39:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.502323+0000", 2024-04-22T18:39:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.1822244340000001, 2024-04-22T18:39:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017670019999999999, 2024-04-22T18:39:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502323+0000", 2024-04-22T18:39:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502323+0000", 2024-04-22T18:39:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502320+0000", 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502337+0000", 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6767000000000001e-05 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502354+0000", 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7180000000000002e-05 2024-04-22T18:39:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502364+0000", 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0179e-05 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503405+0000", 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010405740000000001 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503425+0000", 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0324999999999999e-05 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.504061+0000", 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063580199999999998 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.504090+0000", 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9535000000000001e-05 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:43 3.0 e32/19 3:29860007:::smithi08937929-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 32'30, mlcod=31'28)", 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.628028+0000", 2024-04-22T18:39:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.0565198750000002, 2024-04-22T18:39:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00067497399999999995, 2024-04-22T18:39:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628028+0000", 2024-04-22T18:39:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628028+0000", 2024-04-22T18:39:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628028+0000", 2024-04-22T18:39:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628037+0000", 2024-04-22T18:39:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0119999999999994e-06 2024-04-22T18:39:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628047+0000", 2024-04-22T18:39:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0531e-05 2024-04-22T18:39:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628055+0000", 2024-04-22T18:39:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1550000000000007e-06 2024-04-22T18:39:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628099+0000", 2024-04-22T18:39:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3832000000000003e-05 2024-04-22T18:39:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628117+0000", 2024-04-22T18:39:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7649e-05 2024-04-22T18:39:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628671+0000", 2024-04-22T18:39:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00055402700000000001 2024-04-22T18:39:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.628703+0000", 2024-04-22T18:39:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2177e-05 2024-04-22T18:39:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:73 3.0 e34/19 3:061a7a63:::392f44ec26aa22e85be5f08273a7e45656f3bf4a4854038391a88e880ec01077:head v 34'31, mlcod=31'29)", 2024-04-22T18:39:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.226620966, 2024-04-22T18:39:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103503311, 2024-04-22T18:39:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458193+0000", 2024-04-22T18:39:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026628900000000001 2024-04-22T18:39:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458208+0000", 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4780000000000001e-05 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458217+0000", 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4909999999999994e-06 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459162+0000", 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094440500000000005 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459185+0000", 2024-04-22T18:39:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3685000000000001e-05 2024-04-22T18:39:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561394+0000", 2024-04-22T18:39:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10220874100000001 2024-04-22T18:39:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561430+0000", 2024-04-22T18:39:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6304999999999999e-05 2024-04-22T18:39:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:75 3.0 e34/19 3:18418e0f:::b7c35f165c374d5e075dde9ac2607e47ace3b0a1e2343ec6efb171380f725eea:head v 34'32, mlcod=31'29)", 2024-04-22T18:39:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.2256076930000002, 2024-04-22T18:39:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104798078, 2024-04-22T18:39:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458939+0000", 2024-04-22T18:39:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459092+0000", 2024-04-22T18:39:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153838 2024-04-22T18:39:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459107+0000", 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4703e-05 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459115+0000", 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560093+0000", 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10097840499999999 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560109+0000", 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5631e-05 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563706+0000", 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035972059999999999 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563738+0000", 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2184000000000002e-05 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:77 3.0 e34/19 3:28387c9f:::b6d7840f02d561da04e398fbf94d91b85e9066b426e48baf3b11dd4a4f6096c0:head v 34'33, mlcod=31'29)", 2024-04-22T18:39:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.2246576130000002, 2024-04-22T18:39:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104186079, 2024-04-22T18:39:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459889+0000", 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459963+0000", 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3732999999999994e-05 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459973+0000", 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0285999999999999e-05 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459987+0000", 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3675999999999999e-05 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560515+0000", 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10052781299999999 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560538+0000", 2024-04-22T18:39:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3861000000000001e-05 2024-04-22T18:39:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564044+0000", 2024-04-22T18:39:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035052009999999999 2024-04-22T18:39:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564076+0000", 2024-04-22T18:39:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2737000000000002e-05 2024-04-22T18:39:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:78 3.0 e34/19 3:3a5582ac:::bdd92f7deaa2d50e204891f64f9043469ba687d78b218ed518f2483ba3486a99:head v 34'34, mlcod=31'29)", 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.2239914949999999, 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103547263, 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:07.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460555+0000", 2024-04-22T18:39:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460861+0000", 2024-04-22T18:39:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000305958 2024-04-22T18:39:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460874+0000", 2024-04-22T18:39:07.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2639e-05 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460881+0000", 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2570000000000001e-06 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560990+0000", 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010883199999999 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561011+0000", 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1319e-05 2024-04-22T18:39:07.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564079+0000", 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00306815 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564104+0000", 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4172000000000002e-05 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:39:07.745 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.2233762009999998, 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:39:07.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:39:07.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:39:07.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:39:07.749 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.749 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.749 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.749 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:82 3.0 e34/19 3:0ff8847f:::398c8d727fdbe18b75b96dc6f91eef74b2932b833e12a59cfdb494756f6f91bf:head v 34'36, mlcod=31'29)", 2024-04-22T18:39:07.749 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.664309+0000", 2024-04-22T18:39:07.749 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0202389940000001, 2024-04-22T18:39:07.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00092281999999999998, 2024-04-22T18:39:07.750 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:07.750 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:07.750 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:07.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:07.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.664309+0000", 2024-04-22T18:39:07.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:07.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.664309+0000", 2024-04-22T18:39:07.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:07.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.664307+0000", 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.664387+0000", 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9226000000000003e-05 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.664399+0000", 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2682e-05 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.664407+0000", 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1890000000000001e-06 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.664461+0000", 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4466000000000003e-05 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.664483+0000", 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1651e-05 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:07.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.665200+0000", 2024-04-22T18:39:07.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00071776900000000005 2024-04-22T18:39:07.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:07.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:07.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:07.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.665232+0000", 2024-04-22T18:39:07.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1297999999999997e-05 2024-04-22T18:39:07.753 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.753 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.753 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.753 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:07.753 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:07.753 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:07.754 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:07.755 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:07.755+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:07.855 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:07.855+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:07.858 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:07.858 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:07.858 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:07.858 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:07.867 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:07.956 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:07.956+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:08.010 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:08.010 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:08.010 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:08.010 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:08.010 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:08.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:08.057 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:08.058+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:08.139 INFO:tasks.rados.rados.0.smithi089.stdout:662: set_chunk oid 49 offset: 1219072 length: 980992 target oid c9d0f7fbd1069a9305f3ce9f5945a26245d100bc434845970fa1d15818fe3cca offset: 0 snap -1 2024-04-22T18:39:08.139 INFO:tasks.rados.rados.0.smithi089.stdout:663: read oid 47 snap -1 2024-04-22T18:39:08.139 INFO:tasks.rados.rados.0.smithi089.stdout:663: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-22T18:39:08.139 INFO:tasks.rados.rados.0.smithi089.stdout:664: read oid 41 snap -1 2024-04-22T18:39:08.139 INFO:tasks.rados.rados.0.smithi089.stdout:664: expect (ObjNum 40 snap 0 seq_num 40) 2024-04-22T18:39:08.139 INFO:tasks.rados.rados.0.smithi089.stdout:665: tier_evict oid 40 2024-04-22T18:39:08.139 INFO:tasks.rados.rados.0.smithi089.stdout:665: tier_evict oid 40 snap -1 2024-04-22T18:39:08.139 INFO:tasks.rados.rados.0.smithi089.stdout:666: set_chunk oid 36 target oid 2024-04-22T18:39:08.157 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:08.158+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:08.162 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:6 4.f e24/20 4:f83a159b:::smithi08937929-2:head v 24'2, mlcod=0'0)", 2024-04-22T18:39:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.957018522, 2024-04-22T18:39:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10183012700000001, 2024-04-22T18:39:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.206027+0000", 2024-04-22T18:39:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023705500000000001 2024-04-22T18:39:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.206038+0000", 2024-04-22T18:39:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.148e-05 2024-04-22T18:39:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.206046+0000", 2024-04-22T18:39:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8679999999999999e-06 2024-04-22T18:39:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.306227+0000", 2024-04-22T18:39:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100181223 2024-04-22T18:39:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.306266+0000", 2024-04-22T18:39:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8906000000000003e-05 2024-04-22T18:39:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.307598+0000", 2024-04-22T18:39:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001331561 2024-04-22T18:39:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.307620+0000", 2024-04-22T18:39:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.226e-05 2024-04-22T18:39:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:45 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'1, mlcod=0'0)", 2024-04-22T18:39:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.897696836, 2024-04-22T18:39:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14345418900000001, 2024-04-22T18:39:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:08.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:08.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:08.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:08.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265727+0000", 2024-04-22T18:39:08.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061502799999999999 2024-04-22T18:39:08.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265737+0000", 2024-04-22T18:39:08.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0278999999999999e-05 2024-04-22T18:39:08.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265744+0000", 2024-04-22T18:39:08.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1740000000000003e-06 2024-04-22T18:39:08.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407227+0000", 2024-04-22T18:39:08.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14148266900000001 2024-04-22T18:39:08.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407258+0000", 2024-04-22T18:39:08.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1619000000000003e-05 2024-04-22T18:39:08.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408534+0000", 2024-04-22T18:39:08.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001275829 2024-04-22T18:39:08.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408566+0000", 2024-04-22T18:39:08.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1813999999999997e-05 2024-04-22T18:39:08.171 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.171 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.171 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.171 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:46 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'2, mlcod=0'0)", 2024-04-22T18:39:08.171 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:08.172 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.895072432999999, 2024-04-22T18:39:08.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14304645699999999, 2024-04-22T18:39:08.172 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.172 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.172 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:08.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:08.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:08.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268301+0000", 2024-04-22T18:39:08.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056585700000000001 2024-04-22T18:39:08.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268309+0000", 2024-04-22T18:39:08.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2830000000000004e-06 2024-04-22T18:39:08.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268312+0000", 2024-04-22T18:39:08.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7179999999999998e-06 2024-04-22T18:39:08.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407999+0000", 2024-04-22T18:39:08.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13968636300000001 2024-04-22T18:39:08.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408011+0000", 2024-04-22T18:39:08.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2116999999999999e-05 2024-04-22T18:39:08.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410762+0000", 2024-04-22T18:39:08.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002751524 2024-04-22T18:39:08.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410783+0000", 2024-04-22T18:39:08.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0270000000000001e-05 2024-04-22T18:39:08.175 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.176 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.176 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.176 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:47 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'3, mlcod=0'0)", 2024-04-22T18:39:08.176 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:08.176 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.893339863, 2024-04-22T18:39:08.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14132698599999999, 2024-04-22T18:39:08.176 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.176 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.176 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:08.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:08.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269468+0000", 2024-04-22T18:39:08.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:08.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269473+0000", 2024-04-22T18:39:08.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7790000000000003e-06 2024-04-22T18:39:08.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269478+0000", 2024-04-22T18:39:08.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0080000000000002e-06 2024-04-22T18:39:08.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269489+0000", 2024-04-22T18:39:08.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.075e-05 2024-04-22T18:39:08.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408595+0000", 2024-04-22T18:39:08.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13910662300000001 2024-04-22T18:39:08.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408611+0000", 2024-04-22T18:39:08.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6124e-05 2024-04-22T18:39:08.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410786+0000", 2024-04-22T18:39:08.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021743819999999999 2024-04-22T18:39:08.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410796+0000", 2024-04-22T18:39:08.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.007e-05 2024-04-22T18:39:08.181 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.181 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.181 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.181 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:9 4.f e27/20 4:f2c48a0e:::smithi08937929-3:head v 27'4, mlcod=24'2)", 2024-04-22T18:39:08.181 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.441905+0000", 2024-04-22T18:39:08.181 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.720904074, 2024-04-22T18:39:08.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10138892200000001, 2024-04-22T18:39:08.181 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.181 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.181 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.441905+0000", 2024-04-22T18:39:08.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.441905+0000", 2024-04-22T18:39:08.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.441902+0000", 2024-04-22T18:39:08.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:08.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442053+0000", 2024-04-22T18:39:08.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015104800000000001 2024-04-22T18:39:08.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442067+0000", 2024-04-22T18:39:08.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3964e-05 2024-04-22T18:39:08.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442073+0000", 2024-04-22T18:39:08.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1249999999999998e-06 2024-04-22T18:39:08.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442111+0000", 2024-04-22T18:39:08.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7988000000000003e-05 2024-04-22T18:39:08.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442127+0000", 2024-04-22T18:39:08.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.662e-05 2024-04-22T18:39:08.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543270+0000", 2024-04-22T18:39:08.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101143 2024-04-22T18:39:08.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543293+0000", 2024-04-22T18:39:08.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3147e-05 2024-04-22T18:39:08.187 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.187 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.187 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.188 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:10 4.f e27/20 4:f2c48a0e:::smithi08937929-3:head v 27'5, mlcod=24'2)", 2024-04-22T18:39:08.188 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:08.188 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.720727292999999, 2024-04-22T18:39:08.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10161616800000001, 2024-04-22T18:39:08.188 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.188 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.188 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:08.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:08.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:08.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442159+0000", 2024-04-22T18:39:08.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7935000000000005e-05 2024-04-22T18:39:08.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442163+0000", 2024-04-22T18:39:08.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8430000000000003e-06 2024-04-22T18:39:08.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442166+0000", 2024-04-22T18:39:08.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0759999999999999e-06 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542745+0000", 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10057941200000001 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542770+0000", 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5134e-05 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543679+0000", 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00090833099999999996 2024-04-22T18:39:08.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543697+0000", 2024-04-22T18:39:08.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8876e-05 2024-04-22T18:39:08.192 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.192 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.192 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.193 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:11 4.f e27/20 4:f2c48a0e:::smithi08937929-3:head v 27'6, mlcod=24'2)", 2024-04-22T18:39:08.193 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.442170+0000", 2024-04-22T18:39:08.193 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.720638878999999, 2024-04-22T18:39:08.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101722358, 2024-04-22T18:39:08.193 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.193 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.193 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442170+0000", 2024-04-22T18:39:08.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442170+0000", 2024-04-22T18:39:08.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442169+0000", 2024-04-22T18:39:08.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442174+0000", 2024-04-22T18:39:08.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0350000000000003e-06 2024-04-22T18:39:08.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442177+0000", 2024-04-22T18:39:08.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0580000000000002e-06 2024-04-22T18:39:08.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442181+0000", 2024-04-22T18:39:08.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1649999999999999e-06 2024-04-22T18:39:08.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543319+0000", 2024-04-22T18:39:08.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101138408 2024-04-22T18:39:08.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543336+0000", 2024-04-22T18:39:08.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6509999999999999e-05 2024-04-22T18:39:08.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543880+0000", 2024-04-22T18:39:08.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054429900000000004 2024-04-22T18:39:08.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543892+0000", 2024-04-22T18:39:08.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2138e-05 2024-04-22T18:39:08.196 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.197 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.197 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.197 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:77 4.0 4:01baf5a9:::smithi08937929-20:head [write 719333~462437,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.197 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.505878+0000", 2024-04-22T18:39:08.197 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.656931078, 2024-04-22T18:39:08.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10366277, 2024-04-22T18:39:08.197 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.197 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:08.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:08.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.197 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 77 2024-04-22T18:39:08.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.198 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.505878+0000", 2024-04-22T18:39:08.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.505878+0000", 2024-04-22T18:39:08.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.505876+0000", 2024-04-22T18:39:08.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:08.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506122+0000", 2024-04-22T18:39:08.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024626900000000002 2024-04-22T18:39:08.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506132+0000", 2024-04-22T18:39:08.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0216e-05 2024-04-22T18:39:08.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506139+0000", 2024-04-22T18:39:08.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6030000000000001e-06 2024-04-22T18:39:08.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506191+0000", 2024-04-22T18:39:08.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.185e-05 2024-04-22T18:39:08.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506333+0000", 2024-04-22T18:39:08.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014197299999999999 2024-04-22T18:39:08.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:08.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506480+0000", 2024-04-22T18:39:08.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014780199999999999 2024-04-22T18:39:08.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:08.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.507794+0000", 2024-04-22T18:39:08.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001313587 2024-04-22T18:39:08.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609429+0000", 2024-04-22T18:39:08.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101634658 2024-04-22T18:39:08.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609493+0000", 2024-04-22T18:39:08.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4777999999999995e-05 2024-04-22T18:39:08.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609540+0000", 2024-04-22T18:39:08.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6820000000000002e-05 2024-04-22T18:39:08.202 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:78 4.0 4:01baf5a9:::smithi08937929-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.650873742, 2024-04-22T18:39:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.398522192, 2024-04-22T18:39:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 78 2024-04-22T18:39:08.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.204 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:08.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:08.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511934+0000", 2024-04-22T18:39:08.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:08.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512012+0000", 2024-04-22T18:39:08.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8329000000000002e-05 2024-04-22T18:39:08.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512023+0000", 2024-04-22T18:39:08.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0755e-05 2024-04-22T18:39:08.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512028+0000", 2024-04-22T18:39:08.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6470000000000003e-06 2024-04-22T18:39:08.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512068+0000", 2024-04-22T18:39:08.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9721999999999997e-05 2024-04-22T18:39:08.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512125+0000", 2024-04-22T18:39:08.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6564999999999998e-05 2024-04-22T18:39:08.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:08.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512200+0000", 2024-04-22T18:39:08.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5708000000000001e-05 2024-04-22T18:39:08.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:08.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513131+0000", 2024-04-22T18:39:08.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093063 2024-04-22T18:39:08.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910394+0000", 2024-04-22T18:39:08.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.397263388 2024-04-22T18:39:08.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910430+0000", 2024-04-22T18:39:08.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5485999999999999e-05 2024-04-22T18:39:08.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910457+0000", 2024-04-22T18:39:08.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7086000000000001e-05 2024-04-22T18:39:08.210 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.210 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.211 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.211 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:79 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.211 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:08.211 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.650773085999999, 2024-04-22T18:39:08.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39855419399999997, 2024-04-22T18:39:08.211 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.211 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.212 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:08.212 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:08.212 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.212 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 79 2024-04-22T18:39:08.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.212 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:08.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:08.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512035+0000", 2024-04-22T18:39:08.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512045+0000", 2024-04-22T18:39:08.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0379e-05 2024-04-22T18:39:08.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512051+0000", 2024-04-22T18:39:08.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6030000000000004e-06 2024-04-22T18:39:08.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512057+0000", 2024-04-22T18:39:08.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7270000000000004e-06 2024-04-22T18:39:08.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512756+0000", 2024-04-22T18:39:08.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069954000000000001 2024-04-22T18:39:08.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512806+0000", 2024-04-22T18:39:08.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9922000000000001e-05 2024-04-22T18:39:08.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:08.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512927+0000", 2024-04-22T18:39:08.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012059 2024-04-22T18:39:08.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:08.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.515883+0000", 2024-04-22T18:39:08.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029563739999999999 2024-04-22T18:39:08.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910510+0000", 2024-04-22T18:39:08.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39462745199999999 2024-04-22T18:39:08.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910542+0000", 2024-04-22T18:39:08.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1225e-05 2024-04-22T18:39:08.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910590+0000", 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8052e-05 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:80 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.650747985000001, 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39866376999999997, 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.217 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 80 2024-04-22T18:39:08.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.218 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:08.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512060+0000", 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512063+0000", 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.999e-06 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512069+0000", 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.891e-06 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512073+0000", 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4429999999999996e-06 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513084+0000", 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010102609999999999 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:08.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513123+0000", 2024-04-22T18:39:08.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9237999999999999e-05 2024-04-22T18:39:08.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910618+0000", 2024-04-22T18:39:08.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39749549299999998 2024-04-22T18:39:08.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910670+0000", 2024-04-22T18:39:08.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1221000000000003e-05 2024-04-22T18:39:08.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910724+0000", 2024-04-22T18:39:08.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4777000000000001e-05 2024-04-22T18:39:08.222 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.222 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.222 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.222 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:33 4.f e27/20 4:f0ed3d49:::smithi08937929-9:head v 27'7, mlcod=24'2)", 2024-04-22T18:39:08.222 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.609449+0000", 2024-04-22T18:39:08.222 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.553360051, 2024-04-22T18:39:08.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10183384600000001, 2024-04-22T18:39:08.223 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.223 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.223 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609449+0000", 2024-04-22T18:39:08.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609449+0000", 2024-04-22T18:39:08.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609448+0000", 2024-04-22T18:39:08.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:08.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609626+0000", 2024-04-22T18:39:08.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017809600000000001 2024-04-22T18:39:08.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609638+0000", 2024-04-22T18:39:08.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2045000000000001e-05 2024-04-22T18:39:08.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709771+0000", 2024-04-22T18:39:08.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013347 2024-04-22T18:39:08.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.710008+0000", 2024-04-22T18:39:08.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023696200000000001 2024-04-22T18:39:08.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.710045+0000", 2024-04-22T18:39:08.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6903000000000002e-05 2024-04-22T18:39:08.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.711263+0000", 2024-04-22T18:39:08.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012178110000000001 2024-04-22T18:39:08.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.711282+0000", 2024-04-22T18:39:08.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9525e-05 2024-04-22T18:39:08.229 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.229 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.229 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.229 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:34 4.f e27/20 4:f0ed3d49:::smithi08937929-9:head v 27'8, mlcod=24'2)", 2024-04-22T18:39:08.229 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.709796+0000", 2024-04-22T18:39:08.229 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.453013049999999, 2024-04-22T18:39:08.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10102066, 2024-04-22T18:39:08.229 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.229 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.229 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709796+0000", 2024-04-22T18:39:08.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709796+0000", 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709793+0000", 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709914+0000", 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000120864 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709932+0000", 2024-04-22T18:39:08.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8125999999999999e-05 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709949+0000", 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6480000000000001e-05 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.809972+0000", 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10002336000000001 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.809986+0000", 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3913e-05 2024-04-22T18:39:08.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810789+0000", 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00080341899999999999 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810816+0000", 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6880999999999999e-05 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:35 4.f e27/20 4:f0ed3d49:::smithi08937929-9:head v 27'9, mlcod=24'2)", 2024-04-22T18:39:08.233 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.709964+0000", 2024-04-22T18:39:08.234 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.452844369999999, 2024-04-22T18:39:08.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101040596, 2024-04-22T18:39:08.234 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.234 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.234 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709964+0000", 2024-04-22T18:39:08.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709964+0000", 2024-04-22T18:39:08.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709963+0000", 2024-04-22T18:39:08.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:08.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709975+0000", 2024-04-22T18:39:08.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.236e-05 2024-04-22T18:39:08.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709990+0000", 2024-04-22T18:39:08.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4296e-05 2024-04-22T18:39:08.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810061+0000", 2024-04-22T18:39:08.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10007135 2024-04-22T18:39:08.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810402+0000", 2024-04-22T18:39:08.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00034100299999999999 2024-04-22T18:39:08.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810422+0000", 2024-04-22T18:39:08.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9749e-05 2024-04-22T18:39:08.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810973+0000", 2024-04-22T18:39:08.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00055107799999999998 2024-04-22T18:39:08.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.811005+0000", 2024-04-22T18:39:08.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079999999999998e-05 2024-04-22T18:39:08.238 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.238 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.238 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.238 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:5 3.2 e30/29 3:663b82a1:::smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=0'0)", 2024-04-22T18:39:08.238 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:08.238 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.9353086669999993, 2024-04-22T18:39:08.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20090134500000001, 2024-04-22T18:39:08.238 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.238 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.238 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:08.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:08.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227499+0000", 2024-04-22T18:39:08.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:08.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227506+0000", 2024-04-22T18:39:08.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6540000000000002e-06 2024-04-22T18:39:08.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227516+0000", 2024-04-22T18:39:08.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0054000000000001e-05 2024-04-22T18:39:08.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327607+0000", 2024-04-22T18:39:08.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009167099999999 2024-04-22T18:39:08.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327688+0000", 2024-04-22T18:39:08.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1000000000000004e-05 2024-04-22T18:39:08.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327719+0000", 2024-04-22T18:39:08.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0664000000000003e-05 2024-04-22T18:39:08.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428371+0000", 2024-04-22T18:39:08.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10065239099999999 2024-04-22T18:39:08.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428401+0000", 2024-04-22T18:39:08.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9839e-05 2024-04-22T18:39:08.242 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.242 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.242 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.242 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:19 3.2 e30/29 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'24, mlcod=28'19)", 2024-04-22T18:39:08.242 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.327656+0000", 2024-04-22T18:39:08.242 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.8351524870000002, 2024-04-22T18:39:08.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101011095, 2024-04-22T18:39:08.243 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.243 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.243 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327656+0000", 2024-04-22T18:39:08.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327656+0000", 2024-04-22T18:39:08.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327655+0000", 2024-04-22T18:39:08.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:08.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327667+0000", 2024-04-22T18:39:08.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2480000000000001e-05 2024-04-22T18:39:08.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327679+0000", 2024-04-22T18:39:08.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.22e-05 2024-04-22T18:39:08.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327686+0000", 2024-04-22T18:39:08.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8580000000000002e-06 2024-04-22T18:39:08.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428037+0000", 2024-04-22T18:39:08.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10035092399999999 2024-04-22T18:39:08.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428057+0000", 2024-04-22T18:39:08.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0030999999999998e-05 2024-04-22T18:39:08.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428638+0000", 2024-04-22T18:39:08.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058120100000000005 2024-04-22T18:39:08.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428667+0000", 2024-04-22T18:39:08.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8741000000000001e-05 2024-04-22T18:39:08.250 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.250 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.250 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.251 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:334 4.c e30/20 4:3227431c:::smithi08937929-35:head v 30'18, mlcod=30'16)", 2024-04-22T18:39:08.251 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.698741+0000", 2024-04-22T18:39:08.251 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4640675040000009, 2024-04-22T18:39:08.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10216847599999999, 2024-04-22T18:39:08.252 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.252 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.252 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698741+0000", 2024-04-22T18:39:08.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698741+0000", 2024-04-22T18:39:08.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698740+0000", 2024-04-22T18:39:08.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:08.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698747+0000", 2024-04-22T18:39:08.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9199999999999998e-06 2024-04-22T18:39:08.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698756+0000", 2024-04-22T18:39:08.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1249999999999999e-06 2024-04-22T18:39:08.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698762+0000", 2024-04-22T18:39:08.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:39:08.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798915+0000", 2024-04-22T18:39:08.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10015304899999999 2024-04-22T18:39:08.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798937+0000", 2024-04-22T18:39:08.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1335000000000002e-05 2024-04-22T18:39:08.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.800879+0000", 2024-04-22T18:39:08.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001942088 2024-04-22T18:39:08.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.800910+0000", 2024-04-22T18:39:08.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0892000000000002e-05 2024-04-22T18:39:08.260 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.260 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.260 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.260 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:419 4.0 4:01baf5a9:::smithi08937929-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:39:08.260 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:08.260 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.7605869649999999, 2024-04-22T18:39:08.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30324478199999999, 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 419 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:08.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402219+0000", 2024-04-22T18:39:08.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:08.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402229+0000", 2024-04-22T18:39:08.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2550000000000003e-06 2024-04-22T18:39:08.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402242+0000", 2024-04-22T18:39:08.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2989999999999999e-05 2024-04-22T18:39:08.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402250+0000", 2024-04-22T18:39:08.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7900000000000005e-06 2024-04-22T18:39:08.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404085+0000", 2024-04-22T18:39:08.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001834397 2024-04-22T18:39:08.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:08.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404092+0000", 2024-04-22T18:39:08.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7889999999999997e-06 2024-04-22T18:39:08.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502534+0000", 2024-04-22T18:39:08.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098442331999999994 2024-04-22T18:39:08.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502631+0000", 2024-04-22T18:39:08.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7617999999999999e-05 2024-04-22T18:39:08.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:08.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604414+0000", 2024-04-22T18:39:08.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101782847 2024-04-22T18:39:08.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:08.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604888+0000", 2024-04-22T18:39:08.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047325999999999998 2024-04-22T18:39:08.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705376+0000", 2024-04-22T18:39:08.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100488623 2024-04-22T18:39:08.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705424+0000", 2024-04-22T18:39:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7964999999999999e-05 2024-04-22T18:39:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705466+0000", 2024-04-22T18:39:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2274999999999998e-05 2024-04-22T18:39:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.3.13:13 3.2 e31/29 3:549b9960:::smithi08937929-20 low_tier:head v 31'29, mlcod=30'27)", 2024-04-22T18:39:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.503273+0000", 2024-04-22T18:39:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.6595361079999993, 2024-04-22T18:39:08.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10079894, 2024-04-22T18:39:08.268 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.268 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.268 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503273+0000", 2024-04-22T18:39:08.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503273+0000", 2024-04-22T18:39:08.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503272+0000", 2024-04-22T18:39:08.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503278+0000", 2024-04-22T18:39:08.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3210000000000001e-06 2024-04-22T18:39:08.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503294+0000", 2024-04-22T18:39:08.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.522e-05 2024-04-22T18:39:08.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503301+0000", 2024-04-22T18:39:08.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6879999999999999e-06 2024-04-22T18:39:08.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.603425+0000", 2024-04-22T18:39:08.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100124017 2024-04-22T18:39:08.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.603451+0000", 2024-04-22T18:39:08.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5510000000000001e-05 2024-04-22T18:39:08.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604033+0000", 2024-04-22T18:39:08.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058202199999999996 2024-04-22T18:39:08.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604071+0000", 2024-04-22T18:39:08.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8587000000000002e-05 2024-04-22T18:39:08.271 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.271 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.271 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.272 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:70 3.2 e32/29 3:5f0623ce:::smithi08937929-38 low_tier:head v 32'37, mlcod=31'35)", 2024-04-22T18:39:08.272 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:08.272 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.6343699340000004, 2024-04-22T18:39:08.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20070027900000001, 2024-04-22T18:39:08.272 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:08.272 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.272 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:08.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:08.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:08.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:08.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:08.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:08.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:08.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528438+0000", 2024-04-22T18:39:08.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:08.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528443+0000", 2024-04-22T18:39:08.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8640000000000004e-06 2024-04-22T18:39:08.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:08.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528453+0000", 2024-04-22T18:39:08.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3910000000000004e-06 2024-04-22T18:39:08.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528459+0000", 2024-04-22T18:39:08.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-04-22T18:39:08.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:08.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728627+0000", 2024-04-22T18:39:08.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20016778700000001 2024-04-22T18:39:08.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:08.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728660+0000", 2024-04-22T18:39:08.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3024000000000001e-05 2024-04-22T18:39:08.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:08.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729107+0000", 2024-04-22T18:39:08.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044719200000000001 2024-04-22T18:39:08.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:08.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:08.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:08.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729139+0000", 2024-04-22T18:39:08.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1625000000000003e-05 2024-04-22T18:39:08.276 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.276 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.276 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.276 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:08.276 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:08.276 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:08.277 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:08.172+0000 7fec57bee640 -1 osd.7 36 handle_osd_map injecting map crc failure 2024-04-22T18:39:08.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:08.278 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.5 to 1 (10000) 2024-04-22T18:39:08.279 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:39:08.280 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-22T18:39:08.280 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:08.278+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:08.334 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:08.334+0000 7f8605f32640 -1 osd.3 37 handle_osd_map injecting map crc failure 2024-04-22T18:39:08.378 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:08.379+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:08.395 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:08.395 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:08.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.395 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:944 4.1 4.c4de1ef1 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:08.395 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:08.141207+0000", 2024-04-22T18:39:08.395 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.25575713, 2024-04-22T18:39:08.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.25577783399999998, 2024-04-22T18:39:08.395 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.395 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:08.396 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.396 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.396 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.396 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 944 2024-04-22T18:39:08.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.396 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141207+0000", 2024-04-22T18:39:08.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141207+0000", 2024-04-22T18:39:08.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141205+0000", 2024-04-22T18:39:08.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:08.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141216+0000", 2024-04-22T18:39:08.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0444e-05 2024-04-22T18:39:08.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141229+0000", 2024-04-22T18:39:08.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2995e-05 2024-04-22T18:39:08.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141238+0000", 2024-04-22T18:39:08.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4259999999999992e-06 2024-04-22T18:39:08.398 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.398 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.398 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.398 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.398 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:39:08.398 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 1 2024-04-22T18:39:08.398 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:08.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:08.494 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:08.481+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:08.508 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:08.508 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:08.508 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:08.508 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:08.508 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:08.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:08.539 INFO:tasks.rados.rados.0.smithi089.stdout:666: set_chunk oid 36 offset: 393216 length: 1965056 target oid 746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9 offset: 0 snap -1 2024-04-22T18:39:08.539 INFO:tasks.rados.rados.0.smithi089.stdout:667: set_chunk oid 28 target oid 2024-04-22T18:39:08.550 INFO:tasks.rados.rados.0.smithi089.stdout:667: set_chunk oid 28 offset: 947200 length: 1502208 target oid 6290f8615ebea10d1cc5209b753c07f394b24a3a8dc4397710a3036384c27e26 offset: 0 snap -1 2024-04-22T18:39:08.550 INFO:tasks.rados.rados.0.smithi089.stdout:668: read oid 24 snap -1 2024-04-22T18:39:08.550 INFO:tasks.rados.rados.0.smithi089.stdout:668: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-22T18:39:08.551 INFO:tasks.rados.rados.0.smithi089.stdout:669: write (excl) oid 33 current snap is 0 2024-04-22T18:39:08.551 INFO:tasks.rados.rados.0.smithi089.stdout:655: finishing write tid 3 to smithi08937929-29 2024-04-22T18:39:08.551 INFO:tasks.rados.rados.0.smithi089.stdout:655: finishing write tid 4 to smithi08937929-29 2024-04-22T18:39:08.551 INFO:tasks.rados.rados.0.smithi089.stdout:655: finishing write tid 5 to smithi08937929-29 2024-04-22T18:39:08.551 INFO:tasks.rados.rados.0.smithi089.stdout:655: oid 29 updating version 0 to 38 2024-04-22T18:39:08.551 INFO:tasks.rados.rados.0.smithi089.stdout:655: oid 29 version 38 is already newer than 37 2024-04-22T18:39:08.551 INFO:tasks.rados.rados.0.smithi089.stdout:655: oid 29 updating version 38 to 40 2024-04-22T18:39:08.551 INFO:tasks.rados.rados.0.smithi089.stdout:655: oid 29 version 40 is already newer than 39 2024-04-22T18:39:08.551 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 40 (ObjNum 63 snap 0 seq_num 63) dirty exists 2024-04-22T18:39:08.551 INFO:tasks.rados.rados.0.smithi089.stdout:655: left oid 29 (ObjNum 63 snap 0 seq_num 63) 2024-04-22T18:39:08.551 INFO:tasks.rados.rados.0.smithi089.stdout:669: seq_num 65 ranges {692455=622009,2070506=712007} 2024-04-22T18:39:08.557 INFO:tasks.rados.rados.0.smithi089.stdout:669: writing smithi08937929-33 from 692455 to 1314464 tid 1 2024-04-22T18:39:08.565 INFO:tasks.rados.rados.0.smithi089.stdout:669: writing smithi08937929-33 from 2070506 to 2782513 tid 2 2024-04-22T18:39:08.565 INFO:tasks.rados.rados.0.smithi089.stdout:655: done (13 left) 2024-04-22T18:39:08.565 INFO:tasks.rados.rados.0.smithi089.stdout:670: set_chunk oid 34 target oid 2024-04-22T18:39:08.601 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:08.602+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:08.662 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.499914778999999, 2024-04-22T18:39:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:39:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:39:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:39:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:39:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:39:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:39:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.398355958, 2024-04-22T18:39:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:39:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:39:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:39:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:39:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:39:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:39:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:39:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.398329556, 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:39:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:39:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:39:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:39:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:39:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:39:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:39:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:39:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.999995226999999, 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:39:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:39:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:39:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:39:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:39:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:39:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:39:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:39:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:39:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:39:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:39:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:39:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.899158929, 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:39:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:39:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:39:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:39:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:39:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:39:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:39:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:39:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:39:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:39:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.899001659, 2024-04-22T18:39:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:39:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:39:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:39:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:39:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:39:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:39:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:39:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:39:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:39:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:39:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:39:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:39:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:39:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:39:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:39:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:39:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:39:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:39:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:39:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:39:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:39:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:39:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:39:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.898987050000001, 2024-04-22T18:39:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:39:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:39:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:39:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:39:08.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:39:08.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:39:08.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:39:08.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:39:08.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.698 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.699 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.699 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:08.699 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.898976664999999, 2024-04-22T18:39:08.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:39:08.699 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.699 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.699 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:39:08.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.699 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:08.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:08.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:39:08.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:39:08.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:39:08.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:39:08.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:39:08.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:39:08.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:39:08.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:39:08.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:39:08.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:39:08.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:39:08.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:08.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:39:08.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:39:08.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:39:08.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:39:08.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:39:08.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:39:08.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:39:08.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:39:08.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:39:08.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:39:08.704 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.704 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.704 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.704 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.704 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:08.704 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.898046734999999, 2024-04-22T18:39:08.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:39:08.705 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.705 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.705 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.705 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.705 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.705 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:39:08.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.705 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:08.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:08.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:39:08.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:08.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:39:08.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:39:08.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:39:08.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:39:08.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:08.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:39:08.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:39:08.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:39:08.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:39:08.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:39:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:39:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:39:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:39:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:39:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:39:08.709 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.710 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.710 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.710 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:08.710 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.897584609999999, 2024-04-22T18:39:08.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:39:08.710 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.710 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.710 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.710 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.710 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.710 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:39:08.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.711 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:08.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:08.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:39:08.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:08.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:39:08.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:39:08.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:39:08.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:39:08.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:39:08.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:39:08.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:39:08.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:39:08.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:39:08.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:39:08.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:08.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:39:08.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:39:08.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:39:08.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:39:08.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:39:08.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:39:08.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:39:08.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:39:08.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:39:08.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:39:08.715 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.715 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.715 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.715 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.897571806, 2024-04-22T18:39:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:39:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:39:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:08.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:39:08.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:39:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:39:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:39:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:39:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:39:08.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:39:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:39:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:39:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:39:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:39:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:39:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:39:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:39:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:39:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:109 4.1 4:83c955bf:::smithi08937929-28:head [write 453789~780782,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.694784813, 2024-04-22T18:39:08.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.45086113300000003, 2024-04-22T18:39:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:39:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967588+0000", 2024-04-22T18:39:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.968243+0000", 2024-04-22T18:39:08.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00065488500000000002 2024-04-22T18:39:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.968265+0000", 2024-04-22T18:39:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1997e-05 2024-04-22T18:39:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.068340+0000", 2024-04-22T18:39:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007516 2024-04-22T18:39:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168464+0000", 2024-04-22T18:39:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100124267 2024-04-22T18:39:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168579+0000", 2024-04-22T18:39:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114387 2024-04-22T18:39:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168692+0000", 2024-04-22T18:39:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011335100000000001 2024-04-22T18:39:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.173549+0000", 2024-04-22T18:39:08.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048564430000000002 2024-04-22T18:39:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418367+0000", 2024-04-22T18:39:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.244817962 2024-04-22T18:39:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418419+0000", 2024-04-22T18:39:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2883000000000002e-05 2024-04-22T18:39:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418450+0000", 2024-04-22T18:39:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0278999999999999e-05 2024-04-22T18:39:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:119 4.d 4:bc165c76:::smithi08937929-30:head [setxattr _header (56) in=7b,truncate 2487403] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.069595+0000", 2024-04-22T18:39:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.5927782, 2024-04-22T18:39:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.406949954, 2024-04-22T18:39:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.726 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 119 2024-04-22T18:39:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.069595+0000", 2024-04-22T18:39:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.069595+0000", 2024-04-22T18:39:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.069595+0000", 2024-04-22T18:39:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.069598+0000", 2024-04-22T18:39:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8630000000000001e-06 2024-04-22T18:39:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.069602+0000", 2024-04-22T18:39:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5279999999999999e-06 2024-04-22T18:39:08.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.169672+0000", 2024-04-22T18:39:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100070467 2024-04-22T18:39:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.269862+0000", 2024-04-22T18:39:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100189757 2024-04-22T18:39:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.270027+0000", 2024-04-22T18:39:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165215 2024-04-22T18:39:08.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:39:08.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.270201+0000", 2024-04-22T18:39:08.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017361300000000001 2024-04-22T18:39:08.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.473627+0000", 2024-04-22T18:39:08.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203426361 2024-04-22T18:39:08.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.476448+0000", 2024-04-22T18:39:08.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002820857 2024-04-22T18:39:08.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.476509+0000", 2024-04-22T18:39:08.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1251e-05 2024-04-22T18:39:08.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.476545+0000", 2024-04-22T18:39:08.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5703e-05 2024-04-22T18:39:08.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.731 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.731 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:08.732 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:08.732 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.56979935, 2024-04-22T18:39:08.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:39:08.732 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.732 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.732 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:39:08.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.732 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:08.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:08.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:39:08.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:08.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:39:08.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:39:08.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:39:08.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:39:08.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:39:08.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:39:08.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:39:08.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:39:08.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:39:08.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:39:08.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:08.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:39:08.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:39:08.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:39:08.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:39:08.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:39:08.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:39:08.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:39:08.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:39:08.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:39:08.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:39:08.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.737 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.737 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:08.737 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:08.737 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.5612262379999997, 2024-04-22T18:39:08.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:39:08.737 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.738 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:08.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:08.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:08.738 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:39:08.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.738 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:08.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:08.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:39:08.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:08.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:39:08.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:39:08.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:39:08.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:39:08.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:39:08.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:39:08.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:39:08.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:39:08.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:39:08.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:39:08.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:08.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:39:08.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:39:08.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:39:08.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:39:08.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:39:08.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:39:08.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:39:08.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:39:08.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:39:08.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:39:08.747 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.747 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.747 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.747 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:33 3.3 3:ea25b818:::smithi08937929-10 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:08.747 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.528569+0000", 2024-04-22T18:39:08.747 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.1338041240000001, 2024-04-22T18:39:08.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40827227799999999, 2024-04-22T18:39:08.747 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.747 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.747 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528569+0000", 2024-04-22T18:39:08.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528569+0000", 2024-04-22T18:39:08.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528567+0000", 2024-04-22T18:39:08.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:08.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528582+0000", 2024-04-22T18:39:08.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5565000000000001e-05 2024-04-22T18:39:08.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528597+0000", 2024-04-22T18:39:08.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4918e-05 2024-04-22T18:39:08.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528605+0000", 2024-04-22T18:39:08.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0609999999999996e-06 2024-04-22T18:39:08.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528657+0000", 2024-04-22T18:39:08.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1280999999999999e-05 2024-04-22T18:39:08.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:08.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528662+0000", 2024-04-22T18:39:08.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2129999999999999e-06 2024-04-22T18:39:08.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.733707+0000", 2024-04-22T18:39:08.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20504494000000001 2024-04-22T18:39:08.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.733793+0000", 2024-04-22T18:39:08.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6373999999999997e-05 2024-04-22T18:39:08.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:08.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.733937+0000", 2024-04-22T18:39:08.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143289 2024-04-22T18:39:08.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735740+0000", 2024-04-22T18:39:08.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001803334 2024-04-22T18:39:08.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.936677+0000", 2024-04-22T18:39:08.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20093707699999999 2024-04-22T18:39:08.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.936763+0000", 2024-04-22T18:39:08.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6339000000000001e-05 2024-04-22T18:39:08.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.936842+0000", 2024-04-22T18:39:08.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8237000000000003e-05 2024-04-22T18:39:08.753 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.753 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.753 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.754 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:34 3.3 3:ea25b818:::smithi08937929-10 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:08.754 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.528623+0000", 2024-04-22T18:39:08.754 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.1337502319999999, 2024-04-22T18:39:08.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40832893599999998, 2024-04-22T18:39:08.754 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.754 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.754 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528623+0000", 2024-04-22T18:39:08.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528623+0000", 2024-04-22T18:39:08.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528622+0000", 2024-04-22T18:39:08.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:08.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528627+0000", 2024-04-22T18:39:08.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7169999999999999e-06 2024-04-22T18:39:08.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528635+0000", 2024-04-22T18:39:08.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7549999999999994e-06 2024-04-22T18:39:08.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528652+0000", 2024-04-22T18:39:08.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7090000000000001e-05 2024-04-22T18:39:08.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528728+0000", 2024-04-22T18:39:08.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6401000000000003e-05 2024-04-22T18:39:08.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:08.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528735+0000", 2024-04-22T18:39:08.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2219999999999999e-06 2024-04-22T18:39:08.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.734263+0000", 2024-04-22T18:39:08.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.205528763 2024-04-22T18:39:08.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.734360+0000", 2024-04-22T18:39:08.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6965000000000004e-05 2024-04-22T18:39:08.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:08.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.734561+0000", 2024-04-22T18:39:08.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000200478 2024-04-22T18:39:08.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.736335+0000", 2024-04-22T18:39:08.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017739240000000001 2024-04-22T18:39:08.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.936894+0000", 2024-04-22T18:39:08.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.200559666 2024-04-22T18:39:08.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.936930+0000", 2024-04-22T18:39:08.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6133000000000001e-05 2024-04-22T18:39:08.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.936952+0000", 2024-04-22T18:39:08.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1645999999999999e-05 2024-04-22T18:39:08.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.760 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.760 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:35 3.3 3:ea25b818:::smithi08937929-10 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:08.760 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.528658+0000", 2024-04-22T18:39:08.760 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.1337152239999999, 2024-04-22T18:39:08.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40843963100000003, 2024-04-22T18:39:08.761 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.761 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.761 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528658+0000", 2024-04-22T18:39:08.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528658+0000", 2024-04-22T18:39:08.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528657+0000", 2024-04-22T18:39:08.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:08.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528662+0000", 2024-04-22T18:39:08.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7609999999999998e-06 2024-04-22T18:39:08.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528669+0000", 2024-04-22T18:39:08.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5570000000000001e-06 2024-04-22T18:39:08.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528674+0000", 2024-04-22T18:39:08.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4140000000000002e-06 2024-04-22T18:39:08.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528762+0000", 2024-04-22T18:39:08.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8200000000000003e-05 2024-04-22T18:39:08.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:08.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528767+0000", 2024-04-22T18:39:08.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2880000000000003e-06 2024-04-22T18:39:08.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.734844+0000", 2024-04-22T18:39:08.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.206077327 2024-04-22T18:39:08.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.734934+0000", 2024-04-22T18:39:08.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0016e-05 2024-04-22T18:39:08.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:08.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735118+0000", 2024-04-22T18:39:08.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018423199999999999 2024-04-22T18:39:08.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.738006+0000", 2024-04-22T18:39:08.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028883070000000001 2024-04-22T18:39:08.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937009+0000", 2024-04-22T18:39:08.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19900257099999999 2024-04-22T18:39:08.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937053+0000", 2024-04-22T18:39:08.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4024999999999999e-05 2024-04-22T18:39:08.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937098+0000", 2024-04-22T18:39:08.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4712999999999997e-05 2024-04-22T18:39:08.766 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.766 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.766 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.767 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:36 3.3 3:ea25b818:::smithi08937929-10 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:08.767 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:08.767 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.1336832029999999, 2024-04-22T18:39:08.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40851152800000001, 2024-04-22T18:39:08.767 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.767 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.767 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:08.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:08.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:08.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528694+0000", 2024-04-22T18:39:08.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2459999999999997e-06 2024-04-22T18:39:08.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528701+0000", 2024-04-22T18:39:08.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7950000000000002e-06 2024-04-22T18:39:08.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528709+0000", 2024-04-22T18:39:08.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1000000000000004e-06 2024-04-22T18:39:08.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528782+0000", 2024-04-22T18:39:08.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3699e-05 2024-04-22T18:39:08.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:08.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528785+0000", 2024-04-22T18:39:08.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:39:08.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735367+0000", 2024-04-22T18:39:08.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20658175100000001 2024-04-22T18:39:08.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735443+0000", 2024-04-22T18:39:08.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6106999999999995e-05 2024-04-22T18:39:08.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:08.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735573+0000", 2024-04-22T18:39:08.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129734 2024-04-22T18:39:08.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.740966+0000", 2024-04-22T18:39:08.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053933130000000003 2024-04-22T18:39:08.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937143+0000", 2024-04-22T18:39:08.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19617662399999999 2024-04-22T18:39:08.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937175+0000", 2024-04-22T18:39:08.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1860999999999999e-05 2024-04-22T18:39:08.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937202+0000", 2024-04-22T18:39:08.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7029000000000002e-05 2024-04-22T18:39:08.773 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.773 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.774 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.774 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:37 3.3 3:ea25b818:::smithi08937929-10 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:08.774 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.528720+0000", 2024-04-22T18:39:08.774 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.1336538869999999, 2024-04-22T18:39:08.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40858023100000002, 2024-04-22T18:39:08.774 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:08.774 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:08.774 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:08.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:08.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528720+0000", 2024-04-22T18:39:08.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:08.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528720+0000", 2024-04-22T18:39:08.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:08.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:08.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528719+0000", 2024-04-22T18:39:08.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:08.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:08.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528724+0000", 2024-04-22T18:39:08.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1159999999999998e-06 2024-04-22T18:39:08.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:08.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528730+0000", 2024-04-22T18:39:08.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0480000000000004e-06 2024-04-22T18:39:08.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:08.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.628843+0000", 2024-04-22T18:39:08.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100112425 2024-04-22T18:39:08.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.628917+0000", 2024-04-22T18:39:08.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4385e-05 2024-04-22T18:39:08.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:08.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.628923+0000", 2024-04-22T18:39:08.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4239999999999998e-06 2024-04-22T18:39:08.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:08.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735762+0000", 2024-04-22T18:39:08.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10683828200000001 2024-04-22T18:39:08.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:08.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735826+0000", 2024-04-22T18:39:08.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4144000000000001e-05 2024-04-22T18:39:08.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:08.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735941+0000", 2024-04-22T18:39:08.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011526799999999999 2024-04-22T18:39:08.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:08.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.740973+0000", 2024-04-22T18:39:08.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050320340000000003 2024-04-22T18:39:08.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:08.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937246+0000", 2024-04-22T18:39:08.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19627275499999999 2024-04-22T18:39:08.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:08.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937277+0000", 2024-04-22T18:39:08.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1563999999999998e-05 2024-04-22T18:39:08.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:08.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:08.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:08.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937300+0000", 2024-04-22T18:39:08.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2350000000000001e-05 2024-04-22T18:39:08.780 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.780 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.780 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.780 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:08.780 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:08.780 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:08.781 INFO:tasks.rados.rados.0.smithi089.stdout:670: set_chunk oid 34 offset: 1286144 length: 888832 target oid 08d65e0c0d496f482378d31ac300f29fb91538d506efa351c3d8fcf4f27f8460 offset: 0 snap -1 2024-04-22T18:39:08.781 INFO:tasks.rados.rados.0.smithi089.stdout:671: set_chunk oid 42 target oid 2024-04-22T18:39:08.782 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:08.783 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:08.782+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:08.882 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:08.883+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:08.925 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:08.925 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:08.926 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:08.926 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:08.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:08.983 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:08.984+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:09.036 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:09.036 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:09.036 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:09.036 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:09.036 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:09.043 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:39:09.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:09.087 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:09.088+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:09.160 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:09.161 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:09.175 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.6 to 1 (10000) 2024-04-22T18:39:09.180 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:09.184 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:09.184+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:09.191 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.097583887000001, 2024-04-22T18:39:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:39:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:39:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:39:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:39:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:39:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:39:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:39:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:39:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:39:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:39:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:39:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:39:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:39:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:39:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:39:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2 4.8 4:12d9f826:::smithi08937929-1:head [write 1542129~474678 in=474678b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.996872616999999, 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20435667099999999, 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2 2024-04-22T18:39:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:39:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:39:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194568+0000", 2024-04-22T18:39:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194869+0000", 2024-04-22T18:39:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030115200000000001 2024-04-22T18:39:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194879+0000", 2024-04-22T18:39:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8479999999999996e-06 2024-04-22T18:39:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294954+0000", 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100074702 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296019+0000", 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010649189999999999 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296086+0000", 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7261000000000005e-05 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296173+0000", 2024-04-22T18:39:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6935999999999996e-05 2024-04-22T18:39:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297196+0000", 2024-04-22T18:39:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023415 2024-04-22T18:39:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398879+0000", 2024-04-22T18:39:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101682563 2024-04-22T18:39:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398906+0000", 2024-04-22T18:39:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6526999999999999e-05 2024-04-22T18:39:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398928+0000", 2024-04-22T18:39:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2045e-05 2024-04-22T18:39:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:53 4.3 e27/20 4:ca46fde9:::smithi08937929-14:head v 27'1, mlcod=0'0)", 2024-04-22T18:39:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:39:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.943625414, 2024-04-22T18:39:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20214333400000001, 2024-04-22T18:39:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:39:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:39:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:39:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967296 2024-04-22T18:39:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248504+0000", 2024-04-22T18:39:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000686091 2024-04-22T18:39:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248512+0000", 2024-04-22T18:39:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1850000000000004e-06 2024-04-22T18:39:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248517+0000", 2024-04-22T18:39:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5469999999999998e-06 2024-04-22T18:39:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.348632+0000", 2024-04-22T18:39:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100115069 2024-04-22T18:39:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.348644+0000", 2024-04-22T18:39:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2001000000000001e-05 2024-04-22T18:39:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449930+0000", 2024-04-22T18:39:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101286574 2024-04-22T18:39:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449961+0000", 2024-04-22T18:39:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1046000000000001e-05 2024-04-22T18:39:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:54 4.3 e27/20 4:ca46fde9:::smithi08937929-14:head v 27'2, mlcod=0'0)", 2024-04-22T18:39:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.252046+0000", 2024-04-22T18:39:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.939397871000001, 2024-04-22T18:39:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19793179499999999, 2024-04-22T18:39:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252046+0000", 2024-04-22T18:39:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252046+0000", 2024-04-22T18:39:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252045+0000", 2024-04-22T18:39:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252215+0000", 2024-04-22T18:39:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017004499999999999 2024-04-22T18:39:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252221+0000", 2024-04-22T18:39:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.677e-06 2024-04-22T18:39:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252225+0000", 2024-04-22T18:39:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7730000000000001e-06 2024-04-22T18:39:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.349338+0000", 2024-04-22T18:39:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.097113853999999999 2024-04-22T18:39:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.349350+0000", 2024-04-22T18:39:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1268e-05 2024-04-22T18:39:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449967+0000", 2024-04-22T18:39:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100616754 2024-04-22T18:39:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449977+0000", 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0978999999999999e-05 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:55 4.3 e27/20)", 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.938830219, 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19766915500000001, 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:39:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:39:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:39:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252616+0000", 2024-04-22T18:39:09.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1889999999999999e-06 2024-04-22T18:39:09.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252622+0000", 2024-04-22T18:39:09.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0589999999999999e-06 2024-04-22T18:39:09.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252625+0000", 2024-04-22T18:39:09.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3239999999999999e-06 2024-04-22T18:39:09.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449805+0000", 2024-04-22T18:39:09.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19717992500000001 2024-04-22T18:39:09.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449819+0000", 2024-04-22T18:39:09.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.367e-05 2024-04-22T18:39:09.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.450269+0000", 2024-04-22T18:39:09.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045026300000000001 2024-04-22T18:39:09.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.450282+0000", 2024-04-22T18:39:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3229e-05 2024-04-22T18:39:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=62b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:39:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.896471663, 2024-04-22T18:39:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20408478299999999, 2024-04-22T18:39:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.215 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:09.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:09.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:09.216 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3 2024-04-22T18:39:09.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.216 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:39:09.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:39:09.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294970+0000", 2024-04-22T18:39:09.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:09.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294983+0000", 2024-04-22T18:39:09.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3429e-05 2024-04-22T18:39:09.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294993+0000", 2024-04-22T18:39:09.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0416e-05 2024-04-22T18:39:09.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295007+0000", 2024-04-22T18:39:09.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3742e-05 2024-04-22T18:39:09.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296793+0000", 2024-04-22T18:39:09.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017860420000000001 2024-04-22T18:39:09.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296841+0000", 2024-04-22T18:39:09.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7706e-05 2024-04-22T18:39:09.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:09.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296913+0000", 2024-04-22T18:39:09.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2578999999999997e-05 2024-04-22T18:39:09.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:09.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297946+0000", 2024-04-22T18:39:09.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001032802 2024-04-22T18:39:09.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:09.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498963+0000", 2024-04-22T18:39:09.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20101640400000001 2024-04-22T18:39:09.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498998+0000", 2024-04-22T18:39:09.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5759000000000001e-05 2024-04-22T18:39:09.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499057+0000", 2024-04-22T18:39:09.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8270999999999998e-05 2024-04-22T18:39:09.221 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.221 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.221 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.221 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:09.221 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:39:09.221 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.896431235, 2024-04-22T18:39:09.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20418823899999999, 2024-04-22T18:39:09.221 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.221 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:09.221 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:09.221 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:09.221 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:09.222 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4 2024-04-22T18:39:09.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.222 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:39:09.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:39:09.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295011+0000", 2024-04-22T18:39:09.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:09.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295016+0000", 2024-04-22T18:39:09.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2400000000000001e-06 2024-04-22T18:39:09.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295022+0000", 2024-04-22T18:39:09.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8519999999999998e-06 2024-04-22T18:39:09.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295027+0000", 2024-04-22T18:39:09.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9749999999999998e-06 2024-04-22T18:39:09.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297126+0000", 2024-04-22T18:39:09.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020981070000000001 2024-04-22T18:39:09.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:09.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297167+0000", 2024-04-22T18:39:09.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1842999999999999e-05 2024-04-22T18:39:09.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499096+0000", 2024-04-22T18:39:09.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201928846 2024-04-22T18:39:09.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499142+0000", 2024-04-22T18:39:09.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6152000000000001e-05 2024-04-22T18:39:09.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499201+0000", 2024-04-22T18:39:09.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8190999999999999e-05 2024-04-22T18:39:09.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.226 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.226 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:81 4.8 4:1bc4668a:::smithi08937929-21:head [write 736140~537686,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:09.226 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.549839+0000", 2024-04-22T18:39:09.226 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.641604237999999, 2024-04-22T18:39:09.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20180494099999999, 2024-04-22T18:39:09.226 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.227 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.227 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:09.227 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:09.227 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:09.227 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 81 2024-04-22T18:39:09.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.227 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.549839+0000", 2024-04-22T18:39:09.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.549839+0000", 2024-04-22T18:39:09.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.549837+0000", 2024-04-22T18:39:09.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:09.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.550355+0000", 2024-04-22T18:39:09.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051818999999999997 2024-04-22T18:39:09.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.550370+0000", 2024-04-22T18:39:09.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5073e-05 2024-04-22T18:39:09.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.550378+0000", 2024-04-22T18:39:09.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9580000000000007e-06 2024-04-22T18:39:09.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.650511+0000", 2024-04-22T18:39:09.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100132675 2024-04-22T18:39:09.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.650590+0000", 2024-04-22T18:39:09.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8949999999999995e-05 2024-04-22T18:39:09.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:09.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.650684+0000", 2024-04-22T18:39:09.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4846999999999995e-05 2024-04-22T18:39:09.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:09.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.652918+0000", 2024-04-22T18:39:09.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022340049999999998 2024-04-22T18:39:09.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.751590+0000", 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098671777000000002 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.751620+0000", 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0190999999999999e-05 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.751644+0000", 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3841000000000001e-05 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:82 4.8 4:1bc4668a:::smithi08937929-21:head [write 2020581~315559] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.555572+0000", 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.635871744999999, 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19891041200000001, 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 82 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555572+0000", 2024-04-22T18:39:09.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555572+0000", 2024-04-22T18:39:09.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555570+0000", 2024-04-22T18:39:09.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:09.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555825+0000", 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00025443699999999999 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555836+0000", 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1693e-05 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555843+0000", 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.861e-06 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.651510+0000", 2024-04-22T18:39:09.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.095666449000000001 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.651598+0000", 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7989000000000001e-05 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.651726+0000", 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012811100000000001 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.653977+0000", 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002251074 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:09.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754316+0000", 2024-04-22T18:39:09.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10033908599999999 2024-04-22T18:39:09.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754449+0000", 2024-04-22T18:39:09.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000133497 2024-04-22T18:39:09.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754482+0000", 2024-04-22T18:39:09.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2830000000000002e-05 2024-04-22T18:39:09.236 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.236 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.237 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.237 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:83 4.8 4:1bc4668a:::smithi08937929-21:head [setxattr _header (56) in=7b,truncate 2336140] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:09.237 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.555848+0000", 2024-04-22T18:39:09.237 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.635595255, 2024-04-22T18:39:09.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19872846999999999, 2024-04-22T18:39:09.237 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.237 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.237 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:09.237 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:09.237 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:09.237 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 83 2024-04-22T18:39:09.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.237 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555848+0000", 2024-04-22T18:39:09.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555848+0000", 2024-04-22T18:39:09.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555848+0000", 2024-04-22T18:39:09.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:09.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555852+0000", 2024-04-22T18:39:09.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.69e-06 2024-04-22T18:39:09.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555856+0000", 2024-04-22T18:39:09.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.055e-06 2024-04-22T18:39:09.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555859+0000", 2024-04-22T18:39:09.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1269999999999999e-06 2024-04-22T18:39:09.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.652495+0000", 2024-04-22T18:39:09.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.096636298999999995 2024-04-22T18:39:09.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.652599+0000", 2024-04-22T18:39:09.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010438300000000001 2024-04-22T18:39:09.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:09.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.652736+0000", 2024-04-22T18:39:09.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013642600000000001 2024-04-22T18:39:09.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:09.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.654000+0000", 2024-04-22T18:39:09.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012638370000000001 2024-04-22T18:39:09.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:09.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754516+0000", 2024-04-22T18:39:09.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100516297 2024-04-22T18:39:09.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754547+0000", 2024-04-22T18:39:09.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0673000000000003e-05 2024-04-22T18:39:09.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754577+0000", 2024-04-22T18:39:09.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0207999999999999e-05 2024-04-22T18:39:09.242 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.242 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.242 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.242 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:84 4.8 4:1bc4668a:::smithi08937929-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:09.242 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.555861+0000", 2024-04-22T18:39:09.243 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.63558265, 2024-04-22T18:39:09.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.198839506, 2024-04-22T18:39:09.243 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.243 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:09.243 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:09.243 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:09.243 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:09.243 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 84 2024-04-22T18:39:09.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.243 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555861+0000", 2024-04-22T18:39:09.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555861+0000", 2024-04-22T18:39:09.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555861+0000", 2024-04-22T18:39:09.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:09.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555862+0000", 2024-04-22T18:39:09.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.494e-06 2024-04-22T18:39:09.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555865+0000", 2024-04-22T18:39:09.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.954e-06 2024-04-22T18:39:09.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555870+0000", 2024-04-22T18:39:09.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5070000000000002e-06 2024-04-22T18:39:09.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.652949+0000", 2024-04-22T18:39:09.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.097078988000000005 2024-04-22T18:39:09.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:09.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.653026+0000", 2024-04-22T18:39:09.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7313999999999999e-05 2024-04-22T18:39:09.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754603+0000", 2024-04-22T18:39:09.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101577218 2024-04-22T18:39:09.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754647+0000", 2024-04-22T18:39:09.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3850999999999997e-05 2024-04-22T18:39:09.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754700+0000", 2024-04-22T18:39:09.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3489000000000002e-05 2024-04-22T18:39:09.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.247 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.247 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:70 4.3 e27/20 4:c6fa81cf:::smithi08937929-18:head v 27'8, mlcod=27'3)", 2024-04-22T18:39:09.247 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:09.247 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.233951318999999, 2024-04-22T18:39:09.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20440983300000001, 2024-04-22T18:39:09.248 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.248 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.248 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:09.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:09.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:09.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:09.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957839+0000", 2024-04-22T18:39:09.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000346981 2024-04-22T18:39:09.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957849+0000", 2024-04-22T18:39:09.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8060000000000006e-06 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957856+0000", 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0319999999999998e-06 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958600+0000", 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074474700000000003 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958615+0000", 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4961e-05 2024-04-22T18:39:09.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161867+0000", 2024-04-22T18:39:09.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20325188399999999 2024-04-22T18:39:09.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161902+0000", 2024-04-22T18:39:09.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4884e-05 2024-04-22T18:39:09.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.251 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.251 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:71 4.3 e27/20 4:c6fa81cf:::smithi08937929-18:head v 27'9, mlcod=27'3)", 2024-04-22T18:39:09.252 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:39:09.252 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.233581297000001, 2024-04-22T18:39:09.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20407488600000001, 2024-04-22T18:39:09.252 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.252 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.252 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:39:09.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:39:09.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:39:09.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:09.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957867+0000", 2024-04-22T18:39:09.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8329999999999999e-06 2024-04-22T18:39:09.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957872+0000", 2024-04-22T18:39:09.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4530000000000004e-06 2024-04-22T18:39:09.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957877+0000", 2024-04-22T18:39:09.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8969999999999999e-06 2024-04-22T18:39:09.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959214+0000", 2024-04-22T18:39:09.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001337443 2024-04-22T18:39:09.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959236+0000", 2024-04-22T18:39:09.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1889e-05 2024-04-22T18:39:09.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161910+0000", 2024-04-22T18:39:09.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202673879 2024-04-22T18:39:09.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161937+0000", 2024-04-22T18:39:09.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7202000000000001e-05 2024-04-22T18:39:09.255 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.255 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.255 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:97 4.3 e27/20 4:c93c842e:::smithi08937929-25:head v 27'10, mlcod=27'3)", 2024-04-22T18:39:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:39:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.233563237, 2024-04-22T18:39:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.204078805, 2024-04-22T18:39:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:39:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958337+0000", 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045682399999999999 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958346+0000", 2024-04-22T18:39:09.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6840000000000008e-06 2024-04-22T18:39:09.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958353+0000", 2024-04-22T18:39:09.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0330000000000003e-06 2024-04-22T18:39:09.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959462+0000", 2024-04-22T18:39:09.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011084440000000001 2024-04-22T18:39:09.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959479+0000", 2024-04-22T18:39:09.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6654e-05 2024-04-22T18:39:09.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161940+0000", 2024-04-22T18:39:09.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202461112 2024-04-22T18:39:09.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161959+0000", 2024-04-22T18:39:09.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9477999999999999e-05 2024-04-22T18:39:09.259 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:98 4.3 e27/20 4:c93c842e:::smithi08937929-25:head v 27'11, mlcod=27'3)", 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.233085131999999, 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203627161, 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958880+0000", 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052192199999999997 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958887+0000", 2024-04-22T18:39:09.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:39:09.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958891+0000", 2024-04-22T18:39:09.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2899999999999996e-06 2024-04-22T18:39:09.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.060417+0000", 2024-04-22T18:39:09.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10152593 2024-04-22T18:39:09.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.060433+0000", 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6232e-05 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161964+0000", 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101531234 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161986+0000", 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1090999999999998e-05 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.263 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:99 4.3 e27/20 4:c93c842e:::smithi08937929-25:head v 27'12, mlcod=27'3)", 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.232549217000001, 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20320666600000001, 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:39:09.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:39:09.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:09.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958897+0000", 2024-04-22T18:39:09.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4429999999999999e-06 2024-04-22T18:39:09.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958900+0000", 2024-04-22T18:39:09.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4490000000000002e-06 2024-04-22T18:39:09.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958902+0000", 2024-04-22T18:39:09.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.322e-06 2024-04-22T18:39:09.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161689+0000", 2024-04-22T18:39:09.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20278693 2024-04-22T18:39:09.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161712+0000", 2024-04-22T18:39:09.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2938999999999998e-05 2024-04-22T18:39:09.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.162077+0000", 2024-04-22T18:39:09.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036527800000000002 2024-04-22T18:39:09.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.162101+0000", 2024-04-22T18:39:09.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3785e-05 2024-04-22T18:39:09.268 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.268 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.268 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.268 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:09.268 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:09.268 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.198086011999999, 2024-04-22T18:39:09.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:39:09.268 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.268 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.268 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:39:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:39:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:39:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:39:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:39:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:39:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:39:09.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:39:09.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:39:09.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:39:09.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:39:09.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:39:09.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:39:09.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:39:09.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:39:09.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:09.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:39:09.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:39:09.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:09.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:39:09.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:39:09.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:09.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:39:09.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:39:09.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:39:09.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:39:09.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:39:09.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:39:09.274 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.274 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.274 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.274 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:27 3.2 3:7a6cba00:::smithi08937929-32 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:09.274 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.597045+0000", 2024-04-22T18:39:09.274 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.594398519, 2024-04-22T18:39:09.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20443814799999999, 2024-04-22T18:39:09.274 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.274 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.274 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597045+0000", 2024-04-22T18:39:09.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597045+0000", 2024-04-22T18:39:09.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597043+0000", 2024-04-22T18:39:09.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:09.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597053+0000", 2024-04-22T18:39:09.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0278999999999999e-05 2024-04-22T18:39:09.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597074+0000", 2024-04-22T18:39:09.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0710999999999999e-05 2024-04-22T18:39:09.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.697204+0000", 2024-04-22T18:39:09.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10012958299999999 2024-04-22T18:39:09.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.698362+0000", 2024-04-22T18:39:09.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011587629999999999 2024-04-22T18:39:09.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.698440+0000", 2024-04-22T18:39:09.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7453000000000005e-05 2024-04-22T18:39:09.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:09.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.698593+0000", 2024-04-22T18:39:09.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153449 2024-04-22T18:39:09.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:09.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.699048+0000", 2024-04-22T18:39:09.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000454287 2024-04-22T18:39:09.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:09.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.801433+0000", 2024-04-22T18:39:09.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102385664 2024-04-22T18:39:09.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.801462+0000", 2024-04-22T18:39:09.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9153e-05 2024-04-22T18:39:09.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.801483+0000", 2024-04-22T18:39:09.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0707e-05 2024-04-22T18:39:09.279 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.279 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.279 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.279 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:09.279 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:09.279 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.089786221000001, 2024-04-22T18:39:09.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:39:09.280 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.280 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.280 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:09.280 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:09.280 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:09.280 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:39:09.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.280 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:09.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:09.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:39:09.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:09.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:39:09.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:39:09.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:39:09.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:39:09.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:39:09.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:39:09.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:39:09.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:39:09.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:39:09.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:39:09.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:09.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:39:09.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:39:09.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:09.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:39:09.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:39:09.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:09.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:39:09.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:39:09.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:39:09.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:39:09.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:39:09.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:39:09.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.285 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.285 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:70 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:09.285 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:02.527752+0000", 2024-04-22T18:39:09.285 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.6636919270000003, 2024-04-22T18:39:09.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20158322100000001, 2024-04-22T18:39:09.285 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:09.285 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:09.285 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:09.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527752+0000", 2024-04-22T18:39:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527752+0000", 2024-04-22T18:39:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527750+0000", 2024-04-22T18:39:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:09.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527766+0000", 2024-04-22T18:39:09.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6398000000000001e-05 2024-04-22T18:39:09.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:09.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527780+0000", 2024-04-22T18:39:09.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.327e-05 2024-04-22T18:39:09.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:09.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527788+0000", 2024-04-22T18:39:09.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1729999999999995e-06 2024-04-22T18:39:09.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:09.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527831+0000", 2024-04-22T18:39:09.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3683000000000002e-05 2024-04-22T18:39:09.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:09.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527949+0000", 2024-04-22T18:39:09.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117271 2024-04-22T18:39:09.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:09.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.528131+0000", 2024-04-22T18:39:09.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000182525 2024-04-22T18:39:09.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:09.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.528750+0000", 2024-04-22T18:39:09.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00061904199999999999 2024-04-22T18:39:09.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:09.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.729283+0000", 2024-04-22T18:39:09.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20053246599999999 2024-04-22T18:39:09.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:09.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.729314+0000", 2024-04-22T18:39:09.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0889999999999997e-05 2024-04-22T18:39:09.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:09.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:09.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:09.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.729335+0000", 2024-04-22T18:39:09.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1231999999999999e-05 2024-04-22T18:39:09.290 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.290 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.290 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.290 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:09.290 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:09.290 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:09.292 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-22T18:39:09.292 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:09.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:09.293 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:09.294 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:09.294 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:09.294 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:09.294 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:09.295 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:09.295 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:09.295 INFO:tasks.rados.rados.0.smithi089.stdout:671: set_chunk oid 42 offset: 700928 length: 236544 target oid 8c9e1ab60958a744b00b0ef8a1de299ea73585581a3a2ab5041d3c8648981b49 offset: 0 snap -1 2024-04-22T18:39:09.296 INFO:tasks.rados.rados.0.smithi089.stdout:672: write (excl) oid 5 current snap is 0 2024-04-22T18:39:09.296 INFO:tasks.rados.rados.0.smithi089.stdout:672: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-04-22T18:39:09.296 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:09.296 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:09.297 INFO:tasks.rados.rados.0.smithi089.stdout:672: writing smithi08937929-5 from 709262 to 1406520 tid 1 2024-04-22T18:39:09.297 INFO:tasks.rados.rados.0.smithi089.stdout:672: writing smithi08937929-5 from 2012635 to 2584653 tid 2 2024-04-22T18:39:09.297 INFO:tasks.rados.rados.0.smithi089.stdout:672: writing smithi08937929-5 from 3046388 to 3109262 tid 3 2024-04-22T18:39:09.297 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:09.297 INFO:tasks.rados.rados.0.smithi089.stdout:658: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-22T18:39:09.297 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:09.298 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:09.298 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:09.292+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:09.392 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:09.393+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:09.402 INFO:tasks.rados.rados.0.smithi089.stdout:658: done (15 left) 2024-04-22T18:39:09.402 INFO:tasks.rados.rados.0.smithi089.stdout:661: done (14 left) 2024-04-22T18:39:09.402 INFO:tasks.rados.rados.0.smithi089.stdout:673: set_chunk oid 44 target oid 2024-04-22T18:39:09.403 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:09.403 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:09.412 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:09.487 INFO:tasks.rados.rados.0.smithi089.stdout:673: set_chunk oid 44 offset: 308224 length: 1598464 target oid fe3c84bf5500e4c9f1866e9815017e869a33754ba4e4731fb3bfd02def2c30db offset: 0 snap -1 2024-04-22T18:39:09.487 INFO:tasks.rados.rados.0.smithi089.stdout:674: tier_evict oid 45 2024-04-22T18:39:09.487 INFO:tasks.rados.rados.0.smithi089.stdout:674: tier_evict oid 45 snap -1 2024-04-22T18:39:09.487 INFO:tasks.rados.rados.0.smithi089.stdout:657: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-22T18:39:09.493 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:09.494+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:09.514 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:09.514 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:09.523 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:09.570 INFO:tasks.rados.rados.0.smithi089.stdout:660: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-22T18:39:09.607 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:09.608+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:09.657 INFO:tasks.rados.rados.0.smithi089.stdout:659: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-22T18:39:09.665 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:09.666 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:09.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:09.715 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:09.716+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:09.733 INFO:tasks.rados.rados.0.smithi089.stdout:662: finishing set_chunk to oid 49 2024-04-22T18:39:09.733 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 1219072 length: 980992 2024-04-22T18:39:09.733 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:39:09.734 INFO:tasks.rados.rados.0.smithi089.stdout:657: done (15 left) 2024-04-22T18:39:09.734 INFO:tasks.rados.rados.0.smithi089.stdout:659: done (14 left) 2024-04-22T18:39:09.734 INFO:tasks.rados.rados.0.smithi089.stdout:660: done (13 left) 2024-04-22T18:39:09.734 INFO:tasks.rados.rados.0.smithi089.stdout:662: done (12 left) 2024-04-22T18:39:09.734 INFO:tasks.rados.rados.0.smithi089.stdout:675: set_chunk oid 8 target oid 2024-04-22T18:39:09.734 INFO:tasks.rados.rados.0.smithi089.stdout:675: set_chunk oid 8 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:39:09.734 INFO:tasks.rados.rados.0.smithi089.stdout:676: tier_flush oid 2 2024-04-22T18:39:09.734 INFO:tasks.rados.rados.0.smithi089.stdout:676: tier_flush oid 2 snap -1 2024-04-22T18:39:09.734 INFO:tasks.rados.rados.0.smithi089.stdout:665: got (22) Invalid argument 2024-04-22T18:39:09.734 INFO:tasks.rados.rados.0.smithi089.stdout:663: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-22T18:39:09.795 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:09.796+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:09.816 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:09.817 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:09.823 INFO:tasks.rados.rados.0.smithi089.stdout:664: expect (ObjNum 40 snap 0 seq_num 40) 2024-04-22T18:39:09.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:09.909 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:09.910+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:09.914 INFO:tasks.rados.rados.0.smithi089.stdout:666: finishing set_chunk to oid 36 2024-04-22T18:39:09.914 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 36 set_chunk returned error code -95 offset: 393216 length: 1965056 2024-04-22T18:39:09.914 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-22T18:39:09.914 INFO:tasks.rados.rados.0.smithi089.stdout:667: finishing set_chunk to oid 28 2024-04-22T18:39:09.914 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 28 set_chunk returned error code -95 offset: 947200 length: 1502208 2024-04-22T18:39:09.914 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-22T18:39:09.914 INFO:tasks.rados.rados.0.smithi089.stdout:668: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-22T18:39:09.968 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:09.968 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:09.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:09.993 INFO:tasks.rados.rados.0.smithi089.stdout:669: finishing write tid 1 to smithi08937929-33 2024-04-22T18:39:09.993 INFO:tasks.rados.rados.0.smithi089.stdout:669: finishing write tid 2 to smithi08937929-33 2024-04-22T18:39:09.993 INFO:tasks.rados.rados.0.smithi089.stdout:663: done (13 left) 2024-04-22T18:39:09.993 INFO:tasks.rados.rados.0.smithi089.stdout:664: done (12 left) 2024-04-22T18:39:09.993 INFO:tasks.rados.rados.0.smithi089.stdout:665: done (11 left) 2024-04-22T18:39:09.993 INFO:tasks.rados.rados.0.smithi089.stdout:666: done (10 left) 2024-04-22T18:39:09.993 INFO:tasks.rados.rados.0.smithi089.stdout:667: done (9 left) 2024-04-22T18:39:09.993 INFO:tasks.rados.rados.0.smithi089.stdout:668: done (8 left) 2024-04-22T18:39:09.994 INFO:tasks.rados.rados.0.smithi089.stdout:677: write (excl) oid 12 current snap is 0 2024-04-22T18:39:09.994 INFO:tasks.rados.rados.0.smithi089.stdout:669: finishing write tid 3 to smithi08937929-33 2024-04-22T18:39:09.994 INFO:tasks.rados.rados.0.smithi089.stdout:669: finishing write tid 4 to smithi08937929-33 2024-04-22T18:39:09.994 INFO:tasks.rados.rados.0.smithi089.stdout:669: oid 33 updating version 0 to 18 2024-04-22T18:39:09.994 INFO:tasks.rados.rados.0.smithi089.stdout:669: oid 33 updating version 18 to 19 2024-04-22T18:39:09.994 INFO:tasks.rados.rados.0.smithi089.stdout:669: oid 33 version 19 is already newer than 17 2024-04-22T18:39:09.994 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 19 (ObjNum 65 snap 0 seq_num 65) dirty exists 2024-04-22T18:39:09.994 INFO:tasks.rados.rados.0.smithi089.stdout:669: left oid 33 (ObjNum 65 snap 0 seq_num 65) 2024-04-22T18:39:09.994 INFO:tasks.rados.rados.0.smithi089.stdout:671: finishing set_chunk to oid 42 2024-04-22T18:39:09.994 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 42 set_chunk returned error code -95 offset: 700928 length: 236544 2024-04-22T18:39:09.994 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-22T18:39:09.994 INFO:tasks.rados.rados.0.smithi089.stdout:670: finishing set_chunk to oid 34 2024-04-22T18:39:09.994 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 34 (ObjNum 59 snap 0 seq_num 59) dirty exists 2024-04-22T18:39:09.994 INFO:tasks.rados.rados.0.smithi089.stdout:672: finishing write tid 1 to smithi08937929-5 2024-04-22T18:39:09.995 INFO:tasks.rados.rados.0.smithi089.stdout:672: finishing write tid 2 to smithi08937929-5 2024-04-22T18:39:09.995 INFO:tasks.rados.rados.0.smithi089.stdout:677: seq_num 67 ranges {726069=772507,2071117=715676} 2024-04-22T18:39:10.000 INFO:tasks.rados.rados.0.smithi089.stdout:677: writing smithi08937929-12 from 726069 to 1498576 tid 1 2024-04-22T18:39:10.009 INFO:tasks.rados.rados.0.smithi089.stdout:677: writing smithi08937929-12 from 2071117 to 2786793 tid 2 2024-04-22T18:39:10.009 INFO:tasks.rados.rados.0.smithi089.stdout:669: done (8 left) 2024-04-22T18:39:10.009 INFO:tasks.rados.rados.0.smithi089.stdout:670: done (7 left) 2024-04-22T18:39:10.009 INFO:tasks.rados.rados.0.smithi089.stdout:671: done (6 left) 2024-04-22T18:39:10.009 INFO:tasks.rados.rados.0.smithi089.stdout:678: read oid 40 snap -1 2024-04-22T18:39:10.009 INFO:tasks.rados.rados.0.smithi089.stdout:678: expect (ObjNum 58 snap 0 seq_num 58) 2024-04-22T18:39:10.009 INFO:tasks.rados.rados.0.smithi089.stdout:679: read oid 21 snap -1 2024-04-22T18:39:10.009 INFO:tasks.rados.rados.0.smithi089.stdout:679: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-22T18:39:10.009 INFO:tasks.rados.rados.0.smithi089.stdout:680: write (excl) oid 47 current snap is 0 2024-04-22T18:39:10.009 INFO:tasks.rados.rados.0.smithi089.stdout:672: finishing write tid 3 to smithi08937929-5 2024-04-22T18:39:10.010 INFO:tasks.rados.rados.0.smithi089.stdout:672: finishing write tid 4 to smithi08937929-5 2024-04-22T18:39:10.010 INFO:tasks.rados.rados.0.smithi089.stdout:672: finishing write tid 5 to smithi08937929-5 2024-04-22T18:39:10.010 INFO:tasks.rados.rados.0.smithi089.stdout:672: oid 5 updating version 0 to 43 2024-04-22T18:39:10.010 INFO:tasks.rados.rados.0.smithi089.stdout:672: oid 5 version 43 is already newer than 42 2024-04-22T18:39:10.010 INFO:tasks.rados.rados.0.smithi089.stdout:672: oid 5 updating version 43 to 44 2024-04-22T18:39:10.010 INFO:tasks.rados.rados.0.smithi089.stdout:672: oid 5 version 44 is already newer than 41 2024-04-22T18:39:10.010 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 44 (ObjNum 66 snap 0 seq_num 66) dirty exists 2024-04-22T18:39:10.010 INFO:tasks.rados.rados.0.smithi089.stdout:672: left oid 5 (ObjNum 66 snap 0 seq_num 66) 2024-04-22T18:39:10.010 INFO:tasks.rados.rados.0.smithi089.stdout:674: got (22) Invalid argument 2024-04-22T18:39:10.010 INFO:tasks.rados.rados.0.smithi089.stdout:673: finishing set_chunk to oid 44 2024-04-22T18:39:10.010 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 36 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-04-22T18:39:10.010 INFO:tasks.rados.rados.0.smithi089.stdout:675: finishing set_chunk to oid 8 2024-04-22T18:39:10.010 INFO:tasks.rados.rados.0.smithi089.stdout:675: got expected ENOENT (src dne) 2024-04-22T18:39:10.010 INFO:tasks.rados.rados.0.smithi089.stdout:680: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-04-22T18:39:10.014 INFO:tasks.rados.rados.0.smithi089.stdout:680: writing smithi08937929-47 from 742876 to 1190632 tid 1 2024-04-22T18:39:10.018 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:10.019+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:10.021 INFO:tasks.rados.rados.0.smithi089.stdout:680: writing smithi08937929-47 from 1729599 to 2305286 tid 2 2024-04-22T18:39:10.026 INFO:tasks.rados.rados.0.smithi089.stdout:680: writing smithi08937929-47 from 3101234 to 3142876 tid 3 2024-04-22T18:39:10.026 INFO:tasks.rados.rados.0.smithi089.stdout:672: done (8 left) 2024-04-22T18:39:10.026 INFO:tasks.rados.rados.0.smithi089.stdout:673: done (7 left) 2024-04-22T18:39:10.026 INFO:tasks.rados.rados.0.smithi089.stdout:674: done (6 left) 2024-04-22T18:39:10.026 INFO:tasks.rados.rados.0.smithi089.stdout:675: done (5 left) 2024-04-22T18:39:10.026 INFO:tasks.rados.rados.0.smithi089.stdout:681: write (excl) oid 32 current snap is 0 2024-04-22T18:39:10.026 INFO:tasks.rados.rados.0.smithi089.stdout:677: finishing write tid 1 to smithi08937929-12 2024-04-22T18:39:10.026 INFO:tasks.rados.rados.0.smithi089.stdout:681: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2024-04-22T18:39:10.033 INFO:tasks.rados.rados.0.smithi089.stdout:681: writing smithi08937929-32 from 759683 to 1399041 tid 1 2024-04-22T18:39:10.039 INFO:tasks.rados.rados.0.smithi089.stdout:681: writing smithi08937929-32 from 1904434 to 2623779 tid 2 2024-04-22T18:39:10.043 INFO:tasks.rados.rados.0.smithi089.stdout:681: writing smithi08937929-32 from 3128657 to 3159683 tid 3 2024-04-22T18:39:10.043 INFO:tasks.rados.rados.0.smithi089.stdout:682: tier_evict oid 44 2024-04-22T18:39:10.043 INFO:tasks.rados.rados.0.smithi089.stdout:682: tier_evict oid 44 snap -1 2024-04-22T18:39:10.043 INFO:tasks.rados.rados.0.smithi089.stdout:683: delete oid 15 current snap is 0 2024-04-22T18:39:10.043 INFO:tasks.rados.rados.0.smithi089.stdout:678: expect (ObjNum 58 snap 0 seq_num 58) 2024-04-22T18:39:10.097 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:10.098+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:10.119 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:10.119 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:10.128 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:10.140 INFO:tasks.rados.rados.0.smithi089.stdout:680: finishing write tid 1 to smithi08937929-47 2024-04-22T18:39:10.140 INFO:tasks.rados.rados.0.smithi089.stdout:682: got (22) Invalid argument 2024-04-22T18:39:10.140 INFO:tasks.rados.rados.0.smithi089.stdout:679: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-22T18:39:10.183 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.7 to 1 (10000) 2024-04-22T18:39:10.196 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-22T18:39:10.196 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:39:10.198 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:10.199+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:10.222 INFO:tasks.rados.rados.0.smithi089.stdout:676: got (0) Success 2024-04-22T18:39:10.222 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-22T18:39:10.222 INFO:tasks.rados.rados.0.smithi089.stdout:677: finishing write tid 2 to smithi08937929-12 2024-04-22T18:39:10.234 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:10.235 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:10.244 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:10.251 INFO:tasks.rados.rados.0.smithi089.stdout:681: finishing write tid 1 to smithi08937929-32 2024-04-22T18:39:10.281 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:39:10.298 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:10.299+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:10.326 INFO:tasks.rados.rados.0.smithi089.stdout:680: finishing write tid 2 to smithi08937929-47 2024-04-22T18:39:10.327 INFO:tasks.rados.rados.0.smithi089.stdout:680: finishing write tid 3 to smithi08937929-47 2024-04-22T18:39:10.352 INFO:tasks.rados.rados.0.smithi089.stdout:681: finishing write tid 2 to smithi08937929-32 2024-04-22T18:39:10.352 INFO:tasks.rados.rados.0.smithi089.stdout:681: finishing write tid 3 to smithi08937929-32 2024-04-22T18:39:10.352 INFO:tasks.rados.rados.0.smithi089.stdout:681: finishing write tid 4 to smithi08937929-32 2024-04-22T18:39:10.352 INFO:tasks.rados.rados.0.smithi089.stdout:681: finishing write tid 5 to smithi08937929-32 2024-04-22T18:39:10.352 INFO:tasks.rados.rados.0.smithi089.stdout:681: oid 32 updating version 0 to 23 2024-04-22T18:39:10.352 INFO:tasks.rados.rados.0.smithi089.stdout:681: oid 32 version 23 is already newer than 21 2024-04-22T18:39:10.352 INFO:tasks.rados.rados.0.smithi089.stdout:681: oid 32 version 23 is already newer than 20 2024-04-22T18:39:10.353 INFO:tasks.rados.rados.0.smithi089.stdout:681: oid 32 version 23 is already newer than 22 2024-04-22T18:39:10.353 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 23 (ObjNum 69 snap 0 seq_num 69) dirty exists 2024-04-22T18:39:10.353 INFO:tasks.rados.rados.0.smithi089.stdout:681: left oid 32 (ObjNum 69 snap 0 seq_num 69) 2024-04-22T18:39:10.356 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:10.356 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:10.366 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:10.399 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:10.400+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:10.412 INFO:tasks.rados.rados.0.smithi089.stdout:677: finishing write tid 3 to smithi08937929-12 2024-04-22T18:39:10.412 INFO:tasks.rados.rados.0.smithi089.stdout:677: finishing write tid 4 to smithi08937929-12 2024-04-22T18:39:10.412 INFO:tasks.rados.rados.0.smithi089.stdout:677: oid 12 updating version 0 to 27 2024-04-22T18:39:10.412 INFO:tasks.rados.rados.0.smithi089.stdout:677: oid 12 updating version 27 to 28 2024-04-22T18:39:10.413 INFO:tasks.rados.rados.0.smithi089.stdout:677: oid 12 updating version 28 to 29 2024-04-22T18:39:10.413 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 29 (ObjNum 67 snap 0 seq_num 67) dirty exists 2024-04-22T18:39:10.413 INFO:tasks.rados.rados.0.smithi089.stdout:677: left oid 12 (ObjNum 67 snap 0 seq_num 67) 2024-04-22T18:39:10.500 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:10.501+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:10.509 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:10.509 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:10.515 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:39:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:39:10.518 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:10.527 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2024-04-22T18:39:10.182882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":235193764,"num_objects":116,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":575,"num_read_kb":182513,"num_write":638,"num_write_kb":194969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":44,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":645,"ondisk_log_size":645,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":446656,"kb_used_data":222520,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754528064,"statfs":{"total":773094113280,"available":772636737536,"internally_reserved":0,"allocated":227860480,"data_stored":236530058,"data_compressed":8701134,"data_compressed_allocated":9297920,"data_compressed_original":20545536,"omap_allocated":12744,"internal_metadata":225431096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5252168,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":79648,"num_write":37,"num_write_kb":5607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030802"},"pg_stats":[{"pgid":"4.f","version":"31'16","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.554844+0000","last_change":"2024-04-22T18:38:48.173640+0000","last_active":"2024-04-22T18:39:05.554844+0000","last_peered":"2024-04-22T18:39:05.554844+0000","last_clean":"2024-04-22T18:39:05.554844+0000","last_became_active":"2024-04-22T18:38:48.173369+0000","last_became_peered":"2024-04-22T18:38:48.173369+0000","last_unstale":"2024-04-22T18:39:05.554844+0000","last_undegraded":"2024-04-22T18:39:05.554844+0000","last_fullsized":"2024-04-22T18:39:05.554844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:08.819198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6184877,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":8689,"num_write":12,"num_write_kb":2771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"33'10","reported_seq":36,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.162285+0000","last_change":"2024-04-22T18:38:48.075222+0000","last_active":"2024-04-22T18:39:04.162285+0000","last_peered":"2024-04-22T18:39:04.162285+0000","last_clean":"2024-04-22T18:39:04.162285+0000","last_became_active":"2024-04-22T18:38:48.074901+0000","last_became_peered":"2024-04-22T18:38:48.074901+0000","last_unstale":"2024-04-22T18:39:04.162285+0000","last_undegraded":"2024-04-22T18:39:04.162285+0000","last_fullsized":"2024-04-22T18:39:04.162285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.172332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756315,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":2372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"32'12","reported_seq":44,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.544121+0000","last_change":"2024-04-22T18:38:48.172087+0000","last_active":"2024-04-22T18:39:04.544121+0000","last_peered":"2024-04-22T18:39:04.544121+0000","last_clean":"2024-04-22T18:39:04.544121+0000","last_became_active":"2024-04-22T18:38:48.171797+0000","last_became_peered":"2024-04-22T18:38:48.171797+0000","last_unstale":"2024-04-22T18:39:04.544121+0000","last_undegraded":"2024-04-22T18:39:04.544121+0000","last_fullsized":"2024-04-22T18:39:04.544121+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.134124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":19156,"num_write":8,"num_write_kb":1802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"33'26","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.897206+0000","last_change":"2024-04-22T18:38:48.274538+0000","last_active":"2024-04-22T18:39:05.897206+0000","last_peered":"2024-04-22T18:39:05.897206+0000","last_clean":"2024-04-22T18:39:05.897206+0000","last_became_active":"2024-04-22T18:38:48.174204+0000","last_became_peered":"2024-04-22T18:38:48.174204+0000","last_unstale":"2024-04-22T18:39:05.897206+0000","last_undegraded":"2024-04-22T18:39:05.897206+0000","last_fullsized":"2024-04-22T18:39:05.897206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:11.823691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10437015,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":30,"num_read_kb":23261,"num_write":20,"num_write_kb":5534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"35'36","reported_seq":78,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.892883+0000","last_change":"2024-04-22T18:38:48.269362+0000","last_active":"2024-04-22T18:39:05.892883+0000","last_peered":"2024-04-22T18:39:05.892883+0000","last_clean":"2024-04-22T18:39:05.892883+0000","last_became_active":"2024-04-22T18:38:48.269066+0000","last_became_peered":"2024-04-22T18:38:48.269066+0000","last_unstale":"2024-04-22T18:39:05.892883+0000","last_undegraded":"2024-04-22T18:39:05.892883+0000","last_fullsized":"2024-04-22T18:39:05.892883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:52.098608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13142909,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":24714,"num_write":30,"num_write_kb":8426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"35'19","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.253252+0000","last_change":"2024-04-22T18:38:48.372952+0000","last_active":"2024-04-22T18:39:05.253252+0000","last_peered":"2024-04-22T18:39:05.253252+0000","last_clean":"2024-04-22T18:39:05.253252+0000","last_became_active":"2024-04-22T18:38:48.277922+0000","last_became_peered":"2024-04-22T18:38:48.277922+0000","last_unstale":"2024-04-22T18:39:05.253252+0000","last_undegraded":"2024-04-22T18:39:05.253252+0000","last_fullsized":"2024-04-22T18:39:05.253252+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:15.471951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7092455,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":3383,"num_write":17,"num_write_kb":4007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"35'34","reported_seq":71,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:06.170529+0000","last_change":"2024-04-22T18:38:48.271633+0000","last_active":"2024-04-22T18:39:06.170529+0000","last_peered":"2024-04-22T18:39:06.170529+0000","last_clean":"2024-04-22T18:39:06.170529+0000","last_became_active":"2024-04-22T18:38:48.171245+0000","last_became_peered":"2024-04-22T18:38:48.171245+0000","last_unstale":"2024-04-22T18:39:06.170529+0000","last_undegraded":"2024-04-22T18:39:06.170529+0000","last_fullsized":"2024-04-22T18:39:06.170529+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:47.449912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12537857,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":15230,"num_write":25,"num_write_kb":6740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"34'19","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.901689+0000","last_change":"2024-04-22T18:38:48.149226+0000","last_active":"2024-04-22T18:39:04.901689+0000","last_peered":"2024-04-22T18:39:04.901689+0000","last_clean":"2024-04-22T18:39:04.901689+0000","last_became_active":"2024-04-22T18:38:48.148955+0000","last_became_peered":"2024-04-22T18:38:48.148955+0000","last_unstale":"2024-04-22T18:39:04.901689+0000","last_undegraded":"2024-04-22T18:39:04.901689+0000","last_fullsized":"2024-04-22T18:39:04.901689+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:02.394854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7882384,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":11281,"num_write":17,"num_write_kb":4337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"34'19","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.895680+0000","last_change":"2024-04-22T18:38:48.277905+0000","last_active":"2024-04-22T18:39:05.895680+0000","last_peered":"2024-04-22T18:39:05.895680+0000","last_clean":"2024-04-22T18:39:05.895680+0000","last_became_active":"2024-04-22T18:38:48.277640+0000","last_became_peered":"2024-04-22T18:38:48.277640+0000","last_unstale":"2024-04-22T18:39:05.895680+0000","last_undegraded":"2024-04-22T18:39:05.895680+0000","last_fullsized":"2024-04-22T18:39:05.895680+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:57.153161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5831963,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":16784,"num_write":17,"num_write_kb":4628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":178,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.240945+0000","last_change":"2024-04-22T18:38:38.109358+0000","last_active":"2024-04-22T18:39:04.240945+0000","last_peered":"2024-04-22T18:39:04.240945+0000","last_clean":"2024-04-22T18:39:04.240945+0000","last_became_active":"2024-04-22T18:38:38.108568+0000","last_became_peered":"2024-04-22T18:38:38.108568+0000","last_unstale":"2024-04-22T18:39:04.240945+0000","last_undegraded":"2024-04-22T18:39:04.240945+0000","last_fullsized":"2024-04-22T18:39:04.240945+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:38:36.991764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:46.436569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.241963+0000","last_change":"2024-04-22T18:38:39.010398+0000","last_active":"2024-04-22T18:39:04.241963+0000","last_peered":"2024-04-22T18:39:04.241963+0000","last_clean":"2024-04-22T18:39:04.241963+0000","last_became_active":"2024-04-22T18:38:39.010101+0000","last_became_peered":"2024-04-22T18:38:39.010101+0000","last_unstale":"2024-04-22T18:39:04.241963+0000","last_undegraded":"2024-04-22T18:39:04.241963+0000","last_fullsized":"2024-04-22T18:39:04.241963+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.708134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"34'42","reported_seq":76,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.988839+0000","last_change":"2024-04-22T18:38:58.429682+0000","last_active":"2024-04-22T18:39:04.988839+0000","last_peered":"2024-04-22T18:39:04.988839+0000","last_clean":"2024-04-22T18:39:04.988839+0000","last_became_active":"2024-04-22T18:38:58.125648+0000","last_became_peered":"2024-04-22T18:38:58.125648+0000","last_unstale":"2024-04-22T18:39:04.988839+0000","last_undegraded":"2024-04-22T18:39:04.988839+0000","last_fullsized":"2024-04-22T18:39:04.988839+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:52.346195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28872355,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":30,"num_read_kb":17,"num_write":44,"num_write_kb":28201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"32'19","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.960464+0000","last_change":"2024-04-22T18:38:48.173369+0000","last_active":"2024-04-22T18:39:05.960464+0000","last_peered":"2024-04-22T18:39:05.960464+0000","last_clean":"2024-04-22T18:39:05.960464+0000","last_became_active":"2024-04-22T18:38:48.173090+0000","last_became_peered":"2024-04-22T18:38:48.173090+0000","last_unstale":"2024-04-22T18:39:05.960464+0000","last_undegraded":"2024-04-22T18:39:05.960464+0000","last_fullsized":"2024-04-22T18:39:05.960464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:05.687904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8100875,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":14100,"num_write":17,"num_write_kb":5390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.140702+0000","last_change":"2024-04-22T18:38:39.125885+0000","last_active":"2024-04-22T18:39:04.140702+0000","last_peered":"2024-04-22T18:39:04.140702+0000","last_clean":"2024-04-22T18:39:04.140702+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:39:04.140702+0000","last_undegraded":"2024-04-22T18:39:04.140702+0000","last_fullsized":"2024-04-22T18:39:04.140702+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.321452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"35'60","reported_seq":84,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.454484+0000","last_change":"2024-04-22T18:38:47.173337+0000","last_active":"2024-04-22T18:39:05.454484+0000","last_peered":"2024-04-22T18:39:05.454484+0000","last_clean":"2024-04-22T18:39:05.454484+0000","last_became_active":"2024-04-22T18:38:47.173065+0000","last_became_peered":"2024-04-22T18:38:47.173065+0000","last_unstale":"2024-04-22T18:39:05.454484+0000","last_undegraded":"2024-04-22T18:39:05.454484+0000","last_fullsized":"2024-04-22T18:39:05.454484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:52.987154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25419012,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":47,"num_read_kb":30,"num_write":64,"num_write_kb":31463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"35'16","reported_seq":47,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.354610+0000","last_change":"2024-04-22T18:38:48.278028+0000","last_active":"2024-04-22T18:39:05.354610+0000","last_peered":"2024-04-22T18:39:05.354610+0000","last_clean":"2024-04-22T18:39:05.354610+0000","last_became_active":"2024-04-22T18:38:48.277802+0000","last_became_peered":"2024-04-22T18:38:48.277802+0000","last_unstale":"2024-04-22T18:39:05.354610+0000","last_undegraded":"2024-04-22T18:39:05.354610+0000","last_fullsized":"2024-04-22T18:39:05.354610+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:12.000401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5428595,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":5714,"num_write":13,"num_write_kb":3557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.367651+0000","last_change":"2024-04-22T18:38:39.164369+0000","last_active":"2024-04-22T18:39:04.367651+0000","last_peered":"2024-04-22T18:39:04.367651+0000","last_clean":"2024-04-22T18:39:04.367651+0000","last_became_active":"2024-04-22T18:38:39.164089+0000","last_became_peered":"2024-04-22T18:38:39.164089+0000","last_unstale":"2024-04-22T18:39:04.367651+0000","last_undegraded":"2024-04-22T18:39:04.367651+0000","last_fullsized":"2024-04-22T18:39:04.367651+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:44.825470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"34'40","reported_seq":63,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.002781+0000","last_change":"2024-04-22T18:38:47.172071+0000","last_active":"2024-04-22T18:39:05.002781+0000","last_peered":"2024-04-22T18:39:05.002781+0000","last_clean":"2024-04-22T18:39:05.002781+0000","last_became_active":"2024-04-22T18:38:47.171789+0000","last_became_peered":"2024-04-22T18:38:47.171789+0000","last_unstale":"2024-04-22T18:39:05.002781+0000","last_undegraded":"2024-04-22T18:39:05.002781+0000","last_fullsized":"2024-04-22T18:39:05.002781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:47.501470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24706578,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":31,"num_read_kb":14,"num_write":48,"num_write_kb":24135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"29'4","reported_seq":31,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.793972+0000","last_change":"2024-04-22T18:38:48.171024+0000","last_active":"2024-04-22T18:39:05.793972+0000","last_peered":"2024-04-22T18:39:05.793972+0000","last_clean":"2024-04-22T18:39:05.793972+0000","last_became_active":"2024-04-22T18:38:48.170789+0000","last_became_peered":"2024-04-22T18:38:48.170789+0000","last_unstale":"2024-04-22T18:39:05.793972+0000","last_undegraded":"2024-04-22T18:39:05.793972+0000","last_fullsized":"2024-04-22T18:39:05.793972+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:06.487306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2689087,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11735,"num_write":4,"num_write_kb":1223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.241989+0000","last_change":"2024-04-22T18:38:39.111223+0000","last_active":"2024-04-22T18:39:04.241989+0000","last_peered":"2024-04-22T18:39:04.241989+0000","last_clean":"2024-04-22T18:39:04.241989+0000","last_became_active":"2024-04-22T18:38:39.110914+0000","last_became_peered":"2024-04-22T18:38:39.110914+0000","last_unstale":"2024-04-22T18:39:04.241989+0000","last_undegraded":"2024-04-22T18:39:04.241989+0000","last_fullsized":"2024-04-22T18:39:04.241989+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.570327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"35'74","reported_seq":97,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.555502+0000","last_change":"2024-04-22T18:38:47.173101+0000","last_active":"2024-04-22T18:39:05.555502+0000","last_peered":"2024-04-22T18:39:05.555502+0000","last_clean":"2024-04-22T18:39:05.555502+0000","last_became_active":"2024-04-22T18:38:47.172816+0000","last_became_peered":"2024-04-22T18:38:47.172816+0000","last_unstale":"2024-04-22T18:39:05.555502+0000","last_undegraded":"2024-04-22T18:39:05.555502+0000","last_fullsized":"2024-04-22T18:39:05.555502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:12.128375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36042037,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":54,"num_read_kb":32,"num_write":78,"num_write_kb":42658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"32'13","reported_seq":42,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.222402+0000","last_change":"2024-04-22T18:38:48.372570+0000","last_active":"2024-04-22T18:39:04.222402+0000","last_peered":"2024-04-22T18:39:04.222402+0000","last_clean":"2024-04-22T18:39:04.222402+0000","last_became_active":"2024-04-22T18:38:48.272283+0000","last_became_peered":"2024-04-22T18:38:48.272283+0000","last_unstale":"2024-04-22T18:39:04.222402+0000","last_undegraded":"2024-04-22T18:39:04.222402+0000","last_fullsized":"2024-04-22T18:39:04.222402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:09.183390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4319333,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":6262,"num_write":8,"num_write_kb":2087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.222497+0000","last_change":"2024-04-22T18:38:39.307735+0000","last_active":"2024-04-22T18:39:04.222497+0000","last_peered":"2024-04-22T18:39:04.222497+0000","last_clean":"2024-04-22T18:39:04.222497+0000","last_became_active":"2024-04-22T18:38:39.307459+0000","last_became_peered":"2024-04-22T18:38:39.307459+0000","last_unstale":"2024-04-22T18:39:04.222497+0000","last_undegraded":"2024-04-22T18:39:04.222497+0000","last_fullsized":"2024-04-22T18:39:04.222497+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.680561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"32'24","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.143455+0000","last_change":"2024-04-22T18:38:48.269379+0000","last_active":"2024-04-22T18:39:04.143455+0000","last_peered":"2024-04-22T18:39:04.143455+0000","last_clean":"2024-04-22T18:39:04.143455+0000","last_became_active":"2024-04-22T18:38:48.269070+0000","last_became_peered":"2024-04-22T18:38:48.269070+0000","last_unstale":"2024-04-22T18:39:04.143455+0000","last_undegraded":"2024-04-22T18:39:04.143455+0000","last_fullsized":"2024-04-22T18:39:04.143455+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:08.651956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9176490,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.140674+0000","last_change":"2024-04-22T18:38:39.210344+0000","last_active":"2024-04-22T18:39:04.140674+0000","last_peered":"2024-04-22T18:39:04.140674+0000","last_clean":"2024-04-22T18:39:04.140674+0000","last_became_active":"2024-04-22T18:38:39.210049+0000","last_became_peered":"2024-04-22T18:38:39.210049+0000","last_unstale":"2024-04-22T18:39:04.140674+0000","last_undegraded":"2024-04-22T18:39:04.140674+0000","last_fullsized":"2024-04-22T18:39:04.140674+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.878594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"31'5","reported_seq":30,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.441934+0000","last_change":"2024-04-22T18:38:48.349979+0000","last_active":"2024-04-22T18:39:04.441934+0000","last_peered":"2024-04-22T18:39:04.441934+0000","last_clean":"2024-04-22T18:39:04.441934+0000","last_became_active":"2024-04-22T18:38:48.349706+0000","last_became_peered":"2024-04-22T18:38:48.349706+0000","last_unstale":"2024-04-22T18:39:04.441934+0000","last_undegraded":"2024-04-22T18:39:04.441934+0000","last_fullsized":"2024-04-22T18:39:04.441934+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:54.435973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2319333,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.222414+0000","last_change":"2024-04-22T18:38:39.110499+0000","last_active":"2024-04-22T18:39:04.222414+0000","last_peered":"2024-04-22T18:39:04.222414+0000","last_clean":"2024-04-22T18:39:04.222414+0000","last_became_active":"2024-04-22T18:38:39.010173+0000","last_became_peered":"2024-04-22T18:38:39.010173+0000","last_unstale":"2024-04-22T18:39:04.222414+0000","last_undegraded":"2024-04-22T18:39:04.222414+0000","last_fullsized":"2024-04-22T18:39:04.222414+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:58.222501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"32'40","reported_seq":82,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.162291+0000","last_change":"2024-04-22T18:38:48.372226+0000","last_active":"2024-04-22T18:39:04.162291+0000","last_peered":"2024-04-22T18:39:04.162291+0000","last_clean":"2024-04-22T18:39:04.162291+0000","last_became_active":"2024-04-22T18:38:48.371943+0000","last_became_peered":"2024-04-22T18:38:48.371943+0000","last_unstale":"2024-04-22T18:39:04.162291+0000","last_undegraded":"2024-04-22T18:39:04.162291+0000","last_fullsized":"2024-04-22T18:39:04.162291+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.637563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17075681,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":22004,"num_write":28,"num_write_kb":8178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.423376+0000","last_change":"2024-04-22T18:38:39.026078+0000","last_active":"2024-04-22T18:39:04.423376+0000","last_peered":"2024-04-22T18:39:04.423376+0000","last_clean":"2024-04-22T18:39:04.423376+0000","last_became_active":"2024-04-22T18:38:39.025741+0000","last_became_peered":"2024-04-22T18:38:39.025741+0000","last_unstale":"2024-04-22T18:39:04.423376+0000","last_undegraded":"2024-04-22T18:39:04.423376+0000","last_fullsized":"2024-04-22T18:39:04.423376+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:01.125349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":119563414,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":297,"num_read_kb":182320,"num_write":245,"num_write_kb":66096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":44,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":116834304,"data_stored":115997104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":312,"ondisk_log_size":312,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":115039982,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":162,"num_read_kb":93,"num_write":234,"num_write_kb":126457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":108388352,"data_stored":118340092,"data_compressed":8386446,"data_compressed_allocated":8937472,"data_compressed_original":19062784,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":4964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":5083136,"data_stored":5295281,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.629}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34536,"kb_used_data":7508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337304,"statfs":{"total":96636764160,"available":96601399296,"internally_reserved":0,"allocated":7688192,"data_stored":7422224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62552,"kb_used_data":34708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94309288,"statfs":{"total":96636764160,"available":96572710912,"internally_reserved":0,"allocated":35540992,"data_stored":36611552,"data_compressed":1119849,"data_compressed_allocated":1204224,"data_compressed_original":2621440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":69.058999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80420,"kb_used_data":53120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94291420,"statfs":{"total":96636764160,"available":96554414080,"internally_reserved":0,"allocated":54394880,"data_stored":56725474,"data_compressed":2268767,"data_compressed_allocated":2412544,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49160,"kb_used_data":21852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94322680,"statfs":{"total":96636764160,"available":96586424320,"internally_reserved":0,"allocated":22376448,"data_stored":23499004,"data_compressed":1119849,"data_compressed_allocated":1204224,"data_compressed_original":2621440,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":94.063000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41824,"kb_used_data":14796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330016,"statfs":{"total":96636764160,"available":96593936384,"internally_reserved":0,"allocated":15151104,"data_stored":15950902,"data_compressed":932383,"data_compressed_allocated":991232,"data_compressed_original":2015232,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69828,"kb_used_data":40964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94302012,"statfs":{"total":96636764160,"available":96565260288,"internally_reserved":0,"allocated":41947136,"data_stored":42521455,"data_compressed":932383,"data_compressed_allocated":991232,"data_compressed_original":2015232,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.17899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75728,"kb_used_data":44608,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94296112,"statfs":{"total":96636764160,"available":96559218688,"internally_reserved":0,"allocated":45678592,"data_stored":48504166,"data_compressed":2298335,"data_compressed_allocated":2453504,"data_compressed_original":5636096,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29585408,"data_stored":32190954,"data_compressed":2208471,"data_compressed_allocated":2342912,"data_compressed_original":4993024,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11710464,"data_stored":12715899,"data_compressed":932383,"data_compressed_allocated":991232,"data_compressed_original":2015232,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11710464,"data_stored":12715899,"data_compressed":932383,"data_compressed_allocated":991232,"data_compressed_original":2015232,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13479936,"data_stored":14872515,"data_compressed":1119849,"data_compressed_allocated":1204224,"data_compressed_original":2621440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28422144,"data_stored":30972310,"data_compressed":2073511,"data_compressed_allocated":2203648,"data_compressed_original":4796416,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":13479936,"data_stored":14872515,"data_compressed":1119849,"data_compressed_allocated":1204224,"data_compressed_original":2621440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":15925248,"data_stored":15823002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29888512,"data_stored":29663535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3121152,"data_stored":3095407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8552448,"data_stored":8484022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25604096,"data_stored":25424431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":21757952,"data_stored":21605385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":7323648,"data_stored":7278475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4661248,"data_stored":4622847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:39:10.530 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:39:10.531 INFO:tasks.rados.rados.0.smithi089.stdout:680: finishing write tid 4 to smithi08937929-47 2024-04-22T18:39:10.532 INFO:tasks.rados.rados.0.smithi089.stdout:680: finishing write tid 5 to smithi08937929-47 2024-04-22T18:39:10.532 INFO:tasks.rados.rados.0.smithi089.stdout:680: oid 47 updating version 0 to 21 2024-04-22T18:39:10.532 INFO:tasks.rados.rados.0.smithi089.stdout:680: oid 47 version 21 is already newer than 20 2024-04-22T18:39:10.532 INFO:tasks.rados.rados.0.smithi089.stdout:680: oid 47 updating version 21 to 23 2024-04-22T18:39:10.532 INFO:tasks.rados.rados.0.smithi089.stdout:680: oid 47 version 23 is already newer than 22 2024-04-22T18:39:10.532 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 23 (ObjNum 68 snap 0 seq_num 68) dirty exists 2024-04-22T18:39:10.532 INFO:tasks.rados.rados.0.smithi089.stdout:680: left oid 47 (ObjNum 68 snap 0 seq_num 68) 2024-04-22T18:39:10.532 INFO:tasks.rados.rados.0.smithi089.stdout:676: done (7 left) 2024-04-22T18:39:10.532 INFO:tasks.rados.rados.0.smithi089.stdout:677: done (6 left) 2024-04-22T18:39:10.532 INFO:tasks.rados.rados.0.smithi089.stdout:678: done (5 left) 2024-04-22T18:39:10.532 INFO:tasks.rados.rados.0.smithi089.stdout:679: done (4 left) 2024-04-22T18:39:10.532 INFO:tasks.rados.rados.0.smithi089.stdout:680: done (3 left) 2024-04-22T18:39:10.532 INFO:tasks.rados.rados.0.smithi089.stdout:681: done (2 left) 2024-04-22T18:39:10.533 INFO:tasks.rados.rados.0.smithi089.stdout:682: done (1 left) 2024-04-22T18:39:10.533 INFO:tasks.rados.rados.0.smithi089.stdout:683: done (0 left) 2024-04-22T18:39:10.533 INFO:tasks.rados.rados.0.smithi089.stdout:684: set_chunk oid 38 target oid 2024-04-22T18:39:10.537 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:10.538+0000 7f1e2dd22640 -1 osd.2 39 handle_osd_map injecting map crc failure 2024-04-22T18:39:10.543 INFO:tasks.rados.rados.0.smithi089.stdout:684: set_chunk oid 38 offset: 441344 length: 2052096 target oid d30a6041105f84f6a2298762fe9cd0b487e3c32843d71f4659775f0beecd53cd offset: 0 snap -1 2024-04-22T18:39:10.544 INFO:tasks.rados.rados.0.smithi089.stdout:685: set_chunk oid 2 target oid 2024-04-22T18:39:10.601 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:10.602+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:10.633 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:10.633 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:10.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:10.650 INFO:tasks.rados.rados.0.smithi089.stdout:685: set_chunk oid 2 offset: 708608 length: 515584 target oid 6e5539b767ab6f624c49c9f74b6881fdb47cd6731791cc368a71888e832d72c6 offset: 0 snap -1 2024-04-22T18:39:10.650 INFO:tasks.rados.rados.0.smithi089.stdout:686: write (excl) oid 16 current snap is 0 2024-04-22T18:39:10.650 INFO:tasks.rados.rados.0.smithi089.stdout:684: finishing set_chunk to oid 38 2024-04-22T18:39:10.650 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 38 set_chunk returned error code -95 offset: 441344 length: 2052096 2024-04-22T18:39:10.650 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-22T18:39:10.650 INFO:tasks.rados.rados.0.smithi089.stdout:686: seq_num 71 ranges {793297=789856,2305045=723014} 2024-04-22T18:39:10.658 INFO:tasks.rados.rados.0.smithi089.stdout:686: writing smithi08937929-16 from 793297 to 1583153 tid 1 2024-04-22T18:39:10.666 INFO:tasks.rados.rados.0.smithi089.stdout:686: writing smithi08937929-16 from 2305045 to 3028059 tid 2 2024-04-22T18:39:10.666 INFO:tasks.rados.rados.0.smithi089.stdout:684: done (2 left) 2024-04-22T18:39:10.666 INFO:tasks.rados.rados.0.smithi089.stdout:687: read oid 4 snap -1 2024-04-22T18:39:10.667 INFO:tasks.rados.rados.0.smithi089.stdout:687: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-22T18:39:10.667 INFO:tasks.rados.rados.0.smithi089.stdout:688: delete oid 31 current snap is 0 2024-04-22T18:39:10.702 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:10.702+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:10.750 INFO:tasks.rados.rados.0.smithi089.stdout:685: finishing set_chunk to oid 2 2024-04-22T18:39:10.751 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk returned error code -95 offset: 708608 length: 515584 2024-04-22T18:39:10.751 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-22T18:39:10.751 INFO:tasks.rados.rados.0.smithi089.stdout:685: done (3 left) 2024-04-22T18:39:10.751 INFO:tasks.rados.rados.0.smithi089.stdout:688: done (2 left) 2024-04-22T18:39:10.751 INFO:tasks.rados.rados.0.smithi089.stdout:689: read oid 38 snap -1 2024-04-22T18:39:10.751 INFO:tasks.rados.rados.0.smithi089.stdout:689: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-22T18:39:10.751 INFO:tasks.rados.rados.0.smithi089.stdout:690: write (excl) oid 12 current snap is 0 2024-04-22T18:39:10.751 INFO:tasks.rados.rados.0.smithi089.stdout:690: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-04-22T18:39:10.753 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:10.753 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:10.755 INFO:tasks.rados.rados.0.smithi089.stdout:690: writing smithi08937929-12 from 426911 to 967265 tid 1 2024-04-22T18:39:10.763 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:10.763 INFO:tasks.rados.rados.0.smithi089.stdout:690: writing smithi08937929-12 from 1622009 to 2348692 tid 2 2024-04-22T18:39:10.769 INFO:tasks.rados.rados.0.smithi089.stdout:690: writing smithi08937929-12 from 3121996 to 3226911 tid 3 2024-04-22T18:39:10.769 INFO:tasks.rados.rados.0.smithi089.stdout:691: read oid 35 snap -1 2024-04-22T18:39:10.769 INFO:tasks.rados.rados.0.smithi089.stdout:691: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-22T18:39:10.769 INFO:tasks.rados.rados.0.smithi089.stdout:692: read oid 32 snap -1 2024-04-22T18:39:10.769 INFO:tasks.rados.rados.0.smithi089.stdout:692: expect (ObjNum 69 snap 0 seq_num 69) 2024-04-22T18:39:10.769 INFO:tasks.rados.rados.0.smithi089.stdout:693: read oid 15 snap -1 2024-04-22T18:39:10.769 INFO:tasks.rados.rados.0.smithi089.stdout:693: expect deleted 2024-04-22T18:39:10.769 INFO:tasks.rados.rados.0.smithi089.stdout:694: write oid 31 current snap is 0 2024-04-22T18:39:10.769 INFO:tasks.rados.rados.0.smithi089.stdout:694: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-04-22T18:39:10.774 INFO:tasks.rados.rados.0.smithi089.stdout:694: writing smithi08937929-31 from 443718 to 1059321 tid 1 2024-04-22T18:39:10.779 INFO:tasks.rados.rados.0.smithi089.stdout:694: writing smithi08937929-31 from 1564138 to 2034479 tid 2 2024-04-22T18:39:10.787 INFO:tasks.rados.rados.0.smithi089.stdout:694: writing smithi08937929-31 from 2633066 to 3243718 tid 3 2024-04-22T18:39:10.787 INFO:tasks.rados.rados.0.smithi089.stdout:695: read oid 7 snap -1 2024-04-22T18:39:10.787 INFO:tasks.rados.rados.0.smithi089.stdout:695: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-22T18:39:10.787 INFO:tasks.rados.rados.0.smithi089.stdout:696: read oid 47 snap -1 2024-04-22T18:39:10.787 INFO:tasks.rados.rados.0.smithi089.stdout:696: expect (ObjNum 68 snap 0 seq_num 68) 2024-04-22T18:39:10.787 INFO:tasks.rados.rados.0.smithi089.stdout:697: read oid 46 snap -1 2024-04-22T18:39:10.787 INFO:tasks.rados.rados.0.smithi089.stdout:697: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-22T18:39:10.787 INFO:tasks.rados.rados.0.smithi089.stdout:698: read oid 39 snap -1 2024-04-22T18:39:10.787 INFO:tasks.rados.rados.0.smithi089.stdout:698: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-22T18:39:10.787 INFO:tasks.rados.rados.0.smithi089.stdout:699: write oid 25 current snap is 0 2024-04-22T18:39:10.787 INFO:tasks.rados.rados.0.smithi089.stdout:699: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2024-04-22T18:39:10.793 INFO:tasks.rados.rados.0.smithi089.stdout:699: writing smithi08937929-25 from 460525 to 1151377 tid 1 2024-04-22T18:39:10.800 INFO:tasks.rados.rados.0.smithi089.stdout:699: writing smithi08937929-25 from 1622620 to 2352972 tid 2 2024-04-22T18:39:10.803 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:10.804+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:10.805 INFO:tasks.rados.rados.0.smithi089.stdout:699: writing smithi08937929-25 from 3060489 to 3260525 tid 3 2024-04-22T18:39:10.805 INFO:tasks.rados.rados.0.smithi089.stdout:700: read oid 26 snap -1 2024-04-22T18:39:10.806 INFO:tasks.rados.rados.0.smithi089.stdout:700: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-22T18:39:10.806 INFO:tasks.rados.rados.0.smithi089.stdout:693: done (13 left) 2024-04-22T18:39:10.806 INFO:tasks.rados.rados.0.smithi089.stdout:701: read oid 49 snap -1 2024-04-22T18:39:10.806 INFO:tasks.rados.rados.0.smithi089.stdout:701: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-22T18:39:10.806 INFO:tasks.rados.rados.0.smithi089.stdout:702: set_chunk oid 6 target oid 2024-04-22T18:39:10.830 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:39:10.831 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:39:10.842 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2024-04-22T18:39:10.182882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":235193764,"num_objects":116,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":575,"num_read_kb":182513,"num_write":638,"num_write_kb":194969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":44,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":645,"ondisk_log_size":645,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":446656,"kb_used_data":222520,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754528064,"statfs":{"total":773094113280,"available":772636737536,"internally_reserved":0,"allocated":227860480,"data_stored":236530058,"data_compressed":8701134,"data_compressed_allocated":9297920,"data_compressed_original":20545536,"omap_allocated":12744,"internal_metadata":225431096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5252168,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":79648,"num_write":37,"num_write_kb":5607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030802"},"pg_stats":[{"pgid":"4.f","version":"31'16","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.554844+0000","last_change":"2024-04-22T18:38:48.173640+0000","last_active":"2024-04-22T18:39:05.554844+0000","last_peered":"2024-04-22T18:39:05.554844+0000","last_clean":"2024-04-22T18:39:05.554844+0000","last_became_active":"2024-04-22T18:38:48.173369+0000","last_became_peered":"2024-04-22T18:38:48.173369+0000","last_unstale":"2024-04-22T18:39:05.554844+0000","last_undegraded":"2024-04-22T18:39:05.554844+0000","last_fullsized":"2024-04-22T18:39:05.554844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:08.819198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6184877,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":8689,"num_write":12,"num_write_kb":2771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"33'10","reported_seq":36,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.162285+0000","last_change":"2024-04-22T18:38:48.075222+0000","last_active":"2024-04-22T18:39:04.162285+0000","last_peered":"2024-04-22T18:39:04.162285+0000","last_clean":"2024-04-22T18:39:04.162285+0000","last_became_active":"2024-04-22T18:38:48.074901+0000","last_became_peered":"2024-04-22T18:38:48.074901+0000","last_unstale":"2024-04-22T18:39:04.162285+0000","last_undegraded":"2024-04-22T18:39:04.162285+0000","last_fullsized":"2024-04-22T18:39:04.162285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.172332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756315,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":2372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"32'12","reported_seq":44,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.544121+0000","last_change":"2024-04-22T18:38:48.172087+0000","last_active":"2024-04-22T18:39:04.544121+0000","last_peered":"2024-04-22T18:39:04.544121+0000","last_clean":"2024-04-22T18:39:04.544121+0000","last_became_active":"2024-04-22T18:38:48.171797+0000","last_became_peered":"2024-04-22T18:38:48.171797+0000","last_unstale":"2024-04-22T18:39:04.544121+0000","last_undegraded":"2024-04-22T18:39:04.544121+0000","last_fullsized":"2024-04-22T18:39:04.544121+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.134124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":19156,"num_write":8,"num_write_kb":1802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"33'26","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.897206+0000","last_change":"2024-04-22T18:38:48.274538+0000","last_active":"2024-04-22T18:39:05.897206+0000","last_peered":"2024-04-22T18:39:05.897206+0000","last_clean":"2024-04-22T18:39:05.897206+0000","last_became_active":"2024-04-22T18:38:48.174204+0000","last_became_peered":"2024-04-22T18:38:48.174204+0000","last_unstale":"2024-04-22T18:39:05.897206+0000","last_undegraded":"2024-04-22T18:39:05.897206+0000","last_fullsized":"2024-04-22T18:39:05.897206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:11.823691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10437015,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":30,"num_read_kb":23261,"num_write":20,"num_write_kb":5534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"35'36","reported_seq":78,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.892883+0000","last_change":"2024-04-22T18:38:48.269362+0000","last_active":"2024-04-22T18:39:05.892883+0000","last_peered":"2024-04-22T18:39:05.892883+0000","last_clean":"2024-04-22T18:39:05.892883+0000","last_became_active":"2024-04-22T18:38:48.269066+0000","last_became_peered":"2024-04-22T18:38:48.269066+0000","last_unstale":"2024-04-22T18:39:05.892883+0000","last_undegraded":"2024-04-22T18:39:05.892883+0000","last_fullsized":"2024-04-22T18:39:05.892883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:52.098608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13142909,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":24714,"num_write":30,"num_write_kb":8426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"35'19","reported_seq":53,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.253252+0000","last_change":"2024-04-22T18:38:48.372952+0000","last_active":"2024-04-22T18:39:05.253252+0000","last_peered":"2024-04-22T18:39:05.253252+0000","last_clean":"2024-04-22T18:39:05.253252+0000","last_became_active":"2024-04-22T18:38:48.277922+0000","last_became_peered":"2024-04-22T18:38:48.277922+0000","last_unstale":"2024-04-22T18:39:05.253252+0000","last_undegraded":"2024-04-22T18:39:05.253252+0000","last_fullsized":"2024-04-22T18:39:05.253252+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:15.471951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7092455,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":3383,"num_write":17,"num_write_kb":4007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"35'34","reported_seq":71,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:06.170529+0000","last_change":"2024-04-22T18:38:48.271633+0000","last_active":"2024-04-22T18:39:06.170529+0000","last_peered":"2024-04-22T18:39:06.170529+0000","last_clean":"2024-04-22T18:39:06.170529+0000","last_became_active":"2024-04-22T18:38:48.171245+0000","last_became_peered":"2024-04-22T18:38:48.171245+0000","last_unstale":"2024-04-22T18:39:06.170529+0000","last_undegraded":"2024-04-22T18:39:06.170529+0000","last_fullsized":"2024-04-22T18:39:06.170529+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:47.449912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12537857,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":15230,"num_write":25,"num_write_kb":6740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"34'19","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.901689+0000","last_change":"2024-04-22T18:38:48.149226+0000","last_active":"2024-04-22T18:39:04.901689+0000","last_peered":"2024-04-22T18:39:04.901689+0000","last_clean":"2024-04-22T18:39:04.901689+0000","last_became_active":"2024-04-22T18:38:48.148955+0000","last_became_peered":"2024-04-22T18:38:48.148955+0000","last_unstale":"2024-04-22T18:39:04.901689+0000","last_undegraded":"2024-04-22T18:39:04.901689+0000","last_fullsized":"2024-04-22T18:39:04.901689+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:02.394854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7882384,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":11281,"num_write":17,"num_write_kb":4337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"34'19","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.895680+0000","last_change":"2024-04-22T18:38:48.277905+0000","last_active":"2024-04-22T18:39:05.895680+0000","last_peered":"2024-04-22T18:39:05.895680+0000","last_clean":"2024-04-22T18:39:05.895680+0000","last_became_active":"2024-04-22T18:38:48.277640+0000","last_became_peered":"2024-04-22T18:38:48.277640+0000","last_unstale":"2024-04-22T18:39:05.895680+0000","last_undegraded":"2024-04-22T18:39:05.895680+0000","last_fullsized":"2024-04-22T18:39:05.895680+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:57.153161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5831963,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":16784,"num_write":17,"num_write_kb":4628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":178,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.240945+0000","last_change":"2024-04-22T18:38:38.109358+0000","last_active":"2024-04-22T18:39:04.240945+0000","last_peered":"2024-04-22T18:39:04.240945+0000","last_clean":"2024-04-22T18:39:04.240945+0000","last_became_active":"2024-04-22T18:38:38.108568+0000","last_became_peered":"2024-04-22T18:38:38.108568+0000","last_unstale":"2024-04-22T18:39:04.240945+0000","last_undegraded":"2024-04-22T18:39:04.240945+0000","last_fullsized":"2024-04-22T18:39:04.240945+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:38:36.991764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:46.436569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.241963+0000","last_change":"2024-04-22T18:38:39.010398+0000","last_active":"2024-04-22T18:39:04.241963+0000","last_peered":"2024-04-22T18:39:04.241963+0000","last_clean":"2024-04-22T18:39:04.241963+0000","last_became_active":"2024-04-22T18:38:39.010101+0000","last_became_peered":"2024-04-22T18:38:39.010101+0000","last_unstale":"2024-04-22T18:39:04.241963+0000","last_undegraded":"2024-04-22T18:39:04.241963+0000","last_fullsized":"2024-04-22T18:39:04.241963+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.708134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"34'42","reported_seq":76,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.988839+0000","last_change":"2024-04-22T18:38:58.429682+0000","last_active":"2024-04-22T18:39:04.988839+0000","last_peered":"2024-04-22T18:39:04.988839+0000","last_clean":"2024-04-22T18:39:04.988839+0000","last_became_active":"2024-04-22T18:38:58.125648+0000","last_became_peered":"2024-04-22T18:38:58.125648+0000","last_unstale":"2024-04-22T18:39:04.988839+0000","last_undegraded":"2024-04-22T18:39:04.988839+0000","last_fullsized":"2024-04-22T18:39:04.988839+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:52.346195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28872355,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":30,"num_read_kb":17,"num_write":44,"num_write_kb":28201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"32'19","reported_seq":54,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.960464+0000","last_change":"2024-04-22T18:38:48.173369+0000","last_active":"2024-04-22T18:39:05.960464+0000","last_peered":"2024-04-22T18:39:05.960464+0000","last_clean":"2024-04-22T18:39:05.960464+0000","last_became_active":"2024-04-22T18:38:48.173090+0000","last_became_peered":"2024-04-22T18:38:48.173090+0000","last_unstale":"2024-04-22T18:39:05.960464+0000","last_undegraded":"2024-04-22T18:39:05.960464+0000","last_fullsized":"2024-04-22T18:39:05.960464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:05.687904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8100875,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":14100,"num_write":17,"num_write_kb":5390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.140702+0000","last_change":"2024-04-22T18:38:39.125885+0000","last_active":"2024-04-22T18:39:04.140702+0000","last_peered":"2024-04-22T18:39:04.140702+0000","last_clean":"2024-04-22T18:39:04.140702+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:39:04.140702+0000","last_undegraded":"2024-04-22T18:39:04.140702+0000","last_fullsized":"2024-04-22T18:39:04.140702+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.321452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"35'60","reported_seq":84,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.454484+0000","last_change":"2024-04-22T18:38:47.173337+0000","last_active":"2024-04-22T18:39:05.454484+0000","last_peered":"2024-04-22T18:39:05.454484+0000","last_clean":"2024-04-22T18:39:05.454484+0000","last_became_active":"2024-04-22T18:38:47.173065+0000","last_became_peered":"2024-04-22T18:38:47.173065+0000","last_unstale":"2024-04-22T18:39:05.454484+0000","last_undegraded":"2024-04-22T18:39:05.454484+0000","last_fullsized":"2024-04-22T18:39:05.454484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:52.987154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25419012,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":47,"num_read_kb":30,"num_write":64,"num_write_kb":31463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"35'16","reported_seq":47,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.354610+0000","last_change":"2024-04-22T18:38:48.278028+0000","last_active":"2024-04-22T18:39:05.354610+0000","last_peered":"2024-04-22T18:39:05.354610+0000","last_clean":"2024-04-22T18:39:05.354610+0000","last_became_active":"2024-04-22T18:38:48.277802+0000","last_became_peered":"2024-04-22T18:38:48.277802+0000","last_unstale":"2024-04-22T18:39:05.354610+0000","last_undegraded":"2024-04-22T18:39:05.354610+0000","last_fullsized":"2024-04-22T18:39:05.354610+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:12.000401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5428595,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":5714,"num_write":13,"num_write_kb":3557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.367651+0000","last_change":"2024-04-22T18:38:39.164369+0000","last_active":"2024-04-22T18:39:04.367651+0000","last_peered":"2024-04-22T18:39:04.367651+0000","last_clean":"2024-04-22T18:39:04.367651+0000","last_became_active":"2024-04-22T18:38:39.164089+0000","last_became_peered":"2024-04-22T18:38:39.164089+0000","last_unstale":"2024-04-22T18:39:04.367651+0000","last_undegraded":"2024-04-22T18:39:04.367651+0000","last_fullsized":"2024-04-22T18:39:04.367651+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:44.825470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"34'40","reported_seq":63,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.002781+0000","last_change":"2024-04-22T18:38:47.172071+0000","last_active":"2024-04-22T18:39:05.002781+0000","last_peered":"2024-04-22T18:39:05.002781+0000","last_clean":"2024-04-22T18:39:05.002781+0000","last_became_active":"2024-04-22T18:38:47.171789+0000","last_became_peered":"2024-04-22T18:38:47.171789+0000","last_unstale":"2024-04-22T18:39:05.002781+0000","last_undegraded":"2024-04-22T18:39:05.002781+0000","last_fullsized":"2024-04-22T18:39:05.002781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:47.501470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24706578,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":31,"num_read_kb":14,"num_write":48,"num_write_kb":24135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"29'4","reported_seq":31,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.793972+0000","last_change":"2024-04-22T18:38:48.171024+0000","last_active":"2024-04-22T18:39:05.793972+0000","last_peered":"2024-04-22T18:39:05.793972+0000","last_clean":"2024-04-22T18:39:05.793972+0000","last_became_active":"2024-04-22T18:38:48.170789+0000","last_became_peered":"2024-04-22T18:38:48.170789+0000","last_unstale":"2024-04-22T18:39:05.793972+0000","last_undegraded":"2024-04-22T18:39:05.793972+0000","last_fullsized":"2024-04-22T18:39:05.793972+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:06.487306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2689087,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11735,"num_write":4,"num_write_kb":1223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.241989+0000","last_change":"2024-04-22T18:38:39.111223+0000","last_active":"2024-04-22T18:39:04.241989+0000","last_peered":"2024-04-22T18:39:04.241989+0000","last_clean":"2024-04-22T18:39:04.241989+0000","last_became_active":"2024-04-22T18:38:39.110914+0000","last_became_peered":"2024-04-22T18:38:39.110914+0000","last_unstale":"2024-04-22T18:39:04.241989+0000","last_undegraded":"2024-04-22T18:39:04.241989+0000","last_fullsized":"2024-04-22T18:39:04.241989+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.570327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"35'74","reported_seq":97,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T18:39:05.555502+0000","last_change":"2024-04-22T18:38:47.173101+0000","last_active":"2024-04-22T18:39:05.555502+0000","last_peered":"2024-04-22T18:39:05.555502+0000","last_clean":"2024-04-22T18:39:05.555502+0000","last_became_active":"2024-04-22T18:38:47.172816+0000","last_became_peered":"2024-04-22T18:38:47.172816+0000","last_unstale":"2024-04-22T18:39:05.555502+0000","last_undegraded":"2024-04-22T18:39:05.555502+0000","last_fullsized":"2024-04-22T18:39:05.555502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:12.128375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36042037,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":54,"num_read_kb":32,"num_write":78,"num_write_kb":42658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"32'13","reported_seq":42,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.222402+0000","last_change":"2024-04-22T18:38:48.372570+0000","last_active":"2024-04-22T18:39:04.222402+0000","last_peered":"2024-04-22T18:39:04.222402+0000","last_clean":"2024-04-22T18:39:04.222402+0000","last_became_active":"2024-04-22T18:38:48.272283+0000","last_became_peered":"2024-04-22T18:38:48.272283+0000","last_unstale":"2024-04-22T18:39:04.222402+0000","last_undegraded":"2024-04-22T18:39:04.222402+0000","last_fullsized":"2024-04-22T18:39:04.222402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:09.183390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4319333,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":6262,"num_write":8,"num_write_kb":2087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.222497+0000","last_change":"2024-04-22T18:38:39.307735+0000","last_active":"2024-04-22T18:39:04.222497+0000","last_peered":"2024-04-22T18:39:04.222497+0000","last_clean":"2024-04-22T18:39:04.222497+0000","last_became_active":"2024-04-22T18:38:39.307459+0000","last_became_peered":"2024-04-22T18:38:39.307459+0000","last_unstale":"2024-04-22T18:39:04.222497+0000","last_undegraded":"2024-04-22T18:39:04.222497+0000","last_fullsized":"2024-04-22T18:39:04.222497+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.680561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"32'24","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.143455+0000","last_change":"2024-04-22T18:38:48.269379+0000","last_active":"2024-04-22T18:39:04.143455+0000","last_peered":"2024-04-22T18:39:04.143455+0000","last_clean":"2024-04-22T18:39:04.143455+0000","last_became_active":"2024-04-22T18:38:48.269070+0000","last_became_peered":"2024-04-22T18:38:48.269070+0000","last_unstale":"2024-04-22T18:39:04.143455+0000","last_undegraded":"2024-04-22T18:39:04.143455+0000","last_fullsized":"2024-04-22T18:39:04.143455+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:08.651956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9176490,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.140674+0000","last_change":"2024-04-22T18:38:39.210344+0000","last_active":"2024-04-22T18:39:04.140674+0000","last_peered":"2024-04-22T18:39:04.140674+0000","last_clean":"2024-04-22T18:39:04.140674+0000","last_became_active":"2024-04-22T18:38:39.210049+0000","last_became_peered":"2024-04-22T18:38:39.210049+0000","last_unstale":"2024-04-22T18:39:04.140674+0000","last_undegraded":"2024-04-22T18:39:04.140674+0000","last_fullsized":"2024-04-22T18:39:04.140674+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.878594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"31'5","reported_seq":30,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.441934+0000","last_change":"2024-04-22T18:38:48.349979+0000","last_active":"2024-04-22T18:39:04.441934+0000","last_peered":"2024-04-22T18:39:04.441934+0000","last_clean":"2024-04-22T18:39:04.441934+0000","last_became_active":"2024-04-22T18:38:48.349706+0000","last_became_peered":"2024-04-22T18:38:48.349706+0000","last_unstale":"2024-04-22T18:39:04.441934+0000","last_undegraded":"2024-04-22T18:39:04.441934+0000","last_fullsized":"2024-04-22T18:39:04.441934+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:54.435973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2319333,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.222414+0000","last_change":"2024-04-22T18:38:39.110499+0000","last_active":"2024-04-22T18:39:04.222414+0000","last_peered":"2024-04-22T18:39:04.222414+0000","last_clean":"2024-04-22T18:39:04.222414+0000","last_became_active":"2024-04-22T18:38:39.010173+0000","last_became_peered":"2024-04-22T18:38:39.010173+0000","last_unstale":"2024-04-22T18:39:04.222414+0000","last_undegraded":"2024-04-22T18:39:04.222414+0000","last_fullsized":"2024-04-22T18:39:04.222414+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:58.222501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"32'40","reported_seq":82,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.162291+0000","last_change":"2024-04-22T18:38:48.372226+0000","last_active":"2024-04-22T18:39:04.162291+0000","last_peered":"2024-04-22T18:39:04.162291+0000","last_clean":"2024-04-22T18:39:04.162291+0000","last_became_active":"2024-04-22T18:38:48.371943+0000","last_became_peered":"2024-04-22T18:38:48.371943+0000","last_unstale":"2024-04-22T18:39:04.162291+0000","last_undegraded":"2024-04-22T18:39:04.162291+0000","last_fullsized":"2024-04-22T18:39:04.162291+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.637563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17075681,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":22004,"num_write":28,"num_write_kb":8178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":28,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T18:39:04.423376+0000","last_change":"2024-04-22T18:38:39.026078+0000","last_active":"2024-04-22T18:39:04.423376+0000","last_peered":"2024-04-22T18:39:04.423376+0000","last_clean":"2024-04-22T18:39:04.423376+0000","last_became_active":"2024-04-22T18:38:39.025741+0000","last_became_peered":"2024-04-22T18:38:39.025741+0000","last_unstale":"2024-04-22T18:39:04.423376+0000","last_undegraded":"2024-04-22T18:39:04.423376+0000","last_fullsized":"2024-04-22T18:39:04.423376+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:01.125349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":119563414,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":297,"num_read_kb":182320,"num_write":245,"num_write_kb":66096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":44,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":116834304,"data_stored":115997104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":312,"ondisk_log_size":312,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":115039982,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":162,"num_read_kb":93,"num_write":234,"num_write_kb":126457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":108388352,"data_stored":118340092,"data_compressed":8386446,"data_compressed_allocated":8937472,"data_compressed_original":19062784,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":4964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":5083136,"data_stored":5295281,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.629}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34536,"kb_used_data":7508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337304,"statfs":{"total":96636764160,"available":96601399296,"internally_reserved":0,"allocated":7688192,"data_stored":7422224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62552,"kb_used_data":34708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94309288,"statfs":{"total":96636764160,"available":96572710912,"internally_reserved":0,"allocated":35540992,"data_stored":36611552,"data_compressed":1119849,"data_compressed_allocated":1204224,"data_compressed_original":2621440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":69.058999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80420,"kb_used_data":53120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94291420,"statfs":{"total":96636764160,"available":96554414080,"internally_reserved":0,"allocated":54394880,"data_stored":56725474,"data_compressed":2268767,"data_compressed_allocated":2412544,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49160,"kb_used_data":21852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94322680,"statfs":{"total":96636764160,"available":96586424320,"internally_reserved":0,"allocated":22376448,"data_stored":23499004,"data_compressed":1119849,"data_compressed_allocated":1204224,"data_compressed_original":2621440,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":94.063000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41824,"kb_used_data":14796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330016,"statfs":{"total":96636764160,"available":96593936384,"internally_reserved":0,"allocated":15151104,"data_stored":15950902,"data_compressed":932383,"data_compressed_allocated":991232,"data_compressed_original":2015232,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69828,"kb_used_data":40964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94302012,"statfs":{"total":96636764160,"available":96565260288,"internally_reserved":0,"allocated":41947136,"data_stored":42521455,"data_compressed":932383,"data_compressed_allocated":991232,"data_compressed_original":2015232,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.17899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75728,"kb_used_data":44608,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94296112,"statfs":{"total":96636764160,"available":96559218688,"internally_reserved":0,"allocated":45678592,"data_stored":48504166,"data_compressed":2298335,"data_compressed_allocated":2453504,"data_compressed_original":5636096,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29585408,"data_stored":32190954,"data_compressed":2208471,"data_compressed_allocated":2342912,"data_compressed_original":4993024,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11710464,"data_stored":12715899,"data_compressed":932383,"data_compressed_allocated":991232,"data_compressed_original":2015232,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11710464,"data_stored":12715899,"data_compressed":932383,"data_compressed_allocated":991232,"data_compressed_original":2015232,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13479936,"data_stored":14872515,"data_compressed":1119849,"data_compressed_allocated":1204224,"data_compressed_original":2621440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28422144,"data_stored":30972310,"data_compressed":2073511,"data_compressed_allocated":2203648,"data_compressed_original":4796416,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":13479936,"data_stored":14872515,"data_compressed":1119849,"data_compressed_allocated":1204224,"data_compressed_original":2621440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":15925248,"data_stored":15823002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29888512,"data_stored":29663535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3121152,"data_stored":3095407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8552448,"data_stored":8484022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25604096,"data_stored":25424431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":21757952,"data_stored":21605385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":7323648,"data_stored":7278475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4661248,"data_stored":4622847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:39:10.844 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-22T18:39:10.845 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:39:10.845 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-04-22T18:39:10.845 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-22T18:39:10.845 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-22T18:39:10.845 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-04-22T18:39:10.877 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:10.877 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:10.886 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:10.928 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:10.928+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:10.981 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:10.981 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-22T18:39:10.981 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:11.004 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:11.005+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:11.032 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:11.032 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:11.041 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:11.105 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:11.105+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:11.119 INFO:tasks.rados.rados.0.smithi089.stdout:702: set_chunk oid 6 offset: 69120 length: 1892352 target oid 13bef174a3b1ea9081537f731d20447b852f7154bd4382dfcc23dac8b77fa5a4 offset: 0 snap -1 2024-04-22T18:39:11.119 INFO:tasks.rados.rados.0.smithi089.stdout:703: write oid 36 current snap is 0 2024-04-22T18:39:11.119 INFO:tasks.rados.rados.0.smithi089.stdout:692: expect (ObjNum 69 snap 0 seq_num 69) 2024-04-22T18:39:11.149 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:11.149 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:11.158 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:11.187 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:39:11.241 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:11.242+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:11.263 INFO:tasks.rados.rados.0.smithi089.stdout:703: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2024-04-22T18:39:11.270 INFO:tasks.rados.rados.0.smithi089.stdout:703: writing smithi08937929-36 from 477332 to 1243433 tid 1 2024-04-22T18:39:11.275 INFO:tasks.rados.rados.0.smithi089.stdout:703: writing smithi08937929-36 from 1681102 to 2155112 tid 2 2024-04-22T18:39:11.281 INFO:tasks.rados.rados.0.smithi089.stdout:703: writing smithi08937929-36 from 2687912 to 3277332 tid 3 2024-04-22T18:39:11.281 INFO:tasks.rados.rados.0.smithi089.stdout:692: done (15 left) 2024-04-22T18:39:11.281 INFO:tasks.rados.rados.0.smithi089.stdout:704: write oid 17 current snap is 0 2024-04-22T18:39:11.281 INFO:tasks.rados.rados.0.smithi089.stdout:704: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2024-04-22T18:39:11.286 INFO:tasks.rados.rados.0.smithi089.stdout:704: writing smithi08937929-17 from 494139 to 1051842 tid 1 2024-04-22T18:39:11.290 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:11.290 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:11.292 INFO:tasks.rados.rados.0.smithi089.stdout:704: writing smithi08937929-17 from 1455937 to 2189958 tid 2 2024-04-22T18:39:11.299 INFO:tasks.rados.rados.0.smithi089.stdout:704: writing smithi08937929-17 from 2948041 to 3294139 tid 3 2024-04-22T18:39:11.299 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:11.299 INFO:tasks.rados.rados.0.smithi089.stdout:696: expect (ObjNum 68 snap 0 seq_num 68) 2024-04-22T18:39:11.299 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:39:11.301 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:11.343 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:11.344+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:11.407 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:11.408+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:11.432 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:11.432 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:11.448 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:11.448 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:11.450 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:11.450 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:11.457 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:11.458 INFO:tasks.rados.rados.0.smithi089.stdout:699: finishing write tid 1 to smithi08937929-25 2024-04-22T18:39:11.458 INFO:tasks.rados.rados.0.smithi089.stdout:699: finishing write tid 2 to smithi08937929-25 2024-04-22T18:39:11.458 INFO:tasks.rados.rados.0.smithi089.stdout:686: finishing write tid 1 to smithi08937929-16 2024-04-22T18:39:11.458 INFO:tasks.rados.rados.0.smithi089.stdout:686: finishing write tid 2 to smithi08937929-16 2024-04-22T18:39:11.458 INFO:tasks.rados.rados.0.smithi089.stdout:696: done (15 left) 2024-04-22T18:39:11.458 INFO:tasks.rados.rados.0.smithi089.stdout:705: set_chunk oid 37 target oid 2024-04-22T18:39:11.467 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:11.467 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:11.484 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:11.484 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:11.502 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:11.502 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:11.508 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:11.509+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:11.519 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:11.519 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:11.536 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:11.536 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:11.552 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:11.553 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:11.569 INFO:tasks.rados.rados.0.smithi089.stdout:705: set_chunk oid 37 offset: 185344 length: 2148864 target oid 0eed8da4d38282dc8c76ab4a943555eb43f8a195ca871dec5e4bb51eb5eaa2af offset: 0 snap -1 2024-04-22T18:39:11.569 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:11.569 INFO:tasks.rados.rados.0.smithi089.stdout:694: finishing write tid 1 to smithi08937929-31 2024-04-22T18:39:11.570 INFO:tasks.rados.rados.0.smithi089.stdout:701: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-22T18:39:11.574 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:11.574 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:11.574 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:11.574 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:11.574 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:11.585 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:11.627 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:11.628+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:11.666 INFO:tasks.rados.rados.0.smithi089.stdout:686: finishing write tid 3 to smithi08937929-16 2024-04-22T18:39:11.666 INFO:tasks.rados.rados.0.smithi089.stdout:686: finishing write tid 4 to smithi08937929-16 2024-04-22T18:39:11.666 INFO:tasks.rados.rados.0.smithi089.stdout:686: oid 16 updating version 0 to 42 2024-04-22T18:39:11.667 INFO:tasks.rados.rados.0.smithi089.stdout:686: oid 16 updating version 42 to 44 2024-04-22T18:39:11.667 INFO:tasks.rados.rados.0.smithi089.stdout:686: oid 16 version 44 is already newer than 43 2024-04-22T18:39:11.667 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 44 (ObjNum 71 snap 0 seq_num 71) dirty exists 2024-04-22T18:39:11.667 INFO:tasks.rados.rados.0.smithi089.stdout:686: left oid 16 (ObjNum 71 snap 0 seq_num 71) 2024-04-22T18:39:11.667 INFO:tasks.rados.rados.0.smithi089.stdout:687: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-22T18:39:11.710 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:11.711+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:11.729 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:11.729 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:11.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:11.729 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:11.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.730 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:6 4.f e24/20 4:f83a159b:::smithi08937929-2:head v 24'2, mlcod=0'0)", 2024-04-22T18:39:11.730 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:11.730 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.522988779999999, 2024-04-22T18:39:11.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10183012700000001, 2024-04-22T18:39:11.730 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.731 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.731 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:11.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:11.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:11.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:11.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.206027+0000", 2024-04-22T18:39:11.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023705500000000001 2024-04-22T18:39:11.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.206038+0000", 2024-04-22T18:39:11.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.148e-05 2024-04-22T18:39:11.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.206046+0000", 2024-04-22T18:39:11.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8679999999999999e-06 2024-04-22T18:39:11.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.306227+0000", 2024-04-22T18:39:11.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100181223 2024-04-22T18:39:11.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.306266+0000", 2024-04-22T18:39:11.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8906000000000003e-05 2024-04-22T18:39:11.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.307598+0000", 2024-04-22T18:39:11.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001331561 2024-04-22T18:39:11.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.307620+0000", 2024-04-22T18:39:11.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.226e-05 2024-04-22T18:39:11.739 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.739 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.740 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.740 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:45 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'1, mlcod=0'0)", 2024-04-22T18:39:11.740 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:11.740 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.463667094000002, 2024-04-22T18:39:11.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14345418900000001, 2024-04-22T18:39:11.741 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.741 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.741 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:11.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:11.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:11.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:11.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265727+0000", 2024-04-22T18:39:11.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061502799999999999 2024-04-22T18:39:11.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265737+0000", 2024-04-22T18:39:11.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0278999999999999e-05 2024-04-22T18:39:11.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265744+0000", 2024-04-22T18:39:11.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1740000000000003e-06 2024-04-22T18:39:11.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407227+0000", 2024-04-22T18:39:11.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14148266900000001 2024-04-22T18:39:11.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407258+0000", 2024-04-22T18:39:11.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1619000000000003e-05 2024-04-22T18:39:11.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408534+0000", 2024-04-22T18:39:11.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001275829 2024-04-22T18:39:11.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408566+0000", 2024-04-22T18:39:11.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1813999999999997e-05 2024-04-22T18:39:11.750 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.750 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.750 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.751 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:46 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'2, mlcod=0'0)", 2024-04-22T18:39:11.751 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:11.751 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.461042690999999, 2024-04-22T18:39:11.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14304645699999999, 2024-04-22T18:39:11.752 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.752 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.752 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:11.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:11.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:11.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:11.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268301+0000", 2024-04-22T18:39:11.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056585700000000001 2024-04-22T18:39:11.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268309+0000", 2024-04-22T18:39:11.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2830000000000004e-06 2024-04-22T18:39:11.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268312+0000", 2024-04-22T18:39:11.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7179999999999998e-06 2024-04-22T18:39:11.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407999+0000", 2024-04-22T18:39:11.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13968636300000001 2024-04-22T18:39:11.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408011+0000", 2024-04-22T18:39:11.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2116999999999999e-05 2024-04-22T18:39:11.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410762+0000", 2024-04-22T18:39:11.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002751524 2024-04-22T18:39:11.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410783+0000", 2024-04-22T18:39:11.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0270000000000001e-05 2024-04-22T18:39:11.761 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.761 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.761 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.762 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:47 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'3, mlcod=0'0)", 2024-04-22T18:39:11.762 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:11.762 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.459310121000001, 2024-04-22T18:39:11.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14132698599999999, 2024-04-22T18:39:11.763 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.763 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.763 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:11.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:11.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269468+0000", 2024-04-22T18:39:11.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:11.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269473+0000", 2024-04-22T18:39:11.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7790000000000003e-06 2024-04-22T18:39:11.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269478+0000", 2024-04-22T18:39:11.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0080000000000002e-06 2024-04-22T18:39:11.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269489+0000", 2024-04-22T18:39:11.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.075e-05 2024-04-22T18:39:11.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408595+0000", 2024-04-22T18:39:11.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13910662300000001 2024-04-22T18:39:11.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408611+0000", 2024-04-22T18:39:11.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6124e-05 2024-04-22T18:39:11.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410786+0000", 2024-04-22T18:39:11.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021743819999999999 2024-04-22T18:39:11.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410796+0000", 2024-04-22T18:39:11.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.007e-05 2024-04-22T18:39:11.771 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.771 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.772 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.772 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:9 4.f e27/20 4:f2c48a0e:::smithi08937929-3:head v 27'4, mlcod=24'2)", 2024-04-22T18:39:11.772 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.441905+0000", 2024-04-22T18:39:11.772 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.286874332, 2024-04-22T18:39:11.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10138892200000001, 2024-04-22T18:39:11.773 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.773 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.773 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.441905+0000", 2024-04-22T18:39:11.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.441905+0000", 2024-04-22T18:39:11.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.441902+0000", 2024-04-22T18:39:11.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:11.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442053+0000", 2024-04-22T18:39:11.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015104800000000001 2024-04-22T18:39:11.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442067+0000", 2024-04-22T18:39:11.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3964e-05 2024-04-22T18:39:11.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442073+0000", 2024-04-22T18:39:11.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1249999999999998e-06 2024-04-22T18:39:11.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442111+0000", 2024-04-22T18:39:11.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7988000000000003e-05 2024-04-22T18:39:11.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442127+0000", 2024-04-22T18:39:11.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.662e-05 2024-04-22T18:39:11.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543270+0000", 2024-04-22T18:39:11.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101143 2024-04-22T18:39:11.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543293+0000", 2024-04-22T18:39:11.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3147e-05 2024-04-22T18:39:11.781 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.782 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.782 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.782 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:10 4.f e27/20 4:f2c48a0e:::smithi08937929-3:head v 27'5, mlcod=24'2)", 2024-04-22T18:39:11.783 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:11.783 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.286697551, 2024-04-22T18:39:11.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10161616800000001, 2024-04-22T18:39:11.783 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.783 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.783 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:11.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:11.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:11.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:11.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442159+0000", 2024-04-22T18:39:11.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7935000000000005e-05 2024-04-22T18:39:11.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442163+0000", 2024-04-22T18:39:11.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8430000000000003e-06 2024-04-22T18:39:11.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442166+0000", 2024-04-22T18:39:11.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0759999999999999e-06 2024-04-22T18:39:11.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542745+0000", 2024-04-22T18:39:11.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10057941200000001 2024-04-22T18:39:11.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542770+0000", 2024-04-22T18:39:11.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5134e-05 2024-04-22T18:39:11.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543679+0000", 2024-04-22T18:39:11.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00090833099999999996 2024-04-22T18:39:11.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543697+0000", 2024-04-22T18:39:11.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8876e-05 2024-04-22T18:39:11.790 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.790 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.790 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:11 4.f e27/20 4:f2c48a0e:::smithi08937929-3:head v 27'6, mlcod=24'2)", 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.442170+0000", 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.286609136999999, 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101722358, 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442170+0000", 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442170+0000", 2024-04-22T18:39:11.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442169+0000", 2024-04-22T18:39:11.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:11.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442174+0000", 2024-04-22T18:39:11.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0350000000000003e-06 2024-04-22T18:39:11.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442177+0000", 2024-04-22T18:39:11.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0580000000000002e-06 2024-04-22T18:39:11.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442181+0000", 2024-04-22T18:39:11.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1649999999999999e-06 2024-04-22T18:39:11.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543319+0000", 2024-04-22T18:39:11.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101138408 2024-04-22T18:39:11.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543336+0000", 2024-04-22T18:39:11.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6509999999999999e-05 2024-04-22T18:39:11.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543880+0000", 2024-04-22T18:39:11.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054429900000000004 2024-04-22T18:39:11.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543892+0000", 2024-04-22T18:39:11.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2138e-05 2024-04-22T18:39:11.794 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.794 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.794 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.795 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:77 4.0 4:01baf5a9:::smithi08937929-20:head [write 719333~462437,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:11.795 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.505878+0000", 2024-04-22T18:39:11.795 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.222901336, 2024-04-22T18:39:11.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10366277, 2024-04-22T18:39:11.795 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.795 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:11.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:11.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:11.795 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 77 2024-04-22T18:39:11.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.795 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.505878+0000", 2024-04-22T18:39:11.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.505878+0000", 2024-04-22T18:39:11.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.505876+0000", 2024-04-22T18:39:11.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:11.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506122+0000", 2024-04-22T18:39:11.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024626900000000002 2024-04-22T18:39:11.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506132+0000", 2024-04-22T18:39:11.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0216e-05 2024-04-22T18:39:11.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506139+0000", 2024-04-22T18:39:11.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6030000000000001e-06 2024-04-22T18:39:11.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506191+0000", 2024-04-22T18:39:11.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.185e-05 2024-04-22T18:39:11.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506333+0000", 2024-04-22T18:39:11.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014197299999999999 2024-04-22T18:39:11.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:11.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506480+0000", 2024-04-22T18:39:11.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014780199999999999 2024-04-22T18:39:11.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:11.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.507794+0000", 2024-04-22T18:39:11.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001313587 2024-04-22T18:39:11.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:11.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609429+0000", 2024-04-22T18:39:11.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101634658 2024-04-22T18:39:11.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609493+0000", 2024-04-22T18:39:11.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4777999999999995e-05 2024-04-22T18:39:11.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609540+0000", 2024-04-22T18:39:11.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6820000000000002e-05 2024-04-22T18:39:11.800 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.800 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.800 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.800 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:78 4.0 4:01baf5a9:::smithi08937929-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:11.800 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:11.800 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.216843999999998, 2024-04-22T18:39:11.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.398522192, 2024-04-22T18:39:11.801 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.801 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.801 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:11.801 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:11.801 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:11.801 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 78 2024-04-22T18:39:11.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.801 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:11.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:11.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511934+0000", 2024-04-22T18:39:11.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:11.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512012+0000", 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8329000000000002e-05 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512023+0000", 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0755e-05 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512028+0000", 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6470000000000003e-06 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512068+0000", 2024-04-22T18:39:11.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9721999999999997e-05 2024-04-22T18:39:11.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512125+0000", 2024-04-22T18:39:11.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6564999999999998e-05 2024-04-22T18:39:11.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:11.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512200+0000", 2024-04-22T18:39:11.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5708000000000001e-05 2024-04-22T18:39:11.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:11.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513131+0000", 2024-04-22T18:39:11.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093063 2024-04-22T18:39:11.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:11.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910394+0000", 2024-04-22T18:39:11.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.397263388 2024-04-22T18:39:11.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910430+0000", 2024-04-22T18:39:11.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5485999999999999e-05 2024-04-22T18:39:11.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910457+0000", 2024-04-22T18:39:11.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7086000000000001e-05 2024-04-22T18:39:11.806 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.806 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.806 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.806 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:79 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:11.806 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:11.806 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.216743344000001, 2024-04-22T18:39:11.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39855419399999997, 2024-04-22T18:39:11.806 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.806 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:11.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:11.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:11.807 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 79 2024-04-22T18:39:11.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.807 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:11.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:11.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512035+0000", 2024-04-22T18:39:11.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:11.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512045+0000", 2024-04-22T18:39:11.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0379e-05 2024-04-22T18:39:11.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512051+0000", 2024-04-22T18:39:11.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6030000000000004e-06 2024-04-22T18:39:11.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512057+0000", 2024-04-22T18:39:11.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7270000000000004e-06 2024-04-22T18:39:11.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512756+0000", 2024-04-22T18:39:11.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069954000000000001 2024-04-22T18:39:11.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512806+0000", 2024-04-22T18:39:11.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9922000000000001e-05 2024-04-22T18:39:11.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:11.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512927+0000", 2024-04-22T18:39:11.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012059 2024-04-22T18:39:11.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:11.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.515883+0000", 2024-04-22T18:39:11.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029563739999999999 2024-04-22T18:39:11.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:11.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910510+0000", 2024-04-22T18:39:11.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39462745199999999 2024-04-22T18:39:11.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910542+0000", 2024-04-22T18:39:11.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1225e-05 2024-04-22T18:39:11.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910590+0000", 2024-04-22T18:39:11.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8052e-05 2024-04-22T18:39:11.811 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.811 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.812 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:80 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:11.812 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:11.812 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.216718242999999, 2024-04-22T18:39:11.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39866376999999997, 2024-04-22T18:39:11.812 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.812 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:11.812 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:11.812 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:11.812 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:11.812 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 80 2024-04-22T18:39:11.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.812 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:11.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:11.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512060+0000", 2024-04-22T18:39:11.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:11.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512063+0000", 2024-04-22T18:39:11.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.999e-06 2024-04-22T18:39:11.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512069+0000", 2024-04-22T18:39:11.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.891e-06 2024-04-22T18:39:11.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512073+0000", 2024-04-22T18:39:11.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4429999999999996e-06 2024-04-22T18:39:11.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513084+0000", 2024-04-22T18:39:11.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010102609999999999 2024-04-22T18:39:11.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:11.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513123+0000", 2024-04-22T18:39:11.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9237999999999999e-05 2024-04-22T18:39:11.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910618+0000", 2024-04-22T18:39:11.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39749549299999998 2024-04-22T18:39:11.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910670+0000", 2024-04-22T18:39:11.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1221000000000003e-05 2024-04-22T18:39:11.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910724+0000", 2024-04-22T18:39:11.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4777000000000001e-05 2024-04-22T18:39:11.816 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.816 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.817 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.817 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:33 4.f e27/20 4:f0ed3d49:::smithi08937929-9:head v 27'7, mlcod=24'2)", 2024-04-22T18:39:11.817 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.609449+0000", 2024-04-22T18:39:11.817 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.119330308999999, 2024-04-22T18:39:11.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10183384600000001, 2024-04-22T18:39:11.817 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.817 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.817 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609449+0000", 2024-04-22T18:39:11.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609449+0000", 2024-04-22T18:39:11.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609448+0000", 2024-04-22T18:39:11.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:11.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609626+0000", 2024-04-22T18:39:11.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017809600000000001 2024-04-22T18:39:11.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609638+0000", 2024-04-22T18:39:11.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2045000000000001e-05 2024-04-22T18:39:11.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709771+0000", 2024-04-22T18:39:11.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013347 2024-04-22T18:39:11.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.710008+0000", 2024-04-22T18:39:11.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023696200000000001 2024-04-22T18:39:11.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.710045+0000", 2024-04-22T18:39:11.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6903000000000002e-05 2024-04-22T18:39:11.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.711263+0000", 2024-04-22T18:39:11.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012178110000000001 2024-04-22T18:39:11.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.711282+0000", 2024-04-22T18:39:11.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9525e-05 2024-04-22T18:39:11.821 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.821 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.821 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.821 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:34 4.f e27/20 4:f0ed3d49:::smithi08937929-9:head v 27'8, mlcod=24'2)", 2024-04-22T18:39:11.821 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.709796+0000", 2024-04-22T18:39:11.821 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.018983307999999, 2024-04-22T18:39:11.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10102066, 2024-04-22T18:39:11.821 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.821 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.821 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709796+0000", 2024-04-22T18:39:11.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709796+0000", 2024-04-22T18:39:11.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709793+0000", 2024-04-22T18:39:11.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709914+0000", 2024-04-22T18:39:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000120864 2024-04-22T18:39:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709932+0000", 2024-04-22T18:39:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8125999999999999e-05 2024-04-22T18:39:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709949+0000", 2024-04-22T18:39:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6480000000000001e-05 2024-04-22T18:39:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.809972+0000", 2024-04-22T18:39:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10002336000000001 2024-04-22T18:39:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.809986+0000", 2024-04-22T18:39:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3913e-05 2024-04-22T18:39:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810789+0000", 2024-04-22T18:39:11.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00080341899999999999 2024-04-22T18:39:11.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810816+0000", 2024-04-22T18:39:11.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6880999999999999e-05 2024-04-22T18:39:11.825 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.825 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.825 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.825 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:35 4.f e27/20 4:f0ed3d49:::smithi08937929-9:head v 27'9, mlcod=24'2)", 2024-04-22T18:39:11.825 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.709964+0000", 2024-04-22T18:39:11.825 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.018814628000001, 2024-04-22T18:39:11.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101040596, 2024-04-22T18:39:11.826 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.826 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.826 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709964+0000", 2024-04-22T18:39:11.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709964+0000", 2024-04-22T18:39:11.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709963+0000", 2024-04-22T18:39:11.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:11.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709975+0000", 2024-04-22T18:39:11.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.236e-05 2024-04-22T18:39:11.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709990+0000", 2024-04-22T18:39:11.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4296e-05 2024-04-22T18:39:11.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810061+0000", 2024-04-22T18:39:11.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10007135 2024-04-22T18:39:11.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810402+0000", 2024-04-22T18:39:11.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00034100299999999999 2024-04-22T18:39:11.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810422+0000", 2024-04-22T18:39:11.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9749e-05 2024-04-22T18:39:11.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810973+0000", 2024-04-22T18:39:11.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00055107799999999998 2024-04-22T18:39:11.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.811005+0000", 2024-04-22T18:39:11.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079999999999998e-05 2024-04-22T18:39:11.829 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.829 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.829 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.830 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:5 3.2 e30/29 3:663b82a1:::smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=0'0)", 2024-04-22T18:39:11.830 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:11.830 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.501278924999999, 2024-04-22T18:39:11.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20090134500000001, 2024-04-22T18:39:11.830 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.830 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.830 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:11.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:11.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227499+0000", 2024-04-22T18:39:11.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:11.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227506+0000", 2024-04-22T18:39:11.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6540000000000002e-06 2024-04-22T18:39:11.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227516+0000", 2024-04-22T18:39:11.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0054000000000001e-05 2024-04-22T18:39:11.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327607+0000", 2024-04-22T18:39:11.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009167099999999 2024-04-22T18:39:11.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327688+0000", 2024-04-22T18:39:11.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1000000000000004e-05 2024-04-22T18:39:11.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327719+0000", 2024-04-22T18:39:11.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0664000000000003e-05 2024-04-22T18:39:11.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428371+0000", 2024-04-22T18:39:11.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10065239099999999 2024-04-22T18:39:11.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428401+0000", 2024-04-22T18:39:11.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9839e-05 2024-04-22T18:39:11.833 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.833 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.834 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.834 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:19 3.2 e30/29 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'24, mlcod=28'19)", 2024-04-22T18:39:11.834 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.327656+0000", 2024-04-22T18:39:11.834 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.401122745, 2024-04-22T18:39:11.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101011095, 2024-04-22T18:39:11.834 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.834 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.834 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327656+0000", 2024-04-22T18:39:11.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327656+0000", 2024-04-22T18:39:11.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327655+0000", 2024-04-22T18:39:11.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:11.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327667+0000", 2024-04-22T18:39:11.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2480000000000001e-05 2024-04-22T18:39:11.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327679+0000", 2024-04-22T18:39:11.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.22e-05 2024-04-22T18:39:11.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327686+0000", 2024-04-22T18:39:11.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8580000000000002e-06 2024-04-22T18:39:11.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428037+0000", 2024-04-22T18:39:11.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10035092399999999 2024-04-22T18:39:11.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428057+0000", 2024-04-22T18:39:11.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0030999999999998e-05 2024-04-22T18:39:11.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428638+0000", 2024-04-22T18:39:11.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058120100000000005 2024-04-22T18:39:11.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428667+0000", 2024-04-22T18:39:11.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8741000000000001e-05 2024-04-22T18:39:11.838 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.838 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.838 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.838 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:334 4.c e30/20 4:3227431c:::smithi08937929-35:head v 30'18, mlcod=30'16)", 2024-04-22T18:39:11.838 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.698741+0000", 2024-04-22T18:39:11.838 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.030037761999999, 2024-04-22T18:39:11.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10216847599999999, 2024-04-22T18:39:11.838 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.838 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.839 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698741+0000", 2024-04-22T18:39:11.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698741+0000", 2024-04-22T18:39:11.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698740+0000", 2024-04-22T18:39:11.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:11.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698747+0000", 2024-04-22T18:39:11.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9199999999999998e-06 2024-04-22T18:39:11.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698756+0000", 2024-04-22T18:39:11.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1249999999999999e-06 2024-04-22T18:39:11.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698762+0000", 2024-04-22T18:39:11.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:39:11.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798915+0000", 2024-04-22T18:39:11.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10015304899999999 2024-04-22T18:39:11.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798937+0000", 2024-04-22T18:39:11.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1335000000000002e-05 2024-04-22T18:39:11.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.800879+0000", 2024-04-22T18:39:11.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001942088 2024-04-22T18:39:11.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.800910+0000", 2024-04-22T18:39:11.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0892000000000002e-05 2024-04-22T18:39:11.842 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.842 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.842 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.842 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:419 4.0 4:01baf5a9:::smithi08937929-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:39:11.842 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:11.842 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.326557223, 2024-04-22T18:39:11.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30324478199999999, 2024-04-22T18:39:11.843 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.843 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.843 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:11.843 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:11.843 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:11.843 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 419 2024-04-22T18:39:11.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.843 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:11.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:11.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402219+0000", 2024-04-22T18:39:11.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:11.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402229+0000", 2024-04-22T18:39:11.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2550000000000003e-06 2024-04-22T18:39:11.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402242+0000", 2024-04-22T18:39:11.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2989999999999999e-05 2024-04-22T18:39:11.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402250+0000", 2024-04-22T18:39:11.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7900000000000005e-06 2024-04-22T18:39:11.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404085+0000", 2024-04-22T18:39:11.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001834397 2024-04-22T18:39:11.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:11.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404092+0000", 2024-04-22T18:39:11.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7889999999999997e-06 2024-04-22T18:39:11.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502534+0000", 2024-04-22T18:39:11.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098442331999999994 2024-04-22T18:39:11.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502631+0000", 2024-04-22T18:39:11.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7617999999999999e-05 2024-04-22T18:39:11.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:11.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604414+0000", 2024-04-22T18:39:11.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101782847 2024-04-22T18:39:11.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:11.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604888+0000", 2024-04-22T18:39:11.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047325999999999998 2024-04-22T18:39:11.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705376+0000", 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100488623 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705424+0000", 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7964999999999999e-05 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705466+0000", 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2274999999999998e-05 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.848 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.849 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.849 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.3.13:13 3.2 e31/29 3:549b9960:::smithi08937929-20 low_tier:head v 31'29, mlcod=30'27)", 2024-04-22T18:39:11.849 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.503273+0000", 2024-04-22T18:39:11.849 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.225506365999999, 2024-04-22T18:39:11.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10079894, 2024-04-22T18:39:11.849 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.849 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.849 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503273+0000", 2024-04-22T18:39:11.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503273+0000", 2024-04-22T18:39:11.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503272+0000", 2024-04-22T18:39:11.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:11.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503278+0000", 2024-04-22T18:39:11.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3210000000000001e-06 2024-04-22T18:39:11.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503294+0000", 2024-04-22T18:39:11.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.522e-05 2024-04-22T18:39:11.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503301+0000", 2024-04-22T18:39:11.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6879999999999999e-06 2024-04-22T18:39:11.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.603425+0000", 2024-04-22T18:39:11.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100124017 2024-04-22T18:39:11.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.603451+0000", 2024-04-22T18:39:11.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5510000000000001e-05 2024-04-22T18:39:11.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604033+0000", 2024-04-22T18:39:11.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058202199999999996 2024-04-22T18:39:11.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604071+0000", 2024-04-22T18:39:11.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8587000000000002e-05 2024-04-22T18:39:11.853 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.853 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.853 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.853 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:70 3.2 e32/29 3:5f0623ce:::smithi08937929-38 low_tier:head v 32'37, mlcod=31'35)", 2024-04-22T18:39:11.853 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:11.853 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.2003401920000005, 2024-04-22T18:39:11.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20070027900000001, 2024-04-22T18:39:11.853 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:11.853 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:11.853 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:11.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:11.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:11.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:11.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:11.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:11.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:11.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528438+0000", 2024-04-22T18:39:11.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:11.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:11.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528443+0000", 2024-04-22T18:39:11.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8640000000000004e-06 2024-04-22T18:39:11.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:11.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528453+0000", 2024-04-22T18:39:11.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3910000000000004e-06 2024-04-22T18:39:11.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:11.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528459+0000", 2024-04-22T18:39:11.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-04-22T18:39:11.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:11.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728627+0000", 2024-04-22T18:39:11.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20016778700000001 2024-04-22T18:39:11.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:11.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728660+0000", 2024-04-22T18:39:11.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3024000000000001e-05 2024-04-22T18:39:11.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:11.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729107+0000", 2024-04-22T18:39:11.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044719200000000001 2024-04-22T18:39:11.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:11.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:11.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:11.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729139+0000", 2024-04-22T18:39:11.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1625000000000003e-05 2024-04-22T18:39:11.857 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.857 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.857 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.857 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:11.857 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:11.857 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:11.858 INFO:tasks.rados.rados.0.smithi089.stdout:689: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-22T18:39:11.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:11.859 INFO:tasks.rados.rados.0.smithi089.stdout:691: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-22T18:39:11.860 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:11.859+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:11.897 INFO:tasks.rados.rados.0.smithi089.stdout:695: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-22T18:39:11.960 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:11.961+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:11.965 INFO:tasks.rados.rados.0.smithi089.stdout:698: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-22T18:39:11.970 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:11.970 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:11.970 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:11.970 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:11.979 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:12.049 INFO:tasks.rados.rados.0.smithi089.stdout:700: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-22T18:39:12.060 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:12.061+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:12.081 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:12.081 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:12.081 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:12.081 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:12.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:12.127 INFO:tasks.rados.rados.0.smithi089.stdout:690: finishing write tid 1 to smithi08937929-12 2024-04-22T18:39:12.127 INFO:tasks.rados.rados.0.smithi089.stdout:690: finishing write tid 2 to smithi08937929-12 2024-04-22T18:39:12.127 INFO:tasks.rados.rados.0.smithi089.stdout:699: finishing write tid 3 to smithi08937929-25 2024-04-22T18:39:12.127 INFO:tasks.rados.rados.0.smithi089.stdout:699: finishing write tid 4 to smithi08937929-25 2024-04-22T18:39:12.127 INFO:tasks.rados.rados.0.smithi089.stdout:699: finishing write tid 5 to smithi08937929-25 2024-04-22T18:39:12.127 INFO:tasks.rados.rados.0.smithi089.stdout:699: oid 25 updating version 0 to 28 2024-04-22T18:39:12.127 INFO:tasks.rados.rados.0.smithi089.stdout:699: oid 25 updating version 28 to 29 2024-04-22T18:39:12.127 INFO:tasks.rados.rados.0.smithi089.stdout:699: oid 25 updating version 29 to 30 2024-04-22T18:39:12.127 INFO:tasks.rados.rados.0.smithi089.stdout:699: oid 25 updating version 30 to 31 2024-04-22T18:39:12.127 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 31 (ObjNum 75 snap 0 seq_num 75) dirty exists 2024-04-22T18:39:12.127 INFO:tasks.rados.rados.0.smithi089.stdout:699: left oid 25 (ObjNum 75 snap 0 seq_num 75) 2024-04-22T18:39:12.127 INFO:tasks.rados.rados.0.smithi089.stdout:702: finishing set_chunk to oid 6 2024-04-22T18:39:12.128 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 6 set_chunk returned error code -95 offset: 69120 length: 1892352 2024-04-22T18:39:12.128 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-22T18:39:12.128 INFO:tasks.rados.rados.0.smithi089.stdout:694: finishing write tid 2 to smithi08937929-31 2024-04-22T18:39:12.128 INFO:tasks.rados.rados.0.smithi089.stdout:686: done (15 left) 2024-04-22T18:39:12.128 INFO:tasks.rados.rados.0.smithi089.stdout:687: done (14 left) 2024-04-22T18:39:12.128 INFO:tasks.rados.rados.0.smithi089.stdout:689: done (13 left) 2024-04-22T18:39:12.128 INFO:tasks.rados.rados.0.smithi089.stdout:691: done (12 left) 2024-04-22T18:39:12.128 INFO:tasks.rados.rados.0.smithi089.stdout:695: done (11 left) 2024-04-22T18:39:12.128 INFO:tasks.rados.rados.0.smithi089.stdout:698: done (10 left) 2024-04-22T18:39:12.128 INFO:tasks.rados.rados.0.smithi089.stdout:699: done (9 left) 2024-04-22T18:39:12.128 INFO:tasks.rados.rados.0.smithi089.stdout:700: done (8 left) 2024-04-22T18:39:12.128 INFO:tasks.rados.rados.0.smithi089.stdout:701: done (7 left) 2024-04-22T18:39:12.128 INFO:tasks.rados.rados.0.smithi089.stdout:702: done (6 left) 2024-04-22T18:39:12.128 INFO:tasks.rados.rados.0.smithi089.stdout:706: write oid 32 current snap is 0 2024-04-22T18:39:12.129 INFO:tasks.rados.rados.0.smithi089.stdout:694: finishing write tid 3 to smithi08937929-31 2024-04-22T18:39:12.129 INFO:tasks.rados.rados.0.smithi089.stdout:694: finishing write tid 4 to smithi08937929-31 2024-04-22T18:39:12.129 INFO:tasks.rados.rados.0.smithi089.stdout:697: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-22T18:39:12.161 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:12.163+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:12.211 INFO:tasks.rados.rados.0.smithi089.stdout:690: finishing write tid 3 to smithi08937929-12 2024-04-22T18:39:12.212 INFO:tasks.rados.rados.0.smithi089.stdout:694: finishing write tid 5 to smithi08937929-31 2024-04-22T18:39:12.212 INFO:tasks.rados.rados.0.smithi089.stdout:694: oid 31 updating version 0 to 24 2024-04-22T18:39:12.212 INFO:tasks.rados.rados.0.smithi089.stdout:694: oid 31 version 24 is already newer than 23 2024-04-22T18:39:12.212 INFO:tasks.rados.rados.0.smithi089.stdout:694: oid 31 version 24 is already newer than 22 2024-04-22T18:39:12.212 INFO:tasks.rados.rados.0.smithi089.stdout:694: oid 31 version 24 is already newer than 21 2024-04-22T18:39:12.212 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 24 (ObjNum 74 snap 0 seq_num 74) dirty exists 2024-04-22T18:39:12.212 INFO:tasks.rados.rados.0.smithi089.stdout:694: left oid 31 (ObjNum 74 snap 0 seq_num 74) 2024-04-22T18:39:12.212 INFO:tasks.rados.rados.0.smithi089.stdout:690: finishing write tid 4 to smithi08937929-12 2024-04-22T18:39:12.212 INFO:tasks.rados.rados.0.smithi089.stdout:690: finishing write tid 5 to smithi08937929-12 2024-04-22T18:39:12.212 INFO:tasks.rados.rados.0.smithi089.stdout:690: oid 12 updating version 0 to 32 2024-04-22T18:39:12.212 INFO:tasks.rados.rados.0.smithi089.stdout:690: oid 12 version 32 is already newer than 31 2024-04-22T18:39:12.212 INFO:tasks.rados.rados.0.smithi089.stdout:690: oid 12 version 32 is already newer than 30 2024-04-22T18:39:12.212 INFO:tasks.rados.rados.0.smithi089.stdout:690: oid 12 updating version 32 to 33 2024-04-22T18:39:12.212 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 33 (ObjNum 73 snap 0 seq_num 73) dirty exists 2024-04-22T18:39:12.213 INFO:tasks.rados.rados.0.smithi089.stdout:690: left oid 12 (ObjNum 73 snap 0 seq_num 73) 2024-04-22T18:39:12.213 INFO:tasks.rados.rados.0.smithi089.stdout:703: finishing write tid 1 to smithi08937929-36 2024-04-22T18:39:12.213 INFO:tasks.rados.rados.0.smithi089.stdout:706: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-04-22T18:39:12.217 INFO:tasks.rados.rados.0.smithi089.stdout:706: writing smithi08937929-32 from 510946 to 1143898 tid 1 2024-04-22T18:39:12.224 INFO:tasks.rados.rados.0.smithi089.stdout:706: writing smithi08937929-32 from 1798066 to 2275745 tid 2 2024-04-22T18:39:12.230 INFO:tasks.rados.rados.0.smithi089.stdout:706: writing smithi08937929-32 from 2742758 to 3199975 tid 3 2024-04-22T18:39:12.230 INFO:tasks.rados.rados.0.smithi089.stdout:690: done (6 left) 2024-04-22T18:39:12.230 INFO:tasks.rados.rados.0.smithi089.stdout:694: done (5 left) 2024-04-22T18:39:12.230 INFO:tasks.rados.rados.0.smithi089.stdout:697: done (4 left) 2024-04-22T18:39:12.230 INFO:tasks.rados.rados.0.smithi089.stdout:707: read oid 25 snap -1 2024-04-22T18:39:12.230 INFO:tasks.rados.rados.0.smithi089.stdout:707: expect (ObjNum 75 snap 0 seq_num 75) 2024-04-22T18:39:12.231 INFO:tasks.rados.rados.0.smithi089.stdout:708: tier_evict oid 14 2024-04-22T18:39:12.231 INFO:tasks.rados.rados.0.smithi089.stdout:708: tier_evict oid 14 snap -1 2024-04-22T18:39:12.231 INFO:tasks.rados.rados.0.smithi089.stdout:703: finishing write tid 2 to smithi08937929-36 2024-04-22T18:39:12.231 INFO:tasks.rados.rados.0.smithi089.stdout:704: finishing write tid 1 to smithi08937929-17 2024-04-22T18:39:12.231 INFO:tasks.rados.rados.0.smithi089.stdout:704: finishing write tid 2 to smithi08937929-17 2024-04-22T18:39:12.231 INFO:tasks.rados.rados.0.smithi089.stdout:704: finishing write tid 3 to smithi08937929-17 2024-04-22T18:39:12.231 INFO:tasks.rados.rados.0.smithi089.stdout:704: finishing write tid 4 to smithi08937929-17 2024-04-22T18:39:12.231 INFO:tasks.rados.rados.0.smithi089.stdout:709: delete oid 3 current snap is 0 2024-04-22T18:39:12.231 INFO:tasks.rados.rados.0.smithi089.stdout:704: finishing write tid 5 to smithi08937929-17 2024-04-22T18:39:12.231 INFO:tasks.rados.rados.0.smithi089.stdout:704: oid 17 updating version 0 to 32 2024-04-22T18:39:12.231 INFO:tasks.rados.rados.0.smithi089.stdout:704: oid 17 updating version 32 to 35 2024-04-22T18:39:12.231 INFO:tasks.rados.rados.0.smithi089.stdout:704: oid 17 version 35 is already newer than 33 2024-04-22T18:39:12.231 INFO:tasks.rados.rados.0.smithi089.stdout:704: oid 17 version 35 is already newer than 34 2024-04-22T18:39:12.231 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 35 (ObjNum 77 snap 0 seq_num 77) dirty exists 2024-04-22T18:39:12.232 INFO:tasks.rados.rados.0.smithi089.stdout:704: left oid 17 (ObjNum 77 snap 0 seq_num 77) 2024-04-22T18:39:12.232 INFO:tasks.rados.rados.0.smithi089.stdout:705: finishing set_chunk to oid 37 2024-04-22T18:39:12.232 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 37 set_chunk returned error code -95 offset: 185344 length: 2148864 2024-04-22T18:39:12.232 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-22T18:39:12.232 INFO:tasks.rados.rados.0.smithi089.stdout:703: finishing write tid 3 to smithi08937929-36 2024-04-22T18:39:12.232 INFO:tasks.rados.rados.0.smithi089.stdout:703: finishing write tid 4 to smithi08937929-36 2024-04-22T18:39:12.232 INFO:tasks.rados.rados.0.smithi089.stdout:703: finishing write tid 5 to smithi08937929-36 2024-04-22T18:39:12.232 INFO:tasks.rados.rados.0.smithi089.stdout:703: oid 36 updating version 0 to 22 2024-04-22T18:39:12.232 INFO:tasks.rados.rados.0.smithi089.stdout:703: oid 36 updating version 22 to 23 2024-04-22T18:39:12.232 INFO:tasks.rados.rados.0.smithi089.stdout:703: oid 36 version 23 is already newer than 21 2024-04-22T18:39:12.232 INFO:tasks.rados.rados.0.smithi089.stdout:703: oid 36 version 23 is already newer than 20 2024-04-22T18:39:12.232 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 23 (ObjNum 76 snap 0 seq_num 76) dirty exists 2024-04-22T18:39:12.232 INFO:tasks.rados.rados.0.smithi089.stdout:703: left oid 36 (ObjNum 76 snap 0 seq_num 76) 2024-04-22T18:39:12.235 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:12.235 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:12.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:12.235 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:12.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.235 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.235 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:12.235 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.072278986000001, 2024-04-22T18:39:12.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:39:12.235 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.235 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.235 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.235 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.236 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.236 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:39:12.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.236 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:12.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:12.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:39:12.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:12.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:39:12.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:39:12.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:39:12.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:39:12.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:39:12.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:39:12.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:39:12.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:39:12.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:39:12.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:39:12.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:39:12.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:39:12.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:39:12.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:39:12.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:39:12.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:39:12.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:39:12.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:39:12.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:39:12.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:39:12.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:39:12.240 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.240 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.240 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.241 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.241 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:12.241 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.970720164999999, 2024-04-22T18:39:12.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:39:12.241 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.241 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.241 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:39:12.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.241 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:12.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:12.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:39:12.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:12.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:39:12.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:39:12.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:39:12.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:39:12.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:39:12.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:39:12.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:39:12.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:39:12.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:39:12.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:39:12.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:39:12.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:39:12.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:39:12.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:39:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:39:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:39:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:39:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:39:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:39:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:39:12.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:39:12.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.246 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.246 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.246 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:12.246 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.970693763, 2024-04-22T18:39:12.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:39:12.246 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.246 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:12.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.247 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:39:12.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.247 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:12.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:12.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:39:12.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:39:12.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:39:12.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:39:12.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:39:12.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:39:12.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:39:12.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:39:12.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:39:12.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:12.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:39:12.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:39:12.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:39:12.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:39:12.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:39:12.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:39:12.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:39:12.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:39:12.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.251 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.251 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.251 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:12.251 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.572359433999999, 2024-04-22T18:39:12.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:39:12.251 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.251 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.251 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.251 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.252 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:39:12.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.252 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:12.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:12.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:12.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:39:12.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:39:12.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:39:12.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:39:12.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:39:12.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:39:12.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:39:12.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:39:12.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:39:12.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:39:12.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:12.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:39:12.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:39:12.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:39:12.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:39:12.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:39:12.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:39:12.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:39:12.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:39:12.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:39:12.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:39:12.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.256 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.257 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.257 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:12.257 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.471523135999998, 2024-04-22T18:39:12.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:39:12.257 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.257 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.257 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.257 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.257 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.257 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:39:12.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.257 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:12.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:12.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:39:12.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:12.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:39:12.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:39:12.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:39:12.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:39:12.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:39:12.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:39:12.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:39:12.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:39:12.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:39:12.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:39:12.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:12.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:39:12.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:39:12.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:39:12.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:39:12.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:39:12.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:39:12.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:39:12.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:39:12.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:39:12.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:39:12.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.262 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.262 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.262 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:12.262 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.471365865999999, 2024-04-22T18:39:12.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:39:12.263 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.263 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.263 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:39:12.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.263 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:12.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:12.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:39:12.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:39:12.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:39:12.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:39:12.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:39:12.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:39:12.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:39:12.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:39:12.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:39:12.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:39:12.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:39:12.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:12.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:39:12.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:39:12.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:39:12.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:39:12.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:39:12.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:39:12.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:39:12.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:39:12.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:39:12.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:39:12.268 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.268 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.268 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.268 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.268 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:12.268 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.471351256999998, 2024-04-22T18:39:12.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:39:12.268 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.268 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:12.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.268 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:39:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:39:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:39:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:39:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:39:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:39:12.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:39:12.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:39:12.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:39:12.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:12.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:39:12.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:39:12.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:39:12.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:39:12.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:39:12.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:39:12.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:39:12.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:39:12.272 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.272 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.471340871999999, 2024-04-22T18:39:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:39:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:39:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:39:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:39:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:39:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:39:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:39:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:39:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:39:12.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:39:12.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:39:12.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:39:12.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:12.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:39:12.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:39:12.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:39:12.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:39:12.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:39:12.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:39:12.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:39:12.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:39:12.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:39:12.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:39:12.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.278 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.278 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.278 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:12.278 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.470410942000001, 2024-04-22T18:39:12.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:39:12.279 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.279 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.279 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:39:12.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.279 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:12.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:39:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:39:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:39:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:39:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:39:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:39:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:39:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:39:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:39:12.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:39:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:39:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:39:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:39:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:39:12.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:39:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:39:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:39:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:39:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:39:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:39:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:39:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.469948816999999, 2024-04-22T18:39:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:39:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.284 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:39:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:39:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:39:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:39:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:39:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:39:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:39:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:39:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:39:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:39:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:39:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:39:12.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:39:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:39:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:39:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:39:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:39:12.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:39:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:39:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:39:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:39:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:39:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.469936013000002, 2024-04-22T18:39:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:39:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:39:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:39:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:39:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:39:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:39:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:39:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:39:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:39:12.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:39:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:39:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:39:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:39:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:39:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:39:12.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:39:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:39:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:39:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:39:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:109 4.1 4:83c955bf:::smithi08937929-28:head [write 453789~780782,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.267149020000002, 2024-04-22T18:39:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.45086113300000003, 2024-04-22T18:39:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:39:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967588+0000", 2024-04-22T18:39:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.968243+0000", 2024-04-22T18:39:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00065488500000000002 2024-04-22T18:39:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.968265+0000", 2024-04-22T18:39:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1997e-05 2024-04-22T18:39:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.068340+0000", 2024-04-22T18:39:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007516 2024-04-22T18:39:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168464+0000", 2024-04-22T18:39:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100124267 2024-04-22T18:39:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168579+0000", 2024-04-22T18:39:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114387 2024-04-22T18:39:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168692+0000", 2024-04-22T18:39:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011335100000000001 2024-04-22T18:39:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.173549+0000", 2024-04-22T18:39:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048564430000000002 2024-04-22T18:39:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418367+0000", 2024-04-22T18:39:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.244817962 2024-04-22T18:39:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418419+0000", 2024-04-22T18:39:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2883000000000002e-05 2024-04-22T18:39:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418450+0000", 2024-04-22T18:39:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0278999999999999e-05 2024-04-22T18:39:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:119 4.d 4:bc165c76:::smithi08937929-30:head [setxattr _header (56) in=7b,truncate 2487403] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.069595+0000", 2024-04-22T18:39:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.165142407000001, 2024-04-22T18:39:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.406949954, 2024-04-22T18:39:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.300 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 119 2024-04-22T18:39:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.069595+0000", 2024-04-22T18:39:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.069595+0000", 2024-04-22T18:39:12.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.069595+0000", 2024-04-22T18:39:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.069598+0000", 2024-04-22T18:39:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8630000000000001e-06 2024-04-22T18:39:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.069602+0000", 2024-04-22T18:39:12.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5279999999999999e-06 2024-04-22T18:39:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.169672+0000", 2024-04-22T18:39:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100070467 2024-04-22T18:39:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.269862+0000", 2024-04-22T18:39:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100189757 2024-04-22T18:39:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.270027+0000", 2024-04-22T18:39:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165215 2024-04-22T18:39:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:39:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.270201+0000", 2024-04-22T18:39:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017361300000000001 2024-04-22T18:39:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.473627+0000", 2024-04-22T18:39:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203426361 2024-04-22T18:39:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.476448+0000", 2024-04-22T18:39:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002820857 2024-04-22T18:39:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.476509+0000", 2024-04-22T18:39:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1251e-05 2024-04-22T18:39:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.476545+0000", 2024-04-22T18:39:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5703e-05 2024-04-22T18:39:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.142163557, 2024-04-22T18:39:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:39:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:39:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:39:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:39:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:39:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:39:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:39:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:39:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:39:12.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:39:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:39:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:39:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:39:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:39:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:39:12.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:39:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:39:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:39:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:39:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:39:12.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:39:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:39:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:39:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.133590444999999, 2024-04-22T18:39:12.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:39:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:39:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:39:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:39:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:39:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:39:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:39:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:39:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:39:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:39:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:39:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:39:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:39:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:39:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:39:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:39:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:39:12.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:39:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:39:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:39:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:39:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:39:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:39:12.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:33 3.3 3:ea25b818:::smithi08937929-10 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.528569+0000", 2024-04-22T18:39:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.7061683309999998, 2024-04-22T18:39:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40827227799999999, 2024-04-22T18:39:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528569+0000", 2024-04-22T18:39:12.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528569+0000", 2024-04-22T18:39:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528567+0000", 2024-04-22T18:39:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528582+0000", 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5565000000000001e-05 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528597+0000", 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4918e-05 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528605+0000", 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0609999999999996e-06 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528657+0000", 2024-04-22T18:39:12.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1280999999999999e-05 2024-04-22T18:39:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528662+0000", 2024-04-22T18:39:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2129999999999999e-06 2024-04-22T18:39:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.733707+0000", 2024-04-22T18:39:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20504494000000001 2024-04-22T18:39:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.733793+0000", 2024-04-22T18:39:12.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6373999999999997e-05 2024-04-22T18:39:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.733937+0000", 2024-04-22T18:39:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143289 2024-04-22T18:39:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735740+0000", 2024-04-22T18:39:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001803334 2024-04-22T18:39:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.936677+0000", 2024-04-22T18:39:12.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20093707699999999 2024-04-22T18:39:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.936763+0000", 2024-04-22T18:39:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6339000000000001e-05 2024-04-22T18:39:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.936842+0000", 2024-04-22T18:39:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8237000000000003e-05 2024-04-22T18:39:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:34 3.3 3:ea25b818:::smithi08937929-10 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.528623+0000", 2024-04-22T18:39:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.7061144390000003, 2024-04-22T18:39:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40832893599999998, 2024-04-22T18:39:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528623+0000", 2024-04-22T18:39:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528623+0000", 2024-04-22T18:39:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528622+0000", 2024-04-22T18:39:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528627+0000", 2024-04-22T18:39:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7169999999999999e-06 2024-04-22T18:39:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528635+0000", 2024-04-22T18:39:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7549999999999994e-06 2024-04-22T18:39:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528652+0000", 2024-04-22T18:39:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7090000000000001e-05 2024-04-22T18:39:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528728+0000", 2024-04-22T18:39:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6401000000000003e-05 2024-04-22T18:39:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528735+0000", 2024-04-22T18:39:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2219999999999999e-06 2024-04-22T18:39:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.734263+0000", 2024-04-22T18:39:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.205528763 2024-04-22T18:39:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.734360+0000", 2024-04-22T18:39:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6965000000000004e-05 2024-04-22T18:39:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.734561+0000", 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000200478 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.736335+0000", 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017739240000000001 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.936894+0000", 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.200559666 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.936930+0000", 2024-04-22T18:39:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6133000000000001e-05 2024-04-22T18:39:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.936952+0000", 2024-04-22T18:39:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1645999999999999e-05 2024-04-22T18:39:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:35 3.3 3:ea25b818:::smithi08937929-10 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.528658+0000", 2024-04-22T18:39:12.328 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.706079431, 2024-04-22T18:39:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40843963100000003, 2024-04-22T18:39:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528658+0000", 2024-04-22T18:39:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528658+0000", 2024-04-22T18:39:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528657+0000", 2024-04-22T18:39:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528662+0000", 2024-04-22T18:39:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7609999999999998e-06 2024-04-22T18:39:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528669+0000", 2024-04-22T18:39:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5570000000000001e-06 2024-04-22T18:39:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528674+0000", 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4140000000000002e-06 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528762+0000", 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8200000000000003e-05 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528767+0000", 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2880000000000003e-06 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.734844+0000", 2024-04-22T18:39:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.206077327 2024-04-22T18:39:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.734934+0000", 2024-04-22T18:39:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0016e-05 2024-04-22T18:39:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735118+0000", 2024-04-22T18:39:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018423199999999999 2024-04-22T18:39:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.738006+0000", 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0028883070000000001 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937009+0000", 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19900257099999999 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937053+0000", 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4024999999999999e-05 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937098+0000", 2024-04-22T18:39:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4712999999999997e-05 2024-04-22T18:39:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:36 3.3 3:ea25b818:::smithi08937929-10 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.70604741, 2024-04-22T18:39:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40851152800000001, 2024-04-22T18:39:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528694+0000", 2024-04-22T18:39:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2459999999999997e-06 2024-04-22T18:39:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528701+0000", 2024-04-22T18:39:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7950000000000002e-06 2024-04-22T18:39:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528709+0000", 2024-04-22T18:39:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1000000000000004e-06 2024-04-22T18:39:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528782+0000", 2024-04-22T18:39:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3699e-05 2024-04-22T18:39:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528785+0000", 2024-04-22T18:39:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:39:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735367+0000", 2024-04-22T18:39:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20658175100000001 2024-04-22T18:39:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735443+0000", 2024-04-22T18:39:12.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6106999999999995e-05 2024-04-22T18:39:12.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:12.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735573+0000", 2024-04-22T18:39:12.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129734 2024-04-22T18:39:12.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.740966+0000", 2024-04-22T18:39:12.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053933130000000003 2024-04-22T18:39:12.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937143+0000", 2024-04-22T18:39:12.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19617662399999999 2024-04-22T18:39:12.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937175+0000", 2024-04-22T18:39:12.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1860999999999999e-05 2024-04-22T18:39:12.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937202+0000", 2024-04-22T18:39:12.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7029000000000002e-05 2024-04-22T18:39:12.339 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.339 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:37 3.3 3:ea25b818:::smithi08937929-10 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.528720+0000", 2024-04-22T18:39:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.706018094, 2024-04-22T18:39:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40858023100000002, 2024-04-22T18:39:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528720+0000", 2024-04-22T18:39:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528720+0000", 2024-04-22T18:39:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528719+0000", 2024-04-22T18:39:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528724+0000", 2024-04-22T18:39:12.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1159999999999998e-06 2024-04-22T18:39:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528730+0000", 2024-04-22T18:39:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0480000000000004e-06 2024-04-22T18:39:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.628843+0000", 2024-04-22T18:39:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100112425 2024-04-22T18:39:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.628917+0000", 2024-04-22T18:39:12.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4385e-05 2024-04-22T18:39:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.628923+0000", 2024-04-22T18:39:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4239999999999998e-06 2024-04-22T18:39:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735762+0000", 2024-04-22T18:39:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10683828200000001 2024-04-22T18:39:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735826+0000", 2024-04-22T18:39:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4144000000000001e-05 2024-04-22T18:39:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735941+0000", 2024-04-22T18:39:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011526799999999999 2024-04-22T18:39:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.740973+0000", 2024-04-22T18:39:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050320340000000003 2024-04-22T18:39:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937246+0000", 2024-04-22T18:39:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19627275499999999 2024-04-22T18:39:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937277+0000", 2024-04-22T18:39:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1563999999999998e-05 2024-04-22T18:39:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937300+0000", 2024-04-22T18:39:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2350000000000001e-05 2024-04-22T18:39:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.345 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.346 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.346 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:12.347 INFO:tasks.rados.rados.0.smithi089.stdout:706: finishing write tid 1 to smithi08937929-32 2024-04-22T18:39:12.347 INFO:tasks.rados.rados.0.smithi089.stdout:707: expect (ObjNum 75 snap 0 seq_num 75) 2024-04-22T18:39:12.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:12.348 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:12.348+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:12.447 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:12.448+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:12.490 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:51 3.2 3.d7879dee (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433125+0000", 2024-04-22T18:39:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.058856101000000001, 2024-04-22T18:39:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.058875891, 2024-04-22T18:39:12.490 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433125+0000", 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433125+0000", 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433123+0000", 2024-04-22T18:39:12.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433134+0000", 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0091999999999999e-05 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433145+0000", 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1970000000000001e-05 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433153+0000", 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5079999999999999e-06 2024-04-22T18:39:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:52 3.2 3.d7879dee (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433164+0000", 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.058817870000000001, 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.058865928999999997, 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433164+0000", 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433164+0000", 2024-04-22T18:39:12.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433163+0000", 2024-04-22T18:39:12.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433166+0000", 2024-04-22T18:39:12.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.807e-06 2024-04-22T18:39:12.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433173+0000", 2024-04-22T18:39:12.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6279999999999999e-06 2024-04-22T18:39:12.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433176+0000", 2024-04-22T18:39:12.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.056e-06 2024-04-22T18:39:12.495 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.495 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.495 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3.d7879dee (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.058780536000000001, 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.058853678999999999, 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:12.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:39:12.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:39:12.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:39:12.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:39:12.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:39:12.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:39:12.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:39:12.498 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.498 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.498 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.498 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.498 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:39:12.498 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 3 2024-04-22T18:39:12.498 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:12.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:12.515 INFO:tasks.rados.rados.0.smithi089.stdout:708: got (22) Invalid argument 2024-04-22T18:39:12.515 INFO:tasks.rados.rados.0.smithi089.stdout:706: finishing write tid 2 to smithi08937929-32 2024-04-22T18:39:12.515 INFO:tasks.rados.rados.0.smithi089.stdout:706: finishing write tid 3 to smithi08937929-32 2024-04-22T18:39:12.515 INFO:tasks.rados.rados.0.smithi089.stdout:703: done (6 left) 2024-04-22T18:39:12.515 INFO:tasks.rados.rados.0.smithi089.stdout:704: done (5 left) 2024-04-22T18:39:12.515 INFO:tasks.rados.rados.0.smithi089.stdout:705: done (4 left) 2024-04-22T18:39:12.515 INFO:tasks.rados.rados.0.smithi089.stdout:707: done (3 left) 2024-04-22T18:39:12.515 INFO:tasks.rados.rados.0.smithi089.stdout:708: done (2 left) 2024-04-22T18:39:12.515 INFO:tasks.rados.rados.0.smithi089.stdout:709: done (1 left) 2024-04-22T18:39:12.516 INFO:tasks.rados.rados.0.smithi089.stdout:710: write (excl) oid 36 current snap is 0 2024-04-22T18:39:12.516 INFO:tasks.rados.rados.0.smithi089.stdout:706: finishing write tid 4 to smithi08937929-32 2024-04-22T18:39:12.516 INFO:tasks.rados.rados.0.smithi089.stdout:710: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-04-22T18:39:12.522 INFO:tasks.rados.rados.0.smithi089.stdout:710: writing smithi08937929-36 from 544560 to 1328010 tid 1 2024-04-22T18:39:12.527 INFO:tasks.rados.rados.0.smithi089.stdout:710: writing smithi08937929-36 from 1915030 to 2396378 tid 2 2024-04-22T18:39:12.533 INFO:tasks.rados.rados.0.smithi089.stdout:710: writing smithi08937929-36 from 2797604 to 3277365 tid 3 2024-04-22T18:39:12.533 INFO:tasks.rados.rados.0.smithi089.stdout:711: read oid 3 snap -1 2024-04-22T18:39:12.533 INFO:tasks.rados.rados.0.smithi089.stdout:711: expect deleted 2024-04-22T18:39:12.533 INFO:tasks.rados.rados.0.smithi089.stdout:712: tier_evict oid 5 2024-04-22T18:39:12.533 INFO:tasks.rados.rados.0.smithi089.stdout:712: tier_evict oid 5 snap -1 2024-04-22T18:39:12.533 INFO:tasks.rados.rados.0.smithi089.stdout:713: read oid 33 snap -1 2024-04-22T18:39:12.533 INFO:tasks.rados.rados.0.smithi089.stdout:713: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-22T18:39:12.533 INFO:tasks.rados.rados.0.smithi089.stdout:714: tier_promote oid 37 2024-04-22T18:39:12.534 INFO:tasks.rados.rados.0.smithi089.stdout:715: write (excl) oid 4 current snap is 0 2024-04-22T18:39:12.534 INFO:tasks.rados.rados.0.smithi089.stdout:715: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-04-22T18:39:12.538 INFO:tasks.rados.rados.0.smithi089.stdout:715: writing smithi08937929-4 from 561367 to 1020066 tid 1 2024-04-22T18:39:12.545 INFO:tasks.rados.rados.0.smithi089.stdout:715: writing smithi08937929-4 from 1573512 to 2314871 tid 2 2024-04-22T18:39:12.548 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:12.549+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:12.551 INFO:tasks.rados.rados.0.smithi089.stdout:715: writing smithi08937929-4 from 2941380 to 3361367 tid 3 2024-04-22T18:39:12.551 INFO:tasks.rados.rados.0.smithi089.stdout:716: set_chunk oid 15 target oid 2024-04-22T18:39:12.551 INFO:tasks.rados.rados.0.smithi089.stdout:716: set_chunk oid 15 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:39:12.551 INFO:tasks.rados.rados.0.smithi089.stdout:717: read oid 22 snap -1 2024-04-22T18:39:12.551 INFO:tasks.rados.rados.0.smithi089.stdout:717: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-22T18:39:12.552 INFO:tasks.rados.rados.0.smithi089.stdout:718: write oid 23 current snap is 0 2024-04-22T18:39:12.552 INFO:tasks.rados.rados.0.smithi089.stdout:718: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-04-22T18:39:12.557 INFO:tasks.rados.rados.0.smithi089.stdout:718: writing smithi08937929-23 from 578174 to 1112122 tid 1 2024-04-22T18:39:12.561 INFO:tasks.rados.rados.0.smithi089.stdout:718: writing smithi08937929-23 from 1515641 to 2000658 tid 2 2024-04-22T18:39:12.568 INFO:tasks.rados.rados.0.smithi089.stdout:718: writing smithi08937929-23 from 2736097 to 3238402 tid 3 2024-04-22T18:39:12.568 INFO:tasks.rados.rados.0.smithi089.stdout:719: tier_evict oid 24 2024-04-22T18:39:12.568 INFO:tasks.rados.rados.0.smithi089.stdout:719: tier_evict oid 24 snap -1 2024-04-22T18:39:12.568 INFO:tasks.rados.rados.0.smithi089.stdout:720: write (excl) oid 42 current snap is 0 2024-04-22T18:39:12.568 INFO:tasks.rados.rados.0.smithi089.stdout:720: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-04-22T18:39:12.574 INFO:tasks.rados.rados.0.smithi089.stdout:720: writing smithi08937929-42 from 594981 to 1204178 tid 1 2024-04-22T18:39:12.582 INFO:tasks.rados.rados.0.smithi089.stdout:720: writing smithi08937929-42 from 1974123 to 2719151 tid 2 2024-04-22T18:39:12.586 INFO:tasks.rados.rados.0.smithi089.stdout:720: writing smithi08937929-42 from 3279873 to 3394981 tid 3 2024-04-22T18:39:12.586 INFO:tasks.rados.rados.0.smithi089.stdout:721: set_chunk oid 49 target oid 2024-04-22T18:39:12.646 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:12.647 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:12.647 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:12.647 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:12.647 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:12.650 INFO:tasks.rados.rados.0.smithi089.stdout:721: set_chunk oid 49 offset: 733696 length: 1769472 target oid be18db0093d007ece4c6cc3b64eff1b36375e4987bd4a7debf314b77847a570e offset: 0 snap -1 2024-04-22T18:39:12.650 INFO:tasks.rados.rados.0.smithi089.stdout:722: read oid 31 snap -1 2024-04-22T18:39:12.650 INFO:tasks.rados.rados.0.smithi089.stdout:722: expect (ObjNum 74 snap 0 seq_num 74) 2024-04-22T18:39:12.650 INFO:tasks.rados.rados.0.smithi089.stdout:723: read oid 20 snap -1 2024-04-22T18:39:12.650 INFO:tasks.rados.rados.0.smithi089.stdout:723: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-22T18:39:12.650 INFO:tasks.rados.rados.0.smithi089.stdout:724: read oid 10 snap -1 2024-04-22T18:39:12.650 INFO:tasks.rados.rados.0.smithi089.stdout:724: expect deleted 2024-04-22T18:39:12.650 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:12.650 INFO:tasks.rados.rados.0.smithi089.stdout:720: finishing write tid 1 to smithi08937929-42 2024-04-22T18:39:12.650 INFO:tasks.rados.rados.0.smithi089.stdout:706: finishing write tid 5 to smithi08937929-32 2024-04-22T18:39:12.651 INFO:tasks.rados.rados.0.smithi089.stdout:706: oid 32 updating version 0 to 24 2024-04-22T18:39:12.651 INFO:tasks.rados.rados.0.smithi089.stdout:706: oid 32 updating version 24 to 25 2024-04-22T18:39:12.651 INFO:tasks.rados.rados.0.smithi089.stdout:706: oid 32 updating version 25 to 26 2024-04-22T18:39:12.651 INFO:tasks.rados.rados.0.smithi089.stdout:706: oid 32 updating version 26 to 27 2024-04-22T18:39:12.651 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 27 (ObjNum 78 snap 0 seq_num 78) dirty exists 2024-04-22T18:39:12.651 INFO:tasks.rados.rados.0.smithi089.stdout:706: left oid 32 (ObjNum 78 snap 0 seq_num 78) 2024-04-22T18:39:12.651 INFO:tasks.rados.rados.0.smithi089.stdout:712: got (22) Invalid argument 2024-04-22T18:39:12.651 INFO:tasks.rados.rados.0.smithi089.stdout:713: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-22T18:39:12.651 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:12.650+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:12.656 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:12.750 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:12.752+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:12.801 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:12.801 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.707539258000001, 2024-04-22T18:39:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:39:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:39:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:39:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:39:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:39:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:39:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:39:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:39:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:39:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:39:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:39:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:39:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:39:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:39:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:39:12.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:39:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:39:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:39:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:39:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:39:12.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:39:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:39:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:39:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2 4.8 4:12d9f826:::smithi08937929-1:head [write 1542129~474678 in=474678b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:39:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.606827987999999, 2024-04-22T18:39:12.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20435667099999999, 2024-04-22T18:39:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2 2024-04-22T18:39:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:39:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:39:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194568+0000", 2024-04-22T18:39:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194869+0000", 2024-04-22T18:39:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030115200000000001 2024-04-22T18:39:12.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194879+0000", 2024-04-22T18:39:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8479999999999996e-06 2024-04-22T18:39:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294954+0000", 2024-04-22T18:39:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100074702 2024-04-22T18:39:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296019+0000", 2024-04-22T18:39:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010649189999999999 2024-04-22T18:39:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296086+0000", 2024-04-22T18:39:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7261000000000005e-05 2024-04-22T18:39:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296173+0000", 2024-04-22T18:39:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6935999999999996e-05 2024-04-22T18:39:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297196+0000", 2024-04-22T18:39:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023415 2024-04-22T18:39:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398879+0000", 2024-04-22T18:39:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101682563 2024-04-22T18:39:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398906+0000", 2024-04-22T18:39:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6526999999999999e-05 2024-04-22T18:39:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398928+0000", 2024-04-22T18:39:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2045e-05 2024-04-22T18:39:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.813 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:53 4.3 e27/20 4:ca46fde9:::smithi08937929-14:head v 27'1, mlcod=0'0)", 2024-04-22T18:39:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:39:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.553580785000001, 2024-04-22T18:39:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20214333400000001, 2024-04-22T18:39:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:39:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:39:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:39:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967296 2024-04-22T18:39:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248504+0000", 2024-04-22T18:39:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000686091 2024-04-22T18:39:12.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248512+0000", 2024-04-22T18:39:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1850000000000004e-06 2024-04-22T18:39:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248517+0000", 2024-04-22T18:39:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5469999999999998e-06 2024-04-22T18:39:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.348632+0000", 2024-04-22T18:39:12.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100115069 2024-04-22T18:39:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.348644+0000", 2024-04-22T18:39:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2001000000000001e-05 2024-04-22T18:39:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449930+0000", 2024-04-22T18:39:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101286574 2024-04-22T18:39:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449961+0000", 2024-04-22T18:39:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1046000000000001e-05 2024-04-22T18:39:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:54 4.3 e27/20 4:ca46fde9:::smithi08937929-14:head v 27'2, mlcod=0'0)", 2024-04-22T18:39:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.252046+0000", 2024-04-22T18:39:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.549353241999999, 2024-04-22T18:39:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19793179499999999, 2024-04-22T18:39:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252046+0000", 2024-04-22T18:39:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252046+0000", 2024-04-22T18:39:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252045+0000", 2024-04-22T18:39:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252215+0000", 2024-04-22T18:39:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017004499999999999 2024-04-22T18:39:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252221+0000", 2024-04-22T18:39:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.677e-06 2024-04-22T18:39:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252225+0000", 2024-04-22T18:39:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7730000000000001e-06 2024-04-22T18:39:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.349338+0000", 2024-04-22T18:39:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.097113853999999999 2024-04-22T18:39:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.349350+0000", 2024-04-22T18:39:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1268e-05 2024-04-22T18:39:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449967+0000", 2024-04-22T18:39:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100616754 2024-04-22T18:39:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449977+0000", 2024-04-22T18:39:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0978999999999999e-05 2024-04-22T18:39:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.822 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:55 4.3 e27/20)", 2024-04-22T18:39:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:39:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.548785590000001, 2024-04-22T18:39:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19766915500000001, 2024-04-22T18:39:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:39:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:39:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252613+0000", 2024-04-22T18:39:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252616+0000", 2024-04-22T18:39:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1889999999999999e-06 2024-04-22T18:39:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252622+0000", 2024-04-22T18:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0589999999999999e-06 2024-04-22T18:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.252625+0000", 2024-04-22T18:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3239999999999999e-06 2024-04-22T18:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449805+0000", 2024-04-22T18:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19717992500000001 2024-04-22T18:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449819+0000", 2024-04-22T18:39:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.367e-05 2024-04-22T18:39:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.450269+0000", 2024-04-22T18:39:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045026300000000001 2024-04-22T18:39:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.450282+0000", 2024-04-22T18:39:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3229e-05 2024-04-22T18:39:12.826 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=62b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:39:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.506427034000001, 2024-04-22T18:39:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20408478299999999, 2024-04-22T18:39:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.827 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3 2024-04-22T18:39:12.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.828 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:39:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:39:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294970+0000", 2024-04-22T18:39:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294983+0000", 2024-04-22T18:39:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3429e-05 2024-04-22T18:39:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294993+0000", 2024-04-22T18:39:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0416e-05 2024-04-22T18:39:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295007+0000", 2024-04-22T18:39:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3742e-05 2024-04-22T18:39:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296793+0000", 2024-04-22T18:39:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017860420000000001 2024-04-22T18:39:12.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296841+0000", 2024-04-22T18:39:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7706e-05 2024-04-22T18:39:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296913+0000", 2024-04-22T18:39:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2578999999999997e-05 2024-04-22T18:39:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297946+0000", 2024-04-22T18:39:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001032802 2024-04-22T18:39:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498963+0000", 2024-04-22T18:39:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20101640400000001 2024-04-22T18:39:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498998+0000", 2024-04-22T18:39:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5759000000000001e-05 2024-04-22T18:39:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499057+0000", 2024-04-22T18:39:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8270999999999998e-05 2024-04-22T18:39:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:39:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.506386606, 2024-04-22T18:39:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20418823899999999, 2024-04-22T18:39:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.834 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4 2024-04-22T18:39:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:39:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:39:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295011+0000", 2024-04-22T18:39:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295016+0000", 2024-04-22T18:39:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2400000000000001e-06 2024-04-22T18:39:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295022+0000", 2024-04-22T18:39:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8519999999999998e-06 2024-04-22T18:39:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295027+0000", 2024-04-22T18:39:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9749999999999998e-06 2024-04-22T18:39:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297126+0000", 2024-04-22T18:39:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020981070000000001 2024-04-22T18:39:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297167+0000", 2024-04-22T18:39:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1842999999999999e-05 2024-04-22T18:39:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499096+0000", 2024-04-22T18:39:12.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201928846 2024-04-22T18:39:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499142+0000", 2024-04-22T18:39:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6152000000000001e-05 2024-04-22T18:39:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499201+0000", 2024-04-22T18:39:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8190999999999999e-05 2024-04-22T18:39:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.839 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:81 4.8 4:1bc4668a:::smithi08937929-21:head [write 736140~537686,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.549839+0000", 2024-04-22T18:39:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.251559609000001, 2024-04-22T18:39:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20180494099999999, 2024-04-22T18:39:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 81 2024-04-22T18:39:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.840 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.549839+0000", 2024-04-22T18:39:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.549839+0000", 2024-04-22T18:39:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.549837+0000", 2024-04-22T18:39:12.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.550355+0000", 2024-04-22T18:39:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051818999999999997 2024-04-22T18:39:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.550370+0000", 2024-04-22T18:39:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5073e-05 2024-04-22T18:39:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.550378+0000", 2024-04-22T18:39:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9580000000000007e-06 2024-04-22T18:39:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.650511+0000", 2024-04-22T18:39:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100132675 2024-04-22T18:39:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.650590+0000", 2024-04-22T18:39:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8949999999999995e-05 2024-04-22T18:39:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.650684+0000", 2024-04-22T18:39:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4846999999999995e-05 2024-04-22T18:39:12.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.652918+0000", 2024-04-22T18:39:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022340049999999998 2024-04-22T18:39:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.751590+0000", 2024-04-22T18:39:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098671777000000002 2024-04-22T18:39:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.751620+0000", 2024-04-22T18:39:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0190999999999999e-05 2024-04-22T18:39:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.751644+0000", 2024-04-22T18:39:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3841000000000001e-05 2024-04-22T18:39:12.846 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:82 4.8 4:1bc4668a:::smithi08937929-21:head [write 2020581~315559] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.555572+0000", 2024-04-22T18:39:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.245827116000001, 2024-04-22T18:39:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19891041200000001, 2024-04-22T18:39:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.847 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 82 2024-04-22T18:39:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555572+0000", 2024-04-22T18:39:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555572+0000", 2024-04-22T18:39:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555570+0000", 2024-04-22T18:39:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555825+0000", 2024-04-22T18:39:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00025443699999999999 2024-04-22T18:39:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555836+0000", 2024-04-22T18:39:12.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1693e-05 2024-04-22T18:39:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555843+0000", 2024-04-22T18:39:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.861e-06 2024-04-22T18:39:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.651510+0000", 2024-04-22T18:39:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.095666449000000001 2024-04-22T18:39:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.651598+0000", 2024-04-22T18:39:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7989000000000001e-05 2024-04-22T18:39:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.651726+0000", 2024-04-22T18:39:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012811100000000001 2024-04-22T18:39:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.653977+0000", 2024-04-22T18:39:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002251074 2024-04-22T18:39:12.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754316+0000", 2024-04-22T18:39:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10033908599999999 2024-04-22T18:39:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754449+0000", 2024-04-22T18:39:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000133497 2024-04-22T18:39:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754482+0000", 2024-04-22T18:39:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2830000000000002e-05 2024-04-22T18:39:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:83 4.8 4:1bc4668a:::smithi08937929-21:head [setxattr _header (56) in=7b,truncate 2336140] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.555848+0000", 2024-04-22T18:39:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.245550626, 2024-04-22T18:39:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19872846999999999, 2024-04-22T18:39:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.853 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 83 2024-04-22T18:39:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555848+0000", 2024-04-22T18:39:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555848+0000", 2024-04-22T18:39:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555848+0000", 2024-04-22T18:39:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555852+0000", 2024-04-22T18:39:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.69e-06 2024-04-22T18:39:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555856+0000", 2024-04-22T18:39:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.055e-06 2024-04-22T18:39:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555859+0000", 2024-04-22T18:39:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1269999999999999e-06 2024-04-22T18:39:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.652495+0000", 2024-04-22T18:39:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.096636298999999995 2024-04-22T18:39:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.652599+0000", 2024-04-22T18:39:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010438300000000001 2024-04-22T18:39:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.652736+0000", 2024-04-22T18:39:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013642600000000001 2024-04-22T18:39:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.654000+0000", 2024-04-22T18:39:12.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0012638370000000001 2024-04-22T18:39:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754516+0000", 2024-04-22T18:39:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100516297 2024-04-22T18:39:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754547+0000", 2024-04-22T18:39:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0673000000000003e-05 2024-04-22T18:39:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754577+0000", 2024-04-22T18:39:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0207999999999999e-05 2024-04-22T18:39:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:84 4.8 4:1bc4668a:::smithi08937929-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.555861+0000", 2024-04-22T18:39:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.245538021000002, 2024-04-22T18:39:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.198839506, 2024-04-22T18:39:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.859 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 84 2024-04-22T18:39:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555861+0000", 2024-04-22T18:39:12.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555861+0000", 2024-04-22T18:39:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555861+0000", 2024-04-22T18:39:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555862+0000", 2024-04-22T18:39:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.494e-06 2024-04-22T18:39:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555865+0000", 2024-04-22T18:39:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.954e-06 2024-04-22T18:39:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.555870+0000", 2024-04-22T18:39:12.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5070000000000002e-06 2024-04-22T18:39:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.652949+0000", 2024-04-22T18:39:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.097078988000000005 2024-04-22T18:39:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.653026+0000", 2024-04-22T18:39:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7313999999999999e-05 2024-04-22T18:39:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754603+0000", 2024-04-22T18:39:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101577218 2024-04-22T18:39:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754647+0000", 2024-04-22T18:39:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3850999999999997e-05 2024-04-22T18:39:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.754700+0000", 2024-04-22T18:39:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3489000000000002e-05 2024-04-22T18:39:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:70 4.3 e27/20 4:c6fa81cf:::smithi08937929-18:head v 27'8, mlcod=27'3)", 2024-04-22T18:39:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.843906690000001, 2024-04-22T18:39:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20440983300000001, 2024-04-22T18:39:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.865 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:12.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957839+0000", 2024-04-22T18:39:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000346981 2024-04-22T18:39:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957849+0000", 2024-04-22T18:39:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8060000000000006e-06 2024-04-22T18:39:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957856+0000", 2024-04-22T18:39:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0319999999999998e-06 2024-04-22T18:39:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958600+0000", 2024-04-22T18:39:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074474700000000003 2024-04-22T18:39:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958615+0000", 2024-04-22T18:39:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4961e-05 2024-04-22T18:39:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161867+0000", 2024-04-22T18:39:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20325188399999999 2024-04-22T18:39:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161902+0000", 2024-04-22T18:39:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4884e-05 2024-04-22T18:39:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:71 4.3 e27/20 4:c6fa81cf:::smithi08937929-18:head v 27'9, mlcod=27'3)", 2024-04-22T18:39:12.869 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:39:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.843536667999999, 2024-04-22T18:39:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20407488600000001, 2024-04-22T18:39:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:39:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:39:12.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:39:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957867+0000", 2024-04-22T18:39:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8329999999999999e-06 2024-04-22T18:39:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957872+0000", 2024-04-22T18:39:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4530000000000004e-06 2024-04-22T18:39:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957877+0000", 2024-04-22T18:39:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8969999999999999e-06 2024-04-22T18:39:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959214+0000", 2024-04-22T18:39:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001337443 2024-04-22T18:39:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959236+0000", 2024-04-22T18:39:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1889e-05 2024-04-22T18:39:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161910+0000", 2024-04-22T18:39:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202673879 2024-04-22T18:39:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161937+0000", 2024-04-22T18:39:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7202000000000001e-05 2024-04-22T18:39:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.873 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:97 4.3 e27/20 4:c93c842e:::smithi08937929-25:head v 27'10, mlcod=27'3)", 2024-04-22T18:39:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:39:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.843518608, 2024-04-22T18:39:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.204078805, 2024-04-22T18:39:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:39:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:39:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:39:12.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958337+0000", 2024-04-22T18:39:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045682399999999999 2024-04-22T18:39:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958346+0000", 2024-04-22T18:39:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6840000000000008e-06 2024-04-22T18:39:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958353+0000", 2024-04-22T18:39:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0330000000000003e-06 2024-04-22T18:39:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959462+0000", 2024-04-22T18:39:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011084440000000001 2024-04-22T18:39:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959479+0000", 2024-04-22T18:39:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6654e-05 2024-04-22T18:39:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161940+0000", 2024-04-22T18:39:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202461112 2024-04-22T18:39:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161959+0000", 2024-04-22T18:39:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9477999999999999e-05 2024-04-22T18:39:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:98 4.3 e27/20 4:c93c842e:::smithi08937929-25:head v 27'11, mlcod=27'3)", 2024-04-22T18:39:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:39:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.843040503000001, 2024-04-22T18:39:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203627161, 2024-04-22T18:39:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:39:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:39:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:39:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958880+0000", 2024-04-22T18:39:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052192199999999997 2024-04-22T18:39:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958887+0000", 2024-04-22T18:39:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:39:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958891+0000", 2024-04-22T18:39:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2899999999999996e-06 2024-04-22T18:39:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.060417+0000", 2024-04-22T18:39:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10152593 2024-04-22T18:39:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.060433+0000", 2024-04-22T18:39:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6232e-05 2024-04-22T18:39:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161964+0000", 2024-04-22T18:39:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101531234 2024-04-22T18:39:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161986+0000", 2024-04-22T18:39:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1090999999999998e-05 2024-04-22T18:39:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:99 4.3 e27/20 4:c93c842e:::smithi08937929-25:head v 27'12, mlcod=27'3)", 2024-04-22T18:39:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:39:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.842504588000001, 2024-04-22T18:39:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20320666600000001, 2024-04-22T18:39:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:39:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:39:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:39:12.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958897+0000", 2024-04-22T18:39:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4429999999999999e-06 2024-04-22T18:39:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958900+0000", 2024-04-22T18:39:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4490000000000002e-06 2024-04-22T18:39:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958902+0000", 2024-04-22T18:39:12.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.322e-06 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161689+0000", 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20278693 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161712+0000", 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2938999999999998e-05 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.162077+0000", 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036527800000000002 2024-04-22T18:39:12.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.162101+0000", 2024-04-22T18:39:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3785e-05 2024-04-22T18:39:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.808041382999999, 2024-04-22T18:39:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:39:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.887 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:39:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:39:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:39:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:39:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:39:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:39:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:39:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:39:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:39:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:39:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:39:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:39:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:39:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:39:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:39:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:39:12.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:12.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:39:12.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:39:12.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:39:12.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:39:12.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:39:12.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:39:12.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:39:12.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:39:12.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:39:12.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:39:12.893 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.893 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.893 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.894 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:27 3.2 3:7a6cba00:::smithi08937929-32 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:12.894 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.597045+0000", 2024-04-22T18:39:12.894 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.20435389, 2024-04-22T18:39:12.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20443814799999999, 2024-04-22T18:39:12.894 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.894 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.894 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597045+0000", 2024-04-22T18:39:12.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597045+0000", 2024-04-22T18:39:12.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597043+0000", 2024-04-22T18:39:12.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:12.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597053+0000", 2024-04-22T18:39:12.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0278999999999999e-05 2024-04-22T18:39:12.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597074+0000", 2024-04-22T18:39:12.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0710999999999999e-05 2024-04-22T18:39:12.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.697204+0000", 2024-04-22T18:39:12.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10012958299999999 2024-04-22T18:39:12.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.698362+0000", 2024-04-22T18:39:12.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011587629999999999 2024-04-22T18:39:12.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.698440+0000", 2024-04-22T18:39:12.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7453000000000005e-05 2024-04-22T18:39:12.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:12.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.698593+0000", 2024-04-22T18:39:12.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153449 2024-04-22T18:39:12.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.699048+0000", 2024-04-22T18:39:12.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000454287 2024-04-22T18:39:12.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.801433+0000", 2024-04-22T18:39:12.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102385664 2024-04-22T18:39:12.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.801462+0000", 2024-04-22T18:39:12.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9153e-05 2024-04-22T18:39:12.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.801483+0000", 2024-04-22T18:39:12.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0707e-05 2024-04-22T18:39:12.899 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.899 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.899 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.899 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:12.899 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:12.899 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.699741592000001, 2024-04-22T18:39:12.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:39:12.900 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.900 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.900 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:12.900 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:12.900 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:12.900 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:39:12.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.900 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:12.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:12.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:39:12.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:12.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:39:12.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:39:12.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:39:12.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:39:12.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:39:12.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:39:12.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:39:12.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:39:12.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:39:12.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:39:12.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:12.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:39:12.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:39:12.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:39:12.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:39:12.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:39:12.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:39:12.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:39:12.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:39:12.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:39:12.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:39:12.905 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.905 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.905 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.905 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:70 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:12.905 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:02.527752+0000", 2024-04-22T18:39:12.905 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 10.273647298, 2024-04-22T18:39:12.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20158322100000001, 2024-04-22T18:39:12.905 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:12.905 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:12.906 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:12.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:12.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527752+0000", 2024-04-22T18:39:12.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:12.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527752+0000", 2024-04-22T18:39:12.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:12.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:12.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527750+0000", 2024-04-22T18:39:12.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:12.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:12.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527766+0000", 2024-04-22T18:39:12.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6398000000000001e-05 2024-04-22T18:39:12.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:12.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527780+0000", 2024-04-22T18:39:12.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.327e-05 2024-04-22T18:39:12.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:12.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527788+0000", 2024-04-22T18:39:12.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1729999999999995e-06 2024-04-22T18:39:12.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:12.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527831+0000", 2024-04-22T18:39:12.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3683000000000002e-05 2024-04-22T18:39:12.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:12.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.527949+0000", 2024-04-22T18:39:12.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117271 2024-04-22T18:39:12.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:12.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.528131+0000", 2024-04-22T18:39:12.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000182525 2024-04-22T18:39:12.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:12.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.528750+0000", 2024-04-22T18:39:12.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00061904199999999999 2024-04-22T18:39:12.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:12.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.729283+0000", 2024-04-22T18:39:12.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20053246599999999 2024-04-22T18:39:12.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:12.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.729314+0000", 2024-04-22T18:39:12.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0889999999999997e-05 2024-04-22T18:39:12.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:12.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:12.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:12.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:02.729335+0000", 2024-04-22T18:39:12.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1231999999999999e-05 2024-04-22T18:39:12.910 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.910 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.911 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.911 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:12.911 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:12.911 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:12.913 INFO:tasks.rados.rados.0.smithi089.stdout:717: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-22T18:39:12.913 INFO:tasks.rados.rados.0.smithi089.stdout:714: got (0) Success 2024-04-22T18:39:12.913 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 9 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-22T18:39:12.913 INFO:tasks.rados.rados.0.smithi089.stdout:706: done (15 left) 2024-04-22T18:39:12.913 INFO:tasks.rados.rados.0.smithi089.stdout:711: done (14 left) 2024-04-22T18:39:12.913 INFO:tasks.rados.rados.0.smithi089.stdout:712: done (13 left) 2024-04-22T18:39:12.913 INFO:tasks.rados.rados.0.smithi089.stdout:713: done (12 left) 2024-04-22T18:39:12.913 INFO:tasks.rados.rados.0.smithi089.stdout:714: done (11 left) 2024-04-22T18:39:12.913 INFO:tasks.rados.rados.0.smithi089.stdout:717: done (10 left) 2024-04-22T18:39:12.913 INFO:tasks.rados.rados.0.smithi089.stdout:725: tier_promote oid 13 2024-04-22T18:39:12.913 INFO:tasks.rados.rados.0.smithi089.stdout:721: finishing set_chunk to oid 49 2024-04-22T18:39:12.914 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 733696 length: 1769472 2024-04-22T18:39:12.914 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:39:12.914 INFO:tasks.rados.rados.0.smithi089.stdout:716: finishing set_chunk to oid 15 2024-04-22T18:39:12.914 INFO:tasks.rados.rados.0.smithi089.stdout:716: got expected ENOENT (src dne) 2024-04-22T18:39:12.914 INFO:tasks.rados.rados.0.smithi089.stdout:722: expect (ObjNum 74 snap 0 seq_num 74) 2024-04-22T18:39:12.914 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:12.913+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:12.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:12.995 INFO:tasks.rados.rados.0.smithi089.stdout:723: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-22T18:39:13.013 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:13.014+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:13.016 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:13.016 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:13.016 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:13.017 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:13.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:13.072 INFO:tasks.rados.rados.0.smithi089.stdout:720: finishing write tid 2 to smithi08937929-42 2024-04-22T18:39:13.072 INFO:tasks.rados.rados.0.smithi089.stdout:715: finishing write tid 1 to smithi08937929-4 2024-04-22T18:39:13.072 INFO:tasks.rados.rados.0.smithi089.stdout:715: finishing write tid 2 to smithi08937929-4 2024-04-22T18:39:13.073 INFO:tasks.rados.rados.0.smithi089.stdout:715: finishing write tid 3 to smithi08937929-4 2024-04-22T18:39:13.073 INFO:tasks.rados.rados.0.smithi089.stdout:715: finishing write tid 4 to smithi08937929-4 2024-04-22T18:39:13.073 INFO:tasks.rados.rados.0.smithi089.stdout:715: finishing write tid 5 to smithi08937929-4 2024-04-22T18:39:13.073 INFO:tasks.rados.rados.0.smithi089.stdout:715: oid 4 updating version 0 to 36 2024-04-22T18:39:13.073 INFO:tasks.rados.rados.0.smithi089.stdout:715: oid 4 updating version 36 to 39 2024-04-22T18:39:13.073 INFO:tasks.rados.rados.0.smithi089.stdout:715: oid 4 version 39 is already newer than 37 2024-04-22T18:39:13.073 INFO:tasks.rados.rados.0.smithi089.stdout:715: oid 4 version 39 is already newer than 38 2024-04-22T18:39:13.073 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 39 (ObjNum 81 snap 0 seq_num 81) dirty exists 2024-04-22T18:39:13.073 INFO:tasks.rados.rados.0.smithi089.stdout:715: left oid 4 (ObjNum 81 snap 0 seq_num 81) 2024-04-22T18:39:13.073 INFO:tasks.rados.rados.0.smithi089.stdout:720: finishing write tid 3 to smithi08937929-42 2024-04-22T18:39:13.073 INFO:tasks.rados.rados.0.smithi089.stdout:715: done (10 left) 2024-04-22T18:39:13.073 INFO:tasks.rados.rados.0.smithi089.stdout:716: done (9 left) 2024-04-22T18:39:13.073 INFO:tasks.rados.rados.0.smithi089.stdout:721: done (8 left) 2024-04-22T18:39:13.074 INFO:tasks.rados.rados.0.smithi089.stdout:722: done (7 left) 2024-04-22T18:39:13.074 INFO:tasks.rados.rados.0.smithi089.stdout:723: done (6 left) 2024-04-22T18:39:13.074 INFO:tasks.rados.rados.0.smithi089.stdout:724: done (5 left) 2024-04-22T18:39:13.074 INFO:tasks.rados.rados.0.smithi089.stdout:726: set_chunk oid 8 target oid 2024-04-22T18:39:13.074 INFO:tasks.rados.rados.0.smithi089.stdout:726: set_chunk oid 8 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:39:13.074 INFO:tasks.rados.rados.0.smithi089.stdout:727: set_chunk oid 18 target oid 2024-04-22T18:39:13.113 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:13.114+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:13.166 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:13.166 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:13.166 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:13.166 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:13.167 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:13.175 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:13.200 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:39:13.216 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:13.217+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:13.278 INFO:tasks.rados.rados.0.smithi089.stdout:727: set_chunk oid 18 offset: 1469952 length: 667136 target oid 4a159391123755d93670b0d142ca36f3ee9a35236f6c4396c2aa92521a42a06b offset: 0 snap -1 2024-04-22T18:39:13.278 INFO:tasks.rados.rados.0.smithi089.stdout:728: read oid 38 snap -1 2024-04-22T18:39:13.278 INFO:tasks.rados.rados.0.smithi089.stdout:728: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-22T18:39:13.278 INFO:tasks.rados.rados.0.smithi089.stdout:729: write (excl) oid 37 current snap is 0 2024-04-22T18:39:13.278 INFO:tasks.rados.rados.0.smithi089.stdout:720: finishing write tid 4 to smithi08937929-42 2024-04-22T18:39:13.278 INFO:tasks.rados.rados.0.smithi089.stdout:729: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-04-22T18:39:13.283 INFO:tasks.rados.rados.0.smithi089.stdout:729: writing smithi08937929-37 from 611788 to 1012587 tid 1 2024-04-22T18:39:13.289 INFO:tasks.rados.rados.0.smithi089.stdout:729: writing smithi08937929-37 from 1748958 to 2237644 tid 2 2024-04-22T18:39:13.297 INFO:tasks.rados.rados.0.smithi089.stdout:729: writing smithi08937929-37 from 2907296 to 3411788 tid 3 2024-04-22T18:39:13.297 INFO:tasks.rados.rados.0.smithi089.stdout:730: delete oid 15 current snap is 0 2024-04-22T18:39:13.297 INFO:tasks.rados.rados.0.smithi089.stdout:720: finishing write tid 5 to smithi08937929-42 2024-04-22T18:39:13.297 INFO:tasks.rados.rados.0.smithi089.stdout:720: oid 42 updating version 0 to 7 2024-04-22T18:39:13.297 INFO:tasks.rados.rados.0.smithi089.stdout:720: oid 42 version 7 is already newer than 5 2024-04-22T18:39:13.297 INFO:tasks.rados.rados.0.smithi089.stdout:720: oid 42 updating version 7 to 8 2024-04-22T18:39:13.297 INFO:tasks.rados.rados.0.smithi089.stdout:720: oid 42 version 8 is already newer than 6 2024-04-22T18:39:13.297 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 8 (ObjNum 83 snap 0 seq_num 83) dirty exists 2024-04-22T18:39:13.297 INFO:tasks.rados.rados.0.smithi089.stdout:720: left oid 42 (ObjNum 83 snap 0 seq_num 83) 2024-04-22T18:39:13.298 INFO:tasks.rados.rados.0.smithi089.stdout:710: finishing write tid 1 to smithi08937929-36 2024-04-22T18:39:13.298 INFO:tasks.rados.rados.0.smithi089.stdout:710: finishing write tid 2 to smithi08937929-36 2024-04-22T18:39:13.298 INFO:tasks.rados.rados.0.smithi089.stdout:710: finishing write tid 3 to smithi08937929-36 2024-04-22T18:39:13.298 INFO:tasks.rados.rados.0.smithi089.stdout:710: finishing write tid 4 to smithi08937929-36 2024-04-22T18:39:13.298 INFO:tasks.rados.rados.0.smithi089.stdout:710: finishing write tid 5 to smithi08937929-36 2024-04-22T18:39:13.298 INFO:tasks.rados.rados.0.smithi089.stdout:710: oid 36 updating version 0 to 25 2024-04-22T18:39:13.298 INFO:tasks.rados.rados.0.smithi089.stdout:710: oid 36 version 25 is already newer than 24 2024-04-22T18:39:13.298 INFO:tasks.rados.rados.0.smithi089.stdout:710: oid 36 updating version 25 to 26 2024-04-22T18:39:13.298 INFO:tasks.rados.rados.0.smithi089.stdout:710: oid 36 updating version 26 to 27 2024-04-22T18:39:13.298 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 27 (ObjNum 80 snap 0 seq_num 80) dirty exists 2024-04-22T18:39:13.298 INFO:tasks.rados.rados.0.smithi089.stdout:710: left oid 36 (ObjNum 80 snap 0 seq_num 80) 2024-04-22T18:39:13.298 INFO:tasks.rados.rados.0.smithi089.stdout:718: finishing write tid 1 to smithi08937929-23 2024-04-22T18:39:13.298 INFO:tasks.rados.rados.0.smithi089.stdout:718: finishing write tid 2 to smithi08937929-23 2024-04-22T18:39:13.299 INFO:tasks.rados.rados.0.smithi089.stdout:718: finishing write tid 3 to smithi08937929-23 2024-04-22T18:39:13.299 INFO:tasks.rados.rados.0.smithi089.stdout:719: got (22) Invalid argument 2024-04-22T18:39:13.299 INFO:tasks.rados.rados.0.smithi089.stdout:725: got (2) No such file or directory 2024-04-22T18:39:13.299 INFO:tasks.rados.rados.0.smithi089.stdout:725: got expected ENOENT (src dne) 2024-04-22T18:39:13.299 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 24 (ObjNum 129 snap 0 seq_num 2410262096) clean dne 2024-04-22T18:39:13.299 INFO:tasks.rados.rados.0.smithi089.stdout:718: finishing write tid 4 to smithi08937929-23 2024-04-22T18:39:13.299 INFO:tasks.rados.rados.0.smithi089.stdout:718: finishing write tid 5 to smithi08937929-23 2024-04-22T18:39:13.299 INFO:tasks.rados.rados.0.smithi089.stdout:718: oid 23 updating version 0 to 47 2024-04-22T18:39:13.299 INFO:tasks.rados.rados.0.smithi089.stdout:718: oid 23 updating version 47 to 48 2024-04-22T18:39:13.299 INFO:tasks.rados.rados.0.smithi089.stdout:718: oid 23 version 48 is already newer than 46 2024-04-22T18:39:13.299 INFO:tasks.rados.rados.0.smithi089.stdout:718: oid 23 version 48 is already newer than 45 2024-04-22T18:39:13.299 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 48 (ObjNum 82 snap 0 seq_num 82) dirty exists 2024-04-22T18:39:13.299 INFO:tasks.rados.rados.0.smithi089.stdout:718: left oid 23 (ObjNum 82 snap 0 seq_num 82) 2024-04-22T18:39:13.299 INFO:tasks.rados.rados.0.smithi089.stdout:726: finishing set_chunk to oid 8 2024-04-22T18:39:13.300 INFO:tasks.rados.rados.0.smithi089.stdout:726: got expected ENOENT (src dne) 2024-04-22T18:39:13.300 INFO:tasks.rados.rados.0.smithi089.stdout:728: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-22T18:39:13.315 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:13.316+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:13.320 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:21 4.a 4:5aa97f94:::smithi08937929-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.166683825, 2024-04-22T18:39:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21044934000000001, 2024-04-22T18:39:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.320 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 21 2024-04-22T18:39:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153183+0000", 2024-04-22T18:39:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154022+0000", 2024-04-22T18:39:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00083893300000000004 2024-04-22T18:39:13.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154032+0000", 2024-04-22T18:39:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0433e-05 2024-04-22T18:39:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154051+0000", 2024-04-22T18:39:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8227e-05 2024-04-22T18:39:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154103+0000", 2024-04-22T18:39:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2491999999999998e-05 2024-04-22T18:39:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154111+0000", 2024-04-22T18:39:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3470000000000001e-06 2024-04-22T18:39:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257848+0000", 2024-04-22T18:39:13.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103737703 2024-04-22T18:39:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257981+0000", 2024-04-22T18:39:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013290399999999999 2024-04-22T18:39:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.258138+0000", 2024-04-22T18:39:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015701199999999999 2024-04-22T18:39:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.261944+0000", 2024-04-22T18:39:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038054650000000001 2024-04-22T18:39:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363523+0000", 2024-04-22T18:39:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101579239 2024-04-22T18:39:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363577+0000", 2024-04-22T18:39:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4095000000000003e-05 2024-04-22T18:39:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363635+0000", 2024-04-22T18:39:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8362999999999997e-05 2024-04-22T18:39:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:22 4.a 4:5aa97f94:::smithi08937929-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.162609848999999, 2024-04-22T18:39:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30948398900000001, 2024-04-22T18:39:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.327 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 22 2024-04-22T18:39:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157259+0000", 2024-04-22T18:39:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157318+0000", 2024-04-22T18:39:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8906000000000001e-05 2024-04-22T18:39:13.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157327+0000", 2024-04-22T18:39:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:39:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157332+0000", 2024-04-22T18:39:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4689999999999996e-06 2024-04-22T18:39:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257479+0000", 2024-04-22T18:39:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014700999999999 2024-04-22T18:39:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257492+0000", 2024-04-22T18:39:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2574e-05 2024-04-22T18:39:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259582+0000", 2024-04-22T18:39:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020904019999999999 2024-04-22T18:39:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259668+0000", 2024-04-22T18:39:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5826000000000001e-05 2024-04-22T18:39:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259772+0000", 2024-04-22T18:39:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104048 2024-04-22T18:39:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363453+0000", 2024-04-22T18:39:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103680183 2024-04-22T18:39:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466692+0000", 2024-04-22T18:39:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10323948199999999 2024-04-22T18:39:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466720+0000", 2024-04-22T18:39:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7886e-05 2024-04-22T18:39:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466744+0000", 2024-04-22T18:39:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4049e-05 2024-04-22T18:39:13.333 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:23 4.a 4:5aa97f94:::smithi08937929-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.162532162000002, 2024-04-22T18:39:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39003820300000003, 2024-04-22T18:39:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.334 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 23 2024-04-22T18:39:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157337+0000", 2024-04-22T18:39:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157340+0000", 2024-04-22T18:39:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8609999999999998e-06 2024-04-22T18:39:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157343+0000", 2024-04-22T18:39:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9859999999999999e-06 2024-04-22T18:39:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157346+0000", 2024-04-22T18:39:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8119999999999999e-06 2024-04-22T18:39:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257563+0000", 2024-04-22T18:39:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100217497 2024-04-22T18:39:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257574+0000", 2024-04-22T18:39:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0777e-05 2024-04-22T18:39:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260112+0000", 2024-04-22T18:39:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002537681 2024-04-22T18:39:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260191+0000", 2024-04-22T18:39:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9320999999999995e-05 2024-04-22T18:39:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260333+0000", 2024-04-22T18:39:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014177699999999999 2024-04-22T18:39:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363792+0000", 2024-04-22T18:39:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103458862 2024-04-22T18:39:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547303+0000", 2024-04-22T18:39:13.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18351162100000001 2024-04-22T18:39:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547337+0000", 2024-04-22T18:39:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3640999999999999e-05 2024-04-22T18:39:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547376+0000", 2024-04-22T18:39:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8915000000000003e-05 2024-04-22T18:39:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:24 4.a 4:5aa97f94:::smithi08937929-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.162505731, 2024-04-22T18:39:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39017823499999998, 2024-04-22T18:39:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 24 2024-04-22T18:39:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.341 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157366+0000", 2024-04-22T18:39:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3120000000000001e-06 2024-04-22T18:39:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157370+0000", 2024-04-22T18:39:13.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1300000000000003e-06 2024-04-22T18:39:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157373+0000", 2024-04-22T18:39:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-04-22T18:39:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260553+0000", 2024-04-22T18:39:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103179996 2024-04-22T18:39:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260625+0000", 2024-04-22T18:39:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2236000000000004e-05 2024-04-22T18:39:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547414+0000", 2024-04-22T18:39:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.286789181 2024-04-22T18:39:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547471+0000", 2024-04-22T18:39:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6192e-05 2024-04-22T18:39:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547542+0000", 2024-04-22T18:39:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1622e-05 2024-04-22T18:39:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:49 4.a 4:5d2aee44:::smithi08937929-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.085707072999998, 2024-04-22T18:39:13.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31344442500000003, 2024-04-22T18:39:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 49 2024-04-22T18:39:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234158+0000", 2024-04-22T18:39:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999957 2024-04-22T18:39:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234901+0000", 2024-04-22T18:39:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074250800000000001 2024-04-22T18:39:13.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234914+0000", 2024-04-22T18:39:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3244999999999999e-05 2024-04-22T18:39:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234922+0000", 2024-04-22T18:39:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4590000000000001e-06 2024-04-22T18:39:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260665+0000", 2024-04-22T18:39:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025743423000000001 2024-04-22T18:39:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260779+0000", 2024-04-22T18:39:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114063 2024-04-22T18:39:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260913+0000", 2024-04-22T18:39:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013355499999999999 2024-04-22T18:39:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363798+0000", 2024-04-22T18:39:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102885003 2024-04-22T18:39:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547567+0000", 2024-04-22T18:39:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.183768971 2024-04-22T18:39:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547589+0000", 2024-04-22T18:39:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2507999999999999e-05 2024-04-22T18:39:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547607+0000", 2024-04-22T18:39:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8009999999999999e-05 2024-04-22T18:39:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:50 4.a 4:5d2aee44:::smithi08937929-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.080260893999998, 2024-04-22T18:39:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40856295399999998, 2024-04-22T18:39:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.352 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 50 2024-04-22T18:39:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239607+0000", 2024-04-22T18:39:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240309+0000", 2024-04-22T18:39:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070201599999999997 2024-04-22T18:39:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240320+0000", 2024-04-22T18:39:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1024e-05 2024-04-22T18:39:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240327+0000", 2024-04-22T18:39:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0049999999999998e-06 2024-04-22T18:39:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362065+0000", 2024-04-22T18:39:13.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.121737812 2024-04-22T18:39:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362121+0000", 2024-04-22T18:39:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5868000000000001e-05 2024-04-22T18:39:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362210+0000", 2024-04-22T18:39:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8533000000000001e-05 2024-04-22T18:39:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365322+0000", 2024-04-22T18:39:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031121220000000002 2024-04-22T18:39:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648054+0000", 2024-04-22T18:39:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.282732549 2024-04-22T18:39:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648105+0000", 2024-04-22T18:39:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1276999999999997e-05 2024-04-22T18:39:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648172+0000", 2024-04-22T18:39:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6398999999999999e-05 2024-04-22T18:39:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:51 4.a 4:5d2aee44:::smithi08937929-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.079538360000001, 2024-04-22T18:39:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40793057199999999, 2024-04-22T18:39:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.358 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.359 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.359 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.359 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 51 2024-04-22T18:39:13.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.359 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:13.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:13.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:13.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240336+0000", 2024-04-22T18:39:13.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:39:13.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240339+0000", 2024-04-22T18:39:13.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9009999999999998e-06 2024-04-22T18:39:13.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240342+0000", 2024-04-22T18:39:13.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:39:13.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363080+0000", 2024-04-22T18:39:13.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122738075 2024-04-22T18:39:13.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363164+0000", 2024-04-22T18:39:13.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3722999999999998e-05 2024-04-22T18:39:13.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:13.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363287+0000", 2024-04-22T18:39:13.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012301099999999999 2024-04-22T18:39:13.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466753+0000", 2024-04-22T18:39:13.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103466217 2024-04-22T18:39:13.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648205+0000", 2024-04-22T18:39:13.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.181451788 2024-04-22T18:39:13.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648231+0000", 2024-04-22T18:39:13.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6597999999999998e-05 2024-04-22T18:39:13.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648262+0000", 2024-04-22T18:39:13.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0871000000000003e-05 2024-04-22T18:39:13.364 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.364 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.364 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.364 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:52 4.a 4:5d2aee44:::smithi08937929-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.364 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:13.364 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.079525901, 2024-04-22T18:39:13.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40812731200000002, 2024-04-22T18:39:13.364 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.364 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:13.364 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.364 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.365 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.365 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 52 2024-04-22T18:39:13.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.365 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:13.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:13.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:13.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240345+0000", 2024-04-22T18:39:13.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4300000000000001e-06 2024-04-22T18:39:13.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240349+0000", 2024-04-22T18:39:13.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2570000000000001e-06 2024-04-22T18:39:13.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240358+0000", 2024-04-22T18:39:13.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-22T18:39:13.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363697+0000", 2024-04-22T18:39:13.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12333965299999999 2024-04-22T18:39:13.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:13.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363777+0000", 2024-04-22T18:39:13.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9883999999999996e-05 2024-04-22T18:39:13.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648296+0000", 2024-04-22T18:39:13.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451861299999998 2024-04-22T18:39:13.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648379+0000", 2024-04-22T18:39:13.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3302999999999998e-05 2024-04-22T18:39:13.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648471+0000", 2024-04-22T18:39:13.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2029999999999998e-05 2024-04-22T18:39:13.369 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.369 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.369 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.369 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.369 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:13.369 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.060222566, 2024-04-22T18:39:13.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:39:13.369 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.370 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.370 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.370 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.370 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.370 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:39:13.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.370 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:13.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:13.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:39:13.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:13.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:39:13.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:39:13.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:39:13.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:39:13.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:39:13.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:39:13.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:39:13.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:39:13.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:39:13.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:39:13.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:13.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:39:13.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:39:13.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:39:13.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:39:13.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:39:13.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:39:13.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:39:13.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:39:13.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:39:13.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:39:13.375 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.375 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.375 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.375 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.375 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:13.375 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.055440136000001, 2024-04-22T18:39:13.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:39:13.376 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.376 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.376 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.376 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.376 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.376 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:39:13.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.376 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:39:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:39:13.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:39:13.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:39:13.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:39:13.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:39:13.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:39:13.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:39:13.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:39:13.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:39:13.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:39:13.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:13.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:39:13.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:39:13.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:39:13.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:39:13.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:39:13.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:39:13.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:39:13.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:39:13.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:39:13.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:39:13.381 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.381 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.381 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.381 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.381 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:13.381 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.055345933000002, 2024-04-22T18:39:13.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:39:13.382 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.382 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.382 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.382 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.382 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.382 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:39:13.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.382 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:13.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:13.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:39:13.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:13.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:39:13.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:39:13.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:39:13.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:39:13.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:39:13.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:39:13.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:39:13.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:39:13.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:39:13.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:39:13.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:13.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:39:13.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:39:13.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:39:13.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:39:13.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:39:13.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:39:13.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:39:13.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:39:13.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:39:13.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:39:13.387 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.387 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.387 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.387 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.388 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:13.388 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.055325045, 2024-04-22T18:39:13.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:39:13.388 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.388 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:13.388 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.388 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.388 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.388 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:39:13.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.388 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:13.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:13.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:39:13.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:13.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:39:13.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:39:13.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:39:13.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:39:13.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:39:13.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:39:13.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:39:13.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:39:13.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:13.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:39:13.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:39:13.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.393 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.393 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:13.393 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.272317382000001, 2024-04-22T18:39:13.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:39:13.393 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.393 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.393 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.393 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.393 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.393 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:39:13.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.393 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:13.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:13.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:39:13.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:13.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:39:13.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:39:13.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:39:13.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:39:13.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:39:13.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:39:13.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:39:13.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:39:13.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:39:13.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:39:13.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:13.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:39:13.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:39:13.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:39:13.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:39:13.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:39:13.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:39:13.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:39:13.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:39:13.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:39:13.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:39:13.398 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.398 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.398 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.398 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.399 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:13.399 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.264485976, 2024-04-22T18:39:13.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:39:13.399 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.399 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.399 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.399 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.399 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.399 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:39:13.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.399 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:13.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:13.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:39:13.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:13.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:39:13.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:39:13.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:39:13.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:39:13.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:39:13.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:39:13.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:39:13.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:39:13.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:39:13.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:39:13.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:13.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:39:13.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:39:13.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:39:13.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:39:13.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:39:13.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:39:13.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:39:13.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:39:13.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:39:13.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:39:13.404 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.404 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.404 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.404 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:186 4.a 4:537c89f8:::smithi08937929-47:head [write 1992054~651681] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.404 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:13.404 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.916980441, 2024-04-22T18:39:13.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20325226800000001, 2024-04-22T18:39:13.405 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.405 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.405 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.405 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.405 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.405 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 186 2024-04-22T18:39:13.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.405 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:13.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:13.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402888+0000", 2024-04-22T18:39:13.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:13.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403454+0000", 2024-04-22T18:39:13.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056661000000000005 2024-04-22T18:39:13.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403465+0000", 2024-04-22T18:39:13.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0727000000000001e-05 2024-04-22T18:39:13.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403473+0000", 2024-04-22T18:39:13.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7039999999999994e-06 2024-04-22T18:39:13.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404592+0000", 2024-04-22T18:39:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001118827 2024-04-22T18:39:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404674+0000", 2024-04-22T18:39:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2793999999999994e-05 2024-04-22T18:39:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404779+0000", 2024-04-22T18:39:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104465 2024-04-22T18:39:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.406080+0000", 2024-04-22T18:39:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013006789999999999 2024-04-22T18:39:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606037+0000", 2024-04-22T18:39:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19995755600000001 2024-04-22T18:39:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606098+0000", 2024-04-22T18:39:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1075999999999996e-05 2024-04-22T18:39:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606142+0000", 2024-04-22T18:39:13.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3411999999999997e-05 2024-04-22T18:39:13.410 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.410 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.410 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.410 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:187 4.a 4:537c89f8:::smithi08937929-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:13.410 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:13.410 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.916390301, 2024-04-22T18:39:13.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203643623, 2024-04-22T18:39:13.410 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.410 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.411 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.411 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.411 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 187 2024-04-22T18:39:13.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.411 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:13.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:13.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:13.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:13.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403501+0000", 2024-04-22T18:39:13.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2146000000000001e-05 2024-04-22T18:39:13.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403506+0000", 2024-04-22T18:39:13.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3040000000000004e-06 2024-04-22T18:39:13.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403511+0000", 2024-04-22T18:39:13.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9599999999999999e-06 2024-04-22T18:39:13.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405587+0000", 2024-04-22T18:39:13.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002076296 2024-04-22T18:39:13.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405664+0000", 2024-04-22T18:39:13.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6316000000000006e-05 2024-04-22T18:39:13.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:13.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405785+0000", 2024-04-22T18:39:13.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121276 2024-04-22T18:39:13.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.408030+0000", 2024-04-22T18:39:13.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022452169999999999 2024-04-22T18:39:13.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607059+0000", 2024-04-22T18:39:13.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199029135 2024-04-22T18:39:13.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607094+0000", 2024-04-22T18:39:13.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4686e-05 2024-04-22T18:39:13.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607123+0000", 2024-04-22T18:39:13.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9113000000000001e-05 2024-04-22T18:39:13.415 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:439 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:39:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.905017+0000", 2024-04-22T18:39:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.414852887, 2024-04-22T18:39:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.200452043, 2024-04-22T18:39:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 439 2024-04-22T18:39:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.905017+0000", 2024-04-22T18:39:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.905017+0000", 2024-04-22T18:39:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.905015+0000", 2024-04-22T18:39:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.905023+0000", 2024-04-22T18:39:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2539999999999992e-06 2024-04-22T18:39:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.905035+0000", 2024-04-22T18:39:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2145e-05 2024-04-22T18:39:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:00.005141+0000", 2024-04-22T18:39:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100105518 2024-04-22T18:39:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:00.105286+0000", 2024-04-22T18:39:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100145145 2024-04-22T18:39:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:00.105394+0000", 2024-04-22T18:39:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010877 2024-04-22T18:39:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:00.105469+0000", 2024-04-22T18:39:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4445999999999998e-05 2024-04-22T18:39:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:811 4.5 4:a4a14dff:::smithi08937929-40:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-22T18:39:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.1990478440000008, 2024-04-22T18:39:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242306563, 2024-04-22T18:39:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 811 2024-04-22T18:39:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120821+0000", 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120825+0000", 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.506e-06 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120829+0000", 2024-04-22T18:39:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0849999999999999e-06 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120833+0000", 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9609999999999999e-06 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221263+0000", 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10043045 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221337+0000", 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4010000000000005e-05 2024-04-22T18:39:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221448+0000", 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011065699999999999 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.222541+0000", 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010932610000000001 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.362952+0000", 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.140410535 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.363102+0000", 2024-04-22T18:39:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015041499999999999 2024-04-22T18:39:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.363129+0000", 2024-04-22T18:39:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6574999999999999e-05 2024-04-22T18:39:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:933 4.a 4:5d2aee44:::smithi08937929-13:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:13.426 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:13.426 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.6751242059999996, 2024-04-22T18:39:13.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202744591, 2024-04-22T18:39:13.426 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.426 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:39:13.426 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.426 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.426 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.426 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 933 2024-04-22T18:39:13.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.426 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:13.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:13.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644743+0000", 2024-04-22T18:39:13.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:13.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644754+0000", 2024-04-22T18:39:13.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0807e-05 2024-04-22T18:39:13.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644768+0000", 2024-04-22T18:39:13.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3844e-05 2024-04-22T18:39:13.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.744860+0000", 2024-04-22T18:39:13.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100092835 2024-04-22T18:39:13.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.745084+0000", 2024-04-22T18:39:13.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022401699999999999 2024-04-22T18:39:13.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:13.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.745090+0000", 2024-04-22T18:39:13.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2970000000000003e-06 2024-04-22T18:39:13.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.847381+0000", 2024-04-22T18:39:13.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102291544 2024-04-22T18:39:13.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.847490+0000", 2024-04-22T18:39:13.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010891399999999999 2024-04-22T18:39:13.430 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.430 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.430 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.431 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:941 4.a 4:537c89f8:::smithi08937929-47:head [sparse-read 0~1211253] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:13.431 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:08.141214+0000", 2024-04-22T18:39:13.431 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.1786557890000005, 2024-04-22T18:39:13.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20061749400000001, 2024-04-22T18:39:13.431 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.431 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:13.431 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.431 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:13.431 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:13.431 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 941 2024-04-22T18:39:13.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.431 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141214+0000", 2024-04-22T18:39:13.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141214+0000", 2024-04-22T18:39:13.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141213+0000", 2024-04-22T18:39:13.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:13.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141219+0000", 2024-04-22T18:39:13.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6189999999999999e-06 2024-04-22T18:39:13.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141225+0000", 2024-04-22T18:39:13.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0469999999999999e-06 2024-04-22T18:39:13.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141233+0000", 2024-04-22T18:39:13.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7740000000000004e-06 2024-04-22T18:39:13.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.341440+0000", 2024-04-22T18:39:13.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.200207095 2024-04-22T18:39:13.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.341535+0000", 2024-04-22T18:39:13.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5159000000000002e-05 2024-04-22T18:39:13.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.341831+0000", 2024-04-22T18:39:13.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00029652699999999999 2024-04-22T18:39:13.434 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.435 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.435 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.435 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.435 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.435 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:13.439 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:13.440 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:13.439+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:13.441 INFO:tasks.rados.rados.0.smithi089.stdout:710: done (9 left) 2024-04-22T18:39:13.441 INFO:tasks.rados.rados.0.smithi089.stdout:718: done (8 left) 2024-04-22T18:39:13.441 INFO:tasks.rados.rados.0.smithi089.stdout:719: done (7 left) 2024-04-22T18:39:13.441 INFO:tasks.rados.rados.0.smithi089.stdout:720: done (6 left) 2024-04-22T18:39:13.441 INFO:tasks.rados.rados.0.smithi089.stdout:725: done (5 left) 2024-04-22T18:39:13.441 INFO:tasks.rados.rados.0.smithi089.stdout:726: done (4 left) 2024-04-22T18:39:13.441 INFO:tasks.rados.rados.0.smithi089.stdout:728: done (3 left) 2024-04-22T18:39:13.441 INFO:tasks.rados.rados.0.smithi089.stdout:730: done (2 left) 2024-04-22T18:39:13.441 INFO:tasks.rados.rados.0.smithi089.stdout:731: read oid 23 snap -1 2024-04-22T18:39:13.441 INFO:tasks.rados.rados.0.smithi089.stdout:731: expect (ObjNum 82 snap 0 seq_num 82) 2024-04-22T18:39:13.441 INFO:tasks.rados.rados.0.smithi089.stdout:732: write oid 11 current snap is 0 2024-04-22T18:39:13.441 INFO:tasks.rados.rados.0.smithi089.stdout:732: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-04-22T18:39:13.441 INFO:tasks.rados.rados.0.smithi089.stdout:732: writing smithi08937929-11 from 645402 to 1196699 tid 1 2024-04-22T18:39:13.442 INFO:tasks.rados.rados.0.smithi089.stdout:732: writing smithi08937929-11 from 1749569 to 2241924 tid 2 2024-04-22T18:39:13.442 INFO:tasks.rados.rados.0.smithi089.stdout:732: writing smithi08937929-11 from 2845789 to 3445402 tid 3 2024-04-22T18:39:13.442 INFO:tasks.rados.rados.0.smithi089.stdout:733: read oid 25 snap -1 2024-04-22T18:39:13.442 INFO:tasks.rados.rados.0.smithi089.stdout:733: expect (ObjNum 75 snap 0 seq_num 75) 2024-04-22T18:39:13.442 INFO:tasks.rados.rados.0.smithi089.stdout:734: write (excl) oid 5 current snap is 0 2024-04-22T18:39:13.442 INFO:tasks.rados.rados.0.smithi089.stdout:734: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-04-22T18:39:13.442 INFO:tasks.rados.rados.0.smithi089.stdout:734: writing smithi08937929-5 from 662209 to 1288755 tid 1 2024-04-22T18:39:13.442 INFO:tasks.rados.rados.0.smithi089.stdout:734: writing smithi08937929-5 from 1808051 to 2444064 tid 2 2024-04-22T18:39:13.442 INFO:tasks.rados.rados.0.smithi089.stdout:734: writing smithi08937929-5 from 2873212 to 3462209 tid 3 2024-04-22T18:39:13.442 INFO:tasks.rados.rados.0.smithi089.stdout:735: write (excl) oid 42 current snap is 0 2024-04-22T18:39:13.442 INFO:tasks.rados.rados.0.smithi089.stdout:735: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-04-22T18:39:13.443 INFO:tasks.rados.rados.0.smithi089.stdout:735: writing smithi08937929-42 from 679016 to 1380811 tid 1 2024-04-22T18:39:13.447 INFO:tasks.rados.rados.0.smithi089.stdout:735: writing smithi08937929-42 from 1866533 to 2362557 tid 2 2024-04-22T18:39:13.452 INFO:tasks.rados.rados.0.smithi089.stdout:735: writing smithi08937929-42 from 2900635 to 3303278 tid 3 2024-04-22T18:39:13.452 INFO:tasks.rados.rados.0.smithi089.stdout:736: write oid 50 current snap is 0 2024-04-22T18:39:13.453 INFO:tasks.rados.rados.0.smithi089.stdout:731: expect (ObjNum 82 snap 0 seq_num 82) 2024-04-22T18:39:13.538 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:13.539+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:13.564 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:39:13.583 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:13.583 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:13.583 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:13.583 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:13.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:13.608 INFO:tasks.rados.rados.0.smithi089.stdout:727: finishing set_chunk to oid 18 2024-04-22T18:39:13.608 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 18 set_chunk returned error code -95 offset: 1469952 length: 667136 2024-04-22T18:39:13.608 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-22T18:39:13.608 INFO:tasks.rados.rados.0.smithi089.stdout:729: finishing write tid 1 to smithi08937929-37 2024-04-22T18:39:13.608 INFO:tasks.rados.rados.0.smithi089.stdout:733: expect (ObjNum 75 snap 0 seq_num 75) 2024-04-22T18:39:13.642 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:13.641+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:13.682 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:13.682 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:13.732 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:13.732 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:13.733 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:13.733 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:13.733 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:13.740 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:13.741+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:13.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:13.772 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:13.772 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:13.788 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:13.788 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:13.795 INFO:tasks.rados.rados.0.smithi089.stdout:734: finishing write tid 1 to smithi08937929-5 2024-04-22T18:39:13.795 INFO:tasks.rados.rados.0.smithi089.stdout:734: finishing write tid 2 to smithi08937929-5 2024-04-22T18:39:13.795 INFO:tasks.rados.rados.0.smithi089.stdout:732: finishing write tid 1 to smithi08937929-11 2024-04-22T18:39:13.795 INFO:tasks.rados.rados.0.smithi089.stdout:732: finishing write tid 2 to smithi08937929-11 2024-04-22T18:39:13.795 INFO:tasks.rados.rados.0.smithi089.stdout:735: finishing write tid 1 to smithi08937929-42 2024-04-22T18:39:13.795 INFO:tasks.rados.rados.0.smithi089.stdout:735: finishing write tid 2 to smithi08937929-42 2024-04-22T18:39:13.795 INFO:tasks.rados.rados.0.smithi089.stdout:729: finishing write tid 2 to smithi08937929-37 2024-04-22T18:39:13.795 INFO:tasks.rados.rados.0.smithi089.stdout:729: finishing write tid 3 to smithi08937929-37 2024-04-22T18:39:13.795 INFO:tasks.rados.rados.0.smithi089.stdout:736: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-04-22T18:39:13.802 INFO:tasks.rados.rados.0.smithi089.stdout:736: writing smithi08937929-50 from 695823 to 1472867 tid 1 2024-04-22T18:39:13.803 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:13.804 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:13.808 INFO:tasks.rados.rados.0.smithi089.stdout:736: writing smithi08937929-50 from 1925015 to 2564697 tid 2 2024-04-22T18:39:13.813 INFO:tasks.rados.rados.0.smithi089.stdout:736: writing smithi08937929-50 from 3328058 to 3495823 tid 3 2024-04-22T18:39:13.814 INFO:tasks.rados.rados.0.smithi089.stdout:727: done (7 left) 2024-04-22T18:39:13.814 INFO:tasks.rados.rados.0.smithi089.stdout:731: done (6 left) 2024-04-22T18:39:13.814 INFO:tasks.rados.rados.0.smithi089.stdout:733: done (5 left) 2024-04-22T18:39:13.814 INFO:tasks.rados.rados.0.smithi089.stdout:737: set_chunk oid 28 target oid 2024-04-22T18:39:13.820 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:13.820 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:13.836 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:13.836 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:13.840 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:13.841+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:13.848 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:13.848 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:13.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:13.848 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:13.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.848 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:13.848 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:13.848 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.106008418000002, 2024-04-22T18:39:13.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30520528099999999, 2024-04-22T18:39:13.848 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.848 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.849 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.849 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.849 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.849 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 35 2024-04-22T18:39:13.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.849 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:13.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:13.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741852+0000", 2024-04-22T18:39:13.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:13.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741927+0000", 2024-04-22T18:39:13.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5123000000000006e-05 2024-04-22T18:39:13.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741953+0000", 2024-04-22T18:39:13.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5378999999999999e-05 2024-04-22T18:39:13.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741977+0000", 2024-04-22T18:39:13.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4431999999999999e-05 2024-04-22T18:39:13.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844243+0000", 2024-04-22T18:39:13.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102265525 2024-04-22T18:39:13.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844330+0000", 2024-04-22T18:39:13.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7417999999999995e-05 2024-04-22T18:39:13.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844470+0000", 2024-04-22T18:39:13.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000140197 2024-04-22T18:39:13.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944767+0000", 2024-04-22T18:39:13.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100296409 2024-04-22T18:39:13.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.046968+0000", 2024-04-22T18:39:13.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10220148499999999 2024-04-22T18:39:13.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047018+0000", 2024-04-22T18:39:13.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9206000000000003e-05 2024-04-22T18:39:13.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047059+0000", 2024-04-22T18:39:13.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1496999999999999e-05 2024-04-22T18:39:13.854 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.854 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.854 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.854 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:13.854 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:13.855 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.105877894000002, 2024-04-22T18:39:13.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:39:13.855 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.855 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.855 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.855 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.855 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.855 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:39:13.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.855 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:13.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:13.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:39:13.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:13.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:39:13.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:39:13.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:39:13.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:39:13.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:39:13.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:39:13.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:39:13.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:39:13.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:39:13.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:39:13.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:39:13.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:39:13.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:39:13.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:39:13.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:39:13.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:39:13.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:39:13.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:39:13.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:39:13.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:39:13.860 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.860 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.860 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.860 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:13.860 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:13.860 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.105838202999998, 2024-04-22T18:39:13.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:39:13.861 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.861 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.861 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.861 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.861 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.861 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:39:13.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.861 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:13.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:13.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:39:13.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:13.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:39:13.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:39:13.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:39:13.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:39:13.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:39:13.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:39:13.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:39:13.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:39:13.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:39:13.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:39:13.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:39:13.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:39:13.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:39:13.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:39:13.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:39:13.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:39:13.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:39:13.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:39:13.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:39:13.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:39:13.866 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.866 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.866 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.866 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:13.867 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:13.867 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 31.861657156, 2024-04-22T18:39:13.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402419109, 2024-04-22T18:39:13.867 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.867 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.867 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.867 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.867 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.867 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 68 2024-04-22T18:39:13.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.867 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:13.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:13.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986204+0000", 2024-04-22T18:39:13.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986208+0000", 2024-04-22T18:39:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8149999999999999e-06 2024-04-22T18:39:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986214+0000", 2024-04-22T18:39:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.604e-06 2024-04-22T18:39:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986218+0000", 2024-04-22T18:39:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6689999999999999e-06 2024-04-22T18:39:13.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986833+0000", 2024-04-22T18:39:13.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000615648 2024-04-22T18:39:13.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986891+0000", 2024-04-22T18:39:13.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7915999999999999e-05 2024-04-22T18:39:13.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986981+0000", 2024-04-22T18:39:13.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759999999999994e-05 2024-04-22T18:39:13.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.987584+0000", 2024-04-22T18:39:13.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060270199999999999 2024-04-22T18:39:13.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388527+0000", 2024-04-22T18:39:13.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40094366999999997 2024-04-22T18:39:13.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388583+0000", 2024-04-22T18:39:13.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5383000000000002e-05 2024-04-22T18:39:13.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388624+0000", 2024-04-22T18:39:13.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501999999999997e-05 2024-04-22T18:39:13.873 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.873 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.873 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.873 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:13.873 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:13.873 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 31.861641367000001, 2024-04-22T18:39:13.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:39:13.873 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.873 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.873 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.873 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.874 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.874 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:39:13.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.874 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:13.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:13.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:39:13.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:39:13.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:39:13.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:39:13.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:39:13.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:39:13.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:39:13.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:39:13.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:39:13.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:39:13.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:39:13.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:39:13.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:39:13.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:39:13.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:39:13.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:39:13.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:39:13.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:39:13.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:39:13.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:39:13.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:39:13.879 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.879 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.879 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.879 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:13.879 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:13.879 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.745248358000001, 2024-04-22T18:39:13.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305563733, 2024-04-22T18:39:13.880 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.880 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.880 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.880 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.880 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.880 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 87 2024-04-22T18:39:13.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.880 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:13.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:13.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102613+0000", 2024-04-22T18:39:13.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:13.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102668+0000", 2024-04-22T18:39:13.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4669000000000001e-05 2024-04-22T18:39:13.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102680+0000", 2024-04-22T18:39:13.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2026e-05 2024-04-22T18:39:13.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102687+0000", 2024-04-22T18:39:13.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8339999999999999e-06 2024-04-22T18:39:13.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405644+0000", 2024-04-22T18:39:13.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30295623199999999 2024-04-22T18:39:13.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405743+0000", 2024-04-22T18:39:13.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.98e-05 2024-04-22T18:39:13.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405868+0000", 2024-04-22T18:39:13.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012457899999999999 2024-04-22T18:39:13.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407047+0000", 2024-04-22T18:39:13.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011791449999999999 2024-04-22T18:39:13.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408088+0000", 2024-04-22T18:39:13.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010411680000000001 2024-04-22T18:39:13.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408135+0000", 2024-04-22T18:39:13.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6255999999999999e-05 2024-04-22T18:39:13.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408178+0000", 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2877000000000003e-05 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.745168429, 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30560089000000001, 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.885 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.886 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.886 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.886 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 88 2024-04-22T18:39:13.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.886 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:13.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:13.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102693+0000", 2024-04-22T18:39:13.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:13.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102698+0000", 2024-04-22T18:39:13.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9699999999999998e-06 2024-04-22T18:39:13.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102708+0000", 2024-04-22T18:39:13.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.006e-05 2024-04-22T18:39:13.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102715+0000", 2024-04-22T18:39:13.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6599999999999995e-06 2024-04-22T18:39:13.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406637+0000", 2024-04-22T18:39:13.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392130099999998 2024-04-22T18:39:13.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406696+0000", 2024-04-22T18:39:13.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9066e-05 2024-04-22T18:39:13.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406784+0000", 2024-04-22T18:39:13.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8194000000000003e-05 2024-04-22T18:39:13.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407447+0000", 2024-04-22T18:39:13.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066295700000000004 2024-04-22T18:39:13.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408237+0000", 2024-04-22T18:39:13.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079032600000000005 2024-04-22T18:39:13.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408270+0000", 2024-04-22T18:39:13.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3157000000000001e-05 2024-04-22T18:39:13.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408295+0000", 2024-04-22T18:39:13.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4151999999999998e-05 2024-04-22T18:39:13.891 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.891 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.891 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.891 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:13.891 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:13.891 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.993936304999998, 2024-04-22T18:39:13.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30533819600000001, 2024-04-22T18:39:13.891 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.891 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.892 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.892 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 106 2024-04-22T18:39:13.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.892 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:13.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:13.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853925+0000", 2024-04-22T18:39:13.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853958+0000", 2024-04-22T18:39:13.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2536000000000002e-05 2024-04-22T18:39:13.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853964+0000", 2024-04-22T18:39:13.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6239999999999999e-06 2024-04-22T18:39:13.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853967+0000", 2024-04-22T18:39:13.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9700000000000001e-06 2024-04-22T18:39:13.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855144+0000", 2024-04-22T18:39:13.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001176774 2024-04-22T18:39:13.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855225+0000", 2024-04-22T18:39:13.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0697000000000003e-05 2024-04-22T18:39:13.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855353+0000", 2024-04-22T18:39:13.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012788 2024-04-22T18:39:13.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058003+0000", 2024-04-22T18:39:13.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202649939 2024-04-22T18:39:13.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159184+0000", 2024-04-22T18:39:13.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101181058 2024-04-22T18:39:13.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159238+0000", 2024-04-22T18:39:13.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3967e-05 2024-04-22T18:39:13.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159264+0000", 2024-04-22T18:39:13.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6237999999999998e-05 2024-04-22T18:39:13.896 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.897 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.897 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.897 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:13.897 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:13.897 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.993889455000001, 2024-04-22T18:39:13.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30544294500000002, 2024-04-22T18:39:13.897 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.897 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.897 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.897 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.897 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.898 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 107 2024-04-22T18:39:13.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.898 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:13.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:13.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853972+0000", 2024-04-22T18:39:13.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853975+0000", 2024-04-22T18:39:13.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0939999999999999e-06 2024-04-22T18:39:13.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853978+0000", 2024-04-22T18:39:13.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:39:13.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853981+0000", 2024-04-22T18:39:13.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.858e-06 2024-04-22T18:39:13.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855831+0000", 2024-04-22T18:39:13.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018503689999999999 2024-04-22T18:39:13.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855932+0000", 2024-04-22T18:39:13.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010071 2024-04-22T18:39:13.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856029+0000", 2024-04-22T18:39:13.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6751999999999998e-05 2024-04-22T18:39:13.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058013+0000", 2024-04-22T18:39:13.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20198412600000001 2024-04-22T18:39:13.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159326+0000", 2024-04-22T18:39:13.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101312969 2024-04-22T18:39:13.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159376+0000", 2024-04-22T18:39:13.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9968e-05 2024-04-22T18:39:13.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159416+0000", 2024-04-22T18:39:13.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9641999999999998e-05 2024-04-22T18:39:13.902 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.902 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.902 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.903 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:13.903 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:13.903 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.993801058999999, 2024-04-22T18:39:13.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30548257499999998, 2024-04-22T18:39:13.903 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.903 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.903 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.903 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.903 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.903 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 108 2024-04-22T18:39:13.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.903 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:13.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:13.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:13.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854064+0000", 2024-04-22T18:39:13.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5820000000000002e-06 2024-04-22T18:39:13.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854080+0000", 2024-04-22T18:39:13.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5379999999999998e-05 2024-04-22T18:39:13.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854084+0000", 2024-04-22T18:39:13.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.963e-06 2024-04-22T18:39:13.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856413+0000", 2024-04-22T18:39:13.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023298070000000001 2024-04-22T18:39:13.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856494+0000", 2024-04-22T18:39:13.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0863000000000002e-05 2024-04-22T18:39:13.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856624+0000", 2024-04-22T18:39:13.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129799 2024-04-22T18:39:13.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058947+0000", 2024-04-22T18:39:13.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20232328899999999 2024-04-22T18:39:13.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159468+0000", 2024-04-22T18:39:13.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100520597 2024-04-22T18:39:13.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159508+0000", 2024-04-22T18:39:13.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9830000000000003e-05 2024-04-22T18:39:13.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159544+0000", 2024-04-22T18:39:13.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5944e-05 2024-04-22T18:39:13.908 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.908 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.909 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.909 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:13.909 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:13.909 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.993756566999998, 2024-04-22T18:39:13.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30556502499999999, 2024-04-22T18:39:13.909 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.909 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.909 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.909 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.909 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.909 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:39:13.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.910 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:13.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:13.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854105+0000", 2024-04-22T18:39:13.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854175+0000", 2024-04-22T18:39:13.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0178000000000005e-05 2024-04-22T18:39:13.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854191+0000", 2024-04-22T18:39:13.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5965999999999999e-05 2024-04-22T18:39:13.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854196+0000", 2024-04-22T18:39:13.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.42e-06 2024-04-22T18:39:13.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956526+0000", 2024-04-22T18:39:13.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10233094299999999 2024-04-22T18:39:13.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956634+0000", 2024-04-22T18:39:13.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107398 2024-04-22T18:39:13.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956730+0000", 2024-04-22T18:39:13.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6217000000000004e-05 2024-04-22T18:39:13.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058955+0000", 2024-04-22T18:39:13.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102224749 2024-04-22T18:39:13.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159596+0000", 2024-04-22T18:39:13.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100641297 2024-04-22T18:39:13.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159637+0000", 2024-04-22T18:39:13.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874000000000002e-05 2024-04-22T18:39:13.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159671+0000", 2024-04-22T18:39:13.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3603999999999999e-05 2024-04-22T18:39:13.914 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.914 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.914 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.915 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:13.915 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:13.915 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.99366272, 2024-04-22T18:39:13.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30683048800000001, 2024-04-22T18:39:13.915 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.915 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.915 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.915 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.915 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.915 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110 2024-04-22T18:39:13.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.916 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:13.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:13.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:13.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854202+0000", 2024-04-22T18:39:13.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:39:13.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854206+0000", 2024-04-22T18:39:13.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5870000000000001e-06 2024-04-22T18:39:13.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854208+0000", 2024-04-22T18:39:13.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7199999999999998e-06 2024-04-22T18:39:13.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057285+0000", 2024-04-22T18:39:13.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20307620800000001 2024-04-22T18:39:13.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057414+0000", 2024-04-22T18:39:13.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012896899999999999 2024-04-22T18:39:13.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057588+0000", 2024-04-22T18:39:13.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000174802 2024-04-22T18:39:13.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159429+0000", 2024-04-22T18:39:13.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101840783 2024-04-22T18:39:13.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160947+0000", 2024-04-22T18:39:13.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001518152 2024-04-22T18:39:13.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160992+0000", 2024-04-22T18:39:13.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4725000000000003e-05 2024-04-22T18:39:13.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161030+0000", 2024-04-22T18:39:13.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7789e-05 2024-04-22T18:39:13.920 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.920 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.921 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.921 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:13.921 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:13.921 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.993651538999998, 2024-04-22T18:39:13.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:39:13.921 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.921 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.921 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.921 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.921 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.921 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:39:13.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.921 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:13.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:13.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:39:13.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:39:13.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:39:13.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:39:13.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:39:13.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:39:13.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:39:13.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:39:13.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:39:13.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:39:13.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:39:13.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:39:13.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:39:13.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:39:13.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:39:13.926 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.926 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.926 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.926 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:13.926 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:13.927 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.993635880999999, 2024-04-22T18:39:13.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:39:13.927 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.927 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.927 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.927 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.927 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.927 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:39:13.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.927 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:13.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:13.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:13.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:39:13.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:39:13.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:39:13.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:39:13.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:39:13.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:39:13.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:39:13.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:39:13.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:39:13.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:39:13.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:39:13.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:39:13.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:39:13.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:39:13.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:39:13.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:39:13.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:39:13.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:39:13.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:39:13.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:39:13.932 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.932 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.932 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.932 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:13.932 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:13.932 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.993584719000001, 2024-04-22T18:39:13.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:39:13.933 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.933 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.933 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.933 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.933 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.933 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:39:13.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.933 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:13.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:13.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:13.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:39:13.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:39:13.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:39:13.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:39:13.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:39:13.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:39:13.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:39:13.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:39:13.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:39:13.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:39:13.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:39:13.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:39:13.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:39:13.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:39:13.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:39:13.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:39:13.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:39:13.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:39:13.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:39:13.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:39:13.938 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.938 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.938 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.939 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:13.939 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:13.939 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.993569494999999, 2024-04-22T18:39:13.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:39:13.939 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.939 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.939 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.939 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.939 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.939 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:39:13.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:39:13.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:39:13.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:39:13.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:39:13.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:39:13.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:39:13.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:39:13.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:39:13.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:39:13.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:39:13.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:39:13.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:39:13.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:39:13.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:39:13.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:39:13.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:39:13.944 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.944 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.944 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.944 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:13.944 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:13.944 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.993509918000001, 2024-04-22T18:39:13.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:39:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:39:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:39:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:39:13.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:39:13.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:39:13.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:39:13.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:39:13.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:39:13.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:39:13.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:39:13.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:39:13.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:39:13.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:39:13.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:39:13.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:39:13.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:39:13.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:39:13.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:39:13.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:39:13.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:39:13.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:39:13.950 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.950 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.950 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.951 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:13.951 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:13.951 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.993427602000001, 2024-04-22T18:39:13.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:39:13.951 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.951 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.951 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.951 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.951 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.952 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:39:13.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.952 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:13.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:13.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:39:13.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:39:13.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:39:13.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:39:13.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:39:13.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:39:13.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:39:13.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:39:13.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:39:13.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:39:13.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:39:13.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:39:13.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:39:13.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:39:13.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:39:13.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:39:13.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:39:13.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:39:13.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:39:13.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:39:13.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:39:13.957 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.957 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.957 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.957 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:13.957 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:13.957 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.993417445999999, 2024-04-22T18:39:13.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:39:13.957 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.957 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.957 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.958 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.958 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.958 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:39:13.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.958 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:13.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:13.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:39:13.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:39:13.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:39:13.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:39:13.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:39:13.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:39:13.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:39:13.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:39:13.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:39:13.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:39:13.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:39:13.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:39:13.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:39:13.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:39:13.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:39:13.962 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.962 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.962 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.963 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:161 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:39:13.963 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:13.963 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 25.392923154999998, 2024-04-22T18:39:13.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40239169000000002, 2024-04-22T18:39:13.963 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:13.963 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:13.963 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:13.963 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:13.963 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:13.963 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 161 2024-04-22T18:39:13.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.964 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:13.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:13.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:13.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:13.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:13.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:13.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:13.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454938+0000", 2024-04-22T18:39:13.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:13.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:13.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454982+0000", 2024-04-22T18:39:13.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3470999999999998e-05 2024-04-22T18:39:13.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:13.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454988+0000", 2024-04-22T18:39:13.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7640000000000002e-06 2024-04-22T18:39:13.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:13.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555054+0000", 2024-04-22T18:39:13.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006607300000001 2024-04-22T18:39:13.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:13.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555607+0000", 2024-04-22T18:39:13.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055319500000000003 2024-04-22T18:39:13.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:13.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555689+0000", 2024-04-22T18:39:13.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1817999999999995e-05 2024-04-22T18:39:13.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:13.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555804+0000", 2024-04-22T18:39:13.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115346 2024-04-22T18:39:13.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:13.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556554+0000", 2024-04-22T18:39:13.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000750102 2024-04-22T18:39:13.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:13.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857265+0000", 2024-04-22T18:39:13.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30071122700000003 2024-04-22T18:39:13.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:13.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857302+0000", 2024-04-22T18:39:13.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7069e-05 2024-04-22T18:39:13.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:13.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:13.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:13.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857331+0000", 2024-04-22T18:39:13.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8291999999999999e-05 2024-04-22T18:39:13.968 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.968 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.968 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.968 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:13.968 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:13.968 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:13.969 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:13.970 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:13.971 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:13.971 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:13.971+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:13.971 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:13.972 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:14.021 INFO:tasks.rados.rados.0.smithi089.stdout:737: set_chunk oid 28 offset: 554496 length: 919040 target oid cc7d748d428428ed51050303be64e2b9fd96740d945cc9991ff74b4293170f85 offset: 0 snap -1 2024-04-22T18:39:14.021 INFO:tasks.rados.rados.0.smithi089.stdout:738: set_chunk oid 38 target oid 2024-04-22T18:39:14.033 INFO:tasks.rados.rados.0.smithi089.stdout:738: set_chunk oid 38 offset: 157184 length: 1566208 target oid 35530ab6b54425d8af1fc6e7d77c6e2ddac51b59049ae00aae67c6946c797d95 offset: 0 snap -1 2024-04-22T18:39:14.033 INFO:tasks.rados.rados.0.smithi089.stdout:739: tier_promote oid 34 2024-04-22T18:39:14.033 INFO:tasks.rados.rados.0.smithi089.stdout:729: finishing write tid 4 to smithi08937929-37 2024-04-22T18:39:14.033 INFO:tasks.rados.rados.0.smithi089.stdout:740: read oid 33 snap -1 2024-04-22T18:39:14.033 INFO:tasks.rados.rados.0.smithi089.stdout:740: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-22T18:39:14.033 INFO:tasks.rados.rados.0.smithi089.stdout:741: set_chunk oid 2 target oid 2024-04-22T18:39:14.070 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:14.072+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:14.114 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:14.114 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:14.123 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:14.171 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:14.172+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:14.195 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:39:14.200 INFO:tasks.rados.rados.0.smithi089.stdout:741: set_chunk oid 2 offset: 690688 length: 344064 target oid fa6e1a40c489d40bc69ccb5459abb41ab74051e1780cc4076d3bcebfee447a3b offset: 0 snap -1 2024-04-22T18:39:14.200 INFO:tasks.rados.rados.0.smithi089.stdout:742: read oid 13 snap -1 2024-04-22T18:39:14.200 INFO:tasks.rados.rados.0.smithi089.stdout:742: expect deleted 2024-04-22T18:39:14.200 INFO:tasks.rados.rados.0.smithi089.stdout:743: tier_flush oid 22 2024-04-22T18:39:14.200 INFO:tasks.rados.rados.0.smithi089.stdout:743: tier_flush oid 22 snap -1 2024-04-22T18:39:14.200 INFO:tasks.rados.rados.0.smithi089.stdout:729: finishing write tid 5 to smithi08937929-37 2024-04-22T18:39:14.200 INFO:tasks.rados.rados.0.smithi089.stdout:729: oid 37 updating version 0 to 50 2024-04-22T18:39:14.201 INFO:tasks.rados.rados.0.smithi089.stdout:729: oid 37 version 50 is already newer than 47 2024-04-22T18:39:14.201 INFO:tasks.rados.rados.0.smithi089.stdout:729: oid 37 version 50 is already newer than 48 2024-04-22T18:39:14.201 INFO:tasks.rados.rados.0.smithi089.stdout:729: oid 37 version 50 is already newer than 49 2024-04-22T18:39:14.201 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 50 (ObjNum 84 snap 0 seq_num 84) dirty exists 2024-04-22T18:39:14.201 INFO:tasks.rados.rados.0.smithi089.stdout:729: left oid 37 (ObjNum 84 snap 0 seq_num 84) 2024-04-22T18:39:14.201 INFO:tasks.rados.rados.0.smithi089.stdout:732: finishing write tid 3 to smithi08937929-11 2024-04-22T18:39:14.201 INFO:tasks.rados.rados.0.smithi089.stdout:735: finishing write tid 3 to smithi08937929-42 2024-04-22T18:39:14.201 INFO:tasks.rados.rados.0.smithi089.stdout:734: finishing write tid 3 to smithi08937929-5 2024-04-22T18:39:14.201 INFO:tasks.rados.rados.0.smithi089.stdout:734: finishing write tid 4 to smithi08937929-5 2024-04-22T18:39:14.201 INFO:tasks.rados.rados.0.smithi089.stdout:732: finishing write tid 4 to smithi08937929-11 2024-04-22T18:39:14.201 INFO:tasks.rados.rados.0.smithi089.stdout:732: finishing write tid 5 to smithi08937929-11 2024-04-22T18:39:14.202 INFO:tasks.rados.rados.0.smithi089.stdout:732: oid 11 updating version 0 to 16 2024-04-22T18:39:14.202 INFO:tasks.rados.rados.0.smithi089.stdout:732: oid 11 updating version 16 to 17 2024-04-22T18:39:14.202 INFO:tasks.rados.rados.0.smithi089.stdout:732: oid 11 updating version 17 to 18 2024-04-22T18:39:14.202 INFO:tasks.rados.rados.0.smithi089.stdout:732: oid 11 version 18 is already newer than 15 2024-04-22T18:39:14.202 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 18 (ObjNum 86 snap 0 seq_num 86) dirty exists 2024-04-22T18:39:14.202 INFO:tasks.rados.rados.0.smithi089.stdout:732: left oid 11 (ObjNum 86 snap 0 seq_num 86) 2024-04-22T18:39:14.202 INFO:tasks.rados.rados.0.smithi089.stdout:734: finishing write tid 5 to smithi08937929-5 2024-04-22T18:39:14.202 INFO:tasks.rados.rados.0.smithi089.stdout:734: oid 5 updating version 0 to 51 2024-04-22T18:39:14.202 INFO:tasks.rados.rados.0.smithi089.stdout:734: oid 5 version 51 is already newer than 50 2024-04-22T18:39:14.202 INFO:tasks.rados.rados.0.smithi089.stdout:734: oid 5 version 51 is already newer than 49 2024-04-22T18:39:14.202 INFO:tasks.rados.rados.0.smithi089.stdout:734: oid 5 updating version 51 to 52 2024-04-22T18:39:14.202 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 52 (ObjNum 87 snap 0 seq_num 87) dirty exists 2024-04-22T18:39:14.202 INFO:tasks.rados.rados.0.smithi089.stdout:734: left oid 5 (ObjNum 87 snap 0 seq_num 87) 2024-04-22T18:39:14.203 INFO:tasks.rados.rados.0.smithi089.stdout:735: finishing write tid 4 to smithi08937929-42 2024-04-22T18:39:14.203 INFO:tasks.rados.rados.0.smithi089.stdout:729: done (11 left) 2024-04-22T18:39:14.203 INFO:tasks.rados.rados.0.smithi089.stdout:732: done (10 left) 2024-04-22T18:39:14.203 INFO:tasks.rados.rados.0.smithi089.stdout:734: done (9 left) 2024-04-22T18:39:14.203 INFO:tasks.rados.rados.0.smithi089.stdout:744: delete oid 17 current snap is 0 2024-04-22T18:39:14.203 INFO:tasks.rados.rados.0.smithi089.stdout:735: finishing write tid 5 to smithi08937929-42 2024-04-22T18:39:14.203 INFO:tasks.rados.rados.0.smithi089.stdout:735: oid 42 updating version 0 to 9 2024-04-22T18:39:14.203 INFO:tasks.rados.rados.0.smithi089.stdout:735: oid 42 updating version 9 to 12 2024-04-22T18:39:14.203 INFO:tasks.rados.rados.0.smithi089.stdout:735: oid 42 version 12 is already newer than 11 2024-04-22T18:39:14.203 INFO:tasks.rados.rados.0.smithi089.stdout:735: oid 42 version 12 is already newer than 10 2024-04-22T18:39:14.203 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 12 (ObjNum 88 snap 0 seq_num 88) dirty exists 2024-04-22T18:39:14.203 INFO:tasks.rados.rados.0.smithi089.stdout:735: left oid 42 (ObjNum 88 snap 0 seq_num 88) 2024-04-22T18:39:14.204 INFO:tasks.rados.rados.0.smithi089.stdout:736: finishing write tid 1 to smithi08937929-50 2024-04-22T18:39:14.204 INFO:tasks.rados.rados.0.smithi089.stdout:736: finishing write tid 2 to smithi08937929-50 2024-04-22T18:39:14.204 INFO:tasks.rados.rados.0.smithi089.stdout:736: finishing write tid 3 to smithi08937929-50 2024-04-22T18:39:14.204 INFO:tasks.rados.rados.0.smithi089.stdout:736: finishing write tid 4 to smithi08937929-50 2024-04-22T18:39:14.204 INFO:tasks.rados.rados.0.smithi089.stdout:738: finishing set_chunk to oid 38 2024-04-22T18:39:14.204 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 38 set_chunk returned error code -95 offset: 157184 length: 1566208 2024-04-22T18:39:14.204 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-22T18:39:14.204 INFO:tasks.rados.rados.0.smithi089.stdout:736: finishing write tid 5 to smithi08937929-50 2024-04-22T18:39:14.204 INFO:tasks.rados.rados.0.smithi089.stdout:736: oid 50 updating version 0 to 56 2024-04-22T18:39:14.204 INFO:tasks.rados.rados.0.smithi089.stdout:736: oid 50 version 56 is already newer than 54 2024-04-22T18:39:14.204 INFO:tasks.rados.rados.0.smithi089.stdout:736: oid 50 version 56 is already newer than 55 2024-04-22T18:39:14.204 INFO:tasks.rados.rados.0.smithi089.stdout:736: oid 50 version 56 is already newer than 53 2024-04-22T18:39:14.204 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 56 (ObjNum 89 snap 0 seq_num 89) dirty exists 2024-04-22T18:39:14.205 INFO:tasks.rados.rados.0.smithi089.stdout:736: left oid 50 (ObjNum 89 snap 0 seq_num 89) 2024-04-22T18:39:14.205 INFO:tasks.rados.rados.0.smithi089.stdout:737: finishing set_chunk to oid 28 2024-04-22T18:39:14.205 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 28 set_chunk returned error code -95 offset: 554496 length: 919040 2024-04-22T18:39:14.205 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-22T18:39:14.205 INFO:tasks.rados.rados.0.smithi089.stdout:739: got (0) Success 2024-04-22T18:39:14.205 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 34 (ObjNum 59 snap 0 seq_num 59) dirty exists 2024-04-22T18:39:14.205 INFO:tasks.rados.rados.0.smithi089.stdout:740: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-22T18:39:14.272 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:14.272 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:14.273 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:14.273+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:14.281 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:14.373 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:14.374+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:14.376 INFO:tasks.rados.rados.0.smithi089.stdout:741: finishing set_chunk to oid 2 2024-04-22T18:39:14.377 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk returned error code -95 offset: 690688 length: 344064 2024-04-22T18:39:14.377 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-22T18:39:14.377 INFO:tasks.rados.rados.0.smithi089.stdout:735: done (9 left) 2024-04-22T18:39:14.377 INFO:tasks.rados.rados.0.smithi089.stdout:736: done (8 left) 2024-04-22T18:39:14.377 INFO:tasks.rados.rados.0.smithi089.stdout:737: done (7 left) 2024-04-22T18:39:14.377 INFO:tasks.rados.rados.0.smithi089.stdout:738: done (6 left) 2024-04-22T18:39:14.377 INFO:tasks.rados.rados.0.smithi089.stdout:739: done (5 left) 2024-04-22T18:39:14.377 INFO:tasks.rados.rados.0.smithi089.stdout:740: done (4 left) 2024-04-22T18:39:14.377 INFO:tasks.rados.rados.0.smithi089.stdout:741: done (3 left) 2024-04-22T18:39:14.377 INFO:tasks.rados.rados.0.smithi089.stdout:742: done (2 left) 2024-04-22T18:39:14.377 INFO:tasks.rados.rados.0.smithi089.stdout:744: done (1 left) 2024-04-22T18:39:14.377 INFO:tasks.rados.rados.0.smithi089.stdout:745: write oid 44 current snap is 0 2024-04-22T18:39:14.377 INFO:tasks.rados.rados.0.smithi089.stdout:745: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2024-04-22T18:39:14.383 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:14.383 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:14.383 INFO:tasks.rados.rados.0.smithi089.stdout:745: writing smithi08937929-44 from 729437 to 1256979 tid 1 2024-04-22T18:39:14.390 INFO:tasks.rados.rados.0.smithi089.stdout:745: writing smithi08937929-44 from 1925626 to 2568977 tid 2 2024-04-22T18:39:14.392 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:14.396 INFO:tasks.rados.rados.0.smithi089.stdout:745: writing smithi08937929-44 from 3266551 to 3529437 tid 3 2024-04-22T18:39:14.397 INFO:tasks.rados.rados.0.smithi089.stdout:746: read oid 5 snap -1 2024-04-22T18:39:14.397 INFO:tasks.rados.rados.0.smithi089.stdout:746: expect (ObjNum 87 snap 0 seq_num 87) 2024-04-22T18:39:14.397 INFO:tasks.rados.rados.0.smithi089.stdout:747: read oid 16 snap -1 2024-04-22T18:39:14.397 INFO:tasks.rados.rados.0.smithi089.stdout:747: expect (ObjNum 71 snap 0 seq_num 71) 2024-04-22T18:39:14.397 INFO:tasks.rados.rados.0.smithi089.stdout:748: read oid 38 snap -1 2024-04-22T18:39:14.397 INFO:tasks.rados.rados.0.smithi089.stdout:748: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-22T18:39:14.397 INFO:tasks.rados.rados.0.smithi089.stdout:749: set_chunk oid 27 target oid 2024-04-22T18:39:14.474 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:14.475+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:14.499 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:14.499 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:14.509 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:14.594 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:14.595+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:14.652 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:14.653 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:14.661 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:14.676 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:14.677+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:14.703 INFO:tasks.rados.rados.0.smithi089.stdout:749: set_chunk oid 27 offset: 703488 length: 779776 target oid 008e5603f489687e08b7a89fe7f18a96f1ef1914598c971e454b0158ab3631ec offset: 0 snap -1 2024-04-22T18:39:14.704 INFO:tasks.rados.rados.0.smithi089.stdout:750: tier_evict oid 25 2024-04-22T18:39:14.704 INFO:tasks.rados.rados.0.smithi089.stdout:750: tier_evict oid 25 snap -1 2024-04-22T18:39:14.704 INFO:tasks.rados.rados.0.smithi089.stdout:748: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-22T18:39:14.776 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:14.778+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:14.790 INFO:tasks.rados.rados.0.smithi089.stdout:746: expect (ObjNum 87 snap 0 seq_num 87) 2024-04-22T18:39:14.805 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:14.805 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:14.814 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:14.877 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:14.878+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:14.956 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:14.956 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:14.965 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:14.977 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:14.978+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:14.979 INFO:tasks.rados.rados.0.smithi089.stdout:745: finishing write tid 1 to smithi08937929-44 2024-04-22T18:39:14.980 INFO:tasks.rados.rados.0.smithi089.stdout:745: finishing write tid 2 to smithi08937929-44 2024-04-22T18:39:14.980 INFO:tasks.rados.rados.0.smithi089.stdout:746: done (6 left) 2024-04-22T18:39:14.980 INFO:tasks.rados.rados.0.smithi089.stdout:748: done (5 left) 2024-04-22T18:39:14.980 INFO:tasks.rados.rados.0.smithi089.stdout:751: tier_evict oid 21 2024-04-22T18:39:14.980 INFO:tasks.rados.rados.0.smithi089.stdout:751: tier_evict oid 21 snap -1 2024-04-22T18:39:14.980 INFO:tasks.rados.rados.0.smithi089.stdout:745: finishing write tid 3 to smithi08937929-44 2024-04-22T18:39:14.980 INFO:tasks.rados.rados.0.smithi089.stdout:743: got (0) Success 2024-04-22T18:39:14.980 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-22T18:39:14.980 INFO:tasks.rados.rados.0.smithi089.stdout:745: finishing write tid 4 to smithi08937929-44 2024-04-22T18:39:14.980 INFO:tasks.rados.rados.0.smithi089.stdout:749: finishing set_chunk to oid 27 2024-04-22T18:39:14.980 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 703488 length: 779776 2024-04-22T18:39:14.981 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-22T18:39:14.981 INFO:tasks.rados.rados.0.smithi089.stdout:745: finishing write tid 5 to smithi08937929-44 2024-04-22T18:39:14.981 INFO:tasks.rados.rados.0.smithi089.stdout:745: oid 44 updating version 0 to 54 2024-04-22T18:39:14.981 INFO:tasks.rados.rados.0.smithi089.stdout:745: oid 44 version 54 is already newer than 53 2024-04-22T18:39:14.981 INFO:tasks.rados.rados.0.smithi089.stdout:745: oid 44 version 54 is already newer than 52 2024-04-22T18:39:14.981 INFO:tasks.rados.rados.0.smithi089.stdout:745: oid 44 version 54 is already newer than 51 2024-04-22T18:39:14.981 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 54 (ObjNum 91 snap 0 seq_num 91) dirty exists 2024-04-22T18:39:14.981 INFO:tasks.rados.rados.0.smithi089.stdout:745: left oid 44 (ObjNum 91 snap 0 seq_num 91) 2024-04-22T18:39:14.981 INFO:tasks.rados.rados.0.smithi089.stdout:747: expect (ObjNum 71 snap 0 seq_num 71) 2024-04-22T18:39:15.066 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:15.067 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:15.075 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:15.079 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:15.080+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:15.125 INFO:tasks.rados.rados.0.smithi089.stdout:750: got (22) Invalid argument 2024-04-22T18:39:15.125 INFO:tasks.rados.rados.0.smithi089.stdout:743: done (5 left) 2024-04-22T18:39:15.126 INFO:tasks.rados.rados.0.smithi089.stdout:745: done (4 left) 2024-04-22T18:39:15.126 INFO:tasks.rados.rados.0.smithi089.stdout:747: done (3 left) 2024-04-22T18:39:15.126 INFO:tasks.rados.rados.0.smithi089.stdout:749: done (2 left) 2024-04-22T18:39:15.126 INFO:tasks.rados.rados.0.smithi089.stdout:750: done (1 left) 2024-04-22T18:39:15.126 INFO:tasks.rados.rados.0.smithi089.stdout:752: read oid 46 snap -1 2024-04-22T18:39:15.126 INFO:tasks.rados.rados.0.smithi089.stdout:752: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-22T18:39:15.126 INFO:tasks.rados.rados.0.smithi089.stdout:753: read oid 6 snap -1 2024-04-22T18:39:15.126 INFO:tasks.rados.rados.0.smithi089.stdout:753: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-22T18:39:15.126 INFO:tasks.rados.rados.0.smithi089.stdout:754: tier_evict oid 1 2024-04-22T18:39:15.126 INFO:tasks.rados.rados.0.smithi089.stdout:754: tier_evict oid 1 snap -1 2024-04-22T18:39:15.126 INFO:tasks.rados.rados.0.smithi089.stdout:751: got (22) Invalid argument 2024-04-22T18:39:15.126 INFO:tasks.rados.rados.0.smithi089.stdout:751: done (3 left) 2024-04-22T18:39:15.126 INFO:tasks.rados.rados.0.smithi089.stdout:755: set_chunk oid 7 target oid 2024-04-22T18:39:15.180 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:15.181+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:15.220 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:15.220 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:15.229 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:15.235 INFO:tasks.rados.rados.0.smithi089.stdout:755: set_chunk oid 7 offset: 639488 length: 1070080 target oid 1005c3d470e54e8bc7761b2d8ebd6a3ef6d84decec2f640aec0aa0175021344e offset: 0 snap -1 2024-04-22T18:39:15.236 INFO:tasks.rados.rados.0.smithi089.stdout:756: set_chunk oid 40 target oid 2024-04-22T18:39:15.280 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:15.281+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:15.336 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:15.336 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:15.340 INFO:tasks.rados.rados.0.smithi089.stdout:756: set_chunk oid 40 offset: 2378752 length: 354816 target oid 2436e42d3e2152bf96bede2516b75bde6b3f7bf92d6d3a4dd343b71f9cd22c57 offset: 0 snap -1 2024-04-22T18:39:15.340 INFO:tasks.rados.rados.0.smithi089.stdout:757: read oid 29 snap -1 2024-04-22T18:39:15.340 INFO:tasks.rados.rados.0.smithi089.stdout:757: expect (ObjNum 63 snap 0 seq_num 63) 2024-04-22T18:39:15.340 INFO:tasks.rados.rados.0.smithi089.stdout:758: write (excl) oid 32 current snap is 0 2024-04-22T18:39:15.340 INFO:tasks.rados.rados.0.smithi089.stdout:754: got (22) Invalid argument 2024-04-22T18:39:15.340 INFO:tasks.rados.rados.0.smithi089.stdout:752: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-22T18:39:15.345 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:15.381 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:15.382+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:15.428 INFO:tasks.rados.rados.0.smithi089.stdout:753: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-22T18:39:15.447 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:15.448 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:15.457 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:15.482 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:15.483+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:15.500 INFO:tasks.rados.rados.0.smithi089.stdout:755: finishing set_chunk to oid 7 2024-04-22T18:39:15.500 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 7 set_chunk returned error code -95 offset: 639488 length: 1070080 2024-04-22T18:39:15.500 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-22T18:39:15.500 INFO:tasks.rados.rados.0.smithi089.stdout:757: expect (ObjNum 63 snap 0 seq_num 63) 2024-04-22T18:39:15.559 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:15.559 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:15.568 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:15.613 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:15.614+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:15.640 INFO:tasks.rados.rados.0.smithi089.stdout:758: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2024-04-22T18:39:15.647 INFO:tasks.rados.rados.0.smithi089.stdout:758: writing smithi08937929-32 from 746244 to 1465388 tid 1 2024-04-22T18:39:15.653 INFO:tasks.rados.rados.0.smithi089.stdout:758: writing smithi08937929-32 from 2100461 to 2603823 tid 2 2024-04-22T18:39:15.659 INFO:tasks.rados.rados.0.smithi089.stdout:758: writing smithi08937929-32 from 3126680 to 3546244 tid 3 2024-04-22T18:39:15.659 INFO:tasks.rados.rados.0.smithi089.stdout:752: done (6 left) 2024-04-22T18:39:15.659 INFO:tasks.rados.rados.0.smithi089.stdout:753: done (5 left) 2024-04-22T18:39:15.659 INFO:tasks.rados.rados.0.smithi089.stdout:754: done (4 left) 2024-04-22T18:39:15.659 INFO:tasks.rados.rados.0.smithi089.stdout:755: done (3 left) 2024-04-22T18:39:15.659 INFO:tasks.rados.rados.0.smithi089.stdout:757: done (2 left) 2024-04-22T18:39:15.659 INFO:tasks.rados.rados.0.smithi089.stdout:759: set_chunk oid 49 target oid 2024-04-22T18:39:15.682 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:15.682 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:15.684 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:15.685+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:15.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:15.785 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:15.786+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:15.833 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:15.833 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:15.842 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:15.886 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:15.888+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:15.974 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:39:15.982 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:15.982 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:15.987 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:15.988+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:15.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:15.991 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:39:15.991 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:39:15.992 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-22T18:39:15.992 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-22T18:39:16.056 INFO:tasks.rados.rados.0.smithi089.stdout:759: set_chunk oid 49 offset: 1544704 length: 1047552 target oid f211699942c655c3338e9096b61b2c0e28d03589d96668a88269721993fd5622 offset: 0 snap -1 2024-04-22T18:39:16.056 INFO:tasks.rados.rados.0.smithi089.stdout:760: read oid 44 snap -1 2024-04-22T18:39:16.056 INFO:tasks.rados.rados.0.smithi089.stdout:760: expect (ObjNum 91 snap 0 seq_num 91) 2024-04-22T18:39:16.056 INFO:tasks.rados.rados.0.smithi089.stdout:761: read oid 12 snap -1 2024-04-22T18:39:16.056 INFO:tasks.rados.rados.0.smithi089.stdout:761: expect (ObjNum 73 snap 0 seq_num 73) 2024-04-22T18:39:16.056 INFO:tasks.rados.rados.0.smithi089.stdout:762: delete oid 35 current snap is 0 2024-04-22T18:39:16.056 INFO:tasks.rados.rados.0.smithi089.stdout:756: finishing set_chunk to oid 40 2024-04-22T18:39:16.056 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 19 (ObjNum 58 snap 0 seq_num 58) dirty exists 2024-04-22T18:39:16.057 INFO:tasks.rados.rados.0.smithi089.stdout:758: finishing write tid 1 to smithi08937929-32 2024-04-22T18:39:16.057 INFO:tasks.rados.rados.0.smithi089.stdout:758: finishing write tid 2 to smithi08937929-32 2024-04-22T18:39:16.057 INFO:tasks.rados.rados.0.smithi089.stdout:758: finishing write tid 3 to smithi08937929-32 2024-04-22T18:39:16.057 INFO:tasks.rados.rados.0.smithi089.stdout:758: finishing write tid 4 to smithi08937929-32 2024-04-22T18:39:16.057 INFO:tasks.rados.rados.0.smithi089.stdout:758: finishing write tid 5 to smithi08937929-32 2024-04-22T18:39:16.057 INFO:tasks.rados.rados.0.smithi089.stdout:758: oid 32 updating version 0 to 30 2024-04-22T18:39:16.057 INFO:tasks.rados.rados.0.smithi089.stdout:758: oid 32 version 30 is already newer than 29 2024-04-22T18:39:16.057 INFO:tasks.rados.rados.0.smithi089.stdout:758: oid 32 updating version 30 to 31 2024-04-22T18:39:16.057 INFO:tasks.rados.rados.0.smithi089.stdout:758: oid 32 version 31 is already newer than 28 2024-04-22T18:39:16.057 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 31 (ObjNum 92 snap 0 seq_num 92) dirty exists 2024-04-22T18:39:16.057 INFO:tasks.rados.rados.0.smithi089.stdout:758: left oid 32 (ObjNum 92 snap 0 seq_num 92) 2024-04-22T18:39:16.088 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:16.090+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:16.134 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:16.134 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:16.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:16.156 INFO:tasks.rados.rados.0.smithi089.stdout:759: finishing set_chunk to oid 49 2024-04-22T18:39:16.156 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 1544704 length: 1047552 2024-04-22T18:39:16.156 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:39:16.189 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:16.190+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:16.211 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:39:16.245 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:16.245 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:16.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:16.262 INFO:tasks.rados.rados.0.smithi089.stdout:760: expect (ObjNum 91 snap 0 seq_num 91) 2024-04-22T18:39:16.290 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:16.291+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:16.293 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:16.293 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:16.310 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:16.311 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:16.311 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:39:16.312 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:16.312 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 42, 2024-04-22T18:39:16.312 INFO:teuthology.orchestra.run.smithi089.stdout: "fsid": "9e25744c-eff1-4a4a-8d47-0dda406b2c35", 2024-04-22T18:39:16.312 INFO:teuthology.orchestra.run.smithi089.stdout: "created": "2024-04-22T18:38:26.841687+0000", 2024-04-22T18:39:16.312 INFO:teuthology.orchestra.run.smithi089.stdout: "modified": "2024-04-22T18:39:14.188490+0000", 2024-04-22T18:39:16.312 INFO:teuthology.orchestra.run.smithi089.stdout: "last_up_change": "2024-04-22T18:38:35.971820+0000", 2024-04-22T18:39:16.312 INFO:teuthology.orchestra.run.smithi089.stdout: "last_in_change": "2024-04-22T18:38:28.942710+0000", 2024-04-22T18:39:16.312 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-22T18:39:16.312 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_num": 5806080, 2024-04-22T18:39:16.312 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_set": [ 2024-04-22T18:39:16.312 INFO:teuthology.orchestra.run.smithi089.stdout: "nodeep-scrub", 2024-04-22T18:39:16.312 INFO:teuthology.orchestra.run.smithi089.stdout: "noscrub", 2024-04-22T18:39:16.312 INFO:teuthology.orchestra.run.smithi089.stdout: "pglog_hardlimit", 2024-04-22T18:39:16.313 INFO:teuthology.orchestra.run.smithi089.stdout: "purged_snapdirs", 2024-04-22T18:39:16.313 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_deletes", 2024-04-22T18:39:16.313 INFO:teuthology.orchestra.run.smithi089.stdout: "sortbitwise" 2024-04-22T18:39:16.313 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:39:16.313 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_version": 5, 2024-04-22T18:39:16.313 INFO:teuthology.orchestra.run.smithi089.stdout: "full_ratio": 0.94999998807907104, 2024-04-22T18:39:16.313 INFO:teuthology.orchestra.run.smithi089.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-22T18:39:16.313 INFO:teuthology.orchestra.run.smithi089.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-22T18:39:16.313 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_snapshot": "", 2024-04-22T18:39:16.313 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_max": 4, 2024-04-22T18:39:16.313 INFO:teuthology.orchestra.run.smithi089.stdout: "max_osd": 8, 2024-04-22T18:39:16.313 INFO:teuthology.orchestra.run.smithi089.stdout: "require_min_compat_client": "luminous", 2024-04-22T18:39:16.313 INFO:teuthology.orchestra.run.smithi089.stdout: "min_compat_client": "jewel", 2024-04-22T18:39:16.313 INFO:teuthology.orchestra.run.smithi089.stdout: "require_osd_release": "reef", 2024-04-22T18:39:16.313 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_crimson": false, 2024-04-22T18:39:16.314 INFO:teuthology.orchestra.run.smithi089.stdout: "pools": [ 2024-04-22T18:39:16.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.314 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 1, 2024-04-22T18:39:16.314 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": ".mgr", 2024-04-22T18:39:16.314 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:36.991610+0000", 2024-04-22T18:39:16.314 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:39:16.314 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:39:16.314 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:39:16.314 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:39:16.314 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:39:16.314 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:39:16.314 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:39:16.314 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:39:16.315 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:39:16.315 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:39:16.315 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:39:16.315 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:39:16.315 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 1, 2024-04-22T18:39:16.315 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 1, 2024-04-22T18:39:16.315 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 1, 2024-04-22T18:39:16.315 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 1, 2024-04-22T18:39:16.315 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 1, 2024-04-22T18:39:16.315 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:39:16.315 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:39:16.315 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:39:16.315 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:39:16.315 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:39:16.315 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:39:16.316 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:39:16.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.316 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "16", 2024-04-22T18:39:16.316 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:39:16.316 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:39:16.316 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-04-22T18:39:16.316 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:39:16.316 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:39:16.316 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:39:16.316 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:39:16.316 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:39:16.316 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:39:16.316 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:39:16.316 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:39:16.317 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:39:16.317 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:39:16.317 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:39:16.317 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:39:16.317 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:39:16.317 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:39:16.317 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:39:16.318 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:39:16.318 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:39:16.318 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:39:16.318 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:39:16.318 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:39:16.318 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:39:16.318 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:39:16.318 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:39:16.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.318 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:39:16.319 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:39:16.319 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:39:16.319 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:39:16.319 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:39:16.319 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:39:16.319 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:39:16.319 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:39:16.319 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:39:16.319 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:39:16.319 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:39:16.319 INFO:teuthology.orchestra.run.smithi089.stdout: "options": { 2024-04-22T18:39:16.319 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_max": 32, 2024-04-22T18:39:16.319 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_min": 1 2024-04-22T18:39:16.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.320 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-04-22T18:39:16.320 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": {} 2024-04-22T18:39:16.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.320 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:39:16.320 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 8, 2024-04-22T18:39:16.320 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 8, 2024-04-22T18:39:16.320 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.25, 2024-04-22T18:39:16.320 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2, 2024-04-22T18:39:16.320 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2, 2024-04-22T18:39:16.320 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.98099899291992188, 2024-04-22T18:39:16.320 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-22T18:39:16.320 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-22T18:39:16.320 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.321 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 2, 2024-04-22T18:39:16.321 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "rbd", 2024-04-22T18:39:16.321 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:37.622620+0000", 2024-04-22T18:39:16.321 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:39:16.321 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:39:16.321 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:39:16.321 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:39:16.321 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:39:16.321 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:39:16.321 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:39:16.321 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:39:16.322 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:39:16.322 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:39:16.322 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:39:16.322 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:39:16.322 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 8, 2024-04-22T18:39:16.322 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 8, 2024-04-22T18:39:16.322 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 8, 2024-04-22T18:39:16.322 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 8, 2024-04-22T18:39:16.322 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 8, 2024-04-22T18:39:16.322 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:39:16.322 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:39:16.322 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:39:16.322 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:39:16.323 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:39:16.323 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:39:16.323 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:39:16.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.323 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "17", 2024-04-22T18:39:16.323 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:39:16.323 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:39:16.323 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-04-22T18:39:16.323 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:39:16.323 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:39:16.323 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:39:16.323 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:39:16.323 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:39:16.323 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:39:16.324 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:39:16.324 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:39:16.324 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:39:16.324 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:39:16.324 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:39:16.324 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:39:16.324 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:39:16.324 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:39:16.324 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:39:16.325 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:39:16.325 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:39:16.325 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:39:16.325 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:39:16.325 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:39:16.325 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:39:16.325 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:39:16.325 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:39:16.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.326 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:39:16.326 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:39:16.326 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:39:16.326 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:39:16.326 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:39:16.327 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:39:16.327 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:39:16.327 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:39:16.327 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:39:16.327 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:39:16.327 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:39:16.327 INFO:teuthology.orchestra.run.smithi089.stdout: "options": {}, 2024-04-22T18:39:16.328 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-04-22T18:39:16.328 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd": {} 2024-04-22T18:39:16.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.328 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:39:16.328 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 1.9900000095367432, 2024-04-22T18:39:16.328 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 1.9900000095367432, 2024-04-22T18:39:16.329 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.87999999523162842, 2024-04-22T18:39:16.329 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 1.75, 2024-04-22T18:39:16.329 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 1.75, 2024-04-22T18:39:16.329 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.98099899291992188, 2024-04-22T18:39:16.329 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-22T18:39:16.329 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-22T18:39:16.329 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.329 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 3, 2024-04-22T18:39:16.329 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "low_tier", 2024-04-22T18:39:16.329 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:46.038005+0000", 2024-04-22T18:39:16.329 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:39:16.330 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:39:16.330 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:39:16.330 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:39:16.330 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:39:16.330 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:39:16.330 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:39:16.330 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:39:16.330 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:39:16.330 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:39:16.330 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:39:16.330 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:39:16.330 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 4, 2024-04-22T18:39:16.330 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 4, 2024-04-22T18:39:16.331 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 4, 2024-04-22T18:39:16.331 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 4, 2024-04-22T18:39:16.331 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 4, 2024-04-22T18:39:16.331 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:39:16.331 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:39:16.331 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:39:16.331 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:39:16.331 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:39:16.331 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:39:16.331 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:39:16.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.331 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "21", 2024-04-22T18:39:16.331 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:39:16.331 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:39:16.332 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-04-22T18:39:16.332 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:39:16.332 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:39:16.332 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:39:16.332 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:39:16.332 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:39:16.332 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:39:16.332 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:39:16.332 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:39:16.332 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:39:16.332 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:39:16.332 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:39:16.332 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:39:16.333 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:39:16.334 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:39:16.334 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:39:16.334 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:39:16.334 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:39:16.334 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:39:16.334 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:39:16.334 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:39:16.334 INFO:teuthology.orchestra.run.smithi089.stdout: "options": {}, 2024-04-22T18:39:16.334 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": {}, 2024-04-22T18:39:16.334 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:39:16.334 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 2, 2024-04-22T18:39:16.334 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 2, 2024-04-22T18:39:16.334 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.75, 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 1.5, 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 1.5, 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.98099899291992188, 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 4, 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "unique_pool_0", 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:46.483369+0000", 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:39:16.335 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:39:16.336 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:39:16.336 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:39:16.336 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:39:16.336 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:39:16.336 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:39:16.336 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:39:16.336 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:39:16.336 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 16, 2024-04-22T18:39:16.336 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 16, 2024-04-22T18:39:16.336 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 16, 2024-04-22T18:39:16.336 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 16, 2024-04-22T18:39:16.336 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 16, 2024-04-22T18:39:16.336 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:39:16.336 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:39:16.336 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "27", 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:39:16.337 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:39:16.338 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:39:16.338 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:39:16.338 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:39:16.338 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:39:16.338 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:39:16.338 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:39:16.338 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:39:16.338 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:39:16.338 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:39:16.338 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:39:16.338 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:39:16.338 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:39:16.338 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:39:16.338 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:39:16.339 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:39:16.339 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:39:16.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.339 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:39:16.339 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:39:16.339 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:39:16.339 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:39:16.339 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:39:16.339 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:39:16.339 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:39:16.339 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:39:16.339 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:39:16.339 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:39:16.339 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:39:16.339 INFO:teuthology.orchestra.run.smithi089.stdout: "options": { 2024-04-22T18:39:16.340 INFO:teuthology.orchestra.run.smithi089.stdout: "dedup_cdc_chunk_size": 131072, 2024-04-22T18:39:16.340 INFO:teuthology.orchestra.run.smithi089.stdout: "dedup_chunk_algorithm": "fastcdc", 2024-04-22T18:39:16.340 INFO:teuthology.orchestra.run.smithi089.stdout: "dedup_tier": 3, 2024-04-22T18:39:16.340 INFO:teuthology.orchestra.run.smithi089.stdout: "fingerprint_algorithm": "sha256" 2024-04-22T18:39:16.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.340 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-04-22T18:39:16.340 INFO:teuthology.orchestra.run.smithi089.stdout: "rados": {} 2024-04-22T18:39:16.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.340 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:39:16.340 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 2.5, 2024-04-22T18:39:16.340 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 2.5, 2024-04-22T18:39:16.340 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 1, 2024-04-22T18:39:16.340 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2.5, 2024-04-22T18:39:16.340 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2.5, 2024-04-22T18:39:16.341 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.98099899291992188, 2024-04-22T18:39:16.341 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-22T18:39:16.341 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-22T18:39:16.341 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.341 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.341 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:39:16.341 INFO:teuthology.orchestra.run.smithi089.stdout: "osds": [ 2024-04-22T18:39:16.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.341 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-04-22T18:39:16.341 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "bbf33cfb-0f07-4df6-b90f-83e9edb039a6", 2024-04-22T18:39:16.341 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:39:16.341 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:39:16.341 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:39:16.342 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:39:16.342 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:39:16.342 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:39:16.342 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:39:16.342 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 20, 2024-04-22T18:39:16.342 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:39:16.342 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:39:16.342 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:39:16.342 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.342 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.342 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6800", 2024-04-22T18:39:16.342 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:39:16.343 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.343 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.343 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:39:16.343 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.343 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.343 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6801", 2024-04-22T18:39:16.343 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:39:16.343 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.343 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.343 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:39:16.343 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.344 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.344 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6803", 2024-04-22T18:39:16.344 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:39:16.344 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.344 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.344 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:39:16.344 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.345 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.345 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6802", 2024-04-22T18:39:16.345 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:39:16.345 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.345 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.345 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6800/31890", 2024-04-22T18:39:16.345 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6801/31890", 2024-04-22T18:39:16.345 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6803/31890", 2024-04-22T18:39:16.345 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6802/31890", 2024-04-22T18:39:16.345 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:39:16.345 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:39:16.345 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:39:16.346 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.346 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-04-22T18:39:16.346 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "30415a8e-9f47-416f-9fd0-716c2cb3f562", 2024-04-22T18:39:16.346 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:39:16.346 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:39:16.346 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:39:16.346 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:39:16.346 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:39:16.346 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:39:16.346 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:39:16.346 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 20, 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6804", 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.347 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6805", 2024-04-22T18:39:16.348 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:39:16.348 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.348 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.348 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:39:16.348 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.348 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.348 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6807", 2024-04-22T18:39:16.348 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:39:16.348 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.348 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.348 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:39:16.348 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.349 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.349 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6806", 2024-04-22T18:39:16.349 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:39:16.349 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.349 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.349 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6804/31899", 2024-04-22T18:39:16.349 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6805/31899", 2024-04-22T18:39:16.349 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6807/31899", 2024-04-22T18:39:16.349 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6806/31899", 2024-04-22T18:39:16.349 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:39:16.349 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:39:16.349 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:39:16.350 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.350 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-04-22T18:39:16.350 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "b8da6b88-8355-4225-930d-bfed881089b7", 2024-04-22T18:39:16.350 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:39:16.350 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:39:16.350 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:39:16.350 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:39:16.350 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:39:16.350 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:39:16.350 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:39:16.350 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 15, 2024-04-22T18:39:16.351 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:39:16.351 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:39:16.351 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:39:16.351 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.351 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.351 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6812", 2024-04-22T18:39:16.351 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:39:16.351 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.351 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.351 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:39:16.351 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.351 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.352 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6813", 2024-04-22T18:39:16.352 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:39:16.352 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.352 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.352 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:39:16.352 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.352 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.352 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6815", 2024-04-22T18:39:16.352 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:39:16.352 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.352 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.353 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:39:16.353 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.353 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.353 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6814", 2024-04-22T18:39:16.353 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:39:16.353 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.353 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.353 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6812/31903", 2024-04-22T18:39:16.353 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6813/31903", 2024-04-22T18:39:16.354 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6815/31903", 2024-04-22T18:39:16.354 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6814/31903", 2024-04-22T18:39:16.354 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:39:16.354 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:39:16.354 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:39:16.354 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.354 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-04-22T18:39:16.354 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "3eaec913-9d7d-4b03-a28f-0c334e7c39f1", 2024-04-22T18:39:16.354 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:39:16.354 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:39:16.355 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:39:16.355 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 0.847991943359375, 2024-04-22T18:39:16.355 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:39:16.355 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:39:16.355 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:39:16.355 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 20, 2024-04-22T18:39:16.355 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:39:16.355 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:39:16.355 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:39:16.355 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.355 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.356 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6808", 2024-04-22T18:39:16.356 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:39:16.356 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.356 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.356 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:39:16.356 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.356 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.356 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6809", 2024-04-22T18:39:16.356 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:39:16.356 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.356 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.357 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:39:16.357 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.357 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.357 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6811", 2024-04-22T18:39:16.357 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:39:16.357 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.357 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.357 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:39:16.357 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.357 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.357 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6810", 2024-04-22T18:39:16.357 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:39:16.358 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.358 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.358 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6808/31913", 2024-04-22T18:39:16.358 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6809/31913", 2024-04-22T18:39:16.358 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6811/31913", 2024-04-22T18:39:16.358 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6810/31913", 2024-04-22T18:39:16.358 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:39:16.358 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:39:16.358 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:39:16.358 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.358 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-04-22T18:39:16.359 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "887acc3b-6f6b-49eb-a039-13050a2c7d64", 2024-04-22T18:39:16.359 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:39:16.359 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:39:16.359 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:39:16.359 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:39:16.359 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:39:16.359 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:39:16.359 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:39:16.359 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 20, 2024-04-22T18:39:16.359 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:39:16.359 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:39:16.359 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:39:16.359 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.359 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.360 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6808", 2024-04-22T18:39:16.360 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:39:16.360 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.360 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.360 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:39:16.360 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.360 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.360 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6809", 2024-04-22T18:39:16.360 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:39:16.360 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.360 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.361 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:39:16.361 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.361 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.361 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6811", 2024-04-22T18:39:16.361 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:39:16.361 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.361 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.361 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:39:16.361 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.361 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.361 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6810", 2024-04-22T18:39:16.362 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:39:16.362 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.362 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.362 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6808/31922", 2024-04-22T18:39:16.362 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6809/31922", 2024-04-22T18:39:16.362 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6811/31922", 2024-04-22T18:39:16.362 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6810/31922", 2024-04-22T18:39:16.362 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:39:16.362 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:39:16.362 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:39:16.363 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.363 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-04-22T18:39:16.363 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "dcaddfd0-f8d1-4c28-bda2-d17eed606b60", 2024-04-22T18:39:16.363 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:39:16.363 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:39:16.363 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:39:16.363 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:39:16.363 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:39:16.363 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:39:16.363 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:39:16.363 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 29, 2024-04-22T18:39:16.363 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:39:16.363 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:39:16.364 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:39:16.364 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.364 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.364 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6804", 2024-04-22T18:39:16.364 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:39:16.364 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.364 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.364 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:39:16.364 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.364 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.364 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6805", 2024-04-22T18:39:16.365 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:39:16.365 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.365 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.365 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:39:16.365 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.365 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.365 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6807", 2024-04-22T18:39:16.365 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:39:16.365 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.365 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.366 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:39:16.366 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.366 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.366 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6806", 2024-04-22T18:39:16.366 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:39:16.366 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.366 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.366 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6804/31921", 2024-04-22T18:39:16.366 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6805/31921", 2024-04-22T18:39:16.366 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6807/31921", 2024-04-22T18:39:16.366 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6806/31921", 2024-04-22T18:39:16.366 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "acf0527b-3020-4b0a-b64c-5068d21a5781", 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 20, 2024-04-22T18:39:16.367 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6800", 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6801", 2024-04-22T18:39:16.368 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:39:16.369 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.369 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.369 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:39:16.369 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.369 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.369 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6803", 2024-04-22T18:39:16.369 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:39:16.369 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.369 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.369 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:39:16.369 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.370 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.370 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6802", 2024-04-22T18:39:16.370 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:39:16.370 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.370 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.370 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6800/31920", 2024-04-22T18:39:16.370 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6801/31920", 2024-04-22T18:39:16.370 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6803/31920", 2024-04-22T18:39:16.370 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6802/31920", 2024-04-22T18:39:16.370 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:39:16.371 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:39:16.371 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:39:16.371 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.371 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-04-22T18:39:16.371 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "305394bc-be3d-420b-88d7-d273670540b1", 2024-04-22T18:39:16.371 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:39:16.371 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:39:16.371 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:39:16.371 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:39:16.371 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:39:16.371 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:39:16.371 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:39:16.371 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 15, 2024-04-22T18:39:16.372 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:39:16.372 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:39:16.372 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:39:16.372 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.372 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.372 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6812", 2024-04-22T18:39:16.372 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:39:16.372 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.372 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.372 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:39:16.372 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.373 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.373 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6813", 2024-04-22T18:39:16.373 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:39:16.373 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.373 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.373 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:39:16.373 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.373 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.373 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6815", 2024-04-22T18:39:16.373 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:39:16.373 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.373 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.374 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:39:16.374 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:39:16.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.374 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:39:16.374 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6814", 2024-04-22T18:39:16.374 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:39:16.374 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.374 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.374 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6812/31939", 2024-04-22T18:39:16.374 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6813/31939", 2024-04-22T18:39:16.374 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6815/31939", 2024-04-22T18:39:16.374 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6814/31939", 2024-04-22T18:39:16.374 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:39:16.375 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:39:16.375 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:39:16.375 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:16.375 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.375 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:39:16.375 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_xinfo": [ 2024-04-22T18:39:16.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.375 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-04-22T18:39:16.375 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:39:16.375 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:39:16.375 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:39:16.375 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:39:16.375 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:39:16.375 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.023620+0000", 2024-04-22T18:39:16.375 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.099729+0000", 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:39:16.376 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.074483+0000", 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.122275+0000", 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.378 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-04-22T18:39:16.378 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:39:16.378 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:39:16.378 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:39:16.378 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:39:16.378 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:39:16.378 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.116180+0000", 2024-04-22T18:39:16.379 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:39:16.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.379 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-04-22T18:39:16.379 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:39:16.379 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:39:16.379 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:39:16.379 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:39:16.379 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:39:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.108716+0000", 2024-04-22T18:39:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:39:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-04-22T18:39:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:39:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:39:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:39:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:39:16.381 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:39:16.381 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.005302+0000", 2024-04-22T18:39:16.381 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:39:16.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:16.381 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-04-22T18:39:16.381 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:39:16.381 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:39:16.381 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:39:16.382 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:39:16.382 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:39:16.382 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:38:34.108554+0000", 2024-04-22T18:39:16.382 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:39:16.382 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.382 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:39:16.383 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap": [], 2024-04-22T18:39:16.383 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_items": [], 2024-04-22T18:39:16.383 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_primaries": [], 2024-04-22T18:39:16.383 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_temp": [], 2024-04-22T18:39:16.383 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_temp": [], 2024-04-22T18:39:16.383 INFO:teuthology.orchestra.run.smithi089.stdout: "blocklist": {}, 2024-04-22T18:39:16.383 INFO:teuthology.orchestra.run.smithi089.stdout: "range_blocklist": {}, 2024-04-22T18:39:16.384 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profiles": { 2024-04-22T18:39:16.384 INFO:teuthology.orchestra.run.smithi089.stdout: "default": { 2024-04-22T18:39:16.384 INFO:teuthology.orchestra.run.smithi089.stdout: "crush-failure-domain": "osd", 2024-04-22T18:39:16.384 INFO:teuthology.orchestra.run.smithi089.stdout: "k": "2", 2024-04-22T18:39:16.384 INFO:teuthology.orchestra.run.smithi089.stdout: "m": "1", 2024-04-22T18:39:16.384 INFO:teuthology.orchestra.run.smithi089.stdout: "plugin": "jerasure", 2024-04-22T18:39:16.384 INFO:teuthology.orchestra.run.smithi089.stdout: "technique": "reed_sol_van" 2024-04-22T18:39:16.384 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:16.385 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps_queue": [], 2024-04-22T18:39:16.385 INFO:teuthology.orchestra.run.smithi089.stdout: "new_removed_snaps": [], 2024-04-22T18:39:16.385 INFO:teuthology.orchestra.run.smithi089.stdout: "new_purged_snaps": [], 2024-04-22T18:39:16.385 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_node_flags": {}, 2024-04-22T18:39:16.385 INFO:teuthology.orchestra.run.smithi089.stdout: "device_class_flags": {}, 2024-04-22T18:39:16.385 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode": { 2024-04-22T18:39:16.385 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_enabled": false, 2024-04-22T18:39:16.386 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_bucket_count": 0, 2024-04-22T18:39:16.386 INFO:teuthology.orchestra.run.smithi089.stdout: "degraded_stretch_mode": 0, 2024-04-22T18:39:16.386 INFO:teuthology.orchestra.run.smithi089.stdout: "recovering_stretch_mode": 0, 2024-04-22T18:39:16.386 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_bucket": 0 2024-04-22T18:39:16.386 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:16.386 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:16.389 INFO:tasks.thrashosds.thrasher:j is {'epoch': 42, 'fsid': '9e25744c-eff1-4a4a-8d47-0dda406b2c35', 'created': '2024-04-22T18:38:26.841687+0000', 'modified': '2024-04-22T18:39:14.188490+0000', 'last_up_change': '2024-04-22T18:38:35.971820+0000', 'last_in_change': '2024-04-22T18:38:28.942710+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T18:38:36.991610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9809989929199219, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-22T18:38:37.622620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9809989929199219, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-04-22T18:38:46.038005+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9809989929199219, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-22T18:38:46.483369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'dedup_cdc_chunk_size': 131072, 'dedup_chunk_algorithm': 'fastcdc', 'dedup_tier': 3, 'fingerprint_algorithm': 'sha256'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9809989929199219, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}], 'osds': [{'osd': 0, 'uuid': 'bbf33cfb-0f07-4df6-b90f-83e9edb039a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6800', 'nonce': 31890}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6801', 'nonce': 31890}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6803', 'nonce': 31890}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6802', 'nonce': 31890}]}, 'public_addr': '172.21.15.89:6800/31890', 'cluster_addr': '172.21.15.89:6801/31890', 'heartbeat_back_addr': '172.21.15.89:6803/31890', 'heartbeat_front_addr': '172.21.15.89:6802/31890', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '30415a8e-9f47-416f-9fd0-716c2cb3f562', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6804', 'nonce': 31899}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6805', 'nonce': 31899}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6807', 'nonce': 31899}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6806', 'nonce': 31899}]}, 'public_addr': '172.21.15.89:6804/31899', 'cluster_addr': '172.21.15.89:6805/31899', 'heartbeat_back_addr': '172.21.15.89:6807/31899', 'heartbeat_front_addr': '172.21.15.89:6806/31899', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b8da6b88-8355-4225-930d-bfed881089b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6812', 'nonce': 31903}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6813', 'nonce': 31903}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6815', 'nonce': 31903}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6814', 'nonce': 31903}]}, 'public_addr': '172.21.15.89:6812/31903', 'cluster_addr': '172.21.15.89:6813/31903', 'heartbeat_back_addr': '172.21.15.89:6815/31903', 'heartbeat_front_addr': '172.21.15.89:6814/31903', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3eaec913-9d7d-4b03-a28f-0c334e7c39f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.847991943359375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6808', 'nonce': 31913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6809', 'nonce': 31913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6811', 'nonce': 31913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6810', 'nonce': 31913}]}, 'public_addr': '172.21.15.89:6808/31913', 'cluster_addr': '172.21.15.89:6809/31913', 'heartbeat_back_addr': '172.21.15.89:6811/31913', 'heartbeat_front_addr': '172.21.15.89:6810/31913', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '887acc3b-6f6b-49eb-a039-13050a2c7d64', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 31922}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 31922}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 31922}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 31922}]}, 'public_addr': '172.21.15.121:6808/31922', 'cluster_addr': '172.21.15.121:6809/31922', 'heartbeat_back_addr': '172.21.15.121:6811/31922', 'heartbeat_front_addr': '172.21.15.121:6810/31922', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'dcaddfd0-f8d1-4c28-bda2-d17eed606b60', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 31921}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 31921}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 31921}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 31921}]}, 'public_addr': '172.21.15.121:6804/31921', 'cluster_addr': '172.21.15.121:6805/31921', 'heartbeat_back_addr': '172.21.15.121:6807/31921', 'heartbeat_front_addr': '172.21.15.121:6806/31921', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'acf0527b-3020-4b0a-b64c-5068d21a5781', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 31920}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 31920}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 31920}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 31920}]}, 'public_addr': '172.21.15.121:6800/31920', 'cluster_addr': '172.21.15.121:6801/31920', 'heartbeat_back_addr': '172.21.15.121:6803/31920', 'heartbeat_front_addr': '172.21.15.121:6802/31920', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '305394bc-be3d-420b-88d7-d273670540b1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 31939}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 31939}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 31939}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 31939}]}, 'public_addr': '172.21.15.121:6812/31939', 'cluster_addr': '172.21.15.121:6813/31939', 'heartbeat_back_addr': '172.21.15.121:6815/31939', 'heartbeat_front_addr': '172.21.15.121:6814/31939', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.023620+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.099729+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.074483+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.122275+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.116180+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.108716+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.005302+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:38:34.108554+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-22T18:39:16.389 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-22T18:39:16.389 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:16.390 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:16.390 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:16.390 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:16.391 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:16.391 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:16.391 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:16.392 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:16.392 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:16.393+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:16.401 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:16.401 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:16.402 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:16.403 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:16.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:16.403 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:16.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.403 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:16.403 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:16.403 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.308824019999999, 2024-04-22T18:39:16.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:39:16.403 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.404 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.404 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.404 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.404 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.404 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:39:16.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.404 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:16.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:16.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:39:16.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:16.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:39:16.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:39:16.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:39:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:39:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:39:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:39:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:39:16.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:39:16.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:39:16.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:39:16.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:16.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:39:16.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:39:16.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:39:16.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:39:16.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:39:16.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:39:16.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:39:16.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:39:16.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:39:16.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:39:16.409 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.409 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.409 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.409 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2 4.8 4:12d9f826:::smithi08937929-1:head [write 1542129~474678 in=474678b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:16.410 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:39:16.410 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.208112750000002, 2024-04-22T18:39:16.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20435667099999999, 2024-04-22T18:39:16.410 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.410 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.410 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2 2024-04-22T18:39:16.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.411 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:39:16.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:39:16.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194568+0000", 2024-04-22T18:39:16.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:16.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194869+0000", 2024-04-22T18:39:16.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030115200000000001 2024-04-22T18:39:16.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194879+0000", 2024-04-22T18:39:16.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8479999999999996e-06 2024-04-22T18:39:16.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294954+0000", 2024-04-22T18:39:16.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100074702 2024-04-22T18:39:16.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296019+0000", 2024-04-22T18:39:16.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010649189999999999 2024-04-22T18:39:16.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296086+0000", 2024-04-22T18:39:16.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7261000000000005e-05 2024-04-22T18:39:16.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:16.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296173+0000", 2024-04-22T18:39:16.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6935999999999996e-05 2024-04-22T18:39:16.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297196+0000", 2024-04-22T18:39:16.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023415 2024-04-22T18:39:16.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398879+0000", 2024-04-22T18:39:16.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101682563 2024-04-22T18:39:16.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398906+0000", 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6526999999999999e-05 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398928+0000", 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2045e-05 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:53 4.3 e27/20 4:ca46fde9:::smithi08937929-14:head v 27'1, mlcod=0'0)", 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:39:16.415 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.154865547, 2024-04-22T18:39:16.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20214333400000001, 2024-04-22T18:39:16.416 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.416 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.416 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:39:16.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:39:16.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.247818+0000", 2024-04-22T18:39:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967296 2024-04-22T18:39:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248504+0000", 2024-04-22T18:39:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000686091 2024-04-22T18:39:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248512+0000", 2024-04-22T18:39:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1850000000000004e-06 2024-04-22T18:39:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.248517+0000", 2024-04-22T18:39:16.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5469999999999998e-06 2024-04-22T18:39:16.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.348632+0000", 2024-04-22T18:39:16.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100115069 2024-04-22T18:39:16.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.348644+0000", 2024-04-22T18:39:16.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2001000000000001e-05 2024-04-22T18:39:16.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449930+0000", 2024-04-22T18:39:16.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101286574 2024-04-22T18:39:16.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.449961+0000", 2024-04-22T18:39:16.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1046000000000001e-05 2024-04-22T18:39:16.419 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.419 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.419 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.420 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=62b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:16.420 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:39:16.420 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.107711796, 2024-04-22T18:39:16.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20408478299999999, 2024-04-22T18:39:16.420 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.420 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.420 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.420 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.420 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.420 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3 2024-04-22T18:39:16.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.420 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:39:16.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:39:16.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294970+0000", 2024-04-22T18:39:16.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:16.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294983+0000", 2024-04-22T18:39:16.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3429e-05 2024-04-22T18:39:16.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294993+0000", 2024-04-22T18:39:16.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0416e-05 2024-04-22T18:39:16.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295007+0000", 2024-04-22T18:39:16.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3742e-05 2024-04-22T18:39:16.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296793+0000", 2024-04-22T18:39:16.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017860420000000001 2024-04-22T18:39:16.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296841+0000", 2024-04-22T18:39:16.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7706e-05 2024-04-22T18:39:16.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:16.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296913+0000", 2024-04-22T18:39:16.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2578999999999997e-05 2024-04-22T18:39:16.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297946+0000", 2024-04-22T18:39:16.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001032802 2024-04-22T18:39:16.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498963+0000", 2024-04-22T18:39:16.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20101640400000001 2024-04-22T18:39:16.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498998+0000", 2024-04-22T18:39:16.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5759000000000001e-05 2024-04-22T18:39:16.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499057+0000", 2024-04-22T18:39:16.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8270999999999998e-05 2024-04-22T18:39:16.425 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.425 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.425 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.425 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:16.425 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:39:16.425 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.107671367999998, 2024-04-22T18:39:16.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20418823899999999, 2024-04-22T18:39:16.426 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.426 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:16.426 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.426 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.426 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.426 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4 2024-04-22T18:39:16.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.426 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:39:16.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295011+0000", 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295016+0000", 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2400000000000001e-06 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295022+0000", 2024-04-22T18:39:16.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8519999999999998e-06 2024-04-22T18:39:16.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295027+0000", 2024-04-22T18:39:16.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9749999999999998e-06 2024-04-22T18:39:16.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297126+0000", 2024-04-22T18:39:16.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020981070000000001 2024-04-22T18:39:16.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:16.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297167+0000", 2024-04-22T18:39:16.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1842999999999999e-05 2024-04-22T18:39:16.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499096+0000", 2024-04-22T18:39:16.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201928846 2024-04-22T18:39:16.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499142+0000", 2024-04-22T18:39:16.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6152000000000001e-05 2024-04-22T18:39:16.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499201+0000", 2024-04-22T18:39:16.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8190999999999999e-05 2024-04-22T18:39:16.430 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.430 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.430 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.430 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:70 4.3 e27/20 4:c6fa81cf:::smithi08937929-18:head v 27'8, mlcod=27'3)", 2024-04-22T18:39:16.430 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:16.430 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.445191452, 2024-04-22T18:39:16.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20440983300000001, 2024-04-22T18:39:16.430 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.430 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.431 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:16.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:16.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:16.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:16.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957839+0000", 2024-04-22T18:39:16.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000346981 2024-04-22T18:39:16.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957849+0000", 2024-04-22T18:39:16.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8060000000000006e-06 2024-04-22T18:39:16.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957856+0000", 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0319999999999998e-06 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958600+0000", 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074474700000000003 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958615+0000", 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4961e-05 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161867+0000", 2024-04-22T18:39:16.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20325188399999999 2024-04-22T18:39:16.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161902+0000", 2024-04-22T18:39:16.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4884e-05 2024-04-22T18:39:16.434 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.434 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.434 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.434 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:71 4.3 e27/20 4:c6fa81cf:::smithi08937929-18:head v 27'9, mlcod=27'3)", 2024-04-22T18:39:16.434 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:39:16.434 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.444821430000001, 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20407488600000001, 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957862+0000", 2024-04-22T18:39:16.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:16.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957867+0000", 2024-04-22T18:39:16.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8329999999999999e-06 2024-04-22T18:39:16.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957872+0000", 2024-04-22T18:39:16.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4530000000000004e-06 2024-04-22T18:39:16.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957877+0000", 2024-04-22T18:39:16.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8969999999999999e-06 2024-04-22T18:39:16.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959214+0000", 2024-04-22T18:39:16.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001337443 2024-04-22T18:39:16.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959236+0000", 2024-04-22T18:39:16.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1889e-05 2024-04-22T18:39:16.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161910+0000", 2024-04-22T18:39:16.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202673879 2024-04-22T18:39:16.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161937+0000", 2024-04-22T18:39:16.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7202000000000001e-05 2024-04-22T18:39:16.438 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.438 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.438 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.439 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:97 4.3 e27/20 4:c93c842e:::smithi08937929-25:head v 27'10, mlcod=27'3)", 2024-04-22T18:39:16.439 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:39:16.439 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.444803369999999, 2024-04-22T18:39:16.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.204078805, 2024-04-22T18:39:16.439 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.439 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.439 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:39:16.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:39:16.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957880+0000", 2024-04-22T18:39:16.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:16.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958337+0000", 2024-04-22T18:39:16.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045682399999999999 2024-04-22T18:39:16.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958346+0000", 2024-04-22T18:39:16.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6840000000000008e-06 2024-04-22T18:39:16.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958353+0000", 2024-04-22T18:39:16.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0330000000000003e-06 2024-04-22T18:39:16.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959462+0000", 2024-04-22T18:39:16.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011084440000000001 2024-04-22T18:39:16.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.959479+0000", 2024-04-22T18:39:16.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6654e-05 2024-04-22T18:39:16.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161940+0000", 2024-04-22T18:39:16.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202461112 2024-04-22T18:39:16.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161959+0000", 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9477999999999999e-05 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:98 4.3 e27/20 4:c93c842e:::smithi08937929-25:head v 27'11, mlcod=27'3)", 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.444325265, 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203627161, 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:39:16.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:39:16.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958358+0000", 2024-04-22T18:39:16.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:16.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958880+0000", 2024-04-22T18:39:16.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052192199999999997 2024-04-22T18:39:16.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958887+0000", 2024-04-22T18:39:16.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:39:16.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958891+0000", 2024-04-22T18:39:16.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2899999999999996e-06 2024-04-22T18:39:16.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.060417+0000", 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10152593 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.060433+0000", 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6232e-05 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161964+0000", 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101531234 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161986+0000", 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1090999999999998e-05 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:99 4.3 e27/20 4:c93c842e:::smithi08937929-25:head v 27'12, mlcod=27'3)", 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.443789349999999, 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20320666600000001, 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958894+0000", 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958897+0000", 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4429999999999999e-06 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958900+0000", 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4490000000000002e-06 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958902+0000", 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.322e-06 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161689+0000", 2024-04-22T18:39:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20278693 2024-04-22T18:39:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161712+0000", 2024-04-22T18:39:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2938999999999998e-05 2024-04-22T18:39:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.162077+0000", 2024-04-22T18:39:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036527800000000002 2024-04-22T18:39:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.162101+0000", 2024-04-22T18:39:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3785e-05 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.409326145000001, 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:39:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:39:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:39:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:39:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:39:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:39:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:39:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:39:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:39:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:39:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:39:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:39:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:39:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:39:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:39:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:39:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:39:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:39:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:39:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:39:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:39:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:39:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:39:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:39:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:39:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:39:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:27 3.2 3:7a6cba00:::smithi08937929-32 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.597045+0000", 2024-04-22T18:39:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.805638651999999, 2024-04-22T18:39:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20443814799999999, 2024-04-22T18:39:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597045+0000", 2024-04-22T18:39:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597045+0000", 2024-04-22T18:39:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597043+0000", 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597053+0000", 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0278999999999999e-05 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597074+0000", 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0710999999999999e-05 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.697204+0000", 2024-04-22T18:39:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10012958299999999 2024-04-22T18:39:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.698362+0000", 2024-04-22T18:39:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011587629999999999 2024-04-22T18:39:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.698440+0000", 2024-04-22T18:39:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7453000000000005e-05 2024-04-22T18:39:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.698593+0000", 2024-04-22T18:39:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153449 2024-04-22T18:39:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.699048+0000", 2024-04-22T18:39:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000454287 2024-04-22T18:39:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.801433+0000", 2024-04-22T18:39:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102385664 2024-04-22T18:39:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.801462+0000", 2024-04-22T18:39:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9153e-05 2024-04-22T18:39:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.801483+0000", 2024-04-22T18:39:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0707e-05 2024-04-22T18:39:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.301026354000001, 2024-04-22T18:39:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:39:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:39:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:39:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:39:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:39:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:39:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:39:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:39:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:39:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:39:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:39:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:39:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:52 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433164+0000", 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.9695200599999998, 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201852593, 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.468 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.469 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433164+0000", 2024-04-22T18:39:16.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433164+0000", 2024-04-22T18:39:16.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433163+0000", 2024-04-22T18:39:16.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:16.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433166+0000", 2024-04-22T18:39:16.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.807e-06 2024-04-22T18:39:16.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433173+0000", 2024-04-22T18:39:16.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6279999999999999e-06 2024-04-22T18:39:16.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433176+0000", 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.056e-06 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.533628+0000", 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100452766 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.533687+0000", 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8270000000000003e-05 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.533794+0000", 2024-04-22T18:39:16.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010775199999999999 2024-04-22T18:39:16.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633833+0000", 2024-04-22T18:39:16.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100038839 2024-04-22T18:39:16.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.634932+0000", 2024-04-22T18:39:16.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010983810000000001 2024-04-22T18:39:16.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.634978+0000", 2024-04-22T18:39:16.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6443999999999998e-05 2024-04-22T18:39:16.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635016+0000", 2024-04-22T18:39:16.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8145999999999997e-05 2024-04-22T18:39:16.473 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.473 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.473 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.473 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.9694827259999999, 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:39:16.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:39:16.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:39:16.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:39:16.478 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.478 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.478 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.478 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:16.478 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:16.478 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.878396918, 2024-04-22T18:39:16.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:39:16.479 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.479 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.479 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:39:16.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.479 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:16.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:39:16.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:39:16.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:16.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:39:16.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:39:16.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:39:16.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:39:16.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:39:16.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:39:16.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:39:16.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:39:16.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:39:16.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:39:16.483 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.483 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.483 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.872596658, 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:16.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:39:16.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:39:16.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:39:16.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:39:16.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:39:16.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:39:16.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.867856315, 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:16.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:16.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:39:16.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:16.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:39:16.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:39:16.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:39:16.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:39:16.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:39:16.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:39:16.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:39:16.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:39:16.494 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.494 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.494 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.494 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:16.494 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:16.494 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.8673834170000001, 2024-04-22T18:39:16.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:39:16.494 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.494 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.494 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.494 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.495 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.495 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:39:16.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.495 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:16.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:16.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:39:16.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:16.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:39:16.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:39:16.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:39:16.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:39:16.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:16.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:39:16.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:39:16.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:39:16.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:39:16.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:39:16.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:39:16.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:39:16.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:39:16.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:39:16.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:39:16.499 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.499 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.499 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.500 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:16.500 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:16.500 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.8673634959999998, 2024-04-22T18:39:16.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:39:16.500 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.500 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:16.500 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.500 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.500 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.500 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:39:16.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.500 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:16.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:16.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:16.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:16.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:39:16.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:39:16.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:39:16.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:39:16.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:39:16.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:39:16.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:39:16.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:39:16.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:16.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:39:16.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:39:16.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:39:16.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:39:16.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:39:16.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:39:16.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:39:16.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:39:16.504 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.504 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.505 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.505 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.505 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.505 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:16.505 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:16.507 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:16.507 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:16.507+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:16.507 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:16.513 INFO:tasks.rados.rados.0.smithi089.stdout:761: expect (ObjNum 73 snap 0 seq_num 73) 2024-04-22T18:39:16.607 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:16.608+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:16.650 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:16.650 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:16.650 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:16.650 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:16.659 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:16.707 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:16.708+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:16.739 INFO:tasks.rados.rados.0.smithi089.stdout:756: done (5 left) 2024-04-22T18:39:16.740 INFO:tasks.rados.rados.0.smithi089.stdout:758: done (4 left) 2024-04-22T18:39:16.740 INFO:tasks.rados.rados.0.smithi089.stdout:759: done (3 left) 2024-04-22T18:39:16.740 INFO:tasks.rados.rados.0.smithi089.stdout:760: done (2 left) 2024-04-22T18:39:16.740 INFO:tasks.rados.rados.0.smithi089.stdout:761: done (1 left) 2024-04-22T18:39:16.740 INFO:tasks.rados.rados.0.smithi089.stdout:762: done (0 left) 2024-04-22T18:39:16.740 INFO:tasks.rados.rados.0.smithi089.stdout:763: read oid 39 snap -1 2024-04-22T18:39:16.740 INFO:tasks.rados.rados.0.smithi089.stdout:763: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-22T18:39:16.740 INFO:tasks.rados.rados.0.smithi089.stdout:764: read oid 48 snap -1 2024-04-22T18:39:16.740 INFO:tasks.rados.rados.0.smithi089.stdout:764: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-22T18:39:16.740 INFO:tasks.rados.rados.0.smithi089.stdout:765: tier_promote oid 49 2024-04-22T18:39:16.740 INFO:tasks.rados.rados.0.smithi089.stdout:766: delete oid 3 current snap is 0 2024-04-22T18:39:16.741 INFO:tasks.rados.rados.0.smithi089.stdout:766: done (3 left) 2024-04-22T18:39:16.741 INFO:tasks.rados.rados.0.smithi089.stdout:767: read oid 21 snap -1 2024-04-22T18:39:16.741 INFO:tasks.rados.rados.0.smithi089.stdout:767: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-22T18:39:16.741 INFO:tasks.rados.rados.0.smithi089.stdout:768: read oid 28 snap -1 2024-04-22T18:39:16.742 INFO:tasks.rados.rados.0.smithi089.stdout:768: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-22T18:39:16.742 INFO:tasks.rados.rados.0.smithi089.stdout:769: tier_flush oid 22 2024-04-22T18:39:16.742 INFO:tasks.rados.rados.0.smithi089.stdout:769: tier_flush oid 22 snap -1 2024-04-22T18:39:16.742 INFO:tasks.rados.rados.0.smithi089.stdout:770: read oid 4 snap -1 2024-04-22T18:39:16.742 INFO:tasks.rados.rados.0.smithi089.stdout:770: expect (ObjNum 81 snap 0 seq_num 81) 2024-04-22T18:39:16.742 INFO:tasks.rados.rados.0.smithi089.stdout:771: delete oid 30 current snap is 0 2024-04-22T18:39:16.744 INFO:tasks.rados.rados.0.smithi089.stdout:763: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-22T18:39:16.761 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:16.761 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:16.761 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:16.761 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:16.761 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:16.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:16.814 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:16.815+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:16.829 INFO:tasks.rados.rados.0.smithi089.stdout:770: expect (ObjNum 81 snap 0 seq_num 81) 2024-04-22T18:39:16.909 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:16.910+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:16.917 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:21 4.a 4:5aa97f94:::smithi08937929-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.764553834000001, 2024-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21044934000000001, 2024-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.918 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 21 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153183+0000", 2024-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154022+0000", 2024-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00083893300000000004 2024-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154032+0000", 2024-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0433e-05 2024-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154051+0000", 2024-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8227e-05 2024-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154103+0000", 2024-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2491999999999998e-05 2024-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154111+0000", 2024-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3470000000000001e-06 2024-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257848+0000", 2024-04-22T18:39:16.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103737703 2024-04-22T18:39:16.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257981+0000", 2024-04-22T18:39:16.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013290399999999999 2024-04-22T18:39:16.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:16.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.258138+0000", 2024-04-22T18:39:16.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015701199999999999 2024-04-22T18:39:16.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.261944+0000", 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038054650000000001 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363523+0000", 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101579239 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363577+0000", 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4095000000000003e-05 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363635+0000", 2024-04-22T18:39:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8362999999999997e-05 2024-04-22T18:39:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:22 4.a 4:5aa97f94:::smithi08937929-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.760479858, 2024-04-22T18:39:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30948398900000001, 2024-04-22T18:39:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.925 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.925 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 22 2024-04-22T18:39:16.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.925 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:16.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:16.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157259+0000", 2024-04-22T18:39:16.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:16.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157318+0000", 2024-04-22T18:39:16.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8906000000000001e-05 2024-04-22T18:39:16.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157327+0000", 2024-04-22T18:39:16.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:39:16.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157332+0000", 2024-04-22T18:39:16.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4689999999999996e-06 2024-04-22T18:39:16.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257479+0000", 2024-04-22T18:39:16.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014700999999999 2024-04-22T18:39:16.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:16.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257492+0000", 2024-04-22T18:39:16.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2574e-05 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259582+0000", 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020904019999999999 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259668+0000", 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5826000000000001e-05 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259772+0000", 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104048 2024-04-22T18:39:16.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363453+0000", 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103680183 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466692+0000", 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10323948199999999 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466720+0000", 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7886e-05 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466744+0000", 2024-04-22T18:39:16.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4049e-05 2024-04-22T18:39:16.930 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.930 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.930 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.930 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:23 4.a 4:5aa97f94:::smithi08937929-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:16.930 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:16.930 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.760402170999999, 2024-04-22T18:39:16.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39003820300000003, 2024-04-22T18:39:16.930 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.931 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.931 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 23 2024-04-22T18:39:16.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.931 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:16.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:16.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157337+0000", 2024-04-22T18:39:16.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:16.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157340+0000", 2024-04-22T18:39:16.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8609999999999998e-06 2024-04-22T18:39:16.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157343+0000", 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9859999999999999e-06 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157346+0000", 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8119999999999999e-06 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257563+0000", 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100217497 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:16.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257574+0000", 2024-04-22T18:39:16.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0777e-05 2024-04-22T18:39:16.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260112+0000", 2024-04-22T18:39:16.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002537681 2024-04-22T18:39:16.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260191+0000", 2024-04-22T18:39:16.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9320999999999995e-05 2024-04-22T18:39:16.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260333+0000", 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014177699999999999 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363792+0000", 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103458862 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547303+0000", 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18351162100000001 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547337+0000", 2024-04-22T18:39:16.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3640999999999999e-05 2024-04-22T18:39:16.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547376+0000", 2024-04-22T18:39:16.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8915000000000003e-05 2024-04-22T18:39:16.936 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.936 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.936 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.936 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:24 4.a 4:5aa97f94:::smithi08937929-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:16.936 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:16.936 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.760375740000001, 2024-04-22T18:39:16.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39017823499999998, 2024-04-22T18:39:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 24 2024-04-22T18:39:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157366+0000", 2024-04-22T18:39:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3120000000000001e-06 2024-04-22T18:39:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157370+0000", 2024-04-22T18:39:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1300000000000003e-06 2024-04-22T18:39:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157373+0000", 2024-04-22T18:39:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-04-22T18:39:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260553+0000", 2024-04-22T18:39:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103179996 2024-04-22T18:39:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260625+0000", 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2236000000000004e-05 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547414+0000", 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.286789181 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547471+0000", 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6192e-05 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547542+0000", 2024-04-22T18:39:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1622e-05 2024-04-22T18:39:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:49 4.a 4:5d2aee44:::smithi08937929-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.683577081999999, 2024-04-22T18:39:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31344442500000003, 2024-04-22T18:39:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.941 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 49 2024-04-22T18:39:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:16.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234158+0000", 2024-04-22T18:39:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999957 2024-04-22T18:39:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234901+0000", 2024-04-22T18:39:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074250800000000001 2024-04-22T18:39:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234914+0000", 2024-04-22T18:39:16.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3244999999999999e-05 2024-04-22T18:39:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234922+0000", 2024-04-22T18:39:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4590000000000001e-06 2024-04-22T18:39:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260665+0000", 2024-04-22T18:39:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025743423000000001 2024-04-22T18:39:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260779+0000", 2024-04-22T18:39:16.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114063 2024-04-22T18:39:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260913+0000", 2024-04-22T18:39:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013355499999999999 2024-04-22T18:39:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363798+0000", 2024-04-22T18:39:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102885003 2024-04-22T18:39:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547567+0000", 2024-04-22T18:39:16.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.183768971 2024-04-22T18:39:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547589+0000", 2024-04-22T18:39:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2507999999999999e-05 2024-04-22T18:39:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547607+0000", 2024-04-22T18:39:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8009999999999999e-05 2024-04-22T18:39:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.946 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:50 4.a 4:5d2aee44:::smithi08937929-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.678130903, 2024-04-22T18:39:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40856295399999998, 2024-04-22T18:39:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 50 2024-04-22T18:39:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239607+0000", 2024-04-22T18:39:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240309+0000", 2024-04-22T18:39:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070201599999999997 2024-04-22T18:39:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240320+0000", 2024-04-22T18:39:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1024e-05 2024-04-22T18:39:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240327+0000", 2024-04-22T18:39:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0049999999999998e-06 2024-04-22T18:39:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362065+0000", 2024-04-22T18:39:16.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.121737812 2024-04-22T18:39:16.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362121+0000", 2024-04-22T18:39:16.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5868000000000001e-05 2024-04-22T18:39:16.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:16.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362210+0000", 2024-04-22T18:39:16.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8533000000000001e-05 2024-04-22T18:39:16.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365322+0000", 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031121220000000002 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648054+0000", 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.282732549 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648105+0000", 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1276999999999997e-05 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648172+0000", 2024-04-22T18:39:16.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6398999999999999e-05 2024-04-22T18:39:16.952 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.952 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.952 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.952 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:51 4.a 4:5d2aee44:::smithi08937929-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:16.952 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:16.952 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.677408368999998, 2024-04-22T18:39:16.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40793057199999999, 2024-04-22T18:39:16.952 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.953 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.953 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.953 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.953 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.953 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 51 2024-04-22T18:39:16.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.953 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:16.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:16.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:16.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:16.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240336+0000", 2024-04-22T18:39:16.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:39:16.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240339+0000", 2024-04-22T18:39:16.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9009999999999998e-06 2024-04-22T18:39:16.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240342+0000", 2024-04-22T18:39:16.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:39:16.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363080+0000", 2024-04-22T18:39:16.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122738075 2024-04-22T18:39:16.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363164+0000", 2024-04-22T18:39:16.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3722999999999998e-05 2024-04-22T18:39:16.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:16.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363287+0000", 2024-04-22T18:39:16.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012301099999999999 2024-04-22T18:39:16.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466753+0000", 2024-04-22T18:39:16.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103466217 2024-04-22T18:39:16.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648205+0000", 2024-04-22T18:39:16.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.181451788 2024-04-22T18:39:16.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648231+0000", 2024-04-22T18:39:16.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6597999999999998e-05 2024-04-22T18:39:16.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648262+0000", 2024-04-22T18:39:16.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0871000000000003e-05 2024-04-22T18:39:16.958 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.958 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.958 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.958 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:52 4.a 4:5d2aee44:::smithi08937929-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:16.958 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:16.959 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.677395910000001, 2024-04-22T18:39:16.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40812731200000002, 2024-04-22T18:39:16.959 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.959 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:16.959 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.959 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.959 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.959 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 52 2024-04-22T18:39:16.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.959 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:16.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:16.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:16.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:16.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240345+0000", 2024-04-22T18:39:16.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4300000000000001e-06 2024-04-22T18:39:16.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240349+0000", 2024-04-22T18:39:16.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2570000000000001e-06 2024-04-22T18:39:16.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240358+0000", 2024-04-22T18:39:16.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-22T18:39:16.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363697+0000", 2024-04-22T18:39:16.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12333965299999999 2024-04-22T18:39:16.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:16.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363777+0000", 2024-04-22T18:39:16.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9883999999999996e-05 2024-04-22T18:39:16.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648296+0000", 2024-04-22T18:39:16.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451861299999998 2024-04-22T18:39:16.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648379+0000", 2024-04-22T18:39:16.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3302999999999998e-05 2024-04-22T18:39:16.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648471+0000", 2024-04-22T18:39:16.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2029999999999998e-05 2024-04-22T18:39:16.963 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.963 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.963 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.964 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:16.964 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:16.964 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.658092575000001, 2024-04-22T18:39:16.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:39:16.964 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.964 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.964 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.964 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.964 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.964 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:39:16.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.964 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:16.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:16.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:39:16.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:16.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:39:16.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:39:16.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:39:16.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:39:16.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:39:16.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:39:16.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:39:16.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:39:16.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:39:16.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:39:16.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:16.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:39:16.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:39:16.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:39:16.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:39:16.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:39:16.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:39:16.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:39:16.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:39:16.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:39:16.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:39:16.969 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.969 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.969 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.969 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:16.969 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:16.970 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.653310144999999, 2024-04-22T18:39:16.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:39:16.970 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.970 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.970 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.970 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.970 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.970 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:39:16.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.970 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:16.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:16.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:39:16.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:16.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:39:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:39:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:39:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:39:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:39:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:39:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:39:16.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:39:16.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:39:16.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:39:16.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:16.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:39:16.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:39:16.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:39:16.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:39:16.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:39:16.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:39:16.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:39:16.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:39:16.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:39:16.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:39:16.976 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.976 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.976 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.976 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:16.976 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:16.976 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.653215941999999, 2024-04-22T18:39:16.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:39:16.976 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.976 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.976 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.977 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.977 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.977 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:39:16.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.977 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:16.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:16.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:39:16.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:16.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:39:16.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:39:16.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:39:16.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:39:16.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:39:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:39:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:39:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:39:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:39:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:39:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:16.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:39:16.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:39:16.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:39:16.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:39:16.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:39:16.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:39:16.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:39:16.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:39:16.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:39:16.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:39:16.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.983 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.983 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:16.983 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:16.983 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.653195054000001, 2024-04-22T18:39:16.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:39:16.983 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.983 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:16.983 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.983 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:39:16.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:39:16.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:39:16.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:39:16.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:16.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:39:16.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:39:16.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:39:16.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:39:16.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:39:16.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:39:16.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:39:16.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:39:16.987 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.987 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.987 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.988 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:16.988 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:16.988 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.870187391000002, 2024-04-22T18:39:16.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:39:16.988 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.988 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.988 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.988 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.988 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.988 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:39:16.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.988 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:16.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:16.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:39:16.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:16.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:39:16.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:39:16.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:39:16.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:39:16.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:39:16.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:39:16.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:39:16.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:39:16.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:39:16.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:39:16.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:16.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:39:16.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:39:16.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:16.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:39:16.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:39:16.993 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.993 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:16.993 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:16.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.993 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:16.993 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:16.993 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.862355985000001, 2024-04-22T18:39:16.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:39:16.993 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:16.993 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:16.994 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:16.994 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:16.994 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:16.994 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:39:16.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.994 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:16.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:16.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:16.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:16.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:16.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:16.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:16.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:39:16.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:16.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:16.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:39:16.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:39:16.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:16.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:39:16.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:39:16.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:16.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:39:16.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:39:16.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:16.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:39:16.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:39:16.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:16.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:16.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:16.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:39:16.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:39:16.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:17.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:39:17.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:39:17.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:39:17.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:39:17.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:39:17.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:39:17.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:39:17.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:39:17.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:39:17.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:39:17.001 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.001 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.001 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.002 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:186 4.a 4:537c89f8:::smithi08937929-47:head [write 1992054~651681] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:17.002 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:17.002 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.514850450000001, 2024-04-22T18:39:17.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20325226800000001, 2024-04-22T18:39:17.002 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.002 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.002 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.002 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:17.002 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:17.002 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 186 2024-04-22T18:39:17.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.002 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:17.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:17.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402888+0000", 2024-04-22T18:39:17.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:17.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403454+0000", 2024-04-22T18:39:17.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056661000000000005 2024-04-22T18:39:17.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403465+0000", 2024-04-22T18:39:17.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0727000000000001e-05 2024-04-22T18:39:17.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403473+0000", 2024-04-22T18:39:17.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7039999999999994e-06 2024-04-22T18:39:17.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404592+0000", 2024-04-22T18:39:17.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001118827 2024-04-22T18:39:17.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404674+0000", 2024-04-22T18:39:17.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2793999999999994e-05 2024-04-22T18:39:17.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:17.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404779+0000", 2024-04-22T18:39:17.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104465 2024-04-22T18:39:17.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.406080+0000", 2024-04-22T18:39:17.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013006789999999999 2024-04-22T18:39:17.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606037+0000", 2024-04-22T18:39:17.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19995755600000001 2024-04-22T18:39:17.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606098+0000", 2024-04-22T18:39:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1075999999999996e-05 2024-04-22T18:39:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606142+0000", 2024-04-22T18:39:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3411999999999997e-05 2024-04-22T18:39:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:187 4.a 4:537c89f8:::smithi08937929-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 21.514260310000001, 2024-04-22T18:39:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203643623, 2024-04-22T18:39:17.012 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.012 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.012 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.012 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:17.012 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:17.012 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 187 2024-04-22T18:39:17.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.012 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:17.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:17.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:17.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:17.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403501+0000", 2024-04-22T18:39:17.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2146000000000001e-05 2024-04-22T18:39:17.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403506+0000", 2024-04-22T18:39:17.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3040000000000004e-06 2024-04-22T18:39:17.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403511+0000", 2024-04-22T18:39:17.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9599999999999999e-06 2024-04-22T18:39:17.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405587+0000", 2024-04-22T18:39:17.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002076296 2024-04-22T18:39:17.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405664+0000", 2024-04-22T18:39:17.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6316000000000006e-05 2024-04-22T18:39:17.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:17.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405785+0000", 2024-04-22T18:39:17.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121276 2024-04-22T18:39:17.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.408030+0000", 2024-04-22T18:39:17.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022452169999999999 2024-04-22T18:39:17.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607059+0000", 2024-04-22T18:39:17.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199029135 2024-04-22T18:39:17.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607094+0000", 2024-04-22T18:39:17.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4686e-05 2024-04-22T18:39:17.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607123+0000", 2024-04-22T18:39:17.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9113000000000001e-05 2024-04-22T18:39:17.017 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.017 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.017 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.018 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:439 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:39:17.018 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.905017+0000", 2024-04-22T18:39:17.018 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.012722896, 2024-04-22T18:39:17.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.200452043, 2024-04-22T18:39:17.018 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.018 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:17.018 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.018 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:17.018 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:17.018 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 439 2024-04-22T18:39:17.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.018 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.905017+0000", 2024-04-22T18:39:17.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.905017+0000", 2024-04-22T18:39:17.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.905015+0000", 2024-04-22T18:39:17.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:17.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.905023+0000", 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2539999999999992e-06 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.905035+0000", 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2145e-05 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:00.005141+0000", 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100105518 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:00.105286+0000", 2024-04-22T18:39:17.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100145145 2024-04-22T18:39:17.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:00.105394+0000", 2024-04-22T18:39:17.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010877 2024-04-22T18:39:17.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:00.105469+0000", 2024-04-22T18:39:17.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4445999999999998e-05 2024-04-22T18:39:17.022 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.022 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.022 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.023 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:811 4.5 4:a4a14dff:::smithi08937929-40:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-22T18:39:17.023 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:17.023 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 12.796917853, 2024-04-22T18:39:17.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242306563, 2024-04-22T18:39:17.023 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.023 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:17.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:17.023 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 811 2024-04-22T18:39:17.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.023 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:17.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:17.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120821+0000", 2024-04-22T18:39:17.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:17.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120825+0000", 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.506e-06 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120829+0000", 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0849999999999999e-06 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120833+0000", 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9609999999999999e-06 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221263+0000", 2024-04-22T18:39:17.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10043045 2024-04-22T18:39:17.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221337+0000", 2024-04-22T18:39:17.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4010000000000005e-05 2024-04-22T18:39:17.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:17.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221448+0000", 2024-04-22T18:39:17.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011065699999999999 2024-04-22T18:39:17.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.222541+0000", 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010932610000000001 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.362952+0000", 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.140410535 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.363102+0000", 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015041499999999999 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.363129+0000", 2024-04-22T18:39:17.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6574999999999999e-05 2024-04-22T18:39:17.028 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.028 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.028 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.028 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:933 4.a 4:5d2aee44:::smithi08937929-13:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:17.028 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:17.028 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.2729942150000007, 2024-04-22T18:39:17.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202744591, 2024-04-22T18:39:17.029 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.029 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:39:17.029 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.029 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:17.029 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:17.029 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 933 2024-04-22T18:39:17.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.029 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:17.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:17.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644743+0000", 2024-04-22T18:39:17.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:17.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644754+0000", 2024-04-22T18:39:17.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0807e-05 2024-04-22T18:39:17.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644768+0000", 2024-04-22T18:39:17.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3844e-05 2024-04-22T18:39:17.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.744860+0000", 2024-04-22T18:39:17.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100092835 2024-04-22T18:39:17.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.745084+0000", 2024-04-22T18:39:17.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022401699999999999 2024-04-22T18:39:17.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:17.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.745090+0000", 2024-04-22T18:39:17.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2970000000000003e-06 2024-04-22T18:39:17.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.847381+0000", 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102291544 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.847490+0000", 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010891399999999999 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:941 4.a 4:537c89f8:::smithi08937929-47:head [sparse-read 0~1211253] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:08.141214+0000", 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.7765257979999998, 2024-04-22T18:39:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20061749400000001, 2024-04-22T18:39:17.034 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.034 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:17.034 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.034 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:17.034 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:17.034 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 941 2024-04-22T18:39:17.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.034 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141214+0000", 2024-04-22T18:39:17.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141214+0000", 2024-04-22T18:39:17.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141213+0000", 2024-04-22T18:39:17.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:17.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141219+0000", 2024-04-22T18:39:17.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6189999999999999e-06 2024-04-22T18:39:17.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141225+0000", 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0469999999999999e-06 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.141233+0000", 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7740000000000004e-06 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.341440+0000", 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.200207095 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.341535+0000", 2024-04-22T18:39:17.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5159000000000002e-05 2024-04-22T18:39:17.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:08.341831+0000", 2024-04-22T18:39:17.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00029652699999999999 2024-04-22T18:39:17.037 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.038 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.038 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.038 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.038 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.038 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:17.039 INFO:tasks.rados.rados.0.smithi089.stdout:769: got (0) Success 2024-04-22T18:39:17.039 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-22T18:39:17.039 INFO:tasks.rados.rados.0.smithi089.stdout:764: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-22T18:39:17.040 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:17.040+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:17.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:17.088 INFO:tasks.rados.rados.0.smithi089.stdout:768: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-22T18:39:17.140 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:17.141+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:17.165 INFO:tasks.rados.rados.0.smithi089.stdout:767: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-22T18:39:17.183 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:17.183 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:17.184 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:17.184 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:17.192 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:17.201 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:17.203+0000 7f1e2fd26640 -1 osd.2 42 handle_osd_map injecting map crc failure 2024-04-22T18:39:17.204 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:39:17.246 INFO:tasks.rados.rados.0.smithi089.stdout:765: got (0) Success 2024-04-22T18:39:17.246 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:39:17.246 INFO:tasks.rados.rados.0.smithi089.stdout:763: done (7 left) 2024-04-22T18:39:17.246 INFO:tasks.rados.rados.0.smithi089.stdout:764: done (6 left) 2024-04-22T18:39:17.246 INFO:tasks.rados.rados.0.smithi089.stdout:765: done (5 left) 2024-04-22T18:39:17.246 INFO:tasks.rados.rados.0.smithi089.stdout:767: done (4 left) 2024-04-22T18:39:17.247 INFO:tasks.rados.rados.0.smithi089.stdout:768: done (3 left) 2024-04-22T18:39:17.247 INFO:tasks.rados.rados.0.smithi089.stdout:769: done (2 left) 2024-04-22T18:39:17.247 INFO:tasks.rados.rados.0.smithi089.stdout:770: done (1 left) 2024-04-22T18:39:17.247 INFO:tasks.rados.rados.0.smithi089.stdout:771: done (0 left) 2024-04-22T18:39:17.247 INFO:tasks.rados.rados.0.smithi089.stdout:772: delete oid 3 current snap is 0 2024-04-22T18:39:17.276 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:17.277+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:17.333 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:17.333 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:17.333 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:17.333 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:17.333 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:17.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:17.342 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:17.343+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:17.347 INFO:tasks.rados.rados.0.smithi089.stdout:772: done (0 left) 2024-04-22T18:39:17.348 INFO:tasks.rados.rados.0.smithi089.stdout:773: read oid 19 snap -1 2024-04-22T18:39:17.348 INFO:tasks.rados.rados.0.smithi089.stdout:773: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-22T18:39:17.348 INFO:tasks.rados.rados.0.smithi089.stdout:774: tier_evict oid 47 2024-04-22T18:39:17.348 INFO:tasks.rados.rados.0.smithi089.stdout:774: tier_evict oid 47 snap -1 2024-04-22T18:39:17.348 INFO:tasks.rados.rados.0.smithi089.stdout:775: set_chunk oid 33 target oid 2024-04-22T18:39:17.443 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:17.444+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:17.451 INFO:tasks.rados.rados.0.smithi089.stdout:775: set_chunk oid 33 offset: 199168 length: 189952 target oid 04a7e543b8f78799c6fffbecde30510835425b06ca6ea850e398652b7da4f2cc offset: 0 snap -1 2024-04-22T18:39:17.451 INFO:tasks.rados.rados.0.smithi089.stdout:776: write (excl) oid 38 current snap is 0 2024-04-22T18:39:17.451 INFO:tasks.rados.rados.0.smithi089.stdout:776: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2024-04-22T18:39:17.457 INFO:tasks.rados.rados.0.smithi089.stdout:776: writing smithi08937929-38 from 430279 to 1125668 tid 1 2024-04-22T18:39:17.464 INFO:tasks.rados.rados.0.smithi089.stdout:776: writing smithi08937929-38 from 1876518 to 2530876 tid 2 2024-04-22T18:39:17.470 INFO:tasks.rados.rados.0.smithi089.stdout:776: writing smithi08937929-38 from 3031089 to 3567886 tid 3 2024-04-22T18:39:17.470 INFO:tasks.rados.rados.0.smithi089.stdout:777: read oid 8 snap -1 2024-04-22T18:39:17.470 INFO:tasks.rados.rados.0.smithi089.stdout:777: expect deleted 2024-04-22T18:39:17.470 INFO:tasks.rados.rados.0.smithi089.stdout:778: read oid 10 snap -1 2024-04-22T18:39:17.471 INFO:tasks.rados.rados.0.smithi089.stdout:778: expect deleted 2024-04-22T18:39:17.471 INFO:tasks.rados.rados.0.smithi089.stdout:779: read oid 11 snap -1 2024-04-22T18:39:17.471 INFO:tasks.rados.rados.0.smithi089.stdout:779: expect (ObjNum 86 snap 0 seq_num 86) 2024-04-22T18:39:17.471 INFO:tasks.rados.rados.0.smithi089.stdout:780: tier_flush oid 35 2024-04-22T18:39:17.471 INFO:tasks.rados.rados.0.smithi089.stdout:780: tier_flush oid 35 snap -1 2024-04-22T18:39:17.471 INFO:tasks.rados.rados.0.smithi089.stdout:781: read oid 16 snap -1 2024-04-22T18:39:17.471 INFO:tasks.rados.rados.0.smithi089.stdout:781: expect (ObjNum 71 snap 0 seq_num 71) 2024-04-22T18:39:17.471 INFO:tasks.rados.rados.0.smithi089.stdout:782: delete oid 48 current snap is 0 2024-04-22T18:39:17.471 INFO:tasks.rados.rados.0.smithi089.stdout:773: expect (ObjNum 18 snap 0 seq_num 18) 2024-04-22T18:39:17.486 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.744710396000002, 2024-04-22T18:39:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30520528099999999, 2024-04-22T18:39:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 35 2024-04-22T18:39:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741852+0000", 2024-04-22T18:39:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741927+0000", 2024-04-22T18:39:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5123000000000006e-05 2024-04-22T18:39:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741953+0000", 2024-04-22T18:39:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5378999999999999e-05 2024-04-22T18:39:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741977+0000", 2024-04-22T18:39:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4431999999999999e-05 2024-04-22T18:39:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844243+0000", 2024-04-22T18:39:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102265525 2024-04-22T18:39:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844330+0000", 2024-04-22T18:39:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7417999999999995e-05 2024-04-22T18:39:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844470+0000", 2024-04-22T18:39:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000140197 2024-04-22T18:39:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944767+0000", 2024-04-22T18:39:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100296409 2024-04-22T18:39:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.046968+0000", 2024-04-22T18:39:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10220148499999999 2024-04-22T18:39:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047018+0000", 2024-04-22T18:39:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9206000000000003e-05 2024-04-22T18:39:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047059+0000", 2024-04-22T18:39:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1496999999999999e-05 2024-04-22T18:39:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.744579872000003, 2024-04-22T18:39:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:39:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:39:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:39:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:39:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:39:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:39:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:39:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:39:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:39:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:39:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:39:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:39:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:39:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:39:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:39:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:39:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:39:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:39:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:39:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:39:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:39:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.744540180999998, 2024-04-22T18:39:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:39:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:39:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:39:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:39:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:39:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:39:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:39:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:39:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:39:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:39:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:39:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:39:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:39:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:39:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:39:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:39:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:39:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:39:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:39:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:39:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:39:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:39:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:39:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 35.500359134, 2024-04-22T18:39:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402419109, 2024-04-22T18:39:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 68 2024-04-22T18:39:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986204+0000", 2024-04-22T18:39:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986208+0000", 2024-04-22T18:39:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8149999999999999e-06 2024-04-22T18:39:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986214+0000", 2024-04-22T18:39:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.604e-06 2024-04-22T18:39:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986218+0000", 2024-04-22T18:39:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6689999999999999e-06 2024-04-22T18:39:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986833+0000", 2024-04-22T18:39:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000615648 2024-04-22T18:39:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986891+0000", 2024-04-22T18:39:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7915999999999999e-05 2024-04-22T18:39:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986981+0000", 2024-04-22T18:39:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759999999999994e-05 2024-04-22T18:39:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.987584+0000", 2024-04-22T18:39:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060270199999999999 2024-04-22T18:39:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388527+0000", 2024-04-22T18:39:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40094366999999997 2024-04-22T18:39:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388583+0000", 2024-04-22T18:39:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5383000000000002e-05 2024-04-22T18:39:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388624+0000", 2024-04-22T18:39:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501999999999997e-05 2024-04-22T18:39:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 35.500343344999997, 2024-04-22T18:39:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:39:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:39:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:39:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:39:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:39:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:39:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:39:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:39:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:39:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:39:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:39:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:39:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:39:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:39:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:39:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:39:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:39:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.383950335999998, 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305563733, 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 87 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102613+0000", 2024-04-22T18:39:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102668+0000", 2024-04-22T18:39:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4669000000000001e-05 2024-04-22T18:39:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102680+0000", 2024-04-22T18:39:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2026e-05 2024-04-22T18:39:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102687+0000", 2024-04-22T18:39:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8339999999999999e-06 2024-04-22T18:39:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405644+0000", 2024-04-22T18:39:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30295623199999999 2024-04-22T18:39:17.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405743+0000", 2024-04-22T18:39:17.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.98e-05 2024-04-22T18:39:17.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405868+0000", 2024-04-22T18:39:17.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012457899999999999 2024-04-22T18:39:17.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407047+0000", 2024-04-22T18:39:17.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011791449999999999 2024-04-22T18:39:17.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408088+0000", 2024-04-22T18:39:17.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010411680000000001 2024-04-22T18:39:17.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408135+0000", 2024-04-22T18:39:17.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6255999999999999e-05 2024-04-22T18:39:17.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408178+0000", 2024-04-22T18:39:17.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2877000000000003e-05 2024-04-22T18:39:17.522 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.522 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.522 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.522 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:17.522 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:17.522 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 34.383870407000003, 2024-04-22T18:39:17.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30560089000000001, 2024-04-22T18:39:17.522 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.522 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.522 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.523 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.523 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.523 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 88 2024-04-22T18:39:17.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.523 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:17.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:17.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102693+0000", 2024-04-22T18:39:17.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:17.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102698+0000", 2024-04-22T18:39:17.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9699999999999998e-06 2024-04-22T18:39:17.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102708+0000", 2024-04-22T18:39:17.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.006e-05 2024-04-22T18:39:17.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102715+0000", 2024-04-22T18:39:17.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6599999999999995e-06 2024-04-22T18:39:17.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406637+0000", 2024-04-22T18:39:17.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392130099999998 2024-04-22T18:39:17.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406696+0000", 2024-04-22T18:39:17.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9066e-05 2024-04-22T18:39:17.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406784+0000", 2024-04-22T18:39:17.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8194000000000003e-05 2024-04-22T18:39:17.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407447+0000", 2024-04-22T18:39:17.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066295700000000004 2024-04-22T18:39:17.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408237+0000", 2024-04-22T18:39:17.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079032600000000005 2024-04-22T18:39:17.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408270+0000", 2024-04-22T18:39:17.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3157000000000001e-05 2024-04-22T18:39:17.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408295+0000", 2024-04-22T18:39:17.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4151999999999998e-05 2024-04-22T18:39:17.528 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.528 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.528 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.529 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:17.529 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:17.529 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.632638282999999, 2024-04-22T18:39:17.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30533819600000001, 2024-04-22T18:39:17.529 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.529 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.529 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.529 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.529 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.529 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 106 2024-04-22T18:39:17.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.529 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:17.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:17.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853925+0000", 2024-04-22T18:39:17.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:17.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853958+0000", 2024-04-22T18:39:17.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2536000000000002e-05 2024-04-22T18:39:17.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853964+0000", 2024-04-22T18:39:17.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6239999999999999e-06 2024-04-22T18:39:17.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853967+0000", 2024-04-22T18:39:17.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9700000000000001e-06 2024-04-22T18:39:17.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855144+0000", 2024-04-22T18:39:17.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001176774 2024-04-22T18:39:17.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855225+0000", 2024-04-22T18:39:17.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0697000000000003e-05 2024-04-22T18:39:17.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855353+0000", 2024-04-22T18:39:17.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012788 2024-04-22T18:39:17.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058003+0000", 2024-04-22T18:39:17.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202649939 2024-04-22T18:39:17.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159184+0000", 2024-04-22T18:39:17.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101181058 2024-04-22T18:39:17.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159238+0000", 2024-04-22T18:39:17.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3967e-05 2024-04-22T18:39:17.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159264+0000", 2024-04-22T18:39:17.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6237999999999998e-05 2024-04-22T18:39:17.534 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.534 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.534 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.534 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:17.534 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:17.535 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.632591433000002, 2024-04-22T18:39:17.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30544294500000002, 2024-04-22T18:39:17.535 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.535 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.535 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.535 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.535 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.535 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 107 2024-04-22T18:39:17.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.535 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:17.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853972+0000", 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853975+0000", 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0939999999999999e-06 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853978+0000", 2024-04-22T18:39:17.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:39:17.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853981+0000", 2024-04-22T18:39:17.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.858e-06 2024-04-22T18:39:17.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855831+0000", 2024-04-22T18:39:17.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018503689999999999 2024-04-22T18:39:17.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855932+0000", 2024-04-22T18:39:17.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010071 2024-04-22T18:39:17.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856029+0000", 2024-04-22T18:39:17.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6751999999999998e-05 2024-04-22T18:39:17.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058013+0000", 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20198412600000001 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159326+0000", 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101312969 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159376+0000", 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9968e-05 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159416+0000", 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9641999999999998e-05 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.632503036999999, 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30548257499999998, 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.540 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 108 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:17.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:17.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854064+0000", 2024-04-22T18:39:17.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5820000000000002e-06 2024-04-22T18:39:17.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854080+0000", 2024-04-22T18:39:17.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5379999999999998e-05 2024-04-22T18:39:17.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854084+0000", 2024-04-22T18:39:17.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.963e-06 2024-04-22T18:39:17.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856413+0000", 2024-04-22T18:39:17.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023298070000000001 2024-04-22T18:39:17.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856494+0000", 2024-04-22T18:39:17.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0863000000000002e-05 2024-04-22T18:39:17.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856624+0000", 2024-04-22T18:39:17.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129799 2024-04-22T18:39:17.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058947+0000", 2024-04-22T18:39:17.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20232328899999999 2024-04-22T18:39:17.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159468+0000", 2024-04-22T18:39:17.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100520597 2024-04-22T18:39:17.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159508+0000", 2024-04-22T18:39:17.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9830000000000003e-05 2024-04-22T18:39:17.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159544+0000", 2024-04-22T18:39:17.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5944e-05 2024-04-22T18:39:17.546 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.546 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.546 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.547 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:17.547 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:17.547 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.632458544999999, 2024-04-22T18:39:17.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30556502499999999, 2024-04-22T18:39:17.547 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.547 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.547 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.547 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.547 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.547 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:39:17.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.548 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:17.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:17.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854105+0000", 2024-04-22T18:39:17.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:17.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854175+0000", 2024-04-22T18:39:17.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0178000000000005e-05 2024-04-22T18:39:17.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854191+0000", 2024-04-22T18:39:17.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5965999999999999e-05 2024-04-22T18:39:17.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854196+0000", 2024-04-22T18:39:17.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.42e-06 2024-04-22T18:39:17.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956526+0000", 2024-04-22T18:39:17.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10233094299999999 2024-04-22T18:39:17.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956634+0000", 2024-04-22T18:39:17.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107398 2024-04-22T18:39:17.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956730+0000", 2024-04-22T18:39:17.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6217000000000004e-05 2024-04-22T18:39:17.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058955+0000", 2024-04-22T18:39:17.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102224749 2024-04-22T18:39:17.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159596+0000", 2024-04-22T18:39:17.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100641297 2024-04-22T18:39:17.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159637+0000", 2024-04-22T18:39:17.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874000000000002e-05 2024-04-22T18:39:17.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159671+0000", 2024-04-22T18:39:17.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3603999999999999e-05 2024-04-22T18:39:17.558 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.558 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.558 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.558 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:17.558 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:17.559 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.632364698000004, 2024-04-22T18:39:17.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30683048800000001, 2024-04-22T18:39:17.559 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.559 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.559 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.559 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.559 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.559 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110 2024-04-22T18:39:17.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.559 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:17.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:17.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:17.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:17.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854202+0000", 2024-04-22T18:39:17.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:39:17.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854206+0000", 2024-04-22T18:39:17.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5870000000000001e-06 2024-04-22T18:39:17.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854208+0000", 2024-04-22T18:39:17.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7199999999999998e-06 2024-04-22T18:39:17.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057285+0000", 2024-04-22T18:39:17.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20307620800000001 2024-04-22T18:39:17.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057414+0000", 2024-04-22T18:39:17.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012896899999999999 2024-04-22T18:39:17.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057588+0000", 2024-04-22T18:39:17.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000174802 2024-04-22T18:39:17.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159429+0000", 2024-04-22T18:39:17.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101840783 2024-04-22T18:39:17.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160947+0000", 2024-04-22T18:39:17.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001518152 2024-04-22T18:39:17.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160992+0000", 2024-04-22T18:39:17.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4725000000000003e-05 2024-04-22T18:39:17.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161030+0000", 2024-04-22T18:39:17.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7789e-05 2024-04-22T18:39:17.564 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.564 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.564 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.564 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:17.564 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:17.564 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.632353516999999, 2024-04-22T18:39:17.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:39:17.564 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.565 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.565 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.565 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.565 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.565 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:39:17.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.565 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:17.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:17.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:39:17.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:17.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:39:17.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:39:17.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:39:17.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:39:17.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:39:17.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:39:17.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:39:17.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:39:17.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:39:17.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:39:17.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:39:17.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:39:17.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:39:17.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:39:17.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:39:17.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:39:17.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:39:17.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:39:17.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:39:17.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:39:17.571 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.571 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.571 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.571 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:17.571 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:17.571 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.632337859000003, 2024-04-22T18:39:17.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:39:17.571 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.572 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.572 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.572 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.572 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.572 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:39:17.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.572 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:17.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:17.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:17.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:17.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:39:17.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:39:17.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:39:17.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:39:17.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:39:17.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:39:17.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:39:17.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:39:17.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:39:17.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:39:17.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:39:17.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:39:17.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:39:17.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:39:17.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.577 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.578 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:17.578 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:17.578 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.632286696999998, 2024-04-22T18:39:17.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:39:17.578 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.578 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.578 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.578 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.578 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.579 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:39:17.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.579 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:17.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:17.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:39:17.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:39:17.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:39:17.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:39:17.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:39:17.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:39:17.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:39:17.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:39:17.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:39:17.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.632271473000003, 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:39:17.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.585 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:17.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:17.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:39:17.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:17.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:39:17.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:39:17.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:39:17.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:39:17.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:39:17.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:39:17.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:39:17.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:39:17.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:39:17.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:39:17.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:39:17.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:39:17.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:39:17.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:39:17.589 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.589 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.589 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.589 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:17.590 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:17.590 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.632211896000001, 2024-04-22T18:39:17.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:39:17.590 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.590 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.590 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.590 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.590 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.590 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:39:17.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.590 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:39:17.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:39:17.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:39:17.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:39:17.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:39:17.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:39:17.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:39:17.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:39:17.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:39:17.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:39:17.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:39:17.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:39:17.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:39:17.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:39:17.595 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.595 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.595 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.595 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:17.595 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:17.595 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.632129579999997, 2024-04-22T18:39:17.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:39:17.595 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.595 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.595 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.595 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.595 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.596 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:39:17.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.596 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:17.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:17.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:39:17.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:39:17.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:39:17.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:39:17.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:39:17.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:39:17.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:39:17.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:39:17.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:39:17.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:39:17.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.601 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:17.601 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:17.601 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.632119424000003, 2024-04-22T18:39:17.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:39:17.601 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.601 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.601 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.601 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.601 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.601 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:39:17.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.601 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:17.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:17.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:39:17.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:17.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:39:17.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:39:17.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:39:17.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:39:17.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:39:17.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:39:17.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:39:17.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:39:17.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:39:17.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:39:17.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:39:17.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:39:17.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:39:17.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:39:17.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.607 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.608 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:161 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:39:17.608 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:17.608 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.031625132999999, 2024-04-22T18:39:17.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40239169000000002, 2024-04-22T18:39:17.608 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:17.608 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:17.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:17.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:17.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:17.608 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 161 2024-04-22T18:39:17.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.608 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:17.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:17.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:17.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:17.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:17.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:17.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:17.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454938+0000", 2024-04-22T18:39:17.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:17.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:17.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454982+0000", 2024-04-22T18:39:17.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3470999999999998e-05 2024-04-22T18:39:17.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:17.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454988+0000", 2024-04-22T18:39:17.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7640000000000002e-06 2024-04-22T18:39:17.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:17.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555054+0000", 2024-04-22T18:39:17.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006607300000001 2024-04-22T18:39:17.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:17.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555607+0000", 2024-04-22T18:39:17.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055319500000000003 2024-04-22T18:39:17.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:17.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555689+0000", 2024-04-22T18:39:17.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1817999999999995e-05 2024-04-22T18:39:17.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:17.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555804+0000", 2024-04-22T18:39:17.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115346 2024-04-22T18:39:17.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:17.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556554+0000", 2024-04-22T18:39:17.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000750102 2024-04-22T18:39:17.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:17.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857265+0000", 2024-04-22T18:39:17.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30071122700000003 2024-04-22T18:39:17.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:17.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857302+0000", 2024-04-22T18:39:17.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7069e-05 2024-04-22T18:39:17.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:17.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:17.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:17.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857331+0000", 2024-04-22T18:39:17.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8291999999999999e-05 2024-04-22T18:39:17.614 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.615 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.615 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.615 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:17.615 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:17.615 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:17.616 INFO:tasks.rados.rados.0.smithi089.stdout:774: got (22) Invalid argument 2024-04-22T18:39:17.616 INFO:tasks.rados.rados.0.smithi089.stdout:781: expect (ObjNum 71 snap 0 seq_num 71) 2024-04-22T18:39:17.617 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:17.617+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:17.617 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:17.688 INFO:tasks.rados.rados.0.smithi089.stdout:779: expect (ObjNum 86 snap 0 seq_num 86) 2024-04-22T18:39:17.717 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:17.718+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:17.722 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:17.722 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:17.722 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:17.722 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:17.731 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:17.818 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:17.819+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:17.835 INFO:tasks.rados.rados.0.smithi089.stdout:780: got (2) No such file or directory 2024-04-22T18:39:17.835 INFO:tasks.rados.rados.0.smithi089.stdout:780: got expected ENOENT (src dne) 2024-04-22T18:39:17.835 INFO:tasks.rados.rados.0.smithi089.stdout:776: finishing write tid 1 to smithi08937929-38 2024-04-22T18:39:17.836 INFO:tasks.rados.rados.0.smithi089.stdout:773: done (9 left) 2024-04-22T18:39:17.836 INFO:tasks.rados.rados.0.smithi089.stdout:774: done (8 left) 2024-04-22T18:39:17.836 INFO:tasks.rados.rados.0.smithi089.stdout:777: done (7 left) 2024-04-22T18:39:17.836 INFO:tasks.rados.rados.0.smithi089.stdout:778: done (6 left) 2024-04-22T18:39:17.836 INFO:tasks.rados.rados.0.smithi089.stdout:779: done (5 left) 2024-04-22T18:39:17.836 INFO:tasks.rados.rados.0.smithi089.stdout:780: done (4 left) 2024-04-22T18:39:17.836 INFO:tasks.rados.rados.0.smithi089.stdout:781: done (3 left) 2024-04-22T18:39:17.836 INFO:tasks.rados.rados.0.smithi089.stdout:782: done (2 left) 2024-04-22T18:39:17.836 INFO:tasks.rados.rados.0.smithi089.stdout:783: read oid 18 snap -1 2024-04-22T18:39:17.836 INFO:tasks.rados.rados.0.smithi089.stdout:783: expect (ObjNum 17 snap 0 seq_num 17) 2024-04-22T18:39:17.836 INFO:tasks.rados.rados.0.smithi089.stdout:784: tier_flush oid 7 2024-04-22T18:39:17.836 INFO:tasks.rados.rados.0.smithi089.stdout:784: tier_flush oid 7 snap -1 2024-04-22T18:39:17.836 INFO:tasks.rados.rados.0.smithi089.stdout:785: read oid 40 snap -1 2024-04-22T18:39:17.836 INFO:tasks.rados.rados.0.smithi089.stdout:785: expect (ObjNum 58 snap 0 seq_num 58) 2024-04-22T18:39:17.837 INFO:tasks.rados.rados.0.smithi089.stdout:786: tier_evict oid 11 2024-04-22T18:39:17.837 INFO:tasks.rados.rados.0.smithi089.stdout:786: tier_evict oid 11 snap -1 2024-04-22T18:39:17.837 INFO:tasks.rados.rados.0.smithi089.stdout:787: write (excl) oid 19 current snap is 0 2024-04-22T18:39:17.837 INFO:tasks.rados.rados.0.smithi089.stdout:787: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-04-22T18:39:17.840 INFO:tasks.rados.rados.0.smithi089.stdout:787: writing smithi08937929-19 from 463893 to 1026133 tid 1 2024-04-22T18:39:17.847 INFO:tasks.rados.rados.0.smithi089.stdout:787: writing smithi08937929-19 from 1593482 to 2251509 tid 2 2024-04-22T18:39:17.853 INFO:tasks.rados.rados.0.smithi089.stdout:787: writing smithi08937929-19 from 2685935 to 3245276 tid 3 2024-04-22T18:39:17.853 INFO:tasks.rados.rados.0.smithi089.stdout:788: write oid 30 current snap is 0 2024-04-22T18:39:17.853 INFO:tasks.rados.rados.0.smithi089.stdout:788: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2024-04-22T18:39:17.859 INFO:tasks.rados.rados.0.smithi089.stdout:788: writing smithi08937929-30 from 480700 to 1118189 tid 1 2024-04-22T18:39:17.864 INFO:tasks.rados.rados.0.smithi089.stdout:788: writing smithi08937929-30 from 1651964 to 2053649 tid 2 2024-04-22T18:39:17.872 INFO:tasks.rados.rados.0.smithi089.stdout:788: writing smithi08937929-30 from 2713358 to 3483971 tid 3 2024-04-22T18:39:17.872 INFO:tasks.rados.rados.0.smithi089.stdout:789: read oid 10 snap -1 2024-04-22T18:39:17.872 INFO:tasks.rados.rados.0.smithi089.stdout:789: expect deleted 2024-04-22T18:39:17.872 INFO:tasks.rados.rados.0.smithi089.stdout:790: read oid 23 snap -1 2024-04-22T18:39:17.872 INFO:tasks.rados.rados.0.smithi089.stdout:790: expect (ObjNum 82 snap 0 seq_num 82) 2024-04-22T18:39:17.873 INFO:tasks.rados.rados.0.smithi089.stdout:791: set_chunk oid 45 target oid 2024-04-22T18:39:17.876 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:17.876 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:17.876 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:17.876 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:17.876 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:17.885 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:17.919 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:17.920+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:18.019 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:18.020+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:18.031 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:56 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:39:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.779258515999999, 2024-04-22T18:39:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30396841899999999, 2024-04-22T18:39:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 56 2024-04-22T18:39:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:39:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:39:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:39:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251815+0000", 2024-04-22T18:39:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7800000000000001e-06 2024-04-22T18:39:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251820+0000", 2024-04-22T18:39:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5829999999999997e-06 2024-04-22T18:39:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251825+0000", 2024-04-22T18:39:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7879999999999997e-06 2024-04-22T18:39:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.252721+0000", 2024-04-22T18:39:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089552200000000005 2024-04-22T18:39:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.252794+0000", 2024-04-22T18:39:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3440000000000002e-05 2024-04-22T18:39:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555574+0000", 2024-04-22T18:39:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30278038099999999 2024-04-22T18:39:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555687+0000", 2024-04-22T18:39:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112804 2024-04-22T18:39:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555781+0000", 2024-04-22T18:39:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3624999999999999e-05 2024-04-22T18:39:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:61 4.b 4:d923883e:::smithi08937929-16:head [write 652105~445088,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.760875895000002, 2024-04-22T18:39:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.38578178600000002, 2024-04-22T18:39:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 61 2024-04-22T18:39:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270192+0000", 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270401+0000", 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020921199999999999 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270415+0000", 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4191e-05 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270422+0000", 2024-04-22T18:39:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7329999999999997e-06 2024-04-22T18:39:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270461+0000", 2024-04-22T18:39:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9530999999999998e-05 2024-04-22T18:39:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270550+0000", 2024-04-22T18:39:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8508e-05 2024-04-22T18:39:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270648+0000", 2024-04-22T18:39:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8377999999999999e-05 2024-04-22T18:39:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.271907+0000", 2024-04-22T18:39:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012589599999999999 2024-04-22T18:39:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655904+0000", 2024-04-22T18:39:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.383997115 2024-04-22T18:39:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655933+0000", 2024-04-22T18:39:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9011999999999999e-05 2024-04-22T18:39:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655977+0000", 2024-04-22T18:39:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3578000000000002e-05 2024-04-22T18:39:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:62 4.b 4:d923883e:::smithi08937929-16:head [write 1728171~452988] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:39:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.754917247000002, 2024-04-22T18:39:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.379937407, 2024-04-22T18:39:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62 2024-04-22T18:39:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:39:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:39:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276151+0000", 2024-04-22T18:39:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276370+0000", 2024-04-22T18:39:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000218951 2024-04-22T18:39:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276382+0000", 2024-04-22T18:39:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1517e-05 2024-04-22T18:39:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276389+0000", 2024-04-22T18:39:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8939999999999993e-06 2024-04-22T18:39:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276430+0000", 2024-04-22T18:39:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0958999999999999e-05 2024-04-22T18:39:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276485+0000", 2024-04-22T18:39:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4208e-05 2024-04-22T18:39:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276567+0000", 2024-04-22T18:39:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2006000000000001e-05 2024-04-22T18:39:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277685+0000", 2024-04-22T18:39:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001118237 2024-04-22T18:39:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656024+0000", 2024-04-22T18:39:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.378339588 2024-04-22T18:39:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656054+0000", 2024-04-22T18:39:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9373000000000002e-05 2024-04-22T18:39:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656091+0000", 2024-04-22T18:39:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7168000000000001e-05 2024-04-22T18:39:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:63 4.b 4:d923883e:::smithi08937929-16:head [setxattr _header (56) in=7b,truncate 2252105] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:39:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.754674991000002, 2024-04-22T18:39:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37982964200000002, 2024-04-22T18:39:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 63 2024-04-22T18:39:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:39:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:39:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276395+0000", 2024-04-22T18:39:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276400+0000", 2024-04-22T18:39:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6090000000000003e-06 2024-04-22T18:39:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276407+0000", 2024-04-22T18:39:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5850000000000002e-06 2024-04-22T18:39:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276411+0000", 2024-04-22T18:39:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2569999999999999e-06 2024-04-22T18:39:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277254+0000", 2024-04-22T18:39:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084384300000000005 2024-04-22T18:39:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277312+0000", 2024-04-22T18:39:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7395999999999997e-05 2024-04-22T18:39:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277388+0000", 2024-04-22T18:39:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6604e-05 2024-04-22T18:39:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277870+0000", 2024-04-22T18:39:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000481545 2024-04-22T18:39:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656138+0000", 2024-04-22T18:39:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37826752499999999 2024-04-22T18:39:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656177+0000", 2024-04-22T18:39:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9419999999999999e-05 2024-04-22T18:39:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656226+0000", 2024-04-22T18:39:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8562e-05 2024-04-22T18:39:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:64 4.b 4:d923883e:::smithi08937929-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.754644370000001, 2024-04-22T18:39:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37999559999999999, 2024-04-22T18:39:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 64 2024-04-22T18:39:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:18.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276429+0000", 2024-04-22T18:39:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7329999999999999e-06 2024-04-22T18:39:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276435+0000", 2024-04-22T18:39:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.02e-06 2024-04-22T18:39:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276439+0000", 2024-04-22T18:39:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7659999999999998e-06 2024-04-22T18:39:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277530+0000", 2024-04-22T18:39:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010909470000000001 2024-04-22T18:39:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277603+0000", 2024-04-22T18:39:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2638999999999994e-05 2024-04-22T18:39:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656259+0000", 2024-04-22T18:39:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37865579300000002 2024-04-22T18:39:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656330+0000", 2024-04-22T18:39:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0899e-05 2024-04-22T18:39:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656422+0000", 2024-04-22T18:39:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2397000000000003e-05 2024-04-22T18:39:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.575714294000001, 2024-04-22T18:39:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:39:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:39:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:39:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:39:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:39:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:39:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:39:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:39:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:39:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:39:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:39:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:39:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:39:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:39:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:39:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:39:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:39:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:39:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:39:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:39:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:39:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:39:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:39:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:315 4.b 4:d923883e:::smithi08937929-16:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.039251514, 2024-04-22T18:39:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40309447300000001, 2024-04-22T18:39:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 315 2024-04-22T18:39:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991817+0000", 2024-04-22T18:39:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991825+0000", 2024-04-22T18:39:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7279999999999995e-06 2024-04-22T18:39:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991834+0000", 2024-04-22T18:39:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1980000000000007e-06 2024-04-22T18:39:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991841+0000", 2024-04-22T18:39:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0740000000000004e-06 2024-04-22T18:39:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991881+0000", 2024-04-22T18:39:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-04-22T18:39:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991940+0000", 2024-04-22T18:39:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9382000000000002e-05 2024-04-22T18:39:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394250+0000", 2024-04-22T18:39:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40230947700000003 2024-04-22T18:39:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394752+0000", 2024-04-22T18:39:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050269699999999997 2024-04-22T18:39:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394865+0000", 2024-04-22T18:39:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011267700000000001 2024-04-22T18:39:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394892+0000", 2024-04-22T18:39:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7535999999999999e-05 2024-04-22T18:39:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394914+0000", 2024-04-22T18:39:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1398000000000001e-05 2024-04-22T18:39:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:331 4.6 4:6700eab5:::smithi08937929-32:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:39:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.434514378999999, 2024-04-22T18:39:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.305691029, 2024-04-22T18:39:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 331 2024-04-22T18:39:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:39:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:39:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:39:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596560+0000", 2024-04-22T18:39:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8149999999999996e-06 2024-04-22T18:39:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596568+0000", 2024-04-22T18:39:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1489999999999997e-06 2024-04-22T18:39:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596576+0000", 2024-04-22T18:39:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.191e-06 2024-04-22T18:39:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596619+0000", 2024-04-22T18:39:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3507000000000002e-05 2024-04-22T18:39:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596709+0000", 2024-04-22T18:39:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0043000000000005e-05 2024-04-22T18:39:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.801630+0000", 2024-04-22T18:39:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20492061 2024-04-22T18:39:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.802094+0000", 2024-04-22T18:39:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046431899999999998 2024-04-22T18:39:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.902165+0000", 2024-04-22T18:39:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10007037000000001 2024-04-22T18:39:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.902218+0000", 2024-04-22T18:39:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3897999999999998e-05 2024-04-22T18:39:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.902248+0000", 2024-04-22T18:39:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9023999999999999e-05 2024-04-22T18:39:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:367 4.3 4:c6fa81cf:::smithi08937929-18:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:39:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.233357397999999, 2024-04-22T18:39:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30292533399999999, 2024-04-22T18:39:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 367 2024-04-22T18:39:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:39:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:39:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:39:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797719+0000", 2024-04-22T18:39:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0669999999999997e-06 2024-04-22T18:39:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797725+0000", 2024-04-22T18:39:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3049999999999998e-06 2024-04-22T18:39:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797731+0000", 2024-04-22T18:39:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7649999999999999e-06 2024-04-22T18:39:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.897957+0000", 2024-04-22T18:39:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100226045 2024-04-22T18:39:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.898010+0000", 2024-04-22T18:39:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2679999999999997e-05 2024-04-22T18:39:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.999633+0000", 2024-04-22T18:39:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101623639 2024-04-22T18:39:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.000232+0000", 2024-04-22T18:39:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059861099999999998 2024-04-22T18:39:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100531+0000", 2024-04-22T18:39:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100299112 2024-04-22T18:39:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100599+0000", 2024-04-22T18:39:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7989000000000002e-05 2024-04-22T18:39:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100639+0000", 2024-04-22T18:39:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9919000000000002e-05 2024-04-22T18:39:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:393 4.b 4:defe92fb:::smithi08937929-44:head [assert-version v12,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.929875332000002, 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33050822299999999, 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 393 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:39:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101194+0000", 2024-04-22T18:39:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101201+0000", 2024-04-22T18:39:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1269999999999997e-06 2024-04-22T18:39:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101211+0000", 2024-04-22T18:39:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9380000000000004e-06 2024-04-22T18:39:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101220+0000", 2024-04-22T18:39:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3459999999999994e-06 2024-04-22T18:39:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101324+0000", 2024-04-22T18:39:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104366 2024-04-22T18:39:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101379+0000", 2024-04-22T18:39:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5127999999999998e-05 2024-04-22T18:39:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.330649+0000", 2024-04-22T18:39:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.22927022699999999 2024-04-22T18:39:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.331182+0000", 2024-04-22T18:39:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053250900000000002 2024-04-22T18:39:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.431635+0000", 2024-04-22T18:39:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10045290599999999 2024-04-22T18:39:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.431677+0000", 2024-04-22T18:39:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1816000000000001e-05 2024-04-22T18:39:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.431704+0000", 2024-04-22T18:39:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7132e-05 2024-04-22T18:39:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:415 4.b 4:d923883e:::smithi08937929-16:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:39:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402145+0000", 2024-04-22T18:39:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.628925834, 2024-04-22T18:39:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30209665400000002, 2024-04-22T18:39:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 415 2024-04-22T18:39:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402145+0000", 2024-04-22T18:39:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402145+0000", 2024-04-22T18:39:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402144+0000", 2024-04-22T18:39:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402149+0000", 2024-04-22T18:39:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0760000000000002e-06 2024-04-22T18:39:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402159+0000", 2024-04-22T18:39:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7499999999999998e-06 2024-04-22T18:39:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402165+0000", 2024-04-22T18:39:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0700000000000003e-06 2024-04-22T18:39:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402195+0000", 2024-04-22T18:39:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0485e-05 2024-04-22T18:39:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402293+0000", 2024-04-22T18:39:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7153999999999998e-05 2024-04-22T18:39:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503807+0000", 2024-04-22T18:39:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101514666 2024-04-22T18:39:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604684+0000", 2024-04-22T18:39:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100876493 2024-04-22T18:39:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.704174+0000", 2024-04-22T18:39:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099490331000000001 2024-04-22T18:39:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.704200+0000", 2024-04-22T18:39:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5902e-05 2024-04-22T18:39:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.704242+0000", 2024-04-22T18:39:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1686e-05 2024-04-22T18:39:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:416 4.3 4:c0d4d677:::smithi08937929-17:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:39:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402174+0000", 2024-04-22T18:39:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.628896825999998, 2024-04-22T18:39:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30212472000000001, 2024-04-22T18:39:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 416 2024-04-22T18:39:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402174+0000", 2024-04-22T18:39:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402174+0000", 2024-04-22T18:39:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402173+0000", 2024-04-22T18:39:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402178+0000", 2024-04-22T18:39:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4259999999999996e-06 2024-04-22T18:39:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402184+0000", 2024-04-22T18:39:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.066e-06 2024-04-22T18:39:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402189+0000", 2024-04-22T18:39:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:39:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402426+0000", 2024-04-22T18:39:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023742300000000001 2024-04-22T18:39:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402489+0000", 2024-04-22T18:39:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2891999999999993e-05 2024-04-22T18:39:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.504423+0000", 2024-04-22T18:39:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101933444 2024-04-22T18:39:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.704136+0000", 2024-04-22T18:39:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.199712895 2024-04-22T18:39:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.704250+0000", 2024-04-22T18:39:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114645 2024-04-22T18:39:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.704273+0000", 2024-04-22T18:39:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2736999999999999e-05 2024-04-22T18:39:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.704299+0000", 2024-04-22T18:39:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5833000000000002e-05 2024-04-22T18:39:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:713 4.3 4:ca46fde9:::smithi08937929-14:head [assert-version v3,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.627086+0000", 2024-04-22T18:39:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 15.403984743000001, 2024-04-22T18:39:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30289147900000002, 2024-04-22T18:39:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 713 2024-04-22T18:39:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627086+0000", 2024-04-22T18:39:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627086+0000", 2024-04-22T18:39:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627085+0000", 2024-04-22T18:39:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627098+0000", 2024-04-22T18:39:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2869e-05 2024-04-22T18:39:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627111+0000", 2024-04-22T18:39:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2978e-05 2024-04-22T18:39:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627117+0000", 2024-04-22T18:39:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7209999999999997e-06 2024-04-22T18:39:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627152+0000", 2024-04-22T18:39:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5052000000000003e-05 2024-04-22T18:39:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627204+0000", 2024-04-22T18:39:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1214000000000001e-05 2024-04-22T18:39:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.929177+0000", 2024-04-22T18:39:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30197355399999998 2024-04-22T18:39:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.929785+0000", 2024-04-22T18:39:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060751100000000003 2024-04-22T18:39:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.929922+0000", 2024-04-22T18:39:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013767599999999999 2024-04-22T18:39:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.929952+0000", 2024-04-22T18:39:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9708000000000001e-05 2024-04-22T18:39:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.929978+0000", 2024-04-22T18:39:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5497e-05 2024-04-22T18:39:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:767 4.6 4:6700eab5:::smithi08937929-32:head [read 0~2521017,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.147606+0000", 2024-04-22T18:39:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.883464980999999, 2024-04-22T18:39:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30286525600000003, 2024-04-22T18:39:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 767 2024-04-22T18:39:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.147606+0000", 2024-04-22T18:39:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.147606+0000", 2024-04-22T18:39:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.147605+0000", 2024-04-22T18:39:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.147609+0000", 2024-04-22T18:39:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6770000000000001e-06 2024-04-22T18:39:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.147615+0000", 2024-04-22T18:39:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2539999999999998e-06 2024-04-22T18:39:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.147620+0000", 2024-04-22T18:39:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3549999999999998e-06 2024-04-22T18:39:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.449277+0000", 2024-04-22T18:39:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30165729699999999 2024-04-22T18:39:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.449384+0000", 2024-04-22T18:39:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010679900000000001 2024-04-22T18:39:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.450471+0000", 2024-04-22T18:39:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001087441 2024-04-22T18:39:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:773 4.b 4:defe92fb:::smithi08937929-44:head [write 1826593~546335] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148009+0000", 2024-04-22T18:39:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.883061757, 2024-04-22T18:39:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30494080699999998, 2024-04-22T18:39:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 773 2024-04-22T18:39:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148009+0000", 2024-04-22T18:39:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148009+0000", 2024-04-22T18:39:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148008+0000", 2024-04-22T18:39:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148307+0000", 2024-04-22T18:39:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029880299999999999 2024-04-22T18:39:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148317+0000", 2024-04-22T18:39:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5999999999999996e-06 2024-04-22T18:39:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148323+0000", 2024-04-22T18:39:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9279999999999998e-06 2024-04-22T18:39:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.249871+0000", 2024-04-22T18:39:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10154796200000001 2024-04-22T18:39:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.249960+0000", 2024-04-22T18:39:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9926000000000006e-05 2024-04-22T18:39:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.250093+0000", 2024-04-22T18:39:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013287100000000001 2024-04-22T18:39:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256145+0000", 2024-04-22T18:39:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0060519830000000004 2024-04-22T18:39:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.452777+0000", 2024-04-22T18:39:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.196631417 2024-04-22T18:39:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.452909+0000", 2024-04-22T18:39:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000132488 2024-04-22T18:39:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.452950+0000", 2024-04-22T18:39:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0686999999999999e-05 2024-04-22T18:39:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:774 4.b 4:defe92fb:::smithi08937929-44:head [write 2778819~95145] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:39:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.882741837999999, 2024-04-22T18:39:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40475725000000001, 2024-04-22T18:39:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 774 2024-04-22T18:39:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:39:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:39:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148328+0000", 2024-04-22T18:39:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148392+0000", 2024-04-22T18:39:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3323000000000003e-05 2024-04-22T18:39:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148398+0000", 2024-04-22T18:39:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4420000000000003e-06 2024-04-22T18:39:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148409+0000", 2024-04-22T18:39:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509999999999999e-05 2024-04-22T18:39:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251407+0000", 2024-04-22T18:39:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102998305 2024-04-22T18:39:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251500+0000", 2024-04-22T18:39:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3607e-05 2024-04-22T18:39:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251648+0000", 2024-04-22T18:39:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014788099999999999 2024-04-22T18:39:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256149+0000", 2024-04-22T18:39:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045010980000000003 2024-04-22T18:39:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553016+0000", 2024-04-22T18:39:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29686620800000002 2024-04-22T18:39:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553054+0000", 2024-04-22T18:39:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8414999999999997e-05 2024-04-22T18:39:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553086+0000", 2024-04-22T18:39:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2236000000000001e-05 2024-04-22T18:39:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:39:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.882657046, 2024-04-22T18:39:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:39:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:39:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:39:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:39:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:39:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:39:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:39:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:39:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:39:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:39:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:39:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:39:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:39:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:39:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:39:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:39:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:39:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:39:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:39:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:39:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:39:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:39:18.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:39:18.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:39:18.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:39:18.141 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.141 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.141 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.142 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:18.142 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:18.142 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.882638569999999, 2024-04-22T18:39:18.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:39:18.142 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.142 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:18.142 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.142 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.142 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.142 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:39:18.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:18.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:39:18.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:39:18.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:39:18.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:39:18.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:39:18.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:39:18.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:39:18.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:39:18.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:18.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:39:18.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:39:18.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:39:18.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:39:18.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:39:18.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:39:18.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:39:18.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:39:18.147 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.147 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.147 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.147 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1077 4.6 4:6700eab5:::smithi08937929-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:18.147 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:12.231515+0000", 2024-04-22T18:39:18.147 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.7995561090000001, 2024-04-22T18:39:18.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.303721886, 2024-04-22T18:39:18.147 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.147 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:18.147 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.148 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.148 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.148 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1077 2024-04-22T18:39:18.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.148 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.231515+0000", 2024-04-22T18:39:18.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.231515+0000", 2024-04-22T18:39:18.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.231514+0000", 2024-04-22T18:39:18.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:18.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.231517+0000", 2024-04-22T18:39:18.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1650000000000002e-06 2024-04-22T18:39:18.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.231524+0000", 2024-04-22T18:39:18.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3929999999999996e-06 2024-04-22T18:39:18.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.331593+0000", 2024-04-22T18:39:18.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100069617 2024-04-22T18:39:18.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.331631+0000", 2024-04-22T18:39:18.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7697000000000002e-05 2024-04-22T18:39:18.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:18.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.331693+0000", 2024-04-22T18:39:18.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2038999999999993e-05 2024-04-22T18:39:18.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.535052+0000", 2024-04-22T18:39:18.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20335874200000001 2024-04-22T18:39:18.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.535138+0000", 2024-04-22T18:39:18.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6026999999999995e-05 2024-04-22T18:39:18.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.535237+0000", 2024-04-22T18:39:18.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8875000000000005e-05 2024-04-22T18:39:18.152 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.152 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.152 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.152 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1242 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-22T18:39:18.152 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:39:18.152 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.289393021, 2024-04-22T18:39:18.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40662891899999998, 2024-04-22T18:39:18.152 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.152 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:18.153 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.153 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.153 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.153 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1242 2024-04-22T18:39:18.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.153 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:39:18.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:39:18.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741677+0000", 2024-04-22T18:39:18.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:18.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741680+0000", 2024-04-22T18:39:18.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8059999999999999e-06 2024-04-22T18:39:18.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741685+0000", 2024-04-22T18:39:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0780000000000003e-06 2024-04-22T18:39:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741690+0000", 2024-04-22T18:39:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4020000000000004e-06 2024-04-22T18:39:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741732+0000", 2024-04-22T18:39:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2110000000000002e-05 2024-04-22T18:39:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741798+0000", 2024-04-22T18:39:18.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6432000000000004e-05 2024-04-22T18:39:18.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.148307+0000", 2024-04-22T18:39:18.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40650862100000001 2024-04-22T18:39:18.156 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.156 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.156 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.156 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.156 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.157 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:18.158 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:18.158 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:18.159+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:18.259 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:18.260+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:18.300 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:18.300 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:18.300 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:18.300 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:18.309 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:18.355 INFO:tasks.rados.rados.0.smithi089.stdout:791: set_chunk oid 45 offset: 313344 length: 1283072 target oid 4feb37c20f4a610133b3ed196675ffccd015386c8c0b5d5bbba6fcfea1176e24 offset: 0 snap -1 2024-04-22T18:39:18.355 INFO:tasks.rados.rados.0.smithi089.stdout:792: read oid 48 snap -1 2024-04-22T18:39:18.356 INFO:tasks.rados.rados.0.smithi089.stdout:792: expect deleted 2024-04-22T18:39:18.356 INFO:tasks.rados.rados.0.smithi089.stdout:793: read oid 46 snap -1 2024-04-22T18:39:18.356 INFO:tasks.rados.rados.0.smithi089.stdout:793: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-22T18:39:18.356 INFO:tasks.rados.rados.0.smithi089.stdout:794: write (excl) oid 17 current snap is 0 2024-04-22T18:39:18.356 INFO:tasks.rados.rados.0.smithi089.stdout:775: finishing set_chunk to oid 33 2024-04-22T18:39:18.356 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 19 (ObjNum 65 snap 0 seq_num 65) dirty exists 2024-04-22T18:39:18.356 INFO:tasks.rados.rados.0.smithi089.stdout:786: got (22) Invalid argument 2024-04-22T18:39:18.356 INFO:tasks.rados.rados.0.smithi089.stdout:783: expect (ObjNum 17 snap 0 seq_num 17) 2024-04-22T18:39:18.359 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:18.360+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:18.430 INFO:tasks.rados.rados.0.smithi089.stdout:776: finishing write tid 2 to smithi08937929-38 2024-04-22T18:39:18.431 INFO:tasks.rados.rados.0.smithi089.stdout:776: finishing write tid 3 to smithi08937929-38 2024-04-22T18:39:18.431 INFO:tasks.rados.rados.0.smithi089.stdout:776: finishing write tid 4 to smithi08937929-38 2024-04-22T18:39:18.431 INFO:tasks.rados.rados.0.smithi089.stdout:776: finishing write tid 5 to smithi08937929-38 2024-04-22T18:39:18.431 INFO:tasks.rados.rados.0.smithi089.stdout:776: oid 38 updating version 0 to 38 2024-04-22T18:39:18.431 INFO:tasks.rados.rados.0.smithi089.stdout:776: oid 38 version 38 is already newer than 36 2024-04-22T18:39:18.431 INFO:tasks.rados.rados.0.smithi089.stdout:776: oid 38 updating version 38 to 39 2024-04-22T18:39:18.431 INFO:tasks.rados.rados.0.smithi089.stdout:776: oid 38 version 39 is already newer than 37 2024-04-22T18:39:18.431 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 39 (ObjNum 97 snap 0 seq_num 97) dirty exists 2024-04-22T18:39:18.431 INFO:tasks.rados.rados.0.smithi089.stdout:776: left oid 38 (ObjNum 97 snap 0 seq_num 97) 2024-04-22T18:39:18.431 INFO:tasks.rados.rados.0.smithi089.stdout:787: finishing write tid 1 to smithi08937929-19 2024-04-22T18:39:18.431 INFO:tasks.rados.rados.0.smithi089.stdout:787: finishing write tid 2 to smithi08937929-19 2024-04-22T18:39:18.431 INFO:tasks.rados.rados.0.smithi089.stdout:787: finishing write tid 3 to smithi08937929-19 2024-04-22T18:39:18.431 INFO:tasks.rados.rados.0.smithi089.stdout:787: finishing write tid 4 to smithi08937929-19 2024-04-22T18:39:18.432 INFO:tasks.rados.rados.0.smithi089.stdout:794: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-04-22T18:39:18.437 INFO:tasks.rados.rados.0.smithi089.stdout:794: writing smithi08937929-17 from 497507 to 1210245 tid 1 2024-04-22T18:39:18.443 INFO:tasks.rados.rados.0.smithi089.stdout:794: writing smithi08937929-17 from 1710446 to 2372142 tid 2 2024-04-22T18:39:18.450 INFO:tasks.rados.rados.0.smithi089.stdout:794: writing smithi08937929-17 from 3140781 to 3697507 tid 3 2024-04-22T18:39:18.450 INFO:tasks.rados.rados.0.smithi089.stdout:775: done (13 left) 2024-04-22T18:39:18.450 INFO:tasks.rados.rados.0.smithi089.stdout:776: done (12 left) 2024-04-22T18:39:18.450 INFO:tasks.rados.rados.0.smithi089.stdout:783: done (11 left) 2024-04-22T18:39:18.451 INFO:tasks.rados.rados.0.smithi089.stdout:786: done (10 left) 2024-04-22T18:39:18.451 INFO:tasks.rados.rados.0.smithi089.stdout:789: done (9 left) 2024-04-22T18:39:18.451 INFO:tasks.rados.rados.0.smithi089.stdout:795: write oid 25 current snap is 0 2024-04-22T18:39:18.451 INFO:tasks.rados.rados.0.smithi089.stdout:795: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2024-04-22T18:39:18.452 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:18.452 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:18.453 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:18.453 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:18.453 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:18.457 INFO:tasks.rados.rados.0.smithi089.stdout:795: writing smithi08937929-25 from 514314 to 1302301 tid 1 2024-04-22T18:39:18.461 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:18.462+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:18.462 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:18.463 INFO:tasks.rados.rados.0.smithi089.stdout:795: writing smithi08937929-25 from 1768928 to 2174282 tid 2 2024-04-22T18:39:18.469 INFO:tasks.rados.rados.0.smithi089.stdout:795: writing smithi08937929-25 from 2768204 to 3277714 tid 3 2024-04-22T18:39:18.469 INFO:tasks.rados.rados.0.smithi089.stdout:796: set_chunk oid 5 target oid 2024-04-22T18:39:18.508 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:39:18.561 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:18.562+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:18.575 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:18.575 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:18.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:18.575 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:18.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.575 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:7 4.f 4:f83a159b:::smithi08937929-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:18.575 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.111347+0000", 2024-04-22T18:39:18.575 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.463092422999999, 2024-04-22T18:39:18.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30271893700000002, 2024-04-22T18:39:18.575 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.576 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.576 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.576 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.576 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.576 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 7 2024-04-22T18:39:18.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.576 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111347+0000", 2024-04-22T18:39:18.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111347+0000", 2024-04-22T18:39:18.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111346+0000", 2024-04-22T18:39:18.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:18.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111350+0000", 2024-04-22T18:39:18.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.473e-06 2024-04-22T18:39:18.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111353+0000", 2024-04-22T18:39:18.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9100000000000001e-06 2024-04-22T18:39:18.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111356+0000", 2024-04-22T18:39:18.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2770000000000001e-06 2024-04-22T18:39:18.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.212813+0000", 2024-04-22T18:39:18.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10145739099999999 2024-04-22T18:39:18.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.212869+0000", 2024-04-22T18:39:18.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5630000000000001e-05 2024-04-22T18:39:18.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:18.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.212949+0000", 2024-04-22T18:39:18.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9679999999999996e-05 2024-04-22T18:39:18.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.313864+0000", 2024-04-22T18:39:18.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100915174 2024-04-22T18:39:18.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.413991+0000", 2024-04-22T18:39:18.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100126703 2024-04-22T18:39:18.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414021+0000", 2024-04-22T18:39:18.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0698999999999999e-05 2024-04-22T18:39:18.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414066+0000", 2024-04-22T18:39:18.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4718000000000001e-05 2024-04-22T18:39:18.581 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.581 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.581 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.581 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:8 4.f 4:f83a159b:::smithi08937929-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:18.581 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:18.581 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.463061224, 2024-04-22T18:39:18.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30292182499999998, 2024-04-22T18:39:18.581 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.581 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:18.582 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.582 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.582 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.582 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 8 2024-04-22T18:39:18.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.582 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:18.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111380+0000", 2024-04-22T18:39:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2419999999999999e-06 2024-04-22T18:39:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111384+0000", 2024-04-22T18:39:18.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.715e-06 2024-04-22T18:39:18.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111387+0000", 2024-04-22T18:39:18.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.982e-06 2024-04-22T18:39:18.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414113+0000", 2024-04-22T18:39:18.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30272622700000001 2024-04-22T18:39:18.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414203+0000", 2024-04-22T18:39:18.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0316999999999996e-05 2024-04-22T18:39:18.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414300+0000", 2024-04-22T18:39:18.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6760000000000002e-05 2024-04-22T18:39:18.585 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.585 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.585 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.585 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:9 4.f 4:f2c48a0e:::smithi08937929-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:18.585 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:18.586 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.457665639999998, 2024-04-22T18:39:18.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49290714600000002, 2024-04-22T18:39:18.586 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.586 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.586 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.586 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.586 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.586 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 9 2024-04-22T18:39:18.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.586 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:18.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:18.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116773+0000", 2024-04-22T18:39:18.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:18.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117024+0000", 2024-04-22T18:39:18.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025191299999999999 2024-04-22T18:39:18.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117036+0000", 2024-04-22T18:39:18.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1595e-05 2024-04-22T18:39:18.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117046+0000", 2024-04-22T18:39:18.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0456e-05 2024-04-22T18:39:18.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414324+0000", 2024-04-22T18:39:18.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297277603 2024-04-22T18:39:18.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414404+0000", 2024-04-22T18:39:18.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9738000000000001e-05 2024-04-22T18:39:18.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:18.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414494+0000", 2024-04-22T18:39:18.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9733000000000003e-05 2024-04-22T18:39:18.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518194+0000", 2024-04-22T18:39:18.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10370061 2024-04-22T18:39:18.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609618+0000", 2024-04-22T18:39:18.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091423360999999995 2024-04-22T18:39:18.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609655+0000", 2024-04-22T18:39:18.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7580000000000003e-05 2024-04-22T18:39:18.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609681+0000", 2024-04-22T18:39:18.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5908e-05 2024-04-22T18:39:18.592 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.592 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.592 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.592 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:10 4.f 4:f2c48a0e:::smithi08937929-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:18.592 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:18.592 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.451813380000001, 2024-04-22T18:39:18.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48712681400000002, 2024-04-22T18:39:18.593 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.593 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.593 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.593 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.593 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.593 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 10 2024-04-22T18:39:18.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.593 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:18.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:18.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122625+0000", 2024-04-22T18:39:18.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:18.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122828+0000", 2024-04-22T18:39:18.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020359499999999999 2024-04-22T18:39:18.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122840+0000", 2024-04-22T18:39:18.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1239e-05 2024-04-22T18:39:18.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122845+0000", 2024-04-22T18:39:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2719999999999997e-06 2024-04-22T18:39:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415105+0000", 2024-04-22T18:39:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29225972099999997 2024-04-22T18:39:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415169+0000", 2024-04-22T18:39:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4170000000000004e-05 2024-04-22T18:39:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:18.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415257+0000", 2024-04-22T18:39:18.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8331999999999994e-05 2024-04-22T18:39:18.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518199+0000", 2024-04-22T18:39:18.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10294174 2024-04-22T18:39:18.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609707+0000", 2024-04-22T18:39:18.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091508408999999999 2024-04-22T18:39:18.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609729+0000", 2024-04-22T18:39:18.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1917e-05 2024-04-22T18:39:18.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609753+0000", 2024-04-22T18:39:18.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3776000000000001e-05 2024-04-22T18:39:18.599 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.599 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.599 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.599 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:11 4.f 4:f2c48a0e:::smithi08937929-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:18.599 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:18.599 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.451590007, 2024-04-22T18:39:18.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49452214500000002, 2024-04-22T18:39:18.600 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.600 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.600 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.600 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.600 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.600 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 11 2024-04-22T18:39:18.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.600 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:18.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:18.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122849+0000", 2024-04-22T18:39:18.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:18.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122853+0000", 2024-04-22T18:39:18.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9029999999999997e-06 2024-04-22T18:39:18.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122856+0000", 2024-04-22T18:39:18.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.01e-06 2024-04-22T18:39:18.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122859+0000", 2024-04-22T18:39:18.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:18.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415755+0000", 2024-04-22T18:39:18.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29289606800000001 2024-04-22T18:39:18.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415808+0000", 2024-04-22T18:39:18.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3415999999999998e-05 2024-04-22T18:39:18.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:18.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415888+0000", 2024-04-22T18:39:18.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9061999999999996e-05 2024-04-22T18:39:18.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518203+0000", 2024-04-22T18:39:18.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102315533 2024-04-22T18:39:18.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617164+0000", 2024-04-22T18:39:18.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098960548999999995 2024-04-22T18:39:18.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617241+0000", 2024-04-22T18:39:18.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7292000000000005e-05 2024-04-22T18:39:18.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617372+0000", 2024-04-22T18:39:18.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013075399999999999 2024-04-22T18:39:18.606 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.606 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.606 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.606 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:12 4.f 4:f2c48a0e:::smithi08937929-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:18.607 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:18.607 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.451578142999999, 2024-04-22T18:39:18.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49473910999999998, 2024-04-22T18:39:18.607 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.607 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:18.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.608 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 12 2024-04-22T18:39:18.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.608 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:18.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:18.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:18.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:18.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122863+0000", 2024-04-22T18:39:18.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.863e-06 2024-04-22T18:39:18.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122866+0000", 2024-04-22T18:39:18.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:39:18.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122869+0000", 2024-04-22T18:39:18.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5950000000000001e-06 2024-04-22T18:39:18.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416008+0000", 2024-04-22T18:39:18.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29313975199999998 2024-04-22T18:39:18.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:18.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416048+0000", 2024-04-22T18:39:18.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9888000000000002e-05 2024-04-22T18:39:18.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617447+0000", 2024-04-22T18:39:18.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20139881400000001 2024-04-22T18:39:18.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617537+0000", 2024-04-22T18:39:18.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9606999999999998e-05 2024-04-22T18:39:18.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617600+0000", 2024-04-22T18:39:18.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3884e-05 2024-04-22T18:39:18.613 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.614 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.614 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.614 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:33 4.f 4:f0ed3d49:::smithi08937929-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.614 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:18.614 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.387637942000001, 2024-04-22T18:39:18.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62359692099999997, 2024-04-22T18:39:18.614 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.614 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.614 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.614 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.614 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.615 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 33 2024-04-22T18:39:18.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.615 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:18.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:18.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186799+0000", 2024-04-22T18:39:18.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:18.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187103+0000", 2024-04-22T18:39:18.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030354900000000002 2024-04-22T18:39:18.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187117+0000", 2024-04-22T18:39:18.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4578e-05 2024-04-22T18:39:18.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187129+0000", 2024-04-22T18:39:18.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1421e-05 2024-04-22T18:39:18.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516134+0000", 2024-04-22T18:39:18.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32900553900000001 2024-04-22T18:39:18.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516215+0000", 2024-04-22T18:39:18.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0537999999999994e-05 2024-04-22T18:39:18.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:18.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516302+0000", 2024-04-22T18:39:18.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6601000000000007e-05 2024-04-22T18:39:18.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518207+0000", 2024-04-22T18:39:18.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019058370000000001 2024-04-22T18:39:18.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810302+0000", 2024-04-22T18:39:18.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29209509700000003 2024-04-22T18:39:18.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810348+0000", 2024-04-22T18:39:18.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5992999999999998e-05 2024-04-22T18:39:18.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810398+0000", 2024-04-22T18:39:18.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0046000000000002e-05 2024-04-22T18:39:18.620 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.620 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.620 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.620 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:34 4.f 4:f0ed3d49:::smithi08937929-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.620 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:18.620 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.383035929999998, 2024-04-22T18:39:18.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.61954655800000002, 2024-04-22T18:39:18.621 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.621 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.621 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.621 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.621 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.621 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34 2024-04-22T18:39:18.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.621 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:18.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:18.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191401+0000", 2024-04-22T18:39:18.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:18.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191503+0000", 2024-04-22T18:39:18.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010190400000000001 2024-04-22T18:39:18.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191517+0000", 2024-04-22T18:39:18.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3267e-05 2024-04-22T18:39:18.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191523+0000", 2024-04-22T18:39:18.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-04-22T18:39:18.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517062+0000", 2024-04-22T18:39:18.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32553894300000003 2024-04-22T18:39:18.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517141+0000", 2024-04-22T18:39:18.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9667999999999997e-05 2024-04-22T18:39:18.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:18.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517267+0000", 2024-04-22T18:39:18.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000125494 2024-04-22T18:39:18.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518213+0000", 2024-04-22T18:39:18.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000946275 2024-04-22T18:39:18.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810901+0000", 2024-04-22T18:39:18.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29268821699999997 2024-04-22T18:39:18.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810929+0000", 2024-04-22T18:39:18.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7645999999999999e-05 2024-04-22T18:39:18.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810950+0000", 2024-04-22T18:39:18.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1333999999999999e-05 2024-04-22T18:39:18.626 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.627 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.627 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.627 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.627 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:18.627 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.382912961999999, 2024-04-22T18:39:18.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:39:18.627 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.627 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.627 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.627 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.627 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.627 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:39:18.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.628 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:18.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:18.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:39:18.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:18.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:39:18.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:39:18.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:39:18.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:39:18.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:39:18.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:18.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:39:18.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:39:18.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:39:18.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:39:18.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:18.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:39:18.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:39:18.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:39:18.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:39:18.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:39:18.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:39:18.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:39:18.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:39:18.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:39:18.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:39:18.632 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.632 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.632 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.632 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.632 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:18.633 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.382901886999999, 2024-04-22T18:39:18.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:39:18.633 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.633 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:18.633 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.633 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.633 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.633 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:39:18.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.633 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:18.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:18.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:39:18.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:18.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:39:18.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:39:18.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:39:18.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:39:18.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:39:18.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:39:18.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:39:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:39:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:39:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:39:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:39:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:39:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:39:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:39:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:39:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:39:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:38 4.2 4:4fb26dd8:::smithi08937929-10:head [write 1893632~257631] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.371179283, 2024-04-22T18:39:18.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33863800199999999, 2024-04-22T18:39:18.638 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.638 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.638 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.638 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.638 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.638 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 38 2024-04-22T18:39:18.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.638 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:18.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:18.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203258+0000", 2024-04-22T18:39:18.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:18.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203396+0000", 2024-04-22T18:39:18.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013710799999999999 2024-04-22T18:39:18.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203406+0000", 2024-04-22T18:39:18.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0295000000000001e-05 2024-04-22T18:39:18.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203411+0000", 2024-04-22T18:39:18.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5079999999999996e-06 2024-04-22T18:39:18.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203450+0000", 2024-04-22T18:39:18.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8420999999999997e-05 2024-04-22T18:39:18.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:18.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203455+0000", 2024-04-22T18:39:18.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:39:18.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237071+0000", 2024-04-22T18:39:18.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.033615891000000002 2024-04-22T18:39:18.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237129+0000", 2024-04-22T18:39:18.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8134000000000003e-05 2024-04-22T18:39:18.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:18.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237221+0000", 2024-04-22T18:39:18.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2053999999999997e-05 2024-04-22T18:39:18.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.341419+0000", 2024-04-22T18:39:18.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104197568 2024-04-22T18:39:18.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541792+0000", 2024-04-22T18:39:18.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.200373312 2024-04-22T18:39:18.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541838+0000", 2024-04-22T18:39:18.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6004000000000002e-05 2024-04-22T18:39:18.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541898+0000", 2024-04-22T18:39:18.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0297000000000002e-05 2024-04-22T18:39:18.643 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.643 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.643 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.644 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:39 4.2 4:4fb26dd8:::smithi08937929-10:head [setxattr _header (56) in=7b,truncate 2151263] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.644 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:18.644 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.371022288999999, 2024-04-22T18:39:18.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33862203000000002, 2024-04-22T18:39:18.644 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.644 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.644 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.644 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.644 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.644 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 39 2024-04-22T18:39:18.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.644 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:18.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:18.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:18.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:18.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203423+0000", 2024-04-22T18:39:18.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0589999999999999e-06 2024-04-22T18:39:18.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203429+0000", 2024-04-22T18:39:18.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2559999999999999e-06 2024-04-22T18:39:18.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203434+0000", 2024-04-22T18:39:18.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8049999999999997e-06 2024-04-22T18:39:18.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203475+0000", 2024-04-22T18:39:18.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1205000000000003e-05 2024-04-22T18:39:18.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:18.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203478+0000", 2024-04-22T18:39:18.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7140000000000002e-06 2024-04-22T18:39:18.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239180+0000", 2024-04-22T18:39:18.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035702195999999999 2024-04-22T18:39:18.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239225+0000", 2024-04-22T18:39:18.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5722e-05 2024-04-22T18:39:18.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:18.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239291+0000", 2024-04-22T18:39:18.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5870000000000005e-05 2024-04-22T18:39:18.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342114+0000", 2024-04-22T18:39:18.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102822301 2024-04-22T18:39:18.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541946+0000", 2024-04-22T18:39:18.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.199832544 2024-04-22T18:39:18.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541988+0000", 2024-04-22T18:39:18.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1499999999999999e-05 2024-04-22T18:39:18.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542039+0000", 2024-04-22T18:39:18.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1576000000000002e-05 2024-04-22T18:39:18.649 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.650 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.650 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.650 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:40 4.2 4:4fb26dd8:::smithi08937929-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.650 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:18.650 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.370985029, 2024-04-22T18:39:18.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33924657200000002, 2024-04-22T18:39:18.650 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.650 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:18.650 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.650 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.650 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.650 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 40 2024-04-22T18:39:18.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.651 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:18.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:18.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:18.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:18.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203457+0000", 2024-04-22T18:39:18.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.129e-06 2024-04-22T18:39:18.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203463+0000", 2024-04-22T18:39:18.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7440000000000004e-06 2024-04-22T18:39:18.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203468+0000", 2024-04-22T18:39:18.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2619999999999998e-06 2024-04-22T18:39:18.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203491+0000", 2024-04-22T18:39:18.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3260000000000001e-05 2024-04-22T18:39:18.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:18.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203493+0000", 2024-04-22T18:39:18.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9599999999999999e-06 2024-04-22T18:39:18.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239420+0000", 2024-04-22T18:39:18.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035927212 2024-04-22T18:39:18.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:18.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239462+0000", 2024-04-22T18:39:18.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.159e-05 2024-04-22T18:39:18.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542547+0000", 2024-04-22T18:39:18.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30308492599999998 2024-04-22T18:39:18.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542611+0000", 2024-04-22T18:39:18.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4622000000000006e-05 2024-04-22T18:39:18.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542701+0000", 2024-04-22T18:39:18.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9579000000000004e-05 2024-04-22T18:39:18.656 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.656 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.657 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.657 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:41 4.2 4:409e6456:::smithi08937929-11:head [write 568070~468843,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.657 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:18.657 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.365268757999999, 2024-04-22T18:39:18.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33297193600000002, 2024-04-22T18:39:18.657 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.657 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.657 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.657 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.657 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.657 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 41 2024-04-22T18:39:18.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.657 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:18.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:18.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209168+0000", 2024-04-22T18:39:18.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:18.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209393+0000", 2024-04-22T18:39:18.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022485199999999999 2024-04-22T18:39:18.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209404+0000", 2024-04-22T18:39:18.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1209999999999999e-05 2024-04-22T18:39:18.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209411+0000", 2024-04-22T18:39:18.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8580000000000002e-06 2024-04-22T18:39:18.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209457+0000", 2024-04-22T18:39:18.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6351999999999999e-05 2024-04-22T18:39:18.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:18.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209463+0000", 2024-04-22T18:39:18.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.322e-06 2024-04-22T18:39:18.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339572+0000", 2024-04-22T18:39:18.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13010901999999999 2024-04-22T18:39:18.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339673+0000", 2024-04-22T18:39:18.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101332 2024-04-22T18:39:18.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:18.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339765+0000", 2024-04-22T18:39:18.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1780999999999994e-05 2024-04-22T18:39:18.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342117+0000", 2024-04-22T18:39:18.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002351709 2024-04-22T18:39:18.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542074+0000", 2024-04-22T18:39:18.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995749800000001 2024-04-22T18:39:18.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542112+0000", 2024-04-22T18:39:18.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7891e-05 2024-04-22T18:39:18.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542143+0000", 2024-04-22T18:39:18.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0756000000000002e-05 2024-04-22T18:39:18.663 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.663 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.663 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.663 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:42 4.2 4:409e6456:::smithi08937929-11:head [write 1552114~615956] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.664 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:18.664 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.358930448999999, 2024-04-22T18:39:18.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326735257, 2024-04-22T18:39:18.664 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.664 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.664 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.664 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.664 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.664 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 42 2024-04-22T18:39:18.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.664 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:18.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:18.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215508+0000", 2024-04-22T18:39:18.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:18.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215766+0000", 2024-04-22T18:39:18.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025766999999999999 2024-04-22T18:39:18.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215776+0000", 2024-04-22T18:39:18.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0772e-05 2024-04-22T18:39:18.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215785+0000", 2024-04-22T18:39:18.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6880000000000005e-06 2024-04-22T18:39:18.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215818+0000", 2024-04-22T18:39:18.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2960999999999998e-05 2024-04-22T18:39:18.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:18.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215821+0000", 2024-04-22T18:39:18.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8569999999999999e-06 2024-04-22T18:39:18.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340332+0000", 2024-04-22T18:39:18.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.124511409 2024-04-22T18:39:18.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340416+0000", 2024-04-22T18:39:18.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3793000000000002e-05 2024-04-22T18:39:18.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:18.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340543+0000", 2024-04-22T18:39:18.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012720299999999999 2024-04-22T18:39:18.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342361+0000", 2024-04-22T18:39:18.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018174370000000001 2024-04-22T18:39:18.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542176+0000", 2024-04-22T18:39:18.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19981518400000001 2024-04-22T18:39:18.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542211+0000", 2024-04-22T18:39:18.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5128e-05 2024-04-22T18:39:18.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542244+0000", 2024-04-22T18:39:18.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3256000000000001e-05 2024-04-22T18:39:18.670 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.670 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.670 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.670 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:43 4.2 4:409e6456:::smithi08937929-11:head [setxattr _header (56) in=7b,truncate 2168070] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.670 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:18.670 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.358645675999998, 2024-04-22T18:39:18.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32656540499999998, 2024-04-22T18:39:18.670 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.670 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.670 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.670 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.671 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.671 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 43 2024-04-22T18:39:18.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.671 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:18.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:18.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215793+0000", 2024-04-22T18:39:18.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:18.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215797+0000", 2024-04-22T18:39:18.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4190000000000004e-06 2024-04-22T18:39:18.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215804+0000", 2024-04-22T18:39:18.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:39:18.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215809+0000", 2024-04-22T18:39:18.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3689999999999998e-06 2024-04-22T18:39:18.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215840+0000", 2024-04-22T18:39:18.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-05 2024-04-22T18:39:18.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:18.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215843+0000", 2024-04-22T18:39:18.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8849999999999999e-06 2024-04-22T18:39:18.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342483+0000", 2024-04-22T18:39:18.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126640172 2024-04-22T18:39:18.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342529+0000", 2024-04-22T18:39:18.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5668999999999999e-05 2024-04-22T18:39:18.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:18.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342597+0000", 2024-04-22T18:39:18.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7538000000000003e-05 2024-04-22T18:39:18.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.347102+0000", 2024-04-22T18:39:18.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045058040000000004 2024-04-22T18:39:18.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542278+0000", 2024-04-22T18:39:18.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.195175879 2024-04-22T18:39:18.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542314+0000", 2024-04-22T18:39:18.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6134999999999999e-05 2024-04-22T18:39:18.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542359+0000", 2024-04-22T18:39:18.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4848999999999997e-05 2024-04-22T18:39:18.676 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.676 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.676 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.676 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:44 4.2 4:409e6456:::smithi08937929-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:18.677 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:18.677 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.358625667999998, 2024-04-22T18:39:18.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326706418, 2024-04-22T18:39:18.677 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.677 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:18.677 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.677 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.677 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.677 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 44 2024-04-22T18:39:18.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.677 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:18.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:18.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215813+0000", 2024-04-22T18:39:18.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:18.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215816+0000", 2024-04-22T18:39:18.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9799999999999998e-06 2024-04-22T18:39:18.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215822+0000", 2024-04-22T18:39:18.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5570000000000003e-06 2024-04-22T18:39:18.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215827+0000", 2024-04-22T18:39:18.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2739999999999998e-06 2024-04-22T18:39:18.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215855+0000", 2024-04-22T18:39:18.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7682999999999999e-05 2024-04-22T18:39:18.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:18.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215857+0000", 2024-04-22T18:39:18.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9250000000000002e-06 2024-04-22T18:39:18.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342717+0000", 2024-04-22T18:39:18.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126860696 2024-04-22T18:39:18.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:18.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342758+0000", 2024-04-22T18:39:18.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0295e-05 2024-04-22T18:39:18.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542408+0000", 2024-04-22T18:39:18.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19964990599999999 2024-04-22T18:39:18.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542457+0000", 2024-04-22T18:39:18.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9784999999999999e-05 2024-04-22T18:39:18.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542520+0000", 2024-04-22T18:39:18.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2960999999999996e-05 2024-04-22T18:39:18.682 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.682 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.682 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.682 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:18.682 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:18.682 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.781532392999999, 2024-04-22T18:39:18.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:39:18.683 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.683 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.683 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.683 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.683 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.683 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:39:18.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.683 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:18.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:18.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:39:18.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:18.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:39:18.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:39:18.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:39:18.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:39:18.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:39:18.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:39:18.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:39:18.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:39:18.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:39:18.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:39:18.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:18.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:39:18.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:39:18.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:39:18.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:39:18.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:39:18.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:39:18.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:39:18.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:39:18.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:39:18.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:39:18.688 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.688 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.688 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.688 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:709 4.2 4:4fb26dd8:::smithi08937929-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:18.688 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.528719+0000", 2024-04-22T18:39:18.688 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 16.045720755000001, 2024-04-22T18:39:18.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30036358099999999, 2024-04-22T18:39:18.688 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.688 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.689 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.689 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.689 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 709 2024-04-22T18:39:18.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.689 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528719+0000", 2024-04-22T18:39:18.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528719+0000", 2024-04-22T18:39:18.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528718+0000", 2024-04-22T18:39:18.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:18.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528723+0000", 2024-04-22T18:39:18.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8550000000000001e-06 2024-04-22T18:39:18.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528729+0000", 2024-04-22T18:39:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8680000000000004e-06 2024-04-22T18:39:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528735+0000", 2024-04-22T18:39:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8309999999999997e-06 2024-04-22T18:39:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528766+0000", 2024-04-22T18:39:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1970000000000001e-05 2024-04-22T18:39:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528823+0000", 2024-04-22T18:39:18.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6440000000000002e-05 2024-04-22T18:39:18.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:18.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728211+0000", 2024-04-22T18:39:18.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.199387975 2024-04-22T18:39:18.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728615+0000", 2024-04-22T18:39:18.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000404161 2024-04-22T18:39:18.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.829015+0000", 2024-04-22T18:39:18.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100400197 2024-04-22T18:39:18.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.829053+0000", 2024-04-22T18:39:18.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7373000000000003e-05 2024-04-22T18:39:18.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.829082+0000", 2024-04-22T18:39:18.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9717e-05 2024-04-22T18:39:18.694 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.694 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.694 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.694 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1262 4.7 4:e117a562:::smithi08937929-33:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:18.694 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:18.694 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.1213892169999999, 2024-04-22T18:39:18.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40488836700000003, 2024-04-22T18:39:18.694 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:18.694 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:18.694 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:18.694 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:18.694 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:18.694 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1262 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453048+0000", 2024-04-22T18:39:18.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:18.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:18.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453058+0000", 2024-04-22T18:39:18.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:39:18.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:18.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453069+0000", 2024-04-22T18:39:18.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1117999999999999e-05 2024-04-22T18:39:18.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:18.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453075+0000", 2024-04-22T18:39:18.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0909999999999998e-06 2024-04-22T18:39:18.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453115+0000", 2024-04-22T18:39:18.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0119e-05 2024-04-22T18:39:18.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:18.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453119+0000", 2024-04-22T18:39:18.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3610000000000003e-06 2024-04-22T18:39:18.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:18.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454814+0000", 2024-04-22T18:39:18.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00169436 2024-04-22T18:39:18.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:18.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454861+0000", 2024-04-22T18:39:18.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7184000000000001e-05 2024-04-22T18:39:18.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:18.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756925+0000", 2024-04-22T18:39:18.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30206361799999998 2024-04-22T18:39:18.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:18.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757577+0000", 2024-04-22T18:39:18.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065199599999999996 2024-04-22T18:39:18.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:18.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857846+0000", 2024-04-22T18:39:18.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100269369 2024-04-22T18:39:18.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:18.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857896+0000", 2024-04-22T18:39:18.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9607000000000001e-05 2024-04-22T18:39:18.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:18.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:18.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:18.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857939+0000", 2024-04-22T18:39:18.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3149999999999999e-05 2024-04-22T18:39:18.700 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.700 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.700 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.700 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:18.700 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:18.700 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:18.702 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:18.702 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:18.703+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:18.803 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:18.804+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:18.805 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:18.805 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:18.805 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:18.805 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:18.814 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:18.877 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:18.878 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:18.894 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:18.895 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:18.904 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:18.905+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:18.911 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:18.911 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:18.927 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:18.927 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:18.945 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:18.945 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:18.960 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:18.960 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:18.967 INFO:tasks.rados.rados.0.smithi089.stdout:796: set_chunk oid 5 offset: 1705984 length: 1424384 target oid a3218f4e5a1a7fa66de6f9f94ce512a670fcfbf83687301ae5b7361fcbe1692b offset: 0 snap -1 2024-04-22T18:39:18.967 INFO:tasks.rados.rados.0.smithi089.stdout:797: read oid 28 snap -1 2024-04-22T18:39:18.967 INFO:tasks.rados.rados.0.smithi089.stdout:797: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-22T18:39:18.967 INFO:tasks.rados.rados.0.smithi089.stdout:798: read oid 43 snap -1 2024-04-22T18:39:18.967 INFO:tasks.rados.rados.0.smithi089.stdout:798: expect (ObjNum 51 snap 0 seq_num 51) 2024-04-22T18:39:18.968 INFO:tasks.rados.rados.0.smithi089.stdout:799: read oid 37 snap -1 2024-04-22T18:39:18.968 INFO:tasks.rados.rados.0.smithi089.stdout:799: expect (ObjNum 84 snap 0 seq_num 84) 2024-04-22T18:39:18.968 INFO:tasks.rados.rados.0.smithi089.stdout:800: read oid 16 snap -1 2024-04-22T18:39:18.968 INFO:tasks.rados.rados.0.smithi089.stdout:800: expect (ObjNum 71 snap 0 seq_num 71) 2024-04-22T18:39:18.968 INFO:tasks.rados.rados.0.smithi089.stdout:801: set_chunk oid 38 target oid 2024-04-22T18:39:18.970 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:19.005 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:19.006+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:19.105 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:19.107+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:19.112 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:19.112 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:19.121 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:19.162 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:19.162 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:19.182 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:19.199 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:19.199 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:19.206 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:19.208+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:19.218 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:39:19.270 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:19.270 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:19.280 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:19.308 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:19.309+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:19.408 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:19.410+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:19.425 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:19.425 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:19.434 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:19.473 INFO:tasks.rados.rados.0.smithi089.stdout:801: set_chunk oid 38 offset: 2968576 length: 493056 target oid 1d6726980a706cc596eb40683ac8ec6a790e383c2df9d679d3ebced9aea1af41 offset: 0 snap -1 2024-04-22T18:39:19.473 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:19.473 INFO:tasks.rados.rados.0.smithi089.stdout:787: finishing write tid 5 to smithi08937929-19 2024-04-22T18:39:19.473 INFO:tasks.rados.rados.0.smithi089.stdout:787: oid 19 updating version 0 to 57 2024-04-22T18:39:19.473 INFO:tasks.rados.rados.0.smithi089.stdout:787: oid 19 updating version 57 to 60 2024-04-22T18:39:19.473 INFO:tasks.rados.rados.0.smithi089.stdout:787: oid 19 version 60 is already newer than 59 2024-04-22T18:39:19.474 INFO:tasks.rados.rados.0.smithi089.stdout:787: oid 19 version 60 is already newer than 58 2024-04-22T18:39:19.474 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 60 (ObjNum 99 snap 0 seq_num 99) dirty exists 2024-04-22T18:39:19.474 INFO:tasks.rados.rados.0.smithi089.stdout:787: left oid 19 (ObjNum 99 snap 0 seq_num 99) 2024-04-22T18:39:19.474 INFO:tasks.rados.rados.0.smithi089.stdout:790: expect (ObjNum 82 snap 0 seq_num 82) 2024-04-22T18:39:19.516 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:19.518+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:19.576 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:19.576 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:19.585 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:19.626 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:19.628+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:19.629 INFO:tasks.rados.rados.0.smithi089.stdout:785: expect (ObjNum 58 snap 0 seq_num 58) 2024-04-22T18:39:19.699 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:19.699 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:19.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:19.710 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:19.711+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:19.724 INFO:tasks.rados.rados.0.smithi089.stdout:788: finishing write tid 1 to smithi08937929-30 2024-04-22T18:39:19.724 INFO:tasks.rados.rados.0.smithi089.stdout:784: got (0) Success 2024-04-22T18:39:19.724 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-22T18:39:19.724 INFO:tasks.rados.rados.0.smithi089.stdout:788: finishing write tid 2 to smithi08937929-30 2024-04-22T18:39:19.724 INFO:tasks.rados.rados.0.smithi089.stdout:788: finishing write tid 3 to smithi08937929-30 2024-04-22T18:39:19.724 INFO:tasks.rados.rados.0.smithi089.stdout:784: done (15 left) 2024-04-22T18:39:19.724 INFO:tasks.rados.rados.0.smithi089.stdout:785: done (14 left) 2024-04-22T18:39:19.724 INFO:tasks.rados.rados.0.smithi089.stdout:787: done (13 left) 2024-04-22T18:39:19.724 INFO:tasks.rados.rados.0.smithi089.stdout:790: done (12 left) 2024-04-22T18:39:19.724 INFO:tasks.rados.rados.0.smithi089.stdout:802: read oid 41 snap -1 2024-04-22T18:39:19.724 INFO:tasks.rados.rados.0.smithi089.stdout:802: expect (ObjNum 40 snap 0 seq_num 40) 2024-04-22T18:39:19.724 INFO:tasks.rados.rados.0.smithi089.stdout:803: read oid 32 snap -1 2024-04-22T18:39:19.724 INFO:tasks.rados.rados.0.smithi089.stdout:803: expect (ObjNum 92 snap 0 seq_num 92) 2024-04-22T18:39:19.725 INFO:tasks.rados.rados.0.smithi089.stdout:804: set_chunk oid 26 target oid 2024-04-22T18:39:19.811 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:19.812+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:19.812 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:19.812 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:19.820 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:19.840 INFO:tasks.rados.rados.0.smithi089.stdout:804: set_chunk oid 26 offset: 412160 length: 1193472 target oid 33222c7184512d7b590659344dae543d0c01f6f4a001ee3c55e2356cc0ef7218 offset: 0 snap -1 2024-04-22T18:39:19.840 INFO:tasks.rados.rados.0.smithi089.stdout:805: read oid 10 snap -1 2024-04-22T18:39:19.840 INFO:tasks.rados.rados.0.smithi089.stdout:805: expect deleted 2024-04-22T18:39:19.840 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:19.840 INFO:tasks.rados.rados.0.smithi089.stdout:788: finishing write tid 4 to smithi08937929-30 2024-04-22T18:39:19.840 INFO:tasks.rados.rados.0.smithi089.stdout:788: finishing write tid 5 to smithi08937929-30 2024-04-22T18:39:19.840 INFO:tasks.rados.rados.0.smithi089.stdout:788: oid 30 updating version 0 to 14 2024-04-22T18:39:19.840 INFO:tasks.rados.rados.0.smithi089.stdout:788: oid 30 updating version 14 to 15 2024-04-22T18:39:19.840 INFO:tasks.rados.rados.0.smithi089.stdout:788: oid 30 updating version 15 to 17 2024-04-22T18:39:19.840 INFO:tasks.rados.rados.0.smithi089.stdout:788: oid 30 version 17 is already newer than 16 2024-04-22T18:39:19.840 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 17 (ObjNum 100 snap 0 seq_num 100) dirty exists 2024-04-22T18:39:19.841 INFO:tasks.rados.rados.0.smithi089.stdout:788: left oid 30 (ObjNum 100 snap 0 seq_num 100) 2024-04-22T18:39:19.841 INFO:tasks.rados.rados.0.smithi089.stdout:793: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-22T18:39:19.912 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:19.913+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:19.924 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:19.924 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:19.924 INFO:tasks.rados.rados.0.smithi089.stdout:791: finishing set_chunk to oid 45 2024-04-22T18:39:19.924 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 45 set_chunk returned error code -95 offset: 313344 length: 1283072 2024-04-22T18:39:19.924 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-22T18:39:19.924 INFO:tasks.rados.rados.0.smithi089.stdout:794: finishing write tid 1 to smithi08937929-17 2024-04-22T18:39:19.924 INFO:tasks.rados.rados.0.smithi089.stdout:794: finishing write tid 2 to smithi08937929-17 2024-04-22T18:39:19.924 INFO:tasks.rados.rados.0.smithi089.stdout:799: expect (ObjNum 84 snap 0 seq_num 84) 2024-04-22T18:39:19.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:20.013 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:20.013+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:20.034 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:20.034 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:20.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:20.066 INFO:tasks.rados.rados.0.smithi089.stdout:798: expect (ObjNum 51 snap 0 seq_num 51) 2024-04-22T18:39:20.127 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:20.128+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:20.185 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:20.185 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:20.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:20.198 INFO:tasks.rados.rados.0.smithi089.stdout:794: finishing write tid 3 to smithi08937929-17 2024-04-22T18:39:20.198 INFO:tasks.rados.rados.0.smithi089.stdout:794: finishing write tid 4 to smithi08937929-17 2024-04-22T18:39:20.198 INFO:tasks.rados.rados.0.smithi089.stdout:794: finishing write tid 5 to smithi08937929-17 2024-04-22T18:39:20.198 INFO:tasks.rados.rados.0.smithi089.stdout:794: oid 17 updating version 0 to 38 2024-04-22T18:39:20.198 INFO:tasks.rados.rados.0.smithi089.stdout:794: oid 17 version 38 is already newer than 37 2024-04-22T18:39:20.198 INFO:tasks.rados.rados.0.smithi089.stdout:794: oid 17 updating version 38 to 40 2024-04-22T18:39:20.198 INFO:tasks.rados.rados.0.smithi089.stdout:794: oid 17 version 40 is already newer than 39 2024-04-22T18:39:20.198 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 40 (ObjNum 101 snap 0 seq_num 101) dirty exists 2024-04-22T18:39:20.198 INFO:tasks.rados.rados.0.smithi089.stdout:794: left oid 17 (ObjNum 101 snap 0 seq_num 101) 2024-04-22T18:39:20.199 INFO:tasks.rados.rados.0.smithi089.stdout:800: expect (ObjNum 71 snap 0 seq_num 71) 2024-04-22T18:39:20.219 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:20.220+0000 7fec57bee640 -1 osd.7 43 handle_osd_map injecting map crc failure 2024-04-22T18:39:20.220 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:20.220+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:20.221 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:39:20.316 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:20.317+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:20.320 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:20.321 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:20.329 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:20.344 INFO:tasks.rados.rados.0.smithi089.stdout:797: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-22T18:39:20.417 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:20.418+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:20.423 INFO:tasks.rados.rados.0.smithi089.stdout:796: finishing set_chunk to oid 5 2024-04-22T18:39:20.423 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 52 (ObjNum 87 snap 0 seq_num 87) dirty exists 2024-04-22T18:39:20.423 INFO:tasks.rados.rados.0.smithi089.stdout:795: finishing write tid 1 to smithi08937929-25 2024-04-22T18:39:20.423 INFO:tasks.rados.rados.0.smithi089.stdout:795: finishing write tid 2 to smithi08937929-25 2024-04-22T18:39:20.423 INFO:tasks.rados.rados.0.smithi089.stdout:788: done (15 left) 2024-04-22T18:39:20.423 INFO:tasks.rados.rados.0.smithi089.stdout:791: done (14 left) 2024-04-22T18:39:20.423 INFO:tasks.rados.rados.0.smithi089.stdout:792: done (13 left) 2024-04-22T18:39:20.423 INFO:tasks.rados.rados.0.smithi089.stdout:793: done (12 left) 2024-04-22T18:39:20.423 INFO:tasks.rados.rados.0.smithi089.stdout:794: done (11 left) 2024-04-22T18:39:20.423 INFO:tasks.rados.rados.0.smithi089.stdout:796: done (10 left) 2024-04-22T18:39:20.423 INFO:tasks.rados.rados.0.smithi089.stdout:797: done (9 left) 2024-04-22T18:39:20.423 INFO:tasks.rados.rados.0.smithi089.stdout:798: done (8 left) 2024-04-22T18:39:20.424 INFO:tasks.rados.rados.0.smithi089.stdout:799: done (7 left) 2024-04-22T18:39:20.424 INFO:tasks.rados.rados.0.smithi089.stdout:800: done (6 left) 2024-04-22T18:39:20.424 INFO:tasks.rados.rados.0.smithi089.stdout:806: write oid 30 current snap is 0 2024-04-22T18:39:20.424 INFO:tasks.rados.rados.0.smithi089.stdout:795: finishing write tid 3 to smithi08937929-25 2024-04-22T18:39:20.424 INFO:tasks.rados.rados.0.smithi089.stdout:795: finishing write tid 4 to smithi08937929-25 2024-04-22T18:39:20.424 INFO:tasks.rados.rados.0.smithi089.stdout:795: finishing write tid 5 to smithi08937929-25 2024-04-22T18:39:20.424 INFO:tasks.rados.rados.0.smithi089.stdout:795: oid 25 updating version 0 to 43 2024-04-22T18:39:20.424 INFO:tasks.rados.rados.0.smithi089.stdout:795: oid 25 updating version 43 to 44 2024-04-22T18:39:20.424 INFO:tasks.rados.rados.0.smithi089.stdout:795: oid 25 version 44 is already newer than 41 2024-04-22T18:39:20.424 INFO:tasks.rados.rados.0.smithi089.stdout:795: oid 25 version 44 is already newer than 42 2024-04-22T18:39:20.424 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 44 (ObjNum 102 snap 0 seq_num 102) dirty exists 2024-04-22T18:39:20.424 INFO:tasks.rados.rados.0.smithi089.stdout:795: left oid 25 (ObjNum 102 snap 0 seq_num 102) 2024-04-22T18:39:20.424 INFO:tasks.rados.rados.0.smithi089.stdout:801: finishing set_chunk to oid 38 2024-04-22T18:39:20.424 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 39 (ObjNum 97 snap 0 seq_num 97) dirty exists 2024-04-22T18:39:20.424 INFO:tasks.rados.rados.0.smithi089.stdout:806: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2024-04-22T18:39:20.426 INFO:tasks.rados.rados.0.smithi089.stdout:806: writing smithi08937929-30 from 531121 to 994357 tid 1 2024-04-22T18:39:20.434 INFO:tasks.rados.rados.0.smithi089.stdout:806: writing smithi08937929-30 from 1711057 to 2376422 tid 2 2024-04-22T18:39:20.441 INFO:tasks.rados.rados.0.smithi089.stdout:806: writing smithi08937929-30 from 3079274 to 3731121 tid 3 2024-04-22T18:39:20.441 INFO:tasks.rados.rados.0.smithi089.stdout:795: done (6 left) 2024-04-22T18:39:20.441 INFO:tasks.rados.rados.0.smithi089.stdout:801: done (5 left) 2024-04-22T18:39:20.441 INFO:tasks.rados.rados.0.smithi089.stdout:807: write (excl) oid 28 current snap is 0 2024-04-22T18:39:20.441 INFO:tasks.rados.rados.0.smithi089.stdout:807: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2024-04-22T18:39:20.446 INFO:tasks.rados.rados.0.smithi089.stdout:807: writing smithi08937929-28 from 547928 to 1086413 tid 1 2024-04-22T18:39:20.452 INFO:tasks.rados.rados.0.smithi089.stdout:807: writing smithi08937929-28 from 1769539 to 2178562 tid 2 2024-04-22T18:39:20.458 INFO:tasks.rados.rados.0.smithi089.stdout:807: writing smithi08937929-28 from 2706697 to 3238751 tid 3 2024-04-22T18:39:20.458 INFO:tasks.rados.rados.0.smithi089.stdout:808: read oid 39 snap -1 2024-04-22T18:39:20.458 INFO:tasks.rados.rados.0.smithi089.stdout:808: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-22T18:39:20.458 INFO:tasks.rados.rados.0.smithi089.stdout:809: write oid 42 current snap is 0 2024-04-22T18:39:20.458 INFO:tasks.rados.rados.0.smithi089.stdout:809: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-04-22T18:39:20.464 INFO:tasks.rados.rados.0.smithi089.stdout:809: writing smithi08937929-42 from 564735 to 1178469 tid 1 2024-04-22T18:39:20.471 INFO:tasks.rados.rados.0.smithi089.stdout:809: writing smithi08937929-42 from 1828021 to 2497055 tid 2 2024-04-22T18:39:20.472 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:20.473 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:20.478 INFO:tasks.rados.rados.0.smithi089.stdout:809: writing smithi08937929-42 from 3134120 to 3764735 tid 3 2024-04-22T18:39:20.478 INFO:tasks.rados.rados.0.smithi089.stdout:810: read oid 44 snap -1 2024-04-22T18:39:20.479 INFO:tasks.rados.rados.0.smithi089.stdout:810: expect (ObjNum 91 snap 0 seq_num 91) 2024-04-22T18:39:20.479 INFO:tasks.rados.rados.0.smithi089.stdout:811: set_chunk oid 43 target oid 2024-04-22T18:39:20.482 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:20.518 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:20.520+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:20.541 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:20.542+0000 7f3cfee88640 -1 osd.6 43 handle_osd_map injecting map crc failure 2024-04-22T18:39:20.559 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:20.560+0000 7f1e2dd22640 -1 osd.2 43 handle_osd_map injecting map crc failure 2024-04-22T18:39:20.586 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:20.586 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:20.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:20.619 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:20.620+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:20.698 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:20.699 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:20.707 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:20.751 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:20.752+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:20.810 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:20.810 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:20.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:20.860 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:20.861+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:20.922 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:20.923+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:20.962 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:20.962 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:20.971 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:21.022 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:21.024+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:21.052 INFO:tasks.rados.rados.0.smithi089.stdout:811: set_chunk oid 43 offset: 1376256 length: 933888 target oid dbf6069523e7df612d556bac5eace9a2625d707c6c25a374d4b318bb0498f921 offset: 0 snap -1 2024-04-22T18:39:21.052 INFO:tasks.rados.rados.0.smithi089.stdout:812: write (excl) oid 2 current snap is 0 2024-04-22T18:39:21.053 INFO:tasks.rados.rados.0.smithi089.stdout:803: expect (ObjNum 92 snap 0 seq_num 92) 2024-04-22T18:39:21.113 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:21.113 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:21.122 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:21.123 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:21.124+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:21.210 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:39:21.224 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:21.225+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:21.229 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:21.229 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:21.238 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:21.291 INFO:tasks.rados.rados.0.smithi089.stdout:804: finishing set_chunk to oid 26 2024-04-22T18:39:21.291 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 26 set_chunk returned error code -95 offset: 412160 length: 1193472 2024-04-22T18:39:21.291 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-22T18:39:21.291 INFO:tasks.rados.rados.0.smithi089.stdout:802: expect (ObjNum 40 snap 0 seq_num 40) 2024-04-22T18:39:21.325 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:21.326+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:21.346 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:21.346 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:21.370 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:21.371 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:21.381 INFO:tasks.rados.rados.0.smithi089.stdout:809: finishing write tid 1 to smithi08937929-42 2024-04-22T18:39:21.381 INFO:tasks.rados.rados.0.smithi089.stdout:809: finishing write tid 2 to smithi08937929-42 2024-04-22T18:39:21.381 INFO:tasks.rados.rados.0.smithi089.stdout:809: finishing write tid 3 to smithi08937929-42 2024-04-22T18:39:21.382 INFO:tasks.rados.rados.0.smithi089.stdout:809: finishing write tid 4 to smithi08937929-42 2024-04-22T18:39:21.382 INFO:tasks.rados.rados.0.smithi089.stdout:809: finishing write tid 5 to smithi08937929-42 2024-04-22T18:39:21.382 INFO:tasks.rados.rados.0.smithi089.stdout:809: oid 42 updating version 0 to 16 2024-04-22T18:39:21.382 INFO:tasks.rados.rados.0.smithi089.stdout:809: oid 42 version 16 is already newer than 13 2024-04-22T18:39:21.382 INFO:tasks.rados.rados.0.smithi089.stdout:809: oid 42 version 16 is already newer than 15 2024-04-22T18:39:21.382 INFO:tasks.rados.rados.0.smithi089.stdout:809: oid 42 version 16 is already newer than 14 2024-04-22T18:39:21.382 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 16 (ObjNum 105 snap 0 seq_num 105) dirty exists 2024-04-22T18:39:21.382 INFO:tasks.rados.rados.0.smithi089.stdout:809: left oid 42 (ObjNum 105 snap 0 seq_num 105) 2024-04-22T18:39:21.382 INFO:tasks.rados.rados.0.smithi089.stdout:806: finishing write tid 1 to smithi08937929-30 2024-04-22T18:39:21.382 INFO:tasks.rados.rados.0.smithi089.stdout:812: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-04-22T18:39:21.387 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:21.387 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:21.388 INFO:tasks.rados.rados.0.smithi089.stdout:812: writing smithi08937929-2 from 581542 to 1270525 tid 1 2024-04-22T18:39:21.390 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:39:21.390 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:39:21.390 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:39:21.394 INFO:tasks.rados.rados.0.smithi089.stdout:812: writing smithi08937929-2 from 1886503 to 2299195 tid 2 2024-04-22T18:39:21.402 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:21.402 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:21.402 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:21.402 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:21.402 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:21.403 INFO:tasks.rados.rados.0.smithi089.stdout:812: writing smithi08937929-2 from 2761543 to 3432494 tid 3 2024-04-22T18:39:21.403 INFO:tasks.rados.rados.0.smithi089.stdout:802: done (10 left) 2024-04-22T18:39:21.403 INFO:tasks.rados.rados.0.smithi089.stdout:803: done (9 left) 2024-04-22T18:39:21.403 INFO:tasks.rados.rados.0.smithi089.stdout:804: done (8 left) 2024-04-22T18:39:21.403 INFO:tasks.rados.rados.0.smithi089.stdout:805: done (7 left) 2024-04-22T18:39:21.403 INFO:tasks.rados.rados.0.smithi089.stdout:809: done (6 left) 2024-04-22T18:39:21.403 INFO:tasks.rados.rados.0.smithi089.stdout:813: tier_evict oid 19 2024-04-22T18:39:21.403 INFO:tasks.rados.rados.0.smithi089.stdout:813: tier_evict oid 19 snap -1 2024-04-22T18:39:21.403 INFO:tasks.rados.rados.0.smithi089.stdout:806: finishing write tid 2 to smithi08937929-30 2024-04-22T18:39:21.403 INFO:tasks.rados.rados.0.smithi089.stdout:806: finishing write tid 3 to smithi08937929-30 2024-04-22T18:39:21.404 INFO:tasks.rados.rados.0.smithi089.stdout:810: expect (ObjNum 91 snap 0 seq_num 91) 2024-04-22T18:39:21.404 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:21.405 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:21.409 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:21.422 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:21.422 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:21.426 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:21.427+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:21.438 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:21.438 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:21.455 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:21.456 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:21.472 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:21.473 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:21.526 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:21.528+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:21.536 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:21.537 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:21.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:21.537 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:21.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.537 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:56 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:21.537 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:39:21.537 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.283885322, 2024-04-22T18:39:21.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30396841899999999, 2024-04-22T18:39:21.537 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.537 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:21.538 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.538 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.538 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.538 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 56 2024-04-22T18:39:21.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.538 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:39:21.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:39:21.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251812+0000", 2024-04-22T18:39:21.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:21.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251815+0000", 2024-04-22T18:39:21.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7800000000000001e-06 2024-04-22T18:39:21.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251820+0000", 2024-04-22T18:39:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5829999999999997e-06 2024-04-22T18:39:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.251825+0000", 2024-04-22T18:39:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7879999999999997e-06 2024-04-22T18:39:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.252721+0000", 2024-04-22T18:39:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089552200000000005 2024-04-22T18:39:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.252794+0000", 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3440000000000002e-05 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555574+0000", 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30278038099999999 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555687+0000", 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112804 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.555781+0000", 2024-04-22T18:39:21.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3624999999999999e-05 2024-04-22T18:39:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:61 4.b 4:d923883e:::smithi08937929-16:head [write 652105~445088,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.265502700999999, 2024-04-22T18:39:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.38578178600000002, 2024-04-22T18:39:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.543 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 61 2024-04-22T18:39:21.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.543 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:21.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:21.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270192+0000", 2024-04-22T18:39:21.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:21.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270401+0000", 2024-04-22T18:39:21.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020921199999999999 2024-04-22T18:39:21.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270415+0000", 2024-04-22T18:39:21.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4191e-05 2024-04-22T18:39:21.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270422+0000", 2024-04-22T18:39:21.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7329999999999997e-06 2024-04-22T18:39:21.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270461+0000", 2024-04-22T18:39:21.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9530999999999998e-05 2024-04-22T18:39:21.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270550+0000", 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8508e-05 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270648+0000", 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8377999999999999e-05 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.271907+0000", 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012589599999999999 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655904+0000", 2024-04-22T18:39:21.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.383997115 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655933+0000", 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9011999999999999e-05 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655977+0000", 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3578000000000002e-05 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:62 4.b 4:d923883e:::smithi08937929-16:head [write 1728171~452988] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:21.547 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:39:21.548 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.259544052999999, 2024-04-22T18:39:21.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.379937407, 2024-04-22T18:39:21.548 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.548 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:21.548 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.548 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.548 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.548 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62 2024-04-22T18:39:21.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.548 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:39:21.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:39:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276151+0000", 2024-04-22T18:39:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276370+0000", 2024-04-22T18:39:21.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000218951 2024-04-22T18:39:21.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276382+0000", 2024-04-22T18:39:21.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1517e-05 2024-04-22T18:39:21.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276389+0000", 2024-04-22T18:39:21.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8939999999999993e-06 2024-04-22T18:39:21.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276430+0000", 2024-04-22T18:39:21.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0958999999999999e-05 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276485+0000", 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4208e-05 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276567+0000", 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2006000000000001e-05 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277685+0000", 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001118237 2024-04-22T18:39:21.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656024+0000", 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.378339588 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656054+0000", 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9373000000000002e-05 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656091+0000", 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7168000000000001e-05 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.552 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.553 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:63 4.b 4:d923883e:::smithi08937929-16:head [setxattr _header (56) in=7b,truncate 2252105] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:21.553 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:39:21.553 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.259301796999999, 2024-04-22T18:39:21.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37982964200000002, 2024-04-22T18:39:21.553 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.553 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:21.553 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.553 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.553 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.553 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 63 2024-04-22T18:39:21.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.553 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:39:21.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:39:21.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276395+0000", 2024-04-22T18:39:21.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:21.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276400+0000", 2024-04-22T18:39:21.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6090000000000003e-06 2024-04-22T18:39:21.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276407+0000", 2024-04-22T18:39:21.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5850000000000002e-06 2024-04-22T18:39:21.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276411+0000", 2024-04-22T18:39:21.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2569999999999999e-06 2024-04-22T18:39:21.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277254+0000", 2024-04-22T18:39:21.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084384300000000005 2024-04-22T18:39:21.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277312+0000", 2024-04-22T18:39:21.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7395999999999997e-05 2024-04-22T18:39:21.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:21.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277388+0000", 2024-04-22T18:39:21.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6604e-05 2024-04-22T18:39:21.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:21.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277870+0000", 2024-04-22T18:39:21.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000481545 2024-04-22T18:39:21.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:21.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656138+0000", 2024-04-22T18:39:21.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37826752499999999 2024-04-22T18:39:21.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:21.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656177+0000", 2024-04-22T18:39:21.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9419999999999999e-05 2024-04-22T18:39:21.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656226+0000", 2024-04-22T18:39:21.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8562e-05 2024-04-22T18:39:21.558 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.558 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.558 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.558 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:64 4.b 4:d923883e:::smithi08937929-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:21.558 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:21.558 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.259271175999999, 2024-04-22T18:39:21.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37999559999999999, 2024-04-22T18:39:21.558 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.559 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:21.559 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.559 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.559 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.559 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 64 2024-04-22T18:39:21.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.559 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:21.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:21.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:21.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:21.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276429+0000", 2024-04-22T18:39:21.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7329999999999999e-06 2024-04-22T18:39:21.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276435+0000", 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.02e-06 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276439+0000", 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7659999999999998e-06 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277530+0000", 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010909470000000001 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:21.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277603+0000", 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2638999999999994e-05 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656259+0000", 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37865579300000002 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656330+0000", 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0899e-05 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656422+0000", 2024-04-22T18:39:21.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2397000000000003e-05 2024-04-22T18:39:21.563 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.563 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.563 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.563 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:21.563 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:21.563 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.080341099999998, 2024-04-22T18:39:21.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:39:21.563 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.563 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:21.563 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.563 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.563 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.563 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:39:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:21.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:39:21.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:39:21.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:39:21.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:39:21.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:39:21.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:39:21.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:39:21.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:39:21.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:39:21.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:39:21.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:21.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:39:21.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:39:21.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:21.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:39:21.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:39:21.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:21.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:39:21.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:39:21.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:21.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:39:21.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:39:21.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:39:21.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:39:21.568 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.568 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.568 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.568 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:315 4.b 4:d923883e:::smithi08937929-16:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:21.569 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:21.569 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.543878320000001, 2024-04-22T18:39:21.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40309447300000001, 2024-04-22T18:39:21.569 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.569 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:21.569 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.569 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.569 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.569 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 315 2024-04-22T18:39:21.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.569 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:21.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:21.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991817+0000", 2024-04-22T18:39:21.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:21.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991825+0000", 2024-04-22T18:39:21.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7279999999999995e-06 2024-04-22T18:39:21.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991834+0000", 2024-04-22T18:39:21.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1980000000000007e-06 2024-04-22T18:39:21.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991841+0000", 2024-04-22T18:39:21.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0740000000000004e-06 2024-04-22T18:39:21.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991881+0000", 2024-04-22T18:39:21.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-04-22T18:39:21.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991940+0000", 2024-04-22T18:39:21.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9382000000000002e-05 2024-04-22T18:39:21.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:21.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394250+0000", 2024-04-22T18:39:21.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40230947700000003 2024-04-22T18:39:21.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:21.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394752+0000", 2024-04-22T18:39:21.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050269699999999997 2024-04-22T18:39:21.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:21.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394865+0000", 2024-04-22T18:39:21.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011267700000000001 2024-04-22T18:39:21.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:21.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394892+0000", 2024-04-22T18:39:21.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7535999999999999e-05 2024-04-22T18:39:21.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394914+0000", 2024-04-22T18:39:21.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1398000000000001e-05 2024-04-22T18:39:21.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.574 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.574 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:331 4.6 4:6700eab5:::smithi08937929-32:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:21.574 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:39:21.574 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.939141185, 2024-04-22T18:39:21.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.305691029, 2024-04-22T18:39:21.575 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.575 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:21.575 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.575 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.575 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.575 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 331 2024-04-22T18:39:21.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.575 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:39:21.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:39:21.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596556+0000", 2024-04-22T18:39:21.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:21.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596560+0000", 2024-04-22T18:39:21.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8149999999999996e-06 2024-04-22T18:39:21.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596568+0000", 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1489999999999997e-06 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596576+0000", 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.191e-06 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596619+0000", 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3507000000000002e-05 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.596709+0000", 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0043000000000005e-05 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.801630+0000", 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20492061 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.802094+0000", 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046431899999999998 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:21.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.902165+0000", 2024-04-22T18:39:21.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10007037000000001 2024-04-22T18:39:21.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:21.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.902218+0000", 2024-04-22T18:39:21.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3897999999999998e-05 2024-04-22T18:39:21.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.902248+0000", 2024-04-22T18:39:21.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9023999999999999e-05 2024-04-22T18:39:21.579 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.579 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.579 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.580 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:367 4.3 4:c6fa81cf:::smithi08937929-18:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:21.580 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:39:21.580 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.737984204, 2024-04-22T18:39:21.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30292533399999999, 2024-04-22T18:39:21.580 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.580 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:21.580 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.580 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.580 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.580 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 367 2024-04-22T18:39:21.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.580 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:39:21.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:39:21.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797713+0000", 2024-04-22T18:39:21.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:21.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797719+0000", 2024-04-22T18:39:21.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0669999999999997e-06 2024-04-22T18:39:21.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797725+0000", 2024-04-22T18:39:21.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3049999999999998e-06 2024-04-22T18:39:21.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.797731+0000", 2024-04-22T18:39:21.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7649999999999999e-06 2024-04-22T18:39:21.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.897957+0000", 2024-04-22T18:39:21.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100226045 2024-04-22T18:39:21.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.898010+0000", 2024-04-22T18:39:21.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2679999999999997e-05 2024-04-22T18:39:21.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:21.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.999633+0000", 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101623639 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.000232+0000", 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059861099999999998 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100531+0000", 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100299112 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100599+0000", 2024-04-22T18:39:21.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7989000000000002e-05 2024-04-22T18:39:21.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.100639+0000", 2024-04-22T18:39:21.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9919000000000002e-05 2024-04-22T18:39:21.585 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.585 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.585 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.585 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:393 4.b 4:defe92fb:::smithi08937929-44:head [assert-version v12,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:21.585 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:39:21.585 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.434502137999999, 2024-04-22T18:39:21.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33050822299999999, 2024-04-22T18:39:21.586 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.586 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:21.586 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.586 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.586 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.586 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 393 2024-04-22T18:39:21.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.586 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:39:21.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:39:21.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101194+0000", 2024-04-22T18:39:21.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:21.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101201+0000", 2024-04-22T18:39:21.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1269999999999997e-06 2024-04-22T18:39:21.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101211+0000", 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9380000000000004e-06 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101220+0000", 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3459999999999994e-06 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101324+0000", 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104366 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101379+0000", 2024-04-22T18:39:21.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5127999999999998e-05 2024-04-22T18:39:21.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:21.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.330649+0000", 2024-04-22T18:39:21.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.22927022699999999 2024-04-22T18:39:21.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:21.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.331182+0000", 2024-04-22T18:39:21.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053250900000000002 2024-04-22T18:39:21.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.431635+0000", 2024-04-22T18:39:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10045290599999999 2024-04-22T18:39:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.431677+0000", 2024-04-22T18:39:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1816000000000001e-05 2024-04-22T18:39:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.431704+0000", 2024-04-22T18:39:21.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7132e-05 2024-04-22T18:39:21.591 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.591 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.591 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.591 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:713 4.3 4:ca46fde9:::smithi08937929-14:head [assert-version v3,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:21.591 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.627086+0000", 2024-04-22T18:39:21.591 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.908611549, 2024-04-22T18:39:21.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30289147900000002, 2024-04-22T18:39:21.591 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.591 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:21.591 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.591 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.592 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.592 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 713 2024-04-22T18:39:21.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.592 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627086+0000", 2024-04-22T18:39:21.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627086+0000", 2024-04-22T18:39:21.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627085+0000", 2024-04-22T18:39:21.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:21.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627098+0000", 2024-04-22T18:39:21.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2869e-05 2024-04-22T18:39:21.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627111+0000", 2024-04-22T18:39:21.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2978e-05 2024-04-22T18:39:21.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627117+0000", 2024-04-22T18:39:21.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7209999999999997e-06 2024-04-22T18:39:21.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627152+0000", 2024-04-22T18:39:21.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5052000000000003e-05 2024-04-22T18:39:21.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.627204+0000", 2024-04-22T18:39:21.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1214000000000001e-05 2024-04-22T18:39:21.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:21.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.929177+0000", 2024-04-22T18:39:21.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30197355399999998 2024-04-22T18:39:21.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:21.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.929785+0000", 2024-04-22T18:39:21.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060751100000000003 2024-04-22T18:39:21.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:21.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.929922+0000", 2024-04-22T18:39:21.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013767599999999999 2024-04-22T18:39:21.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:21.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.929952+0000", 2024-04-22T18:39:21.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9708000000000001e-05 2024-04-22T18:39:21.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.929978+0000", 2024-04-22T18:39:21.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5497e-05 2024-04-22T18:39:21.596 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.596 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.596 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.597 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:767 4.6 4:6700eab5:::smithi08937929-32:head [read 0~2521017,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:21.597 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.147606+0000", 2024-04-22T18:39:21.597 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.388091787, 2024-04-22T18:39:21.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30286525600000003, 2024-04-22T18:39:21.597 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.597 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:21.597 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.597 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.597 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.597 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 767 2024-04-22T18:39:21.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.597 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.147606+0000", 2024-04-22T18:39:21.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.147606+0000", 2024-04-22T18:39:21.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.147605+0000", 2024-04-22T18:39:21.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:21.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.147609+0000", 2024-04-22T18:39:21.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6770000000000001e-06 2024-04-22T18:39:21.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.147615+0000", 2024-04-22T18:39:21.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2539999999999998e-06 2024-04-22T18:39:21.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.147620+0000", 2024-04-22T18:39:21.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3549999999999998e-06 2024-04-22T18:39:21.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.449277+0000", 2024-04-22T18:39:21.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30165729699999999 2024-04-22T18:39:21.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.449384+0000", 2024-04-22T18:39:21.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010679900000000001 2024-04-22T18:39:21.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.450471+0000", 2024-04-22T18:39:21.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001087441 2024-04-22T18:39:21.600 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.600 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.601 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.601 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:773 4.b 4:defe92fb:::smithi08937929-44:head [write 1826593~546335] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:21.601 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148009+0000", 2024-04-22T18:39:21.601 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.387688563000001, 2024-04-22T18:39:21.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30494080699999998, 2024-04-22T18:39:21.601 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.601 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:21.601 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.601 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.601 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.601 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 773 2024-04-22T18:39:21.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.602 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148009+0000", 2024-04-22T18:39:21.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148009+0000", 2024-04-22T18:39:21.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148008+0000", 2024-04-22T18:39:21.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:21.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148307+0000", 2024-04-22T18:39:21.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029880299999999999 2024-04-22T18:39:21.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148317+0000", 2024-04-22T18:39:21.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5999999999999996e-06 2024-04-22T18:39:21.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148323+0000", 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9279999999999998e-06 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.249871+0000", 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10154796200000001 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.249960+0000", 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9926000000000006e-05 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:21.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.250093+0000", 2024-04-22T18:39:21.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013287100000000001 2024-04-22T18:39:21.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:21.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256145+0000", 2024-04-22T18:39:21.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0060519830000000004 2024-04-22T18:39:21.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:21.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.452777+0000", 2024-04-22T18:39:21.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.196631417 2024-04-22T18:39:21.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:21.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.452909+0000", 2024-04-22T18:39:21.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000132488 2024-04-22T18:39:21.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.452950+0000", 2024-04-22T18:39:21.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0686999999999999e-05 2024-04-22T18:39:21.607 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.607 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.607 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.607 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:774 4.b 4:defe92fb:::smithi08937929-44:head [write 2778819~95145] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:21.607 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:39:21.607 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.387368643999999, 2024-04-22T18:39:21.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40475725000000001, 2024-04-22T18:39:21.607 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.607 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:21.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.608 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 774 2024-04-22T18:39:21.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.608 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:39:21.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:39:21.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148328+0000", 2024-04-22T18:39:21.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:21.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148392+0000", 2024-04-22T18:39:21.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3323000000000003e-05 2024-04-22T18:39:21.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148398+0000", 2024-04-22T18:39:21.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4420000000000003e-06 2024-04-22T18:39:21.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148409+0000", 2024-04-22T18:39:21.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509999999999999e-05 2024-04-22T18:39:21.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251407+0000", 2024-04-22T18:39:21.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102998305 2024-04-22T18:39:21.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251500+0000", 2024-04-22T18:39:21.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3607e-05 2024-04-22T18:39:21.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:21.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251648+0000", 2024-04-22T18:39:21.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014788099999999999 2024-04-22T18:39:21.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:21.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256149+0000", 2024-04-22T18:39:21.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045010980000000003 2024-04-22T18:39:21.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:21.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553016+0000", 2024-04-22T18:39:21.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29686620800000002 2024-04-22T18:39:21.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:21.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553054+0000", 2024-04-22T18:39:21.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8414999999999997e-05 2024-04-22T18:39:21.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553086+0000", 2024-04-22T18:39:21.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2236000000000001e-05 2024-04-22T18:39:21.612 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.612 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.612 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.387283851999999, 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:39:21.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:39:21.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:39:21.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:21.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:39:21.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:39:21.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:21.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:39:21.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:39:21.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:21.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:39:21.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:39:21.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:39:21.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:39:21.618 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.618 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.618 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.618 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:21.618 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:21.618 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.387265375999998, 2024-04-22T18:39:21.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:39:21.618 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.618 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:21.618 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.618 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.618 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.618 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:39:21.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.619 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:21.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:21.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:21.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:21.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:39:21.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:39:21.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:39:21.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:39:21.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:39:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:39:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:39:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:39:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:39:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:39:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:39:21.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:39:21.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:39:21.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:39:21.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:39:21.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:39:21.622 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.622 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1077 4.6 4:6700eab5:::smithi08937929-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:12.231515+0000", 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.3041829150000002, 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.303721886, 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1077 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.231515+0000", 2024-04-22T18:39:21.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.231515+0000", 2024-04-22T18:39:21.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.231514+0000", 2024-04-22T18:39:21.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:21.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.231517+0000", 2024-04-22T18:39:21.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1650000000000002e-06 2024-04-22T18:39:21.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.231524+0000", 2024-04-22T18:39:21.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3929999999999996e-06 2024-04-22T18:39:21.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.331593+0000", 2024-04-22T18:39:21.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100069617 2024-04-22T18:39:21.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.331631+0000", 2024-04-22T18:39:21.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7697000000000002e-05 2024-04-22T18:39:21.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:21.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.331693+0000", 2024-04-22T18:39:21.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2038999999999993e-05 2024-04-22T18:39:21.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.535052+0000", 2024-04-22T18:39:21.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20335874200000001 2024-04-22T18:39:21.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.535138+0000", 2024-04-22T18:39:21.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6026999999999995e-05 2024-04-22T18:39:21.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.535237+0000", 2024-04-22T18:39:21.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8875000000000005e-05 2024-04-22T18:39:21.627 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.627 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.627 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.627 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1242 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-22T18:39:21.627 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:39:21.628 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.7940198269999996, 2024-04-22T18:39:21.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40662891899999998, 2024-04-22T18:39:21.628 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.628 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:21.628 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.628 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.628 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.628 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1242 2024-04-22T18:39:21.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.628 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:39:21.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:39:21.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741677+0000", 2024-04-22T18:39:21.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:21.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741680+0000", 2024-04-22T18:39:21.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8059999999999999e-06 2024-04-22T18:39:21.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741685+0000", 2024-04-22T18:39:21.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0780000000000003e-06 2024-04-22T18:39:21.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741690+0000", 2024-04-22T18:39:21.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4020000000000004e-06 2024-04-22T18:39:21.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741732+0000", 2024-04-22T18:39:21.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2110000000000002e-05 2024-04-22T18:39:21.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741798+0000", 2024-04-22T18:39:21.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6432000000000004e-05 2024-04-22T18:39:21.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.148307+0000", 2024-04-22T18:39:21.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40650862100000001 2024-04-22T18:39:21.631 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.631 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.631 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.632 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:123 3.1 3:b3ca9fad:::7a14de22115bcf5f1623520f6137ab49508b1483c9b92be35e3a65245166f52a:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:21.632 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.945685+0000", 2024-04-22T18:39:21.632 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.5900128000000002, 2024-04-22T18:39:21.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30758832000000003, 2024-04-22T18:39:21.632 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.632 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:21.632 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.945685+0000", 2024-04-22T18:39:21.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.945685+0000", 2024-04-22T18:39:21.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.945683+0000", 2024-04-22T18:39:21.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:21.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.945750+0000", 2024-04-22T18:39:21.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7497999999999996e-05 2024-04-22T18:39:21.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.945757+0000", 2024-04-22T18:39:21.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6680000000000004e-06 2024-04-22T18:39:21.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.945765+0000", 2024-04-22T18:39:21.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3629999999999994e-06 2024-04-22T18:39:21.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.046699+0000", 2024-04-22T18:39:21.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100933852 2024-04-22T18:39:21.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.046817+0000", 2024-04-22T18:39:21.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000117477 2024-04-22T18:39:21.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:21.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.046965+0000", 2024-04-22T18:39:21.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014835200000000001 2024-04-22T18:39:21.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:21.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.048402+0000", 2024-04-22T18:39:21.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0014373120000000001 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.253161+0000", 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20475876500000001 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.253217+0000", 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6215000000000003e-05 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.253273+0000", 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5875999999999998e-05 2024-04-22T18:39:21.636 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.637 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.637 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.637 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:124 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:21.637 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.945773+0000", 2024-04-22T18:39:21.637 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.5899249600000003, 2024-04-22T18:39:21.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30764503500000001, 2024-04-22T18:39:21.637 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.637 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:21.637 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.945773+0000", 2024-04-22T18:39:21.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.945773+0000", 2024-04-22T18:39:21.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.945772+0000", 2024-04-22T18:39:21.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:21.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.945807+0000", 2024-04-22T18:39:21.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4570999999999998e-05 2024-04-22T18:39:21.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.945813+0000", 2024-04-22T18:39:21.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2879999999999998e-06 2024-04-22T18:39:21.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:21.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.945819+0000", 2024-04-22T18:39:21.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2319999999999998e-06 2024-04-22T18:39:21.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:21.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.047388+0000", 2024-04-22T18:39:21.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156934300000001 2024-04-22T18:39:21.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:21.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.047514+0000", 2024-04-22T18:39:21.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012525500000000001 2024-04-22T18:39:21.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:21.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.047724+0000", 2024-04-22T18:39:21.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021013499999999999 2024-04-22T18:39:21.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:21.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.048676+0000", 2024-04-22T18:39:21.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00095213399999999999 2024-04-22T18:39:21.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:21.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.253316+0000", 2024-04-22T18:39:21.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.204639933 2024-04-22T18:39:21.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:21.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.253361+0000", 2024-04-22T18:39:21.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5253999999999997e-05 2024-04-22T18:39:21.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:21.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.253418+0000", 2024-04-22T18:39:21.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6619000000000001e-05 2024-04-22T18:39:21.642 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.642 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.642 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.642 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.642 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.642 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:21.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:21.644 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:21.644+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:21.652 INFO:tasks.rados.rados.0.smithi089.stdout:808: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-22T18:39:21.689 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:39:21.690 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:39:21.701 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-04-22T18:39:20.968099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":224710720,"num_objects":128,"num_object_clones":0,"num_object_copies":256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":1069,"num_read_kb":632252,"num_write":871,"num_write_kb":261574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":33,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":845,"ondisk_log_size":845,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":467212,"kb_used_data":230772,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754507508,"statfs":{"total":773094113280,"available":772615688192,"internally_reserved":0,"allocated":236310528,"data_stored":243068052,"data_compressed":7601493,"data_compressed_allocated":8101888,"data_compressed_original":17899520,"omap_allocated":12744,"internal_metadata":225431096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1286543,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":216,"num_read_kb":205550,"num_write":129,"num_write_kb":38458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.773508"},"pg_stats":[{"pgid":"4.f","version":"41'18","reported_seq":61,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:14.137057+0000","last_change":"2024-04-22T18:38:48.173640+0000","last_active":"2024-04-22T18:39:14.137057+0000","last_peered":"2024-04-22T18:39:14.137057+0000","last_clean":"2024-04-22T18:39:14.137057+0000","last_became_active":"2024-04-22T18:38:48.173369+0000","last_became_peered":"2024-04-22T18:38:48.173369+0000","last_unstale":"2024-04-22T18:39:14.137057+0000","last_undegraded":"2024-04-22T18:39:14.137057+0000","last_fullsized":"2024-04-22T18:39:14.137057+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:08.819198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4151263,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":9616,"num_write":13,"num_write_kb":2771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"33'10","reported_seq":49,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T18:39:15.129293+0000","last_change":"2024-04-22T18:38:48.075222+0000","last_active":"2024-04-22T18:39:15.129293+0000","last_peered":"2024-04-22T18:39:15.129293+0000","last_clean":"2024-04-22T18:39:15.129293+0000","last_became_active":"2024-04-22T18:38:48.074901+0000","last_became_peered":"2024-04-22T18:38:48.074901+0000","last_unstale":"2024-04-22T18:39:15.129293+0000","last_undegraded":"2024-04-22T18:39:15.129293+0000","last_fullsized":"2024-04-22T18:39:15.129293+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.172332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756315,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":16874,"num_write":9,"num_write_kb":2372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"32'12","reported_seq":55,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T18:39:14.213478+0000","last_change":"2024-04-22T18:38:48.172087+0000","last_active":"2024-04-22T18:39:14.213478+0000","last_peered":"2024-04-22T18:39:14.213478+0000","last_clean":"2024-04-22T18:39:14.213478+0000","last_became_active":"2024-04-22T18:38:48.171797+0000","last_became_peered":"2024-04-22T18:38:48.171797+0000","last_unstale":"2024-04-22T18:39:14.213478+0000","last_undegraded":"2024-04-22T18:39:14.213478+0000","last_fullsized":"2024-04-22T18:39:14.213478+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.134124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":25939,"num_write":8,"num_write_kb":1802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"42'34","reported_seq":110,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T18:39:16.261256+0000","last_change":"2024-04-22T18:38:48.274538+0000","last_active":"2024-04-22T18:39:16.261256+0000","last_peered":"2024-04-22T18:39:16.261256+0000","last_clean":"2024-04-22T18:39:16.261256+0000","last_became_active":"2024-04-22T18:38:48.174204+0000","last_became_peered":"2024-04-22T18:38:48.174204+0000","last_unstale":"2024-04-22T18:39:16.261256+0000","last_undegraded":"2024-04-22T18:39:16.261256+0000","last_fullsized":"2024-04-22T18:39:16.261256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:11.823691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10773155,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":123743,"num_write":29,"num_write_kb":8329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"42'54","reported_seq":127,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T18:39:16.161208+0000","last_change":"2024-04-22T18:38:48.269362+0000","last_active":"2024-04-22T18:39:16.161208+0000","last_peered":"2024-04-22T18:39:16.161208+0000","last_clean":"2024-04-22T18:39:16.161208+0000","last_became_active":"2024-04-22T18:38:48.269066+0000","last_became_peered":"2024-04-22T18:38:48.269066+0000","last_unstale":"2024-04-22T18:39:16.161208+0000","last_undegraded":"2024-04-22T18:39:16.161208+0000","last_fullsized":"2024-04-22T18:39:16.161208+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:52.098608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16000099,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":75855,"num_write":51,"num_write_kb":15222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"41'26","reported_seq":77,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T18:39:15.129742+0000","last_change":"2024-04-22T18:38:48.372952+0000","last_active":"2024-04-22T18:39:15.129742+0000","last_peered":"2024-04-22T18:39:15.129742+0000","last_clean":"2024-04-22T18:39:15.129742+0000","last_became_active":"2024-04-22T18:38:48.277922+0000","last_became_peered":"2024-04-22T18:38:48.277922+0000","last_unstale":"2024-04-22T18:39:15.129742+0000","last_undegraded":"2024-04-22T18:39:15.129742+0000","last_fullsized":"2024-04-22T18:39:15.129742+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:15.471951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":26879,"num_write":23,"num_write_kb":5049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"41'41","reported_seq":111,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T18:39:14.213416+0000","last_change":"2024-04-22T18:38:48.271633+0000","last_active":"2024-04-22T18:39:14.213416+0000","last_peered":"2024-04-22T18:39:14.213416+0000","last_clean":"2024-04-22T18:39:14.213416+0000","last_became_active":"2024-04-22T18:38:48.171245+0000","last_became_peered":"2024-04-22T18:38:48.171245+0000","last_unstale":"2024-04-22T18:39:14.213416+0000","last_undegraded":"2024-04-22T18:39:14.213416+0000","last_fullsized":"2024-04-22T18:39:14.213416+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:47.449912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13848803,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":89,"num_read_kb":70567,"num_write":31,"num_write_kb":9191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"41'27","reported_seq":76,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:12.928750+0000","last_change":"2024-04-22T18:38:48.149226+0000","last_active":"2024-04-22T18:39:12.928750+0000","last_peered":"2024-04-22T18:39:12.928750+0000","last_clean":"2024-04-22T18:39:12.928750+0000","last_became_active":"2024-04-22T18:38:48.148955+0000","last_became_peered":"2024-04-22T18:38:48.148955+0000","last_unstale":"2024-04-22T18:39:12.928750+0000","last_undegraded":"2024-04-22T18:39:12.928750+0000","last_fullsized":"2024-04-22T18:39:12.928750+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:02.394854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8638699,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":39,"num_read_kb":36864,"num_write":27,"num_write_kb":7831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"42'20","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T18:39:15.238368+0000","last_change":"2024-04-22T18:38:48.277905+0000","last_active":"2024-04-22T18:39:15.238368+0000","last_peered":"2024-04-22T18:39:15.238368+0000","last_clean":"2024-04-22T18:39:15.238368+0000","last_became_active":"2024-04-22T18:38:48.277640+0000","last_became_peered":"2024-04-22T18:38:48.277640+0000","last_unstale":"2024-04-22T18:39:15.238368+0000","last_undegraded":"2024-04-22T18:39:15.238368+0000","last_fullsized":"2024-04-22T18:39:15.238368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:57.153161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5831963,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":34571,"num_write":17,"num_write_kb":4628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":184,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:11.187917+0000","last_change":"2024-04-22T18:38:38.109358+0000","last_active":"2024-04-22T18:39:11.187917+0000","last_peered":"2024-04-22T18:39:11.187917+0000","last_clean":"2024-04-22T18:39:11.187917+0000","last_became_active":"2024-04-22T18:38:38.108568+0000","last_became_peered":"2024-04-22T18:38:38.108568+0000","last_unstale":"2024-04-22T18:39:11.187917+0000","last_undegraded":"2024-04-22T18:39:11.187917+0000","last_fullsized":"2024-04-22T18:39:11.187917+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:38:36.991764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:46.436569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":35,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:11.262836+0000","last_change":"2024-04-22T18:38:39.010398+0000","last_active":"2024-04-22T18:39:11.262836+0000","last_peered":"2024-04-22T18:39:11.262836+0000","last_clean":"2024-04-22T18:39:11.262836+0000","last_became_active":"2024-04-22T18:38:39.010101+0000","last_became_peered":"2024-04-22T18:38:39.010101+0000","last_unstale":"2024-04-22T18:39:11.262836+0000","last_undegraded":"2024-04-22T18:39:11.262836+0000","last_fullsized":"2024-04-22T18:39:11.262836+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.708134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"42'62","reported_seq":103,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T18:39:14.410341+0000","last_change":"2024-04-22T18:38:58.429682+0000","last_active":"2024-04-22T18:39:14.410341+0000","last_peered":"2024-04-22T18:39:14.410341+0000","last_clean":"2024-04-22T18:39:14.410341+0000","last_became_active":"2024-04-22T18:38:58.125648+0000","last_became_peered":"2024-04-22T18:38:58.125648+0000","last_unstale":"2024-04-22T18:39:14.410341+0000","last_undegraded":"2024-04-22T18:39:14.410341+0000","last_fullsized":"2024-04-22T18:39:14.410341+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:52.346195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21586174,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":44,"num_read_kb":28,"num_write":67,"num_write_kb":34793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"42'31","reported_seq":83,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T18:39:16.057006+0000","last_change":"2024-04-22T18:38:48.173369+0000","last_active":"2024-04-22T18:39:16.057006+0000","last_peered":"2024-04-22T18:39:16.057006+0000","last_clean":"2024-04-22T18:39:16.057006+0000","last_became_active":"2024-04-22T18:38:48.173090+0000","last_became_peered":"2024-04-22T18:38:48.173090+0000","last_unstale":"2024-04-22T18:39:16.057006+0000","last_undegraded":"2024-04-22T18:39:16.057006+0000","last_fullsized":"2024-04-22T18:39:16.057006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:05.687904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9126102,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":35576,"num_write":32,"num_write_kb":9887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:11.187833+0000","last_change":"2024-04-22T18:38:39.125885+0000","last_active":"2024-04-22T18:39:11.187833+0000","last_peered":"2024-04-22T18:39:11.187833+0000","last_clean":"2024-04-22T18:39:11.187833+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:39:11.187833+0000","last_undegraded":"2024-04-22T18:39:11.187833+0000","last_fullsized":"2024-04-22T18:39:11.187833+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.321452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"42'74","reported_seq":105,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T18:39:14.510058+0000","last_change":"2024-04-22T18:38:47.173337+0000","last_active":"2024-04-22T18:39:14.510058+0000","last_peered":"2024-04-22T18:39:14.510058+0000","last_clean":"2024-04-22T18:39:14.510058+0000","last_became_active":"2024-04-22T18:38:47.173065+0000","last_became_peered":"2024-04-22T18:38:47.173065+0000","last_unstale":"2024-04-22T18:39:14.510058+0000","last_undegraded":"2024-04-22T18:39:14.510058+0000","last_fullsized":"2024-04-22T18:39:14.510058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:52.987154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28941146,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":56,"num_read_kb":33,"num_write":83,"num_write_kb":38649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"40'24","reported_seq":73,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:14.137901+0000","last_change":"2024-04-22T18:38:48.278028+0000","last_active":"2024-04-22T18:39:14.137901+0000","last_peered":"2024-04-22T18:39:14.137901+0000","last_clean":"2024-04-22T18:39:14.137901+0000","last_became_active":"2024-04-22T18:38:48.277802+0000","last_became_peered":"2024-04-22T18:38:48.277802+0000","last_unstale":"2024-04-22T18:39:14.137901+0000","last_undegraded":"2024-04-22T18:39:14.137901+0000","last_fullsized":"2024-04-22T18:39:14.137901+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:12.000401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6336173,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":35008,"num_write":23,"num_write_kb":6520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":33,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:11.493102+0000","last_change":"2024-04-22T18:38:39.164369+0000","last_active":"2024-04-22T18:39:11.493102+0000","last_peered":"2024-04-22T18:39:11.493102+0000","last_clean":"2024-04-22T18:39:11.493102+0000","last_became_active":"2024-04-22T18:38:39.164089+0000","last_became_peered":"2024-04-22T18:38:39.164089+0000","last_unstale":"2024-04-22T18:39:11.493102+0000","last_undegraded":"2024-04-22T18:39:11.493102+0000","last_fullsized":"2024-04-22T18:39:11.493102+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:44.825470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"42'58","reported_seq":88,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T18:39:15.543900+0000","last_change":"2024-04-22T18:38:47.172071+0000","last_active":"2024-04-22T18:39:15.543900+0000","last_peered":"2024-04-22T18:39:15.543900+0000","last_clean":"2024-04-22T18:39:15.543900+0000","last_became_active":"2024-04-22T18:38:47.171789+0000","last_became_peered":"2024-04-22T18:38:47.171789+0000","last_unstale":"2024-04-22T18:39:15.543900+0000","last_undegraded":"2024-04-22T18:39:15.543900+0000","last_fullsized":"2024-04-22T18:39:15.543900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:47.501470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25596602,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":42,"num_read_kb":22,"num_write":68,"num_write_kb":32665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"41'12","reported_seq":50,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:13.856947+0000","last_change":"2024-04-22T18:38:48.171024+0000","last_active":"2024-04-22T18:39:13.856947+0000","last_peered":"2024-04-22T18:39:13.856947+0000","last_clean":"2024-04-22T18:39:13.856947+0000","last_became_active":"2024-04-22T18:38:48.170789+0000","last_became_peered":"2024-04-22T18:38:48.170789+0000","last_unstale":"2024-04-22T18:39:13.856947+0000","last_undegraded":"2024-04-22T18:39:13.856947+0000","last_fullsized":"2024-04-22T18:39:13.856947+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:06.487306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3479016,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":13172,"num_write":14,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":35,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:11.288526+0000","last_change":"2024-04-22T18:38:39.111223+0000","last_active":"2024-04-22T18:39:11.288526+0000","last_peered":"2024-04-22T18:39:11.288526+0000","last_clean":"2024-04-22T18:39:11.288526+0000","last_became_active":"2024-04-22T18:38:39.110914+0000","last_became_peered":"2024-04-22T18:38:39.110914+0000","last_unstale":"2024-04-22T18:39:11.288526+0000","last_undegraded":"2024-04-22T18:39:11.288526+0000","last_fullsized":"2024-04-22T18:39:11.288526+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.570327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"42'110","reported_seq":143,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T18:39:14.807513+0000","last_change":"2024-04-22T18:38:47.173101+0000","last_active":"2024-04-22T18:39:14.807513+0000","last_peered":"2024-04-22T18:39:14.807513+0000","last_clean":"2024-04-22T18:39:14.807513+0000","last_became_active":"2024-04-22T18:38:47.172816+0000","last_became_peered":"2024-04-22T18:38:47.172816+0000","last_unstale":"2024-04-22T18:39:14.807513+0000","last_undegraded":"2024-04-22T18:39:14.807513+0000","last_fullsized":"2024-04-22T18:39:14.807513+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:12.128375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23558854,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":89,"num_read_kb":2231,"num_write":120,"num_write_kb":47648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"41'18","reported_seq":55,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:13.717462+0000","last_change":"2024-04-22T18:38:48.372570+0000","last_active":"2024-04-22T18:39:13.717462+0000","last_peered":"2024-04-22T18:39:13.717462+0000","last_clean":"2024-04-22T18:39:13.717462+0000","last_became_active":"2024-04-22T18:38:48.272283+0000","last_became_peered":"2024-04-22T18:38:48.272283+0000","last_unstale":"2024-04-22T18:39:13.717462+0000","last_undegraded":"2024-04-22T18:39:13.717462+0000","last_fullsized":"2024-04-22T18:39:13.717462+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:09.183390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445402,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":6263,"num_write":14,"num_write_kb":3693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:11.382354+0000","last_change":"2024-04-22T18:38:39.307735+0000","last_active":"2024-04-22T18:39:11.382354+0000","last_peered":"2024-04-22T18:39:11.382354+0000","last_clean":"2024-04-22T18:39:11.382354+0000","last_became_active":"2024-04-22T18:38:39.307459+0000","last_became_peered":"2024-04-22T18:38:39.307459+0000","last_unstale":"2024-04-22T18:39:11.382354+0000","last_undegraded":"2024-04-22T18:39:11.382354+0000","last_fullsized":"2024-04-22T18:39:11.382354+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.680561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"41'36","reported_seq":84,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:14.202618+0000","last_change":"2024-04-22T18:38:48.269379+0000","last_active":"2024-04-22T18:39:14.202618+0000","last_peered":"2024-04-22T18:39:14.202618+0000","last_clean":"2024-04-22T18:39:14.202618+0000","last_became_active":"2024-04-22T18:38:48.269070+0000","last_became_peered":"2024-04-22T18:38:48.269070+0000","last_unstale":"2024-04-22T18:39:14.202618+0000","last_undegraded":"2024-04-22T18:39:14.202618+0000","last_fullsized":"2024-04-22T18:39:14.202618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:08.651956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7764735,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":34705,"num_write":31,"num_write_kb":8758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":33,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:11.287856+0000","last_change":"2024-04-22T18:38:39.210344+0000","last_active":"2024-04-22T18:39:11.287856+0000","last_peered":"2024-04-22T18:39:11.287856+0000","last_clean":"2024-04-22T18:39:11.287856+0000","last_became_active":"2024-04-22T18:38:39.210049+0000","last_became_peered":"2024-04-22T18:38:39.210049+0000","last_unstale":"2024-04-22T18:39:11.287856+0000","last_undegraded":"2024-04-22T18:39:11.287856+0000","last_fullsized":"2024-04-22T18:39:11.287856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.878594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"31'5","reported_seq":40,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:12.654570+0000","last_change":"2024-04-22T18:38:48.349979+0000","last_active":"2024-04-22T18:39:12.654570+0000","last_peered":"2024-04-22T18:39:12.654570+0000","last_clean":"2024-04-22T18:39:12.654570+0000","last_became_active":"2024-04-22T18:38:48.349706+0000","last_became_peered":"2024-04-22T18:38:48.349706+0000","last_unstale":"2024-04-22T18:39:12.654570+0000","last_undegraded":"2024-04-22T18:39:12.654570+0000","last_fullsized":"2024-04-22T18:39:12.654570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:54.435973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2319333,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":13595,"num_write":4,"num_write_kb":801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:11.382398+0000","last_change":"2024-04-22T18:38:39.110499+0000","last_active":"2024-04-22T18:39:11.382398+0000","last_peered":"2024-04-22T18:39:11.382398+0000","last_clean":"2024-04-22T18:39:11.382398+0000","last_became_active":"2024-04-22T18:38:39.010173+0000","last_became_peered":"2024-04-22T18:38:39.010173+0000","last_unstale":"2024-04-22T18:39:11.382398+0000","last_undegraded":"2024-04-22T18:39:11.382398+0000","last_fullsized":"2024-04-22T18:39:11.382398+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:58.222501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"41'56","reported_seq":126,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-22T18:39:14.400398+0000","last_change":"2024-04-22T18:38:48.372226+0000","last_active":"2024-04-22T18:39:14.400398+0000","last_peered":"2024-04-22T18:39:14.400398+0000","last_clean":"2024-04-22T18:39:14.400398+0000","last_became_active":"2024-04-22T18:38:48.371943+0000","last_became_peered":"2024-04-22T18:38:48.371943+0000","last_unstale":"2024-04-22T18:39:14.400398+0000","last_undegraded":"2024-04-22T18:39:14.400398+0000","last_fullsized":"2024-04-22T18:39:14.400398+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.637563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20151362,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":81,"num_read_kb":70611,"num_write":48,"num_write_kb":14325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-04-22T18:39:12.186721+0000","last_change":"2024-04-22T18:38:39.026078+0000","last_active":"2024-04-22T18:39:12.186721+0000","last_peered":"2024-04-22T18:39:12.186721+0000","last_clean":"2024-04-22T18:39:12.186721+0000","last_became_active":"2024-04-22T18:38:39.025741+0000","last_became_peered":"2024-04-22T18:38:39.025741+0000","last_unstale":"2024-04-22T18:39:12.186721+0000","last_undegraded":"2024-04-22T18:39:12.186721+0000","last_fullsized":"2024-04-22T18:39:12.186721+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:01.125349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":124437576,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":722,"num_read_kb":629838,"num_write":374,"num_write_kb":105403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":33,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":136126464,"data_stored":135430408,"data_compressed":106644,"data_compressed_allocated":114688,"data_compressed_original":286720,"omap_allocated":0,"internal_metadata":0},"log_size":424,"ondisk_log_size":424,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":99682776,"num_objects":82,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":231,"num_read_kb":2314,"num_write":338,"num_write_kb":153755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96681984,"data_stored":105152652,"data_compressed":7435713,"data_compressed_allocated":7905280,"data_compressed_original":16588800,"omap_allocated":0,"internal_metadata":0},"log_size":304,"ondisk_log_size":304,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34632,"kb_used_data":5992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337208,"statfs":{"total":96636764160,"available":96601300992,"internally_reserved":0,"allocated":6135808,"data_stored":6298456,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34024,"kb_used_data":7000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337816,"statfs":{"total":96636764160,"available":96601923584,"internally_reserved":0,"allocated":7168000,"data_stored":6858196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67244,"kb_used_data":35056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94304596,"statfs":{"total":96636764160,"available":96567906304,"internally_reserved":0,"allocated":35897344,"data_stored":36443392,"data_compressed":691631,"data_compressed_allocated":741376,"data_compressed_original":1712128,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87208,"kb_used_data":56156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94284632,"statfs":{"total":96636764160,"available":96547463168,"internally_reserved":0,"allocated":57503744,"data_stored":59288319,"data_compressed":1936502,"data_compressed_allocated":2048000,"data_compressed_original":4304896,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44768,"kb_used_data":17748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327072,"statfs":{"total":96636764160,"available":96590921728,"internally_reserved":0,"allocated":18173952,"data_stored":18818438,"data_compressed":691631,"data_compressed_allocated":741376,"data_compressed_original":1712128,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":98.957999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43220,"kb_used_data":16204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328620,"statfs":{"total":96636764160,"available":96592506880,"internally_reserved":0,"allocated":16592896,"data_stored":17446065,"data_compressed":1081783,"data_compressed_allocated":1155072,"data_compressed_original":2277376,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75996,"kb_used_data":46648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94295844,"statfs":{"total":96636764160,"available":96558944256,"internally_reserved":0,"allocated":47767552,"data_stored":48543523,"data_compressed":1205364,"data_compressed_allocated":1286144,"data_compressed_original":2551808,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80120,"kb_used_data":45968,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94291720,"statfs":{"total":96636764160,"available":96554721280,"internally_reserved":0,"allocated":47071232,"data_stored":49371663,"data_compressed":1965014,"data_compressed_allocated":2088960,"data_compressed_original":4829184,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26775552,"data_stored":28936778,"data_compressed":1909313,"data_compressed_allocated":2019328,"data_compressed_original":4239360,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12685312,"data_stored":13846756,"data_compressed":1152042,"data_compressed_allocated":1228800,"data_compressed_original":2408448,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12021760,"data_stored":13125860,"data_compressed":1081783,"data_compressed_allocated":1155072,"data_compressed_original":2277376,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9211904,"data_stored":10153240,"data_compressed":691631,"data_compressed_allocated":741376,"data_compressed_original":1712128,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":26775552,"data_stored":28936778,"data_compressed":1909313,"data_compressed_allocated":2019328,"data_compressed_original":4239360,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9211904,"data_stored":10153240,"data_compressed":691631,"data_compressed_allocated":741376,"data_compressed_original":1712128,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19795968,"data_stored":19725872,"data_compressed":26133,"data_compressed_allocated":28672,"data_compressed_original":77824,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34648064,"data_stored":34517669,"data_compressed":53322,"data_compressed_allocated":57344,"data_compressed_original":143360,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4169728,"data_stored":4143784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8540160,"data_stored":8486158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30330880,"data_stored":30175730,"data_compressed":27189,"data_compressed_allocated":28672,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":26288128,"data_stored":26114376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6717440,"data_stored":6677370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5636096,"data_stored":5589449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:39:21.742 INFO:tasks.rados.rados.0.smithi089.stdout:806: finishing write tid 4 to smithi08937929-30 2024-04-22T18:39:21.742 INFO:tasks.rados.rados.0.smithi089.stdout:806: finishing write tid 5 to smithi08937929-30 2024-04-22T18:39:21.742 INFO:tasks.rados.rados.0.smithi089.stdout:806: oid 30 updating version 0 to 22 2024-04-22T18:39:21.742 INFO:tasks.rados.rados.0.smithi089.stdout:806: oid 30 version 22 is already newer than 21 2024-04-22T18:39:21.743 INFO:tasks.rados.rados.0.smithi089.stdout:806: oid 30 version 22 is already newer than 20 2024-04-22T18:39:21.743 INFO:tasks.rados.rados.0.smithi089.stdout:806: oid 30 version 22 is already newer than 19 2024-04-22T18:39:21.743 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 22 (ObjNum 103 snap 0 seq_num 103) dirty exists 2024-04-22T18:39:21.743 INFO:tasks.rados.rados.0.smithi089.stdout:806: left oid 30 (ObjNum 103 snap 0 seq_num 103) 2024-04-22T18:39:21.743 INFO:tasks.rados.rados.0.smithi089.stdout:807: finishing write tid 1 to smithi08937929-28 2024-04-22T18:39:21.743 INFO:tasks.rados.rados.0.smithi089.stdout:807: finishing write tid 2 to smithi08937929-28 2024-04-22T18:39:21.743 INFO:tasks.rados.rados.0.smithi089.stdout:807: finishing write tid 3 to smithi08937929-28 2024-04-22T18:39:21.743 INFO:tasks.rados.rados.0.smithi089.stdout:807: finishing write tid 4 to smithi08937929-28 2024-04-22T18:39:21.743 INFO:tasks.rados.rados.0.smithi089.stdout:807: finishing write tid 5 to smithi08937929-28 2024-04-22T18:39:21.743 INFO:tasks.rados.rados.0.smithi089.stdout:807: oid 28 updating version 0 to 63 2024-04-22T18:39:21.743 INFO:tasks.rados.rados.0.smithi089.stdout:807: oid 28 version 63 is already newer than 62 2024-04-22T18:39:21.743 INFO:tasks.rados.rados.0.smithi089.stdout:807: oid 28 updating version 63 to 65 2024-04-22T18:39:21.743 INFO:tasks.rados.rados.0.smithi089.stdout:807: oid 28 version 65 is already newer than 64 2024-04-22T18:39:21.743 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 65 (ObjNum 104 snap 0 seq_num 104) dirty exists 2024-04-22T18:39:21.743 INFO:tasks.rados.rados.0.smithi089.stdout:807: left oid 28 (ObjNum 104 snap 0 seq_num 104) 2024-04-22T18:39:21.744 INFO:tasks.rados.rados.0.smithi089.stdout:811: finishing set_chunk to oid 43 2024-04-22T18:39:21.744 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 43 set_chunk returned error code -95 offset: 1376256 length: 933888 2024-04-22T18:39:21.744 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 0 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-04-22T18:39:21.744 INFO:tasks.rados.rados.0.smithi089.stdout:806: done (6 left) 2024-04-22T18:39:21.744 INFO:tasks.rados.rados.0.smithi089.stdout:807: done (5 left) 2024-04-22T18:39:21.744 INFO:tasks.rados.rados.0.smithi089.stdout:808: done (4 left) 2024-04-22T18:39:21.744 INFO:tasks.rados.rados.0.smithi089.stdout:810: done (3 left) 2024-04-22T18:39:21.744 INFO:tasks.rados.rados.0.smithi089.stdout:811: done (2 left) 2024-04-22T18:39:21.744 INFO:tasks.rados.rados.0.smithi089.stdout:814: read oid 47 snap -1 2024-04-22T18:39:21.744 INFO:tasks.rados.rados.0.smithi089.stdout:814: expect (ObjNum 68 snap 0 seq_num 68) 2024-04-22T18:39:21.744 INFO:tasks.rados.rados.0.smithi089.stdout:815: tier_evict oid 18 2024-04-22T18:39:21.744 INFO:tasks.rados.rados.0.smithi089.stdout:815: tier_evict oid 18 snap -1 2024-04-22T18:39:21.744 INFO:tasks.rados.rados.0.smithi089.stdout:813: got (22) Invalid argument 2024-04-22T18:39:21.744 INFO:tasks.rados.rados.0.smithi089.stdout:812: finishing write tid 1 to smithi08937929-2 2024-04-22T18:39:21.744 INFO:tasks.rados.rados.0.smithi089.stdout:812: finishing write tid 2 to smithi08937929-2 2024-04-22T18:39:21.745 INFO:tasks.rados.rados.0.smithi089.stdout:812: finishing write tid 3 to smithi08937929-2 2024-04-22T18:39:21.745 INFO:tasks.rados.rados.0.smithi089.stdout:812: finishing write tid 4 to smithi08937929-2 2024-04-22T18:39:21.745 INFO:tasks.rados.rados.0.smithi089.stdout:812: finishing write tid 5 to smithi08937929-2 2024-04-22T18:39:21.745 INFO:tasks.rados.rados.0.smithi089.stdout:812: oid 2 updating version 0 to 22 2024-04-22T18:39:21.745 INFO:tasks.rados.rados.0.smithi089.stdout:812: oid 2 version 22 is already newer than 21 2024-04-22T18:39:21.745 INFO:tasks.rados.rados.0.smithi089.stdout:812: oid 2 updating version 22 to 23 2024-04-22T18:39:21.745 INFO:tasks.rados.rados.0.smithi089.stdout:812: oid 2 updating version 23 to 24 2024-04-22T18:39:21.745 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 24 (ObjNum 106 snap 0 seq_num 106) dirty exists 2024-04-22T18:39:21.745 INFO:tasks.rados.rados.0.smithi089.stdout:812: left oid 2 (ObjNum 106 snap 0 seq_num 106) 2024-04-22T18:39:21.745 INFO:tasks.rados.rados.0.smithi089.stdout:812: done (3 left) 2024-04-22T18:39:21.745 INFO:tasks.rados.rados.0.smithi089.stdout:813: done (2 left) 2024-04-22T18:39:21.745 INFO:tasks.rados.rados.0.smithi089.stdout:816: tier_evict oid 31 2024-04-22T18:39:21.745 INFO:tasks.rados.rados.0.smithi089.stdout:816: tier_evict oid 31 snap -1 2024-04-22T18:39:21.745 INFO:tasks.rados.rados.0.smithi089.stdout:817: delete oid 35 current snap is 0 2024-04-22T18:39:21.745 INFO:tasks.rados.rados.0.smithi089.stdout:815: got (22) Invalid argument 2024-04-22T18:39:21.746 INFO:tasks.rados.rados.0.smithi089.stdout:816: got (22) Invalid argument 2024-04-22T18:39:21.746 INFO:tasks.rados.rados.0.smithi089.stdout:815: done (3 left) 2024-04-22T18:39:21.746 INFO:tasks.rados.rados.0.smithi089.stdout:816: done (2 left) 2024-04-22T18:39:21.746 INFO:tasks.rados.rados.0.smithi089.stdout:817: done (1 left) 2024-04-22T18:39:21.746 INFO:tasks.rados.rados.0.smithi089.stdout:818: read oid 26 snap -1 2024-04-22T18:39:21.746 INFO:tasks.rados.rados.0.smithi089.stdout:818: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-22T18:39:21.746 INFO:tasks.rados.rados.0.smithi089.stdout:819: set_chunk oid 24 target oid 2024-04-22T18:39:21.747 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:21.748+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:21.756 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1395 3.0 3.3de4582c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:39:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:21.750946+0000", 2024-04-22T18:39:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0070962669999999999, 2024-04-22T18:39:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0071160420000000004, 2024-04-22T18:39:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "no flag points reached", 2024-04-22T18:39:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1395 2024-04-22T18:39:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.750946+0000", 2024-04-22T18:39:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.750946+0000", 2024-04-22T18:39:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.750942+0000", 2024-04-22T18:39:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999957 2024-04-22T18:39:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.751142+0000", 2024-04-22T18:39:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019981 2024-04-22T18:39:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.751174+0000", 2024-04-22T18:39:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2472999999999999e-05 2024-04-22T18:39:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:39:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 1 2024-04-22T18:39:21.759 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:21.765 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:21.851 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:21.853+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:21.907 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:21.907 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:21.907 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:21.907 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:21.907 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:21.915 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:21.948 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:21.949+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:21.954 INFO:tasks.rados.rados.0.smithi089.stdout:819: set_chunk oid 24 offset: 514048 length: 321536 target oid a88abe412631e6e80123641a41a8c3ae4909e589ceddc30fb970e5ab45e57d2f offset: 0 snap -1 2024-04-22T18:39:21.954 INFO:tasks.rados.rados.0.smithi089.stdout:820: set_chunk oid 21 target oid 2024-04-22T18:39:22.049 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:22.050+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:22.057 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:22.057 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:22.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:22.057 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:22.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.058 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:7 4.f 4:f83a159b:::smithi08937929-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:22.058 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.111347+0000", 2024-04-22T18:39:22.058 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.945926767, 2024-04-22T18:39:22.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30271893700000002, 2024-04-22T18:39:22.058 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.058 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.058 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.058 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.058 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.058 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 7 2024-04-22T18:39:22.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.058 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111347+0000", 2024-04-22T18:39:22.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111347+0000", 2024-04-22T18:39:22.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111346+0000", 2024-04-22T18:39:22.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:22.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111350+0000", 2024-04-22T18:39:22.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.473e-06 2024-04-22T18:39:22.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111353+0000", 2024-04-22T18:39:22.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9100000000000001e-06 2024-04-22T18:39:22.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111356+0000", 2024-04-22T18:39:22.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2770000000000001e-06 2024-04-22T18:39:22.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.212813+0000", 2024-04-22T18:39:22.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10145739099999999 2024-04-22T18:39:22.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.212869+0000", 2024-04-22T18:39:22.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5630000000000001e-05 2024-04-22T18:39:22.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:22.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.212949+0000", 2024-04-22T18:39:22.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9679999999999996e-05 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.313864+0000", 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100915174 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.413991+0000", 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100126703 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414021+0000", 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0698999999999999e-05 2024-04-22T18:39:22.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414066+0000", 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4718000000000001e-05 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:8 4.f 4:f83a159b:::smithi08937929-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.945895568000001, 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30292182499999998, 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:22.063 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 8 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:22.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:22.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111380+0000", 2024-04-22T18:39:22.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2419999999999999e-06 2024-04-22T18:39:22.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111384+0000", 2024-04-22T18:39:22.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.715e-06 2024-04-22T18:39:22.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111387+0000", 2024-04-22T18:39:22.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.982e-06 2024-04-22T18:39:22.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414113+0000", 2024-04-22T18:39:22.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30272622700000001 2024-04-22T18:39:22.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414203+0000", 2024-04-22T18:39:22.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0316999999999996e-05 2024-04-22T18:39:22.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414300+0000", 2024-04-22T18:39:22.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6760000000000002e-05 2024-04-22T18:39:22.067 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.067 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.067 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.067 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:9 4.f 4:f2c48a0e:::smithi08937929-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:22.067 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:22.067 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.940499983999999, 2024-04-22T18:39:22.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49290714600000002, 2024-04-22T18:39:22.068 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.068 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.068 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.068 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.068 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.068 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 9 2024-04-22T18:39:22.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.068 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:22.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:22.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116773+0000", 2024-04-22T18:39:22.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:22.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117024+0000", 2024-04-22T18:39:22.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025191299999999999 2024-04-22T18:39:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117036+0000", 2024-04-22T18:39:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1595e-05 2024-04-22T18:39:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117046+0000", 2024-04-22T18:39:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0456e-05 2024-04-22T18:39:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414324+0000", 2024-04-22T18:39:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297277603 2024-04-22T18:39:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414404+0000", 2024-04-22T18:39:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9738000000000001e-05 2024-04-22T18:39:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414494+0000", 2024-04-22T18:39:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9733000000000003e-05 2024-04-22T18:39:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518194+0000", 2024-04-22T18:39:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10370061 2024-04-22T18:39:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609618+0000", 2024-04-22T18:39:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091423360999999995 2024-04-22T18:39:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609655+0000", 2024-04-22T18:39:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7580000000000003e-05 2024-04-22T18:39:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609681+0000", 2024-04-22T18:39:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5908e-05 2024-04-22T18:39:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:10 4.f 4:f2c48a0e:::smithi08937929-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.934647724000001, 2024-04-22T18:39:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48712681400000002, 2024-04-22T18:39:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 10 2024-04-22T18:39:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122625+0000", 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122828+0000", 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020359499999999999 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122840+0000", 2024-04-22T18:39:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1239e-05 2024-04-22T18:39:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122845+0000", 2024-04-22T18:39:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2719999999999997e-06 2024-04-22T18:39:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415105+0000", 2024-04-22T18:39:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29225972099999997 2024-04-22T18:39:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415169+0000", 2024-04-22T18:39:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4170000000000004e-05 2024-04-22T18:39:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415257+0000", 2024-04-22T18:39:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8331999999999994e-05 2024-04-22T18:39:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518199+0000", 2024-04-22T18:39:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10294174 2024-04-22T18:39:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609707+0000", 2024-04-22T18:39:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091508408999999999 2024-04-22T18:39:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609729+0000", 2024-04-22T18:39:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1917e-05 2024-04-22T18:39:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609753+0000", 2024-04-22T18:39:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3776000000000001e-05 2024-04-22T18:39:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:11 4.f 4:f2c48a0e:::smithi08937929-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.934424351000001, 2024-04-22T18:39:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49452214500000002, 2024-04-22T18:39:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 11 2024-04-22T18:39:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122849+0000", 2024-04-22T18:39:22.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:22.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122853+0000", 2024-04-22T18:39:22.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9029999999999997e-06 2024-04-22T18:39:22.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122856+0000", 2024-04-22T18:39:22.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.01e-06 2024-04-22T18:39:22.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122859+0000", 2024-04-22T18:39:22.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:22.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415755+0000", 2024-04-22T18:39:22.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29289606800000001 2024-04-22T18:39:22.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415808+0000", 2024-04-22T18:39:22.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3415999999999998e-05 2024-04-22T18:39:22.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:22.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415888+0000", 2024-04-22T18:39:22.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9061999999999996e-05 2024-04-22T18:39:22.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:22.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518203+0000", 2024-04-22T18:39:22.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102315533 2024-04-22T18:39:22.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:22.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617164+0000", 2024-04-22T18:39:22.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098960548999999995 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617241+0000", 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7292000000000005e-05 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617372+0000", 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013075399999999999 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:12 4.f 4:f2c48a0e:::smithi08937929-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:22.085 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:22.085 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.934412486999999, 2024-04-22T18:39:22.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49473910999999998, 2024-04-22T18:39:22.085 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.085 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:22.085 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.085 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.085 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.085 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 12 2024-04-22T18:39:22.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.085 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:22.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:22.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:22.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:22.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122863+0000", 2024-04-22T18:39:22.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.863e-06 2024-04-22T18:39:22.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122866+0000", 2024-04-22T18:39:22.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:39:22.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122869+0000", 2024-04-22T18:39:22.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5950000000000001e-06 2024-04-22T18:39:22.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416008+0000", 2024-04-22T18:39:22.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29313975199999998 2024-04-22T18:39:22.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:22.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416048+0000", 2024-04-22T18:39:22.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9888000000000002e-05 2024-04-22T18:39:22.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617447+0000", 2024-04-22T18:39:22.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20139881400000001 2024-04-22T18:39:22.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617537+0000", 2024-04-22T18:39:22.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9606999999999998e-05 2024-04-22T18:39:22.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617600+0000", 2024-04-22T18:39:22.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3884e-05 2024-04-22T18:39:22.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.089 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.090 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:33 4.f 4:f0ed3d49:::smithi08937929-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:22.090 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:22.090 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.870472286000002, 2024-04-22T18:39:22.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62359692099999997, 2024-04-22T18:39:22.090 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.090 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.090 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 33 2024-04-22T18:39:22.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.090 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:22.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:22.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186799+0000", 2024-04-22T18:39:22.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:22.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187103+0000", 2024-04-22T18:39:22.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030354900000000002 2024-04-22T18:39:22.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187117+0000", 2024-04-22T18:39:22.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4578e-05 2024-04-22T18:39:22.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187129+0000", 2024-04-22T18:39:22.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1421e-05 2024-04-22T18:39:22.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516134+0000", 2024-04-22T18:39:22.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32900553900000001 2024-04-22T18:39:22.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516215+0000", 2024-04-22T18:39:22.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0537999999999994e-05 2024-04-22T18:39:22.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:22.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516302+0000", 2024-04-22T18:39:22.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6601000000000007e-05 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518207+0000", 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019058370000000001 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810302+0000", 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29209509700000003 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810348+0000", 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5992999999999998e-05 2024-04-22T18:39:22.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810398+0000", 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0046000000000002e-05 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:34 4.f 4:f0ed3d49:::smithi08937929-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.865870273999999, 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.61954655800000002, 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.095 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.096 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.096 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.096 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34 2024-04-22T18:39:22.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.096 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:22.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:22.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191401+0000", 2024-04-22T18:39:22.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:22.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191503+0000", 2024-04-22T18:39:22.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010190400000000001 2024-04-22T18:39:22.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191517+0000", 2024-04-22T18:39:22.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3267e-05 2024-04-22T18:39:22.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191523+0000", 2024-04-22T18:39:22.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-04-22T18:39:22.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517062+0000", 2024-04-22T18:39:22.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32553894300000003 2024-04-22T18:39:22.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517141+0000", 2024-04-22T18:39:22.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9667999999999997e-05 2024-04-22T18:39:22.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:22.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517267+0000", 2024-04-22T18:39:22.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000125494 2024-04-22T18:39:22.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:22.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518213+0000", 2024-04-22T18:39:22.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000946275 2024-04-22T18:39:22.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:22.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810901+0000", 2024-04-22T18:39:22.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29268821699999997 2024-04-22T18:39:22.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810929+0000", 2024-04-22T18:39:22.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7645999999999999e-05 2024-04-22T18:39:22.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810950+0000", 2024-04-22T18:39:22.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1333999999999999e-05 2024-04-22T18:39:22.100 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.100 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.101 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.101 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:22.101 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:22.101 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.865747305999999, 2024-04-22T18:39:22.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:39:22.101 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.101 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.101 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.101 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.101 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.101 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:39:22.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.102 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:22.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:22.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:39:22.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:22.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:39:22.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:39:22.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:39:22.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:39:22.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:22.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:39:22.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:39:22.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:22.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:39:22.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:39:22.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:22.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:39:22.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:39:22.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:39:22.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:39:22.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:39:22.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:39:22.106 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.106 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.106 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.106 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:22.106 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:22.107 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.865736231, 2024-04-22T18:39:22.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:39:22.107 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.107 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:22.107 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.107 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.107 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.107 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:39:22.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.107 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:22.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:22.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:39:22.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:22.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:39:22.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:39:22.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:39:22.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:39:22.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:39:22.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:39:22.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:39:22.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:39:22.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:22.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:39:22.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:39:22.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:39:22.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:39:22.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:39:22.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:39:22.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:39:22.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:39:22.111 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.111 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.111 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.112 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:38 4.2 4:4fb26dd8:::smithi08937929-10:head [write 1893632~257631] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:22.112 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:22.112 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.854013627, 2024-04-22T18:39:22.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33863800199999999, 2024-04-22T18:39:22.112 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.112 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.112 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.112 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.112 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.112 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 38 2024-04-22T18:39:22.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.112 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:22.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:22.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203258+0000", 2024-04-22T18:39:22.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:22.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203396+0000", 2024-04-22T18:39:22.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013710799999999999 2024-04-22T18:39:22.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203406+0000", 2024-04-22T18:39:22.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0295000000000001e-05 2024-04-22T18:39:22.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203411+0000", 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5079999999999996e-06 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203450+0000", 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8420999999999997e-05 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203455+0000", 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237071+0000", 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.033615891000000002 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237129+0000", 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8134000000000003e-05 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237221+0000", 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2053999999999997e-05 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.341419+0000", 2024-04-22T18:39:22.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104197568 2024-04-22T18:39:22.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:22.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541792+0000", 2024-04-22T18:39:22.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.200373312 2024-04-22T18:39:22.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541838+0000", 2024-04-22T18:39:22.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6004000000000002e-05 2024-04-22T18:39:22.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541898+0000", 2024-04-22T18:39:22.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0297000000000002e-05 2024-04-22T18:39:22.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.118 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.118 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:39 4.2 4:4fb26dd8:::smithi08937929-10:head [setxattr _header (56) in=7b,truncate 2151263] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:22.118 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:22.118 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.853856632999999, 2024-04-22T18:39:22.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33862203000000002, 2024-04-22T18:39:22.118 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.118 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.119 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.119 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.119 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.119 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 39 2024-04-22T18:39:22.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.119 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:22.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:22.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:22.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:22.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203423+0000", 2024-04-22T18:39:22.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0589999999999999e-06 2024-04-22T18:39:22.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203429+0000", 2024-04-22T18:39:22.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2559999999999999e-06 2024-04-22T18:39:22.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203434+0000", 2024-04-22T18:39:22.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8049999999999997e-06 2024-04-22T18:39:22.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203475+0000", 2024-04-22T18:39:22.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1205000000000003e-05 2024-04-22T18:39:22.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:22.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203478+0000", 2024-04-22T18:39:22.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7140000000000002e-06 2024-04-22T18:39:22.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239180+0000", 2024-04-22T18:39:22.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035702195999999999 2024-04-22T18:39:22.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239225+0000", 2024-04-22T18:39:22.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5722e-05 2024-04-22T18:39:22.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:22.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239291+0000", 2024-04-22T18:39:22.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5870000000000005e-05 2024-04-22T18:39:22.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:22.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342114+0000", 2024-04-22T18:39:22.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102822301 2024-04-22T18:39:22.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:22.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541946+0000", 2024-04-22T18:39:22.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.199832544 2024-04-22T18:39:22.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541988+0000", 2024-04-22T18:39:22.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1499999999999999e-05 2024-04-22T18:39:22.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542039+0000", 2024-04-22T18:39:22.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1576000000000002e-05 2024-04-22T18:39:22.125 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.125 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.125 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.125 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:40 4.2 4:4fb26dd8:::smithi08937929-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:22.125 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:22.125 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.853819373, 2024-04-22T18:39:22.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33924657200000002, 2024-04-22T18:39:22.125 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.125 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:22.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.126 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 40 2024-04-22T18:39:22.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.126 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:22.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:22.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:22.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:22.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203457+0000", 2024-04-22T18:39:22.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.129e-06 2024-04-22T18:39:22.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203463+0000", 2024-04-22T18:39:22.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7440000000000004e-06 2024-04-22T18:39:22.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203468+0000", 2024-04-22T18:39:22.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2619999999999998e-06 2024-04-22T18:39:22.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203491+0000", 2024-04-22T18:39:22.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3260000000000001e-05 2024-04-22T18:39:22.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:22.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203493+0000", 2024-04-22T18:39:22.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9599999999999999e-06 2024-04-22T18:39:22.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239420+0000", 2024-04-22T18:39:22.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035927212 2024-04-22T18:39:22.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:22.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239462+0000", 2024-04-22T18:39:22.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.159e-05 2024-04-22T18:39:22.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542547+0000", 2024-04-22T18:39:22.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30308492599999998 2024-04-22T18:39:22.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542611+0000", 2024-04-22T18:39:22.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4622000000000006e-05 2024-04-22T18:39:22.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542701+0000", 2024-04-22T18:39:22.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9579000000000004e-05 2024-04-22T18:39:22.130 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.130 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.130 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:41 4.2 4:409e6456:::smithi08937929-11:head [write 568070~468843,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.848103102, 2024-04-22T18:39:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33297193600000002, 2024-04-22T18:39:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 41 2024-04-22T18:39:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:22.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:22.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209168+0000", 2024-04-22T18:39:22.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:22.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209393+0000", 2024-04-22T18:39:22.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022485199999999999 2024-04-22T18:39:22.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209404+0000", 2024-04-22T18:39:22.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1209999999999999e-05 2024-04-22T18:39:22.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209411+0000", 2024-04-22T18:39:22.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8580000000000002e-06 2024-04-22T18:39:22.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209457+0000", 2024-04-22T18:39:22.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6351999999999999e-05 2024-04-22T18:39:22.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:22.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209463+0000", 2024-04-22T18:39:22.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.322e-06 2024-04-22T18:39:22.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339572+0000", 2024-04-22T18:39:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13010901999999999 2024-04-22T18:39:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339673+0000", 2024-04-22T18:39:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101332 2024-04-22T18:39:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339765+0000", 2024-04-22T18:39:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1780999999999994e-05 2024-04-22T18:39:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:22.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342117+0000", 2024-04-22T18:39:22.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002351709 2024-04-22T18:39:22.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:22.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542074+0000", 2024-04-22T18:39:22.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995749800000001 2024-04-22T18:39:22.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542112+0000", 2024-04-22T18:39:22.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7891e-05 2024-04-22T18:39:22.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542143+0000", 2024-04-22T18:39:22.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0756000000000002e-05 2024-04-22T18:39:22.137 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.137 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.137 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.137 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:42 4.2 4:409e6456:::smithi08937929-11:head [write 1552114~615956] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:22.137 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:22.137 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.841764792999999, 2024-04-22T18:39:22.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326735257, 2024-04-22T18:39:22.137 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.138 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.138 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.138 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.138 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.138 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 42 2024-04-22T18:39:22.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.138 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:22.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:22.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215508+0000", 2024-04-22T18:39:22.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:22.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215766+0000", 2024-04-22T18:39:22.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025766999999999999 2024-04-22T18:39:22.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215776+0000", 2024-04-22T18:39:22.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0772e-05 2024-04-22T18:39:22.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215785+0000", 2024-04-22T18:39:22.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6880000000000005e-06 2024-04-22T18:39:22.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215818+0000", 2024-04-22T18:39:22.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2960999999999998e-05 2024-04-22T18:39:22.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:22.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215821+0000", 2024-04-22T18:39:22.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8569999999999999e-06 2024-04-22T18:39:22.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340332+0000", 2024-04-22T18:39:22.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.124511409 2024-04-22T18:39:22.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340416+0000", 2024-04-22T18:39:22.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3793000000000002e-05 2024-04-22T18:39:22.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:22.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340543+0000", 2024-04-22T18:39:22.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012720299999999999 2024-04-22T18:39:22.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:22.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342361+0000", 2024-04-22T18:39:22.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018174370000000001 2024-04-22T18:39:22.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:22.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542176+0000", 2024-04-22T18:39:22.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19981518400000001 2024-04-22T18:39:22.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542211+0000", 2024-04-22T18:39:22.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5128e-05 2024-04-22T18:39:22.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542244+0000", 2024-04-22T18:39:22.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3256000000000001e-05 2024-04-22T18:39:22.143 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.143 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.143 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.144 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:43 4.2 4:409e6456:::smithi08937929-11:head [setxattr _header (56) in=7b,truncate 2168070] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:22.144 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:22.144 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.841480019999999, 2024-04-22T18:39:22.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32656540499999998, 2024-04-22T18:39:22.144 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.144 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.144 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.144 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.144 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.144 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 43 2024-04-22T18:39:22.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.144 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:22.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:22.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215793+0000", 2024-04-22T18:39:22.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:22.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215797+0000", 2024-04-22T18:39:22.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4190000000000004e-06 2024-04-22T18:39:22.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215804+0000", 2024-04-22T18:39:22.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:39:22.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215809+0000", 2024-04-22T18:39:22.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3689999999999998e-06 2024-04-22T18:39:22.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215840+0000", 2024-04-22T18:39:22.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-05 2024-04-22T18:39:22.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:22.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215843+0000", 2024-04-22T18:39:22.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8849999999999999e-06 2024-04-22T18:39:22.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342483+0000", 2024-04-22T18:39:22.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126640172 2024-04-22T18:39:22.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342529+0000", 2024-04-22T18:39:22.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5668999999999999e-05 2024-04-22T18:39:22.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:22.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342597+0000", 2024-04-22T18:39:22.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7538000000000003e-05 2024-04-22T18:39:22.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:22.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.347102+0000", 2024-04-22T18:39:22.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045058040000000004 2024-04-22T18:39:22.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:22.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542278+0000", 2024-04-22T18:39:22.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.195175879 2024-04-22T18:39:22.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542314+0000", 2024-04-22T18:39:22.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6134999999999999e-05 2024-04-22T18:39:22.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542359+0000", 2024-04-22T18:39:22.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4848999999999997e-05 2024-04-22T18:39:22.150 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.150 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.150 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.150 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:44 4.2 4:409e6456:::smithi08937929-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:22.150 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:22.150 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.841460011999999, 2024-04-22T18:39:22.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326706418, 2024-04-22T18:39:22.150 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.150 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:22.151 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.151 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.151 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.151 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 44 2024-04-22T18:39:22.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.151 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:22.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:22.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215813+0000", 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215816+0000", 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9799999999999998e-06 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215822+0000", 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5570000000000003e-06 2024-04-22T18:39:22.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215827+0000", 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2739999999999998e-06 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215855+0000", 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7682999999999999e-05 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215857+0000", 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9250000000000002e-06 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342717+0000", 2024-04-22T18:39:22.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126860696 2024-04-22T18:39:22.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:22.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342758+0000", 2024-04-22T18:39:22.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0295e-05 2024-04-22T18:39:22.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542408+0000", 2024-04-22T18:39:22.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19964990599999999 2024-04-22T18:39:22.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542457+0000", 2024-04-22T18:39:22.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9784999999999999e-05 2024-04-22T18:39:22.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542520+0000", 2024-04-22T18:39:22.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2960999999999996e-05 2024-04-22T18:39:22.155 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.155 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.264366737, 2024-04-22T18:39:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:39:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:39:22.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:39:22.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:39:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:39:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:39:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:39:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:39:22.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:39:22.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:39:22.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:39:22.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:39:22.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:39:22.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:22.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:39:22.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:39:22.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:22.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:39:22.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:39:22.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:22.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:39:22.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:39:22.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:39:22.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:39:22.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:39:22.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:39:22.161 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.161 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.161 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.161 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:709 4.2 4:4fb26dd8:::smithi08937929-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:22.161 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.528719+0000", 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.528555098999998, 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30036358099999999, 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 709 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528719+0000", 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528719+0000", 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528718+0000", 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528723+0000", 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8550000000000001e-06 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528729+0000", 2024-04-22T18:39:22.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8680000000000004e-06 2024-04-22T18:39:22.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528735+0000", 2024-04-22T18:39:22.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8309999999999997e-06 2024-04-22T18:39:22.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528766+0000", 2024-04-22T18:39:22.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1970000000000001e-05 2024-04-22T18:39:22.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528823+0000", 2024-04-22T18:39:22.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6440000000000002e-05 2024-04-22T18:39:22.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:22.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728211+0000", 2024-04-22T18:39:22.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.199387975 2024-04-22T18:39:22.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:22.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728615+0000", 2024-04-22T18:39:22.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000404161 2024-04-22T18:39:22.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:22.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.829015+0000", 2024-04-22T18:39:22.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100400197 2024-04-22T18:39:22.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.829053+0000", 2024-04-22T18:39:22.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7373000000000003e-05 2024-04-22T18:39:22.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.829082+0000", 2024-04-22T18:39:22.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9717e-05 2024-04-22T18:39:22.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.167 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.167 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1262 4.7 4:e117a562:::smithi08937929-33:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:22.167 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:22.167 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.6042235609999995, 2024-04-22T18:39:22.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40488836700000003, 2024-04-22T18:39:22.167 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.167 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.167 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:22.167 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:22.168 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:22.168 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1262 2024-04-22T18:39:22.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.168 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:22.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:22.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453048+0000", 2024-04-22T18:39:22.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:22.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453058+0000", 2024-04-22T18:39:22.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:39:22.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453069+0000", 2024-04-22T18:39:22.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1117999999999999e-05 2024-04-22T18:39:22.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453075+0000", 2024-04-22T18:39:22.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0909999999999998e-06 2024-04-22T18:39:22.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453115+0000", 2024-04-22T18:39:22.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0119e-05 2024-04-22T18:39:22.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:22.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453119+0000", 2024-04-22T18:39:22.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3610000000000003e-06 2024-04-22T18:39:22.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454814+0000", 2024-04-22T18:39:22.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00169436 2024-04-22T18:39:22.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454861+0000", 2024-04-22T18:39:22.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7184000000000001e-05 2024-04-22T18:39:22.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:22.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756925+0000", 2024-04-22T18:39:22.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30206361799999998 2024-04-22T18:39:22.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:22.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757577+0000", 2024-04-22T18:39:22.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065199599999999996 2024-04-22T18:39:22.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:22.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857846+0000", 2024-04-22T18:39:22.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100269369 2024-04-22T18:39:22.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857896+0000", 2024-04-22T18:39:22.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9607000000000001e-05 2024-04-22T18:39:22.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857939+0000", 2024-04-22T18:39:22.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3149999999999999e-05 2024-04-22T18:39:22.173 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.173 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.173 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.173 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.173 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.173 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:22.175 INFO:tasks.rados.rados.0.smithi089.stdout:820: set_chunk oid 21 offset: 1255936 length: 855552 target oid 9d81e38fa45b2f87319d1f6ca039dfc50d4304ceb216fced89f733de95073c57 offset: 0 snap -1 2024-04-22T18:39:22.175 INFO:tasks.rados.rados.0.smithi089.stdout:821: read oid 7 snap -1 2024-04-22T18:39:22.175 INFO:tasks.rados.rados.0.smithi089.stdout:821: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-22T18:39:22.175 INFO:tasks.rados.rados.0.smithi089.stdout:822: read oid 32 snap -1 2024-04-22T18:39:22.175 INFO:tasks.rados.rados.0.smithi089.stdout:822: expect (ObjNum 92 snap 0 seq_num 92) 2024-04-22T18:39:22.175 INFO:tasks.rados.rados.0.smithi089.stdout:823: write (excl) oid 45 current snap is 0 2024-04-22T18:39:22.175 INFO:tasks.rados.rados.0.smithi089.stdout:823: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2024-04-22T18:39:22.175 INFO:tasks.rados.rados.0.smithi089.stdout:823: writing smithi08937929-45 from 615156 to 1170990 tid 1 2024-04-22T18:39:22.176 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:22.177 INFO:tasks.rados.rados.0.smithi089.stdout:823: writing smithi08937929-45 from 1603467 to 2019828 tid 2 2024-04-22T18:39:22.177 INFO:tasks.rados.rados.0.smithi089.stdout:823: writing smithi08937929-45 from 2816389 to 3509884 tid 3 2024-04-22T18:39:22.177 INFO:tasks.rados.rados.0.smithi089.stdout:824: read oid 16 snap -1 2024-04-22T18:39:22.178 INFO:tasks.rados.rados.0.smithi089.stdout:824: expect (ObjNum 71 snap 0 seq_num 71) 2024-04-22T18:39:22.178 INFO:tasks.rados.rados.0.smithi089.stdout:825: write (excl) oid 22 current snap is 0 2024-04-22T18:39:22.178 INFO:tasks.rados.rados.0.smithi089.stdout:818: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-22T18:39:22.178 INFO:tasks.rados.rados.0.smithi089.stdout:814: expect (ObjNum 68 snap 0 seq_num 68) 2024-04-22T18:39:22.178 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:22.176+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:22.235 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:39:22.305 INFO:tasks.rados.rados.0.smithi089.stdout:820: finishing set_chunk to oid 21 2024-04-22T18:39:22.305 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 21 set_chunk returned error code -95 offset: 1255936 length: 855552 2024-04-22T18:39:22.305 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-22T18:39:22.305 INFO:tasks.rados.rados.0.smithi089.stdout:821: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-22T18:39:22.305 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:22.306+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:22.324 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:22.324 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:22.324 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:22.324 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:22.333 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:22.376 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:22.377+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:22.378 INFO:tasks.rados.rados.0.smithi089.stdout:819: finishing set_chunk to oid 24 2024-04-22T18:39:22.379 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 12 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-22T18:39:22.379 INFO:tasks.rados.rados.0.smithi089.stdout:822: expect (ObjNum 92 snap 0 seq_num 92) 2024-04-22T18:39:22.476 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:22.477 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:22.477 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:22.477 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:22.477 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:22.477 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:22.478+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:22.485 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:22.578 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:22.579+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:22.619 INFO:tasks.rados.rados.0.smithi089.stdout:824: expect (ObjNum 71 snap 0 seq_num 71) 2024-04-22T18:39:22.629 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:22.629 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:27 4.d e27/20 4:bee1184f:::smithi08937929-7:head v 27'3, mlcod=0'0)", 2024-04-22T18:39:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.264848264000001, 2024-04-22T18:39:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101135974, 2024-04-22T18:39:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365186+0000", 2024-04-22T18:39:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365194+0000", 2024-04-22T18:39:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3289999999999994e-06 2024-04-22T18:39:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365200+0000", 2024-04-22T18:39:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6590000000000004e-06 2024-04-22T18:39:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465352+0000", 2024-04-22T18:39:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10015208 2024-04-22T18:39:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465511+0000", 2024-04-22T18:39:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015880000000000001 2024-04-22T18:39:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465570+0000", 2024-04-22T18:39:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8952000000000001e-05 2024-04-22T18:39:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466292+0000", 2024-04-22T18:39:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072212100000000005 2024-04-22T18:39:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466323+0000", 2024-04-22T18:39:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0899999999999999e-05 2024-04-22T18:39:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:47 3.0 e30/19 3:30f16451:::smithi08937929-50 low_tier:head v 30'20, mlcod=29'18)", 2024-04-22T18:39:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.031932247, 2024-04-22T18:39:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100803438, 2024-04-22T18:39:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598108+0000", 2024-04-22T18:39:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9119999999999997e-06 2024-04-22T18:39:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598118+0000", 2024-04-22T18:39:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0138e-05 2024-04-22T18:39:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598125+0000", 2024-04-22T18:39:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3010000000000001e-06 2024-04-22T18:39:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698224+0000", 2024-04-22T18:39:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099221 2024-04-22T18:39:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698239+0000", 2024-04-22T18:39:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5053e-05 2024-04-22T18:39:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698875+0000", 2024-04-22T18:39:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063560699999999995 2024-04-22T18:39:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698906+0000", 2024-04-22T18:39:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1547999999999997e-05 2024-04-22T18:39:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.5.13:5 3.0 e30/19 3:1ab765eb:::smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=30'21)", 2024-04-22T18:39:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.427614124000002, 2024-04-22T18:39:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101014042, 2024-04-22T18:39:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202428+0000", 2024-04-22T18:39:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5889999999999997e-06 2024-04-22T18:39:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202446+0000", 2024-04-22T18:39:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8176999999999999e-05 2024-04-22T18:39:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202456+0000", 2024-04-22T18:39:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-04-22T18:39:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302599+0000", 2024-04-22T18:39:22.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100142759 2024-04-22T18:39:22.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302628+0000", 2024-04-22T18:39:22.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9507999999999999e-05 2024-04-22T18:39:22.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303350+0000", 2024-04-22T18:39:22.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072151599999999995 2024-04-22T18:39:22.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303435+0000", 2024-04-22T18:39:22.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5486000000000001e-05 2024-04-22T18:39:22.643 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.643 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.643 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.643 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:53 3.0 e31/19 3:25a201b0:::smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 31'24, mlcod=30'22)", 2024-04-22T18:39:22.643 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:22.643 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.22697179, 2024-04-22T18:39:22.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100751671, 2024-04-22T18:39:22.643 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.644 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.644 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:22.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:22.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403062+0000", 2024-04-22T18:39:22.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:22.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403070+0000", 2024-04-22T18:39:22.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8499999999999994e-06 2024-04-22T18:39:22.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403079+0000", 2024-04-22T18:39:22.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4620000000000002e-06 2024-04-22T18:39:22.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403087+0000", 2024-04-22T18:39:22.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4789999999999999e-06 2024-04-22T18:39:22.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503186+0000", 2024-04-22T18:39:22.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099271 2024-04-22T18:39:22.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503207+0000", 2024-04-22T18:39:22.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0973999999999999e-05 2024-04-22T18:39:22.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503773+0000", 2024-04-22T18:39:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056557600000000003 2024-04-22T18:39:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503815+0000", 2024-04-22T18:39:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2441999999999998e-05 2024-04-22T18:39:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.647 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:73 3.0 e34/19 3:061a7a63:::392f44ec26aa22e85be5f08273a7e45656f3bf4a4854038391a88e880ec01077:head v 34'31, mlcod=31'29)", 2024-04-22T18:39:22.648 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:22.648 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.172108342000001, 2024-04-22T18:39:22.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103503311, 2024-04-22T18:39:22.648 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.648 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.648 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:22.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:22.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:22.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:22.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458193+0000", 2024-04-22T18:39:22.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026628900000000001 2024-04-22T18:39:22.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458208+0000", 2024-04-22T18:39:22.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4780000000000001e-05 2024-04-22T18:39:22.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458217+0000", 2024-04-22T18:39:22.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4909999999999994e-06 2024-04-22T18:39:22.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459162+0000", 2024-04-22T18:39:22.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094440500000000005 2024-04-22T18:39:22.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459185+0000", 2024-04-22T18:39:22.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3685000000000001e-05 2024-04-22T18:39:22.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561394+0000", 2024-04-22T18:39:22.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10220874100000001 2024-04-22T18:39:22.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561430+0000", 2024-04-22T18:39:22.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6304999999999999e-05 2024-04-22T18:39:22.652 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.652 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.652 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.652 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:75 3.0 e34/19 3:18418e0f:::b7c35f165c374d5e075dde9ac2607e47ace3b0a1e2343ec6efb171380f725eea:head v 34'32, mlcod=31'29)", 2024-04-22T18:39:22.652 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:22.652 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.171095069, 2024-04-22T18:39:22.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104798078, 2024-04-22T18:39:22.652 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.652 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.652 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:22.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:22.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458939+0000", 2024-04-22T18:39:22.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:22.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459092+0000", 2024-04-22T18:39:22.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153838 2024-04-22T18:39:22.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459107+0000", 2024-04-22T18:39:22.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4703e-05 2024-04-22T18:39:22.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459115+0000", 2024-04-22T18:39:22.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-04-22T18:39:22.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560093+0000", 2024-04-22T18:39:22.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10097840499999999 2024-04-22T18:39:22.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560109+0000", 2024-04-22T18:39:22.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5631e-05 2024-04-22T18:39:22.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563706+0000", 2024-04-22T18:39:22.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035972059999999999 2024-04-22T18:39:22.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563738+0000", 2024-04-22T18:39:22.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2184000000000002e-05 2024-04-22T18:39:22.656 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.656 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.656 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.656 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:77 3.0 e34/19 3:28387c9f:::b6d7840f02d561da04e398fbf94d91b85e9066b426e48baf3b11dd4a4f6096c0:head v 34'33, mlcod=31'29)", 2024-04-22T18:39:22.656 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:22.656 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.170144989000001, 2024-04-22T18:39:22.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104186079, 2024-04-22T18:39:22.657 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.657 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.657 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:22.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:22.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459889+0000", 2024-04-22T18:39:22.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:22.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459963+0000", 2024-04-22T18:39:22.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3732999999999994e-05 2024-04-22T18:39:22.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459973+0000", 2024-04-22T18:39:22.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0285999999999999e-05 2024-04-22T18:39:22.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459987+0000", 2024-04-22T18:39:22.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3675999999999999e-05 2024-04-22T18:39:22.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560515+0000", 2024-04-22T18:39:22.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10052781299999999 2024-04-22T18:39:22.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560538+0000", 2024-04-22T18:39:22.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3861000000000001e-05 2024-04-22T18:39:22.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564044+0000", 2024-04-22T18:39:22.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035052009999999999 2024-04-22T18:39:22.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564076+0000", 2024-04-22T18:39:22.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2737000000000002e-05 2024-04-22T18:39:22.660 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.660 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.660 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.661 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:78 3.0 e34/19 3:3a5582ac:::bdd92f7deaa2d50e204891f64f9043469ba687d78b218ed518f2483ba3486a99:head v 34'34, mlcod=31'29)", 2024-04-22T18:39:22.661 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:22.661 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.169478870999999, 2024-04-22T18:39:22.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103547263, 2024-04-22T18:39:22.661 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.661 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.661 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:22.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:22.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460555+0000", 2024-04-22T18:39:22.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:22.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460861+0000", 2024-04-22T18:39:22.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000305958 2024-04-22T18:39:22.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460874+0000", 2024-04-22T18:39:22.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2639e-05 2024-04-22T18:39:22.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460881+0000", 2024-04-22T18:39:22.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2570000000000001e-06 2024-04-22T18:39:22.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560990+0000", 2024-04-22T18:39:22.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010883199999999 2024-04-22T18:39:22.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561011+0000", 2024-04-22T18:39:22.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1319e-05 2024-04-22T18:39:22.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564079+0000", 2024-04-22T18:39:22.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00306815 2024-04-22T18:39:22.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564104+0000", 2024-04-22T18:39:22.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4172000000000002e-05 2024-04-22T18:39:22.665 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.665 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.665 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.665 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:39:22.665 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:22.665 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.168863577, 2024-04-22T18:39:22.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:39:22.665 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.665 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.665 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:22.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:22.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:39:22.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:22.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:39:22.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:39:22.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:39:22.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:39:22.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:39:22.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:39:22.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:39:22.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:39:22.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:39:22.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:39:22.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:39:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:39:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:39:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:39:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.669 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:39:22.670 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:22.670 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.292189755000001, 2024-04-22T18:39:22.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:39:22.670 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.670 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.670 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:22.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:22.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:39:22.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:22.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:39:22.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:39:22.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:39:22.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:39:22.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:39:22.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:39:22.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:39:22.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:39:22.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:39:22.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:39:22.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:39:22.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:39:22.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:39:22.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:39:22.673 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.673 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.673 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.674 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1112 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'6, mlcod=29'4)", 2024-04-22T18:39:22.674 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:12.585713+0000", 2024-04-22T18:39:22.674 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.044321847999999, 2024-04-22T18:39:22.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046198259999999996, 2024-04-22T18:39:22.674 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.674 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.674 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.585713+0000", 2024-04-22T18:39:22.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.585713+0000", 2024-04-22T18:39:22.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.585712+0000", 2024-04-22T18:39:22.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:22.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586020+0000", 2024-04-22T18:39:22.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030764900000000001 2024-04-22T18:39:22.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586031+0000", 2024-04-22T18:39:22.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0883000000000001e-05 2024-04-22T18:39:22.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586037+0000", 2024-04-22T18:39:22.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1260000000000003e-06 2024-04-22T18:39:22.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586076+0000", 2024-04-22T18:39:22.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8841999999999998e-05 2024-04-22T18:39:22.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586098+0000", 2024-04-22T18:39:22.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2279000000000001e-05 2024-04-22T18:39:22.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.590302+0000", 2024-04-22T18:39:22.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042039419999999996 2024-04-22T18:39:22.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.590333+0000", 2024-04-22T18:39:22.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1170000000000001e-05 2024-04-22T18:39:22.678 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.678 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.678 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.678 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1131 3.0 e41/19 3:01bdda42:::4a159391123755d93670b0d142ca36f3ee9a35236f6c4396c2aa92521a42a06b:head v 41'50, mlcod=41'48)", 2024-04-22T18:39:22.678 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:22.678 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4524429100000003, 2024-04-22T18:39:22.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101639456, 2024-04-22T18:39:22.678 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.679 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.679 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:22.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:22.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177590+0000", 2024-04-22T18:39:22.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:22.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177880+0000", 2024-04-22T18:39:22.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029001 2024-04-22T18:39:22.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177893+0000", 2024-04-22T18:39:22.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-04-22T18:39:22.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177899+0000", 2024-04-22T18:39:22.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1619999999999996e-06 2024-04-22T18:39:22.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.278054+0000", 2024-04-22T18:39:22.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100154929 2024-04-22T18:39:22.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.278076+0000", 2024-04-22T18:39:22.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1308e-05 2024-04-22T18:39:22.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.279187+0000", 2024-04-22T18:39:22.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011109169999999999 2024-04-22T18:39:22.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.279232+0000", 2024-04-22T18:39:22.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5188000000000002e-05 2024-04-22T18:39:22.682 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.682 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.682 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.682 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1159 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'10, mlcod=41'7)", 2024-04-22T18:39:22.683 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.452702+0000", 2024-04-22T18:39:22.683 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.1773333600000004, 2024-04-22T18:39:22.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050095280000000001, 2024-04-22T18:39:22.683 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.683 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.683 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452702+0000", 2024-04-22T18:39:22.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452702+0000", 2024-04-22T18:39:22.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452701+0000", 2024-04-22T18:39:22.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:22.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452976+0000", 2024-04-22T18:39:22.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027566 2024-04-22T18:39:22.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452989+0000", 2024-04-22T18:39:22.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2615e-05 2024-04-22T18:39:22.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452998+0000", 2024-04-22T18:39:22.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7399999999999993e-06 2024-04-22T18:39:22.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.453040+0000", 2024-04-22T18:39:22.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2395000000000003e-05 2024-04-22T18:39:22.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.453060+0000", 2024-04-22T18:39:22.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0332000000000001e-05 2024-04-22T18:39:22.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.457682+0000", 2024-04-22T18:39:22.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046215010000000001 2024-04-22T18:39:22.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.457711+0000", 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9490999999999999e-05 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1160 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'11, mlcod=41'8)", 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.555212+0000", 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.0748230270000008, 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004833586, 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555212+0000", 2024-04-22T18:39:22.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555212+0000", 2024-04-22T18:39:22.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555209+0000", 2024-04-22T18:39:22.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:22.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555369+0000", 2024-04-22T18:39:22.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015995000000000001 2024-04-22T18:39:22.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555383+0000", 2024-04-22T18:39:22.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3558e-05 2024-04-22T18:39:22.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555389+0000", 2024-04-22T18:39:22.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2759999999999997e-06 2024-04-22T18:39:22.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556393+0000", 2024-04-22T18:39:22.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001003629 2024-04-22T18:39:22.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556418+0000", 2024-04-22T18:39:22.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4915000000000001e-05 2024-04-22T18:39:22.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.560010+0000", 2024-04-22T18:39:22.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003591782 2024-04-22T18:39:22.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.560046+0000", 2024-04-22T18:39:22.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6285000000000002e-05 2024-04-22T18:39:22.691 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.691 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.691 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.692 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1161 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'12, mlcod=41'9)", 2024-04-22T18:39:22.692 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.556330+0000", 2024-04-22T18:39:22.692 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.0737056089999992, 2024-04-22T18:39:22.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099789016999999994, 2024-04-22T18:39:22.692 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.692 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.692 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556330+0000", 2024-04-22T18:39:22.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556330+0000", 2024-04-22T18:39:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556328+0000", 2024-04-22T18:39:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556335+0000", 2024-04-22T18:39:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9770000000000003e-06 2024-04-22T18:39:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556346+0000", 2024-04-22T18:39:22.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0118999999999999e-05 2024-04-22T18:39:22.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556353+0000", 2024-04-22T18:39:22.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3730000000000004e-06 2024-04-22T18:39:22.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.655508+0000", 2024-04-22T18:39:22.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099155219000000003 2024-04-22T18:39:22.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.655536+0000", 2024-04-22T18:39:22.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-05 2024-04-22T18:39:22.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.656078+0000", 2024-04-22T18:39:22.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054138999999999999 2024-04-22T18:39:22.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.656119+0000", 2024-04-22T18:39:22.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1026000000000003e-05 2024-04-22T18:39:22.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.695 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.696 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:125 3.0 e43/19 3:376bc4bb:::8f967acf6e5865f87cd7cada681f96ff59d5fc3f243a710c00c052aacb952df3:head v 43'60, mlcod=42'57)", 2024-04-22T18:39:22.696 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:22.696 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.782469528, 2024-04-22T18:39:22.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100905994, 2024-04-22T18:39:22.696 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.696 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.696 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:22.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:22.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847565+0000", 2024-04-22T18:39:22.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:22.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847610+0000", 2024-04-22T18:39:22.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4984000000000002e-05 2024-04-22T18:39:22.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847619+0000", 2024-04-22T18:39:22.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2189999999999993e-06 2024-04-22T18:39:22.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947667+0000", 2024-04-22T18:39:22.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100047939 2024-04-22T18:39:22.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947716+0000", 2024-04-22T18:39:22.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9228999999999999e-05 2024-04-22T18:39:22.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947738+0000", 2024-04-22T18:39:22.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1963999999999998e-05 2024-04-22T18:39:22.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.948440+0000", 2024-04-22T18:39:22.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070173999999999996 2024-04-22T18:39:22.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.948472+0000", 2024-04-22T18:39:22.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1872999999999998e-05 2024-04-22T18:39:22.700 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.700 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.700 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.700 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1296 4.d e43/20 4:bc165c76:::smithi08937929-30:head v 43'15, mlcod=32'12)", 2024-04-22T18:39:22.700 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:18.060039+0000", 2024-04-22T18:39:22.700 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.569996529, 2024-04-22T18:39:22.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038697890000000002, 2024-04-22T18:39:22.700 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.701 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.701 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060039+0000", 2024-04-22T18:39:22.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060039+0000", 2024-04-22T18:39:22.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060037+0000", 2024-04-22T18:39:22.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:22.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060685+0000", 2024-04-22T18:39:22.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00064793499999999998 2024-04-22T18:39:22.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060697+0000", 2024-04-22T18:39:22.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2653000000000001e-05 2024-04-22T18:39:22.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060705+0000", 2024-04-22T18:39:22.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5320000000000001e-06 2024-04-22T18:39:22.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062088+0000", 2024-04-22T18:39:22.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001383287 2024-04-22T18:39:22.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062105+0000", 2024-04-22T18:39:22.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6595e-05 2024-04-22T18:39:22.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.063879+0000", 2024-04-22T18:39:22.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017744379999999999 2024-04-22T18:39:22.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.063909+0000", 2024-04-22T18:39:22.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9286e-05 2024-04-22T18:39:22.704 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.704 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.704 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.705 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1297 4.d e43/20 4:bc165c76:::smithi08937929-30:head v 43'16, mlcod=32'12)", 2024-04-22T18:39:22.705 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:18.060713+0000", 2024-04-22T18:39:22.705 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.5693223490000001, 2024-04-22T18:39:22.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0034270920000000001, 2024-04-22T18:39:22.705 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.705 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.705 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060713+0000", 2024-04-22T18:39:22.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060713+0000", 2024-04-22T18:39:22.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060712+0000", 2024-04-22T18:39:22.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:22.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.061996+0000", 2024-04-22T18:39:22.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012832869999999999 2024-04-22T18:39:22.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062002+0000", 2024-04-22T18:39:22.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6080000000000001e-06 2024-04-22T18:39:22.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062007+0000", 2024-04-22T18:39:22.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5210000000000004e-06 2024-04-22T18:39:22.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062638+0000", 2024-04-22T18:39:22.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063081699999999999 2024-04-22T18:39:22.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062654+0000", 2024-04-22T18:39:22.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6235e-05 2024-04-22T18:39:22.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.064126+0000", 2024-04-22T18:39:22.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0014719749999999999 2024-04-22T18:39:22.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.064140+0000", 2024-04-22T18:39:22.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4129e-05 2024-04-22T18:39:22.709 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.709 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.709 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.709 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1298 4.d e43/20 4:bc165c76:::smithi08937929-30:head v 43'17, mlcod=32'12)", 2024-04-22T18:39:22.709 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:18.062040+0000", 2024-04-22T18:39:22.709 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.5679957250000003, 2024-04-22T18:39:22.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099368133999999997, 2024-04-22T18:39:22.709 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.709 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.710 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062040+0000", 2024-04-22T18:39:22.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062040+0000", 2024-04-22T18:39:22.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062039+0000", 2024-04-22T18:39:22.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:22.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062056+0000", 2024-04-22T18:39:22.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6911000000000001e-05 2024-04-22T18:39:22.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062060+0000", 2024-04-22T18:39:22.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3000000000000003e-06 2024-04-22T18:39:22.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062065+0000", 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1150000000000002e-06 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.160817+0000", 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098751539999999999 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.160845+0000", 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7908000000000001e-05 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.161378+0000", 2024-04-22T18:39:22.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053300799999999996 2024-04-22T18:39:22.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.161408+0000", 2024-04-22T18:39:22.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.989e-05 2024-04-22T18:39:22.713 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.713 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.713 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.713 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:79 3.0 e44/19 3:05e6fa13:::smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 44'64, mlcod=44'61)", 2024-04-22T18:39:22.713 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:22.713 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.25918823200000002, 2024-04-22T18:39:22.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10093532199999999, 2024-04-22T18:39:22.714 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:22.714 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:22.714 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:22.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:22.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:22.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:22.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:22.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:22.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:22.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370846+0000", 2024-04-22T18:39:22.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:22.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:22.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370857+0000", 2024-04-22T18:39:22.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1155000000000001e-05 2024-04-22T18:39:22.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:22.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370866+0000", 2024-04-22T18:39:22.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8769999999999999e-06 2024-04-22T18:39:22.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:22.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.470937+0000", 2024-04-22T18:39:22.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100071204 2024-04-22T18:39:22.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:22.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.470986+0000", 2024-04-22T18:39:22.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8556000000000001e-05 2024-04-22T18:39:22.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:22.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471014+0000", 2024-04-22T18:39:22.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7982000000000001e-05 2024-04-22T18:39:22.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:22.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471750+0000", 2024-04-22T18:39:22.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073598100000000003 2024-04-22T18:39:22.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:22.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:22.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:22.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471782+0000", 2024-04-22T18:39:22.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339000000000003e-05 2024-04-22T18:39:22.717 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.718 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.718 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.718 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:22.718 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:22.718 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:22.719 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:22.720 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:22.720+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:22.762 INFO:tasks.rados.rados.0.smithi089.stdout:823: finishing write tid 1 to smithi08937929-45 2024-04-22T18:39:22.763 INFO:tasks.rados.rados.0.smithi089.stdout:823: finishing write tid 2 to smithi08937929-45 2024-04-22T18:39:22.763 INFO:tasks.rados.rados.0.smithi089.stdout:823: finishing write tid 3 to smithi08937929-45 2024-04-22T18:39:22.763 INFO:tasks.rados.rados.0.smithi089.stdout:823: finishing write tid 4 to smithi08937929-45 2024-04-22T18:39:22.763 INFO:tasks.rados.rados.0.smithi089.stdout:823: finishing write tid 5 to smithi08937929-45 2024-04-22T18:39:22.763 INFO:tasks.rados.rados.0.smithi089.stdout:823: oid 45 updating version 0 to 35 2024-04-22T18:39:22.763 INFO:tasks.rados.rados.0.smithi089.stdout:823: oid 45 version 35 is already newer than 34 2024-04-22T18:39:22.763 INFO:tasks.rados.rados.0.smithi089.stdout:823: oid 45 updating version 35 to 36 2024-04-22T18:39:22.763 INFO:tasks.rados.rados.0.smithi089.stdout:823: oid 45 version 36 is already newer than 33 2024-04-22T18:39:22.763 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 36 (ObjNum 108 snap 0 seq_num 108) dirty exists 2024-04-22T18:39:22.763 INFO:tasks.rados.rados.0.smithi089.stdout:823: left oid 45 (ObjNum 108 snap 0 seq_num 108) 2024-04-22T18:39:22.763 INFO:tasks.rados.rados.0.smithi089.stdout:825: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2024-04-22T18:39:22.769 INFO:tasks.rados.rados.0.smithi089.stdout:825: writing smithi08937929-22 from 631963 to 1263046 tid 1 2024-04-22T18:39:22.776 INFO:tasks.rados.rados.0.smithi089.stdout:825: writing smithi08937929-22 from 2061949 to 2621968 tid 2 2024-04-22T18:39:22.784 INFO:tasks.rados.rados.0.smithi089.stdout:825: writing smithi08937929-22 from 3243812 to 3748579 tid 3 2024-04-22T18:39:22.784 INFO:tasks.rados.rados.0.smithi089.stdout:814: done (8 left) 2024-04-22T18:39:22.784 INFO:tasks.rados.rados.0.smithi089.stdout:818: done (7 left) 2024-04-22T18:39:22.784 INFO:tasks.rados.rados.0.smithi089.stdout:819: done (6 left) 2024-04-22T18:39:22.784 INFO:tasks.rados.rados.0.smithi089.stdout:820: done (5 left) 2024-04-22T18:39:22.784 INFO:tasks.rados.rados.0.smithi089.stdout:821: done (4 left) 2024-04-22T18:39:22.784 INFO:tasks.rados.rados.0.smithi089.stdout:822: done (3 left) 2024-04-22T18:39:22.784 INFO:tasks.rados.rados.0.smithi089.stdout:823: done (2 left) 2024-04-22T18:39:22.784 INFO:tasks.rados.rados.0.smithi089.stdout:824: done (1 left) 2024-04-22T18:39:22.785 INFO:tasks.rados.rados.0.smithi089.stdout:826: read oid 30 snap -1 2024-04-22T18:39:22.785 INFO:tasks.rados.rados.0.smithi089.stdout:826: expect (ObjNum 103 snap 0 seq_num 103) 2024-04-22T18:39:22.785 INFO:tasks.rados.rados.0.smithi089.stdout:827: read oid 31 snap -1 2024-04-22T18:39:22.785 INFO:tasks.rados.rados.0.smithi089.stdout:827: expect (ObjNum 74 snap 0 seq_num 74) 2024-04-22T18:39:22.785 INFO:tasks.rados.rados.0.smithi089.stdout:828: tier_promote oid 15 2024-04-22T18:39:22.785 INFO:tasks.rados.rados.0.smithi089.stdout:829: read oid 27 snap -1 2024-04-22T18:39:22.785 INFO:tasks.rados.rados.0.smithi089.stdout:829: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-22T18:39:22.785 INFO:tasks.rados.rados.0.smithi089.stdout:830: read oid 4 snap -1 2024-04-22T18:39:22.785 INFO:tasks.rados.rados.0.smithi089.stdout:830: expect (ObjNum 81 snap 0 seq_num 81) 2024-04-22T18:39:22.785 INFO:tasks.rados.rados.0.smithi089.stdout:831: read oid 26 snap -1 2024-04-22T18:39:22.785 INFO:tasks.rados.rados.0.smithi089.stdout:831: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-22T18:39:22.785 INFO:tasks.rados.rados.0.smithi089.stdout:832: set_chunk oid 24 target oid 2024-04-22T18:39:22.820 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:22.821+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:22.863 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:22.863 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:22.872 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:22.956 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:22.957+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:23.016 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:23.016 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:23.022 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:23.023+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:23.025 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:23.122 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:23.123+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:23.168 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:6 4.f e24/20 4:f83a159b:::smithi08937929-2:head v 24'2, mlcod=0'0)", 2024-04-22T18:39:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.963108616, 2024-04-22T18:39:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10183012700000001, 2024-04-22T18:39:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.206027+0000", 2024-04-22T18:39:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023705500000000001 2024-04-22T18:39:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.206038+0000", 2024-04-22T18:39:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.148e-05 2024-04-22T18:39:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.206046+0000", 2024-04-22T18:39:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8679999999999999e-06 2024-04-22T18:39:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.306227+0000", 2024-04-22T18:39:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100181223 2024-04-22T18:39:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.306266+0000", 2024-04-22T18:39:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8906000000000003e-05 2024-04-22T18:39:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.307598+0000", 2024-04-22T18:39:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001331561 2024-04-22T18:39:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.307620+0000", 2024-04-22T18:39:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.226e-05 2024-04-22T18:39:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:45 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'1, mlcod=0'0)", 2024-04-22T18:39:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.903786929999999, 2024-04-22T18:39:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14345418900000001, 2024-04-22T18:39:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265727+0000", 2024-04-22T18:39:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061502799999999999 2024-04-22T18:39:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265737+0000", 2024-04-22T18:39:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0278999999999999e-05 2024-04-22T18:39:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265744+0000", 2024-04-22T18:39:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1740000000000003e-06 2024-04-22T18:39:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407227+0000", 2024-04-22T18:39:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14148266900000001 2024-04-22T18:39:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407258+0000", 2024-04-22T18:39:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1619000000000003e-05 2024-04-22T18:39:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408534+0000", 2024-04-22T18:39:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001275829 2024-04-22T18:39:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408566+0000", 2024-04-22T18:39:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1813999999999997e-05 2024-04-22T18:39:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:46 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'2, mlcod=0'0)", 2024-04-22T18:39:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.901162527, 2024-04-22T18:39:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14304645699999999, 2024-04-22T18:39:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268301+0000", 2024-04-22T18:39:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056585700000000001 2024-04-22T18:39:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268309+0000", 2024-04-22T18:39:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2830000000000004e-06 2024-04-22T18:39:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268312+0000", 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7179999999999998e-06 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407999+0000", 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13968636300000001 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408011+0000", 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2116999999999999e-05 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410762+0000", 2024-04-22T18:39:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002751524 2024-04-22T18:39:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410783+0000", 2024-04-22T18:39:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0270000000000001e-05 2024-04-22T18:39:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:47 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'3, mlcod=0'0)", 2024-04-22T18:39:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.899429956999999, 2024-04-22T18:39:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14132698599999999, 2024-04-22T18:39:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269468+0000", 2024-04-22T18:39:23.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:23.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269473+0000", 2024-04-22T18:39:23.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7790000000000003e-06 2024-04-22T18:39:23.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269478+0000", 2024-04-22T18:39:23.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0080000000000002e-06 2024-04-22T18:39:23.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269489+0000", 2024-04-22T18:39:23.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.075e-05 2024-04-22T18:39:23.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408595+0000", 2024-04-22T18:39:23.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13910662300000001 2024-04-22T18:39:23.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408611+0000", 2024-04-22T18:39:23.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6124e-05 2024-04-22T18:39:23.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410786+0000", 2024-04-22T18:39:23.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021743819999999999 2024-04-22T18:39:23.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410796+0000", 2024-04-22T18:39:23.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.007e-05 2024-04-22T18:39:23.186 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.186 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.186 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:10 4.f e27/20 4:f2c48a0e:::smithi08937929-3:head v 27'5, mlcod=24'2)", 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.726817387000001, 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10161616800000001, 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:23.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:23.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:23.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442159+0000", 2024-04-22T18:39:23.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7935000000000005e-05 2024-04-22T18:39:23.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442163+0000", 2024-04-22T18:39:23.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8430000000000003e-06 2024-04-22T18:39:23.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442166+0000", 2024-04-22T18:39:23.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0759999999999999e-06 2024-04-22T18:39:23.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542745+0000", 2024-04-22T18:39:23.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10057941200000001 2024-04-22T18:39:23.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542770+0000", 2024-04-22T18:39:23.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5134e-05 2024-04-22T18:39:23.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543679+0000", 2024-04-22T18:39:23.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00090833099999999996 2024-04-22T18:39:23.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543697+0000", 2024-04-22T18:39:23.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8876e-05 2024-04-22T18:39:23.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.191 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.191 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:11 4.f e27/20 4:f2c48a0e:::smithi08937929-3:head v 27'6, mlcod=24'2)", 2024-04-22T18:39:23.191 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.442170+0000", 2024-04-22T18:39:23.191 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.726728973, 2024-04-22T18:39:23.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101722358, 2024-04-22T18:39:23.191 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.191 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.191 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442170+0000", 2024-04-22T18:39:23.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442170+0000", 2024-04-22T18:39:23.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442169+0000", 2024-04-22T18:39:23.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:23.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442174+0000", 2024-04-22T18:39:23.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0350000000000003e-06 2024-04-22T18:39:23.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442177+0000", 2024-04-22T18:39:23.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0580000000000002e-06 2024-04-22T18:39:23.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442181+0000", 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1649999999999999e-06 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543319+0000", 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101138408 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543336+0000", 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6509999999999999e-05 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543880+0000", 2024-04-22T18:39:23.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054429900000000004 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543892+0000", 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2138e-05 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:77 4.0 4:01baf5a9:::smithi08937929-20:head [write 719333~462437,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.505878+0000", 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.663021172000001, 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10366277, 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.195 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 77 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.505878+0000", 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.505878+0000", 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.505876+0000", 2024-04-22T18:39:23.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:23.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506122+0000", 2024-04-22T18:39:23.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024626900000000002 2024-04-22T18:39:23.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506132+0000", 2024-04-22T18:39:23.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0216e-05 2024-04-22T18:39:23.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506139+0000", 2024-04-22T18:39:23.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6030000000000001e-06 2024-04-22T18:39:23.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506191+0000", 2024-04-22T18:39:23.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.185e-05 2024-04-22T18:39:23.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506333+0000", 2024-04-22T18:39:23.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014197299999999999 2024-04-22T18:39:23.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:23.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506480+0000", 2024-04-22T18:39:23.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014780199999999999 2024-04-22T18:39:23.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:23.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.507794+0000", 2024-04-22T18:39:23.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001313587 2024-04-22T18:39:23.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609429+0000", 2024-04-22T18:39:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101634658 2024-04-22T18:39:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609493+0000", 2024-04-22T18:39:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4777999999999995e-05 2024-04-22T18:39:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609540+0000", 2024-04-22T18:39:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6820000000000002e-05 2024-04-22T18:39:23.201 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.201 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.201 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.201 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:78 4.0 4:01baf5a9:::smithi08937929-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:23.201 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:23.201 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.656963835999999, 2024-04-22T18:39:23.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.398522192, 2024-04-22T18:39:23.201 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.201 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.201 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:23.201 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:23.202 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:23.202 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 78 2024-04-22T18:39:23.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.202 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:23.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:23.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511934+0000", 2024-04-22T18:39:23.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:23.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512012+0000", 2024-04-22T18:39:23.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8329000000000002e-05 2024-04-22T18:39:23.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512023+0000", 2024-04-22T18:39:23.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0755e-05 2024-04-22T18:39:23.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512028+0000", 2024-04-22T18:39:23.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6470000000000003e-06 2024-04-22T18:39:23.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512068+0000", 2024-04-22T18:39:23.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9721999999999997e-05 2024-04-22T18:39:23.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512125+0000", 2024-04-22T18:39:23.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6564999999999998e-05 2024-04-22T18:39:23.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:23.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512200+0000", 2024-04-22T18:39:23.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5708000000000001e-05 2024-04-22T18:39:23.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:23.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513131+0000", 2024-04-22T18:39:23.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093063 2024-04-22T18:39:23.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:23.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910394+0000", 2024-04-22T18:39:23.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.397263388 2024-04-22T18:39:23.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910430+0000", 2024-04-22T18:39:23.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5485999999999999e-05 2024-04-22T18:39:23.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910457+0000", 2024-04-22T18:39:23.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7086000000000001e-05 2024-04-22T18:39:23.207 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.207 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.207 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.207 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:79 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:23.207 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:23.207 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.656863179999998, 2024-04-22T18:39:23.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39855419399999997, 2024-04-22T18:39:23.208 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.208 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:23.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:23.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:23.208 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 79 2024-04-22T18:39:23.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.208 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:23.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:23.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512035+0000", 2024-04-22T18:39:23.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:23.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512045+0000", 2024-04-22T18:39:23.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0379e-05 2024-04-22T18:39:23.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512051+0000", 2024-04-22T18:39:23.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6030000000000004e-06 2024-04-22T18:39:23.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512057+0000", 2024-04-22T18:39:23.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7270000000000004e-06 2024-04-22T18:39:23.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512756+0000", 2024-04-22T18:39:23.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069954000000000001 2024-04-22T18:39:23.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512806+0000", 2024-04-22T18:39:23.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9922000000000001e-05 2024-04-22T18:39:23.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:23.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512927+0000", 2024-04-22T18:39:23.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012059 2024-04-22T18:39:23.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:23.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.515883+0000", 2024-04-22T18:39:23.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029563739999999999 2024-04-22T18:39:23.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:23.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910510+0000", 2024-04-22T18:39:23.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39462745199999999 2024-04-22T18:39:23.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910542+0000", 2024-04-22T18:39:23.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1225e-05 2024-04-22T18:39:23.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910590+0000", 2024-04-22T18:39:23.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8052e-05 2024-04-22T18:39:23.213 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.213 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.213 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.213 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:80 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:23.213 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:23.213 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.656838079, 2024-04-22T18:39:23.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39866376999999997, 2024-04-22T18:39:23.214 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.214 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:23.214 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:23.214 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:23.214 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:23.214 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 80 2024-04-22T18:39:23.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.214 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:23.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:23.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512060+0000", 2024-04-22T18:39:23.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:23.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512063+0000", 2024-04-22T18:39:23.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.999e-06 2024-04-22T18:39:23.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512069+0000", 2024-04-22T18:39:23.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.891e-06 2024-04-22T18:39:23.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512073+0000", 2024-04-22T18:39:23.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4429999999999996e-06 2024-04-22T18:39:23.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513084+0000", 2024-04-22T18:39:23.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010102609999999999 2024-04-22T18:39:23.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:23.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513123+0000", 2024-04-22T18:39:23.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9237999999999999e-05 2024-04-22T18:39:23.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910618+0000", 2024-04-22T18:39:23.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39749549299999998 2024-04-22T18:39:23.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910670+0000", 2024-04-22T18:39:23.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1221000000000003e-05 2024-04-22T18:39:23.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910724+0000", 2024-04-22T18:39:23.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4777000000000001e-05 2024-04-22T18:39:23.218 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.218 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.218 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.219 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:33 4.f e27/20 4:f0ed3d49:::smithi08937929-9:head v 27'7, mlcod=24'2)", 2024-04-22T18:39:23.219 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.609449+0000", 2024-04-22T18:39:23.219 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.559450145, 2024-04-22T18:39:23.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10183384600000001, 2024-04-22T18:39:23.219 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.219 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.219 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609449+0000", 2024-04-22T18:39:23.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609449+0000", 2024-04-22T18:39:23.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609448+0000", 2024-04-22T18:39:23.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:23.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609626+0000", 2024-04-22T18:39:23.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017809600000000001 2024-04-22T18:39:23.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609638+0000", 2024-04-22T18:39:23.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2045000000000001e-05 2024-04-22T18:39:23.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709771+0000", 2024-04-22T18:39:23.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013347 2024-04-22T18:39:23.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.710008+0000", 2024-04-22T18:39:23.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023696200000000001 2024-04-22T18:39:23.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.710045+0000", 2024-04-22T18:39:23.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6903000000000002e-05 2024-04-22T18:39:23.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.711263+0000", 2024-04-22T18:39:23.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012178110000000001 2024-04-22T18:39:23.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.711282+0000", 2024-04-22T18:39:23.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9525e-05 2024-04-22T18:39:23.223 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.223 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.223 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.223 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:5 3.2 e30/29 3:663b82a1:::smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=0'0)", 2024-04-22T18:39:23.223 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:23.223 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.941398760999999, 2024-04-22T18:39:23.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20090134500000001, 2024-04-22T18:39:23.223 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.223 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.223 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:23.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:23.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227499+0000", 2024-04-22T18:39:23.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:23.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227506+0000", 2024-04-22T18:39:23.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6540000000000002e-06 2024-04-22T18:39:23.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227516+0000", 2024-04-22T18:39:23.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0054000000000001e-05 2024-04-22T18:39:23.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327607+0000", 2024-04-22T18:39:23.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009167099999999 2024-04-22T18:39:23.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327688+0000", 2024-04-22T18:39:23.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1000000000000004e-05 2024-04-22T18:39:23.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327719+0000", 2024-04-22T18:39:23.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0664000000000003e-05 2024-04-22T18:39:23.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428371+0000", 2024-04-22T18:39:23.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10065239099999999 2024-04-22T18:39:23.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428401+0000", 2024-04-22T18:39:23.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9839e-05 2024-04-22T18:39:23.227 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.227 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.227 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.227 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:334 4.c e30/20 4:3227431c:::smithi08937929-35:head v 30'18, mlcod=30'16)", 2024-04-22T18:39:23.227 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.698741+0000", 2024-04-22T18:39:23.228 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.470157598, 2024-04-22T18:39:23.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10216847599999999, 2024-04-22T18:39:23.228 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.228 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.228 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698741+0000", 2024-04-22T18:39:23.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698741+0000", 2024-04-22T18:39:23.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698740+0000", 2024-04-22T18:39:23.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:23.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698747+0000", 2024-04-22T18:39:23.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9199999999999998e-06 2024-04-22T18:39:23.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698756+0000", 2024-04-22T18:39:23.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1249999999999999e-06 2024-04-22T18:39:23.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698762+0000", 2024-04-22T18:39:23.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:39:23.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798915+0000", 2024-04-22T18:39:23.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10015304899999999 2024-04-22T18:39:23.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798937+0000", 2024-04-22T18:39:23.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1335000000000002e-05 2024-04-22T18:39:23.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.800879+0000", 2024-04-22T18:39:23.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001942088 2024-04-22T18:39:23.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.800910+0000", 2024-04-22T18:39:23.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0892000000000002e-05 2024-04-22T18:39:23.231 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.231 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.231 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.232 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:419 4.0 4:01baf5a9:::smithi08937929-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:39:23.232 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:23.232 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.766677058999999, 2024-04-22T18:39:23.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30324478199999999, 2024-04-22T18:39:23.232 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.232 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.232 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:23.232 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:23.232 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:23.232 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 419 2024-04-22T18:39:23.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.232 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:23.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:23.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402219+0000", 2024-04-22T18:39:23.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:23.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402229+0000", 2024-04-22T18:39:23.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2550000000000003e-06 2024-04-22T18:39:23.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402242+0000", 2024-04-22T18:39:23.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2989999999999999e-05 2024-04-22T18:39:23.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402250+0000", 2024-04-22T18:39:23.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7900000000000005e-06 2024-04-22T18:39:23.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404085+0000", 2024-04-22T18:39:23.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001834397 2024-04-22T18:39:23.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:23.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404092+0000", 2024-04-22T18:39:23.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7889999999999997e-06 2024-04-22T18:39:23.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502534+0000", 2024-04-22T18:39:23.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098442331999999994 2024-04-22T18:39:23.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502631+0000", 2024-04-22T18:39:23.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7617999999999999e-05 2024-04-22T18:39:23.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:23.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604414+0000", 2024-04-22T18:39:23.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101782847 2024-04-22T18:39:23.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:23.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604888+0000", 2024-04-22T18:39:23.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047325999999999998 2024-04-22T18:39:23.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:23.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705376+0000", 2024-04-22T18:39:23.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100488623 2024-04-22T18:39:23.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705424+0000", 2024-04-22T18:39:23.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7964999999999999e-05 2024-04-22T18:39:23.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705466+0000", 2024-04-22T18:39:23.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2274999999999998e-05 2024-04-22T18:39:23.241 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.241 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.241 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.241 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:70 3.2 e32/29 3:5f0623ce:::smithi08937929-38 low_tier:head v 32'37, mlcod=31'35)", 2024-04-22T18:39:23.241 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:23.241 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.640460028, 2024-04-22T18:39:23.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20070027900000001, 2024-04-22T18:39:23.241 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.241 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.241 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:23.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:23.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528438+0000", 2024-04-22T18:39:23.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:23.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528443+0000", 2024-04-22T18:39:23.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8640000000000004e-06 2024-04-22T18:39:23.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528453+0000", 2024-04-22T18:39:23.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3910000000000004e-06 2024-04-22T18:39:23.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528459+0000", 2024-04-22T18:39:23.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-04-22T18:39:23.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728627+0000", 2024-04-22T18:39:23.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20016778700000001 2024-04-22T18:39:23.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728660+0000", 2024-04-22T18:39:23.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3024000000000001e-05 2024-04-22T18:39:23.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729107+0000", 2024-04-22T18:39:23.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044719200000000001 2024-04-22T18:39:23.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729139+0000", 2024-04-22T18:39:23.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1625000000000003e-05 2024-04-22T18:39:23.245 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.245 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.245 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.245 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1263 4.c e43/20 4:36664a98:::smithi08937929-38:head v 43'36, mlcod=42'34)", 2024-04-22T18:39:23.245 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.555349+0000", 2024-04-22T18:39:23.245 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.6135495259999999, 2024-04-22T18:39:23.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10300226699999999, 2024-04-22T18:39:23.245 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.246 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.246 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.555349+0000", 2024-04-22T18:39:23.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.555349+0000", 2024-04-22T18:39:23.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.555349+0000", 2024-04-22T18:39:23.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:23.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.556191+0000", 2024-04-22T18:39:23.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084236200000000004 2024-04-22T18:39:23.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.556203+0000", 2024-04-22T18:39:23.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1301000000000001e-05 2024-04-22T18:39:23.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656304+0000", 2024-04-22T18:39:23.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100101022 2024-04-22T18:39:23.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656364+0000", 2024-04-22T18:39:23.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0207000000000001e-05 2024-04-22T18:39:23.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656392+0000", 2024-04-22T18:39:23.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8537000000000001e-05 2024-04-22T18:39:23.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.658324+0000", 2024-04-22T18:39:23.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019315409999999999 2024-04-22T18:39:23.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.658351+0000", 2024-04-22T18:39:23.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7517000000000001e-05 2024-04-22T18:39:23.249 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.249 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.249 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.249 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1264 4.c e43/20 4:36664a98:::smithi08937929-38:head v 43'37, mlcod=42'34)", 2024-04-22T18:39:23.250 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:23.250 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.5125616239999999, 2024-04-22T18:39:23.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20290602399999999, 2024-04-22T18:39:23.250 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.250 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.250 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:23.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:23.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656335+0000", 2024-04-22T18:39:23.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:23.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656850+0000", 2024-04-22T18:39:23.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051499499999999997 2024-04-22T18:39:23.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656861+0000", 2024-04-22T18:39:23.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1119e-05 2024-04-22T18:39:23.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756934+0000", 2024-04-22T18:39:23.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073548 2024-04-22T18:39:23.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756977+0000", 2024-04-22T18:39:23.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2218000000000001e-05 2024-04-22T18:39:23.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756993+0000", 2024-04-22T18:39:23.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6629999999999998e-05 2024-04-22T18:39:23.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859203+0000", 2024-04-22T18:39:23.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102209617 2024-04-22T18:39:23.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859243+0000", 2024-04-22T18:39:23.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0333000000000002e-05 2024-04-22T18:39:23.253 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.254 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.254 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.254 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1265 4.c e43/20 4:36664a98:::smithi08937929-38:head v 43'38, mlcod=42'34)", 2024-04-22T18:39:23.254 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.756947+0000", 2024-04-22T18:39:23.254 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.4119517210000003, 2024-04-22T18:39:23.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102320095, 2024-04-22T18:39:23.254 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.254 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.254 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756947+0000", 2024-04-22T18:39:23.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756947+0000", 2024-04-22T18:39:23.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756946+0000", 2024-04-22T18:39:23.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:23.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757183+0000", 2024-04-22T18:39:23.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000236849 2024-04-22T18:39:23.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757202+0000", 2024-04-22T18:39:23.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9403e-05 2024-04-22T18:39:23.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757209+0000", 2024-04-22T18:39:23.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0539999999999997e-06 2024-04-22T18:39:23.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.758174+0000", 2024-04-22T18:39:23.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00096538199999999998 2024-04-22T18:39:23.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.758195+0000", 2024-04-22T18:39:23.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1043000000000002e-05 2024-04-22T18:39:23.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859248+0000", 2024-04-22T18:39:23.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10105283 2024-04-22T18:39:23.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859267+0000", 2024-04-22T18:39:23.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8774e-05 2024-04-22T18:39:23.258 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.258 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.258 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.258 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1266 4.c e43/20 4:36664a98:::smithi08937929-38:head v 43'39, mlcod=42'34)", 2024-04-22T18:39:23.258 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.757217+0000", 2024-04-22T18:39:23.258 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.4116815230000004, 2024-04-22T18:39:23.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102430021, 2024-04-22T18:39:23.258 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.259 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.259 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757217+0000", 2024-04-22T18:39:23.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757217+0000", 2024-04-22T18:39:23.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757216+0000", 2024-04-22T18:39:23.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:23.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757222+0000", 2024-04-22T18:39:23.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9320000000000001e-06 2024-04-22T18:39:23.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757226+0000", 2024-04-22T18:39:23.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.126e-06 2024-04-22T18:39:23.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757229+0000", 2024-04-22T18:39:23.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.897e-06 2024-04-22T18:39:23.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.858954+0000", 2024-04-22T18:39:23.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101724905 2024-04-22T18:39:23.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.858981+0000", 2024-04-22T18:39:23.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7216000000000002e-05 2024-04-22T18:39:23.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859620+0000", 2024-04-22T18:39:23.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063874799999999996 2024-04-22T18:39:23.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859647+0000", 2024-04-22T18:39:23.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6959000000000001e-05 2024-04-22T18:39:23.262 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.262 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.262 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.262 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1398 3.2 e44/29 3:460a5d9d:::9d81e38fa45b2f87319d1f6ca039dfc50d4304ceb216fced89f733de95073c57:head v 44'70, mlcod=43'68)", 2024-04-22T18:39:23.262 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:21.958373+0000", 2024-04-22T18:39:23.263 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.2105259429999999, 2024-04-22T18:39:23.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102010931, 2024-04-22T18:39:23.263 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:23.263 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:23.263 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:23.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:23.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.958373+0000", 2024-04-22T18:39:23.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:23.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.958373+0000", 2024-04-22T18:39:23.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:23.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:23.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.958372+0000", 2024-04-22T18:39:23.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:23.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:23.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.959307+0000", 2024-04-22T18:39:23.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000934135 2024-04-22T18:39:23.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:23.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.959317+0000", 2024-04-22T18:39:23.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0735e-05 2024-04-22T18:39:23.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.959324+0000", 2024-04-22T18:39:23.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0639999999999996e-06 2024-04-22T18:39:23.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:23.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.059445+0000", 2024-04-22T18:39:23.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100120632 2024-04-22T18:39:23.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:23.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.059471+0000", 2024-04-22T18:39:23.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6057000000000001e-05 2024-04-22T18:39:23.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:23.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.060348+0000", 2024-04-22T18:39:23.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087665200000000001 2024-04-22T18:39:23.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:23.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:23.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:23.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.060384+0000", 2024-04-22T18:39:23.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6016000000000002e-05 2024-04-22T18:39:23.266 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.266 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.266 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.266 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:23.267 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:23.267 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:23.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:23.269 INFO:tasks.rados.rados.0.smithi089.stdout:832: set_chunk oid 24 offset: 883200 length: 382976 target oid 3f4a7193a9504c4cf6fb5b8405366d669aa80de64738a1c18fbec93482e47be3 offset: 0 snap -1 2024-04-22T18:39:23.270 INFO:tasks.rados.rados.0.smithi089.stdout:833: write oid 21 current snap is 0 2024-04-22T18:39:23.270 INFO:tasks.rados.rados.0.smithi089.stdout:833: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2024-04-22T18:39:23.270 INFO:tasks.rados.rados.0.smithi089.stdout:833: writing smithi08937929-21 from 648770 to 1355102 tid 1 2024-04-22T18:39:23.270 INFO:tasks.rados.rados.0.smithi089.stdout:833: writing smithi08937929-21 from 2120431 to 2540461 tid 2 2024-04-22T18:39:23.270 INFO:tasks.rados.rados.0.smithi089.stdout:833: writing smithi08937929-21 from 3271235 to 3703627 tid 3 2024-04-22T18:39:23.270 INFO:tasks.rados.rados.0.smithi089.stdout:834: read oid 17 snap -1 2024-04-22T18:39:23.270 INFO:tasks.rados.rados.0.smithi089.stdout:834: expect (ObjNum 101 snap 0 seq_num 101) 2024-04-22T18:39:23.270 INFO:tasks.rados.rados.0.smithi089.stdout:835: tier_promote oid 43 2024-04-22T18:39:23.270 INFO:tasks.rados.rados.0.smithi089.stdout:827: expect (ObjNum 74 snap 0 seq_num 74) 2024-04-22T18:39:23.271 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:39:23.271 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:23.269+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:23.319 INFO:tasks.rados.rados.0.smithi089.stdout:828: got (2) No such file or directory 2024-04-22T18:39:23.319 INFO:tasks.rados.rados.0.smithi089.stdout:828: got expected ENOENT (src dne) 2024-04-22T18:39:23.319 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 24 (ObjNum 129 snap 0 seq_num 2411949288) clean dne 2024-04-22T18:39:23.319 INFO:tasks.rados.rados.0.smithi089.stdout:827: done (10 left) 2024-04-22T18:39:23.319 INFO:tasks.rados.rados.0.smithi089.stdout:828: done (9 left) 2024-04-22T18:39:23.320 INFO:tasks.rados.rados.0.smithi089.stdout:836: write (excl) oid 45 current snap is 0 2024-04-22T18:39:23.320 INFO:tasks.rados.rados.0.smithi089.stdout:836: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2024-04-22T18:39:23.326 INFO:tasks.rados.rados.0.smithi089.stdout:836: writing smithi08937929-45 from 665577 to 1447158 tid 1 2024-04-22T18:39:23.333 INFO:tasks.rados.rados.0.smithi089.stdout:836: writing smithi08937929-45 from 2062560 to 2626248 tid 2 2024-04-22T18:39:23.339 INFO:tasks.rados.rados.0.smithi089.stdout:836: writing smithi08937929-45 from 3182305 to 3825969 tid 3 2024-04-22T18:39:23.339 INFO:tasks.rados.rados.0.smithi089.stdout:837: set_chunk oid 5 target oid 2024-04-22T18:39:23.368 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:23.369+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:23.379 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:23.379 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:23.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.380 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:145 3.3 3:ee72d5d8:::e9ce7f7524b5011776d2f5e511fccc742fb05c876c8351c5fbcd3a190f117347:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:39:23.380 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:23.177104+0000", 2024-04-22T18:39:23.380 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.204212228, 2024-04-22T18:39:23.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20424612, 2024-04-22T18:39:23.380 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:23.380 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:39:23.380 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:23.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:23.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.177104+0000", 2024-04-22T18:39:23.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:23.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.177104+0000", 2024-04-22T18:39:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.177103+0000", 2024-04-22T18:39:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.177108+0000", 2024-04-22T18:39:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:39:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:23.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.177116+0000", 2024-04-22T18:39:23.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7190000000000001e-06 2024-04-22T18:39:23.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.277194+0000", 2024-04-22T18:39:23.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100078263 2024-04-22T18:39:23.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:23.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.277253+0000", 2024-04-22T18:39:23.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8533999999999999e-05 2024-04-22T18:39:23.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:23.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.277366+0000", 2024-04-22T18:39:23.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113254 2024-04-22T18:39:23.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:23.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.277557+0000", 2024-04-22T18:39:23.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019100599999999999 2024-04-22T18:39:23.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:23.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.278260+0000", 2024-04-22T18:39:23.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070289300000000001 2024-04-22T18:39:23.384 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:23.384 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:23.384 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:23.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.384 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.5.13:10 3.3 3:cef60dab:::smithi08937929-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:39:23.384 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:23.215116+0000", 2024-04-22T18:39:23.384 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.16620009499999999, 2024-04-22T18:39:23.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16629822499999999, 2024-04-22T18:39:23.384 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:23.384 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:39:23.384 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:23.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:23.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215116+0000", 2024-04-22T18:39:23.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:23.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:23.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215116+0000", 2024-04-22T18:39:23.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:23.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:23.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215113+0000", 2024-04-22T18:39:23.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:23.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:23.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215126+0000", 2024-04-22T18:39:23.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3020000000000001e-05 2024-04-22T18:39:23.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:23.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215148+0000", 2024-04-22T18:39:23.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1713e-05 2024-04-22T18:39:23.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215160+0000", 2024-04-22T18:39:23.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2198000000000001e-05 2024-04-22T18:39:23.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215220+0000", 2024-04-22T18:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0497e-05 2024-04-22T18:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215323+0000", 2024-04-22T18:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102773 2024-04-22T18:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215477+0000", 2024-04-22T18:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015396400000000001 2024-04-22T18:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:23.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.216518+0000", 2024-04-22T18:39:23.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001041012 2024-04-22T18:39:23.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:23.388 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:23.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:23.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.388 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.5.13:11 3.3 3:cef60dab:::smithi08937929-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:39:23.388 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:23.215172+0000", 2024-04-22T18:39:23.388 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.16614361499999999, 2024-04-22T18:39:23.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16628359500000001, 2024-04-22T18:39:23.388 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:23.388 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:39:23.389 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:23.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:23.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215172+0000", 2024-04-22T18:39:23.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:23.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:23.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215172+0000", 2024-04-22T18:39:23.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:23.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:23.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215172+0000", 2024-04-22T18:39:23.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:23.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:23.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215184+0000", 2024-04-22T18:39:23.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2678e-05 2024-04-22T18:39:23.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:23.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215190+0000", 2024-04-22T18:39:23.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4469999999999997e-06 2024-04-22T18:39:23.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:23.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215194+0000", 2024-04-22T18:39:23.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0230000000000003e-06 2024-04-22T18:39:23.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:23.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215811+0000", 2024-04-22T18:39:23.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00061693900000000005 2024-04-22T18:39:23.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:23.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.215911+0000", 2024-04-22T18:39:23.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000100098 2024-04-22T18:39:23.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:23.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.216110+0000", 2024-04-22T18:39:23.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019944900000000001 2024-04-22T18:39:23.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:23.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:23.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:23.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:23.216846+0000", 2024-04-22T18:39:23.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073607499999999999 2024-04-22T18:39:23.392 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:23.392 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:23.392 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:23.392 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:23.392 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:39:23.392 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 3 2024-04-22T18:39:23.392 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:23.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:23.478 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:23.479+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:23.484 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:39:23.536 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:23.536 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:23.536 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:23.536 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:23.536 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:23.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:23.570 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:23.571+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:23.603 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:23.604 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:23.620 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:23.620 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:23.636 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:23.636 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:23.653 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:23.653 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:23.669 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:23.670 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:23.670 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:23.671+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:23.686 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:23.792 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:23.792 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:23.793 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:23.793 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:23.794 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:23.794 INFO:tasks.rados.rados.0.smithi089.stdout:837: set_chunk oid 5 offset: 1920000 length: 300544 target oid 7617f7bf3c21180f2f428f6ed01c35a586bd197009d481d32e34b044e2b33d62 offset: 0 snap -1 2024-04-22T18:39:23.795 INFO:tasks.rados.rados.0.smithi089.stdout:838: set_chunk oid 25 target oid 2024-04-22T18:39:23.795 INFO:tasks.rados.rados.0.smithi089.stdout:838: set_chunk oid 25 offset: 3168256 length: 519168 target oid 96e6d307c9f0b5815fc76987e08ec65e3e3d40d39bed18685e4cba0fe0ba48ac offset: 0 snap -1 2024-04-22T18:39:23.795 INFO:tasks.rados.rados.0.smithi089.stdout:839: set_chunk oid 38 target oid 2024-04-22T18:39:23.796 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:23.796 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:23.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:23.797 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:23.797+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:23.896 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:23.898+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:23.938 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:23.938 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:23.943 INFO:tasks.rados.rados.0.smithi089.stdout:839: set_chunk oid 38 offset: 1373184 length: 897024 target oid caa2fee493ffa27d6b1c17aa88694e5c9f90816bfa77695547fbd2bab978c2d4 offset: 0 snap -1 2024-04-22T18:39:23.943 INFO:tasks.rados.rados.0.smithi089.stdout:840: tier_promote oid 46 2024-04-22T18:39:23.943 INFO:tasks.rados.rados.0.smithi089.stdout:826: expect (ObjNum 103 snap 0 seq_num 103) 2024-04-22T18:39:23.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:23.997 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:23.998+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:24.047 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:24.047 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:24.056 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:24.098 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:24.099+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:24.101 INFO:tasks.rados.rados.0.smithi089.stdout:829: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-22T18:39:24.177 INFO:tasks.rados.rados.0.smithi089.stdout:825: finishing write tid 1 to smithi08937929-22 2024-04-22T18:39:24.177 INFO:tasks.rados.rados.0.smithi089.stdout:825: finishing write tid 2 to smithi08937929-22 2024-04-22T18:39:24.177 INFO:tasks.rados.rados.0.smithi089.stdout:830: expect (ObjNum 81 snap 0 seq_num 81) 2024-04-22T18:39:24.198 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:24.199+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:24.200 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:24.200 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:24.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:24.299 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:24.300+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:24.317 INFO:tasks.rados.rados.0.smithi089.stdout:825: finishing write tid 3 to smithi08937929-22 2024-04-22T18:39:24.317 INFO:tasks.rados.rados.0.smithi089.stdout:825: finishing write tid 4 to smithi08937929-22 2024-04-22T18:39:24.317 INFO:tasks.rados.rados.0.smithi089.stdout:825: finishing write tid 5 to smithi08937929-22 2024-04-22T18:39:24.317 INFO:tasks.rados.rados.0.smithi089.stdout:825: oid 22 updating version 0 to 43 2024-04-22T18:39:24.317 INFO:tasks.rados.rados.0.smithi089.stdout:825: oid 22 version 43 is already newer than 42 2024-04-22T18:39:24.318 INFO:tasks.rados.rados.0.smithi089.stdout:825: oid 22 updating version 43 to 44 2024-04-22T18:39:24.318 INFO:tasks.rados.rados.0.smithi089.stdout:825: oid 22 updating version 44 to 45 2024-04-22T18:39:24.318 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 45 (ObjNum 109 snap 0 seq_num 109) dirty exists 2024-04-22T18:39:24.318 INFO:tasks.rados.rados.0.smithi089.stdout:825: left oid 22 (ObjNum 109 snap 0 seq_num 109) 2024-04-22T18:39:24.318 INFO:tasks.rados.rados.0.smithi089.stdout:831: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-22T18:39:24.350 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:24.350 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:24.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:24.394 INFO:tasks.rados.rados.0.smithi089.stdout:833: finishing write tid 1 to smithi08937929-21 2024-04-22T18:39:24.394 INFO:tasks.rados.rados.0.smithi089.stdout:834: expect (ObjNum 101 snap 0 seq_num 101) 2024-04-22T18:39:24.400 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:24.402+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:24.499 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:24.500 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:24.501 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:24.501+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:24.508 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:24.509 INFO:tasks.rados.rados.0.smithi089.stdout:835: got (0) Success 2024-04-22T18:39:24.509 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 15 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-04-22T18:39:24.510 INFO:tasks.rados.rados.0.smithi089.stdout:832: finishing set_chunk to oid 24 2024-04-22T18:39:24.510 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 883200 length: 382976 2024-04-22T18:39:24.510 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-22T18:39:24.510 INFO:tasks.rados.rados.0.smithi089.stdout:833: finishing write tid 2 to smithi08937929-21 2024-04-22T18:39:24.510 INFO:tasks.rados.rados.0.smithi089.stdout:833: finishing write tid 3 to smithi08937929-21 2024-04-22T18:39:24.510 INFO:tasks.rados.rados.0.smithi089.stdout:825: done (13 left) 2024-04-22T18:39:24.510 INFO:tasks.rados.rados.0.smithi089.stdout:826: done (12 left) 2024-04-22T18:39:24.510 INFO:tasks.rados.rados.0.smithi089.stdout:829: done (11 left) 2024-04-22T18:39:24.510 INFO:tasks.rados.rados.0.smithi089.stdout:830: done (10 left) 2024-04-22T18:39:24.510 INFO:tasks.rados.rados.0.smithi089.stdout:831: done (9 left) 2024-04-22T18:39:24.510 INFO:tasks.rados.rados.0.smithi089.stdout:832: done (8 left) 2024-04-22T18:39:24.510 INFO:tasks.rados.rados.0.smithi089.stdout:834: done (7 left) 2024-04-22T18:39:24.510 INFO:tasks.rados.rados.0.smithi089.stdout:835: done (6 left) 2024-04-22T18:39:24.511 INFO:tasks.rados.rados.0.smithi089.stdout:841: write oid 24 current snap is 0 2024-04-22T18:39:24.511 INFO:tasks.rados.rados.0.smithi089.stdout:841: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2024-04-22T18:39:24.514 INFO:tasks.rados.rados.0.smithi089.stdout:841: writing smithi08937929-24 from 682384 to 1139214 tid 1 2024-04-22T18:39:24.520 INFO:tasks.rados.rados.0.smithi089.stdout:841: writing smithi08937929-24 from 1721042 to 2144741 tid 2 2024-04-22T18:39:24.526 INFO:tasks.rados.rados.0.smithi089.stdout:841: writing smithi08937929-24 from 2809728 to 3264664 tid 3 2024-04-22T18:39:24.529 INFO:tasks.rados.rados.0.smithi089.stdout:841: writing smithi08937929-24 from 3731357 to 3882384 tid 4 2024-04-22T18:39:24.529 INFO:tasks.rados.rados.0.smithi089.stdout:842: read oid 44 snap -1 2024-04-22T18:39:24.529 INFO:tasks.rados.rados.0.smithi089.stdout:842: expect (ObjNum 91 snap 0 seq_num 91) 2024-04-22T18:39:24.529 INFO:tasks.rados.rados.0.smithi089.stdout:843: write oid 6 current snap is 0 2024-04-22T18:39:24.529 INFO:tasks.rados.rados.0.smithi089.stdout:843: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-04-22T18:39:24.535 INFO:tasks.rados.rados.0.smithi089.stdout:843: writing smithi08937929-6 from 699191 to 1231270 tid 1 2024-04-22T18:39:24.541 INFO:tasks.rados.rados.0.smithi089.stdout:843: writing smithi08937929-6 from 1779524 to 2346881 tid 2 2024-04-22T18:39:24.549 INFO:tasks.rados.rados.0.smithi089.stdout:843: writing smithi08937929-6 from 2837151 to 3503359 tid 3 2024-04-22T18:39:24.549 INFO:tasks.rados.rados.0.smithi089.stdout:833: finishing write tid 4 to smithi08937929-21 2024-04-22T18:39:24.549 INFO:tasks.rados.rados.0.smithi089.stdout:833: finishing write tid 5 to smithi08937929-21 2024-04-22T18:39:24.549 INFO:tasks.rados.rados.0.smithi089.stdout:833: oid 21 updating version 0 to 31 2024-04-22T18:39:24.549 INFO:tasks.rados.rados.0.smithi089.stdout:833: oid 21 version 31 is already newer than 30 2024-04-22T18:39:24.550 INFO:tasks.rados.rados.0.smithi089.stdout:833: oid 21 version 31 is already newer than 29 2024-04-22T18:39:24.550 INFO:tasks.rados.rados.0.smithi089.stdout:833: oid 21 version 31 is already newer than 28 2024-04-22T18:39:24.550 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 31 (ObjNum 110 snap 0 seq_num 110) dirty exists 2024-04-22T18:39:24.550 INFO:tasks.rados.rados.0.smithi089.stdout:833: left oid 21 (ObjNum 110 snap 0 seq_num 110) 2024-04-22T18:39:24.550 INFO:tasks.rados.rados.0.smithi089.stdout:836: finishing write tid 1 to smithi08937929-45 2024-04-22T18:39:24.550 INFO:tasks.rados.rados.0.smithi089.stdout:836: finishing write tid 2 to smithi08937929-45 2024-04-22T18:39:24.550 INFO:tasks.rados.rados.0.smithi089.stdout:836: finishing write tid 3 to smithi08937929-45 2024-04-22T18:39:24.550 INFO:tasks.rados.rados.0.smithi089.stdout:836: finishing write tid 4 to smithi08937929-45 2024-04-22T18:39:24.550 INFO:tasks.rados.rados.0.smithi089.stdout:836: finishing write tid 5 to smithi08937929-45 2024-04-22T18:39:24.550 INFO:tasks.rados.rados.0.smithi089.stdout:836: oid 45 updating version 0 to 40 2024-04-22T18:39:24.550 INFO:tasks.rados.rados.0.smithi089.stdout:836: oid 45 version 40 is already newer than 39 2024-04-22T18:39:24.550 INFO:tasks.rados.rados.0.smithi089.stdout:836: oid 45 version 40 is already newer than 38 2024-04-22T18:39:24.550 INFO:tasks.rados.rados.0.smithi089.stdout:836: oid 45 version 40 is already newer than 37 2024-04-22T18:39:24.550 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 40 (ObjNum 111 snap 0 seq_num 111) dirty exists 2024-04-22T18:39:24.551 INFO:tasks.rados.rados.0.smithi089.stdout:836: left oid 45 (ObjNum 111 snap 0 seq_num 111) 2024-04-22T18:39:24.551 INFO:tasks.rados.rados.0.smithi089.stdout:837: finishing set_chunk to oid 5 2024-04-22T18:39:24.551 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 5 set_chunk returned error code -95 offset: 1920000 length: 300544 2024-04-22T18:39:24.551 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 0 (ObjNum 87 snap 0 seq_num 87) dirty exists 2024-04-22T18:39:24.551 INFO:tasks.rados.rados.0.smithi089.stdout:838: finishing set_chunk to oid 25 2024-04-22T18:39:24.551 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 44 (ObjNum 102 snap 0 seq_num 102) dirty exists 2024-04-22T18:39:24.551 INFO:tasks.rados.rados.0.smithi089.stdout:839: finishing set_chunk to oid 38 2024-04-22T18:39:24.551 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 38 set_chunk returned error code -95 offset: 1373184 length: 897024 2024-04-22T18:39:24.551 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 0 (ObjNum 97 snap 0 seq_num 97) dirty exists 2024-04-22T18:39:24.551 INFO:tasks.rados.rados.0.smithi089.stdout:840: got (0) Success 2024-04-22T18:39:24.551 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-22T18:39:24.551 INFO:tasks.rados.rados.0.smithi089.stdout:833: done (8 left) 2024-04-22T18:39:24.551 INFO:tasks.rados.rados.0.smithi089.stdout:836: done (7 left) 2024-04-22T18:39:24.551 INFO:tasks.rados.rados.0.smithi089.stdout:837: done (6 left) 2024-04-22T18:39:24.552 INFO:tasks.rados.rados.0.smithi089.stdout:838: done (5 left) 2024-04-22T18:39:24.552 INFO:tasks.rados.rados.0.smithi089.stdout:839: done (4 left) 2024-04-22T18:39:24.552 INFO:tasks.rados.rados.0.smithi089.stdout:840: done (3 left) 2024-04-22T18:39:24.552 INFO:tasks.rados.rados.0.smithi089.stdout:844: tier_evict oid 30 2024-04-22T18:39:24.552 INFO:tasks.rados.rados.0.smithi089.stdout:844: tier_evict oid 30 snap -1 2024-04-22T18:39:24.552 INFO:tasks.rados.rados.0.smithi089.stdout:842: expect (ObjNum 91 snap 0 seq_num 91) 2024-04-22T18:39:24.602 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:24.603+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:24.653 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:24.654 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:24.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:24.707 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:24.708+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:24.765 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:24.765 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:24.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:24.788 INFO:tasks.rados.rados.0.smithi089.stdout:844: got (22) Invalid argument 2024-04-22T18:39:24.788 INFO:tasks.rados.rados.0.smithi089.stdout:843: finishing write tid 1 to smithi08937929-6 2024-04-22T18:39:24.788 INFO:tasks.rados.rados.0.smithi089.stdout:843: finishing write tid 2 to smithi08937929-6 2024-04-22T18:39:24.788 INFO:tasks.rados.rados.0.smithi089.stdout:843: finishing write tid 3 to smithi08937929-6 2024-04-22T18:39:24.788 INFO:tasks.rados.rados.0.smithi089.stdout:843: finishing write tid 4 to smithi08937929-6 2024-04-22T18:39:24.788 INFO:tasks.rados.rados.0.smithi089.stdout:843: finishing write tid 5 to smithi08937929-6 2024-04-22T18:39:24.788 INFO:tasks.rados.rados.0.smithi089.stdout:843: oid 6 updating version 0 to 28 2024-04-22T18:39:24.788 INFO:tasks.rados.rados.0.smithi089.stdout:843: oid 6 updating version 28 to 30 2024-04-22T18:39:24.788 INFO:tasks.rados.rados.0.smithi089.stdout:843: oid 6 version 30 is already newer than 29 2024-04-22T18:39:24.788 INFO:tasks.rados.rados.0.smithi089.stdout:843: oid 6 updating version 30 to 31 2024-04-22T18:39:24.788 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 31 (ObjNum 113 snap 0 seq_num 113) dirty exists 2024-04-22T18:39:24.789 INFO:tasks.rados.rados.0.smithi089.stdout:843: left oid 6 (ObjNum 113 snap 0 seq_num 113) 2024-04-22T18:39:24.789 INFO:tasks.rados.rados.0.smithi089.stdout:842: done (3 left) 2024-04-22T18:39:24.789 INFO:tasks.rados.rados.0.smithi089.stdout:843: done (2 left) 2024-04-22T18:39:24.789 INFO:tasks.rados.rados.0.smithi089.stdout:844: done (1 left) 2024-04-22T18:39:24.789 INFO:tasks.rados.rados.0.smithi089.stdout:845: set_chunk oid 3 target oid 2024-04-22T18:39:24.789 INFO:tasks.rados.rados.0.smithi089.stdout:845: set_chunk oid 3 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:39:24.789 INFO:tasks.rados.rados.0.smithi089.stdout:846: delete oid 5 current snap is 0 2024-04-22T18:39:24.789 INFO:tasks.rados.rados.0.smithi089.stdout:845: finishing set_chunk to oid 3 2024-04-22T18:39:24.789 INFO:tasks.rados.rados.0.smithi089.stdout:845: got expected ENOENT (src dne) 2024-04-22T18:39:24.814 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:24.815+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:24.904 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:24.905+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:24.913 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:24.913 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:24.918 INFO:tasks.rados.rados.0.smithi089.stdout:841: finishing write tid 1 to smithi08937929-24 2024-04-22T18:39:24.918 INFO:tasks.rados.rados.0.smithi089.stdout:841: finishing write tid 2 to smithi08937929-24 2024-04-22T18:39:24.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:25.005 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:25.006+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:25.019 INFO:tasks.rados.rados.0.smithi089.stdout:841: finishing write tid 3 to smithi08937929-24 2024-04-22T18:39:25.020 INFO:tasks.rados.rados.0.smithi089.stdout:841: finishing write tid 4 to smithi08937929-24 2024-04-22T18:39:25.020 INFO:tasks.rados.rados.0.smithi089.stdout:841: finishing write tid 5 to smithi08937929-24 2024-04-22T18:39:25.020 INFO:tasks.rados.rados.0.smithi089.stdout:841: finishing write tid 6 to smithi08937929-24 2024-04-22T18:39:25.020 INFO:tasks.rados.rados.0.smithi089.stdout:841: oid 24 updating version 0 to 71 2024-04-22T18:39:25.020 INFO:tasks.rados.rados.0.smithi089.stdout:841: oid 24 version 71 is already newer than 68 2024-04-22T18:39:25.020 INFO:tasks.rados.rados.0.smithi089.stdout:841: oid 24 version 71 is already newer than 69 2024-04-22T18:39:25.020 INFO:tasks.rados.rados.0.smithi089.stdout:841: oid 24 version 71 is already newer than 70 2024-04-22T18:39:25.020 INFO:tasks.rados.rados.0.smithi089.stdout:841: oid 24 version 71 is already newer than 67 2024-04-22T18:39:25.020 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 71 (ObjNum 112 snap 0 seq_num 112) dirty exists 2024-04-22T18:39:25.020 INFO:tasks.rados.rados.0.smithi089.stdout:841: left oid 24 (ObjNum 112 snap 0 seq_num 112) 2024-04-22T18:39:25.020 INFO:tasks.rados.rados.0.smithi089.stdout:841: done (2 left) 2024-04-22T18:39:25.020 INFO:tasks.rados.rados.0.smithi089.stdout:845: done (1 left) 2024-04-22T18:39:25.020 INFO:tasks.rados.rados.0.smithi089.stdout:846: done (0 left) 2024-04-22T18:39:25.020 INFO:tasks.rados.rados.0.smithi089.stdout:847: read oid 48 snap -1 2024-04-22T18:39:25.021 INFO:tasks.rados.rados.0.smithi089.stdout:847: expect deleted 2024-04-22T18:39:25.021 INFO:tasks.rados.rados.0.smithi089.stdout:848: read oid 37 snap -1 2024-04-22T18:39:25.021 INFO:tasks.rados.rados.0.smithi089.stdout:848: expect (ObjNum 84 snap 0 seq_num 84) 2024-04-22T18:39:25.021 INFO:tasks.rados.rados.0.smithi089.stdout:849: read oid 31 snap -1 2024-04-22T18:39:25.021 INFO:tasks.rados.rados.0.smithi089.stdout:849: expect (ObjNum 74 snap 0 seq_num 74) 2024-04-22T18:39:25.021 INFO:tasks.rados.rados.0.smithi089.stdout:850: tier_promote oid 42 2024-04-22T18:39:25.021 INFO:tasks.rados.rados.0.smithi089.stdout:851: read oid 20 snap -1 2024-04-22T18:39:25.021 INFO:tasks.rados.rados.0.smithi089.stdout:851: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-22T18:39:25.021 INFO:tasks.rados.rados.0.smithi089.stdout:852: write (excl) oid 26 current snap is 0 2024-04-22T18:39:25.021 INFO:tasks.rados.rados.0.smithi089.stdout:852: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2024-04-22T18:39:25.025 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:25.025 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:25.027 INFO:tasks.rados.rados.0.smithi089.stdout:852: writing smithi08937929-26 from 732805 to 1531735 tid 1 2024-04-22T18:39:25.034 INFO:tasks.rados.rados.0.smithi089.stdout:852: writing smithi08937929-26 from 2296488 to 2867514 tid 2 2024-04-22T18:39:25.035 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:25.039 INFO:tasks.rados.rados.0.smithi089.stdout:852: writing smithi08937929-26 from 3291997 to 3697102 tid 3 2024-04-22T18:39:25.039 INFO:tasks.rados.rados.0.smithi089.stdout:853: set_chunk oid 32 target oid 2024-04-22T18:39:25.106 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:25.107+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:25.141 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:25.141 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:25.150 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:25.227 INFO:tasks.rados.rados.0.smithi089.stdout:853: set_chunk oid 32 offset: 1956352 length: 1397248 target oid b6dc202c5497a143deafe136f3271a6a5661c4c68f17e8460f23448ac72c9d44 offset: 0 snap -1 2024-04-22T18:39:25.227 INFO:tasks.rados.rados.0.smithi089.stdout:854: tier_promote oid 12 2024-04-22T18:39:25.227 INFO:tasks.rados.rados.0.smithi089.stdout:847: done (7 left) 2024-04-22T18:39:25.227 INFO:tasks.rados.rados.0.smithi089.stdout:855: read oid 43 snap -1 2024-04-22T18:39:25.227 INFO:tasks.rados.rados.0.smithi089.stdout:855: expect (ObjNum 51 snap 0 seq_num 51) 2024-04-22T18:39:25.227 INFO:tasks.rados.rados.0.smithi089.stdout:856: read oid 10 snap -1 2024-04-22T18:39:25.227 INFO:tasks.rados.rados.0.smithi089.stdout:856: expect deleted 2024-04-22T18:39:25.227 INFO:tasks.rados.rados.0.smithi089.stdout:857: read oid 38 snap -1 2024-04-22T18:39:25.227 INFO:tasks.rados.rados.0.smithi089.stdout:857: expect (ObjNum 97 snap 0 seq_num 97) 2024-04-22T18:39:25.227 INFO:tasks.rados.rados.0.smithi089.stdout:858: write (excl) oid 35 current snap is 0 2024-04-22T18:39:25.228 INFO:tasks.rados.rados.0.smithi089.stdout:850: got (0) Success 2024-04-22T18:39:25.228 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 16 (ObjNum 105 snap 0 seq_num 105) dirty exists 2024-04-22T18:39:25.228 INFO:tasks.rados.rados.0.smithi089.stdout:851: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-22T18:39:25.228 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:25.228+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:25.272 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:39:25.294 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:25.295 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:25.304 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:25.308 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:25.309+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:25.308 INFO:tasks.rados.rados.0.smithi089.stdout:848: expect (ObjNum 84 snap 0 seq_num 84) 2024-04-22T18:39:25.409 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:25.409+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:25.413 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:25.413 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:25.422 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:25.464 INFO:tasks.rados.rados.0.smithi089.stdout:849: expect (ObjNum 74 snap 0 seq_num 74) 2024-04-22T18:39:25.510 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:25.511+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:25.525 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:25.525 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:25.535 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:25.569 INFO:tasks.rados.rados.0.smithi089.stdout:854: got (0) Success 2024-04-22T18:39:25.570 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 33 (ObjNum 73 snap 0 seq_num 73) dirty exists 2024-04-22T18:39:25.570 INFO:tasks.rados.rados.0.smithi089.stdout:852: finishing write tid 1 to smithi08937929-26 2024-04-22T18:39:25.570 INFO:tasks.rados.rados.0.smithi089.stdout:852: finishing write tid 2 to smithi08937929-26 2024-04-22T18:39:25.570 INFO:tasks.rados.rados.0.smithi089.stdout:852: finishing write tid 3 to smithi08937929-26 2024-04-22T18:39:25.570 INFO:tasks.rados.rados.0.smithi089.stdout:855: expect (ObjNum 51 snap 0 seq_num 51) 2024-04-22T18:39:25.613 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:25.614+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:25.671 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:25.671 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:25.680 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:25.705 INFO:tasks.rados.rados.0.smithi089.stdout:852: finishing write tid 4 to smithi08937929-26 2024-04-22T18:39:25.705 INFO:tasks.rados.rados.0.smithi089.stdout:852: finishing write tid 5 to smithi08937929-26 2024-04-22T18:39:25.706 INFO:tasks.rados.rados.0.smithi089.stdout:852: oid 26 updating version 0 to 44 2024-04-22T18:39:25.706 INFO:tasks.rados.rados.0.smithi089.stdout:852: oid 26 version 44 is already newer than 42 2024-04-22T18:39:25.706 INFO:tasks.rados.rados.0.smithi089.stdout:852: oid 26 updating version 44 to 45 2024-04-22T18:39:25.706 INFO:tasks.rados.rados.0.smithi089.stdout:852: oid 26 version 45 is already newer than 43 2024-04-22T18:39:25.706 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 45 (ObjNum 115 snap 0 seq_num 115) dirty exists 2024-04-22T18:39:25.706 INFO:tasks.rados.rados.0.smithi089.stdout:852: left oid 26 (ObjNum 115 snap 0 seq_num 115) 2024-04-22T18:39:25.706 INFO:tasks.rados.rados.0.smithi089.stdout:857: expect (ObjNum 97 snap 0 seq_num 97) 2024-04-22T18:39:25.748 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:25.750+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:25.795 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:39:25.808 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:25.808 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:25.813 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:25.814+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:25.817 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:25.880 INFO:tasks.rados.rados.0.smithi089.stdout:853: finishing set_chunk to oid 32 2024-04-22T18:39:25.880 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 31 (ObjNum 92 snap 0 seq_num 92) dirty exists 2024-04-22T18:39:25.880 INFO:tasks.rados.rados.0.smithi089.stdout:858: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2024-04-22T18:39:25.887 INFO:tasks.rados.rados.0.smithi089.stdout:858: writing smithi08937929-35 from 749612 to 1223791 tid 1 2024-04-22T18:39:25.894 INFO:tasks.rados.rados.0.smithi089.stdout:858: writing smithi08937929-35 from 1954970 to 2386007 tid 2 2024-04-22T18:39:25.901 INFO:tasks.rados.rados.0.smithi089.stdout:858: writing smithi08937929-35 from 2919420 to 3535797 tid 3 2024-04-22T18:39:25.902 INFO:tasks.rados.rados.0.smithi089.stdout:848: done (10 left) 2024-04-22T18:39:25.902 INFO:tasks.rados.rados.0.smithi089.stdout:849: done (9 left) 2024-04-22T18:39:25.902 INFO:tasks.rados.rados.0.smithi089.stdout:850: done (8 left) 2024-04-22T18:39:25.902 INFO:tasks.rados.rados.0.smithi089.stdout:851: done (7 left) 2024-04-22T18:39:25.902 INFO:tasks.rados.rados.0.smithi089.stdout:852: done (6 left) 2024-04-22T18:39:25.902 INFO:tasks.rados.rados.0.smithi089.stdout:853: done (5 left) 2024-04-22T18:39:25.902 INFO:tasks.rados.rados.0.smithi089.stdout:854: done (4 left) 2024-04-22T18:39:25.902 INFO:tasks.rados.rados.0.smithi089.stdout:855: done (3 left) 2024-04-22T18:39:25.902 INFO:tasks.rados.rados.0.smithi089.stdout:856: done (2 left) 2024-04-22T18:39:25.902 INFO:tasks.rados.rados.0.smithi089.stdout:857: done (1 left) 2024-04-22T18:39:25.902 INFO:tasks.rados.rados.0.smithi089.stdout:859: tier_flush oid 39 2024-04-22T18:39:25.902 INFO:tasks.rados.rados.0.smithi089.stdout:859: tier_flush oid 39 snap -1 2024-04-22T18:39:25.903 INFO:tasks.rados.rados.0.smithi089.stdout:858: finishing write tid 1 to smithi08937929-35 2024-04-22T18:39:25.903 INFO:tasks.rados.rados.0.smithi089.stdout:860: delete oid 34 current snap is 0 2024-04-22T18:39:25.903 INFO:tasks.rados.rados.0.smithi089.stdout:858: finishing write tid 2 to smithi08937929-35 2024-04-22T18:39:25.904 INFO:tasks.rados.rados.0.smithi089.stdout:858: finishing write tid 3 to smithi08937929-35 2024-04-22T18:39:25.913 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:25.915+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:25.919 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:25.919 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:25.936 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:25.936 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:25.953 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:25.953 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:25.965 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:25.965 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:25.965 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:25.965 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:25.969 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:25.969 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:25.974 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:25.986 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:25.987 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:26.004 INFO:tasks.rados.rados.0.smithi089.stdout:860: done (2 left) 2024-04-22T18:39:26.004 INFO:tasks.rados.rados.0.smithi089.stdout:861: read oid 44 snap -1 2024-04-22T18:39:26.004 INFO:tasks.rados.rados.0.smithi089.stdout:861: expect (ObjNum 91 snap 0 seq_num 91) 2024-04-22T18:39:26.004 INFO:tasks.rados.rados.0.smithi089.stdout:862: read oid 4 snap -1 2024-04-22T18:39:26.004 INFO:tasks.rados.rados.0.smithi089.stdout:862: expect (ObjNum 81 snap 0 seq_num 81) 2024-04-22T18:39:26.004 INFO:tasks.rados.rados.0.smithi089.stdout:863: read oid 23 snap -1 2024-04-22T18:39:26.004 INFO:tasks.rados.rados.0.smithi089.stdout:863: expect (ObjNum 82 snap 0 seq_num 82) 2024-04-22T18:39:26.004 INFO:tasks.rados.rados.0.smithi089.stdout:864: set_chunk oid 6 target oid 2024-04-22T18:39:26.014 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:26.016+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:26.028 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:26.028 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:26.044 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:26.045 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:26.061 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:26.061 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:26.092 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:26.092 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:26.092 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:26.092 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:26.092 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:26.101 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:26.115 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:26.116+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:26.134 INFO:tasks.rados.rados.0.smithi089.stdout:864: set_chunk oid 6 offset: 16384 length: 3668480 target oid 5e5f26524298f660b881c4824b4b6ba24a4a32392144f963145aad83dada7759 offset: 0 snap -1 2024-04-22T18:39:26.134 INFO:tasks.rados.rados.0.smithi089.stdout:865: read oid 24 snap -1 2024-04-22T18:39:26.134 INFO:tasks.rados.rados.0.smithi089.stdout:865: expect (ObjNum 112 snap 0 seq_num 112) 2024-04-22T18:39:26.134 INFO:tasks.rados.rados.0.smithi089.stdout:866: read oid 33 snap -1 2024-04-22T18:39:26.134 INFO:tasks.rados.rados.0.smithi089.stdout:866: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-22T18:39:26.134 INFO:tasks.rados.rados.0.smithi089.stdout:867: delete oid 34 current snap is 0 2024-04-22T18:39:26.134 INFO:tasks.rados.rados.0.smithi089.stdout:861: expect (ObjNum 91 snap 0 seq_num 91) 2024-04-22T18:39:26.216 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:26.217+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:26.234 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:26.235+0000 7fec57bee640 -1 osd.7 45 handle_osd_map injecting map crc failure 2024-04-22T18:39:26.237 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:39:26.242 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:26.242 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:26.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:26.242 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:26.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.242 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:27 4.d e27/20 4:bee1184f:::smithi08937929-7:head v 27'3, mlcod=0'0)", 2024-04-22T18:39:26.242 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:26.242 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.877088885999999, 2024-04-22T18:39:26.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101135974, 2024-04-22T18:39:26.242 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.243 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.243 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:26.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:26.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365186+0000", 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365194+0000", 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3289999999999994e-06 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365200+0000", 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6590000000000004e-06 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465352+0000", 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10015208 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465511+0000", 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015880000000000001 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465570+0000", 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8952000000000001e-05 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466292+0000", 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072212100000000005 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466323+0000", 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0899999999999999e-05 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:47 3.0 e30/19 3:30f16451:::smithi08937929-50 low_tier:head v 30'20, mlcod=29'18)", 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.644172868999998, 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100803438, 2024-04-22T18:39:26.246 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.247 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.247 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:26.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:26.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598108+0000", 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9119999999999997e-06 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598118+0000", 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0138e-05 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598125+0000", 2024-04-22T18:39:26.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3010000000000001e-06 2024-04-22T18:39:26.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698224+0000", 2024-04-22T18:39:26.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099221 2024-04-22T18:39:26.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698239+0000", 2024-04-22T18:39:26.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5053e-05 2024-04-22T18:39:26.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698875+0000", 2024-04-22T18:39:26.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063560699999999995 2024-04-22T18:39:26.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698906+0000", 2024-04-22T18:39:26.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1547999999999997e-05 2024-04-22T18:39:26.250 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.250 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.250 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.250 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.5.13:5 3.0 e30/19 3:1ab765eb:::smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=30'21)", 2024-04-22T18:39:26.250 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:26.251 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.039854746, 2024-04-22T18:39:26.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101014042, 2024-04-22T18:39:26.251 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.251 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.251 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:26.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:26.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:26.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:26.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202428+0000", 2024-04-22T18:39:26.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5889999999999997e-06 2024-04-22T18:39:26.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202446+0000", 2024-04-22T18:39:26.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8176999999999999e-05 2024-04-22T18:39:26.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202456+0000", 2024-04-22T18:39:26.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-04-22T18:39:26.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302599+0000", 2024-04-22T18:39:26.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100142759 2024-04-22T18:39:26.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302628+0000", 2024-04-22T18:39:26.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9507999999999999e-05 2024-04-22T18:39:26.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303350+0000", 2024-04-22T18:39:26.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072151599999999995 2024-04-22T18:39:26.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303435+0000", 2024-04-22T18:39:26.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5486000000000001e-05 2024-04-22T18:39:26.254 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.255 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.255 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.255 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:53 3.0 e31/19 3:25a201b0:::smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 31'24, mlcod=30'22)", 2024-04-22T18:39:26.255 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:26.255 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.839212411999998, 2024-04-22T18:39:26.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100751671, 2024-04-22T18:39:26.255 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.255 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.255 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:26.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:26.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403062+0000", 2024-04-22T18:39:26.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:26.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403070+0000", 2024-04-22T18:39:26.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8499999999999994e-06 2024-04-22T18:39:26.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403079+0000", 2024-04-22T18:39:26.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4620000000000002e-06 2024-04-22T18:39:26.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403087+0000", 2024-04-22T18:39:26.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4789999999999999e-06 2024-04-22T18:39:26.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503186+0000", 2024-04-22T18:39:26.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099271 2024-04-22T18:39:26.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503207+0000", 2024-04-22T18:39:26.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0973999999999999e-05 2024-04-22T18:39:26.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503773+0000", 2024-04-22T18:39:26.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056557600000000003 2024-04-22T18:39:26.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503815+0000", 2024-04-22T18:39:26.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2441999999999998e-05 2024-04-22T18:39:26.259 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.259 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.259 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.259 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:73 3.0 e34/19 3:061a7a63:::392f44ec26aa22e85be5f08273a7e45656f3bf4a4854038391a88e880ec01077:head v 34'31, mlcod=31'29)", 2024-04-22T18:39:26.259 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:26.259 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.784348963999999, 2024-04-22T18:39:26.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103503311, 2024-04-22T18:39:26.260 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.260 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.260 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:26.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:26.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:26.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:26.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458193+0000", 2024-04-22T18:39:26.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026628900000000001 2024-04-22T18:39:26.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458208+0000", 2024-04-22T18:39:26.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4780000000000001e-05 2024-04-22T18:39:26.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458217+0000", 2024-04-22T18:39:26.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4909999999999994e-06 2024-04-22T18:39:26.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459162+0000", 2024-04-22T18:39:26.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094440500000000005 2024-04-22T18:39:26.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459185+0000", 2024-04-22T18:39:26.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3685000000000001e-05 2024-04-22T18:39:26.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561394+0000", 2024-04-22T18:39:26.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10220874100000001 2024-04-22T18:39:26.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561430+0000", 2024-04-22T18:39:26.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6304999999999999e-05 2024-04-22T18:39:26.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.263 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.264 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:75 3.0 e34/19 3:18418e0f:::b7c35f165c374d5e075dde9ac2607e47ace3b0a1e2343ec6efb171380f725eea:head v 34'32, mlcod=31'29)", 2024-04-22T18:39:26.264 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:26.264 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.783335691000001, 2024-04-22T18:39:26.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104798078, 2024-04-22T18:39:26.264 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.264 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.264 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:26.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:26.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458939+0000", 2024-04-22T18:39:26.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:26.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459092+0000", 2024-04-22T18:39:26.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153838 2024-04-22T18:39:26.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459107+0000", 2024-04-22T18:39:26.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4703e-05 2024-04-22T18:39:26.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459115+0000", 2024-04-22T18:39:26.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-04-22T18:39:26.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560093+0000", 2024-04-22T18:39:26.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10097840499999999 2024-04-22T18:39:26.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560109+0000", 2024-04-22T18:39:26.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5631e-05 2024-04-22T18:39:26.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563706+0000", 2024-04-22T18:39:26.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035972059999999999 2024-04-22T18:39:26.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563738+0000", 2024-04-22T18:39:26.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2184000000000002e-05 2024-04-22T18:39:26.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.268 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.268 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:77 3.0 e34/19 3:28387c9f:::b6d7840f02d561da04e398fbf94d91b85e9066b426e48baf3b11dd4a4f6096c0:head v 34'33, mlcod=31'29)", 2024-04-22T18:39:26.268 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:26.268 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.782385610999999, 2024-04-22T18:39:26.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104186079, 2024-04-22T18:39:26.268 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.269 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.269 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:26.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:26.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459889+0000", 2024-04-22T18:39:26.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:26.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459963+0000", 2024-04-22T18:39:26.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3732999999999994e-05 2024-04-22T18:39:26.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459973+0000", 2024-04-22T18:39:26.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0285999999999999e-05 2024-04-22T18:39:26.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459987+0000", 2024-04-22T18:39:26.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3675999999999999e-05 2024-04-22T18:39:26.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560515+0000", 2024-04-22T18:39:26.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10052781299999999 2024-04-22T18:39:26.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560538+0000", 2024-04-22T18:39:26.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3861000000000001e-05 2024-04-22T18:39:26.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564044+0000", 2024-04-22T18:39:26.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035052009999999999 2024-04-22T18:39:26.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564076+0000", 2024-04-22T18:39:26.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2737000000000002e-05 2024-04-22T18:39:26.272 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.272 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.272 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.272 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:78 3.0 e34/19 3:3a5582ac:::bdd92f7deaa2d50e204891f64f9043469ba687d78b218ed518f2483ba3486a99:head v 34'34, mlcod=31'29)", 2024-04-22T18:39:26.273 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:26.273 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.781719493000001, 2024-04-22T18:39:26.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103547263, 2024-04-22T18:39:26.273 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.273 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.273 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:26.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:26.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460555+0000", 2024-04-22T18:39:26.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:26.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460861+0000", 2024-04-22T18:39:26.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000305958 2024-04-22T18:39:26.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460874+0000", 2024-04-22T18:39:26.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2639e-05 2024-04-22T18:39:26.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460881+0000", 2024-04-22T18:39:26.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2570000000000001e-06 2024-04-22T18:39:26.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560990+0000", 2024-04-22T18:39:26.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010883199999999 2024-04-22T18:39:26.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561011+0000", 2024-04-22T18:39:26.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1319e-05 2024-04-22T18:39:26.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564079+0000", 2024-04-22T18:39:26.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00306815 2024-04-22T18:39:26.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564104+0000", 2024-04-22T18:39:26.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4172000000000002e-05 2024-04-22T18:39:26.277 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.277 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.277 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.277 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:39:26.277 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:26.277 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.781104199000001, 2024-04-22T18:39:26.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:39:26.277 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.277 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.277 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:26.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:26.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:39:26.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:26.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:39:26.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:39:26.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:39:26.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:39:26.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:39:26.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:39:26.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:39:26.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:39:26.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:39:26.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:39:26.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:39:26.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:39:26.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:39:26.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:39:26.281 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.281 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.281 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.281 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:39:26.281 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:26.281 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.904430377000001, 2024-04-22T18:39:26.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:39:26.282 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.282 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.282 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:26.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:26.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:39:26.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:26.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:39:26.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:39:26.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:39:26.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:39:26.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:39:26.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:39:26.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:39:26.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:39:26.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:39:26.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:39:26.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:39:26.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:39:26.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:39:26.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:39:26.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.285 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.286 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.286 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1112 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'6, mlcod=29'4)", 2024-04-22T18:39:26.286 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:12.585713+0000", 2024-04-22T18:39:26.286 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.656562470000001, 2024-04-22T18:39:26.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046198259999999996, 2024-04-22T18:39:26.286 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.286 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.286 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.585713+0000", 2024-04-22T18:39:26.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.585713+0000", 2024-04-22T18:39:26.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.585712+0000", 2024-04-22T18:39:26.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:26.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586020+0000", 2024-04-22T18:39:26.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030764900000000001 2024-04-22T18:39:26.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586031+0000", 2024-04-22T18:39:26.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0883000000000001e-05 2024-04-22T18:39:26.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586037+0000", 2024-04-22T18:39:26.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1260000000000003e-06 2024-04-22T18:39:26.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586076+0000", 2024-04-22T18:39:26.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8841999999999998e-05 2024-04-22T18:39:26.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586098+0000", 2024-04-22T18:39:26.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2279000000000001e-05 2024-04-22T18:39:26.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.590302+0000", 2024-04-22T18:39:26.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042039419999999996 2024-04-22T18:39:26.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.590333+0000", 2024-04-22T18:39:26.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1170000000000001e-05 2024-04-22T18:39:26.290 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.290 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.290 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.290 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1131 3.0 e41/19 3:01bdda42:::4a159391123755d93670b0d142ca36f3ee9a35236f6c4396c2aa92521a42a06b:head v 41'50, mlcod=41'48)", 2024-04-22T18:39:26.290 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:26.290 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.064683532, 2024-04-22T18:39:26.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101639456, 2024-04-22T18:39:26.290 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.291 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.291 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:26.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:26.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177590+0000", 2024-04-22T18:39:26.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:26.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177880+0000", 2024-04-22T18:39:26.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029001 2024-04-22T18:39:26.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177893+0000", 2024-04-22T18:39:26.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-04-22T18:39:26.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177899+0000", 2024-04-22T18:39:26.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1619999999999996e-06 2024-04-22T18:39:26.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.278054+0000", 2024-04-22T18:39:26.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100154929 2024-04-22T18:39:26.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.278076+0000", 2024-04-22T18:39:26.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1308e-05 2024-04-22T18:39:26.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.279187+0000", 2024-04-22T18:39:26.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011109169999999999 2024-04-22T18:39:26.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.279232+0000", 2024-04-22T18:39:26.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5188000000000002e-05 2024-04-22T18:39:26.294 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.294 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.294 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.295 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1159 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'10, mlcod=41'7)", 2024-04-22T18:39:26.295 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.452702+0000", 2024-04-22T18:39:26.295 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.789573982, 2024-04-22T18:39:26.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050095280000000001, 2024-04-22T18:39:26.295 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.295 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.295 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452702+0000", 2024-04-22T18:39:26.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452702+0000", 2024-04-22T18:39:26.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452701+0000", 2024-04-22T18:39:26.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:26.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452976+0000", 2024-04-22T18:39:26.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027566 2024-04-22T18:39:26.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452989+0000", 2024-04-22T18:39:26.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2615e-05 2024-04-22T18:39:26.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452998+0000", 2024-04-22T18:39:26.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7399999999999993e-06 2024-04-22T18:39:26.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.453040+0000", 2024-04-22T18:39:26.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2395000000000003e-05 2024-04-22T18:39:26.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.453060+0000", 2024-04-22T18:39:26.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0332000000000001e-05 2024-04-22T18:39:26.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.457682+0000", 2024-04-22T18:39:26.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046215010000000001 2024-04-22T18:39:26.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.457711+0000", 2024-04-22T18:39:26.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9490999999999999e-05 2024-04-22T18:39:26.299 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.299 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.299 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.299 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1160 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'11, mlcod=41'8)", 2024-04-22T18:39:26.299 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.555212+0000", 2024-04-22T18:39:26.299 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.687063649000001, 2024-04-22T18:39:26.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004833586, 2024-04-22T18:39:26.299 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.299 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.299 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555212+0000", 2024-04-22T18:39:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555212+0000", 2024-04-22T18:39:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555209+0000", 2024-04-22T18:39:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555369+0000", 2024-04-22T18:39:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015995000000000001 2024-04-22T18:39:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555383+0000", 2024-04-22T18:39:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3558e-05 2024-04-22T18:39:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555389+0000", 2024-04-22T18:39:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2759999999999997e-06 2024-04-22T18:39:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556393+0000", 2024-04-22T18:39:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001003629 2024-04-22T18:39:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556418+0000", 2024-04-22T18:39:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4915000000000001e-05 2024-04-22T18:39:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.560010+0000", 2024-04-22T18:39:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003591782 2024-04-22T18:39:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.560046+0000", 2024-04-22T18:39:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6285000000000002e-05 2024-04-22T18:39:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1161 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'12, mlcod=41'9)", 2024-04-22T18:39:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.556330+0000", 2024-04-22T18:39:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.685946230999999, 2024-04-22T18:39:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099789016999999994, 2024-04-22T18:39:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556330+0000", 2024-04-22T18:39:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556330+0000", 2024-04-22T18:39:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556328+0000", 2024-04-22T18:39:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556335+0000", 2024-04-22T18:39:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9770000000000003e-06 2024-04-22T18:39:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556346+0000", 2024-04-22T18:39:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0118999999999999e-05 2024-04-22T18:39:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556353+0000", 2024-04-22T18:39:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3730000000000004e-06 2024-04-22T18:39:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.655508+0000", 2024-04-22T18:39:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099155219000000003 2024-04-22T18:39:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.655536+0000", 2024-04-22T18:39:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-05 2024-04-22T18:39:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.656078+0000", 2024-04-22T18:39:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054138999999999999 2024-04-22T18:39:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.656119+0000", 2024-04-22T18:39:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1026000000000003e-05 2024-04-22T18:39:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:125 3.0 e43/19 3:376bc4bb:::8f967acf6e5865f87cd7cada681f96ff59d5fc3f243a710c00c052aacb952df3:head v 43'60, mlcod=42'57)", 2024-04-22T18:39:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.3947101499999999, 2024-04-22T18:39:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100905994, 2024-04-22T18:39:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847565+0000", 2024-04-22T18:39:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847610+0000", 2024-04-22T18:39:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4984000000000002e-05 2024-04-22T18:39:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847619+0000", 2024-04-22T18:39:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2189999999999993e-06 2024-04-22T18:39:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947667+0000", 2024-04-22T18:39:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100047939 2024-04-22T18:39:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947716+0000", 2024-04-22T18:39:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9228999999999999e-05 2024-04-22T18:39:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947738+0000", 2024-04-22T18:39:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1963999999999998e-05 2024-04-22T18:39:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.948440+0000", 2024-04-22T18:39:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070173999999999996 2024-04-22T18:39:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.948472+0000", 2024-04-22T18:39:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1872999999999998e-05 2024-04-22T18:39:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1296 4.d e43/20 4:bc165c76:::smithi08937929-30:head v 43'15, mlcod=32'12)", 2024-04-22T18:39:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:18.060039+0000", 2024-04-22T18:39:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.1822371510000007, 2024-04-22T18:39:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038697890000000002, 2024-04-22T18:39:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060039+0000", 2024-04-22T18:39:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060039+0000", 2024-04-22T18:39:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060037+0000", 2024-04-22T18:39:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060685+0000", 2024-04-22T18:39:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00064793499999999998 2024-04-22T18:39:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060697+0000", 2024-04-22T18:39:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2653000000000001e-05 2024-04-22T18:39:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060705+0000", 2024-04-22T18:39:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5320000000000001e-06 2024-04-22T18:39:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062088+0000", 2024-04-22T18:39:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001383287 2024-04-22T18:39:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062105+0000", 2024-04-22T18:39:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6595e-05 2024-04-22T18:39:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.063879+0000", 2024-04-22T18:39:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017744379999999999 2024-04-22T18:39:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.063909+0000", 2024-04-22T18:39:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9286e-05 2024-04-22T18:39:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1297 4.d e43/20 4:bc165c76:::smithi08937929-30:head v 43'16, mlcod=32'12)", 2024-04-22T18:39:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:18.060713+0000", 2024-04-22T18:39:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.181562971, 2024-04-22T18:39:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0034270920000000001, 2024-04-22T18:39:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060713+0000", 2024-04-22T18:39:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060713+0000", 2024-04-22T18:39:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.060712+0000", 2024-04-22T18:39:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.061996+0000", 2024-04-22T18:39:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012832869999999999 2024-04-22T18:39:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062002+0000", 2024-04-22T18:39:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6080000000000001e-06 2024-04-22T18:39:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062007+0000", 2024-04-22T18:39:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5210000000000004e-06 2024-04-22T18:39:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062638+0000", 2024-04-22T18:39:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063081699999999999 2024-04-22T18:39:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062654+0000", 2024-04-22T18:39:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6235e-05 2024-04-22T18:39:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.064126+0000", 2024-04-22T18:39:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0014719749999999999 2024-04-22T18:39:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.064140+0000", 2024-04-22T18:39:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4129e-05 2024-04-22T18:39:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1298 4.d e43/20 4:bc165c76:::smithi08937929-30:head v 43'17, mlcod=32'12)", 2024-04-22T18:39:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:18.062040+0000", 2024-04-22T18:39:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.1802363469999992, 2024-04-22T18:39:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099368133999999997, 2024-04-22T18:39:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062040+0000", 2024-04-22T18:39:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062040+0000", 2024-04-22T18:39:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062039+0000", 2024-04-22T18:39:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062056+0000", 2024-04-22T18:39:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6911000000000001e-05 2024-04-22T18:39:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062060+0000", 2024-04-22T18:39:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3000000000000003e-06 2024-04-22T18:39:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062065+0000", 2024-04-22T18:39:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1150000000000002e-06 2024-04-22T18:39:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.160817+0000", 2024-04-22T18:39:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098751539999999999 2024-04-22T18:39:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.160845+0000", 2024-04-22T18:39:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7908000000000001e-05 2024-04-22T18:39:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.161378+0000", 2024-04-22T18:39:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053300799999999996 2024-04-22T18:39:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.161408+0000", 2024-04-22T18:39:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.989e-05 2024-04-22T18:39:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:79 3.0 e44/19 3:05e6fa13:::smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 44'64, mlcod=44'61)", 2024-04-22T18:39:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.8714288539999999, 2024-04-22T18:39:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10093532199999999, 2024-04-22T18:39:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370846+0000", 2024-04-22T18:39:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370857+0000", 2024-04-22T18:39:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1155000000000001e-05 2024-04-22T18:39:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370866+0000", 2024-04-22T18:39:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8769999999999999e-06 2024-04-22T18:39:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.470937+0000", 2024-04-22T18:39:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100071204 2024-04-22T18:39:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.470986+0000", 2024-04-22T18:39:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8556000000000001e-05 2024-04-22T18:39:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471014+0000", 2024-04-22T18:39:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7982000000000001e-05 2024-04-22T18:39:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471750+0000", 2024-04-22T18:39:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073598100000000003 2024-04-22T18:39:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471782+0000", 2024-04-22T18:39:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339000000000003e-05 2024-04-22T18:39:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.329 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:26.330 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:26.285+0000 7fa766d7d640 -1 osd.5 45 handle_osd_map injecting map crc failure 2024-04-22T18:39:26.330 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:26.286+0000 7fa766d7d640 -1 osd.5 45 handle_osd_map injecting map crc failure 2024-04-22T18:39:26.331 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:26.331+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:26.331 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:26.386 INFO:tasks.rados.rados.0.smithi089.stdout:858: finishing write tid 4 to smithi08937929-35 2024-04-22T18:39:26.386 INFO:tasks.rados.rados.0.smithi089.stdout:858: finishing write tid 5 to smithi08937929-35 2024-04-22T18:39:26.386 INFO:tasks.rados.rados.0.smithi089.stdout:858: oid 35 updating version 0 to 48 2024-04-22T18:39:26.386 INFO:tasks.rados.rados.0.smithi089.stdout:858: oid 35 updating version 48 to 49 2024-04-22T18:39:26.386 INFO:tasks.rados.rados.0.smithi089.stdout:858: oid 35 version 49 is already newer than 47 2024-04-22T18:39:26.386 INFO:tasks.rados.rados.0.smithi089.stdout:858: oid 35 updating version 49 to 50 2024-04-22T18:39:26.387 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 50 (ObjNum 116 snap 0 seq_num 116) dirty exists 2024-04-22T18:39:26.387 INFO:tasks.rados.rados.0.smithi089.stdout:858: left oid 35 (ObjNum 116 snap 0 seq_num 116) 2024-04-22T18:39:26.387 INFO:tasks.rados.rados.0.smithi089.stdout:862: expect (ObjNum 81 snap 0 seq_num 81) 2024-04-22T18:39:26.431 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:26.432+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:26.475 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:26.475 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:26.476 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:26.484 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:26.533 INFO:tasks.rados.rados.0.smithi089.stdout:864: finishing set_chunk to oid 6 2024-04-22T18:39:26.533 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 6 set_chunk returned error code -95 offset: 16384 length: 3668480 2024-04-22T18:39:26.533 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 0 (ObjNum 113 snap 0 seq_num 113) dirty exists 2024-04-22T18:39:26.533 INFO:tasks.rados.rados.0.smithi089.stdout:866: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-22T18:39:26.533 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:26.534+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:26.626 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:26.626 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:26.626 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:26.626 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:26.626 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:26.632 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:26.633+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:26.637 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:26.704 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:39:26.704 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:39:26.705 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-04-22T18:39:26.705 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-22T18:39:26.705 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-22T18:39:26.705 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-04-22T18:39:26.708 INFO:tasks.rados.rados.0.smithi089.stdout:863: expect (ObjNum 82 snap 0 seq_num 82) 2024-04-22T18:39:26.732 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:26.733+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:26.785 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:26.785 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:26.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:26.785 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:26.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.785 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:6 4.f e24/20 4:f83a159b:::smithi08937929-2:head v 24'2, mlcod=0'0)", 2024-04-22T18:39:26.785 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:26.785 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.579024840000002, 2024-04-22T18:39:26.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10183012700000001, 2024-04-22T18:39:26.786 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.786 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.786 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:26.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:26.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.205790+0000", 2024-04-22T18:39:26.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:26.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.206027+0000", 2024-04-22T18:39:26.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023705500000000001 2024-04-22T18:39:26.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.206038+0000", 2024-04-22T18:39:26.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.148e-05 2024-04-22T18:39:26.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.206046+0000", 2024-04-22T18:39:26.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8679999999999999e-06 2024-04-22T18:39:26.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.306227+0000", 2024-04-22T18:39:26.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100181223 2024-04-22T18:39:26.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.306266+0000", 2024-04-22T18:39:26.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8906000000000003e-05 2024-04-22T18:39:26.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.307598+0000", 2024-04-22T18:39:26.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001331561 2024-04-22T18:39:26.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.307620+0000", 2024-04-22T18:39:26.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.226e-05 2024-04-22T18:39:26.789 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.789 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.790 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.790 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:45 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'1, mlcod=0'0)", 2024-04-22T18:39:26.790 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:26.790 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.519703153999998, 2024-04-22T18:39:26.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14345418900000001, 2024-04-22T18:39:26.790 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.790 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.790 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:26.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:26.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:26.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:26.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265727+0000", 2024-04-22T18:39:26.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061502799999999999 2024-04-22T18:39:26.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265737+0000", 2024-04-22T18:39:26.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0278999999999999e-05 2024-04-22T18:39:26.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265744+0000", 2024-04-22T18:39:26.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1740000000000003e-06 2024-04-22T18:39:26.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407227+0000", 2024-04-22T18:39:26.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14148266900000001 2024-04-22T18:39:26.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407258+0000", 2024-04-22T18:39:26.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1619000000000003e-05 2024-04-22T18:39:26.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408534+0000", 2024-04-22T18:39:26.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001275829 2024-04-22T18:39:26.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408566+0000", 2024-04-22T18:39:26.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1813999999999997e-05 2024-04-22T18:39:26.794 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.794 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.794 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.794 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:46 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'2, mlcod=0'0)", 2024-04-22T18:39:26.794 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:26.794 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.517078751, 2024-04-22T18:39:26.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14304645699999999, 2024-04-22T18:39:26.794 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.794 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.795 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:26.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:26.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:26.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:26.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268301+0000", 2024-04-22T18:39:26.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056585700000000001 2024-04-22T18:39:26.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268309+0000", 2024-04-22T18:39:26.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2830000000000004e-06 2024-04-22T18:39:26.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268312+0000", 2024-04-22T18:39:26.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7179999999999998e-06 2024-04-22T18:39:26.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407999+0000", 2024-04-22T18:39:26.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13968636300000001 2024-04-22T18:39:26.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408011+0000", 2024-04-22T18:39:26.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2116999999999999e-05 2024-04-22T18:39:26.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410762+0000", 2024-04-22T18:39:26.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002751524 2024-04-22T18:39:26.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410783+0000", 2024-04-22T18:39:26.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0270000000000001e-05 2024-04-22T18:39:26.798 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.798 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.798 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.799 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:47 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'3, mlcod=0'0)", 2024-04-22T18:39:26.799 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:26.799 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.515346180999998, 2024-04-22T18:39:26.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14132698599999999, 2024-04-22T18:39:26.799 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.799 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.799 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:26.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:26.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269468+0000", 2024-04-22T18:39:26.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:26.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269473+0000", 2024-04-22T18:39:26.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7790000000000003e-06 2024-04-22T18:39:26.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269478+0000", 2024-04-22T18:39:26.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0080000000000002e-06 2024-04-22T18:39:26.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269489+0000", 2024-04-22T18:39:26.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.075e-05 2024-04-22T18:39:26.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408595+0000", 2024-04-22T18:39:26.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13910662300000001 2024-04-22T18:39:26.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408611+0000", 2024-04-22T18:39:26.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6124e-05 2024-04-22T18:39:26.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410786+0000", 2024-04-22T18:39:26.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021743819999999999 2024-04-22T18:39:26.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410796+0000", 2024-04-22T18:39:26.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.007e-05 2024-04-22T18:39:26.803 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.803 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.803 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.803 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:10 4.f e27/20 4:f2c48a0e:::smithi08937929-3:head v 27'5, mlcod=24'2)", 2024-04-22T18:39:26.803 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:26.803 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.342733611, 2024-04-22T18:39:26.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10161616800000001, 2024-04-22T18:39:26.803 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.803 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.803 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:26.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:26.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442081+0000", 2024-04-22T18:39:26.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:26.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442159+0000", 2024-04-22T18:39:26.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7935000000000005e-05 2024-04-22T18:39:26.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442163+0000", 2024-04-22T18:39:26.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8430000000000003e-06 2024-04-22T18:39:26.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442166+0000", 2024-04-22T18:39:26.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0759999999999999e-06 2024-04-22T18:39:26.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542745+0000", 2024-04-22T18:39:26.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10057941200000001 2024-04-22T18:39:26.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542770+0000", 2024-04-22T18:39:26.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5134e-05 2024-04-22T18:39:26.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543679+0000", 2024-04-22T18:39:26.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00090833099999999996 2024-04-22T18:39:26.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543697+0000", 2024-04-22T18:39:26.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8876e-05 2024-04-22T18:39:26.807 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.807 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.807 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.808 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:11 4.f e27/20 4:f2c48a0e:::smithi08937929-3:head v 27'6, mlcod=24'2)", 2024-04-22T18:39:26.808 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.442170+0000", 2024-04-22T18:39:26.808 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.342645197000003, 2024-04-22T18:39:26.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101722358, 2024-04-22T18:39:26.808 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.808 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.808 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442170+0000", 2024-04-22T18:39:26.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442170+0000", 2024-04-22T18:39:26.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442169+0000", 2024-04-22T18:39:26.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:26.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442174+0000", 2024-04-22T18:39:26.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0350000000000003e-06 2024-04-22T18:39:26.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442177+0000", 2024-04-22T18:39:26.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0580000000000002e-06 2024-04-22T18:39:26.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.442181+0000", 2024-04-22T18:39:26.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1649999999999999e-06 2024-04-22T18:39:26.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543319+0000", 2024-04-22T18:39:26.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101138408 2024-04-22T18:39:26.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543336+0000", 2024-04-22T18:39:26.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6509999999999999e-05 2024-04-22T18:39:26.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543880+0000", 2024-04-22T18:39:26.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054429900000000004 2024-04-22T18:39:26.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.543892+0000", 2024-04-22T18:39:26.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2138e-05 2024-04-22T18:39:26.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.812 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.812 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:77 4.0 4:01baf5a9:::smithi08937929-20:head [write 719333~462437,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:26.812 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.505878+0000", 2024-04-22T18:39:26.812 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.278937396000003, 2024-04-22T18:39:26.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10366277, 2024-04-22T18:39:26.813 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.813 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.813 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:26.813 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:26.813 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:26.813 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 77 2024-04-22T18:39:26.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.813 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.505878+0000", 2024-04-22T18:39:26.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.505878+0000", 2024-04-22T18:39:26.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.505876+0000", 2024-04-22T18:39:26.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:26.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506122+0000", 2024-04-22T18:39:26.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024626900000000002 2024-04-22T18:39:26.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506132+0000", 2024-04-22T18:39:26.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0216e-05 2024-04-22T18:39:26.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506139+0000", 2024-04-22T18:39:26.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6030000000000001e-06 2024-04-22T18:39:26.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506191+0000", 2024-04-22T18:39:26.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.185e-05 2024-04-22T18:39:26.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506333+0000", 2024-04-22T18:39:26.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014197299999999999 2024-04-22T18:39:26.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:26.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.506480+0000", 2024-04-22T18:39:26.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014780199999999999 2024-04-22T18:39:26.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:26.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.507794+0000", 2024-04-22T18:39:26.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001313587 2024-04-22T18:39:26.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:26.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609429+0000", 2024-04-22T18:39:26.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101634658 2024-04-22T18:39:26.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609493+0000", 2024-04-22T18:39:26.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4777999999999995e-05 2024-04-22T18:39:26.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609540+0000", 2024-04-22T18:39:26.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6820000000000002e-05 2024-04-22T18:39:26.818 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.818 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.818 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.818 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:78 4.0 4:01baf5a9:::smithi08937929-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:26.818 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:26.819 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.272880059999999, 2024-04-22T18:39:26.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.398522192, 2024-04-22T18:39:26.819 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.819 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.819 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:26.819 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:26.819 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:26.819 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 78 2024-04-22T18:39:26.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.819 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:26.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:26.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511934+0000", 2024-04-22T18:39:26.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:26.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512012+0000", 2024-04-22T18:39:26.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8329000000000002e-05 2024-04-22T18:39:26.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512023+0000", 2024-04-22T18:39:26.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0755e-05 2024-04-22T18:39:26.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512028+0000", 2024-04-22T18:39:26.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6470000000000003e-06 2024-04-22T18:39:26.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512068+0000", 2024-04-22T18:39:26.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9721999999999997e-05 2024-04-22T18:39:26.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512125+0000", 2024-04-22T18:39:26.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6564999999999998e-05 2024-04-22T18:39:26.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:26.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512200+0000", 2024-04-22T18:39:26.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5708000000000001e-05 2024-04-22T18:39:26.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:26.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513131+0000", 2024-04-22T18:39:26.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093063 2024-04-22T18:39:26.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:26.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910394+0000", 2024-04-22T18:39:26.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.397263388 2024-04-22T18:39:26.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910430+0000", 2024-04-22T18:39:26.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5485999999999999e-05 2024-04-22T18:39:26.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910457+0000", 2024-04-22T18:39:26.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7086000000000001e-05 2024-04-22T18:39:26.824 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.824 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.824 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.824 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:79 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:26.824 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:26.824 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.272779403999998, 2024-04-22T18:39:26.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39855419399999997, 2024-04-22T18:39:26.824 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.825 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.825 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:26.825 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:26.825 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:26.825 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 79 2024-04-22T18:39:26.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.825 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:26.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:26.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512035+0000", 2024-04-22T18:39:26.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:26.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512045+0000", 2024-04-22T18:39:26.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0379e-05 2024-04-22T18:39:26.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512051+0000", 2024-04-22T18:39:26.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6030000000000004e-06 2024-04-22T18:39:26.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512057+0000", 2024-04-22T18:39:26.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7270000000000004e-06 2024-04-22T18:39:26.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512756+0000", 2024-04-22T18:39:26.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069954000000000001 2024-04-22T18:39:26.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512806+0000", 2024-04-22T18:39:26.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9922000000000001e-05 2024-04-22T18:39:26.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:26.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512927+0000", 2024-04-22T18:39:26.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012059 2024-04-22T18:39:26.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:26.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.515883+0000", 2024-04-22T18:39:26.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029563739999999999 2024-04-22T18:39:26.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:26.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910510+0000", 2024-04-22T18:39:26.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39462745199999999 2024-04-22T18:39:26.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910542+0000", 2024-04-22T18:39:26.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1225e-05 2024-04-22T18:39:26.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910590+0000", 2024-04-22T18:39:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8052e-05 2024-04-22T18:39:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:80 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.272754302999999, 2024-04-22T18:39:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39866376999999997, 2024-04-22T18:39:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 80 2024-04-22T18:39:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512060+0000", 2024-04-22T18:39:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512063+0000", 2024-04-22T18:39:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.999e-06 2024-04-22T18:39:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512069+0000", 2024-04-22T18:39:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.891e-06 2024-04-22T18:39:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512073+0000", 2024-04-22T18:39:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4429999999999996e-06 2024-04-22T18:39:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513084+0000", 2024-04-22T18:39:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010102609999999999 2024-04-22T18:39:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513123+0000", 2024-04-22T18:39:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9237999999999999e-05 2024-04-22T18:39:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910618+0000", 2024-04-22T18:39:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39749549299999998 2024-04-22T18:39:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910670+0000", 2024-04-22T18:39:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1221000000000003e-05 2024-04-22T18:39:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910724+0000", 2024-04-22T18:39:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4777000000000001e-05 2024-04-22T18:39:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:33 4.f e27/20 4:f0ed3d49:::smithi08937929-9:head v 27'7, mlcod=24'2)", 2024-04-22T18:39:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.609449+0000", 2024-04-22T18:39:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.175366369000002, 2024-04-22T18:39:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10183384600000001, 2024-04-22T18:39:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609449+0000", 2024-04-22T18:39:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609449+0000", 2024-04-22T18:39:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609448+0000", 2024-04-22T18:39:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609626+0000", 2024-04-22T18:39:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017809600000000001 2024-04-22T18:39:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609638+0000", 2024-04-22T18:39:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2045000000000001e-05 2024-04-22T18:39:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.709771+0000", 2024-04-22T18:39:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013347 2024-04-22T18:39:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.710008+0000", 2024-04-22T18:39:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023696200000000001 2024-04-22T18:39:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.710045+0000", 2024-04-22T18:39:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6903000000000002e-05 2024-04-22T18:39:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.711263+0000", 2024-04-22T18:39:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012178110000000001 2024-04-22T18:39:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.711282+0000", 2024-04-22T18:39:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9525e-05 2024-04-22T18:39:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:5 3.2 e30/29 3:663b82a1:::smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=0'0)", 2024-04-22T18:39:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.557314985000001, 2024-04-22T18:39:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20090134500000001, 2024-04-22T18:39:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227499+0000", 2024-04-22T18:39:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227506+0000", 2024-04-22T18:39:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6540000000000002e-06 2024-04-22T18:39:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227516+0000", 2024-04-22T18:39:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0054000000000001e-05 2024-04-22T18:39:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327607+0000", 2024-04-22T18:39:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009167099999999 2024-04-22T18:39:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327688+0000", 2024-04-22T18:39:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1000000000000004e-05 2024-04-22T18:39:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327719+0000", 2024-04-22T18:39:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0664000000000003e-05 2024-04-22T18:39:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428371+0000", 2024-04-22T18:39:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10065239099999999 2024-04-22T18:39:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428401+0000", 2024-04-22T18:39:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9839e-05 2024-04-22T18:39:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:334 4.c e30/20 4:3227431c:::smithi08937929-35:head v 30'18, mlcod=30'16)", 2024-04-22T18:39:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.698741+0000", 2024-04-22T18:39:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.086073821999999, 2024-04-22T18:39:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10216847599999999, 2024-04-22T18:39:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698741+0000", 2024-04-22T18:39:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698741+0000", 2024-04-22T18:39:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698740+0000", 2024-04-22T18:39:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698747+0000", 2024-04-22T18:39:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9199999999999998e-06 2024-04-22T18:39:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698756+0000", 2024-04-22T18:39:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1249999999999999e-06 2024-04-22T18:39:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698762+0000", 2024-04-22T18:39:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:39:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798915+0000", 2024-04-22T18:39:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10015304899999999 2024-04-22T18:39:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.798937+0000", 2024-04-22T18:39:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1335000000000002e-05 2024-04-22T18:39:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.800879+0000", 2024-04-22T18:39:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001942088 2024-04-22T18:39:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.800910+0000", 2024-04-22T18:39:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0892000000000002e-05 2024-04-22T18:39:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:419 4.0 4:01baf5a9:::smithi08937929-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:39:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.382593282999999, 2024-04-22T18:39:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30324478199999999, 2024-04-22T18:39:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 419 2024-04-22T18:39:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402219+0000", 2024-04-22T18:39:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402229+0000", 2024-04-22T18:39:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2550000000000003e-06 2024-04-22T18:39:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402242+0000", 2024-04-22T18:39:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2989999999999999e-05 2024-04-22T18:39:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402250+0000", 2024-04-22T18:39:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7900000000000005e-06 2024-04-22T18:39:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404085+0000", 2024-04-22T18:39:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001834397 2024-04-22T18:39:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404092+0000", 2024-04-22T18:39:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7889999999999997e-06 2024-04-22T18:39:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502534+0000", 2024-04-22T18:39:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098442331999999994 2024-04-22T18:39:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502631+0000", 2024-04-22T18:39:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7617999999999999e-05 2024-04-22T18:39:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604414+0000", 2024-04-22T18:39:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101782847 2024-04-22T18:39:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604888+0000", 2024-04-22T18:39:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047325999999999998 2024-04-22T18:39:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705376+0000", 2024-04-22T18:39:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100488623 2024-04-22T18:39:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705424+0000", 2024-04-22T18:39:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7964999999999999e-05 2024-04-22T18:39:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705466+0000", 2024-04-22T18:39:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2274999999999998e-05 2024-04-22T18:39:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:70 3.2 e32/29 3:5f0623ce:::smithi08937929-38 low_tier:head v 32'37, mlcod=31'35)", 2024-04-22T18:39:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.256376251999999, 2024-04-22T18:39:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20070027900000001, 2024-04-22T18:39:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528438+0000", 2024-04-22T18:39:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528443+0000", 2024-04-22T18:39:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8640000000000004e-06 2024-04-22T18:39:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528453+0000", 2024-04-22T18:39:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3910000000000004e-06 2024-04-22T18:39:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528459+0000", 2024-04-22T18:39:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-04-22T18:39:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728627+0000", 2024-04-22T18:39:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20016778700000001 2024-04-22T18:39:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728660+0000", 2024-04-22T18:39:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3024000000000001e-05 2024-04-22T18:39:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729107+0000", 2024-04-22T18:39:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044719200000000001 2024-04-22T18:39:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729139+0000", 2024-04-22T18:39:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1625000000000003e-05 2024-04-22T18:39:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1263 4.c e43/20 4:36664a98:::smithi08937929-38:head v 43'36, mlcod=42'34)", 2024-04-22T18:39:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.555349+0000", 2024-04-22T18:39:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.2294657499999992, 2024-04-22T18:39:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10300226699999999, 2024-04-22T18:39:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.555349+0000", 2024-04-22T18:39:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.555349+0000", 2024-04-22T18:39:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.555349+0000", 2024-04-22T18:39:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.556191+0000", 2024-04-22T18:39:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084236200000000004 2024-04-22T18:39:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.556203+0000", 2024-04-22T18:39:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1301000000000001e-05 2024-04-22T18:39:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656304+0000", 2024-04-22T18:39:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100101022 2024-04-22T18:39:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656364+0000", 2024-04-22T18:39:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0207000000000001e-05 2024-04-22T18:39:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656392+0000", 2024-04-22T18:39:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8537000000000001e-05 2024-04-22T18:39:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.658324+0000", 2024-04-22T18:39:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019315409999999999 2024-04-22T18:39:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.658351+0000", 2024-04-22T18:39:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7517000000000001e-05 2024-04-22T18:39:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1264 4.c e43/20 4:36664a98:::smithi08937929-38:head v 43'37, mlcod=42'34)", 2024-04-22T18:39:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.1284778479999993, 2024-04-22T18:39:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20290602399999999, 2024-04-22T18:39:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656335+0000", 2024-04-22T18:39:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656850+0000", 2024-04-22T18:39:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051499499999999997 2024-04-22T18:39:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656861+0000", 2024-04-22T18:39:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1119e-05 2024-04-22T18:39:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756934+0000", 2024-04-22T18:39:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073548 2024-04-22T18:39:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756977+0000", 2024-04-22T18:39:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2218000000000001e-05 2024-04-22T18:39:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756993+0000", 2024-04-22T18:39:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6629999999999998e-05 2024-04-22T18:39:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859203+0000", 2024-04-22T18:39:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102209617 2024-04-22T18:39:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859243+0000", 2024-04-22T18:39:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0333000000000002e-05 2024-04-22T18:39:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1265 4.c e43/20 4:36664a98:::smithi08937929-38:head v 43'38, mlcod=42'34)", 2024-04-22T18:39:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.756947+0000", 2024-04-22T18:39:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.0278679450000006, 2024-04-22T18:39:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102320095, 2024-04-22T18:39:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756947+0000", 2024-04-22T18:39:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756947+0000", 2024-04-22T18:39:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756946+0000", 2024-04-22T18:39:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757183+0000", 2024-04-22T18:39:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000236849 2024-04-22T18:39:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757202+0000", 2024-04-22T18:39:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9403e-05 2024-04-22T18:39:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757209+0000", 2024-04-22T18:39:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0539999999999997e-06 2024-04-22T18:39:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.758174+0000", 2024-04-22T18:39:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00096538199999999998 2024-04-22T18:39:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.758195+0000", 2024-04-22T18:39:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1043000000000002e-05 2024-04-22T18:39:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859248+0000", 2024-04-22T18:39:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10105283 2024-04-22T18:39:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859267+0000", 2024-04-22T18:39:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8774e-05 2024-04-22T18:39:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1266 4.c e43/20 4:36664a98:::smithi08937929-38:head v 43'39, mlcod=42'34)", 2024-04-22T18:39:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.757217+0000", 2024-04-22T18:39:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.0275977469999997, 2024-04-22T18:39:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102430021, 2024-04-22T18:39:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757217+0000", 2024-04-22T18:39:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757217+0000", 2024-04-22T18:39:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757216+0000", 2024-04-22T18:39:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757222+0000", 2024-04-22T18:39:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9320000000000001e-06 2024-04-22T18:39:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757226+0000", 2024-04-22T18:39:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.126e-06 2024-04-22T18:39:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757229+0000", 2024-04-22T18:39:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.897e-06 2024-04-22T18:39:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.858954+0000", 2024-04-22T18:39:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101724905 2024-04-22T18:39:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.858981+0000", 2024-04-22T18:39:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7216000000000002e-05 2024-04-22T18:39:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859620+0000", 2024-04-22T18:39:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063874799999999996 2024-04-22T18:39:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859647+0000", 2024-04-22T18:39:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6959000000000001e-05 2024-04-22T18:39:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1398 3.2 e44/29 3:460a5d9d:::9d81e38fa45b2f87319d1f6ca039dfc50d4304ceb216fced89f733de95073c57:head v 44'70, mlcod=43'68)", 2024-04-22T18:39:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:21.958373+0000", 2024-04-22T18:39:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.8264421669999997, 2024-04-22T18:39:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102010931, 2024-04-22T18:39:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.958373+0000", 2024-04-22T18:39:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.958373+0000", 2024-04-22T18:39:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.958372+0000", 2024-04-22T18:39:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.959307+0000", 2024-04-22T18:39:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000934135 2024-04-22T18:39:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.959317+0000", 2024-04-22T18:39:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0735e-05 2024-04-22T18:39:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:21.959324+0000", 2024-04-22T18:39:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0639999999999996e-06 2024-04-22T18:39:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.059445+0000", 2024-04-22T18:39:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100120632 2024-04-22T18:39:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.059471+0000", 2024-04-22T18:39:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6057000000000001e-05 2024-04-22T18:39:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.060348+0000", 2024-04-22T18:39:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087665200000000001 2024-04-22T18:39:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.060384+0000", 2024-04-22T18:39:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6016000000000002e-05 2024-04-22T18:39:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:26.882 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:26.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:26.884 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:26.885 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-22T18:39:26.885 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:26.886 INFO:tasks.rados.rados.0.smithi089.stdout:859: got (0) Success 2024-04-22T18:39:26.886 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-22T18:39:26.886 INFO:tasks.rados.rados.0.smithi089.stdout:858: done (8 left) 2024-04-22T18:39:26.886 INFO:tasks.rados.rados.0.smithi089.stdout:859: done (7 left) 2024-04-22T18:39:26.886 INFO:tasks.rados.rados.0.smithi089.stdout:861: done (6 left) 2024-04-22T18:39:26.886 INFO:tasks.rados.rados.0.smithi089.stdout:862: done (5 left) 2024-04-22T18:39:26.886 INFO:tasks.rados.rados.0.smithi089.stdout:863: done (4 left) 2024-04-22T18:39:26.886 INFO:tasks.rados.rados.0.smithi089.stdout:864: done (3 left) 2024-04-22T18:39:26.887 INFO:tasks.rados.rados.0.smithi089.stdout:866: done (2 left) 2024-04-22T18:39:26.887 INFO:tasks.rados.rados.0.smithi089.stdout:867: done (1 left) 2024-04-22T18:39:26.887 INFO:tasks.rados.rados.0.smithi089.stdout:868: read oid 26 snap -1 2024-04-22T18:39:26.887 INFO:tasks.rados.rados.0.smithi089.stdout:868: expect (ObjNum 115 snap 0 seq_num 115) 2024-04-22T18:39:26.887 INFO:tasks.rados.rados.0.smithi089.stdout:869: read oid 37 snap -1 2024-04-22T18:39:26.887 INFO:tasks.rados.rados.0.smithi089.stdout:869: expect (ObjNum 84 snap 0 seq_num 84) 2024-04-22T18:39:26.887 INFO:tasks.rados.rados.0.smithi089.stdout:870: write oid 44 current snap is 0 2024-04-22T18:39:26.887 INFO:tasks.rados.rados.0.smithi089.stdout:870: seq_num 119 ranges {400033=699926,1614063=385970} 2024-04-22T18:39:26.887 INFO:tasks.rados.rados.0.smithi089.stdout:870: writing smithi08937929-44 from 400033 to 1099959 tid 1 2024-04-22T18:39:26.887 INFO:tasks.rados.rados.0.smithi089.stdout:870: writing smithi08937929-44 from 1614063 to 2000033 tid 2 2024-04-22T18:39:26.887 INFO:tasks.rados.rados.0.smithi089.stdout:871: read oid 6 snap -1 2024-04-22T18:39:26.887 INFO:tasks.rados.rados.0.smithi089.stdout:871: expect (ObjNum 113 snap 0 seq_num 113) 2024-04-22T18:39:26.887 INFO:tasks.rados.rados.0.smithi089.stdout:872: tier_promote oid 19 2024-04-22T18:39:26.887 INFO:tasks.rados.rados.0.smithi089.stdout:865: expect (ObjNum 112 snap 0 seq_num 112) 2024-04-22T18:39:26.888 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:26.884+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:26.984 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:26.985+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:26.994 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:26.994 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:26.994 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:26.994 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:27.003 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:27.053 INFO:tasks.rados.rados.0.smithi089.stdout:868: expect (ObjNum 115 snap 0 seq_num 115) 2024-04-22T18:39:27.085 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:27.086+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:27.104 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:27.104 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:27.104 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:27.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:27.196 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:27.197+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:27.227 INFO:tasks.rados.rados.0.smithi089.stdout:871: expect (ObjNum 113 snap 0 seq_num 113) 2024-04-22T18:39:27.255 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:27.255 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:27.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:27.255 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:27.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.256 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:27.256 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:27.256 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.093042154000003, 2024-04-22T18:39:27.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:39:27.256 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.256 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.256 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.256 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.256 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.256 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:39:27.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.256 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:27.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:27.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:39:27.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:27.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:39:27.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:39:27.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:39:27.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:39:27.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:39:27.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:39:27.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:39:27.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:39:27.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:39:27.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:39:27.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:39:27.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:39:27.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:39:27.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:39:27.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:39:27.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:39:27.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:39:27.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:39:27.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:39:27.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:39:27.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:39:27.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.262 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.262 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:27.262 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:27.262 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.991483332999998, 2024-04-22T18:39:27.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:39:27.263 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.263 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.263 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:39:27.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.263 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:27.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:39:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:39:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:39:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:39:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:39:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:39:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:39:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:39:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:39:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:39:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:39:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:39:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:39:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:39:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:39:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:39:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:39:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:39:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:39:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:39:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:39:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:39:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.991456931000002, 2024-04-22T18:39:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:39:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:39:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:39:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:39:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:39:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:39:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:39:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:39:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:39:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:39:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:39:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:39:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:39:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:39:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:39:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:39:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:39:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:39:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:39:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.593122602000001, 2024-04-22T18:39:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:39:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:39:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:39:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:39:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:39:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:39:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:39:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:39:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:39:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:39:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:39:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:39:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:39:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:39:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:39:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:39:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:39:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:39:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:39:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:39:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:39:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:39:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.492286303999997, 2024-04-22T18:39:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:39:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:39:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:39:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:39:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:39:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:39:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:39:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:39:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:39:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:39:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:39:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:39:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:39:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:39:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:39:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:39:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:39:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:39:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:39:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:39:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:39:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:39:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:39:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.492129034000001, 2024-04-22T18:39:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:39:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:39:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:39:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:39:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:39:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:39:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:39:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:39:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:39:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:39:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:39:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:39:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:39:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:39:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:39:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:39:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:39:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:39:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:39:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:39:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:39:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:39:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:39:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.492114424999997, 2024-04-22T18:39:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:39:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:39:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:39:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:39:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:39:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:39:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:39:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:39:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:39:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:39:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:39:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:39:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:39:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:39:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:39:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:39:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:39:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:39:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.492104040000001, 2024-04-22T18:39:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:39:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:39:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:39:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:39:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:39:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:39:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:39:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:39:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:39:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:39:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:39:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:39:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:39:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:39:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:39:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:39:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:39:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:39:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:39:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:39:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:39:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:39:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:39:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.491174110000003, 2024-04-22T18:39:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:39:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:39:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:39:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:39:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:39:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:39:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:39:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:39:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:39:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:39:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:39:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:39:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:39:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:39:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:39:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:39:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:39:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:39:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:39:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:39:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:39:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:39:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:39:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.490711984999997, 2024-04-22T18:39:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:39:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:39:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:39:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:39:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:39:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:39:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:39:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:39:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:39:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:39:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:39:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:39:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:39:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:39:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:39:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:39:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:39:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:39:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:39:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:39:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:39:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:39:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:39:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.490699180999997, 2024-04-22T18:39:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:39:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:39:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:39:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:39:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:39:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:39:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:39:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:39:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:39:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:39:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:39:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:39:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:39:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:39:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:39:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:39:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:39:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:39:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:39:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:109 4.1 4:83c955bf:::smithi08937929-28:head [write 453789~780782,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.287912188, 2024-04-22T18:39:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.45086113300000003, 2024-04-22T18:39:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:39:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967588+0000", 2024-04-22T18:39:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.968243+0000", 2024-04-22T18:39:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00065488500000000002 2024-04-22T18:39:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.968265+0000", 2024-04-22T18:39:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1997e-05 2024-04-22T18:39:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.068340+0000", 2024-04-22T18:39:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007516 2024-04-22T18:39:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168464+0000", 2024-04-22T18:39:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100124267 2024-04-22T18:39:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168579+0000", 2024-04-22T18:39:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114387 2024-04-22T18:39:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168692+0000", 2024-04-22T18:39:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011335100000000001 2024-04-22T18:39:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.173549+0000", 2024-04-22T18:39:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048564430000000002 2024-04-22T18:39:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418367+0000", 2024-04-22T18:39:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.244817962 2024-04-22T18:39:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418419+0000", 2024-04-22T18:39:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2883000000000002e-05 2024-04-22T18:39:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418450+0000", 2024-04-22T18:39:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0278999999999999e-05 2024-04-22T18:39:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.162926724999998, 2024-04-22T18:39:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:39:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:39:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:39:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:39:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:39:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:39:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:39:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:39:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:39:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:39:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:39:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:39:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:39:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:39:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:39:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:39:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:39:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:39:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:39:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:39:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:39:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:39:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:39:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.154353613000001, 2024-04-22T18:39:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:39:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:39:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:39:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:39:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:39:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:39:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:39:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:39:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:39:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:39:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:39:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:39:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:39:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:39:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:39:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:39:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:39:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:39:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:39:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:39:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:39:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:39:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:39:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:36 3.3 3:ea25b818:::smithi08937929-10 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.726810577999998, 2024-04-22T18:39:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40851152800000001, 2024-04-22T18:39:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528694+0000", 2024-04-22T18:39:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2459999999999997e-06 2024-04-22T18:39:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528701+0000", 2024-04-22T18:39:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7950000000000002e-06 2024-04-22T18:39:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528709+0000", 2024-04-22T18:39:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1000000000000004e-06 2024-04-22T18:39:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528782+0000", 2024-04-22T18:39:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3699e-05 2024-04-22T18:39:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528785+0000", 2024-04-22T18:39:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:39:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735367+0000", 2024-04-22T18:39:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20658175100000001 2024-04-22T18:39:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735443+0000", 2024-04-22T18:39:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6106999999999995e-05 2024-04-22T18:39:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735573+0000", 2024-04-22T18:39:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129734 2024-04-22T18:39:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.740966+0000", 2024-04-22T18:39:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053933130000000003 2024-04-22T18:39:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937143+0000", 2024-04-22T18:39:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19617662399999999 2024-04-22T18:39:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937175+0000", 2024-04-22T18:39:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1860999999999999e-05 2024-04-22T18:39:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937202+0000", 2024-04-22T18:39:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7029000000000002e-05 2024-04-22T18:39:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:37 3.3 3:ea25b818:::smithi08937929-10 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.528720+0000", 2024-04-22T18:39:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.726781261999999, 2024-04-22T18:39:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40858023100000002, 2024-04-22T18:39:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528720+0000", 2024-04-22T18:39:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528720+0000", 2024-04-22T18:39:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528719+0000", 2024-04-22T18:39:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528724+0000", 2024-04-22T18:39:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1159999999999998e-06 2024-04-22T18:39:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528730+0000", 2024-04-22T18:39:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0480000000000004e-06 2024-04-22T18:39:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.628843+0000", 2024-04-22T18:39:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100112425 2024-04-22T18:39:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.628917+0000", 2024-04-22T18:39:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4385e-05 2024-04-22T18:39:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.628923+0000", 2024-04-22T18:39:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4239999999999998e-06 2024-04-22T18:39:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735762+0000", 2024-04-22T18:39:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10683828200000001 2024-04-22T18:39:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735826+0000", 2024-04-22T18:39:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4144000000000001e-05 2024-04-22T18:39:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735941+0000", 2024-04-22T18:39:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011526799999999999 2024-04-22T18:39:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.740973+0000", 2024-04-22T18:39:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050320340000000003 2024-04-22T18:39:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937246+0000", 2024-04-22T18:39:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19627275499999999 2024-04-22T18:39:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937277+0000", 2024-04-22T18:39:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1563999999999998e-05 2024-04-22T18:39:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937300+0000", 2024-04-22T18:39:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2350000000000001e-05 2024-04-22T18:39:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1108 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 3378174] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.637716+0000", 2024-04-22T18:39:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.617784783999999, 2024-04-22T18:39:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41175930500000002, 2024-04-22T18:39:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1108 2024-04-22T18:39:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637716+0000", 2024-04-22T18:39:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637716+0000", 2024-04-22T18:39:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637715+0000", 2024-04-22T18:39:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637719+0000", 2024-04-22T18:39:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1710000000000004e-06 2024-04-22T18:39:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637725+0000", 2024-04-22T18:39:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7790000000000001e-06 2024-04-22T18:39:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637730+0000", 2024-04-22T18:39:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7330000000000003e-06 2024-04-22T18:39:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.948647+0000", 2024-04-22T18:39:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31091748400000002 2024-04-22T18:39:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.948704+0000", 2024-04-22T18:39:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7000000000000003e-05 2024-04-22T18:39:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.948802+0000", 2024-04-22T18:39:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8107000000000001e-05 2024-04-22T18:39:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049249+0000", 2024-04-22T18:39:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100446991 2024-04-22T18:39:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049270+0000", 2024-04-22T18:39:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0336e-05 2024-04-22T18:39:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049369+0000", 2024-04-22T18:39:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9256e-05 2024-04-22T18:39:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049475+0000", 2024-04-22T18:39:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010624900000000001 2024-04-22T18:39:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1109 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.637734+0000", 2024-04-22T18:39:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.617767175999999, 2024-04-22T18:39:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41190551399999997, 2024-04-22T18:39:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1109 2024-04-22T18:39:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637734+0000", 2024-04-22T18:39:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637734+0000", 2024-04-22T18:39:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637733+0000", 2024-04-22T18:39:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637736+0000", 2024-04-22T18:39:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4530000000000001e-06 2024-04-22T18:39:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637741+0000", 2024-04-22T18:39:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.118e-06 2024-04-22T18:39:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637745+0000", 2024-04-22T18:39:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4490000000000001e-06 2024-04-22T18:39:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.949123+0000", 2024-04-22T18:39:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31137820900000002 2024-04-22T18:39:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.949205+0000", 2024-04-22T18:39:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1444000000000002e-05 2024-04-22T18:39:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049516+0000", 2024-04-22T18:39:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031090099999999 2024-04-22T18:39:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049573+0000", 2024-04-22T18:39:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7497000000000002e-05 2024-04-22T18:39:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049639+0000", 2024-04-22T18:39:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5906000000000002e-05 2024-04-22T18:39:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1285 4.d 4:bee1184f:::smithi08937929-7:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.4182359390000006, 2024-04-22T18:39:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.42388892500000003, 2024-04-22T18:39:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1285 2024-04-22T18:39:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837262+0000", 2024-04-22T18:39:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837273+0000", 2024-04-22T18:39:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1364e-05 2024-04-22T18:39:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837300+0000", 2024-04-22T18:39:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6928999999999999e-05 2024-04-22T18:39:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837318+0000", 2024-04-22T18:39:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8219999999999998e-05 2024-04-22T18:39:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837445+0000", 2024-04-22T18:39:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000126415 2024-04-22T18:39:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837627+0000", 2024-04-22T18:39:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000182182 2024-04-22T18:39:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.261154+0000", 2024-04-22T18:39:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.42352673499999999 2024-04-22T18:39:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:39:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:39:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.119706098, 2024-04-22T18:39:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:39:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:39:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:39:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:39:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:39:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:39:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:39:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:39:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:39:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:39:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:39:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:39:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:39:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:39:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:39:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:39:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:39:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.371 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:27.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:27.373 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:27.373+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:27.402 INFO:tasks.rados.rados.0.smithi089.stdout:869: expect (ObjNum 84 snap 0 seq_num 84) 2024-04-22T18:39:27.473 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:27.474+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:27.515 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:27.516 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:27.516 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:27.516 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:27.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:27.545 INFO:tasks.rados.rados.0.smithi089.stdout:872: got (0) Success 2024-04-22T18:39:27.545 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 60 (ObjNum 99 snap 0 seq_num 99) dirty exists 2024-04-22T18:39:27.545 INFO:tasks.rados.rados.0.smithi089.stdout:865: done (5 left) 2024-04-22T18:39:27.545 INFO:tasks.rados.rados.0.smithi089.stdout:868: done (4 left) 2024-04-22T18:39:27.545 INFO:tasks.rados.rados.0.smithi089.stdout:869: done (3 left) 2024-04-22T18:39:27.545 INFO:tasks.rados.rados.0.smithi089.stdout:871: done (2 left) 2024-04-22T18:39:27.545 INFO:tasks.rados.rados.0.smithi089.stdout:872: done (1 left) 2024-04-22T18:39:27.545 INFO:tasks.rados.rados.0.smithi089.stdout:873: read oid 8 snap -1 2024-04-22T18:39:27.545 INFO:tasks.rados.rados.0.smithi089.stdout:873: expect deleted 2024-04-22T18:39:27.545 INFO:tasks.rados.rados.0.smithi089.stdout:874: write oid 4 current snap is 0 2024-04-22T18:39:27.546 INFO:tasks.rados.rados.0.smithi089.stdout:870: finishing write tid 1 to smithi08937929-44 2024-04-22T18:39:27.546 INFO:tasks.rados.rados.0.smithi089.stdout:870: finishing write tid 2 to smithi08937929-44 2024-04-22T18:39:27.546 INFO:tasks.rados.rados.0.smithi089.stdout:874: seq_num 120 ranges {416840=775175,1672545=344295} 2024-04-22T18:39:27.551 INFO:tasks.rados.rados.0.smithi089.stdout:874: writing smithi08937929-4 from 416840 to 1192015 tid 1 2024-04-22T18:39:27.555 INFO:tasks.rados.rados.0.smithi089.stdout:874: writing smithi08937929-4 from 1672545 to 2016840 tid 2 2024-04-22T18:39:27.555 INFO:tasks.rados.rados.0.smithi089.stdout:875: write (excl) oid 35 current snap is 0 2024-04-22T18:39:27.555 INFO:tasks.rados.rados.0.smithi089.stdout:875: seq_num 121 ranges {433647=450424,1331027=582033} 2024-04-22T18:39:27.560 INFO:tasks.rados.rados.0.smithi089.stdout:875: writing smithi08937929-35 from 433647 to 884071 tid 1 2024-04-22T18:39:27.565 INFO:tasks.rados.rados.0.smithi089.stdout:875: writing smithi08937929-35 from 1331027 to 1913060 tid 2 2024-04-22T18:39:27.566 INFO:tasks.rados.rados.0.smithi089.stdout:876: read oid 36 snap -1 2024-04-22T18:39:27.566 INFO:tasks.rados.rados.0.smithi089.stdout:876: expect (ObjNum 80 snap 0 seq_num 80) 2024-04-22T18:39:27.566 INFO:tasks.rados.rados.0.smithi089.stdout:877: read oid 42 snap -1 2024-04-22T18:39:27.566 INFO:tasks.rados.rados.0.smithi089.stdout:877: expect (ObjNum 105 snap 0 seq_num 105) 2024-04-22T18:39:27.566 INFO:tasks.rados.rados.0.smithi089.stdout:878: read oid 22 snap -1 2024-04-22T18:39:27.566 INFO:tasks.rados.rados.0.smithi089.stdout:878: expect (ObjNum 109 snap 0 seq_num 109) 2024-04-22T18:39:27.566 INFO:tasks.rados.rados.0.smithi089.stdout:879: write oid 45 current snap is 0 2024-04-22T18:39:27.566 INFO:tasks.rados.rados.0.smithi089.stdout:879: seq_num 122 ranges {450454=642026,1505862=544592} 2024-04-22T18:39:27.571 INFO:tasks.rados.rados.0.smithi089.stdout:879: writing smithi08937929-45 from 450454 to 1092480 tid 1 2024-04-22T18:39:27.574 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:27.576+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:27.578 INFO:tasks.rados.rados.0.smithi089.stdout:879: writing smithi08937929-45 from 1505862 to 2050454 tid 2 2024-04-22T18:39:27.578 INFO:tasks.rados.rados.0.smithi089.stdout:880: read oid 5 snap -1 2024-04-22T18:39:27.578 INFO:tasks.rados.rados.0.smithi089.stdout:880: expect deleted 2024-04-22T18:39:27.578 INFO:tasks.rados.rados.0.smithi089.stdout:881: tier_promote oid 33 2024-04-22T18:39:27.578 INFO:tasks.rados.rados.0.smithi089.stdout:870: finishing write tid 3 to smithi08937929-44 2024-04-22T18:39:27.579 INFO:tasks.rados.rados.0.smithi089.stdout:870: finishing write tid 4 to smithi08937929-44 2024-04-22T18:39:27.579 INFO:tasks.rados.rados.0.smithi089.stdout:870: oid 44 updating version 0 to 57 2024-04-22T18:39:27.579 INFO:tasks.rados.rados.0.smithi089.stdout:870: oid 44 updating version 57 to 58 2024-04-22T18:39:27.579 INFO:tasks.rados.rados.0.smithi089.stdout:870: oid 44 updating version 58 to 59 2024-04-22T18:39:27.579 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 59 (ObjNum 119 snap 0 seq_num 119) dirty exists 2024-04-22T18:39:27.579 INFO:tasks.rados.rados.0.smithi089.stdout:870: left oid 44 (ObjNum 119 snap 0 seq_num 119) 2024-04-22T18:39:27.579 INFO:tasks.rados.rados.0.smithi089.stdout:877: expect (ObjNum 105 snap 0 seq_num 105) 2024-04-22T18:39:27.668 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:27.668 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:27.668 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:27.675 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:27.677+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:27.677 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:27.777 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:27.778+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:27.783 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:27.783 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:27.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:27.783 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:27.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.783 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:27.783 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:27.783 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.689438610000003, 2024-04-22T18:39:27.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:39:27.784 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.784 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.784 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:39:27.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.784 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:27.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:27.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:39:27.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:27.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:39:27.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:39:27.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:39:27.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:39:27.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:39:27.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:39:27.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:39:27.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:39:27.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:39:27.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:39:27.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:27.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:39:27.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:39:27.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:39:27.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:39:27.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:39:27.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:39:27.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:39:27.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:39:27.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:39:27.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:39:27.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.789 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.789 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2 4.8 4:12d9f826:::smithi08937929-1:head [write 1542129~474678 in=474678b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:27.790 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:39:27.790 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.588727339999998, 2024-04-22T18:39:27.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20435667099999999, 2024-04-22T18:39:27.790 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.790 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.790 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2 2024-04-22T18:39:27.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.790 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:39:27.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:39:27.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194568+0000", 2024-04-22T18:39:27.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:27.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194869+0000", 2024-04-22T18:39:27.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030115200000000001 2024-04-22T18:39:27.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194879+0000", 2024-04-22T18:39:27.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8479999999999996e-06 2024-04-22T18:39:27.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294954+0000", 2024-04-22T18:39:27.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100074702 2024-04-22T18:39:27.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296019+0000", 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010649189999999999 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296086+0000", 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7261000000000005e-05 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296173+0000", 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6935999999999996e-05 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297196+0000", 2024-04-22T18:39:27.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023415 2024-04-22T18:39:27.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398879+0000", 2024-04-22T18:39:27.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101682563 2024-04-22T18:39:27.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398906+0000", 2024-04-22T18:39:27.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6526999999999999e-05 2024-04-22T18:39:27.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398928+0000", 2024-04-22T18:39:27.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2045e-05 2024-04-22T18:39:27.795 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.795 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.795 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.795 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=62b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:27.795 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:39:27.795 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.488326385999997, 2024-04-22T18:39:27.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20408478299999999, 2024-04-22T18:39:27.796 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.796 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.796 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.796 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.796 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.796 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3 2024-04-22T18:39:27.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.796 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:39:27.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:39:27.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294970+0000", 2024-04-22T18:39:27.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:27.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294983+0000", 2024-04-22T18:39:27.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3429e-05 2024-04-22T18:39:27.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294993+0000", 2024-04-22T18:39:27.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0416e-05 2024-04-22T18:39:27.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295007+0000", 2024-04-22T18:39:27.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3742e-05 2024-04-22T18:39:27.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296793+0000", 2024-04-22T18:39:27.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017860420000000001 2024-04-22T18:39:27.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296841+0000", 2024-04-22T18:39:27.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7706e-05 2024-04-22T18:39:27.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:27.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296913+0000", 2024-04-22T18:39:27.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2578999999999997e-05 2024-04-22T18:39:27.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297946+0000", 2024-04-22T18:39:27.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001032802 2024-04-22T18:39:27.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498963+0000", 2024-04-22T18:39:27.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20101640400000001 2024-04-22T18:39:27.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498998+0000", 2024-04-22T18:39:27.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5759000000000001e-05 2024-04-22T18:39:27.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499057+0000", 2024-04-22T18:39:27.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8270999999999998e-05 2024-04-22T18:39:27.801 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.801 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.801 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.801 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:27.801 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:39:27.802 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.488285957999999, 2024-04-22T18:39:27.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20418823899999999, 2024-04-22T18:39:27.802 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.802 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:27.802 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.802 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.802 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.802 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4 2024-04-22T18:39:27.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.802 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:39:27.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:39:27.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295011+0000", 2024-04-22T18:39:27.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:27.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295016+0000", 2024-04-22T18:39:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2400000000000001e-06 2024-04-22T18:39:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295022+0000", 2024-04-22T18:39:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8519999999999998e-06 2024-04-22T18:39:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295027+0000", 2024-04-22T18:39:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9749999999999998e-06 2024-04-22T18:39:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297126+0000", 2024-04-22T18:39:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020981070000000001 2024-04-22T18:39:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297167+0000", 2024-04-22T18:39:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1842999999999999e-05 2024-04-22T18:39:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499096+0000", 2024-04-22T18:39:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201928846 2024-04-22T18:39:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499142+0000", 2024-04-22T18:39:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6152000000000001e-05 2024-04-22T18:39:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499201+0000", 2024-04-22T18:39:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8190999999999999e-05 2024-04-22T18:39:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:70 4.3 e27/20 4:c6fa81cf:::smithi08937929-18:head v 27'8, mlcod=27'3)", 2024-04-22T18:39:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.825806041999996, 2024-04-22T18:39:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20440983300000001, 2024-04-22T18:39:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957839+0000", 2024-04-22T18:39:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000346981 2024-04-22T18:39:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957849+0000", 2024-04-22T18:39:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8060000000000006e-06 2024-04-22T18:39:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957856+0000", 2024-04-22T18:39:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0319999999999998e-06 2024-04-22T18:39:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958600+0000", 2024-04-22T18:39:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074474700000000003 2024-04-22T18:39:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958615+0000", 2024-04-22T18:39:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4961e-05 2024-04-22T18:39:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161867+0000", 2024-04-22T18:39:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20325188399999999 2024-04-22T18:39:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161902+0000", 2024-04-22T18:39:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4884e-05 2024-04-22T18:39:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.789940734999998, 2024-04-22T18:39:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:39:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.811 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:39:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:39:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:39:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:39:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:39:27.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:39:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:39:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:39:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:39:27.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:39:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:39:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:39:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:39:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:39:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:39:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:39:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:39:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:39:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:39:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:39:27.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:39:27.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:39:27.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:39:27.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:39:27.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:39:27.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:39:27.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.818 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.818 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:27 3.2 3:7a6cba00:::smithi08937929-32 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:27.818 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.597045+0000", 2024-04-22T18:39:27.818 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.186253241999999, 2024-04-22T18:39:27.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20443814799999999, 2024-04-22T18:39:27.818 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.818 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.818 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597045+0000", 2024-04-22T18:39:27.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597045+0000", 2024-04-22T18:39:27.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597043+0000", 2024-04-22T18:39:27.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597053+0000", 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0278999999999999e-05 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597074+0000", 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0710999999999999e-05 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.697204+0000", 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10012958299999999 2024-04-22T18:39:27.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.698362+0000", 2024-04-22T18:39:27.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011587629999999999 2024-04-22T18:39:27.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.698440+0000", 2024-04-22T18:39:27.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7453000000000005e-05 2024-04-22T18:39:27.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:27.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.698593+0000", 2024-04-22T18:39:27.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153449 2024-04-22T18:39:27.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.699048+0000", 2024-04-22T18:39:27.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000454287 2024-04-22T18:39:27.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.801433+0000", 2024-04-22T18:39:27.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102385664 2024-04-22T18:39:27.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.801462+0000", 2024-04-22T18:39:27.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9153e-05 2024-04-22T18:39:27.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.801483+0000", 2024-04-22T18:39:27.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0707e-05 2024-04-22T18:39:27.823 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.823 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.823 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.823 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:27.823 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:27.823 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.681640944000002, 2024-04-22T18:39:27.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:39:27.824 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.824 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.824 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.824 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.824 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.824 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:39:27.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.824 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:27.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:27.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:39:27.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:27.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:39:27.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:39:27.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:39:27.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:39:27.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:39:27.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:39:27.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:39:27.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:39:27.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:39:27.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:39:27.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:27.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:39:27.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:39:27.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:39:27.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:39:27.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:39:27.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:39:27.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:39:27.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:39:27.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:39:27.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:39:27.829 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.829 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.829 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.829 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:27.829 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:27.829 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.350097315999999, 2024-04-22T18:39:27.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:39:27.829 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.829 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.829 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:27.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:27.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:39:27.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:27.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:39:27.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:39:27.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:39:27.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:39:27.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:39:27.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:39:27.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:39:27.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:39:27.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:39:27.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:39:27.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:39:27.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:39:27.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:39:27.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:39:27.834 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.834 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.834 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.834 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.259011508, 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:39:27.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:39:27.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:27.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:39:27.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:39:27.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:39:27.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:39:27.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:39:27.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:39:27.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:39:27.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:39:27.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:39:27.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:39:27.840 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.840 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.840 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.840 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:27.840 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:27.840 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.253211248, 2024-04-22T18:39:27.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:39:27.840 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.840 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.840 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.840 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.840 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.840 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:39:27.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.841 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:27.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:27.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:39:27.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:27.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:39:27.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:39:27.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:39:27.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:39:27.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:27.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:39:27.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:39:27.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:39:27.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:39:27.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:39:27.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:39:27.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.248470905, 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.845 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.846 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.846 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.846 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.846 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:39:27.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.846 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:27.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:27.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:39:27.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:27.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:39:27.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:39:27.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:39:27.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:39:27.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:39:27.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:39:27.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:39:27.850 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.850 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.850 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.850 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:27.850 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:27.851 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.247998007, 2024-04-22T18:39:27.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:39:27.851 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.851 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.851 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.851 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.851 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.851 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:39:27.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.851 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:27.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:27.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:39:27.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:27.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:39:27.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:39:27.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:39:27.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:39:27.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:39:27.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:39:27.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:39:27.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:39:27.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:39:27.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:39:27.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:27.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:39:27.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:39:27.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:39:27.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:39:27.855 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.856 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.856 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.856 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:27.856 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:27.856 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.247978086, 2024-04-22T18:39:27.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:39:27.856 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.856 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:27.856 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:27.856 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:27.856 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:27.856 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:39:27.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.857 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:27.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:27.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:27.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:27.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:39:27.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:39:27.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:39:27.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:39:27.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:39:27.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:39:27.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:39:27.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:39:27.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:27.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:39:27.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:39:27.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:39:27.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:39:27.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:39:27.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:39:27.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:39:27.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:39:27.860 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.860 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.860 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:39:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.9386543209999996, 2024-04-22T18:39:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:39:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:39:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:39:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:39:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:39:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:39:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:39:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:39:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:39:27.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:39:27.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:27.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:39:27.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:39:27.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:39:27.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:39:27.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:39:27.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:39:27.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:39:27.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:39:27.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:39:27.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:39:27.865 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.866 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.866 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.866 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:27.866 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:39:27.866 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.938446785, 2024-04-22T18:39:27.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:39:27.866 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.866 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.866 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:39:27.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:39:27.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:39:27.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:39:27.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:27.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:39:27.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:39:27.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:39:27.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:39:27.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.870 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.871 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:117 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:27.871 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844946+0000", 2024-04-22T18:39:27.871 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.9383522709999994, 2024-04-22T18:39:27.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22636420400000001, 2024-04-22T18:39:27.871 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.871 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.871 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844946+0000", 2024-04-22T18:39:27.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844946+0000", 2024-04-22T18:39:27.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844945+0000", 2024-04-22T18:39:27.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:27.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845024+0000", 2024-04-22T18:39:27.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8532e-05 2024-04-22T18:39:27.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845031+0000", 2024-04-22T18:39:27.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1269999999999997e-06 2024-04-22T18:39:27.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845039+0000", 2024-04-22T18:39:27.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6809999999999998e-06 2024-04-22T18:39:27.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.946521+0000", 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101482688 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.946610+0000", 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9023999999999993e-05 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.946805+0000", 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019483599999999999 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.950818+0000", 2024-04-22T18:39:27.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040133460000000001 2024-04-22T18:39:27.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071261+0000", 2024-04-22T18:39:27.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12044292500000001 2024-04-22T18:39:27.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071290+0000", 2024-04-22T18:39:27.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8322000000000001e-05 2024-04-22T18:39:27.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071310+0000", 2024-04-22T18:39:27.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0477000000000001e-05 2024-04-22T18:39:27.876 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.876 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.876 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.876 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:121 3.2 3:754415d0:::36414a779a90b1914c27a3701533e6b5d0236736bf30da29dc510853ad2cbd37:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:27.876 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.845046+0000", 2024-04-22T18:39:27.876 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.9382525190000006, 2024-04-22T18:39:27.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22635160700000001, 2024-04-22T18:39:27.876 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.876 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.876 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845046+0000", 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845046+0000", 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845045+0000", 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845081+0000", 2024-04-22T18:39:27.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.595e-05 2024-04-22T18:39:27.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845088+0000", 2024-04-22T18:39:27.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0929999999999997e-06 2024-04-22T18:39:27.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845093+0000", 2024-04-22T18:39:27.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3970000000000002e-06 2024-04-22T18:39:27.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947321+0000", 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102227129 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947400+0000", 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9919000000000005e-05 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947533+0000", 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000132473 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.951155+0000", 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036216730000000002 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071338+0000", 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.120183489 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071380+0000", 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1519e-05 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071397+0000", 2024-04-22T18:39:27.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7740999999999998e-05 2024-04-22T18:39:27.881 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.881 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.881 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.881 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:126 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:27.881 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.957111+0000", 2024-04-22T18:39:27.881 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.8261871290000009, 2024-04-22T18:39:27.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21447909900000001, 2024-04-22T18:39:27.881 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.881 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.881 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957111+0000", 2024-04-22T18:39:27.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957111+0000", 2024-04-22T18:39:27.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957108+0000", 2024-04-22T18:39:27.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:27.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957118+0000", 2024-04-22T18:39:27.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3850000000000008e-06 2024-04-22T18:39:27.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957133+0000", 2024-04-22T18:39:27.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5611999999999999e-05 2024-04-22T18:39:27.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957142+0000", 2024-04-22T18:39:27.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2879999999999993e-06 2024-04-22T18:39:27.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957209+0000", 2024-04-22T18:39:27.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7568000000000001e-05 2024-04-22T18:39:27.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957260+0000", 2024-04-22T18:39:27.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1091999999999998e-05 2024-04-22T18:39:27.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:27.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957362+0000", 2024-04-22T18:39:27.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101889 2024-04-22T18:39:27.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957985+0000", 2024-04-22T18:39:27.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00062231899999999998 2024-04-22T18:39:27.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.171534+0000", 2024-04-22T18:39:27.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21354967 2024-04-22T18:39:27.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.171568+0000", 2024-04-22T18:39:27.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3453e-05 2024-04-22T18:39:27.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.171590+0000", 2024-04-22T18:39:27.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2555e-05 2024-04-22T18:39:27.886 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.886 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.886 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.886 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:127 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:27.886 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.957151+0000", 2024-04-22T18:39:27.886 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.8261469849999994, 2024-04-22T18:39:27.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21456244299999999, 2024-04-22T18:39:27.886 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:27.887 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:27.887 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:27.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:27.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957151+0000", 2024-04-22T18:39:27.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:27.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957151+0000", 2024-04-22T18:39:27.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:27.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:27.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957150+0000", 2024-04-22T18:39:27.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:27.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:27.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957155+0000", 2024-04-22T18:39:27.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4159999999999997e-06 2024-04-22T18:39:27.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:27.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957161+0000", 2024-04-22T18:39:27.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2219999999999999e-06 2024-04-22T18:39:27.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:27.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957167+0000", 2024-04-22T18:39:27.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5029999999999997e-06 2024-04-22T18:39:27.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:27.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957528+0000", 2024-04-22T18:39:27.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000361861 2024-04-22T18:39:27.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:27.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957570+0000", 2024-04-22T18:39:27.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1709000000000003e-05 2024-04-22T18:39:27.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:27.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957669+0000", 2024-04-22T18:39:27.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8758999999999994e-05 2024-04-22T18:39:27.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:27.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.958208+0000", 2024-04-22T18:39:27.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00053896900000000004 2024-04-22T18:39:27.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:27.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.171634+0000", 2024-04-22T18:39:27.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21342636800000001 2024-04-22T18:39:27.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:27.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.171683+0000", 2024-04-22T18:39:27.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8263999999999997e-05 2024-04-22T18:39:27.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:27.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:27.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:27.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.171714+0000", 2024-04-22T18:39:27.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1202999999999999e-05 2024-04-22T18:39:27.891 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.891 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.891 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.891 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:27.891 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:27.891 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:27.892 INFO:tasks.rados.rados.0.smithi089.stdout:876: expect (ObjNum 80 snap 0 seq_num 80) 2024-04-22T18:39:27.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:27.893 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:27.894+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:27.993 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:27.994+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:28.013 INFO:tasks.rados.rados.0.smithi089.stdout:879: finishing write tid 1 to smithi08937929-45 2024-04-22T18:39:28.013 INFO:tasks.rados.rados.0.smithi089.stdout:879: finishing write tid 2 to smithi08937929-45 2024-04-22T18:39:28.013 INFO:tasks.rados.rados.0.smithi089.stdout:879: finishing write tid 3 to smithi08937929-45 2024-04-22T18:39:28.013 INFO:tasks.rados.rados.0.smithi089.stdout:879: finishing write tid 4 to smithi08937929-45 2024-04-22T18:39:28.013 INFO:tasks.rados.rados.0.smithi089.stdout:879: oid 45 updating version 0 to 44 2024-04-22T18:39:28.013 INFO:tasks.rados.rados.0.smithi089.stdout:879: oid 45 version 44 is already newer than 42 2024-04-22T18:39:28.013 INFO:tasks.rados.rados.0.smithi089.stdout:879: oid 45 version 44 is already newer than 43 2024-04-22T18:39:28.013 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 44 (ObjNum 122 snap 0 seq_num 122) dirty exists 2024-04-22T18:39:28.013 INFO:tasks.rados.rados.0.smithi089.stdout:879: left oid 45 (ObjNum 122 snap 0 seq_num 122) 2024-04-22T18:39:28.013 INFO:tasks.rados.rados.0.smithi089.stdout:881: got (0) Success 2024-04-22T18:39:28.014 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 19 (ObjNum 65 snap 0 seq_num 65) dirty exists 2024-04-22T18:39:28.014 INFO:tasks.rados.rados.0.smithi089.stdout:878: expect (ObjNum 109 snap 0 seq_num 109) 2024-04-22T18:39:28.034 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:28.034 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:28.034 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:28.034 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:28.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:28.072 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:39:28.094 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:28.095+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:28.145 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:28.145 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:28.145 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:28.145 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:28.146 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:28.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:28.192 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:28.192 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:28.195 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:28.196+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:28.199 INFO:tasks.rados.rados.0.smithi089.stdout:875: finishing write tid 1 to smithi08937929-35 2024-04-22T18:39:28.199 INFO:tasks.rados.rados.0.smithi089.stdout:875: finishing write tid 2 to smithi08937929-35 2024-04-22T18:39:28.199 INFO:tasks.rados.rados.0.smithi089.stdout:870: done (9 left) 2024-04-22T18:39:28.199 INFO:tasks.rados.rados.0.smithi089.stdout:873: done (8 left) 2024-04-22T18:39:28.199 INFO:tasks.rados.rados.0.smithi089.stdout:876: done (7 left) 2024-04-22T18:39:28.199 INFO:tasks.rados.rados.0.smithi089.stdout:877: done (6 left) 2024-04-22T18:39:28.199 INFO:tasks.rados.rados.0.smithi089.stdout:878: done (5 left) 2024-04-22T18:39:28.199 INFO:tasks.rados.rados.0.smithi089.stdout:879: done (4 left) 2024-04-22T18:39:28.199 INFO:tasks.rados.rados.0.smithi089.stdout:881: done (3 left) 2024-04-22T18:39:28.200 INFO:tasks.rados.rados.0.smithi089.stdout:882: write oid 16 current snap is 0 2024-04-22T18:39:28.200 INFO:tasks.rados.rados.0.smithi089.stdout:875: finishing write tid 3 to smithi08937929-35 2024-04-22T18:39:28.200 INFO:tasks.rados.rados.0.smithi089.stdout:875: finishing write tid 4 to smithi08937929-35 2024-04-22T18:39:28.200 INFO:tasks.rados.rados.0.smithi089.stdout:875: oid 35 updating version 0 to 51 2024-04-22T18:39:28.200 INFO:tasks.rados.rados.0.smithi089.stdout:875: oid 35 updating version 51 to 52 2024-04-22T18:39:28.200 INFO:tasks.rados.rados.0.smithi089.stdout:875: oid 35 updating version 52 to 53 2024-04-22T18:39:28.200 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 53 (ObjNum 121 snap 0 seq_num 121) dirty exists 2024-04-22T18:39:28.200 INFO:tasks.rados.rados.0.smithi089.stdout:875: left oid 35 (ObjNum 121 snap 0 seq_num 121) 2024-04-22T18:39:28.200 INFO:tasks.rados.rados.0.smithi089.stdout:882: seq_num 123 ranges {467261=717275,1847991=219270} 2024-04-22T18:39:28.205 INFO:tasks.rados.rados.0.smithi089.stdout:882: writing smithi08937929-16 from 467261 to 1184536 tid 1 2024-04-22T18:39:28.209 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:28.209 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:28.211 INFO:tasks.rados.rados.0.smithi089.stdout:882: writing smithi08937929-16 from 1847991 to 2067261 tid 2 2024-04-22T18:39:28.211 INFO:tasks.rados.rados.0.smithi089.stdout:875: done (3 left) 2024-04-22T18:39:28.212 INFO:tasks.rados.rados.0.smithi089.stdout:883: tier_flush oid 1 2024-04-22T18:39:28.212 INFO:tasks.rados.rados.0.smithi089.stdout:883: tier_flush oid 1 snap -1 2024-04-22T18:39:28.212 INFO:tasks.rados.rados.0.smithi089.stdout:874: finishing write tid 1 to smithi08937929-4 2024-04-22T18:39:28.212 INFO:tasks.rados.rados.0.smithi089.stdout:874: finishing write tid 2 to smithi08937929-4 2024-04-22T18:39:28.212 INFO:tasks.rados.rados.0.smithi089.stdout:880: done (3 left) 2024-04-22T18:39:28.212 INFO:tasks.rados.rados.0.smithi089.stdout:884: read oid 13 snap -1 2024-04-22T18:39:28.212 INFO:tasks.rados.rados.0.smithi089.stdout:884: expect deleted 2024-04-22T18:39:28.212 INFO:tasks.rados.rados.0.smithi089.stdout:885: read oid 32 snap -1 2024-04-22T18:39:28.212 INFO:tasks.rados.rados.0.smithi089.stdout:885: expect (ObjNum 92 snap 0 seq_num 92) 2024-04-22T18:39:28.212 INFO:tasks.rados.rados.0.smithi089.stdout:886: read oid 2 snap -1 2024-04-22T18:39:28.212 INFO:tasks.rados.rados.0.smithi089.stdout:886: expect (ObjNum 106 snap 0 seq_num 106) 2024-04-22T18:39:28.212 INFO:tasks.rados.rados.0.smithi089.stdout:887: read oid 35 snap -1 2024-04-22T18:39:28.213 INFO:tasks.rados.rados.0.smithi089.stdout:887: expect (ObjNum 121 snap 0 seq_num 121) 2024-04-22T18:39:28.213 INFO:tasks.rados.rados.0.smithi089.stdout:888: write (excl) oid 10 current snap is 0 2024-04-22T18:39:28.213 INFO:tasks.rados.rados.0.smithi089.stdout:888: seq_num 124 ranges {484068=792524,1906473=177595} 2024-04-22T18:39:28.217 INFO:tasks.rados.rados.0.smithi089.stdout:888: writing smithi08937929-10 from 484068 to 1276592 tid 1 2024-04-22T18:39:28.222 INFO:tasks.rados.rados.0.smithi089.stdout:888: writing smithi08937929-10 from 1906473 to 2084068 tid 2 2024-04-22T18:39:28.222 INFO:tasks.rados.rados.0.smithi089.stdout:889: write oid 12 current snap is 0 2024-04-22T18:39:28.222 INFO:tasks.rados.rados.0.smithi089.stdout:889: seq_num 125 ranges {500875=467773,1564955=535920} 2024-04-22T18:39:28.228 INFO:tasks.rados.rados.0.smithi089.stdout:889: writing smithi08937929-12 from 500875 to 968648 tid 1 2024-04-22T18:39:28.228 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:28.228 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:28.234 INFO:tasks.rados.rados.0.smithi089.stdout:889: writing smithi08937929-12 from 1564955 to 2100875 tid 2 2024-04-22T18:39:28.234 INFO:tasks.rados.rados.0.smithi089.stdout:890: tier_evict oid 28 2024-04-22T18:39:28.234 INFO:tasks.rados.rados.0.smithi089.stdout:890: tier_evict oid 28 snap -1 2024-04-22T18:39:28.235 INFO:tasks.rados.rados.0.smithi089.stdout:874: finishing write tid 3 to smithi08937929-4 2024-04-22T18:39:28.235 INFO:tasks.rados.rados.0.smithi089.stdout:874: finishing write tid 4 to smithi08937929-4 2024-04-22T18:39:28.235 INFO:tasks.rados.rados.0.smithi089.stdout:874: oid 4 updating version 0 to 51 2024-04-22T18:39:28.235 INFO:tasks.rados.rados.0.smithi089.stdout:874: oid 4 version 51 is already newer than 50 2024-04-22T18:39:28.235 INFO:tasks.rados.rados.0.smithi089.stdout:874: oid 4 version 51 is already newer than 49 2024-04-22T18:39:28.235 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 51 (ObjNum 120 snap 0 seq_num 120) dirty exists 2024-04-22T18:39:28.235 INFO:tasks.rados.rados.0.smithi089.stdout:874: left oid 4 (ObjNum 120 snap 0 seq_num 120) 2024-04-22T18:39:28.235 INFO:tasks.rados.rados.0.smithi089.stdout:882: finishing write tid 1 to smithi08937929-16 2024-04-22T18:39:28.235 INFO:tasks.rados.rados.0.smithi089.stdout:874: done (9 left) 2024-04-22T18:39:28.235 INFO:tasks.rados.rados.0.smithi089.stdout:891: tier_promote oid 42 2024-04-22T18:39:28.235 INFO:tasks.rados.rados.0.smithi089.stdout:888: finishing write tid 1 to smithi08937929-10 2024-04-22T18:39:28.236 INFO:tasks.rados.rados.0.smithi089.stdout:892: set_chunk oid 47 target oid 2024-04-22T18:39:28.244 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:28.245 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:28.261 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:28.262 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:28.280 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:28.280 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:28.296 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:28.296 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:28.296 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:28.297+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:28.301 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:28.301 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:28.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:28.312 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:28.312 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:28.333 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:39:28.397 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:28.398+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:28.454 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:28.454 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:28.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:28.504 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:28.505+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:28.539 INFO:tasks.rados.rados.0.smithi089.stdout:892: set_chunk oid 47 offset: 1860608 length: 1064448 target oid b3edbfc628e8a5699a713984ce9980479e83e7a93a36b0edb7bcb5322b6bd924 offset: 0 snap -1 2024-04-22T18:39:28.540 INFO:tasks.rados.rados.0.smithi089.stdout:893: tier_evict oid 21 2024-04-22T18:39:28.540 INFO:tasks.rados.rados.0.smithi089.stdout:893: tier_evict oid 21 snap -1 2024-04-22T18:39:28.540 INFO:tasks.rados.rados.0.smithi089.stdout:884: done (11 left) 2024-04-22T18:39:28.540 INFO:tasks.rados.rados.0.smithi089.stdout:894: read oid 27 snap -1 2024-04-22T18:39:28.540 INFO:tasks.rados.rados.0.smithi089.stdout:894: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-22T18:39:28.540 INFO:tasks.rados.rados.0.smithi089.stdout:895: write oid 9 current snap is 0 2024-04-22T18:39:28.540 INFO:tasks.rados.rados.0.smithi089.stdout:890: got (22) Invalid argument 2024-04-22T18:39:28.540 INFO:tasks.rados.rados.0.smithi089.stdout:887: expect (ObjNum 121 snap 0 seq_num 121) 2024-04-22T18:39:28.598 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:28.599+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:28.606 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:28.606 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:28.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:28.632 INFO:tasks.rados.rados.0.smithi089.stdout:885: expect (ObjNum 92 snap 0 seq_num 92) 2024-04-22T18:39:28.635 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:39:28.699 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:28.700+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:28.725 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:28.800 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:28.801+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:28.828 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:28.828 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:28.836 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:28.875 INFO:tasks.rados.rados.0.smithi089.stdout:882: finishing write tid 2 to smithi08937929-16 2024-04-22T18:39:28.875 INFO:tasks.rados.rados.0.smithi089.stdout:888: finishing write tid 2 to smithi08937929-10 2024-04-22T18:39:28.875 INFO:tasks.rados.rados.0.smithi089.stdout:888: finishing write tid 3 to smithi08937929-10 2024-04-22T18:39:28.875 INFO:tasks.rados.rados.0.smithi089.stdout:888: finishing write tid 4 to smithi08937929-10 2024-04-22T18:39:28.875 INFO:tasks.rados.rados.0.smithi089.stdout:888: oid 10 updating version 0 to 20 2024-04-22T18:39:28.875 INFO:tasks.rados.rados.0.smithi089.stdout:888: oid 10 version 20 is already newer than 19 2024-04-22T18:39:28.875 INFO:tasks.rados.rados.0.smithi089.stdout:888: oid 10 updating version 20 to 21 2024-04-22T18:39:28.875 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 21 (ObjNum 124 snap 0 seq_num 124) dirty exists 2024-04-22T18:39:28.875 INFO:tasks.rados.rados.0.smithi089.stdout:888: left oid 10 (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:39:28.876 INFO:tasks.rados.rados.0.smithi089.stdout:886: expect (ObjNum 106 snap 0 seq_num 106) 2024-04-22T18:39:28.918 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:28.920+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:28.977 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:28.977 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:28.986 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:29.002 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:29.003+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:29.049 INFO:tasks.rados.rados.0.smithi089.stdout:889: finishing write tid 1 to smithi08937929-12 2024-04-22T18:39:29.049 INFO:tasks.rados.rados.0.smithi089.stdout:889: finishing write tid 2 to smithi08937929-12 2024-04-22T18:39:29.049 INFO:tasks.rados.rados.0.smithi089.stdout:891: got (0) Success 2024-04-22T18:39:29.049 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 16 (ObjNum 105 snap 0 seq_num 105) dirty exists 2024-04-22T18:39:29.049 INFO:tasks.rados.rados.0.smithi089.stdout:882: finishing write tid 3 to smithi08937929-16 2024-04-22T18:39:29.049 INFO:tasks.rados.rados.0.smithi089.stdout:882: finishing write tid 4 to smithi08937929-16 2024-04-22T18:39:29.049 INFO:tasks.rados.rados.0.smithi089.stdout:882: oid 16 updating version 0 to 60 2024-04-22T18:39:29.049 INFO:tasks.rados.rados.0.smithi089.stdout:882: oid 16 updating version 60 to 62 2024-04-22T18:39:29.049 INFO:tasks.rados.rados.0.smithi089.stdout:882: oid 16 version 62 is already newer than 61 2024-04-22T18:39:29.049 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 62 (ObjNum 123 snap 0 seq_num 123) dirty exists 2024-04-22T18:39:29.049 INFO:tasks.rados.rados.0.smithi089.stdout:882: left oid 16 (ObjNum 123 snap 0 seq_num 123) 2024-04-22T18:39:29.049 INFO:tasks.rados.rados.0.smithi089.stdout:895: seq_num 126 ranges {517682=543022,1623437=494245} 2024-04-22T18:39:29.054 INFO:tasks.rados.rados.0.smithi089.stdout:895: writing smithi08937929-9 from 517682 to 1060704 tid 1 2024-04-22T18:39:29.061 INFO:tasks.rados.rados.0.smithi089.stdout:895: writing smithi08937929-9 from 1623437 to 2117682 tid 2 2024-04-22T18:39:29.061 INFO:tasks.rados.rados.0.smithi089.stdout:882: done (12 left) 2024-04-22T18:39:29.061 INFO:tasks.rados.rados.0.smithi089.stdout:885: done (11 left) 2024-04-22T18:39:29.061 INFO:tasks.rados.rados.0.smithi089.stdout:886: done (10 left) 2024-04-22T18:39:29.061 INFO:tasks.rados.rados.0.smithi089.stdout:887: done (9 left) 2024-04-22T18:39:29.061 INFO:tasks.rados.rados.0.smithi089.stdout:888: done (8 left) 2024-04-22T18:39:29.061 INFO:tasks.rados.rados.0.smithi089.stdout:890: done (7 left) 2024-04-22T18:39:29.061 INFO:tasks.rados.rados.0.smithi089.stdout:891: done (6 left) 2024-04-22T18:39:29.062 INFO:tasks.rados.rados.0.smithi089.stdout:896: read oid 6 snap -1 2024-04-22T18:39:29.062 INFO:tasks.rados.rados.0.smithi089.stdout:896: expect (ObjNum 113 snap 0 seq_num 113) 2024-04-22T18:39:29.062 INFO:tasks.rados.rados.0.smithi089.stdout:897: write oid 34 current snap is 0 2024-04-22T18:39:29.062 INFO:tasks.rados.rados.0.smithi089.stdout:893: got (22) Invalid argument 2024-04-22T18:39:29.062 INFO:tasks.rados.rados.0.smithi089.stdout:889: finishing write tid 3 to smithi08937929-12 2024-04-22T18:39:29.062 INFO:tasks.rados.rados.0.smithi089.stdout:894: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-22T18:39:29.103 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:29.104+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:29.129 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:29.129 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:29.138 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:29.139 INFO:tasks.rados.rados.0.smithi089.stdout:889: finishing write tid 4 to smithi08937929-12 2024-04-22T18:39:29.139 INFO:tasks.rados.rados.0.smithi089.stdout:889: oid 12 updating version 0 to 55 2024-04-22T18:39:29.140 INFO:tasks.rados.rados.0.smithi089.stdout:889: oid 12 version 55 is already newer than 54 2024-04-22T18:39:29.140 INFO:tasks.rados.rados.0.smithi089.stdout:889: oid 12 updating version 55 to 56 2024-04-22T18:39:29.140 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 56 (ObjNum 125 snap 0 seq_num 125) dirty exists 2024-04-22T18:39:29.140 INFO:tasks.rados.rados.0.smithi089.stdout:889: left oid 12 (ObjNum 125 snap 0 seq_num 125) 2024-04-22T18:39:29.140 INFO:tasks.rados.rados.0.smithi089.stdout:892: finishing set_chunk to oid 47 2024-04-22T18:39:29.140 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 23 (ObjNum 68 snap 0 seq_num 68) dirty exists 2024-04-22T18:39:29.140 INFO:tasks.rados.rados.0.smithi089.stdout:883: got (0) Success 2024-04-22T18:39:29.140 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 0 (ObjNum 57 snap 0 seq_num 57) dirty exists 2024-04-22T18:39:29.140 INFO:tasks.rados.rados.0.smithi089.stdout:897: seq_num 127 ranges {534489=618271,1565566=568923} 2024-04-22T18:39:29.145 INFO:tasks.rados.rados.0.smithi089.stdout:897: writing smithi08937929-34 from 534489 to 1152760 tid 1 2024-04-22T18:39:29.151 INFO:tasks.rados.rados.0.smithi089.stdout:897: writing smithi08937929-34 from 1565566 to 2134489 tid 2 2024-04-22T18:39:29.151 INFO:tasks.rados.rados.0.smithi089.stdout:883: done (7 left) 2024-04-22T18:39:29.151 INFO:tasks.rados.rados.0.smithi089.stdout:889: done (6 left) 2024-04-22T18:39:29.151 INFO:tasks.rados.rados.0.smithi089.stdout:892: done (5 left) 2024-04-22T18:39:29.151 INFO:tasks.rados.rados.0.smithi089.stdout:893: done (4 left) 2024-04-22T18:39:29.151 INFO:tasks.rados.rados.0.smithi089.stdout:894: done (3 left) 2024-04-22T18:39:29.152 INFO:tasks.rados.rados.0.smithi089.stdout:898: write (excl) oid 48 current snap is 0 2024-04-22T18:39:29.152 INFO:tasks.rados.rados.0.smithi089.stdout:895: finishing write tid 1 to smithi08937929-9 2024-04-22T18:39:29.152 INFO:tasks.rados.rados.0.smithi089.stdout:896: expect (ObjNum 113 snap 0 seq_num 113) 2024-04-22T18:39:29.204 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:29.205+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:29.241 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:29.241 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:29.250 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:29.333 INFO:tasks.rados.rados.0.smithi089.stdout:895: finishing write tid 2 to smithi08937929-9 2024-04-22T18:39:29.334 INFO:tasks.rados.rados.0.smithi089.stdout:895: finishing write tid 3 to smithi08937929-9 2024-04-22T18:39:29.334 INFO:tasks.rados.rados.0.smithi089.stdout:895: finishing write tid 4 to smithi08937929-9 2024-04-22T18:39:29.334 INFO:tasks.rados.rados.0.smithi089.stdout:895: oid 9 updating version 0 to 27 2024-04-22T18:39:29.334 INFO:tasks.rados.rados.0.smithi089.stdout:895: oid 9 version 27 is already newer than 26 2024-04-22T18:39:29.334 INFO:tasks.rados.rados.0.smithi089.stdout:895: oid 9 version 27 is already newer than 25 2024-04-22T18:39:29.334 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 27 (ObjNum 126 snap 0 seq_num 126) dirty exists 2024-04-22T18:39:29.334 INFO:tasks.rados.rados.0.smithi089.stdout:895: left oid 9 (ObjNum 126 snap 0 seq_num 126) 2024-04-22T18:39:29.334 INFO:tasks.rados.rados.0.smithi089.stdout:898: seq_num 128 ranges {551296=693520,2024048=127248} 2024-04-22T18:39:29.334 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:29.335+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:29.340 INFO:tasks.rados.rados.0.smithi089.stdout:898: writing smithi08937929-48 from 551296 to 1244816 tid 1 2024-04-22T18:39:29.345 INFO:tasks.rados.rados.0.smithi089.stdout:898: writing smithi08937929-48 from 2024048 to 2151296 tid 2 2024-04-22T18:39:29.345 INFO:tasks.rados.rados.0.smithi089.stdout:895: done (3 left) 2024-04-22T18:39:29.345 INFO:tasks.rados.rados.0.smithi089.stdout:896: done (2 left) 2024-04-22T18:39:29.346 INFO:tasks.rados.rados.0.smithi089.stdout:899: set_chunk oid 45 target oid 2024-04-22T18:39:29.392 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:29.392 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:29.401 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:29.442 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:29.443+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:29.465 INFO:tasks.rados.rados.0.smithi089.stdout:899: set_chunk oid 45 offset: 896512 length: 935424 target oid 4524e565e78b6afe126292727b556475f702074dc6da74b22bdd3f5d7b9e5364 offset: 0 snap -1 2024-04-22T18:39:29.465 INFO:tasks.rados.rados.0.smithi089.stdout:900: delete oid 13 current snap is 0 2024-04-22T18:39:29.465 INFO:tasks.rados.rados.0.smithi089.stdout:897: finishing write tid 1 to smithi08937929-34 2024-04-22T18:39:29.465 INFO:tasks.rados.rados.0.smithi089.stdout:897: finishing write tid 2 to smithi08937929-34 2024-04-22T18:39:29.465 INFO:tasks.rados.rados.0.smithi089.stdout:898: finishing write tid 1 to smithi08937929-48 2024-04-22T18:39:29.465 INFO:tasks.rados.rados.0.smithi089.stdout:898: finishing write tid 2 to smithi08937929-48 2024-04-22T18:39:29.465 INFO:tasks.rados.rados.0.smithi089.stdout:897: finishing write tid 3 to smithi08937929-34 2024-04-22T18:39:29.466 INFO:tasks.rados.rados.0.smithi089.stdout:897: finishing write tid 4 to smithi08937929-34 2024-04-22T18:39:29.466 INFO:tasks.rados.rados.0.smithi089.stdout:897: oid 34 updating version 0 to 54 2024-04-22T18:39:29.466 INFO:tasks.rados.rados.0.smithi089.stdout:897: oid 34 version 54 is already newer than 53 2024-04-22T18:39:29.466 INFO:tasks.rados.rados.0.smithi089.stdout:897: oid 34 version 54 is already newer than 52 2024-04-22T18:39:29.466 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 54 (ObjNum 127 snap 0 seq_num 127) dirty exists 2024-04-22T18:39:29.466 INFO:tasks.rados.rados.0.smithi089.stdout:897: left oid 34 (ObjNum 127 snap 0 seq_num 127) 2024-04-22T18:39:29.506 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:29.507+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:29.542 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:29.543 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:29.547 INFO:tasks.rados.rados.0.smithi089.stdout:898: finishing write tid 3 to smithi08937929-48 2024-04-22T18:39:29.547 INFO:tasks.rados.rados.0.smithi089.stdout:898: finishing write tid 4 to smithi08937929-48 2024-04-22T18:39:29.547 INFO:tasks.rados.rados.0.smithi089.stdout:898: oid 48 updating version 0 to 47 2024-04-22T18:39:29.547 INFO:tasks.rados.rados.0.smithi089.stdout:898: oid 48 version 47 is already newer than 45 2024-04-22T18:39:29.547 INFO:tasks.rados.rados.0.smithi089.stdout:898: oid 48 version 47 is already newer than 46 2024-04-22T18:39:29.547 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 47 (ObjNum 128 snap 0 seq_num 128) dirty exists 2024-04-22T18:39:29.548 INFO:tasks.rados.rados.0.smithi089.stdout:898: left oid 48 (ObjNum 128 snap 0 seq_num 128) 2024-04-22T18:39:29.552 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:29.567 INFO:tasks.rados.rados.0.smithi089.stdout:897: done (3 left) 2024-04-22T18:39:29.567 INFO:tasks.rados.rados.0.smithi089.stdout:898: done (2 left) 2024-04-22T18:39:29.567 INFO:tasks.rados.rados.0.smithi089.stdout:900: done (1 left) 2024-04-22T18:39:29.567 INFO:tasks.rados.rados.0.smithi089.stdout:901: read oid 17 snap -1 2024-04-22T18:39:29.567 INFO:tasks.rados.rados.0.smithi089.stdout:901: expect (ObjNum 101 snap 0 seq_num 101) 2024-04-22T18:39:29.567 INFO:tasks.rados.rados.0.smithi089.stdout:902: read oid 11 snap -1 2024-04-22T18:39:29.567 INFO:tasks.rados.rados.0.smithi089.stdout:902: expect (ObjNum 86 snap 0 seq_num 86) 2024-04-22T18:39:29.567 INFO:tasks.rados.rados.0.smithi089.stdout:903: read oid 6 snap -1 2024-04-22T18:39:29.567 INFO:tasks.rados.rados.0.smithi089.stdout:903: expect (ObjNum 113 snap 0 seq_num 113) 2024-04-22T18:39:29.568 INFO:tasks.rados.rados.0.smithi089.stdout:904: write oid 22 current snap is 0 2024-04-22T18:39:29.568 INFO:tasks.rados.rados.0.smithi089.stdout:904: seq_num 130 ranges {584910=560371,1857365=327545} 2024-04-22T18:39:29.573 INFO:tasks.rados.rados.0.smithi089.stdout:904: writing smithi08937929-22 from 584910 to 1145281 tid 1 2024-04-22T18:39:29.579 INFO:tasks.rados.rados.0.smithi089.stdout:904: writing smithi08937929-22 from 1857365 to 2184910 tid 2 2024-04-22T18:39:29.579 INFO:tasks.rados.rados.0.smithi089.stdout:905: tier_evict oid 16 2024-04-22T18:39:29.579 INFO:tasks.rados.rados.0.smithi089.stdout:905: tier_evict oid 16 snap -1 2024-04-22T18:39:29.580 INFO:tasks.rados.rados.0.smithi089.stdout:902: expect (ObjNum 86 snap 0 seq_num 86) 2024-04-22T18:39:29.607 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:29.608+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:29.675 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:29.675 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:29.685 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:29.726 INFO:tasks.rados.rados.0.smithi089.stdout:904: finishing write tid 1 to smithi08937929-22 2024-04-22T18:39:29.726 INFO:tasks.rados.rados.0.smithi089.stdout:903: expect (ObjNum 113 snap 0 seq_num 113) 2024-04-22T18:39:29.727 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:29.728+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:29.808 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:29.809+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:29.827 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:29.827 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:29.836 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:29.909 INFO:tasks.rados.rados.0.smithi089.stdout:905: got (22) Invalid argument 2024-04-22T18:39:29.909 INFO:tasks.rados.rados.0.smithi089.stdout:901: expect (ObjNum 101 snap 0 seq_num 101) 2024-04-22T18:39:29.910 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:29.911+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:29.977 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:29.977 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:29.986 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:30.010 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:30.011+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:30.029 INFO:tasks.rados.rados.0.smithi089.stdout:904: finishing write tid 2 to smithi08937929-22 2024-04-22T18:39:30.029 INFO:tasks.rados.rados.0.smithi089.stdout:904: finishing write tid 3 to smithi08937929-22 2024-04-22T18:39:30.029 INFO:tasks.rados.rados.0.smithi089.stdout:904: finishing write tid 4 to smithi08937929-22 2024-04-22T18:39:30.029 INFO:tasks.rados.rados.0.smithi089.stdout:904: oid 22 updating version 0 to 59 2024-04-22T18:39:30.029 INFO:tasks.rados.rados.0.smithi089.stdout:904: oid 22 version 59 is already newer than 57 2024-04-22T18:39:30.029 INFO:tasks.rados.rados.0.smithi089.stdout:904: oid 22 version 59 is already newer than 58 2024-04-22T18:39:30.029 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 59 (ObjNum 130 snap 0 seq_num 130) dirty exists 2024-04-22T18:39:30.029 INFO:tasks.rados.rados.0.smithi089.stdout:904: left oid 22 (ObjNum 130 snap 0 seq_num 130) 2024-04-22T18:39:30.029 INFO:tasks.rados.rados.0.smithi089.stdout:901: done (5 left) 2024-04-22T18:39:30.029 INFO:tasks.rados.rados.0.smithi089.stdout:902: done (4 left) 2024-04-22T18:39:30.030 INFO:tasks.rados.rados.0.smithi089.stdout:903: done (3 left) 2024-04-22T18:39:30.030 INFO:tasks.rados.rados.0.smithi089.stdout:904: done (2 left) 2024-04-22T18:39:30.030 INFO:tasks.rados.rados.0.smithi089.stdout:905: done (1 left) 2024-04-22T18:39:30.030 INFO:tasks.rados.rados.0.smithi089.stdout:906: read oid 5 snap -1 2024-04-22T18:39:30.030 INFO:tasks.rados.rados.0.smithi089.stdout:906: expect deleted 2024-04-22T18:39:30.030 INFO:tasks.rados.rados.0.smithi089.stdout:907: read oid 13 snap -1 2024-04-22T18:39:30.030 INFO:tasks.rados.rados.0.smithi089.stdout:907: expect deleted 2024-04-22T18:39:30.030 INFO:tasks.rados.rados.0.smithi089.stdout:908: tier_evict oid 25 2024-04-22T18:39:30.030 INFO:tasks.rados.rados.0.smithi089.stdout:908: tier_evict oid 25 snap -1 2024-04-22T18:39:30.030 INFO:tasks.rados.rados.0.smithi089.stdout:899: finishing set_chunk to oid 45 2024-04-22T18:39:30.030 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 44 (ObjNum 122 snap 0 seq_num 122) dirty exists 2024-04-22T18:39:30.030 INFO:tasks.rados.rados.0.smithi089.stdout:899: done (3 left) 2024-04-22T18:39:30.030 INFO:tasks.rados.rados.0.smithi089.stdout:909: write (excl) oid 37 current snap is 0 2024-04-22T18:39:30.030 INFO:tasks.rados.rados.0.smithi089.stdout:909: seq_num 131 ranges {601717=635620,1799494=402223} 2024-04-22T18:39:30.034 INFO:tasks.rados.rados.0.smithi089.stdout:909: writing smithi08937929-37 from 601717 to 1237337 tid 1 2024-04-22T18:39:30.040 INFO:tasks.rados.rados.0.smithi089.stdout:909: writing smithi08937929-37 from 1799494 to 2201717 tid 2 2024-04-22T18:39:30.040 INFO:tasks.rados.rados.0.smithi089.stdout:910: write oid 2 current snap is 0 2024-04-22T18:39:30.040 INFO:tasks.rados.rados.0.smithi089.stdout:908: got (22) Invalid argument 2024-04-22T18:39:30.040 INFO:tasks.rados.rados.0.smithi089.stdout:910: seq_num 132 ranges {618524=710869,1857976=360548} 2024-04-22T18:39:30.047 INFO:tasks.rados.rados.0.smithi089.stdout:910: writing smithi08937929-2 from 618524 to 1329393 tid 1 2024-04-22T18:39:30.052 INFO:tasks.rados.rados.0.smithi089.stdout:910: writing smithi08937929-2 from 1857976 to 2218524 tid 2 2024-04-22T18:39:30.052 INFO:tasks.rados.rados.0.smithi089.stdout:908: done (4 left) 2024-04-22T18:39:30.052 INFO:tasks.rados.rados.0.smithi089.stdout:911: tier_flush oid 29 2024-04-22T18:39:30.052 INFO:tasks.rados.rados.0.smithi089.stdout:911: tier_flush oid 29 snap -1 2024-04-22T18:39:30.052 INFO:tasks.rados.rados.0.smithi089.stdout:912: set_chunk oid 44 target oid 2024-04-22T18:39:30.111 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:30.112+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:30.130 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:30.130 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:30.140 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:30.211 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:30.212+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:30.282 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:30.282 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:30.290 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:30.324 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:39:30.334 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:30.335+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:30.396 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:30.396 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:30.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:30.450 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:30.451+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:30.474 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:30.475 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:30.491 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:30.491 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:30.507 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:30.507 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:30.513 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:30.514+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:30.520 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:30.520 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:30.523 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:30.523 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:30.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:30.538 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:30.539 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:30.555 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:30.555 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:30.558 INFO:tasks.rados.rados.0.smithi089.stdout:912: set_chunk oid 44 offset: 174592 length: 328704 target oid 894a7ed0bd7516900a81b76133f49064f02170ebfef695f8be78ebc126efac79 offset: 0 snap -1 2024-04-22T18:39:30.558 INFO:tasks.rados.rados.0.smithi089.stdout:913: read oid 12 snap -1 2024-04-22T18:39:30.558 INFO:tasks.rados.rados.0.smithi089.stdout:913: expect (ObjNum 125 snap 0 seq_num 125) 2024-04-22T18:39:30.558 INFO:tasks.rados.rados.0.smithi089.stdout:914: write (excl) oid 34 current snap is 0 2024-04-22T18:39:30.559 INFO:tasks.rados.rados.0.smithi089.stdout:910: finishing write tid 1 to smithi08937929-2 2024-04-22T18:39:30.559 INFO:tasks.rados.rados.0.smithi089.stdout:909: finishing write tid 1 to smithi08937929-37 2024-04-22T18:39:30.559 INFO:tasks.rados.rados.0.smithi089.stdout:910: finishing write tid 2 to smithi08937929-2 2024-04-22T18:39:30.559 INFO:tasks.rados.rados.0.smithi089.stdout:909: finishing write tid 2 to smithi08937929-37 2024-04-22T18:39:30.559 INFO:tasks.rados.rados.0.smithi089.stdout:910: finishing write tid 3 to smithi08937929-2 2024-04-22T18:39:30.559 INFO:tasks.rados.rados.0.smithi089.stdout:909: finishing write tid 3 to smithi08937929-37 2024-04-22T18:39:30.559 INFO:tasks.rados.rados.0.smithi089.stdout:909: finishing write tid 4 to smithi08937929-37 2024-04-22T18:39:30.559 INFO:tasks.rados.rados.0.smithi089.stdout:909: oid 37 updating version 0 to 63 2024-04-22T18:39:30.559 INFO:tasks.rados.rados.0.smithi089.stdout:909: oid 37 updating version 63 to 65 2024-04-22T18:39:30.559 INFO:tasks.rados.rados.0.smithi089.stdout:909: oid 37 version 65 is already newer than 64 2024-04-22T18:39:30.559 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 65 (ObjNum 131 snap 0 seq_num 131) dirty exists 2024-04-22T18:39:30.559 INFO:tasks.rados.rados.0.smithi089.stdout:909: left oid 37 (ObjNum 131 snap 0 seq_num 131) 2024-04-22T18:39:30.559 INFO:tasks.rados.rados.0.smithi089.stdout:910: finishing write tid 4 to smithi08937929-2 2024-04-22T18:39:30.559 INFO:tasks.rados.rados.0.smithi089.stdout:910: oid 2 updating version 0 to 29 2024-04-22T18:39:30.560 INFO:tasks.rados.rados.0.smithi089.stdout:910: oid 2 updating version 29 to 30 2024-04-22T18:39:30.560 INFO:tasks.rados.rados.0.smithi089.stdout:910: oid 2 version 30 is already newer than 28 2024-04-22T18:39:30.560 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 30 (ObjNum 132 snap 0 seq_num 132) dirty exists 2024-04-22T18:39:30.560 INFO:tasks.rados.rados.0.smithi089.stdout:910: left oid 2 (ObjNum 132 snap 0 seq_num 132) 2024-04-22T18:39:30.560 INFO:tasks.rados.rados.0.smithi089.stdout:914: seq_num 133 ranges {635331=786118,1916458=318873} 2024-04-22T18:39:30.566 INFO:tasks.rados.rados.0.smithi089.stdout:914: writing smithi08937929-34 from 635331 to 1421449 tid 1 2024-04-22T18:39:30.571 INFO:tasks.rados.rados.0.smithi089.stdout:914: writing smithi08937929-34 from 1916458 to 2235331 tid 2 2024-04-22T18:39:30.571 INFO:tasks.rados.rados.0.smithi089.stdout:906: done (7 left) 2024-04-22T18:39:30.571 INFO:tasks.rados.rados.0.smithi089.stdout:907: done (6 left) 2024-04-22T18:39:30.571 INFO:tasks.rados.rados.0.smithi089.stdout:909: done (5 left) 2024-04-22T18:39:30.571 INFO:tasks.rados.rados.0.smithi089.stdout:910: done (4 left) 2024-04-22T18:39:30.571 INFO:tasks.rados.rados.0.smithi089.stdout:915: read oid 32 snap -1 2024-04-22T18:39:30.571 INFO:tasks.rados.rados.0.smithi089.stdout:915: expect (ObjNum 92 snap 0 seq_num 92) 2024-04-22T18:39:30.571 INFO:tasks.rados.rados.0.smithi089.stdout:916: write oid 39 current snap is 0 2024-04-22T18:39:30.571 INFO:tasks.rados.rados.0.smithi089.stdout:912: finishing set_chunk to oid 44 2024-04-22T18:39:30.571 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 44 set_chunk returned error code -95 offset: 174592 length: 328704 2024-04-22T18:39:30.572 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 0 (ObjNum 119 snap 0 seq_num 119) dirty exists 2024-04-22T18:39:30.572 INFO:tasks.rados.rados.0.smithi089.stdout:916: seq_num 134 ranges {652138=461367,1574940=677198} 2024-04-22T18:39:30.572 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:30.573 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:30.576 INFO:tasks.rados.rados.0.smithi089.stdout:916: writing smithi08937929-39 from 652138 to 1113505 tid 1 2024-04-22T18:39:30.583 INFO:tasks.rados.rados.0.smithi089.stdout:916: writing smithi08937929-39 from 1574940 to 2252138 tid 2 2024-04-22T18:39:30.583 INFO:tasks.rados.rados.0.smithi089.stdout:912: done (5 left) 2024-04-22T18:39:30.583 INFO:tasks.rados.rados.0.smithi089.stdout:917: set_chunk oid 31 target oid 2024-04-22T18:39:30.589 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:30.590 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:30.591 INFO:tasks.rados.rados.0.smithi089.stdout:917: set_chunk oid 31 offset: 1933312 length: 1110528 target oid ab135f61e5165b5223cd7a69e4aaec0526b8ba213f19b11df21408ad44c91bf2 offset: 0 snap -1 2024-04-22T18:39:30.591 INFO:tasks.rados.rados.0.smithi089.stdout:918: write oid 22 current snap is 0 2024-04-22T18:39:30.591 INFO:tasks.rados.rados.0.smithi089.stdout:911: got (0) Success 2024-04-22T18:39:30.591 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 0 (ObjNum 63 snap 0 seq_num 63) dirty exists 2024-04-22T18:39:30.591 INFO:tasks.rados.rados.0.smithi089.stdout:918: seq_num 135 ranges {668945=536616,1633422=491363} 2024-04-22T18:39:30.597 INFO:tasks.rados.rados.0.smithi089.stdout:918: writing smithi08937929-22 from 668945 to 1205561 tid 1 2024-04-22T18:39:30.602 INFO:tasks.rados.rados.0.smithi089.stdout:918: writing smithi08937929-22 from 1633422 to 2124785 tid 2 2024-04-22T18:39:30.603 INFO:tasks.rados.rados.0.smithi089.stdout:911: done (6 left) 2024-04-22T18:39:30.603 INFO:tasks.rados.rados.0.smithi089.stdout:919: read oid 8 snap -1 2024-04-22T18:39:30.603 INFO:tasks.rados.rados.0.smithi089.stdout:919: expect deleted 2024-04-22T18:39:30.603 INFO:tasks.rados.rados.0.smithi089.stdout:920: set_chunk oid 37 target oid 2024-04-22T18:39:30.614 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:30.615+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:30.652 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:39:30.674 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:30.674 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:30.675 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:30.675 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:30.675 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:30.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:30.714 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:30.716+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:30.788 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:30.788 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:30.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:30.788 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:30.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.788 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:30.788 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:30.788 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.625681266000001, 2024-04-22T18:39:30.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:39:30.789 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.789 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:30.789 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.789 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.789 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.789 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:39:30.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.789 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:30.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:30.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:39:30.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:30.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:39:30.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:39:30.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:39:30.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:39:30.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:39:30.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:39:30.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:39:30.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:39:30.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:30.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:39:30.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:39:30.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:30.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:39:30.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:39:30.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:39:30.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:39:30.794 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.794 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.794 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.794 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:30.794 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:30.794 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.524122445000003, 2024-04-22T18:39:30.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:39:30.794 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.794 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:30.794 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.794 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.795 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.795 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:39:30.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.795 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:30.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:30.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:39:30.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:30.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:39:30.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:39:30.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:39:30.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:39:30.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:39:30.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:39:30.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:39:30.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:39:30.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:39:30.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:39:30.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:39:30.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:39:30.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:39:30.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:30.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:39:30.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:39:30.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:30.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:39:30.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:39:30.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:30.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:39:30.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:39:30.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.524096043, 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.801 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:39:30.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.802 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:30.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:30.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:39:30.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:30.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:39:30.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:39:30.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:39:30.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:39:30.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:39:30.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:39:30.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:39:30.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:39:30.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:30.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:39:30.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:39:30.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:39:30.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:39:30.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:39:30.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:39:30.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:39:30.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:39:30.805 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.806 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.806 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.806 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:30.806 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:30.806 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.125761713999999, 2024-04-22T18:39:30.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:39:30.806 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.806 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:30.806 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.806 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.806 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.806 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:39:30.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.807 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:30.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:30.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:30.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:30.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:39:30.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:39:30.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:39:30.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:39:30.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:39:30.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:39:30.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:39:30.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:39:30.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:39:30.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:39:30.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:30.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:39:30.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:39:30.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:30.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:39:30.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:39:30.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:30.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:39:30.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:39:30.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:30.811 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:30.812 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.024925416000002, 2024-04-22T18:39:30.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:39:30.812 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.812 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:30.812 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.812 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.812 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.812 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:39:30.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.812 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:30.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:30.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:39:30.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:30.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:39:30.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:39:30.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:39:30.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:39:30.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:39:30.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:39:30.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:39:30.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:39:30.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:39:30.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:39:30.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:30.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:39:30.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:39:30.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:30.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:39:30.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:39:30.816 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.817 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.817 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.817 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:30.817 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:30.817 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.024768146, 2024-04-22T18:39:30.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:39:30.817 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.817 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:30.817 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.817 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.817 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.817 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:39:30.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.818 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:30.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:30.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:39:30.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:30.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:39:30.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:39:30.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:39:30.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:39:30.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:39:30.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:39:30.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:30.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:39:30.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:39:30.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:30.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:39:30.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:39:30.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:30.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:39:30.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:39:30.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:39:30.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:39:30.822 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.822 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.822 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.822 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:30.822 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:30.822 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.024753537000002, 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:39:30.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:39:30.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:39:30.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:39:30.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:39:30.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:39:30.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:30.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:39:30.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:39:30.826 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.827 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.827 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.827 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:30.827 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:30.827 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.024743151999999, 2024-04-22T18:39:30.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:39:30.827 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.827 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:30.827 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.827 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.827 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.827 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:39:30.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.828 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:30.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:30.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:39:30.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:30.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:39:30.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:39:30.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:39:30.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:39:30.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:39:30.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:39:30.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:39:30.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:39:30.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:39:30.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:39:30.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:30.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:39:30.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:39:30.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:30.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:39:30.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:39:30.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:30.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:39:30.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:39:30.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:30.833 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:30.834 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.023813222000001, 2024-04-22T18:39:30.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:39:30.834 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.834 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:30.834 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.834 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.834 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.834 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:39:30.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.834 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:30.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:39:30.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:39:30.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:30.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:39:30.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:39:30.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:30.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:39:30.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:39:30.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:30.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.839 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:30.839 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:30.839 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.023351097000003, 2024-04-22T18:39:30.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:39:30.839 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.839 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:30.839 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.839 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.839 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.839 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:39:30.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.839 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:39:30.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:39:30.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:39:30.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:39:30.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:30.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:39:30.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:39:30.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:30.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:39:30.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:39:30.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:30.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:39:30.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:39:30.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:30.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:39:30.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:39:30.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:39:30.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:39:30.843 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.023338293000002, 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.844 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:39:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:39:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:39:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:39:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:39:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:39:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:109 4.1 4:83c955bf:::smithi08937929-28:head [write 453789~780782,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 35.820551299999998, 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.45086113300000003, 2024-04-22T18:39:30.848 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:30.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967588+0000", 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.968243+0000", 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00065488500000000002 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.968265+0000", 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1997e-05 2024-04-22T18:39:30.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.068340+0000", 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007516 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168464+0000", 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100124267 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168579+0000", 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114387 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168692+0000", 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011335100000000001 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.173549+0000", 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048564430000000002 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418367+0000", 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.244817962 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:30.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418419+0000", 2024-04-22T18:39:30.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2883000000000002e-05 2024-04-22T18:39:30.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418450+0000", 2024-04-22T18:39:30.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0278999999999999e-05 2024-04-22T18:39:30.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.853 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.853 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:30.853 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:30.853 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.695565836999997, 2024-04-22T18:39:30.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:39:30.854 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.854 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:30.854 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.854 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.854 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.854 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:39:30.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.854 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:30.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:39:30.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:39:30.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:39:30.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:39:30.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:39:30.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:39:30.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:39:30.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.858 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 31.686992725, 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:39:30.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:39:30.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:30.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:39:30.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:39:30.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:30.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:39:30.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:39:30.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:30.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:39:30.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:36 3.3 3:ea25b818:::smithi08937929-10 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.25944969, 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40851152800000001, 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528690+0000", 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528694+0000", 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2459999999999997e-06 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528701+0000", 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7950000000000002e-06 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528709+0000", 2024-04-22T18:39:30.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1000000000000004e-06 2024-04-22T18:39:30.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528782+0000", 2024-04-22T18:39:30.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3699e-05 2024-04-22T18:39:30.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:30.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528785+0000", 2024-04-22T18:39:30.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:39:30.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735367+0000", 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20658175100000001 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735443+0000", 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6106999999999995e-05 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735573+0000", 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129734 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.740966+0000", 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0053933130000000003 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937143+0000", 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19617662399999999 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937175+0000", 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1860999999999999e-05 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937202+0000", 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7029000000000002e-05 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:37 3.3 3:ea25b818:::smithi08937929-10 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.528720+0000", 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.259420374000001, 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40858023100000002, 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528720+0000", 2024-04-22T18:39:30.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528720+0000", 2024-04-22T18:39:30.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528719+0000", 2024-04-22T18:39:30.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:30.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528724+0000", 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1159999999999998e-06 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.528730+0000", 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0480000000000004e-06 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.628843+0000", 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100112425 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.628917+0000", 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4385e-05 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.628923+0000", 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4239999999999998e-06 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735762+0000", 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10683828200000001 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735826+0000", 2024-04-22T18:39:30.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4144000000000001e-05 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.735941+0000", 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011526799999999999 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.740973+0000", 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0050320340000000003 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937246+0000", 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19627275499999999 2024-04-22T18:39:30.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:30.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937277+0000", 2024-04-22T18:39:30.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1563999999999998e-05 2024-04-22T18:39:30.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.937300+0000", 2024-04-22T18:39:30.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2350000000000001e-05 2024-04-22T18:39:30.874 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.874 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.874 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1108 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 3378174] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.637716+0000", 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.150423896, 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41175930500000002, 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1108 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637716+0000", 2024-04-22T18:39:30.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637716+0000", 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637715+0000", 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637719+0000", 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1710000000000004e-06 2024-04-22T18:39:30.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637725+0000", 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7790000000000001e-06 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637730+0000", 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7330000000000003e-06 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.948647+0000", 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31091748400000002 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.948704+0000", 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7000000000000003e-05 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.948802+0000", 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8107000000000001e-05 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049249+0000", 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100446991 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:30.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049270+0000", 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0336e-05 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049369+0000", 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9256e-05 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049475+0000", 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010624900000000001 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1109 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.637734+0000", 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.150406287999999, 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41190551399999997, 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1109 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637734+0000", 2024-04-22T18:39:30.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637734+0000", 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637733+0000", 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637736+0000", 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4530000000000001e-06 2024-04-22T18:39:30.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637741+0000", 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.118e-06 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637745+0000", 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4490000000000001e-06 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.949123+0000", 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31137820900000002 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:30.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.949205+0000", 2024-04-22T18:39:30.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1444000000000002e-05 2024-04-22T18:39:30.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049516+0000", 2024-04-22T18:39:30.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031090099999999 2024-04-22T18:39:30.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049573+0000", 2024-04-22T18:39:30.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7497000000000002e-05 2024-04-22T18:39:30.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049639+0000", 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5906000000000002e-05 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1285 4.d 4:bee1184f:::smithi08937929-7:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 12.950875051000001, 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.42388892500000003, 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.884 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1285 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837262+0000", 2024-04-22T18:39:30.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:30.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837273+0000", 2024-04-22T18:39:30.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1364e-05 2024-04-22T18:39:30.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837300+0000", 2024-04-22T18:39:30.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6928999999999999e-05 2024-04-22T18:39:30.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837318+0000", 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8219999999999998e-05 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837445+0000", 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000126415 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837627+0000", 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000182182 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.261154+0000", 2024-04-22T18:39:30.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.42352673499999999 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.65234521, 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.888 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.891 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:30.892 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:30.892 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:30.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:30.893 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:30.894+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:30.909 INFO:tasks.rados.rados.0.smithi089.stdout:920: set_chunk oid 37 offset: 900608 length: 1066496 target oid e7eb8db0531d9f456e91fe87bc79e47ebd80459f1783dc2a5e53d4822cd34f49 offset: 0 snap -1 2024-04-22T18:39:30.909 INFO:tasks.rados.rados.0.smithi089.stdout:921: write oid 41 current snap is 0 2024-04-22T18:39:30.909 INFO:tasks.rados.rados.0.smithi089.stdout:913: expect (ObjNum 125 snap 0 seq_num 125) 2024-04-22T18:39:30.993 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:30.994+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:31.013 INFO:tasks.rados.rados.0.smithi089.stdout:915: expect (ObjNum 92 snap 0 seq_num 92) 2024-04-22T18:39:31.035 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:31.035 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:31.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.036 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:175 3.2 3:743dff8d:::b16bcd995e8e978339a639e7aefc2bc918da5618224f280957524b4ff91d579c:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:39:31.036 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.878468+0000", 2024-04-22T18:39:31.036 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.15894177300000001, 2024-04-22T18:39:31.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.15896765299999999, 2024-04-22T18:39:31.036 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.036 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:39:31.036 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.878468+0000", 2024-04-22T18:39:31.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.878468+0000", 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.878465+0000", 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.878476+0000", 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1046999999999999e-05 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.878492+0000", 2024-04-22T18:39:31.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5546999999999999e-05 2024-04-22T18:39:31.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.878501+0000", 2024-04-22T18:39:31.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3249999999999997e-06 2024-04-22T18:39:31.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.878551+0000", 2024-04-22T18:39:31.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9651999999999998e-05 2024-04-22T18:39:31.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.878647+0000", 2024-04-22T18:39:31.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6465000000000006e-05 2024-04-22T18:39:31.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:31.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.878816+0000", 2024-04-22T18:39:31.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016820100000000001 2024-04-22T18:39:31.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.879465+0000", 2024-04-22T18:39:31.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064917999999999996 2024-04-22T18:39:31.039 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.039 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.039 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.040 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:177 3.2 3:78b494cf:::fd80e25a1439578d38e7bea01ac41bfa39047c47b65a20e114bb208aa373d88e:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:39:31.040 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.878717+0000", 2024-04-22T18:39:31.040 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.15869280199999999, 2024-04-22T18:39:31.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.15876116100000001, 2024-04-22T18:39:31.040 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.040 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:39:31.040 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.878717+0000", 2024-04-22T18:39:31.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.878717+0000", 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.878716+0000", 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.878723+0000", 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.515e-06 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.878737+0000", 2024-04-22T18:39:31.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4406999999999999e-05 2024-04-22T18:39:31.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.878745+0000", 2024-04-22T18:39:31.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8960000000000003e-06 2024-04-22T18:39:31.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.879128+0000", 2024-04-22T18:39:31.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038307700000000001 2024-04-22T18:39:31.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.879204+0000", 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5680999999999996e-05 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.879288+0000", 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3850999999999994e-05 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.880261+0000", 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00097309999999999996 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.044 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.4.13:175 3.2 e47/29)", 2024-04-22T18:39:31.044 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.979830+0000", 2024-04-22T18:39:31.044 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.057580198999999999, 2024-04-22T18:39:31.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.057681930999999999, 2024-04-22T18:39:31.044 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.044 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:31.044 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.979830+0000", 2024-04-22T18:39:31.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.979830+0000", 2024-04-22T18:39:31.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.979828+0000", 2024-04-22T18:39:31.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:31.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.979837+0000", 2024-04-22T18:39:31.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8779999999999993e-06 2024-04-22T18:39:31.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.979853+0000", 2024-04-22T18:39:31.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5073e-05 2024-04-22T18:39:31.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.979863+0000", 2024-04-22T18:39:31.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0924000000000001e-05 2024-04-22T18:39:31.047 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.047 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.047 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.047 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(osd.4.13:177 3.2 e47/29)", 2024-04-22T18:39:31.047 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.980170+0000", 2024-04-22T18:39:31.047 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.057239722, 2024-04-22T18:39:31.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.057363404999999999, 2024-04-22T18:39:31.048 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.048 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:31.048 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.980170+0000", 2024-04-22T18:39:31.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.980170+0000", 2024-04-22T18:39:31.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.980170+0000", 2024-04-22T18:39:31.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:31.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.980173+0000", 2024-04-22T18:39:31.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8979999999999998e-06 2024-04-22T18:39:31.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.980181+0000", 2024-04-22T18:39:31.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7009999999999996e-06 2024-04-22T18:39:31.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.980187+0000", 2024-04-22T18:39:31.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6869999999999999e-06 2024-04-22T18:39:31.050 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.050 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.050 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.050 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4206.0:1 2.2 2.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:39:31.050 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.981180+0000", 2024-04-22T18:39:31.050 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.056230316000000002, 2024-04-22T18:39:31.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.056378013999999997, 2024-04-22T18:39:31.050 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.050 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:31.050 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.051 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4206", 2024-04-22T18:39:31.051 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/715792134", 2024-04-22T18:39:31.051 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:39:31.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.051 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.981180+0000", 2024-04-22T18:39:31.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.981180+0000", 2024-04-22T18:39:31.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.981178+0000", 2024-04-22T18:39:31.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:31.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.981184+0000", 2024-04-22T18:39:31.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5820000000000001e-06 2024-04-22T18:39:31.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.981191+0000", 2024-04-22T18:39:31.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8700000000000003e-06 2024-04-22T18:39:31.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.981196+0000", 2024-04-22T18:39:31.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9429999999999999e-06 2024-04-22T18:39:31.053 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.053 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.053 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.053 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.053 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:39:31.053 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 5 2024-04-22T18:39:31.053 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:31.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:31.096 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:31.097+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:31.195 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:31.196+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:31.196 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:31.197 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:31.197 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:31.197 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:31.197 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:31.206 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:31.244 INFO:tasks.rados.rados.0.smithi089.stdout:914: finishing write tid 1 to smithi08937929-34 2024-04-22T18:39:31.245 INFO:tasks.rados.rados.0.smithi089.stdout:914: finishing write tid 2 to smithi08937929-34 2024-04-22T18:39:31.245 INFO:tasks.rados.rados.0.smithi089.stdout:917: finishing set_chunk to oid 31 2024-04-22T18:39:31.245 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 24 (ObjNum 74 snap 0 seq_num 74) dirty exists 2024-04-22T18:39:31.245 INFO:tasks.rados.rados.0.smithi089.stdout:914: finishing write tid 3 to smithi08937929-34 2024-04-22T18:39:31.245 INFO:tasks.rados.rados.0.smithi089.stdout:914: finishing write tid 4 to smithi08937929-34 2024-04-22T18:39:31.245 INFO:tasks.rados.rados.0.smithi089.stdout:914: oid 34 updating version 0 to 57 2024-04-22T18:39:31.245 INFO:tasks.rados.rados.0.smithi089.stdout:914: oid 34 version 57 is already newer than 55 2024-04-22T18:39:31.245 INFO:tasks.rados.rados.0.smithi089.stdout:914: oid 34 version 57 is already newer than 56 2024-04-22T18:39:31.245 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 57 (ObjNum 133 snap 0 seq_num 133) dirty exists 2024-04-22T18:39:31.245 INFO:tasks.rados.rados.0.smithi089.stdout:914: left oid 34 (ObjNum 133 snap 0 seq_num 133) 2024-04-22T18:39:31.245 INFO:tasks.rados.rados.0.smithi089.stdout:916: finishing write tid 1 to smithi08937929-39 2024-04-22T18:39:31.245 INFO:tasks.rados.rados.0.smithi089.stdout:921: seq_num 136 ranges {685752=611865,1975551=310201} 2024-04-22T18:39:31.251 INFO:tasks.rados.rados.0.smithi089.stdout:921: writing smithi08937929-41 from 685752 to 1297617 tid 1 2024-04-22T18:39:31.257 INFO:tasks.rados.rados.0.smithi089.stdout:921: writing smithi08937929-41 from 1975551 to 2285752 tid 2 2024-04-22T18:39:31.257 INFO:tasks.rados.rados.0.smithi089.stdout:913: done (8 left) 2024-04-22T18:39:31.257 INFO:tasks.rados.rados.0.smithi089.stdout:914: done (7 left) 2024-04-22T18:39:31.257 INFO:tasks.rados.rados.0.smithi089.stdout:915: done (6 left) 2024-04-22T18:39:31.258 INFO:tasks.rados.rados.0.smithi089.stdout:917: done (5 left) 2024-04-22T18:39:31.258 INFO:tasks.rados.rados.0.smithi089.stdout:919: done (4 left) 2024-04-22T18:39:31.258 INFO:tasks.rados.rados.0.smithi089.stdout:922: read oid 18 snap -1 2024-04-22T18:39:31.258 INFO:tasks.rados.rados.0.smithi089.stdout:922: expect (ObjNum 17 snap 0 seq_num 17) 2024-04-22T18:39:31.258 INFO:tasks.rados.rados.0.smithi089.stdout:923: read oid 7 snap -1 2024-04-22T18:39:31.258 INFO:tasks.rados.rados.0.smithi089.stdout:923: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-22T18:39:31.258 INFO:tasks.rados.rados.0.smithi089.stdout:924: write (excl) oid 11 current snap is 0 2024-04-22T18:39:31.258 INFO:tasks.rados.rados.0.smithi089.stdout:916: finishing write tid 2 to smithi08937929-39 2024-04-22T18:39:31.258 INFO:tasks.rados.rados.0.smithi089.stdout:918: finishing write tid 1 to smithi08937929-22 2024-04-22T18:39:31.258 INFO:tasks.rados.rados.0.smithi089.stdout:918: finishing write tid 2 to smithi08937929-22 2024-04-22T18:39:31.258 INFO:tasks.rados.rados.0.smithi089.stdout:924: seq_num 137 ranges {702559=403467,1750386=495032} 2024-04-22T18:39:31.263 INFO:tasks.rados.rados.0.smithi089.stdout:924: writing smithi08937929-11 from 702559 to 1106026 tid 1 2024-04-22T18:39:31.270 INFO:tasks.rados.rados.0.smithi089.stdout:924: writing smithi08937929-11 from 1750386 to 2245418 tid 2 2024-04-22T18:39:31.270 INFO:tasks.rados.rados.0.smithi089.stdout:925: read oid 42 snap -1 2024-04-22T18:39:31.270 INFO:tasks.rados.rados.0.smithi089.stdout:925: expect (ObjNum 105 snap 0 seq_num 105) 2024-04-22T18:39:31.270 INFO:tasks.rados.rados.0.smithi089.stdout:926: read oid 36 snap -1 2024-04-22T18:39:31.271 INFO:tasks.rados.rados.0.smithi089.stdout:926: expect (ObjNum 80 snap 0 seq_num 80) 2024-04-22T18:39:31.271 INFO:tasks.rados.rados.0.smithi089.stdout:927: set_chunk oid 3 target oid 2024-04-22T18:39:31.271 INFO:tasks.rados.rados.0.smithi089.stdout:916: finishing write tid 3 to smithi08937929-39 2024-04-22T18:39:31.271 INFO:tasks.rados.rados.0.smithi089.stdout:916: finishing write tid 4 to smithi08937929-39 2024-04-22T18:39:31.271 INFO:tasks.rados.rados.0.smithi089.stdout:916: oid 39 updating version 0 to 59 2024-04-22T18:39:31.271 INFO:tasks.rados.rados.0.smithi089.stdout:916: oid 39 version 59 is already newer than 58 2024-04-22T18:39:31.271 INFO:tasks.rados.rados.0.smithi089.stdout:916: oid 39 updating version 59 to 60 2024-04-22T18:39:31.271 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 60 (ObjNum 134 snap 0 seq_num 134) dirty exists 2024-04-22T18:39:31.271 INFO:tasks.rados.rados.0.smithi089.stdout:916: left oid 39 (ObjNum 134 snap 0 seq_num 134) 2024-04-22T18:39:31.271 INFO:tasks.rados.rados.0.smithi089.stdout:927: set_chunk oid 3 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:39:31.271 INFO:tasks.rados.rados.0.smithi089.stdout:916: done (9 left) 2024-04-22T18:39:31.271 INFO:tasks.rados.rados.0.smithi089.stdout:928: read oid 8 snap -1 2024-04-22T18:39:31.271 INFO:tasks.rados.rados.0.smithi089.stdout:928: expect deleted 2024-04-22T18:39:31.271 INFO:tasks.rados.rados.0.smithi089.stdout:929: read oid 17 snap -1 2024-04-22T18:39:31.271 INFO:tasks.rados.rados.0.smithi089.stdout:929: expect (ObjNum 101 snap 0 seq_num 101) 2024-04-22T18:39:31.272 INFO:tasks.rados.rados.0.smithi089.stdout:930: read oid 29 snap -1 2024-04-22T18:39:31.272 INFO:tasks.rados.rados.0.smithi089.stdout:930: expect (ObjNum 63 snap 0 seq_num 63) 2024-04-22T18:39:31.272 INFO:tasks.rados.rados.0.smithi089.stdout:931: read oid 44 snap -1 2024-04-22T18:39:31.272 INFO:tasks.rados.rados.0.smithi089.stdout:931: expect (ObjNum 119 snap 0 seq_num 119) 2024-04-22T18:39:31.272 INFO:tasks.rados.rados.0.smithi089.stdout:932: write oid 28 current snap is 0 2024-04-22T18:39:31.272 INFO:tasks.rados.rados.0.smithi089.stdout:918: finishing write tid 3 to smithi08937929-22 2024-04-22T18:39:31.272 INFO:tasks.rados.rados.0.smithi089.stdout:932: seq_num 138 ranges {719366=478716,1808868=510498} 2024-04-22T18:39:31.275 INFO:tasks.rados.rados.0.smithi089.stdout:932: writing smithi08937929-28 from 719366 to 1198082 tid 1 2024-04-22T18:39:31.282 INFO:tasks.rados.rados.0.smithi089.stdout:932: writing smithi08937929-28 from 1808868 to 2319366 tid 2 2024-04-22T18:39:31.282 INFO:tasks.rados.rados.0.smithi089.stdout:933: set_chunk oid 1 target oid 2024-04-22T18:39:31.296 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:31.297+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:31.349 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:31.349 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:31.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:31.350 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:31.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.350 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:31.350 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:31.350 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.255744596, 2024-04-22T18:39:31.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:39:31.350 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.350 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.350 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.350 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.350 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.350 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:39:31.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.351 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:31.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:31.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:39:31.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:31.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:39:31.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:39:31.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:39:31.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:39:31.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:39:31.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:39:31.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:39:31.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:39:31.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:39:31.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:39:31.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:31.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:39:31.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:39:31.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:39:31.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:39:31.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:39:31.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:39:31.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2 4.8 4:12d9f826:::smithi08937929-1:head [write 1542129~474678 in=474678b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:31.355 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.155033326000002, 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20435667099999999, 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194571+0000", 2024-04-22T18:39:31.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194568+0000", 2024-04-22T18:39:31.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:31.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194869+0000", 2024-04-22T18:39:31.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00030115200000000001 2024-04-22T18:39:31.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194879+0000", 2024-04-22T18:39:31.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8479999999999996e-06 2024-04-22T18:39:31.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294954+0000", 2024-04-22T18:39:31.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100074702 2024-04-22T18:39:31.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296019+0000", 2024-04-22T18:39:31.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010649189999999999 2024-04-22T18:39:31.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296086+0000", 2024-04-22T18:39:31.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7261000000000005e-05 2024-04-22T18:39:31.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:31.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296173+0000", 2024-04-22T18:39:31.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6935999999999996e-05 2024-04-22T18:39:31.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297196+0000", 2024-04-22T18:39:31.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023415 2024-04-22T18:39:31.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398879+0000", 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101682563 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398906+0000", 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6526999999999999e-05 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398928+0000", 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2045e-05 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.360 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.361 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=62b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:31.361 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:39:31.361 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.054632372, 2024-04-22T18:39:31.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20408478299999999, 2024-04-22T18:39:31.361 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.361 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.361 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.361 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.361 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.361 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3 2024-04-22T18:39:31.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.361 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:39:31.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294972+0000", 2024-04-22T18:39:31.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294970+0000", 2024-04-22T18:39:31.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:31.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294983+0000", 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3429e-05 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.294993+0000", 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0416e-05 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295007+0000", 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3742e-05 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296793+0000", 2024-04-22T18:39:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0017860420000000001 2024-04-22T18:39:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296841+0000", 2024-04-22T18:39:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7706e-05 2024-04-22T18:39:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.296913+0000", 2024-04-22T18:39:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2578999999999997e-05 2024-04-22T18:39:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297946+0000", 2024-04-22T18:39:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001032802 2024-04-22T18:39:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498963+0000", 2024-04-22T18:39:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20101640400000001 2024-04-22T18:39:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.498998+0000", 2024-04-22T18:39:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5759000000000001e-05 2024-04-22T18:39:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499057+0000", 2024-04-22T18:39:31.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8270999999999998e-05 2024-04-22T18:39:31.366 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.366 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.366 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.366 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:31.366 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:39:31.366 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.054591944000002, 2024-04-22T18:39:31.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20418823899999999, 2024-04-22T18:39:31.366 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.367 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:31.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.367 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4 2024-04-22T18:39:31.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.367 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:39:31.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295012+0000", 2024-04-22T18:39:31.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295011+0000", 2024-04-22T18:39:31.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:31.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295016+0000", 2024-04-22T18:39:31.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2400000000000001e-06 2024-04-22T18:39:31.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295022+0000", 2024-04-22T18:39:31.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8519999999999998e-06 2024-04-22T18:39:31.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295027+0000", 2024-04-22T18:39:31.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9749999999999998e-06 2024-04-22T18:39:31.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297126+0000", 2024-04-22T18:39:31.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020981070000000001 2024-04-22T18:39:31.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:31.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297167+0000", 2024-04-22T18:39:31.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1842999999999999e-05 2024-04-22T18:39:31.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499096+0000", 2024-04-22T18:39:31.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201928846 2024-04-22T18:39:31.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499142+0000", 2024-04-22T18:39:31.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6152000000000001e-05 2024-04-22T18:39:31.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.499201+0000", 2024-04-22T18:39:31.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8190999999999999e-05 2024-04-22T18:39:31.371 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.371 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.371 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.371 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop(client.4502.0:70 4.3 e27/20 4:c6fa81cf:::smithi08937929-18:head v 27'8, mlcod=27'3)", 2024-04-22T18:39:31.371 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:31.371 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.392112028, 2024-04-22T18:39:31.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20440983300000001, 2024-04-22T18:39:31.371 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.371 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.371 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:31.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:31.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957492+0000", 2024-04-22T18:39:31.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:31.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957839+0000", 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000346981 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957849+0000", 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8060000000000006e-06 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.957856+0000", 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0319999999999998e-06 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958600+0000", 2024-04-22T18:39:31.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074474700000000003 2024-04-22T18:39:31.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.958615+0000", 2024-04-22T18:39:31.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4961e-05 2024-04-22T18:39:31.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161867+0000", 2024-04-22T18:39:31.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20325188399999999 2024-04-22T18:39:31.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.161902+0000", 2024-04-22T18:39:31.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4884e-05 2024-04-22T18:39:31.375 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.375 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.375 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.375 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:31.375 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:31.375 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.356246720999998, 2024-04-22T18:39:31.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:39:31.375 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.376 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.376 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:31.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:31.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:39:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:39:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:39:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:39:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:39:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:39:31.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:39:31.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:39:31.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:39:31.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:39:31.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:39:31.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:39:31.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:39:31.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:39:31.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:39:31.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:39:31.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:31.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:39:31.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:39:31.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:39:31.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:39:31.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:39:31.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:39:31.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:39:31.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:39:31.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:39:31.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:39:31.381 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.381 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.381 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.381 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:27 3.2 3:7a6cba00:::smithi08937929-32 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:31.381 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.597045+0000", 2024-04-22T18:39:31.381 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.752559228000003, 2024-04-22T18:39:31.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20443814799999999, 2024-04-22T18:39:31.381 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.381 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.382 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597045+0000", 2024-04-22T18:39:31.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597045+0000", 2024-04-22T18:39:31.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597043+0000", 2024-04-22T18:39:31.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:31.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597053+0000", 2024-04-22T18:39:31.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0278999999999999e-05 2024-04-22T18:39:31.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.597074+0000", 2024-04-22T18:39:31.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0710999999999999e-05 2024-04-22T18:39:31.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.697204+0000", 2024-04-22T18:39:31.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10012958299999999 2024-04-22T18:39:31.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.698362+0000", 2024-04-22T18:39:31.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011587629999999999 2024-04-22T18:39:31.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.698440+0000", 2024-04-22T18:39:31.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7453000000000005e-05 2024-04-22T18:39:31.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:31.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.698593+0000", 2024-04-22T18:39:31.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153449 2024-04-22T18:39:31.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.699048+0000", 2024-04-22T18:39:31.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000454287 2024-04-22T18:39:31.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.801433+0000", 2024-04-22T18:39:31.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102385664 2024-04-22T18:39:31.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.801462+0000", 2024-04-22T18:39:31.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9153e-05 2024-04-22T18:39:31.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.801483+0000", 2024-04-22T18:39:31.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0707e-05 2024-04-22T18:39:31.386 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.386 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.386 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.387 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:31.387 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:31.387 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.247946929999998, 2024-04-22T18:39:31.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:39:31.387 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.387 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.387 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.387 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.387 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.387 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:39:31.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.387 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:31.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:31.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:39:31.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:31.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:39:31.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:39:31.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:39:31.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:39:31.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:39:31.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:39:31.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:39:31.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:39:31.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:39:31.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:39:31.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:31.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:39:31.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:39:31.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:39:31.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:39:31.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:39:31.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:39:31.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:39:31.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:39:31.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:39:31.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:39:31.392 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.392 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.392 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.392 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:31.392 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:31.393 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.916403301999999, 2024-04-22T18:39:31.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:39:31.393 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.393 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.393 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:31.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:31.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:39:31.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:31.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:39:31.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:39:31.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:39:31.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:39:31.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:39:31.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:39:31.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:39:31.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:39:31.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:39:31.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:39:31.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:31.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:39:31.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:39:31.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:39:31.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:39:31.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:39:31.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:39:31.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:39:31.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:39:31.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:39:31.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:39:31.397 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.398 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.398 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.398 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:31.398 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:31.398 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.825317494, 2024-04-22T18:39:31.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:39:31.398 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.398 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.398 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.398 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.398 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.398 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:39:31.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.399 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:31.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:31.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:39:31.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:31.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:39:31.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:39:31.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:39:31.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:39:31.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:39:31.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:39:31.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:39:31.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:39:31.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:39:31.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:39:31.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:39:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:39:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:39:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:39:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:39:31.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:39:31.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:39:31.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:39:31.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:39:31.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:39:31.403 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.403 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.403 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.404 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:31.404 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:31.404 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.819517233999999, 2024-04-22T18:39:31.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:39:31.404 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.404 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.404 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.404 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.404 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.404 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:39:31.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.404 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:39:31.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:39:31.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:39:31.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:39:31.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:39:31.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:39:31.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:39:31.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:39:31.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:39:31.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:39:31.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:31.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:39:31.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:39:31.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:39:31.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:39:31.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:39:31.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:39:31.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:39:31.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:39:31.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:39:31.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:39:31.409 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.409 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.409 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.410 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:31.410 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:31.410 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.814776891000001, 2024-04-22T18:39:31.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:39:31.410 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.410 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.410 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:39:31.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.410 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:31.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:31.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:39:31.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:31.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:39:31.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:39:31.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:39:31.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:39:31.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:39:31.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:39:31.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:39:31.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:39:31.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:39:31.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:39:31.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:31.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:39:31.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:39:31.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:39:31.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:39:31.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:39:31.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:39:31.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:39:31.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:39:31.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:39:31.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:39:31.415 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.415 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.415 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.415 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:31.415 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:31.415 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.814303992999999, 2024-04-22T18:39:31.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:39:31.416 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.416 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.416 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:39:31.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.416 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:31.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:31.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:39:31.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:31.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:39:31.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:39:31.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:39:31.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:39:31.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:39:31.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:39:31.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:39:31.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:39:31.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:39:31.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:39:31.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:31.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:39:31.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:39:31.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:39:31.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:39:31.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:39:31.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:39:31.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:39:31.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:39:31.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:39:31.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:39:31.421 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.421 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.421 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.421 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:31.421 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:31.421 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.814284072, 2024-04-22T18:39:31.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:39:31.421 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.421 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:31.421 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.421 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.422 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.422 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:39:31.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.422 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:31.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:31.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:31.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:31.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:39:31.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:39:31.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:39:31.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:39:31.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:39:31.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:39:31.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:39:31.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:39:31.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:31.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:39:31.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:39:31.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:39:31.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:39:31.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:39:31.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:39:31.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:39:31.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:39:31.426 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.426 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.426 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.426 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:31.426 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:39:31.426 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.504960306999999, 2024-04-22T18:39:31.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:39:31.426 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.426 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.426 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:39:31.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:39:31.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:39:31.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:31.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:39:31.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:39:31.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:39:31.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:39:31.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:39:31.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:39:31.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:39:31.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:39:31.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:39:31.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:39:31.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:31.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:39:31.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:39:31.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:39:31.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:39:31.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:39:31.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:39:31.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:39:31.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:39:31.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:39:31.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:39:31.431 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.431 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.431 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.431 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:31.431 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:39:31.431 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.504752771, 2024-04-22T18:39:31.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:39:31.432 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.432 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.432 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:39:31.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:39:31.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:39:31.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:31.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:39:31.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:39:31.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:39:31.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:39:31.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:39:31.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:39:31.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:39:31.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:39:31.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:39:31.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:39:31.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:31.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:39:31.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:39:31.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:39:31.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:39:31.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:39:31.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:39:31.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:39:31.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:39:31.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:39:31.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:39:31.436 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.436 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.436 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.437 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:117 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:31.437 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844946+0000", 2024-04-22T18:39:31.437 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.504658256999999, 2024-04-22T18:39:31.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22636420400000001, 2024-04-22T18:39:31.437 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.437 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.437 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844946+0000", 2024-04-22T18:39:31.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844946+0000", 2024-04-22T18:39:31.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844945+0000", 2024-04-22T18:39:31.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:31.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845024+0000", 2024-04-22T18:39:31.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8532e-05 2024-04-22T18:39:31.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845031+0000", 2024-04-22T18:39:31.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1269999999999997e-06 2024-04-22T18:39:31.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845039+0000", 2024-04-22T18:39:31.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6809999999999998e-06 2024-04-22T18:39:31.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.946521+0000", 2024-04-22T18:39:31.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101482688 2024-04-22T18:39:31.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.946610+0000", 2024-04-22T18:39:31.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9023999999999993e-05 2024-04-22T18:39:31.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:31.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.946805+0000", 2024-04-22T18:39:31.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019483599999999999 2024-04-22T18:39:31.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.950818+0000", 2024-04-22T18:39:31.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0040133460000000001 2024-04-22T18:39:31.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071261+0000", 2024-04-22T18:39:31.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12044292500000001 2024-04-22T18:39:31.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071290+0000", 2024-04-22T18:39:31.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8322000000000001e-05 2024-04-22T18:39:31.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071310+0000", 2024-04-22T18:39:31.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0477000000000001e-05 2024-04-22T18:39:31.442 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.442 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.442 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.442 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:121 3.2 3:754415d0:::36414a779a90b1914c27a3701533e6b5d0236736bf30da29dc510853ad2cbd37:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:31.442 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.845046+0000", 2024-04-22T18:39:31.442 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.504558505, 2024-04-22T18:39:31.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22635160700000001, 2024-04-22T18:39:31.442 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.442 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.442 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845046+0000", 2024-04-22T18:39:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845046+0000", 2024-04-22T18:39:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845045+0000", 2024-04-22T18:39:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845081+0000", 2024-04-22T18:39:31.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.595e-05 2024-04-22T18:39:31.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845088+0000", 2024-04-22T18:39:31.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0929999999999997e-06 2024-04-22T18:39:31.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845093+0000", 2024-04-22T18:39:31.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3970000000000002e-06 2024-04-22T18:39:31.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947321+0000", 2024-04-22T18:39:31.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102227129 2024-04-22T18:39:31.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947400+0000", 2024-04-22T18:39:31.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9919000000000005e-05 2024-04-22T18:39:31.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:31.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947533+0000", 2024-04-22T18:39:31.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000132473 2024-04-22T18:39:31.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.951155+0000", 2024-04-22T18:39:31.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0036216730000000002 2024-04-22T18:39:31.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071338+0000", 2024-04-22T18:39:31.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.120183489 2024-04-22T18:39:31.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071380+0000", 2024-04-22T18:39:31.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1519e-05 2024-04-22T18:39:31.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071397+0000", 2024-04-22T18:39:31.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7740999999999998e-05 2024-04-22T18:39:31.447 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.447 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.447 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.447 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:126 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:31.447 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.957111+0000", 2024-04-22T18:39:31.447 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.392493115000001, 2024-04-22T18:39:31.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21447909900000001, 2024-04-22T18:39:31.447 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.448 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.448 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957111+0000", 2024-04-22T18:39:31.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957111+0000", 2024-04-22T18:39:31.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957108+0000", 2024-04-22T18:39:31.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:31.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957118+0000", 2024-04-22T18:39:31.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3850000000000008e-06 2024-04-22T18:39:31.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957133+0000", 2024-04-22T18:39:31.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5611999999999999e-05 2024-04-22T18:39:31.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957142+0000", 2024-04-22T18:39:31.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2879999999999993e-06 2024-04-22T18:39:31.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957209+0000", 2024-04-22T18:39:31.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7568000000000001e-05 2024-04-22T18:39:31.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957260+0000", 2024-04-22T18:39:31.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1091999999999998e-05 2024-04-22T18:39:31.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957362+0000", 2024-04-22T18:39:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101889 2024-04-22T18:39:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957985+0000", 2024-04-22T18:39:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00062231899999999998 2024-04-22T18:39:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.171534+0000", 2024-04-22T18:39:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21354967 2024-04-22T18:39:31.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.171568+0000", 2024-04-22T18:39:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3453e-05 2024-04-22T18:39:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.171590+0000", 2024-04-22T18:39:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2555e-05 2024-04-22T18:39:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:127 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.957151+0000", 2024-04-22T18:39:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.392452971000001, 2024-04-22T18:39:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21456244299999999, 2024-04-22T18:39:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957151+0000", 2024-04-22T18:39:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957151+0000", 2024-04-22T18:39:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957150+0000", 2024-04-22T18:39:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957155+0000", 2024-04-22T18:39:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4159999999999997e-06 2024-04-22T18:39:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957161+0000", 2024-04-22T18:39:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2219999999999999e-06 2024-04-22T18:39:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957167+0000", 2024-04-22T18:39:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5029999999999997e-06 2024-04-22T18:39:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957528+0000", 2024-04-22T18:39:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000361861 2024-04-22T18:39:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957570+0000", 2024-04-22T18:39:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1709000000000003e-05 2024-04-22T18:39:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.957669+0000", 2024-04-22T18:39:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8758999999999994e-05 2024-04-22T18:39:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.958208+0000", 2024-04-22T18:39:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00053896900000000004 2024-04-22T18:39:31.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.171634+0000", 2024-04-22T18:39:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21342636800000001 2024-04-22T18:39:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.171683+0000", 2024-04-22T18:39:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8263999999999997e-05 2024-04-22T18:39:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.171714+0000", 2024-04-22T18:39:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1202999999999999e-05 2024-04-22T18:39:31.457 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.458 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.458 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:31.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:31.460 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:31.460+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:31.559 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:31.560+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:31.578 INFO:tasks.rados.rados.0.smithi089.stdout:933: set_chunk oid 1 offset: 922112 length: 856064 target oid 34336650543f8f8bff225091f9c8c14678dc47baabcb4d8371890fc935dbc3e4 offset: 0 snap -1 2024-04-22T18:39:31.578 INFO:tasks.rados.rados.0.smithi089.stdout:934: write oid 16 current snap is 0 2024-04-22T18:39:31.578 INFO:tasks.rados.rados.0.smithi089.stdout:918: finishing write tid 4 to smithi08937929-22 2024-04-22T18:39:31.579 INFO:tasks.rados.rados.0.smithi089.stdout:918: oid 22 updating version 0 to 60 2024-04-22T18:39:31.579 INFO:tasks.rados.rados.0.smithi089.stdout:918: oid 22 updating version 60 to 62 2024-04-22T18:39:31.579 INFO:tasks.rados.rados.0.smithi089.stdout:918: oid 22 version 62 is already newer than 61 2024-04-22T18:39:31.579 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 62 (ObjNum 135 snap 0 seq_num 135) dirty exists 2024-04-22T18:39:31.579 INFO:tasks.rados.rados.0.smithi089.stdout:918: left oid 22 (ObjNum 135 snap 0 seq_num 135) 2024-04-22T18:39:31.579 INFO:tasks.rados.rados.0.smithi089.stdout:934: seq_num 139 ranges {736173=553965,1867350=468823} 2024-04-22T18:39:31.585 INFO:tasks.rados.rados.0.smithi089.stdout:934: writing smithi08937929-16 from 736173 to 1290138 tid 1 2024-04-22T18:39:31.591 INFO:tasks.rados.rados.0.smithi089.stdout:934: writing smithi08937929-16 from 1867350 to 2336173 tid 2 2024-04-22T18:39:31.591 INFO:tasks.rados.rados.0.smithi089.stdout:918: done (15 left) 2024-04-22T18:39:31.591 INFO:tasks.rados.rados.0.smithi089.stdout:935: read oid 13 snap -1 2024-04-22T18:39:31.591 INFO:tasks.rados.rados.0.smithi089.stdout:935: expect deleted 2024-04-22T18:39:31.591 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:31.591 INFO:tasks.rados.rados.0.smithi089.stdout:927: finishing set_chunk to oid 3 2024-04-22T18:39:31.591 INFO:tasks.rados.rados.0.smithi089.stdout:927: got expected ENOENT (src dne) 2024-04-22T18:39:31.591 INFO:tasks.rados.rados.0.smithi089.stdout:925: expect (ObjNum 105 snap 0 seq_num 105) 2024-04-22T18:39:31.601 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:31.602 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:31.602 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:31.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:31.642 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:39:31.695 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:31.697+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:31.752 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:31.752 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:31.753 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:31.762 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:31.763+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:31.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:31.837 INFO:tasks.rados.rados.0.smithi089.stdout:924: finishing write tid 1 to smithi08937929-11 2024-04-22T18:39:31.837 INFO:tasks.rados.rados.0.smithi089.stdout:925: done (15 left) 2024-04-22T18:39:31.837 INFO:tasks.rados.rados.0.smithi089.stdout:927: done (14 left) 2024-04-22T18:39:31.837 INFO:tasks.rados.rados.0.smithi089.stdout:936: write oid 26 current snap is 0 2024-04-22T18:39:31.837 INFO:tasks.rados.rados.0.smithi089.stdout:926: expect (ObjNum 80 snap 0 seq_num 80) 2024-04-22T18:39:31.862 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:31.864+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:31.866 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:31.867 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:31.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:31.867 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:31.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.867 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:21 4.a 4:5aa97f94:::smithi08937929-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.867 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:31.867 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.713077937000001, 2024-04-22T18:39:31.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21044934000000001, 2024-04-22T18:39:31.867 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.867 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.867 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.867 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.867 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.868 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 21 2024-04-22T18:39:31.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.868 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:31.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:31.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153183+0000", 2024-04-22T18:39:31.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:31.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154022+0000", 2024-04-22T18:39:31.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00083893300000000004 2024-04-22T18:39:31.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154032+0000", 2024-04-22T18:39:31.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0433e-05 2024-04-22T18:39:31.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154051+0000", 2024-04-22T18:39:31.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8227e-05 2024-04-22T18:39:31.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154103+0000", 2024-04-22T18:39:31.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2491999999999998e-05 2024-04-22T18:39:31.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:31.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154111+0000", 2024-04-22T18:39:31.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3470000000000001e-06 2024-04-22T18:39:31.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257848+0000", 2024-04-22T18:39:31.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103737703 2024-04-22T18:39:31.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257981+0000", 2024-04-22T18:39:31.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013290399999999999 2024-04-22T18:39:31.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:31.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.258138+0000", 2024-04-22T18:39:31.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015701199999999999 2024-04-22T18:39:31.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.261944+0000", 2024-04-22T18:39:31.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038054650000000001 2024-04-22T18:39:31.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363523+0000", 2024-04-22T18:39:31.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101579239 2024-04-22T18:39:31.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363577+0000", 2024-04-22T18:39:31.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4095000000000003e-05 2024-04-22T18:39:31.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363635+0000", 2024-04-22T18:39:31.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8362999999999997e-05 2024-04-22T18:39:31.873 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.873 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.873 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.873 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:22 4.a 4:5aa97f94:::smithi08937929-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.873 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:31.873 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.709003961000001, 2024-04-22T18:39:31.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30948398900000001, 2024-04-22T18:39:31.874 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.874 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.874 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.874 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.874 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.874 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 22 2024-04-22T18:39:31.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.874 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.874 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.874 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:31.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157259+0000", 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157318+0000", 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8906000000000001e-05 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157327+0000", 2024-04-22T18:39:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:39:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157332+0000", 2024-04-22T18:39:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4689999999999996e-06 2024-04-22T18:39:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257479+0000", 2024-04-22T18:39:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014700999999999 2024-04-22T18:39:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257492+0000", 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2574e-05 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259582+0000", 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020904019999999999 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259668+0000", 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5826000000000001e-05 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:31.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259772+0000", 2024-04-22T18:39:31.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104048 2024-04-22T18:39:31.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363453+0000", 2024-04-22T18:39:31.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103680183 2024-04-22T18:39:31.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466692+0000", 2024-04-22T18:39:31.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10323948199999999 2024-04-22T18:39:31.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466720+0000", 2024-04-22T18:39:31.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7886e-05 2024-04-22T18:39:31.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466744+0000", 2024-04-22T18:39:31.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4049e-05 2024-04-22T18:39:31.879 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.879 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.879 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.879 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:23 4.a 4:5aa97f94:::smithi08937929-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.879 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:31.880 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.708926274, 2024-04-22T18:39:31.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39003820300000003, 2024-04-22T18:39:31.880 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.880 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.880 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.880 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.880 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.880 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 23 2024-04-22T18:39:31.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.880 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:31.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:31.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157337+0000", 2024-04-22T18:39:31.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:31.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.881 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157340+0000", 2024-04-22T18:39:31.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8609999999999998e-06 2024-04-22T18:39:31.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157343+0000", 2024-04-22T18:39:31.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9859999999999999e-06 2024-04-22T18:39:31.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157346+0000", 2024-04-22T18:39:31.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8119999999999999e-06 2024-04-22T18:39:31.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257563+0000", 2024-04-22T18:39:31.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100217497 2024-04-22T18:39:31.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:31.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257574+0000", 2024-04-22T18:39:31.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0777e-05 2024-04-22T18:39:31.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260112+0000", 2024-04-22T18:39:31.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002537681 2024-04-22T18:39:31.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260191+0000", 2024-04-22T18:39:31.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9320999999999995e-05 2024-04-22T18:39:31.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:31.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260333+0000", 2024-04-22T18:39:31.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014177699999999999 2024-04-22T18:39:31.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363792+0000", 2024-04-22T18:39:31.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103458862 2024-04-22T18:39:31.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547303+0000", 2024-04-22T18:39:31.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18351162100000001 2024-04-22T18:39:31.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547337+0000", 2024-04-22T18:39:31.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3640999999999999e-05 2024-04-22T18:39:31.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547376+0000", 2024-04-22T18:39:31.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8915000000000003e-05 2024-04-22T18:39:31.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.885 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.886 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:24 4.a 4:5aa97f94:::smithi08937929-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.886 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:31.886 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.708899842999998, 2024-04-22T18:39:31.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39017823499999998, 2024-04-22T18:39:31.886 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.886 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:31.886 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.886 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.886 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.886 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 24 2024-04-22T18:39:31.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.886 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:31.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:31.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:31.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:31.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157366+0000", 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3120000000000001e-06 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157370+0000", 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1300000000000003e-06 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157373+0000", 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260553+0000", 2024-04-22T18:39:31.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103179996 2024-04-22T18:39:31.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:31.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260625+0000", 2024-04-22T18:39:31.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2236000000000004e-05 2024-04-22T18:39:31.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547414+0000", 2024-04-22T18:39:31.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.286789181 2024-04-22T18:39:31.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547471+0000", 2024-04-22T18:39:31.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6192e-05 2024-04-22T18:39:31.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547542+0000", 2024-04-22T18:39:31.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1622e-05 2024-04-22T18:39:31.890 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.890 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.890 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.890 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:49 4.a 4:5d2aee44:::smithi08937929-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.890 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:31.891 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.632101185000003, 2024-04-22T18:39:31.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31344442500000003, 2024-04-22T18:39:31.891 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.891 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.891 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 49 2024-04-22T18:39:31.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.891 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:31.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:31.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234158+0000", 2024-04-22T18:39:31.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999957 2024-04-22T18:39:31.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234901+0000", 2024-04-22T18:39:31.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074250800000000001 2024-04-22T18:39:31.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234914+0000", 2024-04-22T18:39:31.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3244999999999999e-05 2024-04-22T18:39:31.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234922+0000", 2024-04-22T18:39:31.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4590000000000001e-06 2024-04-22T18:39:31.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260665+0000", 2024-04-22T18:39:31.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025743423000000001 2024-04-22T18:39:31.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260779+0000", 2024-04-22T18:39:31.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114063 2024-04-22T18:39:31.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:31.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260913+0000", 2024-04-22T18:39:31.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013355499999999999 2024-04-22T18:39:31.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363798+0000", 2024-04-22T18:39:31.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102885003 2024-04-22T18:39:31.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547567+0000", 2024-04-22T18:39:31.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.183768971 2024-04-22T18:39:31.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547589+0000", 2024-04-22T18:39:31.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2507999999999999e-05 2024-04-22T18:39:31.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547607+0000", 2024-04-22T18:39:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8009999999999999e-05 2024-04-22T18:39:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:50 4.a 4:5d2aee44:::smithi08937929-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.626655006, 2024-04-22T18:39:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40856295399999998, 2024-04-22T18:39:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.896 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.897 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.897 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 50 2024-04-22T18:39:31.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.897 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:31.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:31.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239607+0000", 2024-04-22T18:39:31.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:31.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240309+0000", 2024-04-22T18:39:31.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070201599999999997 2024-04-22T18:39:31.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240320+0000", 2024-04-22T18:39:31.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1024e-05 2024-04-22T18:39:31.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240327+0000", 2024-04-22T18:39:31.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0049999999999998e-06 2024-04-22T18:39:31.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362065+0000", 2024-04-22T18:39:31.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.121737812 2024-04-22T18:39:31.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362121+0000", 2024-04-22T18:39:31.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5868000000000001e-05 2024-04-22T18:39:31.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:31.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362210+0000", 2024-04-22T18:39:31.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8533000000000001e-05 2024-04-22T18:39:31.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365322+0000", 2024-04-22T18:39:31.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031121220000000002 2024-04-22T18:39:31.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648054+0000", 2024-04-22T18:39:31.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.282732549 2024-04-22T18:39:31.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648105+0000", 2024-04-22T18:39:31.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1276999999999997e-05 2024-04-22T18:39:31.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648172+0000", 2024-04-22T18:39:31.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6398999999999999e-05 2024-04-22T18:39:31.901 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.901 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.901 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.901 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:51 4.a 4:5d2aee44:::smithi08937929-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.902 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:31.902 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.625932472000002, 2024-04-22T18:39:31.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40793057199999999, 2024-04-22T18:39:31.902 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.902 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.902 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.902 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.902 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.902 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 51 2024-04-22T18:39:31.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.902 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240336+0000", 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240339+0000", 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9009999999999998e-06 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240342+0000", 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363080+0000", 2024-04-22T18:39:31.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122738075 2024-04-22T18:39:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363164+0000", 2024-04-22T18:39:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3722999999999998e-05 2024-04-22T18:39:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363287+0000", 2024-04-22T18:39:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012301099999999999 2024-04-22T18:39:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466753+0000", 2024-04-22T18:39:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103466217 2024-04-22T18:39:31.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648205+0000", 2024-04-22T18:39:31.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.181451788 2024-04-22T18:39:31.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648231+0000", 2024-04-22T18:39:31.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6597999999999998e-05 2024-04-22T18:39:31.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648262+0000", 2024-04-22T18:39:31.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0871000000000003e-05 2024-04-22T18:39:31.907 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.907 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.907 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.907 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:52 4.a 4:5d2aee44:::smithi08937929-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.907 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:31.907 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.625920012999998, 2024-04-22T18:39:31.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40812731200000002, 2024-04-22T18:39:31.907 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.907 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:31.907 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.907 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.908 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.908 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 52 2024-04-22T18:39:31.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.908 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:31.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:31.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:31.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:31.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240345+0000", 2024-04-22T18:39:31.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4300000000000001e-06 2024-04-22T18:39:31.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240349+0000", 2024-04-22T18:39:31.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2570000000000001e-06 2024-04-22T18:39:31.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240358+0000", 2024-04-22T18:39:31.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-22T18:39:31.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363697+0000", 2024-04-22T18:39:31.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12333965299999999 2024-04-22T18:39:31.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:31.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363777+0000", 2024-04-22T18:39:31.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9883999999999996e-05 2024-04-22T18:39:31.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648296+0000", 2024-04-22T18:39:31.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451861299999998 2024-04-22T18:39:31.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648379+0000", 2024-04-22T18:39:31.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3302999999999998e-05 2024-04-22T18:39:31.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648471+0000", 2024-04-22T18:39:31.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2029999999999998e-05 2024-04-22T18:39:31.912 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.912 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.912 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.912 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.912 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:31.912 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.606616678000002, 2024-04-22T18:39:31.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:39:31.912 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.912 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.912 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.912 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.912 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:39:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:39:31.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:31.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:39:31.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:39:31.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:39:31.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:39:31.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:39:31.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:39:31.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:39:31.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:39:31.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:39:31.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:39:31.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:31.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:39:31.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:39:31.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:39:31.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:39:31.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:39:31.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:39:31.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:39:31.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:39:31.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:39:31.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:39:31.917 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.917 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.917 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.917 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.917 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:31.918 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.601834248000003, 2024-04-22T18:39:31.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:39:31.918 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.918 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.918 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.918 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.918 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.918 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:39:31.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.918 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:31.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:31.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:39:31.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:31.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:39:31.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:39:31.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:39:31.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:39:31.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:39:31.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:39:31.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:39:31.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:39:31.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:39:31.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:39:31.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:31.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:39:31.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:39:31.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:39:31.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:39:31.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:39:31.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:39:31.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:39:31.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:39:31.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:39:31.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:39:31.923 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.923 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.923 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.923 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.923 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:31.923 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.601740045, 2024-04-22T18:39:31.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:39:31.923 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.923 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.923 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.923 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:39:31.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:31.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:39:31.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:39:31.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:39:31.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:39:31.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:39:31.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:39:31.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:39:31.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:39:31.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:39:31.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:39:31.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:31.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:39:31.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:39:31.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:39:31.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:39:31.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:39:31.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:39:31.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:39:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:39:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:39:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:39:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.928 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.601719156999998, 2024-04-22T18:39:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:39:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:39:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:31.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:31.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:39:31.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:31.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:39:31.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:39:31.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:39:31.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:39:31.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:39:31.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:39:31.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:39:31.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:39:31.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:31.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:39:31.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:39:31.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:39:31.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:39:31.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:39:31.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:39:31.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:39:31.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:39:31.933 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.933 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.933 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.933 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.933 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:31.933 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.818711493999999, 2024-04-22T18:39:31.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:39:31.934 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.934 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.934 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.934 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.934 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.934 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:39:31.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.934 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:31.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:31.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:39:31.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:31.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:39:31.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:39:31.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:39:31.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:39:31.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:39:31.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:39:31.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:39:31.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:39:31.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:39:31.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:39:31.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:31.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:39:31.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:39:31.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:39:31.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:39:31.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:39:31.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:39:31.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:39:31.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:39:31.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:39:31.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:39:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.810880087999998, 2024-04-22T18:39:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:39:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.939 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:39:31.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.940 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:31.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:31.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:39:31.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:31.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:39:31.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:39:31.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:39:31.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:39:31.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:39:31.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:39:31.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:39:31.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:39:31.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:39:31.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:39:31.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:31.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:39:31.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:39:31.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:39:31.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:39:31.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:39:31.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:39:31.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:39:31.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:39:31.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:39:31.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:39:31.944 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.944 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.944 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.944 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:186 4.a 4:537c89f8:::smithi08937929-47:head [write 1992054~651681] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.944 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.463374553000001, 2024-04-22T18:39:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20325226800000001, 2024-04-22T18:39:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 186 2024-04-22T18:39:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:31.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402888+0000", 2024-04-22T18:39:31.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:31.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403454+0000", 2024-04-22T18:39:31.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056661000000000005 2024-04-22T18:39:31.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403465+0000", 2024-04-22T18:39:31.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0727000000000001e-05 2024-04-22T18:39:31.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403473+0000", 2024-04-22T18:39:31.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7039999999999994e-06 2024-04-22T18:39:31.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404592+0000", 2024-04-22T18:39:31.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001118827 2024-04-22T18:39:31.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404674+0000", 2024-04-22T18:39:31.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2793999999999994e-05 2024-04-22T18:39:31.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:31.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404779+0000", 2024-04-22T18:39:31.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104465 2024-04-22T18:39:31.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.406080+0000", 2024-04-22T18:39:31.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013006789999999999 2024-04-22T18:39:31.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606037+0000", 2024-04-22T18:39:31.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19995755600000001 2024-04-22T18:39:31.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606098+0000", 2024-04-22T18:39:31.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1075999999999996e-05 2024-04-22T18:39:31.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606142+0000", 2024-04-22T18:39:31.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3411999999999997e-05 2024-04-22T18:39:31.950 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.950 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.950 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.950 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:187 4.a 4:537c89f8:::smithi08937929-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:31.950 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:31.950 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.462784413000001, 2024-04-22T18:39:31.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203643623, 2024-04-22T18:39:31.950 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.950 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.950 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.950 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.950 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.951 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 187 2024-04-22T18:39:31.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.951 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:31.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:31.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:31.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:31.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403501+0000", 2024-04-22T18:39:31.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2146000000000001e-05 2024-04-22T18:39:31.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403506+0000", 2024-04-22T18:39:31.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3040000000000004e-06 2024-04-22T18:39:31.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403511+0000", 2024-04-22T18:39:31.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9599999999999999e-06 2024-04-22T18:39:31.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405587+0000", 2024-04-22T18:39:31.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002076296 2024-04-22T18:39:31.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405664+0000", 2024-04-22T18:39:31.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6316000000000006e-05 2024-04-22T18:39:31.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:31.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405785+0000", 2024-04-22T18:39:31.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121276 2024-04-22T18:39:31.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.408030+0000", 2024-04-22T18:39:31.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022452169999999999 2024-04-22T18:39:31.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607059+0000", 2024-04-22T18:39:31.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199029135 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607094+0000", 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4686e-05 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607123+0000", 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9113000000000001e-05 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.955 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:811 4.5 4:a4a14dff:::smithi08937929-40:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-22T18:39:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.745441956000001, 2024-04-22T18:39:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242306563, 2024-04-22T18:39:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 811 2024-04-22T18:39:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:31.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:31.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120821+0000", 2024-04-22T18:39:31.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:31.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120825+0000", 2024-04-22T18:39:31.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.506e-06 2024-04-22T18:39:31.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120829+0000", 2024-04-22T18:39:31.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0849999999999999e-06 2024-04-22T18:39:31.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120833+0000", 2024-04-22T18:39:31.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9609999999999999e-06 2024-04-22T18:39:31.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221263+0000", 2024-04-22T18:39:31.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10043045 2024-04-22T18:39:31.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221337+0000", 2024-04-22T18:39:31.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4010000000000005e-05 2024-04-22T18:39:31.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:31.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221448+0000", 2024-04-22T18:39:31.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011065699999999999 2024-04-22T18:39:31.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:31.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.222541+0000", 2024-04-22T18:39:31.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010932610000000001 2024-04-22T18:39:31.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:31.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.362952+0000", 2024-04-22T18:39:31.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.140410535 2024-04-22T18:39:31.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:31.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.363102+0000", 2024-04-22T18:39:31.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015041499999999999 2024-04-22T18:39:31.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.363129+0000", 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6574999999999999e-05 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:933 4.a 4:5d2aee44:::smithi08937929-13:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.221518318000001, 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202744591, 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.961 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.962 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 933 2024-04-22T18:39:31.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.962 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:31.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:31.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644743+0000", 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644754+0000", 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0807e-05 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644768+0000", 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3844e-05 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.744860+0000", 2024-04-22T18:39:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100092835 2024-04-22T18:39:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.745084+0000", 2024-04-22T18:39:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022401699999999999 2024-04-22T18:39:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.745090+0000", 2024-04-22T18:39:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2970000000000003e-06 2024-04-22T18:39:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.847381+0000", 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102291544 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.847490+0000", 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010891399999999999 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1286 4.5 4:a4a14dff:::smithi08937929-40:head [sparse-read 0~1336255,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.028998811999999, 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202431208, 2024-04-22T18:39:31.965 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.966 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:31.966 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.966 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.966 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.966 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1286 2024-04-22T18:39:31.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.966 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:31.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:31.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837261+0000", 2024-04-22T18:39:31.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999962 2024-04-22T18:39:31.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837278+0000", 2024-04-22T18:39:31.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6359999999999999e-05 2024-04-22T18:39:31.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837307+0000", 2024-04-22T18:39:31.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9828999999999999e-05 2024-04-22T18:39:31.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937448+0000", 2024-04-22T18:39:31.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014044 2024-04-22T18:39:31.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937540+0000", 2024-04-22T18:39:31.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2539000000000003e-05 2024-04-22T18:39:31.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937548+0000", 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4089999999999997e-06 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.039347+0000", 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10179880700000001 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.039505+0000", 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000158758 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.039696+0000", 2024-04-22T18:39:31.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019096599999999999 2024-04-22T18:39:31.970 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.970 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.970 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.970 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1288 4.5 4:a4a14dff:::smithi08937929-40:head [sparse-read 0~1336255] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:31.970 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.937524+0000", 2024-04-22T18:39:31.970 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.928739989, 2024-04-22T18:39:31.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20252358200000001, 2024-04-22T18:39:31.970 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:31.970 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:31.970 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:31.971 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:31.971 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:31.971 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1288 2024-04-22T18:39:31.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.971 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:31.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:31.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937524+0000", 2024-04-22T18:39:31.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:31.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937524+0000", 2024-04-22T18:39:31.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:31.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:31.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937523+0000", 2024-04-22T18:39:31.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:31.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:31.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937529+0000", 2024-04-22T18:39:31.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5369999999999997e-06 2024-04-22T18:39:31.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:31.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937532+0000", 2024-04-22T18:39:31.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.455e-06 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.037667+0000", 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100134238 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.139784+0000", 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102117472 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.139848+0000", 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4257000000000005e-05 2024-04-22T18:39:31.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:31.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:31.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:31.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.140047+0000", 2024-04-22T18:39:31.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000199099 2024-04-22T18:39:31.974 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.974 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.974 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.974 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:31.974 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:31.974 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:31.974 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:39:31.974 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:39:31.975 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:39:31.975 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2024-04-22T18:39:31.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:31.977 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:31.977+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:32.030 INFO:tasks.rados.rados.0.smithi089.stdout:922: expect (ObjNum 17 snap 0 seq_num 17) 2024-04-22T18:39:32.076 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:32.077+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:32.078 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:32.078 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:32.079 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:32.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:32.106 INFO:tasks.rados.rados.0.smithi089.stdout:924: finishing write tid 2 to smithi08937929-11 2024-04-22T18:39:32.106 INFO:tasks.rados.rados.0.smithi089.stdout:924: finishing write tid 3 to smithi08937929-11 2024-04-22T18:39:32.106 INFO:tasks.rados.rados.0.smithi089.stdout:924: finishing write tid 4 to smithi08937929-11 2024-04-22T18:39:32.106 INFO:tasks.rados.rados.0.smithi089.stdout:924: oid 11 updating version 0 to 23 2024-04-22T18:39:32.106 INFO:tasks.rados.rados.0.smithi089.stdout:924: oid 11 version 23 is already newer than 22 2024-04-22T18:39:32.106 INFO:tasks.rados.rados.0.smithi089.stdout:924: oid 11 updating version 23 to 24 2024-04-22T18:39:32.106 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 24 (ObjNum 137 snap 0 seq_num 137) dirty exists 2024-04-22T18:39:32.106 INFO:tasks.rados.rados.0.smithi089.stdout:924: left oid 11 (ObjNum 137 snap 0 seq_num 137) 2024-04-22T18:39:32.106 INFO:tasks.rados.rados.0.smithi089.stdout:936: seq_num 140 ranges {752980=629214,1809479=543501} 2024-04-22T18:39:32.113 INFO:tasks.rados.rados.0.smithi089.stdout:936: writing smithi08937929-26 from 752980 to 1382194 tid 1 2024-04-22T18:39:32.118 INFO:tasks.rados.rados.0.smithi089.stdout:936: writing smithi08937929-26 from 1809479 to 2352980 tid 2 2024-04-22T18:39:32.118 INFO:tasks.rados.rados.0.smithi089.stdout:922: done (14 left) 2024-04-22T18:39:32.119 INFO:tasks.rados.rados.0.smithi089.stdout:924: done (13 left) 2024-04-22T18:39:32.119 INFO:tasks.rados.rados.0.smithi089.stdout:926: done (12 left) 2024-04-22T18:39:32.119 INFO:tasks.rados.rados.0.smithi089.stdout:937: set_chunk oid 21 target oid 2024-04-22T18:39:32.127 INFO:tasks.rados.rados.0.smithi089.stdout:937: set_chunk oid 21 offset: 2119168 length: 791040 target oid 7d2cd25158d554c6c8c75925eb6cd081f4f083cf85fcf7fc3ff51c22b5030976 offset: 0 snap -1 2024-04-22T18:39:32.127 INFO:tasks.rados.rados.0.smithi089.stdout:938: write (excl) oid 12 current snap is 0 2024-04-22T18:39:32.128 INFO:tasks.rados.rados.0.smithi089.stdout:929: expect (ObjNum 101 snap 0 seq_num 101) 2024-04-22T18:39:32.177 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:32.179+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:32.229 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:32.229 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:32.229 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:32.229 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:32.229 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:32.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:32.245 INFO:tasks.rados.rados.0.smithi089.stdout:920: finishing set_chunk to oid 37 2024-04-22T18:39:32.245 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 65 (ObjNum 131 snap 0 seq_num 131) dirty exists 2024-04-22T18:39:32.245 INFO:tasks.rados.rados.0.smithi089.stdout:930: expect (ObjNum 63 snap 0 seq_num 63) 2024-04-22T18:39:32.269 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-22T18:39:32.271+0000 7f9b4bbc8640 -1 mon.a@0(leader).osd e48 definitely_dead 0 2024-04-22T18:39:32.282 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:32.283+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:32.342 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:32.343 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:32.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:32.343 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:32.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.343 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:32.343 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:32.343 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 52.600764177999999, 2024-04-22T18:39:32.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30520528099999999, 2024-04-22T18:39:32.343 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.343 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.343 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.343 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.344 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.344 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 35 2024-04-22T18:39:32.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.344 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:32.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:32.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741852+0000", 2024-04-22T18:39:32.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:32.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741927+0000", 2024-04-22T18:39:32.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5123000000000006e-05 2024-04-22T18:39:32.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741953+0000", 2024-04-22T18:39:32.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5378999999999999e-05 2024-04-22T18:39:32.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741977+0000", 2024-04-22T18:39:32.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4431999999999999e-05 2024-04-22T18:39:32.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844243+0000", 2024-04-22T18:39:32.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102265525 2024-04-22T18:39:32.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844330+0000", 2024-04-22T18:39:32.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7417999999999995e-05 2024-04-22T18:39:32.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844470+0000", 2024-04-22T18:39:32.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000140197 2024-04-22T18:39:32.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944767+0000", 2024-04-22T18:39:32.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100296409 2024-04-22T18:39:32.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.046968+0000", 2024-04-22T18:39:32.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10220148499999999 2024-04-22T18:39:32.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047018+0000", 2024-04-22T18:39:32.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9206000000000003e-05 2024-04-22T18:39:32.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047059+0000", 2024-04-22T18:39:32.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1496999999999999e-05 2024-04-22T18:39:32.348 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.348 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.348 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.349 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:32.349 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:32.349 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 52.600633653999999, 2024-04-22T18:39:32.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:39:32.349 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.349 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.349 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.349 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.349 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.349 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:39:32.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.349 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:32.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:32.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:39:32.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:32.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:39:32.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:39:32.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:39:32.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:39:32.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:39:32.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:39:32.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:39:32.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:39:32.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:39:32.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:39:32.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:39:32.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:39:32.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:39:32.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:39:32.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:39:32.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:39:32.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:39:32.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:39:32.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:39:32.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:39:32.354 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.354 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.354 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.354 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:32.354 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:32.355 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 52.600593963000001, 2024-04-22T18:39:32.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:39:32.355 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.355 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.355 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.355 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.355 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.355 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:39:32.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.355 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:32.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:32.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:39:32.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:32.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:39:32.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:39:32.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:39:32.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:39:32.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:39:32.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:39:32.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:39:32.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:39:32.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:39:32.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:39:32.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:39:32.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:39:32.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:39:32.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:39:32.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:39:32.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:39:32.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:39:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:39:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:39:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:39:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.356412915999996, 2024-04-22T18:39:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402419109, 2024-04-22T18:39:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 68 2024-04-22T18:39:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986204+0000", 2024-04-22T18:39:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986208+0000", 2024-04-22T18:39:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8149999999999999e-06 2024-04-22T18:39:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986214+0000", 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.604e-06 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986218+0000", 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6689999999999999e-06 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986833+0000", 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000615648 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986891+0000", 2024-04-22T18:39:32.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7915999999999999e-05 2024-04-22T18:39:32.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986981+0000", 2024-04-22T18:39:32.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759999999999994e-05 2024-04-22T18:39:32.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.987584+0000", 2024-04-22T18:39:32.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060270199999999999 2024-04-22T18:39:32.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388527+0000", 2024-04-22T18:39:32.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40094366999999997 2024-04-22T18:39:32.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388583+0000", 2024-04-22T18:39:32.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5383000000000002e-05 2024-04-22T18:39:32.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388624+0000", 2024-04-22T18:39:32.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501999999999997e-05 2024-04-22T18:39:32.365 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.365 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.366 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.366 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:32.366 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:32.366 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.356397127000001, 2024-04-22T18:39:32.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:39:32.366 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.366 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.366 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.366 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.366 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.366 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:39:32.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.367 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:32.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:32.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:39:32.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:32.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:39:32.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:39:32.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:39:32.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:39:32.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:39:32.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:39:32.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:39:32.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:39:32.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:39:32.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:39:32.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:39:32.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:39:32.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:39:32.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:39:32.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:39:32.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:39:32.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:39:32.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:39:32.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:39:32.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:39:32.371 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.371 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.371 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.371 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:32.371 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:32.372 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 49.240004118000002, 2024-04-22T18:39:32.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305563733, 2024-04-22T18:39:32.372 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.372 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.372 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.372 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.372 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.372 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 87 2024-04-22T18:39:32.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.372 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:32.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:32.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102613+0000", 2024-04-22T18:39:32.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:32.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102668+0000", 2024-04-22T18:39:32.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4669000000000001e-05 2024-04-22T18:39:32.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102680+0000", 2024-04-22T18:39:32.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2026e-05 2024-04-22T18:39:32.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102687+0000", 2024-04-22T18:39:32.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8339999999999999e-06 2024-04-22T18:39:32.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405644+0000", 2024-04-22T18:39:32.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30295623199999999 2024-04-22T18:39:32.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405743+0000", 2024-04-22T18:39:32.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.98e-05 2024-04-22T18:39:32.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405868+0000", 2024-04-22T18:39:32.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012457899999999999 2024-04-22T18:39:32.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407047+0000", 2024-04-22T18:39:32.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011791449999999999 2024-04-22T18:39:32.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408088+0000", 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010411680000000001 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408135+0000", 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6255999999999999e-05 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408178+0000", 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2877000000000003e-05 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.376 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.377 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.377 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:32.377 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:32.377 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 49.239924189, 2024-04-22T18:39:32.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30560089000000001, 2024-04-22T18:39:32.377 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.377 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.377 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.377 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.377 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.377 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 88 2024-04-22T18:39:32.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.377 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:32.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:32.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102693+0000", 2024-04-22T18:39:32.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:32.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102698+0000", 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9699999999999998e-06 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102708+0000", 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.006e-05 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102715+0000", 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6599999999999995e-06 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406637+0000", 2024-04-22T18:39:32.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392130099999998 2024-04-22T18:39:32.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406696+0000", 2024-04-22T18:39:32.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9066e-05 2024-04-22T18:39:32.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406784+0000", 2024-04-22T18:39:32.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8194000000000003e-05 2024-04-22T18:39:32.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407447+0000", 2024-04-22T18:39:32.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066295700000000004 2024-04-22T18:39:32.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408237+0000", 2024-04-22T18:39:32.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079032600000000005 2024-04-22T18:39:32.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408270+0000", 2024-04-22T18:39:32.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3157000000000001e-05 2024-04-22T18:39:32.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408295+0000", 2024-04-22T18:39:32.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4151999999999998e-05 2024-04-22T18:39:32.382 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.382 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.382 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.382 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:32.382 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:32.382 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.488692065000002, 2024-04-22T18:39:32.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30533819600000001, 2024-04-22T18:39:32.383 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.383 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.383 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.383 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.383 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.383 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 106 2024-04-22T18:39:32.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.383 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:32.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:32.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853925+0000", 2024-04-22T18:39:32.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:32.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853958+0000", 2024-04-22T18:39:32.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2536000000000002e-05 2024-04-22T18:39:32.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853964+0000", 2024-04-22T18:39:32.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6239999999999999e-06 2024-04-22T18:39:32.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853967+0000", 2024-04-22T18:39:32.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9700000000000001e-06 2024-04-22T18:39:32.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855144+0000", 2024-04-22T18:39:32.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001176774 2024-04-22T18:39:32.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855225+0000", 2024-04-22T18:39:32.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0697000000000003e-05 2024-04-22T18:39:32.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855353+0000", 2024-04-22T18:39:32.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012788 2024-04-22T18:39:32.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058003+0000", 2024-04-22T18:39:32.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202649939 2024-04-22T18:39:32.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159184+0000", 2024-04-22T18:39:32.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101181058 2024-04-22T18:39:32.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159238+0000", 2024-04-22T18:39:32.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3967e-05 2024-04-22T18:39:32.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159264+0000", 2024-04-22T18:39:32.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6237999999999998e-05 2024-04-22T18:39:32.387 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.388 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.388 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:32.388 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:32.388 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.488645214999998, 2024-04-22T18:39:32.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30544294500000002, 2024-04-22T18:39:32.388 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.388 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.388 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.388 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.388 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.388 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 107 2024-04-22T18:39:32.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.389 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:32.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:32.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853972+0000", 2024-04-22T18:39:32.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:32.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853975+0000", 2024-04-22T18:39:32.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0939999999999999e-06 2024-04-22T18:39:32.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853978+0000", 2024-04-22T18:39:32.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:39:32.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853981+0000", 2024-04-22T18:39:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.858e-06 2024-04-22T18:39:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855831+0000", 2024-04-22T18:39:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018503689999999999 2024-04-22T18:39:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855932+0000", 2024-04-22T18:39:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010071 2024-04-22T18:39:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856029+0000", 2024-04-22T18:39:32.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6751999999999998e-05 2024-04-22T18:39:32.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058013+0000", 2024-04-22T18:39:32.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20198412600000001 2024-04-22T18:39:32.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159326+0000", 2024-04-22T18:39:32.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101312969 2024-04-22T18:39:32.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159376+0000", 2024-04-22T18:39:32.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9968e-05 2024-04-22T18:39:32.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159416+0000", 2024-04-22T18:39:32.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9641999999999998e-05 2024-04-22T18:39:32.393 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.393 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.393 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.393 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:32.393 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:32.393 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.488556819000003, 2024-04-22T18:39:32.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30548257499999998, 2024-04-22T18:39:32.394 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.394 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.394 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.394 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.394 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.394 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 108 2024-04-22T18:39:32.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.394 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:32.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:32.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:32.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:32.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854064+0000", 2024-04-22T18:39:32.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5820000000000002e-06 2024-04-22T18:39:32.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854080+0000", 2024-04-22T18:39:32.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5379999999999998e-05 2024-04-22T18:39:32.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854084+0000", 2024-04-22T18:39:32.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.963e-06 2024-04-22T18:39:32.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856413+0000", 2024-04-22T18:39:32.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023298070000000001 2024-04-22T18:39:32.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856494+0000", 2024-04-22T18:39:32.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0863000000000002e-05 2024-04-22T18:39:32.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856624+0000", 2024-04-22T18:39:32.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129799 2024-04-22T18:39:32.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058947+0000", 2024-04-22T18:39:32.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20232328899999999 2024-04-22T18:39:32.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159468+0000", 2024-04-22T18:39:32.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100520597 2024-04-22T18:39:32.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159508+0000", 2024-04-22T18:39:32.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9830000000000003e-05 2024-04-22T18:39:32.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159544+0000", 2024-04-22T18:39:32.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5944e-05 2024-04-22T18:39:32.399 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.399 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.399 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.399 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:32.399 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:32.399 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.488512327000002, 2024-04-22T18:39:32.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30556502499999999, 2024-04-22T18:39:32.399 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.399 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.399 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.400 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.400 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.400 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:39:32.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.400 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:32.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:32.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854105+0000", 2024-04-22T18:39:32.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:32.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854175+0000", 2024-04-22T18:39:32.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0178000000000005e-05 2024-04-22T18:39:32.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854191+0000", 2024-04-22T18:39:32.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5965999999999999e-05 2024-04-22T18:39:32.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854196+0000", 2024-04-22T18:39:32.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.42e-06 2024-04-22T18:39:32.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956526+0000", 2024-04-22T18:39:32.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10233094299999999 2024-04-22T18:39:32.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956634+0000", 2024-04-22T18:39:32.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107398 2024-04-22T18:39:32.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956730+0000", 2024-04-22T18:39:32.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6217000000000004e-05 2024-04-22T18:39:32.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058955+0000", 2024-04-22T18:39:32.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102224749 2024-04-22T18:39:32.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159596+0000", 2024-04-22T18:39:32.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100641297 2024-04-22T18:39:32.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159637+0000", 2024-04-22T18:39:32.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874000000000002e-05 2024-04-22T18:39:32.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159671+0000", 2024-04-22T18:39:32.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3603999999999999e-05 2024-04-22T18:39:32.404 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.404 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.404 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.405 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:32.405 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:32.405 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.48841848, 2024-04-22T18:39:32.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30683048800000001, 2024-04-22T18:39:32.405 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.405 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.405 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.405 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.405 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.405 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110 2024-04-22T18:39:32.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.405 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:32.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:32.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:32.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:32.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854202+0000", 2024-04-22T18:39:32.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:39:32.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854206+0000", 2024-04-22T18:39:32.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5870000000000001e-06 2024-04-22T18:39:32.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854208+0000", 2024-04-22T18:39:32.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7199999999999998e-06 2024-04-22T18:39:32.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057285+0000", 2024-04-22T18:39:32.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20307620800000001 2024-04-22T18:39:32.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057414+0000", 2024-04-22T18:39:32.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012896899999999999 2024-04-22T18:39:32.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057588+0000", 2024-04-22T18:39:32.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000174802 2024-04-22T18:39:32.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159429+0000", 2024-04-22T18:39:32.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101840783 2024-04-22T18:39:32.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160947+0000", 2024-04-22T18:39:32.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001518152 2024-04-22T18:39:32.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160992+0000", 2024-04-22T18:39:32.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4725000000000003e-05 2024-04-22T18:39:32.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161030+0000", 2024-04-22T18:39:32.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7789e-05 2024-04-22T18:39:32.410 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.410 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.410 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.410 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:32.410 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:32.410 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.488407299000002, 2024-04-22T18:39:32.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:39:32.411 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.411 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.411 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.411 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.411 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.411 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:39:32.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.411 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:32.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:32.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:39:32.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:32.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:39:32.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:39:32.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:39:32.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:39:32.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:39:32.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:39:32.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:39:32.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:39:32.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:39:32.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:39:32.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:39:32.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:39:32.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:39:32.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:39:32.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:39:32.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:39:32.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:39:32.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:39:32.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:39:32.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:39:32.416 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.416 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.416 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.416 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:32.416 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:32.416 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.488391641, 2024-04-22T18:39:32.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:39:32.416 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.416 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.417 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.417 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:39:32.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.417 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:32.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:32.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:32.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:32.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:39:32.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:39:32.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:39:32.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:39:32.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:39:32.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:39:32.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:39:32.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:39:32.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:39:32.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:39:32.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:39:32.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:39:32.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:39:32.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:39:32.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:39:32.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:39:32.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:39:32.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:39:32.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:39:32.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:39:32.421 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.421 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.421 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.422 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:32.422 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:32.422 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.488340479000001, 2024-04-22T18:39:32.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:39:32.422 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.422 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.422 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.422 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.422 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.422 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:39:32.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.422 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:39:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:39:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:39:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:39:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:39:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:39:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:39:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:39:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:39:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:39:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:39:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:39:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:39:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:39:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:39:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:39:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:39:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:39:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:39:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:39:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.488325254999999, 2024-04-22T18:39:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:39:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:39:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:39:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:39:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:39:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:39:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:39:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:39:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:39:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:39:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:39:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:39:32.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:39:32.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:39:32.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:39:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:39:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:39:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:39:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:39:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:39:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:39:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:39:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:39:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.488265677999998, 2024-04-22T18:39:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:39:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:39:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:32.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:32.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:32.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:39:32.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:39:32.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:39:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:39:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:39:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:39:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:39:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:39:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:39:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:39:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:39:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:39:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:39:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:39:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:39:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:39:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:39:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:39:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:39:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:39:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.488183362000001, 2024-04-22T18:39:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:39:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.442 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.442 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:39:32.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.442 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:32.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:32.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:39:32.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:32.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:39:32.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:39:32.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:39:32.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:39:32.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:39:32.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:39:32.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:39:32.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:39:32.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:39:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:39:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:39:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:39:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:39:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:39:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:39:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:39:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:39:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:39:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:39:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:39:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.488173205999999, 2024-04-22T18:39:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:39:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:39:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:39:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:39:32.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:39:32.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:39:32.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:39:32.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:39:32.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:39:32.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:39:32.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:39:32.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:39:32.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:39:32.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:39:32.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:39:32.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:39:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:39:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:39:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:39:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:39:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:39:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:39:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:39:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:161 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:39:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 43.887678915000002, 2024-04-22T18:39:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40239169000000002, 2024-04-22T18:39:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 161 2024-04-22T18:39:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454938+0000", 2024-04-22T18:39:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454982+0000", 2024-04-22T18:39:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3470999999999998e-05 2024-04-22T18:39:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454988+0000", 2024-04-22T18:39:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7640000000000002e-06 2024-04-22T18:39:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555054+0000", 2024-04-22T18:39:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006607300000001 2024-04-22T18:39:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555607+0000", 2024-04-22T18:39:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055319500000000003 2024-04-22T18:39:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555689+0000", 2024-04-22T18:39:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1817999999999995e-05 2024-04-22T18:39:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555804+0000", 2024-04-22T18:39:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115346 2024-04-22T18:39:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556554+0000", 2024-04-22T18:39:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000750102 2024-04-22T18:39:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857265+0000", 2024-04-22T18:39:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30071122700000003 2024-04-22T18:39:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857302+0000", 2024-04-22T18:39:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7069e-05 2024-04-22T18:39:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857331+0000", 2024-04-22T18:39:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8291999999999999e-05 2024-04-22T18:39:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.458 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:32.458 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:32.458 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:32.459 INFO:tasks.rados.rados.0.smithi089.stdout:931: expect (ObjNum 119 snap 0 seq_num 119) 2024-04-22T18:39:32.459 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:32.460+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:32.460 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:32.482 INFO:tasks.rados.rados.0.smithi089.stdout:938: seq_num 141 ranges {769787=704463,2267961=101826} 2024-04-22T18:39:32.490 INFO:tasks.rados.rados.0.smithi089.stdout:938: writing smithi08937929-12 from 769787 to 1474250 tid 1 2024-04-22T18:39:32.495 INFO:tasks.rados.rados.0.smithi089.stdout:938: writing smithi08937929-12 from 2267961 to 2369787 tid 2 2024-04-22T18:39:32.495 INFO:tasks.rados.rados.0.smithi089.stdout:920: done (13 left) 2024-04-22T18:39:32.495 INFO:tasks.rados.rados.0.smithi089.stdout:929: done (12 left) 2024-04-22T18:39:32.496 INFO:tasks.rados.rados.0.smithi089.stdout:930: done (11 left) 2024-04-22T18:39:32.496 INFO:tasks.rados.rados.0.smithi089.stdout:931: done (10 left) 2024-04-22T18:39:32.496 INFO:tasks.rados.rados.0.smithi089.stdout:939: read oid 31 snap -1 2024-04-22T18:39:32.496 INFO:tasks.rados.rados.0.smithi089.stdout:939: expect (ObjNum 74 snap 0 seq_num 74) 2024-04-22T18:39:32.496 INFO:tasks.rados.rados.0.smithi089.stdout:940: write oid 19 current snap is 0 2024-04-22T18:39:32.496 INFO:tasks.rados.rados.0.smithi089.stdout:933: finishing set_chunk to oid 1 2024-04-22T18:39:32.496 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 1 set_chunk returned error code -95 offset: 922112 length: 856064 2024-04-22T18:39:32.496 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 0 (ObjNum 57 snap 0 seq_num 57) dirty exists 2024-04-22T18:39:32.496 INFO:tasks.rados.rados.0.smithi089.stdout:921: finishing write tid 1 to smithi08937929-41 2024-04-22T18:39:32.496 INFO:tasks.rados.rados.0.smithi089.stdout:940: seq_num 142 ranges {786594=779712,2326443=60151} 2024-04-22T18:39:32.503 INFO:tasks.rados.rados.0.smithi089.stdout:940: writing smithi08937929-19 from 786594 to 1566306 tid 1 2024-04-22T18:39:32.509 INFO:tasks.rados.rados.0.smithi089.stdout:940: writing smithi08937929-19 from 2326443 to 2386594 tid 2 2024-04-22T18:39:32.509 INFO:tasks.rados.rados.0.smithi089.stdout:933: done (11 left) 2024-04-22T18:39:32.509 INFO:tasks.rados.rados.0.smithi089.stdout:935: done (10 left) 2024-04-22T18:39:32.509 INFO:tasks.rados.rados.0.smithi089.stdout:941: read oid 1 snap -1 2024-04-22T18:39:32.509 INFO:tasks.rados.rados.0.smithi089.stdout:941: expect (ObjNum 57 snap 0 seq_num 57) 2024-04-22T18:39:32.509 INFO:tasks.rados.rados.0.smithi089.stdout:942: read oid 36 snap -1 2024-04-22T18:39:32.509 INFO:tasks.rados.rados.0.smithi089.stdout:942: expect (ObjNum 80 snap 0 seq_num 80) 2024-04-22T18:39:32.509 INFO:tasks.rados.rados.0.smithi089.stdout:943: read oid 4 snap -1 2024-04-22T18:39:32.509 INFO:tasks.rados.rados.0.smithi089.stdout:943: expect (ObjNum 120 snap 0 seq_num 120) 2024-04-22T18:39:32.509 INFO:tasks.rados.rados.0.smithi089.stdout:944: delete oid 13 current snap is 0 2024-04-22T18:39:32.509 INFO:tasks.rados.rados.0.smithi089.stdout:921: finishing write tid 2 to smithi08937929-41 2024-04-22T18:39:32.510 INFO:tasks.rados.rados.0.smithi089.stdout:921: finishing write tid 3 to smithi08937929-41 2024-04-22T18:39:32.510 INFO:tasks.rados.rados.0.smithi089.stdout:934: finishing write tid 1 to smithi08937929-16 2024-04-22T18:39:32.510 INFO:tasks.rados.rados.0.smithi089.stdout:934: finishing write tid 2 to smithi08937929-16 2024-04-22T18:39:32.510 INFO:tasks.rados.rados.0.smithi089.stdout:934: finishing write tid 3 to smithi08937929-16 2024-04-22T18:39:32.510 INFO:tasks.rados.rados.0.smithi089.stdout:934: finishing write tid 4 to smithi08937929-16 2024-04-22T18:39:32.510 INFO:tasks.rados.rados.0.smithi089.stdout:934: oid 16 updating version 0 to 70 2024-04-22T18:39:32.510 INFO:tasks.rados.rados.0.smithi089.stdout:934: oid 16 version 70 is already newer than 69 2024-04-22T18:39:32.510 INFO:tasks.rados.rados.0.smithi089.stdout:934: oid 16 version 70 is already newer than 68 2024-04-22T18:39:32.510 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 70 (ObjNum 139 snap 0 seq_num 139) dirty exists 2024-04-22T18:39:32.510 INFO:tasks.rados.rados.0.smithi089.stdout:934: left oid 16 (ObjNum 139 snap 0 seq_num 139) 2024-04-22T18:39:32.510 INFO:tasks.rados.rados.0.smithi089.stdout:923: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-22T18:39:32.560 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:32.561+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:32.568 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:32.568 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:32.568 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:32.568 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:32.577 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:32.580 INFO:tasks.rados.rados.0.smithi089.stdout:921: finishing write tid 4 to smithi08937929-41 2024-04-22T18:39:32.580 INFO:tasks.rados.rados.0.smithi089.stdout:921: oid 41 updating version 0 to 74 2024-04-22T18:39:32.580 INFO:tasks.rados.rados.0.smithi089.stdout:921: oid 41 updating version 74 to 75 2024-04-22T18:39:32.580 INFO:tasks.rados.rados.0.smithi089.stdout:921: oid 41 updating version 75 to 76 2024-04-22T18:39:32.580 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 76 (ObjNum 136 snap 0 seq_num 136) dirty exists 2024-04-22T18:39:32.580 INFO:tasks.rados.rados.0.smithi089.stdout:921: left oid 41 (ObjNum 136 snap 0 seq_num 136) 2024-04-22T18:39:32.580 INFO:tasks.rados.rados.0.smithi089.stdout:932: finishing write tid 1 to smithi08937929-28 2024-04-22T18:39:32.580 INFO:tasks.rados.rados.0.smithi089.stdout:932: finishing write tid 2 to smithi08937929-28 2024-04-22T18:39:32.580 INFO:tasks.rados.rados.0.smithi089.stdout:932: finishing write tid 3 to smithi08937929-28 2024-04-22T18:39:32.580 INFO:tasks.rados.rados.0.smithi089.stdout:932: finishing write tid 4 to smithi08937929-28 2024-04-22T18:39:32.580 INFO:tasks.rados.rados.0.smithi089.stdout:932: oid 28 updating version 0 to 77 2024-04-22T18:39:32.580 INFO:tasks.rados.rados.0.smithi089.stdout:932: oid 28 updating version 77 to 78 2024-04-22T18:39:32.580 INFO:tasks.rados.rados.0.smithi089.stdout:932: oid 28 updating version 78 to 79 2024-04-22T18:39:32.580 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 79 (ObjNum 138 snap 0 seq_num 138) dirty exists 2024-04-22T18:39:32.581 INFO:tasks.rados.rados.0.smithi089.stdout:932: left oid 28 (ObjNum 138 snap 0 seq_num 138) 2024-04-22T18:39:32.581 INFO:tasks.rados.rados.0.smithi089.stdout:936: finishing write tid 1 to smithi08937929-26 2024-04-22T18:39:32.581 INFO:tasks.rados.rados.0.smithi089.stdout:936: finishing write tid 2 to smithi08937929-26 2024-04-22T18:39:32.581 INFO:tasks.rados.rados.0.smithi089.stdout:937: finishing set_chunk to oid 21 2024-04-22T18:39:32.581 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 31 (ObjNum 110 snap 0 seq_num 110) dirty exists 2024-04-22T18:39:32.581 INFO:tasks.rados.rados.0.smithi089.stdout:936: finishing write tid 3 to smithi08937929-26 2024-04-22T18:39:32.581 INFO:tasks.rados.rados.0.smithi089.stdout:936: finishing write tid 4 to smithi08937929-26 2024-04-22T18:39:32.581 INFO:tasks.rados.rados.0.smithi089.stdout:936: oid 26 updating version 0 to 61 2024-04-22T18:39:32.581 INFO:tasks.rados.rados.0.smithi089.stdout:936: oid 26 updating version 61 to 63 2024-04-22T18:39:32.581 INFO:tasks.rados.rados.0.smithi089.stdout:936: oid 26 version 63 is already newer than 62 2024-04-22T18:39:32.581 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 63 (ObjNum 140 snap 0 seq_num 140) dirty exists 2024-04-22T18:39:32.581 INFO:tasks.rados.rados.0.smithi089.stdout:936: left oid 26 (ObjNum 140 snap 0 seq_num 140) 2024-04-22T18:39:32.581 INFO:tasks.rados.rados.0.smithi089.stdout:939: expect (ObjNum 74 snap 0 seq_num 74) 2024-04-22T18:39:32.601 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:39:32.645 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:32.645+0000 7f3093031640 -1 osd.1 48 handle_osd_map injecting map crc failure 2024-04-22T18:39:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:marked out osd.2. 2024-04-22T18:39:32.661 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:32.662+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:32.689 INFO:tasks.rados.rados.0.smithi089.stdout:941: expect (ObjNum 57 snap 0 seq_num 57) 2024-04-22T18:39:32.690 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:32.690 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:32.690 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:32.690 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:32.690 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:32.699 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:32.748 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:32.748 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:32.761 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:32.762+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:32.766 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:32.766 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:32.799 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:32.800 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:32.818 INFO:tasks.rados.rados.0.smithi089.stdout:942: expect (ObjNum 80 snap 0 seq_num 80) 2024-04-22T18:39:32.820 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:32.820 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:32.837 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:32.837 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:32.854 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:32.854 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:32.860 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:32.860 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:32.862 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:32.863+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:32.870 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:32.872 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:32.872 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:32.890 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:32.890 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:32.963 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:32.964+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:32.975 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:32.975 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:32.984 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:33.019 INFO:tasks.rados.rados.0.smithi089.stdout:940: finishing write tid 1 to smithi08937929-19 2024-04-22T18:39:33.019 INFO:tasks.rados.rados.0.smithi089.stdout:940: finishing write tid 2 to smithi08937929-19 2024-04-22T18:39:33.019 INFO:tasks.rados.rados.0.smithi089.stdout:938: finishing write tid 1 to smithi08937929-12 2024-04-22T18:39:33.019 INFO:tasks.rados.rados.0.smithi089.stdout:940: finishing write tid 3 to smithi08937929-19 2024-04-22T18:39:33.019 INFO:tasks.rados.rados.0.smithi089.stdout:940: finishing write tid 4 to smithi08937929-19 2024-04-22T18:39:33.019 INFO:tasks.rados.rados.0.smithi089.stdout:940: oid 19 updating version 0 to 80 2024-04-22T18:39:33.019 INFO:tasks.rados.rados.0.smithi089.stdout:940: oid 19 updating version 80 to 82 2024-04-22T18:39:33.019 INFO:tasks.rados.rados.0.smithi089.stdout:940: oid 19 version 82 is already newer than 81 2024-04-22T18:39:33.019 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 82 (ObjNum 142 snap 0 seq_num 142) dirty exists 2024-04-22T18:39:33.019 INFO:tasks.rados.rados.0.smithi089.stdout:940: left oid 19 (ObjNum 142 snap 0 seq_num 142) 2024-04-22T18:39:33.019 INFO:tasks.rados.rados.0.smithi089.stdout:938: finishing write tid 2 to smithi08937929-12 2024-04-22T18:39:33.019 INFO:tasks.rados.rados.0.smithi089.stdout:938: finishing write tid 3 to smithi08937929-12 2024-04-22T18:39:33.019 INFO:tasks.rados.rados.0.smithi089.stdout:921: done (13 left) 2024-04-22T18:39:33.020 INFO:tasks.rados.rados.0.smithi089.stdout:923: done (12 left) 2024-04-22T18:39:33.020 INFO:tasks.rados.rados.0.smithi089.stdout:928: done (11 left) 2024-04-22T18:39:33.020 INFO:tasks.rados.rados.0.smithi089.stdout:932: done (10 left) 2024-04-22T18:39:33.020 INFO:tasks.rados.rados.0.smithi089.stdout:934: done (9 left) 2024-04-22T18:39:33.020 INFO:tasks.rados.rados.0.smithi089.stdout:936: done (8 left) 2024-04-22T18:39:33.020 INFO:tasks.rados.rados.0.smithi089.stdout:937: done (7 left) 2024-04-22T18:39:33.020 INFO:tasks.rados.rados.0.smithi089.stdout:939: done (6 left) 2024-04-22T18:39:33.020 INFO:tasks.rados.rados.0.smithi089.stdout:940: done (5 left) 2024-04-22T18:39:33.020 INFO:tasks.rados.rados.0.smithi089.stdout:941: done (4 left) 2024-04-22T18:39:33.020 INFO:tasks.rados.rados.0.smithi089.stdout:942: done (3 left) 2024-04-22T18:39:33.020 INFO:tasks.rados.rados.0.smithi089.stdout:944: done (2 left) 2024-04-22T18:39:33.020 INFO:tasks.rados.rados.0.smithi089.stdout:945: read oid 10 snap -1 2024-04-22T18:39:33.020 INFO:tasks.rados.rados.0.smithi089.stdout:945: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:39:33.020 INFO:tasks.rados.rados.0.smithi089.stdout:946: tier_evict oid 45 2024-04-22T18:39:33.021 INFO:tasks.rados.rados.0.smithi089.stdout:946: tier_evict oid 45 snap -1 2024-04-22T18:39:33.021 INFO:tasks.rados.rados.0.smithi089.stdout:938: finishing write tid 4 to smithi08937929-12 2024-04-22T18:39:33.021 INFO:tasks.rados.rados.0.smithi089.stdout:938: oid 12 updating version 0 to 65 2024-04-22T18:39:33.021 INFO:tasks.rados.rados.0.smithi089.stdout:938: oid 12 version 65 is already newer than 64 2024-04-22T18:39:33.021 INFO:tasks.rados.rados.0.smithi089.stdout:938: oid 12 version 65 is already newer than 63 2024-04-22T18:39:33.021 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 65 (ObjNum 141 snap 0 seq_num 141) dirty exists 2024-04-22T18:39:33.021 INFO:tasks.rados.rados.0.smithi089.stdout:938: left oid 12 (ObjNum 141 snap 0 seq_num 141) 2024-04-22T18:39:33.021 INFO:tasks.rados.rados.0.smithi089.stdout:943: expect (ObjNum 120 snap 0 seq_num 120) 2024-04-22T18:39:33.064 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:33.065+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:33.108 INFO:tasks.rados.rados.0.smithi089.stdout:946: got (22) Invalid argument 2024-04-22T18:39:33.108 INFO:tasks.rados.rados.0.smithi089.stdout:938: done (3 left) 2024-04-22T18:39:33.109 INFO:tasks.rados.rados.0.smithi089.stdout:943: done (2 left) 2024-04-22T18:39:33.109 INFO:tasks.rados.rados.0.smithi089.stdout:946: done (1 left) 2024-04-22T18:39:33.109 INFO:tasks.rados.rados.0.smithi089.stdout:947: read oid 44 snap -1 2024-04-22T18:39:33.109 INFO:tasks.rados.rados.0.smithi089.stdout:947: expect (ObjNum 119 snap 0 seq_num 119) 2024-04-22T18:39:33.109 INFO:tasks.rados.rados.0.smithi089.stdout:948: read oid 19 snap -1 2024-04-22T18:39:33.109 INFO:tasks.rados.rados.0.smithi089.stdout:948: expect (ObjNum 142 snap 0 seq_num 142) 2024-04-22T18:39:33.109 INFO:tasks.rados.rados.0.smithi089.stdout:949: read oid 33 snap -1 2024-04-22T18:39:33.109 INFO:tasks.rados.rados.0.smithi089.stdout:949: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-22T18:39:33.109 INFO:tasks.rados.rados.0.smithi089.stdout:950: read oid 40 snap -1 2024-04-22T18:39:33.109 INFO:tasks.rados.rados.0.smithi089.stdout:950: expect (ObjNum 58 snap 0 seq_num 58) 2024-04-22T18:39:33.109 INFO:tasks.rados.rados.0.smithi089.stdout:951: write oid 30 current snap is 0 2024-04-22T18:39:33.109 INFO:tasks.rados.rados.0.smithi089.stdout:951: seq_num 144 ranges {420208=530210,1527054=766050} 2024-04-22T18:39:33.113 INFO:tasks.rados.rados.0.smithi089.stdout:951: writing smithi08937929-30 from 420208 to 950418 tid 1 2024-04-22T18:39:33.121 INFO:tasks.rados.rados.0.smithi089.stdout:951: writing smithi08937929-30 from 1527054 to 2293104 tid 2 2024-04-22T18:39:33.121 INFO:tasks.rados.rados.0.smithi089.stdout:952: write oid 21 current snap is 0 2024-04-22T18:39:33.121 INFO:tasks.rados.rados.0.smithi089.stdout:952: seq_num 145 ranges {437015=721812,1701889=509708} 2024-04-22T18:39:33.123 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:33.123 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:33.127 INFO:tasks.rados.rados.0.smithi089.stdout:952: writing smithi08937929-21 from 437015 to 1158827 tid 1 2024-04-22T18:39:33.133 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:33.134 INFO:tasks.rados.rados.0.smithi089.stdout:952: writing smithi08937929-21 from 1701889 to 2211597 tid 2 2024-04-22T18:39:33.134 INFO:tasks.rados.rados.0.smithi089.stdout:953: write (excl) oid 27 current snap is 0 2024-04-22T18:39:33.134 INFO:tasks.rados.rados.0.smithi089.stdout:953: seq_num 146 ranges {453822=797061,1760371=653366} 2024-04-22T18:39:33.139 INFO:tasks.rados.rados.0.smithi089.stdout:953: writing smithi08937929-27 from 453822 to 1250883 tid 1 2024-04-22T18:39:33.146 INFO:tasks.rados.rados.0.smithi089.stdout:953: writing smithi08937929-27 from 1760371 to 2413737 tid 2 2024-04-22T18:39:33.146 INFO:tasks.rados.rados.0.smithi089.stdout:954: tier_flush oid 39 2024-04-22T18:39:33.146 INFO:tasks.rados.rados.0.smithi089.stdout:954: tier_flush oid 39 snap -1 2024-04-22T18:39:33.146 INFO:tasks.rados.rados.0.smithi089.stdout:955: tier_promote oid 38 2024-04-22T18:39:33.146 INFO:tasks.rados.rados.0.smithi089.stdout:948: expect (ObjNum 142 snap 0 seq_num 142) 2024-04-22T18:39:33.165 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:33.166+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:33.244 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:33.244 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:33.253 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:33.265 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:33.267+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:33.290 INFO:tasks.rados.rados.0.smithi089.stdout:950: expect (ObjNum 58 snap 0 seq_num 58) 2024-04-22T18:39:33.366 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:33.367+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:33.390 INFO:tasks.rados.rados.0.smithi089.stdout:947: expect (ObjNum 119 snap 0 seq_num 119) 2024-04-22T18:39:33.394 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:33.394 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:33.404 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:33.467 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:33.468+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:33.493 INFO:tasks.rados.rados.0.smithi089.stdout:945: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:39:33.549 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:33.549 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:33.558 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:33.563 INFO:tasks.rados.rados.0.smithi089.stdout:949: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-22T18:39:33.568 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:33.569+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:33.658 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:39:33.664 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:33.665 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:33.669 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:33.670+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:33.674 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:33.741 INFO:tasks.rados.rados.0.smithi089.stdout:952: finishing write tid 1 to smithi08937929-21 2024-04-22T18:39:33.741 INFO:tasks.rados.rados.0.smithi089.stdout:952: finishing write tid 2 to smithi08937929-21 2024-04-22T18:39:33.741 INFO:tasks.rados.rados.0.smithi089.stdout:952: finishing write tid 3 to smithi08937929-21 2024-04-22T18:39:33.741 INFO:tasks.rados.rados.0.smithi089.stdout:953: finishing write tid 1 to smithi08937929-27 2024-04-22T18:39:33.741 INFO:tasks.rados.rados.0.smithi089.stdout:953: finishing write tid 2 to smithi08937929-27 2024-04-22T18:39:33.741 INFO:tasks.rados.rados.0.smithi089.stdout:952: finishing write tid 4 to smithi08937929-21 2024-04-22T18:39:33.741 INFO:tasks.rados.rados.0.smithi089.stdout:952: oid 21 updating version 0 to 34 2024-04-22T18:39:33.741 INFO:tasks.rados.rados.0.smithi089.stdout:952: oid 21 updating version 34 to 35 2024-04-22T18:39:33.742 INFO:tasks.rados.rados.0.smithi089.stdout:952: oid 21 updating version 35 to 36 2024-04-22T18:39:33.742 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 36 (ObjNum 145 snap 0 seq_num 145) dirty exists 2024-04-22T18:39:33.742 INFO:tasks.rados.rados.0.smithi089.stdout:952: left oid 21 (ObjNum 145 snap 0 seq_num 145) 2024-04-22T18:39:33.742 INFO:tasks.rados.rados.0.smithi089.stdout:953: finishing write tid 3 to smithi08937929-27 2024-04-22T18:39:33.742 INFO:tasks.rados.rados.0.smithi089.stdout:953: finishing write tid 4 to smithi08937929-27 2024-04-22T18:39:33.742 INFO:tasks.rados.rados.0.smithi089.stdout:953: oid 27 updating version 0 to 66 2024-04-22T18:39:33.742 INFO:tasks.rados.rados.0.smithi089.stdout:953: oid 27 version 66 is already newer than 65 2024-04-22T18:39:33.742 INFO:tasks.rados.rados.0.smithi089.stdout:953: oid 27 version 66 is already newer than 64 2024-04-22T18:39:33.742 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 66 (ObjNum 146 snap 0 seq_num 146) dirty exists 2024-04-22T18:39:33.742 INFO:tasks.rados.rados.0.smithi089.stdout:953: left oid 27 (ObjNum 146 snap 0 seq_num 146) 2024-04-22T18:39:33.742 INFO:tasks.rados.rados.0.smithi089.stdout:955: got (0) Success 2024-04-22T18:39:33.742 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 39 (ObjNum 97 snap 0 seq_num 97) dirty exists 2024-04-22T18:39:33.742 INFO:tasks.rados.rados.0.smithi089.stdout:945: done (9 left) 2024-04-22T18:39:33.743 INFO:tasks.rados.rados.0.smithi089.stdout:947: done (8 left) 2024-04-22T18:39:33.743 INFO:tasks.rados.rados.0.smithi089.stdout:948: done (7 left) 2024-04-22T18:39:33.743 INFO:tasks.rados.rados.0.smithi089.stdout:949: done (6 left) 2024-04-22T18:39:33.743 INFO:tasks.rados.rados.0.smithi089.stdout:950: done (5 left) 2024-04-22T18:39:33.743 INFO:tasks.rados.rados.0.smithi089.stdout:952: done (4 left) 2024-04-22T18:39:33.743 INFO:tasks.rados.rados.0.smithi089.stdout:953: done (3 left) 2024-04-22T18:39:33.743 INFO:tasks.rados.rados.0.smithi089.stdout:955: done (2 left) 2024-04-22T18:39:33.743 INFO:tasks.rados.rados.0.smithi089.stdout:956: read oid 36 snap -1 2024-04-22T18:39:33.743 INFO:tasks.rados.rados.0.smithi089.stdout:956: expect (ObjNum 80 snap 0 seq_num 80) 2024-04-22T18:39:33.743 INFO:tasks.rados.rados.0.smithi089.stdout:957: write oid 47 current snap is 0 2024-04-22T18:39:33.743 INFO:tasks.rados.rados.0.smithi089.stdout:957: seq_num 147 ranges {470629=472310,1418853=513377} 2024-04-22T18:39:33.745 INFO:tasks.rados.rados.0.smithi089.stdout:957: writing smithi08937929-47 from 470629 to 942939 tid 1 2024-04-22T18:39:33.751 INFO:tasks.rados.rados.0.smithi089.stdout:957: writing smithi08937929-47 from 1418853 to 1932230 tid 2 2024-04-22T18:39:33.751 INFO:tasks.rados.rados.0.smithi089.stdout:958: read oid 35 snap -1 2024-04-22T18:39:33.751 INFO:tasks.rados.rados.0.smithi089.stdout:958: expect (ObjNum 121 snap 0 seq_num 121) 2024-04-22T18:39:33.752 INFO:tasks.rados.rados.0.smithi089.stdout:959: set_chunk oid 33 target oid 2024-04-22T18:39:33.770 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:33.771+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:33.825 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:33.826 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:33.835 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:33.870 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:33.871+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:33.960 INFO:tasks.rados.rados.0.smithi089.stdout:959: set_chunk oid 33 offset: 1231360 length: 833536 target oid d2cd654482591d9161cf1caa210fb39af968fec99166bade87d35fd9e35f2bf2 offset: 0 snap -1 2024-04-22T18:39:33.961 INFO:tasks.rados.rados.0.smithi089.stdout:960: read oid 14 snap -1 2024-04-22T18:39:33.961 INFO:tasks.rados.rados.0.smithi089.stdout:960: expect (ObjNum 13 snap 0 seq_num 13) 2024-04-22T18:39:33.961 INFO:tasks.rados.rados.0.smithi089.stdout:961: tier_flush oid 11 2024-04-22T18:39:33.961 INFO:tasks.rados.rados.0.smithi089.stdout:961: tier_flush oid 11 snap -1 2024-04-22T18:39:33.961 INFO:tasks.rados.rados.0.smithi089.stdout:962: tier_flush oid 24 2024-04-22T18:39:33.961 INFO:tasks.rados.rados.0.smithi089.stdout:962: tier_flush oid 24 snap -1 2024-04-22T18:39:33.961 INFO:tasks.rados.rados.0.smithi089.stdout:958: expect (ObjNum 121 snap 0 seq_num 121) 2024-04-22T18:39:33.971 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:33.972+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:33.980 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:33.980 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:33.990 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:34.055 INFO:tasks.rados.rados.0.smithi089.stdout:956: expect (ObjNum 80 snap 0 seq_num 80) 2024-04-22T18:39:34.072 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:34.073+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:34.093 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:34.093 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:34.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:34.173 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:34.174+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:34.212 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:34.213 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:34.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:34.253 INFO:tasks.rados.rados.0.smithi089.stdout:954: got (0) Success 2024-04-22T18:39:34.253 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 0 (ObjNum 134 snap 0 seq_num 134) dirty exists 2024-04-22T18:39:34.253 INFO:tasks.rados.rados.0.smithi089.stdout:954: done (8 left) 2024-04-22T18:39:34.253 INFO:tasks.rados.rados.0.smithi089.stdout:956: done (7 left) 2024-04-22T18:39:34.253 INFO:tasks.rados.rados.0.smithi089.stdout:958: done (6 left) 2024-04-22T18:39:34.253 INFO:tasks.rados.rados.0.smithi089.stdout:963: write oid 50 current snap is 0 2024-04-22T18:39:34.253 INFO:tasks.rados.rados.0.smithi089.stdout:963: seq_num 148 ranges {487436=547559,1760982=657035} 2024-04-22T18:39:34.258 INFO:tasks.rados.rados.0.smithi089.stdout:963: writing smithi08937929-50 from 487436 to 1034995 tid 1 2024-04-22T18:39:34.266 INFO:tasks.rados.rados.0.smithi089.stdout:963: writing smithi08937929-50 from 1760982 to 2418017 tid 2 2024-04-22T18:39:34.266 INFO:tasks.rados.rados.0.smithi089.stdout:964: tier_promote oid 15 2024-04-22T18:39:34.266 INFO:tasks.rados.rados.0.smithi089.stdout:957: finishing write tid 1 to smithi08937929-47 2024-04-22T18:39:34.266 INFO:tasks.rados.rados.0.smithi089.stdout:965: write oid 29 current snap is 0 2024-04-22T18:39:34.266 INFO:tasks.rados.rados.0.smithi089.stdout:957: finishing write tid 2 to smithi08937929-47 2024-04-22T18:39:34.266 INFO:tasks.rados.rados.0.smithi089.stdout:965: seq_num 149 ranges {504243=622808,1819464=517046} 2024-04-22T18:39:34.271 INFO:tasks.rados.rados.0.smithi089.stdout:965: writing smithi08937929-29 from 504243 to 1127051 tid 1 2024-04-22T18:39:34.275 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:34.276+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:34.278 INFO:tasks.rados.rados.0.smithi089.stdout:965: writing smithi08937929-29 from 1819464 to 2336510 tid 2 2024-04-22T18:39:34.278 INFO:tasks.rados.rados.0.smithi089.stdout:966: tier_promote oid 39 2024-04-22T18:39:34.278 INFO:tasks.rados.rados.0.smithi089.stdout:960: expect (ObjNum 13 snap 0 seq_num 13) 2024-04-22T18:39:34.326 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:34.326 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:34.335 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:34.352 INFO:tasks.rados.rados.0.smithi089.stdout:957: finishing write tid 3 to smithi08937929-47 2024-04-22T18:39:34.352 INFO:tasks.rados.rados.0.smithi089.stdout:957: finishing write tid 4 to smithi08937929-47 2024-04-22T18:39:34.352 INFO:tasks.rados.rados.0.smithi089.stdout:957: oid 47 updating version 0 to 35 2024-04-22T18:39:34.352 INFO:tasks.rados.rados.0.smithi089.stdout:957: oid 47 updating version 35 to 36 2024-04-22T18:39:34.352 INFO:tasks.rados.rados.0.smithi089.stdout:957: oid 47 updating version 36 to 37 2024-04-22T18:39:34.352 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 37 (ObjNum 147 snap 0 seq_num 147) dirty exists 2024-04-22T18:39:34.352 INFO:tasks.rados.rados.0.smithi089.stdout:957: left oid 47 (ObjNum 147 snap 0 seq_num 147) 2024-04-22T18:39:34.352 INFO:tasks.rados.rados.0.smithi089.stdout:959: finishing set_chunk to oid 33 2024-04-22T18:39:34.352 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 19 (ObjNum 65 snap 0 seq_num 65) dirty exists 2024-04-22T18:39:34.353 INFO:tasks.rados.rados.0.smithi089.stdout:957: done (9 left) 2024-04-22T18:39:34.353 INFO:tasks.rados.rados.0.smithi089.stdout:959: done (8 left) 2024-04-22T18:39:34.353 INFO:tasks.rados.rados.0.smithi089.stdout:960: done (7 left) 2024-04-22T18:39:34.353 INFO:tasks.rados.rados.0.smithi089.stdout:967: write oid 17 current snap is 0 2024-04-22T18:39:34.353 INFO:tasks.rados.rados.0.smithi089.stdout:967: seq_num 150 ranges {521050=698057,1877946=643104} 2024-04-22T18:39:34.358 INFO:tasks.rados.rados.0.smithi089.stdout:967: writing smithi08937929-17 from 521050 to 1219107 tid 1 2024-04-22T18:39:34.365 INFO:tasks.rados.rados.0.smithi089.stdout:967: writing smithi08937929-17 from 1877946 to 2521050 tid 2 2024-04-22T18:39:34.365 INFO:tasks.rados.rados.0.smithi089.stdout:968: write oid 13 current snap is 0 2024-04-22T18:39:34.365 INFO:tasks.rados.rados.0.smithi089.stdout:968: seq_num 151 ranges {537857=773306,1936428=520715} 2024-04-22T18:39:34.372 INFO:tasks.rados.rados.0.smithi089.stdout:968: writing smithi08937929-13 from 537857 to 1311163 tid 1 2024-04-22T18:39:34.378 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:34.379+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:34.379 INFO:tasks.rados.rados.0.smithi089.stdout:968: writing smithi08937929-13 from 1936428 to 2457143 tid 2 2024-04-22T18:39:34.379 INFO:tasks.rados.rados.0.smithi089.stdout:969: read oid 6 snap -1 2024-04-22T18:39:34.379 INFO:tasks.rados.rados.0.smithi089.stdout:969: expect (ObjNum 113 snap 0 seq_num 113) 2024-04-22T18:39:34.379 INFO:tasks.rados.rados.0.smithi089.stdout:970: set_chunk oid 32 target oid 2024-04-22T18:39:34.441 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:34.441 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:34.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:34.491 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:34.492+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:34.577 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:34.578+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:34.596 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:34.596 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:34.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:34.653 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:34.654+0000 7f7f44bb3640 -1 osd.4 50 handle_osd_map injecting map crc failure 2024-04-22T18:39:34.654 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:39:34.708 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:34.709+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:34.746 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:34.746 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:34.755 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:34.770 INFO:tasks.rados.rados.0.smithi089.stdout:970: set_chunk oid 32 offset: 50176 length: 3318272 target oid 6c3137466225fc07bb05c0c94b49f420f61309dedf1b3d9ebee72fdf88d9145c offset: 0 snap -1 2024-04-22T18:39:34.770 INFO:tasks.rados.rados.0.smithi089.stdout:971: read oid 35 snap -1 2024-04-22T18:39:34.770 INFO:tasks.rados.rados.0.smithi089.stdout:971: expect (ObjNum 121 snap 0 seq_num 121) 2024-04-22T18:39:34.771 INFO:tasks.rados.rados.0.smithi089.stdout:972: write (excl) oid 45 current snap is 0 2024-04-22T18:39:34.771 INFO:tasks.rados.rados.0.smithi089.stdout:972: seq_num 152 ranges {554664=448555,1478557=664373} 2024-04-22T18:39:34.773 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:34.774+0000 7fa768d81640 -1 osd.5 50 handle_osd_map injecting map crc failure 2024-04-22T18:39:34.775 INFO:tasks.rados.rados.0.smithi089.stdout:972: writing smithi08937929-45 from 554664 to 1003219 tid 1 2024-04-22T18:39:34.778 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:34.780+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:34.782 INFO:tasks.rados.rados.0.smithi089.stdout:972: writing smithi08937929-45 from 1478557 to 2142930 tid 2 2024-04-22T18:39:34.782 INFO:tasks.rados.rados.0.smithi089.stdout:973: set_chunk oid 18 target oid 2024-04-22T18:39:34.875 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:34.875 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:34.880 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:34.881+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:34.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:34.902 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:39:34.981 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:34.982+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:34.988 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:34.988 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:34.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:35.020 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:35.021 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:35.076 INFO:tasks.rados.rados.0.smithi089.stdout:973: set_chunk oid 18 offset: 1472000 length: 70144 target oid 160fe14a04c81fc94e208850df8283f9a3736840a418ca51babbd6f1056ffbcf offset: 0 snap -1 2024-04-22T18:39:35.076 INFO:tasks.rados.rados.0.smithi089.stdout:974: read oid 14 snap -1 2024-04-22T18:39:35.076 INFO:tasks.rados.rados.0.smithi089.stdout:974: expect (ObjNum 13 snap 0 seq_num 13) 2024-04-22T18:39:35.076 INFO:tasks.rados.rados.0.smithi089.stdout:975: tier_promote oid 40 2024-04-22T18:39:35.076 INFO:tasks.rados.rados.0.smithi089.stdout:951: finishing write tid 1 to smithi08937929-30 2024-04-22T18:39:35.077 INFO:tasks.rados.rados.0.smithi089.stdout:951: finishing write tid 2 to smithi08937929-30 2024-04-22T18:39:35.077 INFO:tasks.rados.rados.0.smithi089.stdout:951: finishing write tid 3 to smithi08937929-30 2024-04-22T18:39:35.077 INFO:tasks.rados.rados.0.smithi089.stdout:951: finishing write tid 4 to smithi08937929-30 2024-04-22T18:39:35.077 INFO:tasks.rados.rados.0.smithi089.stdout:951: oid 30 updating version 0 to 23 2024-04-22T18:39:35.077 INFO:tasks.rados.rados.0.smithi089.stdout:951: oid 30 updating version 23 to 24 2024-04-22T18:39:35.077 INFO:tasks.rados.rados.0.smithi089.stdout:951: oid 30 updating version 24 to 25 2024-04-22T18:39:35.077 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 25 (ObjNum 144 snap 0 seq_num 144) dirty exists 2024-04-22T18:39:35.077 INFO:tasks.rados.rados.0.smithi089.stdout:951: left oid 30 (ObjNum 144 snap 0 seq_num 144) 2024-04-22T18:39:35.077 INFO:tasks.rados.rados.0.smithi089.stdout:969: expect (ObjNum 113 snap 0 seq_num 113) 2024-04-22T18:39:35.081 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:35.083+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:35.098 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:35.098 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:35.113 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:35.114 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:35.129 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:35.130 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:35.139 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:35.139 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:35.146 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:35.146 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:35.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:35.162 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:35.163 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:35.192 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:35.193+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:35.210 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:35.211 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:35.241 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:35.241 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:35.253 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:35.253 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:35.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:35.253 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:35.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.253 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:21 4.a 4:5aa97f94:::smithi08937929-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.253 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:35.254 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.100029882999998, 2024-04-22T18:39:35.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21044934000000001, 2024-04-22T18:39:35.254 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.254 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.254 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.254 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.254 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.254 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 21 2024-04-22T18:39:35.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.254 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:35.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:35.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153183+0000", 2024-04-22T18:39:35.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:35.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154022+0000", 2024-04-22T18:39:35.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00083893300000000004 2024-04-22T18:39:35.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154032+0000", 2024-04-22T18:39:35.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0433e-05 2024-04-22T18:39:35.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154051+0000", 2024-04-22T18:39:35.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8227e-05 2024-04-22T18:39:35.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154103+0000", 2024-04-22T18:39:35.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2491999999999998e-05 2024-04-22T18:39:35.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:35.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154111+0000", 2024-04-22T18:39:35.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3470000000000001e-06 2024-04-22T18:39:35.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257848+0000", 2024-04-22T18:39:35.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103737703 2024-04-22T18:39:35.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257981+0000", 2024-04-22T18:39:35.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013290399999999999 2024-04-22T18:39:35.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:35.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.258138+0000", 2024-04-22T18:39:35.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015701199999999999 2024-04-22T18:39:35.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.261944+0000", 2024-04-22T18:39:35.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038054650000000001 2024-04-22T18:39:35.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363523+0000", 2024-04-22T18:39:35.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101579239 2024-04-22T18:39:35.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363577+0000", 2024-04-22T18:39:35.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4095000000000003e-05 2024-04-22T18:39:35.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363635+0000", 2024-04-22T18:39:35.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8362999999999997e-05 2024-04-22T18:39:35.259 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.259 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.260 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:22 4.a 4:5aa97f94:::smithi08937929-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.260 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:35.260 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.095955906999997, 2024-04-22T18:39:35.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30948398900000001, 2024-04-22T18:39:35.260 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.260 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.260 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.260 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.260 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.260 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 22 2024-04-22T18:39:35.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.260 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:35.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:35.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157259+0000", 2024-04-22T18:39:35.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:35.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157318+0000", 2024-04-22T18:39:35.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8906000000000001e-05 2024-04-22T18:39:35.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157327+0000", 2024-04-22T18:39:35.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:39:35.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157332+0000", 2024-04-22T18:39:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4689999999999996e-06 2024-04-22T18:39:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257479+0000", 2024-04-22T18:39:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014700999999999 2024-04-22T18:39:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257492+0000", 2024-04-22T18:39:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2574e-05 2024-04-22T18:39:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259582+0000", 2024-04-22T18:39:35.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020904019999999999 2024-04-22T18:39:35.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259668+0000", 2024-04-22T18:39:35.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5826000000000001e-05 2024-04-22T18:39:35.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:35.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259772+0000", 2024-04-22T18:39:35.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104048 2024-04-22T18:39:35.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363453+0000", 2024-04-22T18:39:35.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103680183 2024-04-22T18:39:35.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466692+0000", 2024-04-22T18:39:35.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10323948199999999 2024-04-22T18:39:35.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466720+0000", 2024-04-22T18:39:35.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7886e-05 2024-04-22T18:39:35.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466744+0000", 2024-04-22T18:39:35.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4049e-05 2024-04-22T18:39:35.266 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.266 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.266 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.266 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:23 4.a 4:5aa97f94:::smithi08937929-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.266 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:35.266 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.095878220000003, 2024-04-22T18:39:35.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39003820300000003, 2024-04-22T18:39:35.266 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.266 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.266 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.267 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.267 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.267 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 23 2024-04-22T18:39:35.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.267 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:35.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:35.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157337+0000", 2024-04-22T18:39:35.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157340+0000", 2024-04-22T18:39:35.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8609999999999998e-06 2024-04-22T18:39:35.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157343+0000", 2024-04-22T18:39:35.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9859999999999999e-06 2024-04-22T18:39:35.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157346+0000", 2024-04-22T18:39:35.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8119999999999999e-06 2024-04-22T18:39:35.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257563+0000", 2024-04-22T18:39:35.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100217497 2024-04-22T18:39:35.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:35.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257574+0000", 2024-04-22T18:39:35.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0777e-05 2024-04-22T18:39:35.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260112+0000", 2024-04-22T18:39:35.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002537681 2024-04-22T18:39:35.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260191+0000", 2024-04-22T18:39:35.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9320999999999995e-05 2024-04-22T18:39:35.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:35.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260333+0000", 2024-04-22T18:39:35.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014177699999999999 2024-04-22T18:39:35.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363792+0000", 2024-04-22T18:39:35.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103458862 2024-04-22T18:39:35.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547303+0000", 2024-04-22T18:39:35.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18351162100000001 2024-04-22T18:39:35.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547337+0000", 2024-04-22T18:39:35.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3640999999999999e-05 2024-04-22T18:39:35.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547376+0000", 2024-04-22T18:39:35.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8915000000000003e-05 2024-04-22T18:39:35.272 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.272 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.272 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.272 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:24 4.a 4:5aa97f94:::smithi08937929-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.272 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:35.272 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.095851789000001, 2024-04-22T18:39:35.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39017823499999998, 2024-04-22T18:39:35.273 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.273 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:35.273 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.273 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.273 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.273 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 24 2024-04-22T18:39:35.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.273 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:35.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:35.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:35.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157366+0000", 2024-04-22T18:39:35.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3120000000000001e-06 2024-04-22T18:39:35.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157370+0000", 2024-04-22T18:39:35.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1300000000000003e-06 2024-04-22T18:39:35.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157373+0000", 2024-04-22T18:39:35.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-04-22T18:39:35.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260553+0000", 2024-04-22T18:39:35.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103179996 2024-04-22T18:39:35.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:35.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260625+0000", 2024-04-22T18:39:35.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2236000000000004e-05 2024-04-22T18:39:35.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547414+0000", 2024-04-22T18:39:35.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.286789181 2024-04-22T18:39:35.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547471+0000", 2024-04-22T18:39:35.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6192e-05 2024-04-22T18:39:35.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547542+0000", 2024-04-22T18:39:35.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1622e-05 2024-04-22T18:39:35.277 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.277 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.277 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.277 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:49 4.a 4:5d2aee44:::smithi08937929-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.277 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:35.277 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.019053131, 2024-04-22T18:39:35.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31344442500000003, 2024-04-22T18:39:35.277 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.277 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.278 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.278 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.278 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.278 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 49 2024-04-22T18:39:35.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.278 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:35.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:35.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234158+0000", 2024-04-22T18:39:35.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999957 2024-04-22T18:39:35.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234901+0000", 2024-04-22T18:39:35.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074250800000000001 2024-04-22T18:39:35.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234914+0000", 2024-04-22T18:39:35.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3244999999999999e-05 2024-04-22T18:39:35.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234922+0000", 2024-04-22T18:39:35.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4590000000000001e-06 2024-04-22T18:39:35.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260665+0000", 2024-04-22T18:39:35.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025743423000000001 2024-04-22T18:39:35.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260779+0000", 2024-04-22T18:39:35.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114063 2024-04-22T18:39:35.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:35.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260913+0000", 2024-04-22T18:39:35.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013355499999999999 2024-04-22T18:39:35.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363798+0000", 2024-04-22T18:39:35.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102885003 2024-04-22T18:39:35.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547567+0000", 2024-04-22T18:39:35.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.183768971 2024-04-22T18:39:35.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547589+0000", 2024-04-22T18:39:35.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2507999999999999e-05 2024-04-22T18:39:35.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547607+0000", 2024-04-22T18:39:35.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8009999999999999e-05 2024-04-22T18:39:35.282 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.283 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.283 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.283 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:50 4.a 4:5d2aee44:::smithi08937929-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.283 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:35.283 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.013606952000004, 2024-04-22T18:39:35.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40856295399999998, 2024-04-22T18:39:35.283 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.283 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.283 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.283 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.283 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.283 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 50 2024-04-22T18:39:35.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239607+0000", 2024-04-22T18:39:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240309+0000", 2024-04-22T18:39:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070201599999999997 2024-04-22T18:39:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240320+0000", 2024-04-22T18:39:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1024e-05 2024-04-22T18:39:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240327+0000", 2024-04-22T18:39:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0049999999999998e-06 2024-04-22T18:39:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362065+0000", 2024-04-22T18:39:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.121737812 2024-04-22T18:39:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362121+0000", 2024-04-22T18:39:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5868000000000001e-05 2024-04-22T18:39:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:35.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362210+0000", 2024-04-22T18:39:35.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8533000000000001e-05 2024-04-22T18:39:35.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365322+0000", 2024-04-22T18:39:35.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031121220000000002 2024-04-22T18:39:35.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648054+0000", 2024-04-22T18:39:35.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.282732549 2024-04-22T18:39:35.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648105+0000", 2024-04-22T18:39:35.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1276999999999997e-05 2024-04-22T18:39:35.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648172+0000", 2024-04-22T18:39:35.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6398999999999999e-05 2024-04-22T18:39:35.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.288 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.289 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:51 4.a 4:5d2aee44:::smithi08937929-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.289 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:35.289 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.012884417999999, 2024-04-22T18:39:35.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40793057199999999, 2024-04-22T18:39:35.289 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.289 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.289 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.289 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.289 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.289 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 51 2024-04-22T18:39:35.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.289 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:35.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:35.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:35.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240336+0000", 2024-04-22T18:39:35.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:39:35.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240339+0000", 2024-04-22T18:39:35.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9009999999999998e-06 2024-04-22T18:39:35.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240342+0000", 2024-04-22T18:39:35.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:39:35.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363080+0000", 2024-04-22T18:39:35.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122738075 2024-04-22T18:39:35.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363164+0000", 2024-04-22T18:39:35.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3722999999999998e-05 2024-04-22T18:39:35.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:35.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363287+0000", 2024-04-22T18:39:35.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012301099999999999 2024-04-22T18:39:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466753+0000", 2024-04-22T18:39:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103466217 2024-04-22T18:39:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648205+0000", 2024-04-22T18:39:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.181451788 2024-04-22T18:39:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648231+0000", 2024-04-22T18:39:35.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6597999999999998e-05 2024-04-22T18:39:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648262+0000", 2024-04-22T18:39:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0871000000000003e-05 2024-04-22T18:39:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:52 4.a 4:5d2aee44:::smithi08937929-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.012871959000002, 2024-04-22T18:39:35.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40812731200000002, 2024-04-22T18:39:35.295 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.295 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:35.295 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.295 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.295 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.295 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 52 2024-04-22T18:39:35.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.295 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:35.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:35.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:35.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240345+0000", 2024-04-22T18:39:35.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4300000000000001e-06 2024-04-22T18:39:35.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240349+0000", 2024-04-22T18:39:35.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2570000000000001e-06 2024-04-22T18:39:35.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240358+0000", 2024-04-22T18:39:35.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-22T18:39:35.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363697+0000", 2024-04-22T18:39:35.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12333965299999999 2024-04-22T18:39:35.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:35.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363777+0000", 2024-04-22T18:39:35.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9883999999999996e-05 2024-04-22T18:39:35.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648296+0000", 2024-04-22T18:39:35.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451861299999998 2024-04-22T18:39:35.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648379+0000", 2024-04-22T18:39:35.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3302999999999998e-05 2024-04-22T18:39:35.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648471+0000", 2024-04-22T18:39:35.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2029999999999998e-05 2024-04-22T18:39:35.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.299 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.299 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.299 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:35.299 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.993568623999998, 2024-04-22T18:39:35.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:39:35.299 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.299 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.300 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.300 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.300 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.300 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:39:35.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.300 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:35.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:35.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:39:35.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:35.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:39:35.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:39:35.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:39:35.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:39:35.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:39:35.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:39:35.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:39:35.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:39:35.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:39:35.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:39:35.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:35.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:39:35.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:39:35.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:39:35.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:39:35.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:39:35.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:39:35.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:39:35.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:39:35.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:39:35.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:39:35.304 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.304 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.305 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.305 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.305 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:35.305 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.988786193999999, 2024-04-22T18:39:35.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:39:35.305 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.305 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.305 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.305 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.305 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.305 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:39:35.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.305 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:35.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:35.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:39:35.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:35.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:39:35.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:39:35.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:39:35.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:39:35.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:39:35.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:39:35.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:39:35.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:39:35.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:39:35.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:39:35.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:35.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:39:35.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:39:35.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:39:35.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:39:35.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:39:35.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:39:35.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:39:35.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:39:35.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:39:35.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:39:35.310 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.310 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.310 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.310 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.311 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:35.311 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.988691991000003, 2024-04-22T18:39:35.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:39:35.311 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.311 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.311 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.311 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.311 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.311 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:39:35.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.311 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:35.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:35.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:39:35.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:35.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:39:35.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:39:35.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:39:35.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:39:35.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:39:35.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:39:35.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:39:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:39:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:39:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:39:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:39:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:39:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:39:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:39:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:39:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:39:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:39:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:39:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:39:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:39:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.988671103000001, 2024-04-22T18:39:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:39:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:35.316 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:39:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:39:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:39:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:39:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:39:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:39:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:39:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:39:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:39:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:39:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:39:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:39:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:39:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:39:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:39:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:39:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:39:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:39:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.205663440000002, 2024-04-22T18:39:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:39:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.321 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:39:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:39:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:39:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:39:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:39:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:39:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:39:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:39:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:39:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:39:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:39:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:39:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:39:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:39:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:39:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:39:35.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:39:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:39:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:39:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:39:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:39:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:39:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.197832034000001, 2024-04-22T18:39:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:39:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.327 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:39:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:39:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:39:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:39:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:39:35.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:39:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:39:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:39:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:39:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:39:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:39:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:39:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:39:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:39:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:39:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:39:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:39:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:39:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:39:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:39:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:39:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:39:35.332 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:186 4.a 4:537c89f8:::smithi08937929-47:head [write 1992054~651681] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 39.850326498999998, 2024-04-22T18:39:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20325226800000001, 2024-04-22T18:39:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.333 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 186 2024-04-22T18:39:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402888+0000", 2024-04-22T18:39:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403454+0000", 2024-04-22T18:39:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056661000000000005 2024-04-22T18:39:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403465+0000", 2024-04-22T18:39:35.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0727000000000001e-05 2024-04-22T18:39:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403473+0000", 2024-04-22T18:39:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7039999999999994e-06 2024-04-22T18:39:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404592+0000", 2024-04-22T18:39:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001118827 2024-04-22T18:39:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404674+0000", 2024-04-22T18:39:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2793999999999994e-05 2024-04-22T18:39:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404779+0000", 2024-04-22T18:39:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104465 2024-04-22T18:39:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.406080+0000", 2024-04-22T18:39:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013006789999999999 2024-04-22T18:39:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606037+0000", 2024-04-22T18:39:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19995755600000001 2024-04-22T18:39:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606098+0000", 2024-04-22T18:39:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1075999999999996e-05 2024-04-22T18:39:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606142+0000", 2024-04-22T18:39:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3411999999999997e-05 2024-04-22T18:39:35.338 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:187 4.a 4:537c89f8:::smithi08937929-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 39.849736358999998, 2024-04-22T18:39:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203643623, 2024-04-22T18:39:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 187 2024-04-22T18:39:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403501+0000", 2024-04-22T18:39:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2146000000000001e-05 2024-04-22T18:39:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403506+0000", 2024-04-22T18:39:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3040000000000004e-06 2024-04-22T18:39:35.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403511+0000", 2024-04-22T18:39:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9599999999999999e-06 2024-04-22T18:39:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405587+0000", 2024-04-22T18:39:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002076296 2024-04-22T18:39:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405664+0000", 2024-04-22T18:39:35.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6316000000000006e-05 2024-04-22T18:39:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405785+0000", 2024-04-22T18:39:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121276 2024-04-22T18:39:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.408030+0000", 2024-04-22T18:39:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022452169999999999 2024-04-22T18:39:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607059+0000", 2024-04-22T18:39:35.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199029135 2024-04-22T18:39:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607094+0000", 2024-04-22T18:39:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4686e-05 2024-04-22T18:39:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607123+0000", 2024-04-22T18:39:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9113000000000001e-05 2024-04-22T18:39:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:811 4.5 4:a4a14dff:::smithi08937929-40:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-22T18:39:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 31.132393902, 2024-04-22T18:39:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242306563, 2024-04-22T18:39:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 811 2024-04-22T18:39:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120821+0000", 2024-04-22T18:39:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120825+0000", 2024-04-22T18:39:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.506e-06 2024-04-22T18:39:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120829+0000", 2024-04-22T18:39:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0849999999999999e-06 2024-04-22T18:39:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120833+0000", 2024-04-22T18:39:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9609999999999999e-06 2024-04-22T18:39:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221263+0000", 2024-04-22T18:39:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10043045 2024-04-22T18:39:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221337+0000", 2024-04-22T18:39:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4010000000000005e-05 2024-04-22T18:39:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221448+0000", 2024-04-22T18:39:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011065699999999999 2024-04-22T18:39:35.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.222541+0000", 2024-04-22T18:39:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010932610000000001 2024-04-22T18:39:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.362952+0000", 2024-04-22T18:39:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.140410535 2024-04-22T18:39:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.363102+0000", 2024-04-22T18:39:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015041499999999999 2024-04-22T18:39:35.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.363129+0000", 2024-04-22T18:39:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6574999999999999e-05 2024-04-22T18:39:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:933 4.a 4:5d2aee44:::smithi08937929-13:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.608470264000001, 2024-04-22T18:39:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202744591, 2024-04-22T18:39:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:39:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 933 2024-04-22T18:39:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644743+0000", 2024-04-22T18:39:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644754+0000", 2024-04-22T18:39:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0807e-05 2024-04-22T18:39:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644768+0000", 2024-04-22T18:39:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3844e-05 2024-04-22T18:39:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.744860+0000", 2024-04-22T18:39:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100092835 2024-04-22T18:39:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.745084+0000", 2024-04-22T18:39:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022401699999999999 2024-04-22T18:39:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.745090+0000", 2024-04-22T18:39:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2970000000000003e-06 2024-04-22T18:39:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.847381+0000", 2024-04-22T18:39:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102291544 2024-04-22T18:39:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.847490+0000", 2024-04-22T18:39:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010891399999999999 2024-04-22T18:39:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1286 4.5 4:a4a14dff:::smithi08937929-40:head [sparse-read 0~1336255,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.415950758000001, 2024-04-22T18:39:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202431208, 2024-04-22T18:39:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.355 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1286 2024-04-22T18:39:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837261+0000", 2024-04-22T18:39:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999962 2024-04-22T18:39:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837278+0000", 2024-04-22T18:39:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6359999999999999e-05 2024-04-22T18:39:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837307+0000", 2024-04-22T18:39:35.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9828999999999999e-05 2024-04-22T18:39:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937448+0000", 2024-04-22T18:39:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014044 2024-04-22T18:39:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937540+0000", 2024-04-22T18:39:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2539000000000003e-05 2024-04-22T18:39:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:35.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937548+0000", 2024-04-22T18:39:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4089999999999997e-06 2024-04-22T18:39:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.039347+0000", 2024-04-22T18:39:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10179880700000001 2024-04-22T18:39:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.039505+0000", 2024-04-22T18:39:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000158758 2024-04-22T18:39:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.039696+0000", 2024-04-22T18:39:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00019096599999999999 2024-04-22T18:39:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1288 4.5 4:a4a14dff:::smithi08937929-40:head [sparse-read 0~1336255] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.937524+0000", 2024-04-22T18:39:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.315691935, 2024-04-22T18:39:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20252358200000001, 2024-04-22T18:39:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.360 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1288 2024-04-22T18:39:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937524+0000", 2024-04-22T18:39:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937524+0000", 2024-04-22T18:39:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937523+0000", 2024-04-22T18:39:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937529+0000", 2024-04-22T18:39:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5369999999999997e-06 2024-04-22T18:39:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.937532+0000", 2024-04-22T18:39:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.455e-06 2024-04-22T18:39:35.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.037667+0000", 2024-04-22T18:39:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100134238 2024-04-22T18:39:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.139784+0000", 2024-04-22T18:39:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102117472 2024-04-22T18:39:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.139848+0000", 2024-04-22T18:39:35.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4257000000000005e-05 2024-04-22T18:39:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.140047+0000", 2024-04-22T18:39:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000199099 2024-04-22T18:39:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.364 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.364 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:35.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:35.369 INFO:tasks.rados.rados.0.smithi089.stdout:964: got (2) No such file or directory 2024-04-22T18:39:35.369 INFO:tasks.rados.rados.0.smithi089.stdout:964: got expected ENOENT (src dne) 2024-04-22T18:39:35.370 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 37 (ObjNum 129 snap 0 seq_num 2416848336) clean dne 2024-04-22T18:39:35.370 INFO:tasks.rados.rados.0.smithi089.stdout:963: finishing write tid 1 to smithi08937929-50 2024-04-22T18:39:35.370 INFO:tasks.rados.rados.0.smithi089.stdout:968: finishing write tid 1 to smithi08937929-13 2024-04-22T18:39:35.370 INFO:tasks.rados.rados.0.smithi089.stdout:968: finishing write tid 2 to smithi08937929-13 2024-04-22T18:39:35.370 INFO:tasks.rados.rados.0.smithi089.stdout:968: finishing write tid 3 to smithi08937929-13 2024-04-22T18:39:35.370 INFO:tasks.rados.rados.0.smithi089.stdout:968: finishing write tid 4 to smithi08937929-13 2024-04-22T18:39:35.370 INFO:tasks.rados.rados.0.smithi089.stdout:968: oid 13 updating version 0 to 41 2024-04-22T18:39:35.370 INFO:tasks.rados.rados.0.smithi089.stdout:968: oid 13 version 41 is already newer than 40 2024-04-22T18:39:35.370 INFO:tasks.rados.rados.0.smithi089.stdout:968: oid 13 version 41 is already newer than 39 2024-04-22T18:39:35.370 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 41 (ObjNum 151 snap 0 seq_num 151) dirty exists 2024-04-22T18:39:35.370 INFO:tasks.rados.rados.0.smithi089.stdout:968: left oid 13 (ObjNum 151 snap 0 seq_num 151) 2024-04-22T18:39:35.370 INFO:tasks.rados.rados.0.smithi089.stdout:963: finishing write tid 2 to smithi08937929-50 2024-04-22T18:39:35.370 INFO:tasks.rados.rados.0.smithi089.stdout:951: done (15 left) 2024-04-22T18:39:35.371 INFO:tasks.rados.rados.0.smithi089.stdout:964: done (14 left) 2024-04-22T18:39:35.371 INFO:tasks.rados.rados.0.smithi089.stdout:968: done (13 left) 2024-04-22T18:39:35.371 INFO:tasks.rados.rados.0.smithi089.stdout:969: done (12 left) 2024-04-22T18:39:35.371 INFO:tasks.rados.rados.0.smithi089.stdout:976: read oid 19 snap -1 2024-04-22T18:39:35.371 INFO:tasks.rados.rados.0.smithi089.stdout:976: expect (ObjNum 142 snap 0 seq_num 142) 2024-04-22T18:39:35.371 INFO:tasks.rados.rados.0.smithi089.stdout:977: write (excl) oid 9 current snap is 0 2024-04-22T18:39:35.371 INFO:tasks.rados.rados.0.smithi089.stdout:961: got (0) Success 2024-04-22T18:39:35.371 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 0 (ObjNum 137 snap 0 seq_num 137) dirty exists 2024-04-22T18:39:35.371 INFO:tasks.rados.rados.0.smithi089.stdout:963: finishing write tid 3 to smithi08937929-50 2024-04-22T18:39:35.371 INFO:tasks.rados.rados.0.smithi089.stdout:963: finishing write tid 4 to smithi08937929-50 2024-04-22T18:39:35.371 INFO:tasks.rados.rados.0.smithi089.stdout:963: oid 50 updating version 0 to 85 2024-04-22T18:39:35.371 INFO:tasks.rados.rados.0.smithi089.stdout:963: oid 50 version 85 is already newer than 84 2024-04-22T18:39:35.371 INFO:tasks.rados.rados.0.smithi089.stdout:963: oid 50 version 85 is already newer than 83 2024-04-22T18:39:35.371 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 85 (ObjNum 148 snap 0 seq_num 148) dirty exists 2024-04-22T18:39:35.372 INFO:tasks.rados.rados.0.smithi089.stdout:963: left oid 50 (ObjNum 148 snap 0 seq_num 148) 2024-04-22T18:39:35.372 INFO:tasks.rados.rados.0.smithi089.stdout:965: finishing write tid 1 to smithi08937929-29 2024-04-22T18:39:35.372 INFO:tasks.rados.rados.0.smithi089.stdout:965: finishing write tid 2 to smithi08937929-29 2024-04-22T18:39:35.372 INFO:tasks.rados.rados.0.smithi089.stdout:977: seq_num 153 ranges {571471=640157,1653392=524384} 2024-04-22T18:39:35.372 INFO:tasks.rados.rados.0.smithi089.stdout:977: writing smithi08937929-9 from 571471 to 1211628 tid 1 2024-04-22T18:39:35.372 INFO:tasks.rados.rados.0.smithi089.stdout:977: writing smithi08937929-9 from 1653392 to 2177776 tid 2 2024-04-22T18:39:35.372 INFO:tasks.rados.rados.0.smithi089.stdout:961: done (13 left) 2024-04-22T18:39:35.372 INFO:tasks.rados.rados.0.smithi089.stdout:963: done (12 left) 2024-04-22T18:39:35.372 INFO:tasks.rados.rados.0.smithi089.stdout:978: read oid 41 snap -1 2024-04-22T18:39:35.372 INFO:tasks.rados.rados.0.smithi089.stdout:978: expect (ObjNum 136 snap 0 seq_num 136) 2024-04-22T18:39:35.372 INFO:tasks.rados.rados.0.smithi089.stdout:979: read oid 10 snap -1 2024-04-22T18:39:35.372 INFO:tasks.rados.rados.0.smithi089.stdout:979: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:39:35.372 INFO:tasks.rados.rados.0.smithi089.stdout:980: write (excl) oid 16 current snap is 0 2024-04-22T18:39:35.372 INFO:tasks.rados.rados.0.smithi089.stdout:965: finishing write tid 3 to smithi08937929-29 2024-04-22T18:39:35.373 INFO:tasks.rados.rados.0.smithi089.stdout:965: finishing write tid 4 to smithi08937929-29 2024-04-22T18:39:35.373 INFO:tasks.rados.rados.0.smithi089.stdout:965: oid 29 updating version 0 to 73 2024-04-22T18:39:35.373 INFO:tasks.rados.rados.0.smithi089.stdout:965: oid 29 version 73 is already newer than 71 2024-04-22T18:39:35.373 INFO:tasks.rados.rados.0.smithi089.stdout:965: oid 29 version 73 is already newer than 72 2024-04-22T18:39:35.373 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 73 (ObjNum 149 snap 0 seq_num 149) dirty exists 2024-04-22T18:39:35.373 INFO:tasks.rados.rados.0.smithi089.stdout:965: left oid 29 (ObjNum 149 snap 0 seq_num 149) 2024-04-22T18:39:35.373 INFO:tasks.rados.rados.0.smithi089.stdout:967: finishing write tid 1 to smithi08937929-17 2024-04-22T18:39:35.373 INFO:tasks.rados.rados.0.smithi089.stdout:962: got (0) Success 2024-04-22T18:39:35.373 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 0 (ObjNum 112 snap 0 seq_num 112) dirty exists 2024-04-22T18:39:35.373 INFO:tasks.rados.rados.0.smithi089.stdout:970: finishing set_chunk to oid 32 2024-04-22T18:39:35.373 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 32 set_chunk returned error code -95 offset: 50176 length: 3318272 2024-04-22T18:39:35.373 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 0 (ObjNum 92 snap 0 seq_num 92) dirty exists 2024-04-22T18:39:35.373 INFO:tasks.rados.rados.0.smithi089.stdout:972: finishing write tid 1 to smithi08937929-45 2024-04-22T18:39:35.373 INFO:tasks.rados.rados.0.smithi089.stdout:980: seq_num 154 ranges {588278=715406,1711874=668042} 2024-04-22T18:39:35.374 INFO:tasks.rados.rados.0.smithi089.stdout:980: writing smithi08937929-16 from 588278 to 1303684 tid 1 2024-04-22T18:39:35.374 INFO:tasks.rados.rados.0.smithi089.stdout:980: writing smithi08937929-16 from 1711874 to 2379916 tid 2 2024-04-22T18:39:35.374 INFO:tasks.rados.rados.0.smithi089.stdout:962: done (14 left) 2024-04-22T18:39:35.374 INFO:tasks.rados.rados.0.smithi089.stdout:965: done (13 left) 2024-04-22T18:39:35.374 INFO:tasks.rados.rados.0.smithi089.stdout:970: done (12 left) 2024-04-22T18:39:35.374 INFO:tasks.rados.rados.0.smithi089.stdout:981: write (excl) oid 11 current snap is 0 2024-04-22T18:39:35.374 INFO:tasks.rados.rados.0.smithi089.stdout:971: expect (ObjNum 121 snap 0 seq_num 121) 2024-04-22T18:39:35.374 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:35.367+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:35.400 INFO:tasks.rados.rados.0.smithi089.stdout:972: finishing write tid 2 to smithi08937929-45 2024-04-22T18:39:35.400 INFO:tasks.rados.rados.0.smithi089.stdout:972: finishing write tid 3 to smithi08937929-45 2024-04-22T18:39:35.400 INFO:tasks.rados.rados.0.smithi089.stdout:981: seq_num 155 ranges {605085=790655,2170356=434729} 2024-04-22T18:39:35.408 INFO:tasks.rados.rados.0.smithi089.stdout:981: writing smithi08937929-11 from 605085 to 1395740 tid 1 2024-04-22T18:39:35.415 INFO:tasks.rados.rados.0.smithi089.stdout:981: writing smithi08937929-11 from 2170356 to 2605085 tid 2 2024-04-22T18:39:35.415 INFO:tasks.rados.rados.0.smithi089.stdout:971: done (12 left) 2024-04-22T18:39:35.415 INFO:tasks.rados.rados.0.smithi089.stdout:982: read oid 6 snap -1 2024-04-22T18:39:35.416 INFO:tasks.rados.rados.0.smithi089.stdout:982: expect (ObjNum 113 snap 0 seq_num 113) 2024-04-22T18:39:35.416 INFO:tasks.rados.rados.0.smithi089.stdout:983: read oid 26 snap -1 2024-04-22T18:39:35.416 INFO:tasks.rados.rados.0.smithi089.stdout:983: expect (ObjNum 140 snap 0 seq_num 140) 2024-04-22T18:39:35.416 INFO:tasks.rados.rados.0.smithi089.stdout:984: read oid 12 snap -1 2024-04-22T18:39:35.416 INFO:tasks.rados.rados.0.smithi089.stdout:984: expect (ObjNum 141 snap 0 seq_num 141) 2024-04-22T18:39:35.416 INFO:tasks.rados.rados.0.smithi089.stdout:985: write (excl) oid 2 current snap is 0 2024-04-22T18:39:35.416 INFO:tasks.rados.rados.0.smithi089.stdout:972: finishing write tid 4 to smithi08937929-45 2024-04-22T18:39:35.416 INFO:tasks.rados.rados.0.smithi089.stdout:972: oid 45 updating version 0 to 50 2024-04-22T18:39:35.416 INFO:tasks.rados.rados.0.smithi089.stdout:972: oid 45 version 50 is already newer than 49 2024-04-22T18:39:35.416 INFO:tasks.rados.rados.0.smithi089.stdout:972: oid 45 updating version 50 to 51 2024-04-22T18:39:35.416 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 51 (ObjNum 152 snap 0 seq_num 152) dirty exists 2024-04-22T18:39:35.416 INFO:tasks.rados.rados.0.smithi089.stdout:972: left oid 45 (ObjNum 152 snap 0 seq_num 152) 2024-04-22T18:39:35.416 INFO:tasks.rados.rados.0.smithi089.stdout:966: got (0) Success 2024-04-22T18:39:35.416 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 60 (ObjNum 134 snap 0 seq_num 134) dirty exists 2024-04-22T18:39:35.417 INFO:tasks.rados.rados.0.smithi089.stdout:967: finishing write tid 2 to smithi08937929-17 2024-04-22T18:39:35.417 INFO:tasks.rados.rados.0.smithi089.stdout:967: finishing write tid 3 to smithi08937929-17 2024-04-22T18:39:35.417 INFO:tasks.rados.rados.0.smithi089.stdout:967: finishing write tid 4 to smithi08937929-17 2024-04-22T18:39:35.417 INFO:tasks.rados.rados.0.smithi089.stdout:967: oid 17 updating version 0 to 48 2024-04-22T18:39:35.417 INFO:tasks.rados.rados.0.smithi089.stdout:967: oid 17 version 48 is already newer than 47 2024-04-22T18:39:35.417 INFO:tasks.rados.rados.0.smithi089.stdout:967: oid 17 version 48 is already newer than 46 2024-04-22T18:39:35.417 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 48 (ObjNum 150 snap 0 seq_num 150) dirty exists 2024-04-22T18:39:35.417 INFO:tasks.rados.rados.0.smithi089.stdout:967: left oid 17 (ObjNum 150 snap 0 seq_num 150) 2024-04-22T18:39:35.417 INFO:tasks.rados.rados.0.smithi089.stdout:973: finishing set_chunk to oid 18 2024-04-22T18:39:35.417 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 18 set_chunk returned error code -95 offset: 1472000 length: 70144 2024-04-22T18:39:35.417 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-22T18:39:35.417 INFO:tasks.rados.rados.0.smithi089.stdout:985: seq_num 156 ranges {621892=465904,1712485=671711} 2024-04-22T18:39:35.420 INFO:tasks.rados.rados.0.smithi089.stdout:985: writing smithi08937929-2 from 621892 to 1087796 tid 1 2024-04-22T18:39:35.427 INFO:tasks.rados.rados.0.smithi089.stdout:985: writing smithi08937929-2 from 1712485 to 2384196 tid 2 2024-04-22T18:39:35.427 INFO:tasks.rados.rados.0.smithi089.stdout:966: done (15 left) 2024-04-22T18:39:35.427 INFO:tasks.rados.rados.0.smithi089.stdout:967: done (14 left) 2024-04-22T18:39:35.427 INFO:tasks.rados.rados.0.smithi089.stdout:972: done (13 left) 2024-04-22T18:39:35.428 INFO:tasks.rados.rados.0.smithi089.stdout:973: done (12 left) 2024-04-22T18:39:35.428 INFO:tasks.rados.rados.0.smithi089.stdout:986: write (excl) oid 32 current snap is 0 2024-04-22T18:39:35.428 INFO:tasks.rados.rados.0.smithi089.stdout:986: seq_num 157 ranges {638699=541153,1770967=531722} 2024-04-22T18:39:35.433 INFO:tasks.rados.rados.0.smithi089.stdout:986: writing smithi08937929-32 from 638699 to 1179852 tid 1 2024-04-22T18:39:35.439 INFO:tasks.rados.rados.0.smithi089.stdout:986: writing smithi08937929-32 from 1770967 to 2302689 tid 2 2024-04-22T18:39:35.439 INFO:tasks.rados.rados.0.smithi089.stdout:987: write oid 23 current snap is 0 2024-04-22T18:39:35.440 INFO:tasks.rados.rados.0.smithi089.stdout:974: expect (ObjNum 13 snap 0 seq_num 13) 2024-04-22T18:39:35.466 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:35.467+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:35.511 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:35.511 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:35.511 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:35.511 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:35.514 INFO:tasks.rados.rados.0.smithi089.stdout:976: expect (ObjNum 142 snap 0 seq_num 142) 2024-04-22T18:39:35.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:35.567 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:35.568+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:35.623 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:35.623 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:35.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:35.657 INFO:tasks.rados.rados.0.smithi089.stdout:979: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:39:35.667 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:35.668+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:35.725 INFO:tasks.rados.rados.0.smithi089.stdout:978: expect (ObjNum 136 snap 0 seq_num 136) 2024-04-22T18:39:35.736 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:35.736 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:35.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:35.736 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:35.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.736 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:35.736 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:35.736 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.994090577000001, 2024-04-22T18:39:35.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30520528099999999, 2024-04-22T18:39:35.737 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.737 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.737 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.737 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.737 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.737 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 35 2024-04-22T18:39:35.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.737 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:35.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:35.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741852+0000", 2024-04-22T18:39:35.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:35.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741927+0000", 2024-04-22T18:39:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5123000000000006e-05 2024-04-22T18:39:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741953+0000", 2024-04-22T18:39:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5378999999999999e-05 2024-04-22T18:39:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741977+0000", 2024-04-22T18:39:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4431999999999999e-05 2024-04-22T18:39:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844243+0000", 2024-04-22T18:39:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102265525 2024-04-22T18:39:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844330+0000", 2024-04-22T18:39:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7417999999999995e-05 2024-04-22T18:39:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844470+0000", 2024-04-22T18:39:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000140197 2024-04-22T18:39:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944767+0000", 2024-04-22T18:39:35.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100296409 2024-04-22T18:39:35.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.046968+0000", 2024-04-22T18:39:35.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10220148499999999 2024-04-22T18:39:35.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047018+0000", 2024-04-22T18:39:35.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9206000000000003e-05 2024-04-22T18:39:35.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047059+0000", 2024-04-22T18:39:35.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1496999999999999e-05 2024-04-22T18:39:35.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.742 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.742 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:35.742 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:35.742 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.993960053000002, 2024-04-22T18:39:35.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:39:35.742 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:39:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:39:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:39:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:39:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:39:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:39:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:39:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:39:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:39:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:39:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:39:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:39:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:39:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:39:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:39:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:39:35.747 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.993920361999997, 2024-04-22T18:39:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:39:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.748 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:39:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:39:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:39:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:39:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:39:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:39:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:39:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:39:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:39:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:39:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:39:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:39:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:39:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:39:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:39:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:39:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:39:35.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:39:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:39:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:39:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:39:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:39:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.749739314999999, 2024-04-22T18:39:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402419109, 2024-04-22T18:39:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 68 2024-04-22T18:39:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986204+0000", 2024-04-22T18:39:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986208+0000", 2024-04-22T18:39:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8149999999999999e-06 2024-04-22T18:39:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986214+0000", 2024-04-22T18:39:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.604e-06 2024-04-22T18:39:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986218+0000", 2024-04-22T18:39:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6689999999999999e-06 2024-04-22T18:39:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986833+0000", 2024-04-22T18:39:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000615648 2024-04-22T18:39:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986891+0000", 2024-04-22T18:39:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7915999999999999e-05 2024-04-22T18:39:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986981+0000", 2024-04-22T18:39:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759999999999994e-05 2024-04-22T18:39:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.987584+0000", 2024-04-22T18:39:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060270199999999999 2024-04-22T18:39:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388527+0000", 2024-04-22T18:39:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40094366999999997 2024-04-22T18:39:35.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388583+0000", 2024-04-22T18:39:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5383000000000002e-05 2024-04-22T18:39:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388624+0000", 2024-04-22T18:39:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501999999999997e-05 2024-04-22T18:39:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.749723525999997, 2024-04-22T18:39:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:39:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:39:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:39:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:39:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:39:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:39:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:39:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:39:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:39:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:39:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:39:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:39:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:39:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:39:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:39:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:39:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:39:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:39:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:39:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:39:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:39:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:39:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:39:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.765 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 52.633330516999997, 2024-04-22T18:39:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305563733, 2024-04-22T18:39:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 87 2024-04-22T18:39:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102613+0000", 2024-04-22T18:39:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102668+0000", 2024-04-22T18:39:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4669000000000001e-05 2024-04-22T18:39:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102680+0000", 2024-04-22T18:39:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2026e-05 2024-04-22T18:39:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102687+0000", 2024-04-22T18:39:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8339999999999999e-06 2024-04-22T18:39:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405644+0000", 2024-04-22T18:39:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30295623199999999 2024-04-22T18:39:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405743+0000", 2024-04-22T18:39:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.98e-05 2024-04-22T18:39:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405868+0000", 2024-04-22T18:39:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012457899999999999 2024-04-22T18:39:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407047+0000", 2024-04-22T18:39:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011791449999999999 2024-04-22T18:39:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408088+0000", 2024-04-22T18:39:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010411680000000001 2024-04-22T18:39:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408135+0000", 2024-04-22T18:39:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6255999999999999e-05 2024-04-22T18:39:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408178+0000", 2024-04-22T18:39:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2877000000000003e-05 2024-04-22T18:39:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 52.633250588000003, 2024-04-22T18:39:35.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30560089000000001, 2024-04-22T18:39:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 88 2024-04-22T18:39:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102693+0000", 2024-04-22T18:39:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102698+0000", 2024-04-22T18:39:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9699999999999998e-06 2024-04-22T18:39:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102708+0000", 2024-04-22T18:39:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.006e-05 2024-04-22T18:39:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102715+0000", 2024-04-22T18:39:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6599999999999995e-06 2024-04-22T18:39:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406637+0000", 2024-04-22T18:39:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392130099999998 2024-04-22T18:39:35.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406696+0000", 2024-04-22T18:39:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9066e-05 2024-04-22T18:39:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406784+0000", 2024-04-22T18:39:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8194000000000003e-05 2024-04-22T18:39:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407447+0000", 2024-04-22T18:39:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066295700000000004 2024-04-22T18:39:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408237+0000", 2024-04-22T18:39:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079032600000000005 2024-04-22T18:39:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408270+0000", 2024-04-22T18:39:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3157000000000001e-05 2024-04-22T18:39:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408295+0000", 2024-04-22T18:39:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4151999999999998e-05 2024-04-22T18:39:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.882018463999998, 2024-04-22T18:39:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30533819600000001, 2024-04-22T18:39:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.777 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 106 2024-04-22T18:39:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853925+0000", 2024-04-22T18:39:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853958+0000", 2024-04-22T18:39:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2536000000000002e-05 2024-04-22T18:39:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853964+0000", 2024-04-22T18:39:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6239999999999999e-06 2024-04-22T18:39:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853967+0000", 2024-04-22T18:39:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9700000000000001e-06 2024-04-22T18:39:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855144+0000", 2024-04-22T18:39:35.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001176774 2024-04-22T18:39:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855225+0000", 2024-04-22T18:39:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0697000000000003e-05 2024-04-22T18:39:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855353+0000", 2024-04-22T18:39:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012788 2024-04-22T18:39:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058003+0000", 2024-04-22T18:39:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202649939 2024-04-22T18:39:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159184+0000", 2024-04-22T18:39:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101181058 2024-04-22T18:39:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159238+0000", 2024-04-22T18:39:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3967e-05 2024-04-22T18:39:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159264+0000", 2024-04-22T18:39:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6237999999999998e-05 2024-04-22T18:39:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.881971614000001, 2024-04-22T18:39:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30544294500000002, 2024-04-22T18:39:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.783 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 107 2024-04-22T18:39:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853972+0000", 2024-04-22T18:39:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853975+0000", 2024-04-22T18:39:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0939999999999999e-06 2024-04-22T18:39:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853978+0000", 2024-04-22T18:39:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:39:35.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853981+0000", 2024-04-22T18:39:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.858e-06 2024-04-22T18:39:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855831+0000", 2024-04-22T18:39:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018503689999999999 2024-04-22T18:39:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855932+0000", 2024-04-22T18:39:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010071 2024-04-22T18:39:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856029+0000", 2024-04-22T18:39:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6751999999999998e-05 2024-04-22T18:39:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058013+0000", 2024-04-22T18:39:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20198412600000001 2024-04-22T18:39:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159326+0000", 2024-04-22T18:39:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101312969 2024-04-22T18:39:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159376+0000", 2024-04-22T18:39:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9968e-05 2024-04-22T18:39:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159416+0000", 2024-04-22T18:39:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9641999999999998e-05 2024-04-22T18:39:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.881883217999999, 2024-04-22T18:39:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30548257499999998, 2024-04-22T18:39:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.789 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 108 2024-04-22T18:39:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854064+0000", 2024-04-22T18:39:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5820000000000002e-06 2024-04-22T18:39:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854080+0000", 2024-04-22T18:39:35.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5379999999999998e-05 2024-04-22T18:39:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854084+0000", 2024-04-22T18:39:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.963e-06 2024-04-22T18:39:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856413+0000", 2024-04-22T18:39:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023298070000000001 2024-04-22T18:39:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856494+0000", 2024-04-22T18:39:35.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0863000000000002e-05 2024-04-22T18:39:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856624+0000", 2024-04-22T18:39:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129799 2024-04-22T18:39:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058947+0000", 2024-04-22T18:39:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20232328899999999 2024-04-22T18:39:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159468+0000", 2024-04-22T18:39:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100520597 2024-04-22T18:39:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159508+0000", 2024-04-22T18:39:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9830000000000003e-05 2024-04-22T18:39:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159544+0000", 2024-04-22T18:39:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5944e-05 2024-04-22T18:39:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.881838725999998, 2024-04-22T18:39:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30556502499999999, 2024-04-22T18:39:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.795 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:39:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854105+0000", 2024-04-22T18:39:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854175+0000", 2024-04-22T18:39:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0178000000000005e-05 2024-04-22T18:39:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854191+0000", 2024-04-22T18:39:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5965999999999999e-05 2024-04-22T18:39:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854196+0000", 2024-04-22T18:39:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.42e-06 2024-04-22T18:39:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956526+0000", 2024-04-22T18:39:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10233094299999999 2024-04-22T18:39:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956634+0000", 2024-04-22T18:39:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107398 2024-04-22T18:39:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956730+0000", 2024-04-22T18:39:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6217000000000004e-05 2024-04-22T18:39:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058955+0000", 2024-04-22T18:39:35.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102224749 2024-04-22T18:39:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159596+0000", 2024-04-22T18:39:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100641297 2024-04-22T18:39:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159637+0000", 2024-04-22T18:39:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874000000000002e-05 2024-04-22T18:39:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159671+0000", 2024-04-22T18:39:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3603999999999999e-05 2024-04-22T18:39:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.881744879000003, 2024-04-22T18:39:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30683048800000001, 2024-04-22T18:39:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.801 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110 2024-04-22T18:39:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854202+0000", 2024-04-22T18:39:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:39:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854206+0000", 2024-04-22T18:39:35.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5870000000000001e-06 2024-04-22T18:39:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854208+0000", 2024-04-22T18:39:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7199999999999998e-06 2024-04-22T18:39:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057285+0000", 2024-04-22T18:39:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20307620800000001 2024-04-22T18:39:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057414+0000", 2024-04-22T18:39:35.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012896899999999999 2024-04-22T18:39:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057588+0000", 2024-04-22T18:39:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000174802 2024-04-22T18:39:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159429+0000", 2024-04-22T18:39:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101840783 2024-04-22T18:39:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160947+0000", 2024-04-22T18:39:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001518152 2024-04-22T18:39:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160992+0000", 2024-04-22T18:39:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4725000000000003e-05 2024-04-22T18:39:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161030+0000", 2024-04-22T18:39:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7789e-05 2024-04-22T18:39:35.806 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.881733697999998, 2024-04-22T18:39:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:39:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.807 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:39:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:39:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:39:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:39:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:39:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:39:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:39:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:39:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:39:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:39:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:39:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:39:35.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:39:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:39:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:39:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:39:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:39:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:39:35.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:39:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:39:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:39:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:39:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.881718040000003, 2024-04-22T18:39:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:39:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:39:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.813 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:39:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:39:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:39:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:39:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:39:35.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:39:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:39:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:39:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:39:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:39:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:39:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:39:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:39:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:39:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:39:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:39:35.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:39:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:39:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:39:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:39:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.819 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:35.819 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:35.819 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.881666877999997, 2024-04-22T18:39:35.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:39:35.819 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.819 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.819 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.819 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.819 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.819 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:39:35.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.819 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:35.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:35.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:35.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:39:35.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:39:35.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:39:35.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:39:35.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:39:35.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:39:35.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:39:35.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:39:35.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:39:35.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:39:35.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:39:35.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:39:35.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:39:35.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:39:35.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:39:35.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:39:35.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:39:35.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:39:35.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:39:35.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:39:35.824 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.825 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.825 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.825 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:35.825 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:35.825 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.881651654000002, 2024-04-22T18:39:35.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:39:35.825 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.825 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.825 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.825 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.825 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.826 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:39:35.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.826 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:35.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:35.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:39:35.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:39:35.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:39:35.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:39:35.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:39:35.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:39:35.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:39:35.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:39:35.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:39:35.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:39:35.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:39:35.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:39:35.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:39:35.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:39:35.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:39:35.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:39:35.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:39:35.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:39:35.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:39:35.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:39:35.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:39:35.830 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.830 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.830 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.831 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:35.831 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:35.831 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.881592077000001, 2024-04-22T18:39:35.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:39:35.831 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.831 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.831 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.831 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.831 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.831 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:39:35.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.831 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:35.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:35.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:35.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:39:35.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:39:35.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:39:35.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:39:35.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:39:35.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:39:35.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:39:35.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:39:35.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:39:35.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:39:35.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:39:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:39:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:39:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:39:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.881509760999997, 2024-04-22T18:39:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:39:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:39:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:39:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:39:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:39:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:39:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:39:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:39:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:39:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:39:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:39:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:39:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:39:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:39:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:39:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:39:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:39:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:39:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:39:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:39:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:39:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:39:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:39:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.881499605000002, 2024-04-22T18:39:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:39:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:39:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:39:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:39:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:39:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:39:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:39:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:39:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:39:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:39:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:39:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:39:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:39:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:39:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:161 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:39:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.281005313999998, 2024-04-22T18:39:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40239169000000002, 2024-04-22T18:39:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 161 2024-04-22T18:39:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454938+0000", 2024-04-22T18:39:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454982+0000", 2024-04-22T18:39:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3470999999999998e-05 2024-04-22T18:39:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454988+0000", 2024-04-22T18:39:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7640000000000002e-06 2024-04-22T18:39:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555054+0000", 2024-04-22T18:39:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006607300000001 2024-04-22T18:39:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555607+0000", 2024-04-22T18:39:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055319500000000003 2024-04-22T18:39:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555689+0000", 2024-04-22T18:39:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1817999999999995e-05 2024-04-22T18:39:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555804+0000", 2024-04-22T18:39:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115346 2024-04-22T18:39:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556554+0000", 2024-04-22T18:39:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000750102 2024-04-22T18:39:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857265+0000", 2024-04-22T18:39:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30071122700000003 2024-04-22T18:39:35.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:35.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857302+0000", 2024-04-22T18:39:35.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7069e-05 2024-04-22T18:39:35.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:35.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:35.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:35.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857331+0000", 2024-04-22T18:39:35.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8291999999999999e-05 2024-04-22T18:39:35.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.853 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.853 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:35.853 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:35.854 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:35.856 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:35.856+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:35.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:35.857 INFO:tasks.rados.rados.0.smithi089.stdout:977: finishing write tid 1 to smithi08937929-9 2024-04-22T18:39:35.857 INFO:tasks.rados.rados.0.smithi089.stdout:977: finishing write tid 2 to smithi08937929-9 2024-04-22T18:39:35.857 INFO:tasks.rados.rados.0.smithi089.stdout:977: finishing write tid 3 to smithi08937929-9 2024-04-22T18:39:35.857 INFO:tasks.rados.rados.0.smithi089.stdout:975: got (0) Success 2024-04-22T18:39:35.857 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 19 (ObjNum 58 snap 0 seq_num 58) dirty exists 2024-04-22T18:39:35.857 INFO:tasks.rados.rados.0.smithi089.stdout:987: seq_num 158 ranges {655506=616402,1829449=675380} 2024-04-22T18:39:35.857 INFO:tasks.rados.rados.0.smithi089.stdout:987: writing smithi08937929-23 from 655506 to 1271908 tid 1 2024-04-22T18:39:35.857 INFO:tasks.rados.rados.0.smithi089.stdout:987: writing smithi08937929-23 from 1829449 to 2504829 tid 2 2024-04-22T18:39:35.857 INFO:tasks.rados.rados.0.smithi089.stdout:974: done (13 left) 2024-04-22T18:39:35.857 INFO:tasks.rados.rados.0.smithi089.stdout:975: done (12 left) 2024-04-22T18:39:35.857 INFO:tasks.rados.rados.0.smithi089.stdout:976: done (11 left) 2024-04-22T18:39:35.857 INFO:tasks.rados.rados.0.smithi089.stdout:978: done (10 left) 2024-04-22T18:39:35.857 INFO:tasks.rados.rados.0.smithi089.stdout:979: done (9 left) 2024-04-22T18:39:35.858 INFO:tasks.rados.rados.0.smithi089.stdout:988: read oid 4 snap -1 2024-04-22T18:39:35.858 INFO:tasks.rados.rados.0.smithi089.stdout:988: expect (ObjNum 120 snap 0 seq_num 120) 2024-04-22T18:39:35.858 INFO:tasks.rados.rados.0.smithi089.stdout:989: write (excl) oid 19 current snap is 0 2024-04-22T18:39:35.858 INFO:tasks.rados.rados.0.smithi089.stdout:983: expect (ObjNum 140 snap 0 seq_num 140) 2024-04-22T18:39:35.956 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:35.957+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:35.959 INFO:tasks.rados.rados.0.smithi089.stdout:984: expect (ObjNum 141 snap 0 seq_num 141) 2024-04-22T18:39:35.962 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:35.962 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:35.962 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:35.963 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:35.971 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:36.057 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:36.058+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:36.072 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:36.073 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:36.081 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:36.096 INFO:tasks.rados.rados.0.smithi089.stdout:980: finishing write tid 1 to smithi08937929-16 2024-04-22T18:39:36.096 INFO:tasks.rados.rados.0.smithi089.stdout:980: finishing write tid 2 to smithi08937929-16 2024-04-22T18:39:36.097 INFO:tasks.rados.rados.0.smithi089.stdout:977: finishing write tid 4 to smithi08937929-9 2024-04-22T18:39:36.097 INFO:tasks.rados.rados.0.smithi089.stdout:977: oid 9 updating version 0 to 32 2024-04-22T18:39:36.097 INFO:tasks.rados.rados.0.smithi089.stdout:977: oid 9 version 32 is already newer than 31 2024-04-22T18:39:36.097 INFO:tasks.rados.rados.0.smithi089.stdout:977: oid 9 updating version 32 to 33 2024-04-22T18:39:36.097 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 33 (ObjNum 153 snap 0 seq_num 153) dirty exists 2024-04-22T18:39:36.097 INFO:tasks.rados.rados.0.smithi089.stdout:977: left oid 9 (ObjNum 153 snap 0 seq_num 153) 2024-04-22T18:39:36.097 INFO:tasks.rados.rados.0.smithi089.stdout:980: finishing write tid 3 to smithi08937929-16 2024-04-22T18:39:36.097 INFO:tasks.rados.rados.0.smithi089.stdout:989: seq_num 159 ranges {672313=691651,1887931=419038} 2024-04-22T18:39:36.103 INFO:tasks.rados.rados.0.smithi089.stdout:989: writing smithi08937929-19 from 672313 to 1363964 tid 1 2024-04-22T18:39:36.109 INFO:tasks.rados.rados.0.smithi089.stdout:989: writing smithi08937929-19 from 1887931 to 2306969 tid 2 2024-04-22T18:39:36.109 INFO:tasks.rados.rados.0.smithi089.stdout:977: done (10 left) 2024-04-22T18:39:36.109 INFO:tasks.rados.rados.0.smithi089.stdout:983: done (9 left) 2024-04-22T18:39:36.109 INFO:tasks.rados.rados.0.smithi089.stdout:984: done (8 left) 2024-04-22T18:39:36.109 INFO:tasks.rados.rados.0.smithi089.stdout:990: write oid 27 current snap is 0 2024-04-22T18:39:36.109 INFO:tasks.rados.rados.0.smithi089.stdout:982: expect (ObjNum 113 snap 0 seq_num 113) 2024-04-22T18:39:36.157 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:36.158+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:36.201 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:61 4.b 4:d923883e:::smithi08937929-16:head [write 652105~445088,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.930904492000003, 2024-04-22T18:39:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.38578178600000002, 2024-04-22T18:39:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 61 2024-04-22T18:39:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270192+0000", 2024-04-22T18:39:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270401+0000", 2024-04-22T18:39:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020921199999999999 2024-04-22T18:39:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270415+0000", 2024-04-22T18:39:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4191e-05 2024-04-22T18:39:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270422+0000", 2024-04-22T18:39:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7329999999999997e-06 2024-04-22T18:39:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270461+0000", 2024-04-22T18:39:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9530999999999998e-05 2024-04-22T18:39:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270550+0000", 2024-04-22T18:39:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8508e-05 2024-04-22T18:39:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270648+0000", 2024-04-22T18:39:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8377999999999999e-05 2024-04-22T18:39:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.271907+0000", 2024-04-22T18:39:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012589599999999999 2024-04-22T18:39:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655904+0000", 2024-04-22T18:39:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.383997115 2024-04-22T18:39:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655933+0000", 2024-04-22T18:39:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9011999999999999e-05 2024-04-22T18:39:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655977+0000", 2024-04-22T18:39:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3578000000000002e-05 2024-04-22T18:39:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:62 4.b 4:d923883e:::smithi08937929-16:head [write 1728171~452988] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:39:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.924945844, 2024-04-22T18:39:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.379937407, 2024-04-22T18:39:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62 2024-04-22T18:39:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:39:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:39:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276151+0000", 2024-04-22T18:39:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276370+0000", 2024-04-22T18:39:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000218951 2024-04-22T18:39:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276382+0000", 2024-04-22T18:39:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1517e-05 2024-04-22T18:39:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276389+0000", 2024-04-22T18:39:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8939999999999993e-06 2024-04-22T18:39:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276430+0000", 2024-04-22T18:39:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0958999999999999e-05 2024-04-22T18:39:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276485+0000", 2024-04-22T18:39:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4208e-05 2024-04-22T18:39:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276567+0000", 2024-04-22T18:39:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2006000000000001e-05 2024-04-22T18:39:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277685+0000", 2024-04-22T18:39:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001118237 2024-04-22T18:39:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656024+0000", 2024-04-22T18:39:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.378339588 2024-04-22T18:39:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656054+0000", 2024-04-22T18:39:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9373000000000002e-05 2024-04-22T18:39:36.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656091+0000", 2024-04-22T18:39:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7168000000000001e-05 2024-04-22T18:39:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:63 4.b 4:d923883e:::smithi08937929-16:head [setxattr _header (56) in=7b,truncate 2252105] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:39:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.924703588, 2024-04-22T18:39:36.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37982964200000002, 2024-04-22T18:39:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 63 2024-04-22T18:39:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:39:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:39:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276395+0000", 2024-04-22T18:39:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276400+0000", 2024-04-22T18:39:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6090000000000003e-06 2024-04-22T18:39:36.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276407+0000", 2024-04-22T18:39:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5850000000000002e-06 2024-04-22T18:39:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276411+0000", 2024-04-22T18:39:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2569999999999999e-06 2024-04-22T18:39:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277254+0000", 2024-04-22T18:39:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084384300000000005 2024-04-22T18:39:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277312+0000", 2024-04-22T18:39:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7395999999999997e-05 2024-04-22T18:39:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277388+0000", 2024-04-22T18:39:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6604e-05 2024-04-22T18:39:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277870+0000", 2024-04-22T18:39:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000481545 2024-04-22T18:39:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656138+0000", 2024-04-22T18:39:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37826752499999999 2024-04-22T18:39:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656177+0000", 2024-04-22T18:39:36.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9419999999999999e-05 2024-04-22T18:39:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656226+0000", 2024-04-22T18:39:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8562e-05 2024-04-22T18:39:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:64 4.b 4:d923883e:::smithi08937929-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.924672966999999, 2024-04-22T18:39:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37999559999999999, 2024-04-22T18:39:36.219 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 64 2024-04-22T18:39:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276429+0000", 2024-04-22T18:39:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7329999999999999e-06 2024-04-22T18:39:36.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276435+0000", 2024-04-22T18:39:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.02e-06 2024-04-22T18:39:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276439+0000", 2024-04-22T18:39:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7659999999999998e-06 2024-04-22T18:39:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277530+0000", 2024-04-22T18:39:36.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010909470000000001 2024-04-22T18:39:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277603+0000", 2024-04-22T18:39:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2638999999999994e-05 2024-04-22T18:39:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656259+0000", 2024-04-22T18:39:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37865579300000002 2024-04-22T18:39:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656330+0000", 2024-04-22T18:39:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0899e-05 2024-04-22T18:39:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656422+0000", 2024-04-22T18:39:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2397000000000003e-05 2024-04-22T18:39:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:36.224 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.745742890999999, 2024-04-22T18:39:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:39:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:39:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:39:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:39:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:39:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:39:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:39:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:39:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:39:36.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:39:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:39:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:39:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:39:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:39:36.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:39:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:39:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:39:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:39:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:39:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:39:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:39:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:39:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:39:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:315 4.b 4:d923883e:::smithi08937929-16:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:36.230 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.209280110999998, 2024-04-22T18:39:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40309447300000001, 2024-04-22T18:39:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 315 2024-04-22T18:39:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:36.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991817+0000", 2024-04-22T18:39:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991825+0000", 2024-04-22T18:39:36.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7279999999999995e-06 2024-04-22T18:39:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991834+0000", 2024-04-22T18:39:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1980000000000007e-06 2024-04-22T18:39:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991841+0000", 2024-04-22T18:39:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0740000000000004e-06 2024-04-22T18:39:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991881+0000", 2024-04-22T18:39:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-04-22T18:39:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991940+0000", 2024-04-22T18:39:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9382000000000002e-05 2024-04-22T18:39:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394250+0000", 2024-04-22T18:39:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40230947700000003 2024-04-22T18:39:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394752+0000", 2024-04-22T18:39:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050269699999999997 2024-04-22T18:39:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394865+0000", 2024-04-22T18:39:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011267700000000001 2024-04-22T18:39:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394892+0000", 2024-04-22T18:39:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7535999999999999e-05 2024-04-22T18:39:36.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394914+0000", 2024-04-22T18:39:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1398000000000001e-05 2024-04-22T18:39:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:393 4.b 4:defe92fb:::smithi08937929-44:head [assert-version v12,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:39:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.099903929, 2024-04-22T18:39:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33050822299999999, 2024-04-22T18:39:36.236 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 393 2024-04-22T18:39:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:39:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101196+0000", 2024-04-22T18:39:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101194+0000", 2024-04-22T18:39:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101201+0000", 2024-04-22T18:39:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1269999999999997e-06 2024-04-22T18:39:36.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101211+0000", 2024-04-22T18:39:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9380000000000004e-06 2024-04-22T18:39:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101220+0000", 2024-04-22T18:39:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3459999999999994e-06 2024-04-22T18:39:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101324+0000", 2024-04-22T18:39:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104366 2024-04-22T18:39:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.101379+0000", 2024-04-22T18:39:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5127999999999998e-05 2024-04-22T18:39:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.330649+0000", 2024-04-22T18:39:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.22927022699999999 2024-04-22T18:39:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.331182+0000", 2024-04-22T18:39:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053250900000000002 2024-04-22T18:39:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.431635+0000", 2024-04-22T18:39:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10045290599999999 2024-04-22T18:39:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.431677+0000", 2024-04-22T18:39:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1816000000000001e-05 2024-04-22T18:39:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.431704+0000", 2024-04-22T18:39:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7132e-05 2024-04-22T18:39:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:774 4.b 4:defe92fb:::smithi08937929-44:head [write 2778819~95145] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:39:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.052770434999999, 2024-04-22T18:39:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40475725000000001, 2024-04-22T18:39:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.242 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 774 2024-04-22T18:39:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:39:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:39:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148328+0000", 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148392+0000", 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3323000000000003e-05 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148398+0000", 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4420000000000003e-06 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148409+0000", 2024-04-22T18:39:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509999999999999e-05 2024-04-22T18:39:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251407+0000", 2024-04-22T18:39:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102998305 2024-04-22T18:39:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251500+0000", 2024-04-22T18:39:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3607e-05 2024-04-22T18:39:36.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251648+0000", 2024-04-22T18:39:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014788099999999999 2024-04-22T18:39:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256149+0000", 2024-04-22T18:39:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045010980000000003 2024-04-22T18:39:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553016+0000", 2024-04-22T18:39:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29686620800000002 2024-04-22T18:39:36.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553054+0000", 2024-04-22T18:39:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8414999999999997e-05 2024-04-22T18:39:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553086+0000", 2024-04-22T18:39:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2236000000000001e-05 2024-04-22T18:39:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.247 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:39:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.052685642999997, 2024-04-22T18:39:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:39:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:39:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:39:36.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:39:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:39:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:39:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:39:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:39:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:39:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:39:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:39:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:39:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:39:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:39:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:39:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:39:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:39:36.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:39:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:39:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:39:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:39:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:39:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:39:36.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:39:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:39:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.052667167000003, 2024-04-22T18:39:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:39:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.253 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:39:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:36.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:39:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:39:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:39:36.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:39:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:39:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:39:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:39:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:39:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:36.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:39:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:39:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:39:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:39:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:39:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:39:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:39:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:39:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1242 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-22T18:39:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:39:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.459421618, 2024-04-22T18:39:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40662891899999998, 2024-04-22T18:39:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.258 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1242 2024-04-22T18:39:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:39:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:39:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741677+0000", 2024-04-22T18:39:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741680+0000", 2024-04-22T18:39:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8059999999999999e-06 2024-04-22T18:39:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741685+0000", 2024-04-22T18:39:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0780000000000003e-06 2024-04-22T18:39:36.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741690+0000", 2024-04-22T18:39:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4020000000000004e-06 2024-04-22T18:39:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741732+0000", 2024-04-22T18:39:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2110000000000002e-05 2024-04-22T18:39:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741798+0000", 2024-04-22T18:39:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6432000000000004e-05 2024-04-22T18:39:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.148307+0000", 2024-04-22T18:39:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40650862100000001 2024-04-22T18:39:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1407 4.6 4:630bd8e5:::smithi08937929-45:head [write 1603467~416361] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:39:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161455+0000", 2024-04-22T18:39:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.039644871, 2024-04-22T18:39:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30888835399999998, 2024-04-22T18:39:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1407 2024-04-22T18:39:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161455+0000", 2024-04-22T18:39:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161455+0000", 2024-04-22T18:39:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161454+0000", 2024-04-22T18:39:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161630+0000", 2024-04-22T18:39:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000175878 2024-04-22T18:39:36.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161635+0000", 2024-04-22T18:39:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0799999999999996e-06 2024-04-22T18:39:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161639+0000", 2024-04-22T18:39:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6950000000000002e-06 2024-04-22T18:39:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.264383+0000", 2024-04-22T18:39:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102744717 2024-04-22T18:39:36.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.264487+0000", 2024-04-22T18:39:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010405500000000001 2024-04-22T18:39:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.264631+0000", 2024-04-22T18:39:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014388000000000001 2024-04-22T18:39:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370095+0000", 2024-04-22T18:39:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105463794 2024-04-22T18:39:36.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.470245+0000", 2024-04-22T18:39:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100149475 2024-04-22T18:39:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.470299+0000", 2024-04-22T18:39:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4700999999999997e-05 2024-04-22T18:39:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.470343+0000", 2024-04-22T18:39:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3659999999999999e-05 2024-04-22T18:39:36.267 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1408 4.6 4:630bd8e5:::smithi08937929-45:head [write 2816389~693495] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:39:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:39:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.039451810999999, 2024-04-22T18:39:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408959452, 2024-04-22T18:39:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1408 2024-04-22T18:39:36.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:39:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:39:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161647+0000", 2024-04-22T18:39:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161951+0000", 2024-04-22T18:39:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030389799999999998 2024-04-22T18:39:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161955+0000", 2024-04-22T18:39:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1989999999999999e-06 2024-04-22T18:39:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161958+0000", 2024-04-22T18:39:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2260000000000001e-06 2024-04-22T18:39:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.265867+0000", 2024-04-22T18:39:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103908927 2024-04-22T18:39:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.265960+0000", 2024-04-22T18:39:36.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3016999999999994e-05 2024-04-22T18:39:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.266124+0000", 2024-04-22T18:39:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016347700000000001 2024-04-22T18:39:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570491+0000", 2024-04-22T18:39:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30436694199999997 2024-04-22T18:39:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570502+0000", 2024-04-22T18:39:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1107999999999999e-05 2024-04-22T18:39:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570556+0000", 2024-04-22T18:39:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4388000000000002e-05 2024-04-22T18:39:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570607+0000", 2024-04-22T18:39:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0782999999999998e-05 2024-04-22T18:39:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1409 4.6 4:630bd8e5:::smithi08937929-45:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:39:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:39:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.039138106999999, 2024-04-22T18:39:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40880186200000002, 2024-04-22T18:39:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1409 2024-04-22T18:39:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.274 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:39:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:39:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:39:36.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161964+0000", 2024-04-22T18:39:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5909999999999998e-06 2024-04-22T18:39:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161966+0000", 2024-04-22T18:39:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5979999999999999e-06 2024-04-22T18:39:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161969+0000", 2024-04-22T18:39:36.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7049999999999999e-06 2024-04-22T18:39:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269205+0000", 2024-04-22T18:39:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107236228 2024-04-22T18:39:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269320+0000", 2024-04-22T18:39:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114571 2024-04-22T18:39:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:36.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269476+0000", 2024-04-22T18:39:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155928 2024-04-22T18:39:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570619+0000", 2024-04-22T18:39:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30114322700000001 2024-04-22T18:39:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570672+0000", 2024-04-22T18:39:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3232000000000001e-05 2024-04-22T18:39:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570724+0000", 2024-04-22T18:39:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2258e-05 2024-04-22T18:39:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570763+0000", 2024-04-22T18:39:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8927999999999997e-05 2024-04-22T18:39:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.279 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1410 4.6 4:630bd8e5:::smithi08937929-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:39:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:39:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.039128590000001, 2024-04-22T18:39:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40900504500000001, 2024-04-22T18:39:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1410 2024-04-22T18:39:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:39:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:39:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:39:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161972+0000", 2024-04-22T18:39:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6419999999999999e-06 2024-04-22T18:39:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161976+0000", 2024-04-22T18:39:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9260000000000002e-06 2024-04-22T18:39:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161979+0000", 2024-04-22T18:39:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9359999999999999e-06 2024-04-22T18:39:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570790+0000", 2024-04-22T18:39:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40881099199999998 2024-04-22T18:39:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570879+0000", 2024-04-22T18:39:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8895999999999997e-05 2024-04-22T18:39:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570976+0000", 2024-04-22T18:39:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6923999999999996e-05 2024-04-22T18:39:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.283 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:39:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:39:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.2896188549999996, 2024-04-22T18:39:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:39:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.284 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:39:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:39:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:39:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:39:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:39:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:39:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:39:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:39:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:39:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:39:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:39:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:39:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:39:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:39:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:39:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:39:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:39:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:39:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:39:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:39:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:39:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:39:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:39:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:39:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1868 4.b 4:d923883e:::smithi08937929-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:36.289 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:35.309492+0000", 2024-04-22T18:39:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.89160739499999997, 2024-04-22T18:39:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.36833897100000002, 2024-04-22T18:39:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1868 2024-04-22T18:39:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.309492+0000", 2024-04-22T18:39:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.309492+0000", 2024-04-22T18:39:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.309491+0000", 2024-04-22T18:39:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.309494+0000", 2024-04-22T18:39:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4229999999999999e-06 2024-04-22T18:39:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.309498+0000", 2024-04-22T18:39:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9920000000000001e-06 2024-04-22T18:39:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.309501+0000", 2024-04-22T18:39:36.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8760000000000001e-06 2024-04-22T18:39:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.409603+0000", 2024-04-22T18:39:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100101755 2024-04-22T18:39:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.409669+0000", 2024-04-22T18:39:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6773999999999995e-05 2024-04-22T18:39:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.677698+0000", 2024-04-22T18:39:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.26802832399999998 2024-04-22T18:39:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.677752+0000", 2024-04-22T18:39:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.414e-05 2024-04-22T18:39:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.677831+0000", 2024-04-22T18:39:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9268000000000001e-05 2024-04-22T18:39:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.294 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1887 4.6 4:6700eab5:::smithi08937929-32:head [write 1770967~531722] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:35.441825+0000", 2024-04-22T18:39:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.75927427000000003, 2024-04-22T18:39:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33655051800000002, 2024-04-22T18:39:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1887 2024-04-22T18:39:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.295 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.441825+0000", 2024-04-22T18:39:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.441825+0000", 2024-04-22T18:39:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.441824+0000", 2024-04-22T18:39:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:36.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.441925+0000", 2024-04-22T18:39:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0001014 2024-04-22T18:39:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.441937+0000", 2024-04-22T18:39:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2169e-05 2024-04-22T18:39:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542039+0000", 2024-04-22T18:39:36.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010185000000001 2024-04-22T18:39:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542113+0000", 2024-04-22T18:39:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4362000000000004e-05 2024-04-22T18:39:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542221+0000", 2024-04-22T18:39:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107395 2024-04-22T18:39:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:36.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542403+0000", 2024-04-22T18:39:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018199500000000001 2024-04-22T18:39:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.644759+0000", 2024-04-22T18:39:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102356458 2024-04-22T18:39:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.778120+0000", 2024-04-22T18:39:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13336097799999999 2024-04-22T18:39:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.778303+0000", 2024-04-22T18:39:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000182626 2024-04-22T18:39:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.778376+0000", 2024-04-22T18:39:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2908000000000001e-05 2024-04-22T18:39:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1888 4.6 4:6700eab5:::smithi08937929-32:head [setxattr _header (56) in=7b,truncate 2638699] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:36.300 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:35.542072+0000", 2024-04-22T18:39:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.65902781700000002, 2024-04-22T18:39:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33653544499999999, 2024-04-22T18:39:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1888 2024-04-22T18:39:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542072+0000", 2024-04-22T18:39:36.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542072+0000", 2024-04-22T18:39:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542069+0000", 2024-04-22T18:39:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542081+0000", 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2300000000000001e-05 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542090+0000", 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1300000000000001e-06 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542095+0000", 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1909999999999999e-06 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.644137+0000", 2024-04-22T18:39:36.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102041833 2024-04-22T18:39:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.644198+0000", 2024-04-22T18:39:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1118999999999995e-05 2024-04-22T18:39:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.644287+0000", 2024-04-22T18:39:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9728000000000006e-05 2024-04-22T18:39:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.645071+0000", 2024-04-22T18:39:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00078322999999999999 2024-04-22T18:39:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.878513+0000", 2024-04-22T18:39:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.23344224299999999 2024-04-22T18:39:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.878551+0000", 2024-04-22T18:39:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7993e-05 2024-04-22T18:39:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.878607+0000", 2024-04-22T18:39:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6205999999999997e-05 2024-04-22T18:39:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1889 4.6 4:6700eab5:::smithi08937929-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:35.542099+0000", 2024-04-22T18:39:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.65900047799999995, 2024-04-22T18:39:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33675021900000002, 2024-04-22T18:39:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:36.306 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1889 2024-04-22T18:39:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542099+0000", 2024-04-22T18:39:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542099+0000", 2024-04-22T18:39:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542099+0000", 2024-04-22T18:39:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542101+0000", 2024-04-22T18:39:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2290000000000002e-06 2024-04-22T18:39:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542104+0000", 2024-04-22T18:39:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6550000000000002e-06 2024-04-22T18:39:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.542108+0000", 2024-04-22T18:39:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0510000000000001e-06 2024-04-22T18:39:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.644702+0000", 2024-04-22T18:39:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10259454899999999 2024-04-22T18:39:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.644750+0000", 2024-04-22T18:39:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7809999999999998e-05 2024-04-22T18:39:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.878668+0000", 2024-04-22T18:39:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.233917773 2024-04-22T18:39:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.878755+0000", 2024-04-22T18:39:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7038000000000003e-05 2024-04-22T18:39:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.878849+0000", 2024-04-22T18:39:36.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4498000000000002e-05 2024-04-22T18:39:36.311 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.311 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.311 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.311 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.311 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.311 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:36.313 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:36.314 INFO:tasks.rados.rados.0.smithi089.stdout:981: finishing write tid 1 to smithi08937929-11 2024-04-22T18:39:36.314 INFO:tasks.rados.rados.0.smithi089.stdout:980: finishing write tid 4 to smithi08937929-16 2024-04-22T18:39:36.314 INFO:tasks.rados.rados.0.smithi089.stdout:980: oid 16 updating version 0 to 74 2024-04-22T18:39:36.314 INFO:tasks.rados.rados.0.smithi089.stdout:980: oid 16 updating version 74 to 75 2024-04-22T18:39:36.314 INFO:tasks.rados.rados.0.smithi089.stdout:980: oid 16 updating version 75 to 76 2024-04-22T18:39:36.314 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 76 (ObjNum 154 snap 0 seq_num 154) dirty exists 2024-04-22T18:39:36.314 INFO:tasks.rados.rados.0.smithi089.stdout:980: left oid 16 (ObjNum 154 snap 0 seq_num 154) 2024-04-22T18:39:36.314 INFO:tasks.rados.rados.0.smithi089.stdout:986: finishing write tid 1 to smithi08937929-32 2024-04-22T18:39:36.314 INFO:tasks.rados.rados.0.smithi089.stdout:985: finishing write tid 1 to smithi08937929-2 2024-04-22T18:39:36.314 INFO:tasks.rados.rados.0.smithi089.stdout:990: seq_num 160 ranges {689120=483253,1946413=679049} 2024-04-22T18:39:36.314 INFO:tasks.rados.rados.0.smithi089.stdout:990: writing smithi08937929-27 from 689120 to 1172373 tid 1 2024-04-22T18:39:36.314 INFO:tasks.rados.rados.0.smithi089.stdout:990: writing smithi08937929-27 from 1946413 to 2625462 tid 2 2024-04-22T18:39:36.314 INFO:tasks.rados.rados.0.smithi089.stdout:980: done (8 left) 2024-04-22T18:39:36.315 INFO:tasks.rados.rados.0.smithi089.stdout:982: done (7 left) 2024-04-22T18:39:36.315 INFO:tasks.rados.rados.0.smithi089.stdout:991: write oid 7 current snap is 0 2024-04-22T18:39:36.315 INFO:tasks.rados.rados.0.smithi089.stdout:991: seq_num 161 ranges {705927=558502,2004895=422707} 2024-04-22T18:39:36.315 INFO:tasks.rados.rados.0.smithi089.stdout:991: writing smithi08937929-7 from 705927 to 1264429 tid 1 2024-04-22T18:39:36.315 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:36.313+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:36.319 INFO:tasks.rados.rados.0.smithi089.stdout:991: writing smithi08937929-7 from 2004895 to 2427602 tid 2 2024-04-22T18:39:36.319 INFO:tasks.rados.rados.0.smithi089.stdout:992: read oid 48 snap -1 2024-04-22T18:39:36.319 INFO:tasks.rados.rados.0.smithi089.stdout:992: expect (ObjNum 128 snap 0 seq_num 128) 2024-04-22T18:39:36.319 INFO:tasks.rados.rados.0.smithi089.stdout:993: write oid 35 current snap is 0 2024-04-22T18:39:36.319 INFO:tasks.rados.rados.0.smithi089.stdout:985: finishing write tid 2 to smithi08937929-2 2024-04-22T18:39:36.319 INFO:tasks.rados.rados.0.smithi089.stdout:985: finishing write tid 3 to smithi08937929-2 2024-04-22T18:39:36.319 INFO:tasks.rados.rados.0.smithi089.stdout:985: finishing write tid 4 to smithi08937929-2 2024-04-22T18:39:36.319 INFO:tasks.rados.rados.0.smithi089.stdout:985: oid 2 updating version 0 to 34 2024-04-22T18:39:36.319 INFO:tasks.rados.rados.0.smithi089.stdout:985: oid 2 updating version 34 to 36 2024-04-22T18:39:36.319 INFO:tasks.rados.rados.0.smithi089.stdout:985: oid 2 version 36 is already newer than 35 2024-04-22T18:39:36.320 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 36 (ObjNum 156 snap 0 seq_num 156) dirty exists 2024-04-22T18:39:36.320 INFO:tasks.rados.rados.0.smithi089.stdout:985: left oid 2 (ObjNum 156 snap 0 seq_num 156) 2024-04-22T18:39:36.320 INFO:tasks.rados.rados.0.smithi089.stdout:986: finishing write tid 2 to smithi08937929-32 2024-04-22T18:39:36.320 INFO:tasks.rados.rados.0.smithi089.stdout:981: finishing write tid 2 to smithi08937929-11 2024-04-22T18:39:36.320 INFO:tasks.rados.rados.0.smithi089.stdout:981: finishing write tid 3 to smithi08937929-11 2024-04-22T18:39:36.320 INFO:tasks.rados.rados.0.smithi089.stdout:986: finishing write tid 3 to smithi08937929-32 2024-04-22T18:39:36.320 INFO:tasks.rados.rados.0.smithi089.stdout:986: finishing write tid 4 to smithi08937929-32 2024-04-22T18:39:36.320 INFO:tasks.rados.rados.0.smithi089.stdout:986: oid 32 updating version 0 to 53 2024-04-22T18:39:36.320 INFO:tasks.rados.rados.0.smithi089.stdout:986: oid 32 version 53 is already newer than 52 2024-04-22T18:39:36.320 INFO:tasks.rados.rados.0.smithi089.stdout:986: oid 32 updating version 53 to 54 2024-04-22T18:39:36.320 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 54 (ObjNum 157 snap 0 seq_num 157) dirty exists 2024-04-22T18:39:36.320 INFO:tasks.rados.rados.0.smithi089.stdout:986: left oid 32 (ObjNum 157 snap 0 seq_num 157) 2024-04-22T18:39:36.320 INFO:tasks.rados.rados.0.smithi089.stdout:981: finishing write tid 4 to smithi08937929-11 2024-04-22T18:39:36.320 INFO:tasks.rados.rados.0.smithi089.stdout:981: oid 11 updating version 0 to 26 2024-04-22T18:39:36.320 INFO:tasks.rados.rados.0.smithi089.stdout:981: oid 11 updating version 26 to 27 2024-04-22T18:39:36.321 INFO:tasks.rados.rados.0.smithi089.stdout:981: oid 11 updating version 27 to 28 2024-04-22T18:39:36.321 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 28 (ObjNum 155 snap 0 seq_num 155) dirty exists 2024-04-22T18:39:36.321 INFO:tasks.rados.rados.0.smithi089.stdout:981: left oid 11 (ObjNum 155 snap 0 seq_num 155) 2024-04-22T18:39:36.321 INFO:tasks.rados.rados.0.smithi089.stdout:988: expect (ObjNum 120 snap 0 seq_num 120) 2024-04-22T18:39:36.409 INFO:tasks.rados.rados.0.smithi089.stdout:987: finishing write tid 1 to smithi08937929-23 2024-04-22T18:39:36.410 INFO:tasks.rados.rados.0.smithi089.stdout:987: finishing write tid 2 to smithi08937929-23 2024-04-22T18:39:36.410 INFO:tasks.rados.rados.0.smithi089.stdout:987: finishing write tid 3 to smithi08937929-23 2024-04-22T18:39:36.410 INFO:tasks.rados.rados.0.smithi089.stdout:987: finishing write tid 4 to smithi08937929-23 2024-04-22T18:39:36.410 INFO:tasks.rados.rados.0.smithi089.stdout:987: oid 23 updating version 0 to 87 2024-04-22T18:39:36.410 INFO:tasks.rados.rados.0.smithi089.stdout:987: oid 23 updating version 87 to 88 2024-04-22T18:39:36.410 INFO:tasks.rados.rados.0.smithi089.stdout:987: oid 23 updating version 88 to 89 2024-04-22T18:39:36.410 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 89 (ObjNum 158 snap 0 seq_num 158) dirty exists 2024-04-22T18:39:36.410 INFO:tasks.rados.rados.0.smithi089.stdout:987: left oid 23 (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:39:36.410 INFO:tasks.rados.rados.0.smithi089.stdout:989: finishing write tid 1 to smithi08937929-19 2024-04-22T18:39:36.410 INFO:tasks.rados.rados.0.smithi089.stdout:993: seq_num 162 ranges {722734=633751,2063377=659357} 2024-04-22T18:39:36.412 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:36.414+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:36.417 INFO:tasks.rados.rados.0.smithi089.stdout:993: writing smithi08937929-35 from 722734 to 1356485 tid 1 2024-04-22T18:39:36.424 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:36.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:36.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.424 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1902 4.9 e51/20)", 2024-04-22T18:39:36.424 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.416204+0000", 2024-04-22T18:39:36.424 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0097313150000000008, 2024-04-22T18:39:36.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0097609150000000002, 2024-04-22T18:39:36.424 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.424 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "no flag points reached", 2024-04-22T18:39:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.416204+0000", 2024-04-22T18:39:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.416204+0000", 2024-04-22T18:39:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.416202+0000", 2024-04-22T18:39:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.416731+0000", 2024-04-22T18:39:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00052890499999999998 2024-04-22T18:39:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.416743+0000", 2024-04-22T18:39:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1918e-05 2024-04-22T18:39:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.426 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.427 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.427 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:39:36.427 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 1 2024-04-22T18:39:36.427 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:36.427 INFO:tasks.rados.rados.0.smithi089.stdout:993: writing smithi08937929-35 from 2063377 to 2722734 tid 2 2024-04-22T18:39:36.428 INFO:tasks.rados.rados.0.smithi089.stdout:981: done (9 left) 2024-04-22T18:39:36.428 INFO:tasks.rados.rados.0.smithi089.stdout:985: done (8 left) 2024-04-22T18:39:36.428 INFO:tasks.rados.rados.0.smithi089.stdout:986: done (7 left) 2024-04-22T18:39:36.428 INFO:tasks.rados.rados.0.smithi089.stdout:987: done (6 left) 2024-04-22T18:39:36.428 INFO:tasks.rados.rados.0.smithi089.stdout:988: done (5 left) 2024-04-22T18:39:36.428 INFO:tasks.rados.rados.0.smithi089.stdout:994: write (excl) oid 14 current snap is 0 2024-04-22T18:39:36.428 INFO:tasks.rados.rados.0.smithi089.stdout:994: seq_num 163 ranges {739541=709000,2121859=426376} 2024-04-22T18:39:36.432 INFO:tasks.rados.rados.0.smithi089.stdout:994: writing smithi08937929-14 from 739541 to 1448541 tid 1 2024-04-22T18:39:36.433 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:36.438 INFO:tasks.rados.rados.0.smithi089.stdout:994: writing smithi08937929-14 from 2121859 to 2548235 tid 2 2024-04-22T18:39:36.438 INFO:tasks.rados.rados.0.smithi089.stdout:995: set_chunk oid 49 target oid 2024-04-22T18:39:36.513 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:36.514+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:36.541 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:36.541 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:36.550 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:36.633 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:36.635+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:36.667 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:39:36.680 INFO:tasks.rados.rados.0.smithi089.stdout:995: set_chunk oid 49 offset: 774656 length: 1175040 target oid 7ae638328c5300ed71103b625cc8801a553f2fabeba5f7d90e47e9e6c12387ab offset: 0 snap -1 2024-04-22T18:39:36.680 INFO:tasks.rados.rados.0.smithi089.stdout:996: write (excl) oid 39 current snap is 0 2024-04-22T18:39:36.680 INFO:tasks.rados.rados.0.smithi089.stdout:996: seq_num 164 ranges {756348=784249,2063988=686387} 2024-04-22T18:39:36.688 INFO:tasks.rados.rados.0.smithi089.stdout:996: writing smithi08937929-39 from 756348 to 1540597 tid 1 2024-04-22T18:39:36.695 INFO:tasks.rados.rados.0.smithi089.stdout:996: writing smithi08937929-39 from 2063988 to 2750375 tid 2 2024-04-22T18:39:36.695 INFO:tasks.rados.rados.0.smithi089.stdout:997: read oid 33 snap -1 2024-04-22T18:39:36.695 INFO:tasks.rados.rados.0.smithi089.stdout:997: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-22T18:39:36.695 INFO:tasks.rados.rados.0.smithi089.stdout:998: write oid 18 current snap is 0 2024-04-22T18:39:36.695 INFO:tasks.rados.rados.0.smithi089.stdout:992: expect (ObjNum 128 snap 0 seq_num 128) 2024-04-22T18:39:36.702 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:36.702 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:36.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:36.702 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:36.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.702 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:8 4.f 4:f83a159b:::smithi08937929-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.590701002000003, 2024-04-22T18:39:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30292182499999998, 2024-04-22T18:39:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 8 2024-04-22T18:39:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111380+0000", 2024-04-22T18:39:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2419999999999999e-06 2024-04-22T18:39:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111384+0000", 2024-04-22T18:39:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.715e-06 2024-04-22T18:39:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111387+0000", 2024-04-22T18:39:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.982e-06 2024-04-22T18:39:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414113+0000", 2024-04-22T18:39:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30272622700000001 2024-04-22T18:39:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414203+0000", 2024-04-22T18:39:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0316999999999996e-05 2024-04-22T18:39:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414300+0000", 2024-04-22T18:39:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6760000000000002e-05 2024-04-22T18:39:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:9 4.f 4:f2c48a0e:::smithi08937929-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.585305417999997, 2024-04-22T18:39:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49290714600000002, 2024-04-22T18:39:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.707 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 9 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116773+0000", 2024-04-22T18:39:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117024+0000", 2024-04-22T18:39:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025191299999999999 2024-04-22T18:39:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117036+0000", 2024-04-22T18:39:36.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1595e-05 2024-04-22T18:39:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117046+0000", 2024-04-22T18:39:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0456e-05 2024-04-22T18:39:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414324+0000", 2024-04-22T18:39:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297277603 2024-04-22T18:39:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414404+0000", 2024-04-22T18:39:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9738000000000001e-05 2024-04-22T18:39:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414494+0000", 2024-04-22T18:39:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9733000000000003e-05 2024-04-22T18:39:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518194+0000", 2024-04-22T18:39:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10370061 2024-04-22T18:39:36.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609618+0000", 2024-04-22T18:39:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091423360999999995 2024-04-22T18:39:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609655+0000", 2024-04-22T18:39:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7580000000000003e-05 2024-04-22T18:39:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609681+0000", 2024-04-22T18:39:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5908e-05 2024-04-22T18:39:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:10 4.f 4:f2c48a0e:::smithi08937929-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.579453158, 2024-04-22T18:39:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48712681400000002, 2024-04-22T18:39:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.713 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 10 2024-04-22T18:39:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:36.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122625+0000", 2024-04-22T18:39:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122828+0000", 2024-04-22T18:39:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020359499999999999 2024-04-22T18:39:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122840+0000", 2024-04-22T18:39:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1239e-05 2024-04-22T18:39:36.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122845+0000", 2024-04-22T18:39:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2719999999999997e-06 2024-04-22T18:39:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415105+0000", 2024-04-22T18:39:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29225972099999997 2024-04-22T18:39:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415169+0000", 2024-04-22T18:39:36.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4170000000000004e-05 2024-04-22T18:39:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415257+0000", 2024-04-22T18:39:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8331999999999994e-05 2024-04-22T18:39:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518199+0000", 2024-04-22T18:39:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10294174 2024-04-22T18:39:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609707+0000", 2024-04-22T18:39:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091508408999999999 2024-04-22T18:39:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609729+0000", 2024-04-22T18:39:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1917e-05 2024-04-22T18:39:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609753+0000", 2024-04-22T18:39:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3776000000000001e-05 2024-04-22T18:39:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.721 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:11 4.f 4:f2c48a0e:::smithi08937929-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.579229785000003, 2024-04-22T18:39:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49452214500000002, 2024-04-22T18:39:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 11 2024-04-22T18:39:36.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122849+0000", 2024-04-22T18:39:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122853+0000", 2024-04-22T18:39:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9029999999999997e-06 2024-04-22T18:39:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122856+0000", 2024-04-22T18:39:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.01e-06 2024-04-22T18:39:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122859+0000", 2024-04-22T18:39:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415755+0000", 2024-04-22T18:39:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29289606800000001 2024-04-22T18:39:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415808+0000", 2024-04-22T18:39:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3415999999999998e-05 2024-04-22T18:39:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415888+0000", 2024-04-22T18:39:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9061999999999996e-05 2024-04-22T18:39:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518203+0000", 2024-04-22T18:39:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102315533 2024-04-22T18:39:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617164+0000", 2024-04-22T18:39:36.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098960548999999995 2024-04-22T18:39:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617241+0000", 2024-04-22T18:39:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7292000000000005e-05 2024-04-22T18:39:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617372+0000", 2024-04-22T18:39:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013075399999999999 2024-04-22T18:39:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:12 4.f 4:f2c48a0e:::smithi08937929-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.579217921000001, 2024-04-22T18:39:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49473910999999998, 2024-04-22T18:39:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 12 2024-04-22T18:39:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122863+0000", 2024-04-22T18:39:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.863e-06 2024-04-22T18:39:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122866+0000", 2024-04-22T18:39:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:39:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122869+0000", 2024-04-22T18:39:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5950000000000001e-06 2024-04-22T18:39:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416008+0000", 2024-04-22T18:39:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29313975199999998 2024-04-22T18:39:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416048+0000", 2024-04-22T18:39:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9888000000000002e-05 2024-04-22T18:39:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617447+0000", 2024-04-22T18:39:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20139881400000001 2024-04-22T18:39:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617537+0000", 2024-04-22T18:39:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9606999999999998e-05 2024-04-22T18:39:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617600+0000", 2024-04-22T18:39:36.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3884e-05 2024-04-22T18:39:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:33 4.f 4:f0ed3d49:::smithi08937929-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.51527772, 2024-04-22T18:39:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62359692099999997, 2024-04-22T18:39:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.733 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 33 2024-04-22T18:39:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186799+0000", 2024-04-22T18:39:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187103+0000", 2024-04-22T18:39:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030354900000000002 2024-04-22T18:39:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187117+0000", 2024-04-22T18:39:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4578e-05 2024-04-22T18:39:36.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187129+0000", 2024-04-22T18:39:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1421e-05 2024-04-22T18:39:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516134+0000", 2024-04-22T18:39:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32900553900000001 2024-04-22T18:39:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516215+0000", 2024-04-22T18:39:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0537999999999994e-05 2024-04-22T18:39:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516302+0000", 2024-04-22T18:39:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6601000000000007e-05 2024-04-22T18:39:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518207+0000", 2024-04-22T18:39:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019058370000000001 2024-04-22T18:39:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810302+0000", 2024-04-22T18:39:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29209509700000003 2024-04-22T18:39:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810348+0000", 2024-04-22T18:39:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5992999999999998e-05 2024-04-22T18:39:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810398+0000", 2024-04-22T18:39:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0046000000000002e-05 2024-04-22T18:39:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.738 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.739 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.739 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:34 4.f 4:f0ed3d49:::smithi08937929-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.739 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:36.739 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.510675708000001, 2024-04-22T18:39:36.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.61954655800000002, 2024-04-22T18:39:36.739 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.739 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.739 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.739 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.739 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34 2024-04-22T18:39:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191401+0000", 2024-04-22T18:39:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191503+0000", 2024-04-22T18:39:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010190400000000001 2024-04-22T18:39:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191517+0000", 2024-04-22T18:39:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3267e-05 2024-04-22T18:39:36.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191523+0000", 2024-04-22T18:39:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-04-22T18:39:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517062+0000", 2024-04-22T18:39:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32553894300000003 2024-04-22T18:39:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517141+0000", 2024-04-22T18:39:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9667999999999997e-05 2024-04-22T18:39:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517267+0000", 2024-04-22T18:39:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000125494 2024-04-22T18:39:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518213+0000", 2024-04-22T18:39:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000946275 2024-04-22T18:39:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810901+0000", 2024-04-22T18:39:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29268821699999997 2024-04-22T18:39:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810929+0000", 2024-04-22T18:39:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7645999999999999e-05 2024-04-22T18:39:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810950+0000", 2024-04-22T18:39:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1333999999999999e-05 2024-04-22T18:39:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.744 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.510552740000001, 2024-04-22T18:39:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:39:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.745 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:39:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:39:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:39:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:39:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:39:36.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:39:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:39:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:39:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:39:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:39:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:39:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:39:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:39:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:39:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:39:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:39:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:39:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:39:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:39:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:39:36.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:39:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.510541664999998, 2024-04-22T18:39:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:39:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.751 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:39:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:39:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:39:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:39:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:39:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:39:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:39:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:39:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:39:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:39:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:39:36.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:39:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:39:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:39:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:39:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:39:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:39:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:39:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:38 4.2 4:4fb26dd8:::smithi08937929-10:head [write 1893632~257631] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.498819060999999, 2024-04-22T18:39:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33863800199999999, 2024-04-22T18:39:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.756 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 38 2024-04-22T18:39:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203258+0000", 2024-04-22T18:39:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203396+0000", 2024-04-22T18:39:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013710799999999999 2024-04-22T18:39:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203406+0000", 2024-04-22T18:39:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0295000000000001e-05 2024-04-22T18:39:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203411+0000", 2024-04-22T18:39:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5079999999999996e-06 2024-04-22T18:39:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203450+0000", 2024-04-22T18:39:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8420999999999997e-05 2024-04-22T18:39:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203455+0000", 2024-04-22T18:39:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:39:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237071+0000", 2024-04-22T18:39:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.033615891000000002 2024-04-22T18:39:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237129+0000", 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8134000000000003e-05 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237221+0000", 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2053999999999997e-05 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.341419+0000", 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104197568 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541792+0000", 2024-04-22T18:39:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.200373312 2024-04-22T18:39:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541838+0000", 2024-04-22T18:39:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6004000000000002e-05 2024-04-22T18:39:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541898+0000", 2024-04-22T18:39:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0297000000000002e-05 2024-04-22T18:39:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.762 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:39 4.2 4:4fb26dd8:::smithi08937929-10:head [setxattr _header (56) in=7b,truncate 2151263] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.498662066999998, 2024-04-22T18:39:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33862203000000002, 2024-04-22T18:39:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 39 2024-04-22T18:39:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.763 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203423+0000", 2024-04-22T18:39:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0589999999999999e-06 2024-04-22T18:39:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203429+0000", 2024-04-22T18:39:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2559999999999999e-06 2024-04-22T18:39:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203434+0000", 2024-04-22T18:39:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8049999999999997e-06 2024-04-22T18:39:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203475+0000", 2024-04-22T18:39:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1205000000000003e-05 2024-04-22T18:39:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203478+0000", 2024-04-22T18:39:36.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7140000000000002e-06 2024-04-22T18:39:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239180+0000", 2024-04-22T18:39:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035702195999999999 2024-04-22T18:39:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239225+0000", 2024-04-22T18:39:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5722e-05 2024-04-22T18:39:36.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239291+0000", 2024-04-22T18:39:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5870000000000005e-05 2024-04-22T18:39:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342114+0000", 2024-04-22T18:39:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102822301 2024-04-22T18:39:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541946+0000", 2024-04-22T18:39:36.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.199832544 2024-04-22T18:39:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541988+0000", 2024-04-22T18:39:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1499999999999999e-05 2024-04-22T18:39:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542039+0000", 2024-04-22T18:39:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1576000000000002e-05 2024-04-22T18:39:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.769 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:40 4.2 4:4fb26dd8:::smithi08937929-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.498624806999999, 2024-04-22T18:39:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33924657200000002, 2024-04-22T18:39:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 40 2024-04-22T18:39:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203457+0000", 2024-04-22T18:39:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.129e-06 2024-04-22T18:39:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203463+0000", 2024-04-22T18:39:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7440000000000004e-06 2024-04-22T18:39:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203468+0000", 2024-04-22T18:39:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2619999999999998e-06 2024-04-22T18:39:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203491+0000", 2024-04-22T18:39:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3260000000000001e-05 2024-04-22T18:39:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203493+0000", 2024-04-22T18:39:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9599999999999999e-06 2024-04-22T18:39:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239420+0000", 2024-04-22T18:39:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035927212 2024-04-22T18:39:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239462+0000", 2024-04-22T18:39:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.159e-05 2024-04-22T18:39:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542547+0000", 2024-04-22T18:39:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30308492599999998 2024-04-22T18:39:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542611+0000", 2024-04-22T18:39:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4622000000000006e-05 2024-04-22T18:39:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542701+0000", 2024-04-22T18:39:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9579000000000004e-05 2024-04-22T18:39:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:41 4.2 4:409e6456:::smithi08937929-11:head [write 568070~468843,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.775 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.492908536000002, 2024-04-22T18:39:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33297193600000002, 2024-04-22T18:39:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 41 2024-04-22T18:39:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:36.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209168+0000", 2024-04-22T18:39:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209393+0000", 2024-04-22T18:39:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022485199999999999 2024-04-22T18:39:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209404+0000", 2024-04-22T18:39:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1209999999999999e-05 2024-04-22T18:39:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209411+0000", 2024-04-22T18:39:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8580000000000002e-06 2024-04-22T18:39:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209457+0000", 2024-04-22T18:39:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6351999999999999e-05 2024-04-22T18:39:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209463+0000", 2024-04-22T18:39:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.322e-06 2024-04-22T18:39:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339572+0000", 2024-04-22T18:39:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13010901999999999 2024-04-22T18:39:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339673+0000", 2024-04-22T18:39:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101332 2024-04-22T18:39:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339765+0000", 2024-04-22T18:39:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1780999999999994e-05 2024-04-22T18:39:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342117+0000", 2024-04-22T18:39:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002351709 2024-04-22T18:39:36.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542074+0000", 2024-04-22T18:39:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995749800000001 2024-04-22T18:39:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542112+0000", 2024-04-22T18:39:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7891e-05 2024-04-22T18:39:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542143+0000", 2024-04-22T18:39:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0756000000000002e-05 2024-04-22T18:39:36.781 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:42 4.2 4:409e6456:::smithi08937929-11:head [write 1552114~615956] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.486570227000001, 2024-04-22T18:39:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326735257, 2024-04-22T18:39:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 42 2024-04-22T18:39:36.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215508+0000", 2024-04-22T18:39:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215766+0000", 2024-04-22T18:39:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025766999999999999 2024-04-22T18:39:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215776+0000", 2024-04-22T18:39:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0772e-05 2024-04-22T18:39:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215785+0000", 2024-04-22T18:39:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6880000000000005e-06 2024-04-22T18:39:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215818+0000", 2024-04-22T18:39:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2960999999999998e-05 2024-04-22T18:39:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215821+0000", 2024-04-22T18:39:36.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8569999999999999e-06 2024-04-22T18:39:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340332+0000", 2024-04-22T18:39:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.124511409 2024-04-22T18:39:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340416+0000", 2024-04-22T18:39:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3793000000000002e-05 2024-04-22T18:39:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340543+0000", 2024-04-22T18:39:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012720299999999999 2024-04-22T18:39:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342361+0000", 2024-04-22T18:39:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018174370000000001 2024-04-22T18:39:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542176+0000", 2024-04-22T18:39:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19981518400000001 2024-04-22T18:39:36.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542211+0000", 2024-04-22T18:39:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5128e-05 2024-04-22T18:39:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542244+0000", 2024-04-22T18:39:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3256000000000001e-05 2024-04-22T18:39:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:43 4.2 4:409e6456:::smithi08937929-11:head [setxattr _header (56) in=7b,truncate 2168070] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.486285453999997, 2024-04-22T18:39:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32656540499999998, 2024-04-22T18:39:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 43 2024-04-22T18:39:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215793+0000", 2024-04-22T18:39:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215797+0000", 2024-04-22T18:39:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4190000000000004e-06 2024-04-22T18:39:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215804+0000", 2024-04-22T18:39:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:39:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215809+0000", 2024-04-22T18:39:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3689999999999998e-06 2024-04-22T18:39:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215840+0000", 2024-04-22T18:39:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-05 2024-04-22T18:39:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215843+0000", 2024-04-22T18:39:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8849999999999999e-06 2024-04-22T18:39:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342483+0000", 2024-04-22T18:39:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126640172 2024-04-22T18:39:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342529+0000", 2024-04-22T18:39:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5668999999999999e-05 2024-04-22T18:39:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342597+0000", 2024-04-22T18:39:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7538000000000003e-05 2024-04-22T18:39:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.347102+0000", 2024-04-22T18:39:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045058040000000004 2024-04-22T18:39:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542278+0000", 2024-04-22T18:39:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.195175879 2024-04-22T18:39:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542314+0000", 2024-04-22T18:39:36.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6134999999999999e-05 2024-04-22T18:39:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542359+0000", 2024-04-22T18:39:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4848999999999997e-05 2024-04-22T18:39:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:44 4.2 4:409e6456:::smithi08937929-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.486265445999997, 2024-04-22T18:39:36.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326706418, 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 44 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:36.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215813+0000", 2024-04-22T18:39:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215816+0000", 2024-04-22T18:39:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9799999999999998e-06 2024-04-22T18:39:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215822+0000", 2024-04-22T18:39:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5570000000000003e-06 2024-04-22T18:39:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215827+0000", 2024-04-22T18:39:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2739999999999998e-06 2024-04-22T18:39:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215855+0000", 2024-04-22T18:39:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7682999999999999e-05 2024-04-22T18:39:36.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215857+0000", 2024-04-22T18:39:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9250000000000002e-06 2024-04-22T18:39:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342717+0000", 2024-04-22T18:39:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126860696 2024-04-22T18:39:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:36.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342758+0000", 2024-04-22T18:39:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0295e-05 2024-04-22T18:39:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542408+0000", 2024-04-22T18:39:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19964990599999999 2024-04-22T18:39:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542457+0000", 2024-04-22T18:39:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9784999999999999e-05 2024-04-22T18:39:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542520+0000", 2024-04-22T18:39:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2960999999999996e-05 2024-04-22T18:39:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.909172171000002, 2024-04-22T18:39:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:39:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.801 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:39:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:36.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:39:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:39:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:39:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:39:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:39:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:39:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:39:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:39:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:39:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:39:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:39:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:39:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:39:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:39:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:39:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:39:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:39:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:39:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:39:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:39:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:39:36.806 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1262 4.7 4:e117a562:::smithi08937929-33:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.249028995, 2024-04-22T18:39:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40488836700000003, 2024-04-22T18:39:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1262 2024-04-22T18:39:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453048+0000", 2024-04-22T18:39:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453058+0000", 2024-04-22T18:39:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:39:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453069+0000", 2024-04-22T18:39:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1117999999999999e-05 2024-04-22T18:39:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453075+0000", 2024-04-22T18:39:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0909999999999998e-06 2024-04-22T18:39:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453115+0000", 2024-04-22T18:39:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0119e-05 2024-04-22T18:39:36.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453119+0000", 2024-04-22T18:39:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3610000000000003e-06 2024-04-22T18:39:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454814+0000", 2024-04-22T18:39:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00169436 2024-04-22T18:39:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454861+0000", 2024-04-22T18:39:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7184000000000001e-05 2024-04-22T18:39:36.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756925+0000", 2024-04-22T18:39:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30206361799999998 2024-04-22T18:39:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757577+0000", 2024-04-22T18:39:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065199599999999996 2024-04-22T18:39:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857846+0000", 2024-04-22T18:39:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100269369 2024-04-22T18:39:36.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857896+0000", 2024-04-22T18:39:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9607000000000001e-05 2024-04-22T18:39:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857939+0000", 2024-04-22T18:39:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3149999999999999e-05 2024-04-22T18:39:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1689 3.0 3:1d4bcadf:::e7eb8db0531d9f456e91fe87bc79e47ebd80459f1783dc2a5e53d4822cd34f49:head [write 0~1066496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:39:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.607920+0000", 2024-04-22T18:39:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.0941596210000002, 2024-04-22T18:39:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30339939300000002, 2024-04-22T18:39:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1689 2024-04-22T18:39:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.607920+0000", 2024-04-22T18:39:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.607920+0000", 2024-04-22T18:39:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.607917+0000", 2024-04-22T18:39:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.608327+0000", 2024-04-22T18:39:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000409508 2024-04-22T18:39:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.608344+0000", 2024-04-22T18:39:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6846000000000001e-05 2024-04-22T18:39:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.608351+0000", 2024-04-22T18:39:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8140000000000003e-06 2024-04-22T18:39:36.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708489+0000", 2024-04-22T18:39:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100138536 2024-04-22T18:39:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708651+0000", 2024-04-22T18:39:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000162132 2024-04-22T18:39:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:39:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708793+0000", 2024-04-22T18:39:36.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014147699999999999 2024-04-22T18:39:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.713125+0000", 2024-04-22T18:39:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043321369999999998 2024-04-22T18:39:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911261+0000", 2024-04-22T18:39:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198135744 2024-04-22T18:39:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911293+0000", 2024-04-22T18:39:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1970000000000001e-05 2024-04-22T18:39:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911319+0000", 2024-04-22T18:39:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6440000000000001e-05 2024-04-22T18:39:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1872 4.2 4:409e6456:::smithi08937929-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:36.819 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:35.576858+0000", 2024-04-22T18:39:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.125220868, 2024-04-22T18:39:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.302951992, 2024-04-22T18:39:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1872 2024-04-22T18:39:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576858+0000", 2024-04-22T18:39:36.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576858+0000", 2024-04-22T18:39:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576857+0000", 2024-04-22T18:39:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:36.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576864+0000", 2024-04-22T18:39:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4569999999999999e-06 2024-04-22T18:39:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576871+0000", 2024-04-22T18:39:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1679999999999998e-06 2024-04-22T18:39:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576876+0000", 2024-04-22T18:39:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0440000000000003e-06 2024-04-22T18:39:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.590050+0000", 2024-04-22T18:39:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.013174208 2024-04-22T18:39:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.590142+0000", 2024-04-22T18:39:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1681000000000005e-05 2024-04-22T18:39:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.879685+0000", 2024-04-22T18:39:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.28954336200000003 2024-04-22T18:39:36.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.879745+0000", 2024-04-22T18:39:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9487999999999997e-05 2024-04-22T18:39:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.879810+0000", 2024-04-22T18:39:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5431000000000004e-05 2024-04-22T18:39:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:36.824 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:36.824 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:36.826 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:36.827 INFO:tasks.rados.rados.0.smithi089.stdout:994: finishing write tid 1 to smithi08937929-14 2024-04-22T18:39:36.828 INFO:tasks.rados.rados.0.smithi089.stdout:989: finishing write tid 2 to smithi08937929-19 2024-04-22T18:39:36.828 INFO:tasks.rados.rados.0.smithi089.stdout:989: finishing write tid 3 to smithi08937929-19 2024-04-22T18:39:36.828 INFO:tasks.rados.rados.0.smithi089.stdout:993: finishing write tid 1 to smithi08937929-35 2024-04-22T18:39:36.828 INFO:tasks.rados.rados.0.smithi089.stdout:993: finishing write tid 2 to smithi08937929-35 2024-04-22T18:39:36.828 INFO:tasks.rados.rados.0.smithi089.stdout:998: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2024-04-22T18:39:36.828 INFO:tasks.rados.rados.0.smithi089.stdout:998: writing smithi08937929-18 from 773155 to 1232653 tid 1 2024-04-22T18:39:36.828 INFO:tasks.rados.rados.0.smithi089.stdout:998: writing smithi08937929-18 from 1722470 to 2152515 tid 2 2024-04-22T18:39:36.828 INFO:tasks.rados.rados.0.smithi089.stdout:998: writing smithi08937929-18 from 2648676 to 2773155 tid 3 2024-04-22T18:39:36.828 INFO:tasks.rados.rados.0.smithi089.stdout:992: done (9 left) 2024-04-22T18:39:36.828 INFO:tasks.rados.rados.0.smithi089.stdout:999: write (excl) oid 45 current snap is 0 2024-04-22T18:39:36.828 INFO:tasks.rados.rados.0.smithi089.stdout:989: finishing write tid 4 to smithi08937929-19 2024-04-22T18:39:36.828 INFO:tasks.rados.rados.0.smithi089.stdout:989: oid 19 updating version 0 to 92 2024-04-22T18:39:36.828 INFO:tasks.rados.rados.0.smithi089.stdout:989: oid 19 version 92 is already newer than 90 2024-04-22T18:39:36.828 INFO:tasks.rados.rados.0.smithi089.stdout:989: oid 19 version 92 is already newer than 91 2024-04-22T18:39:36.829 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 92 (ObjNum 159 snap 0 seq_num 159) dirty exists 2024-04-22T18:39:36.829 INFO:tasks.rados.rados.0.smithi089.stdout:989: left oid 19 (ObjNum 159 snap 0 seq_num 159) 2024-04-22T18:39:36.829 INFO:tasks.rados.rados.0.smithi089.stdout:993: finishing write tid 3 to smithi08937929-35 2024-04-22T18:39:36.829 INFO:tasks.rados.rados.0.smithi089.stdout:999: seq_num 166 ranges {789962=534747,1780952=690056} 2024-04-22T18:39:36.829 INFO:tasks.rados.rados.0.smithi089.stdout:999: writing smithi08937929-45 from 789962 to 1324709 tid 1 2024-04-22T18:39:36.829 INFO:tasks.rados.rados.0.smithi089.stdout:999: writing smithi08937929-45 from 1780952 to 2471008 tid 2 2024-04-22T18:39:36.829 INFO:tasks.rados.rados.0.smithi089.stdout:989: done (9 left) 2024-04-22T18:39:36.829 INFO:tasks.rados.rados.0.smithi089.stdout:1000: read oid 2 snap -1 2024-04-22T18:39:36.829 INFO:tasks.rados.rados.0.smithi089.stdout:1000: expect (ObjNum 156 snap 0 seq_num 156) 2024-04-22T18:39:36.829 INFO:tasks.rados.rados.0.smithi089.stdout:1001: set_chunk oid 26 target oid 2024-04-22T18:39:36.829 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:36.827+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:36.926 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:36.927+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:36.977 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:36.977 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:36.987 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:37.030 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:37.032+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:37.089 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:37.089 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:37.089 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:37.089 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:37.089 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:37.098 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:37.127 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:37.128+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:37.226 INFO:tasks.rados.rados.0.smithi089.stdout:1001: set_chunk oid 26 offset: 620544 length: 1673216 target oid 5065fe164b0201b19145d094941407f390b5cf4594ebb2f11fb7d1f1efde2f53 offset: 0 snap -1 2024-04-22T18:39:37.226 INFO:tasks.rados.rados.0.smithi089.stdout:1002: write oid 11 current snap is 0 2024-04-22T18:39:37.226 INFO:tasks.rados.rados.0.smithi089.stdout:990: finishing write tid 1 to smithi08937929-27 2024-04-22T18:39:37.226 INFO:tasks.rados.rados.0.smithi089.stdout:991: finishing write tid 1 to smithi08937929-7 2024-04-22T18:39:37.226 INFO:tasks.rados.rados.0.smithi089.stdout:991: finishing write tid 2 to smithi08937929-7 2024-04-22T18:39:37.226 INFO:tasks.rados.rados.0.smithi089.stdout:993: finishing write tid 4 to smithi08937929-35 2024-04-22T18:39:37.226 INFO:tasks.rados.rados.0.smithi089.stdout:993: oid 35 updating version 0 to 68 2024-04-22T18:39:37.227 INFO:tasks.rados.rados.0.smithi089.stdout:993: oid 35 updating version 68 to 70 2024-04-22T18:39:37.227 INFO:tasks.rados.rados.0.smithi089.stdout:993: oid 35 version 70 is already newer than 69 2024-04-22T18:39:37.227 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 70 (ObjNum 162 snap 0 seq_num 162) dirty exists 2024-04-22T18:39:37.227 INFO:tasks.rados.rados.0.smithi089.stdout:993: left oid 35 (ObjNum 162 snap 0 seq_num 162) 2024-04-22T18:39:37.227 INFO:tasks.rados.rados.0.smithi089.stdout:990: finishing write tid 2 to smithi08937929-27 2024-04-22T18:39:37.227 INFO:tasks.rados.rados.0.smithi089.stdout:991: finishing write tid 3 to smithi08937929-7 2024-04-22T18:39:37.227 INFO:tasks.rados.rados.0.smithi089.stdout:990: finishing write tid 3 to smithi08937929-27 2024-04-22T18:39:37.227 INFO:tasks.rados.rados.0.smithi089.stdout:1002: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2024-04-22T18:39:37.228 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:37.229+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:37.232 INFO:tasks.rados.rados.0.smithi089.stdout:1002: writing smithi08937929-11 from 406769 to 1016765 tid 1 2024-04-22T18:39:37.238 INFO:tasks.rados.rados.0.smithi089.stdout:1002: writing smithi08937929-11 from 1439434 to 1873148 tid 2 2024-04-22T18:39:37.244 INFO:tasks.rados.rados.0.smithi089.stdout:1002: writing smithi08937929-11 from 2303522 to 2806769 tid 3 2024-04-22T18:39:37.244 INFO:tasks.rados.rados.0.smithi089.stdout:993: done (11 left) 2024-04-22T18:39:37.244 INFO:tasks.rados.rados.0.smithi089.stdout:1003: read oid 32 snap -1 2024-04-22T18:39:37.244 INFO:tasks.rados.rados.0.smithi089.stdout:1003: expect (ObjNum 157 snap 0 seq_num 157) 2024-04-22T18:39:37.244 INFO:tasks.rados.rados.0.smithi089.stdout:1004: write (excl) oid 21 current snap is 0 2024-04-22T18:39:37.244 INFO:tasks.rados.rados.0.smithi089.stdout:1004: seq_num 168 ranges {423576=401598,1497916=693725} 2024-04-22T18:39:37.249 INFO:tasks.rados.rados.0.smithi089.stdout:1004: writing smithi08937929-21 from 423576 to 825174 tid 1 2024-04-22T18:39:37.254 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:37.254 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:37.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:37.254 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:27 4.d e27/20 4:bee1184f:::smithi08937929-7:head v 27'3, mlcod=0'0)", 2024-04-22T18:39:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.889495603999997, 2024-04-22T18:39:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101135974, 2024-04-22T18:39:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365186+0000", 2024-04-22T18:39:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365194+0000", 2024-04-22T18:39:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3289999999999994e-06 2024-04-22T18:39:37.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365200+0000", 2024-04-22T18:39:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6590000000000004e-06 2024-04-22T18:39:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465352+0000", 2024-04-22T18:39:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10015208 2024-04-22T18:39:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465511+0000", 2024-04-22T18:39:37.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015880000000000001 2024-04-22T18:39:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465570+0000", 2024-04-22T18:39:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8952000000000001e-05 2024-04-22T18:39:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466292+0000", 2024-04-22T18:39:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072212100000000005 2024-04-22T18:39:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466323+0000", 2024-04-22T18:39:37.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0899999999999999e-05 2024-04-22T18:39:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:47 3.0 e30/19 3:30f16451:::smithi08937929-50 low_tier:head v 30'20, mlcod=29'18)", 2024-04-22T18:39:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.656579587000003, 2024-04-22T18:39:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100803438, 2024-04-22T18:39:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598108+0000", 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9119999999999997e-06 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598118+0000", 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0138e-05 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598125+0000", 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3010000000000001e-06 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698224+0000", 2024-04-22T18:39:37.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099221 2024-04-22T18:39:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698239+0000", 2024-04-22T18:39:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5053e-05 2024-04-22T18:39:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698875+0000", 2024-04-22T18:39:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063560699999999995 2024-04-22T18:39:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698906+0000", 2024-04-22T18:39:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1547999999999997e-05 2024-04-22T18:39:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.5.13:5 3.0 e30/19 3:1ab765eb:::smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=30'21)", 2024-04-22T18:39:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.052261463999997, 2024-04-22T18:39:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101014042, 2024-04-22T18:39:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.263 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202428+0000", 2024-04-22T18:39:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5889999999999997e-06 2024-04-22T18:39:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202446+0000", 2024-04-22T18:39:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8176999999999999e-05 2024-04-22T18:39:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202456+0000", 2024-04-22T18:39:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-04-22T18:39:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302599+0000", 2024-04-22T18:39:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100142759 2024-04-22T18:39:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302628+0000", 2024-04-22T18:39:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9507999999999999e-05 2024-04-22T18:39:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303350+0000", 2024-04-22T18:39:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072151599999999995 2024-04-22T18:39:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303435+0000", 2024-04-22T18:39:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5486000000000001e-05 2024-04-22T18:39:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:53 3.0 e31/19 3:25a201b0:::smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 31'24, mlcod=30'22)", 2024-04-22T18:39:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.851619130000003, 2024-04-22T18:39:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100751671, 2024-04-22T18:39:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:37.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403062+0000", 2024-04-22T18:39:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403070+0000", 2024-04-22T18:39:37.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8499999999999994e-06 2024-04-22T18:39:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403079+0000", 2024-04-22T18:39:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4620000000000002e-06 2024-04-22T18:39:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403087+0000", 2024-04-22T18:39:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4789999999999999e-06 2024-04-22T18:39:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503186+0000", 2024-04-22T18:39:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099271 2024-04-22T18:39:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503207+0000", 2024-04-22T18:39:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0973999999999999e-05 2024-04-22T18:39:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503773+0000", 2024-04-22T18:39:37.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056557600000000003 2024-04-22T18:39:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503815+0000", 2024-04-22T18:39:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2441999999999998e-05 2024-04-22T18:39:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:73 3.0 e34/19 3:061a7a63:::392f44ec26aa22e85be5f08273a7e45656f3bf4a4854038391a88e880ec01077:head v 34'31, mlcod=31'29)", 2024-04-22T18:39:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:37.272 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.796755681999997, 2024-04-22T18:39:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103503311, 2024-04-22T18:39:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458193+0000", 2024-04-22T18:39:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026628900000000001 2024-04-22T18:39:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458208+0000", 2024-04-22T18:39:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4780000000000001e-05 2024-04-22T18:39:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458217+0000", 2024-04-22T18:39:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4909999999999994e-06 2024-04-22T18:39:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459162+0000", 2024-04-22T18:39:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094440500000000005 2024-04-22T18:39:37.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459185+0000", 2024-04-22T18:39:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3685000000000001e-05 2024-04-22T18:39:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561394+0000", 2024-04-22T18:39:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10220874100000001 2024-04-22T18:39:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561430+0000", 2024-04-22T18:39:37.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6304999999999999e-05 2024-04-22T18:39:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:75 3.0 e34/19 3:18418e0f:::b7c35f165c374d5e075dde9ac2607e47ace3b0a1e2343ec6efb171380f725eea:head v 34'32, mlcod=31'29)", 2024-04-22T18:39:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.795742408999999, 2024-04-22T18:39:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104798078, 2024-04-22T18:39:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458939+0000", 2024-04-22T18:39:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459092+0000", 2024-04-22T18:39:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153838 2024-04-22T18:39:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459107+0000", 2024-04-22T18:39:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4703e-05 2024-04-22T18:39:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459115+0000", 2024-04-22T18:39:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-04-22T18:39:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560093+0000", 2024-04-22T18:39:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10097840499999999 2024-04-22T18:39:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560109+0000", 2024-04-22T18:39:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5631e-05 2024-04-22T18:39:37.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563706+0000", 2024-04-22T18:39:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035972059999999999 2024-04-22T18:39:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563738+0000", 2024-04-22T18:39:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2184000000000002e-05 2024-04-22T18:39:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.281 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:77 3.0 e34/19 3:28387c9f:::b6d7840f02d561da04e398fbf94d91b85e9066b426e48baf3b11dd4a4f6096c0:head v 34'33, mlcod=31'29)", 2024-04-22T18:39:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.794792329000003, 2024-04-22T18:39:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104186079, 2024-04-22T18:39:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459889+0000", 2024-04-22T18:39:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459963+0000", 2024-04-22T18:39:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3732999999999994e-05 2024-04-22T18:39:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459973+0000", 2024-04-22T18:39:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0285999999999999e-05 2024-04-22T18:39:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459987+0000", 2024-04-22T18:39:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3675999999999999e-05 2024-04-22T18:39:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560515+0000", 2024-04-22T18:39:37.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10052781299999999 2024-04-22T18:39:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560538+0000", 2024-04-22T18:39:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3861000000000001e-05 2024-04-22T18:39:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564044+0000", 2024-04-22T18:39:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035052009999999999 2024-04-22T18:39:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564076+0000", 2024-04-22T18:39:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2737000000000002e-05 2024-04-22T18:39:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:78 3.0 e34/19 3:3a5582ac:::bdd92f7deaa2d50e204891f64f9043469ba687d78b218ed518f2483ba3486a99:head v 34'34, mlcod=31'29)", 2024-04-22T18:39:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.794126210999998, 2024-04-22T18:39:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103547263, 2024-04-22T18:39:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.286 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460555+0000", 2024-04-22T18:39:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460861+0000", 2024-04-22T18:39:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000305958 2024-04-22T18:39:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460874+0000", 2024-04-22T18:39:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2639e-05 2024-04-22T18:39:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460881+0000", 2024-04-22T18:39:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2570000000000001e-06 2024-04-22T18:39:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560990+0000", 2024-04-22T18:39:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010883199999999 2024-04-22T18:39:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561011+0000", 2024-04-22T18:39:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1319e-05 2024-04-22T18:39:37.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564079+0000", 2024-04-22T18:39:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00306815 2024-04-22T18:39:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564104+0000", 2024-04-22T18:39:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4172000000000002e-05 2024-04-22T18:39:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:39:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.793510916999999, 2024-04-22T18:39:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:39:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:39:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:39:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:39:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:39:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:39:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:39:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:39:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:39:37.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:39:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:39:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:39:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:39:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:39:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:39:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:39:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:39:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.916837094999998, 2024-04-22T18:39:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:39:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.295 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:39:37.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:39:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:39:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:39:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:39:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:39:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:39:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:39:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:39:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:39:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:39:37.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:39:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:39:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:39:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:39:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1112 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'6, mlcod=29'4)", 2024-04-22T18:39:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:12.585713+0000", 2024-04-22T18:39:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.668969187999998, 2024-04-22T18:39:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046198259999999996, 2024-04-22T18:39:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.585713+0000", 2024-04-22T18:39:37.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.585713+0000", 2024-04-22T18:39:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.585712+0000", 2024-04-22T18:39:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586020+0000", 2024-04-22T18:39:37.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030764900000000001 2024-04-22T18:39:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586031+0000", 2024-04-22T18:39:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0883000000000001e-05 2024-04-22T18:39:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586037+0000", 2024-04-22T18:39:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1260000000000003e-06 2024-04-22T18:39:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586076+0000", 2024-04-22T18:39:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8841999999999998e-05 2024-04-22T18:39:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586098+0000", 2024-04-22T18:39:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2279000000000001e-05 2024-04-22T18:39:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.590302+0000", 2024-04-22T18:39:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042039419999999996 2024-04-22T18:39:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.590333+0000", 2024-04-22T18:39:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1170000000000001e-05 2024-04-22T18:39:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1131 3.0 e41/19 3:01bdda42:::4a159391123755d93670b0d142ca36f3ee9a35236f6c4396c2aa92521a42a06b:head v 41'50, mlcod=41'48)", 2024-04-22T18:39:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.077090250000001, 2024-04-22T18:39:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101639456, 2024-04-22T18:39:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.304 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177590+0000", 2024-04-22T18:39:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:37.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177880+0000", 2024-04-22T18:39:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029001 2024-04-22T18:39:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177893+0000", 2024-04-22T18:39:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-04-22T18:39:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177899+0000", 2024-04-22T18:39:37.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1619999999999996e-06 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.278054+0000", 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100154929 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.278076+0000", 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1308e-05 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.279187+0000", 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011109169999999999 2024-04-22T18:39:37.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.279232+0000", 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5188000000000002e-05 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1159 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'10, mlcod=41'7)", 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.452702+0000", 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.801980700000001, 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050095280000000001, 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.308 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452702+0000", 2024-04-22T18:39:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452702+0000", 2024-04-22T18:39:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452701+0000", 2024-04-22T18:39:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452976+0000", 2024-04-22T18:39:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027566 2024-04-22T18:39:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452989+0000", 2024-04-22T18:39:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2615e-05 2024-04-22T18:39:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452998+0000", 2024-04-22T18:39:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7399999999999993e-06 2024-04-22T18:39:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.453040+0000", 2024-04-22T18:39:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2395000000000003e-05 2024-04-22T18:39:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.453060+0000", 2024-04-22T18:39:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0332000000000001e-05 2024-04-22T18:39:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.457682+0000", 2024-04-22T18:39:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046215010000000001 2024-04-22T18:39:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.457711+0000", 2024-04-22T18:39:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9490999999999999e-05 2024-04-22T18:39:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.312 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1160 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'11, mlcod=41'8)", 2024-04-22T18:39:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.555212+0000", 2024-04-22T18:39:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.699470367, 2024-04-22T18:39:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004833586, 2024-04-22T18:39:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555212+0000", 2024-04-22T18:39:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555212+0000", 2024-04-22T18:39:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555209+0000", 2024-04-22T18:39:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555369+0000", 2024-04-22T18:39:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015995000000000001 2024-04-22T18:39:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555383+0000", 2024-04-22T18:39:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3558e-05 2024-04-22T18:39:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555389+0000", 2024-04-22T18:39:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2759999999999997e-06 2024-04-22T18:39:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556393+0000", 2024-04-22T18:39:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001003629 2024-04-22T18:39:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556418+0000", 2024-04-22T18:39:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4915000000000001e-05 2024-04-22T18:39:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.560010+0000", 2024-04-22T18:39:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003591782 2024-04-22T18:39:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.560046+0000", 2024-04-22T18:39:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6285000000000002e-05 2024-04-22T18:39:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1161 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'12, mlcod=41'9)", 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.556330+0000", 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.698352949, 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099789016999999994, 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556330+0000", 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556330+0000", 2024-04-22T18:39:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556328+0000", 2024-04-22T18:39:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556335+0000", 2024-04-22T18:39:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9770000000000003e-06 2024-04-22T18:39:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556346+0000", 2024-04-22T18:39:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0118999999999999e-05 2024-04-22T18:39:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556353+0000", 2024-04-22T18:39:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3730000000000004e-06 2024-04-22T18:39:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.655508+0000", 2024-04-22T18:39:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099155219000000003 2024-04-22T18:39:37.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.655536+0000", 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-05 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.656078+0000", 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054138999999999999 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.656119+0000", 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1026000000000003e-05 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.320 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:125 3.0 e43/19 3:376bc4bb:::8f967acf6e5865f87cd7cada681f96ff59d5fc3f243a710c00c052aacb952df3:head v 43'60, mlcod=42'57)", 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.407116867999999, 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100905994, 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847565+0000", 2024-04-22T18:39:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847610+0000", 2024-04-22T18:39:37.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4984000000000002e-05 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847619+0000", 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2189999999999993e-06 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947667+0000", 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100047939 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947716+0000", 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9228999999999999e-05 2024-04-22T18:39:37.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947738+0000", 2024-04-22T18:39:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1963999999999998e-05 2024-04-22T18:39:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.948440+0000", 2024-04-22T18:39:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070173999999999996 2024-04-22T18:39:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.948472+0000", 2024-04-22T18:39:37.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1872999999999998e-05 2024-04-22T18:39:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1298 4.d e43/20 4:bc165c76:::smithi08937929-30:head v 43'17, mlcod=32'12)", 2024-04-22T18:39:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:18.062040+0000", 2024-04-22T18:39:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.192643064999999, 2024-04-22T18:39:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099368133999999997, 2024-04-22T18:39:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062040+0000", 2024-04-22T18:39:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062040+0000", 2024-04-22T18:39:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062039+0000", 2024-04-22T18:39:37.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062056+0000", 2024-04-22T18:39:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6911000000000001e-05 2024-04-22T18:39:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062060+0000", 2024-04-22T18:39:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3000000000000003e-06 2024-04-22T18:39:37.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062065+0000", 2024-04-22T18:39:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1150000000000002e-06 2024-04-22T18:39:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.160817+0000", 2024-04-22T18:39:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098751539999999999 2024-04-22T18:39:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.160845+0000", 2024-04-22T18:39:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7908000000000001e-05 2024-04-22T18:39:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.161378+0000", 2024-04-22T18:39:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053300799999999996 2024-04-22T18:39:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.161408+0000", 2024-04-22T18:39:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.989e-05 2024-04-22T18:39:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.332 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:79 3.0 e44/19 3:05e6fa13:::smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 44'64, mlcod=44'61)", 2024-04-22T18:39:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.883835572000001, 2024-04-22T18:39:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10093532199999999, 2024-04-22T18:39:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370846+0000", 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370857+0000", 2024-04-22T18:39:37.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1155000000000001e-05 2024-04-22T18:39:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370866+0000", 2024-04-22T18:39:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8769999999999999e-06 2024-04-22T18:39:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.470937+0000", 2024-04-22T18:39:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100071204 2024-04-22T18:39:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.470986+0000", 2024-04-22T18:39:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8556000000000001e-05 2024-04-22T18:39:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471014+0000", 2024-04-22T18:39:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7982000000000001e-05 2024-04-22T18:39:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471750+0000", 2024-04-22T18:39:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073598100000000003 2024-04-22T18:39:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471782+0000", 2024-04-22T18:39:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339000000000003e-05 2024-04-22T18:39:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1689 3.0 e47/19 3:1d4bcadf:::e7eb8db0531d9f456e91fe87bc79e47ebd80459f1783dc2a5e53d4822cd34f49:head v 47'85, mlcod=47'83)", 2024-04-22T18:39:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:39:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.5457946370000002, 2024-04-22T18:39:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102152033, 2024-04-22T18:39:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:39:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:39:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708885+0000", 2024-04-22T18:39:37.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709414+0000", 2024-04-22T18:39:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000528459 2024-04-22T18:39:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709428+0000", 2024-04-22T18:39:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4745e-05 2024-04-22T18:39:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709436+0000", 2024-04-22T18:39:37.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0830000000000004e-06 2024-04-22T18:39:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.809587+0000", 2024-04-22T18:39:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100150641 2024-04-22T18:39:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.809603+0000", 2024-04-22T18:39:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6061000000000001e-05 2024-04-22T18:39:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.811013+0000", 2024-04-22T18:39:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0014094680000000001 2024-04-22T18:39:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.811040+0000", 2024-04-22T18:39:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7474000000000002e-05 2024-04-22T18:39:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.341 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1729 3.0 e47/19 3:27b110a1:::34336650543f8f8bff225091f9c8c14678dc47baabcb4d8371890fc935dbc3e4:head v 47'89, mlcod=47'87)", 2024-04-22T18:39:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:39:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.776740867, 2024-04-22T18:39:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102111412, 2024-04-22T18:39:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:39:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:39:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477940+0000", 2024-04-22T18:39:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478305+0000", 2024-04-22T18:39:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036512200000000001 2024-04-22T18:39:37.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478320+0000", 2024-04-22T18:39:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5475999999999999e-05 2024-04-22T18:39:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478326+0000", 2024-04-22T18:39:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.302e-06 2024-04-22T18:39:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.578536+0000", 2024-04-22T18:39:37.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100209436 2024-04-22T18:39:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.578575+0000", 2024-04-22T18:39:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9223e-05 2024-04-22T18:39:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.580023+0000", 2024-04-22T18:39:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001447463 2024-04-22T18:39:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:37.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.580053+0000", 2024-04-22T18:39:37.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0534999999999998e-05 2024-04-22T18:39:37.346 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.346 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.346 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.346 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.346 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.346 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:37.347 INFO:tasks.rados.rados.0.smithi089.stdout:1004: writing smithi08937929-21 from 1497916 to 2191641 tid 2 2024-04-22T18:39:37.347 INFO:tasks.rados.rados.0.smithi089.stdout:1005: read oid 10 snap -1 2024-04-22T18:39:37.347 INFO:tasks.rados.rados.0.smithi089.stdout:1005: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:39:37.347 INFO:tasks.rados.rados.0.smithi089.stdout:1006: set_chunk oid 42 target oid 2024-04-22T18:39:37.348 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:37.348 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:37.348+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:37.367 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:39:37.448 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:37.449+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:37.497 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:37.497 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:37.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.497 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:87 3.2 e51/29)", 2024-04-22T18:39:37.497 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:37.437515+0000", 2024-04-22T18:39:37.497 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.061514442000000003, 2024-04-22T18:39:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.061531829000000003, 2024-04-22T18:39:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.437515+0000", 2024-04-22T18:39:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.437515+0000", 2024-04-22T18:39:37.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.437513+0000", 2024-04-22T18:39:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.437525+0000", 2024-04-22T18:39:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2115e-05 2024-04-22T18:39:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.437552+0000", 2024-04-22T18:39:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6981000000000001e-05 2024-04-22T18:39:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.437561+0000", 2024-04-22T18:39:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8310000000000007e-06 2024-04-22T18:39:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:88 3.2 e51/29)", 2024-04-22T18:39:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:37.437963+0000", 2024-04-22T18:39:37.500 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.061066286999999997, 2024-04-22T18:39:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.061111784000000002, 2024-04-22T18:39:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.437963+0000", 2024-04-22T18:39:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.437963+0000", 2024-04-22T18:39:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.437962+0000", 2024-04-22T18:39:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.437968+0000", 2024-04-22T18:39:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0719999999999996e-06 2024-04-22T18:39:37.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.437977+0000", 2024-04-22T18:39:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.952e-06 2024-04-22T18:39:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.437982+0000", 2024-04-22T18:39:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5190000000000003e-06 2024-04-22T18:39:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:37.503 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:39:37.504 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 2 2024-04-22T18:39:37.504 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:37.506 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:37.510 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:37.510 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:37.527 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:37.527 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:37.545 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:37.545 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:37.549 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:37.550+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:37.561 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:37.561 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:37.578 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:37.578 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:37.595 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:37.595 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:37.611 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:37.612 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:37.614 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:37.614 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:37.623 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:37.629 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:37.629 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:37.640 INFO:tasks.rados.rados.0.smithi089.stdout:1006: set_chunk oid 42 offset: 1493504 length: 1585152 target oid aba56772c3e6ce16de0d29b109b9d2f300d44dc7efc837466685ff858c4ab4c1 offset: 0 snap -1 2024-04-22T18:39:37.640 INFO:tasks.rados.rados.0.smithi089.stdout:1007: read oid 50 snap -1 2024-04-22T18:39:37.640 INFO:tasks.rados.rados.0.smithi089.stdout:1007: expect (ObjNum 148 snap 0 seq_num 148) 2024-04-22T18:39:37.640 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:37.640 INFO:tasks.rados.rados.0.smithi089.stdout:991: finishing write tid 4 to smithi08937929-7 2024-04-22T18:39:37.640 INFO:tasks.rados.rados.0.smithi089.stdout:991: oid 7 updating version 0 to 26 2024-04-22T18:39:37.640 INFO:tasks.rados.rados.0.smithi089.stdout:991: oid 7 updating version 26 to 27 2024-04-22T18:39:37.640 INFO:tasks.rados.rados.0.smithi089.stdout:991: oid 7 updating version 27 to 28 2024-04-22T18:39:37.640 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 28 (ObjNum 161 snap 0 seq_num 161) dirty exists 2024-04-22T18:39:37.640 INFO:tasks.rados.rados.0.smithi089.stdout:991: left oid 7 (ObjNum 161 snap 0 seq_num 161) 2024-04-22T18:39:37.640 INFO:tasks.rados.rados.0.smithi089.stdout:991: done (15 left) 2024-04-22T18:39:37.641 INFO:tasks.rados.rados.0.smithi089.stdout:1008: set_chunk oid 5 target oid 2024-04-22T18:39:37.641 INFO:tasks.rados.rados.0.smithi089.stdout:1008: set_chunk oid 5 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:39:37.641 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:37.641 INFO:tasks.rados.rados.0.smithi089.stdout:990: finishing write tid 4 to smithi08937929-27 2024-04-22T18:39:37.641 INFO:tasks.rados.rados.0.smithi089.stdout:990: oid 27 updating version 0 to 71 2024-04-22T18:39:37.641 INFO:tasks.rados.rados.0.smithi089.stdout:990: oid 27 version 71 is already newer than 70 2024-04-22T18:39:37.641 INFO:tasks.rados.rados.0.smithi089.stdout:990: oid 27 updating version 71 to 72 2024-04-22T18:39:37.641 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 72 (ObjNum 160 snap 0 seq_num 160) dirty exists 2024-04-22T18:39:37.641 INFO:tasks.rados.rados.0.smithi089.stdout:990: left oid 27 (ObjNum 160 snap 0 seq_num 160) 2024-04-22T18:39:37.641 INFO:tasks.rados.rados.0.smithi089.stdout:994: finishing write tid 2 to smithi08937929-14 2024-04-22T18:39:37.641 INFO:tasks.rados.rados.0.smithi089.stdout:994: finishing write tid 3 to smithi08937929-14 2024-04-22T18:39:37.641 INFO:tasks.rados.rados.0.smithi089.stdout:990: done (15 left) 2024-04-22T18:39:37.641 INFO:tasks.rados.rados.0.smithi089.stdout:1009: read oid 37 snap -1 2024-04-22T18:39:37.641 INFO:tasks.rados.rados.0.smithi089.stdout:1009: expect (ObjNum 131 snap 0 seq_num 131) 2024-04-22T18:39:37.642 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:37.642 INFO:tasks.rados.rados.0.smithi089.stdout:997: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-22T18:39:37.661 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:37.662+0000 7fec57bee640 -1 osd.7 51 handle_osd_map injecting map crc failure 2024-04-22T18:39:37.664 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:39:37.664 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:39:37.665 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-22T18:39:37.665 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-04-22T18:39:37.666 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:39:37.666 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:37.667+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:37.737 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:37.737 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:37.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:37.750 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:37.752+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:37.799 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-04-22T18:39:37.819 INFO:tasks.rados.rados.0.smithi089.stdout:994: finishing write tid 4 to smithi08937929-14 2024-04-22T18:39:37.819 INFO:tasks.rados.rados.0.smithi089.stdout:994: oid 14 updating version 0 to 51 2024-04-22T18:39:37.819 INFO:tasks.rados.rados.0.smithi089.stdout:994: oid 14 version 51 is already newer than 49 2024-04-22T18:39:37.819 INFO:tasks.rados.rados.0.smithi089.stdout:994: oid 14 version 51 is already newer than 50 2024-04-22T18:39:37.819 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 51 (ObjNum 163 snap 0 seq_num 163) dirty exists 2024-04-22T18:39:37.819 INFO:tasks.rados.rados.0.smithi089.stdout:994: left oid 14 (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:39:37.819 INFO:tasks.rados.rados.0.smithi089.stdout:996: finishing write tid 1 to smithi08937929-39 2024-04-22T18:39:37.819 INFO:tasks.rados.rados.0.smithi089.stdout:996: finishing write tid 2 to smithi08937929-39 2024-04-22T18:39:37.819 INFO:tasks.rados.rados.0.smithi089.stdout:996: finishing write tid 3 to smithi08937929-39 2024-04-22T18:39:37.819 INFO:tasks.rados.rados.0.smithi089.stdout:994: done (15 left) 2024-04-22T18:39:37.819 INFO:tasks.rados.rados.0.smithi089.stdout:997: done (14 left) 2024-04-22T18:39:37.819 INFO:tasks.rados.rados.0.smithi089.stdout:1010: read oid 23 snap -1 2024-04-22T18:39:37.820 INFO:tasks.rados.rados.0.smithi089.stdout:1010: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:39:37.820 INFO:tasks.rados.rados.0.smithi089.stdout:1011: tier_evict oid 44 2024-04-22T18:39:37.820 INFO:tasks.rados.rados.0.smithi089.stdout:1011: tier_evict oid 44 snap -1 2024-04-22T18:39:37.820 INFO:tasks.rados.rados.0.smithi089.stdout:996: finishing write tid 4 to smithi08937929-39 2024-04-22T18:39:37.820 INFO:tasks.rados.rados.0.smithi089.stdout:996: oid 39 updating version 0 to 75 2024-04-22T18:39:37.820 INFO:tasks.rados.rados.0.smithi089.stdout:996: oid 39 version 75 is already newer than 74 2024-04-22T18:39:37.820 INFO:tasks.rados.rados.0.smithi089.stdout:996: oid 39 version 75 is already newer than 73 2024-04-22T18:39:37.820 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 75 (ObjNum 164 snap 0 seq_num 164) dirty exists 2024-04-22T18:39:37.820 INFO:tasks.rados.rados.0.smithi089.stdout:996: left oid 39 (ObjNum 164 snap 0 seq_num 164) 2024-04-22T18:39:37.820 INFO:tasks.rados.rados.0.smithi089.stdout:995: finishing set_chunk to oid 49 2024-04-22T18:39:37.820 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 774656 length: 1175040 2024-04-22T18:39:37.820 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-22T18:39:37.820 INFO:tasks.rados.rados.0.smithi089.stdout:998: finishing write tid 1 to smithi08937929-18 2024-04-22T18:39:37.820 INFO:tasks.rados.rados.0.smithi089.stdout:998: finishing write tid 2 to smithi08937929-18 2024-04-22T18:39:37.820 INFO:tasks.rados.rados.0.smithi089.stdout:998: finishing write tid 3 to smithi08937929-18 2024-04-22T18:39:37.821 INFO:tasks.rados.rados.0.smithi089.stdout:1000: expect (ObjNum 156 snap 0 seq_num 156) 2024-04-22T18:39:37.851 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:37.852+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:37.858 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:37.858 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:37.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:37.952 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-04-22T18:39:37.953 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:37.954+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:37.977 INFO:tasks.rados.rados.0.smithi089.stdout:999: finishing write tid 1 to smithi08937929-45 2024-04-22T18:39:37.977 INFO:tasks.rados.rados.0.smithi089.stdout:999: finishing write tid 2 to smithi08937929-45 2024-04-22T18:39:37.977 INFO:tasks.rados.rados.0.smithi089.stdout:999: finishing write tid 3 to smithi08937929-45 2024-04-22T18:39:37.978 INFO:tasks.rados.rados.0.smithi089.stdout:999: finishing write tid 4 to smithi08937929-45 2024-04-22T18:39:37.978 INFO:tasks.rados.rados.0.smithi089.stdout:999: oid 45 updating version 0 to 56 2024-04-22T18:39:37.978 INFO:tasks.rados.rados.0.smithi089.stdout:999: oid 45 updating version 56 to 57 2024-04-22T18:39:37.978 INFO:tasks.rados.rados.0.smithi089.stdout:999: oid 45 version 57 is already newer than 55 2024-04-22T18:39:37.978 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 57 (ObjNum 166 snap 0 seq_num 166) dirty exists 2024-04-22T18:39:37.978 INFO:tasks.rados.rados.0.smithi089.stdout:999: left oid 45 (ObjNum 166 snap 0 seq_num 166) 2024-04-22T18:39:37.978 INFO:tasks.rados.rados.0.smithi089.stdout:998: finishing write tid 4 to smithi08937929-18 2024-04-22T18:39:37.978 INFO:tasks.rados.rados.0.smithi089.stdout:995: done (15 left) 2024-04-22T18:39:37.978 INFO:tasks.rados.rados.0.smithi089.stdout:996: done (14 left) 2024-04-22T18:39:37.978 INFO:tasks.rados.rados.0.smithi089.stdout:999: done (13 left) 2024-04-22T18:39:37.978 INFO:tasks.rados.rados.0.smithi089.stdout:1000: done (12 left) 2024-04-22T18:39:37.978 INFO:tasks.rados.rados.0.smithi089.stdout:1012: read oid 4 snap -1 2024-04-22T18:39:37.979 INFO:tasks.rados.rados.0.smithi089.stdout:1012: expect (ObjNum 120 snap 0 seq_num 120) 2024-04-22T18:39:37.979 INFO:tasks.rados.rados.0.smithi089.stdout:1013: read oid 1 snap -1 2024-04-22T18:39:37.979 INFO:tasks.rados.rados.0.smithi089.stdout:1013: expect (ObjNum 57 snap 0 seq_num 57) 2024-04-22T18:39:37.979 INFO:tasks.rados.rados.0.smithi089.stdout:1014: read oid 40 snap -1 2024-04-22T18:39:37.979 INFO:tasks.rados.rados.0.smithi089.stdout:1014: expect (ObjNum 58 snap 0 seq_num 58) 2024-04-22T18:39:37.979 INFO:tasks.rados.rados.0.smithi089.stdout:1015: read oid 29 snap -1 2024-04-22T18:39:37.979 INFO:tasks.rados.rados.0.smithi089.stdout:1015: expect (ObjNum 149 snap 0 seq_num 149) 2024-04-22T18:39:37.979 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:37.979 INFO:tasks.rados.rados.0.smithi089.stdout:998: finishing write tid 5 to smithi08937929-18 2024-04-22T18:39:37.979 INFO:tasks.rados.rados.0.smithi089.stdout:998: oid 18 updating version 0 to 53 2024-04-22T18:39:37.979 INFO:tasks.rados.rados.0.smithi089.stdout:998: oid 18 updating version 53 to 55 2024-04-22T18:39:37.979 INFO:tasks.rados.rados.0.smithi089.stdout:998: oid 18 version 55 is already newer than 54 2024-04-22T18:39:37.979 INFO:tasks.rados.rados.0.smithi089.stdout:998: oid 18 version 55 is already newer than 52 2024-04-22T18:39:37.980 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 55 (ObjNum 165 snap 0 seq_num 165) dirty exists 2024-04-22T18:39:37.980 INFO:tasks.rados.rados.0.smithi089.stdout:998: left oid 18 (ObjNum 165 snap 0 seq_num 165) 2024-04-22T18:39:37.980 INFO:tasks.rados.rados.0.smithi089.stdout:1004: finishing write tid 1 to smithi08937929-21 2024-04-22T18:39:37.980 INFO:tasks.rados.rados.0.smithi089.stdout:1004: finishing write tid 2 to smithi08937929-21 2024-04-22T18:39:37.980 INFO:tasks.rados.rados.0.smithi089.stdout:1004: finishing write tid 3 to smithi08937929-21 2024-04-22T18:39:37.980 INFO:tasks.rados.rados.0.smithi089.stdout:1003: expect (ObjNum 157 snap 0 seq_num 157) 2024-04-22T18:39:37.980 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:37.978+0000 7f3d00e8c640 -1 osd.6 51 handle_osd_map injecting map crc failure 2024-04-22T18:39:38.008 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:38.009 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:38.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:38.053 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:38.055+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:38.085 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-04-22T18:39:38.120 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:38.120 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:38.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:38.172 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:38.173+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:38.195 INFO:tasks.rados.rados.0.smithi089.stdout:1002: finishing write tid 1 to smithi08937929-11 2024-04-22T18:39:38.195 INFO:tasks.rados.rados.0.smithi089.stdout:1002: finishing write tid 2 to smithi08937929-11 2024-04-22T18:39:38.195 INFO:tasks.rados.rados.0.smithi089.stdout:998: done (15 left) 2024-04-22T18:39:38.196 INFO:tasks.rados.rados.0.smithi089.stdout:1003: done (14 left) 2024-04-22T18:39:38.196 INFO:tasks.rados.rados.0.smithi089.stdout:1016: set_chunk oid 6 target oid 2024-04-22T18:39:38.232 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:38.233 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:38.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:38.255 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:38.256+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:38.257 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-04-22T18:39:38.356 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:38.357+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:38.383 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:38.384 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:38.393 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:38.400 INFO:tasks.rados.rados.0.smithi089.stdout:1016: set_chunk oid 6 offset: 88576 length: 155136 target oid 00c92e678daa60988c0cbc4f4fdc35deae9741a9ef20a27c365e43c3477f2199 offset: 0 snap -1 2024-04-22T18:39:38.400 INFO:tasks.rados.rados.0.smithi089.stdout:1017: read oid 22 snap -1 2024-04-22T18:39:38.400 INFO:tasks.rados.rados.0.smithi089.stdout:1017: expect (ObjNum 135 snap 0 seq_num 135) 2024-04-22T18:39:38.400 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:38.400 INFO:tasks.rados.rados.0.smithi089.stdout:1002: finishing write tid 3 to smithi08937929-11 2024-04-22T18:39:38.400 INFO:tasks.rados.rados.0.smithi089.stdout:1004: finishing write tid 4 to smithi08937929-21 2024-04-22T18:39:38.400 INFO:tasks.rados.rados.0.smithi089.stdout:1004: oid 21 updating version 0 to 38 2024-04-22T18:39:38.400 INFO:tasks.rados.rados.0.smithi089.stdout:1004: oid 21 version 38 is already newer than 37 2024-04-22T18:39:38.400 INFO:tasks.rados.rados.0.smithi089.stdout:1004: oid 21 updating version 38 to 39 2024-04-22T18:39:38.401 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 39 (ObjNum 168 snap 0 seq_num 168) dirty exists 2024-04-22T18:39:38.401 INFO:tasks.rados.rados.0.smithi089.stdout:1004: left oid 21 (ObjNum 168 snap 0 seq_num 168) 2024-04-22T18:39:38.401 INFO:tasks.rados.rados.0.smithi089.stdout:1001: finishing set_chunk to oid 26 2024-04-22T18:39:38.401 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 63 (ObjNum 140 snap 0 seq_num 140) dirty exists 2024-04-22T18:39:38.401 INFO:tasks.rados.rados.0.smithi089.stdout:1002: finishing write tid 4 to smithi08937929-11 2024-04-22T18:39:38.401 INFO:tasks.rados.rados.0.smithi089.stdout:1001: done (15 left) 2024-04-22T18:39:38.401 INFO:tasks.rados.rados.0.smithi089.stdout:1004: done (14 left) 2024-04-22T18:39:38.401 INFO:tasks.rados.rados.0.smithi089.stdout:1018: tier_flush oid 16 2024-04-22T18:39:38.401 INFO:tasks.rados.rados.0.smithi089.stdout:1018: tier_flush oid 16 snap -1 2024-04-22T18:39:38.401 INFO:tasks.rados.rados.0.smithi089.stdout:1002: finishing write tid 5 to smithi08937929-11 2024-04-22T18:39:38.401 INFO:tasks.rados.rados.0.smithi089.stdout:1002: oid 11 updating version 0 to 30 2024-04-22T18:39:38.401 INFO:tasks.rados.rados.0.smithi089.stdout:1002: oid 11 updating version 30 to 32 2024-04-22T18:39:38.401 INFO:tasks.rados.rados.0.smithi089.stdout:1002: oid 11 version 32 is already newer than 29 2024-04-22T18:39:38.401 INFO:tasks.rados.rados.0.smithi089.stdout:1002: oid 11 version 32 is already newer than 31 2024-04-22T18:39:38.401 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 32 (ObjNum 167 snap 0 seq_num 167) dirty exists 2024-04-22T18:39:38.402 INFO:tasks.rados.rados.0.smithi089.stdout:1002: left oid 11 (ObjNum 167 snap 0 seq_num 167) 2024-04-22T18:39:38.402 INFO:tasks.rados.rados.0.smithi089.stdout:1002: done (14 left) 2024-04-22T18:39:38.402 INFO:tasks.rados.rados.0.smithi089.stdout:1019: write oid 33 current snap is 0 2024-04-22T18:39:38.402 INFO:tasks.rados.rados.0.smithi089.stdout:1005: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:39:38.428 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-04-22T18:39:38.469 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:38.471+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:38.470 INFO:tasks.rados.rados.0.smithi089.stdout:1007: expect (ObjNum 148 snap 0 seq_num 148) 2024-04-22T18:39:38.494 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:38.494 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:38.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:38.557 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:38.558+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:38.595 INFO:tasks.rados.rados.0.smithi089.stdout:1008: finishing set_chunk to oid 5 2024-04-22T18:39:38.595 INFO:tasks.rados.rados.0.smithi089.stdout:1008: got expected ENOENT (src dne) 2024-04-22T18:39:38.595 INFO:tasks.rados.rados.0.smithi089.stdout:1009: expect (ObjNum 131 snap 0 seq_num 131) 2024-04-22T18:39:38.602 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-04-22T18:39:38.607 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:38.607 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:38.616 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:38.658 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:38.659+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:38.717 INFO:tasks.rados.rados.0.smithi089.stdout:1011: got (22) Invalid argument 2024-04-22T18:39:38.718 INFO:tasks.rados.rados.0.smithi089.stdout:1010: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:39:38.757 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:38.757 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:38.758 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:38.759+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:38.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:38.859 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:38.860+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:38.863 INFO:tasks.rados.rados.0.smithi089.stdout:1006: finishing set_chunk to oid 42 2024-04-22T18:39:38.863 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 16 (ObjNum 105 snap 0 seq_num 105) dirty exists 2024-04-22T18:39:38.863 INFO:tasks.rados.rados.0.smithi089.stdout:1015: expect (ObjNum 149 snap 0 seq_num 149) 2024-04-22T18:39:38.866 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:38.866 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:38.876 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:38.960 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:38.961+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:39.006 INFO:tasks.rados.rados.0.smithi089.stdout:1012: expect (ObjNum 120 snap 0 seq_num 120) 2024-04-22T18:39:39.019 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:39.019 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:39.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:39.060 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:39.061+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:39.093 INFO:tasks.rados.rados.0.smithi089.stdout:1014: expect (ObjNum 58 snap 0 seq_num 58) 2024-04-22T18:39:39.161 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:39.162+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:39.169 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:39.169 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:39.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:39.188 INFO:tasks.rados.rados.0.smithi089.stdout:1013: expect (ObjNum 57 snap 0 seq_num 57) 2024-04-22T18:39:39.262 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:39.263+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:39.310 INFO:tasks.rados.rados.0.smithi089.stdout:1016: finishing set_chunk to oid 6 2024-04-22T18:39:39.310 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 6 set_chunk returned error code -95 offset: 88576 length: 155136 2024-04-22T18:39:39.311 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 0 (ObjNum 113 snap 0 seq_num 113) dirty exists 2024-04-22T18:39:39.311 INFO:tasks.rados.rados.0.smithi089.stdout:1017: expect (ObjNum 135 snap 0 seq_num 135) 2024-04-22T18:39:39.319 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:39.320 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:39.329 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:39.370 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:39.371+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:39.432 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:39.433 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:39.443 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:39.461 INFO:tasks.rados.rados.0.smithi089.stdout:1018: got (0) Success 2024-04-22T18:39:39.461 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 0 (ObjNum 154 snap 0 seq_num 154) dirty exists 2024-04-22T18:39:39.461 INFO:tasks.rados.rados.0.smithi089.stdout:1019: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2024-04-22T18:39:39.464 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:39.465+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:39.465 INFO:tasks.rados.rados.0.smithi089.stdout:1019: writing smithi08937929-33 from 440383 to 917230 tid 1 2024-04-22T18:39:39.471 INFO:tasks.rados.rados.0.smithi089.stdout:1019: writing smithi08937929-33 from 1556398 to 1993781 tid 2 2024-04-22T18:39:39.477 INFO:tasks.rados.rados.0.smithi089.stdout:1019: writing smithi08937929-33 from 2474721 to 2840383 tid 3 2024-04-22T18:39:39.477 INFO:tasks.rados.rados.0.smithi089.stdout:1005: done (14 left) 2024-04-22T18:39:39.477 INFO:tasks.rados.rados.0.smithi089.stdout:1006: done (13 left) 2024-04-22T18:39:39.477 INFO:tasks.rados.rados.0.smithi089.stdout:1007: done (12 left) 2024-04-22T18:39:39.477 INFO:tasks.rados.rados.0.smithi089.stdout:1008: done (11 left) 2024-04-22T18:39:39.477 INFO:tasks.rados.rados.0.smithi089.stdout:1009: done (10 left) 2024-04-22T18:39:39.477 INFO:tasks.rados.rados.0.smithi089.stdout:1010: done (9 left) 2024-04-22T18:39:39.477 INFO:tasks.rados.rados.0.smithi089.stdout:1011: done (8 left) 2024-04-22T18:39:39.477 INFO:tasks.rados.rados.0.smithi089.stdout:1012: done (7 left) 2024-04-22T18:39:39.477 INFO:tasks.rados.rados.0.smithi089.stdout:1013: done (6 left) 2024-04-22T18:39:39.478 INFO:tasks.rados.rados.0.smithi089.stdout:1014: done (5 left) 2024-04-22T18:39:39.478 INFO:tasks.rados.rados.0.smithi089.stdout:1015: done (4 left) 2024-04-22T18:39:39.478 INFO:tasks.rados.rados.0.smithi089.stdout:1016: done (3 left) 2024-04-22T18:39:39.478 INFO:tasks.rados.rados.0.smithi089.stdout:1017: done (2 left) 2024-04-22T18:39:39.478 INFO:tasks.rados.rados.0.smithi089.stdout:1018: done (1 left) 2024-04-22T18:39:39.478 INFO:tasks.rados.rados.0.smithi089.stdout:1020: read oid 40 snap -1 2024-04-22T18:39:39.478 INFO:tasks.rados.rados.0.smithi089.stdout:1020: expect (ObjNum 58 snap 0 seq_num 58) 2024-04-22T18:39:39.478 INFO:tasks.rados.rados.0.smithi089.stdout:1021: read oid 30 snap -1 2024-04-22T18:39:39.478 INFO:tasks.rados.rados.0.smithi089.stdout:1021: expect (ObjNum 144 snap 0 seq_num 144) 2024-04-22T18:39:39.478 INFO:tasks.rados.rados.0.smithi089.stdout:1022: tier_evict oid 13 2024-04-22T18:39:39.478 INFO:tasks.rados.rados.0.smithi089.stdout:1022: tier_evict oid 13 snap -1 2024-04-22T18:39:39.478 INFO:tasks.rados.rados.0.smithi089.stdout:1023: read oid 23 snap -1 2024-04-22T18:39:39.478 INFO:tasks.rados.rados.0.smithi089.stdout:1023: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:39:39.478 INFO:tasks.rados.rados.0.smithi089.stdout:1024: set_chunk oid 44 target oid 2024-04-22T18:39:39.544 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:39.544 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:39.553 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:39.565 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:39.566+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:39.580 INFO:tasks.rados.rados.0.smithi089.stdout:1024: set_chunk oid 44 offset: 978944 length: 99328 target oid a4d001d1be578489478d6152f4e751182f7725bad2a7fcfc075bc54e6a7463b0 offset: 0 snap -1 2024-04-22T18:39:39.580 INFO:tasks.rados.rados.0.smithi089.stdout:1025: write oid 43 current snap is 0 2024-04-22T18:39:39.580 INFO:tasks.rados.rados.0.smithi089.stdout:1022: got (22) Invalid argument 2024-04-22T18:39:39.580 INFO:tasks.rados.rados.0.smithi089.stdout:1025: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-04-22T18:39:39.585 INFO:tasks.rados.rados.0.smithi089.stdout:1025: writing smithi08937929-43 from 457190 to 1009286 tid 1 2024-04-22T18:39:39.591 INFO:tasks.rados.rados.0.smithi089.stdout:1025: writing smithi08937929-43 from 1614880 to 2195921 tid 2 2024-04-22T18:39:39.595 INFO:tasks.rados.rados.0.smithi089.stdout:1025: writing smithi08937929-43 from 2785791 to 2857190 tid 3 2024-04-22T18:39:39.595 INFO:tasks.rados.rados.0.smithi089.stdout:1022: done (6 left) 2024-04-22T18:39:39.595 INFO:tasks.rados.rados.0.smithi089.stdout:1026: set_chunk oid 36 target oid 2024-04-22T18:39:39.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:39:39.665 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:39.666+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:39.679 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:39:39.688 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:39.688 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:39.698 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:39.783 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:39.785+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:39.787 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:39.787 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:39.810 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:39.811 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:39.812 INFO:tasks.rados.rados.0.smithi089.stdout:1026: set_chunk oid 36 offset: 186368 length: 2181120 target oid d3d7ffd4ecbc22b7939595dfbb7899264507d1b1925eb40e237082ce0a5fc463 offset: 0 snap -1 2024-04-22T18:39:39.812 INFO:tasks.rados.rados.0.smithi089.stdout:1027: write (excl) oid 6 current snap is 0 2024-04-22T18:39:39.812 INFO:tasks.rados.rados.0.smithi089.stdout:1027: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-04-22T18:39:39.819 INFO:tasks.rados.rados.0.smithi089.stdout:1027: writing smithi08937929-6 from 473997 to 1101342 tid 1 2024-04-22T18:39:39.826 INFO:tasks.rados.rados.0.smithi089.stdout:1027: writing smithi08937929-6 from 1673362 to 2114414 tid 2 2024-04-22T18:39:39.831 INFO:tasks.rados.rados.0.smithi089.stdout:1027: writing smithi08937929-6 from 2529567 to 2873997 tid 3 2024-04-22T18:39:39.831 INFO:tasks.rados.rados.0.smithi089.stdout:1028: write (excl) oid 25 current snap is 0 2024-04-22T18:39:39.831 INFO:tasks.rados.rados.0.smithi089.stdout:1028: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2024-04-22T18:39:39.836 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:39.836 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:39.837 INFO:tasks.rados.rados.0.smithi089.stdout:1028: writing smithi08937929-25 from 490804 to 1193398 tid 1 2024-04-22T18:39:39.843 INFO:tasks.rados.rados.0.smithi089.stdout:1028: writing smithi08937929-25 from 1615491 to 2200201 tid 2 2024-04-22T18:39:39.847 INFO:tasks.rados.rados.0.smithi089.stdout:1028: writing smithi08937929-25 from 2724284 to 2890804 tid 3 2024-04-22T18:39:39.847 INFO:tasks.rados.rados.0.smithi089.stdout:1029: set_chunk oid 9 target oid 2024-04-22T18:39:39.855 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:39.856 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:39.856 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:39.856 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:39.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.856 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2030 3.0 3:0f5f80b3:::e65ec2deb1a27f55813533c3e73a682460e829b5dd6586a33a43649649fa77f6:head [write 0~1139200 in=1139200b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-04-22T18:39:39.856 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:39.853326+0000", 2024-04-22T18:39:39.856 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0043587809999999999, 2024-04-22T18:39:39.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043868500000000003, 2024-04-22T18:39:39.856 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:39.856 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:39:39.856 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2030 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.853326+0000", 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.853326+0000", 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.853325+0000", 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.853780+0000", 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00045480500000000002 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.853793+0000", 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3162999999999999e-05 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:39.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.853800+0000", 2024-04-22T18:39:39.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2690000000000002e-06 2024-04-22T18:39:39.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:39.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.853856+0000", 2024-04-22T18:39:39.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6200999999999999e-05 2024-04-22T18:39:39.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:39.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.854017+0000", 2024-04-22T18:39:39.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016022599999999999 2024-04-22T18:39:39.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.854190+0000", 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000173124 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.856872+0000", 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002682163 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop_reply(client.4502.0:2030 3.0 e52/49)", 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:39.857074+0000", 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.00061074800000000004, 2024-04-22T18:39:39.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070907399999999997, 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.857074+0000", 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.857074+0000", 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:39.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.857073+0000", 2024-04-22T18:39:39.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:39.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:39.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.857081+0000", 2024-04-22T18:39:39.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7169999999999997e-06 2024-04-22T18:39:39.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:39.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.857094+0000", 2024-04-22T18:39:39.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2927e-05 2024-04-22T18:39:39.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:39.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:39.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:39.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:39.857105+0000", 2024-04-22T18:39:39.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0767e-05 2024-04-22T18:39:39.863 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:39.863 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:39.863 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:39.863 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:39.863 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:39:39.863 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 2 2024-04-22T18:39:39.863 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:39.865 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:39.867 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:39.868+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:39.870 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:39.871+0000 7fa766d7d640 -1 osd.5 52 handle_osd_map injecting map crc failure 2024-04-22T18:39:39.955 INFO:tasks.rados.rados.0.smithi089.stdout:1029: set_chunk oid 9 offset: 686592 length: 1139200 target oid e65ec2deb1a27f55813533c3e73a682460e829b5dd6586a33a43649649fa77f6 offset: 0 snap -1 2024-04-22T18:39:39.955 INFO:tasks.rados.rados.0.smithi089.stdout:1030: set_chunk oid 50 target oid 2024-04-22T18:39:39.969 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:39.969+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:40.013 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:40.013 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:40.024 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:40.024 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:40.025 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:40.025 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:40.025 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:40.032 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:40.032 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:40.034 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:40.050 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:40.050 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:40.066 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:40.066 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:40.070 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:40.071+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:40.072 INFO:tasks.rados.rados.0.smithi089.stdout:1030: set_chunk oid 50 offset: 1239040 length: 89088 target oid e2b0c0d51b4d45a92a8cb137f60a630d36829659cde4be63adad410894a9aec5 offset: 0 snap -1 2024-04-22T18:39:40.073 INFO:tasks.rados.rados.0.smithi089.stdout:1031: write oid 34 current snap is 0 2024-04-22T18:39:40.073 INFO:tasks.rados.rados.0.smithi089.stdout:1031: seq_num 173 ranges {507611=777843,2073973=444721} 2024-04-22T18:39:40.073 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:40.074+0000 7f8605f32640 -1 osd.3 52 handle_osd_map injecting map crc failure 2024-04-22T18:39:40.079 INFO:tasks.rados.rados.0.smithi089.stdout:1031: writing smithi08937929-34 from 507611 to 1285454 tid 1 2024-04-22T18:39:40.087 INFO:tasks.rados.rados.0.smithi089.stdout:1031: writing smithi08937929-34 from 2073973 to 2518694 tid 2 2024-04-22T18:39:40.087 INFO:tasks.rados.rados.0.smithi089.stdout:1032: write oid 24 current snap is 0 2024-04-22T18:39:40.087 INFO:tasks.rados.rados.0.smithi089.stdout:1023: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:39:40.141 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:40.141 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:40.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:40.141 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:40.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.142 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:8 4.f 4:f83a159b:::smithi08937929-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:40.142 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:40.142 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.029749420000002, 2024-04-22T18:39:40.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30292182499999998, 2024-04-22T18:39:40.142 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.142 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:40.142 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.142 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.142 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.142 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 8 2024-04-22T18:39:40.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.143 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:40.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:40.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111378+0000", 2024-04-22T18:39:40.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:40.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111380+0000", 2024-04-22T18:39:40.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2419999999999999e-06 2024-04-22T18:39:40.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111384+0000", 2024-04-22T18:39:40.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.715e-06 2024-04-22T18:39:40.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.111387+0000", 2024-04-22T18:39:40.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.982e-06 2024-04-22T18:39:40.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414113+0000", 2024-04-22T18:39:40.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30272622700000001 2024-04-22T18:39:40.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414203+0000", 2024-04-22T18:39:40.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0316999999999996e-05 2024-04-22T18:39:40.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414300+0000", 2024-04-22T18:39:40.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6760000000000002e-05 2024-04-22T18:39:40.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.146 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.146 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:9 4.f 4:f2c48a0e:::smithi08937929-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:40.147 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:40.147 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.024353836000003, 2024-04-22T18:39:40.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49290714600000002, 2024-04-22T18:39:40.147 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.147 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.147 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.147 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.147 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.147 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 9 2024-04-22T18:39:40.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.147 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:40.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:40.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116773+0000", 2024-04-22T18:39:40.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:40.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117024+0000", 2024-04-22T18:39:40.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025191299999999999 2024-04-22T18:39:40.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117036+0000", 2024-04-22T18:39:40.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1595e-05 2024-04-22T18:39:40.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117046+0000", 2024-04-22T18:39:40.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0456e-05 2024-04-22T18:39:40.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414324+0000", 2024-04-22T18:39:40.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297277603 2024-04-22T18:39:40.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414404+0000", 2024-04-22T18:39:40.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9738000000000001e-05 2024-04-22T18:39:40.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:40.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414494+0000", 2024-04-22T18:39:40.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9733000000000003e-05 2024-04-22T18:39:40.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:40.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518194+0000", 2024-04-22T18:39:40.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10370061 2024-04-22T18:39:40.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:40.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609618+0000", 2024-04-22T18:39:40.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091423360999999995 2024-04-22T18:39:40.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609655+0000", 2024-04-22T18:39:40.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7580000000000003e-05 2024-04-22T18:39:40.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609681+0000", 2024-04-22T18:39:40.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5908e-05 2024-04-22T18:39:40.153 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.153 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.153 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.153 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:10 4.f 4:f2c48a0e:::smithi08937929-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:40.153 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:40.153 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.018501575999998, 2024-04-22T18:39:40.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48712681400000002, 2024-04-22T18:39:40.154 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.154 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.154 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.154 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.154 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.154 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 10 2024-04-22T18:39:40.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.154 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:40.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:40.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122625+0000", 2024-04-22T18:39:40.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:40.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122828+0000", 2024-04-22T18:39:40.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020359499999999999 2024-04-22T18:39:40.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122840+0000", 2024-04-22T18:39:40.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1239e-05 2024-04-22T18:39:40.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122845+0000", 2024-04-22T18:39:40.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2719999999999997e-06 2024-04-22T18:39:40.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415105+0000", 2024-04-22T18:39:40.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29225972099999997 2024-04-22T18:39:40.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415169+0000", 2024-04-22T18:39:40.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4170000000000004e-05 2024-04-22T18:39:40.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:40.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415257+0000", 2024-04-22T18:39:40.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8331999999999994e-05 2024-04-22T18:39:40.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:40.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518199+0000", 2024-04-22T18:39:40.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10294174 2024-04-22T18:39:40.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:40.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609707+0000", 2024-04-22T18:39:40.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091508408999999999 2024-04-22T18:39:40.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609729+0000", 2024-04-22T18:39:40.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1917e-05 2024-04-22T18:39:40.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609753+0000", 2024-04-22T18:39:40.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3776000000000001e-05 2024-04-22T18:39:40.160 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.160 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.160 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.160 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:11 4.f 4:f2c48a0e:::smithi08937929-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:40.160 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:40.160 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.018278203000001, 2024-04-22T18:39:40.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49452214500000002, 2024-04-22T18:39:40.160 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.160 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.160 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.161 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.161 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.161 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 11 2024-04-22T18:39:40.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.161 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:40.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:40.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122849+0000", 2024-04-22T18:39:40.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:40.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122853+0000", 2024-04-22T18:39:40.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9029999999999997e-06 2024-04-22T18:39:40.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122856+0000", 2024-04-22T18:39:40.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.01e-06 2024-04-22T18:39:40.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122859+0000", 2024-04-22T18:39:40.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:40.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415755+0000", 2024-04-22T18:39:40.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29289606800000001 2024-04-22T18:39:40.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415808+0000", 2024-04-22T18:39:40.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3415999999999998e-05 2024-04-22T18:39:40.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:40.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415888+0000", 2024-04-22T18:39:40.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9061999999999996e-05 2024-04-22T18:39:40.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:40.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518203+0000", 2024-04-22T18:39:40.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102315533 2024-04-22T18:39:40.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:40.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617164+0000", 2024-04-22T18:39:40.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098960548999999995 2024-04-22T18:39:40.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617241+0000", 2024-04-22T18:39:40.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7292000000000005e-05 2024-04-22T18:39:40.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617372+0000", 2024-04-22T18:39:40.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013075399999999999 2024-04-22T18:39:40.166 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.166 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.166 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.167 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:12 4.f 4:f2c48a0e:::smithi08937929-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:40.167 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:40.167 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.018266339, 2024-04-22T18:39:40.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49473910999999998, 2024-04-22T18:39:40.167 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.167 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:40.167 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.167 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.167 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.168 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 12 2024-04-22T18:39:40.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.168 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:40.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:40.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:40.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:40.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122863+0000", 2024-04-22T18:39:40.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.863e-06 2024-04-22T18:39:40.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122866+0000", 2024-04-22T18:39:40.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:39:40.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122869+0000", 2024-04-22T18:39:40.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5950000000000001e-06 2024-04-22T18:39:40.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416008+0000", 2024-04-22T18:39:40.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29313975199999998 2024-04-22T18:39:40.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:40.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416048+0000", 2024-04-22T18:39:40.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9888000000000002e-05 2024-04-22T18:39:40.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617447+0000", 2024-04-22T18:39:40.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20139881400000001 2024-04-22T18:39:40.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617537+0000", 2024-04-22T18:39:40.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9606999999999998e-05 2024-04-22T18:39:40.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617600+0000", 2024-04-22T18:39:40.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3884e-05 2024-04-22T18:39:40.172 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.173 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.173 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.173 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:33 4.f 4:f0ed3d49:::smithi08937929-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:40.173 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:40.173 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.954326137999999, 2024-04-22T18:39:40.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62359692099999997, 2024-04-22T18:39:40.173 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.173 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.173 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.173 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.174 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.174 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 33 2024-04-22T18:39:40.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.174 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:40.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:40.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186799+0000", 2024-04-22T18:39:40.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:40.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187103+0000", 2024-04-22T18:39:40.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030354900000000002 2024-04-22T18:39:40.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187117+0000", 2024-04-22T18:39:40.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4578e-05 2024-04-22T18:39:40.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187129+0000", 2024-04-22T18:39:40.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1421e-05 2024-04-22T18:39:40.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516134+0000", 2024-04-22T18:39:40.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32900553900000001 2024-04-22T18:39:40.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516215+0000", 2024-04-22T18:39:40.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0537999999999994e-05 2024-04-22T18:39:40.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:40.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516302+0000", 2024-04-22T18:39:40.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6601000000000007e-05 2024-04-22T18:39:40.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:40.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518207+0000", 2024-04-22T18:39:40.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019058370000000001 2024-04-22T18:39:40.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:40.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810302+0000", 2024-04-22T18:39:40.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29209509700000003 2024-04-22T18:39:40.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810348+0000", 2024-04-22T18:39:40.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5992999999999998e-05 2024-04-22T18:39:40.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810398+0000", 2024-04-22T18:39:40.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0046000000000002e-05 2024-04-22T18:39:40.179 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.180 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.180 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.180 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:34 4.f 4:f0ed3d49:::smithi08937929-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:40.180 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:40.180 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.949724126, 2024-04-22T18:39:40.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.61954655800000002, 2024-04-22T18:39:40.180 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.180 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.180 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.180 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.180 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.181 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34 2024-04-22T18:39:40.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.181 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:40.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:40.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191401+0000", 2024-04-22T18:39:40.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:40.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191503+0000", 2024-04-22T18:39:40.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010190400000000001 2024-04-22T18:39:40.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191517+0000", 2024-04-22T18:39:40.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3267e-05 2024-04-22T18:39:40.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191523+0000", 2024-04-22T18:39:40.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-04-22T18:39:40.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517062+0000", 2024-04-22T18:39:40.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32553894300000003 2024-04-22T18:39:40.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517141+0000", 2024-04-22T18:39:40.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9667999999999997e-05 2024-04-22T18:39:40.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:40.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517267+0000", 2024-04-22T18:39:40.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000125494 2024-04-22T18:39:40.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:40.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518213+0000", 2024-04-22T18:39:40.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000946275 2024-04-22T18:39:40.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:40.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810901+0000", 2024-04-22T18:39:40.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29268821699999997 2024-04-22T18:39:40.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810929+0000", 2024-04-22T18:39:40.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7645999999999999e-05 2024-04-22T18:39:40.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810950+0000", 2024-04-22T18:39:40.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1333999999999999e-05 2024-04-22T18:39:40.186 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.186 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.186 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.186 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:40.187 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:40.187 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.949601158, 2024-04-22T18:39:40.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:39:40.187 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.187 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.187 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.187 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.187 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.187 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:39:40.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.187 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:40.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:40.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:39:40.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:40.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:39:40.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:39:40.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:39:40.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:39:40.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:39:40.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:40.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:39:40.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:39:40.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:39:40.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:39:40.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:40.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:39:40.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:39:40.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:40.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:39:40.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:39:40.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:40.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:39:40.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:39:40.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:39:40.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:39:40.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:39:40.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:39:40.193 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.193 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.193 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.193 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:40.193 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:40.193 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.949590082999997, 2024-04-22T18:39:40.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:39:40.194 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.194 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:40.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.194 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:39:40.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.194 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:40.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:40.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:39:40.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:40.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:39:40.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:39:40.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:39:40.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:39:40.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:39:40.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:39:40.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:39:40.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:39:40.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:40.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:39:40.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:39:40.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:39:40.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:39:40.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:39:40.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:39:40.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:39:40.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:39:40.199 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.199 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.199 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.199 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:38 4.2 4:4fb26dd8:::smithi08937929-10:head [write 1893632~257631] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:40.199 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:40.199 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.937867478999998, 2024-04-22T18:39:40.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33863800199999999, 2024-04-22T18:39:40.199 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.200 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.200 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.200 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.200 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.200 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 38 2024-04-22T18:39:40.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.200 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:40.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:40.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203258+0000", 2024-04-22T18:39:40.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:40.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203396+0000", 2024-04-22T18:39:40.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013710799999999999 2024-04-22T18:39:40.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203406+0000", 2024-04-22T18:39:40.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0295000000000001e-05 2024-04-22T18:39:40.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203411+0000", 2024-04-22T18:39:40.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5079999999999996e-06 2024-04-22T18:39:40.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203450+0000", 2024-04-22T18:39:40.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8420999999999997e-05 2024-04-22T18:39:40.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:40.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203455+0000", 2024-04-22T18:39:40.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:39:40.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237071+0000", 2024-04-22T18:39:40.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.033615891000000002 2024-04-22T18:39:40.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237129+0000", 2024-04-22T18:39:40.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8134000000000003e-05 2024-04-22T18:39:40.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:40.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237221+0000", 2024-04-22T18:39:40.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2053999999999997e-05 2024-04-22T18:39:40.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:40.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.341419+0000", 2024-04-22T18:39:40.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104197568 2024-04-22T18:39:40.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:40.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541792+0000", 2024-04-22T18:39:40.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.200373312 2024-04-22T18:39:40.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541838+0000", 2024-04-22T18:39:40.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6004000000000002e-05 2024-04-22T18:39:40.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541898+0000", 2024-04-22T18:39:40.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0297000000000002e-05 2024-04-22T18:39:40.207 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.207 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.207 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.207 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:39 4.2 4:4fb26dd8:::smithi08937929-10:head [setxattr _header (56) in=7b,truncate 2151263] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:40.207 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:40.207 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.937710484999997, 2024-04-22T18:39:40.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33862203000000002, 2024-04-22T18:39:40.207 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.207 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.208 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 39 2024-04-22T18:39:40.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.208 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:40.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:40.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:40.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:40.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203423+0000", 2024-04-22T18:39:40.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0589999999999999e-06 2024-04-22T18:39:40.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203429+0000", 2024-04-22T18:39:40.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2559999999999999e-06 2024-04-22T18:39:40.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203434+0000", 2024-04-22T18:39:40.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8049999999999997e-06 2024-04-22T18:39:40.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203475+0000", 2024-04-22T18:39:40.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1205000000000003e-05 2024-04-22T18:39:40.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:40.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203478+0000", 2024-04-22T18:39:40.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7140000000000002e-06 2024-04-22T18:39:40.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239180+0000", 2024-04-22T18:39:40.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035702195999999999 2024-04-22T18:39:40.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239225+0000", 2024-04-22T18:39:40.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5722e-05 2024-04-22T18:39:40.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:40.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239291+0000", 2024-04-22T18:39:40.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5870000000000005e-05 2024-04-22T18:39:40.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:40.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342114+0000", 2024-04-22T18:39:40.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102822301 2024-04-22T18:39:40.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:40.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541946+0000", 2024-04-22T18:39:40.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.199832544 2024-04-22T18:39:40.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541988+0000", 2024-04-22T18:39:40.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1499999999999999e-05 2024-04-22T18:39:40.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542039+0000", 2024-04-22T18:39:40.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1576000000000002e-05 2024-04-22T18:39:40.214 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.214 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.214 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.215 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:40 4.2 4:4fb26dd8:::smithi08937929-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:40.215 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:40.215 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.937673224999998, 2024-04-22T18:39:40.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33924657200000002, 2024-04-22T18:39:40.215 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.215 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:40.215 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.215 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.215 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.215 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 40 2024-04-22T18:39:40.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.216 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:40.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:40.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:40.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:40.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203457+0000", 2024-04-22T18:39:40.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.129e-06 2024-04-22T18:39:40.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203463+0000", 2024-04-22T18:39:40.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7440000000000004e-06 2024-04-22T18:39:40.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203468+0000", 2024-04-22T18:39:40.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2619999999999998e-06 2024-04-22T18:39:40.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203491+0000", 2024-04-22T18:39:40.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3260000000000001e-05 2024-04-22T18:39:40.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:40.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203493+0000", 2024-04-22T18:39:40.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9599999999999999e-06 2024-04-22T18:39:40.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239420+0000", 2024-04-22T18:39:40.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035927212 2024-04-22T18:39:40.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:40.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239462+0000", 2024-04-22T18:39:40.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.159e-05 2024-04-22T18:39:40.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542547+0000", 2024-04-22T18:39:40.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30308492599999998 2024-04-22T18:39:40.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542611+0000", 2024-04-22T18:39:40.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4622000000000006e-05 2024-04-22T18:39:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542701+0000", 2024-04-22T18:39:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9579000000000004e-05 2024-04-22T18:39:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:41 4.2 4:409e6456:::smithi08937929-11:head [write 568070~468843,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:40.222 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.931956954, 2024-04-22T18:39:40.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33297193600000002, 2024-04-22T18:39:40.222 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.222 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.222 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.222 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.222 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.222 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 41 2024-04-22T18:39:40.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.222 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:40.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:40.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209168+0000", 2024-04-22T18:39:40.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:40.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209393+0000", 2024-04-22T18:39:40.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022485199999999999 2024-04-22T18:39:40.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209404+0000", 2024-04-22T18:39:40.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1209999999999999e-05 2024-04-22T18:39:40.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209411+0000", 2024-04-22T18:39:40.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8580000000000002e-06 2024-04-22T18:39:40.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209457+0000", 2024-04-22T18:39:40.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6351999999999999e-05 2024-04-22T18:39:40.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:40.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209463+0000", 2024-04-22T18:39:40.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.322e-06 2024-04-22T18:39:40.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339572+0000", 2024-04-22T18:39:40.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13010901999999999 2024-04-22T18:39:40.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339673+0000", 2024-04-22T18:39:40.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101332 2024-04-22T18:39:40.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:40.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339765+0000", 2024-04-22T18:39:40.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1780999999999994e-05 2024-04-22T18:39:40.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:40.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342117+0000", 2024-04-22T18:39:40.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002351709 2024-04-22T18:39:40.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:40.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542074+0000", 2024-04-22T18:39:40.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995749800000001 2024-04-22T18:39:40.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542112+0000", 2024-04-22T18:39:40.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7891e-05 2024-04-22T18:39:40.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542143+0000", 2024-04-22T18:39:40.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0756000000000002e-05 2024-04-22T18:39:40.229 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.229 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.229 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.229 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:42 4.2 4:409e6456:::smithi08937929-11:head [write 1552114~615956] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:40.229 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:40.229 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.925618645, 2024-04-22T18:39:40.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326735257, 2024-04-22T18:39:40.230 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.230 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.230 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.230 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.230 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.230 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 42 2024-04-22T18:39:40.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.230 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:40.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:40.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215508+0000", 2024-04-22T18:39:40.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:40.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215766+0000", 2024-04-22T18:39:40.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025766999999999999 2024-04-22T18:39:40.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215776+0000", 2024-04-22T18:39:40.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0772e-05 2024-04-22T18:39:40.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215785+0000", 2024-04-22T18:39:40.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6880000000000005e-06 2024-04-22T18:39:40.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215818+0000", 2024-04-22T18:39:40.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2960999999999998e-05 2024-04-22T18:39:40.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:40.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215821+0000", 2024-04-22T18:39:40.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8569999999999999e-06 2024-04-22T18:39:40.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340332+0000", 2024-04-22T18:39:40.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.124511409 2024-04-22T18:39:40.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340416+0000", 2024-04-22T18:39:40.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3793000000000002e-05 2024-04-22T18:39:40.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:40.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340543+0000", 2024-04-22T18:39:40.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012720299999999999 2024-04-22T18:39:40.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:40.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342361+0000", 2024-04-22T18:39:40.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018174370000000001 2024-04-22T18:39:40.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:40.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542176+0000", 2024-04-22T18:39:40.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19981518400000001 2024-04-22T18:39:40.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542211+0000", 2024-04-22T18:39:40.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5128e-05 2024-04-22T18:39:40.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542244+0000", 2024-04-22T18:39:40.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3256000000000001e-05 2024-04-22T18:39:40.236 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.237 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.237 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:43 4.2 4:409e6456:::smithi08937929-11:head [setxattr _header (56) in=7b,truncate 2168070] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:40.237 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:40.237 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.925333872000003, 2024-04-22T18:39:40.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32656540499999998, 2024-04-22T18:39:40.237 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.237 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.237 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.237 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.238 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.238 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 43 2024-04-22T18:39:40.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.238 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:40.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:40.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215793+0000", 2024-04-22T18:39:40.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:40.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215797+0000", 2024-04-22T18:39:40.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4190000000000004e-06 2024-04-22T18:39:40.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215804+0000", 2024-04-22T18:39:40.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:39:40.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215809+0000", 2024-04-22T18:39:40.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3689999999999998e-06 2024-04-22T18:39:40.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215840+0000", 2024-04-22T18:39:40.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-05 2024-04-22T18:39:40.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:40.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215843+0000", 2024-04-22T18:39:40.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8849999999999999e-06 2024-04-22T18:39:40.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342483+0000", 2024-04-22T18:39:40.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126640172 2024-04-22T18:39:40.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342529+0000", 2024-04-22T18:39:40.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5668999999999999e-05 2024-04-22T18:39:40.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:40.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342597+0000", 2024-04-22T18:39:40.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7538000000000003e-05 2024-04-22T18:39:40.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:40.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.347102+0000", 2024-04-22T18:39:40.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045058040000000004 2024-04-22T18:39:40.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:40.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542278+0000", 2024-04-22T18:39:40.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.195175879 2024-04-22T18:39:40.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542314+0000", 2024-04-22T18:39:40.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6134999999999999e-05 2024-04-22T18:39:40.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542359+0000", 2024-04-22T18:39:40.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4848999999999997e-05 2024-04-22T18:39:40.244 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.244 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.244 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.245 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:44 4.2 4:409e6456:::smithi08937929-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:40.245 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:40.245 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.925313864000003, 2024-04-22T18:39:40.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326706418, 2024-04-22T18:39:40.245 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.245 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:40.245 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.245 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.245 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.245 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 44 2024-04-22T18:39:40.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.245 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:40.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:40.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215813+0000", 2024-04-22T18:39:40.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:40.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215816+0000", 2024-04-22T18:39:40.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9799999999999998e-06 2024-04-22T18:39:40.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215822+0000", 2024-04-22T18:39:40.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5570000000000003e-06 2024-04-22T18:39:40.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215827+0000", 2024-04-22T18:39:40.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2739999999999998e-06 2024-04-22T18:39:40.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215855+0000", 2024-04-22T18:39:40.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7682999999999999e-05 2024-04-22T18:39:40.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:40.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215857+0000", 2024-04-22T18:39:40.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9250000000000002e-06 2024-04-22T18:39:40.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342717+0000", 2024-04-22T18:39:40.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126860696 2024-04-22T18:39:40.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:40.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342758+0000", 2024-04-22T18:39:40.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0295e-05 2024-04-22T18:39:40.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542408+0000", 2024-04-22T18:39:40.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19964990599999999 2024-04-22T18:39:40.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542457+0000", 2024-04-22T18:39:40.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9784999999999999e-05 2024-04-22T18:39:40.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542520+0000", 2024-04-22T18:39:40.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2960999999999996e-05 2024-04-22T18:39:40.251 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.251 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.251 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.251 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:40.251 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:40.251 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.348220589, 2024-04-22T18:39:40.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:39:40.252 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.252 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.252 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.252 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.252 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.252 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:39:40.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.252 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:40.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:40.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:39:40.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:40.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:39:40.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:39:40.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:39:40.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:39:40.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:39:40.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:39:40.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:39:40.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:39:40.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:39:40.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:39:40.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:40.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:39:40.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:39:40.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:40.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:39:40.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:39:40.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:40.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:39:40.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:39:40.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:39:40.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:39:40.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:39:40.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:39:40.257 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.257 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.257 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.258 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1262 4.7 4:e117a562:::smithi08937929-33:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:40.258 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:40.258 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.688077412999998, 2024-04-22T18:39:40.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40488836700000003, 2024-04-22T18:39:40.258 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.258 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.258 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.258 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.258 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.258 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1262 2024-04-22T18:39:40.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.259 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:40.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:40.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453048+0000", 2024-04-22T18:39:40.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:40.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453058+0000", 2024-04-22T18:39:40.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:39:40.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453069+0000", 2024-04-22T18:39:40.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1117999999999999e-05 2024-04-22T18:39:40.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453075+0000", 2024-04-22T18:39:40.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0909999999999998e-06 2024-04-22T18:39:40.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453115+0000", 2024-04-22T18:39:40.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0119e-05 2024-04-22T18:39:40.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:40.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453119+0000", 2024-04-22T18:39:40.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3610000000000003e-06 2024-04-22T18:39:40.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454814+0000", 2024-04-22T18:39:40.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00169436 2024-04-22T18:39:40.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454861+0000", 2024-04-22T18:39:40.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7184000000000001e-05 2024-04-22T18:39:40.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:40.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756925+0000", 2024-04-22T18:39:40.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30206361799999998 2024-04-22T18:39:40.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:40.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757577+0000", 2024-04-22T18:39:40.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065199599999999996 2024-04-22T18:39:40.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:40.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857846+0000", 2024-04-22T18:39:40.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100269369 2024-04-22T18:39:40.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857896+0000", 2024-04-22T18:39:40.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9607000000000001e-05 2024-04-22T18:39:40.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857939+0000", 2024-04-22T18:39:40.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3149999999999999e-05 2024-04-22T18:39:40.265 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.265 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.265 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.265 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1689 3.0 3:1d4bcadf:::e7eb8db0531d9f456e91fe87bc79e47ebd80459f1783dc2a5e53d4822cd34f49:head [write 0~1066496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:39:40.266 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.607920+0000", 2024-04-22T18:39:40.266 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.5332080389999998, 2024-04-22T18:39:40.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30339939300000002, 2024-04-22T18:39:40.266 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.266 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.266 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.266 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.266 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.266 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1689 2024-04-22T18:39:40.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.266 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.607920+0000", 2024-04-22T18:39:40.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.607920+0000", 2024-04-22T18:39:40.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.607917+0000", 2024-04-22T18:39:40.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:40.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.608327+0000", 2024-04-22T18:39:40.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000409508 2024-04-22T18:39:40.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.608344+0000", 2024-04-22T18:39:40.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6846000000000001e-05 2024-04-22T18:39:40.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.608351+0000", 2024-04-22T18:39:40.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8140000000000003e-06 2024-04-22T18:39:40.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708489+0000", 2024-04-22T18:39:40.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100138536 2024-04-22T18:39:40.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708651+0000", 2024-04-22T18:39:40.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000162132 2024-04-22T18:39:40.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:39:40.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708793+0000", 2024-04-22T18:39:40.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014147699999999999 2024-04-22T18:39:40.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:40.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.713125+0000", 2024-04-22T18:39:40.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043321369999999998 2024-04-22T18:39:40.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:40.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911261+0000", 2024-04-22T18:39:40.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198135744 2024-04-22T18:39:40.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911293+0000", 2024-04-22T18:39:40.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1970000000000001e-05 2024-04-22T18:39:40.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911319+0000", 2024-04-22T18:39:40.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6440000000000001e-05 2024-04-22T18:39:40.272 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.272 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.272 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.273 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1872 4.2 4:409e6456:::smithi08937929-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:40.273 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:35.576858+0000", 2024-04-22T18:39:40.273 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.564269286, 2024-04-22T18:39:40.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.302951992, 2024-04-22T18:39:40.273 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.273 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:40.273 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:40.273 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:40.273 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:40.273 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1872 2024-04-22T18:39:40.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.274 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576858+0000", 2024-04-22T18:39:40.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576858+0000", 2024-04-22T18:39:40.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576857+0000", 2024-04-22T18:39:40.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:40.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576864+0000", 2024-04-22T18:39:40.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4569999999999999e-06 2024-04-22T18:39:40.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576871+0000", 2024-04-22T18:39:40.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1679999999999998e-06 2024-04-22T18:39:40.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576876+0000", 2024-04-22T18:39:40.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0440000000000003e-06 2024-04-22T18:39:40.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.590050+0000", 2024-04-22T18:39:40.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.013174208 2024-04-22T18:39:40.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:40.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.590142+0000", 2024-04-22T18:39:40.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1681000000000005e-05 2024-04-22T18:39:40.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.879685+0000", 2024-04-22T18:39:40.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.28954336200000003 2024-04-22T18:39:40.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.879745+0000", 2024-04-22T18:39:40.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9487999999999997e-05 2024-04-22T18:39:40.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.879810+0000", 2024-04-22T18:39:40.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5431000000000004e-05 2024-04-22T18:39:40.278 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.278 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.278 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.278 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.279 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.279 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:40.280 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:40.280 INFO:tasks.rados.rados.0.smithi089.stdout:1021: expect (ObjNum 144 snap 0 seq_num 144) 2024-04-22T18:39:40.281 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:40.280+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:40.363 INFO:tasks.rados.rados.0.smithi089.stdout:1020: expect (ObjNum 58 snap 0 seq_num 58) 2024-04-22T18:39:40.380 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:40.382+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:40.423 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:40.424 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:40.432 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:40.462 INFO:tasks.rados.rados.0.smithi089.stdout:1019: finishing write tid 1 to smithi08937929-33 2024-04-22T18:39:40.462 INFO:tasks.rados.rados.0.smithi089.stdout:1019: finishing write tid 2 to smithi08937929-33 2024-04-22T18:39:40.462 INFO:tasks.rados.rados.0.smithi089.stdout:1019: finishing write tid 3 to smithi08937929-33 2024-04-22T18:39:40.462 INFO:tasks.rados.rados.0.smithi089.stdout:1019: finishing write tid 4 to smithi08937929-33 2024-04-22T18:39:40.462 INFO:tasks.rados.rados.0.smithi089.stdout:1019: finishing write tid 5 to smithi08937929-33 2024-04-22T18:39:40.462 INFO:tasks.rados.rados.0.smithi089.stdout:1019: oid 33 updating version 0 to 31 2024-04-22T18:39:40.463 INFO:tasks.rados.rados.0.smithi089.stdout:1019: oid 33 updating version 31 to 33 2024-04-22T18:39:40.463 INFO:tasks.rados.rados.0.smithi089.stdout:1019: oid 33 version 33 is already newer than 30 2024-04-22T18:39:40.463 INFO:tasks.rados.rados.0.smithi089.stdout:1019: oid 33 version 33 is already newer than 32 2024-04-22T18:39:40.463 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 33 (ObjNum 169 snap 0 seq_num 169) dirty exists 2024-04-22T18:39:40.463 INFO:tasks.rados.rados.0.smithi089.stdout:1019: left oid 33 (ObjNum 169 snap 0 seq_num 169) 2024-04-22T18:39:40.463 INFO:tasks.rados.rados.0.smithi089.stdout:1026: finishing set_chunk to oid 36 2024-04-22T18:39:40.463 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 27 (ObjNum 80 snap 0 seq_num 80) dirty exists 2024-04-22T18:39:40.463 INFO:tasks.rados.rados.0.smithi089.stdout:1024: finishing set_chunk to oid 44 2024-04-22T18:39:40.463 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 59 (ObjNum 119 snap 0 seq_num 119) dirty exists 2024-04-22T18:39:40.463 INFO:tasks.rados.rados.0.smithi089.stdout:1028: finishing write tid 1 to smithi08937929-25 2024-04-22T18:39:40.463 INFO:tasks.rados.rados.0.smithi089.stdout:1032: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2024-04-22T18:39:40.467 INFO:tasks.rados.rados.0.smithi089.stdout:1032: writing smithi08937929-24 from 524418 to 977510 tid 1 2024-04-22T18:39:40.474 INFO:tasks.rados.rados.0.smithi089.stdout:1032: writing smithi08937929-24 from 1732455 to 2320834 tid 2 2024-04-22T18:39:40.479 INFO:tasks.rados.rados.0.smithi089.stdout:1032: writing smithi08937929-24 from 2779130 to 2924418 tid 3 2024-04-22T18:39:40.479 INFO:tasks.rados.rados.0.smithi089.stdout:1019: done (12 left) 2024-04-22T18:39:40.479 INFO:tasks.rados.rados.0.smithi089.stdout:1020: done (11 left) 2024-04-22T18:39:40.479 INFO:tasks.rados.rados.0.smithi089.stdout:1021: done (10 left) 2024-04-22T18:39:40.479 INFO:tasks.rados.rados.0.smithi089.stdout:1023: done (9 left) 2024-04-22T18:39:40.479 INFO:tasks.rados.rados.0.smithi089.stdout:1024: done (8 left) 2024-04-22T18:39:40.479 INFO:tasks.rados.rados.0.smithi089.stdout:1026: done (7 left) 2024-04-22T18:39:40.479 INFO:tasks.rados.rados.0.smithi089.stdout:1033: read oid 20 snap -1 2024-04-22T18:39:40.479 INFO:tasks.rados.rados.0.smithi089.stdout:1033: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-22T18:39:40.479 INFO:tasks.rados.rados.0.smithi089.stdout:1034: read oid 32 snap -1 2024-04-22T18:39:40.479 INFO:tasks.rados.rados.0.smithi089.stdout:1034: expect (ObjNum 157 snap 0 seq_num 157) 2024-04-22T18:39:40.479 INFO:tasks.rados.rados.0.smithi089.stdout:1035: set_chunk oid 40 target oid 2024-04-22T18:39:40.481 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:40.483+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:40.564 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:40.564 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:40.564 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:40.564 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:40.564 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:40.572 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:40.576 INFO:tasks.rados.rados.0.smithi089.stdout:1035: set_chunk oid 40 offset: 291328 length: 408064 target oid 135c6aa059b9e3c2921ec4ea2f362554ad75b3ac2d718d3ada3d933bc352b588 offset: 0 snap -1 2024-04-22T18:39:40.576 INFO:tasks.rados.rados.0.smithi089.stdout:1036: read oid 35 snap -1 2024-04-22T18:39:40.576 INFO:tasks.rados.rados.0.smithi089.stdout:1036: expect (ObjNum 162 snap 0 seq_num 162) 2024-04-22T18:39:40.576 INFO:tasks.rados.rados.0.smithi089.stdout:1037: write oid 37 current snap is 0 2024-04-22T18:39:40.576 INFO:tasks.rados.rados.0.smithi089.stdout:1025: finishing write tid 1 to smithi08937929-43 2024-04-22T18:39:40.576 INFO:tasks.rados.rados.0.smithi089.stdout:1025: finishing write tid 2 to smithi08937929-43 2024-04-22T18:39:40.576 INFO:tasks.rados.rados.0.smithi089.stdout:1027: finishing write tid 1 to smithi08937929-6 2024-04-22T18:39:40.576 INFO:tasks.rados.rados.0.smithi089.stdout:1028: finishing write tid 2 to smithi08937929-25 2024-04-22T18:39:40.576 INFO:tasks.rados.rados.0.smithi089.stdout:1025: finishing write tid 3 to smithi08937929-43 2024-04-22T18:39:40.576 INFO:tasks.rados.rados.0.smithi089.stdout:1031: finishing write tid 1 to smithi08937929-34 2024-04-22T18:39:40.576 INFO:tasks.rados.rados.0.smithi089.stdout:1027: finishing write tid 2 to smithi08937929-6 2024-04-22T18:39:40.577 INFO:tasks.rados.rados.0.smithi089.stdout:1027: finishing write tid 3 to smithi08937929-6 2024-04-22T18:39:40.577 INFO:tasks.rados.rados.0.smithi089.stdout:1030: finishing set_chunk to oid 50 2024-04-22T18:39:40.577 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 85 (ObjNum 148 snap 0 seq_num 148) dirty exists 2024-04-22T18:39:40.577 INFO:tasks.rados.rados.0.smithi089.stdout:1028: finishing write tid 3 to smithi08937929-25 2024-04-22T18:39:40.577 INFO:tasks.rados.rados.0.smithi089.stdout:1028: finishing write tid 4 to smithi08937929-25 2024-04-22T18:39:40.577 INFO:tasks.rados.rados.0.smithi089.stdout:1037: seq_num 175 ranges {541225=644694,1907290=448390} 2024-04-22T18:39:40.582 INFO:tasks.rados.rados.0.smithi089.stdout:1037: writing smithi08937929-37 from 541225 to 1185919 tid 1 2024-04-22T18:39:40.582 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:40.584+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:40.589 INFO:tasks.rados.rados.0.smithi089.stdout:1037: writing smithi08937929-37 from 1907290 to 2355680 tid 2 2024-04-22T18:39:40.589 INFO:tasks.rados.rados.0.smithi089.stdout:1030: done (11 left) 2024-04-22T18:39:40.589 INFO:tasks.rados.rados.0.smithi089.stdout:1038: read oid 2 snap -1 2024-04-22T18:39:40.590 INFO:tasks.rados.rados.0.smithi089.stdout:1038: expect (ObjNum 156 snap 0 seq_num 156) 2024-04-22T18:39:40.590 INFO:tasks.rados.rados.0.smithi089.stdout:1039: read oid 28 snap -1 2024-04-22T18:39:40.590 INFO:tasks.rados.rados.0.smithi089.stdout:1039: expect (ObjNum 138 snap 0 seq_num 138) 2024-04-22T18:39:40.590 INFO:tasks.rados.rados.0.smithi089.stdout:1040: read oid 44 snap -1 2024-04-22T18:39:40.590 INFO:tasks.rados.rados.0.smithi089.stdout:1040: expect (ObjNum 119 snap 0 seq_num 119) 2024-04-22T18:39:40.590 INFO:tasks.rados.rados.0.smithi089.stdout:1041: read oid 50 snap -1 2024-04-22T18:39:40.590 INFO:tasks.rados.rados.0.smithi089.stdout:1041: expect (ObjNum 148 snap 0 seq_num 148) 2024-04-22T18:39:40.590 INFO:tasks.rados.rados.0.smithi089.stdout:1042: write oid 1 current snap is 0 2024-04-22T18:39:40.590 INFO:tasks.rados.rados.0.smithi089.stdout:1028: finishing write tid 5 to smithi08937929-25 2024-04-22T18:39:40.590 INFO:tasks.rados.rados.0.smithi089.stdout:1028: oid 25 updating version 0 to 57 2024-04-22T18:39:40.590 INFO:tasks.rados.rados.0.smithi089.stdout:1028: oid 25 version 57 is already newer than 56 2024-04-22T18:39:40.590 INFO:tasks.rados.rados.0.smithi089.stdout:1028: oid 25 updating version 57 to 59 2024-04-22T18:39:40.590 INFO:tasks.rados.rados.0.smithi089.stdout:1028: oid 25 version 59 is already newer than 58 2024-04-22T18:39:40.590 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 59 (ObjNum 172 snap 0 seq_num 172) dirty exists 2024-04-22T18:39:40.591 INFO:tasks.rados.rados.0.smithi089.stdout:1028: left oid 25 (ObjNum 172 snap 0 seq_num 172) 2024-04-22T18:39:40.591 INFO:tasks.rados.rados.0.smithi089.stdout:1025: finishing write tid 4 to smithi08937929-43 2024-04-22T18:39:40.591 INFO:tasks.rados.rados.0.smithi089.stdout:1042: seq_num 176 ranges {558032=719943,1849419=592048} 2024-04-22T18:39:40.595 INFO:tasks.rados.rados.0.smithi089.stdout:1042: writing smithi08937929-1 from 558032 to 1277975 tid 1 2024-04-22T18:39:40.603 INFO:tasks.rados.rados.0.smithi089.stdout:1042: writing smithi08937929-1 from 1849419 to 2441467 tid 2 2024-04-22T18:39:40.603 INFO:tasks.rados.rados.0.smithi089.stdout:1028: done (15 left) 2024-04-22T18:39:40.603 INFO:tasks.rados.rados.0.smithi089.stdout:1043: read oid 14 snap -1 2024-04-22T18:39:40.603 INFO:tasks.rados.rados.0.smithi089.stdout:1043: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:39:40.603 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:40.603 INFO:tasks.rados.rados.0.smithi089.stdout:1029: finishing set_chunk to oid 9 2024-04-22T18:39:40.603 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 33 (ObjNum 153 snap 0 seq_num 153) dirty exists 2024-04-22T18:39:40.603 INFO:tasks.rados.rados.0.smithi089.stdout:1027: finishing write tid 4 to smithi08937929-6 2024-04-22T18:39:40.603 INFO:tasks.rados.rados.0.smithi089.stdout:1027: finishing write tid 5 to smithi08937929-6 2024-04-22T18:39:40.603 INFO:tasks.rados.rados.0.smithi089.stdout:1027: oid 6 updating version 0 to 44 2024-04-22T18:39:40.604 INFO:tasks.rados.rados.0.smithi089.stdout:1027: oid 6 version 44 is already newer than 43 2024-04-22T18:39:40.604 INFO:tasks.rados.rados.0.smithi089.stdout:1027: oid 6 version 44 is already newer than 42 2024-04-22T18:39:40.604 INFO:tasks.rados.rados.0.smithi089.stdout:1027: oid 6 updating version 44 to 45 2024-04-22T18:39:40.604 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 45 (ObjNum 171 snap 0 seq_num 171) dirty exists 2024-04-22T18:39:40.604 INFO:tasks.rados.rados.0.smithi089.stdout:1027: left oid 6 (ObjNum 171 snap 0 seq_num 171) 2024-04-22T18:39:40.604 INFO:tasks.rados.rados.0.smithi089.stdout:1027: done (15 left) 2024-04-22T18:39:40.604 INFO:tasks.rados.rados.0.smithi089.stdout:1029: done (14 left) 2024-04-22T18:39:40.604 INFO:tasks.rados.rados.0.smithi089.stdout:1044: read oid 6 snap -1 2024-04-22T18:39:40.604 INFO:tasks.rados.rados.0.smithi089.stdout:1044: expect (ObjNum 171 snap 0 seq_num 171) 2024-04-22T18:39:40.604 INFO:tasks.rados.rados.0.smithi089.stdout:1045: set_chunk oid 42 target oid 2024-04-22T18:39:40.669 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:39:40.681 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:40.681 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:40.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:40.681 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:40.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.681 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:27 4.d e27/20 4:bee1184f:::smithi08937929-7:head v 27'3, mlcod=0'0)", 2024-04-22T18:39:40.681 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:40.681 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.315409871999996, 2024-04-22T18:39:40.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101135974, 2024-04-22T18:39:40.681 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.681 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.681 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:40.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:40.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365186+0000", 2024-04-22T18:39:40.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:40.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365194+0000", 2024-04-22T18:39:40.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3289999999999994e-06 2024-04-22T18:39:40.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365200+0000", 2024-04-22T18:39:40.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6590000000000004e-06 2024-04-22T18:39:40.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465352+0000", 2024-04-22T18:39:40.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10015208 2024-04-22T18:39:40.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465511+0000", 2024-04-22T18:39:40.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015880000000000001 2024-04-22T18:39:40.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465570+0000", 2024-04-22T18:39:40.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8952000000000001e-05 2024-04-22T18:39:40.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466292+0000", 2024-04-22T18:39:40.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072212100000000005 2024-04-22T18:39:40.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466323+0000", 2024-04-22T18:39:40.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0899999999999999e-05 2024-04-22T18:39:40.685 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.685 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.685 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.685 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:47 3.0 e30/19 3:30f16451:::smithi08937929-50 low_tier:head v 30'20, mlcod=29'18)", 2024-04-22T18:39:40.685 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:40.685 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.082493855000003, 2024-04-22T18:39:40.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100803438, 2024-04-22T18:39:40.685 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.686 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.686 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:40.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:40.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:40.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:40.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598108+0000", 2024-04-22T18:39:40.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9119999999999997e-06 2024-04-22T18:39:40.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598118+0000", 2024-04-22T18:39:40.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0138e-05 2024-04-22T18:39:40.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598125+0000", 2024-04-22T18:39:40.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3010000000000001e-06 2024-04-22T18:39:40.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698224+0000", 2024-04-22T18:39:40.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099221 2024-04-22T18:39:40.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698239+0000", 2024-04-22T18:39:40.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5053e-05 2024-04-22T18:39:40.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698875+0000", 2024-04-22T18:39:40.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063560699999999995 2024-04-22T18:39:40.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698906+0000", 2024-04-22T18:39:40.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1547999999999997e-05 2024-04-22T18:39:40.689 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.689 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.689 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.690 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.5.13:5 3.0 e30/19 3:1ab765eb:::smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=30'21)", 2024-04-22T18:39:40.690 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:40.690 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.478175731999997, 2024-04-22T18:39:40.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101014042, 2024-04-22T18:39:40.690 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.690 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.690 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:40.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:40.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:40.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:40.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202428+0000", 2024-04-22T18:39:40.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5889999999999997e-06 2024-04-22T18:39:40.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202446+0000", 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8176999999999999e-05 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202456+0000", 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302599+0000", 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100142759 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302628+0000", 2024-04-22T18:39:40.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9507999999999999e-05 2024-04-22T18:39:40.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303350+0000", 2024-04-22T18:39:40.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072151599999999995 2024-04-22T18:39:40.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303435+0000", 2024-04-22T18:39:40.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5486000000000001e-05 2024-04-22T18:39:40.693 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.693 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.694 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.694 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:53 3.0 e31/19 3:25a201b0:::smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 31'24, mlcod=30'22)", 2024-04-22T18:39:40.694 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:40.694 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.277533398000003, 2024-04-22T18:39:40.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100751671, 2024-04-22T18:39:40.694 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.694 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.694 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:40.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403062+0000", 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403070+0000", 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8499999999999994e-06 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403079+0000", 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4620000000000002e-06 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403087+0000", 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4789999999999999e-06 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503186+0000", 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099271 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503207+0000", 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0973999999999999e-05 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503773+0000", 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056557600000000003 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503815+0000", 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2441999999999998e-05 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.698 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:73 3.0 e34/19 3:061a7a63:::392f44ec26aa22e85be5f08273a7e45656f3bf4a4854038391a88e880ec01077:head v 34'31, mlcod=31'29)", 2024-04-22T18:39:40.698 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:40.698 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.222669949999997, 2024-04-22T18:39:40.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103503311, 2024-04-22T18:39:40.698 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.698 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.698 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:40.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:40.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:40.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:40.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458193+0000", 2024-04-22T18:39:40.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026628900000000001 2024-04-22T18:39:40.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458208+0000", 2024-04-22T18:39:40.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4780000000000001e-05 2024-04-22T18:39:40.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458217+0000", 2024-04-22T18:39:40.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4909999999999994e-06 2024-04-22T18:39:40.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459162+0000", 2024-04-22T18:39:40.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094440500000000005 2024-04-22T18:39:40.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459185+0000", 2024-04-22T18:39:40.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3685000000000001e-05 2024-04-22T18:39:40.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561394+0000", 2024-04-22T18:39:40.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10220874100000001 2024-04-22T18:39:40.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561430+0000", 2024-04-22T18:39:40.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6304999999999999e-05 2024-04-22T18:39:40.701 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.701 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.701 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.702 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:75 3.0 e34/19 3:18418e0f:::b7c35f165c374d5e075dde9ac2607e47ace3b0a1e2343ec6efb171380f725eea:head v 34'32, mlcod=31'29)", 2024-04-22T18:39:40.702 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:40.702 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.221656676999999, 2024-04-22T18:39:40.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104798078, 2024-04-22T18:39:40.702 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.702 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.702 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:40.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:40.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458939+0000", 2024-04-22T18:39:40.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:40.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459092+0000", 2024-04-22T18:39:40.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153838 2024-04-22T18:39:40.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459107+0000", 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4703e-05 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459115+0000", 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560093+0000", 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10097840499999999 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560109+0000", 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5631e-05 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563706+0000", 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035972059999999999 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563738+0000", 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2184000000000002e-05 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.706 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:77 3.0 e34/19 3:28387c9f:::b6d7840f02d561da04e398fbf94d91b85e9066b426e48baf3b11dd4a4f6096c0:head v 34'33, mlcod=31'29)", 2024-04-22T18:39:40.706 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:40.706 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.220706597000003, 2024-04-22T18:39:40.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104186079, 2024-04-22T18:39:40.706 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.706 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.706 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:40.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:40.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459889+0000", 2024-04-22T18:39:40.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:40.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459963+0000", 2024-04-22T18:39:40.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3732999999999994e-05 2024-04-22T18:39:40.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459973+0000", 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0285999999999999e-05 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459987+0000", 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3675999999999999e-05 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560515+0000", 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10052781299999999 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560538+0000", 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3861000000000001e-05 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564044+0000", 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035052009999999999 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564076+0000", 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2737000000000002e-05 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.710 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:78 3.0 e34/19 3:3a5582ac:::bdd92f7deaa2d50e204891f64f9043469ba687d78b218ed518f2483ba3486a99:head v 34'34, mlcod=31'29)", 2024-04-22T18:39:40.710 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:40.710 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.220040478999998, 2024-04-22T18:39:40.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103547263, 2024-04-22T18:39:40.710 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.710 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.710 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:40.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:40.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460555+0000", 2024-04-22T18:39:40.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:40.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460861+0000", 2024-04-22T18:39:40.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000305958 2024-04-22T18:39:40.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460874+0000", 2024-04-22T18:39:40.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2639e-05 2024-04-22T18:39:40.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460881+0000", 2024-04-22T18:39:40.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2570000000000001e-06 2024-04-22T18:39:40.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560990+0000", 2024-04-22T18:39:40.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010883199999999 2024-04-22T18:39:40.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561011+0000", 2024-04-22T18:39:40.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1319e-05 2024-04-22T18:39:40.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564079+0000", 2024-04-22T18:39:40.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00306815 2024-04-22T18:39:40.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564104+0000", 2024-04-22T18:39:40.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4172000000000002e-05 2024-04-22T18:39:40.713 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.714 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.714 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.714 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:39:40.714 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:40.714 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.219425184999999, 2024-04-22T18:39:40.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:39:40.714 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.714 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.714 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:40.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:40.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:39:40.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:40.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:39:40.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:39:40.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:39:40.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:39:40.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:39:40.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:39:40.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:39:40.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:39:40.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:39:40.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:39:40.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:39:40.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:39:40.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.342751362999998, 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:39:40.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:39:40.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:39:40.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:39:40.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:39:40.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:39:40.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:39:40.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:39:40.722 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.722 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.722 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.722 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1112 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'6, mlcod=29'4)", 2024-04-22T18:39:40.722 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:12.585713+0000", 2024-04-22T18:39:40.722 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.094883456000002, 2024-04-22T18:39:40.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046198259999999996, 2024-04-22T18:39:40.722 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.722 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.722 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.585713+0000", 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.585713+0000", 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.585712+0000", 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586020+0000", 2024-04-22T18:39:40.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030764900000000001 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586031+0000", 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0883000000000001e-05 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586037+0000", 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1260000000000003e-06 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586076+0000", 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8841999999999998e-05 2024-04-22T18:39:40.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.586098+0000", 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2279000000000001e-05 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.590302+0000", 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042039419999999996 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:12.590333+0000", 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1170000000000001e-05 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.725 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1131 3.0 e41/19 3:01bdda42:::4a159391123755d93670b0d142ca36f3ee9a35236f6c4396c2aa92521a42a06b:head v 41'50, mlcod=41'48)", 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.503004518000001, 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101639456, 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177590+0000", 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177880+0000", 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029001 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177893+0000", 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177899+0000", 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1619999999999996e-06 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.278054+0000", 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100154929 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.278076+0000", 2024-04-22T18:39:40.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1308e-05 2024-04-22T18:39:40.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.279187+0000", 2024-04-22T18:39:40.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011109169999999999 2024-04-22T18:39:40.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.279232+0000", 2024-04-22T18:39:40.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5188000000000002e-05 2024-04-22T18:39:40.729 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.729 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.729 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.730 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1159 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'10, mlcod=41'7)", 2024-04-22T18:39:40.730 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.452702+0000", 2024-04-22T18:39:40.730 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.227894968000001, 2024-04-22T18:39:40.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050095280000000001, 2024-04-22T18:39:40.730 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.730 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.730 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452702+0000", 2024-04-22T18:39:40.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452702+0000", 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452701+0000", 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452976+0000", 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027566 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452989+0000", 2024-04-22T18:39:40.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2615e-05 2024-04-22T18:39:40.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.452998+0000", 2024-04-22T18:39:40.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7399999999999993e-06 2024-04-22T18:39:40.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.453040+0000", 2024-04-22T18:39:40.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2395000000000003e-05 2024-04-22T18:39:40.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.453060+0000", 2024-04-22T18:39:40.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0332000000000001e-05 2024-04-22T18:39:40.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.457682+0000", 2024-04-22T18:39:40.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046215010000000001 2024-04-22T18:39:40.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.457711+0000", 2024-04-22T18:39:40.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9490999999999999e-05 2024-04-22T18:39:40.733 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.734 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.734 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.734 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1160 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'11, mlcod=41'8)", 2024-04-22T18:39:40.734 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.555212+0000", 2024-04-22T18:39:40.734 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.125384635, 2024-04-22T18:39:40.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004833586, 2024-04-22T18:39:40.734 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.734 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.734 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555212+0000", 2024-04-22T18:39:40.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555212+0000", 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555209+0000", 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555369+0000", 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015995000000000001 2024-04-22T18:39:40.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555383+0000", 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3558e-05 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.555389+0000", 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2759999999999997e-06 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556393+0000", 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001003629 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556418+0000", 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4915000000000001e-05 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.560010+0000", 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003591782 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.560046+0000", 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6285000000000002e-05 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.738 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1161 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'12, mlcod=41'9)", 2024-04-22T18:39:40.738 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.556330+0000", 2024-04-22T18:39:40.738 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.124267217, 2024-04-22T18:39:40.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099789016999999994, 2024-04-22T18:39:40.738 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.738 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.738 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556330+0000", 2024-04-22T18:39:40.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556330+0000", 2024-04-22T18:39:40.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556328+0000", 2024-04-22T18:39:40.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:40.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556335+0000", 2024-04-22T18:39:40.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9770000000000003e-06 2024-04-22T18:39:40.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556346+0000", 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0118999999999999e-05 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556353+0000", 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3730000000000004e-06 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.655508+0000", 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099155219000000003 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.655536+0000", 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-05 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.656078+0000", 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054138999999999999 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.656119+0000", 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1026000000000003e-05 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:125 3.0 e43/19 3:376bc4bb:::8f967acf6e5865f87cd7cada681f96ff59d5fc3f243a710c00c052aacb952df3:head v 43'60, mlcod=42'57)", 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.833031135999999, 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100905994, 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847565+0000", 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847610+0000", 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4984000000000002e-05 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847619+0000", 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2189999999999993e-06 2024-04-22T18:39:40.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947667+0000", 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100047939 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947716+0000", 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9228999999999999e-05 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947738+0000", 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1963999999999998e-05 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.948440+0000", 2024-04-22T18:39:40.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070173999999999996 2024-04-22T18:39:40.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.948472+0000", 2024-04-22T18:39:40.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1872999999999998e-05 2024-04-22T18:39:40.745 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.745 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.745 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.745 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1298 4.d e43/20 4:bc165c76:::smithi08937929-30:head v 43'17, mlcod=32'12)", 2024-04-22T18:39:40.745 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:18.062040+0000", 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.618557332999998, 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099368133999999997, 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062040+0000", 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062040+0000", 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062039+0000", 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062056+0000", 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6911000000000001e-05 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062060+0000", 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3000000000000003e-06 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.062065+0000", 2024-04-22T18:39:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1150000000000002e-06 2024-04-22T18:39:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.160817+0000", 2024-04-22T18:39:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098751539999999999 2024-04-22T18:39:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.160845+0000", 2024-04-22T18:39:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7908000000000001e-05 2024-04-22T18:39:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.161378+0000", 2024-04-22T18:39:40.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053300799999999996 2024-04-22T18:39:40.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:18.161408+0000", 2024-04-22T18:39:40.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.989e-05 2024-04-22T18:39:40.749 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.749 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.749 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.749 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:79 3.0 e44/19 3:05e6fa13:::smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 44'64, mlcod=44'61)", 2024-04-22T18:39:40.749 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:40.749 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.309749839999998, 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10093532199999999, 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370846+0000", 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370857+0000", 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1155000000000001e-05 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370866+0000", 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8769999999999999e-06 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.470937+0000", 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100071204 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.470986+0000", 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8556000000000001e-05 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471014+0000", 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7982000000000001e-05 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471750+0000", 2024-04-22T18:39:40.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073598100000000003 2024-04-22T18:39:40.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471782+0000", 2024-04-22T18:39:40.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339000000000003e-05 2024-04-22T18:39:40.753 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.753 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.753 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.753 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1689 3.0 e47/19 3:1d4bcadf:::e7eb8db0531d9f456e91fe87bc79e47ebd80459f1783dc2a5e53d4822cd34f49:head v 47'85, mlcod=47'83)", 2024-04-22T18:39:40.753 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:39:40.753 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.9717089049999998, 2024-04-22T18:39:40.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102152033, 2024-04-22T18:39:40.753 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.754 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.754 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:39:40.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:39:40.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708885+0000", 2024-04-22T18:39:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709414+0000", 2024-04-22T18:39:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000528459 2024-04-22T18:39:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709428+0000", 2024-04-22T18:39:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4745e-05 2024-04-22T18:39:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709436+0000", 2024-04-22T18:39:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0830000000000004e-06 2024-04-22T18:39:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.809587+0000", 2024-04-22T18:39:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100150641 2024-04-22T18:39:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.809603+0000", 2024-04-22T18:39:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6061000000000001e-05 2024-04-22T18:39:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.811013+0000", 2024-04-22T18:39:40.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0014094680000000001 2024-04-22T18:39:40.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.811040+0000", 2024-04-22T18:39:40.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7474000000000002e-05 2024-04-22T18:39:40.757 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.757 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.757 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.757 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1729 3.0 e47/19 3:27b110a1:::34336650543f8f8bff225091f9c8c14678dc47baabcb4d8371890fc935dbc3e4:head v 47'89, mlcod=47'87)", 2024-04-22T18:39:40.757 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:39:40.757 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.2026551350000005, 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102111412, 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477940+0000", 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478305+0000", 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036512200000000001 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478320+0000", 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5475999999999999e-05 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478326+0000", 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.302e-06 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.578536+0000", 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100209436 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.578575+0000", 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9223e-05 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.580023+0000", 2024-04-22T18:39:40.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001447463 2024-04-22T18:39:40.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:40.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.580053+0000", 2024-04-22T18:39:40.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0534999999999998e-05 2024-04-22T18:39:40.761 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.761 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.761 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.761 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.761 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.761 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:40.763 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:40.764 INFO:tasks.rados.rados.0.smithi089.stdout:1045: set_chunk oid 42 offset: 1775104 length: 923136 target oid b3a39b52fe44698617319691398e55963f15de3d676658e9990bcce8620354d3 offset: 0 snap -1 2024-04-22T18:39:40.764 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:40.764 INFO:tasks.rados.rados.0.smithi089.stdout:1031: finishing write tid 2 to smithi08937929-34 2024-04-22T18:39:40.764 INFO:tasks.rados.rados.0.smithi089.stdout:1031: finishing write tid 3 to smithi08937929-34 2024-04-22T18:39:40.764 INFO:tasks.rados.rados.0.smithi089.stdout:1031: finishing write tid 4 to smithi08937929-34 2024-04-22T18:39:40.764 INFO:tasks.rados.rados.0.smithi089.stdout:1031: oid 34 updating version 0 to 77 2024-04-22T18:39:40.764 INFO:tasks.rados.rados.0.smithi089.stdout:1031: oid 34 updating version 77 to 79 2024-04-22T18:39:40.764 INFO:tasks.rados.rados.0.smithi089.stdout:1031: oid 34 version 79 is already newer than 78 2024-04-22T18:39:40.764 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 79 (ObjNum 173 snap 0 seq_num 173) dirty exists 2024-04-22T18:39:40.764 INFO:tasks.rados.rados.0.smithi089.stdout:1031: left oid 34 (ObjNum 173 snap 0 seq_num 173) 2024-04-22T18:39:40.765 INFO:tasks.rados.rados.0.smithi089.stdout:1025: finishing write tid 5 to smithi08937929-43 2024-04-22T18:39:40.765 INFO:tasks.rados.rados.0.smithi089.stdout:1025: oid 43 updating version 0 to 25 2024-04-22T18:39:40.765 INFO:tasks.rados.rados.0.smithi089.stdout:1025: oid 43 updating version 25 to 28 2024-04-22T18:39:40.765 INFO:tasks.rados.rados.0.smithi089.stdout:1025: oid 43 version 28 is already newer than 27 2024-04-22T18:39:40.765 INFO:tasks.rados.rados.0.smithi089.stdout:1025: oid 43 version 28 is already newer than 26 2024-04-22T18:39:40.765 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 28 (ObjNum 170 snap 0 seq_num 170) dirty exists 2024-04-22T18:39:40.765 INFO:tasks.rados.rados.0.smithi089.stdout:1025: left oid 43 (ObjNum 170 snap 0 seq_num 170) 2024-04-22T18:39:40.765 INFO:tasks.rados.rados.0.smithi089.stdout:1025: done (15 left) 2024-04-22T18:39:40.765 INFO:tasks.rados.rados.0.smithi089.stdout:1031: done (14 left) 2024-04-22T18:39:40.765 INFO:tasks.rados.rados.0.smithi089.stdout:1046: set_chunk oid 47 target oid 2024-04-22T18:39:40.765 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:40.763+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:40.863 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:40.864+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:40.905 INFO:tasks.rados.rados.0.smithi089.stdout:1046: set_chunk oid 47 offset: 826880 length: 1231360 target oid eb9565077ec88edb51a31aa14b6ca59a52ef672d87ece233082a6c0a52f762e2 offset: 0 snap -1 2024-04-22T18:39:40.905 INFO:tasks.rados.rados.0.smithi089.stdout:1047: set_chunk oid 48 target oid 2024-04-22T18:39:40.907 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:40.907 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:40.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.907 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:248 3.2 e54/29)", 2024-04-22T18:39:40.907 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.881057+0000", 2024-04-22T18:39:40.907 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.028071743, 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028087766, 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.881057+0000", 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.881057+0000", 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.881055+0000", 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.881066+0000", 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1286e-05 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.881082+0000", 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5446000000000001e-05 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:40.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.881091+0000", 2024-04-22T18:39:40.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9900000000000003e-06 2024-04-22T18:39:40.910 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.910 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:40.910 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.910 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:40.910 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:39:40.910 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 1 2024-04-22T18:39:40.910 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:40.916 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:40.964 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:40.964+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:41.021 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:41.021 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:41.029 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:41.071 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:41.071+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:41.166 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:41.166+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:41.174 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:41.174 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:41.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:41.174 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:41.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.174 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:45 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'1, mlcod=0'0)", 2024-04-22T18:39:41.174 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:41.174 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.909360794999998, 2024-04-22T18:39:41.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14345418900000001, 2024-04-22T18:39:41.174 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.175 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.175 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:41.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:41.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265727+0000", 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061502799999999999 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265737+0000", 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0278999999999999e-05 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265744+0000", 2024-04-22T18:39:41.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1740000000000003e-06 2024-04-22T18:39:41.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407227+0000", 2024-04-22T18:39:41.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14148266900000001 2024-04-22T18:39:41.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407258+0000", 2024-04-22T18:39:41.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1619000000000003e-05 2024-04-22T18:39:41.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408534+0000", 2024-04-22T18:39:41.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001275829 2024-04-22T18:39:41.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408566+0000", 2024-04-22T18:39:41.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1813999999999997e-05 2024-04-22T18:39:41.178 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.178 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.178 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.178 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:46 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'2, mlcod=0'0)", 2024-04-22T18:39:41.178 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:41.178 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.906736391999999, 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14304645699999999, 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268301+0000", 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056585700000000001 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268309+0000", 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2830000000000004e-06 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268312+0000", 2024-04-22T18:39:41.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7179999999999998e-06 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407999+0000", 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13968636300000001 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408011+0000", 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2116999999999999e-05 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410762+0000", 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002751524 2024-04-22T18:39:41.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410783+0000", 2024-04-22T18:39:41.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0270000000000001e-05 2024-04-22T18:39:41.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.182 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.182 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:47 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'3, mlcod=0'0)", 2024-04-22T18:39:41.182 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:41.182 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.905003821999998, 2024-04-22T18:39:41.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14132698599999999, 2024-04-22T18:39:41.182 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.182 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.183 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:41.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:41.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269468+0000", 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269473+0000", 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7790000000000003e-06 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269478+0000", 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0080000000000002e-06 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269489+0000", 2024-04-22T18:39:41.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.075e-05 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408595+0000", 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13910662300000001 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408611+0000", 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6124e-05 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410786+0000", 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021743819999999999 2024-04-22T18:39:41.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410796+0000", 2024-04-22T18:39:41.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.007e-05 2024-04-22T18:39:41.186 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.186 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.186 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.186 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:78 4.0 4:01baf5a9:::smithi08937929-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:41.186 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:41.186 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.662537700999998, 2024-04-22T18:39:41.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.398522192, 2024-04-22T18:39:41.186 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.187 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.187 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:41.187 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:41.187 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.187 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 78 2024-04-22T18:39:41.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.187 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:41.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:41.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511934+0000", 2024-04-22T18:39:41.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:41.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512012+0000", 2024-04-22T18:39:41.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8329000000000002e-05 2024-04-22T18:39:41.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512023+0000", 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0755e-05 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512028+0000", 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6470000000000003e-06 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512068+0000", 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9721999999999997e-05 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512125+0000", 2024-04-22T18:39:41.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6564999999999998e-05 2024-04-22T18:39:41.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:41.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512200+0000", 2024-04-22T18:39:41.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5708000000000001e-05 2024-04-22T18:39:41.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:41.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513131+0000", 2024-04-22T18:39:41.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093063 2024-04-22T18:39:41.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910394+0000", 2024-04-22T18:39:41.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.397263388 2024-04-22T18:39:41.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910430+0000", 2024-04-22T18:39:41.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5485999999999999e-05 2024-04-22T18:39:41.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910457+0000", 2024-04-22T18:39:41.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7086000000000001e-05 2024-04-22T18:39:41.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.191 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:79 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.662437044999997, 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39855419399999997, 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 79 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:41.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:41.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512035+0000", 2024-04-22T18:39:41.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512045+0000", 2024-04-22T18:39:41.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0379e-05 2024-04-22T18:39:41.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512051+0000", 2024-04-22T18:39:41.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6030000000000004e-06 2024-04-22T18:39:41.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512057+0000", 2024-04-22T18:39:41.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7270000000000004e-06 2024-04-22T18:39:41.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512756+0000", 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069954000000000001 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512806+0000", 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9922000000000001e-05 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512927+0000", 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012059 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:41.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.515883+0000", 2024-04-22T18:39:41.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029563739999999999 2024-04-22T18:39:41.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910510+0000", 2024-04-22T18:39:41.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39462745199999999 2024-04-22T18:39:41.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910542+0000", 2024-04-22T18:39:41.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1225e-05 2024-04-22T18:39:41.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910590+0000", 2024-04-22T18:39:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8052e-05 2024-04-22T18:39:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:80 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.662411943999999, 2024-04-22T18:39:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39866376999999997, 2024-04-22T18:39:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:41.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.198 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 80 2024-04-22T18:39:41.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.198 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:41.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:41.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512060+0000", 2024-04-22T18:39:41.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512063+0000", 2024-04-22T18:39:41.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.999e-06 2024-04-22T18:39:41.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512069+0000", 2024-04-22T18:39:41.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.891e-06 2024-04-22T18:39:41.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512073+0000", 2024-04-22T18:39:41.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4429999999999996e-06 2024-04-22T18:39:41.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513084+0000", 2024-04-22T18:39:41.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010102609999999999 2024-04-22T18:39:41.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:41.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513123+0000", 2024-04-22T18:39:41.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9237999999999999e-05 2024-04-22T18:39:41.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910618+0000", 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39749549299999998 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910670+0000", 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1221000000000003e-05 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910724+0000", 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4777000000000001e-05 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.201 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.202 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.202 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:5 3.2 e30/29 3:663b82a1:::smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=0'0)", 2024-04-22T18:39:41.202 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:41.202 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.946972625999997, 2024-04-22T18:39:41.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20090134500000001, 2024-04-22T18:39:41.202 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.202 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.202 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227499+0000", 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227506+0000", 2024-04-22T18:39:41.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6540000000000002e-06 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227516+0000", 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0054000000000001e-05 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327607+0000", 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009167099999999 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327688+0000", 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1000000000000004e-05 2024-04-22T18:39:41.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327719+0000", 2024-04-22T18:39:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0664000000000003e-05 2024-04-22T18:39:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428371+0000", 2024-04-22T18:39:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10065239099999999 2024-04-22T18:39:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428401+0000", 2024-04-22T18:39:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9839e-05 2024-04-22T18:39:41.205 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:419 4.0 4:01baf5a9:::smithi08937929-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:39:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.772250923999998, 2024-04-22T18:39:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30324478199999999, 2024-04-22T18:39:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.206 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 419 2024-04-22T18:39:41.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.207 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:41.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:41.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402219+0000", 2024-04-22T18:39:41.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:41.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402229+0000", 2024-04-22T18:39:41.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2550000000000003e-06 2024-04-22T18:39:41.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402242+0000", 2024-04-22T18:39:41.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2989999999999999e-05 2024-04-22T18:39:41.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402250+0000", 2024-04-22T18:39:41.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7900000000000005e-06 2024-04-22T18:39:41.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404085+0000", 2024-04-22T18:39:41.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001834397 2024-04-22T18:39:41.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:41.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404092+0000", 2024-04-22T18:39:41.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7889999999999997e-06 2024-04-22T18:39:41.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502534+0000", 2024-04-22T18:39:41.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098442331999999994 2024-04-22T18:39:41.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502631+0000", 2024-04-22T18:39:41.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7617999999999999e-05 2024-04-22T18:39:41.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:41.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604414+0000", 2024-04-22T18:39:41.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101782847 2024-04-22T18:39:41.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604888+0000", 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047325999999999998 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705376+0000", 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100488623 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705424+0000", 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7964999999999999e-05 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705466+0000", 2024-04-22T18:39:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2274999999999998e-05 2024-04-22T18:39:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:70 3.2 e32/29 3:5f0623ce:::smithi08937929-38 low_tier:head v 32'37, mlcod=31'35)", 2024-04-22T18:39:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.646033893000002, 2024-04-22T18:39:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20070027900000001, 2024-04-22T18:39:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:41.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:41.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528438+0000", 2024-04-22T18:39:41.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528443+0000", 2024-04-22T18:39:41.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8640000000000004e-06 2024-04-22T18:39:41.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528453+0000", 2024-04-22T18:39:41.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3910000000000004e-06 2024-04-22T18:39:41.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528459+0000", 2024-04-22T18:39:41.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-04-22T18:39:41.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728627+0000", 2024-04-22T18:39:41.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20016778700000001 2024-04-22T18:39:41.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728660+0000", 2024-04-22T18:39:41.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3024000000000001e-05 2024-04-22T18:39:41.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729107+0000", 2024-04-22T18:39:41.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044719200000000001 2024-04-22T18:39:41.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729139+0000", 2024-04-22T18:39:41.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1625000000000003e-05 2024-04-22T18:39:41.216 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.216 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.216 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.216 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1264 4.c e43/20 4:36664a98:::smithi08937929-38:head v 43'37, mlcod=42'34)", 2024-04-22T18:39:41.216 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:41.216 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.518135488999999, 2024-04-22T18:39:41.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20290602399999999, 2024-04-22T18:39:41.216 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.216 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.217 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:41.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:41.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656335+0000", 2024-04-22T18:39:41.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:41.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656850+0000", 2024-04-22T18:39:41.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051499499999999997 2024-04-22T18:39:41.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656861+0000", 2024-04-22T18:39:41.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1119e-05 2024-04-22T18:39:41.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756934+0000", 2024-04-22T18:39:41.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073548 2024-04-22T18:39:41.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756977+0000", 2024-04-22T18:39:41.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2218000000000001e-05 2024-04-22T18:39:41.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756993+0000", 2024-04-22T18:39:41.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6629999999999998e-05 2024-04-22T18:39:41.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859203+0000", 2024-04-22T18:39:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102209617 2024-04-22T18:39:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859243+0000", 2024-04-22T18:39:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0333000000000002e-05 2024-04-22T18:39:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.220 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1745 4.c e48/20 4:35eda2ca:::smithi08937929-12:head v 48'63, mlcod=47'61)", 2024-04-22T18:39:41.221 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:32.493414+0000", 2024-04-22T18:39:41.221 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.6810589559999993, 2024-04-22T18:39:41.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107006161, 2024-04-22T18:39:41.221 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.221 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.221 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.493414+0000", 2024-04-22T18:39:41.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.493414+0000", 2024-04-22T18:39:41.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.493410+0000", 2024-04-22T18:39:41.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:41.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.494215+0000", 2024-04-22T18:39:41.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00080466700000000001 2024-04-22T18:39:41.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.494239+0000", 2024-04-22T18:39:41.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3754999999999999e-05 2024-04-22T18:39:41.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.494251+0000", 2024-04-22T18:39:41.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2082999999999999e-05 2024-04-22T18:39:41.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597906+0000", 2024-04-22T18:39:41.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.1036551 2024-04-22T18:39:41.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597926+0000", 2024-04-22T18:39:41.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0254999999999998e-05 2024-04-22T18:39:41.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.600390+0000", 2024-04-22T18:39:41.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0024633430000000002 2024-04-22T18:39:41.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.600420+0000", 2024-04-22T18:39:41.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0077e-05 2024-04-22T18:39:41.224 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.224 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.224 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.225 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1747 4.c e48/20 4:35eda2ca:::smithi08937929-12:head v 48'65, mlcod=47'61)", 2024-04-22T18:39:41.225 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:32.597827+0000", 2024-04-22T18:39:41.225 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5766453469999995, 2024-04-22T18:39:41.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20085196499999999, 2024-04-22T18:39:41.225 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.225 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.225 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597827+0000", 2024-04-22T18:39:41.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597827+0000", 2024-04-22T18:39:41.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597825+0000", 2024-04-22T18:39:41.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:41.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597839+0000", 2024-04-22T18:39:41.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4329e-05 2024-04-22T18:39:41.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597856+0000", 2024-04-22T18:39:41.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6858e-05 2024-04-22T18:39:41.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.697948+0000", 2024-04-22T18:39:41.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092367 2024-04-22T18:39:41.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798102+0000", 2024-04-22T18:39:41.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100153411 2024-04-22T18:39:41.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798122+0000", 2024-04-22T18:39:41.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0069e-05 2024-04-22T18:39:41.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798628+0000", 2024-04-22T18:39:41.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050619700000000001 2024-04-22T18:39:41.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798679+0000", 2024-04-22T18:39:41.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1307000000000002e-05 2024-04-22T18:39:41.228 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.229 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.229 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.229 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1855 4.f e51/20 4:f0ed3d49:::smithi08937929-9:head v 51'31, mlcod=47'29)", 2024-04-22T18:39:41.229 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:35.291419+0000", 2024-04-22T18:39:41.229 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.8830534239999999, 2024-04-22T18:39:41.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10796507700000001, 2024-04-22T18:39:41.229 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.229 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.229 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.291419+0000", 2024-04-22T18:39:41.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.291419+0000", 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.291419+0000", 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.291707+0000", 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000288592 2024-04-22T18:39:41.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.291716+0000", 2024-04-22T18:39:41.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0049999999999993e-06 2024-04-22T18:39:41.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.291725+0000", 2024-04-22T18:39:41.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1869999999999997e-06 2024-04-22T18:39:41.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.297113+0000", 2024-04-22T18:39:41.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053880330000000004 2024-04-22T18:39:41.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.297140+0000", 2024-04-22T18:39:41.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7126000000000001e-05 2024-04-22T18:39:41.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.399352+0000", 2024-04-22T18:39:41.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102212637 2024-04-22T18:39:41.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.399384+0000", 2024-04-22T18:39:41.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1859000000000001e-05 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1883 4.f e51/20 4:f83a159b:::smithi08937929-2:head v 51'35, mlcod=51'32)", 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:35.579245+0000", 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.5952275379999996, 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10463091400000001, 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.579245+0000", 2024-04-22T18:39:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.579245+0000", 2024-04-22T18:39:41.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.579244+0000", 2024-04-22T18:39:41.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:41.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.579593+0000", 2024-04-22T18:39:41.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000348833 2024-04-22T18:39:41.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.579604+0000", 2024-04-22T18:39:41.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1626e-05 2024-04-22T18:39:41.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.579613+0000", 2024-04-22T18:39:41.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4230000000000005e-06 2024-04-22T18:39:41.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.679989+0000", 2024-04-22T18:39:41.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100376099 2024-04-22T18:39:41.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.680005+0000", 2024-04-22T18:39:41.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6237999999999999e-05 2024-04-22T18:39:41.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.683849+0000", 2024-04-22T18:39:41.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038441270000000001 2024-04-22T18:39:41.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.683876+0000", 2024-04-22T18:39:41.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6874999999999999e-05 2024-04-22T18:39:41.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.237 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.237 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:229 3.2 e51/29 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head v 51'117, mlcod=51'115)", 2024-04-22T18:39:41.237 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:39:41.237 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.3634298859999996, 2024-04-22T18:39:41.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31548799399999999, 2024-04-22T18:39:41.237 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.237 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.237 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:39:41.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:39:41.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811041+0000", 2024-04-22T18:39:41.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:41.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811054+0000", 2024-04-22T18:39:41.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3188e-05 2024-04-22T18:39:41.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811068+0000", 2024-04-22T18:39:41.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4239999999999999e-05 2024-04-22T18:39:41.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.911141+0000", 2024-04-22T18:39:41.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100072851 2024-04-22T18:39:41.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025053+0000", 2024-04-22T18:39:41.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.11391237 2024-04-22T18:39:41.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025083+0000", 2024-04-22T18:39:41.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9703999999999999e-05 2024-04-22T18:39:41.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126504+0000", 2024-04-22T18:39:41.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10142053500000001 2024-04-22T18:39:41.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126531+0000", 2024-04-22T18:39:41.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7172999999999999e-05 2024-04-22T18:39:41.241 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.241 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.241 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.241 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:235 3.2 e51/29 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head v 51'118, mlcod=51'115)", 2024-04-22T18:39:41.241 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.924871+0000", 2024-04-22T18:39:41.241 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.249601685, 2024-04-22T18:39:41.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20167500599999999, 2024-04-22T18:39:41.242 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.242 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.242 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924871+0000", 2024-04-22T18:39:41.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924871+0000", 2024-04-22T18:39:41.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924869+0000", 2024-04-22T18:39:41.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:41.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924883+0000", 2024-04-22T18:39:41.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3725e-05 2024-04-22T18:39:41.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924899+0000", 2024-04-22T18:39:41.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6344000000000001e-05 2024-04-22T18:39:41.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924907+0000", 2024-04-22T18:39:41.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0909999999999993e-06 2024-04-22T18:39:41.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025426+0000", 2024-04-22T18:39:41.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100519109 2024-04-22T18:39:41.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025451+0000", 2024-04-22T18:39:41.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4338e-05 2024-04-22T18:39:41.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126534+0000", 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101083083 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126546+0000", 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2122e-05 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:238 3.2 e51/29 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head v 51'119, mlcod=51'115)", 2024-04-22T18:39:41.245 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.925398+0000", 2024-04-22T18:39:41.246 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.2490749030000003, 2024-04-22T18:39:41.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.201162223, 2024-04-22T18:39:41.246 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.246 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.246 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925398+0000", 2024-04-22T18:39:41.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925398+0000", 2024-04-22T18:39:41.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925396+0000", 2024-04-22T18:39:41.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:41.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925405+0000", 2024-04-22T18:39:41.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6600000000000001e-06 2024-04-22T18:39:41.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925417+0000", 2024-04-22T18:39:41.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2697000000000001e-05 2024-04-22T18:39:41.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925425+0000", 2024-04-22T18:39:41.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3000000000000004e-06 2024-04-22T18:39:41.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.125722+0000", 2024-04-22T18:39:41.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20029754299999999 2024-04-22T18:39:41.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.125763+0000", 2024-04-22T18:39:41.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0408999999999999e-05 2024-04-22T18:39:41.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126548+0000", 2024-04-22T18:39:41.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00078496600000000001 2024-04-22T18:39:41.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126560+0000", 2024-04-22T18:39:41.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2227e-05 2024-04-22T18:39:41.250 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.250 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.250 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.250 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:118 3.2 e51/29 3:43d85947:::smithi08937929-18 low_tier:head v 51'120, mlcod=51'115)", 2024-04-22T18:39:41.250 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.983526+0000", 2024-04-22T18:39:41.250 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.1909469159999997, 2024-04-22T18:39:41.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14328775899999999, 2024-04-22T18:39:41.250 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.250 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.250 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983526+0000", 2024-04-22T18:39:41.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983526+0000", 2024-04-22T18:39:41.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983524+0000", 2024-04-22T18:39:41.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:41.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983538+0000", 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3858e-05 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983552+0000", 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4466e-05 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983562+0000", 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3899999999999999e-06 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126168+0000", 2024-04-22T18:39:41.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14260677899999999 2024-04-22T18:39:41.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126185+0000", 2024-04-22T18:39:41.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6625000000000001e-05 2024-04-22T18:39:41.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126794+0000", 2024-04-22T18:39:41.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060855999999999996 2024-04-22T18:39:41.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126813+0000", 2024-04-22T18:39:41.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9851e-05 2024-04-22T18:39:41.254 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.254 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.254 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.254 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:119 3.2 e51/29 3:43d85947:::smithi08937929-18 low_tier:head v 51'121, mlcod=51'115)", 2024-04-22T18:39:41.254 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:41.254 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.1906242159999998, 2024-04-22T18:39:41.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.145803025, 2024-04-22T18:39:41.254 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.254 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.255 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:41.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:41.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:41.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:41.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983854+0000", 2024-04-22T18:39:41.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.207e-06 2024-04-22T18:39:41.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983860+0000", 2024-04-22T18:39:41.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5350000000000001e-06 2024-04-22T18:39:41.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983865+0000", 2024-04-22T18:39:41.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0170000000000004e-06 2024-04-22T18:39:41.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126364+0000", 2024-04-22T18:39:41.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.142498491 2024-04-22T18:39:41.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126378+0000", 2024-04-22T18:39:41.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4134999999999999e-05 2024-04-22T18:39:41.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129628+0000", 2024-04-22T18:39:41.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032498940000000001 2024-04-22T18:39:41.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129651+0000", 2024-04-22T18:39:41.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3493999999999999e-05 2024-04-22T18:39:41.258 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.258 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.258 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.258 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:120 3.2 e51/29 3:43d85947:::smithi08937929-18 low_tier:head v 51'122, mlcod=51'115)", 2024-04-22T18:39:41.258 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.984299+0000", 2024-04-22T18:39:41.259 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.1901738210000001, 2024-04-22T18:39:41.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14536710899999999, 2024-04-22T18:39:41.259 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:41.259 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.259 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:41.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:41.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984299+0000", 2024-04-22T18:39:41.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:41.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984299+0000", 2024-04-22T18:39:41.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:41.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:41.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984298+0000", 2024-04-22T18:39:41.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:41.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984302+0000", 2024-04-22T18:39:41.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.929e-06 2024-04-22T18:39:41.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:41.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984309+0000", 2024-04-22T18:39:41.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2080000000000003e-06 2024-04-22T18:39:41.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984316+0000", 2024-04-22T18:39:41.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7329999999999997e-06 2024-04-22T18:39:41.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:41.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126582+0000", 2024-04-22T18:39:41.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14226603400000001 2024-04-22T18:39:41.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:41.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126595+0000", 2024-04-22T18:39:41.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3458000000000001e-05 2024-04-22T18:39:41.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:41.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129653+0000", 2024-04-22T18:39:41.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030578860000000001 2024-04-22T18:39:41.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:41.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:41.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:41.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129666+0000", 2024-04-22T18:39:41.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2483999999999999e-05 2024-04-22T18:39:41.262 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.262 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.262 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:41.263 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:41.263 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:41.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:41.267 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:41.267+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:41.293 INFO:tasks.rados.rados.0.smithi089.stdout:1047: set_chunk oid 48 offset: 302080 length: 1179648 target oid d15b035c5252c2f3ce4c1e10de8a773fdee97bffb1c7d1c25cf1ba121181b6da offset: 0 snap -1 2024-04-22T18:39:41.294 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:41.294 INFO:tasks.rados.rados.0.smithi089.stdout:1032: finishing write tid 1 to smithi08937929-24 2024-04-22T18:39:41.294 INFO:tasks.rados.rados.0.smithi089.stdout:1032: finishing write tid 2 to smithi08937929-24 2024-04-22T18:39:41.294 INFO:tasks.rados.rados.0.smithi089.stdout:1033: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-22T18:39:41.367 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:41.368+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:41.370 INFO:tasks.rados.rados.0.smithi089.stdout:1038: expect (ObjNum 156 snap 0 seq_num 156) 2024-04-22T18:39:41.377 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:41.377 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:41.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.377 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:136 3.3 3.46ca8ef3 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e54)", 2024-04-22T18:39:41.377 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:41.296135+0000", 2024-04-22T18:39:41.377 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.082827259, 2024-04-22T18:39:41.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.082850438999999998, 2024-04-22T18:39:41.377 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.377 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "no flag points reached", 2024-04-22T18:39:41.377 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296135+0000", 2024-04-22T18:39:41.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296135+0000", 2024-04-22T18:39:41.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296132+0000", 2024-04-22T18:39:41.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:41.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296144+0000", 2024-04-22T18:39:41.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1516e-05 2024-04-22T18:39:41.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296159+0000", 2024-04-22T18:39:41.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5338000000000001e-05 2024-04-22T18:39:41.379 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.379 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.379 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.379 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.379 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:39:41.379 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 1 2024-04-22T18:39:41.379 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:41.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:41.470 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:41.471+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:41.521 INFO:tasks.rados.rados.0.smithi089.stdout:1044: expect (ObjNum 171 snap 0 seq_num 171) 2024-04-22T18:39:41.528 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:41.528 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:41.528 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:41.528 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:41.528 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:41.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:41.568 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:41.568+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:41.668 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:41.669+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:41.680 INFO:tasks.rados.rados.0.smithi089.stdout:1034: expect (ObjNum 157 snap 0 seq_num 157) 2024-04-22T18:39:41.681 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:41.681 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:41.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:41.681 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:41.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.681 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:41.682 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:41.682 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.518838019, 2024-04-22T18:39:41.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:39:41.682 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.682 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.682 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:39:41.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.682 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:41.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:41.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:39:41.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:41.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:39:41.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:39:41.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:39:41.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:39:41.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:39:41.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:39:41.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:39:41.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:39:41.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:39:41.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:39:41.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:39:41.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:39:41.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:39:41.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:41.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:39:41.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:39:41.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:39:41.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:39:41.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:41.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:39:41.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:39:41.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:39:41.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:39:41.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.687 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:41.687 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:41.688 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.417279198000003, 2024-04-22T18:39:41.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:39:41.688 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.688 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.688 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:39:41.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.688 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:41.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:41.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:39:41.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:41.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:39:41.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:39:41.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:39:41.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:39:41.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:39:41.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:39:41.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:39:41.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:39:41.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:39:41.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:39:41.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:39:41.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:39:41.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:39:41.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:39:41.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:39:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.417252796, 2024-04-22T18:39:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:39:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:39:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:39:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:39:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:39:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:39:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:39:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:39:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:39:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:39:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:39:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:39:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:39:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:39:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:39:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:39:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:39:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:39:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:39:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.018918466999999, 2024-04-22T18:39:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:39:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:39:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:39:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:39:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:39:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:39:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:39:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:39:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:39:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:39:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:39:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:39:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:39:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:39:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:39:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:39:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:39:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:39:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:39:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.918082169000002, 2024-04-22T18:39:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:39:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:39:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:39:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:39:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:39:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:39:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:39:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:39:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:39:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:39:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:39:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:39:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:39:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:39:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:39:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:39:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:39:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.917924898999999, 2024-04-22T18:39:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:39:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:39:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:39:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:39:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:39:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:39:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:39:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:39:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:39:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:39:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:39:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:39:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:39:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:39:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:39:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:39:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:39:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:39:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:39:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:39:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:39:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:39:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:39:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.917910290000002, 2024-04-22T18:39:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:39:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:39:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:39:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:39:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:39:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:39:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:39:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:39:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:39:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:39:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:39:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:39:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:39:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:39:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:39:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:39:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:39:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:39:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.917899904999999, 2024-04-22T18:39:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:39:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:39:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:39:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:39:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:39:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:39:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:39:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:39:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:39:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:39:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:39:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:39:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:39:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:39:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:39:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:39:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:39:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:39:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:39:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:39:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:39:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:39:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:39:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.916969975000001, 2024-04-22T18:39:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:39:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:39:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:39:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:39:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:39:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:39:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:39:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:39:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:39:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:39:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:39:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:39:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:39:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:39:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:39:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:39:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:39:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.916507850000002, 2024-04-22T18:39:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:39:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:39:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:39:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:39:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:39:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:39:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:39:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:39:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:39:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:39:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:39:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:39:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:39:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:39:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:39:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:39:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:39:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:39:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:39:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.916495046000001, 2024-04-22T18:39:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:39:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:39:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:39:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:39:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:39:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:39:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:39:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:39:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:39:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:39:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:39:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:39:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:39:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:39:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:39:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:39:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:39:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:39:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:39:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:109 4.1 4:83c955bf:::smithi08937929-28:head [write 453789~780782,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.713708052999998, 2024-04-22T18:39:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.45086113300000003, 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967588+0000", 2024-04-22T18:39:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.968243+0000", 2024-04-22T18:39:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00065488500000000002 2024-04-22T18:39:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.968265+0000", 2024-04-22T18:39:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1997e-05 2024-04-22T18:39:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.068340+0000", 2024-04-22T18:39:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007516 2024-04-22T18:39:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168464+0000", 2024-04-22T18:39:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100124267 2024-04-22T18:39:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168579+0000", 2024-04-22T18:39:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114387 2024-04-22T18:39:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168692+0000", 2024-04-22T18:39:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011335100000000001 2024-04-22T18:39:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.173549+0000", 2024-04-22T18:39:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048564430000000002 2024-04-22T18:39:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418367+0000", 2024-04-22T18:39:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.244817962 2024-04-22T18:39:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418419+0000", 2024-04-22T18:39:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2883000000000002e-05 2024-04-22T18:39:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418450+0000", 2024-04-22T18:39:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0278999999999999e-05 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 43.588722590000003, 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:39:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:39:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:39:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:39:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:39:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:39:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:39:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:39:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:39:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:39:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:39:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:39:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:39:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:39:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:39:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:39:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 42.580149478000003, 2024-04-22T18:39:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:39:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:39:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:39:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:39:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:39:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:39:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:39:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:39:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:39:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:39:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:39:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:39:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:39:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:39:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1108 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 3378174] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.637716+0000", 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.043580648999999, 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41175930500000002, 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1108 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637716+0000", 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637716+0000", 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637715+0000", 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637719+0000", 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1710000000000004e-06 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637725+0000", 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7790000000000001e-06 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637730+0000", 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7330000000000003e-06 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.948647+0000", 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31091748400000002 2024-04-22T18:39:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.948704+0000", 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7000000000000003e-05 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.948802+0000", 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8107000000000001e-05 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049249+0000", 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100446991 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049270+0000", 2024-04-22T18:39:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0336e-05 2024-04-22T18:39:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049369+0000", 2024-04-22T18:39:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9256e-05 2024-04-22T18:39:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049475+0000", 2024-04-22T18:39:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010624900000000001 2024-04-22T18:39:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1109 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.637734+0000", 2024-04-22T18:39:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.043563040999999, 2024-04-22T18:39:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41190551399999997, 2024-04-22T18:39:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1109 2024-04-22T18:39:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637734+0000", 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637734+0000", 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637733+0000", 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637736+0000", 2024-04-22T18:39:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4530000000000001e-06 2024-04-22T18:39:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637741+0000", 2024-04-22T18:39:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.118e-06 2024-04-22T18:39:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637745+0000", 2024-04-22T18:39:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4490000000000001e-06 2024-04-22T18:39:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.949123+0000", 2024-04-22T18:39:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31137820900000002 2024-04-22T18:39:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.949205+0000", 2024-04-22T18:39:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1444000000000002e-05 2024-04-22T18:39:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049516+0000", 2024-04-22T18:39:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031090099999999 2024-04-22T18:39:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049573+0000", 2024-04-22T18:39:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7497000000000002e-05 2024-04-22T18:39:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049639+0000", 2024-04-22T18:39:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5906000000000002e-05 2024-04-22T18:39:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1285 4.d 4:bee1184f:::smithi08937929-7:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.844031804, 2024-04-22T18:39:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.42388892500000003, 2024-04-22T18:39:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1285 2024-04-22T18:39:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837262+0000", 2024-04-22T18:39:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837273+0000", 2024-04-22T18:39:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1364e-05 2024-04-22T18:39:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837300+0000", 2024-04-22T18:39:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6928999999999999e-05 2024-04-22T18:39:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837318+0000", 2024-04-22T18:39:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8219999999999998e-05 2024-04-22T18:39:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837445+0000", 2024-04-22T18:39:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000126415 2024-04-22T18:39:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837627+0000", 2024-04-22T18:39:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000182182 2024-04-22T18:39:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.261154+0000", 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.42352673499999999 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.545501963, 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:39:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:39:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:39:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:39:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:39:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:39:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:39:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:39:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:39:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:39:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:39:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:39:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:39:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1684 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:39:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:39:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.076017999999999, 2024-04-22T18:39:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987885199999998, 2024-04-22T18:39:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1684 2024-04-22T18:39:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:39:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:39:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605278+0000", 2024-04-22T18:39:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605280+0000", 2024-04-22T18:39:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1770000000000001e-06 2024-04-22T18:39:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605285+0000", 2024-04-22T18:39:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-04-22T18:39:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605289+0000", 2024-04-22T18:39:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.687e-06 2024-04-22T18:39:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705352+0000", 2024-04-22T18:39:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006322199999999 2024-04-22T18:39:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705423+0000", 2024-04-22T18:39:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1136999999999993e-05 2024-04-22T18:39:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105017+0000", 2024-04-22T18:39:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.399593268 2024-04-22T18:39:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105081+0000", 2024-04-22T18:39:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4703999999999996e-05 2024-04-22T18:39:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105157+0000", 2024-04-22T18:39:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6205999999999995e-05 2024-04-22T18:39:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1902 4.9 4:9ec4daef:::smithi08937929-27:head [write 1946413~679049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.307159+0000", 2024-04-22T18:39:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.3741376159999996, 2024-04-22T18:39:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41122182200000001, 2024-04-22T18:39:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1902 2024-04-22T18:39:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.307159+0000", 2024-04-22T18:39:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.307159+0000", 2024-04-22T18:39:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.307156+0000", 2024-04-22T18:39:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.307698+0000", 2024-04-22T18:39:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054243300000000004 2024-04-22T18:39:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.307711+0000", 2024-04-22T18:39:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.252e-05 2024-04-22T18:39:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.307717+0000", 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.049e-06 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.316502+0000", 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087848209999999999 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.316571+0000", 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9133000000000004e-05 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.316663+0000", 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2230000000000003e-05 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.320733+0000", 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004069909 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.718304+0000", 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39757116999999997 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.718335+0000", 2024-04-22T18:39:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1331999999999997e-05 2024-04-22T18:39:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.718381+0000", 2024-04-22T18:39:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5389000000000002e-05 2024-04-22T18:39:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:41.787 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:41.788 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:41.717+0000 7f1e2dd22640 -1 osd.2 54 handle_osd_map injecting map crc failure 2024-04-22T18:39:41.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:41.789 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:41.790+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:41.889 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:41.889+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:41.889 INFO:tasks.rados.rados.0.smithi089.stdout:1032: finishing write tid 3 to smithi08937929-24 2024-04-22T18:39:41.889 INFO:tasks.rados.rados.0.smithi089.stdout:1032: finishing write tid 4 to smithi08937929-24 2024-04-22T18:39:41.889 INFO:tasks.rados.rados.0.smithi089.stdout:1036: expect (ObjNum 162 snap 0 seq_num 162) 2024-04-22T18:39:41.931 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:41.931 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:41.931 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:41.931 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:41.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:41.989 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:41.989+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:42.027 INFO:tasks.rados.rados.0.smithi089.stdout:1032: finishing write tid 5 to smithi08937929-24 2024-04-22T18:39:42.028 INFO:tasks.rados.rados.0.smithi089.stdout:1032: oid 24 updating version 0 to 95 2024-04-22T18:39:42.028 INFO:tasks.rados.rados.0.smithi089.stdout:1032: oid 24 version 95 is already newer than 94 2024-04-22T18:39:42.028 INFO:tasks.rados.rados.0.smithi089.stdout:1032: oid 24 updating version 95 to 97 2024-04-22T18:39:42.028 INFO:tasks.rados.rados.0.smithi089.stdout:1032: oid 24 version 97 is already newer than 96 2024-04-22T18:39:42.028 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 97 (ObjNum 174 snap 0 seq_num 174) dirty exists 2024-04-22T18:39:42.028 INFO:tasks.rados.rados.0.smithi089.stdout:1032: left oid 24 (ObjNum 174 snap 0 seq_num 174) 2024-04-22T18:39:42.028 INFO:tasks.rados.rados.0.smithi089.stdout:1032: done (15 left) 2024-04-22T18:39:42.028 INFO:tasks.rados.rados.0.smithi089.stdout:1033: done (14 left) 2024-04-22T18:39:42.028 INFO:tasks.rados.rados.0.smithi089.stdout:1034: done (13 left) 2024-04-22T18:39:42.028 INFO:tasks.rados.rados.0.smithi089.stdout:1036: done (12 left) 2024-04-22T18:39:42.028 INFO:tasks.rados.rados.0.smithi089.stdout:1038: done (11 left) 2024-04-22T18:39:42.028 INFO:tasks.rados.rados.0.smithi089.stdout:1044: done (10 left) 2024-04-22T18:39:42.029 INFO:tasks.rados.rados.0.smithi089.stdout:1048: tier_promote oid 22 2024-04-22T18:39:42.029 INFO:tasks.rados.rados.0.smithi089.stdout:1040: expect (ObjNum 119 snap 0 seq_num 119) 2024-04-22T18:39:42.041 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:42.041 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:42.041 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:42.041 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:42.041 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:42.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:42.079 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:39:42.094 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:42.095+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:42.128 INFO:tasks.rados.rados.0.smithi089.stdout:1037: finishing write tid 1 to smithi08937929-37 2024-04-22T18:39:42.128 INFO:tasks.rados.rados.0.smithi089.stdout:1042: finishing write tid 1 to smithi08937929-1 2024-04-22T18:39:42.128 INFO:tasks.rados.rados.0.smithi089.stdout:1039: expect (ObjNum 138 snap 0 seq_num 138) 2024-04-22T18:39:42.155 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:42.156 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:42.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:42.156 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:42.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.156 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:42.156 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:42.156 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 48.061853550000002, 2024-04-22T18:39:42.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:39:42.156 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.156 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.156 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:42.156 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:42.156 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:42.156 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:39:42.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.157 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:42.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:42.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:39:42.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:42.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:39:42.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:39:42.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:39:42.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:39:42.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:39:42.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:39:42.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:39:42.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:39:42.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:39:42.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:39:42.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:42.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:39:42.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:39:42.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:39:42.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:39:42.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:39:42.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:39:42.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:39:42.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:39:42.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:39:42.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:39:42.161 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.161 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.162 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.162 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:42.162 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:42.162 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 44.162355675000001, 2024-04-22T18:39:42.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:39:42.162 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.162 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.162 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:42.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:39:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:39:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:39:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:39:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:39:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:39:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:39:42.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:39:42.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:39:42.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:39:42.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:39:42.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:39:42.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:39:42.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:42.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:39:42.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:39:42.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:39:42.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:39:42.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:39:42.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:39:42.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:39:42.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:39:42.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:39:42.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:39:42.167 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.167 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.167 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.167 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 43.054055884, 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:42.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:39:42.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:42.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:39:42.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:39:42.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:39:42.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:39:42.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:39:42.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:39:42.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:39:42.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:39:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:39:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:39:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:39:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:39:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:39:42.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:39:42.172 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.173 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.173 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.173 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:42.173 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:42.173 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.722512256000002, 2024-04-22T18:39:42.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:39:42.173 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.173 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.173 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:42.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:42.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:39:42.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:42.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:39:42.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:39:42.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:39:42.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:39:42.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:39:42.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:39:42.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:39:42.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:39:42.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:39:42.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:39:42.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:42.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:39:42.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:39:42.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:39:42.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:39:42.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:39:42.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:39:42.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:39:42.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:39:42.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:39:42.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:39:42.178 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.178 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.178 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.178 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:42.178 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:42.178 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.631426447999999, 2024-04-22T18:39:42.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:39:42.179 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.179 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.179 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:42.179 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:42.179 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:42.179 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:39:42.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.179 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:42.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.625626187999998, 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:42.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:39:42.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:39:42.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:39:42.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:39:42.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:39:42.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:39:42.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:39:42.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:39:42.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:39:42.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:42.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:39:42.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:39:42.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:39:42.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:39:42.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:39:42.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:39:42.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:39:42.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:39:42.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:39:42.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:39:42.189 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.189 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.189 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.189 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:42.189 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:42.189 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.620885845, 2024-04-22T18:39:42.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:39:42.189 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.189 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.189 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:42.189 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:39:42.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.620412946999998, 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:42.194 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:39:42.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:39:42.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:39:42.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:39:42.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:39:42.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:39:42.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:39:42.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:39:42.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:39:42.199 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.199 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.199 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.199 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:42.199 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:42.199 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.620393025999999, 2024-04-22T18:39:42.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:39:42.199 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.204 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:42.204 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:39:42.204 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.311069261, 2024-04-22T18:39:42.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:39:42.204 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.204 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.204 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:39:42.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:39:42.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:39:42.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:39:42.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:39:42.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:39:42.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:39:42.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:39:42.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:39:42.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:42.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:39:42.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:39:42.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:39:42.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:39:42.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.310861724999999, 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:39:42.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:42.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:39:42.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:39:42.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:39:42.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:39:42.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:42.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:39:42.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:39:42.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:39:42.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:39:42.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:39:42.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:39:42.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:39:42.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:39:42.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:39:42.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:39:42.213 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.213 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.213 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.213 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:39:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.4452119919999999, 2024-04-22T18:39:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:39:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:39:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:39:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:39:42.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:42.218 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:39:42.219 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.2313736070000001, 2024-04-22T18:39:42.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:39:42.219 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.219 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.219 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:39:42.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:39:42.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:39:42.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:42.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:39:42.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:39:42.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:39:42.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:39:42.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:39:42.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:39:42.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:39:42.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:39:42.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:39:42.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:39:42.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:42.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:39:42.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:39:42.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:39:42.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:39:42.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.2313401969999997, 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:39:42.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:39:42.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:42.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:39:42.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:39:42.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:39:42.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:39:42.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:39:42.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:39:42.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:39:42.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:39:42.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:39:42.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:39:42.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:42.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:39:42.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:39:42.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:39:42.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:39:42.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:39:42.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:42.228 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:39:42.229 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.1724575430000002, 2024-04-22T18:39:42.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:39:42.229 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.229 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.229 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:39:42.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:39:42.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:39:42.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:42.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:39:42.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:39:42.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:39:42.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:39:42.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:39:42.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:39:42.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:39:42.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:39:42.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:39:42.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:39:42.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:42.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:39:42.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:39:42.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:39:42.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:39:42.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:39:42.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:39:42.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:39:42.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:39:42.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:39:42.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:39:42.233 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.233 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.233 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:39:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.1723189270000001, 2024-04-22T18:39:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:39:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:39:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:39:42.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:39:42.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:42.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:39:42.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:39:42.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:39:42.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:39:42.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:39:42.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:39:42.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:39:42.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:39:42.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:39:42.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:39:42.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:42.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:39:42.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:39:42.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:39:42.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:39:42.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:39:42.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:39:42.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:39:42.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:39:42.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:39:42.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:39:42.238 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.1722999239999998, 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:39:42.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:39:42.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:42.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:39:42.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:39:42.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:39:42.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:39:42.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:39:42.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:39:42.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:39:42.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:39:42.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.243 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.551287391, 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:39:42.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:39:42.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:39:42.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:39:42.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:39:42.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:39:42.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:39:42.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.248 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.5507678, 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:39:42.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:39:42.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:39:42.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:39:42.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:39:42.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:39:42.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:39:42.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:39:42.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:39:42.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:39:42.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:42.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:39:42.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:39:42.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:39:42.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:39:42.254 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.254 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.254 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.254 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:39:42.254 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:39:42.254 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.450614917, 2024-04-22T18:39:42.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:39:42.254 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:42.254 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:42.254 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:42.254 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:42.254 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:39:42.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:39:42.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:42.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:39:42.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:39:42.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:42.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:42.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:42.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:39:42.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:39:42.258 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.258 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.258 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.258 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:42.258 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:42.258 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:42.259 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:42.260 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:42.260 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:42.260 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:42.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:42.261 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:42.261+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:42.261 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:42.262 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:42.262 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:42.262 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:42.263 INFO:tasks.rados.rados.0.smithi089.stdout:1037: finishing write tid 2 to smithi08937929-37 2024-04-22T18:39:42.263 INFO:tasks.rados.rados.0.smithi089.stdout:1037: finishing write tid 3 to smithi08937929-37 2024-04-22T18:39:42.263 INFO:tasks.rados.rados.0.smithi089.stdout:1043: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:39:42.263 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:42.263 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:42.278 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:42.278 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:42.294 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:42.295 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:42.311 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:42.311 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:42.360 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:42.361+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:42.383 INFO:tasks.rados.rados.0.smithi089.stdout:1042: finishing write tid 2 to smithi08937929-1 2024-04-22T18:39:42.383 INFO:tasks.rados.rados.0.smithi089.stdout:1037: finishing write tid 4 to smithi08937929-37 2024-04-22T18:39:42.383 INFO:tasks.rados.rados.0.smithi089.stdout:1037: oid 37 updating version 0 to 80 2024-04-22T18:39:42.383 INFO:tasks.rados.rados.0.smithi089.stdout:1037: oid 37 updating version 80 to 81 2024-04-22T18:39:42.383 INFO:tasks.rados.rados.0.smithi089.stdout:1037: oid 37 version 81 is already newer than 79 2024-04-22T18:39:42.383 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 81 (ObjNum 175 snap 0 seq_num 175) dirty exists 2024-04-22T18:39:42.383 INFO:tasks.rados.rados.0.smithi089.stdout:1037: left oid 37 (ObjNum 175 snap 0 seq_num 175) 2024-04-22T18:39:42.383 INFO:tasks.rados.rados.0.smithi089.stdout:1045: finishing set_chunk to oid 42 2024-04-22T18:39:42.384 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 42 set_chunk returned error code -95 offset: 1775104 length: 923136 2024-04-22T18:39:42.384 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 0 (ObjNum 105 snap 0 seq_num 105) dirty exists 2024-04-22T18:39:42.384 INFO:tasks.rados.rados.0.smithi089.stdout:1041: expect (ObjNum 148 snap 0 seq_num 148) 2024-04-22T18:39:42.402 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:42.402 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:42.411 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:42.462 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:42.462+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:42.501 INFO:tasks.rados.rados.0.smithi089.stdout:1035: finishing set_chunk to oid 40 2024-04-22T18:39:42.501 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 19 (ObjNum 58 snap 0 seq_num 58) dirty exists 2024-04-22T18:39:42.501 INFO:tasks.rados.rados.0.smithi089.stdout:1042: finishing write tid 3 to smithi08937929-1 2024-04-22T18:39:42.501 INFO:tasks.rados.rados.0.smithi089.stdout:1042: finishing write tid 4 to smithi08937929-1 2024-04-22T18:39:42.501 INFO:tasks.rados.rados.0.smithi089.stdout:1042: oid 1 updating version 0 to 43 2024-04-22T18:39:42.501 INFO:tasks.rados.rados.0.smithi089.stdout:1042: oid 1 version 43 is already newer than 42 2024-04-22T18:39:42.501 INFO:tasks.rados.rados.0.smithi089.stdout:1042: oid 1 version 43 is already newer than 41 2024-04-22T18:39:42.501 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 43 (ObjNum 176 snap 0 seq_num 176) dirty exists 2024-04-22T18:39:42.502 INFO:tasks.rados.rados.0.smithi089.stdout:1042: left oid 1 (ObjNum 176 snap 0 seq_num 176) 2024-04-22T18:39:42.502 INFO:tasks.rados.rados.0.smithi089.stdout:1046: finishing set_chunk to oid 47 2024-04-22T18:39:42.502 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 37 (ObjNum 147 snap 0 seq_num 147) dirty exists 2024-04-22T18:39:42.502 INFO:tasks.rados.rados.0.smithi089.stdout:1047: finishing set_chunk to oid 48 2024-04-22T18:39:42.502 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 47 (ObjNum 128 snap 0 seq_num 128) dirty exists 2024-04-22T18:39:42.502 INFO:tasks.rados.rados.0.smithi089.stdout:1035: done (10 left) 2024-04-22T18:39:42.502 INFO:tasks.rados.rados.0.smithi089.stdout:1037: done (9 left) 2024-04-22T18:39:42.502 INFO:tasks.rados.rados.0.smithi089.stdout:1039: done (8 left) 2024-04-22T18:39:42.502 INFO:tasks.rados.rados.0.smithi089.stdout:1040: done (7 left) 2024-04-22T18:39:42.502 INFO:tasks.rados.rados.0.smithi089.stdout:1041: done (6 left) 2024-04-22T18:39:42.502 INFO:tasks.rados.rados.0.smithi089.stdout:1042: done (5 left) 2024-04-22T18:39:42.502 INFO:tasks.rados.rados.0.smithi089.stdout:1043: done (4 left) 2024-04-22T18:39:42.502 INFO:tasks.rados.rados.0.smithi089.stdout:1045: done (3 left) 2024-04-22T18:39:42.503 INFO:tasks.rados.rados.0.smithi089.stdout:1046: done (2 left) 2024-04-22T18:39:42.503 INFO:tasks.rados.rados.0.smithi089.stdout:1047: done (1 left) 2024-04-22T18:39:42.503 INFO:tasks.rados.rados.0.smithi089.stdout:1049: read oid 47 snap -1 2024-04-22T18:39:42.503 INFO:tasks.rados.rados.0.smithi089.stdout:1049: expect (ObjNum 147 snap 0 seq_num 147) 2024-04-22T18:39:42.503 INFO:tasks.rados.rados.0.smithi089.stdout:1050: set_chunk oid 3 target oid 2024-04-22T18:39:42.503 INFO:tasks.rados.rados.0.smithi089.stdout:1050: set_chunk oid 3 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:39:42.503 INFO:tasks.rados.rados.0.smithi089.stdout:1048: got (0) Success 2024-04-22T18:39:42.503 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 62 (ObjNum 135 snap 0 seq_num 135) dirty exists 2024-04-22T18:39:42.503 INFO:tasks.rados.rados.0.smithi089.stdout:1051: read oid 25 snap -1 2024-04-22T18:39:42.503 INFO:tasks.rados.rados.0.smithi089.stdout:1051: expect (ObjNum 172 snap 0 seq_num 172) 2024-04-22T18:39:42.503 INFO:tasks.rados.rados.0.smithi089.stdout:1048: done (3 left) 2024-04-22T18:39:42.503 INFO:tasks.rados.rados.0.smithi089.stdout:1052: set_chunk oid 12 target oid 2024-04-22T18:39:42.507 INFO:tasks.rados.rados.0.smithi089.stdout:1052: set_chunk oid 12 offset: 1437696 length: 548864 target oid 74988d658df6826c43828222038c5a12dc981d3a8d1809883dd13037db523e1b offset: 0 snap -1 2024-04-22T18:39:42.507 INFO:tasks.rados.rados.0.smithi089.stdout:1053: delete oid 13 current snap is 0 2024-04-22T18:39:42.507 INFO:tasks.rados.rados.0.smithi089.stdout:1051: expect (ObjNum 172 snap 0 seq_num 172) 2024-04-22T18:39:42.513 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:42.514 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:42.523 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:42.567 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:42.568+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:42.625 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:42.625 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:42.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:42.663 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:42.664+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:42.722 INFO:tasks.rados.rados.0.smithi089.stdout:1050: finishing set_chunk to oid 3 2024-04-22T18:39:42.722 INFO:tasks.rados.rados.0.smithi089.stdout:1050: got expected ENOENT (src dne) 2024-04-22T18:39:42.722 INFO:tasks.rados.rados.0.smithi089.stdout:1049: expect (ObjNum 147 snap 0 seq_num 147) 2024-04-22T18:39:42.763 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:39:42.764 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:42.765+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:42.775 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:42.775 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:42.784 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:42.865 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:42.865+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:42.869 INFO:tasks.rados.rados.0.smithi089.stdout:1052: finishing set_chunk to oid 12 2024-04-22T18:39:42.870 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 65 (ObjNum 141 snap 0 seq_num 141) dirty exists 2024-04-22T18:39:42.870 INFO:tasks.rados.rados.0.smithi089.stdout:1049: done (4 left) 2024-04-22T18:39:42.870 INFO:tasks.rados.rados.0.smithi089.stdout:1050: done (3 left) 2024-04-22T18:39:42.870 INFO:tasks.rados.rados.0.smithi089.stdout:1051: done (2 left) 2024-04-22T18:39:42.870 INFO:tasks.rados.rados.0.smithi089.stdout:1052: done (1 left) 2024-04-22T18:39:42.870 INFO:tasks.rados.rados.0.smithi089.stdout:1053: done (0 left) 2024-04-22T18:39:42.870 INFO:tasks.rados.rados.0.smithi089.stdout:1054: read oid 26 snap -1 2024-04-22T18:39:42.870 INFO:tasks.rados.rados.0.smithi089.stdout:1054: expect (ObjNum 140 snap 0 seq_num 140) 2024-04-22T18:39:42.870 INFO:tasks.rados.rados.0.smithi089.stdout:1055: read oid 44 snap -1 2024-04-22T18:39:42.870 INFO:tasks.rados.rados.0.smithi089.stdout:1055: expect (ObjNum 119 snap 0 seq_num 119) 2024-04-22T18:39:42.870 INFO:tasks.rados.rados.0.smithi089.stdout:1056: write oid 3 current snap is 0 2024-04-22T18:39:42.870 INFO:tasks.rados.rados.0.smithi089.stdout:1056: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2024-04-22T18:39:42.875 INFO:tasks.rados.rados.0.smithi089.stdout:1056: writing smithi08937929-3 from 591646 to 1062087 tid 1 2024-04-22T18:39:42.881 INFO:tasks.rados.rados.0.smithi089.stdout:1056: writing smithi08937929-3 from 1566383 to 2162100 tid 2 2024-04-22T18:39:42.886 INFO:tasks.rados.rados.0.smithi089.stdout:1056: writing smithi08937929-3 from 2888822 to 2991646 tid 3 2024-04-22T18:39:42.886 INFO:tasks.rados.rados.0.smithi089.stdout:1057: read oid 5 snap -1 2024-04-22T18:39:42.887 INFO:tasks.rados.rados.0.smithi089.stdout:1057: expect deleted 2024-04-22T18:39:42.887 INFO:tasks.rados.rados.0.smithi089.stdout:1058: read oid 18 snap -1 2024-04-22T18:39:42.887 INFO:tasks.rados.rados.0.smithi089.stdout:1058: expect (ObjNum 165 snap 0 seq_num 165) 2024-04-22T18:39:42.887 INFO:tasks.rados.rados.0.smithi089.stdout:1059: set_chunk oid 46 target oid 2024-04-22T18:39:42.926 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:42.926 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:42.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:42.979 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:42.980+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:43.037 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:43.037 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:43.046 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:43.087 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:43.088+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:43.149 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:43.149 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:43.158 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:43.167 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:43.168+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:43.262 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:43.263 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:43.268 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:43.269+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:43.272 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:43.297 INFO:tasks.rados.rados.0.smithi089.stdout:1059: set_chunk oid 46 offset: 1286656 length: 823808 target oid 21e6062cdc4e7409ce639d1edda35407909b9c1d8cbf0fcbd76a4393cf0b326a offset: 0 snap -1 2024-04-22T18:39:43.297 INFO:tasks.rados.rados.0.smithi089.stdout:1060: read oid 30 snap -1 2024-04-22T18:39:43.297 INFO:tasks.rados.rados.0.smithi089.stdout:1060: expect (ObjNum 144 snap 0 seq_num 144) 2024-04-22T18:39:43.297 INFO:tasks.rados.rados.0.smithi089.stdout:1061: write (excl) oid 36 current snap is 0 2024-04-22T18:39:43.298 INFO:tasks.rados.rados.0.smithi089.stdout:1061: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2024-04-22T18:39:43.304 INFO:tasks.rados.rados.0.smithi089.stdout:1061: writing smithi08937929-36 from 608453 to 1154143 tid 1 2024-04-22T18:39:43.309 INFO:tasks.rados.rados.0.smithi089.stdout:1061: writing smithi08937929-36 from 1624865 to 2080593 tid 2 2024-04-22T18:39:43.316 INFO:tasks.rados.rados.0.smithi089.stdout:1061: writing smithi08937929-36 from 2632598 to 3008453 tid 3 2024-04-22T18:39:43.316 INFO:tasks.rados.rados.0.smithi089.stdout:1062: delete oid 32 current snap is 0 2024-04-22T18:39:43.316 INFO:tasks.rados.rados.0.smithi089.stdout:1056: finishing write tid 1 to smithi08937929-3 2024-04-22T18:39:43.316 INFO:tasks.rados.rados.0.smithi089.stdout:1056: finishing write tid 2 to smithi08937929-3 2024-04-22T18:39:43.317 INFO:tasks.rados.rados.0.smithi089.stdout:1055: expect (ObjNum 119 snap 0 seq_num 119) 2024-04-22T18:39:43.369 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:43.370+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:43.381 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:43.381 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:43.390 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:43.420 INFO:tasks.rados.rados.0.smithi089.stdout:1058: expect (ObjNum 165 snap 0 seq_num 165) 2024-04-22T18:39:43.469 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:43.470+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:43.521 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:43.521 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:43.531 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:43.547 INFO:tasks.rados.rados.0.smithi089.stdout:1056: finishing write tid 3 to smithi08937929-3 2024-04-22T18:39:43.547 INFO:tasks.rados.rados.0.smithi089.stdout:1056: finishing write tid 4 to smithi08937929-3 2024-04-22T18:39:43.547 INFO:tasks.rados.rados.0.smithi089.stdout:1056: finishing write tid 5 to smithi08937929-3 2024-04-22T18:39:43.547 INFO:tasks.rados.rados.0.smithi089.stdout:1056: oid 3 updating version 0 to 40 2024-04-22T18:39:43.547 INFO:tasks.rados.rados.0.smithi089.stdout:1056: oid 3 version 40 is already newer than 39 2024-04-22T18:39:43.547 INFO:tasks.rados.rados.0.smithi089.stdout:1056: oid 3 updating version 40 to 41 2024-04-22T18:39:43.547 INFO:tasks.rados.rados.0.smithi089.stdout:1056: oid 3 version 41 is already newer than 38 2024-04-22T18:39:43.547 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 41 (ObjNum 178 snap 0 seq_num 178) dirty exists 2024-04-22T18:39:43.547 INFO:tasks.rados.rados.0.smithi089.stdout:1056: left oid 3 (ObjNum 178 snap 0 seq_num 178) 2024-04-22T18:39:43.547 INFO:tasks.rados.rados.0.smithi089.stdout:1054: expect (ObjNum 140 snap 0 seq_num 140) 2024-04-22T18:39:43.590 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:43.591+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:43.648 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:43.648 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:43.657 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:43.671 INFO:tasks.rados.rados.0.smithi089.stdout:1059: finishing set_chunk to oid 46 2024-04-22T18:39:43.671 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 46 set_chunk returned error code -95 offset: 1286656 length: 823808 2024-04-22T18:39:43.672 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-22T18:39:43.672 INFO:tasks.rados.rados.0.smithi089.stdout:1061: finishing write tid 1 to smithi08937929-36 2024-04-22T18:39:43.672 INFO:tasks.rados.rados.0.smithi089.stdout:1061: finishing write tid 2 to smithi08937929-36 2024-04-22T18:39:43.672 INFO:tasks.rados.rados.0.smithi089.stdout:1054: done (8 left) 2024-04-22T18:39:43.672 INFO:tasks.rados.rados.0.smithi089.stdout:1055: done (7 left) 2024-04-22T18:39:43.672 INFO:tasks.rados.rados.0.smithi089.stdout:1056: done (6 left) 2024-04-22T18:39:43.672 INFO:tasks.rados.rados.0.smithi089.stdout:1057: done (5 left) 2024-04-22T18:39:43.672 INFO:tasks.rados.rados.0.smithi089.stdout:1058: done (4 left) 2024-04-22T18:39:43.672 INFO:tasks.rados.rados.0.smithi089.stdout:1059: done (3 left) 2024-04-22T18:39:43.672 INFO:tasks.rados.rados.0.smithi089.stdout:1062: done (2 left) 2024-04-22T18:39:43.672 INFO:tasks.rados.rados.0.smithi089.stdout:1063: read oid 39 snap -1 2024-04-22T18:39:43.672 INFO:tasks.rados.rados.0.smithi089.stdout:1063: expect (ObjNum 164 snap 0 seq_num 164) 2024-04-22T18:39:43.672 INFO:tasks.rados.rados.0.smithi089.stdout:1064: read oid 16 snap -1 2024-04-22T18:39:43.672 INFO:tasks.rados.rados.0.smithi089.stdout:1064: expect (ObjNum 154 snap 0 seq_num 154) 2024-04-22T18:39:43.672 INFO:tasks.rados.rados.0.smithi089.stdout:1065: read oid 45 snap -1 2024-04-22T18:39:43.673 INFO:tasks.rados.rados.0.smithi089.stdout:1065: expect (ObjNum 166 snap 0 seq_num 166) 2024-04-22T18:39:43.673 INFO:tasks.rados.rados.0.smithi089.stdout:1066: write oid 18 current snap is 0 2024-04-22T18:39:43.673 INFO:tasks.rados.rados.0.smithi089.stdout:1061: finishing write tid 3 to smithi08937929-36 2024-04-22T18:39:43.673 INFO:tasks.rados.rados.0.smithi089.stdout:1060: expect (ObjNum 144 snap 0 seq_num 144) 2024-04-22T18:39:43.673 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:43.671+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:43.686 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:43.687+0000 7f7f44bb3640 -1 osd.4 56 handle_osd_map injecting map crc failure 2024-04-22T18:39:43.688 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:39:43.735 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:39:43.735 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:39:43.736 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-04-22T18:39:43.736 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:39:43.771 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:43.772+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:43.777 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:43.777 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:43.786 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:43.804 INFO:tasks.rados.rados.0.smithi089.stdout:1061: finishing write tid 4 to smithi08937929-36 2024-04-22T18:39:43.804 INFO:tasks.rados.rados.0.smithi089.stdout:1061: finishing write tid 5 to smithi08937929-36 2024-04-22T18:39:43.804 INFO:tasks.rados.rados.0.smithi089.stdout:1061: oid 36 updating version 0 to 46 2024-04-22T18:39:43.804 INFO:tasks.rados.rados.0.smithi089.stdout:1061: oid 36 version 46 is already newer than 45 2024-04-22T18:39:43.804 INFO:tasks.rados.rados.0.smithi089.stdout:1061: oid 36 updating version 46 to 47 2024-04-22T18:39:43.804 INFO:tasks.rados.rados.0.smithi089.stdout:1061: oid 36 version 47 is already newer than 44 2024-04-22T18:39:43.804 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 47 (ObjNum 179 snap 0 seq_num 179) dirty exists 2024-04-22T18:39:43.804 INFO:tasks.rados.rados.0.smithi089.stdout:1061: left oid 36 (ObjNum 179 snap 0 seq_num 179) 2024-04-22T18:39:43.804 INFO:tasks.rados.rados.0.smithi089.stdout:1066: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-04-22T18:39:43.811 INFO:tasks.rados.rados.0.smithi089.stdout:1066: writing smithi08937929-18 from 642067 to 1338255 tid 1 2024-04-22T18:39:43.812 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:43.813+0000 7fa766d7d640 -1 osd.5 56 handle_osd_map injecting map crc failure 2024-04-22T18:39:43.817 INFO:tasks.rados.rados.0.smithi089.stdout:1066: writing smithi08937929-18 from 2025476 to 2484873 tid 2 2024-04-22T18:39:43.821 INFO:tasks.rados.rados.0.smithi089.stdout:1066: writing smithi08937929-18 from 2971091 to 3042067 tid 3 2024-04-22T18:39:43.821 INFO:tasks.rados.rados.0.smithi089.stdout:1060: done (5 left) 2024-04-22T18:39:43.821 INFO:tasks.rados.rados.0.smithi089.stdout:1061: done (4 left) 2024-04-22T18:39:43.821 INFO:tasks.rados.rados.0.smithi089.stdout:1067: read oid 13 snap -1 2024-04-22T18:39:43.821 INFO:tasks.rados.rados.0.smithi089.stdout:1067: expect deleted 2024-04-22T18:39:43.821 INFO:tasks.rados.rados.0.smithi089.stdout:1068: read oid 36 snap -1 2024-04-22T18:39:43.821 INFO:tasks.rados.rados.0.smithi089.stdout:1068: expect (ObjNum 179 snap 0 seq_num 179) 2024-04-22T18:39:43.822 INFO:tasks.rados.rados.0.smithi089.stdout:1069: read oid 22 snap -1 2024-04-22T18:39:43.822 INFO:tasks.rados.rados.0.smithi089.stdout:1069: expect (ObjNum 135 snap 0 seq_num 135) 2024-04-22T18:39:43.822 INFO:tasks.rados.rados.0.smithi089.stdout:1070: read oid 9 snap -1 2024-04-22T18:39:43.822 INFO:tasks.rados.rados.0.smithi089.stdout:1070: expect (ObjNum 153 snap 0 seq_num 153) 2024-04-22T18:39:43.822 INFO:tasks.rados.rados.0.smithi089.stdout:1071: read oid 10 snap -1 2024-04-22T18:39:43.822 INFO:tasks.rados.rados.0.smithi089.stdout:1071: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:39:43.822 INFO:tasks.rados.rados.0.smithi089.stdout:1072: read oid 34 snap -1 2024-04-22T18:39:43.822 INFO:tasks.rados.rados.0.smithi089.stdout:1072: expect (ObjNum 173 snap 0 seq_num 173) 2024-04-22T18:39:43.822 INFO:tasks.rados.rados.0.smithi089.stdout:1073: set_chunk oid 5 target oid 2024-04-22T18:39:43.822 INFO:tasks.rados.rados.0.smithi089.stdout:1073: set_chunk oid 5 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:39:43.822 INFO:tasks.rados.rados.0.smithi089.stdout:1074: write (excl) oid 2 current snap is 0 2024-04-22T18:39:43.822 INFO:tasks.rados.rados.0.smithi089.stdout:1074: seq_num 182 ranges {658874=771437,2083958=603055} 2024-04-22T18:39:43.829 INFO:tasks.rados.rados.0.smithi089.stdout:1074: writing smithi08937929-2 from 658874 to 1430311 tid 1 2024-04-22T18:39:43.836 INFO:tasks.rados.rados.0.smithi089.stdout:1074: writing smithi08937929-2 from 2083958 to 2687013 tid 2 2024-04-22T18:39:43.837 INFO:tasks.rados.rados.0.smithi089.stdout:1075: read oid 27 snap -1 2024-04-22T18:39:43.837 INFO:tasks.rados.rados.0.smithi089.stdout:1075: expect (ObjNum 160 snap 0 seq_num 160) 2024-04-22T18:39:43.837 INFO:tasks.rados.rados.0.smithi089.stdout:1076: tier_evict oid 14 2024-04-22T18:39:43.837 INFO:tasks.rados.rados.0.smithi089.stdout:1076: tier_evict oid 14 snap -1 2024-04-22T18:39:43.837 INFO:tasks.rados.rados.0.smithi089.stdout:1077: read oid 26 snap -1 2024-04-22T18:39:43.837 INFO:tasks.rados.rados.0.smithi089.stdout:1077: expect (ObjNum 140 snap 0 seq_num 140) 2024-04-22T18:39:43.837 INFO:tasks.rados.rados.0.smithi089.stdout:1078: read oid 20 snap -1 2024-04-22T18:39:43.837 INFO:tasks.rados.rados.0.smithi089.stdout:1078: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-22T18:39:43.837 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:43.837 INFO:tasks.rados.rados.0.smithi089.stdout:1065: expect (ObjNum 166 snap 0 seq_num 166) 2024-04-22T18:39:43.872 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:43.873+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:43.913 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:43.913 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:43.923 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:44.007 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:44.007+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:44.032 INFO:tasks.rados.rados.0.smithi089.stdout:1064: expect (ObjNum 154 snap 0 seq_num 154) 2024-04-22T18:39:44.051 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:39:44.051 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:39:44.065 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-04-22T18:39:43.688484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":217478587,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":2448,"num_read_kb":1935215,"num_write":1595,"num_write_kb":413598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":16490875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":28,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1440,"ondisk_log_size":1440,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":473772,"kb_used_data":254460,"kb_used_omap":10,"kb_used_meta":201333,"kb_avail":660129108,"statfs":{"total":676457349120,"available":675972206592,"internally_reserved":0,"allocated":260567040,"data_stored":267218307,"data_compressed":7677874,"data_compressed_allocated":8200192,"data_compressed_original":18222709,"omap_allocated":11150,"internal_metadata":206165106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,11,0,16],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3735983,"num_objects":-10,"num_object_clones":0,"num_object_copies":-20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2,"num_objects_unfound":0,"num_objects_dirty":-7,"num_whiteouts":0,"num_read":305,"num_read_kb":266494,"num_write":157,"num_write_kb":33450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1463808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019330"},"pg_stats":[{"pgid":"4.f","version":"53'37","reported_seq":107,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.592583+0000","last_change":"2024-04-22T18:38:48.173640+0000","last_active":"2024-04-22T18:39:40.592583+0000","last_peered":"2024-04-22T18:39:40.592583+0000","last_clean":"2024-04-22T18:39:40.592583+0000","last_became_active":"2024-04-22T18:38:48.173369+0000","last_became_peered":"2024-04-22T18:38:48.173369+0000","last_unstale":"2024-04-22T18:39:40.592583+0000","last_undegraded":"2024-04-22T18:39:40.592583+0000","last_fullsized":"2024-04-22T18:39:40.592583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:08.819198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5193363,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":46506,"num_write":34,"num_write_kb":8816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"45'12","reported_seq":66,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:39.670415+0000","last_change":"2024-04-22T18:38:48.075222+0000","last_active":"2024-04-22T18:39:39.670415+0000","last_peered":"2024-04-22T18:39:39.670415+0000","last_clean":"2024-04-22T18:39:39.670415+0000","last_became_active":"2024-04-22T18:38:48.074901+0000","last_became_peered":"2024-04-22T18:38:48.074901+0000","last_unstale":"2024-04-22T18:39:39.670415+0000","last_undegraded":"2024-04-22T18:39:39.670415+0000","last_fullsized":"2024-04-22T18:39:39.670415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.172332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756315,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":21450,"num_write":11,"num_write_kb":3819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"51'28","reported_seq":113,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:39.670397+0000","last_change":"2024-04-22T18:39:34.390908+0000","last_active":"2024-04-22T18:39:39.670397+0000","last_peered":"2024-04-22T18:39:39.670397+0000","last_clean":"2024-04-22T18:39:39.670397+0000","last_became_active":"2024-04-22T18:39:33.856648+0000","last_became_peered":"2024-04-22T18:39:33.856648+0000","last_unstale":"2024-04-22T18:39:39.670397+0000","last_undegraded":"2024-04-22T18:39:39.670397+0000","last_fullsized":"2024-04-22T18:39:39.670397+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:02.104769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5126135,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":55367,"num_write":27,"num_write_kb":7537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3213860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"51'70","reported_seq":198,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.679469+0000","last_change":"2024-04-22T18:38:48.274538+0000","last_active":"2024-04-22T18:39:40.679469+0000","last_peered":"2024-04-22T18:39:40.679469+0000","last_clean":"2024-04-22T18:39:40.679469+0000","last_became_active":"2024-04-22T18:38:48.174204+0000","last_became_peered":"2024-04-22T18:38:48.174204+0000","last_unstale":"2024-04-22T18:39:40.679469+0000","last_undegraded":"2024-04-22T18:39:40.679469+0000","last_fullsized":"2024-04-22T18:39:40.679469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:11.823691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10991745,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":208,"num_read_kb":215756,"num_write":71,"num_write_kb":19760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"53'81","reported_seq":222,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.892039+0000","last_change":"2024-04-22T18:38:48.269362+0000","last_active":"2024-04-22T18:39:40.892039+0000","last_peered":"2024-04-22T18:39:40.892039+0000","last_clean":"2024-04-22T18:39:40.892039+0000","last_became_active":"2024-04-22T18:38:48.269066+0000","last_became_peered":"2024-04-22T18:38:48.269066+0000","last_unstale":"2024-04-22T18:39:40.892039+0000","last_undegraded":"2024-04-22T18:39:40.892039+0000","last_fullsized":"2024-04-22T18:39:40.892039+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:52.098608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12840515,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":220,"num_read_kb":235121,"num_write":81,"num_write_kb":24731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"52'45","reported_seq":137,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.712875+0000","last_change":"2024-04-22T18:38:48.372952+0000","last_active":"2024-04-22T18:39:40.712875+0000","last_peered":"2024-04-22T18:39:40.712875+0000","last_clean":"2024-04-22T18:39:40.712875+0000","last_became_active":"2024-04-22T18:38:48.277922+0000","last_became_peered":"2024-04-22T18:38:48.277922+0000","last_unstale":"2024-04-22T18:39:40.712875+0000","last_undegraded":"2024-04-22T18:39:40.712875+0000","last_fullsized":"2024-04-22T18:39:40.712875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:15.471951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7882483,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":130486,"num_write":41,"num_write_kb":10385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"53'79","reported_seq":215,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.292297+0000","last_change":"2024-04-22T18:38:48.271633+0000","last_active":"2024-04-22T18:39:40.292297+0000","last_peered":"2024-04-22T18:39:40.292297+0000","last_clean":"2024-04-22T18:39:40.292297+0000","last_became_active":"2024-04-22T18:38:48.171245+0000","last_became_peered":"2024-04-22T18:38:48.171245+0000","last_unstale":"2024-04-22T18:39:40.292297+0000","last_undegraded":"2024-04-22T18:39:40.292297+0000","last_fullsized":"2024-04-22T18:39:40.292297+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:47.449912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12722899,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":225,"num_read_kb":211770,"num_write":83,"num_write_kb":23908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"54'43","reported_seq":133,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T18:39:41.072446+0000","last_change":"2024-04-22T18:38:48.149226+0000","last_active":"2024-04-22T18:39:41.072446+0000","last_peered":"2024-04-22T18:39:41.072446+0000","last_clean":"2024-04-22T18:39:41.072446+0000","last_became_active":"2024-04-22T18:38:48.148955+0000","last_became_peered":"2024-04-22T18:38:48.148955+0000","last_unstale":"2024-04-22T18:39:41.072446+0000","last_undegraded":"2024-04-22T18:39:41.072446+0000","last_fullsized":"2024-04-22T18:39:41.072446+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:02.394854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9126168,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":143106,"num_write":44,"num_write_kb":12912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"54'29","reported_seq":107,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.712893+0000","last_change":"2024-04-22T18:38:48.277905+0000","last_active":"2024-04-22T18:39:40.712893+0000","last_peered":"2024-04-22T18:39:40.712893+0000","last_clean":"2024-04-22T18:39:40.712893+0000","last_became_active":"2024-04-22T18:38:48.277640+0000","last_became_peered":"2024-04-22T18:38:48.277640+0000","last_unstale":"2024-04-22T18:39:40.712893+0000","last_undegraded":"2024-04-22T18:39:40.712893+0000","last_fullsized":"2024-04-22T18:39:40.712893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:57.153161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5831996,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":90920,"num_write":24,"num_write_kb":6203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":196,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:39.670004+0000","last_change":"2024-04-22T18:38:38.109358+0000","last_active":"2024-04-22T18:39:39.670004+0000","last_peered":"2024-04-22T18:39:39.670004+0000","last_clean":"2024-04-22T18:39:39.670004+0000","last_became_active":"2024-04-22T18:38:38.108568+0000","last_became_peered":"2024-04-22T18:38:38.108568+0000","last_unstale":"2024-04-22T18:39:39.670004+0000","last_undegraded":"2024-04-22T18:39:39.670004+0000","last_fullsized":"2024-04-22T18:39:39.670004+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:38:36.991764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:46.436569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":55,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:39.875279+0000","last_change":"2024-04-22T18:39:34.035584+0000","last_active":"2024-04-22T18:39:39.875279+0000","last_peered":"2024-04-22T18:39:39.875279+0000","last_clean":"2024-04-22T18:39:39.875279+0000","last_became_active":"2024-04-22T18:39:33.935251+0000","last_became_peered":"2024-04-22T18:39:33.935251+0000","last_unstale":"2024-04-22T18:39:39.875279+0000","last_undegraded":"2024-04-22T18:39:39.875279+0000","last_fullsized":"2024-04-22T18:39:39.875279+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:51.995118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"54'137","reported_seq":197,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.982124+0000","last_change":"2024-04-22T18:38:58.429682+0000","last_active":"2024-04-22T18:39:40.982124+0000","last_peered":"2024-04-22T18:39:40.982124+0000","last_clean":"2024-04-22T18:39:40.982124+0000","last_became_active":"2024-04-22T18:38:58.125648+0000","last_became_peered":"2024-04-22T18:38:58.125648+0000","last_unstale":"2024-04-22T18:39:40.982124+0000","last_undegraded":"2024-04-22T18:39:40.982124+0000","last_fullsized":"2024-04-22T18:39:40.982124+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:52.346195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27535737,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":118,"num_read_kb":1588,"num_write":169,"num_write_kb":49040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"51'57","reported_seq":157,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.782910+0000","last_change":"2024-04-22T18:38:48.173369+0000","last_active":"2024-04-22T18:39:40.782910+0000","last_peered":"2024-04-22T18:39:40.782910+0000","last_clean":"2024-04-22T18:39:40.782910+0000","last_became_active":"2024-04-22T18:38:48.173090+0000","last_became_peered":"2024-04-22T18:38:48.173090+0000","last_unstale":"2024-04-22T18:39:40.782910+0000","last_undegraded":"2024-04-22T18:39:40.782910+0000","last_fullsized":"2024-04-22T18:39:40.782910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:05.687904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7579957,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":131,"num_read_kb":140425,"num_write":63,"num_write_kb":18755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":45,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:39.669960+0000","last_change":"2024-04-22T18:38:39.125885+0000","last_active":"2024-04-22T18:39:39.669960+0000","last_peered":"2024-04-22T18:39:39.669960+0000","last_clean":"2024-04-22T18:39:39.669960+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:39:39.669960+0000","last_undegraded":"2024-04-22T18:39:39.669960+0000","last_fullsized":"2024-04-22T18:39:39.669960+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:04.321452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"53'156","reported_seq":207,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T18:39:41.093369+0000","last_change":"2024-04-22T18:38:47.173337+0000","last_active":"2024-04-22T18:39:41.093369+0000","last_peered":"2024-04-22T18:39:41.093369+0000","last_clean":"2024-04-22T18:39:41.093369+0000","last_became_active":"2024-04-22T18:38:47.173065+0000","last_became_peered":"2024-04-22T18:38:47.173065+0000","last_unstale":"2024-04-22T18:39:41.093369+0000","last_undegraded":"2024-04-22T18:39:41.093369+0000","last_fullsized":"2024-04-22T18:39:41.093369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:52.987154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24882753,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":136,"num_read_kb":4355,"num_write":189,"num_write_kb":52696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"52'33","reported_seq":116,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:39.974450+0000","last_change":"2024-04-22T18:38:48.278028+0000","last_active":"2024-04-22T18:39:39.974450+0000","last_peered":"2024-04-22T18:39:39.974450+0000","last_clean":"2024-04-22T18:39:39.974450+0000","last_became_active":"2024-04-22T18:38:48.277802+0000","last_became_peered":"2024-04-22T18:38:48.277802+0000","last_unstale":"2024-04-22T18:39:39.974450+0000","last_undegraded":"2024-04-22T18:39:39.974450+0000","last_fullsized":"2024-04-22T18:39:39.974450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:12.000401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6084101,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":105222,"num_write":29,"num_write_kb":7958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.177761+0000","last_change":"2024-04-22T18:39:33.654430+0000","last_active":"2024-04-22T18:39:40.177761+0000","last_peered":"2024-04-22T18:39:40.177761+0000","last_clean":"2024-04-22T18:39:40.177761+0000","last_became_active":"2024-04-22T18:39:33.654150+0000","last_became_peered":"2024-04-22T18:39:33.654150+0000","last_unstale":"2024-04-22T18:39:40.177761+0000","last_undegraded":"2024-04-22T18:39:40.177761+0000","last_fullsized":"2024-04-22T18:39:40.177761+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.252385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"54'117","reported_seq":248,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T18:39:41.007988+0000","last_change":"2024-04-22T18:39:36.328830+0000","last_active":"2024-04-22T18:39:41.007988+0000","last_peered":"2024-04-22T18:39:41.007988+0000","last_clean":"2024-04-22T18:39:41.007988+0000","last_became_active":"2024-04-22T18:39:33.854362+0000","last_became_peered":"2024-04-22T18:39:33.854362+0000","last_unstale":"2024-04-22T18:39:41.007988+0000","last_undegraded":"2024-04-22T18:39:41.007988+0000","last_fullsized":"2024-04-22T18:39:41.007988+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.501273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17070740,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":97,"num_read_kb":810,"num_write":143,"num_write_kb":40721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":10577332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"51'19","reported_seq":90,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.604760+0000","last_change":"2024-04-22T18:39:34.143591+0000","last_active":"2024-04-22T18:39:40.604760+0000","last_peered":"2024-04-22T18:39:40.604760+0000","last_clean":"2024-04-22T18:39:40.604760+0000","last_became_active":"2024-04-22T18:39:33.731848+0000","last_became_peered":"2024-04-22T18:39:33.731848+0000","last_unstale":"2024-04-22T18:39:40.604760+0000","last_undegraded":"2024-04-22T18:39:40.604760+0000","last_fullsized":"2024-04-22T18:39:40.604760+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:59.939375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764735,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":37,"num_read_kb":45604,"num_write":19,"num_write_kb":6094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2699683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":47,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:39.875119+0000","last_change":"2024-04-22T18:38:39.111223+0000","last_active":"2024-04-22T18:39:39.875119+0000","last_peered":"2024-04-22T18:39:39.875119+0000","last_clean":"2024-04-22T18:39:39.875119+0000","last_became_active":"2024-04-22T18:38:39.110914+0000","last_became_peered":"2024-04-22T18:38:39.110914+0000","last_unstale":"2024-04-22T18:39:39.875119+0000","last_undegraded":"2024-04-22T18:39:39.875119+0000","last_fullsized":"2024-04-22T18:39:39.875119+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.570327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"54'187","reported_seq":233,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T18:39:41.295569+0000","last_change":"2024-04-22T18:38:47.173101+0000","last_active":"2024-04-22T18:39:41.295569+0000","last_peered":"2024-04-22T18:39:41.295569+0000","last_clean":"2024-04-22T18:39:41.295569+0000","last_became_active":"2024-04-22T18:38:47.172816+0000","last_became_peered":"2024-04-22T18:38:47.172816+0000","last_unstale":"2024-04-22T18:39:41.295569+0000","last_undegraded":"2024-04-22T18:39:41.295569+0000","last_fullsized":"2024-04-22T18:39:41.295569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:12.128375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24019066,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":156,"num_read_kb":2452,"num_write":220,"num_write_kb":65333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"51'32","reported_seq":100,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.074386+0000","last_change":"2024-04-22T18:38:48.372570+0000","last_active":"2024-04-22T18:39:40.074386+0000","last_peered":"2024-04-22T18:39:40.074386+0000","last_clean":"2024-04-22T18:39:40.074386+0000","last_became_active":"2024-04-22T18:38:48.272283+0000","last_became_peered":"2024-04-22T18:38:48.272283+0000","last_unstale":"2024-04-22T18:39:40.074386+0000","last_undegraded":"2024-04-22T18:39:40.074386+0000","last_fullsized":"2024-04-22T18:39:40.074386+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:09.183390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4890837,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":75335,"num_write":31,"num_write_kb":8230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":46,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:39.974392+0000","last_change":"2024-04-22T18:38:39.307735+0000","last_active":"2024-04-22T18:39:39.974392+0000","last_peered":"2024-04-22T18:39:39.974392+0000","last_clean":"2024-04-22T18:39:39.974392+0000","last_became_active":"2024-04-22T18:38:39.307459+0000","last_became_peered":"2024-04-22T18:38:39.307459+0000","last_unstale":"2024-04-22T18:39:39.974392+0000","last_undegraded":"2024-04-22T18:39:39.974392+0000","last_fullsized":"2024-04-22T18:39:39.974392+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.680561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"53'59","reported_seq":151,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.892605+0000","last_change":"2024-04-22T18:38:48.269379+0000","last_active":"2024-04-22T18:39:40.892605+0000","last_peered":"2024-04-22T18:39:40.892605+0000","last_clean":"2024-04-22T18:39:40.892605+0000","last_became_active":"2024-04-22T18:38:48.269070+0000","last_became_peered":"2024-04-22T18:38:48.269070+0000","last_unstale":"2024-04-22T18:39:40.892605+0000","last_undegraded":"2024-04-22T18:39:40.892605+0000","last_fullsized":"2024-04-22T18:39:40.892605+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:08.651956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10924550,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":115,"num_read_kb":140118,"num_write":59,"num_write_kb":17144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:39.669919+0000","last_change":"2024-04-22T18:38:39.210344+0000","last_active":"2024-04-22T18:39:39.669919+0000","last_peered":"2024-04-22T18:39:39.669919+0000","last_clean":"2024-04-22T18:39:39.669919+0000","last_became_active":"2024-04-22T18:38:39.210049+0000","last_became_peered":"2024-04-22T18:38:39.210049+0000","last_unstale":"2024-04-22T18:39:39.669919+0000","last_undegraded":"2024-04-22T18:39:39.669919+0000","last_fullsized":"2024-04-22T18:39:39.669919+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:03.878594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"31'5","reported_seq":58,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.483227+0000","last_change":"2024-04-22T18:38:48.349979+0000","last_active":"2024-04-22T18:39:40.483227+0000","last_peered":"2024-04-22T18:39:40.483227+0000","last_clean":"2024-04-22T18:39:40.483227+0000","last_became_active":"2024-04-22T18:38:48.349706+0000","last_became_peered":"2024-04-22T18:38:48.349706+0000","last_unstale":"2024-04-22T18:39:40.483227+0000","last_undegraded":"2024-04-22T18:39:40.483227+0000","last_fullsized":"2024-04-22T18:39:40.483227+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:54.435973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2319333,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":33325,"num_write":4,"num_write_kb":801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":46,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.074487+0000","last_change":"2024-04-22T18:38:39.110499+0000","last_active":"2024-04-22T18:39:40.074487+0000","last_peered":"2024-04-22T18:39:40.074487+0000","last_clean":"2024-04-22T18:39:40.074487+0000","last_became_active":"2024-04-22T18:38:39.010173+0000","last_became_peered":"2024-04-22T18:38:39.010173+0000","last_unstale":"2024-04-22T18:39:40.074487+0000","last_undegraded":"2024-04-22T18:39:40.074487+0000","last_fullsized":"2024-04-22T18:39:40.074487+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:39:58.222501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"53'97","reported_seq":239,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.891973+0000","last_change":"2024-04-22T18:38:48.372226+0000","last_active":"2024-04-22T18:39:40.891973+0000","last_peered":"2024-04-22T18:39:40.891973+0000","last_clean":"2024-04-22T18:39:40.891973+0000","last_became_active":"2024-04-22T18:38:48.371943+0000","last_became_peered":"2024-04-22T18:38:48.371943+0000","last_unstale":"2024-04-22T18:39:40.891973+0000","last_undegraded":"2024-04-22T18:39:40.891973+0000","last_fullsized":"2024-04-22T18:39:40.891973+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:00.637563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15344791,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":229,"num_read_kb":235399,"num_write":94,"num_write_kb":26339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":46,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-22T18:39:40.030137+0000","last_change":"2024-04-22T18:38:39.026078+0000","last_active":"2024-04-22T18:39:40.030137+0000","last_peered":"2024-04-22T18:39:40.030137+0000","last_clean":"2024-04-22T18:39:40.030137+0000","last_became_active":"2024-04-22T18:38:39.025741+0000","last_became_peered":"2024-04-22T18:38:39.025741+0000","last_unstale":"2024-04-22T18:39:40.030137+0000","last_undegraded":"2024-04-22T18:39:40.030137+0000","last_fullsized":"2024-04-22T18:39:40.030137+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:01.125349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":123379923,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":1825,"num_read_kb":1925910,"num_write":715,"num_write_kb":203392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5913543,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":28,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159150080,"data_stored":158894835,"data_compressed":406984,"data_compressed_allocated":442368,"data_compressed_original":1048576,"omap_allocated":0,"internal_metadata":0},"log_size":726,"ondisk_log_size":726,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":93508296,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":507,"num_read_kb":9205,"num_write":721,"num_write_kb":207790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":10577332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":111296512,"data_stored":120010012,"data_compressed":7809809,"data_compressed_allocated":8314880,"data_compressed_original":17313792,"omap_allocated":0,"internal_metadata":0},"log_size":597,"ondisk_log_size":597,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574863,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35416,"kb_used_data":7428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336424,"statfs":{"total":96636764160,"available":96600498176,"internally_reserved":0,"allocated":7606272,"data_stored":7703445,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.33300000000000002},{"interface":"front","average":{"1min":4.508,"5min":4.508,"15min":4.508},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":85.542000000000002,"5min":85.542000000000002,"15min":85.542000000000002},"last":0.38200000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.8650000000000002,"5min":4.8650000000000002,"15min":4.8650000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":79.927999999999997,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.36899999999999999},{"interface":"front","average":{"1min":13.048,"5min":13.048,"15min":13.048},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":99.388999999999996,"5min":99.388999999999996,"15min":99.388999999999996},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.085,"5min":4.085,"15min":4.085},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":77.320999999999998,"5min":77.320999999999998,"15min":77.320999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.40300000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.1509999999999998,"5min":4.1509999999999998,"15min":4.1509999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":77.361000000000004,"5min":77.361000000000004,"15min":77.361000000000004},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.436}]},{"osd":4,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.4589999999999996,"5min":4.4589999999999996,"15min":4.4589999999999996},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":85.509,"5min":85.509,"15min":85.509},"last":0.30599999999999999},{"interface":"front","average":{"1min":22.872,"5min":22.872,"15min":22.872},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":327.77300000000002,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.39300000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":3.0710000000000002,"5min":3.0710000000000002,"15min":3.0710000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":55.116999999999997,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.27300000000000002},{"interface":"front","average":{"1min":7.1619999999999999,"5min":7.1619999999999999,"15min":7.1619999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":85.475999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.44700000000000001}]},{"osd":6,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":13.276,"5min":13.276,"15min":13.276},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":99.620999999999995,"5min":99.620999999999995,"15min":99.620999999999995},"last":0.34599999999999997},{"interface":"front","average":{"1min":4.133,"5min":4.133,"15min":4.133},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":77.405000000000001,"5min":77.405000000000001,"15min":77.405000000000001},"last":0.42599999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574863,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49244,"kb_used_data":19680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94322596,"statfs":{"total":96636764160,"available":96586338304,"internally_reserved":0,"allocated":20152320,"data_stored":20865559,"data_compressed":1034509,"data_compressed_allocated":1105920,"data_compressed_original":2232320,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.794,"15min":1.794},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":32.097999999999999,"5min":32.097999999999999,"15min":32.097999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":25.808,"5min":25.808,"15min":25.808},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":166.85499999999999,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.33100000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":37.161999999999999,"5min":37.161999999999999,"15min":37.161999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":10.092000000000001,"5min":10.092000000000001,"15min":10.092000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":222.55199999999999,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.29899999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.34300000000000003},{"interface":"front","average":{"1min":18.803999999999998,"5min":18.803999999999998,"15min":18.803999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":166.81800000000001,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.45800000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":21.518999999999998,"5min":21.518999999999998,"15min":21.518999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":171.99100000000001,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.52300000000000002}]},{"osd":4,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":7.6029999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":81.697000000000003,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.27100000000000002},{"interface":"front","average":{"1min":19.327999999999999,"5min":19.327999999999999,"15min":19.327999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":166.733,"5min":166.733,"15min":166.733},"last":0.496}]},{"osd":5,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.316},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.48299999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":18.850999999999999,"5min":18.850999999999999,"15min":18.850999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":166.79400000000001,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.42999999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74452,"kb_used_data":42112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94297388,"statfs":{"total":96636764160,"available":96560525312,"internally_reserved":0,"allocated":43122688,"data_stored":43804457,"data_compressed":1024829,"data_compressed_allocated":1097728,"data_compressed_original":2281472,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":13.391999999999999,"5min":13.391999999999999,"15min":13.391999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":74.019999999999996,"5min":74.019999999999996,"15min":74.019999999999996},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.3919999999999995,"5min":8.3919999999999995,"15min":8.3919999999999995},"last":0.439}]},{"osd":1,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":10.538,"5min":10.538,"15min":10.538},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":94.263999999999996,"5min":94.263999999999996,"15min":94.263999999999996},"last":78.322999999999993},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.42799999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.41799999999999998},{"interface":"front","average":{"1min":5.0389999999999997,"5min":5.0389999999999997,"15min":5.0389999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":78.296999999999997,"5min":78.296999999999997,"15min":78.296999999999997},"last":78.296999999999997}]},{"osd":3,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":5.7489999999999997,"5min":5.7489999999999997,"15min":5.7489999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":89.688000000000002,"5min":89.688000000000002,"15min":89.688000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":10.365,"5min":10.365,"15min":10.365},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":89.718999999999994,"5min":89.718999999999994,"15min":89.718999999999994},"last":78.197000000000003}]},{"osd":4,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":17.721,"5min":17.721,"15min":17.721},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":202.983,"5min":202.983,"15min":202.983},"last":0.30499999999999999},{"interface":"front","average":{"1min":17.783000000000001,"5min":17.783000000000001,"15min":17.783000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":202.999,"5min":202.999,"15min":202.999},"last":0.33100000000000002}]},{"osd":6,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":5.0369999999999999,"5min":5.0369999999999999,"15min":5.0369999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":78.366,"5min":78.366,"15min":78.366},"last":78.366},{"interface":"front","average":{"1min":6.3289999999999997,"5min":6.3289999999999997,"15min":6.3289999999999997},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":100.54900000000001,"5min":100.54900000000001,"15min":100.54900000000001},"last":0.26600000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.28499999999999998},{"interface":"front","average":{"1min":5.0800000000000001,"5min":5.0800000000000001,"15min":5.0800000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":78.340999999999994,"5min":78.340999999999994,"15min":78.340999999999994},"last":78.340999999999994}]}]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91172,"kb_used_data":60888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94280668,"statfs":{"total":96636764160,"available":96543404032,"internally_reserved":0,"allocated":62349312,"data_stored":64135669,"data_compressed":1803700,"data_compressed_allocated":1916928,"data_compressed_original":4239360,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,1,0,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":16.648,"5min":16.648,"15min":16.648},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":86.712999999999994,"5min":86.712999999999994,"15min":86.712999999999994},"last":78.963999999999999},{"interface":"front","average":{"1min":8.5679999999999996,"5min":8.5679999999999996,"15min":8.5679999999999996},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":78.158000000000001,"5min":78.158000000000001,"15min":78.158000000000001},"last":0.35999999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.318},{"interface":"front","average":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":78.866,"5min":78.866,"15min":78.866},"last":78.866}]},{"osd":2,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":8.5700000000000003,"5min":8.5700000000000003,"15min":8.5700000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":76.724999999999994,"5min":76.724999999999994,"15min":76.724999999999994},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":4.0220000000000002,"5min":4.0220000000000002,"15min":4.0220000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":78.912000000000006,"5min":78.912000000000006,"15min":78.912000000000006},"last":78.912000000000006},{"interface":"front","average":{"1min":20.968,"5min":20.968,"15min":20.968},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":227.41200000000001,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.378}]},{"osd":5,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":4.0380000000000003,"5min":4.0380000000000003,"15min":4.0380000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":78.811999999999998,"5min":78.811999999999998,"15min":78.811999999999998},"last":78.811999999999998}]},{"osd":6,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":13.16,"5min":13.16,"15min":13.16},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":124.303,"5min":124.303,"15min":124.303},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.28599999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":8.5600000000000005,"5min":8.5600000000000005,"15min":8.5600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":76.697000000000003,"5min":76.697000000000003,"15min":76.697000000000003},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.34399999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57948,"kb_used_data":26828,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94313892,"statfs":{"total":96636764160,"available":96577425408,"internally_reserved":0,"allocated":27471872,"data_stored":28244668,"data_compressed":993281,"data_compressed_allocated":1064960,"data_compressed_original":2281472,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":12.199999999999999,"5min":12.199999999999999,"15min":12.199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":98.957999999999998,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.26100000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":5.0449999999999999,"5min":5.0449999999999999,"15min":5.0449999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":94.063000000000002,"5min":94.063000000000002,"15min":94.063000000000002},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.40899999999999997}]},{"osd":2,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":12.901999999999999,"5min":12.901999999999999,"15min":12.901999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.34200000000000003}]},{"osd":4,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":4.1079999999999997,"5min":4.1079999999999997,"15min":4.1079999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":71.998000000000005,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.43099999999999999},{"interface":"front","average":{"1min":4.1379999999999999,"5min":4.1379999999999999,"15min":4.1379999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":72.227000000000004,"5min":72.227000000000004,"15min":72.227000000000004},"last":0.502}]},{"osd":5,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":13.94,"5min":13.94,"15min":13.94},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":13.965,"5min":13.965,"15min":13.965},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.36699999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.39500000000000002}]}]},{"osd":2,"up_from":0,"seq":55834574863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86556,"kb_used_data":50524,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94285284,"statfs":{"total":96636764160,"available":96548130816,"internally_reserved":0,"allocated":51736576,"data_stored":52256346,"data_compressed":1022279,"data_compressed_allocated":1089536,"data_compressed_original":2186869,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":7.859,"5min":7.859,"15min":7.859},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":88.171000000000006,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.19500000000000001},{"interface":"front","average":{"1min":7.8630000000000004,"5min":7.8630000000000004,"15min":7.8630000000000004},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":88.156000000000006,"5min":88.156000000000006,"15min":88.156000000000006},"last":0.27700000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":8.5999999999999996,"5min":8.5999999999999996,"15min":8.5999999999999996},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":183.04400000000001,"5min":183.04400000000001,"15min":183.04400000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.251}]},{"osd":3,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.308},{"interface":"front","average":{"1min":14.492000000000001,"5min":14.492000000000001,"15min":14.492000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":231.864,"5min":231.864,"15min":231.864},"last":0.20699999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":12.984,"5min":12.984,"15min":12.984},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":192.42099999999999,"5min":192.42099999999999,"15min":192.42099999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.41199999999999998}]},{"osd":5,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":14.071,"5min":14.071,"15min":14.071},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":138.089,"5min":138.089,"15min":138.089},"last":0.49199999999999999},{"interface":"front","average":{"1min":20.077000000000002,"5min":20.077000000000002,"15min":20.077000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":183.08199999999999,"5min":183.08199999999999,"15min":183.08199999999999},"last":0.48299999999999998}]},{"osd":6,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":51.070999999999998,"5min":51.070999999999998,"15min":51.070999999999998},"last":0.439},{"interface":"front","average":{"1min":16.532,"5min":16.532,"15min":16.532},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":183.10900000000001,"5min":183.10900000000001,"15min":183.10900000000001},"last":0.44800000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":3.0390000000000001,"5min":3.0390000000000001,"15min":3.0390000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":51.097000000000001,"5min":51.097000000000001,"15min":51.097000000000001},"last":0.51900000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574863,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78984,"kb_used_data":47000,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94292856,"statfs":{"total":96636764160,"available":96555884544,"internally_reserved":0,"allocated":48128000,"data_stored":50208163,"data_compressed":1769708,"data_compressed_allocated":1884160,"data_compressed_original":4489216,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,6,0,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":6.7599999999999998,"5min":6.7599999999999998,"15min":6.7599999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":133.09899999999999,"5min":133.09899999999999,"15min":133.09899999999999},"last":67.096999999999994},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":25.827999999999999,"5min":25.827999999999999,"15min":25.827999999999999},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":25.873000000000001,"5min":25.873000000000001,"15min":25.873000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.318,"15min":0.318},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.35099999999999998}]},{"osd":3,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":67.147999999999996},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":25.898,"5min":25.898,"15min":25.898},"last":0.29599999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":9.9120000000000008,"5min":9.9120000000000008,"15min":9.9120000000000008},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":133.036,"5min":133.036,"15min":133.036},"last":67.328999999999994}]},{"osd":5,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":26.16,"5min":26.16,"15min":26.16},"last":41.645000000000003},{"interface":"front","average":{"1min":13.955,"5min":13.955,"15min":13.955},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":282.197,"5min":282.197,"15min":282.197},"last":0.47699999999999998}]},{"osd":6,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":3.198,"5min":3.198,"15min":3.198},"last":67.281999999999996},{"interface":"front","average":{"1min":1.748,"5min":1.748,"15min":1.748},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":26.190000000000001,"5min":26.190000000000001,"15min":26.190000000000001},"last":0.51500000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":4.6779999999999999,"5min":4.6779999999999999,"15min":4.6779999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":87.209999999999994,"5min":87.209999999999994,"15min":87.209999999999994},"last":0.495},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.2440000000000002,"5min":3.2440000000000002,"15min":3.2440000000000002},"last":67.373999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28848128,"data_stored":30855014,"data_compressed":1670174,"data_compressed_allocated":1769472,"data_compressed_original":3846144,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9097216,"data_stored":9877789,"data_compressed":850587,"data_compressed_allocated":905216,"data_compressed_original":1708032,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9674752,"data_stored":10577332,"data_compressed":795625,"data_compressed_allocated":851968,"data_compressed_original":1773568,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12718080,"data_stored":13837246,"data_compressed":986331,"data_compressed_allocated":1056768,"data_compressed_original":2215936,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28848128,"data_stored":30855014,"data_compressed":1670174,"data_compressed_allocated":1769472,"data_compressed_original":3846144,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":13012992,"data_stored":14129828,"data_compressed":986331,"data_compressed_allocated":1056768,"data_compressed_original":2215936,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":9097216,"data_stored":9877789,"data_compressed":850587,"data_compressed_allocated":905216,"data_compressed_original":1708032,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18632704,"data_stored":18580399,"data_compressed":69966,"data_compressed_allocated":73728,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41705472,"data_stored":41592745,"data_compressed":88971,"data_compressed_allocated":94208,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5943296,"data_stored":5913543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13897728,"data_stored":13872601,"data_compressed":6950,"data_compressed_allocated":8192,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32960512,"data_stored":33041351,"data_compressed":133526,"data_compressed_allocated":147456,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29405184,"data_stored":29239290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":9625600,"data_stored":9722985,"data_compressed":107571,"data_compressed_allocated":118784,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6979584,"data_stored":6931921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:39:44.068 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-04-22T18:39:44.068 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-04-22T18:39:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:44.076 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:44.076 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:44.076+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:44.173 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:44.173+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:44.189 INFO:tasks.rados.rados.0.smithi089.stdout:1063: expect (ObjNum 164 snap 0 seq_num 164) 2024-04-22T18:39:44.242 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:44.242 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:44.251 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:44.274 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:44.275+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:44.322 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:39:44.325 INFO:tasks.rados.rados.0.smithi089.stdout:1073: finishing set_chunk to oid 5 2024-04-22T18:39:44.326 INFO:tasks.rados.rados.0.smithi089.stdout:1073: got expected ENOENT (src dne) 2024-04-22T18:39:44.326 INFO:tasks.rados.rados.0.smithi089.stdout:1068: expect (ObjNum 179 snap 0 seq_num 179) 2024-04-22T18:39:44.375 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:44.376+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:44.410 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:44.410 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:44.420 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:44.448 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:44.449 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:44.467 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:44.467 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:44.475 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:44.475+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:44.483 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:44.483 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:44.499 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:44.499 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:44.510 INFO:tasks.rados.rados.0.smithi089.stdout:1072: expect (ObjNum 173 snap 0 seq_num 173) 2024-04-22T18:39:44.516 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:44.516 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:44.526 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:44.526 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:44.526 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:44.526 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:44.526 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:44.534 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:44.534 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:44.535 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:44.551 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:44.551 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:44.568 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:44.568 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:44.576 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:44.578+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:44.656 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:44.656 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:44.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:44.656 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:44.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.656 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:45 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'1, mlcod=0'0)", 2024-04-22T18:39:44.656 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:44.656 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.390696466999998, 2024-04-22T18:39:44.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14345418900000001, 2024-04-22T18:39:44.657 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.657 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.657 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:44.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:44.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:44.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:44.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265727+0000", 2024-04-22T18:39:44.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061502799999999999 2024-04-22T18:39:44.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265737+0000", 2024-04-22T18:39:44.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0278999999999999e-05 2024-04-22T18:39:44.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265744+0000", 2024-04-22T18:39:44.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1740000000000003e-06 2024-04-22T18:39:44.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407227+0000", 2024-04-22T18:39:44.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14148266900000001 2024-04-22T18:39:44.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407258+0000", 2024-04-22T18:39:44.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1619000000000003e-05 2024-04-22T18:39:44.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408534+0000", 2024-04-22T18:39:44.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001275829 2024-04-22T18:39:44.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408566+0000", 2024-04-22T18:39:44.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1813999999999997e-05 2024-04-22T18:39:44.660 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.660 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.660 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.661 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:46 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'2, mlcod=0'0)", 2024-04-22T18:39:44.661 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:44.661 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.388072063999999, 2024-04-22T18:39:44.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14304645699999999, 2024-04-22T18:39:44.661 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.661 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.661 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:44.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:44.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.267736+0000", 2024-04-22T18:39:44.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:44.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268301+0000", 2024-04-22T18:39:44.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056585700000000001 2024-04-22T18:39:44.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268309+0000", 2024-04-22T18:39:44.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2830000000000004e-06 2024-04-22T18:39:44.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.268312+0000", 2024-04-22T18:39:44.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7179999999999998e-06 2024-04-22T18:39:44.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407999+0000", 2024-04-22T18:39:44.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13968636300000001 2024-04-22T18:39:44.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408011+0000", 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2116999999999999e-05 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410762+0000", 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002751524 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410783+0000", 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0270000000000001e-05 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.664 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:47 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'3, mlcod=0'0)", 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.386339493999998, 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14132698599999999, 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269469+0000", 2024-04-22T18:39:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269468+0000", 2024-04-22T18:39:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269473+0000", 2024-04-22T18:39:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7790000000000003e-06 2024-04-22T18:39:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269478+0000", 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0080000000000002e-06 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.269489+0000", 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.075e-05 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408595+0000", 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13910662300000001 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408611+0000", 2024-04-22T18:39:44.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6124e-05 2024-04-22T18:39:44.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410786+0000", 2024-04-22T18:39:44.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021743819999999999 2024-04-22T18:39:44.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.410796+0000", 2024-04-22T18:39:44.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.007e-05 2024-04-22T18:39:44.668 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.668 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.669 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.669 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:78 4.0 4:01baf5a9:::smithi08937929-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:44.669 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:44.669 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.143873372999998, 2024-04-22T18:39:44.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.398522192, 2024-04-22T18:39:44.669 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.669 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.669 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:44.669 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:44.669 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:44.669 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 78 2024-04-22T18:39:44.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.670 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:44.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:44.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511934+0000", 2024-04-22T18:39:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512012+0000", 2024-04-22T18:39:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8329000000000002e-05 2024-04-22T18:39:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512023+0000", 2024-04-22T18:39:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0755e-05 2024-04-22T18:39:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512028+0000", 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6470000000000003e-06 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512068+0000", 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9721999999999997e-05 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512125+0000", 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6564999999999998e-05 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512200+0000", 2024-04-22T18:39:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5708000000000001e-05 2024-04-22T18:39:44.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:44.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513131+0000", 2024-04-22T18:39:44.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093063 2024-04-22T18:39:44.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:44.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910394+0000", 2024-04-22T18:39:44.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.397263388 2024-04-22T18:39:44.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910430+0000", 2024-04-22T18:39:44.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5485999999999999e-05 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910457+0000", 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7086000000000001e-05 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:79 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.143772716999997, 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39855419399999997, 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.674 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.675 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:44.675 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:44.675 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:44.675 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 79 2024-04-22T18:39:44.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.675 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:44.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:44.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512035+0000", 2024-04-22T18:39:44.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:44.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512045+0000", 2024-04-22T18:39:44.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0379e-05 2024-04-22T18:39:44.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512051+0000", 2024-04-22T18:39:44.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6030000000000004e-06 2024-04-22T18:39:44.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512057+0000", 2024-04-22T18:39:44.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7270000000000004e-06 2024-04-22T18:39:44.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512756+0000", 2024-04-22T18:39:44.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069954000000000001 2024-04-22T18:39:44.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512806+0000", 2024-04-22T18:39:44.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9922000000000001e-05 2024-04-22T18:39:44.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:44.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512927+0000", 2024-04-22T18:39:44.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012059 2024-04-22T18:39:44.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:44.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.515883+0000", 2024-04-22T18:39:44.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029563739999999999 2024-04-22T18:39:44.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:44.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910510+0000", 2024-04-22T18:39:44.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39462745199999999 2024-04-22T18:39:44.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910542+0000", 2024-04-22T18:39:44.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1225e-05 2024-04-22T18:39:44.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910590+0000", 2024-04-22T18:39:44.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8052e-05 2024-04-22T18:39:44.680 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.680 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.680 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.680 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:80 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:44.680 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:44.681 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.143747615999999, 2024-04-22T18:39:44.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39866376999999997, 2024-04-22T18:39:44.681 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.681 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:44.681 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:44.681 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:44.681 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:44.681 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 80 2024-04-22T18:39:44.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.681 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:44.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512060+0000", 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512063+0000", 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.999e-06 2024-04-22T18:39:44.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512069+0000", 2024-04-22T18:39:44.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.891e-06 2024-04-22T18:39:44.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512073+0000", 2024-04-22T18:39:44.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4429999999999996e-06 2024-04-22T18:39:44.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513084+0000", 2024-04-22T18:39:44.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010102609999999999 2024-04-22T18:39:44.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513123+0000", 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9237999999999999e-05 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910618+0000", 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39749549299999998 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910670+0000", 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1221000000000003e-05 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910724+0000", 2024-04-22T18:39:44.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4777000000000001e-05 2024-04-22T18:39:44.685 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.685 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.685 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.685 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:5 3.2 e30/29 3:663b82a1:::smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=0'0)", 2024-04-22T18:39:44.685 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:44.685 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.428308297999997, 2024-04-22T18:39:44.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20090134500000001, 2024-04-22T18:39:44.685 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.685 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.685 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227499+0000", 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227506+0000", 2024-04-22T18:39:44.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6540000000000002e-06 2024-04-22T18:39:44.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227516+0000", 2024-04-22T18:39:44.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0054000000000001e-05 2024-04-22T18:39:44.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327607+0000", 2024-04-22T18:39:44.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009167099999999 2024-04-22T18:39:44.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327688+0000", 2024-04-22T18:39:44.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1000000000000004e-05 2024-04-22T18:39:44.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327719+0000", 2024-04-22T18:39:44.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0664000000000003e-05 2024-04-22T18:39:44.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428371+0000", 2024-04-22T18:39:44.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10065239099999999 2024-04-22T18:39:44.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428401+0000", 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9839e-05 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:419 4.0 4:01baf5a9:::smithi08937929-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.253586595999998, 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30324478199999999, 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:44.689 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:44.690 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:44.690 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 419 2024-04-22T18:39:44.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.690 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:44.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:44.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402219+0000", 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402229+0000", 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2550000000000003e-06 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402242+0000", 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2989999999999999e-05 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402250+0000", 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7900000000000005e-06 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404085+0000", 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001834397 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404092+0000", 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7889999999999997e-06 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502534+0000", 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098442331999999994 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502631+0000", 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7617999999999999e-05 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604414+0000", 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101782847 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604888+0000", 2024-04-22T18:39:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047325999999999998 2024-04-22T18:39:44.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:44.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705376+0000", 2024-04-22T18:39:44.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100488623 2024-04-22T18:39:44.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705424+0000", 2024-04-22T18:39:44.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7964999999999999e-05 2024-04-22T18:39:44.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705466+0000", 2024-04-22T18:39:44.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2274999999999998e-05 2024-04-22T18:39:44.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.695 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.695 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:70 3.2 e32/29 3:5f0623ce:::smithi08937929-38 low_tier:head v 32'37, mlcod=31'35)", 2024-04-22T18:39:44.695 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:44.695 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.127369565000002, 2024-04-22T18:39:44.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20070027900000001, 2024-04-22T18:39:44.695 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.695 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.695 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:44.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:44.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528438+0000", 2024-04-22T18:39:44.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:44.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528443+0000", 2024-04-22T18:39:44.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8640000000000004e-06 2024-04-22T18:39:44.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528453+0000", 2024-04-22T18:39:44.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3910000000000004e-06 2024-04-22T18:39:44.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528459+0000", 2024-04-22T18:39:44.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-04-22T18:39:44.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728627+0000", 2024-04-22T18:39:44.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20016778700000001 2024-04-22T18:39:44.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728660+0000", 2024-04-22T18:39:44.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3024000000000001e-05 2024-04-22T18:39:44.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729107+0000", 2024-04-22T18:39:44.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044719200000000001 2024-04-22T18:39:44.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729139+0000", 2024-04-22T18:39:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1625000000000003e-05 2024-04-22T18:39:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1264 4.c e43/20 4:36664a98:::smithi08937929-38:head v 43'37, mlcod=42'34)", 2024-04-22T18:39:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.999471160999999, 2024-04-22T18:39:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20290602399999999, 2024-04-22T18:39:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656335+0000", 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656850+0000", 2024-04-22T18:39:44.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051499499999999997 2024-04-22T18:39:44.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656861+0000", 2024-04-22T18:39:44.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1119e-05 2024-04-22T18:39:44.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756934+0000", 2024-04-22T18:39:44.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073548 2024-04-22T18:39:44.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756977+0000", 2024-04-22T18:39:44.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2218000000000001e-05 2024-04-22T18:39:44.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756993+0000", 2024-04-22T18:39:44.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6629999999999998e-05 2024-04-22T18:39:44.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859203+0000", 2024-04-22T18:39:44.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102209617 2024-04-22T18:39:44.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859243+0000", 2024-04-22T18:39:44.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0333000000000002e-05 2024-04-22T18:39:44.703 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.703 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.703 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.703 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1745 4.c e48/20 4:35eda2ca:::smithi08937929-12:head v 48'63, mlcod=47'61)", 2024-04-22T18:39:44.703 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:32.493414+0000", 2024-04-22T18:39:44.703 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.162394627999999, 2024-04-22T18:39:44.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107006161, 2024-04-22T18:39:44.703 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.704 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.704 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.493414+0000", 2024-04-22T18:39:44.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.493414+0000", 2024-04-22T18:39:44.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.493410+0000", 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.494215+0000", 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00080466700000000001 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.494239+0000", 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3754999999999999e-05 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.494251+0000", 2024-04-22T18:39:44.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2082999999999999e-05 2024-04-22T18:39:44.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597906+0000", 2024-04-22T18:39:44.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.1036551 2024-04-22T18:39:44.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597926+0000", 2024-04-22T18:39:44.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0254999999999998e-05 2024-04-22T18:39:44.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.600390+0000", 2024-04-22T18:39:44.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0024633430000000002 2024-04-22T18:39:44.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.600420+0000", 2024-04-22T18:39:44.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0077e-05 2024-04-22T18:39:44.707 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.707 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.707 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.707 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1747 4.c e48/20 4:35eda2ca:::smithi08937929-12:head v 48'65, mlcod=47'61)", 2024-04-22T18:39:44.707 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:32.597827+0000", 2024-04-22T18:39:44.707 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.057981019, 2024-04-22T18:39:44.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20085196499999999, 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597827+0000", 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597827+0000", 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597825+0000", 2024-04-22T18:39:44.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:44.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597839+0000", 2024-04-22T18:39:44.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4329e-05 2024-04-22T18:39:44.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597856+0000", 2024-04-22T18:39:44.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6858e-05 2024-04-22T18:39:44.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.697948+0000", 2024-04-22T18:39:44.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092367 2024-04-22T18:39:44.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798102+0000", 2024-04-22T18:39:44.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100153411 2024-04-22T18:39:44.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798122+0000", 2024-04-22T18:39:44.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0069e-05 2024-04-22T18:39:44.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798628+0000", 2024-04-22T18:39:44.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050619700000000001 2024-04-22T18:39:44.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798679+0000", 2024-04-22T18:39:44.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1307000000000002e-05 2024-04-22T18:39:44.711 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.711 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.711 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.711 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1855 4.f e51/20 4:f0ed3d49:::smithi08937929-9:head v 51'31, mlcod=47'29)", 2024-04-22T18:39:44.711 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:35.291419+0000", 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.364389096, 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10796507700000001, 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.291419+0000", 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.291419+0000", 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.291419+0000", 2024-04-22T18:39:44.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:44.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.291707+0000", 2024-04-22T18:39:44.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000288592 2024-04-22T18:39:44.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.291716+0000", 2024-04-22T18:39:44.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0049999999999993e-06 2024-04-22T18:39:44.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.291725+0000", 2024-04-22T18:39:44.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1869999999999997e-06 2024-04-22T18:39:44.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.297113+0000", 2024-04-22T18:39:44.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053880330000000004 2024-04-22T18:39:44.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.297140+0000", 2024-04-22T18:39:44.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7126000000000001e-05 2024-04-22T18:39:44.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.399352+0000", 2024-04-22T18:39:44.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102212637 2024-04-22T18:39:44.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.399384+0000", 2024-04-22T18:39:44.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1859000000000001e-05 2024-04-22T18:39:44.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.715 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.716 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1883 4.f e51/20 4:f83a159b:::smithi08937929-2:head v 51'35, mlcod=51'32)", 2024-04-22T18:39:44.716 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:35.579245+0000", 2024-04-22T18:39:44.716 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.0765632099999998, 2024-04-22T18:39:44.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10463091400000001, 2024-04-22T18:39:44.716 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.716 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.716 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.579245+0000", 2024-04-22T18:39:44.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.579245+0000", 2024-04-22T18:39:44.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.579244+0000", 2024-04-22T18:39:44.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:44.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.579593+0000", 2024-04-22T18:39:44.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000348833 2024-04-22T18:39:44.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.579604+0000", 2024-04-22T18:39:44.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1626e-05 2024-04-22T18:39:44.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.579613+0000", 2024-04-22T18:39:44.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4230000000000005e-06 2024-04-22T18:39:44.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.679989+0000", 2024-04-22T18:39:44.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100376099 2024-04-22T18:39:44.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.680005+0000", 2024-04-22T18:39:44.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6237999999999999e-05 2024-04-22T18:39:44.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.683849+0000", 2024-04-22T18:39:44.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038441270000000001 2024-04-22T18:39:44.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.683876+0000", 2024-04-22T18:39:44.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6874999999999999e-05 2024-04-22T18:39:44.719 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.719 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.720 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.720 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:229 3.2 e51/29 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head v 51'117, mlcod=51'115)", 2024-04-22T18:39:44.720 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:39:44.720 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.8447655579999997, 2024-04-22T18:39:44.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31548799399999999, 2024-04-22T18:39:44.720 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.720 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.720 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:39:44.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:39:44.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811041+0000", 2024-04-22T18:39:44.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:44.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811054+0000", 2024-04-22T18:39:44.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3188e-05 2024-04-22T18:39:44.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811068+0000", 2024-04-22T18:39:44.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4239999999999999e-05 2024-04-22T18:39:44.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.911141+0000", 2024-04-22T18:39:44.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100072851 2024-04-22T18:39:44.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025053+0000", 2024-04-22T18:39:44.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.11391237 2024-04-22T18:39:44.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025083+0000", 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9703999999999999e-05 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126504+0000", 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10142053500000001 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126531+0000", 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7172999999999999e-05 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.723 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:235 3.2 e51/29 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head v 51'118, mlcod=51'115)", 2024-04-22T18:39:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.924871+0000", 2024-04-22T18:39:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.7309373570000002, 2024-04-22T18:39:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20167500599999999, 2024-04-22T18:39:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924871+0000", 2024-04-22T18:39:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924871+0000", 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924869+0000", 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924883+0000", 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3725e-05 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924899+0000", 2024-04-22T18:39:44.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6344000000000001e-05 2024-04-22T18:39:44.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924907+0000", 2024-04-22T18:39:44.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0909999999999993e-06 2024-04-22T18:39:44.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025426+0000", 2024-04-22T18:39:44.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100519109 2024-04-22T18:39:44.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025451+0000", 2024-04-22T18:39:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4338e-05 2024-04-22T18:39:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126534+0000", 2024-04-22T18:39:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101083083 2024-04-22T18:39:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126546+0000", 2024-04-22T18:39:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2122e-05 2024-04-22T18:39:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.727 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:238 3.2 e51/29 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head v 51'119, mlcod=51'115)", 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.925398+0000", 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.7304105749999996, 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.201162223, 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925398+0000", 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925398+0000", 2024-04-22T18:39:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925396+0000", 2024-04-22T18:39:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925405+0000", 2024-04-22T18:39:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6600000000000001e-06 2024-04-22T18:39:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925417+0000", 2024-04-22T18:39:44.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2697000000000001e-05 2024-04-22T18:39:44.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925425+0000", 2024-04-22T18:39:44.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3000000000000004e-06 2024-04-22T18:39:44.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.125722+0000", 2024-04-22T18:39:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20029754299999999 2024-04-22T18:39:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.125763+0000", 2024-04-22T18:39:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0408999999999999e-05 2024-04-22T18:39:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126548+0000", 2024-04-22T18:39:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00078496600000000001 2024-04-22T18:39:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126560+0000", 2024-04-22T18:39:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2227e-05 2024-04-22T18:39:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:118 3.2 e51/29 3:43d85947:::smithi08937929-18 low_tier:head v 51'120, mlcod=51'115)", 2024-04-22T18:39:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.983526+0000", 2024-04-22T18:39:44.732 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.6722825879999998, 2024-04-22T18:39:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14328775899999999, 2024-04-22T18:39:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983526+0000", 2024-04-22T18:39:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983526+0000", 2024-04-22T18:39:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983524+0000", 2024-04-22T18:39:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983538+0000", 2024-04-22T18:39:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3858e-05 2024-04-22T18:39:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983552+0000", 2024-04-22T18:39:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4466e-05 2024-04-22T18:39:44.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983562+0000", 2024-04-22T18:39:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3899999999999999e-06 2024-04-22T18:39:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126168+0000", 2024-04-22T18:39:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14260677899999999 2024-04-22T18:39:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126185+0000", 2024-04-22T18:39:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6625000000000001e-05 2024-04-22T18:39:44.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126794+0000", 2024-04-22T18:39:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060855999999999996 2024-04-22T18:39:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126813+0000", 2024-04-22T18:39:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9851e-05 2024-04-22T18:39:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:119 3.2 e51/29 3:43d85947:::smithi08937929-18 low_tier:head v 51'121, mlcod=51'115)", 2024-04-22T18:39:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.6719598879999999, 2024-04-22T18:39:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.145803025, 2024-04-22T18:39:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983854+0000", 2024-04-22T18:39:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.207e-06 2024-04-22T18:39:44.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983860+0000", 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5350000000000001e-06 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983865+0000", 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0170000000000004e-06 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126364+0000", 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.142498491 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126378+0000", 2024-04-22T18:39:44.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4134999999999999e-05 2024-04-22T18:39:44.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129628+0000", 2024-04-22T18:39:44.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032498940000000001 2024-04-22T18:39:44.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129651+0000", 2024-04-22T18:39:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3493999999999999e-05 2024-04-22T18:39:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:120 3.2 e51/29 3:43d85947:::smithi08937929-18 low_tier:head v 51'122, mlcod=51'115)", 2024-04-22T18:39:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.984299+0000", 2024-04-22T18:39:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.6715094930000003, 2024-04-22T18:39:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14536710899999999, 2024-04-22T18:39:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:44.741 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984299+0000", 2024-04-22T18:39:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984299+0000", 2024-04-22T18:39:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:44.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984298+0000", 2024-04-22T18:39:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984302+0000", 2024-04-22T18:39:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.929e-06 2024-04-22T18:39:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984309+0000", 2024-04-22T18:39:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2080000000000003e-06 2024-04-22T18:39:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984316+0000", 2024-04-22T18:39:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7329999999999997e-06 2024-04-22T18:39:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126582+0000", 2024-04-22T18:39:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14226603400000001 2024-04-22T18:39:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126595+0000", 2024-04-22T18:39:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3458000000000001e-05 2024-04-22T18:39:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129653+0000", 2024-04-22T18:39:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030578860000000001 2024-04-22T18:39:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129666+0000", 2024-04-22T18:39:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2483999999999999e-05 2024-04-22T18:39:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:44.745 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:44.745 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:44.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:44.747 INFO:tasks.rados.rados.0.smithi089.stdout:1075: expect (ObjNum 160 snap 0 seq_num 160) 2024-04-22T18:39:44.748 INFO:teuthology.orchestra.run.smithi089.stderr:set pool 4 pg_num to 26 2024-04-22T18:39:44.748 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:44.747+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:44.804 INFO:tasks.rados.rados.0.smithi089.stdout:1069: expect (ObjNum 135 snap 0 seq_num 135) 2024-04-22T18:39:44.847 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:44.847+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:44.860 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:44.860 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:44.860 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:44.860 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:44.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:44.948 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:44.949+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:44.948 INFO:tasks.rados.rados.0.smithi089.stdout:1076: got (22) Invalid argument 2024-04-22T18:39:44.948 INFO:tasks.rados.rados.0.smithi089.stdout:1071: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:39:44.972 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:44.972 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:44.972 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:44.972 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:44.972 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:44.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:45.014 INFO:tasks.rados.rados.0.smithi089.stdout:1078: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-22T18:39:45.048 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:45.049+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:45.091 INFO:tasks.rados.rados.0.smithi089.stdout:1077: expect (ObjNum 140 snap 0 seq_num 140) 2024-04-22T18:39:45.124 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:45.124 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:45.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.962344756, 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:45.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:45.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:39:45.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:45.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:39:45.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:39:45.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:39:45.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:39:45.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:39:45.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:39:45.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:39:45.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:39:45.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:39:45.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:39:45.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:39:45.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:39:45.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:39:45.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:39:45.130 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.130 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.130 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.130 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:45.130 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:45.131 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.860785935000003, 2024-04-22T18:39:45.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:39:45.131 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.131 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.131 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.131 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.131 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.131 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:39:45.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.132 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:45.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:45.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:39:45.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:45.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:39:45.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:39:45.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:39:45.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:39:45.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:39:45.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:39:45.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:39:45.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:39:45.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:39:45.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:39:45.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:39:45.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:39:45.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:39:45.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:39:45.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:39:45.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:39:45.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:39:45.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:39:45.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:39:45.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:39:45.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:39:45.137 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.137 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.137 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.137 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:45.137 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:45.137 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.860759533, 2024-04-22T18:39:45.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:39:45.137 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.137 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:45.137 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.137 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.137 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.138 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:39:45.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.138 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:45.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:45.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:39:45.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:45.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:39:45.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:39:45.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:39:45.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:39:45.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:39:45.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:39:45.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:39:45.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:39:45.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:45.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:39:45.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:39:45.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:39:45.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:39:45.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:39:45.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:39:45.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.462425203999999, 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.142 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:45.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:45.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:39:45.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:39:45.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:39:45.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:39:45.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:45.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:39:45.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:39:45.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:39:45.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:39:45.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:39:45.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:39:45.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.361588906000001, 2024-04-22T18:39:45.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:39:45.148 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.148 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.148 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.148 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.148 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.148 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:39:45.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.148 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:45.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:45.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:39:45.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:45.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:39:45.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:39:45.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:39:45.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:39:45.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:39:45.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:39:45.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:39:45.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:39:45.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:39:45.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:39:45.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:45.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:39:45.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:39:45.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:39:45.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:39:45.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:39:45.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:39:45.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:39:45.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:39:45.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:39:45.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:39:45.153 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.153 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.153 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.153 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:45.153 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:45.153 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.361431635999999, 2024-04-22T18:39:45.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:39:45.154 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.154 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.154 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.154 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.154 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.154 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:39:45.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.154 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:45.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:45.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:39:45.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:45.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:39:45.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:39:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:39:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:39:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:39:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:39:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:39:45.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:39:45.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:39:45.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:39:45.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:45.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:39:45.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:39:45.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:39:45.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:39:45.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:39:45.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:39:45.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:39:45.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:39:45.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:39:45.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:39:45.159 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.159 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.159 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.159 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:45.159 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:45.159 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.361417027000002, 2024-04-22T18:39:45.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:39:45.159 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.159 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:45.159 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.159 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.159 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.160 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:39:45.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.160 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:45.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:45.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:39:45.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:39:45.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:39:45.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:39:45.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:45.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:39:45.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:39:45.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:39:45.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:39:45.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:39:45.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:39:45.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:39:45.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:39:45.163 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.164 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.164 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.164 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:45.164 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:45.164 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.361406641999999, 2024-04-22T18:39:45.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:39:45.164 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.164 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.164 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.164 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.164 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.164 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:39:45.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.165 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:45.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:45.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:39:45.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:45.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:39:45.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:39:45.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:39:45.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:39:45.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:39:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:39:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:39:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:39:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:39:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:39:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:45.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:39:45.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:39:45.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:39:45.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:39:45.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:39:45.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:39:45.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:45.169 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:45.170 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.360476712000001, 2024-04-22T18:39:45.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:39:45.170 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.170 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.170 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.170 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.170 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.170 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:39:45.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.170 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:45.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:45.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:39:45.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:45.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:39:45.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:39:45.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:39:45.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:39:45.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:39:45.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:39:45.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:39:45.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:39:45.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:39:45.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:39:45.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:45.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:39:45.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:39:45.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:39:45.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:39:45.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:39:45.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:39:45.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:39:45.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:39:45.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:39:45.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:39:45.175 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.175 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.175 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.176 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:45.176 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:45.176 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.360014587000002, 2024-04-22T18:39:45.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:39:45.176 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.176 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.176 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.176 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.176 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.177 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:39:45.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.177 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:45.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:45.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:39:45.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:45.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:39:45.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:39:45.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:39:45.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:39:45.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:45.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:39:45.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:39:45.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:39:45.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:39:45.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:39:45.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:39:45.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:39:45.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:39:45.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:39:45.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:39:45.181 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.181 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.181 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.182 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:45.182 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:45.182 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.360001783000001, 2024-04-22T18:39:45.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:39:45.182 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.182 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:45.182 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.182 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.182 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.182 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:39:45.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.182 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:45.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:45.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:39:45.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:45.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:39:45.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:39:45.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:39:45.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:39:45.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:39:45.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:39:45.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:39:45.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:39:45.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:45.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:39:45.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:39:45.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:39:45.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:39:45.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:39:45.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:39:45.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:39:45.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:39:45.186 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.186 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.186 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.187 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:109 4.1 4:83c955bf:::smithi08937929-28:head [write 453789~780782,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:45.187 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:45.187 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.157214789999998, 2024-04-22T18:39:45.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.45086113300000003, 2024-04-22T18:39:45.187 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.187 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.187 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.187 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.187 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.187 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:39:45.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.187 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:45.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967589+0000", 2024-04-22T18:39:45.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.967588+0000", 2024-04-22T18:39:45.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:45.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.968243+0000", 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00065488500000000002 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.968265+0000", 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1997e-05 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.068340+0000", 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007516 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168464+0000", 2024-04-22T18:39:45.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100124267 2024-04-22T18:39:45.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168579+0000", 2024-04-22T18:39:45.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114387 2024-04-22T18:39:45.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:45.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.168692+0000", 2024-04-22T18:39:45.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011335100000000001 2024-04-22T18:39:45.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.173549+0000", 2024-04-22T18:39:45.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0048564430000000002 2024-04-22T18:39:45.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418367+0000", 2024-04-22T18:39:45.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.244817962 2024-04-22T18:39:45.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418419+0000", 2024-04-22T18:39:45.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2883000000000002e-05 2024-04-22T18:39:45.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.418450+0000", 2024-04-22T18:39:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0278999999999999e-05 2024-04-22T18:39:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.032229327000003, 2024-04-22T18:39:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:39:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.193 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.193 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.193 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:39:45.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.193 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:45.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:45.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:39:45.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:45.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:39:45.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:39:45.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:39:45.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:39:45.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:45.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:39:45.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:39:45.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:39:45.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:39:45.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:39:45.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:39:45.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:39:45.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:39:45.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:39:45.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:39:45.197 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.197 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.197 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.198 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:45.198 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:45.198 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.023656215000003, 2024-04-22T18:39:45.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:39:45.198 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.198 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.198 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.198 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.198 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.198 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:39:45.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.198 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:45.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:45.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:39:45.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:45.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:39:45.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:39:45.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:39:45.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:39:45.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:39:45.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:39:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:39:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:39:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:39:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:39:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:39:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:39:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:39:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:39:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:39:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:39:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:39:45.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:39:45.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:39:45.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:39:45.203 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.203 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.203 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.203 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1108 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 3378174] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:45.203 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.637716+0000", 2024-04-22T18:39:45.203 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.487087385999999, 2024-04-22T18:39:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41175930500000002, 2024-04-22T18:39:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1108 2024-04-22T18:39:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637716+0000", 2024-04-22T18:39:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637716+0000", 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637715+0000", 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637719+0000", 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1710000000000004e-06 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637725+0000", 2024-04-22T18:39:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7790000000000001e-06 2024-04-22T18:39:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637730+0000", 2024-04-22T18:39:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7330000000000003e-06 2024-04-22T18:39:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.948647+0000", 2024-04-22T18:39:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31091748400000002 2024-04-22T18:39:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.948704+0000", 2024-04-22T18:39:45.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7000000000000003e-05 2024-04-22T18:39:45.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:45.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.948802+0000", 2024-04-22T18:39:45.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8107000000000001e-05 2024-04-22T18:39:45.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049249+0000", 2024-04-22T18:39:45.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100446991 2024-04-22T18:39:45.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049270+0000", 2024-04-22T18:39:45.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0336e-05 2024-04-22T18:39:45.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049369+0000", 2024-04-22T18:39:45.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9256e-05 2024-04-22T18:39:45.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049475+0000", 2024-04-22T18:39:45.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010624900000000001 2024-04-22T18:39:45.208 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1109 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.637734+0000", 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.487069777999999, 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41190551399999997, 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1109 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637734+0000", 2024-04-22T18:39:45.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637734+0000", 2024-04-22T18:39:45.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637733+0000", 2024-04-22T18:39:45.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:45.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637736+0000", 2024-04-22T18:39:45.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4530000000000001e-06 2024-04-22T18:39:45.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637741+0000", 2024-04-22T18:39:45.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.118e-06 2024-04-22T18:39:45.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.637745+0000", 2024-04-22T18:39:45.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4490000000000001e-06 2024-04-22T18:39:45.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.949123+0000", 2024-04-22T18:39:45.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31137820900000002 2024-04-22T18:39:45.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:45.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.949205+0000", 2024-04-22T18:39:45.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1444000000000002e-05 2024-04-22T18:39:45.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049516+0000", 2024-04-22T18:39:45.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031090099999999 2024-04-22T18:39:45.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049573+0000", 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7497000000000002e-05 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:13.049639+0000", 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5906000000000002e-05 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1285 4.d 4:bee1184f:::smithi08937929-7:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:45.213 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.287538541, 2024-04-22T18:39:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.42388892500000003, 2024-04-22T18:39:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1285 2024-04-22T18:39:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:45.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837265+0000", 2024-04-22T18:39:45.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837262+0000", 2024-04-22T18:39:45.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:45.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837273+0000", 2024-04-22T18:39:45.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1364e-05 2024-04-22T18:39:45.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837300+0000", 2024-04-22T18:39:45.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6928999999999999e-05 2024-04-22T18:39:45.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837318+0000", 2024-04-22T18:39:45.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8219999999999998e-05 2024-04-22T18:39:45.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837445+0000", 2024-04-22T18:39:45.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000126415 2024-04-22T18:39:45.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.837627+0000", 2024-04-22T18:39:45.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000182182 2024-04-22T18:39:45.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.261154+0000", 2024-04-22T18:39:45.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.42352673499999999 2024-04-22T18:39:45.217 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.217 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:39:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:39:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.989008699999999, 2024-04-22T18:39:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:39:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:39:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.218 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:39:45.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:39:45.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:39:45.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:39:45.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:39:45.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:39:45.221 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.221 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.222 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.222 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1684 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:39:45.222 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:39:45.222 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.519524736999999, 2024-04-22T18:39:45.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987885199999998, 2024-04-22T18:39:45.222 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.222 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:45.222 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.222 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.222 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.222 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1684 2024-04-22T18:39:45.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.223 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:39:45.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:39:45.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605278+0000", 2024-04-22T18:39:45.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:45.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605280+0000", 2024-04-22T18:39:45.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1770000000000001e-06 2024-04-22T18:39:45.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605285+0000", 2024-04-22T18:39:45.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-04-22T18:39:45.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605289+0000", 2024-04-22T18:39:45.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.687e-06 2024-04-22T18:39:45.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705352+0000", 2024-04-22T18:39:45.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006322199999999 2024-04-22T18:39:45.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:45.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705423+0000", 2024-04-22T18:39:45.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1136999999999993e-05 2024-04-22T18:39:45.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105017+0000", 2024-04-22T18:39:45.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.399593268 2024-04-22T18:39:45.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105081+0000", 2024-04-22T18:39:45.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4703999999999996e-05 2024-04-22T18:39:45.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105157+0000", 2024-04-22T18:39:45.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6205999999999995e-05 2024-04-22T18:39:45.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.226 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.227 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1902 4.9 4:9ec4daef:::smithi08937929-27:head [write 1946413~679049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:45.227 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.307159+0000", 2024-04-22T18:39:45.227 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.8176443530000004, 2024-04-22T18:39:45.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41122182200000001, 2024-04-22T18:39:45.227 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.227 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.227 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.227 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.227 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.227 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1902 2024-04-22T18:39:45.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.227 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.307159+0000", 2024-04-22T18:39:45.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.307159+0000", 2024-04-22T18:39:45.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.307156+0000", 2024-04-22T18:39:45.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:45.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.307698+0000", 2024-04-22T18:39:45.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054243300000000004 2024-04-22T18:39:45.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.307711+0000", 2024-04-22T18:39:45.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.252e-05 2024-04-22T18:39:45.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.307717+0000", 2024-04-22T18:39:45.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.049e-06 2024-04-22T18:39:45.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.316502+0000", 2024-04-22T18:39:45.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0087848209999999999 2024-04-22T18:39:45.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.316571+0000", 2024-04-22T18:39:45.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9133000000000004e-05 2024-04-22T18:39:45.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:45.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.316663+0000", 2024-04-22T18:39:45.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2230000000000003e-05 2024-04-22T18:39:45.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.320733+0000", 2024-04-22T18:39:45.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004069909 2024-04-22T18:39:45.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.718304+0000", 2024-04-22T18:39:45.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39757116999999997 2024-04-22T18:39:45.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.718335+0000", 2024-04-22T18:39:45.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1331999999999997e-05 2024-04-22T18:39:45.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.718381+0000", 2024-04-22T18:39:45.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5389000000000002e-05 2024-04-22T18:39:45.233 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.233 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.233 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.233 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.233 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.234 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:45.234 INFO:tasks.rados.rados.0.smithi089.stdout:1070: expect (ObjNum 153 snap 0 seq_num 153) 2024-04-22T18:39:45.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:45.236 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:45.236+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:45.335 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:45.336+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:45.345 INFO:tasks.rados.rados.0.smithi089.stdout:1066: finishing write tid 1 to smithi08937929-18 2024-04-22T18:39:45.345 INFO:tasks.rados.rados.0.smithi089.stdout:1066: finishing write tid 2 to smithi08937929-18 2024-04-22T18:39:45.345 INFO:tasks.rados.rados.0.smithi089.stdout:1066: finishing write tid 3 to smithi08937929-18 2024-04-22T18:39:45.345 INFO:tasks.rados.rados.0.smithi089.stdout:1066: finishing write tid 4 to smithi08937929-18 2024-04-22T18:39:45.345 INFO:tasks.rados.rados.0.smithi089.stdout:1066: finishing write tid 5 to smithi08937929-18 2024-04-22T18:39:45.345 INFO:tasks.rados.rados.0.smithi089.stdout:1066: oid 18 updating version 0 to 60 2024-04-22T18:39:45.346 INFO:tasks.rados.rados.0.smithi089.stdout:1066: oid 18 updating version 60 to 63 2024-04-22T18:39:45.346 INFO:tasks.rados.rados.0.smithi089.stdout:1066: oid 18 version 63 is already newer than 61 2024-04-22T18:39:45.346 INFO:tasks.rados.rados.0.smithi089.stdout:1066: oid 18 version 63 is already newer than 62 2024-04-22T18:39:45.346 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 63 (ObjNum 181 snap 0 seq_num 181) dirty exists 2024-04-22T18:39:45.346 INFO:tasks.rados.rados.0.smithi089.stdout:1066: left oid 18 (ObjNum 181 snap 0 seq_num 181) 2024-04-22T18:39:45.346 INFO:tasks.rados.rados.0.smithi089.stdout:1074: finishing write tid 1 to smithi08937929-2 2024-04-22T18:39:45.346 INFO:tasks.rados.rados.0.smithi089.stdout:1074: finishing write tid 2 to smithi08937929-2 2024-04-22T18:39:45.346 INFO:tasks.rados.rados.0.smithi089.stdout:1063: done (15 left) 2024-04-22T18:39:45.346 INFO:tasks.rados.rados.0.smithi089.stdout:1064: done (14 left) 2024-04-22T18:39:45.346 INFO:tasks.rados.rados.0.smithi089.stdout:1065: done (13 left) 2024-04-22T18:39:45.346 INFO:tasks.rados.rados.0.smithi089.stdout:1066: done (12 left) 2024-04-22T18:39:45.346 INFO:tasks.rados.rados.0.smithi089.stdout:1067: done (11 left) 2024-04-22T18:39:45.346 INFO:tasks.rados.rados.0.smithi089.stdout:1068: done (10 left) 2024-04-22T18:39:45.347 INFO:tasks.rados.rados.0.smithi089.stdout:1069: done (9 left) 2024-04-22T18:39:45.347 INFO:tasks.rados.rados.0.smithi089.stdout:1070: done (8 left) 2024-04-22T18:39:45.347 INFO:tasks.rados.rados.0.smithi089.stdout:1071: done (7 left) 2024-04-22T18:39:45.347 INFO:tasks.rados.rados.0.smithi089.stdout:1072: done (6 left) 2024-04-22T18:39:45.347 INFO:tasks.rados.rados.0.smithi089.stdout:1073: done (5 left) 2024-04-22T18:39:45.347 INFO:tasks.rados.rados.0.smithi089.stdout:1075: done (4 left) 2024-04-22T18:39:45.347 INFO:tasks.rados.rados.0.smithi089.stdout:1076: done (3 left) 2024-04-22T18:39:45.347 INFO:tasks.rados.rados.0.smithi089.stdout:1077: done (2 left) 2024-04-22T18:39:45.347 INFO:tasks.rados.rados.0.smithi089.stdout:1078: done (1 left) 2024-04-22T18:39:45.347 INFO:tasks.rados.rados.0.smithi089.stdout:1079: read oid 22 snap -1 2024-04-22T18:39:45.347 INFO:tasks.rados.rados.0.smithi089.stdout:1079: expect (ObjNum 135 snap 0 seq_num 135) 2024-04-22T18:39:45.347 INFO:tasks.rados.rados.0.smithi089.stdout:1080: write (excl) oid 50 current snap is 0 2024-04-22T18:39:45.347 INFO:tasks.rados.rados.0.smithi089.stdout:1074: finishing write tid 3 to smithi08937929-2 2024-04-22T18:39:45.347 INFO:tasks.rados.rados.0.smithi089.stdout:1074: finishing write tid 4 to smithi08937929-2 2024-04-22T18:39:45.347 INFO:tasks.rados.rados.0.smithi089.stdout:1074: oid 2 updating version 0 to 42 2024-04-22T18:39:45.348 INFO:tasks.rados.rados.0.smithi089.stdout:1074: oid 2 updating version 42 to 43 2024-04-22T18:39:45.348 INFO:tasks.rados.rados.0.smithi089.stdout:1074: oid 2 updating version 43 to 44 2024-04-22T18:39:45.348 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 44 (ObjNum 182 snap 0 seq_num 182) dirty exists 2024-04-22T18:39:45.348 INFO:tasks.rados.rados.0.smithi089.stdout:1074: left oid 2 (ObjNum 182 snap 0 seq_num 182) 2024-04-22T18:39:45.348 INFO:tasks.rados.rados.0.smithi089.stdout:1080: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-04-22T18:39:45.350 INFO:tasks.rados.rados.0.smithi089.stdout:1080: writing smithi08937929-50 from 675681 to 1238720 tid 1 2024-04-22T18:39:45.358 INFO:tasks.rados.rados.0.smithi089.stdout:1080: writing smithi08937929-50 from 1858793 to 2605506 tid 2 2024-04-22T18:39:45.360 INFO:tasks.rados.rados.0.smithi089.stdout:1080: writing smithi08937929-50 from 3025937 to 3075681 tid 3 2024-04-22T18:39:45.360 INFO:tasks.rados.rados.0.smithi089.stdout:1074: done (2 left) 2024-04-22T18:39:45.360 INFO:tasks.rados.rados.0.smithi089.stdout:1081: tier_evict oid 37 2024-04-22T18:39:45.360 INFO:tasks.rados.rados.0.smithi089.stdout:1081: tier_evict oid 37 snap -1 2024-04-22T18:39:45.361 INFO:tasks.rados.rados.0.smithi089.stdout:1082: write oid 28 current snap is 0 2024-04-22T18:39:45.361 INFO:tasks.rados.rados.0.smithi089.stdout:1082: seq_num 184 ranges {692488=638288,1917275=606724} 2024-04-22T18:39:45.367 INFO:tasks.rados.rados.0.smithi089.stdout:1082: writing smithi08937929-28 from 692488 to 1330776 tid 1 2024-04-22T18:39:45.373 INFO:tasks.rados.rados.0.smithi089.stdout:1082: writing smithi08937929-28 from 1917275 to 2523999 tid 2 2024-04-22T18:39:45.373 INFO:tasks.rados.rados.0.smithi089.stdout:1083: read oid 40 snap -1 2024-04-22T18:39:45.373 INFO:tasks.rados.rados.0.smithi089.stdout:1083: expect (ObjNum 58 snap 0 seq_num 58) 2024-04-22T18:39:45.374 INFO:tasks.rados.rados.0.smithi089.stdout:1084: write (excl) oid 30 current snap is 0 2024-04-22T18:39:45.374 INFO:tasks.rados.rados.0.smithi089.stdout:1084: seq_num 185 ranges {709295=713537,1859404=750382} 2024-04-22T18:39:45.379 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:45.379 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:45.379 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:45.382 INFO:tasks.rados.rados.0.smithi089.stdout:1084: writing smithi08937929-30 from 709295 to 1422832 tid 1 2024-04-22T18:39:45.388 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:45.389 INFO:tasks.rados.rados.0.smithi089.stdout:1084: writing smithi08937929-30 from 1859404 to 2609786 tid 2 2024-04-22T18:39:45.390 INFO:tasks.rados.rados.0.smithi089.stdout:1085: read oid 36 snap -1 2024-04-22T18:39:45.390 INFO:tasks.rados.rados.0.smithi089.stdout:1085: expect (ObjNum 179 snap 0 seq_num 179) 2024-04-22T18:39:45.390 INFO:tasks.rados.rados.0.smithi089.stdout:1086: read oid 3 snap -1 2024-04-22T18:39:45.390 INFO:tasks.rados.rados.0.smithi089.stdout:1086: expect (ObjNum 178 snap 0 seq_num 178) 2024-04-22T18:39:45.390 INFO:tasks.rados.rados.0.smithi089.stdout:1087: tier_promote oid 10 2024-04-22T18:39:45.390 INFO:tasks.rados.rados.0.smithi089.stdout:1083: expect (ObjNum 58 snap 0 seq_num 58) 2024-04-22T18:39:45.437 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:45.438+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:45.483 INFO:tasks.rados.rados.0.smithi089.stdout:1081: got (22) Invalid argument 2024-04-22T18:39:45.483 INFO:tasks.rados.rados.0.smithi089.stdout:1081: done (8 left) 2024-04-22T18:39:45.483 INFO:tasks.rados.rados.0.smithi089.stdout:1083: done (7 left) 2024-04-22T18:39:45.484 INFO:tasks.rados.rados.0.smithi089.stdout:1088: read oid 9 snap -1 2024-04-22T18:39:45.484 INFO:tasks.rados.rados.0.smithi089.stdout:1088: expect (ObjNum 153 snap 0 seq_num 153) 2024-04-22T18:39:45.484 INFO:tasks.rados.rados.0.smithi089.stdout:1089: read oid 27 snap -1 2024-04-22T18:39:45.484 INFO:tasks.rados.rados.0.smithi089.stdout:1089: expect (ObjNum 160 snap 0 seq_num 160) 2024-04-22T18:39:45.484 INFO:tasks.rados.rados.0.smithi089.stdout:1090: read oid 34 snap -1 2024-04-22T18:39:45.484 INFO:tasks.rados.rados.0.smithi089.stdout:1090: expect (ObjNum 173 snap 0 seq_num 173) 2024-04-22T18:39:45.484 INFO:tasks.rados.rados.0.smithi089.stdout:1091: delete oid 16 current snap is 0 2024-04-22T18:39:45.489 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:45.490+0000 7fa768d81640 -1 osd.5 57 handle_osd_map injecting map crc failure 2024-04-22T18:39:45.493 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:45.494 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:45.494 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:45.494 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:45.494 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:45.495 INFO:tasks.rados.rados.0.smithi089.stdout:1086: expect (ObjNum 178 snap 0 seq_num 178) 2024-04-22T18:39:45.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:45.537 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:45.538+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:45.593 INFO:tasks.rados.rados.0.smithi089.stdout:1079: expect (ObjNum 135 snap 0 seq_num 135) 2024-04-22T18:39:45.639 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:45.640+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:45.646 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:45.646 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:45.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:45.646 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:45.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.646 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:45.647 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:45.647 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.552491431999997, 2024-04-22T18:39:45.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:39:45.647 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.647 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.647 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.647 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.647 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.647 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:39:45.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.647 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:45.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:45.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:39:45.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:45.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:39:45.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:39:45.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:39:45.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:39:45.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:39:45.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:39:45.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:39:45.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:39:45.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:39:45.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:39:45.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:45.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:39:45.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:39:45.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:39:45.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:39:45.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:39:45.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:39:45.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:39:45.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:39:45.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:39:45.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:39:45.652 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.652 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.652 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.652 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:45.653 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:45.653 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.652993557000002, 2024-04-22T18:39:45.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:39:45.653 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.653 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.653 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:45.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:45.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:39:45.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:45.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:39:45.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:39:45.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:39:45.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:39:45.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:39:45.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:39:45.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:39:45.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:39:45.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:45.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:39:45.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:39:45.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:39:45.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:39:45.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:39:45.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:39:45.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:39:45.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:39:45.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:39:45.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:39:45.658 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.658 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.658 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.658 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:45.658 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:45.659 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.544693766000002, 2024-04-22T18:39:45.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:39:45.659 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.659 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.659 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.659 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.659 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.659 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:39:45.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.659 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:45.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:45.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:39:45.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:45.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:39:45.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:39:45.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:39:45.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:39:45.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:39:45.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:39:45.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:39:45.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:39:45.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:39:45.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:39:45.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:45.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:39:45.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:39:45.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:39:45.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:39:45.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:39:45.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:39:45.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:39:45.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:39:45.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:39:45.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:39:45.664 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.664 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.664 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.213150138000003, 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:45.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:39:45.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:45.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:39:45.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:39:45.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:39:45.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:39:45.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:39:45.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:39:45.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:39:45.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:39:45.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:39:45.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:39:45.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:45.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:39:45.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:39:45.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:39:45.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:39:45.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:39:45.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:39:45.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:39:45.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:39:45.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:39:45.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:39:45.670 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.670 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.670 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.670 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:45.670 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:45.670 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.122064330000001, 2024-04-22T18:39:45.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:39:45.670 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.671 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.671 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.671 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.671 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.671 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:39:45.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.671 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:45.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:45.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:39:45.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:45.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:39:45.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:39:45.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:39:45.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:39:45.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:39:45.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:39:45.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:39:45.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:39:45.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:39:45.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:39:45.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:45.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:39:45.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:39:45.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:39:45.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:39:45.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.11626407, 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:39:45.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:39:45.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:39:45.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:39:45.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:39:45.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:39:45.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:39:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:39:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:39:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:39:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:39:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:39:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:39:45.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:39:45.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:39:45.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:39:45.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:39:45.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:39:45.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:39:45.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:39:45.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.681 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.681 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:45.681 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:45.681 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.111523726999998, 2024-04-22T18:39:45.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:39:45.681 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.681 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.682 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:39:45.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.682 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:45.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:45.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:39:45.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:45.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:39:45.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:39:45.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:39:45.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:39:45.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:39:45.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:39:45.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:39:45.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:39:45.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:39:45.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:39:45.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:45.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:39:45.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:39:45.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:39:45.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:39:45.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:39:45.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:39:45.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:39:45.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:39:45.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:39:45.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:39:45.686 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.686 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.687 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:45.687 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:45.687 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.111050829, 2024-04-22T18:39:45.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:39:45.687 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.687 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.687 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.687 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.687 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.687 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:39:45.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.688 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:45.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:45.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:39:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:39:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:39:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:39:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:39:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:39:45.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:39:45.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:39:45.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:39:45.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:39:45.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:39:45.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:45.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:39:45.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:39:45.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:39:45.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:39:45.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:39:45.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:39:45.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:39:45.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:39:45.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:39:45.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:39:45.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.692 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.692 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:45.692 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:45.692 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.111030907999996, 2024-04-22T18:39:45.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:39:45.693 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.693 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:45.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.693 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:39:45.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.693 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:45.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:45.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:45.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:45.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:39:45.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:39:45.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:39:45.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:39:45.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:39:45.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:39:45.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:39:45.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:39:45.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:45.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:39:45.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:39:45.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:39:45.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:39:45.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:39:45.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:39:45.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:39:45.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:39:45.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.699 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.699 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:45.699 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:39:45.699 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.801707143000002, 2024-04-22T18:39:45.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:39:45.699 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.699 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.700 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:39:45.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:39:45.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:39:45.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:45.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:39:45.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:39:45.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:39:45.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:39:45.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:39:45.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:39:45.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:39:45.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:39:45.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:39:45.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:39:45.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:45.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:39:45.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:39:45.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:39:45.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:39:45.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:39:45.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:39:45.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:39:45.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:39:45.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:39:45.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:39:45.704 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.704 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.704 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.705 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:45.705 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:39:45.705 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.801499607, 2024-04-22T18:39:45.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:39:45.705 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.705 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.705 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:39:45.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:39:45.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:39:45.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:45.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:39:45.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:39:45.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:39:45.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:39:45.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:39:45.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:39:45.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:39:45.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:39:45.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:39:45.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:39:45.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:45.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:39:45.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:39:45.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:39:45.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:39:45.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:39:45.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:39:45.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:39:45.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:39:45.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:39:45.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:39:45.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.710 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.710 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:45.710 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:39:45.710 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.9358498740000005, 2024-04-22T18:39:45.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:39:45.710 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.710 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.710 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:39:45.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:39:45.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:39:45.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:45.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:39:45.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:39:45.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:39:45.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:39:45.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:39:45.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:39:45.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:39:45.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:39:45.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:39:45.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:39:45.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:45.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:39:45.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:39:45.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:39:45.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:39:45.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:39:45.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:39:45.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:39:45.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:39:45.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:39:45.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:39:45.715 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.716 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.716 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.716 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:45.716 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:39:45.716 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.7220114889999998, 2024-04-22T18:39:45.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:39:45.716 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.716 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.716 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:39:45.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:39:45.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:39:45.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:45.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:39:45.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:39:45.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:39:45.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:39:45.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:39:45.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:39:45.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:39:45.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:39:45.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:39:45.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:39:45.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:45.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:39:45.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:39:45.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:39:45.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:39:45.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:39:45.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:39:45.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:39:45.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:39:45.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:39:45.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:39:45.721 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.721 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.721 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.721 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:45.721 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:39:45.721 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.7219780789999994, 2024-04-22T18:39:45.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:39:45.722 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.722 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.722 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:39:45.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:39:45.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:39:45.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:39:45.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:39:45.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:39:45.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:39:45.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:39:45.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:45.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:39:45.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:39:45.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:39:45.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:39:45.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:39:45.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:39:45.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:39:45.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:39:45.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:39:45.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:39:45.726 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.726 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.726 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.726 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:45.727 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:39:45.727 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.6630954249999998, 2024-04-22T18:39:45.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:39:45.727 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.727 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.727 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:39:45.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:39:45.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:39:45.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:45.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:39:45.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:39:45.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:39:45.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:39:45.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:39:45.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:39:45.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:39:45.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:39:45.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:39:45.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:39:45.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:45.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:39:45.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:39:45.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:39:45.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:39:45.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:39:45.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:39:45.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:39:45.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:39:45.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:39:45.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:39:45.732 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.732 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.732 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.732 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:45.732 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:39:45.732 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.6629568090000006, 2024-04-22T18:39:45.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:39:45.732 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.732 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.733 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:39:45.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:39:45.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:39:45.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:45.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:39:45.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:39:45.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:39:45.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:39:45.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:39:45.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:39:45.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:39:45.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:39:45.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:39:45.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:39:45.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:45.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:39:45.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:39:45.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:39:45.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:39:45.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:39:45.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:39:45.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:39:45.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:39:45.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:39:45.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:39:45.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.737 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.738 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:45.738 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:39:45.738 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.6629378060000004, 2024-04-22T18:39:45.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:39:45.738 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.738 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.738 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:39:45.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:39:45.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:39:45.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:45.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:39:45.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:39:45.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:39:45.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:39:45.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:39:45.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:39:45.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:39:45.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:39:45.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:39:45.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:39:45.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:45.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:39:45.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:39:45.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:39:45.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:39:45.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:39:45.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:39:45.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:39:45.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:39:45.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:39:45.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:39:45.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.743 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.744 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:39:45.744 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:39:45.744 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.0419252730000004, 2024-04-22T18:39:45.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:39:45.744 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.744 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.744 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:39:45.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.744 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:39:45.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:39:45.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:39:45.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:45.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:39:45.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:39:45.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:39:45.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:39:45.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:39:45.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:39:45.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:39:45.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:39:45.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:39:45.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:39:45.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:45.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:39:45.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:39:45.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:39:45.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:39:45.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:39:45.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:39:45.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:39:45.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:39:45.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:39:45.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:39:45.749 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.749 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.749 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.749 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:39:45.750 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:39:45.750 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.0414056819999997, 2024-04-22T18:39:45.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:39:45.750 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.750 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:45.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.750 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:39:45.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.750 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:39:45.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:39:45.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:39:45.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:45.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:39:45.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:39:45.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:39:45.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:39:45.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:39:45.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:39:45.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:39:45.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:39:45.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:39:45.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:39:45.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:45.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:39:45.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:39:45.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:45.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:39:45.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:39:45.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:45.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:39:45.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:39:45.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:45.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:39:45.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:39:45.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:39:45.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:39:45.755 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.755 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.755 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.755 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:39:45.755 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:39:45.755 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.9412527989999999, 2024-04-22T18:39:45.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:39:45.755 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:45.756 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:45.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:45.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:45.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:45.756 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:39:45.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.756 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:45.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:45.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:39:45.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:45.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:39:45.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:45.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:45.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:39:45.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:45.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:45.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:39:45.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:39:45.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:45.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:39:45.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:39:45.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:45.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:39:45.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:39:45.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:39:45.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:39:45.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:45.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:39:45.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:39:45.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:45.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:39:45.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:39:45.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:45.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:39:45.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:39:45.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:45.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:45.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:45.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:39:45.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:39:45.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.760 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:45.760 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:45.760 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:45.761 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:39:45.762 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:45.695+0000 7f3e7be88640 -1 osd.0 58 handle_osd_map injecting map crc failure 2024-04-22T18:39:45.762 INFO:tasks.rados.rados.0.smithi089.stdout:1080: finishing write tid 1 to smithi08937929-50 2024-04-22T18:39:45.762 INFO:tasks.rados.rados.0.smithi089.stdout:1080: finishing write tid 2 to smithi08937929-50 2024-04-22T18:39:45.763 INFO:tasks.rados.rados.0.smithi089.stdout:1088: expect (ObjNum 153 snap 0 seq_num 153) 2024-04-22T18:39:45.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:45.764 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:45.763+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:45.862 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:45.863+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:45.882 INFO:tasks.rados.rados.0.smithi089.stdout:1085: expect (ObjNum 179 snap 0 seq_num 179) 2024-04-22T18:39:45.907 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:45.907 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:45.907 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:45.907 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:45.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:45.963 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:45.964+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:46.018 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:46.018 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:46.027 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:46.060 INFO:tasks.rados.rados.0.smithi089.stdout:1080: finishing write tid 3 to smithi08937929-50 2024-04-22T18:39:46.060 INFO:tasks.rados.rados.0.smithi089.stdout:1080: finishing write tid 4 to smithi08937929-50 2024-04-22T18:39:46.060 INFO:tasks.rados.rados.0.smithi089.stdout:1080: finishing write tid 5 to smithi08937929-50 2024-04-22T18:39:46.060 INFO:tasks.rados.rados.0.smithi089.stdout:1080: oid 50 updating version 0 to 101 2024-04-22T18:39:46.060 INFO:tasks.rados.rados.0.smithi089.stdout:1080: oid 50 version 101 is already newer than 99 2024-04-22T18:39:46.060 INFO:tasks.rados.rados.0.smithi089.stdout:1080: oid 50 version 101 is already newer than 98 2024-04-22T18:39:46.060 INFO:tasks.rados.rados.0.smithi089.stdout:1080: oid 50 version 101 is already newer than 100 2024-04-22T18:39:46.060 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 101 (ObjNum 183 snap 0 seq_num 183) dirty exists 2024-04-22T18:39:46.061 INFO:tasks.rados.rados.0.smithi089.stdout:1080: left oid 50 (ObjNum 183 snap 0 seq_num 183) 2024-04-22T18:39:46.061 INFO:tasks.rados.rados.0.smithi089.stdout:1084: finishing write tid 1 to smithi08937929-30 2024-04-22T18:39:46.061 INFO:tasks.rados.rados.0.smithi089.stdout:1079: done (10 left) 2024-04-22T18:39:46.061 INFO:tasks.rados.rados.0.smithi089.stdout:1080: done (9 left) 2024-04-22T18:39:46.061 INFO:tasks.rados.rados.0.smithi089.stdout:1085: done (8 left) 2024-04-22T18:39:46.061 INFO:tasks.rados.rados.0.smithi089.stdout:1086: done (7 left) 2024-04-22T18:39:46.061 INFO:tasks.rados.rados.0.smithi089.stdout:1088: done (6 left) 2024-04-22T18:39:46.061 INFO:tasks.rados.rados.0.smithi089.stdout:1091: done (5 left) 2024-04-22T18:39:46.061 INFO:tasks.rados.rados.0.smithi089.stdout:1092: write (excl) oid 44 current snap is 0 2024-04-22T18:39:46.061 INFO:tasks.rados.rados.0.smithi089.stdout:1084: finishing write tid 2 to smithi08937929-30 2024-04-22T18:39:46.061 INFO:tasks.rados.rados.0.smithi089.stdout:1092: seq_num 187 ranges {742909=464035,1976368=754051} 2024-04-22T18:39:46.064 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:46.065+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:46.066 INFO:tasks.rados.rados.0.smithi089.stdout:1092: writing smithi08937929-44 from 742909 to 1206944 tid 1 2024-04-22T18:39:46.074 INFO:tasks.rados.rados.0.smithi089.stdout:1092: writing smithi08937929-44 from 1976368 to 2730419 tid 2 2024-04-22T18:39:46.075 INFO:tasks.rados.rados.0.smithi089.stdout:1093: write (excl) oid 39 current snap is 0 2024-04-22T18:39:46.075 INFO:tasks.rados.rados.0.smithi089.stdout:1093: seq_num 188 ranges {759716=539284,2034850=614062} 2024-04-22T18:39:46.081 INFO:tasks.rados.rados.0.smithi089.stdout:1093: writing smithi08937929-39 from 759716 to 1299000 tid 1 2024-04-22T18:39:46.089 INFO:tasks.rados.rados.0.smithi089.stdout:1093: writing smithi08937929-39 from 2034850 to 2648912 tid 2 2024-04-22T18:39:46.089 INFO:tasks.rados.rados.0.smithi089.stdout:1094: tier_evict oid 6 2024-04-22T18:39:46.089 INFO:tasks.rados.rados.0.smithi089.stdout:1094: tier_evict oid 6 snap -1 2024-04-22T18:39:46.090 INFO:tasks.rados.rados.0.smithi089.stdout:1084: finishing write tid 3 to smithi08937929-30 2024-04-22T18:39:46.090 INFO:tasks.rados.rados.0.smithi089.stdout:1084: finishing write tid 4 to smithi08937929-30 2024-04-22T18:39:46.090 INFO:tasks.rados.rados.0.smithi089.stdout:1084: oid 30 updating version 0 to 31 2024-04-22T18:39:46.090 INFO:tasks.rados.rados.0.smithi089.stdout:1084: oid 30 version 31 is already newer than 29 2024-04-22T18:39:46.090 INFO:tasks.rados.rados.0.smithi089.stdout:1084: oid 30 version 31 is already newer than 30 2024-04-22T18:39:46.090 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 31 (ObjNum 185 snap 0 seq_num 185) dirty exists 2024-04-22T18:39:46.090 INFO:tasks.rados.rados.0.smithi089.stdout:1084: left oid 30 (ObjNum 185 snap 0 seq_num 185) 2024-04-22T18:39:46.090 INFO:tasks.rados.rados.0.smithi089.stdout:1092: finishing write tid 1 to smithi08937929-44 2024-04-22T18:39:46.090 INFO:tasks.rados.rados.0.smithi089.stdout:1084: done (7 left) 2024-04-22T18:39:46.090 INFO:tasks.rados.rados.0.smithi089.stdout:1095: read oid 7 snap -1 2024-04-22T18:39:46.090 INFO:tasks.rados.rados.0.smithi089.stdout:1095: expect (ObjNum 161 snap 0 seq_num 161) 2024-04-22T18:39:46.090 INFO:tasks.rados.rados.0.smithi089.stdout:1096: write (excl) oid 50 current snap is 0 2024-04-22T18:39:46.091 INFO:tasks.rados.rados.0.smithi089.stdout:1092: finishing write tid 2 to smithi08937929-44 2024-04-22T18:39:46.091 INFO:tasks.rados.rados.0.smithi089.stdout:1096: seq_num 189 ranges {776523=614533,1976979=757720} 2024-04-22T18:39:46.097 INFO:tasks.rados.rados.0.smithi089.stdout:1096: writing smithi08937929-50 from 776523 to 1391056 tid 1 2024-04-22T18:39:46.105 INFO:tasks.rados.rados.0.smithi089.stdout:1096: writing smithi08937929-50 from 1976979 to 2734699 tid 2 2024-04-22T18:39:46.105 INFO:tasks.rados.rados.0.smithi089.stdout:1097: delete oid 19 current snap is 0 2024-04-22T18:39:46.105 INFO:tasks.rados.rados.0.smithi089.stdout:1092: finishing write tid 3 to smithi08937929-44 2024-04-22T18:39:46.106 INFO:tasks.rados.rados.0.smithi089.stdout:1092: finishing write tid 4 to smithi08937929-44 2024-04-22T18:39:46.106 INFO:tasks.rados.rados.0.smithi089.stdout:1092: oid 44 updating version 0 to 84 2024-04-22T18:39:46.106 INFO:tasks.rados.rados.0.smithi089.stdout:1092: oid 44 updating version 84 to 85 2024-04-22T18:39:46.106 INFO:tasks.rados.rados.0.smithi089.stdout:1092: oid 44 version 85 is already newer than 83 2024-04-22T18:39:46.106 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 85 (ObjNum 187 snap 0 seq_num 187) dirty exists 2024-04-22T18:39:46.106 INFO:tasks.rados.rados.0.smithi089.stdout:1092: left oid 44 (ObjNum 187 snap 0 seq_num 187) 2024-04-22T18:39:46.106 INFO:tasks.rados.rados.0.smithi089.stdout:1094: got (22) Invalid argument 2024-04-22T18:39:46.106 INFO:tasks.rados.rados.0.smithi089.stdout:1095: expect (ObjNum 161 snap 0 seq_num 161) 2024-04-22T18:39:46.165 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:46.166+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:46.173 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:46.173 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:46.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:46.173 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:46.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.173 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:21 4.a 4:5aa97f94:::smithi08937929-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.173 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:46.173 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 52.019877999000002, 2024-04-22T18:39:46.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21044934000000001, 2024-04-22T18:39:46.174 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.174 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.174 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.174 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.174 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.174 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 21 2024-04-22T18:39:46.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.174 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:46.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:46.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153183+0000", 2024-04-22T18:39:46.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:46.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154022+0000", 2024-04-22T18:39:46.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00083893300000000004 2024-04-22T18:39:46.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154032+0000", 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0433e-05 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154051+0000", 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8227e-05 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154103+0000", 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2491999999999998e-05 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:46.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154111+0000", 2024-04-22T18:39:46.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3470000000000001e-06 2024-04-22T18:39:46.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257848+0000", 2024-04-22T18:39:46.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103737703 2024-04-22T18:39:46.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257981+0000", 2024-04-22T18:39:46.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013290399999999999 2024-04-22T18:39:46.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:46.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.258138+0000", 2024-04-22T18:39:46.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015701199999999999 2024-04-22T18:39:46.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.261944+0000", 2024-04-22T18:39:46.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038054650000000001 2024-04-22T18:39:46.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363523+0000", 2024-04-22T18:39:46.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101579239 2024-04-22T18:39:46.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363577+0000", 2024-04-22T18:39:46.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4095000000000003e-05 2024-04-22T18:39:46.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363635+0000", 2024-04-22T18:39:46.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8362999999999997e-05 2024-04-22T18:39:46.179 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.179 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.179 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.179 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:22 4.a 4:5aa97f94:::smithi08937929-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.180 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:46.180 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 52.015804023000001, 2024-04-22T18:39:46.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30948398900000001, 2024-04-22T18:39:46.180 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.180 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.180 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.180 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.180 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.180 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 22 2024-04-22T18:39:46.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.180 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:46.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:46.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157259+0000", 2024-04-22T18:39:46.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:46.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157318+0000", 2024-04-22T18:39:46.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8906000000000001e-05 2024-04-22T18:39:46.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157327+0000", 2024-04-22T18:39:46.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:39:46.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157332+0000", 2024-04-22T18:39:46.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4689999999999996e-06 2024-04-22T18:39:46.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257479+0000", 2024-04-22T18:39:46.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014700999999999 2024-04-22T18:39:46.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:46.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257492+0000", 2024-04-22T18:39:46.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2574e-05 2024-04-22T18:39:46.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259582+0000", 2024-04-22T18:39:46.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020904019999999999 2024-04-22T18:39:46.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259668+0000", 2024-04-22T18:39:46.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5826000000000001e-05 2024-04-22T18:39:46.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:46.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259772+0000", 2024-04-22T18:39:46.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104048 2024-04-22T18:39:46.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363453+0000", 2024-04-22T18:39:46.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103680183 2024-04-22T18:39:46.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466692+0000", 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10323948199999999 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466720+0000", 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7886e-05 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466744+0000", 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4049e-05 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.185 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.186 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:23 4.a 4:5aa97f94:::smithi08937929-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.186 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:46.186 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 52.015726336, 2024-04-22T18:39:46.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39003820300000003, 2024-04-22T18:39:46.186 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.186 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.186 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 23 2024-04-22T18:39:46.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.186 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157337+0000", 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157340+0000", 2024-04-22T18:39:46.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8609999999999998e-06 2024-04-22T18:39:46.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157343+0000", 2024-04-22T18:39:46.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9859999999999999e-06 2024-04-22T18:39:46.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157346+0000", 2024-04-22T18:39:46.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8119999999999999e-06 2024-04-22T18:39:46.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257563+0000", 2024-04-22T18:39:46.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100217497 2024-04-22T18:39:46.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:46.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257574+0000", 2024-04-22T18:39:46.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0777e-05 2024-04-22T18:39:46.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260112+0000", 2024-04-22T18:39:46.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002537681 2024-04-22T18:39:46.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260191+0000", 2024-04-22T18:39:46.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9320999999999995e-05 2024-04-22T18:39:46.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:46.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260333+0000", 2024-04-22T18:39:46.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014177699999999999 2024-04-22T18:39:46.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363792+0000", 2024-04-22T18:39:46.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103458862 2024-04-22T18:39:46.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547303+0000", 2024-04-22T18:39:46.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18351162100000001 2024-04-22T18:39:46.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547337+0000", 2024-04-22T18:39:46.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3640999999999999e-05 2024-04-22T18:39:46.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547376+0000", 2024-04-22T18:39:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8915000000000003e-05 2024-04-22T18:39:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:24 4.a 4:5aa97f94:::smithi08937929-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 52.015699904999998, 2024-04-22T18:39:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39017823499999998, 2024-04-22T18:39:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:46.192 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.193 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.193 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.193 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 24 2024-04-22T18:39:46.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.193 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:46.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:46.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:46.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157366+0000", 2024-04-22T18:39:46.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3120000000000001e-06 2024-04-22T18:39:46.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157370+0000", 2024-04-22T18:39:46.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1300000000000003e-06 2024-04-22T18:39:46.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157373+0000", 2024-04-22T18:39:46.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-04-22T18:39:46.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260553+0000", 2024-04-22T18:39:46.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103179996 2024-04-22T18:39:46.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:46.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260625+0000", 2024-04-22T18:39:46.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2236000000000004e-05 2024-04-22T18:39:46.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547414+0000", 2024-04-22T18:39:46.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.286789181 2024-04-22T18:39:46.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547471+0000", 2024-04-22T18:39:46.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6192e-05 2024-04-22T18:39:46.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547542+0000", 2024-04-22T18:39:46.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1622e-05 2024-04-22T18:39:46.197 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.197 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.197 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.197 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:49 4.a 4:5d2aee44:::smithi08937929-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.197 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:46.197 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.938901246999997, 2024-04-22T18:39:46.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31344442500000003, 2024-04-22T18:39:46.197 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.197 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.198 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.198 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.198 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 49 2024-04-22T18:39:46.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.198 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:46.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:46.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234158+0000", 2024-04-22T18:39:46.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999957 2024-04-22T18:39:46.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234901+0000", 2024-04-22T18:39:46.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074250800000000001 2024-04-22T18:39:46.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234914+0000", 2024-04-22T18:39:46.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3244999999999999e-05 2024-04-22T18:39:46.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234922+0000", 2024-04-22T18:39:46.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4590000000000001e-06 2024-04-22T18:39:46.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260665+0000", 2024-04-22T18:39:46.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025743423000000001 2024-04-22T18:39:46.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260779+0000", 2024-04-22T18:39:46.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114063 2024-04-22T18:39:46.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:46.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260913+0000", 2024-04-22T18:39:46.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013355499999999999 2024-04-22T18:39:46.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363798+0000", 2024-04-22T18:39:46.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102885003 2024-04-22T18:39:46.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547567+0000", 2024-04-22T18:39:46.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.183768971 2024-04-22T18:39:46.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547589+0000", 2024-04-22T18:39:46.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2507999999999999e-05 2024-04-22T18:39:46.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547607+0000", 2024-04-22T18:39:46.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8009999999999999e-05 2024-04-22T18:39:46.202 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.202 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:50 4.a 4:5d2aee44:::smithi08937929-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.933455068000001, 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40856295399999998, 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 50 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239607+0000", 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240309+0000", 2024-04-22T18:39:46.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070201599999999997 2024-04-22T18:39:46.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240320+0000", 2024-04-22T18:39:46.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1024e-05 2024-04-22T18:39:46.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240327+0000", 2024-04-22T18:39:46.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0049999999999998e-06 2024-04-22T18:39:46.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362065+0000", 2024-04-22T18:39:46.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.121737812 2024-04-22T18:39:46.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362121+0000", 2024-04-22T18:39:46.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5868000000000001e-05 2024-04-22T18:39:46.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:46.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362210+0000", 2024-04-22T18:39:46.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8533000000000001e-05 2024-04-22T18:39:46.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365322+0000", 2024-04-22T18:39:46.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031121220000000002 2024-04-22T18:39:46.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648054+0000", 2024-04-22T18:39:46.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.282732549 2024-04-22T18:39:46.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648105+0000", 2024-04-22T18:39:46.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1276999999999997e-05 2024-04-22T18:39:46.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648172+0000", 2024-04-22T18:39:46.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6398999999999999e-05 2024-04-22T18:39:46.208 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.208 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.208 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.208 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:51 4.a 4:5d2aee44:::smithi08937929-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.208 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:46.208 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.932732534000003, 2024-04-22T18:39:46.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40793057199999999, 2024-04-22T18:39:46.209 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.209 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.209 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.209 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.209 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.209 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 51 2024-04-22T18:39:46.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.209 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:46.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:46.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:46.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240336+0000", 2024-04-22T18:39:46.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:39:46.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240339+0000", 2024-04-22T18:39:46.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9009999999999998e-06 2024-04-22T18:39:46.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240342+0000", 2024-04-22T18:39:46.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:39:46.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363080+0000", 2024-04-22T18:39:46.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122738075 2024-04-22T18:39:46.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363164+0000", 2024-04-22T18:39:46.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3722999999999998e-05 2024-04-22T18:39:46.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:46.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363287+0000", 2024-04-22T18:39:46.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012301099999999999 2024-04-22T18:39:46.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466753+0000", 2024-04-22T18:39:46.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103466217 2024-04-22T18:39:46.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648205+0000", 2024-04-22T18:39:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.181451788 2024-04-22T18:39:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648231+0000", 2024-04-22T18:39:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6597999999999998e-05 2024-04-22T18:39:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648262+0000", 2024-04-22T18:39:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0871000000000003e-05 2024-04-22T18:39:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.213 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.214 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.214 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:52 4.a 4:5d2aee44:::smithi08937929-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.214 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:46.214 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.932720074999999, 2024-04-22T18:39:46.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40812731200000002, 2024-04-22T18:39:46.214 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.214 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:46.214 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.214 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.214 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.214 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 52 2024-04-22T18:39:46.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240345+0000", 2024-04-22T18:39:46.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4300000000000001e-06 2024-04-22T18:39:46.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240349+0000", 2024-04-22T18:39:46.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2570000000000001e-06 2024-04-22T18:39:46.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240358+0000", 2024-04-22T18:39:46.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-22T18:39:46.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363697+0000", 2024-04-22T18:39:46.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12333965299999999 2024-04-22T18:39:46.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:46.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363777+0000", 2024-04-22T18:39:46.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9883999999999996e-05 2024-04-22T18:39:46.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648296+0000", 2024-04-22T18:39:46.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451861299999998 2024-04-22T18:39:46.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648379+0000", 2024-04-22T18:39:46.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3302999999999998e-05 2024-04-22T18:39:46.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648471+0000", 2024-04-22T18:39:46.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2029999999999998e-05 2024-04-22T18:39:46.218 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.218 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.218 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.219 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.219 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:46.219 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.913416740000002, 2024-04-22T18:39:46.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:39:46.219 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.219 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.219 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:39:46.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.220 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:46.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:46.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:39:46.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:46.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:39:46.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:39:46.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:39:46.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:39:46.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:39:46.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:39:46.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:39:46.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:39:46.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:39:46.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:39:46.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:46.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:39:46.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:39:46.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:39:46.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:39:46.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.225 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.225 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:46.225 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.908634309999997, 2024-04-22T18:39:46.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:39:46.225 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.225 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.225 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:39:46.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.225 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:46.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:46.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:39:46.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:46.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:39:46.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:39:46.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:39:46.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:39:46.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:39:46.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:39:46.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:39:46.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:39:46.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:39:46.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:39:46.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:46.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:39:46.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:39:46.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:39:46.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:39:46.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:39:46.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:39:46.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:39:46.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:39:46.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:39:46.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:39:46.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.230 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.230 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.230 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:46.230 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.908540107, 2024-04-22T18:39:46.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:39:46.231 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.231 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.231 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.231 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.231 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.231 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:39:46.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.231 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:46.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:46.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:39:46.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:46.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:39:46.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:39:46.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:39:46.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:39:46.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:39:46.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:39:46.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:39:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:39:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:39:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:39:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:39:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:39:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:39:46.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:39:46.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:39:46.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:39:46.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:39:46.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:39:46.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:39:46.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:39:46.236 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.236 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.236 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.236 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.236 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:46.236 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.908519218999999, 2024-04-22T18:39:46.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:39:46.236 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.236 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:46.236 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.236 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.237 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.237 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:39:46.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.237 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:46.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:46.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:39:46.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:46.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:39:46.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:39:46.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:39:46.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:39:46.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:39:46.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:39:46.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:39:46.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:39:46.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:46.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:39:46.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:39:46.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:39:46.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:39:46.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:39:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:39:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:39:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:39:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:46.242 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.125511555999999, 2024-04-22T18:39:46.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:39:46.242 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.242 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.242 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:39:46.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.242 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:46.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:46.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:39:46.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:46.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:39:46.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:39:46.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:39:46.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:39:46.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:46.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:39:46.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:39:46.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:39:46.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:39:46.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:39:46.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:39:46.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:39:46.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:39:46.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:39:46.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:39:46.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.247 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.247 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.247 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:46.247 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.117680149999998, 2024-04-22T18:39:46.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:39:46.247 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.247 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.248 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:39:46.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.248 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:46.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:46.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:39:46.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:46.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:39:46.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:39:46.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:39:46.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:39:46.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:39:46.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:39:46.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:39:46.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:39:46.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:39:46.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:39:46.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:186 4.a 4:537c89f8:::smithi08937929-47:head [write 1992054~651681] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:46.252 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.770174615000002, 2024-04-22T18:39:46.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20325226800000001, 2024-04-22T18:39:46.253 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.253 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.253 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.253 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.253 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.253 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 186 2024-04-22T18:39:46.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.253 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:46.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:46.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402888+0000", 2024-04-22T18:39:46.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:46.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403454+0000", 2024-04-22T18:39:46.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056661000000000005 2024-04-22T18:39:46.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403465+0000", 2024-04-22T18:39:46.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0727000000000001e-05 2024-04-22T18:39:46.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403473+0000", 2024-04-22T18:39:46.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7039999999999994e-06 2024-04-22T18:39:46.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404592+0000", 2024-04-22T18:39:46.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001118827 2024-04-22T18:39:46.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404674+0000", 2024-04-22T18:39:46.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2793999999999994e-05 2024-04-22T18:39:46.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:46.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404779+0000", 2024-04-22T18:39:46.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104465 2024-04-22T18:39:46.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.406080+0000", 2024-04-22T18:39:46.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013006789999999999 2024-04-22T18:39:46.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606037+0000", 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19995755600000001 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606098+0000", 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1075999999999996e-05 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606142+0000", 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3411999999999997e-05 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.257 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.258 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.258 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:187 4.a 4:537c89f8:::smithi08937929-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:46.258 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:46.258 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.769584475000002, 2024-04-22T18:39:46.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203643623, 2024-04-22T18:39:46.258 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.258 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.258 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.258 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.258 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.258 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 187 2024-04-22T18:39:46.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.259 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:46.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:46.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:46.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:46.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403501+0000", 2024-04-22T18:39:46.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2146000000000001e-05 2024-04-22T18:39:46.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403506+0000", 2024-04-22T18:39:46.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3040000000000004e-06 2024-04-22T18:39:46.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403511+0000", 2024-04-22T18:39:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9599999999999999e-06 2024-04-22T18:39:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405587+0000", 2024-04-22T18:39:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002076296 2024-04-22T18:39:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405664+0000", 2024-04-22T18:39:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6316000000000006e-05 2024-04-22T18:39:46.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:46.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405785+0000", 2024-04-22T18:39:46.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121276 2024-04-22T18:39:46.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.408030+0000", 2024-04-22T18:39:46.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022452169999999999 2024-04-22T18:39:46.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607059+0000", 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199029135 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607094+0000", 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4686e-05 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607123+0000", 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9113000000000001e-05 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.263 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:811 4.5 4:a4a14dff:::smithi08937929-40:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 42.052242018000001, 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242306563, 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 811 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120821+0000", 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120825+0000", 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.506e-06 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120829+0000", 2024-04-22T18:39:46.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0849999999999999e-06 2024-04-22T18:39:46.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120833+0000", 2024-04-22T18:39:46.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9609999999999999e-06 2024-04-22T18:39:46.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221263+0000", 2024-04-22T18:39:46.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10043045 2024-04-22T18:39:46.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221337+0000", 2024-04-22T18:39:46.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4010000000000005e-05 2024-04-22T18:39:46.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:46.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221448+0000", 2024-04-22T18:39:46.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011065699999999999 2024-04-22T18:39:46.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.222541+0000", 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010932610000000001 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.362952+0000", 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.140410535 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.363102+0000", 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015041499999999999 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.363129+0000", 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6574999999999999e-05 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:933 4.a 4:5d2aee44:::smithi08937929-13:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 38.528318380000002, 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202744591, 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.269 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 933 2024-04-22T18:39:46.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.270 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:46.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:46.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644743+0000", 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644754+0000", 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0807e-05 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644768+0000", 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3844e-05 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.744860+0000", 2024-04-22T18:39:46.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100092835 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.745084+0000", 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022401699999999999 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.745090+0000", 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2970000000000003e-06 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.847381+0000", 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102291544 2024-04-22T18:39:46.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.847490+0000", 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010891399999999999 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2052 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.5955683690000004, 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43058958600000002, 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.273 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2052 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577494+0000", 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577500+0000", 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3279999999999996e-06 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577510+0000", 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9450000000000005e-06 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577515+0000", 2024-04-22T18:39:46.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7799999999999997e-06 2024-04-22T18:39:46.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677622+0000", 2024-04-22T18:39:46.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010654099999999 2024-04-22T18:39:46.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677710+0000", 2024-04-22T18:39:46.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8472000000000003e-05 2024-04-22T18:39:46.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:46.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.906619+0000", 2024-04-22T18:39:46.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.228908519 2024-04-22T18:39:46.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907343+0000", 2024-04-22T18:39:46.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072442600000000002 2024-04-22T18:39:46.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.007972+0000", 2024-04-22T18:39:46.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100628781 2024-04-22T18:39:46.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008022+0000", 2024-04-22T18:39:46.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9843999999999999e-05 2024-04-22T18:39:46.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008085+0000", 2024-04-22T18:39:46.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3256999999999994e-05 2024-04-22T18:39:46.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.278 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.278 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2087 4.a 4:537c89f8:::smithi08937929-47:head [assert-version v37,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-04-22T18:39:46.278 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.2656516089999998, 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38904324299999998, 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2087 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:39:46.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907410+0000", 2024-04-22T18:39:46.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:46.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907417+0000", 2024-04-22T18:39:46.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.306e-06 2024-04-22T18:39:46.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907427+0000", 2024-04-22T18:39:46.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4930000000000004e-06 2024-04-22T18:39:46.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907432+0000", 2024-04-22T18:39:46.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5400000000000003e-06 2024-04-22T18:39:46.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907477+0000", 2024-04-22T18:39:46.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4851999999999997e-05 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907537+0000", 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9460999999999999e-05 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.295587+0000", 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38805073200000001 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296281+0000", 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00069370800000000002 2024-04-22T18:39:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296400+0000", 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118579 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296431+0000", 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1627000000000001e-05 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296456+0000", 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4331000000000001e-05 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.283 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.284 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.284 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.284 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.284 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:46.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:46.286 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:46.286+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:46.386 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:46.387+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:46.427 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:46.427 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:46.427 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:46.428 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:46.436 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:46.486 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:46.486+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:46.536 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:46.536 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:46.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:46.579 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:39:46.624 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:46.624+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:46.648 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 66.906451215000004, 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30520528099999999, 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 35 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:46.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741852+0000", 2024-04-22T18:39:46.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:46.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741927+0000", 2024-04-22T18:39:46.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5123000000000006e-05 2024-04-22T18:39:46.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741953+0000", 2024-04-22T18:39:46.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5378999999999999e-05 2024-04-22T18:39:46.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741977+0000", 2024-04-22T18:39:46.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4431999999999999e-05 2024-04-22T18:39:46.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844243+0000", 2024-04-22T18:39:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102265525 2024-04-22T18:39:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844330+0000", 2024-04-22T18:39:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7417999999999995e-05 2024-04-22T18:39:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844470+0000", 2024-04-22T18:39:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000140197 2024-04-22T18:39:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944767+0000", 2024-04-22T18:39:46.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100296409 2024-04-22T18:39:46.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.046968+0000", 2024-04-22T18:39:46.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10220148499999999 2024-04-22T18:39:46.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047018+0000", 2024-04-22T18:39:46.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9206000000000003e-05 2024-04-22T18:39:46.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047059+0000", 2024-04-22T18:39:46.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1496999999999999e-05 2024-04-22T18:39:46.654 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.654 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.654 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.654 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:46.654 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:46.654 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 66.906320691000005, 2024-04-22T18:39:46.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:39:46.654 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.654 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.654 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.654 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:39:46.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:46.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:39:46.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:39:46.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:39:46.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:39:46.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:39:46.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:39:46.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:39:46.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:39:46.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:39:46.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:39:46.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:39:46.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:39:46.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:39:46.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:39:46.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:39:46.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:39:46.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:46.659 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:46.660 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 66.906281000000007, 2024-04-22T18:39:46.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:39:46.660 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.660 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.660 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:39:46.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.660 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:46.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:46.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:39:46.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:46.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:39:46.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:39:46.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:39:46.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:39:46.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:39:46.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:39:46.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:39:46.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:39:46.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:39:46.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:39:46.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:39:46.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:39:46.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:39:46.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:39:46.665 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.665 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.665 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.665 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:46.665 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:46.665 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.662099952999995, 2024-04-22T18:39:46.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402419109, 2024-04-22T18:39:46.665 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.665 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.665 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.665 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.665 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 68 2024-04-22T18:39:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986204+0000", 2024-04-22T18:39:46.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986208+0000", 2024-04-22T18:39:46.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8149999999999999e-06 2024-04-22T18:39:46.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986214+0000", 2024-04-22T18:39:46.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.604e-06 2024-04-22T18:39:46.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986218+0000", 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6689999999999999e-06 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986833+0000", 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000615648 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986891+0000", 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7915999999999999e-05 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986981+0000", 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759999999999994e-05 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.987584+0000", 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060270199999999999 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388527+0000", 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40094366999999997 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388583+0000", 2024-04-22T18:39:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5383000000000002e-05 2024-04-22T18:39:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388624+0000", 2024-04-22T18:39:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501999999999997e-05 2024-04-22T18:39:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.662084164000007, 2024-04-22T18:39:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:39:46.670 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.671 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.671 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.671 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.671 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.671 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:39:46.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.671 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:46.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:46.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:39:46.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:39:46.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:39:46.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:39:46.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:39:46.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:39:46.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:39:46.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:39:46.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:39:46.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:39:46.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:39:46.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:39:46.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:39:46.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:39:46.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:39:46.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:39:46.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:39:46.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:39:46.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:39:46.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:39:46.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:39:46.675 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.675 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.675 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 63.545691155, 2024-04-22T18:39:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305563733, 2024-04-22T18:39:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 87 2024-04-22T18:39:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:46.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:46.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102613+0000", 2024-04-22T18:39:46.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:46.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102668+0000", 2024-04-22T18:39:46.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4669000000000001e-05 2024-04-22T18:39:46.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102680+0000", 2024-04-22T18:39:46.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2026e-05 2024-04-22T18:39:46.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102687+0000", 2024-04-22T18:39:46.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8339999999999999e-06 2024-04-22T18:39:46.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405644+0000", 2024-04-22T18:39:46.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30295623199999999 2024-04-22T18:39:46.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405743+0000", 2024-04-22T18:39:46.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.98e-05 2024-04-22T18:39:46.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405868+0000", 2024-04-22T18:39:46.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012457899999999999 2024-04-22T18:39:46.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407047+0000", 2024-04-22T18:39:46.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011791449999999999 2024-04-22T18:39:46.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408088+0000", 2024-04-22T18:39:46.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010411680000000001 2024-04-22T18:39:46.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408135+0000", 2024-04-22T18:39:46.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6255999999999999e-05 2024-04-22T18:39:46.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408178+0000", 2024-04-22T18:39:46.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2877000000000003e-05 2024-04-22T18:39:46.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.681 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.681 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:46.681 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:46.681 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 63.545611225999998, 2024-04-22T18:39:46.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30560089000000001, 2024-04-22T18:39:46.682 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.682 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.682 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.682 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 88 2024-04-22T18:39:46.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.682 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:46.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:46.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102693+0000", 2024-04-22T18:39:46.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:46.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102698+0000", 2024-04-22T18:39:46.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9699999999999998e-06 2024-04-22T18:39:46.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102708+0000", 2024-04-22T18:39:46.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.006e-05 2024-04-22T18:39:46.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102715+0000", 2024-04-22T18:39:46.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6599999999999995e-06 2024-04-22T18:39:46.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406637+0000", 2024-04-22T18:39:46.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392130099999998 2024-04-22T18:39:46.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406696+0000", 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9066e-05 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406784+0000", 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8194000000000003e-05 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407447+0000", 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066295700000000004 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408237+0000", 2024-04-22T18:39:46.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079032600000000005 2024-04-22T18:39:46.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408270+0000", 2024-04-22T18:39:46.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3157000000000001e-05 2024-04-22T18:39:46.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408295+0000", 2024-04-22T18:39:46.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4151999999999998e-05 2024-04-22T18:39:46.686 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.686 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.686 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.687 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:46.687 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:46.687 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.794379102000001, 2024-04-22T18:39:46.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30533819600000001, 2024-04-22T18:39:46.687 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.687 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.687 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.687 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.687 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.687 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 106 2024-04-22T18:39:46.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.687 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:46.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:46.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853925+0000", 2024-04-22T18:39:46.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853958+0000", 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2536000000000002e-05 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853964+0000", 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6239999999999999e-06 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853967+0000", 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9700000000000001e-06 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855144+0000", 2024-04-22T18:39:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001176774 2024-04-22T18:39:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855225+0000", 2024-04-22T18:39:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0697000000000003e-05 2024-04-22T18:39:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855353+0000", 2024-04-22T18:39:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012788 2024-04-22T18:39:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058003+0000", 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202649939 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159184+0000", 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101181058 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159238+0000", 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3967e-05 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159264+0000", 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6237999999999998e-05 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.794332251999997, 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30544294500000002, 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.692 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 107 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853972+0000", 2024-04-22T18:39:46.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853975+0000", 2024-04-22T18:39:46.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0939999999999999e-06 2024-04-22T18:39:46.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853978+0000", 2024-04-22T18:39:46.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:39:46.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853981+0000", 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.858e-06 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855831+0000", 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018503689999999999 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855932+0000", 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010071 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856029+0000", 2024-04-22T18:39:46.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6751999999999998e-05 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058013+0000", 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20198412600000001 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159326+0000", 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101312969 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159376+0000", 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9968e-05 2024-04-22T18:39:46.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159416+0000", 2024-04-22T18:39:46.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9641999999999998e-05 2024-04-22T18:39:46.697 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.697 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.697 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.697 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:46.697 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:46.697 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.794243856000001, 2024-04-22T18:39:46.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30548257499999998, 2024-04-22T18:39:46.697 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.697 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.698 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.698 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.698 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.698 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 108 2024-04-22T18:39:46.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.698 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:46.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:46.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:46.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854064+0000", 2024-04-22T18:39:46.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5820000000000002e-06 2024-04-22T18:39:46.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854080+0000", 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5379999999999998e-05 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854084+0000", 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.963e-06 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856413+0000", 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023298070000000001 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856494+0000", 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0863000000000002e-05 2024-04-22T18:39:46.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856624+0000", 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129799 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058947+0000", 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20232328899999999 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159468+0000", 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100520597 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159508+0000", 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9830000000000003e-05 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159544+0000", 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5944e-05 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:46.702 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.794199364000001, 2024-04-22T18:39:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30556502499999999, 2024-04-22T18:39:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:39:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854105+0000", 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854175+0000", 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0178000000000005e-05 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854191+0000", 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5965999999999999e-05 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854196+0000", 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.42e-06 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956526+0000", 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10233094299999999 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956634+0000", 2024-04-22T18:39:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107398 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956730+0000", 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6217000000000004e-05 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058955+0000", 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102224749 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159596+0000", 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100641297 2024-04-22T18:39:46.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159637+0000", 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874000000000002e-05 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159671+0000", 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3603999999999999e-05 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:46.707 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.794105516999998, 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30683048800000001, 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:46.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:46.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854202+0000", 2024-04-22T18:39:46.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:39:46.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854206+0000", 2024-04-22T18:39:46.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5870000000000001e-06 2024-04-22T18:39:46.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854208+0000", 2024-04-22T18:39:46.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7199999999999998e-06 2024-04-22T18:39:46.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057285+0000", 2024-04-22T18:39:46.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20307620800000001 2024-04-22T18:39:46.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057414+0000", 2024-04-22T18:39:46.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012896899999999999 2024-04-22T18:39:46.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057588+0000", 2024-04-22T18:39:46.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000174802 2024-04-22T18:39:46.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159429+0000", 2024-04-22T18:39:46.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101840783 2024-04-22T18:39:46.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160947+0000", 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001518152 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160992+0000", 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4725000000000003e-05 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161030+0000", 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7789e-05 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.712 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.713 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.713 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:46.713 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:46.713 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.794094336000001, 2024-04-22T18:39:46.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:39:46.713 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.713 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.713 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.713 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.713 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.713 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:39:46.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.713 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:39:46.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:39:46.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:39:46.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:39:46.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:39:46.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:39:46.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:39:46.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:39:46.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:39:46.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:39:46.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:39:46.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:39:46.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:39:46.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:39:46.719 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.719 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.719 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.719 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:46.719 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:46.719 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.794078677999998, 2024-04-22T18:39:46.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:39:46.719 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.719 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.719 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.720 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.720 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.720 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:39:46.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.720 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:46.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:46.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:46.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:39:46.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:39:46.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:39:46.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:39:46.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:39:46.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:39:46.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:39:46.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:39:46.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:39:46.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:39:46.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:39:46.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:39:46.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:39:46.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:39:46.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.794027516, 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:39:46.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:39:46.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:39:46.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:39:46.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:39:46.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:39:46.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:39:46.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:39:46.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:39:46.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:39:46.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:39:46.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:39:46.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:39:46.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:39:46.730 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.730 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.730 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.730 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:46.730 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:46.730 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.794012291999998, 2024-04-22T18:39:46.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:39:46.730 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.730 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.730 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.730 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.730 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.730 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:39:46.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.731 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:46.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:46.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:39:46.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:39:46.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:39:46.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:39:46.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:39:46.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:39:46.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:39:46.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:39:46.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:39:46.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:39:46.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:39:46.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:39:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:39:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:39:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:39:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.793952715000003, 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:39:46.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:39:46.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:39:46.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:39:46.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:39:46.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:39:46.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:39:46.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:39:46.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:39:46.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:39:46.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:39:46.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.740 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.793870398999999, 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:39:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:39:46.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:39:46.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:39:46.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:39:46.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:39:46.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:39:46.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:39:46.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:39:46.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:39:46.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:39:46.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:39:46.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:39:46.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:39:46.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:39:46.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:39:46.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:39:46.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:39:46.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:39:46.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:39:46.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:39:46.746 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.746 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.746 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.746 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:46.746 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:46.746 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.793860242999997, 2024-04-22T18:39:46.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:39:46.746 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.747 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.747 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.747 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:39:46.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.747 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:46.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:39:46.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:39:46.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:39:46.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:39:46.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:39:46.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:39:46.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:39:46.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.751 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:161 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 58.193365952000001, 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40239169000000002, 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 161 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454938+0000", 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454982+0000", 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3470999999999998e-05 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454988+0000", 2024-04-22T18:39:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7640000000000002e-06 2024-04-22T18:39:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555054+0000", 2024-04-22T18:39:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006607300000001 2024-04-22T18:39:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555607+0000", 2024-04-22T18:39:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055319500000000003 2024-04-22T18:39:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555689+0000", 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1817999999999995e-05 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555804+0000", 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115346 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556554+0000", 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000750102 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:46.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857265+0000", 2024-04-22T18:39:46.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30071122700000003 2024-04-22T18:39:46.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:46.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857302+0000", 2024-04-22T18:39:46.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7069e-05 2024-04-22T18:39:46.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:46.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:46.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:46.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857331+0000", 2024-04-22T18:39:46.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8291999999999999e-05 2024-04-22T18:39:46.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.756 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:46.757 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:46.757 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:46.757 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:46.758 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:46.759 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:46.701+0000 7f7f44bb3640 -1 osd.4 59 handle_osd_map injecting map crc failure 2024-04-22T18:39:46.759 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:46.760 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:46.703+0000 7fa768d81640 -1 osd.5 59 handle_osd_map injecting map crc failure 2024-04-22T18:39:46.760 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:39:46.760 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:46.703+0000 7f3d00e8c640 -1 osd.6 59 handle_osd_map injecting map crc failure 2024-04-22T18:39:46.760 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:46.761 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:46.759+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:46.761 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:46.761 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:46.761 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:46.762 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:46.762 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:46.767 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:46.767 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:46.784 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:46.784 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:46.801 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:46.801 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:46.819 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:46.819 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:46.870 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:46.871+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:46.873 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:46.873 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:46.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:46.959 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:46.961+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:47.007 INFO:tasks.rados.rados.0.smithi089.stdout:1087: got (0) Success 2024-04-22T18:39:47.007 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 0 (ObjNum 124 snap 0 seq_num 124) dirty exists 2024-04-22T18:39:47.015 INFO:tasks.rados.rados.0.smithi089.stdout:1089: expect (ObjNum 160 snap 0 seq_num 160) 2024-04-22T18:39:47.028 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:47.028 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:47.037 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:47.060 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:47.061+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:47.142 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:47.142 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:47.149 INFO:tasks.rados.rados.0.smithi089.stdout:1090: expect (ObjNum 173 snap 0 seq_num 173) 2024-04-22T18:39:47.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:47.160 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:47.161+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:47.253 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:47.253 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:47.261 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:47.262+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:47.262 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:47.312 INFO:tasks.rados.rados.0.smithi089.stdout:1082: finishing write tid 1 to smithi08937929-28 2024-04-22T18:39:47.312 INFO:tasks.rados.rados.0.smithi089.stdout:1093: finishing write tid 1 to smithi08937929-39 2024-04-22T18:39:47.312 INFO:tasks.rados.rados.0.smithi089.stdout:1082: finishing write tid 2 to smithi08937929-28 2024-04-22T18:39:47.312 INFO:tasks.rados.rados.0.smithi089.stdout:1087: done (9 left) 2024-04-22T18:39:47.312 INFO:tasks.rados.rados.0.smithi089.stdout:1089: done (8 left) 2024-04-22T18:39:47.313 INFO:tasks.rados.rados.0.smithi089.stdout:1090: done (7 left) 2024-04-22T18:39:47.313 INFO:tasks.rados.rados.0.smithi089.stdout:1092: done (6 left) 2024-04-22T18:39:47.313 INFO:tasks.rados.rados.0.smithi089.stdout:1094: done (5 left) 2024-04-22T18:39:47.313 INFO:tasks.rados.rados.0.smithi089.stdout:1095: done (4 left) 2024-04-22T18:39:47.313 INFO:tasks.rados.rados.0.smithi089.stdout:1097: done (3 left) 2024-04-22T18:39:47.313 INFO:tasks.rados.rados.0.smithi089.stdout:1098: set_chunk oid 30 target oid 2024-04-22T18:39:47.362 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:47.363+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:47.363 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:47.364 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:47.372 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:47.463 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:47.463+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:47.474 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:47.474 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:47.483 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:47.519 INFO:tasks.rados.rados.0.smithi089.stdout:1098: set_chunk oid 30 offset: 368640 length: 603648 target oid 6b4a7eb3172e0828da79015a68c76620d66536f110f031992150494399736e0c offset: 0 snap -1 2024-04-22T18:39:47.519 INFO:tasks.rados.rados.0.smithi089.stdout:1099: write oid 20 current snap is 0 2024-04-22T18:39:47.519 INFO:tasks.rados.rados.0.smithi089.stdout:1099: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-04-22T18:39:47.524 INFO:tasks.rados.rados.0.smithi089.stdout:1099: writing smithi08937929-20 from 410137 to 891521 tid 1 2024-04-22T18:39:47.531 INFO:tasks.rados.rados.0.smithi089.stdout:1099: writing smithi08937929-20 from 1410296 to 2171685 tid 2 2024-04-22T18:39:47.537 INFO:tasks.rados.rados.0.smithi089.stdout:1099: writing smithi08937929-20 from 2728968 to 3210137 tid 3 2024-04-22T18:39:47.537 INFO:tasks.rados.rados.0.smithi089.stdout:1100: write oid 17 current snap is 0 2024-04-22T18:39:47.537 INFO:tasks.rados.rados.0.smithi089.stdout:1082: finishing write tid 3 to smithi08937929-28 2024-04-22T18:39:47.537 INFO:tasks.rados.rados.0.smithi089.stdout:1100: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2024-04-22T18:39:47.542 INFO:tasks.rados.rados.0.smithi089.stdout:1100: writing smithi08937929-17 from 426944 to 983577 tid 1 2024-04-22T18:39:47.548 INFO:tasks.rados.rados.0.smithi089.stdout:1100: writing smithi08937929-17 from 1468778 to 2090178 tid 2 2024-04-22T18:39:47.555 INFO:tasks.rados.rados.0.smithi089.stdout:1100: writing smithi08937929-17 from 2872744 to 3226944 tid 3 2024-04-22T18:39:47.555 INFO:tasks.rados.rados.0.smithi089.stdout:1101: read oid 10 snap -1 2024-04-22T18:39:47.555 INFO:tasks.rados.rados.0.smithi089.stdout:1101: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:39:47.555 INFO:tasks.rados.rados.0.smithi089.stdout:1102: delete oid 40 current snap is 0 2024-04-22T18:39:47.555 INFO:tasks.rados.rados.0.smithi089.stdout:1082: finishing write tid 4 to smithi08937929-28 2024-04-22T18:39:47.555 INFO:tasks.rados.rados.0.smithi089.stdout:1082: oid 28 updating version 0 to 104 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:1082: oid 28 version 104 is already newer than 103 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:1082: oid 28 version 104 is already newer than 102 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 104 (ObjNum 184 snap 0 seq_num 184) dirty exists 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:1082: left oid 28 (ObjNum 184 snap 0 seq_num 184) 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:1096: finishing write tid 1 to smithi08937929-50 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:1096: finishing write tid 2 to smithi08937929-50 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:1096: finishing write tid 3 to smithi08937929-50 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:1096: finishing write tid 4 to smithi08937929-50 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:1096: oid 50 updating version 0 to 107 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:1096: oid 50 version 107 is already newer than 106 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:1096: oid 50 version 107 is already newer than 105 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 107 (ObjNum 189 snap 0 seq_num 189) dirty exists 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:1096: left oid 50 (ObjNum 189 snap 0 seq_num 189) 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:1093: finishing write tid 2 to smithi08937929-39 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:1093: finishing write tid 3 to smithi08937929-39 2024-04-22T18:39:47.556 INFO:tasks.rados.rados.0.smithi089.stdout:1093: finishing write tid 4 to smithi08937929-39 2024-04-22T18:39:47.557 INFO:tasks.rados.rados.0.smithi089.stdout:1093: oid 39 updating version 0 to 81 2024-04-22T18:39:47.557 INFO:tasks.rados.rados.0.smithi089.stdout:1093: oid 39 version 81 is already newer than 80 2024-04-22T18:39:47.557 INFO:tasks.rados.rados.0.smithi089.stdout:1093: oid 39 updating version 81 to 82 2024-04-22T18:39:47.557 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 82 (ObjNum 188 snap 0 seq_num 188) dirty exists 2024-04-22T18:39:47.557 INFO:tasks.rados.rados.0.smithi089.stdout:1093: left oid 39 (ObjNum 188 snap 0 seq_num 188) 2024-04-22T18:39:47.557 INFO:tasks.rados.rados.0.smithi089.stdout:1100: finishing write tid 1 to smithi08937929-17 2024-04-22T18:39:47.557 INFO:tasks.rados.rados.0.smithi089.stdout:1101: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:39:47.563 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:47.564+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:47.623 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:47.623 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:47.627 INFO:tasks.rados.rados.0.smithi089.stdout:1098: finishing set_chunk to oid 30 2024-04-22T18:39:47.627 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 31 (ObjNum 185 snap 0 seq_num 185) dirty exists 2024-04-22T18:39:47.633 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:47.657 INFO:tasks.rados.rados.0.smithi089.stdout:1100: finishing write tid 2 to smithi08937929-17 2024-04-22T18:39:47.657 INFO:tasks.rados.rados.0.smithi089.stdout:1100: finishing write tid 3 to smithi08937929-17 2024-04-22T18:39:47.665 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:47.666+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:47.724 INFO:tasks.rados.rados.0.smithi089.stdout:1082: done (7 left) 2024-04-22T18:39:47.724 INFO:tasks.rados.rados.0.smithi089.stdout:1093: done (6 left) 2024-04-22T18:39:47.724 INFO:tasks.rados.rados.0.smithi089.stdout:1096: done (5 left) 2024-04-22T18:39:47.724 INFO:tasks.rados.rados.0.smithi089.stdout:1098: done (4 left) 2024-04-22T18:39:47.724 INFO:tasks.rados.rados.0.smithi089.stdout:1101: done (3 left) 2024-04-22T18:39:47.724 INFO:tasks.rados.rados.0.smithi089.stdout:1102: done (2 left) 2024-04-22T18:39:47.724 INFO:tasks.rados.rados.0.smithi089.stdout:1103: read oid 3 snap -1 2024-04-22T18:39:47.724 INFO:tasks.rados.rados.0.smithi089.stdout:1103: expect (ObjNum 178 snap 0 seq_num 178) 2024-04-22T18:39:47.725 INFO:tasks.rados.rados.0.smithi089.stdout:1104: read oid 46 snap -1 2024-04-22T18:39:47.725 INFO:tasks.rados.rados.0.smithi089.stdout:1104: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-22T18:39:47.725 INFO:tasks.rados.rados.0.smithi089.stdout:1105: read oid 4 snap -1 2024-04-22T18:39:47.725 INFO:tasks.rados.rados.0.smithi089.stdout:1105: expect (ObjNum 120 snap 0 seq_num 120) 2024-04-22T18:39:47.725 INFO:tasks.rados.rados.0.smithi089.stdout:1106: read oid 10 snap -1 2024-04-22T18:39:47.725 INFO:tasks.rados.rados.0.smithi089.stdout:1106: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:39:47.725 INFO:tasks.rados.rados.0.smithi089.stdout:1107: read oid 29 snap -1 2024-04-22T18:39:47.725 INFO:tasks.rados.rados.0.smithi089.stdout:1107: expect (ObjNum 149 snap 0 seq_num 149) 2024-04-22T18:39:47.725 INFO:tasks.rados.rados.0.smithi089.stdout:1108: read oid 16 snap -1 2024-04-22T18:39:47.725 INFO:tasks.rados.rados.0.smithi089.stdout:1108: expect deleted 2024-04-22T18:39:47.725 INFO:tasks.rados.rados.0.smithi089.stdout:1109: write oid 37 current snap is 0 2024-04-22T18:39:47.725 INFO:tasks.rados.rados.0.smithi089.stdout:1109: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-04-22T18:39:47.731 INFO:tasks.rados.rados.0.smithi089.stdout:1109: writing smithi08937929-37 from 460558 to 1167689 tid 1 2024-04-22T18:39:47.738 INFO:tasks.rados.rados.0.smithi089.stdout:1109: writing smithi08937929-37 from 1869389 to 2378105 tid 2 2024-04-22T18:39:47.743 INFO:tasks.rados.rados.0.smithi089.stdout:1109: writing smithi08937929-37 from 3094884 to 3260558 tid 3 2024-04-22T18:39:47.743 INFO:tasks.rados.rados.0.smithi089.stdout:1110: read oid 49 snap -1 2024-04-22T18:39:47.744 INFO:tasks.rados.rados.0.smithi089.stdout:1110: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-22T18:39:47.744 INFO:tasks.rados.rados.0.smithi089.stdout:1111: set_chunk oid 13 target oid 2024-04-22T18:39:47.744 INFO:tasks.rados.rados.0.smithi089.stdout:1111: set_chunk oid 13 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:39:47.744 INFO:tasks.rados.rados.0.smithi089.stdout:1112: read oid 9 snap -1 2024-04-22T18:39:47.744 INFO:tasks.rados.rados.0.smithi089.stdout:1112: expect (ObjNum 153 snap 0 seq_num 153) 2024-04-22T18:39:47.744 INFO:tasks.rados.rados.0.smithi089.stdout:1113: write (excl) oid 19 current snap is 0 2024-04-22T18:39:47.744 INFO:tasks.rados.rados.0.smithi089.stdout:1113: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-04-22T18:39:47.750 INFO:tasks.rados.rados.0.smithi089.stdout:1113: writing smithi08937929-19 from 477365 to 1259745 tid 1 2024-04-22T18:39:47.757 INFO:tasks.rados.rados.0.smithi089.stdout:1113: writing smithi08937929-19 from 1927871 to 2696598 tid 2 2024-04-22T18:39:47.761 INFO:tasks.rados.rados.0.smithi089.stdout:1113: writing smithi08937929-19 from 3122307 to 3277365 tid 3 2024-04-22T18:39:47.761 INFO:tasks.rados.rados.0.smithi089.stdout:1114: write (excl) oid 11 current snap is 0 2024-04-22T18:39:47.761 INFO:tasks.rados.rados.0.smithi089.stdout:1114: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-04-22T18:39:47.762 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:47.762 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:47.766 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:47.766+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:47.766 INFO:tasks.rados.rados.0.smithi089.stdout:1114: writing smithi08937929-11 from 494172 to 951801 tid 1 2024-04-22T18:39:47.772 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:47.772 INFO:tasks.rados.rados.0.smithi089.stdout:1114: writing smithi08937929-11 from 1586353 to 2098738 tid 2 2024-04-22T18:39:47.779 INFO:tasks.rados.rados.0.smithi089.stdout:1114: writing smithi08937929-11 from 2749730 to 3294172 tid 3 2024-04-22T18:39:47.779 INFO:tasks.rados.rados.0.smithi089.stdout:1115: write oid 42 current snap is 0 2024-04-22T18:39:47.780 INFO:tasks.rados.rados.0.smithi089.stdout:1105: expect (ObjNum 120 snap 0 seq_num 120) 2024-04-22T18:39:47.867 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:47.868+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:47.869 INFO:tasks.rados.rados.0.smithi089.stdout:1099: finishing write tid 1 to smithi08937929-20 2024-04-22T18:39:47.870 INFO:tasks.rados.rados.0.smithi089.stdout:1099: finishing write tid 2 to smithi08937929-20 2024-04-22T18:39:47.870 INFO:tasks.rados.rados.0.smithi089.stdout:1115: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-04-22T18:39:47.875 INFO:tasks.rados.rados.0.smithi089.stdout:1115: writing smithi08937929-42 from 510979 to 1043857 tid 1 2024-04-22T18:39:47.877 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:47.877 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:47.882 INFO:tasks.rados.rados.0.smithi089.stdout:1115: writing smithi08937929-42 from 1528482 to 2300878 tid 2 2024-04-22T18:39:47.886 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:47.888 INFO:tasks.rados.rados.0.smithi089.stdout:1115: writing smithi08937929-42 from 3060800 to 3310979 tid 3 2024-04-22T18:39:47.888 INFO:tasks.rados.rados.0.smithi089.stdout:1105: done (14 left) 2024-04-22T18:39:47.888 INFO:tasks.rados.rados.0.smithi089.stdout:1116: delete oid 48 current snap is 0 2024-04-22T18:39:47.889 INFO:tasks.rados.rados.0.smithi089.stdout:1111: finishing set_chunk to oid 13 2024-04-22T18:39:47.889 INFO:tasks.rados.rados.0.smithi089.stdout:1111: got expected ENOENT (src dne) 2024-04-22T18:39:47.889 INFO:tasks.rados.rados.0.smithi089.stdout:1103: expect (ObjNum 178 snap 0 seq_num 178) 2024-04-22T18:39:47.968 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:47.969+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:47.993 INFO:tasks.rados.rados.0.smithi089.stdout:1104: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-22T18:39:47.996 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:47.996 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:48.005 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:48.069 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:48.070+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:48.082 INFO:tasks.rados.rados.0.smithi089.stdout:1099: finishing write tid 3 to smithi08937929-20 2024-04-22T18:39:48.082 INFO:tasks.rados.rados.0.smithi089.stdout:1113: finishing write tid 1 to smithi08937929-19 2024-04-22T18:39:48.082 INFO:tasks.rados.rados.0.smithi089.stdout:1107: expect (ObjNum 149 snap 0 seq_num 149) 2024-04-22T18:39:48.109 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:48.109 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:48.118 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:48.170 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:48.171+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:48.221 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:48.221 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:48.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:48.232 INFO:tasks.rados.rados.0.smithi089.stdout:1100: finishing write tid 4 to smithi08937929-17 2024-04-22T18:39:48.232 INFO:tasks.rados.rados.0.smithi089.stdout:1100: finishing write tid 5 to smithi08937929-17 2024-04-22T18:39:48.232 INFO:tasks.rados.rados.0.smithi089.stdout:1100: oid 17 updating version 0 to 64 2024-04-22T18:39:48.232 INFO:tasks.rados.rados.0.smithi089.stdout:1100: oid 17 updating version 64 to 66 2024-04-22T18:39:48.232 INFO:tasks.rados.rados.0.smithi089.stdout:1100: oid 17 version 66 is already newer than 65 2024-04-22T18:39:48.233 INFO:tasks.rados.rados.0.smithi089.stdout:1100: oid 17 updating version 66 to 67 2024-04-22T18:39:48.233 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 67 (ObjNum 192 snap 0 seq_num 192) dirty exists 2024-04-22T18:39:48.233 INFO:tasks.rados.rados.0.smithi089.stdout:1100: left oid 17 (ObjNum 192 snap 0 seq_num 192) 2024-04-22T18:39:48.233 INFO:tasks.rados.rados.0.smithi089.stdout:1099: finishing write tid 4 to smithi08937929-20 2024-04-22T18:39:48.233 INFO:tasks.rados.rados.0.smithi089.stdout:1099: finishing write tid 5 to smithi08937929-20 2024-04-22T18:39:48.233 INFO:tasks.rados.rados.0.smithi089.stdout:1099: oid 20 updating version 0 to 8 2024-04-22T18:39:48.233 INFO:tasks.rados.rados.0.smithi089.stdout:1099: oid 20 version 8 is already newer than 6 2024-04-22T18:39:48.233 INFO:tasks.rados.rados.0.smithi089.stdout:1099: oid 20 version 8 is already newer than 7 2024-04-22T18:39:48.233 INFO:tasks.rados.rados.0.smithi089.stdout:1099: oid 20 updating version 8 to 9 2024-04-22T18:39:48.233 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 9 (ObjNum 191 snap 0 seq_num 191) dirty exists 2024-04-22T18:39:48.233 INFO:tasks.rados.rados.0.smithi089.stdout:1099: left oid 20 (ObjNum 191 snap 0 seq_num 191) 2024-04-22T18:39:48.233 INFO:tasks.rados.rados.0.smithi089.stdout:1113: finishing write tid 2 to smithi08937929-19 2024-04-22T18:39:48.233 INFO:tasks.rados.rados.0.smithi089.stdout:1113: finishing write tid 3 to smithi08937929-19 2024-04-22T18:39:48.233 INFO:tasks.rados.rados.0.smithi089.stdout:1113: finishing write tid 4 to smithi08937929-19 2024-04-22T18:39:48.233 INFO:tasks.rados.rados.0.smithi089.stdout:1110: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-22T18:39:48.271 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:48.272+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:48.321 INFO:tasks.rados.rados.0.smithi089.stdout:1112: expect (ObjNum 153 snap 0 seq_num 153) 2024-04-22T18:39:48.332 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:48.333 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:48.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:48.385 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:48.386+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:48.442 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:48.442 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:48.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:48.457 INFO:tasks.rados.rados.0.smithi089.stdout:1106: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:39:48.473 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:48.473+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:48.525 INFO:tasks.rados.rados.0.smithi089.stdout:1114: finishing write tid 1 to smithi08937929-11 2024-04-22T18:39:48.526 INFO:tasks.rados.rados.0.smithi089.stdout:1114: finishing write tid 2 to smithi08937929-11 2024-04-22T18:39:48.526 INFO:tasks.rados.rados.0.smithi089.stdout:1114: finishing write tid 3 to smithi08937929-11 2024-04-22T18:39:48.526 INFO:tasks.rados.rados.0.smithi089.stdout:1114: finishing write tid 4 to smithi08937929-11 2024-04-22T18:39:48.526 INFO:tasks.rados.rados.0.smithi089.stdout:1109: finishing write tid 1 to smithi08937929-37 2024-04-22T18:39:48.526 INFO:tasks.rados.rados.0.smithi089.stdout:1109: finishing write tid 2 to smithi08937929-37 2024-04-22T18:39:48.526 INFO:tasks.rados.rados.0.smithi089.stdout:1113: finishing write tid 5 to smithi08937929-19 2024-04-22T18:39:48.526 INFO:tasks.rados.rados.0.smithi089.stdout:1113: oid 19 updating version 0 to 106 2024-04-22T18:39:48.526 INFO:tasks.rados.rados.0.smithi089.stdout:1113: oid 19 updating version 106 to 107 2024-04-22T18:39:48.526 INFO:tasks.rados.rados.0.smithi089.stdout:1113: oid 19 updating version 107 to 108 2024-04-22T18:39:48.526 INFO:tasks.rados.rados.0.smithi089.stdout:1113: oid 19 version 108 is already newer than 105 2024-04-22T18:39:48.526 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 108 (ObjNum 195 snap 0 seq_num 195) dirty exists 2024-04-22T18:39:48.526 INFO:tasks.rados.rados.0.smithi089.stdout:1113: left oid 19 (ObjNum 195 snap 0 seq_num 195) 2024-04-22T18:39:48.526 INFO:tasks.rados.rados.0.smithi089.stdout:1114: finishing write tid 5 to smithi08937929-11 2024-04-22T18:39:48.527 INFO:tasks.rados.rados.0.smithi089.stdout:1114: oid 11 updating version 0 to 36 2024-04-22T18:39:48.527 INFO:tasks.rados.rados.0.smithi089.stdout:1114: oid 11 version 36 is already newer than 34 2024-04-22T18:39:48.527 INFO:tasks.rados.rados.0.smithi089.stdout:1114: oid 11 version 36 is already newer than 35 2024-04-22T18:39:48.527 INFO:tasks.rados.rados.0.smithi089.stdout:1114: oid 11 updating version 36 to 37 2024-04-22T18:39:48.527 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 37 (ObjNum 196 snap 0 seq_num 196) dirty exists 2024-04-22T18:39:48.527 INFO:tasks.rados.rados.0.smithi089.stdout:1114: left oid 11 (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:39:48.527 INFO:tasks.rados.rados.0.smithi089.stdout:1115: finishing write tid 1 to smithi08937929-42 2024-04-22T18:39:48.527 INFO:tasks.rados.rados.0.smithi089.stdout:1115: finishing write tid 2 to smithi08937929-42 2024-04-22T18:39:48.527 INFO:tasks.rados.rados.0.smithi089.stdout:1099: done (14 left) 2024-04-22T18:39:48.527 INFO:tasks.rados.rados.0.smithi089.stdout:1100: done (13 left) 2024-04-22T18:39:48.527 INFO:tasks.rados.rados.0.smithi089.stdout:1103: done (12 left) 2024-04-22T18:39:48.527 INFO:tasks.rados.rados.0.smithi089.stdout:1104: done (11 left) 2024-04-22T18:39:48.527 INFO:tasks.rados.rados.0.smithi089.stdout:1106: done (10 left) 2024-04-22T18:39:48.527 INFO:tasks.rados.rados.0.smithi089.stdout:1107: done (9 left) 2024-04-22T18:39:48.528 INFO:tasks.rados.rados.0.smithi089.stdout:1108: done (8 left) 2024-04-22T18:39:48.528 INFO:tasks.rados.rados.0.smithi089.stdout:1110: done (7 left) 2024-04-22T18:39:48.528 INFO:tasks.rados.rados.0.smithi089.stdout:1111: done (6 left) 2024-04-22T18:39:48.528 INFO:tasks.rados.rados.0.smithi089.stdout:1112: done (5 left) 2024-04-22T18:39:48.528 INFO:tasks.rados.rados.0.smithi089.stdout:1113: done (4 left) 2024-04-22T18:39:48.528 INFO:tasks.rados.rados.0.smithi089.stdout:1114: done (3 left) 2024-04-22T18:39:48.528 INFO:tasks.rados.rados.0.smithi089.stdout:1116: done (2 left) 2024-04-22T18:39:48.528 INFO:tasks.rados.rados.0.smithi089.stdout:1117: read oid 12 snap -1 2024-04-22T18:39:48.528 INFO:tasks.rados.rados.0.smithi089.stdout:1117: expect (ObjNum 141 snap 0 seq_num 141) 2024-04-22T18:39:48.528 INFO:tasks.rados.rados.0.smithi089.stdout:1118: read oid 4 snap -1 2024-04-22T18:39:48.528 INFO:tasks.rados.rados.0.smithi089.stdout:1118: expect (ObjNum 120 snap 0 seq_num 120) 2024-04-22T18:39:48.528 INFO:tasks.rados.rados.0.smithi089.stdout:1119: tier_evict oid 34 2024-04-22T18:39:48.528 INFO:tasks.rados.rados.0.smithi089.stdout:1119: tier_evict oid 34 snap -1 2024-04-22T18:39:48.528 INFO:tasks.rados.rados.0.smithi089.stdout:1115: finishing write tid 3 to smithi08937929-42 2024-04-22T18:39:48.529 INFO:tasks.rados.rados.0.smithi089.stdout:1115: finishing write tid 4 to smithi08937929-42 2024-04-22T18:39:48.529 INFO:tasks.rados.rados.0.smithi089.stdout:1115: finishing write tid 5 to smithi08937929-42 2024-04-22T18:39:48.529 INFO:tasks.rados.rados.0.smithi089.stdout:1115: oid 42 updating version 0 to 22 2024-04-22T18:39:48.529 INFO:tasks.rados.rados.0.smithi089.stdout:1115: oid 42 updating version 22 to 23 2024-04-22T18:39:48.529 INFO:tasks.rados.rados.0.smithi089.stdout:1115: oid 42 version 23 is already newer than 20 2024-04-22T18:39:48.529 INFO:tasks.rados.rados.0.smithi089.stdout:1115: oid 42 version 23 is already newer than 21 2024-04-22T18:39:48.529 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 23 (ObjNum 197 snap 0 seq_num 197) dirty exists 2024-04-22T18:39:48.529 INFO:tasks.rados.rados.0.smithi089.stdout:1115: left oid 42 (ObjNum 197 snap 0 seq_num 197) 2024-04-22T18:39:48.529 INFO:tasks.rados.rados.0.smithi089.stdout:1109: finishing write tid 3 to smithi08937929-37 2024-04-22T18:39:48.529 INFO:tasks.rados.rados.0.smithi089.stdout:1109: finishing write tid 4 to smithi08937929-37 2024-04-22T18:39:48.529 INFO:tasks.rados.rados.0.smithi089.stdout:1109: finishing write tid 5 to smithi08937929-37 2024-04-22T18:39:48.529 INFO:tasks.rados.rados.0.smithi089.stdout:1109: oid 37 updating version 0 to 89 2024-04-22T18:39:48.529 INFO:tasks.rados.rados.0.smithi089.stdout:1109: oid 37 version 89 is already newer than 86 2024-04-22T18:39:48.530 INFO:tasks.rados.rados.0.smithi089.stdout:1109: oid 37 version 89 is already newer than 88 2024-04-22T18:39:48.530 INFO:tasks.rados.rados.0.smithi089.stdout:1109: oid 37 version 89 is already newer than 87 2024-04-22T18:39:48.530 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 89 (ObjNum 194 snap 0 seq_num 194) dirty exists 2024-04-22T18:39:48.530 INFO:tasks.rados.rados.0.smithi089.stdout:1109: left oid 37 (ObjNum 194 snap 0 seq_num 194) 2024-04-22T18:39:48.530 INFO:tasks.rados.rados.0.smithi089.stdout:1109: done (4 left) 2024-04-22T18:39:48.530 INFO:tasks.rados.rados.0.smithi089.stdout:1115: done (3 left) 2024-04-22T18:39:48.530 INFO:tasks.rados.rados.0.smithi089.stdout:1120: write (excl) oid 6 current snap is 0 2024-04-22T18:39:48.530 INFO:tasks.rados.rados.0.smithi089.stdout:1120: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2024-04-22T18:39:48.533 INFO:tasks.rados.rados.0.smithi089.stdout:1120: writing smithi08937929-6 from 544593 to 1344322 tid 1 2024-04-22T18:39:48.539 INFO:tasks.rados.rados.0.smithi089.stdout:1120: writing smithi08937929-6 from 1761799 to 2537864 tid 2 2024-04-22T18:39:48.546 INFO:tasks.rados.rados.0.smithi089.stdout:1120: writing smithi08937929-6 from 2948352 to 3344593 tid 3 2024-04-22T18:39:48.546 INFO:tasks.rados.rados.0.smithi089.stdout:1121: tier_flush oid 39 2024-04-22T18:39:48.546 INFO:tasks.rados.rados.0.smithi089.stdout:1121: tier_flush oid 39 snap -1 2024-04-22T18:39:48.546 INFO:tasks.rados.rados.0.smithi089.stdout:1122: read oid 37 snap -1 2024-04-22T18:39:48.546 INFO:tasks.rados.rados.0.smithi089.stdout:1122: expect (ObjNum 194 snap 0 seq_num 194) 2024-04-22T18:39:48.546 INFO:tasks.rados.rados.0.smithi089.stdout:1123: read oid 15 snap -1 2024-04-22T18:39:48.546 INFO:tasks.rados.rados.0.smithi089.stdout:1123: expect deleted 2024-04-22T18:39:48.546 INFO:tasks.rados.rados.0.smithi089.stdout:1124: read oid 1 snap -1 2024-04-22T18:39:48.546 INFO:tasks.rados.rados.0.smithi089.stdout:1124: expect (ObjNum 176 snap 0 seq_num 176) 2024-04-22T18:39:48.546 INFO:tasks.rados.rados.0.smithi089.stdout:1125: read oid 16 snap -1 2024-04-22T18:39:48.546 INFO:tasks.rados.rados.0.smithi089.stdout:1125: expect deleted 2024-04-22T18:39:48.546 INFO:tasks.rados.rados.0.smithi089.stdout:1126: read oid 29 snap -1 2024-04-22T18:39:48.547 INFO:tasks.rados.rados.0.smithi089.stdout:1126: expect (ObjNum 149 snap 0 seq_num 149) 2024-04-22T18:39:48.547 INFO:tasks.rados.rados.0.smithi089.stdout:1127: read oid 45 snap -1 2024-04-22T18:39:48.547 INFO:tasks.rados.rados.0.smithi089.stdout:1127: expect (ObjNum 166 snap 0 seq_num 166) 2024-04-22T18:39:48.547 INFO:tasks.rados.rados.0.smithi089.stdout:1128: read oid 41 snap -1 2024-04-22T18:39:48.547 INFO:tasks.rados.rados.0.smithi089.stdout:1128: expect (ObjNum 136 snap 0 seq_num 136) 2024-04-22T18:39:48.547 INFO:tasks.rados.rados.0.smithi089.stdout:1129: read oid 5 snap -1 2024-04-22T18:39:48.547 INFO:tasks.rados.rados.0.smithi089.stdout:1129: expect deleted 2024-04-22T18:39:48.547 INFO:tasks.rados.rados.0.smithi089.stdout:1130: read oid 47 snap -1 2024-04-22T18:39:48.547 INFO:tasks.rados.rados.0.smithi089.stdout:1130: expect (ObjNum 147 snap 0 seq_num 147) 2024-04-22T18:39:48.547 INFO:tasks.rados.rados.0.smithi089.stdout:1131: read oid 23 snap -1 2024-04-22T18:39:48.547 INFO:tasks.rados.rados.0.smithi089.stdout:1131: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:39:48.547 INFO:tasks.rados.rados.0.smithi089.stdout:1132: write (excl) oid 3 current snap is 0 2024-04-22T18:39:48.547 INFO:tasks.rados.rados.0.smithi089.stdout:1117: expect (ObjNum 141 snap 0 seq_num 141) 2024-04-22T18:39:48.573 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:48.574+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:48.597 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:48.597 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:48.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:48.674 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:48.675+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:48.678 INFO:tasks.rados.rados.0.smithi089.stdout:1119: got (22) Invalid argument 2024-04-22T18:39:48.678 INFO:tasks.rados.rados.0.smithi089.stdout:1124: expect (ObjNum 176 snap 0 seq_num 176) 2024-04-22T18:39:48.737 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:48.737+0000 7f3093031640 -1 osd.1 60 handle_osd_map injecting map crc failure 2024-04-22T18:39:48.756 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:48.757 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:48.761 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:39:48.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:48.805 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:48.805+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:48.830 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:39:48.858 INFO:tasks.rados.rados.0.smithi089.stdout:1118: expect (ObjNum 120 snap 0 seq_num 120) 2024-04-22T18:39:48.867 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:48.867 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:48.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:48.878 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:48.877+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:48.956 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:48.956 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:48.959 INFO:tasks.rados.rados.0.smithi089.stdout:1120: finishing write tid 1 to smithi08937929-6 2024-04-22T18:39:48.959 INFO:tasks.rados.rados.0.smithi089.stdout:1132: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-04-22T18:39:48.965 INFO:tasks.rados.rados.0.smithi089.stdout:1132: writing smithi08937929-3 from 561400 to 1036378 tid 1 2024-04-22T18:39:48.972 INFO:tasks.rados.rados.0.smithi089.stdout:1132: writing smithi08937929-3 from 1820281 to 2340004 tid 2 2024-04-22T18:39:48.972 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:48.973 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:48.978 INFO:tasks.rados.rados.0.smithi089.stdout:1132: writing smithi08937929-3 from 2859422 to 3317001 tid 3 2024-04-22T18:39:48.978 INFO:tasks.rados.rados.0.smithi089.stdout:1117: done (15 left) 2024-04-22T18:39:48.978 INFO:tasks.rados.rados.0.smithi089.stdout:1118: done (14 left) 2024-04-22T18:39:48.978 INFO:tasks.rados.rados.0.smithi089.stdout:1119: done (13 left) 2024-04-22T18:39:48.978 INFO:tasks.rados.rados.0.smithi089.stdout:1124: done (12 left) 2024-04-22T18:39:48.978 INFO:tasks.rados.rados.0.smithi089.stdout:1133: write (excl) oid 20 current snap is 0 2024-04-22T18:39:48.978 INFO:tasks.rados.rados.0.smithi089.stdout:1128: expect (ObjNum 136 snap 0 seq_num 136) 2024-04-22T18:39:48.978 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:48.978+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:48.979 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:48.979 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:48.988 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:48.989 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:48.989 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:49.006 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:49.007 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:49.022 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:49.023 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:49.038 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:49.039 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:49.055 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:49.055 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:49.073 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:49.073 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:49.078 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:49.078+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:49.078 INFO:tasks.rados.rados.0.smithi089.stdout:1133: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2024-04-22T18:39:49.085 INFO:tasks.rados.rados.0.smithi089.stdout:1133: writing smithi08937929-20 from 578207 to 1128434 tid 1 2024-04-22T18:39:49.092 INFO:tasks.rados.rados.0.smithi089.stdout:1133: writing smithi08937929-20 from 1762410 to 2542144 tid 2 2024-04-22T18:39:49.093 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:49.093 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:49.097 INFO:tasks.rados.rados.0.smithi089.stdout:1133: writing smithi08937929-20 from 3286845 to 3378207 tid 3 2024-04-22T18:39:49.097 INFO:tasks.rados.rados.0.smithi089.stdout:1123: done (12 left) 2024-04-22T18:39:49.098 INFO:tasks.rados.rados.0.smithi089.stdout:1128: done (11 left) 2024-04-22T18:39:49.098 INFO:tasks.rados.rados.0.smithi089.stdout:1129: done (10 left) 2024-04-22T18:39:49.098 INFO:tasks.rados.rados.0.smithi089.stdout:1134: read oid 13 snap -1 2024-04-22T18:39:49.098 INFO:tasks.rados.rados.0.smithi089.stdout:1134: expect deleted 2024-04-22T18:39:49.098 INFO:tasks.rados.rados.0.smithi089.stdout:1135: read oid 7 snap -1 2024-04-22T18:39:49.098 INFO:tasks.rados.rados.0.smithi089.stdout:1135: expect (ObjNum 161 snap 0 seq_num 161) 2024-04-22T18:39:49.098 INFO:tasks.rados.rados.0.smithi089.stdout:1136: set_chunk oid 24 target oid 2024-04-22T18:39:49.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:49.178 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:49.179+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:49.205 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:49.205 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:49.205 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:49.205 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:49.205 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:49.206 INFO:tasks.rados.rados.0.smithi089.stdout:1136: set_chunk oid 24 offset: 619520 length: 277504 target oid 390d2c9fb6744dbdab527ec172a4441077a5b0e1ebff7ba21c57ce18f7b18023 offset: 0 snap -1 2024-04-22T18:39:49.207 INFO:tasks.rados.rados.0.smithi089.stdout:1137: read oid 35 snap -1 2024-04-22T18:39:49.207 INFO:tasks.rados.rados.0.smithi089.stdout:1137: expect (ObjNum 162 snap 0 seq_num 162) 2024-04-22T18:39:49.207 INFO:tasks.rados.rados.0.smithi089.stdout:1138: set_chunk oid 8 target oid 2024-04-22T18:39:49.207 INFO:tasks.rados.rados.0.smithi089.stdout:1138: set_chunk oid 8 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:39:49.207 INFO:tasks.rados.rados.0.smithi089.stdout:1139: set_chunk oid 46 target oid 2024-04-22T18:39:49.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:49.279 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:49.280+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:49.325 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:49.379 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:49.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:49.379 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:49.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.380 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:21 4.a 4:5aa97f94:::smithi08937929-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.380 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:49.380 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.166968945000001, 2024-04-22T18:39:49.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.21044934000000001, 2024-04-22T18:39:49.380 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.380 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.380 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.380 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.380 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.380 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 21 2024-04-22T18:39:49.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.380 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:49.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153186+0000", 2024-04-22T18:39:49.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.153183+0000", 2024-04-22T18:39:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154022+0000", 2024-04-22T18:39:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00083893300000000004 2024-04-22T18:39:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154032+0000", 2024-04-22T18:39:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0433e-05 2024-04-22T18:39:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154051+0000", 2024-04-22T18:39:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8227e-05 2024-04-22T18:39:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154103+0000", 2024-04-22T18:39:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2491999999999998e-05 2024-04-22T18:39:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.154111+0000", 2024-04-22T18:39:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3470000000000001e-06 2024-04-22T18:39:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257848+0000", 2024-04-22T18:39:49.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103737703 2024-04-22T18:39:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257981+0000", 2024-04-22T18:39:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013290399999999999 2024-04-22T18:39:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.258138+0000", 2024-04-22T18:39:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015701199999999999 2024-04-22T18:39:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.261944+0000", 2024-04-22T18:39:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0038054650000000001 2024-04-22T18:39:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363523+0000", 2024-04-22T18:39:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101579239 2024-04-22T18:39:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363577+0000", 2024-04-22T18:39:49.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4095000000000003e-05 2024-04-22T18:39:49.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363635+0000", 2024-04-22T18:39:49.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8362999999999997e-05 2024-04-22T18:39:49.387 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.387 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.387 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.388 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:22 4.a 4:5aa97f94:::smithi08937929-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.388 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:49.388 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.162894969, 2024-04-22T18:39:49.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30948398900000001, 2024-04-22T18:39:49.388 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.388 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.388 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.388 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.389 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.389 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 22 2024-04-22T18:39:49.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.389 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:49.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:39:49.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157259+0000", 2024-04-22T18:39:49.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:49.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157318+0000", 2024-04-22T18:39:49.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8906000000000001e-05 2024-04-22T18:39:49.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157327+0000", 2024-04-22T18:39:49.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:39:49.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157332+0000", 2024-04-22T18:39:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4689999999999996e-06 2024-04-22T18:39:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257479+0000", 2024-04-22T18:39:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014700999999999 2024-04-22T18:39:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257492+0000", 2024-04-22T18:39:49.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2574e-05 2024-04-22T18:39:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259582+0000", 2024-04-22T18:39:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020904019999999999 2024-04-22T18:39:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259668+0000", 2024-04-22T18:39:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5826000000000001e-05 2024-04-22T18:39:49.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:49.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259772+0000", 2024-04-22T18:39:49.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104048 2024-04-22T18:39:49.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363453+0000", 2024-04-22T18:39:49.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103680183 2024-04-22T18:39:49.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466692+0000", 2024-04-22T18:39:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10323948199999999 2024-04-22T18:39:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466720+0000", 2024-04-22T18:39:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7886e-05 2024-04-22T18:39:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466744+0000", 2024-04-22T18:39:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4049e-05 2024-04-22T18:39:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:23 4.a 4:5aa97f94:::smithi08937929-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.162817281999999, 2024-04-22T18:39:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39003820300000003, 2024-04-22T18:39:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 23 2024-04-22T18:39:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:39:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157337+0000", 2024-04-22T18:39:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157340+0000", 2024-04-22T18:39:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8609999999999998e-06 2024-04-22T18:39:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157343+0000", 2024-04-22T18:39:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9859999999999999e-06 2024-04-22T18:39:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157346+0000", 2024-04-22T18:39:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8119999999999999e-06 2024-04-22T18:39:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257563+0000", 2024-04-22T18:39:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100217497 2024-04-22T18:39:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257574+0000", 2024-04-22T18:39:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0777e-05 2024-04-22T18:39:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260112+0000", 2024-04-22T18:39:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002537681 2024-04-22T18:39:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260191+0000", 2024-04-22T18:39:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9320999999999995e-05 2024-04-22T18:39:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260333+0000", 2024-04-22T18:39:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014177699999999999 2024-04-22T18:39:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363792+0000", 2024-04-22T18:39:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103458862 2024-04-22T18:39:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547303+0000", 2024-04-22T18:39:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18351162100000001 2024-04-22T18:39:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547337+0000", 2024-04-22T18:39:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3640999999999999e-05 2024-04-22T18:39:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547376+0000", 2024-04-22T18:39:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8915000000000003e-05 2024-04-22T18:39:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:24 4.a 4:5aa97f94:::smithi08937929-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.162790850999997, 2024-04-22T18:39:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39017823499999998, 2024-04-22T18:39:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 24 2024-04-22T18:39:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:39:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157366+0000", 2024-04-22T18:39:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3120000000000001e-06 2024-04-22T18:39:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157370+0000", 2024-04-22T18:39:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1300000000000003e-06 2024-04-22T18:39:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157373+0000", 2024-04-22T18:39:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-04-22T18:39:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260553+0000", 2024-04-22T18:39:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103179996 2024-04-22T18:39:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260625+0000", 2024-04-22T18:39:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2236000000000004e-05 2024-04-22T18:39:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547414+0000", 2024-04-22T18:39:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.286789181 2024-04-22T18:39:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547471+0000", 2024-04-22T18:39:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6192e-05 2024-04-22T18:39:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547542+0000", 2024-04-22T18:39:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1622e-05 2024-04-22T18:39:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:49 4.a 4:5d2aee44:::smithi08937929-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.085992193000003, 2024-04-22T18:39:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31344442500000003, 2024-04-22T18:39:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 49 2024-04-22T18:39:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:39:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234158+0000", 2024-04-22T18:39:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999957 2024-04-22T18:39:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234901+0000", 2024-04-22T18:39:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074250800000000001 2024-04-22T18:39:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234914+0000", 2024-04-22T18:39:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3244999999999999e-05 2024-04-22T18:39:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234922+0000", 2024-04-22T18:39:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4590000000000001e-06 2024-04-22T18:39:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260665+0000", 2024-04-22T18:39:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025743423000000001 2024-04-22T18:39:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260779+0000", 2024-04-22T18:39:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114063 2024-04-22T18:39:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260913+0000", 2024-04-22T18:39:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013355499999999999 2024-04-22T18:39:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363798+0000", 2024-04-22T18:39:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102885003 2024-04-22T18:39:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547567+0000", 2024-04-22T18:39:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.183768971 2024-04-22T18:39:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547589+0000", 2024-04-22T18:39:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2507999999999999e-05 2024-04-22T18:39:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547607+0000", 2024-04-22T18:39:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8009999999999999e-05 2024-04-22T18:39:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:50 4.a 4:5d2aee44:::smithi08937929-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.080546013999999, 2024-04-22T18:39:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40856295399999998, 2024-04-22T18:39:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 50 2024-04-22T18:39:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:39:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239607+0000", 2024-04-22T18:39:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240309+0000", 2024-04-22T18:39:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070201599999999997 2024-04-22T18:39:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240320+0000", 2024-04-22T18:39:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1024e-05 2024-04-22T18:39:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240327+0000", 2024-04-22T18:39:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0049999999999998e-06 2024-04-22T18:39:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362065+0000", 2024-04-22T18:39:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.121737812 2024-04-22T18:39:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362121+0000", 2024-04-22T18:39:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5868000000000001e-05 2024-04-22T18:39:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362210+0000", 2024-04-22T18:39:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8533000000000001e-05 2024-04-22T18:39:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365322+0000", 2024-04-22T18:39:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031121220000000002 2024-04-22T18:39:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648054+0000", 2024-04-22T18:39:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.282732549 2024-04-22T18:39:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648105+0000", 2024-04-22T18:39:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1276999999999997e-05 2024-04-22T18:39:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648172+0000", 2024-04-22T18:39:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6398999999999999e-05 2024-04-22T18:39:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:51 4.a 4:5d2aee44:::smithi08937929-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.079823480000002, 2024-04-22T18:39:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40793057199999999, 2024-04-22T18:39:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 51 2024-04-22T18:39:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:39:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240336+0000", 2024-04-22T18:39:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:39:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240339+0000", 2024-04-22T18:39:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9009999999999998e-06 2024-04-22T18:39:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240342+0000", 2024-04-22T18:39:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:39:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363080+0000", 2024-04-22T18:39:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122738075 2024-04-22T18:39:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363164+0000", 2024-04-22T18:39:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3722999999999998e-05 2024-04-22T18:39:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363287+0000", 2024-04-22T18:39:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012301099999999999 2024-04-22T18:39:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466753+0000", 2024-04-22T18:39:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103466217 2024-04-22T18:39:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648205+0000", 2024-04-22T18:39:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.181451788 2024-04-22T18:39:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648231+0000", 2024-04-22T18:39:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6597999999999998e-05 2024-04-22T18:39:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648262+0000", 2024-04-22T18:39:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0871000000000003e-05 2024-04-22T18:39:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:52 4.a 4:5d2aee44:::smithi08937929-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.079811020999998, 2024-04-22T18:39:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40812731200000002, 2024-04-22T18:39:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 52 2024-04-22T18:39:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:39:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240345+0000", 2024-04-22T18:39:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4300000000000001e-06 2024-04-22T18:39:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240349+0000", 2024-04-22T18:39:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2570000000000001e-06 2024-04-22T18:39:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240358+0000", 2024-04-22T18:39:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-22T18:39:49.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363697+0000", 2024-04-22T18:39:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12333965299999999 2024-04-22T18:39:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363777+0000", 2024-04-22T18:39:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9883999999999996e-05 2024-04-22T18:39:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648296+0000", 2024-04-22T18:39:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451861299999998 2024-04-22T18:39:49.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648379+0000", 2024-04-22T18:39:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3302999999999998e-05 2024-04-22T18:39:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648471+0000", 2024-04-22T18:39:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2029999999999998e-05 2024-04-22T18:39:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.432 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.060507686000001, 2024-04-22T18:39:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:39:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:39:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.433 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:39:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:39:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:39:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:39:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:39:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:39:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:39:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:39:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:39:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:39:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:39:49.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:39:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:39:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:39:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:39:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:39:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:39:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:39:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:39:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:39:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:39:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:39:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.438 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.055725256000002, 2024-04-22T18:39:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:39:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:39:49.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:39:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:39:49.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:39:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:39:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:39:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:39:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:39:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:39:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:39:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:39:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:39:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:39:49.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:39:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:39:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:39:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:39:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:39:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:39:49.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:39:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:39:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:39:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:39:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.444 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.055631052999999, 2024-04-22T18:39:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:39:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:39:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:39:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:39:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:39:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:39:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:39:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:39:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:39:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:39:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:39:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:39:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:39:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:39:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:39:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:39:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:39:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:39:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:39:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:39:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:39:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:39:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:39:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:39:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.055610164999997, 2024-04-22T18:39:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:39:49.450 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.451 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:49.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.451 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:39:49.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.451 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:39:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:39:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:39:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:39:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:39:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:39:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:39:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:39:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:39:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:39:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:39:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:39:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:39:49.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:39:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:39:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:39:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:39:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:39:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.455 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 54.272602501999998, 2024-04-22T18:39:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:39:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:39:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:39:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:39:49.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:39:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:39:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:39:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:39:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:39:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:39:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:39:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:39:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:39:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:39:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:39:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:39:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:39:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:39:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:39:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:39:49.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:39:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:39:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:39:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:39:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 54.264771095999997, 2024-04-22T18:39:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:39:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:39:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:39:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:39:49.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:39:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:39:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:39:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:39:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:39:49.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:39:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:39:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:39:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:39:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:39:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:49.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:39:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:39:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:39:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:39:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:39:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:39:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:39:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:39:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:39:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:39:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:186 4.a 4:537c89f8:::smithi08937929-47:head [write 1992054~651681] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.917265561000001, 2024-04-22T18:39:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20325226800000001, 2024-04-22T18:39:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 186 2024-04-22T18:39:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402889+0000", 2024-04-22T18:39:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.402888+0000", 2024-04-22T18:39:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403454+0000", 2024-04-22T18:39:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056661000000000005 2024-04-22T18:39:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403465+0000", 2024-04-22T18:39:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0727000000000001e-05 2024-04-22T18:39:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403473+0000", 2024-04-22T18:39:49.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7039999999999994e-06 2024-04-22T18:39:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404592+0000", 2024-04-22T18:39:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001118827 2024-04-22T18:39:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404674+0000", 2024-04-22T18:39:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2793999999999994e-05 2024-04-22T18:39:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:49.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.404779+0000", 2024-04-22T18:39:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104465 2024-04-22T18:39:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.406080+0000", 2024-04-22T18:39:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013006789999999999 2024-04-22T18:39:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606037+0000", 2024-04-22T18:39:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19995755600000001 2024-04-22T18:39:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606098+0000", 2024-04-22T18:39:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1075999999999996e-05 2024-04-22T18:39:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606142+0000", 2024-04-22T18:39:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3411999999999997e-05 2024-04-22T18:39:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:187 4.a 4:537c89f8:::smithi08937929-47:head [setxattr _header (56) in=7b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:49.473 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.916675421000001, 2024-04-22T18:39:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203643623, 2024-04-22T18:39:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 187 2024-04-22T18:39:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403479+0000", 2024-04-22T18:39:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403501+0000", 2024-04-22T18:39:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2146000000000001e-05 2024-04-22T18:39:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403506+0000", 2024-04-22T18:39:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3040000000000004e-06 2024-04-22T18:39:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.403511+0000", 2024-04-22T18:39:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9599999999999999e-06 2024-04-22T18:39:49.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405587+0000", 2024-04-22T18:39:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002076296 2024-04-22T18:39:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405664+0000", 2024-04-22T18:39:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6316000000000006e-05 2024-04-22T18:39:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:49.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.405785+0000", 2024-04-22T18:39:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000121276 2024-04-22T18:39:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.408030+0000", 2024-04-22T18:39:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022452169999999999 2024-04-22T18:39:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607059+0000", 2024-04-22T18:39:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199029135 2024-04-22T18:39:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607094+0000", 2024-04-22T18:39:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4686e-05 2024-04-22T18:39:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.607123+0000", 2024-04-22T18:39:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9113000000000001e-05 2024-04-22T18:39:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.479 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:811 4.5 4:a4a14dff:::smithi08937929-40:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-04-22T18:39:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 45.199332964, 2024-04-22T18:39:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242306563, 2024-04-22T18:39:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 811 2024-04-22T18:39:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120822+0000", 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120821+0000", 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120825+0000", 2024-04-22T18:39:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.506e-06 2024-04-22T18:39:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120829+0000", 2024-04-22T18:39:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0849999999999999e-06 2024-04-22T18:39:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.120833+0000", 2024-04-22T18:39:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9609999999999999e-06 2024-04-22T18:39:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221263+0000", 2024-04-22T18:39:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10043045 2024-04-22T18:39:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221337+0000", 2024-04-22T18:39:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4010000000000005e-05 2024-04-22T18:39:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.221448+0000", 2024-04-22T18:39:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011065699999999999 2024-04-22T18:39:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.222541+0000", 2024-04-22T18:39:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010932610000000001 2024-04-22T18:39:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.362952+0000", 2024-04-22T18:39:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.140410535 2024-04-22T18:39:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.363102+0000", 2024-04-22T18:39:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015041499999999999 2024-04-22T18:39:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:04.363129+0000", 2024-04-22T18:39:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6574999999999999e-05 2024-04-22T18:39:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:933 4.a 4:5d2aee44:::smithi08937929-13:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-22T18:39:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.675409326, 2024-04-22T18:39:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202744591, 2024-04-22T18:39:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:39:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 933 2024-04-22T18:39:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644746+0000", 2024-04-22T18:39:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644743+0000", 2024-04-22T18:39:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644754+0000", 2024-04-22T18:39:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0807e-05 2024-04-22T18:39:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.644768+0000", 2024-04-22T18:39:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3844e-05 2024-04-22T18:39:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.744860+0000", 2024-04-22T18:39:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100092835 2024-04-22T18:39:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.745084+0000", 2024-04-22T18:39:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022401699999999999 2024-04-22T18:39:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op must wait for map", 2024-04-22T18:39:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.745090+0000", 2024-04-22T18:39:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2970000000000003e-06 2024-04-22T18:39:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.847381+0000", 2024-04-22T18:39:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102291544 2024-04-22T18:39:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:07.847490+0000", 2024-04-22T18:39:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010891399999999999 2024-04-22T18:39:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2052 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:39:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:39:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.7426593150000009, 2024-04-22T18:39:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43058958600000002, 2024-04-22T18:39:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2052 2024-04-22T18:39:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:39:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:39:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577494+0000", 2024-04-22T18:39:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577500+0000", 2024-04-22T18:39:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3279999999999996e-06 2024-04-22T18:39:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577510+0000", 2024-04-22T18:39:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9450000000000005e-06 2024-04-22T18:39:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577515+0000", 2024-04-22T18:39:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7799999999999997e-06 2024-04-22T18:39:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677622+0000", 2024-04-22T18:39:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010654099999999 2024-04-22T18:39:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677710+0000", 2024-04-22T18:39:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8472000000000003e-05 2024-04-22T18:39:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.906619+0000", 2024-04-22T18:39:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.228908519 2024-04-22T18:39:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907343+0000", 2024-04-22T18:39:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072442600000000002 2024-04-22T18:39:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.007972+0000", 2024-04-22T18:39:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100628781 2024-04-22T18:39:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008022+0000", 2024-04-22T18:39:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9843999999999999e-05 2024-04-22T18:39:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008085+0000", 2024-04-22T18:39:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3256999999999994e-05 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2087 4.a 4:537c89f8:::smithi08937929-47:head [assert-version v37,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.4127425549999995, 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38904324299999998, 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2087 2024-04-22T18:39:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:39:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:39:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907410+0000", 2024-04-22T18:39:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907417+0000", 2024-04-22T18:39:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.306e-06 2024-04-22T18:39:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907427+0000", 2024-04-22T18:39:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4930000000000004e-06 2024-04-22T18:39:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907432+0000", 2024-04-22T18:39:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5400000000000003e-06 2024-04-22T18:39:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907477+0000", 2024-04-22T18:39:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4851999999999997e-05 2024-04-22T18:39:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907537+0000", 2024-04-22T18:39:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9460999999999999e-05 2024-04-22T18:39:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.295587+0000", 2024-04-22T18:39:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38805073200000001 2024-04-22T18:39:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296281+0000", 2024-04-22T18:39:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00069370800000000002 2024-04-22T18:39:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296400+0000", 2024-04-22T18:39:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118579 2024-04-22T18:39:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296431+0000", 2024-04-22T18:39:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1627000000000001e-05 2024-04-22T18:39:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296456+0000", 2024-04-22T18:39:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4331000000000001e-05 2024-04-22T18:39:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.502 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:49.503 INFO:tasks.rados.rados.0.smithi089.stdout:1139: set_chunk oid 46 offset: 754688 length: 94208 target oid 806612eda42721d2484b48ba74b4821099b30d2739ba45de814bad4d036f2c48 offset: 0 snap -1 2024-04-22T18:39:49.503 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:49.503 INFO:tasks.rados.rados.0.smithi089.stdout:1127: expect (ObjNum 166 snap 0 seq_num 166) 2024-04-22T18:39:49.503 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:49.503+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:49.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:49.526 INFO:tasks.rados.rados.0.smithi089.stdout:1131: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:39:49.603 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:49.604+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:49.606 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:49.606 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:49.606 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:49.606 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:49.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:49.670 INFO:tasks.rados.rados.0.smithi089.stdout:1130: expect (ObjNum 147 snap 0 seq_num 147) 2024-04-22T18:39:49.705 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:49.705+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:49.710 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:49.711+0000 7fec57bee640 -1 osd.7 61 handle_osd_map injecting map crc failure 2024-04-22T18:39:49.711 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:49.711+0000 7f7f44bb3640 -1 osd.4 61 handle_osd_map injecting map crc failure 2024-04-22T18:39:49.712 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:39:49.718 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:49.718 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:49.718 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:49.718 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:49.718 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:49.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:49.749 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:39:49.749 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:39:49.749 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-04-22T18:39:49.749 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 2 2024-04-22T18:39:49.809 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:49.810+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:49.821 INFO:tasks.rados.rados.0.smithi089.stdout:1127: done (15 left) 2024-04-22T18:39:49.821 INFO:tasks.rados.rados.0.smithi089.stdout:1130: done (14 left) 2024-04-22T18:39:49.821 INFO:tasks.rados.rados.0.smithi089.stdout:1131: done (13 left) 2024-04-22T18:39:49.821 INFO:tasks.rados.rados.0.smithi089.stdout:1140: read oid 23 snap -1 2024-04-22T18:39:49.822 INFO:tasks.rados.rados.0.smithi089.stdout:1140: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:39:49.822 INFO:tasks.rados.rados.0.smithi089.stdout:1141: write (excl) oid 1 current snap is 0 2024-04-22T18:39:49.822 INFO:tasks.rados.rados.0.smithi089.stdout:1120: finishing write tid 2 to smithi08937929-6 2024-04-22T18:39:49.822 INFO:tasks.rados.rados.0.smithi089.stdout:1141: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2024-04-22T18:39:49.828 INFO:tasks.rados.rados.0.smithi089.stdout:1141: writing smithi08937929-1 from 595014 to 1220490 tid 1 2024-04-22T18:39:49.834 INFO:tasks.rados.rados.0.smithi089.stdout:1141: writing smithi08937929-1 from 1820892 to 2344284 tid 2 2024-04-22T18:39:49.839 INFO:tasks.rados.rados.0.smithi089.stdout:1141: writing smithi08937929-1 from 2797915 to 3394391 tid 3 2024-04-22T18:39:49.840 INFO:tasks.rados.rados.0.smithi089.stdout:1142: write (excl) oid 30 current snap is 0 2024-04-22T18:39:49.840 INFO:tasks.rados.rados.0.smithi089.stdout:1120: finishing write tid 3 to smithi08937929-6 2024-04-22T18:39:49.840 INFO:tasks.rados.rados.0.smithi089.stdout:1120: finishing write tid 4 to smithi08937929-6 2024-04-22T18:39:49.840 INFO:tasks.rados.rados.0.smithi089.stdout:1120: finishing write tid 5 to smithi08937929-6 2024-04-22T18:39:49.840 INFO:tasks.rados.rados.0.smithi089.stdout:1120: oid 6 updating version 0 to 50 2024-04-22T18:39:49.840 INFO:tasks.rados.rados.0.smithi089.stdout:1120: oid 6 updating version 50 to 51 2024-04-22T18:39:49.840 INFO:tasks.rados.rados.0.smithi089.stdout:1120: oid 6 version 51 is already newer than 48 2024-04-22T18:39:49.840 INFO:tasks.rados.rados.0.smithi089.stdout:1120: oid 6 version 51 is already newer than 49 2024-04-22T18:39:49.840 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 51 (ObjNum 199 snap 0 seq_num 199) dirty exists 2024-04-22T18:39:49.840 INFO:tasks.rados.rados.0.smithi089.stdout:1120: left oid 6 (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:39:49.840 INFO:tasks.rados.rados.0.smithi089.stdout:1122: expect (ObjNum 194 snap 0 seq_num 194) 2024-04-22T18:39:49.874 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:49.874 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:49.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:49.875 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:49.875 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.875 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:49.875 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:49.875 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 70.132770041000001, 2024-04-22T18:39:49.875 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30520528099999999, 2024-04-22T18:39:49.875 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.875 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.875 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.875 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.875 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.875 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 35 2024-04-22T18:39:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:39:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741852+0000", 2024-04-22T18:39:49.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:49.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741927+0000", 2024-04-22T18:39:49.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5123000000000006e-05 2024-04-22T18:39:49.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.877 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741953+0000", 2024-04-22T18:39:49.877 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5378999999999999e-05 2024-04-22T18:39:49.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.877 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.877 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741977+0000", 2024-04-22T18:39:49.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4431999999999999e-05 2024-04-22T18:39:49.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844243+0000", 2024-04-22T18:39:49.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102265525 2024-04-22T18:39:49.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.878 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.878 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844330+0000", 2024-04-22T18:39:49.878 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7417999999999995e-05 2024-04-22T18:39:49.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.878 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844470+0000", 2024-04-22T18:39:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000140197 2024-04-22T18:39:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944767+0000", 2024-04-22T18:39:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100296409 2024-04-22T18:39:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.046968+0000", 2024-04-22T18:39:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10220148499999999 2024-04-22T18:39:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047018+0000", 2024-04-22T18:39:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9206000000000003e-05 2024-04-22T18:39:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047059+0000", 2024-04-22T18:39:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1496999999999999e-05 2024-04-22T18:39:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 70.132639517000001, 2024-04-22T18:39:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:39:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:39:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:39:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:39:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:39:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:39:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:39:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:39:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:39:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:39:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:39:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:39:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:39:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:39:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:39:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:39:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:39:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:39:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:39:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:39:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:39:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:39:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:39:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:39:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:39:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:49.887 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 70.132599826000003, 2024-04-22T18:39:49.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:39:49.887 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.887 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.887 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.887 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.887 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.887 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:39:49.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.887 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.887 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.887 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:49.887 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:39:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:39:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:39:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:39:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:39:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:39:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:39:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:39:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:39:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:39:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:39:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:39:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:39:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:39:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:39:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:39:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:39:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:39:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:39:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:39:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:39:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:39:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 67.888418779000006, 2024-04-22T18:39:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402419109, 2024-04-22T18:39:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 68 2024-04-22T18:39:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:39:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986204+0000", 2024-04-22T18:39:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986208+0000", 2024-04-22T18:39:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8149999999999999e-06 2024-04-22T18:39:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986214+0000", 2024-04-22T18:39:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.604e-06 2024-04-22T18:39:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986218+0000", 2024-04-22T18:39:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6689999999999999e-06 2024-04-22T18:39:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986833+0000", 2024-04-22T18:39:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000615648 2024-04-22T18:39:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986891+0000", 2024-04-22T18:39:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7915999999999999e-05 2024-04-22T18:39:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986981+0000", 2024-04-22T18:39:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759999999999994e-05 2024-04-22T18:39:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.987584+0000", 2024-04-22T18:39:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060270199999999999 2024-04-22T18:39:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388527+0000", 2024-04-22T18:39:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40094366999999997 2024-04-22T18:39:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388583+0000", 2024-04-22T18:39:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5383000000000002e-05 2024-04-22T18:39:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388624+0000", 2024-04-22T18:39:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501999999999997e-05 2024-04-22T18:39:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 67.888402990000003, 2024-04-22T18:39:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:39:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:39:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:39:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:39:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:39:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:39:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:39:49.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:39:49.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:39:49.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:39:49.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:39:49.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:39:49.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:39:49.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:39:49.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:39:49.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:39:49.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:39:49.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:39:49.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:39:49.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:39:49.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:39:49.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:39:49.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:39:49.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:39:49.904 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.904 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.904 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.904 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:49.904 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:49.904 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 66.772009980999997, 2024-04-22T18:39:49.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305563733, 2024-04-22T18:39:49.904 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.904 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.904 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.904 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.904 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.904 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 87 2024-04-22T18:39:49.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.905 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:49.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:39:49.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102613+0000", 2024-04-22T18:39:49.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:49.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102668+0000", 2024-04-22T18:39:49.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4669000000000001e-05 2024-04-22T18:39:49.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102680+0000", 2024-04-22T18:39:49.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2026e-05 2024-04-22T18:39:49.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102687+0000", 2024-04-22T18:39:49.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8339999999999999e-06 2024-04-22T18:39:49.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405644+0000", 2024-04-22T18:39:49.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30295623199999999 2024-04-22T18:39:49.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405743+0000", 2024-04-22T18:39:49.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.98e-05 2024-04-22T18:39:49.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405868+0000", 2024-04-22T18:39:49.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012457899999999999 2024-04-22T18:39:49.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407047+0000", 2024-04-22T18:39:49.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011791449999999999 2024-04-22T18:39:49.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408088+0000", 2024-04-22T18:39:49.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010411680000000001 2024-04-22T18:39:49.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408135+0000", 2024-04-22T18:39:49.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6255999999999999e-05 2024-04-22T18:39:49.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408178+0000", 2024-04-22T18:39:49.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2877000000000003e-05 2024-04-22T18:39:49.909 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.909 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.909 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.910 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:39:49.910 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:49.910 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 66.771930052000002, 2024-04-22T18:39:49.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30560089000000001, 2024-04-22T18:39:49.910 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.910 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.910 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.910 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.910 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.910 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 88 2024-04-22T18:39:49.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.910 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:49.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:39:49.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102693+0000", 2024-04-22T18:39:49.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:49.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102698+0000", 2024-04-22T18:39:49.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9699999999999998e-06 2024-04-22T18:39:49.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102708+0000", 2024-04-22T18:39:49.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.006e-05 2024-04-22T18:39:49.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102715+0000", 2024-04-22T18:39:49.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6599999999999995e-06 2024-04-22T18:39:49.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406637+0000", 2024-04-22T18:39:49.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392130099999998 2024-04-22T18:39:49.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406696+0000", 2024-04-22T18:39:49.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9066e-05 2024-04-22T18:39:49.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406784+0000", 2024-04-22T18:39:49.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8194000000000003e-05 2024-04-22T18:39:49.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407447+0000", 2024-04-22T18:39:49.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066295700000000004 2024-04-22T18:39:49.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408237+0000", 2024-04-22T18:39:49.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079032600000000005 2024-04-22T18:39:49.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408270+0000", 2024-04-22T18:39:49.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3157000000000001e-05 2024-04-22T18:39:49.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408295+0000", 2024-04-22T18:39:49.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4151999999999998e-05 2024-04-22T18:39:49.915 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.915 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.915 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.915 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:49.915 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:49.916 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.020697928000004, 2024-04-22T18:39:49.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30533819600000001, 2024-04-22T18:39:49.916 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.916 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.916 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.916 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.916 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.916 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 106 2024-04-22T18:39:49.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.916 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:49.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:39:49.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853925+0000", 2024-04-22T18:39:49.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853958+0000", 2024-04-22T18:39:49.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2536000000000002e-05 2024-04-22T18:39:49.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853964+0000", 2024-04-22T18:39:49.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6239999999999999e-06 2024-04-22T18:39:49.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853967+0000", 2024-04-22T18:39:49.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9700000000000001e-06 2024-04-22T18:39:49.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855144+0000", 2024-04-22T18:39:49.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001176774 2024-04-22T18:39:49.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855225+0000", 2024-04-22T18:39:49.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0697000000000003e-05 2024-04-22T18:39:49.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855353+0000", 2024-04-22T18:39:49.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012788 2024-04-22T18:39:49.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058003+0000", 2024-04-22T18:39:49.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202649939 2024-04-22T18:39:49.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159184+0000", 2024-04-22T18:39:49.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101181058 2024-04-22T18:39:49.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159238+0000", 2024-04-22T18:39:49.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3967e-05 2024-04-22T18:39:49.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159264+0000", 2024-04-22T18:39:49.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6237999999999998e-05 2024-04-22T18:39:49.921 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.921 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.921 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.921 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:49.921 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:49.921 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.020651078, 2024-04-22T18:39:49.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30544294500000002, 2024-04-22T18:39:49.922 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.922 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.922 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.922 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.922 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.922 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 107 2024-04-22T18:39:49.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.922 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:49.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:39:49.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853972+0000", 2024-04-22T18:39:49.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853975+0000", 2024-04-22T18:39:49.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0939999999999999e-06 2024-04-22T18:39:49.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853978+0000", 2024-04-22T18:39:49.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:39:49.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853981+0000", 2024-04-22T18:39:49.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.858e-06 2024-04-22T18:39:49.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855831+0000", 2024-04-22T18:39:49.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018503689999999999 2024-04-22T18:39:49.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855932+0000", 2024-04-22T18:39:49.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010071 2024-04-22T18:39:49.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856029+0000", 2024-04-22T18:39:49.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6751999999999998e-05 2024-04-22T18:39:49.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058013+0000", 2024-04-22T18:39:49.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20198412600000001 2024-04-22T18:39:49.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159326+0000", 2024-04-22T18:39:49.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101312969 2024-04-22T18:39:49.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159376+0000", 2024-04-22T18:39:49.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9968e-05 2024-04-22T18:39:49.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159416+0000", 2024-04-22T18:39:49.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9641999999999998e-05 2024-04-22T18:39:49.927 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.927 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.927 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.927 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:49.927 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:49.927 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.020562682000005, 2024-04-22T18:39:49.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30548257499999998, 2024-04-22T18:39:49.927 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.928 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.928 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.928 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.928 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.928 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 108 2024-04-22T18:39:49.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.928 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:49.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:49.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:39:49.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854064+0000", 2024-04-22T18:39:49.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5820000000000002e-06 2024-04-22T18:39:49.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854080+0000", 2024-04-22T18:39:49.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5379999999999998e-05 2024-04-22T18:39:49.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854084+0000", 2024-04-22T18:39:49.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.963e-06 2024-04-22T18:39:49.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856413+0000", 2024-04-22T18:39:49.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023298070000000001 2024-04-22T18:39:49.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856494+0000", 2024-04-22T18:39:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0863000000000002e-05 2024-04-22T18:39:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856624+0000", 2024-04-22T18:39:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129799 2024-04-22T18:39:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058947+0000", 2024-04-22T18:39:49.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20232328899999999 2024-04-22T18:39:49.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159468+0000", 2024-04-22T18:39:49.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100520597 2024-04-22T18:39:49.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159508+0000", 2024-04-22T18:39:49.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9830000000000003e-05 2024-04-22T18:39:49.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159544+0000", 2024-04-22T18:39:49.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5944e-05 2024-04-22T18:39:49.933 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.933 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.933 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.933 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:49.933 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:49.933 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.020518190000004, 2024-04-22T18:39:49.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30556502499999999, 2024-04-22T18:39:49.933 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.933 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.933 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.934 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.934 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.934 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:39:49.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.934 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:49.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:39:49.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854105+0000", 2024-04-22T18:39:49.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854175+0000", 2024-04-22T18:39:49.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0178000000000005e-05 2024-04-22T18:39:49.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854191+0000", 2024-04-22T18:39:49.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5965999999999999e-05 2024-04-22T18:39:49.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854196+0000", 2024-04-22T18:39:49.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.42e-06 2024-04-22T18:39:49.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956526+0000", 2024-04-22T18:39:49.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10233094299999999 2024-04-22T18:39:49.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956634+0000", 2024-04-22T18:39:49.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107398 2024-04-22T18:39:49.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956730+0000", 2024-04-22T18:39:49.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6217000000000004e-05 2024-04-22T18:39:49.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058955+0000", 2024-04-22T18:39:49.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102224749 2024-04-22T18:39:49.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159596+0000", 2024-04-22T18:39:49.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100641297 2024-04-22T18:39:49.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159637+0000", 2024-04-22T18:39:49.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874000000000002e-05 2024-04-22T18:39:49.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159671+0000", 2024-04-22T18:39:49.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3603999999999999e-05 2024-04-22T18:39:49.938 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.939 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.939 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.939 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:49.939 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:49.939 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.020424343000002, 2024-04-22T18:39:49.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30683048800000001, 2024-04-22T18:39:49.939 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.939 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.939 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.939 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.939 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.939 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110 2024-04-22T18:39:49.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.940 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:49.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:49.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:39:49.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854202+0000", 2024-04-22T18:39:49.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:39:49.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854206+0000", 2024-04-22T18:39:49.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5870000000000001e-06 2024-04-22T18:39:49.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854208+0000", 2024-04-22T18:39:49.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7199999999999998e-06 2024-04-22T18:39:49.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057285+0000", 2024-04-22T18:39:49.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20307620800000001 2024-04-22T18:39:49.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057414+0000", 2024-04-22T18:39:49.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012896899999999999 2024-04-22T18:39:49.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057588+0000", 2024-04-22T18:39:49.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000174802 2024-04-22T18:39:49.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159429+0000", 2024-04-22T18:39:49.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101840783 2024-04-22T18:39:49.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160947+0000", 2024-04-22T18:39:49.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001518152 2024-04-22T18:39:49.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160992+0000", 2024-04-22T18:39:49.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4725000000000003e-05 2024-04-22T18:39:49.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161030+0000", 2024-04-22T18:39:49.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7789e-05 2024-04-22T18:39:49.944 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.944 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.944 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.945 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:49.945 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:49.945 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.020413161999997, 2024-04-22T18:39:49.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:39:49.945 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.945 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.945 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.945 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.945 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.945 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:39:49.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.945 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:49.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:39:49.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:39:49.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:39:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:39:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:39:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:39:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:39:49.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:39:49.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:39:49.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:39:49.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:39:49.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:39:49.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:39:49.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:39:49.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:39:49.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:39:49.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:39:49.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:39:49.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:39:49.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:39:49.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:39:49.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:39:49.950 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.950 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.950 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.950 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:49.950 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:49.951 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.020397504000002, 2024-04-22T18:39:49.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:39:49.951 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.951 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.951 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.951 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.951 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.951 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:39:49.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.951 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:49.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:49.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:39:49.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:39:49.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:39:49.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:39:49.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:39:49.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:39:49.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:39:49.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:39:49.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:39:49.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:39:49.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:39:49.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:39:49.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:39:49.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:39:49.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:39:49.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:39:49.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:39:49.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:39:49.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:39:49.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:39:49.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:39:49.956 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.956 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.956 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.957 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:49.957 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:49.957 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.020346341999996, 2024-04-22T18:39:49.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:39:49.957 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.957 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.957 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.957 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.957 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.957 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:39:49.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.957 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:49.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:49.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:39:49.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:39:49.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:39:49.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:39:49.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:39:49.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:39:49.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:39:49.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:39:49.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:39:49.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:39:49.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:39:49.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:39:49.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:39:49.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:39:49.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:39:49.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:39:49.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:39:49.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:39:49.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:39:49.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:39:49.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:39:49.962 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.962 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.962 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.962 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:49.962 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:49.963 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.020331118000001, 2024-04-22T18:39:49.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:39:49.963 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.963 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.963 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.963 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.963 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.963 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:39:49.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.963 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:49.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:39:49.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:39:49.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:39:49.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:39:49.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:39:49.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:39:49.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:39:49.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:39:49.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:39:49.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:39:49.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:39:49.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:39:49.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:39:49.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:39:49.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:39:49.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:39:49.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:39:49.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:39:49.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:39:49.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:39:49.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:39:49.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:39:49.968 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.968 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.968 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.968 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:49.968 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:49.968 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.020271541, 2024-04-22T18:39:49.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:39:49.968 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.969 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.969 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.969 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.969 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.969 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:39:49.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.969 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:49.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:49.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:39:49.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:39:49.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:39:49.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:39:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:39:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:39:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:39:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:39:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:39:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:39:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:39:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:39:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:39:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:39:49.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:39:49.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:39:49.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:39:49.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:39:49.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:39:49.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:39:49.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:39:49.974 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.974 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.974 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.974 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:49.974 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:49.974 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.020189224999996, 2024-04-22T18:39:49.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:39:49.974 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.974 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:39:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:39:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:39:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:39:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:39:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:39:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:39:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:39:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:39:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:39:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:39:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:39:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:39:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:39:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:39:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:39:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:39:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:39:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:39:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:39:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:39:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:39:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:39:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:39:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.020179068999994, 2024-04-22T18:39:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:39:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:39:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:39:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:39:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:39:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:39:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:39:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:39:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:39:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:39:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:39:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:39:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:39:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:39:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:39:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:39:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:39:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:39:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:39:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:39:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:39:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:39:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:39:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:39:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:161 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:39:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.419684777999997, 2024-04-22T18:39:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40239169000000002, 2024-04-22T18:39:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:39:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:39:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 161 2024-04-22T18:39:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:39:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454938+0000", 2024-04-22T18:39:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454982+0000", 2024-04-22T18:39:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3470999999999998e-05 2024-04-22T18:39:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454988+0000", 2024-04-22T18:39:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7640000000000002e-06 2024-04-22T18:39:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555054+0000", 2024-04-22T18:39:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006607300000001 2024-04-22T18:39:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555607+0000", 2024-04-22T18:39:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055319500000000003 2024-04-22T18:39:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555689+0000", 2024-04-22T18:39:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1817999999999995e-05 2024-04-22T18:39:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555804+0000", 2024-04-22T18:39:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115346 2024-04-22T18:39:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556554+0000", 2024-04-22T18:39:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000750102 2024-04-22T18:39:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857265+0000", 2024-04-22T18:39:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30071122700000003 2024-04-22T18:39:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857302+0000", 2024-04-22T18:39:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7069e-05 2024-04-22T18:39:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857331+0000", 2024-04-22T18:39:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8291999999999999e-05 2024-04-22T18:39:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:49.991 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:49.993 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:49.994 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:49.993+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:50.079 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-22T18:39:50.079+0000 7f9b4bbc8640 -1 mon.a@0(leader).osd e62 definitely_dead 0 2024-04-22T18:39:50.090 INFO:tasks.rados.rados.0.smithi089.stdout:1126: expect (ObjNum 149 snap 0 seq_num 149) 2024-04-22T18:39:50.093 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:50.094+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:50.096 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.5.13:37 3.3 e62/19)", 2024-04-22T18:39:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.045662+0000", 2024-04-22T18:39:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.052144674000000002, 2024-04-22T18:39:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.052162515, 2024-04-22T18:39:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.045662+0000", 2024-04-22T18:39:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.045662+0000", 2024-04-22T18:39:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.045662+0000", 2024-04-22T18:39:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.045670+0000", 2024-04-22T18:39:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.208e-06 2024-04-22T18:39:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.045692+0000", 2024-04-22T18:39:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2351999999999999e-05 2024-04-22T18:39:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.045703+0000", 2024-04-22T18:39:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1122e-05 2024-04-22T18:39:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.5.13:38 3.3 e62/19)", 2024-04-22T18:39:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.045855+0000", 2024-04-22T18:39:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.051951340999999998, 2024-04-22T18:39:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.051997450000000001, 2024-04-22T18:39:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.045855+0000", 2024-04-22T18:39:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.045855+0000", 2024-04-22T18:39:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.045854+0000", 2024-04-22T18:39:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.045861+0000", 2024-04-22T18:39:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5579999999999997e-06 2024-04-22T18:39:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.045872+0000", 2024-04-22T18:39:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0849000000000001e-05 2024-04-22T18:39:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.045878+0000", 2024-04-22T18:39:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6810000000000001e-06 2024-04-22T18:39:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:39:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 2 2024-04-22T18:39:50.102 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:50.105 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:50.194 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:50.194+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:50.207 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:50.207 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:50.216 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:50.241 INFO:tasks.rados.rados.0.smithi089.stdout:1132: finishing write tid 1 to smithi08937929-3 2024-04-22T18:39:50.241 INFO:tasks.rados.rados.0.smithi089.stdout:1132: finishing write tid 2 to smithi08937929-3 2024-04-22T18:39:50.241 INFO:tasks.rados.rados.0.smithi089.stdout:1132: finishing write tid 3 to smithi08937929-3 2024-04-22T18:39:50.241 INFO:tasks.rados.rados.0.smithi089.stdout:1132: finishing write tid 4 to smithi08937929-3 2024-04-22T18:39:50.241 INFO:tasks.rados.rados.0.smithi089.stdout:1132: finishing write tid 5 to smithi08937929-3 2024-04-22T18:39:50.241 INFO:tasks.rados.rados.0.smithi089.stdout:1132: oid 3 updating version 0 to 47 2024-04-22T18:39:50.241 INFO:tasks.rados.rados.0.smithi089.stdout:1132: oid 3 version 47 is already newer than 46 2024-04-22T18:39:50.241 INFO:tasks.rados.rados.0.smithi089.stdout:1132: oid 3 updating version 47 to 48 2024-04-22T18:39:50.241 INFO:tasks.rados.rados.0.smithi089.stdout:1132: oid 3 version 48 is already newer than 45 2024-04-22T18:39:50.241 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 48 (ObjNum 200 snap 0 seq_num 200) dirty exists 2024-04-22T18:39:50.241 INFO:tasks.rados.rados.0.smithi089.stdout:1132: left oid 3 (ObjNum 200 snap 0 seq_num 200) 2024-04-22T18:39:50.241 INFO:tasks.rados.rados.0.smithi089.stdout:1142: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2024-04-22T18:39:50.247 INFO:tasks.rados.rados.0.smithi089.stdout:1142: writing smithi08937929-30 from 611821 to 1312546 tid 1 2024-04-22T18:39:50.255 INFO:tasks.rados.rados.0.smithi089.stdout:1142: writing smithi08937929-30 from 1879374 to 2662777 tid 2 2024-04-22T18:39:50.259 INFO:tasks.rados.rados.0.smithi089.stdout:1142: writing smithi08937929-30 from 3341691 to 3411821 tid 3 2024-04-22T18:39:50.260 INFO:tasks.rados.rados.0.smithi089.stdout:1120: done (15 left) 2024-04-22T18:39:50.260 INFO:tasks.rados.rados.0.smithi089.stdout:1122: done (14 left) 2024-04-22T18:39:50.260 INFO:tasks.rados.rados.0.smithi089.stdout:1125: done (13 left) 2024-04-22T18:39:50.260 INFO:tasks.rados.rados.0.smithi089.stdout:1126: done (12 left) 2024-04-22T18:39:50.260 INFO:tasks.rados.rados.0.smithi089.stdout:1132: done (11 left) 2024-04-22T18:39:50.260 INFO:tasks.rados.rados.0.smithi089.stdout:1143: read oid 17 snap -1 2024-04-22T18:39:50.260 INFO:tasks.rados.rados.0.smithi089.stdout:1143: expect (ObjNum 192 snap 0 seq_num 192) 2024-04-22T18:39:50.260 INFO:tasks.rados.rados.0.smithi089.stdout:1144: set_chunk oid 2 target oid 2024-04-22T18:39:50.272 INFO:tasks.rados.rados.0.smithi089.stdout:1144: set_chunk oid 2 offset: 650240 length: 1679360 target oid 3b3d57544944a65507a3893f53f6a1021067aa03b5851d967a6a721a24f8ebde offset: 0 snap -1 2024-04-22T18:39:50.272 INFO:tasks.rados.rados.0.smithi089.stdout:1145: write oid 25 current snap is 0 2024-04-22T18:39:50.272 INFO:tasks.rados.rados.0.smithi089.stdout:1145: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2024-04-22T18:39:50.279 INFO:tasks.rados.rados.0.smithi089.stdout:1145: writing smithi08937929-25 from 628628 to 1404602 tid 1 2024-04-22T18:39:50.285 INFO:tasks.rados.rados.0.smithi089.stdout:1145: writing smithi08937929-25 from 1937856 to 2464917 tid 2 2024-04-22T18:39:50.291 INFO:tasks.rados.rados.0.smithi089.stdout:1145: writing smithi08937929-25 from 3252761 to 3428628 tid 3 2024-04-22T18:39:50.291 INFO:tasks.rados.rados.0.smithi089.stdout:1146: read oid 14 snap -1 2024-04-22T18:39:50.291 INFO:tasks.rados.rados.0.smithi089.stdout:1146: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:39:50.291 INFO:tasks.rados.rados.0.smithi089.stdout:1147: read oid 9 snap -1 2024-04-22T18:39:50.291 INFO:tasks.rados.rados.0.smithi089.stdout:1147: expect (ObjNum 153 snap 0 seq_num 153) 2024-04-22T18:39:50.291 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:50.291 INFO:tasks.rados.rados.0.smithi089.stdout:1134: done (15 left) 2024-04-22T18:39:50.291 INFO:tasks.rados.rados.0.smithi089.stdout:1148: read oid 16 snap -1 2024-04-22T18:39:50.292 INFO:tasks.rados.rados.0.smithi089.stdout:1148: expect deleted 2024-04-22T18:39:50.292 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:50.292 INFO:tasks.rados.rados.0.smithi089.stdout:1135: expect (ObjNum 161 snap 0 seq_num 161) 2024-04-22T18:39:50.295 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:50.296+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:50.348 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:61 4.b 4:d923883e:::smithi08937929-16:head [write 652105~445088,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.077661489999997, 2024-04-22T18:39:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.38578178600000002, 2024-04-22T18:39:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 61 2024-04-22T18:39:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270192+0000", 2024-04-22T18:39:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270401+0000", 2024-04-22T18:39:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020921199999999999 2024-04-22T18:39:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270415+0000", 2024-04-22T18:39:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4191e-05 2024-04-22T18:39:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270422+0000", 2024-04-22T18:39:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7329999999999997e-06 2024-04-22T18:39:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270461+0000", 2024-04-22T18:39:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9530999999999998e-05 2024-04-22T18:39:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270550+0000", 2024-04-22T18:39:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8508e-05 2024-04-22T18:39:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270648+0000", 2024-04-22T18:39:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8377999999999999e-05 2024-04-22T18:39:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.271907+0000", 2024-04-22T18:39:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012589599999999999 2024-04-22T18:39:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655904+0000", 2024-04-22T18:39:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.383997115 2024-04-22T18:39:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655933+0000", 2024-04-22T18:39:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9011999999999999e-05 2024-04-22T18:39:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655977+0000", 2024-04-22T18:39:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3578000000000002e-05 2024-04-22T18:39:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:62 4.b 4:d923883e:::smithi08937929-16:head [write 1728171~452988] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:39:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.071702842000001, 2024-04-22T18:39:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.379937407, 2024-04-22T18:39:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62 2024-04-22T18:39:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:39:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276154+0000", 2024-04-22T18:39:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276151+0000", 2024-04-22T18:39:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276370+0000", 2024-04-22T18:39:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000218951 2024-04-22T18:39:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276382+0000", 2024-04-22T18:39:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1517e-05 2024-04-22T18:39:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276389+0000", 2024-04-22T18:39:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8939999999999993e-06 2024-04-22T18:39:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276430+0000", 2024-04-22T18:39:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0958999999999999e-05 2024-04-22T18:39:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276485+0000", 2024-04-22T18:39:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4208e-05 2024-04-22T18:39:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276567+0000", 2024-04-22T18:39:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2006000000000001e-05 2024-04-22T18:39:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277685+0000", 2024-04-22T18:39:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001118237 2024-04-22T18:39:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656024+0000", 2024-04-22T18:39:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.378339588 2024-04-22T18:39:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656054+0000", 2024-04-22T18:39:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9373000000000002e-05 2024-04-22T18:39:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656091+0000", 2024-04-22T18:39:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7168000000000001e-05 2024-04-22T18:39:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:63 4.b 4:d923883e:::smithi08937929-16:head [setxattr _header (56) in=7b,truncate 2252105] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:39:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.071460586000001, 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37982964200000002, 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 63 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276396+0000", 2024-04-22T18:39:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276395+0000", 2024-04-22T18:39:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276400+0000", 2024-04-22T18:39:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6090000000000003e-06 2024-04-22T18:39:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276407+0000", 2024-04-22T18:39:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5850000000000002e-06 2024-04-22T18:39:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276411+0000", 2024-04-22T18:39:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2569999999999999e-06 2024-04-22T18:39:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277254+0000", 2024-04-22T18:39:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084384300000000005 2024-04-22T18:39:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277312+0000", 2024-04-22T18:39:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7395999999999997e-05 2024-04-22T18:39:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277388+0000", 2024-04-22T18:39:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6604e-05 2024-04-22T18:39:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277870+0000", 2024-04-22T18:39:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000481545 2024-04-22T18:39:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656138+0000", 2024-04-22T18:39:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37826752499999999 2024-04-22T18:39:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656177+0000", 2024-04-22T18:39:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9419999999999999e-05 2024-04-22T18:39:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656226+0000", 2024-04-22T18:39:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8562e-05 2024-04-22T18:39:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:64 4.b 4:d923883e:::smithi08937929-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.071429965, 2024-04-22T18:39:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37999559999999999, 2024-04-22T18:39:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 64 2024-04-22T18:39:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276426+0000", 2024-04-22T18:39:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276429+0000", 2024-04-22T18:39:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7329999999999999e-06 2024-04-22T18:39:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276435+0000", 2024-04-22T18:39:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.02e-06 2024-04-22T18:39:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.276439+0000", 2024-04-22T18:39:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7659999999999998e-06 2024-04-22T18:39:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277530+0000", 2024-04-22T18:39:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010909470000000001 2024-04-22T18:39:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.277603+0000", 2024-04-22T18:39:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2638999999999994e-05 2024-04-22T18:39:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656259+0000", 2024-04-22T18:39:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37865579300000002 2024-04-22T18:39:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656330+0000", 2024-04-22T18:39:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0899e-05 2024-04-22T18:39:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.656422+0000", 2024-04-22T18:39:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2397000000000003e-05 2024-04-22T18:39:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.892499889, 2024-04-22T18:39:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:39:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:39:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:39:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:39:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:39:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:39:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:39:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:39:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:39:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:39:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:39:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:39:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:39:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:39:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:39:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:39:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:39:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:39:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:39:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:39:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:39:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:39:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:39:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:315 4.b 4:d923883e:::smithi08937929-16:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.356037108999999, 2024-04-22T18:39:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40309447300000001, 2024-04-22T18:39:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 315 2024-04-22T18:39:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991817+0000", 2024-04-22T18:39:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991825+0000", 2024-04-22T18:39:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7279999999999995e-06 2024-04-22T18:39:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991834+0000", 2024-04-22T18:39:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1980000000000007e-06 2024-04-22T18:39:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991841+0000", 2024-04-22T18:39:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0740000000000004e-06 2024-04-22T18:39:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991881+0000", 2024-04-22T18:39:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-04-22T18:39:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991940+0000", 2024-04-22T18:39:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9382000000000002e-05 2024-04-22T18:39:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394250+0000", 2024-04-22T18:39:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40230947700000003 2024-04-22T18:39:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394752+0000", 2024-04-22T18:39:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050269699999999997 2024-04-22T18:39:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394865+0000", 2024-04-22T18:39:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011267700000000001 2024-04-22T18:39:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394892+0000", 2024-04-22T18:39:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7535999999999999e-05 2024-04-22T18:39:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394914+0000", 2024-04-22T18:39:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1398000000000001e-05 2024-04-22T18:39:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:774 4.b 4:defe92fb:::smithi08937929-44:head [write 2778819~95145] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:39:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 47.199527433, 2024-04-22T18:39:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40475725000000001, 2024-04-22T18:39:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 774 2024-04-22T18:39:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:39:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:39:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148328+0000", 2024-04-22T18:39:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148392+0000", 2024-04-22T18:39:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3323000000000003e-05 2024-04-22T18:39:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148398+0000", 2024-04-22T18:39:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4420000000000003e-06 2024-04-22T18:39:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148409+0000", 2024-04-22T18:39:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509999999999999e-05 2024-04-22T18:39:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251407+0000", 2024-04-22T18:39:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102998305 2024-04-22T18:39:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251500+0000", 2024-04-22T18:39:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3607e-05 2024-04-22T18:39:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251648+0000", 2024-04-22T18:39:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014788099999999999 2024-04-22T18:39:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256149+0000", 2024-04-22T18:39:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045010980000000003 2024-04-22T18:39:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553016+0000", 2024-04-22T18:39:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29686620800000002 2024-04-22T18:39:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553054+0000", 2024-04-22T18:39:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8414999999999997e-05 2024-04-22T18:39:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553086+0000", 2024-04-22T18:39:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2236000000000001e-05 2024-04-22T18:39:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:39:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 47.199442640999997, 2024-04-22T18:39:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:39:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:39:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:39:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:39:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:39:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:39:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:39:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:39:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:39:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:39:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:39:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:39:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:39:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:39:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:39:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:39:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:39:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:39:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:39:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:39:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:39:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:39:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:39:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:39:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:39:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 47.199424165000003, 2024-04-22T18:39:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:39:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:39:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:39:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:39:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:39:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:39:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:39:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:39:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:39:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:39:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:39:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:39:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:39:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:39:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:39:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:39:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:39:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:39:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1242 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-22T18:39:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:39:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.606178616000001, 2024-04-22T18:39:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40662891899999998, 2024-04-22T18:39:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1242 2024-04-22T18:39:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:39:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:39:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741677+0000", 2024-04-22T18:39:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741680+0000", 2024-04-22T18:39:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8059999999999999e-06 2024-04-22T18:39:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741685+0000", 2024-04-22T18:39:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0780000000000003e-06 2024-04-22T18:39:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741690+0000", 2024-04-22T18:39:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4020000000000004e-06 2024-04-22T18:39:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741732+0000", 2024-04-22T18:39:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2110000000000002e-05 2024-04-22T18:39:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741798+0000", 2024-04-22T18:39:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6432000000000004e-05 2024-04-22T18:39:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.148307+0000", 2024-04-22T18:39:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40650862100000001 2024-04-22T18:39:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1408 4.6 4:630bd8e5:::smithi08937929-45:head [write 2816389~693495] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:39:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:39:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.186208809, 2024-04-22T18:39:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408959452, 2024-04-22T18:39:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1408 2024-04-22T18:39:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161647+0000", 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161951+0000", 2024-04-22T18:39:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030389799999999998 2024-04-22T18:39:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161955+0000", 2024-04-22T18:39:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1989999999999999e-06 2024-04-22T18:39:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161958+0000", 2024-04-22T18:39:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2260000000000001e-06 2024-04-22T18:39:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.265867+0000", 2024-04-22T18:39:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103908927 2024-04-22T18:39:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.265960+0000", 2024-04-22T18:39:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3016999999999994e-05 2024-04-22T18:39:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.266124+0000", 2024-04-22T18:39:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016347700000000001 2024-04-22T18:39:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570491+0000", 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30436694199999997 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570502+0000", 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1107999999999999e-05 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570556+0000", 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4388000000000002e-05 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570607+0000", 2024-04-22T18:39:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0782999999999998e-05 2024-04-22T18:39:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1409 4.6 4:630bd8e5:::smithi08937929-45:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:39:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:39:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.185895105, 2024-04-22T18:39:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40880186200000002, 2024-04-22T18:39:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1409 2024-04-22T18:39:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:39:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:39:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:39:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161964+0000", 2024-04-22T18:39:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5909999999999998e-06 2024-04-22T18:39:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161966+0000", 2024-04-22T18:39:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5979999999999999e-06 2024-04-22T18:39:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161969+0000", 2024-04-22T18:39:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7049999999999999e-06 2024-04-22T18:39:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269205+0000", 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107236228 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269320+0000", 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114571 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269476+0000", 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155928 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570619+0000", 2024-04-22T18:39:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30114322700000001 2024-04-22T18:39:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570672+0000", 2024-04-22T18:39:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3232000000000001e-05 2024-04-22T18:39:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570724+0000", 2024-04-22T18:39:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2258e-05 2024-04-22T18:39:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570763+0000", 2024-04-22T18:39:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8927999999999997e-05 2024-04-22T18:39:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1410 4.6 4:630bd8e5:::smithi08937929-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:39:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:39:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.185885588000001, 2024-04-22T18:39:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40900504500000001, 2024-04-22T18:39:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1410 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:39:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:39:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161972+0000", 2024-04-22T18:39:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6419999999999999e-06 2024-04-22T18:39:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161976+0000", 2024-04-22T18:39:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9260000000000002e-06 2024-04-22T18:39:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161979+0000", 2024-04-22T18:39:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9359999999999999e-06 2024-04-22T18:39:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570790+0000", 2024-04-22T18:39:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40881099199999998 2024-04-22T18:39:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570879+0000", 2024-04-22T18:39:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8895999999999997e-05 2024-04-22T18:39:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570976+0000", 2024-04-22T18:39:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6923999999999996e-05 2024-04-22T18:39:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:39:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:39:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.436375853000001, 2024-04-22T18:39:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:39:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:39:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:39:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:39:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:39:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:39:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:39:50.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:39:50.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:39:50.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:39:50.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:39:50.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:39:50.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:39:50.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:39:50.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:39:50.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:50.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:39:50.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:39:50.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.907216526999999, 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.429 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:39:50.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.430 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:39:50.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:39:50.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:39:50.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:39:50.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:39:50.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:39:50.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:39:50.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:39:50.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:39:50.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:39:50.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:39:50.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:50.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:39:50.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:39:50.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:39:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:39:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:39:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:39:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:39:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:39:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.434 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.434 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:50.434 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:39:50.434 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.729509132, 2024-04-22T18:39:50.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:39:50.434 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.434 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.434 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:39:50.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:39:50.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:39:50.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:50.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:39:50.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:39:50.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:39:50.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:39:50.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:39:50.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:39:50.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:39:50.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:39:50.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.438 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.439 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.439 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:50.439 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:39:50.439 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.72944086, 2024-04-22T18:39:50.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:39:50.439 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.439 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.439 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:39:50.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:39:50.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:39:50.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:39:50.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:39:50.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:39:50.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:39:50.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:39:50.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:39:50.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:39:50.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:39:50.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.443 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.444 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.444 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:50.444 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:39:50.444 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.729273019000001, 2024-04-22T18:39:50.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:39:50.444 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.444 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.444 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:39:50.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:39:50.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:39:50.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:39:50.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:39:50.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:39:50.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:39:50.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:39:50.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:39:50.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:39:50.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:39:50.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:39:50.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:39:50.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:39:50.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:39:50.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:39:50.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:39:50.449 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.449 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.449 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.449 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:39:50.449 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:39:50.449 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.7639475440000005, 2024-04-22T18:39:50.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:39:50.449 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.449 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.449 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:39:50.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:39:50.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:39:50.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:50.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:39:50.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:39:50.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:39:50.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:39:50.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:39:50.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:39:50.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:39:50.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:39:50.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:39:50.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:39:50.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:50.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:39:50.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:39:50.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:39:50.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:39:50.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:39:50.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:39:50.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:39:50.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:39:50.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:39:50.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:39:50.454 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.454 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.454 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.454 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:39:50.454 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:39:50.454 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.6637953569999997, 2024-04-22T18:39:50.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:39:50.455 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.455 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.455 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:39:50.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:39:50.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:39:50.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:50.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:39:50.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:39:50.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:39:50.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:39:50.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:39:50.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:39:50.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:39:50.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:39:50.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:39:50.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:39:50.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:50.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:39:50.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:39:50.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:39:50.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:39:50.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:39:50.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:39:50.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:39:50.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:39:50.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:39:50.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:39:50.459 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.459 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.459 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.460 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.460 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.460 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:50.461 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:50.462 INFO:tasks.rados.rados.0.smithi089.stdout:1138: finishing set_chunk to oid 8 2024-04-22T18:39:50.462 INFO:tasks.rados.rados.0.smithi089.stdout:1138: got expected ENOENT (src dne) 2024-04-22T18:39:50.462 INFO:tasks.rados.rados.0.smithi089.stdout:1133: finishing write tid 1 to smithi08937929-20 2024-04-22T18:39:50.462 INFO:tasks.rados.rados.0.smithi089.stdout:1137: expect (ObjNum 162 snap 0 seq_num 162) 2024-04-22T18:39:50.462 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:50.462+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:50.559 INFO:tasks.rados.rados.0.smithi089.stdout:1121: got (0) Success 2024-04-22T18:39:50.559 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 0 (ObjNum 188 snap 0 seq_num 188) dirty exists 2024-04-22T18:39:50.559 INFO:tasks.rados.rados.0.smithi089.stdout:1139: finishing set_chunk to oid 46 2024-04-22T18:39:50.559 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 46 set_chunk returned error code -95 offset: 754688 length: 94208 2024-04-22T18:39:50.559 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-22T18:39:50.559 INFO:tasks.rados.rados.0.smithi089.stdout:1133: finishing write tid 2 to smithi08937929-20 2024-04-22T18:39:50.559 INFO:tasks.rados.rados.0.smithi089.stdout:1133: finishing write tid 3 to smithi08937929-20 2024-04-22T18:39:50.559 INFO:tasks.rados.rados.0.smithi089.stdout:1136: finishing set_chunk to oid 24 2024-04-22T18:39:50.559 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 97 (ObjNum 174 snap 0 seq_num 174) dirty exists 2024-04-22T18:39:50.559 INFO:tasks.rados.rados.0.smithi089.stdout:1133: finishing write tid 4 to smithi08937929-20 2024-04-22T18:39:50.559 INFO:tasks.rados.rados.0.smithi089.stdout:1133: finishing write tid 5 to smithi08937929-20 2024-04-22T18:39:50.559 INFO:tasks.rados.rados.0.smithi089.stdout:1133: oid 20 updating version 0 to 12 2024-04-22T18:39:50.559 INFO:tasks.rados.rados.0.smithi089.stdout:1133: oid 20 version 12 is already newer than 10 2024-04-22T18:39:50.559 INFO:tasks.rados.rados.0.smithi089.stdout:1133: oid 20 version 12 is already newer than 11 2024-04-22T18:39:50.560 INFO:tasks.rados.rados.0.smithi089.stdout:1133: oid 20 updating version 12 to 13 2024-04-22T18:39:50.560 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 13 (ObjNum 201 snap 0 seq_num 201) dirty exists 2024-04-22T18:39:50.560 INFO:tasks.rados.rados.0.smithi089.stdout:1133: left oid 20 (ObjNum 201 snap 0 seq_num 201) 2024-04-22T18:39:50.560 INFO:tasks.rados.rados.0.smithi089.stdout:1121: done (15 left) 2024-04-22T18:39:50.560 INFO:tasks.rados.rados.0.smithi089.stdout:1133: done (14 left) 2024-04-22T18:39:50.560 INFO:tasks.rados.rados.0.smithi089.stdout:1135: done (13 left) 2024-04-22T18:39:50.560 INFO:tasks.rados.rados.0.smithi089.stdout:1136: done (12 left) 2024-04-22T18:39:50.560 INFO:tasks.rados.rados.0.smithi089.stdout:1137: done (11 left) 2024-04-22T18:39:50.560 INFO:tasks.rados.rados.0.smithi089.stdout:1138: done (10 left) 2024-04-22T18:39:50.560 INFO:tasks.rados.rados.0.smithi089.stdout:1139: done (9 left) 2024-04-22T18:39:50.560 INFO:tasks.rados.rados.0.smithi089.stdout:1149: tier_promote oid 5 2024-04-22T18:39:50.560 INFO:tasks.rados.rados.0.smithi089.stdout:1140: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:39:50.561 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:50.561+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:50.605 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:50.605 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:50.614 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:50.662 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:50.663+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:50.709 INFO:tasks.rados.rados.0.smithi089.stdout:1141: finishing write tid 1 to smithi08937929-1 2024-04-22T18:39:50.709 INFO:tasks.rados.rados.0.smithi089.stdout:1141: finishing write tid 2 to smithi08937929-1 2024-04-22T18:39:50.710 INFO:tasks.rados.rados.0.smithi089.stdout:1140: done (9 left) 2024-04-22T18:39:50.710 INFO:tasks.rados.rados.0.smithi089.stdout:1150: read oid 23 snap -1 2024-04-22T18:39:50.710 INFO:tasks.rados.rados.0.smithi089.stdout:1150: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:39:50.710 INFO:tasks.rados.rados.0.smithi089.stdout:1151: set_chunk oid 8 target oid 2024-04-22T18:39:50.710 INFO:tasks.rados.rados.0.smithi089.stdout:1151: set_chunk oid 8 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:39:50.710 INFO:tasks.rados.rados.0.smithi089.stdout:1152: read oid 26 snap -1 2024-04-22T18:39:50.710 INFO:tasks.rados.rados.0.smithi089.stdout:1152: expect (ObjNum 140 snap 0 seq_num 140) 2024-04-22T18:39:50.710 INFO:tasks.rados.rados.0.smithi089.stdout:1153: write oid 34 current snap is 0 2024-04-22T18:39:50.710 INFO:tasks.rados.rados.0.smithi089.stdout:1141: finishing write tid 3 to smithi08937929-1 2024-04-22T18:39:50.710 INFO:tasks.rados.rados.0.smithi089.stdout:1141: finishing write tid 4 to smithi08937929-1 2024-04-22T18:39:50.710 INFO:tasks.rados.rados.0.smithi089.stdout:1153: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-04-22T18:39:50.715 INFO:tasks.rados.rados.0.smithi089.stdout:1153: writing smithi08937929-34 from 645435 to 1096658 tid 1 2024-04-22T18:39:50.724 INFO:tasks.rados.rados.0.smithi089.stdout:1153: writing smithi08937929-34 from 1879985 to 2667057 tid 2 2024-04-22T18:39:50.728 INFO:teuthology.orchestra.run.smithi089.stderr:marked in osd.2. 2024-04-22T18:39:50.729 INFO:tasks.rados.rados.0.smithi089.stdout:1153: writing smithi08937929-34 from 3280184 to 3445435 tid 3 2024-04-22T18:39:50.729 INFO:tasks.rados.rados.0.smithi089.stdout:1154: tier_evict oid 20 2024-04-22T18:39:50.729 INFO:tasks.rados.rados.0.smithi089.stdout:1154: tier_evict oid 20 snap -1 2024-04-22T18:39:50.729 INFO:tasks.rados.rados.0.smithi089.stdout:1141: finishing write tid 5 to smithi08937929-1 2024-04-22T18:39:50.729 INFO:tasks.rados.rados.0.smithi089.stdout:1141: oid 1 updating version 0 to 51 2024-04-22T18:39:50.729 INFO:tasks.rados.rados.0.smithi089.stdout:1141: oid 1 version 51 is already newer than 50 2024-04-22T18:39:50.729 INFO:tasks.rados.rados.0.smithi089.stdout:1141: oid 1 version 51 is already newer than 49 2024-04-22T18:39:50.729 INFO:tasks.rados.rados.0.smithi089.stdout:1141: oid 1 version 51 is already newer than 48 2024-04-22T18:39:50.729 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 51 (ObjNum 202 snap 0 seq_num 202) dirty exists 2024-04-22T18:39:50.729 INFO:tasks.rados.rados.0.smithi089.stdout:1141: left oid 1 (ObjNum 202 snap 0 seq_num 202) 2024-04-22T18:39:50.730 INFO:tasks.rados.rados.0.smithi089.stdout:1143: expect (ObjNum 192 snap 0 seq_num 192) 2024-04-22T18:39:50.741 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-04-22T18:39:50.762 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:50.763+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:50.763 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:50.763 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:50.772 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:50.863 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:50.863+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:50.870 INFO:tasks.rados.rados.0.smithi089.stdout:1142: finishing write tid 1 to smithi08937929-30 2024-04-22T18:39:50.870 INFO:tasks.rados.rados.0.smithi089.stdout:1147: expect (ObjNum 153 snap 0 seq_num 153) 2024-04-22T18:39:50.920 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:50.920 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:50.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:50.920 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:50.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.920 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:9 4.f 4:f2c48a0e:::smithi08937929-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:50.920 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:50.920 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.801944601000002, 2024-04-22T18:39:50.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49290714600000002, 2024-04-22T18:39:50.920 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.920 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.921 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.921 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.921 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.921 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 9 2024-04-22T18:39:50.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.921 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:50.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:50.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116773+0000", 2024-04-22T18:39:50.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:50.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117024+0000", 2024-04-22T18:39:50.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025191299999999999 2024-04-22T18:39:50.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117036+0000", 2024-04-22T18:39:50.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1595e-05 2024-04-22T18:39:50.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117046+0000", 2024-04-22T18:39:50.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0456e-05 2024-04-22T18:39:50.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414324+0000", 2024-04-22T18:39:50.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297277603 2024-04-22T18:39:50.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414404+0000", 2024-04-22T18:39:50.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9738000000000001e-05 2024-04-22T18:39:50.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:50.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414494+0000", 2024-04-22T18:39:50.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9733000000000003e-05 2024-04-22T18:39:50.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518194+0000", 2024-04-22T18:39:50.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10370061 2024-04-22T18:39:50.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609618+0000", 2024-04-22T18:39:50.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091423360999999995 2024-04-22T18:39:50.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609655+0000", 2024-04-22T18:39:50.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7580000000000003e-05 2024-04-22T18:39:50.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609681+0000", 2024-04-22T18:39:50.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5908e-05 2024-04-22T18:39:50.927 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.927 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.927 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.927 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:10 4.f 4:f2c48a0e:::smithi08937929-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:50.927 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:50.927 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.796092340999998, 2024-04-22T18:39:50.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48712681400000002, 2024-04-22T18:39:50.927 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.927 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.928 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.928 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.928 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.928 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 10 2024-04-22T18:39:50.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.928 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:50.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:50.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122625+0000", 2024-04-22T18:39:50.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:50.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122828+0000", 2024-04-22T18:39:50.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020359499999999999 2024-04-22T18:39:50.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122840+0000", 2024-04-22T18:39:50.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1239e-05 2024-04-22T18:39:50.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122845+0000", 2024-04-22T18:39:50.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2719999999999997e-06 2024-04-22T18:39:50.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415105+0000", 2024-04-22T18:39:50.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29225972099999997 2024-04-22T18:39:50.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415169+0000", 2024-04-22T18:39:50.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4170000000000004e-05 2024-04-22T18:39:50.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:50.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415257+0000", 2024-04-22T18:39:50.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8331999999999994e-05 2024-04-22T18:39:50.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518199+0000", 2024-04-22T18:39:50.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10294174 2024-04-22T18:39:50.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609707+0000", 2024-04-22T18:39:50.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091508408999999999 2024-04-22T18:39:50.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609729+0000", 2024-04-22T18:39:50.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1917e-05 2024-04-22T18:39:50.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609753+0000", 2024-04-22T18:39:50.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3776000000000001e-05 2024-04-22T18:39:50.934 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.934 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.934 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.934 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:11 4.f 4:f2c48a0e:::smithi08937929-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:50.934 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:50.934 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.795868968000001, 2024-04-22T18:39:50.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49452214500000002, 2024-04-22T18:39:50.934 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.934 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.935 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.935 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.935 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.935 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 11 2024-04-22T18:39:50.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.935 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:50.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:50.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122849+0000", 2024-04-22T18:39:50.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122853+0000", 2024-04-22T18:39:50.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9029999999999997e-06 2024-04-22T18:39:50.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122856+0000", 2024-04-22T18:39:50.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.01e-06 2024-04-22T18:39:50.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122859+0000", 2024-04-22T18:39:50.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:50.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415755+0000", 2024-04-22T18:39:50.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29289606800000001 2024-04-22T18:39:50.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415808+0000", 2024-04-22T18:39:50.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3415999999999998e-05 2024-04-22T18:39:50.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:50.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415888+0000", 2024-04-22T18:39:50.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9061999999999996e-05 2024-04-22T18:39:50.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518203+0000", 2024-04-22T18:39:50.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102315533 2024-04-22T18:39:50.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617164+0000", 2024-04-22T18:39:50.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098960548999999995 2024-04-22T18:39:50.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617241+0000", 2024-04-22T18:39:50.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7292000000000005e-05 2024-04-22T18:39:50.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617372+0000", 2024-04-22T18:39:50.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013075399999999999 2024-04-22T18:39:50.940 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.940 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.940 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.941 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:12 4.f 4:f2c48a0e:::smithi08937929-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:50.941 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:50.941 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.795857104, 2024-04-22T18:39:50.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49473910999999998, 2024-04-22T18:39:50.941 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.941 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:50.941 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.941 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.941 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.942 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 12 2024-04-22T18:39:50.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.942 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:50.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:50.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:50.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122863+0000", 2024-04-22T18:39:50.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.863e-06 2024-04-22T18:39:50.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122866+0000", 2024-04-22T18:39:50.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:39:50.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122869+0000", 2024-04-22T18:39:50.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5950000000000001e-06 2024-04-22T18:39:50.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416008+0000", 2024-04-22T18:39:50.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29313975199999998 2024-04-22T18:39:50.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:50.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416048+0000", 2024-04-22T18:39:50.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9888000000000002e-05 2024-04-22T18:39:50.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617447+0000", 2024-04-22T18:39:50.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20139881400000001 2024-04-22T18:39:50.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617537+0000", 2024-04-22T18:39:50.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9606999999999998e-05 2024-04-22T18:39:50.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617600+0000", 2024-04-22T18:39:50.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3884e-05 2024-04-22T18:39:50.946 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.946 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.946 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.947 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:33 4.f 4:f0ed3d49:::smithi08937929-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:50.947 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:50.947 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.731916902999998, 2024-04-22T18:39:50.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62359692099999997, 2024-04-22T18:39:50.947 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.947 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.947 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.947 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.947 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.947 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 33 2024-04-22T18:39:50.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.948 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:50.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:50.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186799+0000", 2024-04-22T18:39:50.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:50.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187103+0000", 2024-04-22T18:39:50.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030354900000000002 2024-04-22T18:39:50.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187117+0000", 2024-04-22T18:39:50.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4578e-05 2024-04-22T18:39:50.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187129+0000", 2024-04-22T18:39:50.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1421e-05 2024-04-22T18:39:50.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516134+0000", 2024-04-22T18:39:50.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32900553900000001 2024-04-22T18:39:50.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516215+0000", 2024-04-22T18:39:50.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0537999999999994e-05 2024-04-22T18:39:50.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:50.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516302+0000", 2024-04-22T18:39:50.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6601000000000007e-05 2024-04-22T18:39:50.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518207+0000", 2024-04-22T18:39:50.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019058370000000001 2024-04-22T18:39:50.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810302+0000", 2024-04-22T18:39:50.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29209509700000003 2024-04-22T18:39:50.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810348+0000", 2024-04-22T18:39:50.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5992999999999998e-05 2024-04-22T18:39:50.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810398+0000", 2024-04-22T18:39:50.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0046000000000002e-05 2024-04-22T18:39:50.953 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.953 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.953 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.953 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:34 4.f 4:f0ed3d49:::smithi08937929-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:50.953 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:50.953 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.727314890999999, 2024-04-22T18:39:50.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.61954655800000002, 2024-04-22T18:39:50.954 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.954 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.954 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.954 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.954 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.954 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34 2024-04-22T18:39:50.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.954 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:50.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:50.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191401+0000", 2024-04-22T18:39:50.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:50.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191503+0000", 2024-04-22T18:39:50.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010190400000000001 2024-04-22T18:39:50.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191517+0000", 2024-04-22T18:39:50.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3267e-05 2024-04-22T18:39:50.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191523+0000", 2024-04-22T18:39:50.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-04-22T18:39:50.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517062+0000", 2024-04-22T18:39:50.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32553894300000003 2024-04-22T18:39:50.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517141+0000", 2024-04-22T18:39:50.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9667999999999997e-05 2024-04-22T18:39:50.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:50.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517267+0000", 2024-04-22T18:39:50.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000125494 2024-04-22T18:39:50.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518213+0000", 2024-04-22T18:39:50.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000946275 2024-04-22T18:39:50.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810901+0000", 2024-04-22T18:39:50.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29268821699999997 2024-04-22T18:39:50.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810929+0000", 2024-04-22T18:39:50.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7645999999999999e-05 2024-04-22T18:39:50.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810950+0000", 2024-04-22T18:39:50.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1333999999999999e-05 2024-04-22T18:39:50.959 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.960 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.960 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.960 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:50.960 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:50.960 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.727191922999999, 2024-04-22T18:39:50.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:39:50.960 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.960 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.960 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.960 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.961 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.961 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:39:50.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.961 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:50.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:50.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:39:50.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:39:50.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:39:50.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:39:50.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:39:50.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:39:50.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:50.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:39:50.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:39:50.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:39:50.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:39:50.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:50.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:39:50.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:39:50.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:39:50.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:39:50.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:39:50.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:39:50.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:39:50.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:39:50.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:39:50.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:39:50.966 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.966 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.966 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.967 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:50.967 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:50.967 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.727180848000003, 2024-04-22T18:39:50.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:39:50.967 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.967 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:50.967 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.967 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.967 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.967 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:39:50.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.967 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:50.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:50.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:39:50.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:39:50.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:39:50.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:39:50.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:39:50.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:39:50.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:39:50.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:39:50.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:39:50.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:50.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:39:50.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:39:50.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:39:50.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:39:50.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:39:50.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:39:50.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:39:50.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:39:50.972 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.972 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.972 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.973 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:38 4.2 4:4fb26dd8:::smithi08937929-10:head [write 1893632~257631] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:50.973 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:50.973 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.715458243999997, 2024-04-22T18:39:50.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33863800199999999, 2024-04-22T18:39:50.973 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.973 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.973 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.973 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.973 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.974 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 38 2024-04-22T18:39:50.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.974 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:50.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:50.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203258+0000", 2024-04-22T18:39:50.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:50.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203396+0000", 2024-04-22T18:39:50.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013710799999999999 2024-04-22T18:39:50.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203406+0000", 2024-04-22T18:39:50.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0295000000000001e-05 2024-04-22T18:39:50.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203411+0000", 2024-04-22T18:39:50.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5079999999999996e-06 2024-04-22T18:39:50.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203450+0000", 2024-04-22T18:39:50.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8420999999999997e-05 2024-04-22T18:39:50.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:50.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203455+0000", 2024-04-22T18:39:50.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:39:50.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237071+0000", 2024-04-22T18:39:50.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.033615891000000002 2024-04-22T18:39:50.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237129+0000", 2024-04-22T18:39:50.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8134000000000003e-05 2024-04-22T18:39:50.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:50.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237221+0000", 2024-04-22T18:39:50.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2053999999999997e-05 2024-04-22T18:39:50.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.341419+0000", 2024-04-22T18:39:50.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104197568 2024-04-22T18:39:50.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541792+0000", 2024-04-22T18:39:50.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.200373312 2024-04-22T18:39:50.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541838+0000", 2024-04-22T18:39:50.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6004000000000002e-05 2024-04-22T18:39:50.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541898+0000", 2024-04-22T18:39:50.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0297000000000002e-05 2024-04-22T18:39:50.980 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.980 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.980 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.980 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:39 4.2 4:4fb26dd8:::smithi08937929-10:head [setxattr _header (56) in=7b,truncate 2151263] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:50.981 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:50.981 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.715301250000003, 2024-04-22T18:39:50.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33862203000000002, 2024-04-22T18:39:50.981 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.981 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.981 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.981 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.981 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.981 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 39 2024-04-22T18:39:50.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.981 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:50.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:50.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:50.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:50.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203423+0000", 2024-04-22T18:39:50.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0589999999999999e-06 2024-04-22T18:39:50.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203429+0000", 2024-04-22T18:39:50.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2559999999999999e-06 2024-04-22T18:39:50.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203434+0000", 2024-04-22T18:39:50.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8049999999999997e-06 2024-04-22T18:39:50.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203475+0000", 2024-04-22T18:39:50.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1205000000000003e-05 2024-04-22T18:39:50.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:50.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203478+0000", 2024-04-22T18:39:50.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7140000000000002e-06 2024-04-22T18:39:50.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239180+0000", 2024-04-22T18:39:50.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035702195999999999 2024-04-22T18:39:50.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239225+0000", 2024-04-22T18:39:50.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5722e-05 2024-04-22T18:39:50.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:50.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239291+0000", 2024-04-22T18:39:50.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5870000000000005e-05 2024-04-22T18:39:50.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:50.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342114+0000", 2024-04-22T18:39:50.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102822301 2024-04-22T18:39:50.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:50.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541946+0000", 2024-04-22T18:39:50.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.199832544 2024-04-22T18:39:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541988+0000", 2024-04-22T18:39:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1499999999999999e-05 2024-04-22T18:39:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542039+0000", 2024-04-22T18:39:50.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1576000000000002e-05 2024-04-22T18:39:50.988 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.988 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.988 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.988 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:40 4.2 4:4fb26dd8:::smithi08937929-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:50.988 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:50.988 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.715263989999997, 2024-04-22T18:39:50.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33924657200000002, 2024-04-22T18:39:50.988 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.988 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:50.988 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.988 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.989 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.989 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 40 2024-04-22T18:39:50.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.989 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:50.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:50.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:50.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:50.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203457+0000", 2024-04-22T18:39:50.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.129e-06 2024-04-22T18:39:50.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203463+0000", 2024-04-22T18:39:50.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7440000000000004e-06 2024-04-22T18:39:50.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203468+0000", 2024-04-22T18:39:50.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2619999999999998e-06 2024-04-22T18:39:50.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203491+0000", 2024-04-22T18:39:50.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3260000000000001e-05 2024-04-22T18:39:50.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:50.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203493+0000", 2024-04-22T18:39:50.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9599999999999999e-06 2024-04-22T18:39:50.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239420+0000", 2024-04-22T18:39:50.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035927212 2024-04-22T18:39:50.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:50.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239462+0000", 2024-04-22T18:39:50.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.159e-05 2024-04-22T18:39:50.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542547+0000", 2024-04-22T18:39:50.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30308492599999998 2024-04-22T18:39:50.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:50.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542611+0000", 2024-04-22T18:39:50.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4622000000000006e-05 2024-04-22T18:39:50.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:50.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542701+0000", 2024-04-22T18:39:50.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9579000000000004e-05 2024-04-22T18:39:50.994 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.994 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:50.994 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:50.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.995 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:41 4.2 4:409e6456:::smithi08937929-11:head [write 568070~468843,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:50.995 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:50.995 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.709547719, 2024-04-22T18:39:50.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33297193600000002, 2024-04-22T18:39:50.995 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:50.995 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:50.995 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:50.995 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:50.995 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:50.995 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 41 2024-04-22T18:39:50.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.995 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:50.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:50.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:50.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:50.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:50.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:50.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:50.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209168+0000", 2024-04-22T18:39:50.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:50.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:50.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209393+0000", 2024-04-22T18:39:50.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022485199999999999 2024-04-22T18:39:50.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:50.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209404+0000", 2024-04-22T18:39:50.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1209999999999999e-05 2024-04-22T18:39:50.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:50.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209411+0000", 2024-04-22T18:39:50.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8580000000000002e-06 2024-04-22T18:39:50.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209457+0000", 2024-04-22T18:39:50.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6351999999999999e-05 2024-04-22T18:39:50.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:50.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209463+0000", 2024-04-22T18:39:50.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.322e-06 2024-04-22T18:39:50.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:50.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:50.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:50.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339572+0000", 2024-04-22T18:39:51.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13010901999999999 2024-04-22T18:39:51.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:51.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339673+0000", 2024-04-22T18:39:51.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101332 2024-04-22T18:39:51.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:51.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339765+0000", 2024-04-22T18:39:51.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1780999999999994e-05 2024-04-22T18:39:51.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:51.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342117+0000", 2024-04-22T18:39:51.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002351709 2024-04-22T18:39:51.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:51.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542074+0000", 2024-04-22T18:39:51.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995749800000001 2024-04-22T18:39:51.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:51.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542112+0000", 2024-04-22T18:39:51.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7891e-05 2024-04-22T18:39:51.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:51.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542143+0000", 2024-04-22T18:39:51.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0756000000000002e-05 2024-04-22T18:39:51.002 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.002 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:51.002 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.003 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:42 4.2 4:409e6456:::smithi08937929-11:head [write 1552114~615956] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:51.003 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:51.003 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.703209409999999, 2024-04-22T18:39:51.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326735257, 2024-04-22T18:39:51.003 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:51.003 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:51.003 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:51.003 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:51.003 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:51.004 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 42 2024-04-22T18:39:51.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.004 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:51.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:51.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:51.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:51.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:51.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:51.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215508+0000", 2024-04-22T18:39:51.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:51.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:51.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215766+0000", 2024-04-22T18:39:51.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025766999999999999 2024-04-22T18:39:51.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:51.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215776+0000", 2024-04-22T18:39:51.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0772e-05 2024-04-22T18:39:51.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:51.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215785+0000", 2024-04-22T18:39:51.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6880000000000005e-06 2024-04-22T18:39:51.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215818+0000", 2024-04-22T18:39:51.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2960999999999998e-05 2024-04-22T18:39:51.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:51.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215821+0000", 2024-04-22T18:39:51.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8569999999999999e-06 2024-04-22T18:39:51.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340332+0000", 2024-04-22T18:39:51.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.124511409 2024-04-22T18:39:51.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:51.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340416+0000", 2024-04-22T18:39:51.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3793000000000002e-05 2024-04-22T18:39:51.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:51.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340543+0000", 2024-04-22T18:39:51.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012720299999999999 2024-04-22T18:39:51.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:51.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342361+0000", 2024-04-22T18:39:51.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018174370000000001 2024-04-22T18:39:51.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:51.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542176+0000", 2024-04-22T18:39:51.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19981518400000001 2024-04-22T18:39:51.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:51.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542211+0000", 2024-04-22T18:39:51.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5128e-05 2024-04-22T18:39:51.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:51.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542244+0000", 2024-04-22T18:39:51.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3256000000000001e-05 2024-04-22T18:39:51.010 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.010 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:51.010 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.011 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:43 4.2 4:409e6456:::smithi08937929-11:head [setxattr _header (56) in=7b,truncate 2168070] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:51.011 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:51.011 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.702924637000002, 2024-04-22T18:39:51.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32656540499999998, 2024-04-22T18:39:51.011 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:51.011 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:51.011 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:51.011 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:51.011 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:51.011 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 43 2024-04-22T18:39:51.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.012 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:51.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:51.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:51.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:51.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:51.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:51.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215793+0000", 2024-04-22T18:39:51.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:51.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:51.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215797+0000", 2024-04-22T18:39:51.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4190000000000004e-06 2024-04-22T18:39:51.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:51.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215804+0000", 2024-04-22T18:39:51.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:39:51.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:51.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215809+0000", 2024-04-22T18:39:51.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3689999999999998e-06 2024-04-22T18:39:51.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215840+0000", 2024-04-22T18:39:51.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-05 2024-04-22T18:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215843+0000", 2024-04-22T18:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8849999999999999e-06 2024-04-22T18:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342483+0000", 2024-04-22T18:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126640172 2024-04-22T18:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342529+0000", 2024-04-22T18:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5668999999999999e-05 2024-04-22T18:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342597+0000", 2024-04-22T18:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7538000000000003e-05 2024-04-22T18:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.347102+0000", 2024-04-22T18:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045058040000000004 2024-04-22T18:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542278+0000", 2024-04-22T18:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.195175879 2024-04-22T18:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542314+0000", 2024-04-22T18:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6134999999999999e-05 2024-04-22T18:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542359+0000", 2024-04-22T18:39:51.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4848999999999997e-05 2024-04-22T18:39:51.018 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.018 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:51.018 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.018 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:44 4.2 4:409e6456:::smithi08937929-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:51.018 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:51.018 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.702904629000003, 2024-04-22T18:39:51.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326706418, 2024-04-22T18:39:51.018 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:51.018 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:51.018 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:51.019 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:51.019 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:51.019 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 44 2024-04-22T18:39:51.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.019 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:51.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:51.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:51.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:51.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:51.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:51.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215813+0000", 2024-04-22T18:39:51.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:51.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:51.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215816+0000", 2024-04-22T18:39:51.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9799999999999998e-06 2024-04-22T18:39:51.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:51.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215822+0000", 2024-04-22T18:39:51.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5570000000000003e-06 2024-04-22T18:39:51.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:51.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215827+0000", 2024-04-22T18:39:51.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2739999999999998e-06 2024-04-22T18:39:51.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215855+0000", 2024-04-22T18:39:51.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7682999999999999e-05 2024-04-22T18:39:51.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:51.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215857+0000", 2024-04-22T18:39:51.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9250000000000002e-06 2024-04-22T18:39:51.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342717+0000", 2024-04-22T18:39:51.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126860696 2024-04-22T18:39:51.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:51.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342758+0000", 2024-04-22T18:39:51.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0295e-05 2024-04-22T18:39:51.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542408+0000", 2024-04-22T18:39:51.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19964990599999999 2024-04-22T18:39:51.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:51.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542457+0000", 2024-04-22T18:39:51.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9784999999999999e-05 2024-04-22T18:39:51.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:51.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542520+0000", 2024-04-22T18:39:51.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2960999999999996e-05 2024-04-22T18:39:51.024 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.024 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:51.024 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.024 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:51.024 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:51.025 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 53.125811354, 2024-04-22T18:39:51.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:39:51.025 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:51.025 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:51.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:51.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:51.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:51.025 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:39:51.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.025 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:51.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:51.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:51.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:51.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:51.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:51.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:39:51.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:51.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:51.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:39:51.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:39:51.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:51.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:39:51.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:39:51.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:51.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:39:51.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:39:51.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:39:51.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:39:51.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:51.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:39:51.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:39:51.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:51.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:39:51.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:39:51.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:51.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:39:51.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:39:51.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:51.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:39:51.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:39:51.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:51.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:39:51.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:39:51.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:51.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:39:51.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:39:51.030 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.031 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:51.031 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.031 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1262 4.7 4:e117a562:::smithi08937929-33:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:51.031 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:51.031 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.465668178000001, 2024-04-22T18:39:51.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40488836700000003, 2024-04-22T18:39:51.031 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:51.031 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:51.031 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:51.031 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:51.031 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:51.031 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1262 2024-04-22T18:39:51.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.032 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:51.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:51.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:51.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:51.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:51.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:51.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453048+0000", 2024-04-22T18:39:51.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:51.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:51.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453058+0000", 2024-04-22T18:39:51.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:39:51.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:51.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453069+0000", 2024-04-22T18:39:51.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1117999999999999e-05 2024-04-22T18:39:51.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:51.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453075+0000", 2024-04-22T18:39:51.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0909999999999998e-06 2024-04-22T18:39:51.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453115+0000", 2024-04-22T18:39:51.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0119e-05 2024-04-22T18:39:51.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:51.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453119+0000", 2024-04-22T18:39:51.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3610000000000003e-06 2024-04-22T18:39:51.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454814+0000", 2024-04-22T18:39:51.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00169436 2024-04-22T18:39:51.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:51.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454861+0000", 2024-04-22T18:39:51.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7184000000000001e-05 2024-04-22T18:39:51.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:51.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756925+0000", 2024-04-22T18:39:51.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30206361799999998 2024-04-22T18:39:51.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:51.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757577+0000", 2024-04-22T18:39:51.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065199599999999996 2024-04-22T18:39:51.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:51.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857846+0000", 2024-04-22T18:39:51.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100269369 2024-04-22T18:39:51.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:51.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857896+0000", 2024-04-22T18:39:51.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9607000000000001e-05 2024-04-22T18:39:51.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:51.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857939+0000", 2024-04-22T18:39:51.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3149999999999999e-05 2024-04-22T18:39:51.038 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.038 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:51.038 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.038 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1689 3.0 3:1d4bcadf:::e7eb8db0531d9f456e91fe87bc79e47ebd80459f1783dc2a5e53d4822cd34f49:head [write 0~1066496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:39:51.038 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.607920+0000", 2024-04-22T18:39:51.038 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.310798804000001, 2024-04-22T18:39:51.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30339939300000002, 2024-04-22T18:39:51.039 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:51.039 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:51.039 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:51.039 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:51.039 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:51.039 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1689 2024-04-22T18:39:51.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.039 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:51.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:51.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.607920+0000", 2024-04-22T18:39:51.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:51.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.607920+0000", 2024-04-22T18:39:51.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:51.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.607917+0000", 2024-04-22T18:39:51.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:51.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:51.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.608327+0000", 2024-04-22T18:39:51.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000409508 2024-04-22T18:39:51.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:51.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.608344+0000", 2024-04-22T18:39:51.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6846000000000001e-05 2024-04-22T18:39:51.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:51.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.608351+0000", 2024-04-22T18:39:51.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8140000000000003e-06 2024-04-22T18:39:51.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708489+0000", 2024-04-22T18:39:51.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100138536 2024-04-22T18:39:51.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:51.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708651+0000", 2024-04-22T18:39:51.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000162132 2024-04-22T18:39:51.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:39:51.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708793+0000", 2024-04-22T18:39:51.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014147699999999999 2024-04-22T18:39:51.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:51.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.713125+0000", 2024-04-22T18:39:51.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043321369999999998 2024-04-22T18:39:51.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:51.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911261+0000", 2024-04-22T18:39:51.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198135744 2024-04-22T18:39:51.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:51.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911293+0000", 2024-04-22T18:39:51.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1970000000000001e-05 2024-04-22T18:39:51.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:51.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911319+0000", 2024-04-22T18:39:51.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6440000000000001e-05 2024-04-22T18:39:51.045 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.045 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:51.045 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.045 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1872 4.2 4:409e6456:::smithi08937929-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:51.045 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:35.576858+0000", 2024-04-22T18:39:51.045 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 15.341860050999999, 2024-04-22T18:39:51.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.302951992, 2024-04-22T18:39:51.046 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:51.046 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:51.046 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:51.046 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:51.046 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:51.046 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1872 2024-04-22T18:39:51.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.046 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:51.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:51.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576858+0000", 2024-04-22T18:39:51.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:51.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576858+0000", 2024-04-22T18:39:51.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:51.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576857+0000", 2024-04-22T18:39:51.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:51.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:51.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576864+0000", 2024-04-22T18:39:51.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4569999999999999e-06 2024-04-22T18:39:51.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:51.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576871+0000", 2024-04-22T18:39:51.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1679999999999998e-06 2024-04-22T18:39:51.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:51.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576876+0000", 2024-04-22T18:39:51.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0440000000000003e-06 2024-04-22T18:39:51.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.590050+0000", 2024-04-22T18:39:51.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.013174208 2024-04-22T18:39:51.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:51.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.590142+0000", 2024-04-22T18:39:51.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1681000000000005e-05 2024-04-22T18:39:51.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.879685+0000", 2024-04-22T18:39:51.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.28954336200000003 2024-04-22T18:39:51.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:51.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.879745+0000", 2024-04-22T18:39:51.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9487999999999997e-05 2024-04-22T18:39:51.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:51.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.879810+0000", 2024-04-22T18:39:51.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5431000000000004e-05 2024-04-22T18:39:51.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.051 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:51.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.051 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2205 4.2 4:4fb26dd8:::smithi08937929-10:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:39:51.051 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:39:51.051 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.5281230770000001, 2024-04-22T18:39:51.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.413791136, 2024-04-22T18:39:51.052 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:51.052 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:39:51.052 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:51.052 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:51.052 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:51.052 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2205 2024-04-22T18:39:51.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.052 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:51.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:51.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:39:51.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:51.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:39:51.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:51.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:51.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:39:51.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:51.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:51.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390598+0000", 2024-04-22T18:39:51.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7630000000000001e-06 2024-04-22T18:39:51.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:51.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390603+0000", 2024-04-22T18:39:51.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.293e-06 2024-04-22T18:39:51.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:51.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390608+0000", 2024-04-22T18:39:51.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8790000000000002e-06 2024-04-22T18:39:51.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390653+0000", 2024-04-22T18:39:51.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5661999999999998e-05 2024-04-22T18:39:51.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:51.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390657+0000", 2024-04-22T18:39:51.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9709999999999998e-06 2024-04-22T18:39:51.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.592250+0000", 2024-04-22T18:39:51.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20159280600000001 2024-04-22T18:39:51.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:51.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.592322+0000", 2024-04-22T18:39:51.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2171999999999999e-05 2024-04-22T18:39:51.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:51.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.804373+0000", 2024-04-22T18:39:51.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.212050297 2024-04-22T18:39:51.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:51.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:51.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:51.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.804387+0000", 2024-04-22T18:39:51.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3859999999999999e-05 2024-04-22T18:39:51.057 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.057 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:51.057 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.058 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:51.058 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:51.058 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:51.059 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:51.060 INFO:tasks.rados.rados.0.smithi089.stdout:1142: finishing write tid 2 to smithi08937929-30 2024-04-22T18:39:51.060 INFO:tasks.rados.rados.0.smithi089.stdout:1142: finishing write tid 3 to smithi08937929-30 2024-04-22T18:39:51.060 INFO:tasks.rados.rados.0.smithi089.stdout:1142: finishing write tid 4 to smithi08937929-30 2024-04-22T18:39:51.060 INFO:tasks.rados.rados.0.smithi089.stdout:1142: finishing write tid 5 to smithi08937929-30 2024-04-22T18:39:51.060 INFO:tasks.rados.rados.0.smithi089.stdout:1142: oid 30 updating version 0 to 34 2024-04-22T18:39:51.060 INFO:tasks.rados.rados.0.smithi089.stdout:1142: oid 30 version 34 is already newer than 33 2024-04-22T18:39:51.061 INFO:tasks.rados.rados.0.smithi089.stdout:1142: oid 30 updating version 34 to 36 2024-04-22T18:39:51.061 INFO:tasks.rados.rados.0.smithi089.stdout:1142: oid 30 version 36 is already newer than 35 2024-04-22T18:39:51.061 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 36 (ObjNum 203 snap 0 seq_num 203) dirty exists 2024-04-22T18:39:51.061 INFO:tasks.rados.rados.0.smithi089.stdout:1142: left oid 30 (ObjNum 203 snap 0 seq_num 203) 2024-04-22T18:39:51.061 INFO:tasks.rados.rados.0.smithi089.stdout:1145: finishing write tid 1 to smithi08937929-25 2024-04-22T18:39:51.061 INFO:tasks.rados.rados.0.smithi089.stdout:1149: got (2) No such file or directory 2024-04-22T18:39:51.061 INFO:tasks.rados.rados.0.smithi089.stdout:1149: got expected ENOENT (src dne) 2024-04-22T18:39:51.061 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 107 (ObjNum 603697 snap 0 seq_num 2411296176) clean dne 2024-04-22T18:39:51.061 INFO:tasks.rados.rados.0.smithi089.stdout:1144: finishing set_chunk to oid 2 2024-04-22T18:39:51.061 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 44 (ObjNum 182 snap 0 seq_num 182) dirty exists 2024-04-22T18:39:51.061 INFO:tasks.rados.rados.0.smithi089.stdout:1146: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:39:51.062 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:51.059+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:51.085 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:39:51.135 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:51.136+0000 7f3cfee88640 -1 osd.6 62 handle_osd_map injecting map crc failure 2024-04-22T18:39:51.140 INFO:tasks.rados.rados.0.smithi089.stdout:1145: finishing write tid 2 to smithi08937929-25 2024-04-22T18:39:51.140 INFO:tasks.rados.rados.0.smithi089.stdout:1145: finishing write tid 3 to smithi08937929-25 2024-04-22T18:39:51.140 INFO:tasks.rados.rados.0.smithi089.stdout:1151: finishing set_chunk to oid 8 2024-04-22T18:39:51.140 INFO:tasks.rados.rados.0.smithi089.stdout:1151: got expected ENOENT (src dne) 2024-04-22T18:39:51.140 INFO:tasks.rados.rados.0.smithi089.stdout:1150: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:39:51.181 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:51.181+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:51.206 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:51.207 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:51.207 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:51.207 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:51.216 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:51.220 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:51.220 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:51.238 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:51.238 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:51.255 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:51.256 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:51.260 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:51.260+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:51.271 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:51.272 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:51.288 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:51.289 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:51.292 INFO:tasks.rados.rados.0.smithi089.stdout:1152: expect (ObjNum 140 snap 0 seq_num 140) 2024-04-22T18:39:51.305 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:51.306 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:51.322 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:51.322 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:51.323 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:51.323 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:51.332 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:51.339 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:51.339 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:51.392 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:51.392+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:51.418 INFO:tasks.rados.rados.0.smithi089.stdout:1153: finishing write tid 1 to smithi08937929-34 2024-04-22T18:39:51.418 INFO:tasks.rados.rados.0.smithi089.stdout:1145: finishing write tid 4 to smithi08937929-25 2024-04-22T18:39:51.418 INFO:tasks.rados.rados.0.smithi089.stdout:1141: done (13 left) 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1142: done (12 left) 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1143: done (11 left) 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1144: done (10 left) 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1146: done (9 left) 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1147: done (8 left) 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1148: done (7 left) 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1149: done (6 left) 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1150: done (5 left) 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1151: done (4 left) 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1152: done (3 left) 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1155: read oid 8 snap -1 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1155: expect deleted 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1156: write (excl) oid 13 current snap is 0 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1154: got (22) Invalid argument 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1145: finishing write tid 5 to smithi08937929-25 2024-04-22T18:39:51.419 INFO:tasks.rados.rados.0.smithi089.stdout:1145: oid 25 updating version 0 to 65 2024-04-22T18:39:51.420 INFO:tasks.rados.rados.0.smithi089.stdout:1145: oid 25 updating version 65 to 67 2024-04-22T18:39:51.420 INFO:tasks.rados.rados.0.smithi089.stdout:1145: oid 25 version 67 is already newer than 64 2024-04-22T18:39:51.420 INFO:tasks.rados.rados.0.smithi089.stdout:1145: oid 25 version 67 is already newer than 66 2024-04-22T18:39:51.420 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 67 (ObjNum 204 snap 0 seq_num 204) dirty exists 2024-04-22T18:39:51.420 INFO:tasks.rados.rados.0.smithi089.stdout:1145: left oid 25 (ObjNum 204 snap 0 seq_num 204) 2024-04-22T18:39:51.420 INFO:tasks.rados.rados.0.smithi089.stdout:1153: finishing write tid 2 to smithi08937929-34 2024-04-22T18:39:51.420 INFO:tasks.rados.rados.0.smithi089.stdout:1153: finishing write tid 3 to smithi08937929-34 2024-04-22T18:39:51.420 INFO:tasks.rados.rados.0.smithi089.stdout:1156: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2024-04-22T18:39:51.424 INFO:tasks.rados.rados.0.smithi089.stdout:1156: writing smithi08937929-13 from 662242 to 1305067 tid 1 2024-04-22T18:39:51.431 INFO:tasks.rados.rados.0.smithi089.stdout:1156: writing smithi08937929-13 from 2054820 to 2585550 tid 2 2024-04-22T18:39:51.436 INFO:tasks.rados.rados.0.smithi089.stdout:1156: writing smithi08937929-13 from 3307607 to 3462242 tid 3 2024-04-22T18:39:51.436 INFO:tasks.rados.rados.0.smithi089.stdout:1145: done (4 left) 2024-04-22T18:39:51.436 INFO:tasks.rados.rados.0.smithi089.stdout:1154: done (3 left) 2024-04-22T18:39:51.436 INFO:tasks.rados.rados.0.smithi089.stdout:1157: write (excl) oid 43 current snap is 0 2024-04-22T18:39:51.437 INFO:tasks.rados.rados.0.smithi089.stdout:1157: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2024-04-22T18:39:51.441 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:51.441 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:51.443 INFO:tasks.rados.rados.0.smithi089.stdout:1157: writing smithi08937929-43 from 679049 to 1397123 tid 1 2024-04-22T18:39:51.450 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:51.451 INFO:tasks.rados.rados.0.smithi089.stdout:1157: writing smithi08937929-43 from 2113302 to 2787690 tid 2 2024-04-22T18:39:51.455 INFO:tasks.rados.rados.0.smithi089.stdout:1157: writing smithi08937929-43 from 3335030 to 3479049 tid 3 2024-04-22T18:39:51.455 INFO:tasks.rados.rados.0.smithi089.stdout:1158: write oid 36 current snap is 0 2024-04-22T18:39:51.455 INFO:tasks.rados.rados.0.smithi089.stdout:1153: finishing write tid 4 to smithi08937929-34 2024-04-22T18:39:51.455 INFO:tasks.rados.rados.0.smithi089.stdout:1158: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2024-04-22T18:39:51.462 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:51.462+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:51.463 INFO:tasks.rados.rados.0.smithi089.stdout:1158: writing smithi08937929-36 from 695856 to 1489179 tid 1 2024-04-22T18:39:51.469 INFO:tasks.rados.rados.0.smithi089.stdout:1158: writing smithi08937929-36 from 2171784 to 2706183 tid 2 2024-04-22T18:39:51.475 INFO:tasks.rados.rados.0.smithi089.stdout:1158: writing smithi08937929-36 from 3362453 to 3495856 tid 3 2024-04-22T18:39:51.475 INFO:tasks.rados.rados.0.smithi089.stdout:1159: read oid 27 snap -1 2024-04-22T18:39:51.475 INFO:tasks.rados.rados.0.smithi089.stdout:1159: expect (ObjNum 160 snap 0 seq_num 160) 2024-04-22T18:39:51.475 INFO:tasks.rados.rados.0.smithi089.stdout:1160: read oid 3 snap -1 2024-04-22T18:39:51.475 INFO:tasks.rados.rados.0.smithi089.stdout:1160: expect (ObjNum 200 snap 0 seq_num 200) 2024-04-22T18:39:51.476 INFO:tasks.rados.rados.0.smithi089.stdout:1161: read oid 46 snap -1 2024-04-22T18:39:51.476 INFO:tasks.rados.rados.0.smithi089.stdout:1161: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-22T18:39:51.476 INFO:tasks.rados.rados.0.smithi089.stdout:1162: read oid 38 snap -1 2024-04-22T18:39:51.476 INFO:tasks.rados.rados.0.smithi089.stdout:1162: expect (ObjNum 97 snap 0 seq_num 97) 2024-04-22T18:39:51.476 INFO:tasks.rados.rados.0.smithi089.stdout:1163: read oid 10 snap -1 2024-04-22T18:39:51.476 INFO:tasks.rados.rados.0.smithi089.stdout:1163: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:39:51.476 INFO:tasks.rados.rados.0.smithi089.stdout:1164: read oid 21 snap -1 2024-04-22T18:39:51.476 INFO:tasks.rados.rados.0.smithi089.stdout:1164: expect (ObjNum 168 snap 0 seq_num 168) 2024-04-22T18:39:51.476 INFO:tasks.rados.rados.0.smithi089.stdout:1165: read oid 41 snap -1 2024-04-22T18:39:51.476 INFO:tasks.rados.rados.0.smithi089.stdout:1165: expect (ObjNum 136 snap 0 seq_num 136) 2024-04-22T18:39:51.476 INFO:tasks.rados.rados.0.smithi089.stdout:1166: read oid 17 snap -1 2024-04-22T18:39:51.476 INFO:tasks.rados.rados.0.smithi089.stdout:1166: expect (ObjNum 192 snap 0 seq_num 192) 2024-04-22T18:39:51.476 INFO:tasks.rados.rados.0.smithi089.stdout:1167: tier_promote oid 14 2024-04-22T18:39:51.476 INFO:tasks.rados.rados.0.smithi089.stdout:1153: finishing write tid 5 to smithi08937929-34 2024-04-22T18:39:51.477 INFO:tasks.rados.rados.0.smithi089.stdout:1153: oid 34 updating version 0 to 82 2024-04-22T18:39:51.477 INFO:tasks.rados.rados.0.smithi089.stdout:1153: oid 34 version 82 is already newer than 81 2024-04-22T18:39:51.477 INFO:tasks.rados.rados.0.smithi089.stdout:1153: oid 34 updating version 82 to 83 2024-04-22T18:39:51.477 INFO:tasks.rados.rados.0.smithi089.stdout:1153: oid 34 version 83 is already newer than 80 2024-04-22T18:39:51.477 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 83 (ObjNum 205 snap 0 seq_num 205) dirty exists 2024-04-22T18:39:51.477 INFO:tasks.rados.rados.0.smithi089.stdout:1153: left oid 34 (ObjNum 205 snap 0 seq_num 205) 2024-04-22T18:39:51.477 INFO:tasks.rados.rados.0.smithi089.stdout:1158: finishing write tid 1 to smithi08937929-36 2024-04-22T18:39:51.477 INFO:tasks.rados.rados.0.smithi089.stdout:1153: done (13 left) 2024-04-22T18:39:51.477 INFO:tasks.rados.rados.0.smithi089.stdout:1168: set_chunk oid 50 target oid 2024-04-22T18:39:51.553 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:51.554 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:51.562 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:51.563+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:51.563 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:51.662 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:51.663+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:51.706 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:51.706 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:51.715 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:51.726 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:39:51.727 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:51.726+0000 7f3e7be88640 -1 osd.0 63 handle_osd_map injecting map crc failure 2024-04-22T18:39:51.732 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:51.733+0000 7fa768d81640 -1 osd.5 63 handle_osd_map injecting map crc failure 2024-04-22T18:39:51.763 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:51.764+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:51.821 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:51.821 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:51.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:51.864 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:51.865+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:51.940 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:51.940 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:51.941 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:51.942+0000 7f3cfee88640 -1 osd.6 63 handle_osd_map injecting map crc failure 2024-04-22T18:39:51.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:51.993 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:51.994+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:52.052 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:52.052 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:52.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:52.066 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:52.066+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:52.150 INFO:tasks.rados.rados.0.smithi089.stdout:1168: set_chunk oid 50 offset: 1643520 length: 421888 target oid cc67062468f36dba156369cdc59a2d804c4b4d000df5e55b1bb4f3703ea47e98 offset: 0 snap -1 2024-04-22T18:39:52.150 INFO:tasks.rados.rados.0.smithi089.stdout:1166: expect (ObjNum 192 snap 0 seq_num 192) 2024-04-22T18:39:52.166 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:52.167+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:52.203 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:52.204 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:52.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:52.267 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:52.268+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:52.287 INFO:tasks.rados.rados.0.smithi089.stdout:1157: finishing write tid 1 to smithi08937929-43 2024-04-22T18:39:52.287 INFO:tasks.rados.rados.0.smithi089.stdout:1157: finishing write tid 2 to smithi08937929-43 2024-04-22T18:39:52.287 INFO:tasks.rados.rados.0.smithi089.stdout:1157: finishing write tid 3 to smithi08937929-43 2024-04-22T18:39:52.287 INFO:tasks.rados.rados.0.smithi089.stdout:1160: expect (ObjNum 200 snap 0 seq_num 200) 2024-04-22T18:39:52.314 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:52.314 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:52.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:52.368 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:52.369+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:52.423 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:52.424 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:52.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:52.443 INFO:tasks.rados.rados.0.smithi089.stdout:1158: finishing write tid 2 to smithi08937929-36 2024-04-22T18:39:52.443 INFO:tasks.rados.rados.0.smithi089.stdout:1155: done (13 left) 2024-04-22T18:39:52.444 INFO:tasks.rados.rados.0.smithi089.stdout:1160: done (12 left) 2024-04-22T18:39:52.444 INFO:tasks.rados.rados.0.smithi089.stdout:1166: done (11 left) 2024-04-22T18:39:52.444 INFO:tasks.rados.rados.0.smithi089.stdout:1169: read oid 25 snap -1 2024-04-22T18:39:52.444 INFO:tasks.rados.rados.0.smithi089.stdout:1169: expect (ObjNum 204 snap 0 seq_num 204) 2024-04-22T18:39:52.444 INFO:tasks.rados.rados.0.smithi089.stdout:1170: read oid 30 snap -1 2024-04-22T18:39:52.444 INFO:tasks.rados.rados.0.smithi089.stdout:1170: expect (ObjNum 203 snap 0 seq_num 203) 2024-04-22T18:39:52.444 INFO:tasks.rados.rados.0.smithi089.stdout:1171: set_chunk oid 11 target oid 2024-04-22T18:39:52.444 INFO:tasks.rados.rados.0.smithi089.stdout:1156: finishing write tid 1 to smithi08937929-13 2024-04-22T18:39:52.444 INFO:tasks.rados.rados.0.smithi089.stdout:1157: finishing write tid 4 to smithi08937929-43 2024-04-22T18:39:52.444 INFO:tasks.rados.rados.0.smithi089.stdout:1157: finishing write tid 5 to smithi08937929-43 2024-04-22T18:39:52.444 INFO:tasks.rados.rados.0.smithi089.stdout:1157: oid 43 updating version 0 to 33 2024-04-22T18:39:52.444 INFO:tasks.rados.rados.0.smithi089.stdout:1157: oid 43 version 33 is already newer than 32 2024-04-22T18:39:52.444 INFO:tasks.rados.rados.0.smithi089.stdout:1157: oid 43 updating version 33 to 34 2024-04-22T18:39:52.444 INFO:tasks.rados.rados.0.smithi089.stdout:1157: oid 43 version 34 is already newer than 31 2024-04-22T18:39:52.445 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 34 (ObjNum 207 snap 0 seq_num 207) dirty exists 2024-04-22T18:39:52.445 INFO:tasks.rados.rados.0.smithi089.stdout:1157: left oid 43 (ObjNum 207 snap 0 seq_num 207) 2024-04-22T18:39:52.445 INFO:tasks.rados.rados.0.smithi089.stdout:1158: finishing write tid 3 to smithi08937929-36 2024-04-22T18:39:52.445 INFO:tasks.rados.rados.0.smithi089.stdout:1159: expect (ObjNum 160 snap 0 seq_num 160) 2024-04-22T18:39:52.469 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:52.469+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:52.570 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:52.570+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:52.573 INFO:tasks.rados.rados.0.smithi089.stdout:1161: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-22T18:39:52.573 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:52.573 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:52.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:52.659 INFO:tasks.rados.rados.0.smithi089.stdout:1165: expect (ObjNum 136 snap 0 seq_num 136) 2024-04-22T18:39:52.671 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:52.672+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:52.725 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:52.725 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:52.730 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:39:52.735 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:52.756 INFO:tasks.rados.rados.0.smithi089.stdout:1156: finishing write tid 2 to smithi08937929-13 2024-04-22T18:39:52.757 INFO:tasks.rados.rados.0.smithi089.stdout:1156: finishing write tid 3 to smithi08937929-13 2024-04-22T18:39:52.757 INFO:tasks.rados.rados.0.smithi089.stdout:1156: finishing write tid 4 to smithi08937929-13 2024-04-22T18:39:52.772 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:52.772+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:52.872 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:52.874+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:52.876 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:52.876 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:52.885 INFO:tasks.rados.rados.0.smithi089.stdout:1171: set_chunk oid 11 offset: 2622976 length: 489984 target oid 40eac48652b5846f4338f6c6e4dc17848e5110e5d10851cd5309bac2ec231da0 offset: 0 snap -1 2024-04-22T18:39:52.885 INFO:tasks.rados.rados.0.smithi089.stdout:1157: done (13 left) 2024-04-22T18:39:52.885 INFO:tasks.rados.rados.0.smithi089.stdout:1159: done (12 left) 2024-04-22T18:39:52.885 INFO:tasks.rados.rados.0.smithi089.stdout:1161: done (11 left) 2024-04-22T18:39:52.885 INFO:tasks.rados.rados.0.smithi089.stdout:1165: done (10 left) 2024-04-22T18:39:52.885 INFO:tasks.rados.rados.0.smithi089.stdout:1172: set_chunk oid 31 target oid 2024-04-22T18:39:52.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:52.974 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:52.975+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:52.987 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:52.987 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:52.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:53.080 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:53.081+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:53.091 INFO:tasks.rados.rados.0.smithi089.stdout:1172: set_chunk oid 31 offset: 62976 length: 850944 target oid b4288df75b8148ed492787eb76c8ece0c94303d4960b7832674dbea657b7d216 offset: 0 snap -1 2024-04-22T18:39:53.091 INFO:tasks.rados.rados.0.smithi089.stdout:1173: set_chunk oid 19 target oid 2024-04-22T18:39:53.137 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:53.138 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:53.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:53.188 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:53.189+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:53.277 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:53.278+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:53.288 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:53.288 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:53.294 INFO:tasks.rados.rados.0.smithi089.stdout:1173: set_chunk oid 19 offset: 984064 length: 115712 target oid 728af1051a21755eaf20a51646c6212806eefeec76ba817ad4f1f49df73c9f14 offset: 0 snap -1 2024-04-22T18:39:53.294 INFO:tasks.rados.rados.0.smithi089.stdout:1174: read oid 45 snap -1 2024-04-22T18:39:53.295 INFO:tasks.rados.rados.0.smithi089.stdout:1174: expect (ObjNum 166 snap 0 seq_num 166) 2024-04-22T18:39:53.295 INFO:tasks.rados.rados.0.smithi089.stdout:1175: set_chunk oid 44 target oid 2024-04-22T18:39:53.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:53.351 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:39:53.394 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:53.395+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:53.440 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:53.440 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:53.448 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:53.470 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:53.470 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:53.477 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:53.478+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:53.487 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:53.487 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:53.509 INFO:tasks.rados.rados.0.smithi089.stdout:1175: set_chunk oid 44 offset: 1474048 length: 1528320 target oid da47ce2e5fe20e35f6c34fab2040ae87cca2fc9a81a834505a750fc83b0ea6a8 offset: 0 snap -1 2024-04-22T18:39:53.510 INFO:tasks.rados.rados.0.smithi089.stdout:1176: read oid 29 snap -1 2024-04-22T18:39:53.510 INFO:tasks.rados.rados.0.smithi089.stdout:1176: expect (ObjNum 149 snap 0 seq_num 149) 2024-04-22T18:39:53.510 INFO:tasks.rados.rados.0.smithi089.stdout:1177: read oid 12 snap -1 2024-04-22T18:39:53.510 INFO:tasks.rados.rados.0.smithi089.stdout:1177: expect (ObjNum 141 snap 0 seq_num 141) 2024-04-22T18:39:53.510 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:53.510 INFO:tasks.rados.rados.0.smithi089.stdout:1156: finishing write tid 5 to smithi08937929-13 2024-04-22T18:39:53.510 INFO:tasks.rados.rados.0.smithi089.stdout:1156: oid 13 updating version 0 to 55 2024-04-22T18:39:53.510 INFO:tasks.rados.rados.0.smithi089.stdout:1156: oid 13 version 55 is already newer than 54 2024-04-22T18:39:53.510 INFO:tasks.rados.rados.0.smithi089.stdout:1156: oid 13 version 55 is already newer than 53 2024-04-22T18:39:53.510 INFO:tasks.rados.rados.0.smithi089.stdout:1156: oid 13 version 55 is already newer than 52 2024-04-22T18:39:53.510 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 55 (ObjNum 206 snap 0 seq_num 206) dirty exists 2024-04-22T18:39:53.510 INFO:tasks.rados.rados.0.smithi089.stdout:1156: left oid 13 (ObjNum 206 snap 0 seq_num 206) 2024-04-22T18:39:53.510 INFO:tasks.rados.rados.0.smithi089.stdout:1162: expect (ObjNum 97 snap 0 seq_num 97) 2024-04-22T18:39:53.528 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:53.528 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:53.549 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:53.550 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:53.566 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:53.567 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2478 4.b 4:defe92fb:::smithi08937929-44:head [assert-version v85,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:53.511375+0000", 2024-04-22T18:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.058177321999999997, 2024-04-22T18:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058212451999999998, 2024-04-22T18:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2478 2024-04-22T18:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.568 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511375+0000", 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511375+0000", 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511372+0000", 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511386+0000", 2024-04-22T18:39:53.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4056000000000001e-05 2024-04-22T18:39:53.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511403+0000", 2024-04-22T18:39:53.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6793e-05 2024-04-22T18:39:53.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511412+0000", 2024-04-22T18:39:53.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0899999999999994e-06 2024-04-22T18:39:53.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511484+0000", 2024-04-22T18:39:53.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1594999999999995e-05 2024-04-22T18:39:53.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511577+0000", 2024-04-22T18:39:53.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3468999999999996e-05 2024-04-22T18:39:53.571 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.571 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.571 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.571 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2479 4.b 4.c77160cb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:53.511437+0000", 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.058114752999999998, 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058217671999999998, 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2479 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511437+0000", 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511437+0000", 2024-04-22T18:39:53.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511437+0000", 2024-04-22T18:39:53.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:53.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511442+0000", 2024-04-22T18:39:53.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5500000000000002e-06 2024-04-22T18:39:53.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511456+0000", 2024-04-22T18:39:53.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.362e-05 2024-04-22T18:39:53.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511463+0000", 2024-04-22T18:39:53.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1300000000000003e-06 2024-04-22T18:39:53.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.574 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2480 4.b 4.c77160cb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:53.511478+0000", 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.058074654000000003, 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058214052000000002, 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2480 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511478+0000", 2024-04-22T18:39:53.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511478+0000", 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511477+0000", 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511481+0000", 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3170000000000003e-06 2024-04-22T18:39:53.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511486+0000", 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1850000000000003e-06 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511491+0000", 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0050000000000004e-06 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2481 4.b 4.c77160cb (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:53.511496+0000", 2024-04-22T18:39:53.577 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.058056654999999999, 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058249785999999998, 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2481 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511496+0000", 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511496+0000", 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511495+0000", 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511498+0000", 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8150000000000002e-06 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511502+0000", 2024-04-22T18:39:53.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4710000000000001e-06 2024-04-22T18:39:53.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.511507+0000", 2024-04-22T18:39:53.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0150000000000003e-06 2024-04-22T18:39:53.580 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.580 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.580 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.580 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.580 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:39:53.580 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 4 2024-04-22T18:39:53.580 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:53.581 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:53.582 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:53.582+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:53.583 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:53.583 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:53.599 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:53.600 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:53.618 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:53.618 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:53.682 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:53.683+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:53.686 INFO:tasks.rados.rados.0.smithi089.stdout:1163: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:39:53.689 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:53.689 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:53.698 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:53.758 INFO:tasks.rados.rados.0.smithi089.stdout:1158: finishing write tid 4 to smithi08937929-36 2024-04-22T18:39:53.759 INFO:tasks.rados.rados.0.smithi089.stdout:1158: finishing write tid 5 to smithi08937929-36 2024-04-22T18:39:53.759 INFO:tasks.rados.rados.0.smithi089.stdout:1158: oid 36 updating version 0 to 50 2024-04-22T18:39:53.759 INFO:tasks.rados.rados.0.smithi089.stdout:1158: oid 36 version 50 is already newer than 49 2024-04-22T18:39:53.759 INFO:tasks.rados.rados.0.smithi089.stdout:1158: oid 36 updating version 50 to 51 2024-04-22T18:39:53.759 INFO:tasks.rados.rados.0.smithi089.stdout:1158: oid 36 version 51 is already newer than 48 2024-04-22T18:39:53.759 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 51 (ObjNum 208 snap 0 seq_num 208) dirty exists 2024-04-22T18:39:53.759 INFO:tasks.rados.rados.0.smithi089.stdout:1158: left oid 36 (ObjNum 208 snap 0 seq_num 208) 2024-04-22T18:39:53.759 INFO:tasks.rados.rados.0.smithi089.stdout:1164: expect (ObjNum 168 snap 0 seq_num 168) 2024-04-22T18:39:53.783 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:53.783+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:53.805 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:53.805 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:53.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:53.805 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:53.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.805 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:61 4.b 4:d923883e:::smithi08937929-16:head [write 652105~445088,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:53.805 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:53.805 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.534920071999998, 2024-04-22T18:39:53.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.38578178600000002, 2024-04-22T18:39:53.805 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.805 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 61 2024-04-22T18:39:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270195+0000", 2024-04-22T18:39:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270192+0000", 2024-04-22T18:39:53.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:53.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270401+0000", 2024-04-22T18:39:53.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020921199999999999 2024-04-22T18:39:53.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270415+0000", 2024-04-22T18:39:53.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4191e-05 2024-04-22T18:39:53.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270422+0000", 2024-04-22T18:39:53.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7329999999999997e-06 2024-04-22T18:39:53.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270461+0000", 2024-04-22T18:39:53.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9530999999999998e-05 2024-04-22T18:39:53.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270550+0000", 2024-04-22T18:39:53.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8508e-05 2024-04-22T18:39:53.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:53.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.270648+0000", 2024-04-22T18:39:53.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8377999999999999e-05 2024-04-22T18:39:53.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:53.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.271907+0000", 2024-04-22T18:39:53.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012589599999999999 2024-04-22T18:39:53.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:53.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655904+0000", 2024-04-22T18:39:53.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.383997115 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655933+0000", 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9011999999999999e-05 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.655977+0000", 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3578000000000002e-05 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.810 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:53.811 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:53.811 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.349758471000001, 2024-04-22T18:39:53.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:39:53.811 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.811 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:53.811 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.811 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.811 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.811 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:39:53.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.811 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:53.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:39:53.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:39:53.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:53.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:39:53.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:39:53.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:39:53.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:39:53.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:39:53.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:39:53.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:39:53.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:39:53.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:39:53.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:39:53.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:53.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:39:53.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:39:53.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:53.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:39:53.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:39:53.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:53.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:39:53.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:39:53.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:53.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:39:53.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:39:53.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:39:53.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:39:53.816 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.816 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.816 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.816 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:315 4.b 4:d923883e:::smithi08937929-16:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:53.816 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:53.816 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.813295691, 2024-04-22T18:39:53.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40309447300000001, 2024-04-22T18:39:53.816 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.816 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:53.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 315 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991817+0000", 2024-04-22T18:39:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991825+0000", 2024-04-22T18:39:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7279999999999995e-06 2024-04-22T18:39:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991834+0000", 2024-04-22T18:39:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1980000000000007e-06 2024-04-22T18:39:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991841+0000", 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0740000000000004e-06 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991881+0000", 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991940+0000", 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9382000000000002e-05 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394250+0000", 2024-04-22T18:39:53.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40230947700000003 2024-04-22T18:39:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394752+0000", 2024-04-22T18:39:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050269699999999997 2024-04-22T18:39:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394865+0000", 2024-04-22T18:39:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011267700000000001 2024-04-22T18:39:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394892+0000", 2024-04-22T18:39:53.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7535999999999999e-05 2024-04-22T18:39:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394914+0000", 2024-04-22T18:39:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1398000000000001e-05 2024-04-22T18:39:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:774 4.b 4:defe92fb:::smithi08937929-44:head [write 2778819~95145] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:39:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.656786015000002, 2024-04-22T18:39:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40475725000000001, 2024-04-22T18:39:53.821 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 774 2024-04-22T18:39:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:39:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148328+0000", 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148392+0000", 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3323000000000003e-05 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148398+0000", 2024-04-22T18:39:53.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4420000000000003e-06 2024-04-22T18:39:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148409+0000", 2024-04-22T18:39:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509999999999999e-05 2024-04-22T18:39:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251407+0000", 2024-04-22T18:39:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102998305 2024-04-22T18:39:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251500+0000", 2024-04-22T18:39:53.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3607e-05 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251648+0000", 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014788099999999999 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256149+0000", 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045010980000000003 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553016+0000", 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29686620800000002 2024-04-22T18:39:53.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553054+0000", 2024-04-22T18:39:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8414999999999997e-05 2024-04-22T18:39:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553086+0000", 2024-04-22T18:39:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2236000000000001e-05 2024-04-22T18:39:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.826 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.656701222999999, 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:39:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:39:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:39:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:39:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:39:53.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:39:53.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:53.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:39:53.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:39:53.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:53.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:39:53.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:39:53.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.656682746999998, 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:39:53.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:39:53.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:39:53.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:39:53.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:39:53.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:39:53.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:39:53.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:39:53.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1242 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.063437198000003, 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40662891899999998, 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:53.836 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.837 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.837 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.837 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1242 2024-04-22T18:39:53.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.837 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:39:53.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:39:53.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741677+0000", 2024-04-22T18:39:53.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:53.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741680+0000", 2024-04-22T18:39:53.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8059999999999999e-06 2024-04-22T18:39:53.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741685+0000", 2024-04-22T18:39:53.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0780000000000003e-06 2024-04-22T18:39:53.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741690+0000", 2024-04-22T18:39:53.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4020000000000004e-06 2024-04-22T18:39:53.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741732+0000", 2024-04-22T18:39:53.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2110000000000002e-05 2024-04-22T18:39:53.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741798+0000", 2024-04-22T18:39:53.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6432000000000004e-05 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.148307+0000", 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40650862100000001 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1408 4.6 4:630bd8e5:::smithi08937929-45:head [write 2816389~693495] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.643467391000001, 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408959452, 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.840 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:53.841 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.841 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.841 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.841 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1408 2024-04-22T18:39:53.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.841 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:39:53.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:39:53.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161647+0000", 2024-04-22T18:39:53.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:53.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161951+0000", 2024-04-22T18:39:53.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030389799999999998 2024-04-22T18:39:53.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161955+0000", 2024-04-22T18:39:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1989999999999999e-06 2024-04-22T18:39:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161958+0000", 2024-04-22T18:39:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2260000000000001e-06 2024-04-22T18:39:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.265867+0000", 2024-04-22T18:39:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103908927 2024-04-22T18:39:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.265960+0000", 2024-04-22T18:39:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3016999999999994e-05 2024-04-22T18:39:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.266124+0000", 2024-04-22T18:39:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016347700000000001 2024-04-22T18:39:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570491+0000", 2024-04-22T18:39:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30436694199999997 2024-04-22T18:39:53.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570502+0000", 2024-04-22T18:39:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1107999999999999e-05 2024-04-22T18:39:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570556+0000", 2024-04-22T18:39:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4388000000000002e-05 2024-04-22T18:39:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570607+0000", 2024-04-22T18:39:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0782999999999998e-05 2024-04-22T18:39:53.845 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1409 4.6 4:630bd8e5:::smithi08937929-45:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:39:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:39:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.643153687000002, 2024-04-22T18:39:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40880186200000002, 2024-04-22T18:39:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1409 2024-04-22T18:39:53.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:53.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161964+0000", 2024-04-22T18:39:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5909999999999998e-06 2024-04-22T18:39:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161966+0000", 2024-04-22T18:39:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5979999999999999e-06 2024-04-22T18:39:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161969+0000", 2024-04-22T18:39:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7049999999999999e-06 2024-04-22T18:39:53.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269205+0000", 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107236228 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269320+0000", 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114571 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269476+0000", 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155928 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570619+0000", 2024-04-22T18:39:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30114322700000001 2024-04-22T18:39:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570672+0000", 2024-04-22T18:39:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3232000000000001e-05 2024-04-22T18:39:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570724+0000", 2024-04-22T18:39:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2258e-05 2024-04-22T18:39:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570763+0000", 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8927999999999997e-05 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1410 4.6 4:630bd8e5:::smithi08937929-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.643144169999999, 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40900504500000001, 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.851 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1410 2024-04-22T18:39:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:39:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:39:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161972+0000", 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6419999999999999e-06 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161976+0000", 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9260000000000002e-06 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161979+0000", 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9359999999999999e-06 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570790+0000", 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40881099199999998 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570879+0000", 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8895999999999997e-05 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570976+0000", 2024-04-22T18:39:53.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6923999999999996e-05 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.893634434999999, 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:39:53.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.856 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:39:53.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:39:53.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:39:53.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:53.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:39:53.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:39:53.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:39:53.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:39:53.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:53.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:39:53.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.364475109000001, 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.860 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:39:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:39:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:39:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:39:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:39:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:39:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:39:53.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:39:53.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:39:53.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:39:53.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:39:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:39:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:39:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:39:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:39:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:39:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:39:53.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:39:53.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:39:53.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:39:53.867 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.867 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.867 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.867 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:53.867 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:39:53.867 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.186767713999998, 2024-04-22T18:39:53.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:39:53.868 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.868 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:53.868 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:39:53.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:39:53.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:39:53.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:53.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:39:53.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:39:53.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:39:53.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:39:53.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:39:53.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:39:53.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:39:53.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:39:53.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:39:53.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:39:53.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:53.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:39:53.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:39:53.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:53.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:39:53.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:39:53.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:53.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:39:53.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:39:53.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:53.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:39:53.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:39:53.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:39:53.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:39:53.872 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.873 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.873 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.873 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:53.873 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:39:53.873 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.186699441999998, 2024-04-22T18:39:53.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:39:53.873 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.873 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:53.873 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:39:53.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:39:53.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:39:53.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:53.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:39:53.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:39:53.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:39:53.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:39:53.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:39:53.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:39:53.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:39:53.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:39:53.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:39:53.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:39:53.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:53.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:39:53.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:39:53.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:53.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:39:53.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:39:53.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:53.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:39:53.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:39:53.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:53.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:39:53.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:39:53.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:39:53.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:39:53.878 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.878 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.878 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.878 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:53.878 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:39:53.878 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.186531600999999, 2024-04-22T18:39:53.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:39:53.878 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.878 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:53.878 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:39:53.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:39:53.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:39:53.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:53.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:39:53.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:39:53.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:39:53.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:39:53.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:39:53.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:39:53.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:53.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:39:53.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:39:53.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:53.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:39:53.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:39:53.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:53.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:39:53.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:39:53.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:39:53.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:39:53.883 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.883 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.883 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.883 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:39:53.883 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:39:53.883 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.221206126, 2024-04-22T18:39:53.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:39:53.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:53.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:39:53.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:39:53.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:39:53.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:39:53.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:39:53.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:39:53.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:39:53.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:39:53.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:39:53.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:39:53.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:53.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:39:53.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:39:53.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:53.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:39:53.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:39:53.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:53.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:39:53.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:39:53.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:53.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:39:53.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:39:53.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:39:53.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:39:53.888 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.888 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.888 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.889 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:39:53.889 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:39:53.889 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.121053938999999, 2024-04-22T18:39:53.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:39:53.889 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.889 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:53.889 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:39:53.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:39:53.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:39:53.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:53.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:39:53.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:39:53.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:39:53.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:39:53.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:39:53.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:39:53.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:39:53.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:39:53.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:39:53.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:39:53.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:39:53.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:39:53.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:39:53.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:53.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:39:53.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:39:53.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:53.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:39:53.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:39:53.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:53.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:39:53.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:39:53.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:39:53.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:39:53.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.894 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.894 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.894 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2385 4.13 4:c93c842e:::smithi08937929-25:head [write 1937856~527061] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:39:53.894 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.287418+0000", 2024-04-22T18:39:53.894 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.517696881, 2024-04-22T18:39:53.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39986265300000001, 2024-04-22T18:39:53.894 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.894 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:53.894 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.894 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.894 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2385 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287418+0000", 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287418+0000", 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287417+0000", 2024-04-22T18:39:53.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:53.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287636+0000", 2024-04-22T18:39:53.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000219089 2024-04-22T18:39:53.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287654+0000", 2024-04-22T18:39:53.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8294000000000001e-05 2024-04-22T18:39:53.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287661+0000", 2024-04-22T18:39:53.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7780000000000002e-06 2024-04-22T18:39:53.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.483858+0000", 2024-04-22T18:39:53.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19619666199999999 2024-04-22T18:39:53.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.483917+0000", 2024-04-22T18:39:53.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9454999999999999e-05 2024-04-22T18:39:53.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:53.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.483999+0000", 2024-04-22T18:39:53.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1359999999999994e-05 2024-04-22T18:39:53.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:53.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586037+0000", 2024-04-22T18:39:53.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102038217 2024-04-22T18:39:53.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:53.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687214+0000", 2024-04-22T18:39:53.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101177195 2024-04-22T18:39:53.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687253+0000", 2024-04-22T18:39:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9409999999999997e-05 2024-04-22T18:39:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687281+0000", 2024-04-22T18:39:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7478e-05 2024-04-22T18:39:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2386 4.13 4:c93c842e:::smithi08937929-25:head [write 3252761~175867] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:39:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.292661+0000", 2024-04-22T18:39:53.900 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.5124542330000001, 2024-04-22T18:39:53.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39468563099999998, 2024-04-22T18:39:53.900 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.900 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:53.900 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.900 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.900 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.900 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2386 2024-04-22T18:39:53.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.900 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.292661+0000", 2024-04-22T18:39:53.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.292661+0000", 2024-04-22T18:39:53.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.292659+0000", 2024-04-22T18:39:53.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:53.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.292788+0000", 2024-04-22T18:39:53.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012955 2024-04-22T18:39:53.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.292806+0000", 2024-04-22T18:39:53.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7357999999999999e-05 2024-04-22T18:39:53.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392900+0000", 2024-04-22T18:39:53.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100093799 2024-04-22T18:39:53.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.485125+0000", 2024-04-22T18:39:53.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.092225336000000005 2024-04-22T18:39:53.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.485172+0000", 2024-04-22T18:39:53.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7216000000000003e-05 2024-04-22T18:39:53.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:53.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.485244+0000", 2024-04-22T18:39:53.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1888e-05 2024-04-22T18:39:53.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:53.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.587565+0000", 2024-04-22T18:39:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102321 2024-04-22T18:39:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687303+0000", 2024-04-22T18:39:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099737563000000001 2024-04-22T18:39:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687324+0000", 2024-04-22T18:39:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0868000000000001e-05 2024-04-22T18:39:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687346+0000", 2024-04-22T18:39:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2915999999999999e-05 2024-04-22T18:39:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:39:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:39:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.4121894190000002, 2024-04-22T18:39:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:39:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:53.905 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:53.906 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:53.906 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:39:53.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.906 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:53.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:53.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:39:53.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:53.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:39:53.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:53.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:53.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:39:53.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:53.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:53.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:39:53.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:39:53.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:53.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:39:53.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:39:53.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:53.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:39:53.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:39:53.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:39:53.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:39:53.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:53.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:39:53.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:39:53.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:53.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:39:53.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:39:53.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:53.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:39:53.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:39:53.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:53.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:53.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:53.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:39:53.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:39:53.909 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.910 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.910 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.910 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:53.910 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:53.910 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:53.911 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:53.912 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:53.911+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:53.923 INFO:tasks.rados.rados.0.smithi089.stdout:1167: got (0) Success 2024-04-22T18:39:53.923 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 51 (ObjNum 163 snap 0 seq_num 163) dirty exists 2024-04-22T18:39:53.923 INFO:tasks.rados.rados.0.smithi089.stdout:1156: done (15 left) 2024-04-22T18:39:53.923 INFO:tasks.rados.rados.0.smithi089.stdout:1158: done (14 left) 2024-04-22T18:39:53.923 INFO:tasks.rados.rados.0.smithi089.stdout:1162: done (13 left) 2024-04-22T18:39:53.923 INFO:tasks.rados.rados.0.smithi089.stdout:1163: done (12 left) 2024-04-22T18:39:53.923 INFO:tasks.rados.rados.0.smithi089.stdout:1164: done (11 left) 2024-04-22T18:39:53.923 INFO:tasks.rados.rados.0.smithi089.stdout:1167: done (10 left) 2024-04-22T18:39:53.923 INFO:tasks.rados.rados.0.smithi089.stdout:1178: read oid 24 snap -1 2024-04-22T18:39:53.923 INFO:tasks.rados.rados.0.smithi089.stdout:1178: expect (ObjNum 174 snap 0 seq_num 174) 2024-04-22T18:39:53.923 INFO:tasks.rados.rados.0.smithi089.stdout:1179: read oid 28 snap -1 2024-04-22T18:39:53.923 INFO:tasks.rados.rados.0.smithi089.stdout:1179: expect (ObjNum 184 snap 0 seq_num 184) 2024-04-22T18:39:53.923 INFO:tasks.rados.rados.0.smithi089.stdout:1168: finishing set_chunk to oid 50 2024-04-22T18:39:53.924 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 107 (ObjNum 189 snap 0 seq_num 189) dirty exists 2024-04-22T18:39:53.924 INFO:tasks.rados.rados.0.smithi089.stdout:1169: expect (ObjNum 204 snap 0 seq_num 204) 2024-04-22T18:39:54.012 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:54.012+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:54.026 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:54.026 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:54.026 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:54.026 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:54.036 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:54.112 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:54.113+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:54.144 INFO:tasks.rados.rados.0.smithi089.stdout:1170: expect (ObjNum 203 snap 0 seq_num 203) 2024-04-22T18:39:54.178 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:54.178 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:54.187 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:54.213 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:54.215+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:54.293 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:54.293 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:54.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:54.293 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:54.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.294 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:9 4.f 4:f2c48a0e:::smithi08937929-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:54.294 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:54.294 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.175912965999999, 2024-04-22T18:39:54.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49290714600000002, 2024-04-22T18:39:54.294 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.294 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.294 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.294 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.294 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.295 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 9 2024-04-22T18:39:54.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.295 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:54.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:39:54.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116773+0000", 2024-04-22T18:39:54.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:54.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117024+0000", 2024-04-22T18:39:54.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025191299999999999 2024-04-22T18:39:54.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117036+0000", 2024-04-22T18:39:54.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1595e-05 2024-04-22T18:39:54.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117046+0000", 2024-04-22T18:39:54.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0456e-05 2024-04-22T18:39:54.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414324+0000", 2024-04-22T18:39:54.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297277603 2024-04-22T18:39:54.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414404+0000", 2024-04-22T18:39:54.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9738000000000001e-05 2024-04-22T18:39:54.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:54.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414494+0000", 2024-04-22T18:39:54.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9733000000000003e-05 2024-04-22T18:39:54.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:54.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518194+0000", 2024-04-22T18:39:54.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10370061 2024-04-22T18:39:54.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:54.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609618+0000", 2024-04-22T18:39:54.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091423360999999995 2024-04-22T18:39:54.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609655+0000", 2024-04-22T18:39:54.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7580000000000003e-05 2024-04-22T18:39:54.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609681+0000", 2024-04-22T18:39:54.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5908e-05 2024-04-22T18:39:54.300 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.300 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.300 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.300 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:10 4.f 4:f2c48a0e:::smithi08937929-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:54.300 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:54.300 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.170060706000001, 2024-04-22T18:39:54.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48712681400000002, 2024-04-22T18:39:54.301 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.301 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.301 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.301 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.301 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.301 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 10 2024-04-22T18:39:54.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.301 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:54.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:39:54.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122625+0000", 2024-04-22T18:39:54.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:54.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122828+0000", 2024-04-22T18:39:54.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020359499999999999 2024-04-22T18:39:54.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122840+0000", 2024-04-22T18:39:54.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1239e-05 2024-04-22T18:39:54.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122845+0000", 2024-04-22T18:39:54.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2719999999999997e-06 2024-04-22T18:39:54.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415105+0000", 2024-04-22T18:39:54.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29225972099999997 2024-04-22T18:39:54.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415169+0000", 2024-04-22T18:39:54.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4170000000000004e-05 2024-04-22T18:39:54.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:54.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415257+0000", 2024-04-22T18:39:54.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8331999999999994e-05 2024-04-22T18:39:54.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:54.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518199+0000", 2024-04-22T18:39:54.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10294174 2024-04-22T18:39:54.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:54.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609707+0000", 2024-04-22T18:39:54.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091508408999999999 2024-04-22T18:39:54.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609729+0000", 2024-04-22T18:39:54.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1917e-05 2024-04-22T18:39:54.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609753+0000", 2024-04-22T18:39:54.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3776000000000001e-05 2024-04-22T18:39:54.307 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.307 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.307 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.307 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:11 4.f 4:f2c48a0e:::smithi08937929-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:54.307 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:54.307 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.169837332999997, 2024-04-22T18:39:54.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49452214500000002, 2024-04-22T18:39:54.307 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.308 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.308 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.308 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.308 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.308 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 11 2024-04-22T18:39:54.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.308 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:54.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:39:54.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122849+0000", 2024-04-22T18:39:54.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:54.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122853+0000", 2024-04-22T18:39:54.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9029999999999997e-06 2024-04-22T18:39:54.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122856+0000", 2024-04-22T18:39:54.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.01e-06 2024-04-22T18:39:54.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122859+0000", 2024-04-22T18:39:54.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:54.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415755+0000", 2024-04-22T18:39:54.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29289606800000001 2024-04-22T18:39:54.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415808+0000", 2024-04-22T18:39:54.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3415999999999998e-05 2024-04-22T18:39:54.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:54.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415888+0000", 2024-04-22T18:39:54.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9061999999999996e-05 2024-04-22T18:39:54.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:54.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518203+0000", 2024-04-22T18:39:54.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102315533 2024-04-22T18:39:54.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:54.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617164+0000", 2024-04-22T18:39:54.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098960548999999995 2024-04-22T18:39:54.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617241+0000", 2024-04-22T18:39:54.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7292000000000005e-05 2024-04-22T18:39:54.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617372+0000", 2024-04-22T18:39:54.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013075399999999999 2024-04-22T18:39:54.314 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.314 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.314 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.314 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:12 4.f 4:f2c48a0e:::smithi08937929-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:39:54.314 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:54.314 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.169825469000003, 2024-04-22T18:39:54.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49473910999999998, 2024-04-22T18:39:54.314 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.315 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:54.315 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.315 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.315 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.315 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 12 2024-04-22T18:39:54.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.315 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:54.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:54.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:39:54.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:54.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122863+0000", 2024-04-22T18:39:54.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.863e-06 2024-04-22T18:39:54.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122866+0000", 2024-04-22T18:39:54.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:39:54.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122869+0000", 2024-04-22T18:39:54.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5950000000000001e-06 2024-04-22T18:39:54.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416008+0000", 2024-04-22T18:39:54.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29313975199999998 2024-04-22T18:39:54.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:54.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416048+0000", 2024-04-22T18:39:54.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9888000000000002e-05 2024-04-22T18:39:54.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617447+0000", 2024-04-22T18:39:54.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20139881400000001 2024-04-22T18:39:54.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617537+0000", 2024-04-22T18:39:54.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9606999999999998e-05 2024-04-22T18:39:54.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617600+0000", 2024-04-22T18:39:54.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3884e-05 2024-04-22T18:39:54.320 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.320 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.320 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.320 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:33 4.f 4:f0ed3d49:::smithi08937929-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:54.320 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:54.320 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.105885268000002, 2024-04-22T18:39:54.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62359692099999997, 2024-04-22T18:39:54.320 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.321 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.321 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.321 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.321 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.321 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 33 2024-04-22T18:39:54.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.321 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:54.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:39:54.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186799+0000", 2024-04-22T18:39:54.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:54.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187103+0000", 2024-04-22T18:39:54.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030354900000000002 2024-04-22T18:39:54.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187117+0000", 2024-04-22T18:39:54.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4578e-05 2024-04-22T18:39:54.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187129+0000", 2024-04-22T18:39:54.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1421e-05 2024-04-22T18:39:54.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516134+0000", 2024-04-22T18:39:54.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32900553900000001 2024-04-22T18:39:54.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516215+0000", 2024-04-22T18:39:54.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0537999999999994e-05 2024-04-22T18:39:54.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:54.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516302+0000", 2024-04-22T18:39:54.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6601000000000007e-05 2024-04-22T18:39:54.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:54.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518207+0000", 2024-04-22T18:39:54.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019058370000000001 2024-04-22T18:39:54.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:54.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810302+0000", 2024-04-22T18:39:54.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29209509700000003 2024-04-22T18:39:54.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810348+0000", 2024-04-22T18:39:54.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5992999999999998e-05 2024-04-22T18:39:54.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810398+0000", 2024-04-22T18:39:54.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0046000000000002e-05 2024-04-22T18:39:54.326 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.326 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.326 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.327 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:34 4.f 4:f0ed3d49:::smithi08937929-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:54.327 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:54.327 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.101283256000002, 2024-04-22T18:39:54.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.61954655800000002, 2024-04-22T18:39:54.327 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.327 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.327 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.327 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.327 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.327 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34 2024-04-22T18:39:54.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.327 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:54.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:39:54.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191401+0000", 2024-04-22T18:39:54.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:54.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191503+0000", 2024-04-22T18:39:54.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010190400000000001 2024-04-22T18:39:54.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191517+0000", 2024-04-22T18:39:54.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3267e-05 2024-04-22T18:39:54.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191523+0000", 2024-04-22T18:39:54.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-04-22T18:39:54.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517062+0000", 2024-04-22T18:39:54.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32553894300000003 2024-04-22T18:39:54.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517141+0000", 2024-04-22T18:39:54.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9667999999999997e-05 2024-04-22T18:39:54.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:54.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517267+0000", 2024-04-22T18:39:54.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000125494 2024-04-22T18:39:54.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:54.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518213+0000", 2024-04-22T18:39:54.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000946275 2024-04-22T18:39:54.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:54.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810901+0000", 2024-04-22T18:39:54.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29268821699999997 2024-04-22T18:39:54.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810929+0000", 2024-04-22T18:39:54.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7645999999999999e-05 2024-04-22T18:39:54.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810950+0000", 2024-04-22T18:39:54.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1333999999999999e-05 2024-04-22T18:39:54.333 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.333 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.333 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.333 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:54.333 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:54.333 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.101160288000003, 2024-04-22T18:39:54.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:39:54.333 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.333 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.333 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.334 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.334 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.334 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:39:54.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.334 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:54.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:39:54.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:39:54.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:54.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:39:54.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:39:54.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:39:54.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:39:54.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:39:54.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:54.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:39:54.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:39:54.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:39:54.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:39:54.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:54.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:39:54.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:39:54.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:54.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:39:54.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:39:54.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:54.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:39:54.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:39:54.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:39:54.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:39:54.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:39:54.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:39:54.339 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.339 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.339 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.340 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:54.340 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:54.340 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.101149212999999, 2024-04-22T18:39:54.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:39:54.340 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.340 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:54.340 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.340 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.340 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.340 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:39:54.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.341 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:54.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:39:54.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:39:54.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:54.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:39:54.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:39:54.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:39:54.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:39:54.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:39:54.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:39:54.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:39:54.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:39:54.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:54.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:39:54.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:39:54.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:39:54.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:39:54.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:39:54.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:39:54.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:39:54.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:39:54.345 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.345 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.346 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.346 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:38 4.2 4:4fb26dd8:::smithi08937929-10:head [write 1893632~257631] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:54.346 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:54.346 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.089426609, 2024-04-22T18:39:54.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33863800199999999, 2024-04-22T18:39:54.346 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.346 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.346 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.347 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.347 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.347 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 38 2024-04-22T18:39:54.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.347 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:54.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203260+0000", 2024-04-22T18:39:54.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203258+0000", 2024-04-22T18:39:54.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:54.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203396+0000", 2024-04-22T18:39:54.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013710799999999999 2024-04-22T18:39:54.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203406+0000", 2024-04-22T18:39:54.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0295000000000001e-05 2024-04-22T18:39:54.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203411+0000", 2024-04-22T18:39:54.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5079999999999996e-06 2024-04-22T18:39:54.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203450+0000", 2024-04-22T18:39:54.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8420999999999997e-05 2024-04-22T18:39:54.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:54.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203455+0000", 2024-04-22T18:39:54.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:39:54.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237071+0000", 2024-04-22T18:39:54.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.033615891000000002 2024-04-22T18:39:54.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237129+0000", 2024-04-22T18:39:54.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8134000000000003e-05 2024-04-22T18:39:54.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:54.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.237221+0000", 2024-04-22T18:39:54.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2053999999999997e-05 2024-04-22T18:39:54.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:54.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.341419+0000", 2024-04-22T18:39:54.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104197568 2024-04-22T18:39:54.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:54.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541792+0000", 2024-04-22T18:39:54.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.200373312 2024-04-22T18:39:54.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541838+0000", 2024-04-22T18:39:54.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6004000000000002e-05 2024-04-22T18:39:54.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541898+0000", 2024-04-22T18:39:54.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0297000000000002e-05 2024-04-22T18:39:54.354 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.354 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.354 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.354 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:39 4.2 4:4fb26dd8:::smithi08937929-10:head [setxattr _header (56) in=7b,truncate 2151263] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:54.354 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:54.354 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.089269614999999, 2024-04-22T18:39:54.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33862203000000002, 2024-04-22T18:39:54.354 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.355 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.355 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.355 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.355 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.355 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 39 2024-04-22T18:39:54.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.355 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:54.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:54.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203417+0000", 2024-04-22T18:39:54.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:54.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203423+0000", 2024-04-22T18:39:54.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0589999999999999e-06 2024-04-22T18:39:54.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203429+0000", 2024-04-22T18:39:54.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2559999999999999e-06 2024-04-22T18:39:54.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203434+0000", 2024-04-22T18:39:54.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8049999999999997e-06 2024-04-22T18:39:54.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203475+0000", 2024-04-22T18:39:54.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1205000000000003e-05 2024-04-22T18:39:54.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:54.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203478+0000", 2024-04-22T18:39:54.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7140000000000002e-06 2024-04-22T18:39:54.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239180+0000", 2024-04-22T18:39:54.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035702195999999999 2024-04-22T18:39:54.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239225+0000", 2024-04-22T18:39:54.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5722e-05 2024-04-22T18:39:54.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:54.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239291+0000", 2024-04-22T18:39:54.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5870000000000005e-05 2024-04-22T18:39:54.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:54.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342114+0000", 2024-04-22T18:39:54.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102822301 2024-04-22T18:39:54.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:54.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541946+0000", 2024-04-22T18:39:54.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.199832544 2024-04-22T18:39:54.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.541988+0000", 2024-04-22T18:39:54.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1499999999999999e-05 2024-04-22T18:39:54.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542039+0000", 2024-04-22T18:39:54.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1576000000000002e-05 2024-04-22T18:39:54.362 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.362 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.362 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.362 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:40 4.2 4:4fb26dd8:::smithi08937929-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:54.362 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:54.362 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.089232355, 2024-04-22T18:39:54.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33924657200000002, 2024-04-22T18:39:54.362 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.362 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:54.363 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.363 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.363 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.363 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 40 2024-04-22T18:39:54.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.363 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:54.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:54.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203454+0000", 2024-04-22T18:39:54.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:54.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203457+0000", 2024-04-22T18:39:54.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.129e-06 2024-04-22T18:39:54.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203463+0000", 2024-04-22T18:39:54.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7440000000000004e-06 2024-04-22T18:39:54.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203468+0000", 2024-04-22T18:39:54.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2619999999999998e-06 2024-04-22T18:39:54.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203491+0000", 2024-04-22T18:39:54.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3260000000000001e-05 2024-04-22T18:39:54.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:54.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.203493+0000", 2024-04-22T18:39:54.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9599999999999999e-06 2024-04-22T18:39:54.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239420+0000", 2024-04-22T18:39:54.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035927212 2024-04-22T18:39:54.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:54.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.239462+0000", 2024-04-22T18:39:54.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.159e-05 2024-04-22T18:39:54.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542547+0000", 2024-04-22T18:39:54.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30308492599999998 2024-04-22T18:39:54.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542611+0000", 2024-04-22T18:39:54.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4622000000000006e-05 2024-04-22T18:39:54.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542701+0000", 2024-04-22T18:39:54.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9579000000000004e-05 2024-04-22T18:39:54.368 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.368 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.368 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.369 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:41 4.2 4:409e6456:::smithi08937929-11:head [write 568070~468843,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:54.369 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:54.369 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.083516084000003, 2024-04-22T18:39:54.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33297193600000002, 2024-04-22T18:39:54.369 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.369 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.369 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.369 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.369 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.369 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 41 2024-04-22T18:39:54.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.369 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:54.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209171+0000", 2024-04-22T18:39:54.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209168+0000", 2024-04-22T18:39:54.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:54.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209393+0000", 2024-04-22T18:39:54.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022485199999999999 2024-04-22T18:39:54.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209404+0000", 2024-04-22T18:39:54.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1209999999999999e-05 2024-04-22T18:39:54.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209411+0000", 2024-04-22T18:39:54.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8580000000000002e-06 2024-04-22T18:39:54.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209457+0000", 2024-04-22T18:39:54.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6351999999999999e-05 2024-04-22T18:39:54.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:54.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.209463+0000", 2024-04-22T18:39:54.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.322e-06 2024-04-22T18:39:54.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339572+0000", 2024-04-22T18:39:54.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13010901999999999 2024-04-22T18:39:54.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339673+0000", 2024-04-22T18:39:54.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101332 2024-04-22T18:39:54.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:54.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.339765+0000", 2024-04-22T18:39:54.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1780999999999994e-05 2024-04-22T18:39:54.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:54.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342117+0000", 2024-04-22T18:39:54.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002351709 2024-04-22T18:39:54.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:54.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542074+0000", 2024-04-22T18:39:54.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995749800000001 2024-04-22T18:39:54.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542112+0000", 2024-04-22T18:39:54.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7891e-05 2024-04-22T18:39:54.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542143+0000", 2024-04-22T18:39:54.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0756000000000002e-05 2024-04-22T18:39:54.376 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.376 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.376 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.376 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:42 4.2 4:409e6456:::smithi08937929-11:head [write 1552114~615956] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:54.376 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:54.376 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.077177775000003, 2024-04-22T18:39:54.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326735257, 2024-04-22T18:39:54.377 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.377 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.377 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.377 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.377 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.377 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 42 2024-04-22T18:39:54.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.377 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:54.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215509+0000", 2024-04-22T18:39:54.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215508+0000", 2024-04-22T18:39:54.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:54.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215766+0000", 2024-04-22T18:39:54.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025766999999999999 2024-04-22T18:39:54.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215776+0000", 2024-04-22T18:39:54.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0772e-05 2024-04-22T18:39:54.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215785+0000", 2024-04-22T18:39:54.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6880000000000005e-06 2024-04-22T18:39:54.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215818+0000", 2024-04-22T18:39:54.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2960999999999998e-05 2024-04-22T18:39:54.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:54.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215821+0000", 2024-04-22T18:39:54.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8569999999999999e-06 2024-04-22T18:39:54.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340332+0000", 2024-04-22T18:39:54.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.124511409 2024-04-22T18:39:54.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340416+0000", 2024-04-22T18:39:54.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3793000000000002e-05 2024-04-22T18:39:54.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:54.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.340543+0000", 2024-04-22T18:39:54.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012720299999999999 2024-04-22T18:39:54.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:54.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342361+0000", 2024-04-22T18:39:54.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018174370000000001 2024-04-22T18:39:54.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:54.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542176+0000", 2024-04-22T18:39:54.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19981518400000001 2024-04-22T18:39:54.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542211+0000", 2024-04-22T18:39:54.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5128e-05 2024-04-22T18:39:54.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542244+0000", 2024-04-22T18:39:54.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3256000000000001e-05 2024-04-22T18:39:54.384 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.384 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.384 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.384 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:43 4.2 4:409e6456:::smithi08937929-11:head [setxattr _header (56) in=7b,truncate 2168070] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:54.384 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:54.384 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.076893001999998, 2024-04-22T18:39:54.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32656540499999998, 2024-04-22T18:39:54.385 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.385 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.385 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.385 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.385 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.385 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 43 2024-04-22T18:39:54.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.385 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:54.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215794+0000", 2024-04-22T18:39:54.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215793+0000", 2024-04-22T18:39:54.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:54.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215797+0000", 2024-04-22T18:39:54.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4190000000000004e-06 2024-04-22T18:39:54.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215804+0000", 2024-04-22T18:39:54.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:39:54.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215809+0000", 2024-04-22T18:39:54.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3689999999999998e-06 2024-04-22T18:39:54.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215840+0000", 2024-04-22T18:39:54.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-05 2024-04-22T18:39:54.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:54.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215843+0000", 2024-04-22T18:39:54.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8849999999999999e-06 2024-04-22T18:39:54.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342483+0000", 2024-04-22T18:39:54.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126640172 2024-04-22T18:39:54.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342529+0000", 2024-04-22T18:39:54.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5668999999999999e-05 2024-04-22T18:39:54.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:54.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342597+0000", 2024-04-22T18:39:54.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7538000000000003e-05 2024-04-22T18:39:54.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:54.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.347102+0000", 2024-04-22T18:39:54.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045058040000000004 2024-04-22T18:39:54.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:54.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542278+0000", 2024-04-22T18:39:54.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.195175879 2024-04-22T18:39:54.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542314+0000", 2024-04-22T18:39:54.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6134999999999999e-05 2024-04-22T18:39:54.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542359+0000", 2024-04-22T18:39:54.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4848999999999997e-05 2024-04-22T18:39:54.392 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.392 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.392 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.392 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:44 4.2 4:409e6456:::smithi08937929-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:54.392 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:54.392 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.076872993999999, 2024-04-22T18:39:54.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326706418, 2024-04-22T18:39:54.393 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.393 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:54.393 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.393 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.393 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.393 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 44 2024-04-22T18:39:54.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.393 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:54.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215814+0000", 2024-04-22T18:39:54.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215813+0000", 2024-04-22T18:39:54.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:54.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215816+0000", 2024-04-22T18:39:54.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9799999999999998e-06 2024-04-22T18:39:54.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215822+0000", 2024-04-22T18:39:54.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5570000000000003e-06 2024-04-22T18:39:54.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215827+0000", 2024-04-22T18:39:54.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2739999999999998e-06 2024-04-22T18:39:54.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215855+0000", 2024-04-22T18:39:54.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7682999999999999e-05 2024-04-22T18:39:54.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:54.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.215857+0000", 2024-04-22T18:39:54.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9250000000000002e-06 2024-04-22T18:39:54.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342717+0000", 2024-04-22T18:39:54.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.126860696 2024-04-22T18:39:54.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:54.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.342758+0000", 2024-04-22T18:39:54.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0295e-05 2024-04-22T18:39:54.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542408+0000", 2024-04-22T18:39:54.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19964990599999999 2024-04-22T18:39:54.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542457+0000", 2024-04-22T18:39:54.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9784999999999999e-05 2024-04-22T18:39:54.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.542520+0000", 2024-04-22T18:39:54.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2960999999999996e-05 2024-04-22T18:39:54.399 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.399 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.399 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.399 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:54.399 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:54.399 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.499779719000003, 2024-04-22T18:39:54.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:39:54.400 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.400 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.400 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.400 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.400 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.400 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:39:54.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.400 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:54.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:39:54.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:39:54.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:54.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:39:54.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:39:54.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:39:54.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:39:54.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:39:54.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:39:54.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:39:54.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:39:54.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:39:54.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:39:54.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:54.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:39:54.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:39:54.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:54.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:39:54.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:39:54.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:54.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:39:54.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:39:54.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:39:54.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:39:54.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:39:54.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:39:54.406 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.406 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.406 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.406 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1262 4.7 4:e117a562:::smithi08937929-33:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:54.406 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:54.407 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.839636542999997, 2024-04-22T18:39:54.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40488836700000003, 2024-04-22T18:39:54.407 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.407 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.407 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.407 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.407 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.407 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1262 2024-04-22T18:39:54.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.407 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:54.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:39:54.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453048+0000", 2024-04-22T18:39:54.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:54.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453058+0000", 2024-04-22T18:39:54.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:39:54.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453069+0000", 2024-04-22T18:39:54.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1117999999999999e-05 2024-04-22T18:39:54.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453075+0000", 2024-04-22T18:39:54.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0909999999999998e-06 2024-04-22T18:39:54.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453115+0000", 2024-04-22T18:39:54.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0119e-05 2024-04-22T18:39:54.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:54.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453119+0000", 2024-04-22T18:39:54.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3610000000000003e-06 2024-04-22T18:39:54.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454814+0000", 2024-04-22T18:39:54.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00169436 2024-04-22T18:39:54.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454861+0000", 2024-04-22T18:39:54.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7184000000000001e-05 2024-04-22T18:39:54.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:54.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756925+0000", 2024-04-22T18:39:54.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30206361799999998 2024-04-22T18:39:54.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:54.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757577+0000", 2024-04-22T18:39:54.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065199599999999996 2024-04-22T18:39:54.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:54.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857846+0000", 2024-04-22T18:39:54.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100269369 2024-04-22T18:39:54.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857896+0000", 2024-04-22T18:39:54.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9607000000000001e-05 2024-04-22T18:39:54.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857939+0000", 2024-04-22T18:39:54.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3149999999999999e-05 2024-04-22T18:39:54.414 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.414 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.414 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.414 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1689 3.0 3:1d4bcadf:::e7eb8db0531d9f456e91fe87bc79e47ebd80459f1783dc2a5e53d4822cd34f49:head [write 0~1066496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:39:54.414 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.607920+0000", 2024-04-22T18:39:54.414 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.684767169000001, 2024-04-22T18:39:54.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30339939300000002, 2024-04-22T18:39:54.415 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.415 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.415 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.415 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.415 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.415 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1689 2024-04-22T18:39:54.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.415 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.607920+0000", 2024-04-22T18:39:54.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.607920+0000", 2024-04-22T18:39:54.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.607917+0000", 2024-04-22T18:39:54.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:54.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.608327+0000", 2024-04-22T18:39:54.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000409508 2024-04-22T18:39:54.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.608344+0000", 2024-04-22T18:39:54.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6846000000000001e-05 2024-04-22T18:39:54.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.608351+0000", 2024-04-22T18:39:54.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8140000000000003e-06 2024-04-22T18:39:54.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708489+0000", 2024-04-22T18:39:54.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100138536 2024-04-22T18:39:54.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708651+0000", 2024-04-22T18:39:54.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000162132 2024-04-22T18:39:54.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:39:54.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708793+0000", 2024-04-22T18:39:54.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014147699999999999 2024-04-22T18:39:54.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:54.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.713125+0000", 2024-04-22T18:39:54.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043321369999999998 2024-04-22T18:39:54.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:54.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911261+0000", 2024-04-22T18:39:54.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198135744 2024-04-22T18:39:54.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911293+0000", 2024-04-22T18:39:54.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1970000000000001e-05 2024-04-22T18:39:54.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911319+0000", 2024-04-22T18:39:54.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6440000000000001e-05 2024-04-22T18:39:54.421 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.421 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.421 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.421 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1872 4.2 4:409e6456:::smithi08937929-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:54.421 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:35.576858+0000", 2024-04-22T18:39:54.421 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.715828416000001, 2024-04-22T18:39:54.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.302951992, 2024-04-22T18:39:54.421 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.421 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:54.422 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.422 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.422 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.422 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1872 2024-04-22T18:39:54.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.422 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576858+0000", 2024-04-22T18:39:54.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576858+0000", 2024-04-22T18:39:54.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576857+0000", 2024-04-22T18:39:54.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:54.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576864+0000", 2024-04-22T18:39:54.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4569999999999999e-06 2024-04-22T18:39:54.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576871+0000", 2024-04-22T18:39:54.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1679999999999998e-06 2024-04-22T18:39:54.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.576876+0000", 2024-04-22T18:39:54.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0440000000000003e-06 2024-04-22T18:39:54.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.590050+0000", 2024-04-22T18:39:54.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.013174208 2024-04-22T18:39:54.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:54.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.590142+0000", 2024-04-22T18:39:54.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1681000000000005e-05 2024-04-22T18:39:54.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.879685+0000", 2024-04-22T18:39:54.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.28954336200000003 2024-04-22T18:39:54.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.879745+0000", 2024-04-22T18:39:54.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9487999999999997e-05 2024-04-22T18:39:54.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:35.879810+0000", 2024-04-22T18:39:54.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5431000000000004e-05 2024-04-22T18:39:54.427 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.427 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.427 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.427 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2205 4.2 4:4fb26dd8:::smithi08937929-10:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:39:54.427 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:39:54.427 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.9020914419999997, 2024-04-22T18:39:54.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.413791136, 2024-04-22T18:39:54.427 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.428 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:39:54.428 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:54.428 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:54.428 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:54.428 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2205 2024-04-22T18:39:54.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.428 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:39:54.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:39:54.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:39:54.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:54.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390598+0000", 2024-04-22T18:39:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7630000000000001e-06 2024-04-22T18:39:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390603+0000", 2024-04-22T18:39:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.293e-06 2024-04-22T18:39:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390608+0000", 2024-04-22T18:39:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8790000000000002e-06 2024-04-22T18:39:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390653+0000", 2024-04-22T18:39:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5661999999999998e-05 2024-04-22T18:39:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:54.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390657+0000", 2024-04-22T18:39:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9709999999999998e-06 2024-04-22T18:39:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.592250+0000", 2024-04-22T18:39:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20159280600000001 2024-04-22T18:39:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.592322+0000", 2024-04-22T18:39:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2171999999999999e-05 2024-04-22T18:39:54.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.804373+0000", 2024-04-22T18:39:54.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.212050297 2024-04-22T18:39:54.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.804387+0000", 2024-04-22T18:39:54.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3859999999999999e-05 2024-04-22T18:39:54.433 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.433 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.433 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.433 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.434 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.434 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:54.435 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:54.436 INFO:tasks.rados.rados.0.smithi089.stdout:1171: finishing set_chunk to oid 11 2024-04-22T18:39:54.436 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 37 (ObjNum 196 snap 0 seq_num 196) dirty exists 2024-04-22T18:39:54.436 INFO:tasks.rados.rados.0.smithi089.stdout:1172: finishing set_chunk to oid 31 2024-04-22T18:39:54.436 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 24 (ObjNum 74 snap 0 seq_num 74) dirty exists 2024-04-22T18:39:54.436 INFO:tasks.rados.rados.0.smithi089.stdout:1174: expect (ObjNum 166 snap 0 seq_num 166) 2024-04-22T18:39:54.437 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:54.436+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:54.530 INFO:tasks.rados.rados.0.smithi089.stdout:1173: finishing set_chunk to oid 19 2024-04-22T18:39:54.530 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 108 (ObjNum 195 snap 0 seq_num 195) dirty exists 2024-04-22T18:39:54.530 INFO:tasks.rados.rados.0.smithi089.stdout:1168: done (11 left) 2024-04-22T18:39:54.530 INFO:tasks.rados.rados.0.smithi089.stdout:1169: done (10 left) 2024-04-22T18:39:54.530 INFO:tasks.rados.rados.0.smithi089.stdout:1170: done (9 left) 2024-04-22T18:39:54.530 INFO:tasks.rados.rados.0.smithi089.stdout:1171: done (8 left) 2024-04-22T18:39:54.531 INFO:tasks.rados.rados.0.smithi089.stdout:1172: done (7 left) 2024-04-22T18:39:54.531 INFO:tasks.rados.rados.0.smithi089.stdout:1173: done (6 left) 2024-04-22T18:39:54.531 INFO:tasks.rados.rados.0.smithi089.stdout:1174: done (5 left) 2024-04-22T18:39:54.531 INFO:tasks.rados.rados.0.smithi089.stdout:1180: write (excl) oid 32 current snap is 0 2024-04-22T18:39:54.531 INFO:tasks.rados.rados.0.smithi089.stdout:1176: expect (ObjNum 149 snap 0 seq_num 149) 2024-04-22T18:39:54.536 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:54.536+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:54.540 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:54.540 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:54.540 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:54.540 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:54.549 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:54.637 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:54.637+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:54.681 INFO:tasks.rados.rados.0.smithi089.stdout:1177: expect (ObjNum 141 snap 0 seq_num 141) 2024-04-22T18:39:54.693 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:54.693 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:54.702 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:54.744 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:54.744+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:54.744 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:39:54.821 INFO:tasks.rados.rados.0.smithi089.stdout:1175: finishing set_chunk to oid 44 2024-04-22T18:39:54.821 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 85 (ObjNum 187 snap 0 seq_num 187) dirty exists 2024-04-22T18:39:54.821 INFO:tasks.rados.rados.0.smithi089.stdout:1178: expect (ObjNum 174 snap 0 seq_num 174) 2024-04-22T18:39:54.838 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:54.839+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:54.852 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:54.852 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:54.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:54.852 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:54.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.852 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:27 4.d e27/20 4:bee1184f:::smithi08937929-7:head v 27'3, mlcod=0'0)", 2024-04-22T18:39:54.852 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:54.852 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.487207695999999, 2024-04-22T18:39:54.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101135974, 2024-04-22T18:39:54.852 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.853 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.853 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:54.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:54.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365186+0000", 2024-04-22T18:39:54.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:54.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365194+0000", 2024-04-22T18:39:54.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3289999999999994e-06 2024-04-22T18:39:54.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365200+0000", 2024-04-22T18:39:54.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6590000000000004e-06 2024-04-22T18:39:54.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465352+0000", 2024-04-22T18:39:54.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10015208 2024-04-22T18:39:54.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465511+0000", 2024-04-22T18:39:54.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015880000000000001 2024-04-22T18:39:54.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465570+0000", 2024-04-22T18:39:54.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8952000000000001e-05 2024-04-22T18:39:54.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466292+0000", 2024-04-22T18:39:54.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072212100000000005 2024-04-22T18:39:54.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466323+0000", 2024-04-22T18:39:54.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0899999999999999e-05 2024-04-22T18:39:54.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.857 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.857 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.857 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:47 3.0 e30/19 3:30f16451:::smithi08937929-50 low_tier:head v 30'20, mlcod=29'18)", 2024-04-22T18:39:54.857 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:54.857 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.254291678999998, 2024-04-22T18:39:54.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100803438, 2024-04-22T18:39:54.857 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.857 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.857 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:54.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:54.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:54.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:54.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598108+0000", 2024-04-22T18:39:54.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9119999999999997e-06 2024-04-22T18:39:54.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598118+0000", 2024-04-22T18:39:54.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0138e-05 2024-04-22T18:39:54.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598125+0000", 2024-04-22T18:39:54.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3010000000000001e-06 2024-04-22T18:39:54.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698224+0000", 2024-04-22T18:39:54.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099221 2024-04-22T18:39:54.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698239+0000", 2024-04-22T18:39:54.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5053e-05 2024-04-22T18:39:54.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698875+0000", 2024-04-22T18:39:54.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063560699999999995 2024-04-22T18:39:54.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698906+0000", 2024-04-22T18:39:54.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1547999999999997e-05 2024-04-22T18:39:54.861 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.861 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.861 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.861 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.5.13:5 3.0 e30/19 3:1ab765eb:::smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=30'21)", 2024-04-22T18:39:54.862 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:54.862 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.649973555999999, 2024-04-22T18:39:54.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101014042, 2024-04-22T18:39:54.862 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.862 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.862 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:54.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:54.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:54.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:54.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202428+0000", 2024-04-22T18:39:54.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5889999999999997e-06 2024-04-22T18:39:54.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202446+0000", 2024-04-22T18:39:54.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8176999999999999e-05 2024-04-22T18:39:54.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202456+0000", 2024-04-22T18:39:54.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-04-22T18:39:54.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302599+0000", 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100142759 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302628+0000", 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9507999999999999e-05 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303350+0000", 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072151599999999995 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303435+0000", 2024-04-22T18:39:54.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5486000000000001e-05 2024-04-22T18:39:54.866 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.866 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.866 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.866 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:53 3.0 e31/19 3:25a201b0:::smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 31'24, mlcod=30'22)", 2024-04-22T18:39:54.866 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:54.866 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.449331221999998, 2024-04-22T18:39:54.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100751671, 2024-04-22T18:39:54.866 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.867 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.867 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:54.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:54.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403062+0000", 2024-04-22T18:39:54.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:54.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403070+0000", 2024-04-22T18:39:54.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8499999999999994e-06 2024-04-22T18:39:54.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403079+0000", 2024-04-22T18:39:54.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4620000000000002e-06 2024-04-22T18:39:54.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403087+0000", 2024-04-22T18:39:54.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4789999999999999e-06 2024-04-22T18:39:54.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503186+0000", 2024-04-22T18:39:54.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099271 2024-04-22T18:39:54.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503207+0000", 2024-04-22T18:39:54.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0973999999999999e-05 2024-04-22T18:39:54.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503773+0000", 2024-04-22T18:39:54.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056557600000000003 2024-04-22T18:39:54.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503815+0000", 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2441999999999998e-05 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:73 3.0 e34/19 3:061a7a63:::392f44ec26aa22e85be5f08273a7e45656f3bf4a4854038391a88e880ec01077:head v 34'31, mlcod=31'29)", 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.394467773999999, 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103503311, 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:54.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:54.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:54.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:54.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458193+0000", 2024-04-22T18:39:54.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026628900000000001 2024-04-22T18:39:54.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458208+0000", 2024-04-22T18:39:54.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4780000000000001e-05 2024-04-22T18:39:54.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458217+0000", 2024-04-22T18:39:54.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4909999999999994e-06 2024-04-22T18:39:54.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459162+0000", 2024-04-22T18:39:54.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094440500000000005 2024-04-22T18:39:54.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459185+0000", 2024-04-22T18:39:54.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3685000000000001e-05 2024-04-22T18:39:54.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561394+0000", 2024-04-22T18:39:54.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10220874100000001 2024-04-22T18:39:54.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561430+0000", 2024-04-22T18:39:54.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6304999999999999e-05 2024-04-22T18:39:54.876 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.876 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.876 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.876 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:75 3.0 e34/19 3:18418e0f:::b7c35f165c374d5e075dde9ac2607e47ace3b0a1e2343ec6efb171380f725eea:head v 34'32, mlcod=31'29)", 2024-04-22T18:39:54.876 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:54.876 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.393454501000001, 2024-04-22T18:39:54.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104798078, 2024-04-22T18:39:54.876 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.876 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.876 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:54.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:54.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458939+0000", 2024-04-22T18:39:54.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:54.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459092+0000", 2024-04-22T18:39:54.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153838 2024-04-22T18:39:54.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459107+0000", 2024-04-22T18:39:54.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4703e-05 2024-04-22T18:39:54.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459115+0000", 2024-04-22T18:39:54.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-04-22T18:39:54.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560093+0000", 2024-04-22T18:39:54.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10097840499999999 2024-04-22T18:39:54.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560109+0000", 2024-04-22T18:39:54.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5631e-05 2024-04-22T18:39:54.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563706+0000", 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035972059999999999 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563738+0000", 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2184000000000002e-05 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:77 3.0 e34/19 3:28387c9f:::b6d7840f02d561da04e398fbf94d91b85e9066b426e48baf3b11dd4a4f6096c0:head v 34'33, mlcod=31'29)", 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.392504420999998, 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104186079, 2024-04-22T18:39:54.880 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459889+0000", 2024-04-22T18:39:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459963+0000", 2024-04-22T18:39:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3732999999999994e-05 2024-04-22T18:39:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459973+0000", 2024-04-22T18:39:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0285999999999999e-05 2024-04-22T18:39:54.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459987+0000", 2024-04-22T18:39:54.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3675999999999999e-05 2024-04-22T18:39:54.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560515+0000", 2024-04-22T18:39:54.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10052781299999999 2024-04-22T18:39:54.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560538+0000", 2024-04-22T18:39:54.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3861000000000001e-05 2024-04-22T18:39:54.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564044+0000", 2024-04-22T18:39:54.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035052009999999999 2024-04-22T18:39:54.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564076+0000", 2024-04-22T18:39:54.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2737000000000002e-05 2024-04-22T18:39:54.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.886 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.886 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.886 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:78 3.0 e34/19 3:3a5582ac:::bdd92f7deaa2d50e204891f64f9043469ba687d78b218ed518f2483ba3486a99:head v 34'34, mlcod=31'29)", 2024-04-22T18:39:54.886 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:54.886 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.391838303, 2024-04-22T18:39:54.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103547263, 2024-04-22T18:39:54.886 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.886 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.886 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:54.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460555+0000", 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460861+0000", 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000305958 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460874+0000", 2024-04-22T18:39:54.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2639e-05 2024-04-22T18:39:54.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460881+0000", 2024-04-22T18:39:54.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2570000000000001e-06 2024-04-22T18:39:54.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560990+0000", 2024-04-22T18:39:54.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010883199999999 2024-04-22T18:39:54.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561011+0000", 2024-04-22T18:39:54.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1319e-05 2024-04-22T18:39:54.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564079+0000", 2024-04-22T18:39:54.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00306815 2024-04-22T18:39:54.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564104+0000", 2024-04-22T18:39:54.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4172000000000002e-05 2024-04-22T18:39:54.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.890 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.890 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:39:54.890 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:54.890 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.391223009000001, 2024-04-22T18:39:54.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:39:54.891 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.891 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.891 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:54.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:54.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:39:54.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:54.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:39:54.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:39:54.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:39:54.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:39:54.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:39:54.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:39:54.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:39:54.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:39:54.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:39:54.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:39:54.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:39:54.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:39:54.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:39:54.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:39:54.894 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.894 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.894 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.895 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:39:54.895 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:54.895 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.514549187, 2024-04-22T18:39:54.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:39:54.895 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.895 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.895 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:54.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:54.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:39:54.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:54.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:39:54.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:39:54.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:39:54.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:39:54.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:39:54.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:39:54.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:39:54.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:39:54.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:39:54.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:39:54.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:39:54.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:39:54.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:39:54.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:39:54.899 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.899 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.899 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.899 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1131 3.0 e41/19 3:01bdda42:::4a159391123755d93670b0d142ca36f3ee9a35236f6c4396c2aa92521a42a06b:head v 41'50, mlcod=41'48)", 2024-04-22T18:39:54.899 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:54.899 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.674802342, 2024-04-22T18:39:54.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101639456, 2024-04-22T18:39:54.899 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.899 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.899 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177590+0000", 2024-04-22T18:39:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177880+0000", 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029001 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177893+0000", 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177899+0000", 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1619999999999996e-06 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.278054+0000", 2024-04-22T18:39:54.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100154929 2024-04-22T18:39:54.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.278076+0000", 2024-04-22T18:39:54.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1308e-05 2024-04-22T18:39:54.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.279187+0000", 2024-04-22T18:39:54.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011109169999999999 2024-04-22T18:39:54.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.279232+0000", 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5188000000000002e-05 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1161 4.4 e41/20 4:24717d05:::smithi08937929-42:head v 41'12, mlcod=41'9)", 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.556330+0000", 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.296065040999999, 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099789016999999994, 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556330+0000", 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556330+0000", 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556328+0000", 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556335+0000", 2024-04-22T18:39:54.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9770000000000003e-06 2024-04-22T18:39:54.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556346+0000", 2024-04-22T18:39:54.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0118999999999999e-05 2024-04-22T18:39:54.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.556353+0000", 2024-04-22T18:39:54.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3730000000000004e-06 2024-04-22T18:39:54.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.655508+0000", 2024-04-22T18:39:54.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099155219000000003 2024-04-22T18:39:54.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.655536+0000", 2024-04-22T18:39:54.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-05 2024-04-22T18:39:54.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.656078+0000", 2024-04-22T18:39:54.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054138999999999999 2024-04-22T18:39:54.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.656119+0000", 2024-04-22T18:39:54.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1026000000000003e-05 2024-04-22T18:39:54.907 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.907 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.907 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.907 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:125 3.0 e43/19 3:376bc4bb:::8f967acf6e5865f87cd7cada681f96ff59d5fc3f243a710c00c052aacb952df3:head v 43'60, mlcod=42'57)", 2024-04-22T18:39:54.907 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:54.907 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.004828959999998, 2024-04-22T18:39:54.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100905994, 2024-04-22T18:39:54.907 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.907 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.907 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:54.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:54.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847565+0000", 2024-04-22T18:39:54.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:54.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847610+0000", 2024-04-22T18:39:54.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4984000000000002e-05 2024-04-22T18:39:54.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847619+0000", 2024-04-22T18:39:54.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2189999999999993e-06 2024-04-22T18:39:54.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947667+0000", 2024-04-22T18:39:54.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100047939 2024-04-22T18:39:54.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947716+0000", 2024-04-22T18:39:54.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9228999999999999e-05 2024-04-22T18:39:54.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947738+0000", 2024-04-22T18:39:54.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1963999999999998e-05 2024-04-22T18:39:54.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.948440+0000", 2024-04-22T18:39:54.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070173999999999996 2024-04-22T18:39:54.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.948472+0000", 2024-04-22T18:39:54.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1872999999999998e-05 2024-04-22T18:39:54.911 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.911 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.912 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.912 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:79 3.0 e44/19 3:05e6fa13:::smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 44'64, mlcod=44'61)", 2024-04-22T18:39:54.912 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:54.912 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.481547663999997, 2024-04-22T18:39:54.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10093532199999999, 2024-04-22T18:39:54.912 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.912 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.912 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:54.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370846+0000", 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370857+0000", 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1155000000000001e-05 2024-04-22T18:39:54.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370866+0000", 2024-04-22T18:39:54.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8769999999999999e-06 2024-04-22T18:39:54.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.470937+0000", 2024-04-22T18:39:54.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100071204 2024-04-22T18:39:54.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.470986+0000", 2024-04-22T18:39:54.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8556000000000001e-05 2024-04-22T18:39:54.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471014+0000", 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7982000000000001e-05 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471750+0000", 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073598100000000003 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471782+0000", 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339000000000003e-05 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.915 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1689 3.0 e47/19 3:1d4bcadf:::e7eb8db0531d9f456e91fe87bc79e47ebd80459f1783dc2a5e53d4822cd34f49:head v 47'85, mlcod=47'83)", 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.143506728999999, 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102152033, 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:39:54.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708885+0000", 2024-04-22T18:39:54.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:54.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709414+0000", 2024-04-22T18:39:54.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000528459 2024-04-22T18:39:54.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709428+0000", 2024-04-22T18:39:54.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4745e-05 2024-04-22T18:39:54.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709436+0000", 2024-04-22T18:39:54.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0830000000000004e-06 2024-04-22T18:39:54.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.809587+0000", 2024-04-22T18:39:54.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100150641 2024-04-22T18:39:54.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.809603+0000", 2024-04-22T18:39:54.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6061000000000001e-05 2024-04-22T18:39:54.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.811013+0000", 2024-04-22T18:39:54.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0014094680000000001 2024-04-22T18:39:54.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.811040+0000", 2024-04-22T18:39:54.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7474000000000002e-05 2024-04-22T18:39:54.920 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.920 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.920 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.920 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1729 3.0 e47/19 3:27b110a1:::34336650543f8f8bff225091f9c8c14678dc47baabcb4d8371890fc935dbc3e4:head v 47'89, mlcod=47'87)", 2024-04-22T18:39:54.920 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:39:54.920 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.374452958999999, 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102111412, 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477940+0000", 2024-04-22T18:39:54.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:54.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478305+0000", 2024-04-22T18:39:54.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036512200000000001 2024-04-22T18:39:54.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478320+0000", 2024-04-22T18:39:54.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5475999999999999e-05 2024-04-22T18:39:54.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478326+0000", 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.302e-06 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.578536+0000", 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100209436 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.578575+0000", 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9223e-05 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:54.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.580023+0000", 2024-04-22T18:39:54.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001447463 2024-04-22T18:39:54.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.580053+0000", 2024-04-22T18:39:54.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0534999999999998e-05 2024-04-22T18:39:54.924 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.924 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.924 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.924 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGRecoveryDelete(3.0 e65,63 3:255bc945:::0cf7ba283f6e98a1dfe8fb74a0dc7464561757ac03eac554ddfa7f49f708eaf6:head,47'87)", 2024-04-22T18:39:54.924 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:53.725840+0000", 2024-04-22T18:39:54.924 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.1265543920000001, 2024-04-22T18:39:54.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10024833900000001, 2024-04-22T18:39:54.925 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.925 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:54.925 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.725840+0000", 2024-04-22T18:39:54.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.725840+0000", 2024-04-22T18:39:54.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.725839+0000", 2024-04-22T18:39:54.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:54.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.725845+0000", 2024-04-22T18:39:54.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.516e-06 2024-04-22T18:39:54.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.725858+0000", 2024-04-22T18:39:54.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.277e-05 2024-04-22T18:39:54.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.725868+0000", 2024-04-22T18:39:54.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0028e-05 2024-04-22T18:39:54.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.825987+0000", 2024-04-22T18:39:54.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100118847 2024-04-22T18:39:54.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.825996+0000", 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2350000000000005e-06 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.826089+0000", 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2292000000000003e-05 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGRecoveryDelete(3.0 e65,63 3:15548ae8:::smithi08937929-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head,48'91)", 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:53.931993+0000", 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.92040132200000002, 2024-04-22T18:39:54.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100310439, 2024-04-22T18:39:54.929 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.929 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:54.929 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.931993+0000", 2024-04-22T18:39:54.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.931993+0000", 2024-04-22T18:39:54.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.931991+0000", 2024-04-22T18:39:54.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:54.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.932003+0000", 2024-04-22T18:39:54.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1620999999999999e-05 2024-04-22T18:39:54.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.932020+0000", 2024-04-22T18:39:54.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7325000000000001e-05 2024-04-22T18:39:54.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.932033+0000", 2024-04-22T18:39:54.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3076e-05 2024-04-22T18:39:54.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.032180+0000", 2024-04-22T18:39:54.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100146548 2024-04-22T18:39:54.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.032193+0000", 2024-04-22T18:39:54.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3006999999999999e-05 2024-04-22T18:39:54.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.032304+0000", 2024-04-22T18:39:54.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000111234 2024-04-22T18:39:54.932 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.932 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.932 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.932 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGRecoveryDelete(3.0 e65,63 3:1ea7c794:::bb3bf927b22c6786279d3d3abcc10e44cfd13a6005614d74d8894d5eccd91ad3:head,51'98)", 2024-04-22T18:39:54.932 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:54.437078+0000", 2024-04-22T18:39:54.932 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.41531706200000001, 2024-04-22T18:39:54.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100233899, 2024-04-22T18:39:54.933 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.933 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:54.933 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.437078+0000", 2024-04-22T18:39:54.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.437078+0000", 2024-04-22T18:39:54.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.437076+0000", 2024-04-22T18:39:54.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:54.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.437083+0000", 2024-04-22T18:39:54.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7270000000000001e-06 2024-04-22T18:39:54.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.437095+0000", 2024-04-22T18:39:54.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1855000000000001e-05 2024-04-22T18:39:54.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.437103+0000", 2024-04-22T18:39:54.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7649999999999993e-06 2024-04-22T18:39:54.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.537234+0000", 2024-04-22T18:39:54.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100131132 2024-04-22T18:39:54.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.537242+0000", 2024-04-22T18:39:54.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5720000000000003e-06 2024-04-22T18:39:54.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.537312+0000", 2024-04-22T18:39:54.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9055999999999997e-05 2024-04-22T18:39:54.936 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.936 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.936 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGRecoveryDelete(3.0 e65,63 3:1f6d22b5:::3612666ea96266e6d282dc1f55eda8fa070760fc41b15fcf9dccb7b58ba10707:head,51'101)", 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:54.746293+0000", 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.10610122399999999, 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100233402, 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.746293+0000", 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:54.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.746293+0000", 2024-04-22T18:39:54.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:54.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:54.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.746291+0000", 2024-04-22T18:39:54.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:54.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:54.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.746302+0000", 2024-04-22T18:39:54.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1236000000000001e-05 2024-04-22T18:39:54.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:54.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.746317+0000", 2024-04-22T18:39:54.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4535000000000001e-05 2024-04-22T18:39:54.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:54.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.746327+0000", 2024-04-22T18:39:54.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.013e-05 2024-04-22T18:39:54.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:54.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.846441+0000", 2024-04-22T18:39:54.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100113811 2024-04-22T18:39:54.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:54.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.846448+0000", 2024-04-22T18:39:54.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4429999999999997e-06 2024-04-22T18:39:54.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:54.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:54.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:54.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.846527+0000", 2024-04-22T18:39:54.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8585999999999996e-05 2024-04-22T18:39:54.940 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.940 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.940 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.940 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:54.940 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:54.940 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:54.942 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:54.942 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:54.943+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:54.986 INFO:tasks.rados.rados.0.smithi089.stdout:1179: expect (ObjNum 184 snap 0 seq_num 184) 2024-04-22T18:39:55.042 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:55.044+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:55.084 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:55.084 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:55.084 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:55.084 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:55.093 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:55.143 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:55.144+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:55.167 INFO:tasks.rados.rados.0.smithi089.stdout:1180: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2024-04-22T18:39:55.173 INFO:tasks.rados.rados.0.smithi089.stdout:1180: writing smithi08937929-32 from 712663 to 1181235 tid 1 2024-04-22T18:39:55.180 INFO:tasks.rados.rados.0.smithi089.stdout:1180: writing smithi08937929-32 from 1713913 to 2391970 tid 2 2024-04-22T18:39:55.187 INFO:tasks.rados.rados.0.smithi089.stdout:1180: writing smithi08937929-32 from 2873523 to 3465256 tid 3 2024-04-22T18:39:55.187 INFO:tasks.rados.rados.0.smithi089.stdout:1175: done (5 left) 2024-04-22T18:39:55.187 INFO:tasks.rados.rados.0.smithi089.stdout:1176: done (4 left) 2024-04-22T18:39:55.187 INFO:tasks.rados.rados.0.smithi089.stdout:1177: done (3 left) 2024-04-22T18:39:55.187 INFO:tasks.rados.rados.0.smithi089.stdout:1178: done (2 left) 2024-04-22T18:39:55.187 INFO:tasks.rados.rados.0.smithi089.stdout:1179: done (1 left) 2024-04-22T18:39:55.187 INFO:tasks.rados.rados.0.smithi089.stdout:1181: read oid 20 snap -1 2024-04-22T18:39:55.187 INFO:tasks.rados.rados.0.smithi089.stdout:1181: expect (ObjNum 201 snap 0 seq_num 201) 2024-04-22T18:39:55.187 INFO:tasks.rados.rados.0.smithi089.stdout:1182: read oid 50 snap -1 2024-04-22T18:39:55.187 INFO:tasks.rados.rados.0.smithi089.stdout:1182: expect (ObjNum 189 snap 0 seq_num 189) 2024-04-22T18:39:55.187 INFO:tasks.rados.rados.0.smithi089.stdout:1183: write oid 43 current snap is 0 2024-04-22T18:39:55.187 INFO:tasks.rados.rados.0.smithi089.stdout:1183: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2024-04-22T18:39:55.194 INFO:tasks.rados.rados.0.smithi089.stdout:1183: writing smithi08937929-43 from 729470 to 1273291 tid 1 2024-04-22T18:39:55.199 INFO:tasks.rados.rados.0.smithi089.stdout:1183: writing smithi08937929-43 from 1772395 to 2310463 tid 2 2024-04-22T18:39:55.205 INFO:tasks.rados.rados.0.smithi089.stdout:1183: writing smithi08937929-43 from 2900946 to 3303951 tid 3 2024-04-22T18:39:55.205 INFO:tasks.rados.rados.0.smithi089.stdout:1184: write oid 4 current snap is 0 2024-04-22T18:39:55.205 INFO:tasks.rados.rados.0.smithi089.stdout:1184: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-04-22T18:39:55.212 INFO:tasks.rados.rados.0.smithi089.stdout:1184: writing smithi08937929-4 from 746277 to 1365347 tid 1 2024-04-22T18:39:55.218 INFO:tasks.rados.rados.0.smithi089.stdout:1184: writing smithi08937929-4 from 1830877 to 2512603 tid 2 2024-04-22T18:39:55.224 INFO:tasks.rados.rados.0.smithi089.stdout:1184: writing smithi08937929-4 from 2928369 to 3546277 tid 3 2024-04-22T18:39:55.224 INFO:tasks.rados.rados.0.smithi089.stdout:1185: read oid 34 snap -1 2024-04-22T18:39:55.224 INFO:tasks.rados.rados.0.smithi089.stdout:1185: expect (ObjNum 205 snap 0 seq_num 205) 2024-04-22T18:39:55.224 INFO:tasks.rados.rados.0.smithi089.stdout:1186: write (excl) oid 35 current snap is 0 2024-04-22T18:39:55.225 INFO:tasks.rados.rados.0.smithi089.stdout:1186: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2024-04-22T18:39:55.232 INFO:tasks.rados.rados.0.smithi089.stdout:1186: writing smithi08937929-35 from 763084 to 1457403 tid 1 2024-04-22T18:39:55.237 INFO:tasks.rados.rados.0.smithi089.stdout:1186: writing smithi08937929-35 from 1889359 to 2431096 tid 2 2024-04-22T18:39:55.243 INFO:tasks.rados.rados.0.smithi089.stdout:1186: writing smithi08937929-35 from 2955792 to 3497694 tid 3 2024-04-22T18:39:55.243 INFO:tasks.rados.rados.0.smithi089.stdout:1187: delete oid 31 current snap is 0 2024-04-22T18:39:55.243 INFO:tasks.rados.rados.0.smithi089.stdout:1182: expect (ObjNum 189 snap 0 seq_num 189) 2024-04-22T18:39:55.244 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:55.244+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:55.245 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:55.245 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:55.245 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:55.246 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:55.246 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:55.255 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:55.344 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:55.345+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:55.400 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:55.400 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:55.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:55.400 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:55.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.400 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:45 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'1, mlcod=0'0)", 2024-04-22T18:39:55.400 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:55.400 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.134875745000002, 2024-04-22T18:39:55.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14345418900000001, 2024-04-22T18:39:55.400 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265727+0000", 2024-04-22T18:39:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061502799999999999 2024-04-22T18:39:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265737+0000", 2024-04-22T18:39:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0278999999999999e-05 2024-04-22T18:39:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265744+0000", 2024-04-22T18:39:55.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1740000000000003e-06 2024-04-22T18:39:55.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407227+0000", 2024-04-22T18:39:55.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14148266900000001 2024-04-22T18:39:55.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407258+0000", 2024-04-22T18:39:55.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1619000000000003e-05 2024-04-22T18:39:55.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408534+0000", 2024-04-22T18:39:55.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001275829 2024-04-22T18:39:55.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408566+0000", 2024-04-22T18:39:55.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1813999999999997e-05 2024-04-22T18:39:55.404 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.404 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.404 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.404 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:78 4.0 4:01baf5a9:::smithi08937929-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:55.404 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:55.405 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.888052651000002, 2024-04-22T18:39:55.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.398522192, 2024-04-22T18:39:55.405 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.405 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.405 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:55.405 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:55.405 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:55.405 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 78 2024-04-22T18:39:55.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.405 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:55.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:55.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511934+0000", 2024-04-22T18:39:55.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:55.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512012+0000", 2024-04-22T18:39:55.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8329000000000002e-05 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512023+0000", 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0755e-05 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512028+0000", 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6470000000000003e-06 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512068+0000", 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9721999999999997e-05 2024-04-22T18:39:55.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512125+0000", 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6564999999999998e-05 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512200+0000", 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5708000000000001e-05 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513131+0000", 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093063 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910394+0000", 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.397263388 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910430+0000", 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5485999999999999e-05 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910457+0000", 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7086000000000001e-05 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.410 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:79 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:55.410 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:55.410 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.887951995000002, 2024-04-22T18:39:55.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39855419399999997, 2024-04-22T18:39:55.410 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.410 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.410 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:55.410 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:55.410 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:55.410 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 79 2024-04-22T18:39:55.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.410 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512035+0000", 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512045+0000", 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0379e-05 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512051+0000", 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6030000000000004e-06 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512057+0000", 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7270000000000004e-06 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512756+0000", 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069954000000000001 2024-04-22T18:39:55.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512806+0000", 2024-04-22T18:39:55.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9922000000000001e-05 2024-04-22T18:39:55.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:55.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512927+0000", 2024-04-22T18:39:55.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012059 2024-04-22T18:39:55.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:55.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.515883+0000", 2024-04-22T18:39:55.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029563739999999999 2024-04-22T18:39:55.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:55.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910510+0000", 2024-04-22T18:39:55.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39462745199999999 2024-04-22T18:39:55.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910542+0000", 2024-04-22T18:39:55.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1225e-05 2024-04-22T18:39:55.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910590+0000", 2024-04-22T18:39:55.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8052e-05 2024-04-22T18:39:55.414 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:80 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.887926894000003, 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39866376999999997, 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 80 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512060+0000", 2024-04-22T18:39:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512063+0000", 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.999e-06 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512069+0000", 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.891e-06 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512073+0000", 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4429999999999996e-06 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513084+0000", 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010102609999999999 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513123+0000", 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9237999999999999e-05 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910618+0000", 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39749549299999998 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910670+0000", 2024-04-22T18:39:55.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1221000000000003e-05 2024-04-22T18:39:55.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910724+0000", 2024-04-22T18:39:55.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4777000000000001e-05 2024-04-22T18:39:55.419 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.419 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.419 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.419 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:5 3.2 e30/29 3:663b82a1:::smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=0'0)", 2024-04-22T18:39:55.419 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:55.419 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 57.172487576000002, 2024-04-22T18:39:55.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20090134500000001, 2024-04-22T18:39:55.420 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.420 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.420 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:55.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:55.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227499+0000", 2024-04-22T18:39:55.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:55.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227506+0000", 2024-04-22T18:39:55.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6540000000000002e-06 2024-04-22T18:39:55.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227516+0000", 2024-04-22T18:39:55.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0054000000000001e-05 2024-04-22T18:39:55.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327607+0000", 2024-04-22T18:39:55.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009167099999999 2024-04-22T18:39:55.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327688+0000", 2024-04-22T18:39:55.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1000000000000004e-05 2024-04-22T18:39:55.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327719+0000", 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0664000000000003e-05 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428371+0000", 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10065239099999999 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428401+0000", 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9839e-05 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:419 4.0 4:01baf5a9:::smithi08937929-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.997765874000002, 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30324478199999999, 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 419 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:55.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402219+0000", 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402229+0000", 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2550000000000003e-06 2024-04-22T18:39:55.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402242+0000", 2024-04-22T18:39:55.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2989999999999999e-05 2024-04-22T18:39:55.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402250+0000", 2024-04-22T18:39:55.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7900000000000005e-06 2024-04-22T18:39:55.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404085+0000", 2024-04-22T18:39:55.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001834397 2024-04-22T18:39:55.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:55.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404092+0000", 2024-04-22T18:39:55.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7889999999999997e-06 2024-04-22T18:39:55.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502534+0000", 2024-04-22T18:39:55.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098442331999999994 2024-04-22T18:39:55.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502631+0000", 2024-04-22T18:39:55.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7617999999999999e-05 2024-04-22T18:39:55.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:55.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604414+0000", 2024-04-22T18:39:55.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101782847 2024-04-22T18:39:55.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:55.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604888+0000", 2024-04-22T18:39:55.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047325999999999998 2024-04-22T18:39:55.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:55.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705376+0000", 2024-04-22T18:39:55.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100488623 2024-04-22T18:39:55.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705424+0000", 2024-04-22T18:39:55.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7964999999999999e-05 2024-04-22T18:39:55.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705466+0000", 2024-04-22T18:39:55.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2274999999999998e-05 2024-04-22T18:39:55.429 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.429 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.429 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:70 3.2 e32/29 3:5f0623ce:::smithi08937929-38 low_tier:head v 32'37, mlcod=31'35)", 2024-04-22T18:39:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.871548842999999, 2024-04-22T18:39:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20070027900000001, 2024-04-22T18:39:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528438+0000", 2024-04-22T18:39:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528443+0000", 2024-04-22T18:39:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8640000000000004e-06 2024-04-22T18:39:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528453+0000", 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3910000000000004e-06 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528459+0000", 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728627+0000", 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20016778700000001 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728660+0000", 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3024000000000001e-05 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729107+0000", 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044719200000000001 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729139+0000", 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1625000000000003e-05 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1264 4.c e43/20 4:36664a98:::smithi08937929-38:head v 43'37, mlcod=42'34)", 2024-04-22T18:39:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.743650439, 2024-04-22T18:39:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20290602399999999, 2024-04-22T18:39:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656335+0000", 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656850+0000", 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051499499999999997 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656861+0000", 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1119e-05 2024-04-22T18:39:55.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756934+0000", 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073548 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756977+0000", 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2218000000000001e-05 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756993+0000", 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6629999999999998e-05 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859203+0000", 2024-04-22T18:39:55.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102209617 2024-04-22T18:39:55.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859243+0000", 2024-04-22T18:39:55.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0333000000000002e-05 2024-04-22T18:39:55.437 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.437 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.437 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.437 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1747 4.c e48/20 4:35eda2ca:::smithi08937929-12:head v 48'65, mlcod=47'61)", 2024-04-22T18:39:55.437 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:32.597827+0000", 2024-04-22T18:39:55.438 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.802160297, 2024-04-22T18:39:55.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20085196499999999, 2024-04-22T18:39:55.438 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.438 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.438 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597827+0000", 2024-04-22T18:39:55.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597827+0000", 2024-04-22T18:39:55.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597825+0000", 2024-04-22T18:39:55.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:55.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597839+0000", 2024-04-22T18:39:55.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4329e-05 2024-04-22T18:39:55.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597856+0000", 2024-04-22T18:39:55.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6858e-05 2024-04-22T18:39:55.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.697948+0000", 2024-04-22T18:39:55.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092367 2024-04-22T18:39:55.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798102+0000", 2024-04-22T18:39:55.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100153411 2024-04-22T18:39:55.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798122+0000", 2024-04-22T18:39:55.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0069e-05 2024-04-22T18:39:55.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798628+0000", 2024-04-22T18:39:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050619700000000001 2024-04-22T18:39:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798679+0000", 2024-04-22T18:39:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1307000000000002e-05 2024-04-22T18:39:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.441 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:229 3.2 e51/29 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head v 51'117, mlcod=51'115)", 2024-04-22T18:39:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:39:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.588944836, 2024-04-22T18:39:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31548799399999999, 2024-04-22T18:39:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:39:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:39:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811041+0000", 2024-04-22T18:39:55.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:55.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811054+0000", 2024-04-22T18:39:55.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3188e-05 2024-04-22T18:39:55.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811068+0000", 2024-04-22T18:39:55.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4239999999999999e-05 2024-04-22T18:39:55.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.911141+0000", 2024-04-22T18:39:55.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100072851 2024-04-22T18:39:55.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025053+0000", 2024-04-22T18:39:55.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.11391237 2024-04-22T18:39:55.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025083+0000", 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9703999999999999e-05 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126504+0000", 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10142053500000001 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126531+0000", 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7172999999999999e-05 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.446 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:235 3.2 e51/29 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head v 51'118, mlcod=51'115)", 2024-04-22T18:39:55.446 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.924871+0000", 2024-04-22T18:39:55.446 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.475116634999999, 2024-04-22T18:39:55.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20167500599999999, 2024-04-22T18:39:55.446 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.446 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.446 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924871+0000", 2024-04-22T18:39:55.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924871+0000", 2024-04-22T18:39:55.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924869+0000", 2024-04-22T18:39:55.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:55.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924883+0000", 2024-04-22T18:39:55.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3725e-05 2024-04-22T18:39:55.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924899+0000", 2024-04-22T18:39:55.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6344000000000001e-05 2024-04-22T18:39:55.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924907+0000", 2024-04-22T18:39:55.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0909999999999993e-06 2024-04-22T18:39:55.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025426+0000", 2024-04-22T18:39:55.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100519109 2024-04-22T18:39:55.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025451+0000", 2024-04-22T18:39:55.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4338e-05 2024-04-22T18:39:55.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126534+0000", 2024-04-22T18:39:55.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101083083 2024-04-22T18:39:55.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126546+0000", 2024-04-22T18:39:55.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2122e-05 2024-04-22T18:39:55.450 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.450 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.450 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.450 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:238 3.2 e51/29 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head v 51'119, mlcod=51'115)", 2024-04-22T18:39:55.450 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.925398+0000", 2024-04-22T18:39:55.450 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.474589853000001, 2024-04-22T18:39:55.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.201162223, 2024-04-22T18:39:55.450 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.450 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.450 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925398+0000", 2024-04-22T18:39:55.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925398+0000", 2024-04-22T18:39:55.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925396+0000", 2024-04-22T18:39:55.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:55.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925405+0000", 2024-04-22T18:39:55.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6600000000000001e-06 2024-04-22T18:39:55.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925417+0000", 2024-04-22T18:39:55.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2697000000000001e-05 2024-04-22T18:39:55.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925425+0000", 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3000000000000004e-06 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.125722+0000", 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20029754299999999 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.125763+0000", 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0408999999999999e-05 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126548+0000", 2024-04-22T18:39:55.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00078496600000000001 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126560+0000", 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2227e-05 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:118 3.2 e51/29 3:43d85947:::smithi08937929-18 low_tier:head v 51'120, mlcod=51'115)", 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.983526+0000", 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.416461865999999, 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14328775899999999, 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.454 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.455 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983526+0000", 2024-04-22T18:39:55.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983526+0000", 2024-04-22T18:39:55.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983524+0000", 2024-04-22T18:39:55.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:55.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983538+0000", 2024-04-22T18:39:55.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3858e-05 2024-04-22T18:39:55.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983552+0000", 2024-04-22T18:39:55.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4466e-05 2024-04-22T18:39:55.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983562+0000", 2024-04-22T18:39:55.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3899999999999999e-06 2024-04-22T18:39:55.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126168+0000", 2024-04-22T18:39:55.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14260677899999999 2024-04-22T18:39:55.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126185+0000", 2024-04-22T18:39:55.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6625000000000001e-05 2024-04-22T18:39:55.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126794+0000", 2024-04-22T18:39:55.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060855999999999996 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126813+0000", 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9851e-05 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:119 3.2 e51/29 3:43d85947:::smithi08937929-18 low_tier:head v 51'121, mlcod=51'115)", 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.416139166000001, 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.145803025, 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.458 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:55.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983854+0000", 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.207e-06 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983860+0000", 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5350000000000001e-06 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983865+0000", 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0170000000000004e-06 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126364+0000", 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.142498491 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126378+0000", 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4134999999999999e-05 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129628+0000", 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032498940000000001 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129651+0000", 2024-04-22T18:39:55.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3493999999999999e-05 2024-04-22T18:39:55.462 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.462 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.462 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.462 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:120 3.2 e51/29 3:43d85947:::smithi08937929-18 low_tier:head v 51'122, mlcod=51'115)", 2024-04-22T18:39:55.462 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.984299+0000", 2024-04-22T18:39:55.462 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.415688770999999, 2024-04-22T18:39:55.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14536710899999999, 2024-04-22T18:39:55.462 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.462 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.462 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984299+0000", 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984299+0000", 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984298+0000", 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984302+0000", 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.929e-06 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984309+0000", 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2080000000000003e-06 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984316+0000", 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7329999999999997e-06 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126582+0000", 2024-04-22T18:39:55.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14226603400000001 2024-04-22T18:39:55.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126595+0000", 2024-04-22T18:39:55.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3458000000000001e-05 2024-04-22T18:39:55.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129653+0000", 2024-04-22T18:39:55.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030578860000000001 2024-04-22T18:39:55.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129666+0000", 2024-04-22T18:39:55.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2483999999999999e-05 2024-04-22T18:39:55.466 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.466 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.466 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.466 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2344 4.0 4:01baf5a9:::smithi08937929-20:head [write 578207~550227,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:39:55.466 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.087243+0000", 2024-04-22T18:39:55.466 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.3127441969999998, 2024-04-22T18:39:55.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20456982100000001, 2024-04-22T18:39:55.466 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.466 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.466 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:55.467 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:55.467 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:55.467 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2344 2024-04-22T18:39:55.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.467 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.087243+0000", 2024-04-22T18:39:55.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.087243+0000", 2024-04-22T18:39:55.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.087240+0000", 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.087373+0000", 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013313999999999999 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.087390+0000", 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7538e-05 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187547+0000", 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100156021 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187667+0000", 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000120364 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187724+0000", 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6984999999999997e-05 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:55.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187823+0000", 2024-04-22T18:39:55.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8652000000000003e-05 2024-04-22T18:39:55.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:55.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291749+0000", 2024-04-22T18:39:55.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103926662 2024-04-22T18:39:55.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:55.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291757+0000", 2024-04-22T18:39:55.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0660000000000004e-06 2024-04-22T18:39:55.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291789+0000", 2024-04-22T18:39:55.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1701e-05 2024-04-22T18:39:55.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291813+0000", 2024-04-22T18:39:55.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4199e-05 2024-04-22T18:39:55.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.471 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.471 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2345 4.0 4:01baf5a9:::smithi08937929-20:head [write 1762410~779734] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:39:55.471 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187577+0000", 2024-04-22T18:39:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.2124101989999998, 2024-04-22T18:39:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20497090500000001, 2024-04-22T18:39:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2345 2024-04-22T18:39:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187577+0000", 2024-04-22T18:39:55.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187577+0000", 2024-04-22T18:39:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187575+0000", 2024-04-22T18:39:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187749+0000", 2024-04-22T18:39:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000173812 2024-04-22T18:39:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187756+0000", 2024-04-22T18:39:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6340000000000001e-06 2024-04-22T18:39:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187761+0000", 2024-04-22T18:39:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9760000000000003e-06 2024-04-22T18:39:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.289300+0000", 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10153891399999999 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.289359+0000", 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8866999999999997e-05 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.289465+0000", 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010571199999999999 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:55.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292263+0000", 2024-04-22T18:39:55.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0027986259999999998 2024-04-22T18:39:55.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:55.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.392486+0000", 2024-04-22T18:39:55.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100222441 2024-04-22T18:39:55.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.392523+0000", 2024-04-22T18:39:55.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6792999999999998e-05 2024-04-22T18:39:55.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.392548+0000", 2024-04-22T18:39:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.55e-05 2024-04-22T18:39:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2346 4.0 4:01baf5a9:::smithi08937929-20:head [write 3286845~91362] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:39:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187766+0000", 2024-04-22T18:39:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.2122218150000004, 2024-04-22T18:39:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20486648599999999, 2024-04-22T18:39:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:55.478 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2346 2024-04-22T18:39:55.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.478 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187766+0000", 2024-04-22T18:39:55.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187766+0000", 2024-04-22T18:39:55.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187765+0000", 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187808+0000", 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2911999999999998e-05 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187813+0000", 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.848e-06 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187818+0000", 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9819999999999999e-06 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291118+0000", 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103300054 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291176+0000", 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8350000000000002e-05 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:55.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291267+0000", 2024-04-22T18:39:55.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0248999999999996e-05 2024-04-22T18:39:55.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:55.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292266+0000", 2024-04-22T18:39:55.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00099966599999999997 2024-04-22T18:39:55.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:55.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.392581+0000", 2024-04-22T18:39:55.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100314291 2024-04-22T18:39:55.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.392612+0000", 2024-04-22T18:39:55.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1044000000000003e-05 2024-04-22T18:39:55.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.392632+0000", 2024-04-22T18:39:55.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0632000000000002e-05 2024-04-22T18:39:55.482 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.482 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.482 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.482 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2347 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 3378207] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:39:55.482 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:39:55.482 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.2121666109999998, 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30520418599999999, 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2347 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:39:55.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187820+0000", 2024-04-22T18:39:55.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:55.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187823+0000", 2024-04-22T18:39:55.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6299999999999998e-06 2024-04-22T18:39:55.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187827+0000", 2024-04-22T18:39:55.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3739999999999998e-06 2024-04-22T18:39:55.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187829+0000", 2024-04-22T18:39:55.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6290000000000002e-06 2024-04-22T18:39:55.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291848+0000", 2024-04-22T18:39:55.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104018812 2024-04-22T18:39:55.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291924+0000", 2024-04-22T18:39:55.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6421000000000006e-05 2024-04-22T18:39:55.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:55.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292050+0000", 2024-04-22T18:39:55.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012571500000000001 2024-04-22T18:39:55.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:55.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292407+0000", 2024-04-22T18:39:55.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035709199999999999 2024-04-22T18:39:55.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:55.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.492962+0000", 2024-04-22T18:39:55.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20055479200000001 2024-04-22T18:39:55.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:55.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.492993+0000", 2024-04-22T18:39:55.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1041000000000003e-05 2024-04-22T18:39:55.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493025+0000", 2024-04-22T18:39:55.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2092999999999999e-05 2024-04-22T18:39:55.487 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.487 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2348 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.2121552949999996, 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30532242999999998, 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2348 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:55.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:39:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:39:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:39:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:55.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187833+0000", 2024-04-22T18:39:55.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.437e-06 2024-04-22T18:39:55.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:55.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187836+0000", 2024-04-22T18:39:55.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.897e-06 2024-04-22T18:39:55.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187839+0000", 2024-04-22T18:39:55.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7250000000000002e-06 2024-04-22T18:39:55.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292213+0000", 2024-04-22T18:39:55.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104373761 2024-04-22T18:39:55.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:55.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292256+0000", 2024-04-22T18:39:55.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2896999999999999e-05 2024-04-22T18:39:55.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:55.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493044+0000", 2024-04-22T18:39:55.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20078818700000001 2024-04-22T18:39:55.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:55.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493099+0000", 2024-04-22T18:39:55.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5009000000000001e-05 2024-04-22T18:39:55.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:55.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:55.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:55.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493155+0000", 2024-04-22T18:39:55.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5807000000000003e-05 2024-04-22T18:39:55.492 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.492 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.492 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.492 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:55.492 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:55.492 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:55.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:55.494 INFO:tasks.rados.rados.0.smithi089.stdout:1180: finishing write tid 1 to smithi08937929-32 2024-04-22T18:39:55.494 INFO:tasks.rados.rados.0.smithi089.stdout:1180: finishing write tid 2 to smithi08937929-32 2024-04-22T18:39:55.495 INFO:tasks.rados.rados.0.smithi089.stdout:1180: finishing write tid 3 to smithi08937929-32 2024-04-22T18:39:55.495 INFO:tasks.rados.rados.0.smithi089.stdout:1181: expect (ObjNum 201 snap 0 seq_num 201) 2024-04-22T18:39:55.495 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:55.494+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:55.594 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:55.595+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:55.607 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2510 4.9 4:96737e19:::smithi08937929-4:head [setxattr _header (55) in=62b,truncate 3546277] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:39:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.316723+0000", 2024-04-22T18:39:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.29194582099999999, 2024-04-22T18:39:55.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29197061899999999, 2024-04-22T18:39:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:39:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2510 2024-04-22T18:39:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.316723+0000", 2024-04-22T18:39:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.316723+0000", 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.316722+0000", 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.316728+0000", 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4400000000000002e-06 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:55.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.316734+0000", 2024-04-22T18:39:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.173e-06 2024-04-22T18:39:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.316741+0000", 2024-04-22T18:39:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3890000000000002e-06 2024-04-22T18:39:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.421301+0000", 2024-04-22T18:39:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104560115 2024-04-22T18:39:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:55.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.421366+0000", 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4537999999999998e-05 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.421465+0000", 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9145e-05 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.425894+0000", 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0044292929999999999 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2511 4.9 4:96737e19:::smithi08937929-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:39:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.316745+0000", 2024-04-22T18:39:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.29192405900000001, 2024-04-22T18:39:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29199691999999999, 2024-04-22T18:39:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for rw locks", 2024-04-22T18:39:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2511 2024-04-22T18:39:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:55.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.316745+0000", 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.316745+0000", 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.316744+0000", 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.316747+0000", 2024-04-22T18:39:55.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:39:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.316753+0000", 2024-04-22T18:39:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0329999999999997e-06 2024-04-22T18:39:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.316759+0000", 2024-04-22T18:39:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7100000000000004e-06 2024-04-22T18:39:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.421699+0000", 2024-04-22T18:39:55.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104940144 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.421755+0000", 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5810000000000003e-05 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2517 4.c 4:3227431c:::smithi08937929-35:head [write 2955792~541902 in=541902b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.318112+0000", 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.290557179, 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29066655800000002, 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:55.615 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:39:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2517 2024-04-22T18:39:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318112+0000", 2024-04-22T18:39:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318112+0000", 2024-04-22T18:39:55.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318110+0000", 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318385+0000", 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027495500000000001 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318394+0000", 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.952e-06 2024-04-22T18:39:55.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318412+0000", 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8249000000000001e-05 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.422178+0000", 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103765307 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.422270+0000", 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2348000000000004e-05 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.422372+0000", 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101994 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.524295+0000", 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101923184 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2518 4.c 4:3227431c:::smithi08937929-35:head [setxattr _header (56) in=63b,truncate 3563084] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.318419+0000", 2024-04-22T18:39:55.619 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.29024972999999998, 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29039990799999998, 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2518 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318419+0000", 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318419+0000", 2024-04-22T18:39:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318418+0000", 2024-04-22T18:39:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318429+0000", 2024-04-22T18:39:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0560000000000001e-05 2024-04-22T18:39:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318435+0000", 2024-04-22T18:39:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2539999999999998e-06 2024-04-22T18:39:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318441+0000", 2024-04-22T18:39:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6620000000000002e-06 2024-04-22T18:39:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.523583+0000", 2024-04-22T18:39:55.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20514217100000001 2024-04-22T18:39:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.523663+0000", 2024-04-22T18:39:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9835999999999999e-05 2024-04-22T18:39:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.523786+0000", 2024-04-22T18:39:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122967 2024-04-22T18:39:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.524415+0000", 2024-04-22T18:39:55.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00062897799999999998 2024-04-22T18:39:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2519 4.c 4:3227431c:::smithi08937929-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:39:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.318445+0000", 2024-04-22T18:39:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.290223703, 2024-04-22T18:39:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.290412803, 2024-04-22T18:39:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for rw locks", 2024-04-22T18:39:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:55.624 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2519 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318445+0000", 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318445+0000", 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318445+0000", 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:55.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318448+0000", 2024-04-22T18:39:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2739999999999999e-06 2024-04-22T18:39:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.318454+0000", 2024-04-22T18:39:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4479999999999999e-06 2024-04-22T18:39:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.418611+0000", 2024-04-22T18:39:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100156566 2024-04-22T18:39:55.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.524075+0000", 2024-04-22T18:39:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105464321 2024-04-22T18:39:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.524165+0000", 2024-04-22T18:39:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0154000000000005e-05 2024-04-22T18:39:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.627 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.4502.0:2510 4.9 e65/59)", 2024-04-22T18:39:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.518207+0000", 2024-04-22T18:39:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.090462205000000004, 2024-04-22T18:39:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090683142999999994, 2024-04-22T18:39:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-22T18:39:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.518207+0000", 2024-04-22T18:39:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:55.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.518207+0000", 2024-04-22T18:39:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.518206+0000", 2024-04-22T18:39:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.518209+0000", 2024-04-22T18:39:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1710000000000001e-06 2024-04-22T18:39:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:55.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.518213+0000", 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3100000000000002e-06 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.518216+0000", 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.18e-06 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:281 3.3 3:fa6cfc8d:::smithi08937929-4 low_tier:head [call cas.chunk_put_ref in=77b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.518363+0000", 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.090305670000000005, 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090552990999999999, 2024-04-22T18:39:55.630 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.518363+0000", 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.518363+0000", 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.518361+0000", 2024-04-22T18:39:55.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.518370+0000", 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7260000000000008e-06 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.518379+0000", 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4940000000000003e-06 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.518384+0000", 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1590000000000001e-06 2024-04-22T18:39:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.548235+0000", 2024-04-22T18:39:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.029851022000000001 2024-04-22T18:39:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.548355+0000", 2024-04-22T18:39:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120525 2024-04-22T18:39:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:39:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.548583+0000", 2024-04-22T18:39:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00022801000000000001 2024-04-22T18:39:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.549779+0000", 2024-04-22T18:39:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011961020000000001 2024-04-22T18:39:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_repop_reply(client.4502.0:2517 4.c e65/20)", 2024-04-22T18:39:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.525523+0000", 2024-04-22T18:39:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.083145575999999999, 2024-04-22T18:39:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.083427157000000002, 2024-04-22T18:39:55.634 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "no flag points reached", 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.525523+0000", 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.525523+0000", 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.525522+0000", 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:55.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.525533+0000", 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1103e-05 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.525548+0000", 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5014000000000001e-05 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 8 2024-04-22T18:39:55.636 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:55.637 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:39:55.638 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:55.656 INFO:tasks.rados.rados.0.smithi089.stdout:1183: finishing write tid 1 to smithi08937929-43 2024-04-22T18:39:55.656 INFO:tasks.rados.rados.0.smithi089.stdout:1183: finishing write tid 2 to smithi08937929-43 2024-04-22T18:39:55.656 INFO:tasks.rados.rados.0.smithi089.stdout:1180: finishing write tid 4 to smithi08937929-32 2024-04-22T18:39:55.656 INFO:tasks.rados.rados.0.smithi089.stdout:1180: finishing write tid 5 to smithi08937929-32 2024-04-22T18:39:55.656 INFO:tasks.rados.rados.0.smithi089.stdout:1180: oid 32 updating version 0 to 63 2024-04-22T18:39:55.656 INFO:tasks.rados.rados.0.smithi089.stdout:1180: oid 32 version 63 is already newer than 60 2024-04-22T18:39:55.656 INFO:tasks.rados.rados.0.smithi089.stdout:1180: oid 32 version 63 is already newer than 61 2024-04-22T18:39:55.656 INFO:tasks.rados.rados.0.smithi089.stdout:1180: oid 32 version 63 is already newer than 62 2024-04-22T18:39:55.656 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 63 (ObjNum 209 snap 0 seq_num 209) dirty exists 2024-04-22T18:39:55.656 INFO:tasks.rados.rados.0.smithi089.stdout:1180: left oid 32 (ObjNum 209 snap 0 seq_num 209) 2024-04-22T18:39:55.657 INFO:tasks.rados.rados.0.smithi089.stdout:1185: expect (ObjNum 205 snap 0 seq_num 205) 2024-04-22T18:39:55.694 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:55.695+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:55.743 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:39:55.743 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:39:55.743 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-04-22T18:39:55.743 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-22T18:39:55.743 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-22T18:39:55.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2024-04-22T18:39:55.754 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:39:55.766 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:55.766 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:55.786 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:55.786 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:55.794 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:55.794 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:55.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:55.796 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:55.796+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:55.810 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:55.810 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:55.825 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:55.825 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:55.847 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:55.848 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-22T18:39:55.848 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:55.850 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:55.851 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:55.867 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:55.867 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:55.883 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:55.883 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:55.898 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:55.899+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:55.899 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:55.899 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:55.905 INFO:tasks.rados.rados.0.smithi089.stdout:1184: finishing write tid 1 to smithi08937929-4 2024-04-22T18:39:55.905 INFO:tasks.rados.rados.0.smithi089.stdout:1184: finishing write tid 2 to smithi08937929-4 2024-04-22T18:39:55.905 INFO:tasks.rados.rados.0.smithi089.stdout:1184: finishing write tid 3 to smithi08937929-4 2024-04-22T18:39:55.905 INFO:tasks.rados.rados.0.smithi089.stdout:1186: finishing write tid 1 to smithi08937929-35 2024-04-22T18:39:55.905 INFO:tasks.rados.rados.0.smithi089.stdout:1186: finishing write tid 2 to smithi08937929-35 2024-04-22T18:39:55.905 INFO:tasks.rados.rados.0.smithi089.stdout:1184: finishing write tid 4 to smithi08937929-4 2024-04-22T18:39:55.905 INFO:tasks.rados.rados.0.smithi089.stdout:1184: finishing write tid 5 to smithi08937929-4 2024-04-22T18:39:55.905 INFO:tasks.rados.rados.0.smithi089.stdout:1184: oid 4 updating version 0 to 85 2024-04-22T18:39:55.905 INFO:tasks.rados.rados.0.smithi089.stdout:1184: oid 4 version 85 is already newer than 84 2024-04-22T18:39:55.905 INFO:tasks.rados.rados.0.smithi089.stdout:1184: oid 4 updating version 85 to 87 2024-04-22T18:39:55.906 INFO:tasks.rados.rados.0.smithi089.stdout:1184: oid 4 version 87 is already newer than 86 2024-04-22T18:39:55.906 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 87 (ObjNum 211 snap 0 seq_num 211) dirty exists 2024-04-22T18:39:55.906 INFO:tasks.rados.rados.0.smithi089.stdout:1184: left oid 4 (ObjNum 211 snap 0 seq_num 211) 2024-04-22T18:39:55.906 INFO:tasks.rados.rados.0.smithi089.stdout:1186: finishing write tid 3 to smithi08937929-35 2024-04-22T18:39:55.906 INFO:tasks.rados.rados.0.smithi089.stdout:1180: done (7 left) 2024-04-22T18:39:55.906 INFO:tasks.rados.rados.0.smithi089.stdout:1181: done (6 left) 2024-04-22T18:39:55.906 INFO:tasks.rados.rados.0.smithi089.stdout:1182: done (5 left) 2024-04-22T18:39:55.906 INFO:tasks.rados.rados.0.smithi089.stdout:1184: done (4 left) 2024-04-22T18:39:55.906 INFO:tasks.rados.rados.0.smithi089.stdout:1185: done (3 left) 2024-04-22T18:39:55.906 INFO:tasks.rados.rados.0.smithi089.stdout:1187: done (2 left) 2024-04-22T18:39:55.906 INFO:tasks.rados.rados.0.smithi089.stdout:1188: read oid 24 snap -1 2024-04-22T18:39:55.906 INFO:tasks.rados.rados.0.smithi089.stdout:1188: expect (ObjNum 174 snap 0 seq_num 174) 2024-04-22T18:39:55.906 INFO:tasks.rados.rados.0.smithi089.stdout:1189: read oid 19 snap -1 2024-04-22T18:39:55.906 INFO:tasks.rados.rados.0.smithi089.stdout:1189: expect (ObjNum 195 snap 0 seq_num 195) 2024-04-22T18:39:55.906 INFO:tasks.rados.rados.0.smithi089.stdout:1190: read oid 38 snap -1 2024-04-22T18:39:55.907 INFO:tasks.rados.rados.0.smithi089.stdout:1190: expect (ObjNum 97 snap 0 seq_num 97) 2024-04-22T18:39:55.907 INFO:tasks.rados.rados.0.smithi089.stdout:1191: read oid 49 snap -1 2024-04-22T18:39:55.907 INFO:tasks.rados.rados.0.smithi089.stdout:1191: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-22T18:39:55.907 INFO:tasks.rados.rados.0.smithi089.stdout:1183: finishing write tid 3 to smithi08937929-43 2024-04-22T18:39:55.907 INFO:tasks.rados.rados.0.smithi089.stdout:1183: finishing write tid 4 to smithi08937929-43 2024-04-22T18:39:55.907 INFO:tasks.rados.rados.0.smithi089.stdout:1183: finishing write tid 5 to smithi08937929-43 2024-04-22T18:39:55.907 INFO:tasks.rados.rados.0.smithi089.stdout:1183: oid 43 updating version 0 to 36 2024-04-22T18:39:55.907 INFO:tasks.rados.rados.0.smithi089.stdout:1183: oid 43 version 36 is already newer than 35 2024-04-22T18:39:55.907 INFO:tasks.rados.rados.0.smithi089.stdout:1183: oid 43 updating version 36 to 38 2024-04-22T18:39:55.907 INFO:tasks.rados.rados.0.smithi089.stdout:1183: oid 43 version 38 is already newer than 37 2024-04-22T18:39:55.907 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 38 (ObjNum 210 snap 0 seq_num 210) dirty exists 2024-04-22T18:39:55.907 INFO:tasks.rados.rados.0.smithi089.stdout:1183: left oid 43 (ObjNum 210 snap 0 seq_num 210) 2024-04-22T18:39:55.907 INFO:tasks.rados.rados.0.smithi089.stdout:1186: finishing write tid 4 to smithi08937929-35 2024-04-22T18:39:55.907 INFO:tasks.rados.rados.0.smithi089.stdout:1186: finishing write tid 5 to smithi08937929-35 2024-04-22T18:39:55.908 INFO:tasks.rados.rados.0.smithi089.stdout:1186: oid 35 updating version 0 to 74 2024-04-22T18:39:55.908 INFO:tasks.rados.rados.0.smithi089.stdout:1186: oid 35 updating version 74 to 76 2024-04-22T18:39:55.908 INFO:tasks.rados.rados.0.smithi089.stdout:1186: oid 35 version 76 is already newer than 75 2024-04-22T18:39:55.908 INFO:tasks.rados.rados.0.smithi089.stdout:1186: oid 35 updating version 76 to 77 2024-04-22T18:39:55.908 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 77 (ObjNum 212 snap 0 seq_num 212) dirty exists 2024-04-22T18:39:55.908 INFO:tasks.rados.rados.0.smithi089.stdout:1186: left oid 35 (ObjNum 212 snap 0 seq_num 212) 2024-04-22T18:39:55.908 INFO:tasks.rados.rados.0.smithi089.stdout:1192: read oid 6 snap -1 2024-04-22T18:39:55.908 INFO:tasks.rados.rados.0.smithi089.stdout:1192: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:39:55.908 INFO:tasks.rados.rados.0.smithi089.stdout:1183: done (6 left) 2024-04-22T18:39:55.908 INFO:tasks.rados.rados.0.smithi089.stdout:1186: done (5 left) 2024-04-22T18:39:55.908 INFO:tasks.rados.rados.0.smithi089.stdout:1193: read oid 23 snap -1 2024-04-22T18:39:55.908 INFO:tasks.rados.rados.0.smithi089.stdout:1193: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:39:55.908 INFO:tasks.rados.rados.0.smithi089.stdout:1194: read oid 27 snap -1 2024-04-22T18:39:55.908 INFO:tasks.rados.rados.0.smithi089.stdout:1194: expect (ObjNum 160 snap 0 seq_num 160) 2024-04-22T18:39:55.908 INFO:tasks.rados.rados.0.smithi089.stdout:1195: read oid 35 snap -1 2024-04-22T18:39:55.909 INFO:tasks.rados.rados.0.smithi089.stdout:1195: expect (ObjNum 212 snap 0 seq_num 212) 2024-04-22T18:39:55.909 INFO:tasks.rados.rados.0.smithi089.stdout:1196: write (excl) oid 2 current snap is 0 2024-04-22T18:39:55.909 INFO:tasks.rados.rados.0.smithi089.stdout:1196: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2024-04-22T18:39:55.910 INFO:tasks.rados.rados.0.smithi089.stdout:1196: writing smithi08937929-2 from 796698 to 1357868 tid 1 2024-04-22T18:39:55.917 INFO:tasks.rados.rados.0.smithi089.stdout:1196: writing smithi08937929-2 from 2006323 to 2551729 tid 2 2024-04-22T18:39:55.923 INFO:tasks.rados.rados.0.smithi089.stdout:1196: writing smithi08937929-2 from 3126991 to 3596698 tid 3 2024-04-22T18:39:55.923 INFO:tasks.rados.rados.0.smithi089.stdout:1197: write (excl) oid 7 current snap is 0 2024-04-22T18:39:55.923 INFO:tasks.rados.rados.0.smithi089.stdout:1197: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-04-22T18:39:55.928 INFO:tasks.rados.rados.0.smithi089.stdout:1197: writing smithi08937929-7 from 413505 to 1049924 tid 1 2024-04-22T18:39:55.935 INFO:tasks.rados.rados.0.smithi089.stdout:1197: writing smithi08937929-7 from 1664805 to 2353869 tid 2 2024-04-22T18:39:55.940 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:55.940 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:55.942 INFO:tasks.rados.rados.0.smithi089.stdout:1197: writing smithi08937929-7 from 3038061 to 3530132 tid 3 2024-04-22T18:39:55.942 INFO:tasks.rados.rados.0.smithi089.stdout:1198: read oid 5 snap -1 2024-04-22T18:39:55.942 INFO:tasks.rados.rados.0.smithi089.stdout:1198: expect deleted 2024-04-22T18:39:55.942 INFO:tasks.rados.rados.0.smithi089.stdout:1199: write oid 33 current snap is 0 2024-04-22T18:39:55.943 INFO:tasks.rados.rados.0.smithi089.stdout:1199: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2024-04-22T18:39:55.948 INFO:tasks.rados.rados.0.smithi089.stdout:1199: writing smithi08937929-33 from 430312 to 1141980 tid 1 2024-04-22T18:39:55.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:55.954 INFO:tasks.rados.rados.0.smithi089.stdout:1199: writing smithi08937929-33 from 1723287 to 2272362 tid 2 2024-04-22T18:39:55.961 INFO:tasks.rados.rados.0.smithi089.stdout:1199: writing smithi08937929-33 from 2781837 to 3485180 tid 3 2024-04-22T18:39:55.961 INFO:tasks.rados.rados.0.smithi089.stdout:1200: read oid 4 snap -1 2024-04-22T18:39:55.961 INFO:tasks.rados.rados.0.smithi089.stdout:1200: expect (ObjNum 211 snap 0 seq_num 211) 2024-04-22T18:39:55.961 INFO:tasks.rados.rados.0.smithi089.stdout:1201: read oid 47 snap -1 2024-04-22T18:39:55.961 INFO:tasks.rados.rados.0.smithi089.stdout:1201: expect (ObjNum 147 snap 0 seq_num 147) 2024-04-22T18:39:55.961 INFO:tasks.rados.rados.0.smithi089.stdout:1202: read oid 46 snap -1 2024-04-22T18:39:55.961 INFO:tasks.rados.rados.0.smithi089.stdout:1202: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-22T18:39:55.961 INFO:tasks.rados.rados.0.smithi089.stdout:1203: read oid 34 snap -1 2024-04-22T18:39:55.961 INFO:tasks.rados.rados.0.smithi089.stdout:1203: expect (ObjNum 205 snap 0 seq_num 205) 2024-04-22T18:39:55.997 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:55.999+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:56.091 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:56.092 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:56.099 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:56.100+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:56.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:56.110 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:56.110 INFO:tasks.rados.rados.0.smithi089.stdout:1191: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-22T18:39:56.200 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:56.200+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:56.200 INFO:tasks.rados.rados.0.smithi089.stdout:1192: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:39:56.244 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:56.245 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:56.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:56.300 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:56.300+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:56.343 INFO:tasks.rados.rados.0.smithi089.stdout:1201: expect (ObjNum 147 snap 0 seq_num 147) 2024-04-22T18:39:56.363 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:56.363 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:56.372 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:56.400 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:56.401+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:56.474 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:56.474 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:56.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:56.488 INFO:tasks.rados.rados.0.smithi089.stdout:1196: finishing write tid 1 to smithi08937929-2 2024-04-22T18:39:56.488 INFO:tasks.rados.rados.0.smithi089.stdout:1196: finishing write tid 2 to smithi08937929-2 2024-04-22T18:39:56.488 INFO:tasks.rados.rados.0.smithi089.stdout:1188: expect (ObjNum 174 snap 0 seq_num 174) 2024-04-22T18:39:56.502 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:56.501+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:56.583 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:56.583 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:56.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:39:56.602 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:56.602+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:56.645 INFO:tasks.rados.rados.0.smithi089.stdout:1196: finishing write tid 3 to smithi08937929-2 2024-04-22T18:39:56.645 INFO:tasks.rados.rados.0.smithi089.stdout:1196: finishing write tid 4 to smithi08937929-2 2024-04-22T18:39:56.645 INFO:tasks.rados.rados.0.smithi089.stdout:1199: finishing write tid 1 to smithi08937929-33 2024-04-22T18:39:56.645 INFO:tasks.rados.rados.0.smithi089.stdout:1196: finishing write tid 5 to smithi08937929-2 2024-04-22T18:39:56.645 INFO:tasks.rados.rados.0.smithi089.stdout:1196: oid 2 updating version 0 to 52 2024-04-22T18:39:56.645 INFO:tasks.rados.rados.0.smithi089.stdout:1196: oid 2 version 52 is already newer than 51 2024-04-22T18:39:56.646 INFO:tasks.rados.rados.0.smithi089.stdout:1196: oid 2 version 52 is already newer than 50 2024-04-22T18:39:56.646 INFO:tasks.rados.rados.0.smithi089.stdout:1196: oid 2 updating version 52 to 53 2024-04-22T18:39:56.646 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 53 (ObjNum 214 snap 0 seq_num 214) dirty exists 2024-04-22T18:39:56.646 INFO:tasks.rados.rados.0.smithi089.stdout:1196: left oid 2 (ObjNum 214 snap 0 seq_num 214) 2024-04-22T18:39:56.646 INFO:tasks.rados.rados.0.smithi089.stdout:1199: finishing write tid 2 to smithi08937929-33 2024-04-22T18:39:56.646 INFO:tasks.rados.rados.0.smithi089.stdout:1188: done (15 left) 2024-04-22T18:39:56.646 INFO:tasks.rados.rados.0.smithi089.stdout:1191: done (14 left) 2024-04-22T18:39:56.646 INFO:tasks.rados.rados.0.smithi089.stdout:1192: done (13 left) 2024-04-22T18:39:56.646 INFO:tasks.rados.rados.0.smithi089.stdout:1196: done (12 left) 2024-04-22T18:39:56.646 INFO:tasks.rados.rados.0.smithi089.stdout:1201: done (11 left) 2024-04-22T18:39:56.646 INFO:tasks.rados.rados.0.smithi089.stdout:1204: read oid 42 snap -1 2024-04-22T18:39:56.646 INFO:tasks.rados.rados.0.smithi089.stdout:1204: expect (ObjNum 197 snap 0 seq_num 197) 2024-04-22T18:39:56.646 INFO:tasks.rados.rados.0.smithi089.stdout:1205: tier_promote oid 18 2024-04-22T18:39:56.646 INFO:tasks.rados.rados.0.smithi089.stdout:1199: finishing write tid 3 to smithi08937929-33 2024-04-22T18:39:56.646 INFO:tasks.rados.rados.0.smithi089.stdout:1190: expect (ObjNum 97 snap 0 seq_num 97) 2024-04-22T18:39:56.694 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:56.694 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:56.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:39:56.704 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:56.703+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:56.804 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:56.804 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:56.804 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:56.805+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:56.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:39:56.813 INFO:tasks.rados.rados.0.smithi089.stdout:1189: expect (ObjNum 195 snap 0 seq_num 195) 2024-04-22T18:39:56.904 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:56.905+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:56.916 INFO:tasks.rados.rados.0.smithi089.stdout:1199: finishing write tid 4 to smithi08937929-33 2024-04-22T18:39:56.916 INFO:tasks.rados.rados.0.smithi089.stdout:1199: finishing write tid 5 to smithi08937929-33 2024-04-22T18:39:56.916 INFO:tasks.rados.rados.0.smithi089.stdout:1199: oid 33 updating version 0 to 36 2024-04-22T18:39:56.916 INFO:tasks.rados.rados.0.smithi089.stdout:1199: oid 33 updating version 36 to 38 2024-04-22T18:39:56.916 INFO:tasks.rados.rados.0.smithi089.stdout:1199: oid 33 updating version 38 to 39 2024-04-22T18:39:56.916 INFO:tasks.rados.rados.0.smithi089.stdout:1199: oid 33 version 39 is already newer than 37 2024-04-22T18:39:56.916 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 39 (ObjNum 216 snap 0 seq_num 216) dirty exists 2024-04-22T18:39:56.916 INFO:tasks.rados.rados.0.smithi089.stdout:1199: left oid 33 (ObjNum 216 snap 0 seq_num 216) 2024-04-22T18:39:56.916 INFO:tasks.rados.rados.0.smithi089.stdout:1193: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:39:56.954 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:56.954 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:56.963 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:39:57.007 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:57.008+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:57.062 INFO:tasks.rados.rados.0.smithi089.stdout:1194: expect (ObjNum 160 snap 0 seq_num 160) 2024-04-22T18:39:57.065 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:57.065 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:57.075 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:39:57.116 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:57.117+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:57.174 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:57.175+0000 7f1e2fd26640 -1 osd.2 66 handle_osd_map injecting map crc failure 2024-04-22T18:39:57.179 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:57.179 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:57.188 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:39:57.197 INFO:tasks.rados.rados.0.smithi089.stdout:1189: done (12 left) 2024-04-22T18:39:57.197 INFO:tasks.rados.rados.0.smithi089.stdout:1190: done (11 left) 2024-04-22T18:39:57.197 INFO:tasks.rados.rados.0.smithi089.stdout:1193: done (10 left) 2024-04-22T18:39:57.197 INFO:tasks.rados.rados.0.smithi089.stdout:1194: done (9 left) 2024-04-22T18:39:57.198 INFO:tasks.rados.rados.0.smithi089.stdout:1198: done (8 left) 2024-04-22T18:39:57.198 INFO:tasks.rados.rados.0.smithi089.stdout:1199: done (7 left) 2024-04-22T18:39:57.198 INFO:tasks.rados.rados.0.smithi089.stdout:1206: read oid 37 snap -1 2024-04-22T18:39:57.198 INFO:tasks.rados.rados.0.smithi089.stdout:1206: expect (ObjNum 194 snap 0 seq_num 194) 2024-04-22T18:39:57.198 INFO:tasks.rados.rados.0.smithi089.stdout:1207: read oid 38 snap -1 2024-04-22T18:39:57.198 INFO:tasks.rados.rados.0.smithi089.stdout:1207: expect (ObjNum 97 snap 0 seq_num 97) 2024-04-22T18:39:57.198 INFO:tasks.rados.rados.0.smithi089.stdout:1208: read oid 22 snap -1 2024-04-22T18:39:57.198 INFO:tasks.rados.rados.0.smithi089.stdout:1208: expect (ObjNum 135 snap 0 seq_num 135) 2024-04-22T18:39:57.198 INFO:tasks.rados.rados.0.smithi089.stdout:1209: read oid 49 snap -1 2024-04-22T18:39:57.198 INFO:tasks.rados.rados.0.smithi089.stdout:1209: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-22T18:39:57.198 INFO:tasks.rados.rados.0.smithi089.stdout:1210: set_chunk oid 36 target oid 2024-04-22T18:39:57.207 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:57.209+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:57.293 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:57.293 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:57.297 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:57.299+0000 7f8605f32640 -1 osd.3 66 handle_osd_map injecting map crc failure 2024-04-22T18:39:57.302 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:39:57.309 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:57.310+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:57.409 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:57.410+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:57.446 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:57.446 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:57.455 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:39:57.510 INFO:tasks.rados.rados.0.smithi089.stdout:1210: set_chunk oid 36 offset: 937472 length: 1154560 target oid a03e23a0250e4c2522ea11ba5360a3fbc52ade9532ca6d9feb4c719865cc5a27 offset: 0 snap -1 2024-04-22T18:39:57.510 INFO:tasks.rados.rados.0.smithi089.stdout:1211: read oid 13 snap -1 2024-04-22T18:39:57.510 INFO:tasks.rados.rados.0.smithi089.stdout:1211: expect (ObjNum 206 snap 0 seq_num 206) 2024-04-22T18:39:57.510 INFO:tasks.rados.rados.0.smithi089.stdout:1212: read oid 31 snap -1 2024-04-22T18:39:57.510 INFO:tasks.rados.rados.0.smithi089.stdout:1212: expect deleted 2024-04-22T18:39:57.510 INFO:tasks.rados.rados.0.smithi089.stdout:1213: write (excl) oid 5 current snap is 0 2024-04-22T18:39:57.510 INFO:tasks.rados.rados.0.smithi089.stdout:1195: expect (ObjNum 212 snap 0 seq_num 212) 2024-04-22T18:39:57.511 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:57.512+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:57.559 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:57.559 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:57.568 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:39:57.653 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:57.654+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:57.698 INFO:tasks.rados.rados.0.smithi089.stdout:1200: expect (ObjNum 211 snap 0 seq_num 211) 2024-04-22T18:39:57.711 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:57.711 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:57.712 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:57.713+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:57.720 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:39:57.767 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:39:57.848 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:57.849+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:57.865 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:57.865 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:57.875 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:39:57.876 INFO:tasks.rados.rados.0.smithi089.stdout:1202: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-22T18:39:57.913 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:39:57.919 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:57.920+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:57.966 INFO:tasks.rados.rados.0.smithi089.stdout:1203: expect (ObjNum 205 snap 0 seq_num 205) 2024-04-22T18:39:57.986 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:39:57.986 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:39:57.995 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:39:58.015 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:58.016+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:58.039 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:39:58.039 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:39:58.056 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:39:58.056 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:39:58.073 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:39:58.074 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:39:58.090 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:39:58.090 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:39:58.107 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:39:58.107 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:39:58.116 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:58.118+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:58.124 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:39:58.124 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:39:58.141 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:39:58.141 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:39:58.142 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:58.142 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:58.142 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:58.142 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:58.142 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:58.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:39:58.157 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:39:58.158 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:39:58.218 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:58.219+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:39:58.224 INFO:tasks.rados.rados.0.smithi089.stdout:1197: finishing write tid 1 to smithi08937929-7 2024-04-22T18:39:58.224 INFO:tasks.rados.rados.0.smithi089.stdout:1197: finishing write tid 2 to smithi08937929-7 2024-04-22T18:39:58.224 INFO:tasks.rados.rados.0.smithi089.stdout:1197: finishing write tid 3 to smithi08937929-7 2024-04-22T18:39:58.224 INFO:tasks.rados.rados.0.smithi089.stdout:1197: finishing write tid 4 to smithi08937929-7 2024-04-22T18:39:58.224 INFO:tasks.rados.rados.0.smithi089.stdout:1197: finishing write tid 5 to smithi08937929-7 2024-04-22T18:39:58.224 INFO:tasks.rados.rados.0.smithi089.stdout:1197: oid 7 updating version 0 to 38 2024-04-22T18:39:58.224 INFO:tasks.rados.rados.0.smithi089.stdout:1197: oid 7 updating version 38 to 39 2024-04-22T18:39:58.224 INFO:tasks.rados.rados.0.smithi089.stdout:1197: oid 7 version 39 is already newer than 37 2024-04-22T18:39:58.224 INFO:tasks.rados.rados.0.smithi089.stdout:1197: oid 7 updating version 39 to 40 2024-04-22T18:39:58.224 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 40 (ObjNum 215 snap 0 seq_num 215) dirty exists 2024-04-22T18:39:58.225 INFO:tasks.rados.rados.0.smithi089.stdout:1197: left oid 7 (ObjNum 215 snap 0 seq_num 215) 2024-04-22T18:39:58.225 INFO:tasks.rados.rados.0.smithi089.stdout:1213: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2024-04-22T18:39:58.230 INFO:tasks.rados.rados.0.smithi089.stdout:1213: writing smithi08937929-5 from 447119 to 1234036 tid 1 2024-04-22T18:39:58.236 INFO:tasks.rados.rados.0.smithi089.stdout:1213: writing smithi08937929-5 from 1665416 to 2358149 tid 2 2024-04-22T18:39:58.243 INFO:tasks.rados.rados.0.smithi089.stdout:1213: writing smithi08937929-5 from 2976554 to 3491169 tid 3 2024-04-22T18:39:58.243 INFO:tasks.rados.rados.0.smithi089.stdout:1195: done (14 left) 2024-04-22T18:39:58.243 INFO:tasks.rados.rados.0.smithi089.stdout:1197: done (13 left) 2024-04-22T18:39:58.243 INFO:tasks.rados.rados.0.smithi089.stdout:1200: done (12 left) 2024-04-22T18:39:58.243 INFO:tasks.rados.rados.0.smithi089.stdout:1202: done (11 left) 2024-04-22T18:39:58.244 INFO:tasks.rados.rados.0.smithi089.stdout:1203: done (10 left) 2024-04-22T18:39:58.244 INFO:tasks.rados.rados.0.smithi089.stdout:1214: set_chunk oid 48 target oid 2024-04-22T18:39:58.244 INFO:tasks.rados.rados.0.smithi089.stdout:1214: set_chunk oid 48 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:39:58.244 INFO:tasks.rados.rados.0.smithi089.stdout:1215: read oid 41 snap -1 2024-04-22T18:39:58.244 INFO:tasks.rados.rados.0.smithi089.stdout:1215: expect (ObjNum 136 snap 0 seq_num 136) 2024-04-22T18:39:58.244 INFO:tasks.rados.rados.0.smithi089.stdout:1216: set_chunk oid 32 target oid 2024-04-22T18:39:58.262 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:58.262 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:58.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:58.262 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:58.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.263 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:27 4.d e27/20 4:bee1184f:::smithi08937929-7:head v 27'3, mlcod=0'0)", 2024-04-22T18:39:58.263 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:58.263 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 63.897446189, 2024-04-22T18:39:58.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101135974, 2024-04-22T18:39:58.263 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.263 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.263 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:58.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365187+0000", 2024-04-22T18:39:58.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365186+0000", 2024-04-22T18:39:58.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:58.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365194+0000", 2024-04-22T18:39:58.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3289999999999994e-06 2024-04-22T18:39:58.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.365200+0000", 2024-04-22T18:39:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6590000000000004e-06 2024-04-22T18:39:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465352+0000", 2024-04-22T18:39:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10015208 2024-04-22T18:39:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465511+0000", 2024-04-22T18:39:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015880000000000001 2024-04-22T18:39:58.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.465570+0000", 2024-04-22T18:39:58.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8952000000000001e-05 2024-04-22T18:39:58.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466292+0000", 2024-04-22T18:39:58.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072212100000000005 2024-04-22T18:39:58.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.466323+0000", 2024-04-22T18:39:58.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0899999999999999e-05 2024-04-22T18:39:58.266 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.267 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.267 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.267 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:47 3.0 e30/19 3:30f16451:::smithi08937929-50 low_tier:head v 30'20, mlcod=29'18)", 2024-04-22T18:39:58.267 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:58.267 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.664530171999999, 2024-04-22T18:39:58.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100803438, 2024-04-22T18:39:58.267 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.267 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.267 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:58.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:58.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598103+0000", 2024-04-22T18:39:58.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:58.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598108+0000", 2024-04-22T18:39:58.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9119999999999997e-06 2024-04-22T18:39:58.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598118+0000", 2024-04-22T18:39:58.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0138e-05 2024-04-22T18:39:58.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.598125+0000", 2024-04-22T18:39:58.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3010000000000001e-06 2024-04-22T18:39:58.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698224+0000", 2024-04-22T18:39:58.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099221 2024-04-22T18:39:58.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698239+0000", 2024-04-22T18:39:58.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5053e-05 2024-04-22T18:39:58.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698875+0000", 2024-04-22T18:39:58.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063560699999999995 2024-04-22T18:39:58.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.698906+0000", 2024-04-22T18:39:58.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1547999999999997e-05 2024-04-22T18:39:58.271 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.271 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.271 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.271 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.5.13:5 3.0 e30/19 3:1ab765eb:::smithi08937929-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=30'21)", 2024-04-22T18:39:58.271 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:58.271 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.060212049, 2024-04-22T18:39:58.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101014042, 2024-04-22T18:39:58.271 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.271 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.271 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:58.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:58.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202421+0000", 2024-04-22T18:39:58.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:58.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202428+0000", 2024-04-22T18:39:58.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5889999999999997e-06 2024-04-22T18:39:58.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202446+0000", 2024-04-22T18:39:58.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8176999999999999e-05 2024-04-22T18:39:58.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.202456+0000", 2024-04-22T18:39:58.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-04-22T18:39:58.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302599+0000", 2024-04-22T18:39:58.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100142759 2024-04-22T18:39:58.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.302628+0000", 2024-04-22T18:39:58.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9507999999999999e-05 2024-04-22T18:39:58.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303350+0000", 2024-04-22T18:39:58.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072151599999999995 2024-04-22T18:39:58.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.303435+0000", 2024-04-22T18:39:58.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5486000000000001e-05 2024-04-22T18:39:58.275 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.275 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.275 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.275 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:53 3.0 e31/19 3:25a201b0:::smithi08937929-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 31'24, mlcod=30'22)", 2024-04-22T18:39:58.275 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:58.275 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.859569714999999, 2024-04-22T18:39:58.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100751671, 2024-04-22T18:39:58.275 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.276 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.276 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:58.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403063+0000", 2024-04-22T18:39:58.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403062+0000", 2024-04-22T18:39:58.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:58.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403070+0000", 2024-04-22T18:39:58.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8499999999999994e-06 2024-04-22T18:39:58.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403079+0000", 2024-04-22T18:39:58.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4620000000000002e-06 2024-04-22T18:39:58.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.403087+0000", 2024-04-22T18:39:58.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4789999999999999e-06 2024-04-22T18:39:58.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503186+0000", 2024-04-22T18:39:58.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100099271 2024-04-22T18:39:58.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503207+0000", 2024-04-22T18:39:58.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0973999999999999e-05 2024-04-22T18:39:58.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503773+0000", 2024-04-22T18:39:58.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056557600000000003 2024-04-22T18:39:58.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.503815+0000", 2024-04-22T18:39:58.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2441999999999998e-05 2024-04-22T18:39:58.279 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.279 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.280 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.280 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:73 3.0 e34/19 3:061a7a63:::392f44ec26aa22e85be5f08273a7e45656f3bf4a4854038391a88e880ec01077:head v 34'31, mlcod=31'29)", 2024-04-22T18:39:58.280 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:58.280 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 53.804706267, 2024-04-22T18:39:58.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103503311, 2024-04-22T18:39:58.280 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.280 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.280 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:58.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:58.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:39:58.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:58.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458193+0000", 2024-04-22T18:39:58.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026628900000000001 2024-04-22T18:39:58.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458208+0000", 2024-04-22T18:39:58.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4780000000000001e-05 2024-04-22T18:39:58.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458217+0000", 2024-04-22T18:39:58.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4909999999999994e-06 2024-04-22T18:39:58.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459162+0000", 2024-04-22T18:39:58.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094440500000000005 2024-04-22T18:39:58.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459185+0000", 2024-04-22T18:39:58.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3685000000000001e-05 2024-04-22T18:39:58.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561394+0000", 2024-04-22T18:39:58.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10220874100000001 2024-04-22T18:39:58.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561430+0000", 2024-04-22T18:39:58.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6304999999999999e-05 2024-04-22T18:39:58.283 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.284 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.284 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.284 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:75 3.0 e34/19 3:18418e0f:::b7c35f165c374d5e075dde9ac2607e47ace3b0a1e2343ec6efb171380f725eea:head v 34'32, mlcod=31'29)", 2024-04-22T18:39:58.284 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:58.284 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 53.803692994000002, 2024-04-22T18:39:58.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104798078, 2024-04-22T18:39:58.284 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.284 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.284 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:58.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:39:58.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458939+0000", 2024-04-22T18:39:58.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:58.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459092+0000", 2024-04-22T18:39:58.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153838 2024-04-22T18:39:58.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459107+0000", 2024-04-22T18:39:58.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4703e-05 2024-04-22T18:39:58.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459115+0000", 2024-04-22T18:39:58.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-04-22T18:39:58.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560093+0000", 2024-04-22T18:39:58.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10097840499999999 2024-04-22T18:39:58.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560109+0000", 2024-04-22T18:39:58.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5631e-05 2024-04-22T18:39:58.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563706+0000", 2024-04-22T18:39:58.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035972059999999999 2024-04-22T18:39:58.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563738+0000", 2024-04-22T18:39:58.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2184000000000002e-05 2024-04-22T18:39:58.288 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.288 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.288 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.288 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:77 3.0 e34/19 3:28387c9f:::b6d7840f02d561da04e398fbf94d91b85e9066b426e48baf3b11dd4a4f6096c0:head v 34'33, mlcod=31'29)", 2024-04-22T18:39:58.288 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:58.288 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 53.802742914, 2024-04-22T18:39:58.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104186079, 2024-04-22T18:39:58.288 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.288 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.289 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:58.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:39:58.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459889+0000", 2024-04-22T18:39:58.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:58.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459963+0000", 2024-04-22T18:39:58.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3732999999999994e-05 2024-04-22T18:39:58.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459973+0000", 2024-04-22T18:39:58.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0285999999999999e-05 2024-04-22T18:39:58.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459987+0000", 2024-04-22T18:39:58.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3675999999999999e-05 2024-04-22T18:39:58.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560515+0000", 2024-04-22T18:39:58.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10052781299999999 2024-04-22T18:39:58.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560538+0000", 2024-04-22T18:39:58.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3861000000000001e-05 2024-04-22T18:39:58.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564044+0000", 2024-04-22T18:39:58.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035052009999999999 2024-04-22T18:39:58.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564076+0000", 2024-04-22T18:39:58.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2737000000000002e-05 2024-04-22T18:39:58.292 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.292 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.292 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.292 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:78 3.0 e34/19 3:3a5582ac:::bdd92f7deaa2d50e204891f64f9043469ba687d78b218ed518f2483ba3486a99:head v 34'34, mlcod=31'29)", 2024-04-22T18:39:58.292 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:58.293 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 53.802076796000001, 2024-04-22T18:39:58.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103547263, 2024-04-22T18:39:58.293 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.293 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.293 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:58.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:39:58.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460555+0000", 2024-04-22T18:39:58.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:58.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460861+0000", 2024-04-22T18:39:58.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000305958 2024-04-22T18:39:58.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460874+0000", 2024-04-22T18:39:58.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2639e-05 2024-04-22T18:39:58.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460881+0000", 2024-04-22T18:39:58.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2570000000000001e-06 2024-04-22T18:39:58.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560990+0000", 2024-04-22T18:39:58.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010883199999999 2024-04-22T18:39:58.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561011+0000", 2024-04-22T18:39:58.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1319e-05 2024-04-22T18:39:58.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564079+0000", 2024-04-22T18:39:58.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00306815 2024-04-22T18:39:58.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564104+0000", 2024-04-22T18:39:58.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4172000000000002e-05 2024-04-22T18:39:58.296 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.296 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.296 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.297 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:39:58.297 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:58.297 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 53.801461502000002, 2024-04-22T18:39:58.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:39:58.297 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.297 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.297 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:58.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:39:58.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:39:58.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:58.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:39:58.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:39:58.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:39:58.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:39:58.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:39:58.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:39:58.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:39:58.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:39:58.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.300 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.301 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.301 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:39:58.301 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:58.301 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 49.924787680000001, 2024-04-22T18:39:58.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:39:58.301 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.301 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.301 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:58.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:39:58.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:39:58.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:58.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:39:58.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:39:58.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:39:58.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:39:58.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:39:58.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:39:58.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:39:58.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:39:58.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:39:58.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:39:58.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:39:58.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:39:58.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:39:58.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:39:58.305 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.305 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.305 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.305 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1131 3.0 e41/19 3:01bdda42:::4a159391123755d93670b0d142ca36f3ee9a35236f6c4396c2aa92521a42a06b:head v 41'50, mlcod=41'48)", 2024-04-22T18:39:58.305 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:58.305 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.085040835000001, 2024-04-22T18:39:58.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101639456, 2024-04-22T18:39:58.305 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.305 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.305 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:58.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177592+0000", 2024-04-22T18:39:58.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177590+0000", 2024-04-22T18:39:58.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:58.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177880+0000", 2024-04-22T18:39:58.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029001 2024-04-22T18:39:58.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177893+0000", 2024-04-22T18:39:58.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-04-22T18:39:58.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.177899+0000", 2024-04-22T18:39:58.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1619999999999996e-06 2024-04-22T18:39:58.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.278054+0000", 2024-04-22T18:39:58.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100154929 2024-04-22T18:39:58.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.278076+0000", 2024-04-22T18:39:58.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1308e-05 2024-04-22T18:39:58.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.279187+0000", 2024-04-22T18:39:58.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011109169999999999 2024-04-22T18:39:58.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:13.279232+0000", 2024-04-22T18:39:58.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5188000000000002e-05 2024-04-22T18:39:58.309 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.309 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.309 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.309 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:125 3.0 e43/19 3:376bc4bb:::8f967acf6e5865f87cd7cada681f96ff59d5fc3f243a710c00c052aacb952df3:head v 43'60, mlcod=42'57)", 2024-04-22T18:39:58.309 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:58.309 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.415067452999999, 2024-04-22T18:39:58.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100905994, 2024-04-22T18:39:58.310 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.310 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.310 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:58.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847566+0000", 2024-04-22T18:39:58.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847565+0000", 2024-04-22T18:39:58.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:58.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847610+0000", 2024-04-22T18:39:58.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4984000000000002e-05 2024-04-22T18:39:58.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.847619+0000", 2024-04-22T18:39:58.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2189999999999993e-06 2024-04-22T18:39:58.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947667+0000", 2024-04-22T18:39:58.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100047939 2024-04-22T18:39:58.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947716+0000", 2024-04-22T18:39:58.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9228999999999999e-05 2024-04-22T18:39:58.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.947738+0000", 2024-04-22T18:39:58.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1963999999999998e-05 2024-04-22T18:39:58.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.948440+0000", 2024-04-22T18:39:58.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070173999999999996 2024-04-22T18:39:58.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.948472+0000", 2024-04-22T18:39:58.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1872999999999998e-05 2024-04-22T18:39:58.313 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.313 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.313 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.314 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:79 3.0 e44/19 3:05e6fa13:::smithi08937929-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 44'64, mlcod=44'61)", 2024-04-22T18:39:58.314 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:58.314 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.891786156999999, 2024-04-22T18:39:58.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10093532199999999, 2024-04-22T18:39:58.314 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.314 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.314 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:58.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370847+0000", 2024-04-22T18:39:58.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370846+0000", 2024-04-22T18:39:58.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:58.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370857+0000", 2024-04-22T18:39:58.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1155000000000001e-05 2024-04-22T18:39:58.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.370866+0000", 2024-04-22T18:39:58.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8769999999999999e-06 2024-04-22T18:39:58.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.470937+0000", 2024-04-22T18:39:58.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100071204 2024-04-22T18:39:58.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.470986+0000", 2024-04-22T18:39:58.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8556000000000001e-05 2024-04-22T18:39:58.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471014+0000", 2024-04-22T18:39:58.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7982000000000001e-05 2024-04-22T18:39:58.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471750+0000", 2024-04-22T18:39:58.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073598100000000003 2024-04-22T18:39:58.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.471782+0000", 2024-04-22T18:39:58.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339000000000003e-05 2024-04-22T18:39:58.317 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.318 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.318 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.318 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1689 3.0 e47/19 3:1d4bcadf:::e7eb8db0531d9f456e91fe87bc79e47ebd80459f1783dc2a5e53d4822cd34f49:head v 47'85, mlcod=47'83)", 2024-04-22T18:39:58.318 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:39:58.318 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.553745222, 2024-04-22T18:39:58.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102152033, 2024-04-22T18:39:58.318 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.318 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.318 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:39:58.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:39:58.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708885+0000", 2024-04-22T18:39:58.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:58.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709414+0000", 2024-04-22T18:39:58.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000528459 2024-04-22T18:39:58.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709428+0000", 2024-04-22T18:39:58.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4745e-05 2024-04-22T18:39:58.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709436+0000", 2024-04-22T18:39:58.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0830000000000004e-06 2024-04-22T18:39:58.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.809587+0000", 2024-04-22T18:39:58.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100150641 2024-04-22T18:39:58.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.809603+0000", 2024-04-22T18:39:58.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6061000000000001e-05 2024-04-22T18:39:58.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.811013+0000", 2024-04-22T18:39:58.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0014094680000000001 2024-04-22T18:39:58.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.811040+0000", 2024-04-22T18:39:58.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7474000000000002e-05 2024-04-22T18:39:58.322 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.322 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.322 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.322 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1729 3.0 e47/19 3:27b110a1:::34336650543f8f8bff225091f9c8c14678dc47baabcb4d8371890fc935dbc3e4:head v 47'89, mlcod=47'87)", 2024-04-22T18:39:58.322 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:39:58.322 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.784691452000001, 2024-04-22T18:39:58.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102111412, 2024-04-22T18:39:58.322 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.322 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.322 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:39:58.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:39:58.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477940+0000", 2024-04-22T18:39:58.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:58.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478305+0000", 2024-04-22T18:39:58.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036512200000000001 2024-04-22T18:39:58.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478320+0000", 2024-04-22T18:39:58.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5475999999999999e-05 2024-04-22T18:39:58.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478326+0000", 2024-04-22T18:39:58.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.302e-06 2024-04-22T18:39:58.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.578536+0000", 2024-04-22T18:39:58.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100209436 2024-04-22T18:39:58.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.578575+0000", 2024-04-22T18:39:58.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9223e-05 2024-04-22T18:39:58.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.580023+0000", 2024-04-22T18:39:58.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001447463 2024-04-22T18:39:58.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.580053+0000", 2024-04-22T18:39:58.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0534999999999998e-05 2024-04-22T18:39:58.326 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.326 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.326 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.326 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGRecoveryDelete(3.0 e65,63 3:255bc945:::0cf7ba283f6e98a1dfe8fb74a0dc7464561757ac03eac554ddfa7f49f708eaf6:head,47'87)", 2024-04-22T18:39:58.326 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:53.725840+0000", 2024-04-22T18:39:58.326 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.5367928849999997, 2024-04-22T18:39:58.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10024833900000001, 2024-04-22T18:39:58.326 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.327 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:58.327 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.725840+0000", 2024-04-22T18:39:58.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.725840+0000", 2024-04-22T18:39:58.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.725839+0000", 2024-04-22T18:39:58.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:58.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.725845+0000", 2024-04-22T18:39:58.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.516e-06 2024-04-22T18:39:58.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.725858+0000", 2024-04-22T18:39:58.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.277e-05 2024-04-22T18:39:58.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.725868+0000", 2024-04-22T18:39:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0028e-05 2024-04-22T18:39:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.825987+0000", 2024-04-22T18:39:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100118847 2024-04-22T18:39:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.825996+0000", 2024-04-22T18:39:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2350000000000005e-06 2024-04-22T18:39:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.826089+0000", 2024-04-22T18:39:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2292000000000003e-05 2024-04-22T18:39:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGRecoveryDelete(3.0 e65,63 3:15548ae8:::smithi08937929-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head,48'91)", 2024-04-22T18:39:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:53.931993+0000", 2024-04-22T18:39:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.3306398149999996, 2024-04-22T18:39:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100310439, 2024-04-22T18:39:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.330 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:58.331 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.931993+0000", 2024-04-22T18:39:58.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.931993+0000", 2024-04-22T18:39:58.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.931991+0000", 2024-04-22T18:39:58.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:58.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.932003+0000", 2024-04-22T18:39:58.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1620999999999999e-05 2024-04-22T18:39:58.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.932020+0000", 2024-04-22T18:39:58.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7325000000000001e-05 2024-04-22T18:39:58.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:53.932033+0000", 2024-04-22T18:39:58.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3076e-05 2024-04-22T18:39:58.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.032180+0000", 2024-04-22T18:39:58.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100146548 2024-04-22T18:39:58.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.032193+0000", 2024-04-22T18:39:58.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3006999999999999e-05 2024-04-22T18:39:58.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.032304+0000", 2024-04-22T18:39:58.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000111234 2024-04-22T18:39:58.334 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.334 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.334 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.334 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGRecoveryDelete(3.0 e65,63 3:1ea7c794:::bb3bf927b22c6786279d3d3abcc10e44cfd13a6005614d74d8894d5eccd91ad3:head,51'98)", 2024-04-22T18:39:58.334 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:54.437078+0000", 2024-04-22T18:39:58.334 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.8255555550000002, 2024-04-22T18:39:58.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100233899, 2024-04-22T18:39:58.334 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.334 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:58.334 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.437078+0000", 2024-04-22T18:39:58.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.437078+0000", 2024-04-22T18:39:58.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.437076+0000", 2024-04-22T18:39:58.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:58.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.437083+0000", 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7270000000000001e-06 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.437095+0000", 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1855000000000001e-05 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.437103+0000", 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7649999999999993e-06 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.537234+0000", 2024-04-22T18:39:58.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100131132 2024-04-22T18:39:58.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.537242+0000", 2024-04-22T18:39:58.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5720000000000003e-06 2024-04-22T18:39:58.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.537312+0000", 2024-04-22T18:39:58.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9055999999999997e-05 2024-04-22T18:39:58.337 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.337 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.338 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.338 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(3.0 65/63 [PushOp(3:0f5f80b3:::e65ec2deb1a27f55813533c3e73a682460e829b5dd6586a33a43649649fa77f6:head, version: 53'111, data_included: [0~589824,917504~221696], data_size: 811520, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:0f5f80b3:::e65ec2deb1a27f55813533c3e73a682460e829b5dd6586a33a43649649fa77f6:head@53'111, size: 1139200, copy_subset: [0~1139200], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1139200, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-22T18:39:58.338 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:54.953508+0000", 2024-04-22T18:39:58.338 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.3091250830000001, 2024-04-22T18:39:58.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101969016, 2024-04-22T18:39:58.338 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.338 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:58.338 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.953508+0000", 2024-04-22T18:39:58.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.953508+0000", 2024-04-22T18:39:58.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.953506+0000", 2024-04-22T18:39:58.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:58.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.953962+0000", 2024-04-22T18:39:58.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00045575900000000003 2024-04-22T18:39:58.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.953984+0000", 2024-04-22T18:39:58.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1579000000000001e-05 2024-04-22T18:39:58.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.054112+0000", 2024-04-22T18:39:58.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100128071 2024-04-22T18:39:58.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.054190+0000", 2024-04-22T18:39:58.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8047999999999995e-05 2024-04-22T18:39:58.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.054205+0000", 2024-04-22T18:39:58.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5273000000000001e-05 2024-04-22T18:39:58.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.055477+0000", 2024-04-22T18:39:58.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001272053 2024-04-22T18:39:58.341 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.341 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.341 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.342 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGRecoveryDelete(3.0 e65,63 3:17da094e:::3d3923760e594ff77139d30cd9223a6d60a2b336dcb3619696144cbaff02d806:head,53'113)", 2024-04-22T18:39:58.342 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:55.057311+0000", 2024-04-22T18:39:58.342 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.2053226690000001, 2024-04-22T18:39:58.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100257334, 2024-04-22T18:39:58.342 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.342 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:58.342 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.057311+0000", 2024-04-22T18:39:58.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.057311+0000", 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.057309+0000", 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.057315+0000", 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6699999999999999e-06 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.057326+0000", 2024-04-22T18:39:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0679000000000001e-05 2024-04-22T18:39:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.057332+0000", 2024-04-22T18:39:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6730000000000003e-06 2024-04-22T18:39:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.157452+0000", 2024-04-22T18:39:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10011998 2024-04-22T18:39:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.157463+0000", 2024-04-22T18:39:58.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0947999999999999e-05 2024-04-22T18:39:58.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.157568+0000", 2024-04-22T18:39:58.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104572 2024-04-22T18:39:58.345 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.345 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.345 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.345 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.345 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.345 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:58.347 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:39:58.348 INFO:tasks.rados.rados.0.smithi089.stdout:1216: set_chunk oid 32 offset: 1466880 length: 101888 target oid 13c003fc5ca06cdda7dc0073ff6ae998b2f66cdf1dd02403b6e4ff1d2854ef8e offset: 0 snap -1 2024-04-22T18:39:58.348 INFO:tasks.rados.rados.0.smithi089.stdout:1217: read oid 11 snap -1 2024-04-22T18:39:58.348 INFO:tasks.rados.rados.0.smithi089.stdout:1217: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:39:58.348 INFO:tasks.rados.rados.0.smithi089.stdout:1218: write oid 43 current snap is 0 2024-04-22T18:39:58.348 INFO:tasks.rados.rados.0.smithi089.stdout:1218: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2024-04-22T18:39:58.348 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:58.348+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:58.350 INFO:tasks.rados.rados.0.smithi089.stdout:1218: writing smithi08937929-43 from 463926 to 926092 tid 1 2024-04-22T18:39:58.357 INFO:tasks.rados.rados.0.smithi089.stdout:1218: writing smithi08937929-43 from 1723898 to 2160289 tid 2 2024-04-22T18:39:58.363 INFO:tasks.rados.rados.0.smithi089.stdout:1218: writing smithi08937929-43 from 2603977 to 3329864 tid 3 2024-04-22T18:39:58.364 INFO:tasks.rados.rados.0.smithi089.stdout:1219: read oid 39 snap -1 2024-04-22T18:39:58.364 INFO:tasks.rados.rados.0.smithi089.stdout:1219: expect (ObjNum 188 snap 0 seq_num 188) 2024-04-22T18:39:58.364 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:39:58.364 INFO:tasks.rados.rados.0.smithi089.stdout:1204: expect (ObjNum 197 snap 0 seq_num 197) 2024-04-22T18:39:58.447 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:58.448+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:58.455 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:39:58.455 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:58.465 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:39:58.548 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:39:58.550+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:39:58.608 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:58.608 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:39:58.608 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:39:58.608 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:39:58.609 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:58.617 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:39:58.643 INFO:tasks.rados.rados.0.smithi089.stdout:1205: got (0) Success 2024-04-22T18:39:58.643 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 63 (ObjNum 181 snap 0 seq_num 181) dirty exists 2024-04-22T18:39:58.643 INFO:tasks.rados.rados.0.smithi089.stdout:1206: expect (ObjNum 194 snap 0 seq_num 194) 2024-04-22T18:39:58.649 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:58.651+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:58.721 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:39:58.721 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:39:58.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:39:58.722 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:39:58.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.722 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:45 4.c e27/20 4:35eda2ca:::smithi08937929-12:head v 27'1, mlcod=0'0)", 2024-04-22T18:39:58.722 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:58.722 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 64.456581803000006, 2024-04-22T18:39:58.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14345418900000001, 2024-04-22T18:39:58.722 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.722 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.722 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265112+0000", 2024-04-22T18:39:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:39:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265727+0000", 2024-04-22T18:39:58.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061502799999999999 2024-04-22T18:39:58.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265737+0000", 2024-04-22T18:39:58.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0278999999999999e-05 2024-04-22T18:39:58.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.265744+0000", 2024-04-22T18:39:58.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1740000000000003e-06 2024-04-22T18:39:58.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407227+0000", 2024-04-22T18:39:58.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14148266900000001 2024-04-22T18:39:58.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.407258+0000", 2024-04-22T18:39:58.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1619000000000003e-05 2024-04-22T18:39:58.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408534+0000", 2024-04-22T18:39:58.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001275829 2024-04-22T18:39:58.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.408566+0000", 2024-04-22T18:39:58.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1813999999999997e-05 2024-04-22T18:39:58.726 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.726 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.726 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.726 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:78 4.0 4:01baf5a9:::smithi08937929-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:58.726 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:58.726 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 64.209758708999999, 2024-04-22T18:39:58.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.398522192, 2024-04-22T18:39:58.727 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.727 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.727 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:58.727 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:58.727 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:58.727 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 78 2024-04-22T18:39:58.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.727 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:58.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511934+0000", 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512012+0000", 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8329000000000002e-05 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512023+0000", 2024-04-22T18:39:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0755e-05 2024-04-22T18:39:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512028+0000", 2024-04-22T18:39:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6470000000000003e-06 2024-04-22T18:39:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512068+0000", 2024-04-22T18:39:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9721999999999997e-05 2024-04-22T18:39:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512125+0000", 2024-04-22T18:39:58.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6564999999999998e-05 2024-04-22T18:39:58.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:58.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512200+0000", 2024-04-22T18:39:58.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5708000000000001e-05 2024-04-22T18:39:58.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:58.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513131+0000", 2024-04-22T18:39:58.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093063 2024-04-22T18:39:58.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:58.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910394+0000", 2024-04-22T18:39:58.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.397263388 2024-04-22T18:39:58.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910430+0000", 2024-04-22T18:39:58.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5485999999999999e-05 2024-04-22T18:39:58.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910457+0000", 2024-04-22T18:39:58.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7086000000000001e-05 2024-04-22T18:39:58.731 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:79 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 64.209658052999998, 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39855419399999997, 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 79 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.732 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:58.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:39:58.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512035+0000", 2024-04-22T18:39:58.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:58.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512045+0000", 2024-04-22T18:39:58.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0379e-05 2024-04-22T18:39:58.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512051+0000", 2024-04-22T18:39:58.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6030000000000004e-06 2024-04-22T18:39:58.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512057+0000", 2024-04-22T18:39:58.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7270000000000004e-06 2024-04-22T18:39:58.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512756+0000", 2024-04-22T18:39:58.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069954000000000001 2024-04-22T18:39:58.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512806+0000", 2024-04-22T18:39:58.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9922000000000001e-05 2024-04-22T18:39:58.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:58.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512927+0000", 2024-04-22T18:39:58.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012059 2024-04-22T18:39:58.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:58.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.515883+0000", 2024-04-22T18:39:58.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029563739999999999 2024-04-22T18:39:58.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:58.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910510+0000", 2024-04-22T18:39:58.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39462745199999999 2024-04-22T18:39:58.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910542+0000", 2024-04-22T18:39:58.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1225e-05 2024-04-22T18:39:58.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910590+0000", 2024-04-22T18:39:58.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8052e-05 2024-04-22T18:39:58.737 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.737 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.737 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.737 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:80 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:58.737 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:58.737 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 64.209632952000007, 2024-04-22T18:39:58.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39866376999999997, 2024-04-22T18:39:58.738 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.738 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:58.738 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:58.738 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:58.738 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:58.738 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 80 2024-04-22T18:39:58.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.738 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:58.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:39:58.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512060+0000", 2024-04-22T18:39:58.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:58.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512063+0000", 2024-04-22T18:39:58.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.999e-06 2024-04-22T18:39:58.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512069+0000", 2024-04-22T18:39:58.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.891e-06 2024-04-22T18:39:58.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512073+0000", 2024-04-22T18:39:58.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4429999999999996e-06 2024-04-22T18:39:58.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513084+0000", 2024-04-22T18:39:58.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010102609999999999 2024-04-22T18:39:58.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:58.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513123+0000", 2024-04-22T18:39:58.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9237999999999999e-05 2024-04-22T18:39:58.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910618+0000", 2024-04-22T18:39:58.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39749549299999998 2024-04-22T18:39:58.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910670+0000", 2024-04-22T18:39:58.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1221000000000003e-05 2024-04-22T18:39:58.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910724+0000", 2024-04-22T18:39:58.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4777000000000001e-05 2024-04-22T18:39:58.742 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.742 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.742 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.742 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:5 3.2 e30/29 3:663b82a1:::smithi08937929-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 30'23, mlcod=0'0)", 2024-04-22T18:39:58.742 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:58.742 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.494193633999998, 2024-04-22T18:39:58.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20090134500000001, 2024-04-22T18:39:58.743 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.743 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.743 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:58.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227500+0000", 2024-04-22T18:39:58.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227499+0000", 2024-04-22T18:39:58.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:58.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227506+0000", 2024-04-22T18:39:58.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6540000000000002e-06 2024-04-22T18:39:58.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.227516+0000", 2024-04-22T18:39:58.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0054000000000001e-05 2024-04-22T18:39:58.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327607+0000", 2024-04-22T18:39:58.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009167099999999 2024-04-22T18:39:58.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327688+0000", 2024-04-22T18:39:58.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1000000000000004e-05 2024-04-22T18:39:58.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.327719+0000", 2024-04-22T18:39:58.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0664000000000003e-05 2024-04-22T18:39:58.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428371+0000", 2024-04-22T18:39:58.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10065239099999999 2024-04-22T18:39:58.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.428401+0000", 2024-04-22T18:39:58.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9839e-05 2024-04-22T18:39:58.746 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.746 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.746 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.746 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:419 4.0 4:01baf5a9:::smithi08937929-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:39:58.747 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:58.747 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.319471931999999, 2024-04-22T18:39:58.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30324478199999999, 2024-04-22T18:39:58.747 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.747 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.747 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:58.747 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:58.747 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:58.747 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 419 2024-04-22T18:39:58.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.747 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:58.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:39:58.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402219+0000", 2024-04-22T18:39:58.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:58.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402229+0000", 2024-04-22T18:39:58.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2550000000000003e-06 2024-04-22T18:39:58.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402242+0000", 2024-04-22T18:39:58.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2989999999999999e-05 2024-04-22T18:39:58.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402250+0000", 2024-04-22T18:39:58.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7900000000000005e-06 2024-04-22T18:39:58.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404085+0000", 2024-04-22T18:39:58.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001834397 2024-04-22T18:39:58.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:39:58.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404092+0000", 2024-04-22T18:39:58.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7889999999999997e-06 2024-04-22T18:39:58.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502534+0000", 2024-04-22T18:39:58.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098442331999999994 2024-04-22T18:39:58.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502631+0000", 2024-04-22T18:39:58.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7617999999999999e-05 2024-04-22T18:39:58.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:58.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604414+0000", 2024-04-22T18:39:58.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101782847 2024-04-22T18:39:58.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:58.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604888+0000", 2024-04-22T18:39:58.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047325999999999998 2024-04-22T18:39:58.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:58.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705376+0000", 2024-04-22T18:39:58.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100488623 2024-04-22T18:39:58.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705424+0000", 2024-04-22T18:39:58.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7964999999999999e-05 2024-04-22T18:39:58.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705466+0000", 2024-04-22T18:39:58.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2274999999999998e-05 2024-04-22T18:39:58.752 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.752 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.753 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.753 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:70 3.2 e32/29 3:5f0623ce:::smithi08937929-38 low_tier:head v 32'37, mlcod=31'35)", 2024-04-22T18:39:58.753 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:58.753 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.193254901000003, 2024-04-22T18:39:58.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20070027900000001, 2024-04-22T18:39:58.753 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.753 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.753 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:58.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528439+0000", 2024-04-22T18:39:58.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528438+0000", 2024-04-22T18:39:58.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:58.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528443+0000", 2024-04-22T18:39:58.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8640000000000004e-06 2024-04-22T18:39:58.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528453+0000", 2024-04-22T18:39:58.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3910000000000004e-06 2024-04-22T18:39:58.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.528459+0000", 2024-04-22T18:39:58.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-04-22T18:39:58.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728627+0000", 2024-04-22T18:39:58.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20016778700000001 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.728660+0000", 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3024000000000001e-05 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729107+0000", 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044719200000000001 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:02.729139+0000", 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1625000000000003e-05 2024-04-22T18:39:58.756 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.757 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.757 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.757 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1264 4.c e43/20 4:36664a98:::smithi08937929-38:head v 43'37, mlcod=42'34)", 2024-04-22T18:39:58.757 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:58.757 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.065356497000003, 2024-04-22T18:39:58.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20290602399999999, 2024-04-22T18:39:58.757 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.757 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.757 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:58.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656337+0000", 2024-04-22T18:39:58.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656335+0000", 2024-04-22T18:39:58.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:58.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656850+0000", 2024-04-22T18:39:58.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051499499999999997 2024-04-22T18:39:58.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.656861+0000", 2024-04-22T18:39:58.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1119e-05 2024-04-22T18:39:58.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756934+0000", 2024-04-22T18:39:58.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073548 2024-04-22T18:39:58.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756977+0000", 2024-04-22T18:39:58.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2218000000000001e-05 2024-04-22T18:39:58.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756993+0000", 2024-04-22T18:39:58.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6629999999999998e-05 2024-04-22T18:39:58.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859203+0000", 2024-04-22T18:39:58.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102209617 2024-04-22T18:39:58.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.859243+0000", 2024-04-22T18:39:58.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0333000000000002e-05 2024-04-22T18:39:58.762 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.762 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.762 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.763 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1747 4.c e48/20 4:35eda2ca:::smithi08937929-12:head v 48'65, mlcod=47'61)", 2024-04-22T18:39:58.763 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:32.597827+0000", 2024-04-22T18:39:58.763 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.123866355000001, 2024-04-22T18:39:58.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20085196499999999, 2024-04-22T18:39:58.763 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.763 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.763 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597827+0000", 2024-04-22T18:39:58.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597827+0000", 2024-04-22T18:39:58.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597825+0000", 2024-04-22T18:39:58.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:58.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597839+0000", 2024-04-22T18:39:58.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4329e-05 2024-04-22T18:39:58.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.597856+0000", 2024-04-22T18:39:58.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6858e-05 2024-04-22T18:39:58.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.697948+0000", 2024-04-22T18:39:58.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092367 2024-04-22T18:39:58.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798102+0000", 2024-04-22T18:39:58.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100153411 2024-04-22T18:39:58.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798122+0000", 2024-04-22T18:39:58.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0069e-05 2024-04-22T18:39:58.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798628+0000", 2024-04-22T18:39:58.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050619700000000001 2024-04-22T18:39:58.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:32.798679+0000", 2024-04-22T18:39:58.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1307000000000002e-05 2024-04-22T18:39:58.766 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:229 3.2 e51/29 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head v 51'117, mlcod=51'115)", 2024-04-22T18:39:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:39:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.910650894, 2024-04-22T18:39:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31548799399999999, 2024-04-22T18:39:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:39:58.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:39:58.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811041+0000", 2024-04-22T18:39:58.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:58.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811054+0000", 2024-04-22T18:39:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3188e-05 2024-04-22T18:39:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811068+0000", 2024-04-22T18:39:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4239999999999999e-05 2024-04-22T18:39:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.911141+0000", 2024-04-22T18:39:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100072851 2024-04-22T18:39:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025053+0000", 2024-04-22T18:39:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.11391237 2024-04-22T18:39:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025083+0000", 2024-04-22T18:39:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9703999999999999e-05 2024-04-22T18:39:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126504+0000", 2024-04-22T18:39:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10142053500000001 2024-04-22T18:39:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126531+0000", 2024-04-22T18:39:58.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7172999999999999e-05 2024-04-22T18:39:58.771 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.771 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.771 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.771 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:235 3.2 e51/29 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head v 51'118, mlcod=51'115)", 2024-04-22T18:39:58.771 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.924871+0000", 2024-04-22T18:39:58.771 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.796822692999999, 2024-04-22T18:39:58.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20167500599999999, 2024-04-22T18:39:58.771 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.771 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.771 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924871+0000", 2024-04-22T18:39:58.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924871+0000", 2024-04-22T18:39:58.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924869+0000", 2024-04-22T18:39:58.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:58.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924883+0000", 2024-04-22T18:39:58.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3725e-05 2024-04-22T18:39:58.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924899+0000", 2024-04-22T18:39:58.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6344000000000001e-05 2024-04-22T18:39:58.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.924907+0000", 2024-04-22T18:39:58.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0909999999999993e-06 2024-04-22T18:39:58.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025426+0000", 2024-04-22T18:39:58.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100519109 2024-04-22T18:39:58.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025451+0000", 2024-04-22T18:39:58.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4338e-05 2024-04-22T18:39:58.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126534+0000", 2024-04-22T18:39:58.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101083083 2024-04-22T18:39:58.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126546+0000", 2024-04-22T18:39:58.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2122e-05 2024-04-22T18:39:58.775 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.775 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.775 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.775 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:238 3.2 e51/29 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head v 51'119, mlcod=51'115)", 2024-04-22T18:39:58.775 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.925398+0000", 2024-04-22T18:39:58.775 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.796295911000001, 2024-04-22T18:39:58.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.201162223, 2024-04-22T18:39:58.776 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.776 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.776 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925398+0000", 2024-04-22T18:39:58.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925398+0000", 2024-04-22T18:39:58.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925396+0000", 2024-04-22T18:39:58.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:58.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925405+0000", 2024-04-22T18:39:58.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6600000000000001e-06 2024-04-22T18:39:58.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925417+0000", 2024-04-22T18:39:58.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2697000000000001e-05 2024-04-22T18:39:58.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.925425+0000", 2024-04-22T18:39:58.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3000000000000004e-06 2024-04-22T18:39:58.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.125722+0000", 2024-04-22T18:39:58.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20029754299999999 2024-04-22T18:39:58.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.125763+0000", 2024-04-22T18:39:58.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0408999999999999e-05 2024-04-22T18:39:58.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126548+0000", 2024-04-22T18:39:58.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00078496600000000001 2024-04-22T18:39:58.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126560+0000", 2024-04-22T18:39:58.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2227e-05 2024-04-22T18:39:58.779 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.779 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.779 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.780 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:118 3.2 e51/29 3:43d85947:::smithi08937929-18 low_tier:head v 51'120, mlcod=51'115)", 2024-04-22T18:39:58.780 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.983526+0000", 2024-04-22T18:39:58.780 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.738167923999999, 2024-04-22T18:39:58.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14328775899999999, 2024-04-22T18:39:58.780 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.780 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.780 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983526+0000", 2024-04-22T18:39:58.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983526+0000", 2024-04-22T18:39:58.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983524+0000", 2024-04-22T18:39:58.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:58.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983538+0000", 2024-04-22T18:39:58.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3858e-05 2024-04-22T18:39:58.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983552+0000", 2024-04-22T18:39:58.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4466e-05 2024-04-22T18:39:58.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983562+0000", 2024-04-22T18:39:58.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3899999999999999e-06 2024-04-22T18:39:58.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126168+0000", 2024-04-22T18:39:58.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14260677899999999 2024-04-22T18:39:58.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126185+0000", 2024-04-22T18:39:58.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6625000000000001e-05 2024-04-22T18:39:58.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126794+0000", 2024-04-22T18:39:58.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060855999999999996 2024-04-22T18:39:58.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126813+0000", 2024-04-22T18:39:58.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9851e-05 2024-04-22T18:39:58.784 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.784 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.784 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.784 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:119 3.2 e51/29 3:43d85947:::smithi08937929-18 low_tier:head v 51'121, mlcod=51'115)", 2024-04-22T18:39:58.784 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:58.784 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.737845224000001, 2024-04-22T18:39:58.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.145803025, 2024-04-22T18:39:58.784 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.784 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.784 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:58.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:58.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983848+0000", 2024-04-22T18:39:58.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:39:58.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983854+0000", 2024-04-22T18:39:58.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.207e-06 2024-04-22T18:39:58.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983860+0000", 2024-04-22T18:39:58.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5350000000000001e-06 2024-04-22T18:39:58.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.983865+0000", 2024-04-22T18:39:58.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0170000000000004e-06 2024-04-22T18:39:58.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126364+0000", 2024-04-22T18:39:58.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.142498491 2024-04-22T18:39:58.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126378+0000", 2024-04-22T18:39:58.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4134999999999999e-05 2024-04-22T18:39:58.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129628+0000", 2024-04-22T18:39:58.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032498940000000001 2024-04-22T18:39:58.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129651+0000", 2024-04-22T18:39:58.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3493999999999999e-05 2024-04-22T18:39:58.788 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.788 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.788 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.788 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:120 3.2 e51/29 3:43d85947:::smithi08937929-18 low_tier:head v 51'122, mlcod=51'115)", 2024-04-22T18:39:58.788 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.984299+0000", 2024-04-22T18:39:58.788 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.737394828999999, 2024-04-22T18:39:58.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14536710899999999, 2024-04-22T18:39:58.788 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984299+0000", 2024-04-22T18:39:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984299+0000", 2024-04-22T18:39:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984298+0000", 2024-04-22T18:39:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984302+0000", 2024-04-22T18:39:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.929e-06 2024-04-22T18:39:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984309+0000", 2024-04-22T18:39:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2080000000000003e-06 2024-04-22T18:39:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.984316+0000", 2024-04-22T18:39:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7329999999999997e-06 2024-04-22T18:39:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126582+0000", 2024-04-22T18:39:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14226603400000001 2024-04-22T18:39:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126595+0000", 2024-04-22T18:39:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3458000000000001e-05 2024-04-22T18:39:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129653+0000", 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030578860000000001 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.129666+0000", 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2483999999999999e-05 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2344 4.0 4:01baf5a9:::smithi08937929-20:head [write 578207~550227,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:39:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.087243+0000", 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.6344502550000009, 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20456982100000001, 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2344 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.087243+0000", 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.087243+0000", 2024-04-22T18:39:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.087240+0000", 2024-04-22T18:39:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.087373+0000", 2024-04-22T18:39:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013313999999999999 2024-04-22T18:39:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.087390+0000", 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7538e-05 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187547+0000", 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100156021 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187667+0000", 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000120364 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187724+0000", 2024-04-22T18:39:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6984999999999997e-05 2024-04-22T18:39:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187823+0000", 2024-04-22T18:39:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8652000000000003e-05 2024-04-22T18:39:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291749+0000", 2024-04-22T18:39:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103926662 2024-04-22T18:39:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291757+0000", 2024-04-22T18:39:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0660000000000004e-06 2024-04-22T18:39:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291789+0000", 2024-04-22T18:39:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1701e-05 2024-04-22T18:39:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291813+0000", 2024-04-22T18:39:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4199e-05 2024-04-22T18:39:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2345 4.0 4:01baf5a9:::smithi08937929-20:head [write 1762410~779734] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:39:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187577+0000", 2024-04-22T18:39:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.5341162570000009, 2024-04-22T18:39:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20497090500000001, 2024-04-22T18:39:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2345 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187577+0000", 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187577+0000", 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187575+0000", 2024-04-22T18:39:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187749+0000", 2024-04-22T18:39:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000173812 2024-04-22T18:39:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187756+0000", 2024-04-22T18:39:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6340000000000001e-06 2024-04-22T18:39:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187761+0000", 2024-04-22T18:39:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9760000000000003e-06 2024-04-22T18:39:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.289300+0000", 2024-04-22T18:39:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10153891399999999 2024-04-22T18:39:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.289359+0000", 2024-04-22T18:39:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8866999999999997e-05 2024-04-22T18:39:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.289465+0000", 2024-04-22T18:39:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010571199999999999 2024-04-22T18:39:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292263+0000", 2024-04-22T18:39:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0027986259999999998 2024-04-22T18:39:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.392486+0000", 2024-04-22T18:39:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100222441 2024-04-22T18:39:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.392523+0000", 2024-04-22T18:39:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6792999999999998e-05 2024-04-22T18:39:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.392548+0000", 2024-04-22T18:39:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.55e-05 2024-04-22T18:39:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2346 4.0 4:01baf5a9:::smithi08937929-20:head [write 3286845~91362] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:39:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187766+0000", 2024-04-22T18:39:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.5339278729999997, 2024-04-22T18:39:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20486648599999999, 2024-04-22T18:39:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2346 2024-04-22T18:39:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187766+0000", 2024-04-22T18:39:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187766+0000", 2024-04-22T18:39:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187765+0000", 2024-04-22T18:39:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187808+0000", 2024-04-22T18:39:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2911999999999998e-05 2024-04-22T18:39:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187813+0000", 2024-04-22T18:39:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.848e-06 2024-04-22T18:39:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187818+0000", 2024-04-22T18:39:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9819999999999999e-06 2024-04-22T18:39:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291118+0000", 2024-04-22T18:39:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103300054 2024-04-22T18:39:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291176+0000", 2024-04-22T18:39:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8350000000000002e-05 2024-04-22T18:39:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291267+0000", 2024-04-22T18:39:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0248999999999996e-05 2024-04-22T18:39:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292266+0000", 2024-04-22T18:39:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00099966599999999997 2024-04-22T18:39:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.392581+0000", 2024-04-22T18:39:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100314291 2024-04-22T18:39:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.392612+0000", 2024-04-22T18:39:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1044000000000003e-05 2024-04-22T18:39:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.392632+0000", 2024-04-22T18:39:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0632000000000002e-05 2024-04-22T18:39:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2347 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 3378207] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:39:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:39:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.5338726690000009, 2024-04-22T18:39:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30520418599999999, 2024-04-22T18:39:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2347 2024-04-22T18:39:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:39:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:39:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187820+0000", 2024-04-22T18:39:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187823+0000", 2024-04-22T18:39:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6299999999999998e-06 2024-04-22T18:39:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187827+0000", 2024-04-22T18:39:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3739999999999998e-06 2024-04-22T18:39:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187829+0000", 2024-04-22T18:39:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6290000000000002e-06 2024-04-22T18:39:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291848+0000", 2024-04-22T18:39:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104018812 2024-04-22T18:39:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291924+0000", 2024-04-22T18:39:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6421000000000006e-05 2024-04-22T18:39:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292050+0000", 2024-04-22T18:39:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012571500000000001 2024-04-22T18:39:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:39:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292407+0000", 2024-04-22T18:39:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035709199999999999 2024-04-22T18:39:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.492962+0000", 2024-04-22T18:39:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20055479200000001 2024-04-22T18:39:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:39:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.492993+0000", 2024-04-22T18:39:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1041000000000003e-05 2024-04-22T18:39:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493025+0000", 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2092999999999999e-05 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2348 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.5338613530000007, 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30532242999999998, 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2348 2024-04-22T18:39:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:39:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:39:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:39:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:39:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:39:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:39:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:39:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:39:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:39:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187833+0000", 2024-04-22T18:39:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.437e-06 2024-04-22T18:39:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:39:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187836+0000", 2024-04-22T18:39:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.897e-06 2024-04-22T18:39:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:39:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187839+0000", 2024-04-22T18:39:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7250000000000002e-06 2024-04-22T18:39:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292213+0000", 2024-04-22T18:39:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104373761 2024-04-22T18:39:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292256+0000", 2024-04-22T18:39:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2896999999999999e-05 2024-04-22T18:39:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493044+0000", 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20078818700000001 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493099+0000", 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5009000000000001e-05 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493155+0000", 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5807000000000003e-05 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:39:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:39:58.820 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:39:58.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:39:58.822 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:58.762+0000 7f1e2fd26640 -1 osd.2 68 handle_osd_map injecting map crc failure 2024-04-22T18:39:58.822 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:39:58.823 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:58.822+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:58.884 INFO:tasks.rados.rados.0.smithi089.stdout:1209: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-22T18:39:58.921 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:58.922+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:58.934 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:58.934 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:58.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:39:58.972 INFO:tasks.rados.rados.0.smithi089.stdout:1207: expect (ObjNum 97 snap 0 seq_num 97) 2024-04-22T18:39:59.022 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:39:59.023+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:39:59.045 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:59.045 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:59.045 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:59.045 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:59.045 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:59.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:39:59.123 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:39:59.124+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:39:59.140 INFO:tasks.rados.rados.0.smithi089.stdout:1208: expect (ObjNum 135 snap 0 seq_num 135) 2024-04-22T18:39:59.156 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:59.156 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:59.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:59.156 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:59.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.156 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:59.157 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:59.157 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.993993673000006, 2024-04-22T18:39:59.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:39:59.157 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.157 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.157 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.157 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.157 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.157 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:39:59.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.157 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:59.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:39:59.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:39:59.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:39:59.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:39:59.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:39:59.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:39:59.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:39:59.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:39:59.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:39:59.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:39:59.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:39:59.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:39:59.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:39:59.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:39:59.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:39:59.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:39:59.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:39:59.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:39:59.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:39:59.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:39:59.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:39:59.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:39:59.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:39:59.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:39:59.162 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.162 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.162 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.162 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:59.162 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:59.162 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.892434851999994, 2024-04-22T18:39:59.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:39:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:39:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:39:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:39:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:39:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:39:59.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:39:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:39:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:39:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:39:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:39:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:39:59.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:39:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:39:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:39:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:39:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:39:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:39:59.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:39:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:39:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:39:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:39:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:39:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:39:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:39:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.892408450000005, 2024-04-22T18:39:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:39:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:39:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:39:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:39:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:39:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:39:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:39:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:39:59.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:39:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:39:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:39:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:39:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:59.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:39:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:39:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:39:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:39:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:39:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:39:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:39:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:39:59.174 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.494074120999997, 2024-04-22T18:39:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:39:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:39:59.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:39:59.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:39:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:39:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:39:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:39:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:39:59.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:39:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:39:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:39:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:39:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:39:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:39:59.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:39:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:39:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:39:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.393237823000007, 2024-04-22T18:39:59.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:39:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:39:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:39:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:39:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:39:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:39:59.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:39:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:39:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:39:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:39:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:39:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:39:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:39:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:39:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:39:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:39:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:39:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:39:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:39:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:39:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:39:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:39:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:39:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:39:59.185 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.393080553000004, 2024-04-22T18:39:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:39:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.186 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:39:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:39:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:39:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:39:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:39:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:39:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:39:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:39:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:39:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:39:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:39:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:39:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:39:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:39:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:39:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:39:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:39:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:39:59.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:39:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:39:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:39:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:39:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:39:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.393065944, 2024-04-22T18:39:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:39:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:39:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:39:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:39:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:39:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:39:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:39:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:39:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:39:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:39:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:39:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:39:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:39:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:39:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:39:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:39:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:39:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:39:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:39:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:59.196 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.393055559000004, 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:39:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:39:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:39:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:39:59.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:39:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:39:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:39:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:39:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:39:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:39:59.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:39:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:39:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:39:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:39:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:39:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:39:59.201 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.392125629000006, 2024-04-22T18:39:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:39:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.202 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:39:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:39:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:39:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:39:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:39:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:39:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:39:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:39:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:39:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:39:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:39:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:39:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:39:59.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:39:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:39:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:39:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:39:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:39:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:39:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:39:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:39:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:39:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:39:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.391663503999993, 2024-04-22T18:39:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:39:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:39:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:39:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:39:59.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:39:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:39:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:39:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:39:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:39:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:39:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:39:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:39:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:39:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:39:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:39:59.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:39:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:39:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:39:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:39:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:39:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:39:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:39:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:39:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:39:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:39:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:39:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.3916507, 2024-04-22T18:39:59.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:39:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:39:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:39:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:39:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:39:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:39:59.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:39:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:39:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:39:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:39:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:39:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:39:59.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:39:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:39:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:39:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:39:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:39:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:39:59.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:39:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:39:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.063878244000001, 2024-04-22T18:39:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:39:59.218 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:39:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:39:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:39:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:39:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:39:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:39:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:39:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:39:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:39:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:39:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:39:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:39:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:39:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:39:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:39:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:39:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:39:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:39:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:39:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:39:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:39:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:39:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:39:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.223 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.055305132000001, 2024-04-22T18:39:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:39:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:39:59.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:39:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:39:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:39:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:39:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:39:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:39:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:39:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:39:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:39:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:39:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:39:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:39:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:39:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:39:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:39:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:39:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:39:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:39:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:39:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:39:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:39:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:39:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:39:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:39:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:39:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.020657616999998, 2024-04-22T18:39:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:39:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:39:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:39:59.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:39:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:39:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:39:59.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:39:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:39:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:39:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:39:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:39:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:39:59.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:39:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:39:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:39:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:39:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:39:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1684 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:39:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:39:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.551173653999999, 2024-04-22T18:39:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987885199999998, 2024-04-22T18:39:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1684 2024-04-22T18:39:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:39:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:39:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605278+0000", 2024-04-22T18:39:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605280+0000", 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1770000000000001e-06 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605285+0000", 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605289+0000", 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.687e-06 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705352+0000", 2024-04-22T18:39:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006322199999999 2024-04-22T18:39:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705423+0000", 2024-04-22T18:39:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1136999999999993e-05 2024-04-22T18:39:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105017+0000", 2024-04-22T18:39:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.399593268 2024-04-22T18:39:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105081+0000", 2024-04-22T18:39:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4703999999999996e-05 2024-04-22T18:39:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105157+0000", 2024-04-22T18:39:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6205999999999995e-05 2024-04-22T18:39:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2191 4.1 4.fdaa93c1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-04-22T18:39:59.238 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:39:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.606581404, 2024-04-22T18:39:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51064384399999996, 2024-04-22T18:39:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:39:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2191 2024-04-22T18:39:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:39:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:39:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:39:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549872+0000", 2024-04-22T18:39:59.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.387e-06 2024-04-22T18:39:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549875+0000", 2024-04-22T18:39:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:39:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549880+0000", 2024-04-22T18:39:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2529999999999998e-06 2024-04-22T18:39:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060496+0000", 2024-04-22T18:39:59.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51061597999999997 2024-04-22T18:39:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060515+0000", 2024-04-22T18:39:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9130000000000001e-05 2024-04-22T18:39:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2198 4.d 4:bc165c76:::smithi08937929-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:39:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:39:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.605313703, 2024-04-22T18:39:59.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50502289899999997, 2024-04-22T18:39:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2198 2024-04-22T18:39:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:39:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:39:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551138+0000", 2024-04-22T18:39:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551143+0000", 2024-04-22T18:39:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8949999999999997e-06 2024-04-22T18:39:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551151+0000", 2024-04-22T18:39:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8729999999999996e-06 2024-04-22T18:39:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551158+0000", 2024-04-22T18:39:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3099999999999997e-06 2024-04-22T18:39:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551180+0000", 2024-04-22T18:39:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2197000000000001e-05 2024-04-22T18:39:59.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:39:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551185+0000", 2024-04-22T18:39:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7010000000000004e-06 2024-04-22T18:39:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657133+0000", 2024-04-22T18:39:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105948498 2024-04-22T18:39:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657183+0000", 2024-04-22T18:39:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0124000000000003e-05 2024-04-22T18:39:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.055924+0000", 2024-04-22T18:39:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39874079299999998 2024-04-22T18:39:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056018+0000", 2024-04-22T18:39:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4032999999999999e-05 2024-04-22T18:39:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056161+0000", 2024-04-22T18:39:59.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143385 2024-04-22T18:39:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2212 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:39:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:39:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.605244061, 2024-04-22T18:39:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50989936400000002, 2024-04-22T18:39:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:39:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.248 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2212 2024-04-22T18:39:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:39:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:39:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:39:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551211+0000", 2024-04-22T18:39:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6280000000000001e-06 2024-04-22T18:39:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551214+0000", 2024-04-22T18:39:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1209999999999998e-06 2024-04-22T18:39:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551217+0000", 2024-04-22T18:39:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8490000000000002e-06 2024-04-22T18:39:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061086+0000", 2024-04-22T18:39:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50986975099999998 2024-04-22T18:39:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061108+0000", 2024-04-22T18:39:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1319e-05 2024-04-22T18:39:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2213 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:39:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:39:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.605233644, 2024-04-22T18:39:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50998486099999996, 2024-04-22T18:39:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:39:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2213 2024-04-22T18:39:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.252 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:39:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:39:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551218+0000", 2024-04-22T18:39:59.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551220+0000", 2024-04-22T18:39:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.319e-06 2024-04-22T18:39:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551223+0000", 2024-04-22T18:39:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9660000000000001e-06 2024-04-22T18:39:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551226+0000", 2024-04-22T18:39:59.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9050000000000001e-06 2024-04-22T18:39:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061162+0000", 2024-04-22T18:39:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.509936838 2024-04-22T18:39:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061204+0000", 2024-04-22T18:39:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1094999999999999e-05 2024-04-22T18:39:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2214 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:39:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:39:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.605224655000001, 2024-04-22T18:39:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50997458100000004, 2024-04-22T18:39:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:39:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2214 2024-04-22T18:39:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:39:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:39:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551227+0000", 2024-04-22T18:39:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551229+0000", 2024-04-22T18:39:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8649999999999999e-06 2024-04-22T18:39:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551232+0000", 2024-04-22T18:39:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:39:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551235+0000", 2024-04-22T18:39:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:39:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061194+0000", 2024-04-22T18:39:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50995819099999995 2024-04-22T18:39:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061202+0000", 2024-04-22T18:39:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.541e-06 2024-04-22T18:39:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.259 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.259 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:59.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:39:59.261 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:59.261+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:59.283 INFO:tasks.rados.rados.0.smithi089.stdout:1210: finishing set_chunk to oid 36 2024-04-22T18:39:59.283 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 51 (ObjNum 208 snap 0 seq_num 208) dirty exists 2024-04-22T18:39:59.283 INFO:tasks.rados.rados.0.smithi089.stdout:1211: expect (ObjNum 206 snap 0 seq_num 206) 2024-04-22T18:39:59.361 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:59.363+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:59.397 INFO:tasks.rados.rados.0.smithi089.stdout:1214: finishing set_chunk to oid 48 2024-04-22T18:39:59.397 INFO:tasks.rados.rados.0.smithi089.stdout:1214: got expected ENOENT (src dne) 2024-04-22T18:39:59.397 INFO:tasks.rados.rados.0.smithi089.stdout:1217: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:39:59.402 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:59.402 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:59.403 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:59.403 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:59.411 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:39:59.462 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:59.463+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:59.512 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:59.513 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:59.513 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:39:59.513 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:39:59.513 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:59.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:39:59.563 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:39:59.564+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:39:59.597 INFO:tasks.rados.rados.0.smithi089.stdout:1213: finishing write tid 1 to smithi08937929-5 2024-04-22T18:39:59.597 INFO:tasks.rados.rados.0.smithi089.stdout:1213: finishing write tid 2 to smithi08937929-5 2024-04-22T18:39:59.597 INFO:tasks.rados.rados.0.smithi089.stdout:1216: finishing set_chunk to oid 32 2024-04-22T18:39:59.597 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 63 (ObjNum 209 snap 0 seq_num 209) dirty exists 2024-04-22T18:39:59.597 INFO:tasks.rados.rados.0.smithi089.stdout:1215: expect (ObjNum 136 snap 0 seq_num 136) 2024-04-22T18:39:59.664 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:59.664 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:39:59.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:39:59.664 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:39:59.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.664 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:39:59.664 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:59.665 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.570391173000004, 2024-04-22T18:39:59.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:39:59.665 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.665 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.665 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.665 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.665 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.665 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:39:59.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.665 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:59.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:39:59.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:39:59.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:59.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:39:59.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:39:59.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:39:59.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:39:59.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:39:59.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:39:59.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:39:59.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:39:59.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:39:59.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:39:59.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:59.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:39:59.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:39:59.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:39:59.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:39:59.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:39:59.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:39:59.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:39:59.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:39:59.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:39:59.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:39:59.670 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.670 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.670 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.670 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:39:59.670 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:59.670 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.670893298000003, 2024-04-22T18:39:59.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:39:59.671 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.671 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.671 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:59.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:39:59.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:39:59.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:39:59.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:39:59.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:39:59.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:39:59.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:39:59.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:39:59.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:39:59.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:39:59.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:39:59.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:39:59.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:39:59.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:39:59.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:39:59.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:39:59.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:39:59.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:39:59.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:59.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:39:59.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:39:59.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:39:59.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:39:59.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:39:59.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:39:59.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:39:59.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:39:59.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:39:59.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:39:59.676 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.676 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.676 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.677 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:39:59.677 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:59.677 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.562593507000003, 2024-04-22T18:39:59.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:39:59.677 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.677 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.677 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.677 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.677 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.677 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:39:59.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.677 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:59.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:39:59.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:39:59.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:59.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:39:59.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:39:59.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:39:59.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:39:59.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:39:59.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:39:59.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:39:59.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:39:59.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:39:59.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:39:59.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:59.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:39:59.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:39:59.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:39:59.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:39:59.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:39:59.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:39:59.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:39:59.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:39:59.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:39:59.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:39:59.682 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.682 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.682 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.682 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:59.682 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:59.682 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.231049878999997, 2024-04-22T18:39:59.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:39:59.683 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.683 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.683 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:59.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:39:59.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:39:59.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:39:59.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:39:59.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:39:59.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:39:59.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:39:59.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:39:59.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:39:59.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:39:59.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:39:59.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:39:59.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:59.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:39:59.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:39:59.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:39:59.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:39:59.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:39:59.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:39:59.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:39:59.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:39:59.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:39:59.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:39:59.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.688 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:59.688 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:59.688 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.139964071000001, 2024-04-22T18:39:59.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:39:59.688 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.688 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.688 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:39:59.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.688 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:59.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:39:59.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:39:59.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:39:59.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:39:59.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:39:59.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:39:59.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:39:59.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:39:59.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:39:59.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:39:59.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:39:59.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:39:59.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:39:59.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:59.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:39:59.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:39:59.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:39:59.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:39:59.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:39:59.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:39:59.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:39:59.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:39:59.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:39:59.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:39:59.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.693 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.693 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:59.693 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:59.694 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.134163811000001, 2024-04-22T18:39:59.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:39:59.694 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.694 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.694 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:39:59.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.694 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:59.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:39:59.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:39:59.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:59.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:39:59.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:39:59.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:39:59.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:39:59.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:39:59.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:39:59.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:39:59.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:39:59.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:39:59.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:39:59.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:59.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:39:59.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:39:59.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:39:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:39:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:39:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:39:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:39:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:39:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:39:59.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:39:59.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.699 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.699 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:59.699 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:59.699 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.129423467999999, 2024-04-22T18:39:59.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:39:59.699 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.699 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:39:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:39:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:39:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:39:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:39:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:39:59.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:39:59.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:39:59.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:39:59.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:39:59.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:39:59.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:59.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:39:59.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:39:59.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:39:59.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:39:59.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:39:59.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:39:59.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:39:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:39:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:39:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:39:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.128950570000001, 2024-04-22T18:39:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:39:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:39:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:59.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:39:59.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:39:59.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:59.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:39:59.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:39:59.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:39:59.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:39:59.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:39:59.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:39:59.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:39:59.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:39:59.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:39:59.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:39:59.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:59.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:39:59.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:39:59.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:39:59.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:39:59.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:39:59.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:39:59.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:39:59.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:39:59.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:39:59.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:39:59.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.710 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.710 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:39:59.710 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:59.710 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.128930648999997, 2024-04-22T18:39:59.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:39:59.710 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.711 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:59.711 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.711 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.711 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.711 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:39:59.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.711 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:59.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:39:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:39:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:39:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:39:59.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:39:59.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:39:59.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:39:59.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:39:59.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:39:59.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:59.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:39:59.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:39:59.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:39:59.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:39:59.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:39:59.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:39:59.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:39:59.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:39:59.715 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.715 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.715 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.715 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:59.715 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:39:59.715 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.819606884000002, 2024-04-22T18:39:59.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:39:59.715 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.715 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.716 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:39:59.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:39:59.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:39:59.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:59.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:39:59.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:39:59.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:39:59.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:39:59.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:39:59.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:39:59.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:39:59.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:39:59.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:39:59.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:39:59.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:59.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:39:59.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:39:59.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:39:59.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:39:59.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:39:59.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:39:59.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:39:59.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:39:59.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:39:59.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:39:59.720 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.720 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.720 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.720 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:39:59.721 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:39:59.721 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.819399347999997, 2024-04-22T18:39:59.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:39:59.721 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.721 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.721 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:39:59.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:39:59.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:39:59.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:59.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:39:59.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:39:59.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:39:59.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:39:59.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:39:59.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:39:59.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:39:59.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:39:59.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:39:59.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:39:59.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:59.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:39:59.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:39:59.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:39:59.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:39:59.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:39:59.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:39:59.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:39:59.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:39:59.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:39:59.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:39:59.725 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.725 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.726 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.726 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:59.726 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:39:59.726 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.953749615, 2024-04-22T18:39:59.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:39:59.726 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.726 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.726 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:39:59.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:39:59.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:39:59.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:59.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:39:59.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:39:59.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:39:59.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:39:59.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:39:59.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:39:59.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:39:59.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:39:59.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:39:59.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:39:59.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:59.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:39:59.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:39:59.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:39:59.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:39:59.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:39:59.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:39:59.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:39:59.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:39:59.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:39:59.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:39:59.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.731 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.731 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:59.731 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:39:59.731 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.739911230000001, 2024-04-22T18:39:59.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:39:59.732 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.732 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.732 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:39:59.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:39:59.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:39:59.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:59.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:39:59.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:39:59.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:39:59.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:39:59.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:39:59.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:39:59.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:39:59.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:39:59.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:39:59.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:39:59.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:59.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:39:59.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:39:59.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:39:59.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:39:59.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:39:59.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:39:59.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:39:59.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:39:59.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:39:59.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:39:59.736 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.736 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.736 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.737 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:59.737 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:39:59.737 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.73987782, 2024-04-22T18:39:59.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:39:59.737 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.737 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.737 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:39:59.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:39:59.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:39:59.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:39:59.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:39:59.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:39:59.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:39:59.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:39:59.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:39:59.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:39:59.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:39:59.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:39:59.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:39:59.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:59.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:39:59.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:39:59.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:39:59.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:39:59.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:39:59.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:39:59.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:39:59.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:39:59.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:39:59.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:39:59.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.742 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.742 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:59.742 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:39:59.742 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.680995165999999, 2024-04-22T18:39:59.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:39:59.742 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.742 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.742 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:39:59.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:39:59.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:39:59.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:39:59.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:39:59.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:39:59.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:39:59.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:39:59.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:39:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:39:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:39:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:39:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:39:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:39:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:39:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:39:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:39:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:39:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:39:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:39:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:39:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:39:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:39:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:39:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:39:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.680856550000001, 2024-04-22T18:39:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:39:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:39:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:39:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:39:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:39:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:39:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:39:59.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:39:59.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:39:59.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:39:59.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:39:59.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:39:59.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:39:59.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:39:59.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:59.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:39:59.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:39:59.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:39:59.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:39:59.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:39:59.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:39:59.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:39:59.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:39:59.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:39:59.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:39:59.753 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.753 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.753 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.753 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:39:59.753 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:39:59.753 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.680837546999999, 2024-04-22T18:39:59.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:39:59.753 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.753 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.753 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:39:59.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:39:59.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:39:59.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:39:59.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:39:59.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:39:59.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:39:59.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:39:59.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:39:59.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:39:59.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:39:59.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:39:59.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:39:59.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:39:59.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:39:59.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:39:59.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:39:59.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:39:59.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:39:59.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:39:59.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:39:59.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:39:59.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:39:59.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:39:59.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:39:59.758 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.758 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.758 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.758 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:39:59.758 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:39:59.759 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.059825014000001, 2024-04-22T18:39:59.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:39:59.759 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.759 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.759 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.759 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.759 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.759 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:39:59.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.759 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:39:59.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:39:59.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:39:59.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:59.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:39:59.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:39:59.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:39:59.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:39:59.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:39:59.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:39:59.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:39:59.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:39:59.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:39:59.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:39:59.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:59.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:39:59.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:39:59.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:39:59.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:39:59.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:39:59.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:39:59.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:39:59.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:39:59.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:39:59.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:39:59.764 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.764 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.764 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.764 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:39:59.764 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:39:59.764 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.059305423000001, 2024-04-22T18:39:59.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:39:59.764 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.764 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:39:59.765 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.765 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.765 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.765 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:39:59.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.765 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:39:59.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:39:59.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:39:59.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:39:59.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:39:59.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:39:59.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:39:59.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:39:59.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:39:59.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:39:59.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:39:59.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:39:59.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:39:59.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:39:59.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:39:59.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:39:59.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:39:59.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:39:59.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:39:59.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:39:59.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:39:59.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:39:59.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:39:59.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:39:59.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:39:59.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:39:59.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:39:59.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:39:59.769 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.770 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.770 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.770 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:39:59.770 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:39:59.770 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.959152540000002, 2024-04-22T18:39:59.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:39:59.770 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:39:59.770 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:39:59.770 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:39:59.770 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:39:59.770 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:39:59.771 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:39:59.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.771 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:39:59.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:39:59.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:39:59.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:39:59.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:39:59.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:39:59.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:39:59.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:39:59.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:39:59.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:39:59.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:39:59.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:39:59.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:39:59.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:39:59.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:39:59.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:39:59.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:39:59.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:39:59.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:39:59.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:39:59.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:39:59.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:39:59.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:39:59.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:39:59.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:39:59.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:39:59.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:39:59.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:39:59.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:39:59.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:39:59.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:39:59.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:39:59.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:39:59.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:39:59.775 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.775 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.775 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.775 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:39:59.775 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:39:59.775 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:59.776 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:39:59.665+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:39:59.776 INFO:tasks.rados.rados.0.smithi089.stdout:1213: finishing write tid 3 to smithi08937929-5 2024-04-22T18:39:59.777 INFO:tasks.rados.rados.0.smithi089.stdout:1213: finishing write tid 4 to smithi08937929-5 2024-04-22T18:39:59.777 INFO:tasks.rados.rados.0.smithi089.stdout:1213: finishing write tid 5 to smithi08937929-5 2024-04-22T18:39:59.777 INFO:tasks.rados.rados.0.smithi089.stdout:1213: oid 5 updating version 0 to 113 2024-04-22T18:39:59.777 INFO:tasks.rados.rados.0.smithi089.stdout:1213: oid 5 version 113 is already newer than 112 2024-04-22T18:39:59.777 INFO:tasks.rados.rados.0.smithi089.stdout:1213: oid 5 version 113 is already newer than 111 2024-04-22T18:39:59.777 INFO:tasks.rados.rados.0.smithi089.stdout:1213: oid 5 updating version 113 to 114 2024-04-22T18:39:59.777 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 114 (ObjNum 217 snap 0 seq_num 217) dirty exists 2024-04-22T18:39:59.777 INFO:tasks.rados.rados.0.smithi089.stdout:1213: left oid 5 (ObjNum 217 snap 0 seq_num 217) 2024-04-22T18:39:59.777 INFO:tasks.rados.rados.0.smithi089.stdout:1219: expect (ObjNum 188 snap 0 seq_num 188) 2024-04-22T18:39:59.778 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:39:59.778 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:39:59.778+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:39:59.877 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:39:59.878+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:39:59.884 INFO:tasks.rados.rados.0.smithi089.stdout:1218: finishing write tid 1 to smithi08937929-43 2024-04-22T18:39:59.884 INFO:tasks.rados.rados.0.smithi089.stdout:1218: finishing write tid 2 to smithi08937929-43 2024-04-22T18:39:59.885 INFO:tasks.rados.rados.0.smithi089.stdout:1218: finishing write tid 3 to smithi08937929-43 2024-04-22T18:39:59.885 INFO:tasks.rados.rados.0.smithi089.stdout:1218: finishing write tid 4 to smithi08937929-43 2024-04-22T18:39:59.885 INFO:tasks.rados.rados.0.smithi089.stdout:1218: finishing write tid 5 to smithi08937929-43 2024-04-22T18:39:59.885 INFO:tasks.rados.rados.0.smithi089.stdout:1218: oid 43 updating version 0 to 39 2024-04-22T18:39:59.885 INFO:tasks.rados.rados.0.smithi089.stdout:1218: oid 43 updating version 39 to 40 2024-04-22T18:39:59.885 INFO:tasks.rados.rados.0.smithi089.stdout:1218: oid 43 updating version 40 to 41 2024-04-22T18:39:59.885 INFO:tasks.rados.rados.0.smithi089.stdout:1218: oid 43 updating version 41 to 42 2024-04-22T18:39:59.885 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 42 (ObjNum 218 snap 0 seq_num 218) dirty exists 2024-04-22T18:39:59.885 INFO:tasks.rados.rados.0.smithi089.stdout:1218: left oid 43 (ObjNum 218 snap 0 seq_num 218) 2024-04-22T18:39:59.885 INFO:tasks.rados.rados.0.smithi089.stdout:1204: done (15 left) 2024-04-22T18:39:59.885 INFO:tasks.rados.rados.0.smithi089.stdout:1205: done (14 left) 2024-04-22T18:39:59.885 INFO:tasks.rados.rados.0.smithi089.stdout:1206: done (13 left) 2024-04-22T18:39:59.885 INFO:tasks.rados.rados.0.smithi089.stdout:1207: done (12 left) 2024-04-22T18:39:59.885 INFO:tasks.rados.rados.0.smithi089.stdout:1208: done (11 left) 2024-04-22T18:39:59.886 INFO:tasks.rados.rados.0.smithi089.stdout:1209: done (10 left) 2024-04-22T18:39:59.886 INFO:tasks.rados.rados.0.smithi089.stdout:1210: done (9 left) 2024-04-22T18:39:59.886 INFO:tasks.rados.rados.0.smithi089.stdout:1211: done (8 left) 2024-04-22T18:39:59.886 INFO:tasks.rados.rados.0.smithi089.stdout:1212: done (7 left) 2024-04-22T18:39:59.886 INFO:tasks.rados.rados.0.smithi089.stdout:1213: done (6 left) 2024-04-22T18:39:59.886 INFO:tasks.rados.rados.0.smithi089.stdout:1214: done (5 left) 2024-04-22T18:39:59.886 INFO:tasks.rados.rados.0.smithi089.stdout:1215: done (4 left) 2024-04-22T18:39:59.886 INFO:tasks.rados.rados.0.smithi089.stdout:1216: done (3 left) 2024-04-22T18:39:59.886 INFO:tasks.rados.rados.0.smithi089.stdout:1217: done (2 left) 2024-04-22T18:39:59.886 INFO:tasks.rados.rados.0.smithi089.stdout:1218: done (1 left) 2024-04-22T18:39:59.886 INFO:tasks.rados.rados.0.smithi089.stdout:1219: done (0 left) 2024-04-22T18:39:59.886 INFO:tasks.rados.rados.0.smithi089.stdout:1220: tier_evict oid 24 2024-04-22T18:39:59.886 INFO:tasks.rados.rados.0.smithi089.stdout:1220: tier_evict oid 24 snap -1 2024-04-22T18:39:59.886 INFO:tasks.rados.rados.0.smithi089.stdout:1221: set_chunk oid 29 target oid 2024-04-22T18:39:59.921 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:39:59.921 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:39:59.921 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:39:59.921 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:39:59.930 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:39:59.978 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:39:59.979+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:00.031 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:00.031 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:00.031 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:00.031 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:00.031 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:00.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:00.071 INFO:tasks.rados.rados.0.smithi089.stdout:1221: set_chunk oid 29 offset: 1155584 length: 701440 target oid 761bfdb14ccc2cf6443119f578484f28153d7108c6f2dc5d06ddd39e19c9608c offset: 0 snap -1 2024-04-22T18:40:00.071 INFO:tasks.rados.rados.0.smithi089.stdout:1222: read oid 36 snap -1 2024-04-22T18:40:00.071 INFO:tasks.rados.rados.0.smithi089.stdout:1222: expect (ObjNum 208 snap 0 seq_num 208) 2024-04-22T18:40:00.071 INFO:tasks.rados.rados.0.smithi089.stdout:1223: write (excl) oid 8 current snap is 0 2024-04-22T18:40:00.071 INFO:tasks.rados.rados.0.smithi089.stdout:1220: got (22) Invalid argument 2024-04-22T18:40:00.071 INFO:tasks.rados.rados.0.smithi089.stdout:1223: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2024-04-22T18:40:00.076 INFO:tasks.rados.rados.0.smithi089.stdout:1223: writing smithi08937929-8 from 480733 to 1018148 tid 1 2024-04-22T18:40:00.079 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:00.080+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:00.084 INFO:tasks.rados.rados.0.smithi089.stdout:1223: writing smithi08937929-8 from 1782380 to 2478782 tid 2 2024-04-22T18:40:00.091 INFO:tasks.rados.rados.0.smithi089.stdout:1223: writing smithi08937929-8 from 3031400 to 3568559 tid 3 2024-04-22T18:40:00.091 INFO:tasks.rados.rados.0.smithi089.stdout:1220: done (3 left) 2024-04-22T18:40:00.091 INFO:tasks.rados.rados.0.smithi089.stdout:1224: read oid 38 snap -1 2024-04-22T18:40:00.091 INFO:tasks.rados.rados.0.smithi089.stdout:1224: expect (ObjNum 97 snap 0 seq_num 97) 2024-04-22T18:40:00.091 INFO:tasks.rados.rados.0.smithi089.stdout:1225: read oid 30 snap -1 2024-04-22T18:40:00.091 INFO:tasks.rados.rados.0.smithi089.stdout:1225: expect (ObjNum 203 snap 0 seq_num 203) 2024-04-22T18:40:00.091 INFO:tasks.rados.rados.0.smithi089.stdout:1226: read oid 24 snap -1 2024-04-22T18:40:00.091 INFO:tasks.rados.rados.0.smithi089.stdout:1226: expect (ObjNum 174 snap 0 seq_num 174) 2024-04-22T18:40:00.091 INFO:tasks.rados.rados.0.smithi089.stdout:1227: read oid 13 snap -1 2024-04-22T18:40:00.091 INFO:tasks.rados.rados.0.smithi089.stdout:1227: expect (ObjNum 206 snap 0 seq_num 206) 2024-04-22T18:40:00.091 INFO:tasks.rados.rados.0.smithi089.stdout:1221: finishing set_chunk to oid 29 2024-04-22T18:40:00.091 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 29 set_chunk returned error code -95 offset: 1155584 length: 701440 2024-04-22T18:40:00.091 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 0 (ObjNum 149 snap 0 seq_num 149) dirty exists 2024-04-22T18:40:00.092 INFO:tasks.rados.rados.0.smithi089.stdout:1228: read oid 28 snap -1 2024-04-22T18:40:00.092 INFO:tasks.rados.rados.0.smithi089.stdout:1228: expect (ObjNum 184 snap 0 seq_num 184) 2024-04-22T18:40:00.092 INFO:tasks.rados.rados.0.smithi089.stdout:1221: done (7 left) 2024-04-22T18:40:00.092 INFO:tasks.rados.rados.0.smithi089.stdout:1229: set_chunk oid 44 target oid 2024-04-22T18:40:00.169 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:40:00.180 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:00.181+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:00.190 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:00.190 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:00.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:00.190 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:00.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.190 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:22 4.a 4:5aa97f94:::smithi08937929-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:00.190 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:40:00.190 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 66.032514547999995, 2024-04-22T18:40:00.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30948398900000001, 2024-04-22T18:40:00.190 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.190 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.190 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.191 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.191 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.191 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 22 2024-04-22T18:40:00.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.191 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:40:00.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:40:00.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157259+0000", 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157318+0000", 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8906000000000001e-05 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157327+0000", 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:40:00.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157332+0000", 2024-04-22T18:40:00.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4689999999999996e-06 2024-04-22T18:40:00.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257479+0000", 2024-04-22T18:40:00.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014700999999999 2024-04-22T18:40:00.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:00.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257492+0000", 2024-04-22T18:40:00.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2574e-05 2024-04-22T18:40:00.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259582+0000", 2024-04-22T18:40:00.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020904019999999999 2024-04-22T18:40:00.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259668+0000", 2024-04-22T18:40:00.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5826000000000001e-05 2024-04-22T18:40:00.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:00.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259772+0000", 2024-04-22T18:40:00.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104048 2024-04-22T18:40:00.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363453+0000", 2024-04-22T18:40:00.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103680183 2024-04-22T18:40:00.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466692+0000", 2024-04-22T18:40:00.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10323948199999999 2024-04-22T18:40:00.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466720+0000", 2024-04-22T18:40:00.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7886e-05 2024-04-22T18:40:00.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466744+0000", 2024-04-22T18:40:00.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4049e-05 2024-04-22T18:40:00.196 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.196 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.196 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.196 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:23 4.a 4:5aa97f94:::smithi08937929-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:00.196 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:40:00.196 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 66.032436860999994, 2024-04-22T18:40:00.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39003820300000003, 2024-04-22T18:40:00.197 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.197 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.197 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 23 2024-04-22T18:40:00.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.197 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:40:00.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157337+0000", 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157340+0000", 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8609999999999998e-06 2024-04-22T18:40:00.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157343+0000", 2024-04-22T18:40:00.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9859999999999999e-06 2024-04-22T18:40:00.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157346+0000", 2024-04-22T18:40:00.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8119999999999999e-06 2024-04-22T18:40:00.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257563+0000", 2024-04-22T18:40:00.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100217497 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257574+0000", 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0777e-05 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260112+0000", 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002537681 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260191+0000", 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9320999999999995e-05 2024-04-22T18:40:00.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:00.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260333+0000", 2024-04-22T18:40:00.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014177699999999999 2024-04-22T18:40:00.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363792+0000", 2024-04-22T18:40:00.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103458862 2024-04-22T18:40:00.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547303+0000", 2024-04-22T18:40:00.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18351162100000001 2024-04-22T18:40:00.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547337+0000", 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3640999999999999e-05 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547376+0000", 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8915000000000003e-05 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:24 4.a 4:5aa97f94:::smithi08937929-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:00.203 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 66.032410429999999, 2024-04-22T18:40:00.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39017823499999998, 2024-04-22T18:40:00.204 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.204 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:00.204 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.204 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.204 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.204 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 24 2024-04-22T18:40:00.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.204 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:00.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157366+0000", 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3120000000000001e-06 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157370+0000", 2024-04-22T18:40:00.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1300000000000003e-06 2024-04-22T18:40:00.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157373+0000", 2024-04-22T18:40:00.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-04-22T18:40:00.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260553+0000", 2024-04-22T18:40:00.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103179996 2024-04-22T18:40:00.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:00.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260625+0000", 2024-04-22T18:40:00.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2236000000000004e-05 2024-04-22T18:40:00.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547414+0000", 2024-04-22T18:40:00.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.286789181 2024-04-22T18:40:00.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547471+0000", 2024-04-22T18:40:00.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6192e-05 2024-04-22T18:40:00.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547542+0000", 2024-04-22T18:40:00.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1622e-05 2024-04-22T18:40:00.208 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.208 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.208 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.208 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:49 4.a 4:5d2aee44:::smithi08937929-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:00.208 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:40:00.208 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.955611771999997, 2024-04-22T18:40:00.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31344442500000003, 2024-04-22T18:40:00.208 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.208 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.209 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.209 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.209 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.209 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 49 2024-04-22T18:40:00.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.209 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:40:00.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:40:00.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234158+0000", 2024-04-22T18:40:00.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999957 2024-04-22T18:40:00.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234901+0000", 2024-04-22T18:40:00.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074250800000000001 2024-04-22T18:40:00.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234914+0000", 2024-04-22T18:40:00.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3244999999999999e-05 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234922+0000", 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4590000000000001e-06 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260665+0000", 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025743423000000001 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260779+0000", 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114063 2024-04-22T18:40:00.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:00.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260913+0000", 2024-04-22T18:40:00.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013355499999999999 2024-04-22T18:40:00.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363798+0000", 2024-04-22T18:40:00.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102885003 2024-04-22T18:40:00.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547567+0000", 2024-04-22T18:40:00.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.183768971 2024-04-22T18:40:00.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547589+0000", 2024-04-22T18:40:00.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2507999999999999e-05 2024-04-22T18:40:00.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547607+0000", 2024-04-22T18:40:00.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8009999999999999e-05 2024-04-22T18:40:00.213 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.213 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.213 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.213 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:50 4.a 4:5d2aee44:::smithi08937929-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:00.214 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:00.214 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.950165592999994, 2024-04-22T18:40:00.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40856295399999998, 2024-04-22T18:40:00.214 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.214 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.214 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.214 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.214 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.214 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 50 2024-04-22T18:40:00.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.214 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:00.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:00.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239607+0000", 2024-04-22T18:40:00.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:00.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240309+0000", 2024-04-22T18:40:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070201599999999997 2024-04-22T18:40:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240320+0000", 2024-04-22T18:40:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1024e-05 2024-04-22T18:40:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240327+0000", 2024-04-22T18:40:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0049999999999998e-06 2024-04-22T18:40:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362065+0000", 2024-04-22T18:40:00.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.121737812 2024-04-22T18:40:00.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362121+0000", 2024-04-22T18:40:00.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5868000000000001e-05 2024-04-22T18:40:00.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:00.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362210+0000", 2024-04-22T18:40:00.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8533000000000001e-05 2024-04-22T18:40:00.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365322+0000", 2024-04-22T18:40:00.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031121220000000002 2024-04-22T18:40:00.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648054+0000", 2024-04-22T18:40:00.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.282732549 2024-04-22T18:40:00.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648105+0000", 2024-04-22T18:40:00.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1276999999999997e-05 2024-04-22T18:40:00.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648172+0000", 2024-04-22T18:40:00.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6398999999999999e-05 2024-04-22T18:40:00.219 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.219 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.219 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.219 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:51 4.a 4:5d2aee44:::smithi08937929-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:00.219 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:00.219 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.949443059000004, 2024-04-22T18:40:00.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40793057199999999, 2024-04-22T18:40:00.219 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.219 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.220 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.220 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.220 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.220 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 51 2024-04-22T18:40:00.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.220 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:00.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:00.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240336+0000", 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240339+0000", 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9009999999999998e-06 2024-04-22T18:40:00.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240342+0000", 2024-04-22T18:40:00.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:40:00.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363080+0000", 2024-04-22T18:40:00.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122738075 2024-04-22T18:40:00.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363164+0000", 2024-04-22T18:40:00.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3722999999999998e-05 2024-04-22T18:40:00.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:00.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363287+0000", 2024-04-22T18:40:00.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012301099999999999 2024-04-22T18:40:00.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466753+0000", 2024-04-22T18:40:00.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103466217 2024-04-22T18:40:00.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648205+0000", 2024-04-22T18:40:00.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.181451788 2024-04-22T18:40:00.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648231+0000", 2024-04-22T18:40:00.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6597999999999998e-05 2024-04-22T18:40:00.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648262+0000", 2024-04-22T18:40:00.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0871000000000003e-05 2024-04-22T18:40:00.224 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.224 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.224 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.225 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:52 4.a 4:5d2aee44:::smithi08937929-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:00.225 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:00.225 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.949430599999999, 2024-04-22T18:40:00.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40812731200000002, 2024-04-22T18:40:00.225 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.225 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:00.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.225 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 52 2024-04-22T18:40:00.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.225 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:00.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:00.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:00.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:00.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240345+0000", 2024-04-22T18:40:00.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4300000000000001e-06 2024-04-22T18:40:00.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240349+0000", 2024-04-22T18:40:00.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2570000000000001e-06 2024-04-22T18:40:00.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240358+0000", 2024-04-22T18:40:00.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-22T18:40:00.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363697+0000", 2024-04-22T18:40:00.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12333965299999999 2024-04-22T18:40:00.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:00.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363777+0000", 2024-04-22T18:40:00.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9883999999999996e-05 2024-04-22T18:40:00.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648296+0000", 2024-04-22T18:40:00.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451861299999998 2024-04-22T18:40:00.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648379+0000", 2024-04-22T18:40:00.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3302999999999998e-05 2024-04-22T18:40:00.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648471+0000", 2024-04-22T18:40:00.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2029999999999998e-05 2024-04-22T18:40:00.229 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.229 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.229 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.230 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:00.230 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:00.230 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.930127264999996, 2024-04-22T18:40:00.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:40:00.230 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.230 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.230 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.230 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.230 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.230 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:40:00.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.230 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:40:00.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:40:00.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:40:00.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:40:00.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:40:00.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:40:00.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:40:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:40:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:40:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:40:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:40:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:40:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:40:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:40:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:40:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:40:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:40:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:40:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:40:00.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:40:00.235 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.235 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.235 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.235 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:00.235 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:00.235 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.925344835000004, 2024-04-22T18:40:00.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:40:00.235 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.235 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.235 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:40:00.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:00.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:40:00.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:40:00.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:40:00.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:40:00.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:40:00.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:40:00.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:40:00.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:40:00.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:40:00.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:40:00.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:40:00.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:40:00.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:40:00.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:40:00.240 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.240 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.240 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.925250632000001, 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:00.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:00.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:40:00.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:00.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:40:00.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:40:00.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:40:00.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:40:00.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:40:00.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:40:00.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:40:00.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:40:00.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:40:00.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:40:00.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:00.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:40:00.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:40:00.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:40:00.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:40:00.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:40:00.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:40:00.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:40:00.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:40:00.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:40:00.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:40:00.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.246 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.246 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:00.246 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:00.246 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.925229744000006, 2024-04-22T18:40:00.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:40:00.246 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.246 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:00.246 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.246 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:40:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:40:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:40:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:40:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:40:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:40:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:40:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:40:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:40:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:40:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:40:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:40:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.142222081, 2024-04-22T18:40:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:40:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:40:00.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.252 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:00.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:00.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:40:00.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:00.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:40:00.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:40:00.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:40:00.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:40:00.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:40:00.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:40:00.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:40:00.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:40:00.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:40:00.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:40:00.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:00.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:40:00.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:40:00.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:40:00.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:40:00.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:40:00.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:40:00.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:40:00.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:40:00.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:40:00.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:40:00.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.256 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.256 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:00.256 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.134390675000006, 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:00.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:40:00.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:00.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:40:00.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:40:00.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:40:00.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:40:00.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:40:00.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:40:00.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:40:00.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:40:00.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:40:00.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:40:00.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:00.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:40:00.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:40:00.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:40:00.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:40:00.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:40:00.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:40:00.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:40:00.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:40:00.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:40:00.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:40:00.261 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.262 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.262 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2052 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:00.262 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:00.262 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.612278893999999, 2024-04-22T18:40:00.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43058958600000002, 2024-04-22T18:40:00.262 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.262 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.262 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.262 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.262 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.262 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2052 2024-04-22T18:40:00.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.263 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:00.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:00.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577494+0000", 2024-04-22T18:40:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577500+0000", 2024-04-22T18:40:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3279999999999996e-06 2024-04-22T18:40:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577510+0000", 2024-04-22T18:40:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9450000000000005e-06 2024-04-22T18:40:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577515+0000", 2024-04-22T18:40:00.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7799999999999997e-06 2024-04-22T18:40:00.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677622+0000", 2024-04-22T18:40:00.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010654099999999 2024-04-22T18:40:00.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677710+0000", 2024-04-22T18:40:00.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8472000000000003e-05 2024-04-22T18:40:00.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:00.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.906619+0000", 2024-04-22T18:40:00.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.228908519 2024-04-22T18:40:00.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907343+0000", 2024-04-22T18:40:00.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072442600000000002 2024-04-22T18:40:00.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.007972+0000", 2024-04-22T18:40:00.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100628781 2024-04-22T18:40:00.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008022+0000", 2024-04-22T18:40:00.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9843999999999999e-05 2024-04-22T18:40:00.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008085+0000", 2024-04-22T18:40:00.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3256999999999994e-05 2024-04-22T18:40:00.267 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.267 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.267 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.267 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2087 4.a 4:537c89f8:::smithi08937929-47:head [assert-version v37,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-04-22T18:40:00.267 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:40:00.268 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.282362134, 2024-04-22T18:40:00.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38904324299999998, 2024-04-22T18:40:00.268 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.268 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.268 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2087 2024-04-22T18:40:00.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.268 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:40:00.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:40:00.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907410+0000", 2024-04-22T18:40:00.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:00.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907417+0000", 2024-04-22T18:40:00.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.306e-06 2024-04-22T18:40:00.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907427+0000", 2024-04-22T18:40:00.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4930000000000004e-06 2024-04-22T18:40:00.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907432+0000", 2024-04-22T18:40:00.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5400000000000003e-06 2024-04-22T18:40:00.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907477+0000", 2024-04-22T18:40:00.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4851999999999997e-05 2024-04-22T18:40:00.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907537+0000", 2024-04-22T18:40:00.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9460999999999999e-05 2024-04-22T18:40:00.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:00.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.295587+0000", 2024-04-22T18:40:00.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38805073200000001 2024-04-22T18:40:00.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296281+0000", 2024-04-22T18:40:00.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00069370800000000002 2024-04-22T18:40:00.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296400+0000", 2024-04-22T18:40:00.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118579 2024-04-22T18:40:00.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296431+0000", 2024-04-22T18:40:00.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1627000000000001e-05 2024-04-22T18:40:00.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296456+0000", 2024-04-22T18:40:00.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4331000000000001e-05 2024-04-22T18:40:00.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.273 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.273 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:00.273 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:00.273 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.982518045, 2024-04-22T18:40:00.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:40:00.273 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.273 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.273 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:40:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:00.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:40:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:40:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:40:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:40:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:40:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:40:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:40:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:40:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:40:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:40:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:40:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:40:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:40:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:40:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:40:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:40:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:40:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:40:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:40:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:40:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:40:00.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2505 4.5 4:a6d237d2:::smithi08937929-43:head [setxattr _header (56) in=7b,truncate 3529470] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.9821253500000005, 2024-04-22T18:40:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43706744400000003, 2024-04-22T18:40:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.279 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2505 2024-04-22T18:40:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207648+0000", 2024-04-22T18:40:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207653+0000", 2024-04-22T18:40:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2989999999999998e-06 2024-04-22T18:40:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207656+0000", 2024-04-22T18:40:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0199999999999999e-06 2024-04-22T18:40:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207659+0000", 2024-04-22T18:40:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2270000000000002e-06 2024-04-22T18:40:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243213+0000", 2024-04-22T18:40:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035553653999999997 2024-04-22T18:40:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243273+0000", 2024-04-22T18:40:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0300000000000002e-05 2024-04-22T18:40:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243359+0000", 2024-04-22T18:40:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6275999999999999e-05 2024-04-22T18:40:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297443+0000", 2024-04-22T18:40:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.054083930000000002 2024-04-22T18:40:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644651+0000", 2024-04-22T18:40:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.347208181 2024-04-22T18:40:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644684+0000", 2024-04-22T18:40:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3080999999999997e-05 2024-04-22T18:40:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644717+0000", 2024-04-22T18:40:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2264000000000001e-05 2024-04-22T18:40:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:00.284 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.9821131369999998, 2024-04-22T18:40:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:40:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:40:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:40:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:40:00.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:40:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:40:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:40:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:40:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:40:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:40:00.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:00.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:40:00.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:40:00.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:40:00.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:40:00.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:40:00.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:40:00.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:40:00.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:40:00.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.289 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.289 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2615 4.5 4:a6d237d2:::smithi08937929-43:head [write 463926~462166] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-22T18:40:00.289 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:58.352901+0000", 2024-04-22T18:40:00.289 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.8368736889999999, 2024-04-22T18:40:00.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.307052514, 2024-04-22T18:40:00.289 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.290 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.290 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.290 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.290 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.290 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2615 2024-04-22T18:40:00.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.290 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.352901+0000", 2024-04-22T18:40:00.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.352901+0000", 2024-04-22T18:40:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.352897+0000", 2024-04-22T18:40:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999962 2024-04-22T18:40:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.353421+0000", 2024-04-22T18:40:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052391999999999998 2024-04-22T18:40:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.353433+0000", 2024-04-22T18:40:00.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2128e-05 2024-04-22T18:40:00.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.353460+0000", 2024-04-22T18:40:00.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7211000000000001e-05 2024-04-22T18:40:00.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.453670+0000", 2024-04-22T18:40:00.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10020929100000001 2024-04-22T18:40:00.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.453809+0000", 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013967800000000001 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.454021+0000", 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021190900000000001 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659805+0000", 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20578357999999999 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659886+0000", 2024-04-22T18:40:00.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1310000000000006e-05 2024-04-22T18:40:00.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659925+0000", 2024-04-22T18:40:00.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9233000000000002e-05 2024-04-22T18:40:00.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659953+0000", 2024-04-22T18:40:00.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7894000000000001e-05 2024-04-22T18:40:00.294 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.294 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.295 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.295 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2616 4.5 4:a6d237d2:::smithi08937929-43:head [write 1723898~436391] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-22T18:40:00.295 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:58.359646+0000", 2024-04-22T18:40:00.295 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.8301287930000001, 2024-04-22T18:40:00.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.300417604, 2024-04-22T18:40:00.295 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:00.295 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:00.295 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:00.295 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:00.295 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:00.295 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2616 2024-04-22T18:40:00.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.295 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:00.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:00.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.359646+0000", 2024-04-22T18:40:00.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:00.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.359646+0000", 2024-04-22T18:40:00.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:00.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:00.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.359642+0000", 2024-04-22T18:40:00.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:00.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:00.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.359916+0000", 2024-04-22T18:40:00.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027331399999999998 2024-04-22T18:40:00.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:00.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.359928+0000", 2024-04-22T18:40:00.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2063e-05 2024-04-22T18:40:00.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:00.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.359952+0000", 2024-04-22T18:40:00.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4002000000000002e-05 2024-04-22T18:40:00.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:00.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.455280+0000", 2024-04-22T18:40:00.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.095328063000000005 2024-04-22T18:40:00.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:00.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.455339+0000", 2024-04-22T18:40:00.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9534999999999998e-05 2024-04-22T18:40:00.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:00.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.455419+0000", 2024-04-22T18:40:00.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9923e-05 2024-04-22T18:40:00.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:00.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659831+0000", 2024-04-22T18:40:00.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.204412027 2024-04-22T18:40:00.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:00.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.660001+0000", 2024-04-22T18:40:00.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016913299999999999 2024-04-22T18:40:00.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:00.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.660042+0000", 2024-04-22T18:40:00.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1402000000000001e-05 2024-04-22T18:40:00.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:00.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:00.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:00.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.660063+0000", 2024-04-22T18:40:00.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1367999999999999e-05 2024-04-22T18:40:00.300 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.300 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.300 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.300 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:00.300 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:00.300 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:00.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:00.303 INFO:tasks.rados.rados.0.smithi089.stdout:1229: set_chunk oid 44 offset: 551424 length: 842240 target oid 5092bc1805791a6a55008f6e6eac653218aa21b11ecb8e2b33c0b00580d66f12 offset: 0 snap -1 2024-04-22T18:40:00.303 INFO:tasks.rados.rados.0.smithi089.stdout:1230: write oid 5 current snap is 0 2024-04-22T18:40:00.303 INFO:tasks.rados.rados.0.smithi089.stdout:1230: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-04-22T18:40:00.303 INFO:tasks.rados.rados.0.smithi089.stdout:1230: writing smithi08937929-5 from 497540 to 1110204 tid 1 2024-04-22T18:40:00.304 INFO:tasks.rados.rados.0.smithi089.stdout:1230: writing smithi08937929-5 from 1840862 to 2280922 tid 2 2024-04-22T18:40:00.304 INFO:tasks.rados.rados.0.smithi089.stdout:1230: writing smithi08937929-5 from 3058823 to 3523607 tid 3 2024-04-22T18:40:00.304 INFO:tasks.rados.rados.0.smithi089.stdout:1231: write oid 41 current snap is 0 2024-04-22T18:40:00.304 INFO:tasks.rados.rados.0.smithi089.stdout:1231: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-04-22T18:40:00.304 INFO:tasks.rados.rados.0.smithi089.stdout:1231: writing smithi08937929-41 from 514347 to 918613 tid 1 2024-04-22T18:40:00.304 INFO:tasks.rados.rados.0.smithi089.stdout:1231: writing smithi08937929-41 from 1499344 to 2199415 tid 2 2024-04-22T18:40:00.304 INFO:tasks.rados.rados.0.smithi089.stdout:1231: writing smithi08937929-41 from 2686246 to 3362302 tid 3 2024-04-22T18:40:00.304 INFO:tasks.rados.rados.0.smithi089.stdout:1232: write (excl) oid 43 current snap is 0 2024-04-22T18:40:00.304 INFO:tasks.rados.rados.0.smithi089.stdout:1222: expect (ObjNum 208 snap 0 seq_num 208) 2024-04-22T18:40:00.304 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:00.303+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:00.318 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:00.319 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:00.338 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:00.339 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:00.354 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:00.354 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:00.378 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:00.378 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:00.394 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:00.394 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:00.403 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:00.404+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:00.410 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:00.411 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:00.427 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:00.427 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:00.443 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:00.443 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:00.446 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:00.446 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:00.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:00.467 INFO:tasks.rados.rados.0.smithi089.stdout:1227: expect (ObjNum 206 snap 0 seq_num 206) 2024-04-22T18:40:00.539 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:00.540+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:00.580 INFO:tasks.rados.rados.0.smithi089.stdout:1225: expect (ObjNum 203 snap 0 seq_num 203) 2024-04-22T18:40:00.595 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:00.596 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:00.605 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:00.605+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:00.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:00.704 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:00.706+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:00.746 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:00.746 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:00.755 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:00.772 INFO:tasks.rados.rados.0.smithi089.stdout:1223: finishing write tid 1 to smithi08937929-8 2024-04-22T18:40:00.772 INFO:tasks.rados.rados.0.smithi089.stdout:1226: expect (ObjNum 174 snap 0 seq_num 174) 2024-04-22T18:40:00.773 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:00.773+0000 7f3e7be88640 -1 osd.0 70 handle_osd_map injecting map crc failure 2024-04-22T18:40:00.805 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:00.807+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:00.824 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:40:00.858 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:40:00.858 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:40:00.858 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 3, 4, 5, 6, 7, 2] 2024-04-22T18:40:00.858 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-04-22T18:40:00.863 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:00.863 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:00.872 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:00.876 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:00.877+0000 7fec55bea640 -1 osd.7 70 handle_osd_map injecting map crc failure 2024-04-22T18:40:00.913 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:00.915+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:00.937 INFO:tasks.rados.rados.0.smithi089.stdout:1224: expect (ObjNum 97 snap 0 seq_num 97) 2024-04-22T18:40:01.007 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:01.009+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:01.043 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:01.043 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:01.056 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:01.108 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:01.110+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:01.117 INFO:tasks.rados.rados.0.smithi089.stdout:1228: expect (ObjNum 184 snap 0 seq_num 184) 2024-04-22T18:40:01.199 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:01.199 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:01.207 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:01.212 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-22T18:40:01.213+0000 7f9b4bbc8640 -1 mon.a@0(leader).osd e71 definitely_dead 0 2024-04-22T18:40:01.212 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:01.213+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:01.296 INFO:tasks.rados.rados.0.smithi089.stdout:1230: finishing write tid 1 to smithi08937929-5 2024-04-22T18:40:01.296 INFO:tasks.rados.rados.0.smithi089.stdout:1230: finishing write tid 2 to smithi08937929-5 2024-04-22T18:40:01.297 INFO:tasks.rados.rados.0.smithi089.stdout:1223: finishing write tid 2 to smithi08937929-8 2024-04-22T18:40:01.297 INFO:tasks.rados.rados.0.smithi089.stdout:1223: finishing write tid 3 to smithi08937929-8 2024-04-22T18:40:01.297 INFO:tasks.rados.rados.0.smithi089.stdout:1223: finishing write tid 4 to smithi08937929-8 2024-04-22T18:40:01.297 INFO:tasks.rados.rados.0.smithi089.stdout:1223: finishing write tid 5 to smithi08937929-8 2024-04-22T18:40:01.297 INFO:tasks.rados.rados.0.smithi089.stdout:1223: oid 8 updating version 0 to 14 2024-04-22T18:40:01.297 INFO:tasks.rados.rados.0.smithi089.stdout:1223: oid 8 version 14 is already newer than 13 2024-04-22T18:40:01.297 INFO:tasks.rados.rados.0.smithi089.stdout:1223: oid 8 updating version 14 to 16 2024-04-22T18:40:01.297 INFO:tasks.rados.rados.0.smithi089.stdout:1223: oid 8 version 16 is already newer than 15 2024-04-22T18:40:01.297 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 16 (ObjNum 219 snap 0 seq_num 219) dirty exists 2024-04-22T18:40:01.297 INFO:tasks.rados.rados.0.smithi089.stdout:1223: left oid 8 (ObjNum 219 snap 0 seq_num 219) 2024-04-22T18:40:01.297 INFO:tasks.rados.rados.0.smithi089.stdout:1229: finishing set_chunk to oid 44 2024-04-22T18:40:01.297 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 85 (ObjNum 187 snap 0 seq_num 187) dirty exists 2024-04-22T18:40:01.297 INFO:tasks.rados.rados.0.smithi089.stdout:1230: finishing write tid 3 to smithi08937929-5 2024-04-22T18:40:01.297 INFO:tasks.rados.rados.0.smithi089.stdout:1232: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2024-04-22T18:40:01.302 INFO:tasks.rados.rados.0.smithi089.stdout:1232: writing smithi08937929-43 from 531154 to 1010669 tid 1 2024-04-22T18:40:01.307 INFO:tasks.rados.rados.0.smithi089.stdout:1232: writing smithi08937929-43 from 1557826 to 2001555 tid 2 2024-04-22T18:40:01.309 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:01.310+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:01.313 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:01.313 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:01.314 INFO:tasks.rados.rados.0.smithi089.stdout:1232: writing smithi08937929-43 from 2713669 to 3200997 tid 3 2024-04-22T18:40:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:1232: writing smithi08937929-43 from 3679295 to 3731154 tid 4 2024-04-22T18:40:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:1222: done (10 left) 2024-04-22T18:40:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:1223: done (9 left) 2024-04-22T18:40:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:1224: done (8 left) 2024-04-22T18:40:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:1225: done (7 left) 2024-04-22T18:40:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:1226: done (6 left) 2024-04-22T18:40:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:1227: done (5 left) 2024-04-22T18:40:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:1228: done (4 left) 2024-04-22T18:40:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:1229: done (3 left) 2024-04-22T18:40:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:1233: tier_promote oid 20 2024-04-22T18:40:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:1230: finishing write tid 4 to smithi08937929-5 2024-04-22T18:40:01.319 INFO:tasks.rados.rados.0.smithi089.stdout:1230: finishing write tid 5 to smithi08937929-5 2024-04-22T18:40:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:1230: oid 5 updating version 0 to 118 2024-04-22T18:40:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:1230: oid 5 version 118 is already newer than 116 2024-04-22T18:40:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:1230: oid 5 version 118 is already newer than 117 2024-04-22T18:40:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:1230: oid 5 version 118 is already newer than 115 2024-04-22T18:40:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 118 (ObjNum 220 snap 0 seq_num 220) dirty exists 2024-04-22T18:40:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:1230: left oid 5 (ObjNum 220 snap 0 seq_num 220) 2024-04-22T18:40:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:1232: finishing write tid 1 to smithi08937929-43 2024-04-22T18:40:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:1230: done (3 left) 2024-04-22T18:40:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:1234: read oid 13 snap -1 2024-04-22T18:40:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:1234: expect (ObjNum 206 snap 0 seq_num 206) 2024-04-22T18:40:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:1235: tier_flush oid 36 2024-04-22T18:40:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:1235: tier_flush oid 36 snap -1 2024-04-22T18:40:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:1236: read oid 3 snap -1 2024-04-22T18:40:01.320 INFO:tasks.rados.rados.0.smithi089.stdout:1236: expect (ObjNum 200 snap 0 seq_num 200) 2024-04-22T18:40:01.321 INFO:tasks.rados.rados.0.smithi089.stdout:1237: write (excl) oid 12 current snap is 0 2024-04-22T18:40:01.321 INFO:tasks.rados.rados.0.smithi089.stdout:1237: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2024-04-22T18:40:01.323 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:01.324 INFO:tasks.rados.rados.0.smithi089.stdout:1237: writing smithi08937929-12 from 547961 to 1102725 tid 1 2024-04-22T18:40:01.330 INFO:tasks.rados.rados.0.smithi089.stdout:1237: writing smithi08937929-12 from 1616308 to 2320048 tid 2 2024-04-22T18:40:01.336 INFO:tasks.rados.rados.0.smithi089.stdout:1237: writing smithi08937929-12 from 2741092 to 3439692 tid 3 2024-04-22T18:40:01.336 INFO:tasks.rados.rados.0.smithi089.stdout:1238: tier_evict oid 10 2024-04-22T18:40:01.336 INFO:tasks.rados.rados.0.smithi089.stdout:1238: tier_evict oid 10 snap -1 2024-04-22T18:40:01.336 INFO:tasks.rados.rados.0.smithi089.stdout:1236: expect (ObjNum 200 snap 0 seq_num 200) 2024-04-22T18:40:01.410 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:01.411+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:01.425 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:01.425 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:01.435 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:01.501 INFO:tasks.rados.rados.0.smithi089.stdout:1238: got (22) Invalid argument 2024-04-22T18:40:01.501 INFO:tasks.rados.rados.0.smithi089.stdout:1232: finishing write tid 2 to smithi08937929-43 2024-04-22T18:40:01.501 INFO:tasks.rados.rados.0.smithi089.stdout:1236: done (7 left) 2024-04-22T18:40:01.501 INFO:tasks.rados.rados.0.smithi089.stdout:1238: done (6 left) 2024-04-22T18:40:01.501 INFO:tasks.rados.rados.0.smithi089.stdout:1239: read oid 25 snap -1 2024-04-22T18:40:01.501 INFO:tasks.rados.rados.0.smithi089.stdout:1239: expect (ObjNum 204 snap 0 seq_num 204) 2024-04-22T18:40:01.501 INFO:tasks.rados.rados.0.smithi089.stdout:1240: read oid 33 snap -1 2024-04-22T18:40:01.501 INFO:tasks.rados.rados.0.smithi089.stdout:1240: expect (ObjNum 216 snap 0 seq_num 216) 2024-04-22T18:40:01.502 INFO:tasks.rados.rados.0.smithi089.stdout:1241: delete oid 21 current snap is 0 2024-04-22T18:40:01.502 INFO:tasks.rados.rados.0.smithi089.stdout:1233: got (0) Success 2024-04-22T18:40:01.502 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 13 (ObjNum 201 snap 0 seq_num 201) dirty exists 2024-04-22T18:40:01.502 INFO:tasks.rados.rados.0.smithi089.stdout:1233: done (8 left) 2024-04-22T18:40:01.502 INFO:tasks.rados.rados.0.smithi089.stdout:1241: done (7 left) 2024-04-22T18:40:01.502 INFO:tasks.rados.rados.0.smithi089.stdout:1242: delete oid 38 current snap is 0 2024-04-22T18:40:01.506 INFO:tasks.rados.rados.0.smithi089.stdout:1240: expect (ObjNum 216 snap 0 seq_num 216) 2024-04-22T18:40:01.510 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:01.512+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:01.611 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:01.612+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:01.614 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:01.614 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:01.623 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:01.635 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:40:01.636 INFO:teuthology.orchestra.run.smithi089.stderr:marked out osd.5. 2024-04-22T18:40:01.644 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:01.645+0000 7f1e2dd22640 -1 osd.2 71 handle_osd_map injecting map crc failure 2024-04-22T18:40:01.712 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:01.713+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:01.740 INFO:tasks.rados.rados.0.smithi089.stdout:1234: expect (ObjNum 206 snap 0 seq_num 206) 2024-04-22T18:40:01.744 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:01.744 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:01.753 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:01.813 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:01.814+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:01.857 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:01.857 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:01.861 INFO:tasks.rados.rados.0.smithi089.stdout:1232: finishing write tid 3 to smithi08937929-43 2024-04-22T18:40:01.861 INFO:tasks.rados.rados.0.smithi089.stdout:1232: finishing write tid 4 to smithi08937929-43 2024-04-22T18:40:01.861 INFO:tasks.rados.rados.0.smithi089.stdout:1232: finishing write tid 5 to smithi08937929-43 2024-04-22T18:40:01.861 INFO:tasks.rados.rados.0.smithi089.stdout:1232: finishing write tid 6 to smithi08937929-43 2024-04-22T18:40:01.861 INFO:tasks.rados.rados.0.smithi089.stdout:1232: oid 43 updating version 0 to 45 2024-04-22T18:40:01.861 INFO:tasks.rados.rados.0.smithi089.stdout:1232: oid 43 updating version 45 to 46 2024-04-22T18:40:01.861 INFO:tasks.rados.rados.0.smithi089.stdout:1232: oid 43 version 46 is already newer than 43 2024-04-22T18:40:01.861 INFO:tasks.rados.rados.0.smithi089.stdout:1232: oid 43 updating version 46 to 47 2024-04-22T18:40:01.861 INFO:tasks.rados.rados.0.smithi089.stdout:1232: oid 43 version 47 is already newer than 44 2024-04-22T18:40:01.861 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 47 (ObjNum 222 snap 0 seq_num 222) dirty exists 2024-04-22T18:40:01.861 INFO:tasks.rados.rados.0.smithi089.stdout:1232: left oid 43 (ObjNum 222 snap 0 seq_num 222) 2024-04-22T18:40:01.861 INFO:tasks.rados.rados.0.smithi089.stdout:1237: finishing write tid 1 to smithi08937929-12 2024-04-22T18:40:01.861 INFO:tasks.rados.rados.0.smithi089.stdout:1237: finishing write tid 2 to smithi08937929-12 2024-04-22T18:40:01.861 INFO:tasks.rados.rados.0.smithi089.stdout:1237: finishing write tid 3 to smithi08937929-12 2024-04-22T18:40:01.862 INFO:tasks.rados.rados.0.smithi089.stdout:1237: finishing write tid 4 to smithi08937929-12 2024-04-22T18:40:01.862 INFO:tasks.rados.rados.0.smithi089.stdout:1232: done (7 left) 2024-04-22T18:40:01.862 INFO:tasks.rados.rados.0.smithi089.stdout:1234: done (6 left) 2024-04-22T18:40:01.862 INFO:tasks.rados.rados.0.smithi089.stdout:1240: done (5 left) 2024-04-22T18:40:01.862 INFO:tasks.rados.rados.0.smithi089.stdout:1242: done (4 left) 2024-04-22T18:40:01.862 INFO:tasks.rados.rados.0.smithi089.stdout:1243: set_chunk oid 23 target oid 2024-04-22T18:40:01.867 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:01.913 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:01.914+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:01.968 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:01.968 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:01.977 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:02.019 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:02.020+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:02.114 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:02.116+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:02.120 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:02.120 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:02.128 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:02.215 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:02.216+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:02.271 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:02.272 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:02.281 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:02.325 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:02.326+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:02.383 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:02.383 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:02.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:02.433 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:02.434+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:02.454 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:40:02.505 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:02.506 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:02.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:02.517 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:02.518+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:02.571 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:02.571 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:02.587 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:02.587 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:02.603 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:02.603 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:02.616 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:02.616 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:02.618 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:02.619+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:02.619 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:02.619 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:02.625 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:02.634 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:02.635 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:02.655 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:02.655 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:02.671 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:02.671 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:02.687 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:02.688 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:02.719 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:02.720+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:02.734 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:02.734 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:02.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:02.734 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:02.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.734 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:02.734 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:02.734 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 68.571896331000005, 2024-04-22T18:40:02.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:40:02.734 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.734 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:02.735 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.735 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.735 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.735 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:40:02.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.735 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:02.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:40:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:40:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:40:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:40:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:40:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:40:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:40:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:40:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:40:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:40:02.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:40:02.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:02.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:40:02.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:40:02.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:02.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:40:02.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:40:02.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:02.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:40:02.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:40:02.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:02.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:40:02.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:40:02.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:40:02.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:40:02.739 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.739 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 68.470337509999993, 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:40:02.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:40:02.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:40:02.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:40:02.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:40:02.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:40:02.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:40:02.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:40:02.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:40:02.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:40:02.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:02.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:40:02.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:40:02.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:02.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:40:02.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:40:02.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:02.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:40:02.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:40:02.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:02.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:40:02.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:40:02.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:40:02.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:40:02.745 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.745 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.745 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.745 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:02.745 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:02.745 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 68.470311108000004, 2024-04-22T18:40:02.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:40:02.745 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.745 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:02.745 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.746 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.746 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.746 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:40:02.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.746 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:02.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:02.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:40:02.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:02.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:40:02.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:40:02.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:40:02.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:40:02.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:40:02.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:40:02.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:40:02.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:40:02.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:02.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:40:02.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:40:02.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:40:02.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:40:02.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:40:02.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:40:02.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:40:02.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:40:02.750 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.750 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.750 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.750 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:02.750 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:02.750 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 68.071976778999996, 2024-04-22T18:40:02.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:40:02.750 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.750 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:02.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.750 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.751 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.751 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:40:02.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.751 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:02.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:02.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:40:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:40:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:40:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:40:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:40:02.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:40:02.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:40:02.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:40:02.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:40:02.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:40:02.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:02.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:40:02.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:40:02.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:02.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:40:02.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:40:02.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:02.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:40:02.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:40:02.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:02.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:40:02.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:40:02.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:40:02.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:40:02.755 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.755 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.755 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.756 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:02.756 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:02.756 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 67.971140481000006, 2024-04-22T18:40:02.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:40:02.756 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.756 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:02.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.756 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.756 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:40:02.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.756 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:02.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:02.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:40:02.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:02.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:40:02.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:40:02.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:40:02.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:40:02.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:40:02.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:40:02.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:40:02.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:40:02.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:40:02.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:40:02.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:02.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:40:02.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:40:02.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:02.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:40:02.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:40:02.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:02.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:40:02.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:40:02.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:02.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:40:02.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:40:02.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:40:02.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:40:02.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.761 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.761 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:02.761 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:02.761 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 67.970983211000004, 2024-04-22T18:40:02.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:40:02.761 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.761 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:02.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.762 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:40:02.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.762 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:02.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:02.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:40:02.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:40:02.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:40:02.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:40:02.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:40:02.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:40:02.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:40:02.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:02.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:40:02.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:40:02.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:40:02.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:40:02.766 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.766 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.766 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.766 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 67.970968601999999, 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:40:02.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:02.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:40:02.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:40:02.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:40:02.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:40:02.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:02.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:40:02.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:40:02.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:40:02.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:40:02.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:40:02.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 67.970958217000003, 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.771 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:02.772 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.772 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.772 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.772 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:40:02.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.772 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:02.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:02.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:40:02.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:02.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:40:02.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:40:02.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:40:02.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:40:02.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:40:02.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:40:02.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:40:02.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:40:02.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:40:02.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:40:02.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:02.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:40:02.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:40:02.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:02.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:40:02.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:40:02.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:02.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:40:02.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:40:02.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:02.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:40:02.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:40:02.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:40:02.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:40:02.776 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.776 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.776 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.777 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:02.777 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:02.777 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 67.970028287000005, 2024-04-22T18:40:02.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:40:02.777 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.777 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:02.777 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.777 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.777 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.777 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:40:02.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.777 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:02.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:02.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:40:02.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:02.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:40:02.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:40:02.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:40:02.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:40:02.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:40:02.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:40:02.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:40:02.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:40:02.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:40:02.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:40:02.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:02.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:40:02.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:40:02.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:40:02.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:40:02.782 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.782 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.782 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.782 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:02.782 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:02.782 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 67.969566162000007, 2024-04-22T18:40:02.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:40:02.782 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.782 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:02.782 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:40:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:40:02.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:02.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:40:02.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:40:02.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:40:02.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:40:02.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:02.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:40:02.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:40:02.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:02.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:40:02.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:40:02.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:02.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:40:02.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:40:02.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:02.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:40:02.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:40:02.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:40:02.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:40:02.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.787 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.788 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:02.788 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:02.788 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 67.969553357999999, 2024-04-22T18:40:02.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:40:02.788 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.788 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:02.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.788 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:40:02.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.788 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:02.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:02.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:40:02.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:02.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:40:02.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:40:02.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:40:02.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:40:02.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:40:02.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:40:02.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:40:02.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:40:02.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:02.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:40:02.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:40:02.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:40:02.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:40:02.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:40:02.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:40:02.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:40:02.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:40:02.792 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.792 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.792 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.792 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:02.792 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:02.793 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.641780901999994, 2024-04-22T18:40:02.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:40:02.793 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.793 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:02.793 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.793 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.793 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.793 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:40:02.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.793 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:02.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:02.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:40:02.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:02.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:40:02.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:40:02.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:40:02.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:40:02.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:40:02.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:40:02.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:40:02.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:40:02.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:40:02.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:40:02.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:02.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:40:02.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:40:02.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:02.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:40:02.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:40:02.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:02.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:40:02.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:40:02.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:02.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:40:02.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:40:02.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:40:02.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:40:02.798 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.798 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.798 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.798 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:40:02.798 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:02.798 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 63.63320779, 2024-04-22T18:40:02.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:40:02.798 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.798 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:02.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.799 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.799 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:40:02.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.799 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:02.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:02.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:40:02.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:02.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:40:02.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:40:02.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:40:02.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:40:02.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:40:02.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:40:02.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:40:02.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:40:02.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:40:02.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:40:02.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:02.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:40:02.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:40:02.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:02.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:40:02.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:40:02.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:02.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:40:02.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:40:02.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:02.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:40:02.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:40:02.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:40:02.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:40:02.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.803 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.804 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:40:02.804 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:02.804 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 36.598560274999997, 2024-04-22T18:40:02.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:40:02.804 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.804 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:02.804 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.804 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.804 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.804 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:40:02.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.804 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:40:02.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.807 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1684 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:02.808 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:02.808 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.129076312000002, 2024-04-22T18:40:02.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987885199999998, 2024-04-22T18:40:02.808 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.808 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:02.808 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.808 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.808 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.808 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1684 2024-04-22T18:40:02.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.808 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:02.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605278+0000", 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605280+0000", 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1770000000000001e-06 2024-04-22T18:40:02.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605285+0000", 2024-04-22T18:40:02.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-04-22T18:40:02.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605289+0000", 2024-04-22T18:40:02.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.687e-06 2024-04-22T18:40:02.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705352+0000", 2024-04-22T18:40:02.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006322199999999 2024-04-22T18:40:02.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:02.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705423+0000", 2024-04-22T18:40:02.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1136999999999993e-05 2024-04-22T18:40:02.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105017+0000", 2024-04-22T18:40:02.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.399593268 2024-04-22T18:40:02.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105081+0000", 2024-04-22T18:40:02.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4703999999999996e-05 2024-04-22T18:40:02.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105157+0000", 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6205999999999995e-05 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2191 4.1 4.fdaa93c1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.184484061999999, 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51064384399999996, 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2191 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549872+0000", 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.387e-06 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549875+0000", 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549880+0000", 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2529999999999998e-06 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060496+0000", 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51061597999999997 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060515+0000", 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9130000000000001e-05 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2198 4.d 4:bc165c76:::smithi08937929-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.183216360999999, 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50502289899999997, 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2198 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551138+0000", 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551143+0000", 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8949999999999997e-06 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551151+0000", 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8729999999999996e-06 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551158+0000", 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3099999999999997e-06 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551180+0000", 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2197000000000001e-05 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551185+0000", 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7010000000000004e-06 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657133+0000", 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105948498 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657183+0000", 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0124000000000003e-05 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.055924+0000", 2024-04-22T18:40:02.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39874079299999998 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056018+0000", 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4032999999999999e-05 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056161+0000", 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143385 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.820 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2212 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.183146719, 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50989936400000002, 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2212 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:02.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:02.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:02.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551211+0000", 2024-04-22T18:40:02.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6280000000000001e-06 2024-04-22T18:40:02.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551214+0000", 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1209999999999998e-06 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551217+0000", 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8490000000000002e-06 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061086+0000", 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50986975099999998 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061108+0000", 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1319e-05 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2213 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.183136302000001, 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50998486099999996, 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2213 2024-04-22T18:40:02.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551218+0000", 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:02.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551220+0000", 2024-04-22T18:40:02.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.319e-06 2024-04-22T18:40:02.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551223+0000", 2024-04-22T18:40:02.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9660000000000001e-06 2024-04-22T18:40:02.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.826 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551226+0000", 2024-04-22T18:40:02.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9050000000000001e-06 2024-04-22T18:40:02.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061162+0000", 2024-04-22T18:40:02.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.509936838 2024-04-22T18:40:02.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061204+0000", 2024-04-22T18:40:02.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1094999999999999e-05 2024-04-22T18:40:02.827 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.827 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.827 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.827 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2214 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 17.183127313, 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50997458100000004, 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2214 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551227+0000", 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551229+0000", 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8649999999999999e-06 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551232+0000", 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551235+0000", 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061194+0000", 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50995819099999995 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:02.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061202+0000", 2024-04-22T18:40:02.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.541e-06 2024-04-22T18:40:02.831 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.831 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.831 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.831 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:02.831 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:02.831 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:02.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:02.833 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:02.834+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:02.933 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:02.934+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:02.976 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:02.976 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:02.976 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:02.977 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:02.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:03.034 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:03.035+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:03.092 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:03.092 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:03.092 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:03.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:03.134 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:03.136+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:03.160 INFO:tasks.rados.rados.0.smithi089.stdout:1243: set_chunk oid 23 offset: 617984 length: 1690624 target oid 705f73a574b5b7a90aefb96e81056d644c4a06245d4a0a723bf12e6db5421115 offset: 0 snap -1 2024-04-22T18:40:03.160 INFO:tasks.rados.rados.0.smithi089.stdout:1244: set_chunk oid 21 target oid 2024-04-22T18:40:03.160 INFO:tasks.rados.rados.0.smithi089.stdout:1244: set_chunk oid 21 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:40:03.160 INFO:tasks.rados.rados.0.smithi089.stdout:1245: read oid 17 snap -1 2024-04-22T18:40:03.160 INFO:tasks.rados.rados.0.smithi089.stdout:1245: expect (ObjNum 192 snap 0 seq_num 192) 2024-04-22T18:40:03.160 INFO:tasks.rados.rados.0.smithi089.stdout:1246: set_chunk oid 22 target oid 2024-04-22T18:40:03.235 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:03.236+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:03.246 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:03.246 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:03.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:03.246 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:03.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.246 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:40:03.247 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:03.247 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.152445182999998, 2024-04-22T18:40:03.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:40:03.247 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.247 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.247 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:40:03.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.247 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:03.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:03.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:40:03.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:03.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:40:03.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:40:03.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:40:03.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:40:03.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:40:03.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:40:03.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:40:03.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:40:03.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:40:03.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:40:03.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:40:03.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:40:03.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:40:03.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.252947308000003, 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:03.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:40:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:40:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:40:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:40:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:40:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:40:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:03.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:40:03.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:40:03.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:40:03.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:40:03.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:40:03.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:40:03.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.144647516999996, 2024-04-22T18:40:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:40:03.258 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.258 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.258 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.258 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.258 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.258 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:40:03.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.258 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:03.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:03.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:40:03.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:03.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:40:03.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:40:03.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:40:03.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:40:03.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:40:03.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:40:03.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:40:03.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:40:03.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:40:03.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:40:03.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:40:03.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:40:03.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:40:03.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:40:03.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.262 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.263 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.263 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:03.263 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:03.263 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.813103888999997, 2024-04-22T18:40:03.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:40:03.263 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.263 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.263 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:03.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:40:03.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:40:03.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:03.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:40:03.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:40:03.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:40:03.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:40:03.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:40:03.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:40:03.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:40:03.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:40:03.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:40:03.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:40:03.267 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.267 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.267 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.268 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:03.268 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:03.268 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.722018081000002, 2024-04-22T18:40:03.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:40:03.268 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.268 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.268 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:40:03.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.268 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:03.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:03.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:40:03.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:03.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:40:03.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:40:03.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:40:03.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:40:03.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:40:03.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:40:03.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:40:03.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:40:03.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:40:03.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:40:03.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:03.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:40:03.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:40:03.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:40:03.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:40:03.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:40:03.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:40:03.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:40:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:40:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:40:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:40:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.716217821000001, 2024-04-22T18:40:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:40:03.274 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.274 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.274 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:40:03.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.274 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:03.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:40:03.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:40:03.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:40:03.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:40:03.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:40:03.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:40:03.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:40:03.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:40:03.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:40:03.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:40:03.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:40:03.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:40:03.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.278 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.279 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:03.279 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:03.279 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.711477477999999, 2024-04-22T18:40:03.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:40:03.279 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.279 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.279 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.279 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:40:03.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.279 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:40:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:40:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:40:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:40:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:40:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:40:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:40:03.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:40:03.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:40:03.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:40:03.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:03.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:40:03.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:40:03.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:40:03.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:40:03.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:40:03.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:40:03.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:40:03.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:40:03.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:40:03.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:40:03.284 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.284 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.284 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.284 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:03.284 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:03.284 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.711004580000001, 2024-04-22T18:40:03.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:40:03.284 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.284 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.284 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.284 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.285 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.285 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:40:03.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.285 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:03.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:03.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 50.710984658999998, 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:40:03.289 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.290 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:03.290 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.290 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.290 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.290 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:40:03.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.290 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:03.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:40:03.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:40:03.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 45.401660894000003, 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:03.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:40:03.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:03.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:40:03.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:40:03.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:40:03.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:40:03.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:03.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:40:03.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:40:03.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:40:03.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:40:03.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:40:03.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:40:03.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:40:03.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:40:03.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:40:03.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:40:03.298 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.298 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.298 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 45.401453357999998, 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:03.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:40:03.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:03.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:40:03.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:40:03.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:40:03.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:40:03.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:40:03.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:40:03.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:40:03.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:40:03.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:40:03.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:40:03.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:03.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:40:03.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:40:03.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:40:03.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:40:03.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.535803625, 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:40:03.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:40:03.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:03.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:40:03.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:40:03.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:40:03.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:40:03.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:40:03.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:40:03.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:03.308 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:03.309 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.321965240000001, 2024-04-22T18:40:03.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:40:03.309 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.309 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.309 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:03.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:03.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:40:03.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:03.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:40:03.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:40:03.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:40:03.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:40:03.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:03.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:40:03.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:40:03.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:40:03.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:40:03.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:40:03.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.313 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:03.314 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:03.314 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.32193183, 2024-04-22T18:40:03.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:40:03.314 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.314 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.314 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:03.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:03.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:40:03.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:40:03.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:40:03.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:40:03.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:40:03.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:40:03.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:03.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:40:03.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:40:03.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:40:03.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:40:03.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:40:03.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:40:03.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:40:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:40:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:40:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:40:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.318 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.263049175999999, 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:40:03.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:03.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:40:03.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:40:03.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:40:03.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:40:03.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:03.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:40:03.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:40:03.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:40:03.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:40:03.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:40:03.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:40:03.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:03.323 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.262910560000002, 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:40:03.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:40:03.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:40:03.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:40:03.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:40:03.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:40:03.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:40:03.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:40:03.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:40:03.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:40:03.328 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.328 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.328 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.328 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:03.328 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:03.328 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.262891557, 2024-04-22T18:40:03.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:40:03.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:40:03.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:40:03.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:40:03.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:40:03.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:40:03.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:40:03.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.641879024000001, 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.333 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:40:03.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:40:03.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:40:03.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:40:03.338 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.338 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.338 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.338 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:03.338 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:03.338 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.641359433000002, 2024-04-22T18:40:03.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:40:03.338 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.338 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:40:03.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:03.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:40:03.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:40:03.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:40:03.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:40:03.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:40:03.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:40:03.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:40:03.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:40:03.343 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.343 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.343 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.343 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:03.343 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:03.343 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.541206549999998, 2024-04-22T18:40:03.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:40:03.344 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.344 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:03.344 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.344 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.344 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.344 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:40:03.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.344 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:03.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:40:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.347 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.348 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.348 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.348 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.348 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:03.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:03.349 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:03.350+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:03.450 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:03.451+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:03.452 INFO:tasks.rados.rados.0.smithi089.stdout:1246: set_chunk oid 22 offset: 506880 length: 1746432 target oid 73f6cd42eb17b7fbc29bb083f5443f1d5c65f51bcb8aa3d092b3b80aa1d4c657 offset: 0 snap -1 2024-04-22T18:40:03.452 INFO:tasks.rados.rados.0.smithi089.stdout:1247: tier_promote oid 42 2024-04-22T18:40:03.452 INFO:tasks.rados.rados.0.smithi089.stdout:1237: finishing write tid 5 to smithi08937929-12 2024-04-22T18:40:03.452 INFO:tasks.rados.rados.0.smithi089.stdout:1237: oid 12 updating version 0 to 79 2024-04-22T18:40:03.452 INFO:tasks.rados.rados.0.smithi089.stdout:1237: oid 12 updating version 79 to 80 2024-04-22T18:40:03.453 INFO:tasks.rados.rados.0.smithi089.stdout:1237: oid 12 version 80 is already newer than 78 2024-04-22T18:40:03.453 INFO:tasks.rados.rados.0.smithi089.stdout:1237: oid 12 updating version 80 to 81 2024-04-22T18:40:03.453 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 81 (ObjNum 223 snap 0 seq_num 223) dirty exists 2024-04-22T18:40:03.453 INFO:tasks.rados.rados.0.smithi089.stdout:1237: left oid 12 (ObjNum 223 snap 0 seq_num 223) 2024-04-22T18:40:03.453 INFO:tasks.rados.rados.0.smithi089.stdout:1237: done (8 left) 2024-04-22T18:40:03.453 INFO:tasks.rados.rados.0.smithi089.stdout:1248: read oid 10 snap -1 2024-04-22T18:40:03.453 INFO:tasks.rados.rados.0.smithi089.stdout:1248: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:40:03.453 INFO:tasks.rados.rados.0.smithi089.stdout:1249: read oid 45 snap -1 2024-04-22T18:40:03.453 INFO:tasks.rados.rados.0.smithi089.stdout:1249: expect (ObjNum 166 snap 0 seq_num 166) 2024-04-22T18:40:03.453 INFO:tasks.rados.rados.0.smithi089.stdout:1250: read oid 11 snap -1 2024-04-22T18:40:03.453 INFO:tasks.rados.rados.0.smithi089.stdout:1250: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:40:03.453 INFO:tasks.rados.rados.0.smithi089.stdout:1251: write oid 43 current snap is 0 2024-04-22T18:40:03.453 INFO:tasks.rados.rados.0.smithi089.stdout:1239: expect (ObjNum 204 snap 0 seq_num 204) 2024-04-22T18:40:03.492 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:03.493 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:03.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.493 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGPush(4.1 73/72 [PushOp(4:809dfd21:::smithi08937929-5:head, version: 70'118, data_included: [447119~786917,1665416~261659], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:809dfd21:::smithi08937929-5:head@70'118, size: 3697540, copy_subset: [0~3697540], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1927075, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-22T18:40:03.493 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:03.491543+0000", 2024-04-22T18:40:03.493 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0029659199999999999, 2024-04-22T18:40:03.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029934800000000002, 2024-04-22T18:40:03.493 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.493 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:03.493 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.491543+0000", 2024-04-22T18:40:03.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.491543+0000", 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.491541+0000", 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.491750+0000", 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00020853500000000001 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.491764+0000", 2024-04-22T18:40:03.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4358999999999999e-05 2024-04-22T18:40:03.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.491775+0000", 2024-04-22T18:40:03.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0812000000000001e-05 2024-04-22T18:40:03.495 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.495 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.495 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.495 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.495 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:40:03.495 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 1 2024-04-22T18:40:03.495 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:03.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:03.551 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:03.552+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:03.604 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:03.605 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:03.605 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:03.605 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:03.605 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:03.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:03.641 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:03.643+0000 7f3e7be88640 -1 osd.0 73 handle_osd_map injecting map crc failure 2024-04-22T18:40:03.651 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:03.653+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:03.653 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:40:03.669 INFO:tasks.rados.rados.0.smithi089.stdout:1231: finishing write tid 1 to smithi08937929-41 2024-04-22T18:40:03.669 INFO:tasks.rados.rados.0.smithi089.stdout:1231: finishing write tid 2 to smithi08937929-41 2024-04-22T18:40:03.669 INFO:tasks.rados.rados.0.smithi089.stdout:1231: finishing write tid 3 to smithi08937929-41 2024-04-22T18:40:03.669 INFO:tasks.rados.rados.0.smithi089.stdout:1251: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-04-22T18:40:03.676 INFO:tasks.rados.rados.0.smithi089.stdout:1251: writing smithi08937929-43 from 598382 to 1378893 tid 1 2024-04-22T18:40:03.683 INFO:tasks.rados.rados.0.smithi089.stdout:1251: writing smithi08937929-43 from 2075401 to 2526468 tid 2 2024-04-22T18:40:03.691 INFO:tasks.rados.rados.0.smithi089.stdout:1251: writing smithi08937929-43 from 3107008 to 3755777 tid 3 2024-04-22T18:40:03.691 INFO:tasks.rados.rados.0.smithi089.stdout:1239: done (11 left) 2024-04-22T18:40:03.691 INFO:tasks.rados.rados.0.smithi089.stdout:1252: write (excl) oid 40 current snap is 0 2024-04-22T18:40:03.691 INFO:tasks.rados.rados.0.smithi089.stdout:1231: finishing write tid 4 to smithi08937929-41 2024-04-22T18:40:03.691 INFO:tasks.rados.rados.0.smithi089.stdout:1231: finishing write tid 5 to smithi08937929-41 2024-04-22T18:40:03.691 INFO:tasks.rados.rados.0.smithi089.stdout:1231: oid 41 updating version 0 to 112 2024-04-22T18:40:03.691 INFO:tasks.rados.rados.0.smithi089.stdout:1231: oid 41 updating version 112 to 113 2024-04-22T18:40:03.691 INFO:tasks.rados.rados.0.smithi089.stdout:1231: oid 41 version 113 is already newer than 111 2024-04-22T18:40:03.691 INFO:tasks.rados.rados.0.smithi089.stdout:1231: oid 41 version 113 is already newer than 110 2024-04-22T18:40:03.691 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 113 (ObjNum 221 snap 0 seq_num 221) dirty exists 2024-04-22T18:40:03.691 INFO:tasks.rados.rados.0.smithi089.stdout:1231: left oid 41 (ObjNum 221 snap 0 seq_num 221) 2024-04-22T18:40:03.691 INFO:tasks.rados.rados.0.smithi089.stdout:1244: finishing set_chunk to oid 21 2024-04-22T18:40:03.691 INFO:tasks.rados.rados.0.smithi089.stdout:1244: got expected ENOENT (src dne) 2024-04-22T18:40:03.692 INFO:tasks.rados.rados.0.smithi089.stdout:1243: finishing set_chunk to oid 23 2024-04-22T18:40:03.692 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 89 (ObjNum 158 snap 0 seq_num 158) dirty exists 2024-04-22T18:40:03.692 INFO:tasks.rados.rados.0.smithi089.stdout:1246: finishing set_chunk to oid 22 2024-04-22T18:40:03.692 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 22 set_chunk returned error code -95 offset: 506880 length: 1746432 2024-04-22T18:40:03.692 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 0 (ObjNum 135 snap 0 seq_num 135) dirty exists 2024-04-22T18:40:03.692 INFO:tasks.rados.rados.0.smithi089.stdout:1247: got (0) Success 2024-04-22T18:40:03.692 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 23 (ObjNum 197 snap 0 seq_num 197) dirty exists 2024-04-22T18:40:03.692 INFO:tasks.rados.rados.0.smithi089.stdout:1249: expect (ObjNum 166 snap 0 seq_num 166) 2024-04-22T18:40:03.753 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:03.753+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:03.761 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:03.761 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:03.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:03.762 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:03.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.762 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:22 4.a 4:5aa97f94:::smithi08937929-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:03.762 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:40:03.762 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.6045436, 2024-04-22T18:40:03.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30948398900000001, 2024-04-22T18:40:03.762 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.762 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.763 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.763 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.763 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 22 2024-04-22T18:40:03.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.764 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:40:03.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:40:03.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157259+0000", 2024-04-22T18:40:03.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:03.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157318+0000", 2024-04-22T18:40:03.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8906000000000001e-05 2024-04-22T18:40:03.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157327+0000", 2024-04-22T18:40:03.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:40:03.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157332+0000", 2024-04-22T18:40:03.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4689999999999996e-06 2024-04-22T18:40:03.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257479+0000", 2024-04-22T18:40:03.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014700999999999 2024-04-22T18:40:03.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:03.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257492+0000", 2024-04-22T18:40:03.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2574e-05 2024-04-22T18:40:03.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259582+0000", 2024-04-22T18:40:03.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020904019999999999 2024-04-22T18:40:03.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259668+0000", 2024-04-22T18:40:03.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5826000000000001e-05 2024-04-22T18:40:03.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:03.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259772+0000", 2024-04-22T18:40:03.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104048 2024-04-22T18:40:03.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363453+0000", 2024-04-22T18:40:03.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103680183 2024-04-22T18:40:03.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466692+0000", 2024-04-22T18:40:03.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10323948199999999 2024-04-22T18:40:03.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466720+0000", 2024-04-22T18:40:03.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7886e-05 2024-04-22T18:40:03.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466744+0000", 2024-04-22T18:40:03.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4049e-05 2024-04-22T18:40:03.769 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.769 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.769 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.769 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:23 4.a 4:5aa97f94:::smithi08937929-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:03.769 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:40:03.769 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.604465912999999, 2024-04-22T18:40:03.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39003820300000003, 2024-04-22T18:40:03.769 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.769 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.769 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.770 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.770 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.770 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 23 2024-04-22T18:40:03.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.770 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:40:03.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:40:03.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157337+0000", 2024-04-22T18:40:03.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:03.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157340+0000", 2024-04-22T18:40:03.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8609999999999998e-06 2024-04-22T18:40:03.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157343+0000", 2024-04-22T18:40:03.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9859999999999999e-06 2024-04-22T18:40:03.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157346+0000", 2024-04-22T18:40:03.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8119999999999999e-06 2024-04-22T18:40:03.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257563+0000", 2024-04-22T18:40:03.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100217497 2024-04-22T18:40:03.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:03.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257574+0000", 2024-04-22T18:40:03.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0777e-05 2024-04-22T18:40:03.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260112+0000", 2024-04-22T18:40:03.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002537681 2024-04-22T18:40:03.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260191+0000", 2024-04-22T18:40:03.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9320999999999995e-05 2024-04-22T18:40:03.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:03.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260333+0000", 2024-04-22T18:40:03.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014177699999999999 2024-04-22T18:40:03.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363792+0000", 2024-04-22T18:40:03.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103458862 2024-04-22T18:40:03.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547303+0000", 2024-04-22T18:40:03.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18351162100000001 2024-04-22T18:40:03.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547337+0000", 2024-04-22T18:40:03.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3640999999999999e-05 2024-04-22T18:40:03.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547376+0000", 2024-04-22T18:40:03.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8915000000000003e-05 2024-04-22T18:40:03.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.778 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.778 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:24 4.a 4:5aa97f94:::smithi08937929-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:03.778 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:03.778 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.604439482000004, 2024-04-22T18:40:03.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39017823499999998, 2024-04-22T18:40:03.778 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.778 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:03.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.779 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 24 2024-04-22T18:40:03.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.779 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:03.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:03.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:03.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:03.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157366+0000", 2024-04-22T18:40:03.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3120000000000001e-06 2024-04-22T18:40:03.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157370+0000", 2024-04-22T18:40:03.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1300000000000003e-06 2024-04-22T18:40:03.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157373+0000", 2024-04-22T18:40:03.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-04-22T18:40:03.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260553+0000", 2024-04-22T18:40:03.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103179996 2024-04-22T18:40:03.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:03.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260625+0000", 2024-04-22T18:40:03.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2236000000000004e-05 2024-04-22T18:40:03.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547414+0000", 2024-04-22T18:40:03.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.286789181 2024-04-22T18:40:03.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547471+0000", 2024-04-22T18:40:03.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6192e-05 2024-04-22T18:40:03.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547542+0000", 2024-04-22T18:40:03.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1622e-05 2024-04-22T18:40:03.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.783 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.783 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:49 4.a 4:5d2aee44:::smithi08937929-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:03.783 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:40:03.783 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.527640824000002, 2024-04-22T18:40:03.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31344442500000003, 2024-04-22T18:40:03.783 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.783 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.784 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 49 2024-04-22T18:40:03.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.784 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:40:03.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:40:03.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234158+0000", 2024-04-22T18:40:03.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999957 2024-04-22T18:40:03.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234901+0000", 2024-04-22T18:40:03.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074250800000000001 2024-04-22T18:40:03.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234914+0000", 2024-04-22T18:40:03.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3244999999999999e-05 2024-04-22T18:40:03.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234922+0000", 2024-04-22T18:40:03.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4590000000000001e-06 2024-04-22T18:40:03.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260665+0000", 2024-04-22T18:40:03.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025743423000000001 2024-04-22T18:40:03.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260779+0000", 2024-04-22T18:40:03.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114063 2024-04-22T18:40:03.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:03.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260913+0000", 2024-04-22T18:40:03.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013355499999999999 2024-04-22T18:40:03.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363798+0000", 2024-04-22T18:40:03.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102885003 2024-04-22T18:40:03.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547567+0000", 2024-04-22T18:40:03.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.183768971 2024-04-22T18:40:03.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547589+0000", 2024-04-22T18:40:03.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2507999999999999e-05 2024-04-22T18:40:03.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547607+0000", 2024-04-22T18:40:03.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8009999999999999e-05 2024-04-22T18:40:03.788 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.788 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.789 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:50 4.a 4:5d2aee44:::smithi08937929-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:03.789 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:03.789 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.522194644999999, 2024-04-22T18:40:03.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40856295399999998, 2024-04-22T18:40:03.789 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.789 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.789 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.789 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.789 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.789 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 50 2024-04-22T18:40:03.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.789 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:03.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:03.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239607+0000", 2024-04-22T18:40:03.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:03.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240309+0000", 2024-04-22T18:40:03.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070201599999999997 2024-04-22T18:40:03.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240320+0000", 2024-04-22T18:40:03.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1024e-05 2024-04-22T18:40:03.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240327+0000", 2024-04-22T18:40:03.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0049999999999998e-06 2024-04-22T18:40:03.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362065+0000", 2024-04-22T18:40:03.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.121737812 2024-04-22T18:40:03.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362121+0000", 2024-04-22T18:40:03.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5868000000000001e-05 2024-04-22T18:40:03.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:03.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362210+0000", 2024-04-22T18:40:03.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8533000000000001e-05 2024-04-22T18:40:03.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365322+0000", 2024-04-22T18:40:03.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031121220000000002 2024-04-22T18:40:03.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648054+0000", 2024-04-22T18:40:03.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.282732549 2024-04-22T18:40:03.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648105+0000", 2024-04-22T18:40:03.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1276999999999997e-05 2024-04-22T18:40:03.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648172+0000", 2024-04-22T18:40:03.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6398999999999999e-05 2024-04-22T18:40:03.794 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.794 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.794 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.794 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:51 4.a 4:5d2aee44:::smithi08937929-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:03.794 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:03.795 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.521472110999994, 2024-04-22T18:40:03.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40793057199999999, 2024-04-22T18:40:03.795 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.795 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.795 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.795 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.795 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.795 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 51 2024-04-22T18:40:03.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.795 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:03.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:03.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:03.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:03.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240336+0000", 2024-04-22T18:40:03.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:40:03.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240339+0000", 2024-04-22T18:40:03.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9009999999999998e-06 2024-04-22T18:40:03.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240342+0000", 2024-04-22T18:40:03.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:40:03.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363080+0000", 2024-04-22T18:40:03.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122738075 2024-04-22T18:40:03.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363164+0000", 2024-04-22T18:40:03.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3722999999999998e-05 2024-04-22T18:40:03.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:03.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363287+0000", 2024-04-22T18:40:03.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012301099999999999 2024-04-22T18:40:03.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466753+0000", 2024-04-22T18:40:03.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103466217 2024-04-22T18:40:03.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648205+0000", 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.181451788 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648231+0000", 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6597999999999998e-05 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648262+0000", 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0871000000000003e-05 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.799 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.800 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.800 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:52 4.a 4:5d2aee44:::smithi08937929-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:03.800 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:03.800 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.521459652000004, 2024-04-22T18:40:03.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40812731200000002, 2024-04-22T18:40:03.800 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.800 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:03.800 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.800 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.800 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.800 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 52 2024-04-22T18:40:03.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240345+0000", 2024-04-22T18:40:03.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4300000000000001e-06 2024-04-22T18:40:03.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240349+0000", 2024-04-22T18:40:03.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2570000000000001e-06 2024-04-22T18:40:03.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240358+0000", 2024-04-22T18:40:03.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-22T18:40:03.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363697+0000", 2024-04-22T18:40:03.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12333965299999999 2024-04-22T18:40:03.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:03.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363777+0000", 2024-04-22T18:40:03.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9883999999999996e-05 2024-04-22T18:40:03.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648296+0000", 2024-04-22T18:40:03.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451861299999998 2024-04-22T18:40:03.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648379+0000", 2024-04-22T18:40:03.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3302999999999998e-05 2024-04-22T18:40:03.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648471+0000", 2024-04-22T18:40:03.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2029999999999998e-05 2024-04-22T18:40:03.804 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.804 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.804 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.804 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:03.805 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:03.805 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.502156317000001, 2024-04-22T18:40:03.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:40:03.805 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.805 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.805 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.805 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.805 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.805 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:40:03.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.805 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:03.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:03.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:40:03.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:40:03.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:40:03.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:40:03.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:40:03.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:40:03.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:03.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:40:03.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:40:03.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:40:03.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:40:03.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:40:03.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:40:03.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:40:03.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:40:03.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:40:03.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:40:03.810 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.810 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.810 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.810 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:03.810 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:03.810 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.497373886999995, 2024-04-22T18:40:03.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:40:03.810 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:40:03.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:03.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:40:03.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:40:03.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:40:03.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:40:03.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:40:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:40:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:40:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:40:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:40:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.497279684000006, 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:40:03.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:40:03.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:40:03.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:40:03.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:40:03.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:40:03.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:40:03.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:40:03.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:03.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:40:03.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:40:03.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:40:03.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:40:03.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:40:03.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:40:03.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:40:03.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:40:03.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:40:03.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:40:03.821 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.821 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.821 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.821 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:03.821 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:03.821 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.497258795999997, 2024-04-22T18:40:03.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:40:03.821 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.821 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:03.822 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.822 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.822 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.822 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:40:03.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.822 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:03.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:40:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:40:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:40:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:40:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:40:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:40:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:40:03.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:40:03.826 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.826 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.826 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.826 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:03.826 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:03.826 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 68.714251133000005, 2024-04-22T18:40:03.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:40:03.826 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.826 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.826 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.826 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.826 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:40:03.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:03.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:40:03.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:40:03.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:40:03.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:40:03.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:40:03.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:40:03.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:40:03.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:40:03.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:40:03.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:40:03.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:03.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:40:03.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:40:03.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:40:03.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:40:03.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:40:03.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:40:03.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:03.831 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 68.706419726999997, 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:40:03.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:40:03.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:40:03.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:40:03.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:40:03.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:40:03.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:40:03.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:40:03.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:40:03.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:40:03.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:40:03.836 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.836 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.836 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.836 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2052 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:03.837 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:03.837 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.184307946000001, 2024-04-22T18:40:03.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43058958600000002, 2024-04-22T18:40:03.837 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.837 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.837 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.837 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.837 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.837 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2052 2024-04-22T18:40:03.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.837 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:03.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:03.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577494+0000", 2024-04-22T18:40:03.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:03.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577500+0000", 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3279999999999996e-06 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577510+0000", 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9450000000000005e-06 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577515+0000", 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7799999999999997e-06 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677622+0000", 2024-04-22T18:40:03.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010654099999999 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677710+0000", 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8472000000000003e-05 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.906619+0000", 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.228908519 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907343+0000", 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072442600000000002 2024-04-22T18:40:03.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.007972+0000", 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100628781 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008022+0000", 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9843999999999999e-05 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008085+0000", 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3256999999999994e-05 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.841 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.842 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.842 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2087 4.a 4:537c89f8:::smithi08937929-47:head [assert-version v37,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-04-22T18:40:03.842 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:40:03.842 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.854391186000001, 2024-04-22T18:40:03.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38904324299999998, 2024-04-22T18:40:03.842 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.842 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.842 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.842 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.842 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.842 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2087 2024-04-22T18:40:03.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.843 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:40:03.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:40:03.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907410+0000", 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907417+0000", 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.306e-06 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907427+0000", 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4930000000000004e-06 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907432+0000", 2024-04-22T18:40:03.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5400000000000003e-06 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907477+0000", 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4851999999999997e-05 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907537+0000", 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9460999999999999e-05 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.295587+0000", 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38805073200000001 2024-04-22T18:40:03.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296281+0000", 2024-04-22T18:40:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00069370800000000002 2024-04-22T18:40:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296400+0000", 2024-04-22T18:40:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118579 2024-04-22T18:40:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296431+0000", 2024-04-22T18:40:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1627000000000001e-05 2024-04-22T18:40:03.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296456+0000", 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4331000000000001e-05 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.5545470970000004, 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.847 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:40:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:03.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:40:03.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:03.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:40:03.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:40:03.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:40:03.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:40:03.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:40:03.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:40:03.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:40:03.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:40:03.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:40:03.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:40:03.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:03.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:40:03.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:40:03.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:40:03.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:40:03.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:40:03.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:40:03.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:40:03.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:40:03.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:40:03.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:40:03.852 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.852 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.852 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.853 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2505 4.5 4:a6d237d2:::smithi08937929-43:head [setxattr _header (56) in=7b,truncate 3529470] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:03.853 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:03.853 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.554154402, 2024-04-22T18:40:03.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43706744400000003, 2024-04-22T18:40:03.853 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.853 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.853 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.853 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.853 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.853 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2505 2024-04-22T18:40:03.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.853 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:03.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:03.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207648+0000", 2024-04-22T18:40:03.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:03.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207653+0000", 2024-04-22T18:40:03.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2989999999999998e-06 2024-04-22T18:40:03.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207656+0000", 2024-04-22T18:40:03.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0199999999999999e-06 2024-04-22T18:40:03.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207659+0000", 2024-04-22T18:40:03.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2270000000000002e-06 2024-04-22T18:40:03.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243213+0000", 2024-04-22T18:40:03.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035553653999999997 2024-04-22T18:40:03.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243273+0000", 2024-04-22T18:40:03.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0300000000000002e-05 2024-04-22T18:40:03.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:03.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243359+0000", 2024-04-22T18:40:03.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6275999999999999e-05 2024-04-22T18:40:03.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297443+0000", 2024-04-22T18:40:03.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.054083930000000002 2024-04-22T18:40:03.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644651+0000", 2024-04-22T18:40:03.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.347208181 2024-04-22T18:40:03.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.857 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644684+0000", 2024-04-22T18:40:03.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3080999999999997e-05 2024-04-22T18:40:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644717+0000", 2024-04-22T18:40:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2264000000000001e-05 2024-04-22T18:40:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 8.5541421890000002, 2024-04-22T18:40:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:40:03.858 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:03.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:40:03.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:40:03.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:40:03.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:40:03.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:40:03.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:03.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:40:03.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:40:03.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:40:03.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:40:03.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:40:03.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:40:03.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:40:03.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:40:03.862 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.863 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.863 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.863 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2615 4.5 4:a6d237d2:::smithi08937929-43:head [write 463926~462166] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-22T18:40:03.863 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:58.352901+0000", 2024-04-22T18:40:03.863 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.4089027410000003, 2024-04-22T18:40:03.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.307052514, 2024-04-22T18:40:03.863 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.863 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.863 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.863 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.863 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.863 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2615 2024-04-22T18:40:03.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.864 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.352901+0000", 2024-04-22T18:40:03.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.352901+0000", 2024-04-22T18:40:03.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.352897+0000", 2024-04-22T18:40:03.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999962 2024-04-22T18:40:03.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.353421+0000", 2024-04-22T18:40:03.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052391999999999998 2024-04-22T18:40:03.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.353433+0000", 2024-04-22T18:40:03.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2128e-05 2024-04-22T18:40:03.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.353460+0000", 2024-04-22T18:40:03.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7211000000000001e-05 2024-04-22T18:40:03.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.453670+0000", 2024-04-22T18:40:03.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10020929100000001 2024-04-22T18:40:03.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.453809+0000", 2024-04-22T18:40:03.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013967800000000001 2024-04-22T18:40:03.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:03.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.454021+0000", 2024-04-22T18:40:03.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021190900000000001 2024-04-22T18:40:03.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659805+0000", 2024-04-22T18:40:03.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20578357999999999 2024-04-22T18:40:03.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659886+0000", 2024-04-22T18:40:03.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1310000000000006e-05 2024-04-22T18:40:03.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659925+0000", 2024-04-22T18:40:03.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9233000000000002e-05 2024-04-22T18:40:03.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659953+0000", 2024-04-22T18:40:03.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7894000000000001e-05 2024-04-22T18:40:03.868 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.868 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.868 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.868 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2616 4.5 4:a6d237d2:::smithi08937929-43:head [write 1723898~436391] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-22T18:40:03.868 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:58.359646+0000", 2024-04-22T18:40:03.869 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.4021578449999996, 2024-04-22T18:40:03.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.300417604, 2024-04-22T18:40:03.869 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:03.869 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:03.869 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:03.869 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:03.869 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:03.869 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2616 2024-04-22T18:40:03.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.869 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:03.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:03.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.359646+0000", 2024-04-22T18:40:03.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:03.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.359646+0000", 2024-04-22T18:40:03.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:03.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:03.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.359642+0000", 2024-04-22T18:40:03.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:03.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:03.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.359916+0000", 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027331399999999998 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.359928+0000", 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2063e-05 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.359952+0000", 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4002000000000002e-05 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.455280+0000", 2024-04-22T18:40:03.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.095328063000000005 2024-04-22T18:40:03.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:03.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.455339+0000", 2024-04-22T18:40:03.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9534999999999998e-05 2024-04-22T18:40:03.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:03.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.455419+0000", 2024-04-22T18:40:03.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9923e-05 2024-04-22T18:40:03.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:03.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659831+0000", 2024-04-22T18:40:03.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.204412027 2024-04-22T18:40:03.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:03.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.660001+0000", 2024-04-22T18:40:03.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016913299999999999 2024-04-22T18:40:03.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:03.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.660042+0000", 2024-04-22T18:40:03.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1402000000000001e-05 2024-04-22T18:40:03.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:03.873 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:03.873 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:03.874 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.660063+0000", 2024-04-22T18:40:03.874 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1367999999999999e-05 2024-04-22T18:40:03.874 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.874 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.874 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.874 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:03.874 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:03.874 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:03.876 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:03.876 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:03.877+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:03.880 INFO:tasks.rados.rados.0.smithi089.stdout:1245: expect (ObjNum 192 snap 0 seq_num 192) 2024-04-22T18:40:03.976 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:03.978+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:04.017 INFO:tasks.rados.rados.0.smithi089.stdout:1248: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:40:04.018 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:04.019 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:04.027 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:04.078 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:04.079+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:04.084 INFO:tasks.rados.rados.0.smithi089.stdout:1251: finishing write tid 1 to smithi08937929-43 2024-04-22T18:40:04.084 INFO:tasks.rados.rados.0.smithi089.stdout:1251: finishing write tid 2 to smithi08937929-43 2024-04-22T18:40:04.084 INFO:tasks.rados.rados.0.smithi089.stdout:1251: finishing write tid 3 to smithi08937929-43 2024-04-22T18:40:04.084 INFO:tasks.rados.rados.0.smithi089.stdout:1250: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:40:04.129 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:04.129 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:04.129 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:04.129 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:04.129 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:04.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:04.181 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:04.182+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:04.242 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:04.242 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:04.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:04.243 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:04.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.243 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:04.243 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:40:04.243 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.500517497999994, 2024-04-22T18:40:04.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30520528099999999, 2024-04-22T18:40:04.243 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.243 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.243 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.243 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.243 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.243 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 35 2024-04-22T18:40:04.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.244 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:40:04.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:40:04.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741852+0000", 2024-04-22T18:40:04.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:04.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741927+0000", 2024-04-22T18:40:04.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5123000000000006e-05 2024-04-22T18:40:04.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741953+0000", 2024-04-22T18:40:04.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5378999999999999e-05 2024-04-22T18:40:04.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741977+0000", 2024-04-22T18:40:04.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4431999999999999e-05 2024-04-22T18:40:04.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844243+0000", 2024-04-22T18:40:04.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102265525 2024-04-22T18:40:04.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844330+0000", 2024-04-22T18:40:04.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7417999999999995e-05 2024-04-22T18:40:04.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844470+0000", 2024-04-22T18:40:04.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000140197 2024-04-22T18:40:04.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944767+0000", 2024-04-22T18:40:04.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100296409 2024-04-22T18:40:04.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.046968+0000", 2024-04-22T18:40:04.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10220148499999999 2024-04-22T18:40:04.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047018+0000", 2024-04-22T18:40:04.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9206000000000003e-05 2024-04-22T18:40:04.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047059+0000", 2024-04-22T18:40:04.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1496999999999999e-05 2024-04-22T18:40:04.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.248 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.248 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:04.248 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:04.248 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.500386973999994, 2024-04-22T18:40:04.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:40:04.249 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.249 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.249 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.249 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.249 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.249 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:40:04.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.249 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:04.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:04.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:40:04.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:04.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:40:04.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:40:04.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:40:04.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:40:04.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:40:04.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:40:04.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:40:04.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:40:04.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:40:04.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:40:04.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:40:04.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:40:04.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:40:04.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:40:04.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:40:04.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:40:04.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:40:04.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:40:04.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:40:04.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:40:04.253 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.253 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.254 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.254 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:04.254 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:04.254 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.500347282999996, 2024-04-22T18:40:04.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:40:04.254 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.254 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.254 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.254 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.254 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.254 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:40:04.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.255 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:04.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:04.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:40:04.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:04.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:40:04.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:40:04.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:40:04.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:40:04.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:40:04.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:40:04.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:40:04.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:40:04.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:40:04.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:40:04.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:40:04.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:40:04.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:40:04.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:40:04.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:40:04.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:40:04.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:40:04.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:40:04.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:40:04.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:40:04.259 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.259 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.259 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.259 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:04.259 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:40:04.259 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 82.256166235999999, 2024-04-22T18:40:04.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402419109, 2024-04-22T18:40:04.260 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.260 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.260 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.260 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.260 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.260 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 68 2024-04-22T18:40:04.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.260 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:40:04.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:40:04.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986204+0000", 2024-04-22T18:40:04.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986208+0000", 2024-04-22T18:40:04.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8149999999999999e-06 2024-04-22T18:40:04.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986214+0000", 2024-04-22T18:40:04.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.604e-06 2024-04-22T18:40:04.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986218+0000", 2024-04-22T18:40:04.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6689999999999999e-06 2024-04-22T18:40:04.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986833+0000", 2024-04-22T18:40:04.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000615648 2024-04-22T18:40:04.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986891+0000", 2024-04-22T18:40:04.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7915999999999999e-05 2024-04-22T18:40:04.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986981+0000", 2024-04-22T18:40:04.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759999999999994e-05 2024-04-22T18:40:04.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.987584+0000", 2024-04-22T18:40:04.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060270199999999999 2024-04-22T18:40:04.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388527+0000", 2024-04-22T18:40:04.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40094366999999997 2024-04-22T18:40:04.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388583+0000", 2024-04-22T18:40:04.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5383000000000002e-05 2024-04-22T18:40:04.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388624+0000", 2024-04-22T18:40:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501999999999997e-05 2024-04-22T18:40:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 82.256150446999996, 2024-04-22T18:40:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:40:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.266 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.266 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:40:04.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.266 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:04.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:04.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:40:04.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:40:04.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:40:04.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:40:04.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:40:04.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:40:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:40:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:40:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:40:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:40:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:40:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:40:04.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:40:04.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:40:04.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:40:04.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:40:04.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:40:04.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:40:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:40:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:40:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:40:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.271 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:04.271 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:40:04.271 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 81.139757438000004, 2024-04-22T18:40:04.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305563733, 2024-04-22T18:40:04.271 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.271 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.271 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.271 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.271 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.271 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 87 2024-04-22T18:40:04.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.271 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:40:04.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:40:04.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102613+0000", 2024-04-22T18:40:04.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:04.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102668+0000", 2024-04-22T18:40:04.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4669000000000001e-05 2024-04-22T18:40:04.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102680+0000", 2024-04-22T18:40:04.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2026e-05 2024-04-22T18:40:04.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102687+0000", 2024-04-22T18:40:04.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8339999999999999e-06 2024-04-22T18:40:04.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405644+0000", 2024-04-22T18:40:04.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30295623199999999 2024-04-22T18:40:04.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405743+0000", 2024-04-22T18:40:04.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.98e-05 2024-04-22T18:40:04.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405868+0000", 2024-04-22T18:40:04.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012457899999999999 2024-04-22T18:40:04.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407047+0000", 2024-04-22T18:40:04.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011791449999999999 2024-04-22T18:40:04.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408088+0000", 2024-04-22T18:40:04.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010411680000000001 2024-04-22T18:40:04.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408135+0000", 2024-04-22T18:40:04.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6255999999999999e-05 2024-04-22T18:40:04.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408178+0000", 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2877000000000003e-05 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 81.139677508999995, 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30560089000000001, 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.276 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.277 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.277 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 88 2024-04-22T18:40:04.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.277 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:40:04.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:40:04.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102693+0000", 2024-04-22T18:40:04.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:04.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102698+0000", 2024-04-22T18:40:04.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9699999999999998e-06 2024-04-22T18:40:04.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102708+0000", 2024-04-22T18:40:04.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.006e-05 2024-04-22T18:40:04.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102715+0000", 2024-04-22T18:40:04.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6599999999999995e-06 2024-04-22T18:40:04.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406637+0000", 2024-04-22T18:40:04.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392130099999998 2024-04-22T18:40:04.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406696+0000", 2024-04-22T18:40:04.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9066e-05 2024-04-22T18:40:04.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406784+0000", 2024-04-22T18:40:04.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8194000000000003e-05 2024-04-22T18:40:04.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407447+0000", 2024-04-22T18:40:04.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066295700000000004 2024-04-22T18:40:04.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408237+0000", 2024-04-22T18:40:04.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079032600000000005 2024-04-22T18:40:04.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408270+0000", 2024-04-22T18:40:04.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3157000000000001e-05 2024-04-22T18:40:04.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408295+0000", 2024-04-22T18:40:04.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4151999999999998e-05 2024-04-22T18:40:04.281 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.281 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.281 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.282 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:04.282 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:40:04.282 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.388445384999997, 2024-04-22T18:40:04.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30533819600000001, 2024-04-22T18:40:04.282 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.282 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.282 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.282 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.282 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.282 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 106 2024-04-22T18:40:04.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.282 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:40:04.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:40:04.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853925+0000", 2024-04-22T18:40:04.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853958+0000", 2024-04-22T18:40:04.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2536000000000002e-05 2024-04-22T18:40:04.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853964+0000", 2024-04-22T18:40:04.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6239999999999999e-06 2024-04-22T18:40:04.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853967+0000", 2024-04-22T18:40:04.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9700000000000001e-06 2024-04-22T18:40:04.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855144+0000", 2024-04-22T18:40:04.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001176774 2024-04-22T18:40:04.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855225+0000", 2024-04-22T18:40:04.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0697000000000003e-05 2024-04-22T18:40:04.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855353+0000", 2024-04-22T18:40:04.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012788 2024-04-22T18:40:04.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058003+0000", 2024-04-22T18:40:04.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202649939 2024-04-22T18:40:04.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159184+0000", 2024-04-22T18:40:04.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101181058 2024-04-22T18:40:04.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159238+0000", 2024-04-22T18:40:04.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3967e-05 2024-04-22T18:40:04.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159264+0000", 2024-04-22T18:40:04.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6237999999999998e-05 2024-04-22T18:40:04.287 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.287 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.287 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.287 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:04.287 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:40:04.287 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.388398534999993, 2024-04-22T18:40:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30544294500000002, 2024-04-22T18:40:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 107 2024-04-22T18:40:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:40:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:40:04.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853972+0000", 2024-04-22T18:40:04.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853975+0000", 2024-04-22T18:40:04.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0939999999999999e-06 2024-04-22T18:40:04.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853978+0000", 2024-04-22T18:40:04.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:40:04.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853981+0000", 2024-04-22T18:40:04.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.858e-06 2024-04-22T18:40:04.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855831+0000", 2024-04-22T18:40:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018503689999999999 2024-04-22T18:40:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855932+0000", 2024-04-22T18:40:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010071 2024-04-22T18:40:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856029+0000", 2024-04-22T18:40:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6751999999999998e-05 2024-04-22T18:40:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058013+0000", 2024-04-22T18:40:04.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20198412600000001 2024-04-22T18:40:04.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159326+0000", 2024-04-22T18:40:04.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101312969 2024-04-22T18:40:04.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159376+0000", 2024-04-22T18:40:04.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9968e-05 2024-04-22T18:40:04.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159416+0000", 2024-04-22T18:40:04.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9641999999999998e-05 2024-04-22T18:40:04.293 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.293 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.293 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.293 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:04.293 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:04.293 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.388310138999998, 2024-04-22T18:40:04.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30548257499999998, 2024-04-22T18:40:04.293 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.293 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.293 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.293 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.293 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.294 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 108 2024-04-22T18:40:04.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.294 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:04.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:04.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:04.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854064+0000", 2024-04-22T18:40:04.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5820000000000002e-06 2024-04-22T18:40:04.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854080+0000", 2024-04-22T18:40:04.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5379999999999998e-05 2024-04-22T18:40:04.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854084+0000", 2024-04-22T18:40:04.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.963e-06 2024-04-22T18:40:04.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856413+0000", 2024-04-22T18:40:04.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023298070000000001 2024-04-22T18:40:04.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856494+0000", 2024-04-22T18:40:04.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0863000000000002e-05 2024-04-22T18:40:04.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856624+0000", 2024-04-22T18:40:04.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129799 2024-04-22T18:40:04.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058947+0000", 2024-04-22T18:40:04.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20232328899999999 2024-04-22T18:40:04.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159468+0000", 2024-04-22T18:40:04.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100520597 2024-04-22T18:40:04.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159508+0000", 2024-04-22T18:40:04.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9830000000000003e-05 2024-04-22T18:40:04.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159544+0000", 2024-04-22T18:40:04.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5944e-05 2024-04-22T18:40:04.298 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.299 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.299 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:04.299 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:40:04.299 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.388265646999997, 2024-04-22T18:40:04.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30556502499999999, 2024-04-22T18:40:04.299 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.299 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.299 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.299 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.299 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.299 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:40:04.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.300 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:40:04.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:40:04.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854105+0000", 2024-04-22T18:40:04.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854175+0000", 2024-04-22T18:40:04.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0178000000000005e-05 2024-04-22T18:40:04.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854191+0000", 2024-04-22T18:40:04.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5965999999999999e-05 2024-04-22T18:40:04.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854196+0000", 2024-04-22T18:40:04.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.42e-06 2024-04-22T18:40:04.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956526+0000", 2024-04-22T18:40:04.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10233094299999999 2024-04-22T18:40:04.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956634+0000", 2024-04-22T18:40:04.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107398 2024-04-22T18:40:04.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956730+0000", 2024-04-22T18:40:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6217000000000004e-05 2024-04-22T18:40:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058955+0000", 2024-04-22T18:40:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102224749 2024-04-22T18:40:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159596+0000", 2024-04-22T18:40:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100641297 2024-04-22T18:40:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159637+0000", 2024-04-22T18:40:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874000000000002e-05 2024-04-22T18:40:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159671+0000", 2024-04-22T18:40:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3603999999999999e-05 2024-04-22T18:40:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:04.304 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:04.305 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.388171799999995, 2024-04-22T18:40:04.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30683048800000001, 2024-04-22T18:40:04.305 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.305 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.305 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.305 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.305 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.305 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110 2024-04-22T18:40:04.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.305 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:04.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:04.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:04.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854202+0000", 2024-04-22T18:40:04.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:40:04.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854206+0000", 2024-04-22T18:40:04.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5870000000000001e-06 2024-04-22T18:40:04.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854208+0000", 2024-04-22T18:40:04.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7199999999999998e-06 2024-04-22T18:40:04.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057285+0000", 2024-04-22T18:40:04.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20307620800000001 2024-04-22T18:40:04.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057414+0000", 2024-04-22T18:40:04.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012896899999999999 2024-04-22T18:40:04.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057588+0000", 2024-04-22T18:40:04.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000174802 2024-04-22T18:40:04.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159429+0000", 2024-04-22T18:40:04.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101840783 2024-04-22T18:40:04.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160947+0000", 2024-04-22T18:40:04.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001518152 2024-04-22T18:40:04.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160992+0000", 2024-04-22T18:40:04.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4725000000000003e-05 2024-04-22T18:40:04.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161030+0000", 2024-04-22T18:40:04.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7789e-05 2024-04-22T18:40:04.310 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.310 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.310 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.310 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:04.310 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:04.310 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.388160619000004, 2024-04-22T18:40:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:40:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:40:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:04.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:40:04.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:40:04.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:40:04.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:40:04.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:40:04.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:40:04.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:40:04.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:40:04.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:40:04.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:40:04.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:40:04.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:40:04.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:40:04.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:40:04.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:40:04.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:40:04.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:40:04.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:40:04.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:40:04.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:40:04.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:40:04.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.316 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.316 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:04.316 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:04.316 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.388144960999995, 2024-04-22T18:40:04.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:40:04.316 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.316 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.316 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.316 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.316 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.316 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:40:04.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.317 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:04.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:04.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:04.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:40:04.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:04.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:40:04.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:40:04.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:40:04.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:40:04.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:40:04.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:40:04.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:40:04.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:40:04.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:40:04.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:40:04.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:40:04.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:40:04.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:40:04.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:40:04.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:40:04.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:40:04.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:40:04.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:40:04.321 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.321 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.321 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.321 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:04.321 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:04.321 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.388093799000004, 2024-04-22T18:40:04.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:40:04.322 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.322 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.322 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.322 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.322 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.322 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:40:04.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.322 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:04.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:04.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:04.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:40:04.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:40:04.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:40:04.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:40:04.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:40:04.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:40:04.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:40:04.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:40:04.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:40:04.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:40:04.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:40:04.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:40:04.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:40:04.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:40:04.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:40:04.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:40:04.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:40:04.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:40:04.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.388078574999994, 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.327 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:40:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:40:04.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:40:04.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:40:04.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:40:04.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:40:04.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:40:04.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:40:04.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:40:04.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:40:04.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:40:04.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:40:04.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:40:04.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:40:04.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:40:04.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:40:04.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:40:04.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:40:04.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:40:04.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:40:04.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:40:04.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:40:04.332 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.332 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.332 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.332 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:04.332 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:04.333 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.388018998000007, 2024-04-22T18:40:04.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:40:04.333 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.333 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.333 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.333 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.333 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.333 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:40:04.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.333 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:04.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:04.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:04.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:40:04.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:40:04.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:40:04.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:40:04.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:40:04.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:40:04.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:40:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:40:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:40:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:40:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:40:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:40:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:40:04.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:40:04.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:40:04.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:40:04.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:40:04.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:40:04.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:40:04.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:40:04.338 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.338 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.338 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.338 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:04.338 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:04.338 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.387936682000003, 2024-04-22T18:40:04.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:40:04.338 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.338 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.338 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.339 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:40:04.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.339 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:04.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:04.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:40:04.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:40:04.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:40:04.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:40:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:40:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:40:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:40:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:40:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:40:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:40:04.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:40:04.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:40:04.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:40:04.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:40:04.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:40:04.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:40:04.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:40:04.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:40:04.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:40:04.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:40:04.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:40:04.343 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.343 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.343 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.344 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:04.344 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:04.344 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.387926526000001, 2024-04-22T18:40:04.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:40:04.344 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.344 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.344 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.344 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.344 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.344 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:40:04.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.344 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:04.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:04.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:40:04.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:40:04.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:40:04.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:40:04.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:40:04.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:40:04.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:40:04.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:161 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 75.787432234999997, 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40239169000000002, 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:04.349 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:04.350 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 161 2024-04-22T18:40:04.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.350 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:04.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:04.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:40:04.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:04.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:40:04.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:04.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:04.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454938+0000", 2024-04-22T18:40:04.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:04.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454982+0000", 2024-04-22T18:40:04.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3470999999999998e-05 2024-04-22T18:40:04.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:04.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454988+0000", 2024-04-22T18:40:04.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7640000000000002e-06 2024-04-22T18:40:04.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555054+0000", 2024-04-22T18:40:04.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006607300000001 2024-04-22T18:40:04.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:04.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555607+0000", 2024-04-22T18:40:04.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055319500000000003 2024-04-22T18:40:04.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:04.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555689+0000", 2024-04-22T18:40:04.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1817999999999995e-05 2024-04-22T18:40:04.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:04.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555804+0000", 2024-04-22T18:40:04.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115346 2024-04-22T18:40:04.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:04.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556554+0000", 2024-04-22T18:40:04.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000750102 2024-04-22T18:40:04.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857265+0000", 2024-04-22T18:40:04.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30071122700000003 2024-04-22T18:40:04.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:04.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857302+0000", 2024-04-22T18:40:04.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7069e-05 2024-04-22T18:40:04.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:04.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:04.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:04.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857331+0000", 2024-04-22T18:40:04.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8291999999999999e-05 2024-04-22T18:40:04.354 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.354 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.354 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.354 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:04.354 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:04.354 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:04.356 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:04.356+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:04.357 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:04.358 INFO:tasks.rados.rados.0.smithi089.stdout:1251: finishing write tid 4 to smithi08937929-43 2024-04-22T18:40:04.358 INFO:tasks.rados.rados.0.smithi089.stdout:1251: finishing write tid 5 to smithi08937929-43 2024-04-22T18:40:04.358 INFO:tasks.rados.rados.0.smithi089.stdout:1251: oid 43 updating version 0 to 48 2024-04-22T18:40:04.358 INFO:tasks.rados.rados.0.smithi089.stdout:1251: oid 43 updating version 48 to 50 2024-04-22T18:40:04.358 INFO:tasks.rados.rados.0.smithi089.stdout:1251: oid 43 updating version 50 to 51 2024-04-22T18:40:04.358 INFO:tasks.rados.rados.0.smithi089.stdout:1251: oid 43 version 51 is already newer than 49 2024-04-22T18:40:04.358 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 51 (ObjNum 226 snap 0 seq_num 226) dirty exists 2024-04-22T18:40:04.358 INFO:tasks.rados.rados.0.smithi089.stdout:1251: left oid 43 (ObjNum 226 snap 0 seq_num 226) 2024-04-22T18:40:04.358 INFO:tasks.rados.rados.0.smithi089.stdout:1252: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-04-22T18:40:04.358 INFO:tasks.rados.rados.0.smithi089.stdout:1252: writing smithi08937929-40 from 615189 to 1070949 tid 1 2024-04-22T18:40:04.358 INFO:tasks.rados.rados.0.smithi089.stdout:1252: writing smithi08937929-40 from 1733883 to 2444961 tid 2 2024-04-22T18:40:04.358 INFO:tasks.rados.rados.0.smithi089.stdout:1252: writing smithi08937929-40 from 3134431 to 3594472 tid 3 2024-04-22T18:40:04.359 INFO:tasks.rados.rados.0.smithi089.stdout:1231: done (11 left) 2024-04-22T18:40:04.359 INFO:tasks.rados.rados.0.smithi089.stdout:1243: done (10 left) 2024-04-22T18:40:04.359 INFO:tasks.rados.rados.0.smithi089.stdout:1244: done (9 left) 2024-04-22T18:40:04.359 INFO:tasks.rados.rados.0.smithi089.stdout:1245: done (8 left) 2024-04-22T18:40:04.359 INFO:tasks.rados.rados.0.smithi089.stdout:1246: done (7 left) 2024-04-22T18:40:04.359 INFO:tasks.rados.rados.0.smithi089.stdout:1247: done (6 left) 2024-04-22T18:40:04.359 INFO:tasks.rados.rados.0.smithi089.stdout:1248: done (5 left) 2024-04-22T18:40:04.359 INFO:tasks.rados.rados.0.smithi089.stdout:1249: done (4 left) 2024-04-22T18:40:04.359 INFO:tasks.rados.rados.0.smithi089.stdout:1250: done (3 left) 2024-04-22T18:40:04.359 INFO:tasks.rados.rados.0.smithi089.stdout:1251: done (2 left) 2024-04-22T18:40:04.359 INFO:tasks.rados.rados.0.smithi089.stdout:1253: read oid 4 snap -1 2024-04-22T18:40:04.359 INFO:tasks.rados.rados.0.smithi089.stdout:1253: expect (ObjNum 211 snap 0 seq_num 211) 2024-04-22T18:40:04.359 INFO:tasks.rados.rados.0.smithi089.stdout:1254: read oid 33 snap -1 2024-04-22T18:40:04.359 INFO:tasks.rados.rados.0.smithi089.stdout:1254: expect (ObjNum 216 snap 0 seq_num 216) 2024-04-22T18:40:04.359 INFO:tasks.rados.rados.0.smithi089.stdout:1255: set_chunk oid 2 target oid 2024-04-22T18:40:04.421 INFO:tasks.rados.rados.0.smithi089.stdout:1255: set_chunk oid 2 offset: 205824 length: 1811968 target oid 0ae01291051f56c7d1a3b0e184abc7e68e2ff6ca15d638fa18ec451ba856e186 offset: 0 snap -1 2024-04-22T18:40:04.421 INFO:tasks.rados.rados.0.smithi089.stdout:1256: read oid 30 snap -1 2024-04-22T18:40:04.421 INFO:tasks.rados.rados.0.smithi089.stdout:1256: expect (ObjNum 203 snap 0 seq_num 203) 2024-04-22T18:40:04.421 INFO:tasks.rados.rados.0.smithi089.stdout:1257: read oid 50 snap -1 2024-04-22T18:40:04.421 INFO:tasks.rados.rados.0.smithi089.stdout:1257: expect (ObjNum 189 snap 0 seq_num 189) 2024-04-22T18:40:04.421 INFO:tasks.rados.rados.0.smithi089.stdout:1258: read oid 7 snap -1 2024-04-22T18:40:04.421 INFO:tasks.rados.rados.0.smithi089.stdout:1258: expect (ObjNum 215 snap 0 seq_num 215) 2024-04-22T18:40:04.421 INFO:tasks.rados.rados.0.smithi089.stdout:1259: read oid 21 snap -1 2024-04-22T18:40:04.421 INFO:tasks.rados.rados.0.smithi089.stdout:1259: expect deleted 2024-04-22T18:40:04.421 INFO:tasks.rados.rados.0.smithi089.stdout:1260: write (excl) oid 42 current snap is 0 2024-04-22T18:40:04.421 INFO:tasks.rados.rados.0.smithi089.stdout:1252: finishing write tid 1 to smithi08937929-40 2024-04-22T18:40:04.422 INFO:tasks.rados.rados.0.smithi089.stdout:1260: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2024-04-22T18:40:04.427 INFO:tasks.rados.rados.0.smithi089.stdout:1260: writing smithi08937929-42 from 631996 to 1163005 tid 1 2024-04-22T18:40:04.433 INFO:tasks.rados.rados.0.smithi089.stdout:1260: writing smithi08937929-42 from 1792365 to 2247101 tid 2 2024-04-22T18:40:04.439 INFO:tasks.rados.rados.0.smithi089.stdout:1260: writing smithi08937929-42 from 2761854 to 3433167 tid 3 2024-04-22T18:40:04.439 INFO:tasks.rados.rados.0.smithi089.stdout:1261: set_chunk oid 5 target oid 2024-04-22T18:40:04.456 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:04.458+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:04.469 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:04.469 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:04.469 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:04.470 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:04.478 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:04.557 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:04.558+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:04.621 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:04.622 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:04.630 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:04.640 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:40:04.658 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:04.659+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:04.699 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:40:04.737 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:04.737 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:04.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:04.738 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:04.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.738 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:04.738 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:04.738 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 70.282151838999994, 2024-04-22T18:40:04.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:40:04.738 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.738 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.738 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:04.738 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:04.738 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:04.738 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:40:04.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.739 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:04.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:04.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:40:04.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:04.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:40:04.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:40:04.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:40:04.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:40:04.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:40:04.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:40:04.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:40:04.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:40:04.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:40:04.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:40:04.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:04.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:40:04.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:40:04.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:04.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:40:04.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:40:04.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:40:04.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:40:04.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:04.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:40:04.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:40:04.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:40:04.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:40:04.743 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.743 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.743 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.744 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:315 4.b 4:d923883e:::smithi08937929-16:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:04.744 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:40:04.744 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 66.745689059, 2024-04-22T18:40:04.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40309447300000001, 2024-04-22T18:40:04.744 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.744 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.744 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:04.744 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:04.744 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:04.744 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 315 2024-04-22T18:40:04.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.745 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:40:04.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:40:04.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991817+0000", 2024-04-22T18:40:04.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:04.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991825+0000", 2024-04-22T18:40:04.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7279999999999995e-06 2024-04-22T18:40:04.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991834+0000", 2024-04-22T18:40:04.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1980000000000007e-06 2024-04-22T18:40:04.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991841+0000", 2024-04-22T18:40:04.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0740000000000004e-06 2024-04-22T18:40:04.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991881+0000", 2024-04-22T18:40:04.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-04-22T18:40:04.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991940+0000", 2024-04-22T18:40:04.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9382000000000002e-05 2024-04-22T18:40:04.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:04.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394250+0000", 2024-04-22T18:40:04.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40230947700000003 2024-04-22T18:40:04.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:04.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394752+0000", 2024-04-22T18:40:04.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050269699999999997 2024-04-22T18:40:04.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394865+0000", 2024-04-22T18:40:04.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011267700000000001 2024-04-22T18:40:04.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:04.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394892+0000", 2024-04-22T18:40:04.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7535999999999999e-05 2024-04-22T18:40:04.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394914+0000", 2024-04-22T18:40:04.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1398000000000001e-05 2024-04-22T18:40:04.749 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.750 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.750 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.750 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:774 4.b 4:defe92fb:::smithi08937929-44:head [write 2778819~95145] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:04.750 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:40:04.750 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.589179383000001, 2024-04-22T18:40:04.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40475725000000001, 2024-04-22T18:40:04.750 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.750 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.750 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:04.750 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:04.751 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:04.751 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 774 2024-04-22T18:40:04.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.751 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:40:04.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:40:04.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148328+0000", 2024-04-22T18:40:04.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:04.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148392+0000", 2024-04-22T18:40:04.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3323000000000003e-05 2024-04-22T18:40:04.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148398+0000", 2024-04-22T18:40:04.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4420000000000003e-06 2024-04-22T18:40:04.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148409+0000", 2024-04-22T18:40:04.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509999999999999e-05 2024-04-22T18:40:04.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251407+0000", 2024-04-22T18:40:04.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102998305 2024-04-22T18:40:04.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251500+0000", 2024-04-22T18:40:04.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3607e-05 2024-04-22T18:40:04.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:04.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251648+0000", 2024-04-22T18:40:04.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014788099999999999 2024-04-22T18:40:04.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:04.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256149+0000", 2024-04-22T18:40:04.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045010980000000003 2024-04-22T18:40:04.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553016+0000", 2024-04-22T18:40:04.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29686620800000002 2024-04-22T18:40:04.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:04.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553054+0000", 2024-04-22T18:40:04.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8414999999999997e-05 2024-04-22T18:40:04.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553086+0000", 2024-04-22T18:40:04.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2236000000000001e-05 2024-04-22T18:40:04.756 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.756 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.756 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.756 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:04.756 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:04.756 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.589094590999999, 2024-04-22T18:40:04.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:40:04.756 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.756 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.756 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:04.756 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:04.756 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:04.757 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:40:04.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.757 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:04.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:04.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:40:04.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:40:04.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:40:04.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:40:04.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:40:04.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:40:04.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:40:04.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:40:04.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:40:04.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:40:04.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:40:04.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:04.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:40:04.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:40:04.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:40:04.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:40:04.761 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.761 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.761 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.762 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:04.762 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:04.762 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.589076114999997, 2024-04-22T18:40:04.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:40:04.762 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.762 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:04.762 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:04.762 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:04.762 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:04.762 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:40:04.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.762 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:04.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:04.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:04.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:40:04.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:40:04.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:40:04.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:40:04.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:40:04.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:40:04.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:40:04.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:40:04.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:04.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:40:04.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:40:04.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:40:04.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:40:04.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:40:04.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:40:04.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:40:04.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:40:04.766 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.766 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1242 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 47.995830566000002, 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40662891899999998, 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1242 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:40:04.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:40:04.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741677+0000", 2024-04-22T18:40:04.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741680+0000", 2024-04-22T18:40:04.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8059999999999999e-06 2024-04-22T18:40:04.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741685+0000", 2024-04-22T18:40:04.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0780000000000003e-06 2024-04-22T18:40:04.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741690+0000", 2024-04-22T18:40:04.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4020000000000004e-06 2024-04-22T18:40:04.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741732+0000", 2024-04-22T18:40:04.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2110000000000002e-05 2024-04-22T18:40:04.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741798+0000", 2024-04-22T18:40:04.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6432000000000004e-05 2024-04-22T18:40:04.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.148307+0000", 2024-04-22T18:40:04.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40650862100000001 2024-04-22T18:40:04.771 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.771 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.771 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.771 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1408 4.6 4:630bd8e5:::smithi08937929-45:head [write 2816389~693495] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:40:04.771 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:40:04.771 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.575860759000001, 2024-04-22T18:40:04.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408959452, 2024-04-22T18:40:04.771 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.771 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.772 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:04.772 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:04.772 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:04.772 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1408 2024-04-22T18:40:04.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.772 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:40:04.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:40:04.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161647+0000", 2024-04-22T18:40:04.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161951+0000", 2024-04-22T18:40:04.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030389799999999998 2024-04-22T18:40:04.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161955+0000", 2024-04-22T18:40:04.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1989999999999999e-06 2024-04-22T18:40:04.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161958+0000", 2024-04-22T18:40:04.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2260000000000001e-06 2024-04-22T18:40:04.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.265867+0000", 2024-04-22T18:40:04.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103908927 2024-04-22T18:40:04.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.265960+0000", 2024-04-22T18:40:04.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3016999999999994e-05 2024-04-22T18:40:04.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:04.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.266124+0000", 2024-04-22T18:40:04.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016347700000000001 2024-04-22T18:40:04.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570491+0000", 2024-04-22T18:40:04.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30436694199999997 2024-04-22T18:40:04.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:04.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570502+0000", 2024-04-22T18:40:04.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1107999999999999e-05 2024-04-22T18:40:04.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:04.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570556+0000", 2024-04-22T18:40:04.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4388000000000002e-05 2024-04-22T18:40:04.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570607+0000", 2024-04-22T18:40:04.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0782999999999998e-05 2024-04-22T18:40:04.777 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.777 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.777 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.777 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1409 4.6 4:630bd8e5:::smithi08937929-45:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:40:04.777 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:04.778 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.575547055000001, 2024-04-22T18:40:04.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40880186200000002, 2024-04-22T18:40:04.778 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.778 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.778 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:04.778 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:04.778 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:04.778 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1409 2024-04-22T18:40:04.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.778 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:04.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:04.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:04.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161964+0000", 2024-04-22T18:40:04.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5909999999999998e-06 2024-04-22T18:40:04.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161966+0000", 2024-04-22T18:40:04.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5979999999999999e-06 2024-04-22T18:40:04.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161969+0000", 2024-04-22T18:40:04.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7049999999999999e-06 2024-04-22T18:40:04.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269205+0000", 2024-04-22T18:40:04.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107236228 2024-04-22T18:40:04.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269320+0000", 2024-04-22T18:40:04.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114571 2024-04-22T18:40:04.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:04.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269476+0000", 2024-04-22T18:40:04.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155928 2024-04-22T18:40:04.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:04.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570619+0000", 2024-04-22T18:40:04.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30114322700000001 2024-04-22T18:40:04.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570672+0000", 2024-04-22T18:40:04.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3232000000000001e-05 2024-04-22T18:40:04.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:04.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570724+0000", 2024-04-22T18:40:04.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2258e-05 2024-04-22T18:40:04.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570763+0000", 2024-04-22T18:40:04.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8927999999999997e-05 2024-04-22T18:40:04.783 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.784 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.784 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.784 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1410 4.6 4:630bd8e5:::smithi08937929-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:40:04.784 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:04.784 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.575537537999999, 2024-04-22T18:40:04.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40900504500000001, 2024-04-22T18:40:04.784 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.784 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:04.784 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:04.784 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:04.784 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:04.784 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1410 2024-04-22T18:40:04.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.785 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:04.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:04.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:04.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161972+0000", 2024-04-22T18:40:04.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6419999999999999e-06 2024-04-22T18:40:04.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161976+0000", 2024-04-22T18:40:04.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9260000000000002e-06 2024-04-22T18:40:04.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161979+0000", 2024-04-22T18:40:04.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9359999999999999e-06 2024-04-22T18:40:04.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570790+0000", 2024-04-22T18:40:04.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40881099199999998 2024-04-22T18:40:04.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570879+0000", 2024-04-22T18:40:04.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8895999999999997e-05 2024-04-22T18:40:04.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570976+0000", 2024-04-22T18:40:04.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6923999999999996e-05 2024-04-22T18:40:04.788 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.788 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.788 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.788 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:04.788 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:04.789 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.826027803000002, 2024-04-22T18:40:04.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:40:04.789 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.789 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.789 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:04.789 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:04.789 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:04.789 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:40:04.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.789 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:04.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:04.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:40:04.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:04.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:40:04.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:40:04.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:40:04.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:40:04.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:40:04.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:40:04.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:40:04.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:40:04.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:40:04.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:40:04.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:04.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:40:04.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:40:04.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:40:04.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:40:04.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:04.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:40:04.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:40:04.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:04.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:40:04.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:40:04.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:40:04.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:40:04.794 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.794 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.794 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.794 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:04.795 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:04.795 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.296868477, 2024-04-22T18:40:04.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:40:04.795 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.795 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:04.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:04.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:04.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:04.795 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:40:04.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.795 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:40:04.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:40:04.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:40:04.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:40:04.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:40:04.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:40:04.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:40:04.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:40:04.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:04.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:40:04.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:40:04.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:40:04.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:40:04.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:40:04.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:40:04.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:40:04.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:40:04.799 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.799 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.799 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.799 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:04.800 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:04.800 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.119161082000002, 2024-04-22T18:40:04.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:40:04.800 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.800 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.800 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:04.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:04.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:40:04.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:04.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:40:04.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:40:04.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:40:04.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:40:04.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:40:04.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:40:04.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:40:04.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:40:04.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:40:04.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:40:04.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:04.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:40:04.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:40:04.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:04.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:40:04.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:40:04.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:40:04.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:40:04.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:04.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:40:04.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:40:04.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:40:04.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:40:04.805 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.805 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.805 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.805 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:04.805 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:04.805 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.119092810000001, 2024-04-22T18:40:04.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:40:04.805 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.805 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.805 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:04.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:04.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:40:04.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:40:04.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:40:04.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:40:04.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:40:04.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:40:04.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:40:04.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:40:04.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:40:04.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:40:04.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:40:04.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:04.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:40:04.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:40:04.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:04.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:40:04.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:40:04.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:40:04.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:40:04.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:04.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:40:04.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:40:04.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:40:04.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:04.810 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.810 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.811 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.811 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:04.811 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:04.811 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.118924968999998, 2024-04-22T18:40:04.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:40:04.811 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.811 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.811 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:04.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:04.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:04.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:40:04.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:40:04.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:40:04.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:40:04.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:40:04.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:40:04.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:40:04.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:40:04.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:40:04.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:40:04.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:04.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:40:04.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:40:04.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:04.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:40:04.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:40:04.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:40:04.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:40:04.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:04.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:40:04.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:40:04.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:40:04.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:40:04.816 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.816 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.816 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.817 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:04.817 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:04.817 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.153599494000002, 2024-04-22T18:40:04.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:40:04.817 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.817 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.817 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:04.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:04.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:40:04.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:04.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:40:04.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:40:04.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:40:04.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:40:04.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:40:04.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:40:04.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:40:04.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:40:04.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:40:04.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:40:04.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:04.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:40:04.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:40:04.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:04.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:40:04.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:40:04.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:40:04.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:40:04.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:04.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:40:04.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:40:04.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:40:04.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:40:04.822 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.822 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.822 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.822 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:04.822 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:04.822 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.053447306999999, 2024-04-22T18:40:04.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:40:04.823 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.823 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.823 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:04.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:04.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:40:04.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:04.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:40:04.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:40:04.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:40:04.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:40:04.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:40:04.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:40:04.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:40:04.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:40:04.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:40:04.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:40:04.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:04.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:40:04.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:40:04.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:04.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:40:04.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:40:04.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:40:04.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:40:04.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:04.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:40:04.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:40:04.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:40:04.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:40:04.827 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.827 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.828 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.828 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2385 4.13 4:c93c842e:::smithi08937929-25:head [write 1937856~527061] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:40:04.828 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.287418+0000", 2024-04-22T18:40:04.828 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.450090249, 2024-04-22T18:40:04.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39986265300000001, 2024-04-22T18:40:04.828 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.828 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.828 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:04.828 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:04.828 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:04.828 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2385 2024-04-22T18:40:04.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.829 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287418+0000", 2024-04-22T18:40:04.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287418+0000", 2024-04-22T18:40:04.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287417+0000", 2024-04-22T18:40:04.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:04.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287636+0000", 2024-04-22T18:40:04.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000219089 2024-04-22T18:40:04.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287654+0000", 2024-04-22T18:40:04.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8294000000000001e-05 2024-04-22T18:40:04.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287661+0000", 2024-04-22T18:40:04.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7780000000000002e-06 2024-04-22T18:40:04.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.483858+0000", 2024-04-22T18:40:04.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19619666199999999 2024-04-22T18:40:04.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.483917+0000", 2024-04-22T18:40:04.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9454999999999999e-05 2024-04-22T18:40:04.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:04.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.483999+0000", 2024-04-22T18:40:04.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1359999999999994e-05 2024-04-22T18:40:04.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:04.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586037+0000", 2024-04-22T18:40:04.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102038217 2024-04-22T18:40:04.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687214+0000", 2024-04-22T18:40:04.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101177195 2024-04-22T18:40:04.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:04.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687253+0000", 2024-04-22T18:40:04.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9409999999999997e-05 2024-04-22T18:40:04.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687281+0000", 2024-04-22T18:40:04.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7478e-05 2024-04-22T18:40:04.833 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.833 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.833 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.834 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2386 4.13 4:c93c842e:::smithi08937929-25:head [write 3252761~175867] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:40:04.834 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.292661+0000", 2024-04-22T18:40:04.834 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.444847600999999, 2024-04-22T18:40:04.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39468563099999998, 2024-04-22T18:40:04.834 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.834 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:04.834 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:04.834 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:04.834 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:04.834 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2386 2024-04-22T18:40:04.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.834 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.292661+0000", 2024-04-22T18:40:04.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.292661+0000", 2024-04-22T18:40:04.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.292659+0000", 2024-04-22T18:40:04.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:04.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.292788+0000", 2024-04-22T18:40:04.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012955 2024-04-22T18:40:04.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.292806+0000", 2024-04-22T18:40:04.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7357999999999999e-05 2024-04-22T18:40:04.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392900+0000", 2024-04-22T18:40:04.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100093799 2024-04-22T18:40:04.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.485125+0000", 2024-04-22T18:40:04.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.092225336000000005 2024-04-22T18:40:04.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.485172+0000", 2024-04-22T18:40:04.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7216000000000003e-05 2024-04-22T18:40:04.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:04.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.485244+0000", 2024-04-22T18:40:04.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1888e-05 2024-04-22T18:40:04.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:04.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.587565+0000", 2024-04-22T18:40:04.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102321 2024-04-22T18:40:04.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:04.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687303+0000", 2024-04-22T18:40:04.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099737563000000001 2024-04-22T18:40:04.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:04.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687324+0000", 2024-04-22T18:40:04.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0868000000000001e-05 2024-04-22T18:40:04.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687346+0000", 2024-04-22T18:40:04.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2915999999999999e-05 2024-04-22T18:40:04.839 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.839 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.839 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.839 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:40:04.839 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:04.839 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.344582787, 2024-04-22T18:40:04.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:40:04.840 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.840 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:04.840 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:04.840 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:04.840 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:04.840 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:40:04.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.840 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:04.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:04.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:40:04.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:40:04.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:40:04.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:40:04.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:40:04.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:40:04.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:40:04.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:40:04.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:40:04.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:04.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:40:04.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:40:04.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:40:04.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:40:04.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:40:04.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:40:04.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:40:04.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:40:04.844 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.844 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.844 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.844 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2697 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~2198635] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:04.844 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.162267+0000", 2024-04-22T18:40:04.844 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.5752416, 2024-04-22T18:40:04.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39410373599999998, 2024-04-22T18:40:04.845 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:04.845 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:04.845 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:04.845 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:04.845 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:04.845 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2697 2024-04-22T18:40:04.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.845 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:04.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:04.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.162267+0000", 2024-04-22T18:40:04.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:04.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.162267+0000", 2024-04-22T18:40:04.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:04.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:04.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.162266+0000", 2024-04-22T18:40:04.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:04.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:04.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.162268+0000", 2024-04-22T18:40:04.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.829e-06 2024-04-22T18:40:04.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:04.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.162272+0000", 2024-04-22T18:40:04.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3859999999999999e-06 2024-04-22T18:40:04.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:04.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.162274+0000", 2024-04-22T18:40:04.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7439999999999999e-06 2024-04-22T18:40:04.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:04.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.556026+0000", 2024-04-22T18:40:04.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39375122299999998 2024-04-22T18:40:04.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:04.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.556094+0000", 2024-04-22T18:40:04.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8283999999999999e-05 2024-04-22T18:40:04.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:04.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:04.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:04.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.556371+0000", 2024-04-22T18:40:04.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027660299999999999 2024-04-22T18:40:04.848 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.848 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.848 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.849 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:04.849 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:04.849 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:04.850 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:04.851 INFO:tasks.rados.rados.0.smithi089.stdout:1261: set_chunk oid 5 offset: 2152448 length: 88064 target oid 53386adc8b27d3e945fe858cb0b6ad947143e08521c2af8a61a6f85eaf67b4b5 offset: 0 snap -1 2024-04-22T18:40:04.852 INFO:tasks.rados.rados.0.smithi089.stdout:1262: tier_evict oid 22 2024-04-22T18:40:04.852 INFO:tasks.rados.rados.0.smithi089.stdout:1262: tier_evict oid 22 snap -1 2024-04-22T18:40:04.852 INFO:tasks.rados.rados.0.smithi089.stdout:1252: finishing write tid 2 to smithi08937929-40 2024-04-22T18:40:04.852 INFO:tasks.rados.rados.0.smithi089.stdout:1252: finishing write tid 3 to smithi08937929-40 2024-04-22T18:40:04.852 INFO:tasks.rados.rados.0.smithi089.stdout:1263: read oid 43 snap -1 2024-04-22T18:40:04.852 INFO:tasks.rados.rados.0.smithi089.stdout:1263: expect (ObjNum 226 snap 0 seq_num 226) 2024-04-22T18:40:04.852 INFO:tasks.rados.rados.0.smithi089.stdout:1264: write (excl) oid 39 current snap is 0 2024-04-22T18:40:04.852 INFO:tasks.rados.rados.0.smithi089.stdout:1253: expect (ObjNum 211 snap 0 seq_num 211) 2024-04-22T18:40:04.852 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:04.853 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:04.853 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:04.851+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:04.862 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:04.863 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:04.880 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:04.881 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:04.897 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:04.898 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:04.914 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:04.915 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:04.931 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:04.931 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:04.947 INFO:tasks.rados.rados.0.smithi089.stdout:1252: finishing write tid 4 to smithi08937929-40 2024-04-22T18:40:04.947 INFO:tasks.rados.rados.0.smithi089.stdout:1255: finishing set_chunk to oid 2 2024-04-22T18:40:04.947 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk returned error code -95 offset: 205824 length: 1811968 2024-04-22T18:40:04.948 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 214 snap 0 seq_num 214) dirty exists 2024-04-22T18:40:04.948 INFO:tasks.rados.rados.0.smithi089.stdout:1264: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-04-22T18:40:04.948 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:04.948 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:04.950 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:04.952+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:04.955 INFO:tasks.rados.rados.0.smithi089.stdout:1264: writing smithi08937929-39 from 648803 to 1371414 tid 1 2024-04-22T18:40:04.962 INFO:tasks.rados.rados.0.smithi089.stdout:1264: writing smithi08937929-39 from 1967200 to 2681947 tid 2 2024-04-22T18:40:04.966 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:04.967 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:04.968 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:04.968 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:04.969 INFO:tasks.rados.rados.0.smithi089.stdout:1264: writing smithi08937929-39 from 3421983 to 3848803 tid 3 2024-04-22T18:40:04.969 INFO:tasks.rados.rados.0.smithi089.stdout:1253: done (13 left) 2024-04-22T18:40:04.969 INFO:tasks.rados.rados.0.smithi089.stdout:1255: done (12 left) 2024-04-22T18:40:04.969 INFO:tasks.rados.rados.0.smithi089.stdout:1259: done (11 left) 2024-04-22T18:40:04.969 INFO:tasks.rados.rados.0.smithi089.stdout:1265: set_chunk oid 29 target oid 2024-04-22T18:40:04.979 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:05.063 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:05.064+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:05.123 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:05.123 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:05.132 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:05.173 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:05.174+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:05.178 INFO:tasks.rados.rados.0.smithi089.stdout:1265: set_chunk oid 29 offset: 22528 length: 2162176 target oid fddd16c9954119b334d154936ecb29d739e0d6cf3f38a320d100111a4ddc77d5 offset: 0 snap -1 2024-04-22T18:40:05.178 INFO:tasks.rados.rados.0.smithi089.stdout:1266: write oid 38 current snap is 0 2024-04-22T18:40:05.178 INFO:tasks.rados.rados.0.smithi089.stdout:1266: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2024-04-22T18:40:05.186 INFO:tasks.rados.rados.0.smithi089.stdout:1266: writing smithi08937929-38 from 665610 to 1463470 tid 1 2024-04-22T18:40:05.191 INFO:tasks.rados.rados.0.smithi089.stdout:1266: writing smithi08937929-38 from 1909329 to 2367734 tid 2 2024-04-22T18:40:05.196 INFO:tasks.rados.rados.0.smithi089.stdout:1266: writing smithi08937929-38 from 2816700 to 3226910 tid 3 2024-04-22T18:40:05.196 INFO:tasks.rados.rados.0.smithi089.stdout:1267: read oid 21 snap -1 2024-04-22T18:40:05.196 INFO:tasks.rados.rados.0.smithi089.stdout:1267: expect deleted 2024-04-22T18:40:05.196 INFO:tasks.rados.rados.0.smithi089.stdout:1268: set_chunk oid 45 target oid 2024-04-22T18:40:05.253 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:05.254+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:05.274 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:05.275 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:05.284 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:05.354 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:05.356+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:05.369 INFO:tasks.rados.rados.0.smithi089.stdout:1268: set_chunk oid 45 offset: 1423360 length: 189440 target oid 40b3874f17e3b043a354e1ae1e875047fa47972bfcea195b70fc992a4f14dd53 offset: 0 snap -1 2024-04-22T18:40:05.369 INFO:tasks.rados.rados.0.smithi089.stdout:1269: set_chunk oid 19 target oid 2024-04-22T18:40:05.427 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:05.427 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:05.437 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:05.481 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:05.482+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:05.538 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:05.538 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:05.546 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:05.555 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:05.556+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:05.577 INFO:tasks.rados.rados.0.smithi089.stdout:1269: set_chunk oid 19 offset: 153088 length: 1296896 target oid 51b571a3e0c802e79beeab895f0420989d17a49c94d78ddd06c97f1d19aeee13 offset: 0 snap -1 2024-04-22T18:40:05.578 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:05.578 INFO:tasks.rados.rados.0.smithi089.stdout:1256: expect (ObjNum 203 snap 0 seq_num 203) 2024-04-22T18:40:05.655 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:05.656+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:05.681 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:05.682 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:05.690 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:05.757 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:05.758+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:05.778 INFO:tasks.rados.rados.0.smithi089.stdout:1260: finishing write tid 1 to smithi08937929-42 2024-04-22T18:40:05.778 INFO:tasks.rados.rados.0.smithi089.stdout:1260: finishing write tid 2 to smithi08937929-42 2024-04-22T18:40:05.778 INFO:tasks.rados.rados.0.smithi089.stdout:1257: expect (ObjNum 189 snap 0 seq_num 189) 2024-04-22T18:40:05.793 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:05.793 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:05.802 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:05.887 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:05.888+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:05.945 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:05.945 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:05.954 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:05.996 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:05.997+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:06.001 INFO:tasks.rados.rados.0.smithi089.stdout:1258: expect (ObjNum 215 snap 0 seq_num 215) 2024-04-22T18:40:06.060 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:06.061+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:06.098 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:06.098 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:06.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:06.160 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:06.161+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:06.210 INFO:tasks.rados.rados.0.smithi089.stdout:1254: expect (ObjNum 216 snap 0 seq_num 216) 2024-04-22T18:40:06.214 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:06.214 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:06.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:06.261 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:06.262+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:06.362 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:06.363+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:06.365 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:06.365 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:06.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:06.406 INFO:tasks.rados.rados.0.smithi089.stdout:1252: finishing write tid 5 to smithi08937929-40 2024-04-22T18:40:06.406 INFO:tasks.rados.rados.0.smithi089.stdout:1252: oid 40 updating version 0 to 54 2024-04-22T18:40:06.406 INFO:tasks.rados.rados.0.smithi089.stdout:1252: oid 40 version 54 is already newer than 53 2024-04-22T18:40:06.406 INFO:tasks.rados.rados.0.smithi089.stdout:1252: oid 40 updating version 54 to 55 2024-04-22T18:40:06.407 INFO:tasks.rados.rados.0.smithi089.stdout:1252: oid 40 version 55 is already newer than 52 2024-04-22T18:40:06.407 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 55 (ObjNum 227 snap 0 seq_num 227) dirty exists 2024-04-22T18:40:06.407 INFO:tasks.rados.rados.0.smithi089.stdout:1252: left oid 40 (ObjNum 227 snap 0 seq_num 227) 2024-04-22T18:40:06.407 INFO:tasks.rados.rados.0.smithi089.stdout:1260: finishing write tid 3 to smithi08937929-42 2024-04-22T18:40:06.407 INFO:tasks.rados.rados.0.smithi089.stdout:1235: got (0) Success 2024-04-22T18:40:06.407 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 0 (ObjNum 208 snap 0 seq_num 208) dirty exists 2024-04-22T18:40:06.407 INFO:tasks.rados.rados.0.smithi089.stdout:1262: got (22) Invalid argument 2024-04-22T18:40:06.407 INFO:tasks.rados.rados.0.smithi089.stdout:1260: finishing write tid 4 to smithi08937929-42 2024-04-22T18:40:06.407 INFO:tasks.rados.rados.0.smithi089.stdout:1260: finishing write tid 5 to smithi08937929-42 2024-04-22T18:40:06.407 INFO:tasks.rados.rados.0.smithi089.stdout:1260: oid 42 updating version 0 to 28 2024-04-22T18:40:06.407 INFO:tasks.rados.rados.0.smithi089.stdout:1260: oid 42 version 28 is already newer than 27 2024-04-22T18:40:06.407 INFO:tasks.rados.rados.0.smithi089.stdout:1260: oid 42 version 28 is already newer than 26 2024-04-22T18:40:06.407 INFO:tasks.rados.rados.0.smithi089.stdout:1260: oid 42 version 28 is already newer than 25 2024-04-22T18:40:06.407 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 28 (ObjNum 228 snap 0 seq_num 228) dirty exists 2024-04-22T18:40:06.408 INFO:tasks.rados.rados.0.smithi089.stdout:1260: left oid 42 (ObjNum 228 snap 0 seq_num 228) 2024-04-22T18:40:06.408 INFO:tasks.rados.rados.0.smithi089.stdout:1263: expect (ObjNum 226 snap 0 seq_num 226) 2024-04-22T18:40:06.463 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:06.464+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:06.517 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:06.517 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:06.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:06.564 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:06.566+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:06.653 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 6, 7, 2] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:40:06.654 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:40:06.654 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-22T18:40:06.654 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-22T18:40:06.655 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:40:06.664 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:06.665+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:06.670 INFO:tasks.rados.rados.0.smithi089.stdout:1264: finishing write tid 1 to smithi08937929-39 2024-04-22T18:40:06.670 INFO:tasks.rados.rados.0.smithi089.stdout:1265: finishing set_chunk to oid 29 2024-04-22T18:40:06.670 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 29 set_chunk returned error code -95 offset: 22528 length: 2162176 2024-04-22T18:40:06.670 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 0 (ObjNum 149 snap 0 seq_num 149) dirty exists 2024-04-22T18:40:06.670 INFO:tasks.rados.rados.0.smithi089.stdout:1266: finishing write tid 1 to smithi08937929-38 2024-04-22T18:40:06.670 INFO:tasks.rados.rados.0.smithi089.stdout:1266: finishing write tid 2 to smithi08937929-38 2024-04-22T18:40:06.670 INFO:tasks.rados.rados.0.smithi089.stdout:1266: finishing write tid 3 to smithi08937929-38 2024-04-22T18:40:06.670 INFO:tasks.rados.rados.0.smithi089.stdout:1235: done (15 left) 2024-04-22T18:40:06.671 INFO:tasks.rados.rados.0.smithi089.stdout:1252: done (14 left) 2024-04-22T18:40:06.671 INFO:tasks.rados.rados.0.smithi089.stdout:1254: done (13 left) 2024-04-22T18:40:06.671 INFO:tasks.rados.rados.0.smithi089.stdout:1256: done (12 left) 2024-04-22T18:40:06.671 INFO:tasks.rados.rados.0.smithi089.stdout:1257: done (11 left) 2024-04-22T18:40:06.671 INFO:tasks.rados.rados.0.smithi089.stdout:1258: done (10 left) 2024-04-22T18:40:06.671 INFO:tasks.rados.rados.0.smithi089.stdout:1260: done (9 left) 2024-04-22T18:40:06.671 INFO:tasks.rados.rados.0.smithi089.stdout:1262: done (8 left) 2024-04-22T18:40:06.671 INFO:tasks.rados.rados.0.smithi089.stdout:1263: done (7 left) 2024-04-22T18:40:06.671 INFO:tasks.rados.rados.0.smithi089.stdout:1265: done (6 left) 2024-04-22T18:40:06.671 INFO:tasks.rados.rados.0.smithi089.stdout:1267: done (5 left) 2024-04-22T18:40:06.671 INFO:tasks.rados.rados.0.smithi089.stdout:1270: write oid 48 current snap is 0 2024-04-22T18:40:06.671 INFO:tasks.rados.rados.0.smithi089.stdout:1264: finishing write tid 2 to smithi08937929-39 2024-04-22T18:40:06.671 INFO:tasks.rados.rados.0.smithi089.stdout:1270: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2024-04-22T18:40:06.672 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:06.672 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:06.676 INFO:tasks.rados.rados.0.smithi089.stdout:1270: writing smithi08937929-48 from 682417 to 1155526 tid 1 2024-04-22T18:40:06.679 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:06.682 INFO:tasks.rados.rados.0.smithi089.stdout:1270: writing smithi08937929-48 from 1567811 to 2169874 tid 2 2024-04-22T18:40:06.690 INFO:tasks.rados.rados.0.smithi089.stdout:1270: writing smithi08937929-48 from 2844123 to 3465605 tid 3 2024-04-22T18:40:06.690 INFO:tasks.rados.rados.0.smithi089.stdout:1271: write oid 25 current snap is 0 2024-04-22T18:40:06.690 INFO:tasks.rados.rados.0.smithi089.stdout:1264: finishing write tid 3 to smithi08937929-39 2024-04-22T18:40:06.690 INFO:tasks.rados.rados.0.smithi089.stdout:1264: finishing write tid 4 to smithi08937929-39 2024-04-22T18:40:06.690 INFO:tasks.rados.rados.0.smithi089.stdout:1264: finishing write tid 5 to smithi08937929-39 2024-04-22T18:40:06.690 INFO:tasks.rados.rados.0.smithi089.stdout:1264: oid 39 updating version 0 to 91 2024-04-22T18:40:06.690 INFO:tasks.rados.rados.0.smithi089.stdout:1264: oid 39 version 91 is already newer than 89 2024-04-22T18:40:06.690 INFO:tasks.rados.rados.0.smithi089.stdout:1264: oid 39 version 91 is already newer than 88 2024-04-22T18:40:06.690 INFO:tasks.rados.rados.0.smithi089.stdout:1264: oid 39 version 91 is already newer than 90 2024-04-22T18:40:06.690 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 91 (ObjNum 229 snap 0 seq_num 229) dirty exists 2024-04-22T18:40:06.690 INFO:tasks.rados.rados.0.smithi089.stdout:1264: left oid 39 (ObjNum 229 snap 0 seq_num 229) 2024-04-22T18:40:06.691 INFO:tasks.rados.rados.0.smithi089.stdout:1266: finishing write tid 4 to smithi08937929-38 2024-04-22T18:40:06.691 INFO:tasks.rados.rados.0.smithi089.stdout:1271: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-04-22T18:40:06.696 INFO:tasks.rados.rados.0.smithi089.stdout:1271: writing smithi08937929-25 from 699224 to 1247582 tid 1 2024-04-22T18:40:06.703 INFO:tasks.rados.rados.0.smithi089.stdout:1271: writing smithi08937929-25 from 2026293 to 2488367 tid 2 2024-04-22T18:40:06.709 INFO:tasks.rados.rados.0.smithi089.stdout:1271: writing smithi08937929-25 from 3271546 to 3704300 tid 3 2024-04-22T18:40:06.710 INFO:tasks.rados.rados.0.smithi089.stdout:1264: done (6 left) 2024-04-22T18:40:06.710 INFO:tasks.rados.rados.0.smithi089.stdout:1272: set_chunk oid 35 target oid 2024-04-22T18:40:06.766 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:06.767+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:06.824 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:06.824 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:06.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:06.866 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:06.868+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:06.943 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:40:06.943 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:06.943 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 76, 2024-04-22T18:40:06.943 INFO:teuthology.orchestra.run.smithi089.stdout: "fsid": "9e25744c-eff1-4a4a-8d47-0dda406b2c35", 2024-04-22T18:40:06.943 INFO:teuthology.orchestra.run.smithi089.stdout: "created": "2024-04-22T18:38:26.841687+0000", 2024-04-22T18:40:06.943 INFO:teuthology.orchestra.run.smithi089.stdout: "modified": "2024-04-22T18:40:06.635264+0000", 2024-04-22T18:40:06.944 INFO:teuthology.orchestra.run.smithi089.stdout: "last_up_change": "2024-04-22T18:38:35.971820+0000", 2024-04-22T18:40:06.944 INFO:teuthology.orchestra.run.smithi089.stdout: "last_in_change": "2024-04-22T18:40:01.634193+0000", 2024-04-22T18:40:06.944 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-22T18:40:06.944 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_num": 5804032, 2024-04-22T18:40:06.944 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_set": [ 2024-04-22T18:40:06.944 INFO:teuthology.orchestra.run.smithi089.stdout: "nodeep-scrub", 2024-04-22T18:40:06.944 INFO:teuthology.orchestra.run.smithi089.stdout: "pglog_hardlimit", 2024-04-22T18:40:06.944 INFO:teuthology.orchestra.run.smithi089.stdout: "purged_snapdirs", 2024-04-22T18:40:06.944 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_deletes", 2024-04-22T18:40:06.944 INFO:teuthology.orchestra.run.smithi089.stdout: "sortbitwise" 2024-04-22T18:40:06.944 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:06.944 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_version": 5, 2024-04-22T18:40:06.944 INFO:teuthology.orchestra.run.smithi089.stdout: "full_ratio": 0.94999998807907104, 2024-04-22T18:40:06.944 INFO:teuthology.orchestra.run.smithi089.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-22T18:40:06.945 INFO:teuthology.orchestra.run.smithi089.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-22T18:40:06.945 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_snapshot": "", 2024-04-22T18:40:06.945 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_max": 4, 2024-04-22T18:40:06.945 INFO:teuthology.orchestra.run.smithi089.stdout: "max_osd": 8, 2024-04-22T18:40:06.945 INFO:teuthology.orchestra.run.smithi089.stdout: "require_min_compat_client": "luminous", 2024-04-22T18:40:06.945 INFO:teuthology.orchestra.run.smithi089.stdout: "min_compat_client": "jewel", 2024-04-22T18:40:06.945 INFO:teuthology.orchestra.run.smithi089.stdout: "require_osd_release": "reef", 2024-04-22T18:40:06.945 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_crimson": false, 2024-04-22T18:40:06.945 INFO:teuthology.orchestra.run.smithi089.stdout: "pools": [ 2024-04-22T18:40:06.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.945 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 1, 2024-04-22T18:40:06.945 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": ".mgr", 2024-04-22T18:40:06.945 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:36.991610+0000", 2024-04-22T18:40:06.945 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:40:06.946 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:40:06.946 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:40:06.946 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:40:06.946 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:40:06.946 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:40:06.946 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:40:06.946 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:40:06.946 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:40:06.946 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:40:06.946 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:40:06.946 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:40:06.947 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 1, 2024-04-22T18:40:06.947 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 1, 2024-04-22T18:40:06.947 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 1, 2024-04-22T18:40:06.947 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 1, 2024-04-22T18:40:06.947 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 1, 2024-04-22T18:40:06.947 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:40:06.947 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:40:06.947 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:40:06.947 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:40:06.947 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:40:06.947 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:40:06.947 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:40:06.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.947 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "16", 2024-04-22T18:40:06.948 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:40:06.948 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:40:06.948 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-04-22T18:40:06.948 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:40:06.948 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:40:06.948 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:40:06.948 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:40:06.948 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:40:06.948 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:40:06.948 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:40:06.948 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:40:06.948 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:40:06.948 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:40:06.948 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:40:06.948 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:40:06.949 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:40:06.949 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:40:06.949 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:40:06.949 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:40:06.949 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:40:06.949 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:40:06.949 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:40:06.949 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:40:06.949 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:40:06.949 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:40:06.949 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:40:06.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.949 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:40:06.949 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:40:06.950 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:40:06.950 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:40:06.950 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:40:06.950 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:40:06.950 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:40:06.950 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:40:06.950 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:40:06.950 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:40:06.950 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:40:06.950 INFO:teuthology.orchestra.run.smithi089.stdout: "options": { 2024-04-22T18:40:06.950 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_max": 32, 2024-04-22T18:40:06.950 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_min": 1 2024-04-22T18:40:06.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.951 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-04-22T18:40:06.951 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": {} 2024-04-22T18:40:06.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.951 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:40:06.951 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 8, 2024-04-22T18:40:06.951 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 8, 2024-04-22T18:40:06.951 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.25, 2024-04-22T18:40:06.951 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2, 2024-04-22T18:40:06.951 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2, 2024-04-22T18:40:06.951 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.98099899291992188, 2024-04-22T18:40:06.951 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-22T18:40:06.952 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-22T18:40:06.952 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.952 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 2, 2024-04-22T18:40:06.952 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "rbd", 2024-04-22T18:40:06.952 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:37.622620+0000", 2024-04-22T18:40:06.952 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:40:06.952 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:40:06.952 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:40:06.952 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:40:06.952 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:40:06.952 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:40:06.953 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:40:06.953 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:40:06.953 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:40:06.953 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:40:06.953 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:40:06.953 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:40:06.953 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 8, 2024-04-22T18:40:06.953 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 8, 2024-04-22T18:40:06.953 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 8, 2024-04-22T18:40:06.953 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 8, 2024-04-22T18:40:06.953 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 8, 2024-04-22T18:40:06.953 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:40:06.953 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:40:06.954 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:40:06.954 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:40:06.954 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:40:06.954 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:40:06.954 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:40:06.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.954 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "17", 2024-04-22T18:40:06.954 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:40:06.954 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:40:06.954 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-04-22T18:40:06.954 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:40:06.954 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:40:06.955 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:40:06.955 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:40:06.955 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:40:06.955 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:40:06.955 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:40:06.955 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:40:06.955 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:40:06.955 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:40:06.955 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:40:06.955 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:40:06.955 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:40:06.955 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:40:06.955 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:40:06.956 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:40:06.956 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:40:06.956 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:40:06.956 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:40:06.956 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:40:06.956 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:40:06.956 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:40:06.956 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:40:06.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.956 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:40:06.956 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:40:06.956 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:40:06.956 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:40:06.956 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:40:06.957 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:40:06.957 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:40:06.957 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:40:06.957 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:40:06.957 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:40:06.957 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:40:06.957 INFO:teuthology.orchestra.run.smithi089.stdout: "options": {}, 2024-04-22T18:40:06.957 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-04-22T18:40:06.957 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd": {} 2024-04-22T18:40:06.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.957 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:40:06.957 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 3, 2024-04-22T18:40:06.957 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 3, 2024-04-22T18:40:06.957 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.75, 2024-04-22T18:40:06.958 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2.25, 2024-04-22T18:40:06.958 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2.25, 2024-04-22T18:40:06.958 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.98099899291992188, 2024-04-22T18:40:06.958 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-22T18:40:06.958 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-22T18:40:06.958 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.958 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 3, 2024-04-22T18:40:06.958 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "low_tier", 2024-04-22T18:40:06.958 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:46.038005+0000", 2024-04-22T18:40:06.958 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:40:06.958 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:40:06.959 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:40:06.959 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:40:06.959 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:40:06.959 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:40:06.959 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:40:06.959 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:40:06.959 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:40:06.959 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:40:06.959 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:40:06.959 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:40:06.959 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 4, 2024-04-22T18:40:06.959 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 4, 2024-04-22T18:40:06.959 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 4, 2024-04-22T18:40:06.959 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 4, 2024-04-22T18:40:06.960 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 4, 2024-04-22T18:40:06.960 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:40:06.960 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:40:06.960 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:40:06.960 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:40:06.960 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:40:06.960 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:40:06.960 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:40:06.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.960 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "21", 2024-04-22T18:40:06.960 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:40:06.960 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:40:06.960 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-04-22T18:40:06.961 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:40:06.961 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:40:06.961 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:40:06.961 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:40:06.961 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:40:06.961 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:40:06.961 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:40:06.961 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:40:06.961 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:40:06.961 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:40:06.961 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:40:06.961 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:40:06.961 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:40:06.961 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:40:06.962 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:40:06.962 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:40:06.962 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:40:06.962 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:40:06.962 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:40:06.962 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:40:06.962 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:40:06.962 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:40:06.962 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:40:06.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.962 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:40:06.962 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:40:06.962 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:40:06.962 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:40:06.963 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:40:06.963 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:40:06.963 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:40:06.963 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:40:06.963 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:40:06.963 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:40:06.963 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:40:06.963 INFO:teuthology.orchestra.run.smithi089.stdout: "options": {}, 2024-04-22T18:40:06.963 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": {}, 2024-04-22T18:40:06.963 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:40:06.963 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 2, 2024-04-22T18:40:06.963 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 2, 2024-04-22T18:40:06.963 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.75, 2024-04-22T18:40:06.963 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 1.5, 2024-04-22T18:40:06.963 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 1.5, 2024-04-22T18:40:06.964 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.98099899291992188, 2024-04-22T18:40:06.964 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-22T18:40:06.964 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-22T18:40:06.964 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.964 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 4, 2024-04-22T18:40:06.964 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "unique_pool_0", 2024-04-22T18:40:06.964 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:46.483369+0000", 2024-04-22T18:40:06.964 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:40:06.964 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:40:06.964 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:40:06.965 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:40:06.965 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:40:06.965 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:40:06.965 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:40:06.965 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:40:06.965 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:40:06.965 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:40:06.965 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:40:06.965 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:40:06.965 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 26, 2024-04-22T18:40:06.965 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 19, 2024-04-22T18:40:06.965 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 26, 2024-04-22T18:40:06.965 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 26, 2024-04-22T18:40:06.966 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 26, 2024-04-22T18:40:06.966 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:40:06.966 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:40:06.966 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:40:06.966 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:40:06.966 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:40:06.966 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:40:06.966 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:40:06.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.966 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "72", 2024-04-22T18:40:06.966 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:40:06.966 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:40:06.967 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "59", 2024-04-22T18:40:06.967 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:40:06.967 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:40:06.967 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:40:06.967 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:40:06.967 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:40:06.967 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:40:06.967 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:40:06.967 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:40:06.967 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:40:06.967 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:40:06.967 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:40:06.967 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:40:06.967 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:40:06.967 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:40:06.968 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:40:06.968 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:40:06.968 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:40:06.968 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:40:06.968 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:40:06.968 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:40:06.968 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:40:06.968 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:40:06.968 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:40:06.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.968 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:40:06.968 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:40:06.968 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:40:06.968 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:40:06.969 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:40:06.969 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:40:06.969 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:40:06.969 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:40:06.969 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:40:06.969 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:40:06.969 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:40:06.969 INFO:teuthology.orchestra.run.smithi089.stdout: "options": { 2024-04-22T18:40:06.969 INFO:teuthology.orchestra.run.smithi089.stdout: "dedup_cdc_chunk_size": 131072, 2024-04-22T18:40:06.969 INFO:teuthology.orchestra.run.smithi089.stdout: "dedup_chunk_algorithm": "fastcdc", 2024-04-22T18:40:06.969 INFO:teuthology.orchestra.run.smithi089.stdout: "dedup_tier": 3, 2024-04-22T18:40:06.969 INFO:teuthology.orchestra.run.smithi089.stdout: "fingerprint_algorithm": "sha256" 2024-04-22T18:40:06.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.970 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-04-22T18:40:06.970 INFO:teuthology.orchestra.run.smithi089.stdout: "rados": {} 2024-04-22T18:40:06.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.970 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:40:06.970 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 2.1400001049041748, 2024-04-22T18:40:06.970 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 2.1400001049041748, 2024-04-22T18:40:06.970 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.87999999523162842, 2024-04-22T18:40:06.970 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 1.8799999952316284, 2024-04-22T18:40:06.970 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 1.8799999952316284, 2024-04-22T18:40:06.970 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.98099899291992188, 2024-04-22T18:40:06.970 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-22T18:40:06.970 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-22T18:40:06.971 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.971 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.971 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:06.971 INFO:teuthology.orchestra.run.smithi089.stdout: "osds": [ 2024-04-22T18:40:06.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.971 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-04-22T18:40:06.971 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "bbf33cfb-0f07-4df6-b90f-83e9edb039a6", 2024-04-22T18:40:06.971 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:06.971 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:40:06.971 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:40:06.971 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:40:06.971 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:06.971 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:06.971 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:06.972 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 72, 2024-04-22T18:40:06.972 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:06.972 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:06.972 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:06.972 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.972 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.972 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6800", 2024-04-22T18:40:06.972 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:40:06.972 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.972 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.972 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:06.972 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6801", 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6803", 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.973 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:06.974 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.974 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.974 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6802", 2024-04-22T18:40:06.974 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:40:06.974 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.974 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.974 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6800/31890", 2024-04-22T18:40:06.974 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6801/31890", 2024-04-22T18:40:06.974 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6803/31890", 2024-04-22T18:40:06.974 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6802/31890", 2024-04-22T18:40:06.974 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:06.974 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:06.975 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:06.975 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.975 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-04-22T18:40:06.975 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "30415a8e-9f47-416f-9fd0-716c2cb3f562", 2024-04-22T18:40:06.975 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:06.975 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:40:06.975 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:40:06.975 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:40:06.975 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:06.975 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:06.975 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:06.975 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 72, 2024-04-22T18:40:06.976 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:06.976 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:06.976 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:06.976 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.976 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.976 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6804", 2024-04-22T18:40:06.976 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:40:06.976 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.976 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.976 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:06.976 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.976 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.977 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6805", 2024-04-22T18:40:06.977 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:40:06.977 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.977 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.977 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:06.977 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.977 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.977 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6807", 2024-04-22T18:40:06.977 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:40:06.977 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.977 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.978 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:06.978 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.978 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.978 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6806", 2024-04-22T18:40:06.978 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:40:06.978 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.978 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.978 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6804/31899", 2024-04-22T18:40:06.978 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6805/31899", 2024-04-22T18:40:06.978 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6807/31899", 2024-04-22T18:40:06.978 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6806/31899", 2024-04-22T18:40:06.978 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:06.979 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:06.979 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:06.979 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.979 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-04-22T18:40:06.979 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "b8da6b88-8355-4225-930d-bfed881089b7", 2024-04-22T18:40:06.979 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:06.979 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:40:06.979 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:40:06.979 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:40:06.979 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:06.979 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:06.979 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:06.980 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 72, 2024-04-22T18:40:06.980 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:06.980 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:06.980 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:06.980 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.980 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.980 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6812", 2024-04-22T18:40:06.980 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:40:06.980 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.980 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.980 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:06.980 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.981 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.981 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6813", 2024-04-22T18:40:06.981 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:40:06.981 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.981 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.981 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:06.981 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.981 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.981 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6815", 2024-04-22T18:40:06.981 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:40:06.981 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.981 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.982 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:06.982 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.982 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.982 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6814", 2024-04-22T18:40:06.982 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:40:06.982 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.982 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.982 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6812/31903", 2024-04-22T18:40:06.982 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6813/31903", 2024-04-22T18:40:06.982 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6815/31903", 2024-04-22T18:40:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6814/31903", 2024-04-22T18:40:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-04-22T18:40:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "3eaec913-9d7d-4b03-a28f-0c334e7c39f1", 2024-04-22T18:40:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:40:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:40:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 0.847991943359375, 2024-04-22T18:40:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 74, 2024-04-22T18:40:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6808", 2024-04-22T18:40:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:40:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6809", 2024-04-22T18:40:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:40:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6811", 2024-04-22T18:40:06.986 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:40:06.986 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.986 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.986 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:06.986 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.986 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.986 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6810", 2024-04-22T18:40:06.986 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:40:06.986 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.986 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.987 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6808/31913", 2024-04-22T18:40:06.987 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6809/31913", 2024-04-22T18:40:06.987 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6811/31913", 2024-04-22T18:40:06.987 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6810/31913", 2024-04-22T18:40:06.987 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:06.987 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:06.987 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:06.987 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.987 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-04-22T18:40:06.987 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "887acc3b-6f6b-49eb-a039-13050a2c7d64", 2024-04-22T18:40:06.987 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:06.988 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:40:06.988 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:40:06.988 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:40:06.988 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:06.988 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:06.988 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:06.988 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 73, 2024-04-22T18:40:06.988 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:06.988 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:06.988 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:06.988 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.988 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.988 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6808", 2024-04-22T18:40:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:40:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6809", 2024-04-22T18:40:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:40:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:06.989 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.990 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.990 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6811", 2024-04-22T18:40:06.990 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:40:06.990 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.990 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.990 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:06.990 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.990 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.990 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6810", 2024-04-22T18:40:06.990 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6808/31922", 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6809/31922", 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6811/31922", 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6810/31922", 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "dcaddfd0-f8d1-4c28-bda2-d17eed606b60", 2024-04-22T18:40:06.991 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:06.992 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 0, 2024-04-22T18:40:06.992 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 0, 2024-04-22T18:40:06.992 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:40:06.992 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:06.992 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:06.992 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:06.992 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 63, 2024-04-22T18:40:06.992 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:06.992 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:06.992 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:06.992 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.992 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.992 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6804", 2024-04-22T18:40:06.992 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:40:06.993 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.993 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.993 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:06.993 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.993 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.993 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6805", 2024-04-22T18:40:06.993 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:40:06.993 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.993 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.993 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:06.993 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6807", 2024-04-22T18:40:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:40:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6806", 2024-04-22T18:40:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:40:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.995 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6804/31921", 2024-04-22T18:40:06.995 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6805/31921", 2024-04-22T18:40:06.995 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6807/31921", 2024-04-22T18:40:06.995 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6806/31921", 2024-04-22T18:40:06.995 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:06.995 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:06.995 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:06.995 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.995 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-04-22T18:40:06.995 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "acf0527b-3020-4b0a-b64c-5068d21a5781", 2024-04-22T18:40:06.995 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:06.995 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:40:06.996 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:40:06.996 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:40:06.996 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:06.996 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:06.996 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:06.996 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 72, 2024-04-22T18:40:06.996 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:06.996 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:06.996 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:06.996 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.996 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.996 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6800", 2024-04-22T18:40:06.996 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:40:06.997 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.997 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.997 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:06.997 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.997 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.997 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6801", 2024-04-22T18:40:06.997 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:40:06.997 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.997 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.997 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:06.997 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.998 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.998 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6803", 2024-04-22T18:40:06.998 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:40:06.998 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.998 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.998 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:06.998 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:06.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.998 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:06.998 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6802", 2024-04-22T18:40:06.998 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:40:06.998 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:06.999 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.999 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6800/31920", 2024-04-22T18:40:06.999 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6801/31920", 2024-04-22T18:40:06.999 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6803/31920", 2024-04-22T18:40:06.999 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6802/31920", 2024-04-22T18:40:06.999 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:06.999 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:06.999 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:06.999 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:06.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:06.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:06.999 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-04-22T18:40:06.999 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "305394bc-be3d-420b-88d7-d273670540b1", 2024-04-22T18:40:07.000 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:07.000 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:40:07.000 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:40:07.000 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:40:07.000 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:07.000 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:07.000 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:07.000 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 72, 2024-04-22T18:40:07.000 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:07.000 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:07.000 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:07.000 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:07.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:07.000 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:07.001 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6812", 2024-04-22T18:40:07.001 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:40:07.001 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:07.001 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:07.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:07.001 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:07.001 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:07.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:07.001 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:07.001 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6813", 2024-04-22T18:40:07.001 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:40:07.001 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:07.001 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:07.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:07.002 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:07.002 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:07.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:07.002 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:07.002 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6815", 2024-04-22T18:40:07.002 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:40:07.002 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:07.002 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:07.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:07.002 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:07.002 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:07.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:07.002 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:07.002 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6814", 2024-04-22T18:40:07.003 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:40:07.003 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:07.003 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:07.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:07.003 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6812/31939", 2024-04-22T18:40:07.003 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6813/31939", 2024-04-22T18:40:07.003 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6815/31939", 2024-04-22T18:40:07.003 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6814/31939", 2024-04-22T18:40:07.003 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:07.003 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:07.003 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:07.003 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:07.003 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:07.003 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:07.004 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_xinfo": [ 2024-04-22T18:40:07.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:07.004 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-04-22T18:40:07.004 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:07.004 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:07.004 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:07.004 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:07.004 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:40:07.004 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:48.032183+0000", 2024-04-22T18:40:07.004 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:07.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:07.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:07.004 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-04-22T18:40:07.004 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:07.004 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:07.005 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:07.005 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:07.005 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:40:07.005 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:48.108964+0000", 2024-04-22T18:40:07.005 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:07.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:07.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:07.005 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-04-22T18:40:07.005 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:07.005 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:07.005 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:07.005 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:07.005 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:40:07.005 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:50.436792+0000", 2024-04-22T18:40:07.005 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:07.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:07.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:07.006 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-04-22T18:40:07.006 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:07.006 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:07.006 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:07.006 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:07.006 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:40:07.006 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:52.033545+0000", 2024-04-22T18:40:07.006 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:07.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:07.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:07.006 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-04-22T18:40:07.006 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:07.007 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:07.007 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:07.007 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:07.007 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:40:07.007 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:39.194178+0000", 2024-04-22T18:40:07.007 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:07.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:07.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:07.007 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-04-22T18:40:07.007 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:07.007 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:07.007 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:07.007 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:07.007 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 65536, 2024-04-22T18:40:07.008 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:40.812449+0000", 2024-04-22T18:40:07.008 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:07.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:07.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:07.008 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-04-22T18:40:07.008 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:07.008 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:07.008 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:07.008 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:07.008 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:40:07.008 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:44.814263+0000", 2024-04-22T18:40:07.008 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:07.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:07.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:07.009 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-04-22T18:40:07.009 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:07.009 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:07.009 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:07.009 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:07.009 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:40:07.009 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:42.667515+0000", 2024-04-22T18:40:07.009 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:07.009 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:07.009 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:07.009 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap": [], 2024-04-22T18:40:07.009 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_items": [], 2024-04-22T18:40:07.009 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_primaries": [], 2024-04-22T18:40:07.009 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_temp": [ 2024-04-22T18:40:07.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:07.010 INFO:teuthology.orchestra.run.smithi089.stdout: "pgid": "3.2", 2024-04-22T18:40:07.010 INFO:teuthology.orchestra.run.smithi089.stdout: "osds": [ 2024-04-22T18:40:07.010 INFO:teuthology.orchestra.run.smithi089.stdout: 3, 2024-04-22T18:40:07.010 INFO:teuthology.orchestra.run.smithi089.stdout: 6 2024-04-22T18:40:07.010 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:07.010 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:07.010 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:07.010 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_temp": [], 2024-04-22T18:40:07.010 INFO:teuthology.orchestra.run.smithi089.stdout: "blocklist": {}, 2024-04-22T18:40:07.010 INFO:teuthology.orchestra.run.smithi089.stdout: "range_blocklist": {}, 2024-04-22T18:40:07.010 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profiles": { 2024-04-22T18:40:07.010 INFO:teuthology.orchestra.run.smithi089.stdout: "default": { 2024-04-22T18:40:07.011 INFO:teuthology.orchestra.run.smithi089.stdout: "crush-failure-domain": "osd", 2024-04-22T18:40:07.011 INFO:teuthology.orchestra.run.smithi089.stdout: "k": "2", 2024-04-22T18:40:07.011 INFO:teuthology.orchestra.run.smithi089.stdout: "m": "1", 2024-04-22T18:40:07.011 INFO:teuthology.orchestra.run.smithi089.stdout: "plugin": "jerasure", 2024-04-22T18:40:07.011 INFO:teuthology.orchestra.run.smithi089.stdout: "technique": "reed_sol_van" 2024-04-22T18:40:07.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:07.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:07.011 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps_queue": [], 2024-04-22T18:40:07.011 INFO:teuthology.orchestra.run.smithi089.stdout: "new_removed_snaps": [], 2024-04-22T18:40:07.011 INFO:teuthology.orchestra.run.smithi089.stdout: "new_purged_snaps": [], 2024-04-22T18:40:07.011 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_node_flags": {}, 2024-04-22T18:40:07.011 INFO:teuthology.orchestra.run.smithi089.stdout: "device_class_flags": {}, 2024-04-22T18:40:07.011 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode": { 2024-04-22T18:40:07.012 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_enabled": false, 2024-04-22T18:40:07.012 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_bucket_count": 0, 2024-04-22T18:40:07.012 INFO:teuthology.orchestra.run.smithi089.stdout: "degraded_stretch_mode": 0, 2024-04-22T18:40:07.012 INFO:teuthology.orchestra.run.smithi089.stdout: "recovering_stretch_mode": 0, 2024-04-22T18:40:07.012 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_bucket": 0 2024-04-22T18:40:07.012 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:07.012 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:07.012 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:40:07.014 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:07.014 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:07.015 INFO:tasks.thrashosds.thrasher:j is {'epoch': 76, 'fsid': '9e25744c-eff1-4a4a-8d47-0dda406b2c35', 'created': '2024-04-22T18:38:26.841687+0000', 'modified': '2024-04-22T18:40:06.635264+0000', 'last_up_change': '2024-04-22T18:38:35.971820+0000', 'last_in_change': '2024-04-22T18:40:01.634193+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T18:38:36.991610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9809989929199219, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-22T18:38:37.622620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9809989929199219, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-04-22T18:38:46.038005+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9809989929199219, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-22T18:38:46.483369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 19, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '59', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'dedup_cdc_chunk_size': 131072, 'dedup_chunk_algorithm': 'fastcdc', 'dedup_tier': 3, 'fingerprint_algorithm': 'sha256'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_acting': 2.140000104904175, 'score_stable': 2.140000104904175, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9809989929199219, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}], 'osds': [{'osd': 0, 'uuid': 'bbf33cfb-0f07-4df6-b90f-83e9edb039a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6800', 'nonce': 31890}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6801', 'nonce': 31890}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6803', 'nonce': 31890}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6802', 'nonce': 31890}]}, 'public_addr': '172.21.15.89:6800/31890', 'cluster_addr': '172.21.15.89:6801/31890', 'heartbeat_back_addr': '172.21.15.89:6803/31890', 'heartbeat_front_addr': '172.21.15.89:6802/31890', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '30415a8e-9f47-416f-9fd0-716c2cb3f562', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6804', 'nonce': 31899}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6805', 'nonce': 31899}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6807', 'nonce': 31899}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6806', 'nonce': 31899}]}, 'public_addr': '172.21.15.89:6804/31899', 'cluster_addr': '172.21.15.89:6805/31899', 'heartbeat_back_addr': '172.21.15.89:6807/31899', 'heartbeat_front_addr': '172.21.15.89:6806/31899', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b8da6b88-8355-4225-930d-bfed881089b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6812', 'nonce': 31903}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6813', 'nonce': 31903}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6815', 'nonce': 31903}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6814', 'nonce': 31903}]}, 'public_addr': '172.21.15.89:6812/31903', 'cluster_addr': '172.21.15.89:6813/31903', 'heartbeat_back_addr': '172.21.15.89:6815/31903', 'heartbeat_front_addr': '172.21.15.89:6814/31903', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3eaec913-9d7d-4b03-a28f-0c334e7c39f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.847991943359375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 74, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6808', 'nonce': 31913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6809', 'nonce': 31913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6811', 'nonce': 31913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6810', 'nonce': 31913}]}, 'public_addr': '172.21.15.89:6808/31913', 'cluster_addr': '172.21.15.89:6809/31913', 'heartbeat_back_addr': '172.21.15.89:6811/31913', 'heartbeat_front_addr': '172.21.15.89:6810/31913', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '887acc3b-6f6b-49eb-a039-13050a2c7d64', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 31922}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 31922}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 31922}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 31922}]}, 'public_addr': '172.21.15.121:6808/31922', 'cluster_addr': '172.21.15.121:6809/31922', 'heartbeat_back_addr': '172.21.15.121:6811/31922', 'heartbeat_front_addr': '172.21.15.121:6810/31922', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'dcaddfd0-f8d1-4c28-bda2-d17eed606b60', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 63, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 31921}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 31921}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 31921}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 31921}]}, 'public_addr': '172.21.15.121:6804/31921', 'cluster_addr': '172.21.15.121:6805/31921', 'heartbeat_back_addr': '172.21.15.121:6807/31921', 'heartbeat_front_addr': '172.21.15.121:6806/31921', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'acf0527b-3020-4b0a-b64c-5068d21a5781', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 31920}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 31920}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 31920}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 31920}]}, 'public_addr': '172.21.15.121:6800/31920', 'cluster_addr': '172.21.15.121:6801/31920', 'heartbeat_back_addr': '172.21.15.121:6803/31920', 'heartbeat_front_addr': '172.21.15.121:6802/31920', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '305394bc-be3d-420b-88d7-d273670540b1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 31939}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 31939}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 31939}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 31939}]}, 'public_addr': '172.21.15.121:6812/31939', 'cluster_addr': '172.21.15.121:6813/31939', 'heartbeat_back_addr': '172.21.15.121:6815/31939', 'heartbeat_front_addr': '172.21.15.121:6814/31939', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:39:48.032183+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:39:48.108964+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:39:50.436792+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:39:52.033545+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:39:39.194178+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-22T18:39:40.812449+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:39:44.814263+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:39:42.667515+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.2', 'osds': [3, 6]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-22T18:40:07.015 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:40:07.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:07.016 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:07.015+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:07.018 INFO:tasks.rados.rados.0.smithi089.stdout:1272: set_chunk oid 35 offset: 2634752 length: 523776 target oid ad394e1e2818f082c0a581413eb2cc6038b3f9f736b58494c78181ceefa63321 offset: 0 snap -1 2024-04-22T18:40:07.018 INFO:tasks.rados.rados.0.smithi089.stdout:1273: tier_evict oid 26 2024-04-22T18:40:07.018 INFO:tasks.rados.rados.0.smithi089.stdout:1273: tier_evict oid 26 snap -1 2024-04-22T18:40:07.018 INFO:tasks.rados.rados.0.smithi089.stdout:1266: finishing write tid 5 to smithi08937929-38 2024-04-22T18:40:07.018 INFO:tasks.rados.rados.0.smithi089.stdout:1266: oid 38 updating version 0 to 83 2024-04-22T18:40:07.018 INFO:tasks.rados.rados.0.smithi089.stdout:1266: oid 38 updating version 83 to 86 2024-04-22T18:40:07.018 INFO:tasks.rados.rados.0.smithi089.stdout:1266: oid 38 version 86 is already newer than 84 2024-04-22T18:40:07.018 INFO:tasks.rados.rados.0.smithi089.stdout:1266: oid 38 version 86 is already newer than 85 2024-04-22T18:40:07.019 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 86 (ObjNum 230 snap 0 seq_num 230) dirty exists 2024-04-22T18:40:07.019 INFO:tasks.rados.rados.0.smithi089.stdout:1266: left oid 38 (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:40:07.019 INFO:tasks.rados.rados.0.smithi089.stdout:1261: finishing set_chunk to oid 5 2024-04-22T18:40:07.019 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 118 (ObjNum 220 snap 0 seq_num 220) dirty exists 2024-04-22T18:40:07.019 INFO:tasks.rados.rados.0.smithi089.stdout:1269: finishing set_chunk to oid 19 2024-04-22T18:40:07.019 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 19 set_chunk returned error code -95 offset: 153088 length: 1296896 2024-04-22T18:40:07.019 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 0 (ObjNum 195 snap 0 seq_num 195) dirty exists 2024-04-22T18:40:07.019 INFO:tasks.rados.rados.0.smithi089.stdout:1268: finishing set_chunk to oid 45 2024-04-22T18:40:07.019 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 57 (ObjNum 166 snap 0 seq_num 166) dirty exists 2024-04-22T18:40:07.019 INFO:tasks.rados.rados.0.smithi089.stdout:1261: done (7 left) 2024-04-22T18:40:07.019 INFO:tasks.rados.rados.0.smithi089.stdout:1266: done (6 left) 2024-04-22T18:40:07.019 INFO:tasks.rados.rados.0.smithi089.stdout:1268: done (5 left) 2024-04-22T18:40:07.019 INFO:tasks.rados.rados.0.smithi089.stdout:1269: done (4 left) 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:1274: delete oid 28 current snap is 0 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:1270: finishing write tid 1 to smithi08937929-48 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:1270: finishing write tid 2 to smithi08937929-48 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:1270: finishing write tid 3 to smithi08937929-48 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:1270: finishing write tid 4 to smithi08937929-48 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:1270: finishing write tid 5 to smithi08937929-48 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:1270: oid 48 updating version 0 to 64 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:1270: oid 48 version 64 is already newer than 63 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:1270: oid 48 version 64 is already newer than 62 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:1270: oid 48 version 64 is already newer than 61 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 64 (ObjNum 231 snap 0 seq_num 231) dirty exists 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:1270: left oid 48 (ObjNum 231 snap 0 seq_num 231) 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:1271: finishing write tid 1 to smithi08937929-25 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:1271: finishing write tid 2 to smithi08937929-25 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:1271: finishing write tid 3 to smithi08937929-25 2024-04-22T18:40:07.020 INFO:tasks.rados.rados.0.smithi089.stdout:1271: finishing write tid 4 to smithi08937929-25 2024-04-22T18:40:07.021 INFO:tasks.rados.rados.0.smithi089.stdout:1271: finishing write tid 5 to smithi08937929-25 2024-04-22T18:40:07.021 INFO:tasks.rados.rados.0.smithi089.stdout:1271: oid 25 updating version 0 to 72 2024-04-22T18:40:07.022 INFO:tasks.rados.rados.0.smithi089.stdout:1271: oid 25 updating version 72 to 73 2024-04-22T18:40:07.022 INFO:tasks.rados.rados.0.smithi089.stdout:1271: oid 25 version 73 is already newer than 71 2024-04-22T18:40:07.022 INFO:tasks.rados.rados.0.smithi089.stdout:1271: oid 25 version 73 is already newer than 70 2024-04-22T18:40:07.022 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 73 (ObjNum 232 snap 0 seq_num 232) dirty exists 2024-04-22T18:40:07.022 INFO:tasks.rados.rados.0.smithi089.stdout:1271: left oid 25 (ObjNum 232 snap 0 seq_num 232) 2024-04-22T18:40:07.022 INFO:tasks.rados.rados.0.smithi089.stdout:1273: got (22) Invalid argument 2024-04-22T18:40:07.113 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:07.115+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:07.117 INFO:tasks.rados.rados.0.smithi089.stdout:1270: done (4 left) 2024-04-22T18:40:07.117 INFO:tasks.rados.rados.0.smithi089.stdout:1271: done (3 left) 2024-04-22T18:40:07.118 INFO:tasks.rados.rados.0.smithi089.stdout:1273: done (2 left) 2024-04-22T18:40:07.118 INFO:tasks.rados.rados.0.smithi089.stdout:1274: done (1 left) 2024-04-22T18:40:07.118 INFO:tasks.rados.rados.0.smithi089.stdout:1275: write (excl) oid 13 current snap is 0 2024-04-22T18:40:07.118 INFO:tasks.rados.rados.0.smithi089.stdout:1275: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-04-22T18:40:07.125 INFO:tasks.rados.rados.0.smithi089.stdout:1275: writing smithi08937929-13 from 732838 to 1431694 tid 1 2024-04-22T18:40:07.131 INFO:tasks.rados.rados.0.smithi089.stdout:1275: writing smithi08937929-13 from 2026904 to 2492647 tid 2 2024-04-22T18:40:07.134 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:07.135 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:07.138 INFO:tasks.rados.rados.0.smithi089.stdout:1275: writing smithi08937929-13 from 3210039 to 3781690 tid 3 2024-04-22T18:40:07.138 INFO:tasks.rados.rados.0.smithi089.stdout:1276: set_chunk oid 14 target oid 2024-04-22T18:40:07.143 INFO:tasks.rados.rados.0.smithi089.stdout:1276: set_chunk oid 14 offset: 2120192 length: 517120 target oid 3d77163f318b147b3e9de9290d851a4f1c84f3796904c3d69b6fcbbe7aaea0fd offset: 0 snap -1 2024-04-22T18:40:07.143 INFO:tasks.rados.rados.0.smithi089.stdout:1277: write oid 34 current snap is 0 2024-04-22T18:40:07.143 INFO:tasks.rados.rados.0.smithi089.stdout:1272: finishing set_chunk to oid 35 2024-04-22T18:40:07.143 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 77 (ObjNum 212 snap 0 seq_num 212) dirty exists 2024-04-22T18:40:07.143 INFO:tasks.rados.rados.0.smithi089.stdout:1277: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2024-04-22T18:40:07.151 INFO:tasks.rados.rados.0.smithi089.stdout:1277: writing smithi08937929-34 from 749645 to 1523750 tid 1 2024-04-22T18:40:07.153 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:07.153 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:07.157 INFO:tasks.rados.rados.0.smithi089.stdout:1277: writing smithi08937929-34 from 2085386 to 2694787 tid 2 2024-04-22T18:40:07.161 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:07.161 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:07.164 INFO:tasks.rados.rados.0.smithi089.stdout:1277: writing smithi08937929-34 from 3237462 to 3949645 tid 3 2024-04-22T18:40:07.164 INFO:tasks.rados.rados.0.smithi089.stdout:1272: done (3 left) 2024-04-22T18:40:07.164 INFO:tasks.rados.rados.0.smithi089.stdout:1278: write oid 47 current snap is 0 2024-04-22T18:40:07.164 INFO:tasks.rados.rados.0.smithi089.stdout:1278: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2024-04-22T18:40:07.172 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:07.172 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:07.173 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:07.173 INFO:tasks.rados.rados.0.smithi089.stdout:1278: writing smithi08937929-47 from 766452 to 1332159 tid 1 2024-04-22T18:40:07.178 INFO:tasks.rados.rados.0.smithi089.stdout:1278: writing smithi08937929-47 from 1860221 to 2329633 tid 2 2024-04-22T18:40:07.187 INFO:tasks.rados.rados.0.smithi089.stdout:1278: writing smithi08937929-47 from 2981238 to 3575433 tid 3 2024-04-22T18:40:07.187 INFO:tasks.rados.rados.0.smithi089.stdout:1279: read oid 16 snap -1 2024-04-22T18:40:07.187 INFO:tasks.rados.rados.0.smithi089.stdout:1279: expect deleted 2024-04-22T18:40:07.187 INFO:tasks.rados.rados.0.smithi089.stdout:1280: read oid 33 snap -1 2024-04-22T18:40:07.187 INFO:tasks.rados.rados.0.smithi089.stdout:1280: expect (ObjNum 216 snap 0 seq_num 216) 2024-04-22T18:40:07.187 INFO:tasks.rados.rados.0.smithi089.stdout:1281: read oid 15 snap -1 2024-04-22T18:40:07.187 INFO:tasks.rados.rados.0.smithi089.stdout:1281: expect deleted 2024-04-22T18:40:07.187 INFO:tasks.rados.rados.0.smithi089.stdout:1282: write oid 3 current snap is 0 2024-04-22T18:40:07.187 INFO:tasks.rados.rados.0.smithi089.stdout:1282: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2024-04-22T18:40:07.188 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:07.188 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:07.196 INFO:tasks.rados.rados.0.smithi089.stdout:1282: writing smithi08937929-3 from 783259 to 1424215 tid 1 2024-04-22T18:40:07.203 INFO:tasks.rados.rados.0.smithi089.stdout:1282: writing smithi08937929-3 from 1918703 to 2531773 tid 2 2024-04-22T18:40:07.208 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:07.208 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:07.209 INFO:tasks.rados.rados.0.smithi089.stdout:1282: writing smithi08937929-3 from 3008661 to 3414128 tid 3 2024-04-22T18:40:07.214 INFO:tasks.rados.rados.0.smithi089.stdout:1282: writing smithi08937929-3 from 3821939 to 3983259 tid 4 2024-04-22T18:40:07.214 INFO:tasks.rados.rados.0.smithi089.stdout:1283: read oid 42 snap -1 2024-04-22T18:40:07.214 INFO:tasks.rados.rados.0.smithi089.stdout:1283: expect (ObjNum 228 snap 0 seq_num 228) 2024-04-22T18:40:07.214 INFO:tasks.rados.rados.0.smithi089.stdout:1284: tier_flush oid 32 2024-04-22T18:40:07.214 INFO:tasks.rados.rados.0.smithi089.stdout:1284: tier_flush oid 32 snap -1 2024-04-22T18:40:07.214 INFO:tasks.rados.rados.0.smithi089.stdout:1285: set_chunk oid 23 target oid 2024-04-22T18:40:07.217 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:07.218+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:07.226 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:07.226 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:07.276 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:07.276 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:07.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:07.306 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:07.306 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:07.314 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:07.316+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:07.324 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:07.324 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:07.381 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:40:07.382 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:40:07.393 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-04-22T18:40:06.976044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":245558766,"num_objects":151,"num_object_clones":0,"num_object_copies":302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":2807,"num_read_kb":2337993,"num_write":1742,"num_write_kb":433003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":72453531,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":32,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2269,"ondisk_log_size":2269,"up":78,"acting":78,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":502048,"kb_used_data":283712,"kb_used_omap":10,"kb_used_meta":209525,"kb_avail":660100832,"statfs":{"total":676457349120,"available":675943251968,"internally_reserved":0,"allocated":290521088,"data_stored":295147759,"data_compressed":7045456,"data_compressed_allocated":7516160,"data_compressed_original":16538822,"omap_allocated":11150,"internal_metadata":214553714},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11312843,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":-12,"num_read_kb":-14450,"num_write":60,"num_write_kb":8011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":39528339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.338316"},"pg_stats":[{"pgid":"4.19","version":"62'83","reported_seq":295,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.642393+0000","last_change":"2024-04-22T18:39:57.558463+0000","last_active":"2024-04-22T18:40:03.642393+0000","last_peered":"2024-04-22T18:40:03.642393+0000","last_clean":"2024-04-22T18:40:03.642393+0000","last_became_active":"2024-04-22T18:39:47.010533+0000","last_became_peered":"2024-04-22T18:39:47.010533+0000","last_unstale":"2024-04-22T18:40:03.642393+0000","last_undegraded":"2024-04-22T18:40:03.642393+0000","last_fullsized":"2024-04-22T18:40:03.642393+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'83","last_scrub_stamp":"2024-04-22T18:39:57.558302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.558302+0000","objects_scrubbed":2,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:14.300038+0000","scrub_duration":0.31429868900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6134555,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"75'54","reported_seq":202,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T18:40:04.760752+0000","last_change":"2024-04-22T18:40:03.272899+0000","last_active":"2024-04-22T18:40:04.760752+0000","last_peered":"2024-04-22T18:40:04.760752+0000","last_clean":"2024-04-22T18:40:04.760752+0000","last_became_active":"2024-04-22T18:40:02.645235+0000","last_became_peered":"2024-04-22T18:40:02.645235+0000","last_unstale":"2024-04-22T18:40:04.760752+0000","last_undegraded":"2024-04-22T18:40:04.760752+0000","last_fullsized":"2024-04-22T18:40:04.760752+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2058858,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":100483,"num_write":31,"num_write_kb":8558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2528838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"52'33","reported_seq":153,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.696955+0000","last_change":"2024-04-22T18:40:02.646106+0000","last_active":"2024-04-22T18:40:03.696955+0000","last_peered":"2024-04-22T18:40:03.696955+0000","last_clean":"2024-04-22T18:40:03.696955+0000","last_became_active":"2024-04-22T18:40:02.645801+0000","last_became_peered":"2024-04-22T18:40:02.645801+0000","last_unstale":"2024-04-22T18:40:03.696955+0000","last_undegraded":"2024-04-22T18:40:03.696955+0000","last_fullsized":"2024-04-22T18:40:03.696955+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042051,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":49,"num_read_kb":52611,"num_write":15,"num_write_kb":3979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"60'60","reported_seq":193,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.646938+0000","last_change":"2024-04-22T18:39:47.105023+0000","last_active":"2024-04-22T18:40:03.646938+0000","last_peered":"2024-04-22T18:40:03.646938+0000","last_clean":"2024-04-22T18:40:03.646938+0000","last_became_active":"2024-04-22T18:39:47.104695+0000","last_became_peered":"2024-04-22T18:39:47.104695+0000","last_unstale":"2024-04-22T18:40:03.646938+0000","last_undegraded":"2024-04-22T18:40:03.646938+0000","last_fullsized":"2024-04-22T18:40:03.646938+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319333,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":74883,"num_write":33,"num_write_kb":9378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"54'29","reported_seq":151,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.743269+0000","last_change":"2024-04-22T18:39:56.614792+0000","last_active":"2024-04-22T18:40:03.743269+0000","last_peered":"2024-04-22T18:40:03.743269+0000","last_clean":"2024-04-22T18:40:03.743269+0000","last_became_active":"2024-04-22T18:39:47.007675+0000","last_became_peered":"2024-04-22T18:39:47.007675+0000","last_unstale":"2024-04-22T18:40:03.743269+0000","last_undegraded":"2024-04-22T18:40:03.743269+0000","last_fullsized":"2024-04-22T18:40:03.743269+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'29","last_scrub_stamp":"2024-04-22T18:39:56.614637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:56.614637+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:22.964555+0000","scrub_duration":0.37681601799999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"51'19","reported_seq":127,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.696804+0000","last_change":"2024-04-22T18:39:51.834064+0000","last_active":"2024-04-22T18:40:03.696804+0000","last_peered":"2024-04-22T18:40:03.696804+0000","last_clean":"2024-04-22T18:40:03.696804+0000","last_became_active":"2024-04-22T18:39:51.833751+0000","last_became_peered":"2024-04-22T18:39:51.833751+0000","last_unstale":"2024-04-22T18:40:03.696804+0000","last_undegraded":"2024-04-22T18:40:03.696804+0000","last_fullsized":"2024-04-22T18:40:03.696804+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1882368,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":22802,"num_write":10,"num_write_kb":3047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1349842,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"64'69","reported_seq":222,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.646904+0000","last_change":"2024-04-22T18:40:03.558989+0000","last_active":"2024-04-22T18:40:03.646904+0000","last_peered":"2024-04-22T18:40:03.646904+0000","last_clean":"2024-04-22T18:40:03.646904+0000","last_became_active":"2024-04-22T18:40:02.745206+0000","last_became_peered":"2024-04-22T18:40:02.745206+0000","last_unstale":"2024-04-22T18:40:03.646904+0000","last_undegraded":"2024-04-22T18:40:03.646904+0000","last_fullsized":"2024-04-22T18:40:03.646904+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6134555,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":117672,"num_write":38,"num_write_kb":10746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3762988,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"58'33","reported_seq":157,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.755864+0000","last_change":"2024-04-22T18:40:03.451290+0000","last_active":"2024-04-22T18:40:03.755864+0000","last_peered":"2024-04-22T18:40:03.755864+0000","last_clean":"2024-04-22T18:40:03.755864+0000","last_became_active":"2024-04-22T18:40:02.844371+0000","last_became_peered":"2024-04-22T18:40:02.844371+0000","last_unstale":"2024-04-22T18:40:03.755864+0000","last_undegraded":"2024-04-22T18:40:03.755864+0000","last_fullsized":"2024-04-22T18:40:03.755864+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2445419,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":67666,"num_write":16,"num_write_kb":4115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":970119,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"73'113","reported_seq":317,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T18:40:05.372878+0000","last_change":"2024-04-22T18:40:04.645761+0000","last_active":"2024-04-22T18:40:05.372878+0000","last_peered":"2024-04-22T18:40:05.372878+0000","last_clean":"2024-04-22T18:40:05.372878+0000","last_became_active":"2024-04-22T18:40:04.645418+0000","last_became_peered":"2024-04-22T18:40:04.645418+0000","last_unstale":"2024-04-22T18:40:05.372878+0000","last_undegraded":"2024-04-22T18:40:05.372878+0000","last_fullsized":"2024-04-22T18:40:05.372878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10102482,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":154,"num_read_kb":160206,"num_write":62,"num_write_kb":17851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1101926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"31'5","reported_seq":97,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.746601+0000","last_change":"2024-04-22T18:39:59.265419+0000","last_active":"2024-04-22T18:40:03.746601+0000","last_peered":"2024-04-22T18:40:03.746601+0000","last_clean":"2024-04-22T18:40:03.746601+0000","last_became_active":"2024-04-22T18:39:59.265189+0000","last_became_peered":"2024-04-22T18:39:59.265189+0000","last_unstale":"2024-04-22T18:40:03.746601+0000","last_undegraded":"2024-04-22T18:40:03.746601+0000","last_fullsized":"2024-04-22T18:40:03.746601+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1159667,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":23460,"num_write":2,"num_write_kb":401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.743187+0000","last_change":"2024-04-22T18:38:39.026078+0000","last_active":"2024-04-22T18:40:03.743187+0000","last_peered":"2024-04-22T18:40:03.743187+0000","last_clean":"2024-04-22T18:40:03.743187+0000","last_became_active":"2024-04-22T18:38:39.025741+0000","last_became_peered":"2024-04-22T18:38:39.025741+0000","last_unstale":"2024-04-22T18:40:03.743187+0000","last_undegraded":"2024-04-22T18:40:03.743187+0000","last_fullsized":"2024-04-22T18:40:03.743187+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"75'120","reported_seq":364,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T18:40:04.959590+0000","last_change":"2024-04-22T18:40:04.048901+0000","last_active":"2024-04-22T18:40:04.959590+0000","last_peered":"2024-04-22T18:40:04.959590+0000","last_clean":"2024-04-22T18:40:04.959590+0000","last_became_active":"2024-04-22T18:40:02.858745+0000","last_became_peered":"2024-04-22T18:40:02.858745+0000","last_unstale":"2024-04-22T18:40:04.959590+0000","last_undegraded":"2024-04-22T18:40:04.959590+0000","last_fullsized":"2024-04-22T18:40:04.959590+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:07.943249+0000","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15546475,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":74,"num_read_kb":96954,"num_write":10,"num_write_kb":3432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9086896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":67,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.696906+0000","last_change":"2024-04-22T18:38:39.110499+0000","last_active":"2024-04-22T18:40:03.696906+0000","last_peered":"2024-04-22T18:40:03.696906+0000","last_clean":"2024-04-22T18:40:03.696906+0000","last_became_active":"2024-04-22T18:38:39.010173+0000","last_became_peered":"2024-04-22T18:38:39.010173+0000","last_unstale":"2024-04-22T18:40:03.696906+0000","last_undegraded":"2024-04-22T18:40:03.696906+0000","last_fullsized":"2024-04-22T18:40:03.696906+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"71'14","reported_seq":116,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.746536+0000","last_change":"2024-04-22T18:39:54.378188+0000","last_active":"2024-04-22T18:40:03.746536+0000","last_peered":"2024-04-22T18:40:03.746536+0000","last_clean":"2024-04-22T18:40:03.746536+0000","last_became_active":"2024-04-22T18:39:46.805569+0000","last_became_peered":"2024-04-22T18:39:46.805569+0000","last_unstale":"2024-04-22T18:40:03.746536+0000","last_undegraded":"2024-04-22T18:40:03.746536+0000","last_fullsized":"2024-04-22T18:40:03.746536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'13","last_scrub_stamp":"2024-04-22T18:39:54.377980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.377980+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:00.921987+0000","scrub_duration":0.30260881099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3378207,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":19804,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":65,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.744734+0000","last_change":"2024-04-22T18:38:39.210344+0000","last_active":"2024-04-22T18:40:03.744734+0000","last_peered":"2024-04-22T18:40:03.744734+0000","last_clean":"2024-04-22T18:40:03.744734+0000","last_became_active":"2024-04-22T18:38:39.210049+0000","last_became_peered":"2024-04-22T18:38:39.210049+0000","last_unstale":"2024-04-22T18:40:03.744734+0000","last_undegraded":"2024-04-22T18:40:03.744734+0000","last_fullsized":"2024-04-22T18:40:03.744734+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"67'69","reported_seq":234,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.764142+0000","last_change":"2024-04-22T18:40:03.763662+0000","last_active":"2024-04-22T18:40:03.764142+0000","last_peered":"2024-04-22T18:40:03.764142+0000","last_clean":"2024-04-22T18:40:03.764142+0000","last_became_active":"2024-04-22T18:40:02.739213+0000","last_became_peered":"2024-04-22T18:40:02.739213+0000","last_unstale":"2024-04-22T18:40:03.764142+0000","last_undegraded":"2024-04-22T18:40:03.764142+0000","last_fullsized":"2024-04-22T18:40:03.764142+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'67","last_scrub_stamp":"2024-04-22T18:39:53.916646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.916646+0000","objects_scrubbed":2,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:06.252157+0000","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269011,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":14760,"num_write":1,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3912084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":67,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.696874+0000","last_change":"2024-04-22T18:38:39.307735+0000","last_active":"2024-04-22T18:40:03.696874+0000","last_peered":"2024-04-22T18:40:03.696874+0000","last_clean":"2024-04-22T18:40:03.696874+0000","last_became_active":"2024-04-22T18:38:39.307459+0000","last_became_peered":"2024-04-22T18:38:39.307459+0000","last_unstale":"2024-04-22T18:40:03.696874+0000","last_undegraded":"2024-04-22T18:40:03.696874+0000","last_fullsized":"2024-04-22T18:40:03.696874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"64'38","reported_seq":158,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:04.058265+0000","last_change":"2024-04-22T18:40:04.057863+0000","last_active":"2024-04-22T18:40:04.058265+0000","last_peered":"2024-04-22T18:40:04.058265+0000","last_clean":"2024-04-22T18:40:04.058265+0000","last_became_active":"2024-04-22T18:40:02.844289+0000","last_became_peered":"2024-04-22T18:40:02.844289+0000","last_unstale":"2024-04-22T18:40:04.058265+0000","last_undegraded":"2024-04-22T18:40:04.058265+0000","last_fullsized":"2024-04-22T18:40:04.058265+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2932821,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":72433,"num_write":20,"num_write_kb":5595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2843709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"75'230","reported_seq":298,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T18:40:05.378574+0000","last_change":"2024-04-22T18:38:47.173101+0000","last_active":"2024-04-22T18:40:05.378574+0000","last_peered":"2024-04-22T18:40:05.378574+0000","last_clean":"2024-04-22T18:40:05.378574+0000","last_became_active":"2024-04-22T18:38:47.172816+0000","last_became_peered":"2024-04-22T18:38:47.172816+0000","last_unstale":"2024-04-22T18:40:05.378574+0000","last_undegraded":"2024-04-22T18:40:05.378574+0000","last_fullsized":"2024-04-22T18:40:05.378574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:12.128375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23831923,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":192,"num_read_kb":2602,"num_write":271,"num_write_kb":74409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T18:40:04.847509+0000","last_change":"2024-04-22T18:40:03.245549+0000","last_active":"2024-04-22T18:40:04.847509+0000","last_peered":"2024-04-22T18:40:04.847509+0000","last_clean":"2024-04-22T18:40:04.847509+0000","last_became_active":"2024-04-22T18:40:02.945025+0000","last_became_peered":"2024-04-22T18:40:02.945025+0000","last_unstale":"2024-04-22T18:40:04.847509+0000","last_undegraded":"2024-04-22T18:40:04.847509+0000","last_fullsized":"2024-04-22T18:40:04.847509+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":146,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:04.846053+0000","last_change":"2024-04-22T18:39:52.361604+0000","last_active":"2024-04-22T18:40:04.846053+0000","last_peered":"2024-04-22T18:40:04.846053+0000","last_clean":"2024-04-22T18:40:04.846053+0000","last_became_active":"2024-04-22T18:39:51.943071+0000","last_became_peered":"2024-04-22T18:39:51.943071+0000","last_unstale":"2024-04-22T18:40:04.846053+0000","last_undegraded":"2024-04-22T18:40:04.846053+0000","last_fullsized":"2024-04-22T18:40:04.846053+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1949628,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":38157,"num_write":19,"num_write_kb":6188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":3820686,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"75'151","reported_seq":425,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T18:40:05.375373+0000","last_change":"2024-04-22T18:39:56.824261+0000","last_active":"2024-04-22T18:40:05.375373+0000","last_peered":"2024-04-22T18:40:05.375373+0000","last_clean":"2024-04-22T18:40:05.375373+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:40:05.375373+0000","last_undegraded":"2024-04-22T18:40:05.375373+0000","last_fullsized":"2024-04-22T18:40:05.375373+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243172,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":124,"num_read_kb":826,"num_write":184,"num_write_kb":46912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":75,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T18:40:04.849143+0000","last_change":"2024-04-22T18:40:05.068266+0000","last_active":"2024-04-22T18:40:04.849143+0000","last_peered":"2024-04-22T18:40:04.849143+0000","last_clean":"2024-04-22T18:40:04.849143+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:40:04.849143+0000","last_undegraded":"2024-04-22T18:40:04.849143+0000","last_fullsized":"2024-04-22T18:40:04.849143+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"66'39","reported_seq":186,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:04.508992+0000","last_change":"2024-04-22T18:40:03.361557+0000","last_active":"2024-04-22T18:40:04.508992+0000","last_peered":"2024-04-22T18:40:04.508992+0000","last_clean":"2024-04-22T18:40:04.508992+0000","last_became_active":"2024-04-22T18:40:02.744089+0000","last_became_peered":"2024-04-22T18:40:02.744089+0000","last_unstale":"2024-04-22T18:40:04.508992+0000","last_undegraded":"2024-04-22T18:40:04.508992+0000","last_fullsized":"2024-04-22T18:40:04.508992+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:16.501156+0000","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3630312,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":16936,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2812934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"75'180","reported_seq":252,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T18:40:04.641867+0000","last_change":"2024-04-22T18:38:47.173337+0000","last_active":"2024-04-22T18:40:04.641867+0000","last_peered":"2024-04-22T18:40:04.641867+0000","last_clean":"2024-04-22T18:40:04.641867+0000","last_became_active":"2024-04-22T18:38:47.173065+0000","last_became_peered":"2024-04-22T18:38:47.173065+0000","last_unstale":"2024-04-22T18:40:04.641867+0000","last_undegraded":"2024-04-22T18:40:04.641867+0000","last_fullsized":"2024-04-22T18:40:04.641867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26065629,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":156,"num_read_kb":4368,"num_write":217,"num_write_kb":58593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.744791+0000","last_change":"2024-04-22T18:38:39.125885+0000","last_active":"2024-04-22T18:40:03.744791+0000","last_peered":"2024-04-22T18:40:03.744791+0000","last_clean":"2024-04-22T18:40:03.744791+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:40:03.744791+0000","last_undegraded":"2024-04-22T18:40:03.744791+0000","last_fullsized":"2024-04-22T18:40:03.744791+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'65","reported_seq":210,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T18:40:05.198432+0000","last_change":"2024-04-22T18:39:46.810152+0000","last_active":"2024-04-22T18:40:05.198432+0000","last_peered":"2024-04-22T18:40:05.198432+0000","last_clean":"2024-04-22T18:40:05.198432+0000","last_became_active":"2024-04-22T18:39:46.809886+0000","last_became_peered":"2024-04-22T18:39:46.809886+0000","last_unstale":"2024-04-22T18:40:05.198432+0000","last_undegraded":"2024-04-22T18:40:05.198432+0000","last_fullsized":"2024-04-22T18:40:05.198432+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5983292,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":110202,"num_write":37,"num_write_kb":11076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"75'182","reported_seq":335,"reported_epoch":75,"state":"active+recovering+remapped","last_fresh":"2024-04-22T18:40:06.107704+0000","last_change":"2024-04-22T18:40:04.352637+0000","last_active":"2024-04-22T18:40:06.107704+0000","last_peered":"2024-04-22T18:40:06.107704+0000","last_clean":"2024-04-22T18:40:01.536496+0000","last_became_active":"2024-04-22T18:40:03.846259+0000","last_became_peered":"2024-04-22T18:40:03.846259+0000","last_unstale":"2024-04-22T18:40:06.107704+0000","last_undegraded":"2024-04-22T18:40:06.107704+0000","last_fullsized":"2024-04-22T18:40:06.107704+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23235544,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":162,"num_read_kb":2156,"num_write":226,"num_write_kb":52487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":9986235,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[3,6],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":9},{"shards":"3,6","objects":21}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":78,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T18:40:04.849160+0000","last_change":"2024-04-22T18:40:02.742106+0000","last_active":"2024-04-22T18:40:04.849160+0000","last_peered":"2024-04-22T18:40:04.849160+0000","last_clean":"2024-04-22T18:40:04.849160+0000","last_became_active":"2024-04-22T18:40:02.741862+0000","last_became_peered":"2024-04-22T18:40:02.741862+0000","last_unstale":"2024-04-22T18:40:04.849160+0000","last_undegraded":"2024-04-22T18:40:04.849160+0000","last_fullsized":"2024-04-22T18:40:04.849160+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":227,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.744766+0000","last_change":"2024-04-22T18:39:53.346906+0000","last_active":"2024-04-22T18:40:03.744766+0000","last_peered":"2024-04-22T18:40:03.744766+0000","last_clean":"2024-04-22T18:40:03.744766+0000","last_became_active":"2024-04-22T18:38:38.108568+0000","last_became_peered":"2024-04-22T18:38:38.108568+0000","last_unstale":"2024-04-22T18:40:03.744766+0000","last_undegraded":"2024-04-22T18:40:03.744766+0000","last_fullsized":"2024-04-22T18:40:03.744766+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:39:53.346787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.346787+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:17.891739+0000","scrub_duration":0.20273052499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'55","reported_seq":187,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T18:40:04.860674+0000","last_change":"2024-04-22T18:39:55.746188+0000","last_active":"2024-04-22T18:40:04.860674+0000","last_peered":"2024-04-22T18:40:04.860674+0000","last_clean":"2024-04-22T18:40:04.860674+0000","last_became_active":"2024-04-22T18:39:47.108040+0000","last_became_peered":"2024-04-22T18:39:47.108040+0000","last_unstale":"2024-04-22T18:40:04.860674+0000","last_undegraded":"2024-04-22T18:40:04.860674+0000","last_fullsized":"2024-04-22T18:40:04.860674+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'38","last_scrub_stamp":"2024-04-22T18:39:55.746067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:55.746067+0000","objects_scrubbed":1,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:04.875560+0000","scrub_duration":0.60877772299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7613571,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":17914,"num_write":21,"num_write_kb":6447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"62'51","reported_seq":181,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T18:40:05.263306+0000","last_change":"2024-04-22T18:40:06.087720+0000","last_active":"2024-04-22T18:40:05.263306+0000","last_peered":"2024-04-22T18:40:05.263306+0000","last_clean":"2024-04-22T18:40:05.263306+0000","last_became_active":"2024-04-22T18:40:02.645205+0000","last_became_peered":"2024-04-22T18:40:02.645205+0000","last_unstale":"2024-04-22T18:40:05.263306+0000","last_undegraded":"2024-04-22T18:40:05.263306+0000","last_fullsized":"2024-04-22T18:40:05.263306+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4832012,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":92005,"num_write":29,"num_write_kb":8835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2522610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'91","reported_seq":306,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T18:40:05.576764+0000","last_change":"2024-04-22T18:40:05.576934+0000","last_active":"2024-04-22T18:40:05.576764+0000","last_peered":"2024-04-22T18:40:05.576764+0000","last_clean":"2024-04-22T18:40:05.576764+0000","last_became_active":"2024-04-22T18:39:46.807956+0000","last_became_peered":"2024-04-22T18:39:46.807956+0000","last_unstale":"2024-04-22T18:40:05.576764+0000","last_undegraded":"2024-04-22T18:40:05.576764+0000","last_fullsized":"2024-04-22T18:40:05.576764+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:11.922250+0000","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9748060,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"63'55","reported_seq":191,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.943581+0000","last_change":"2024-04-22T18:38:48.372952+0000","last_active":"2024-04-22T18:40:03.943581+0000","last_peered":"2024-04-22T18:40:03.943581+0000","last_clean":"2024-04-22T18:40:03.943581+0000","last_became_active":"2024-04-22T18:38:48.277922+0000","last_became_peered":"2024-04-22T18:38:48.277922+0000","last_unstale":"2024-04-22T18:40:03.943581+0000","last_undegraded":"2024-04-22T18:40:03.943581+0000","last_fullsized":"2024-04-22T18:40:03.943581+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:15.471951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9277464,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":174,"num_read_kb":204065,"num_write":52,"num_write_kb":13610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"70'91","reported_seq":289,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T18:40:04.641952+0000","last_change":"2024-04-22T18:38:48.269362+0000","last_active":"2024-04-22T18:40:04.641952+0000","last_peered":"2024-04-22T18:40:04.641952+0000","last_clean":"2024-04-22T18:40:04.641952+0000","last_became_active":"2024-04-22T18:38:48.269066+0000","last_became_peered":"2024-04-22T18:38:48.269066+0000","last_unstale":"2024-04-22T18:40:04.641952+0000","last_undegraded":"2024-04-22T18:40:04.641952+0000","last_fullsized":"2024-04-22T18:40:04.641952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11714446,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":311,"num_read_kb":338042,"num_write":91,"num_write_kb":27272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"75'86","reported_seq":273,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T18:40:05.401010+0000","last_change":"2024-04-22T18:38:48.274538+0000","last_active":"2024-04-22T18:40:05.401010+0000","last_peered":"2024-04-22T18:40:05.401010+0000","last_clean":"2024-04-22T18:40:05.401010+0000","last_became_active":"2024-04-22T18:38:48.174204+0000","last_became_peered":"2024-04-22T18:38:48.174204+0000","last_unstale":"2024-04-22T18:40:05.401010+0000","last_undegraded":"2024-04-22T18:40:05.401010+0000","last_fullsized":"2024-04-22T18:40:05.401010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:11.823691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13445600,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":315,"num_read_kb":364124,"num_write":88,"num_write_kb":25553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"66'40","reported_seq":185,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:04.529351+0000","last_change":"2024-04-22T18:39:52.257129+0000","last_active":"2024-04-22T18:40:04.529351+0000","last_peered":"2024-04-22T18:40:04.529351+0000","last_clean":"2024-04-22T18:40:04.529351+0000","last_became_active":"2024-04-22T18:39:51.934575+0000","last_became_peered":"2024-04-22T18:39:51.934575+0000","last_unstale":"2024-04-22T18:40:04.529351+0000","last_undegraded":"2024-04-22T18:40:04.529351+0000","last_fullsized":"2024-04-22T18:40:04.529351+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:08.036615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7025326,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":132,"num_read_kb":135577,"num_write":41,"num_write_kb":12263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":6895340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"70'16","reported_seq":103,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:03.642553+0000","last_change":"2024-04-22T18:38:48.075222+0000","last_active":"2024-04-22T18:40:03.642553+0000","last_peered":"2024-04-22T18:40:03.642553+0000","last_clean":"2024-04-22T18:40:03.642553+0000","last_became_active":"2024-04-22T18:38:48.074901+0000","last_became_peered":"2024-04-22T18:38:48.074901+0000","last_unstale":"2024-04-22T18:40:03.642553+0000","last_undegraded":"2024-04-22T18:40:03.642553+0000","last_fullsized":"2024-04-22T18:40:03.642553+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6437048,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":63372,"num_write":16,"num_write_kb":5550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"66'53","reported_seq":174,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-22T18:40:04.408762+0000","last_change":"2024-04-22T18:38:48.173640+0000","last_active":"2024-04-22T18:40:04.408762+0000","last_peered":"2024-04-22T18:40:04.408762+0000","last_clean":"2024-04-22T18:40:04.408762+0000","last_became_active":"2024-04-22T18:38:48.173369+0000","last_became_peered":"2024-04-22T18:38:48.173369+0000","last_unstale":"2024-04-22T18:40:04.408762+0000","last_undegraded":"2024-04-22T18:40:04.408762+0000","last_fullsized":"2024-04-22T18:40:04.408762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:08.819198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9529569,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":135,"num_read_kb":123812,"num_write":53,"num_write_kb":14262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":152592130,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":2057,"num_read_kb":2327941,"num_write":685,"num_write_kb":198186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":41607972,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":32,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215867392,"data_stored":215211380,"data_compressed":409038,"data_compressed_allocated":442368,"data_compressed_original":896198,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":52,"acting":52,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":92376268,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":634,"num_read_kb":9952,"num_write":898,"num_write_kb":232401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":30845559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":115843072,"data_stored":124191533,"data_compressed":7515406,"data_compressed_allocated":7991296,"data_compressed_original":16662528,"omap_allocated":0,"internal_metadata":0},"log_size":743,"ondisk_log_size":743,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50952,"kb_used_data":19832,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94320888,"statfs":{"total":96636764160,"available":96584589312,"internally_reserved":0,"allocated":20307968,"data_stored":20205929,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.376},{"interface":"front","average":{"1min":4.508,"5min":4.508,"15min":4.508},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":85.542000000000002,"5min":85.542000000000002,"15min":85.542000000000002},"last":0.45400000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.8650000000000002,"5min":4.8650000000000002,"15min":4.8650000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":79.927999999999997,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.41999999999999998},{"interface":"front","average":{"1min":13.048,"5min":13.048,"15min":13.048},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":99.388999999999996,"5min":99.388999999999996,"15min":99.388999999999996},"last":0.433}]},{"osd":2,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.085,"5min":4.085,"15min":4.085},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":77.320999999999998,"5min":77.320999999999998,"15min":77.320999999999998},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.46300000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.1509999999999998,"5min":4.1509999999999998,"15min":4.1509999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":77.361000000000004,"5min":77.361000000000004,"15min":77.361000000000004},"last":0.442},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.48099999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.4589999999999996,"5min":4.4589999999999996,"15min":4.4589999999999996},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":85.509,"5min":85.509,"15min":85.509},"last":0.38500000000000001},{"interface":"front","average":{"1min":22.872,"5min":22.872,"15min":22.872},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":327.77300000000002,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.311}]},{"osd":5,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":3.0710000000000002,"5min":3.0710000000000002,"15min":3.0710000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":55.116999999999997,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.29599999999999999},{"interface":"front","average":{"1min":7.1619999999999999,"5min":7.1619999999999999,"15min":7.1619999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":85.475999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.41099999999999998}]},{"osd":6,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":13.276,"5min":13.276,"15min":13.276},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":99.620999999999995,"5min":99.620999999999995,"15min":99.620999999999995},"last":0.32500000000000001},{"interface":"front","average":{"1min":4.133,"5min":4.133,"15min":4.133},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":77.405000000000001,"5min":77.405000000000001,"15min":77.405000000000001},"last":0.34599999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65412,"kb_used_data":36784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94306428,"statfs":{"total":96636764160,"available":96569782272,"internally_reserved":0,"allocated":37666816,"data_stored":37988987,"data_compressed":694545,"data_compressed_allocated":745472,"data_compressed_original":1720320,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.794,"15min":1.794},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":32.097999999999999,"5min":32.097999999999999,"15min":32.097999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":25.808,"5min":25.808,"15min":25.808},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":166.85499999999999,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":37.161999999999999,"5min":37.161999999999999,"15min":37.161999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":10.092000000000001,"5min":10.092000000000001,"15min":10.092000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":222.55199999999999,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.38400000000000001},{"interface":"front","average":{"1min":18.803999999999998,"5min":18.803999999999998,"15min":18.803999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":166.81800000000001,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.39700000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":21.518999999999998,"5min":21.518999999999998,"15min":21.518999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":171.99100000000001,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":4.6289999999999996}]},{"osd":4,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":7.6029999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":81.697000000000003,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":19.327999999999999,"5min":19.327999999999999,"15min":19.327999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":166.733,"5min":166.733,"15min":166.733},"last":0.441}]},{"osd":5,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.43099999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":18.850999999999999,"5min":18.850999999999999,"15min":18.850999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":166.79400000000001,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.499},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.45200000000000001}]}]},{"osd":5,"up_from":0,"seq":55834574868,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107824,"kb_used_data":73712,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94264016,"statfs":{"total":96636764160,"available":96526352384,"internally_reserved":0,"allocated":75481088,"data_stored":76877102,"data_compressed":1685288,"data_compressed_allocated":1794048,"data_compressed_original":3892835,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":16.648,"5min":16.648,"15min":16.648},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":86.712999999999994,"5min":86.712999999999994,"15min":86.712999999999994},"last":0.52700000000000002},{"interface":"front","average":{"1min":8.5679999999999996,"5min":8.5679999999999996,"15min":8.5679999999999996},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":78.158000000000001,"5min":78.158000000000001,"15min":78.158000000000001},"last":0.51800000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":96.486999999999995},{"interface":"front","average":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":78.866,"5min":78.866,"15min":78.866},"last":0.46899999999999997}]},{"osd":2,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":8.5700000000000003,"5min":8.5700000000000003,"15min":8.5700000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":76.724999999999994,"5min":76.724999999999994,"15min":76.724999999999994},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":96.471999999999994}]},{"osd":3,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":4.0220000000000002,"5min":4.0220000000000002,"15min":4.0220000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":78.912000000000006,"5min":78.912000000000006,"15min":78.912000000000006},"last":0.505},{"interface":"front","average":{"1min":20.968,"5min":20.968,"15min":20.968},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":227.41200000000001,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.57399999999999995}]},{"osd":5,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":96.408000000000001},{"interface":"front","average":{"1min":4.0380000000000003,"5min":4.0380000000000003,"15min":4.0380000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":78.811999999999998,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.36699999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":13.16,"5min":13.16,"15min":13.16},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":124.303,"5min":124.303,"15min":124.303},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":96.438000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":8.5600000000000005,"5min":8.5600000000000005,"15min":8.5600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":76.697000000000003,"5min":76.697000000000003,"15min":76.697000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":96.456999999999994}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61148,"kb_used_data":30028,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94310692,"statfs":{"total":96636764160,"available":96574148608,"internally_reserved":0,"allocated":30748672,"data_stored":31098721,"data_compressed":780309,"data_compressed_allocated":835584,"data_compressed_original":1785856,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":12.199999999999999,"5min":12.199999999999999,"15min":12.199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":98.957999999999998,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.24399999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":5.0449999999999999,"5min":5.0449999999999999,"15min":5.0449999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":94.063000000000002,"5min":94.063000000000002,"15min":94.063000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.28599999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":12.901999999999999,"5min":12.901999999999999,"15min":12.901999999999999},"last":0.221},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.41799999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":4.1079999999999997,"5min":4.1079999999999997,"15min":4.1079999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":71.998000000000005,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.32300000000000001},{"interface":"front","average":{"1min":4.1379999999999999,"5min":4.1379999999999999,"15min":4.1379999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":72.227000000000004,"5min":72.227000000000004,"15min":72.227000000000004},"last":0.433}]},{"osd":5,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":13.94,"5min":13.94,"15min":13.94},"last":0.38100000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":13.965,"5min":13.965,"15min":13.965},"last":0.441}]},{"osd":6,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.40600000000000003}]},{"osd":7,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.373}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47388,"kb_used_data":19008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324452,"statfs":{"total":96636764160,"available":96588238848,"internally_reserved":0,"allocated":19464192,"data_stored":19972343,"data_compressed":1019561,"data_compressed_allocated":1081344,"data_compressed_original":2080768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":149.03,"5min":149.03,"15min":149.03},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":31.866,"5min":31.866,"15min":31.866},"last":0.27600000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":13.868,"5min":13.868,"15min":13.868},"last":52.121000000000002},{"interface":"front","average":{"1min":14.885,"5min":14.885,"15min":14.885},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":160.37100000000001,"5min":160.37100000000001,"15min":160.37100000000001},"last":0.23899999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":147.721},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.28899999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":18.771999999999998,"5min":18.771999999999998,"15min":18.771999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":160.584,"5min":160.584,"15min":160.584},"last":52.683},{"interface":"front","average":{"1min":2.448,"5min":2.448,"15min":2.448},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":36.841000000000001,"5min":36.841000000000001,"15min":36.841000000000001},"last":0.437}]},{"osd":5,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":3.052,"5min":3.052,"15min":3.052},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":48.631,"5min":48.631,"15min":48.631},"last":0.42799999999999999},{"interface":"front","average":{"1min":3.476,"5min":3.476,"15min":3.476},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":56.18,"5min":56.18,"15min":56.18},"last":0.49099999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.51400000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.503}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86460,"kb_used_data":52600,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94285380,"statfs":{"total":96636764160,"available":96548229120,"internally_reserved":0,"allocated":53862400,"data_stored":54204332,"data_compressed":1078602,"data_compressed_allocated":1142784,"data_compressed_original":2205283,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":7.859,"5min":7.859,"15min":7.859},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":88.171000000000006,"5min":88.171000000000006,"15min":88.171000000000006},"last":66.480999999999995},{"interface":"front","average":{"1min":7.8630000000000004,"5min":7.8630000000000004,"15min":7.8630000000000004},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":88.156000000000006,"5min":88.156000000000006,"15min":88.156000000000006},"last":66.498000000000005}]},{"osd":2,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":8.5999999999999996,"5min":8.5999999999999996,"15min":8.5999999999999996},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":183.04400000000001,"5min":183.04400000000001,"15min":183.04400000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.377}]},{"osd":3,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":14.492000000000001,"5min":14.492000000000001,"15min":14.492000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":231.864,"5min":231.864,"15min":231.864},"last":0.39500000000000002}]},{"osd":4,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":12.984,"5min":12.984,"15min":12.984},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":192.42099999999999,"5min":192.42099999999999,"15min":192.42099999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.501}]},{"osd":5,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":14.071,"5min":14.071,"15min":14.071},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":138.089,"5min":138.089,"15min":138.089},"last":0.54000000000000004},{"interface":"front","average":{"1min":20.077000000000002,"5min":20.077000000000002,"15min":20.077000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":183.08199999999999,"5min":183.08199999999999,"15min":183.08199999999999},"last":0.56200000000000006}]},{"osd":6,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":51.070999999999998,"5min":51.070999999999998,"15min":51.070999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":16.532,"5min":16.532,"15min":16.532},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":183.10900000000001,"5min":183.10900000000001,"15min":183.10900000000001},"last":0.58599999999999997}]},{"osd":7,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.59099999999999997},{"interface":"front","average":{"1min":3.0390000000000001,"5min":3.0390000000000001,"15min":3.0390000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":51.097000000000001,"5min":51.097000000000001,"15min":51.097000000000001},"last":0.61299999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82864,"kb_used_data":51748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94288976,"statfs":{"total":96636764160,"available":96551911424,"internally_reserved":0,"allocated":52989952,"data_stored":54800345,"data_compressed":1757583,"data_compressed_allocated":1875968,"data_compressed_original":4341760,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":6.7599999999999998,"5min":6.7599999999999998,"15min":6.7599999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":133.09899999999999,"5min":133.09899999999999,"15min":133.09899999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":25.827999999999999,"5min":25.827999999999999,"15min":25.827999999999999},"last":0.439}]},{"osd":2,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":25.873000000000001,"5min":25.873000000000001,"15min":25.873000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.318,"5min":0.318,"15min":0.318},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.47799999999999998}]},{"osd":3,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":25.898,"5min":25.898,"15min":25.898},"last":0.61099999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":9.9120000000000008,"5min":9.9120000000000008,"15min":9.9120000000000008},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":133.036,"5min":133.036,"15min":133.036},"last":0.70399999999999996}]},{"osd":5,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":26.16,"5min":26.16,"15min":26.16},"last":0.67800000000000005},{"interface":"front","average":{"1min":13.955,"5min":13.955,"15min":13.955},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":282.197,"5min":282.197,"15min":282.197},"last":0.71899999999999997}]},{"osd":6,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":3.198,"5min":3.198,"15min":3.198},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.748,"5min":1.748,"15min":1.748},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":26.190000000000001,"5min":26.190000000000001,"15min":26.190000000000001},"last":52.762}]},{"osd":7,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":4.6779999999999999,"5min":4.6779999999999999,"15min":4.6779999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":87.209999999999994,"5min":87.209999999999994,"15min":87.209999999999994},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.2440000000000002,"5min":3.2440000000000002,"15min":3.2440000000000002},"last":0.74099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31186944,"data_stored":33049851,"data_compressed":1604392,"data_compressed_allocated":1703936,"data_compressed_original":3637248,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11223040,"data_stored":12165933,"data_compressed":989631,"data_compressed_allocated":1048576,"data_compressed_original":2015232,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11223040,"data_stored":12165933,"data_compressed":989631,"data_compressed_allocated":1048576,"data_compressed_original":2015232,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10956800,"data_stored":11886865,"data_compressed":816986,"data_compressed_allocated":872448,"data_compressed_original":1851392,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":31186944,"data_stored":33049851,"data_compressed":1604392,"data_compressed_allocated":1703936,"data_compressed_original":3637248,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12304384,"data_stored":13295755,"data_compressed":877720,"data_compressed_allocated":937984,"data_compressed_original":1982464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":7761920,"data_stored":8577345,"data_compressed":632654,"data_compressed_allocated":675840,"data_compressed_original":1523712,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20914176,"data_stored":20864898,"data_compressed":123623,"data_compressed_allocated":131072,"data_compressed_original":192512,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41807872,"data_stored":41682224,"data_compressed":88971,"data_compressed_allocated":94208,"data_compressed_original":190051,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7086080,"data_stored":7046083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20754432,"data_stored":20648760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":43511808,"data_stored":43474993,"data_compressed":80896,"data_compressed_allocated":90112,"data_compressed_original":255587,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33280000,"data_stored":33118894,"data_compressed":53657,"data_compressed_allocated":57344,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":29077504,"data_stored":29054709,"data_compressed":61891,"data_compressed_allocated":69632,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":19435520,"data_stored":19320819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:40:07.397 INFO:tasks.thrashosds.thrasher:Setting 4.7 to [4, 1, 7, 3] 2024-04-22T18:40:07.397 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.7', '4', '1', '7', '3'] 2024-04-22T18:40:07.397 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 4.7 4 1 7 3 2024-04-22T18:40:07.415 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:07.416+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:07.432 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:07.432 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:07.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:07.432 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:07.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.432 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:22 4.a 4:5aa97f94:::smithi08937929-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:07.432 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:40:07.432 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.274842039999996, 2024-04-22T18:40:07.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30948398900000001, 2024-04-22T18:40:07.432 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.432 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.433 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.433 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.433 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.433 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 22 2024-04-22T18:40:07.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.433 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:40:07.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:40:07.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157259+0000", 2024-04-22T18:40:07.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:07.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157318+0000", 2024-04-22T18:40:07.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8906000000000001e-05 2024-04-22T18:40:07.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157327+0000", 2024-04-22T18:40:07.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:40:07.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157332+0000", 2024-04-22T18:40:07.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4689999999999996e-06 2024-04-22T18:40:07.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257479+0000", 2024-04-22T18:40:07.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014700999999999 2024-04-22T18:40:07.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:07.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257492+0000", 2024-04-22T18:40:07.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2574e-05 2024-04-22T18:40:07.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259582+0000", 2024-04-22T18:40:07.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020904019999999999 2024-04-22T18:40:07.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259668+0000", 2024-04-22T18:40:07.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5826000000000001e-05 2024-04-22T18:40:07.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:07.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259772+0000", 2024-04-22T18:40:07.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104048 2024-04-22T18:40:07.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363453+0000", 2024-04-22T18:40:07.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103680183 2024-04-22T18:40:07.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466692+0000", 2024-04-22T18:40:07.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10323948199999999 2024-04-22T18:40:07.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466720+0000", 2024-04-22T18:40:07.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7886e-05 2024-04-22T18:40:07.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466744+0000", 2024-04-22T18:40:07.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4049e-05 2024-04-22T18:40:07.438 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.438 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.438 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.439 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:23 4.a 4:5aa97f94:::smithi08937929-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:07.439 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:40:07.439 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.274764352999995, 2024-04-22T18:40:07.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39003820300000003, 2024-04-22T18:40:07.439 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.439 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.439 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 23 2024-04-22T18:40:07.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.439 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:40:07.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:40:07.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157337+0000", 2024-04-22T18:40:07.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:07.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157340+0000", 2024-04-22T18:40:07.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8609999999999998e-06 2024-04-22T18:40:07.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157343+0000", 2024-04-22T18:40:07.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9859999999999999e-06 2024-04-22T18:40:07.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157346+0000", 2024-04-22T18:40:07.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8119999999999999e-06 2024-04-22T18:40:07.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257563+0000", 2024-04-22T18:40:07.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100217497 2024-04-22T18:40:07.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:07.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257574+0000", 2024-04-22T18:40:07.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0777e-05 2024-04-22T18:40:07.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260112+0000", 2024-04-22T18:40:07.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002537681 2024-04-22T18:40:07.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260191+0000", 2024-04-22T18:40:07.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9320999999999995e-05 2024-04-22T18:40:07.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:07.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260333+0000", 2024-04-22T18:40:07.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014177699999999999 2024-04-22T18:40:07.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363792+0000", 2024-04-22T18:40:07.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103458862 2024-04-22T18:40:07.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547303+0000", 2024-04-22T18:40:07.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18351162100000001 2024-04-22T18:40:07.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547337+0000", 2024-04-22T18:40:07.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3640999999999999e-05 2024-04-22T18:40:07.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547376+0000", 2024-04-22T18:40:07.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8915000000000003e-05 2024-04-22T18:40:07.445 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.445 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.445 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.445 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:24 4.a 4:5aa97f94:::smithi08937929-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:07.445 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:07.445 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.274737922, 2024-04-22T18:40:07.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39017823499999998, 2024-04-22T18:40:07.445 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.445 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:07.446 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.446 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.446 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.446 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 24 2024-04-22T18:40:07.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.446 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:07.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:07.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:07.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:07.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157366+0000", 2024-04-22T18:40:07.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3120000000000001e-06 2024-04-22T18:40:07.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157370+0000", 2024-04-22T18:40:07.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1300000000000003e-06 2024-04-22T18:40:07.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157373+0000", 2024-04-22T18:40:07.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-04-22T18:40:07.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260553+0000", 2024-04-22T18:40:07.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103179996 2024-04-22T18:40:07.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:07.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260625+0000", 2024-04-22T18:40:07.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2236000000000004e-05 2024-04-22T18:40:07.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547414+0000", 2024-04-22T18:40:07.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.286789181 2024-04-22T18:40:07.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547471+0000", 2024-04-22T18:40:07.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6192e-05 2024-04-22T18:40:07.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547542+0000", 2024-04-22T18:40:07.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1622e-05 2024-04-22T18:40:07.450 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.450 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.450 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.450 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:49 4.a 4:5d2aee44:::smithi08937929-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:07.450 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:40:07.450 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.197939263999999, 2024-04-22T18:40:07.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31344442500000003, 2024-04-22T18:40:07.450 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.451 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.451 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 49 2024-04-22T18:40:07.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.451 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:40:07.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:40:07.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234158+0000", 2024-04-22T18:40:07.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999957 2024-04-22T18:40:07.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234901+0000", 2024-04-22T18:40:07.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074250800000000001 2024-04-22T18:40:07.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234914+0000", 2024-04-22T18:40:07.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3244999999999999e-05 2024-04-22T18:40:07.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234922+0000", 2024-04-22T18:40:07.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4590000000000001e-06 2024-04-22T18:40:07.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260665+0000", 2024-04-22T18:40:07.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025743423000000001 2024-04-22T18:40:07.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260779+0000", 2024-04-22T18:40:07.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114063 2024-04-22T18:40:07.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:07.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260913+0000", 2024-04-22T18:40:07.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013355499999999999 2024-04-22T18:40:07.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363798+0000", 2024-04-22T18:40:07.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102885003 2024-04-22T18:40:07.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547567+0000", 2024-04-22T18:40:07.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.183768971 2024-04-22T18:40:07.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547589+0000", 2024-04-22T18:40:07.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2507999999999999e-05 2024-04-22T18:40:07.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547607+0000", 2024-04-22T18:40:07.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8009999999999999e-05 2024-04-22T18:40:07.456 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.456 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.456 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.456 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:50 4.a 4:5d2aee44:::smithi08937929-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:07.456 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:07.456 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.192493084999995, 2024-04-22T18:40:07.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40856295399999998, 2024-04-22T18:40:07.456 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.456 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.456 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.456 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.457 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.457 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 50 2024-04-22T18:40:07.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.457 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:07.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:07.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239607+0000", 2024-04-22T18:40:07.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:07.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240309+0000", 2024-04-22T18:40:07.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070201599999999997 2024-04-22T18:40:07.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240320+0000", 2024-04-22T18:40:07.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1024e-05 2024-04-22T18:40:07.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240327+0000", 2024-04-22T18:40:07.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0049999999999998e-06 2024-04-22T18:40:07.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362065+0000", 2024-04-22T18:40:07.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.121737812 2024-04-22T18:40:07.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362121+0000", 2024-04-22T18:40:07.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5868000000000001e-05 2024-04-22T18:40:07.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:07.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362210+0000", 2024-04-22T18:40:07.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8533000000000001e-05 2024-04-22T18:40:07.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365322+0000", 2024-04-22T18:40:07.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031121220000000002 2024-04-22T18:40:07.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648054+0000", 2024-04-22T18:40:07.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.282732549 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648105+0000", 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1276999999999997e-05 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648172+0000", 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6398999999999999e-05 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.461 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:51 4.a 4:5d2aee44:::smithi08937929-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:07.462 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:07.462 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.191770551000005, 2024-04-22T18:40:07.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40793057199999999, 2024-04-22T18:40:07.462 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.462 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.462 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 51 2024-04-22T18:40:07.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.462 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240336+0000", 2024-04-22T18:40:07.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:40:07.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240339+0000", 2024-04-22T18:40:07.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9009999999999998e-06 2024-04-22T18:40:07.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240342+0000", 2024-04-22T18:40:07.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:40:07.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363080+0000", 2024-04-22T18:40:07.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122738075 2024-04-22T18:40:07.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363164+0000", 2024-04-22T18:40:07.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3722999999999998e-05 2024-04-22T18:40:07.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:07.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363287+0000", 2024-04-22T18:40:07.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012301099999999999 2024-04-22T18:40:07.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466753+0000", 2024-04-22T18:40:07.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103466217 2024-04-22T18:40:07.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648205+0000", 2024-04-22T18:40:07.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.181451788 2024-04-22T18:40:07.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648231+0000", 2024-04-22T18:40:07.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6597999999999998e-05 2024-04-22T18:40:07.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648262+0000", 2024-04-22T18:40:07.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0871000000000003e-05 2024-04-22T18:40:07.467 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.467 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.467 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.467 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:52 4.a 4:5d2aee44:::smithi08937929-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:07.468 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:07.468 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.191758092000001, 2024-04-22T18:40:07.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40812731200000002, 2024-04-22T18:40:07.468 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.468 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:07.468 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.468 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.468 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.468 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 52 2024-04-22T18:40:07.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.468 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:07.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:07.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:07.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:07.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240345+0000", 2024-04-22T18:40:07.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4300000000000001e-06 2024-04-22T18:40:07.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240349+0000", 2024-04-22T18:40:07.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2570000000000001e-06 2024-04-22T18:40:07.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240358+0000", 2024-04-22T18:40:07.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-22T18:40:07.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363697+0000", 2024-04-22T18:40:07.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12333965299999999 2024-04-22T18:40:07.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:07.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363777+0000", 2024-04-22T18:40:07.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9883999999999996e-05 2024-04-22T18:40:07.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648296+0000", 2024-04-22T18:40:07.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451861299999998 2024-04-22T18:40:07.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648379+0000", 2024-04-22T18:40:07.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3302999999999998e-05 2024-04-22T18:40:07.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648471+0000", 2024-04-22T18:40:07.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2029999999999998e-05 2024-04-22T18:40:07.472 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.472 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.472 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.472 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:07.473 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:07.473 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.172454756999997, 2024-04-22T18:40:07.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:40:07.473 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.473 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.473 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.473 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.473 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.473 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:40:07.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.473 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:07.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:07.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:40:07.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:07.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:40:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:40:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:40:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:40:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:40:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:40:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:40:07.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:40:07.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:40:07.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:40:07.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:07.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:40:07.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:40:07.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:40:07.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:40:07.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:40:07.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:40:07.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:40:07.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:40:07.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:40:07.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:40:07.478 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.478 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.478 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.478 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:07.478 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:07.478 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.167672327000005, 2024-04-22T18:40:07.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:40:07.479 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.479 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.479 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:40:07.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.479 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:07.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:07.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:40:07.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:07.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:40:07.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:40:07.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:40:07.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:40:07.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:40:07.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:40:07.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:40:07.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:40:07.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:40:07.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:40:07.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:07.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:40:07.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:40:07.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:40:07.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:40:07.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:40:07.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:40:07.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:40:07.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:40:07.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:40:07.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:40:07.484 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.484 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.484 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.484 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:07.484 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:07.484 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.167578124000002, 2024-04-22T18:40:07.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:40:07.484 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.484 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.484 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.484 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.484 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.485 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:40:07.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.485 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:07.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:07.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:40:07.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:07.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:40:07.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:40:07.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:40:07.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:40:07.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:40:07.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:40:07.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:40:07.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:40:07.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:40:07.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:40:07.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:07.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:40:07.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:40:07.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:40:07.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:40:07.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:40:07.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:40:07.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:40:07.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:40:07.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:40:07.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:40:07.489 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.489 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.489 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.489 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:07.490 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:07.490 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.167557235999993, 2024-04-22T18:40:07.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:40:07.490 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.490 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:07.490 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.490 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.490 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.490 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:40:07.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.490 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:07.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:07.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:40:07.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:07.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:40:07.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:40:07.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:40:07.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:40:07.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:40:07.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:40:07.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:40:07.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:40:07.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:07.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:40:07.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:40:07.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:40:07.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:40:07.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:40:07.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:40:07.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:40:07.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:40:07.494 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.494 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.494 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.495 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:07.495 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:07.495 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.384549573000001, 2024-04-22T18:40:07.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:40:07.495 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.495 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.495 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.495 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.495 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.495 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:40:07.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.495 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:07.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:07.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:40:07.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:07.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:40:07.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:40:07.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:40:07.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:40:07.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:40:07.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:40:07.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:40:07.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:40:07.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:40:07.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:40:07.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:07.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:40:07.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:40:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:40:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:40:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:40:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:40:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:40:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:40:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:40:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:40:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.376718167000007, 2024-04-22T18:40:07.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:40:07.501 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.501 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.501 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.501 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.501 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.501 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:40:07.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.501 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:07.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:40:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:40:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:40:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:40:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:40:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:40:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:40:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:40:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:40:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:40:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:40:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:40:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:40:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:40:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:40:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:40:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:40:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:40:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:40:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:40:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:40:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.505 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.506 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.506 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2052 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:07.506 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:07.506 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.854606386, 2024-04-22T18:40:07.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43058958600000002, 2024-04-22T18:40:07.506 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.506 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.506 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.506 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.506 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.506 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2052 2024-04-22T18:40:07.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.507 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:07.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:07.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577494+0000", 2024-04-22T18:40:07.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:07.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577500+0000", 2024-04-22T18:40:07.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3279999999999996e-06 2024-04-22T18:40:07.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577510+0000", 2024-04-22T18:40:07.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9450000000000005e-06 2024-04-22T18:40:07.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577515+0000", 2024-04-22T18:40:07.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7799999999999997e-06 2024-04-22T18:40:07.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677622+0000", 2024-04-22T18:40:07.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010654099999999 2024-04-22T18:40:07.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677710+0000", 2024-04-22T18:40:07.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8472000000000003e-05 2024-04-22T18:40:07.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:07.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.906619+0000", 2024-04-22T18:40:07.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.228908519 2024-04-22T18:40:07.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907343+0000", 2024-04-22T18:40:07.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072442600000000002 2024-04-22T18:40:07.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.007972+0000", 2024-04-22T18:40:07.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100628781 2024-04-22T18:40:07.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008022+0000", 2024-04-22T18:40:07.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9843999999999999e-05 2024-04-22T18:40:07.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008085+0000", 2024-04-22T18:40:07.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3256999999999994e-05 2024-04-22T18:40:07.511 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.511 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.511 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.511 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2087 4.a 4:537c89f8:::smithi08937929-47:head [assert-version v37,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-04-22T18:40:07.511 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:40:07.512 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.524689626000001, 2024-04-22T18:40:07.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38904324299999998, 2024-04-22T18:40:07.512 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.512 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.512 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.512 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.512 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.512 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2087 2024-04-22T18:40:07.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.512 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:40:07.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:40:07.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907410+0000", 2024-04-22T18:40:07.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:07.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907417+0000", 2024-04-22T18:40:07.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.306e-06 2024-04-22T18:40:07.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907427+0000", 2024-04-22T18:40:07.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4930000000000004e-06 2024-04-22T18:40:07.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907432+0000", 2024-04-22T18:40:07.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5400000000000003e-06 2024-04-22T18:40:07.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907477+0000", 2024-04-22T18:40:07.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4851999999999997e-05 2024-04-22T18:40:07.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907537+0000", 2024-04-22T18:40:07.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9460999999999999e-05 2024-04-22T18:40:07.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:07.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.295587+0000", 2024-04-22T18:40:07.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38805073200000001 2024-04-22T18:40:07.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296281+0000", 2024-04-22T18:40:07.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00069370800000000002 2024-04-22T18:40:07.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296400+0000", 2024-04-22T18:40:07.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118579 2024-04-22T18:40:07.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296431+0000", 2024-04-22T18:40:07.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1627000000000001e-05 2024-04-22T18:40:07.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296456+0000", 2024-04-22T18:40:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4331000000000001e-05 2024-04-22T18:40:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 12.224845537, 2024-04-22T18:40:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:40:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.517 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.518 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.518 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.518 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:40:07.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.518 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:07.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:07.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:40:07.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:07.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:40:07.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:40:07.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:40:07.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:40:07.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:40:07.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:40:07.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:40:07.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:40:07.522 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.522 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.522 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.523 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2505 4.5 4:a6d237d2:::smithi08937929-43:head [setxattr _header (56) in=7b,truncate 3529470] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:07.523 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:07.523 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 12.224452842, 2024-04-22T18:40:07.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43706744400000003, 2024-04-22T18:40:07.523 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.523 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.523 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.523 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.523 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.523 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2505 2024-04-22T18:40:07.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.523 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:07.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:07.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207648+0000", 2024-04-22T18:40:07.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207653+0000", 2024-04-22T18:40:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2989999999999998e-06 2024-04-22T18:40:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207656+0000", 2024-04-22T18:40:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0199999999999999e-06 2024-04-22T18:40:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207659+0000", 2024-04-22T18:40:07.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2270000000000002e-06 2024-04-22T18:40:07.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243213+0000", 2024-04-22T18:40:07.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035553653999999997 2024-04-22T18:40:07.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243273+0000", 2024-04-22T18:40:07.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0300000000000002e-05 2024-04-22T18:40:07.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:07.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243359+0000", 2024-04-22T18:40:07.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6275999999999999e-05 2024-04-22T18:40:07.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297443+0000", 2024-04-22T18:40:07.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.054083930000000002 2024-04-22T18:40:07.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644651+0000", 2024-04-22T18:40:07.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.347208181 2024-04-22T18:40:07.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644684+0000", 2024-04-22T18:40:07.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3080999999999997e-05 2024-04-22T18:40:07.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644717+0000", 2024-04-22T18:40:07.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2264000000000001e-05 2024-04-22T18:40:07.528 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.528 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.528 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.528 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:07.528 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:07.529 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 12.224440629, 2024-04-22T18:40:07.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:40:07.529 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.529 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:07.529 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.529 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.529 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.529 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:40:07.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.529 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:07.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:07.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:07.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:07.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:40:07.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:40:07.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:40:07.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:40:07.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:40:07.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:40:07.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:40:07.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:40:07.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:07.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:40:07.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:40:07.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:40:07.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:40:07.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:40:07.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:40:07.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:40:07.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:40:07.533 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.533 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.533 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.534 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2615 4.5 4:a6d237d2:::smithi08937929-43:head [write 463926~462166] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-22T18:40:07.534 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:58.352901+0000", 2024-04-22T18:40:07.534 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.0792011810000002, 2024-04-22T18:40:07.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.307052514, 2024-04-22T18:40:07.534 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.534 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.534 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.534 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.534 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.534 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2615 2024-04-22T18:40:07.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.534 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.352901+0000", 2024-04-22T18:40:07.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.352901+0000", 2024-04-22T18:40:07.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.352897+0000", 2024-04-22T18:40:07.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999962 2024-04-22T18:40:07.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.353421+0000", 2024-04-22T18:40:07.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052391999999999998 2024-04-22T18:40:07.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.353433+0000", 2024-04-22T18:40:07.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2128e-05 2024-04-22T18:40:07.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.353460+0000", 2024-04-22T18:40:07.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7211000000000001e-05 2024-04-22T18:40:07.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.453670+0000", 2024-04-22T18:40:07.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10020929100000001 2024-04-22T18:40:07.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.453809+0000", 2024-04-22T18:40:07.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013967800000000001 2024-04-22T18:40:07.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:07.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.454021+0000", 2024-04-22T18:40:07.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021190900000000001 2024-04-22T18:40:07.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659805+0000", 2024-04-22T18:40:07.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20578357999999999 2024-04-22T18:40:07.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659886+0000", 2024-04-22T18:40:07.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1310000000000006e-05 2024-04-22T18:40:07.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659925+0000", 2024-04-22T18:40:07.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9233000000000002e-05 2024-04-22T18:40:07.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659953+0000", 2024-04-22T18:40:07.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7894000000000001e-05 2024-04-22T18:40:07.539 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.539 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.539 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.539 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2715 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:07.539 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:03.692698+0000", 2024-04-22T18:40:07.540 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 3.7394042870000002, 2024-04-22T18:40:07.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40343488, 2024-04-22T18:40:07.540 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.540 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:07.540 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.540 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.540 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.540 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2715 2024-04-22T18:40:07.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.540 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.692698+0000", 2024-04-22T18:40:07.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.692698+0000", 2024-04-22T18:40:07.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.692697+0000", 2024-04-22T18:40:07.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:07.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.692700+0000", 2024-04-22T18:40:07.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3410000000000001e-06 2024-04-22T18:40:07.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.692703+0000", 2024-04-22T18:40:07.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7230000000000002e-06 2024-04-22T18:40:07.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.792867+0000", 2024-04-22T18:40:07.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10016380499999999 2024-04-22T18:40:07.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.793431+0000", 2024-04-22T18:40:07.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056344299999999995 2024-04-22T18:40:07.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:07.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.793496+0000", 2024-04-22T18:40:07.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5415999999999998e-05 2024-04-22T18:40:07.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:04.095990+0000", 2024-04-22T18:40:07.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30249418099999997 2024-04-22T18:40:07.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:04.096045+0000", 2024-04-22T18:40:07.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4486e-05 2024-04-22T18:40:07.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:04.096133+0000", 2024-04-22T18:40:07.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7956999999999998e-05 2024-04-22T18:40:07.544 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.544 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.544 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.544 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.544 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.544 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:07.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:07.546 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:07.547+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:07.603 INFO:tasks.rados.rados.0.smithi089.stdout:1285: set_chunk oid 23 offset: 1676288 length: 337408 target oid fc7ee0e7787ce4e86cf93601d7f74659053df97087a10042788e62296d3f9f5f offset: 0 snap -1 2024-04-22T18:40:07.603 INFO:tasks.rados.rados.0.smithi089.stdout:1286: write oid 37 current snap is 0 2024-04-22T18:40:07.603 INFO:tasks.rados.rados.0.smithi089.stdout:1277: finishing write tid 1 to smithi08937929-34 2024-04-22T18:40:07.603 INFO:tasks.rados.rados.0.smithi089.stdout:1286: seq_num 238 ranges {400066=716205,1860832=139234} 2024-04-22T18:40:07.609 INFO:tasks.rados.rados.0.smithi089.stdout:1286: writing smithi08937929-37 from 400066 to 1116271 tid 1 2024-04-22T18:40:07.614 INFO:tasks.rados.rados.0.smithi089.stdout:1286: writing smithi08937929-37 from 1860832 to 2000066 tid 2 2024-04-22T18:40:07.614 INFO:tasks.rados.rados.0.smithi089.stdout:1287: tier_promote oid 36 2024-04-22T18:40:07.614 INFO:tasks.rados.rados.0.smithi089.stdout:1277: finishing write tid 2 to smithi08937929-34 2024-04-22T18:40:07.614 INFO:tasks.rados.rados.0.smithi089.stdout:1277: finishing write tid 3 to smithi08937929-34 2024-04-22T18:40:07.614 INFO:tasks.rados.rados.0.smithi089.stdout:1277: finishing write tid 4 to smithi08937929-34 2024-04-22T18:40:07.614 INFO:tasks.rados.rados.0.smithi089.stdout:1280: expect (ObjNum 216 snap 0 seq_num 216) 2024-04-22T18:40:07.646 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:07.648+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:07.657 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:40:07.689 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:07.689 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:07.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.689 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2835 4.18 4.835b5e18 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:07.689 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:07.689 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.074848461000000005, 2024-04-22T18:40:07.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.074868204999999993, 2024-04-22T18:40:07.690 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.690 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:07.690 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.690 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:07.690 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:07.690 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2835 2024-04-22T18:40:07.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.690 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:07.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:07.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616367+0000", 2024-04-22T18:40:07.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:07.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616377+0000", 2024-04-22T18:40:07.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0618999999999999e-05 2024-04-22T18:40:07.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616410+0000", 2024-04-22T18:40:07.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3021000000000001e-05 2024-04-22T18:40:07.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616437+0000", 2024-04-22T18:40:07.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7158e-05 2024-04-22T18:40:07.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.692 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.693 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:40:07.693 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 1 2024-04-22T18:40:07.693 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:07.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:07.750 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:07.751+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:07.801 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:07.801 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:07.801 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:07.801 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:07.801 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:07.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:07.819 INFO:tasks.rados.rados.0.smithi089.stdout:1282: finishing write tid 1 to smithi08937929-3 2024-04-22T18:40:07.819 INFO:tasks.rados.rados.0.smithi089.stdout:1276: finishing set_chunk to oid 14 2024-04-22T18:40:07.819 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 51 (ObjNum 163 snap 0 seq_num 163) dirty exists 2024-04-22T18:40:07.819 INFO:tasks.rados.rados.0.smithi089.stdout:1275: finishing write tid 1 to smithi08937929-13 2024-04-22T18:40:07.819 INFO:tasks.rados.rados.0.smithi089.stdout:1275: finishing write tid 2 to smithi08937929-13 2024-04-22T18:40:07.819 INFO:tasks.rados.rados.0.smithi089.stdout:1276: done (12 left) 2024-04-22T18:40:07.820 INFO:tasks.rados.rados.0.smithi089.stdout:1279: done (11 left) 2024-04-22T18:40:07.820 INFO:tasks.rados.rados.0.smithi089.stdout:1280: done (10 left) 2024-04-22T18:40:07.820 INFO:tasks.rados.rados.0.smithi089.stdout:1288: write oid 22 current snap is 0 2024-04-22T18:40:07.820 INFO:tasks.rados.rados.0.smithi089.stdout:1288: seq_num 239 ranges {416873=791454,1919314=97559} 2024-04-22T18:40:07.825 INFO:tasks.rados.rados.0.smithi089.stdout:1288: writing smithi08937929-22 from 416873 to 1208327 tid 1 2024-04-22T18:40:07.830 INFO:tasks.rados.rados.0.smithi089.stdout:1288: writing smithi08937929-22 from 1919314 to 2016873 tid 2 2024-04-22T18:40:07.830 INFO:tasks.rados.rados.0.smithi089.stdout:1289: read oid 48 snap -1 2024-04-22T18:40:07.830 INFO:tasks.rados.rados.0.smithi089.stdout:1289: expect (ObjNum 231 snap 0 seq_num 231) 2024-04-22T18:40:07.830 INFO:tasks.rados.rados.0.smithi089.stdout:1290: write oid 33 current snap is 0 2024-04-22T18:40:07.830 INFO:tasks.rados.rados.0.smithi089.stdout:1277: finishing write tid 5 to smithi08937929-34 2024-04-22T18:40:07.830 INFO:tasks.rados.rados.0.smithi089.stdout:1277: oid 34 updating version 0 to 85 2024-04-22T18:40:07.830 INFO:tasks.rados.rados.0.smithi089.stdout:1277: oid 34 updating version 85 to 86 2024-04-22T18:40:07.830 INFO:tasks.rados.rados.0.smithi089.stdout:1277: oid 34 version 86 is already newer than 84 2024-04-22T18:40:07.831 INFO:tasks.rados.rados.0.smithi089.stdout:1277: oid 34 updating version 86 to 87 2024-04-22T18:40:07.831 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 87 (ObjNum 235 snap 0 seq_num 235) dirty exists 2024-04-22T18:40:07.831 INFO:tasks.rados.rados.0.smithi089.stdout:1277: left oid 34 (ObjNum 235 snap 0 seq_num 235) 2024-04-22T18:40:07.831 INFO:tasks.rados.rados.0.smithi089.stdout:1283: expect (ObjNum 228 snap 0 seq_num 228) 2024-04-22T18:40:07.851 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:07.852+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:07.951 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:07.952+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:07.959 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:40:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 88.217047479000001, 2024-04-22T18:40:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30520528099999999, 2024-04-22T18:40:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.959 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.960 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.960 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:07.960 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:07.960 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 35 2024-04-22T18:40:07.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.960 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:40:07.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741854+0000", 2024-04-22T18:40:07.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741852+0000", 2024-04-22T18:40:07.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:07.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741927+0000", 2024-04-22T18:40:07.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5123000000000006e-05 2024-04-22T18:40:07.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741953+0000", 2024-04-22T18:40:07.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5378999999999999e-05 2024-04-22T18:40:07.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741977+0000", 2024-04-22T18:40:07.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4431999999999999e-05 2024-04-22T18:40:07.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844243+0000", 2024-04-22T18:40:07.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102265525 2024-04-22T18:40:07.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844330+0000", 2024-04-22T18:40:07.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7417999999999995e-05 2024-04-22T18:40:07.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:07.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.844470+0000", 2024-04-22T18:40:07.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000140197 2024-04-22T18:40:07.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944767+0000", 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100296409 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.046968+0000", 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10220148499999999 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047018+0000", 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9206000000000003e-05 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.047059+0000", 2024-04-22T18:40:07.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1496999999999999e-05 2024-04-22T18:40:07.965 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.965 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.965 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.965 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:07.965 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:07.965 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 88.216916955000002, 2024-04-22T18:40:07.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:40:07.965 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.965 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.965 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.965 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:07.965 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:07.965 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:40:07.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.966 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:07.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:07.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:40:07.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:07.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:40:07.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:40:07.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:40:07.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:40:07.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:40:07.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:40:07.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:40:07.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:40:07.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:40:07.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:40:07.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:07.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:40:07.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:40:07.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:40:07.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:40:07.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:40:07.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:40:07.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:40:07.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:40:07.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:40:07.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:40:07.971 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.971 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.971 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.971 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:07.971 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:07.972 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 88.216877264000004, 2024-04-22T18:40:07.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:40:07.972 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.972 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.972 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.972 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:07.972 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:07.972 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:40:07.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.972 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:07.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:07.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:40:07.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:07.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:40:07.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:40:07.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:40:07.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:40:07.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:40:07.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:40:07.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:40:07.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:40:07.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:40:07.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:40:07.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:07.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:40:07.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:40:07.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:40:07.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:40:07.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:40:07.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:40:07.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:40:07.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:40:07.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:40:07.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:40:07.977 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.977 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.978 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.978 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:07.978 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:40:07.978 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 85.972696217000006, 2024-04-22T18:40:07.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402419109, 2024-04-22T18:40:07.978 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.978 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.978 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.978 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:07.978 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:07.978 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 68 2024-04-22T18:40:07.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.979 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:40:07.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:40:07.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986204+0000", 2024-04-22T18:40:07.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:07.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986208+0000", 2024-04-22T18:40:07.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8149999999999999e-06 2024-04-22T18:40:07.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986214+0000", 2024-04-22T18:40:07.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.604e-06 2024-04-22T18:40:07.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986218+0000", 2024-04-22T18:40:07.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6689999999999999e-06 2024-04-22T18:40:07.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986833+0000", 2024-04-22T18:40:07.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000615648 2024-04-22T18:40:07.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986891+0000", 2024-04-22T18:40:07.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7915999999999999e-05 2024-04-22T18:40:07.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:07.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986981+0000", 2024-04-22T18:40:07.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759999999999994e-05 2024-04-22T18:40:07.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.987584+0000", 2024-04-22T18:40:07.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060270199999999999 2024-04-22T18:40:07.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388527+0000", 2024-04-22T18:40:07.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40094366999999997 2024-04-22T18:40:07.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388583+0000", 2024-04-22T18:40:07.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5383000000000002e-05 2024-04-22T18:40:07.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388624+0000", 2024-04-22T18:40:07.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501999999999997e-05 2024-04-22T18:40:07.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.983 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.984 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:07.984 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:07.984 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 85.972680428000004, 2024-04-22T18:40:07.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:40:07.984 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.984 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.984 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.984 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:07.984 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:07.984 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:40:07.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.984 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:07.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:07.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:40:07.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:07.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:40:07.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:40:07.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:40:07.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:40:07.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:40:07.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:40:07.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:40:07.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:40:07.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:40:07.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:40:07.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:07.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:40:07.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:40:07.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:40:07.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:40:07.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:40:07.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:40:07.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:40:07.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:40:07.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:40:07.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:40:07.990 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.990 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.990 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.990 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:07.990 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:40:07.990 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.856287418999997, 2024-04-22T18:40:07.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305563733, 2024-04-22T18:40:07.990 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.990 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.990 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.990 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:07.991 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:07.991 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 87 2024-04-22T18:40:07.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.991 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:40:07.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:40:07.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102613+0000", 2024-04-22T18:40:07.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:07.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102668+0000", 2024-04-22T18:40:07.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4669000000000001e-05 2024-04-22T18:40:07.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102680+0000", 2024-04-22T18:40:07.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2026e-05 2024-04-22T18:40:07.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102687+0000", 2024-04-22T18:40:07.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8339999999999999e-06 2024-04-22T18:40:07.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:07.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405644+0000", 2024-04-22T18:40:07.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30295623199999999 2024-04-22T18:40:07.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:07.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405743+0000", 2024-04-22T18:40:07.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.98e-05 2024-04-22T18:40:07.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:07.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405868+0000", 2024-04-22T18:40:07.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012457899999999999 2024-04-22T18:40:07.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:07.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407047+0000", 2024-04-22T18:40:07.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011791449999999999 2024-04-22T18:40:07.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:07.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408088+0000", 2024-04-22T18:40:07.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010411680000000001 2024-04-22T18:40:07.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:07.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408135+0000", 2024-04-22T18:40:07.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6255999999999999e-05 2024-04-22T18:40:07.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:07.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408178+0000", 2024-04-22T18:40:07.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2877000000000003e-05 2024-04-22T18:40:07.996 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.996 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:07.996 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:07.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.996 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:07.996 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:40:07.996 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.856207490000003, 2024-04-22T18:40:07.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30560089000000001, 2024-04-22T18:40:07.996 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:07.997 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:07.997 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:07.997 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:07.997 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:07.997 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 88 2024-04-22T18:40:07.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.997 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:07.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:07.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:40:07.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:07.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:40:07.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:07.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:07.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102693+0000", 2024-04-22T18:40:07.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:07.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:07.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102698+0000", 2024-04-22T18:40:07.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9699999999999998e-06 2024-04-22T18:40:07.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:07.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102708+0000", 2024-04-22T18:40:07.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.006e-05 2024-04-22T18:40:07.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:07.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102715+0000", 2024-04-22T18:40:07.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6599999999999995e-06 2024-04-22T18:40:07.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:07.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:07.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:08.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406637+0000", 2024-04-22T18:40:08.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392130099999998 2024-04-22T18:40:08.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:08.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406696+0000", 2024-04-22T18:40:08.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9066e-05 2024-04-22T18:40:08.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:08.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406784+0000", 2024-04-22T18:40:08.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8194000000000003e-05 2024-04-22T18:40:08.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:08.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407447+0000", 2024-04-22T18:40:08.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066295700000000004 2024-04-22T18:40:08.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408237+0000", 2024-04-22T18:40:08.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079032600000000005 2024-04-22T18:40:08.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:08.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408270+0000", 2024-04-22T18:40:08.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3157000000000001e-05 2024-04-22T18:40:08.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:08.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408295+0000", 2024-04-22T18:40:08.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4151999999999998e-05 2024-04-22T18:40:08.002 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.002 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:08.002 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:40:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.104975366000005, 2024-04-22T18:40:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30533819600000001, 2024-04-22T18:40:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 106 2024-04-22T18:40:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:08.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:40:08.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:08.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:40:08.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:08.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853925+0000", 2024-04-22T18:40:08.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:08.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853958+0000", 2024-04-22T18:40:08.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2536000000000002e-05 2024-04-22T18:40:08.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:08.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853964+0000", 2024-04-22T18:40:08.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6239999999999999e-06 2024-04-22T18:40:08.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853967+0000", 2024-04-22T18:40:08.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9700000000000001e-06 2024-04-22T18:40:08.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:08.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855144+0000", 2024-04-22T18:40:08.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001176774 2024-04-22T18:40:08.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:08.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855225+0000", 2024-04-22T18:40:08.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0697000000000003e-05 2024-04-22T18:40:08.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:08.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855353+0000", 2024-04-22T18:40:08.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012788 2024-04-22T18:40:08.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:08.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058003+0000", 2024-04-22T18:40:08.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202649939 2024-04-22T18:40:08.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159184+0000", 2024-04-22T18:40:08.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101181058 2024-04-22T18:40:08.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:08.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159238+0000", 2024-04-22T18:40:08.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3967e-05 2024-04-22T18:40:08.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:08.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159264+0000", 2024-04-22T18:40:08.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6237999999999998e-05 2024-04-22T18:40:08.008 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.008 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:08.008 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.008 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:08.008 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:40:08.008 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.104928516000001, 2024-04-22T18:40:08.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30544294500000002, 2024-04-22T18:40:08.009 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:08.009 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.009 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:08.009 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:08.009 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:08.009 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 107 2024-04-22T18:40:08.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.009 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:08.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:08.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:40:08.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:08.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:40:08.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:08.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853972+0000", 2024-04-22T18:40:08.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:08.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853975+0000", 2024-04-22T18:40:08.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0939999999999999e-06 2024-04-22T18:40:08.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:08.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853978+0000", 2024-04-22T18:40:08.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:40:08.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853981+0000", 2024-04-22T18:40:08.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.858e-06 2024-04-22T18:40:08.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:08.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855831+0000", 2024-04-22T18:40:08.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018503689999999999 2024-04-22T18:40:08.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:08.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855932+0000", 2024-04-22T18:40:08.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010071 2024-04-22T18:40:08.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:08.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856029+0000", 2024-04-22T18:40:08.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6751999999999998e-05 2024-04-22T18:40:08.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:08.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058013+0000", 2024-04-22T18:40:08.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20198412600000001 2024-04-22T18:40:08.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159326+0000", 2024-04-22T18:40:08.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101312969 2024-04-22T18:40:08.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:08.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159376+0000", 2024-04-22T18:40:08.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9968e-05 2024-04-22T18:40:08.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:08.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159416+0000", 2024-04-22T18:40:08.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9641999999999998e-05 2024-04-22T18:40:08.014 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.014 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:08.014 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.014 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:08.014 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:08.014 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.104840120000006, 2024-04-22T18:40:08.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30548257499999998, 2024-04-22T18:40:08.014 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:08.014 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.015 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:08.015 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:08.015 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:08.015 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 108 2024-04-22T18:40:08.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.015 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:08.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:08.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:08.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:08.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:08.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:08.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:08.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:08.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854064+0000", 2024-04-22T18:40:08.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5820000000000002e-06 2024-04-22T18:40:08.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:08.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854080+0000", 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5379999999999998e-05 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854084+0000", 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.963e-06 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856413+0000", 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023298070000000001 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856494+0000", 2024-04-22T18:40:08.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0863000000000002e-05 2024-04-22T18:40:08.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:08.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856624+0000", 2024-04-22T18:40:08.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129799 2024-04-22T18:40:08.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:08.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058947+0000", 2024-04-22T18:40:08.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20232328899999999 2024-04-22T18:40:08.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159468+0000", 2024-04-22T18:40:08.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100520597 2024-04-22T18:40:08.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:08.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159508+0000", 2024-04-22T18:40:08.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9830000000000003e-05 2024-04-22T18:40:08.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:08.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159544+0000", 2024-04-22T18:40:08.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5944e-05 2024-04-22T18:40:08.020 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.020 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:08.020 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.020 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:08.020 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:40:08.020 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.104795628000005, 2024-04-22T18:40:08.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30556502499999999, 2024-04-22T18:40:08.020 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:08.021 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.021 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:08.021 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:08.021 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:08.021 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:40:08.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.021 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:08.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:08.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:40:08.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:08.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:40:08.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:08.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854105+0000", 2024-04-22T18:40:08.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:08.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854175+0000", 2024-04-22T18:40:08.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0178000000000005e-05 2024-04-22T18:40:08.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:08.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854191+0000", 2024-04-22T18:40:08.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5965999999999999e-05 2024-04-22T18:40:08.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854196+0000", 2024-04-22T18:40:08.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.42e-06 2024-04-22T18:40:08.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:08.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956526+0000", 2024-04-22T18:40:08.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10233094299999999 2024-04-22T18:40:08.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:08.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956634+0000", 2024-04-22T18:40:08.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107398 2024-04-22T18:40:08.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:08.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956730+0000", 2024-04-22T18:40:08.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6217000000000004e-05 2024-04-22T18:40:08.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:08.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058955+0000", 2024-04-22T18:40:08.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102224749 2024-04-22T18:40:08.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159596+0000", 2024-04-22T18:40:08.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100641297 2024-04-22T18:40:08.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:08.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159637+0000", 2024-04-22T18:40:08.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874000000000002e-05 2024-04-22T18:40:08.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:08.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159671+0000", 2024-04-22T18:40:08.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3603999999999999e-05 2024-04-22T18:40:08.026 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.026 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:08.026 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.026 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:08.026 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:08.026 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.104701781000003, 2024-04-22T18:40:08.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30683048800000001, 2024-04-22T18:40:08.026 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:08.026 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.027 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:08.027 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:08.027 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:08.027 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110 2024-04-22T18:40:08.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.027 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:08.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:08.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:08.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:08.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:08.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:08.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:08.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:08.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854202+0000", 2024-04-22T18:40:08.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:40:08.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:08.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854206+0000", 2024-04-22T18:40:08.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5870000000000001e-06 2024-04-22T18:40:08.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854208+0000", 2024-04-22T18:40:08.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7199999999999998e-06 2024-04-22T18:40:08.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:08.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057285+0000", 2024-04-22T18:40:08.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20307620800000001 2024-04-22T18:40:08.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:08.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057414+0000", 2024-04-22T18:40:08.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012896899999999999 2024-04-22T18:40:08.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:08.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057588+0000", 2024-04-22T18:40:08.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000174802 2024-04-22T18:40:08.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:08.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159429+0000", 2024-04-22T18:40:08.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101840783 2024-04-22T18:40:08.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160947+0000", 2024-04-22T18:40:08.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001518152 2024-04-22T18:40:08.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:08.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160992+0000", 2024-04-22T18:40:08.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4725000000000003e-05 2024-04-22T18:40:08.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:08.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161030+0000", 2024-04-22T18:40:08.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7789e-05 2024-04-22T18:40:08.032 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.032 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:08.032 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.032 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:08.032 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:08.032 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.104690599999998, 2024-04-22T18:40:08.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:40:08.032 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:08.033 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.033 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:08.033 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:08.033 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:08.033 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:40:08.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.033 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:08.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:08.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:08.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:08.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:08.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:08.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:40:08.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:08.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:40:08.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:40:08.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:08.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:40:08.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:40:08.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:40:08.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:40:08.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:08.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:40:08.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:40:08.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:08.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:40:08.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:40:08.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:08.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:40:08.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:40:08.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:08.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:40:08.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:40:08.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:40:08.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:40:08.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:08.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:40:08.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:40:08.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:08.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:40:08.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:40:08.038 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.038 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:08.038 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.038 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:08.038 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:08.039 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.104674942000003, 2024-04-22T18:40:08.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:40:08.039 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:08.039 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.039 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:08.039 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:08.039 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:08.039 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:40:08.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.039 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:08.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:08.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:40:08.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:08.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:08.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:40:08.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:40:08.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:40:08.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:40:08.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:08.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:40:08.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:40:08.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:08.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:40:08.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:40:08.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:08.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:40:08.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:40:08.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:08.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:40:08.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:40:08.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:40:08.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:40:08.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:08.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:40:08.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:40:08.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:08.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:40:08.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:40:08.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.044 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:08.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.104623779999997, 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:08.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:08.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:08.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:08.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:08.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:08.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:08.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:40:08.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:40:08.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:08.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:40:08.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:40:08.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:40:08.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:40:08.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:08.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:40:08.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:40:08.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:08.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:40:08.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:40:08.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:08.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:40:08.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:40:08.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:08.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:40:08.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:40:08.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:40:08.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:40:08.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:08.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:40:08.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:40:08.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:08.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:40:08.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:40:08.050 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.050 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:08.051 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.051 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:08.051 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:08.051 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.104608556000002, 2024-04-22T18:40:08.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:40:08.051 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:08.051 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.051 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:08.051 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:08.051 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:08.051 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:40:08.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.052 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:08.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:08.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:08.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:08.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:08.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:08.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:40:08.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:08.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:40:08.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:40:08.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:08.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:40:08.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:40:08.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:40:08.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:40:08.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:08.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:40:08.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:40:08.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:08.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:40:08.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:40:08.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:08.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:40:08.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:40:08.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:08.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:40:08.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:40:08.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:40:08.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:40:08.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:08.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:40:08.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:40:08.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:08.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:40:08.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:40:08.057 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.057 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:08.057 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.057 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:08.057 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:08.057 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.104548979, 2024-04-22T18:40:08.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:40:08.057 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:08.057 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.057 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:08.057 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:08.058 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:08.058 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:40:08.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.058 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:08.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:08.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:08.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:08.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:08.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:40:08.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:40:08.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:40:08.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:08.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:40:08.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:40:08.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:08.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:40:08.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:40:08.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:08.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:40:08.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:40:08.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:08.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:40:08.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:40:08.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:40:08.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:40:08.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:08.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:40:08.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:40:08.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:08.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:40:08.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:40:08.062 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.062 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:08.062 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.063 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:08.063 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:08.063 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.104466662999997, 2024-04-22T18:40:08.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:40:08.063 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:08.063 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.063 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:08.063 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:08.063 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:08.063 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:40:08.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.063 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:08.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:08.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:08.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:08.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:08.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:08.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:40:08.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:08.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:40:08.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:40:08.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:08.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:40:08.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:40:08.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:40:08.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:40:08.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:08.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:40:08.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:40:08.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:08.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:40:08.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:40:08.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:08.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:40:08.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:40:08.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:08.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:40:08.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:40:08.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:40:08.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:40:08.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:08.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:40:08.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:40:08.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:08.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:40:08.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:40:08.068 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.068 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:08.068 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.069 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:08.069 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:08.069 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.104456506999995, 2024-04-22T18:40:08.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:40:08.069 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:08.069 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.069 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:08.069 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:08.069 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:08.069 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:40:08.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.069 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:08.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:08.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:08.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:08.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:08.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:08.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:40:08.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:40:08.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:08.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:40:08.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:40:08.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:40:08.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:40:08.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:08.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:40:08.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:40:08.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:08.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:40:08.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:40:08.074 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.074 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:08.074 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.074 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:161 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:40:08.074 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:40:08.074 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.503962216000005, 2024-04-22T18:40:08.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40239169000000002, 2024-04-22T18:40:08.075 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:08.075 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.075 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:08.075 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:08.075 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:08.075 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 161 2024-04-22T18:40:08.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.075 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:08.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:08.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:40:08.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:08.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:40:08.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:08.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:08.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454938+0000", 2024-04-22T18:40:08.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:08.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454982+0000", 2024-04-22T18:40:08.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3470999999999998e-05 2024-04-22T18:40:08.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:08.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454988+0000", 2024-04-22T18:40:08.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7640000000000002e-06 2024-04-22T18:40:08.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555054+0000", 2024-04-22T18:40:08.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006607300000001 2024-04-22T18:40:08.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:08.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555607+0000", 2024-04-22T18:40:08.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055319500000000003 2024-04-22T18:40:08.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:08.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555689+0000", 2024-04-22T18:40:08.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1817999999999995e-05 2024-04-22T18:40:08.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:08.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555804+0000", 2024-04-22T18:40:08.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115346 2024-04-22T18:40:08.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:08.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556554+0000", 2024-04-22T18:40:08.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000750102 2024-04-22T18:40:08.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857265+0000", 2024-04-22T18:40:08.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30071122700000003 2024-04-22T18:40:08.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:08.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857302+0000", 2024-04-22T18:40:08.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7069e-05 2024-04-22T18:40:08.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:08.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:08.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:08.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857331+0000", 2024-04-22T18:40:08.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8291999999999999e-05 2024-04-22T18:40:08.080 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.080 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:08.080 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.080 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:08.080 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:08.081 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:08.082 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:08.083+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:08.082 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:08.144 INFO:tasks.rados.rados.0.smithi089.stdout:1282: finishing write tid 2 to smithi08937929-3 2024-04-22T18:40:08.145 INFO:tasks.rados.rados.0.smithi089.stdout:1282: finishing write tid 3 to smithi08937929-3 2024-04-22T18:40:08.145 INFO:tasks.rados.rados.0.smithi089.stdout:1290: seq_num 240 ranges {433680=466703,1577796=455884} 2024-04-22T18:40:08.149 INFO:tasks.rados.rados.0.smithi089.stdout:1290: writing smithi08937929-33 from 433680 to 900383 tid 1 2024-04-22T18:40:08.155 INFO:tasks.rados.rados.0.smithi089.stdout:1290: writing smithi08937929-33 from 1577796 to 2033680 tid 2 2024-04-22T18:40:08.156 INFO:tasks.rados.rados.0.smithi089.stdout:1277: done (12 left) 2024-04-22T18:40:08.156 INFO:tasks.rados.rados.0.smithi089.stdout:1281: done (11 left) 2024-04-22T18:40:08.156 INFO:tasks.rados.rados.0.smithi089.stdout:1283: done (10 left) 2024-04-22T18:40:08.156 INFO:tasks.rados.rados.0.smithi089.stdout:1291: set_chunk oid 19 target oid 2024-04-22T18:40:08.168 INFO:tasks.rados.rados.0.smithi089.stdout:1291: set_chunk oid 19 offset: 409600 length: 2280448 target oid 0b62904818f4d34f1c2ce002af6499a3c483748515ee22c1383076393eabc4aa offset: 0 snap -1 2024-04-22T18:40:08.168 INFO:tasks.rados.rados.0.smithi089.stdout:1292: set_chunk oid 40 target oid 2024-04-22T18:40:08.183 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:08.184+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:08.187 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:08.187 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:08.188 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:08.188 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:08.196 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:08.271 INFO:tasks.rados.rados.0.smithi089.stdout:1292: set_chunk oid 40 offset: 255488 length: 52736 target oid e3212364168602da08b394c88bdcbd1f4ece713005f5f0aa7b28de4cefc5b317 offset: 0 snap -1 2024-04-22T18:40:08.271 INFO:tasks.rados.rados.0.smithi089.stdout:1293: read oid 11 snap -1 2024-04-22T18:40:08.271 INFO:tasks.rados.rados.0.smithi089.stdout:1293: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:40:08.271 INFO:tasks.rados.rados.0.smithi089.stdout:1294: read oid 35 snap -1 2024-04-22T18:40:08.271 INFO:tasks.rados.rados.0.smithi089.stdout:1294: expect (ObjNum 212 snap 0 seq_num 212) 2024-04-22T18:40:08.271 INFO:tasks.rados.rados.0.smithi089.stdout:1295: set_chunk oid 46 target oid 2024-04-22T18:40:08.283 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:08.285+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:08.295 INFO:tasks.rados.rados.0.smithi089.stdout:1295: set_chunk oid 46 offset: 74240 length: 1376256 target oid c5feefbbbd6d76f02938b831bb6b0d962285502a0748ace480f44943d5faa162 offset: 0 snap -1 2024-04-22T18:40:08.295 INFO:tasks.rados.rados.0.smithi089.stdout:1296: read oid 6 snap -1 2024-04-22T18:40:08.295 INFO:tasks.rados.rados.0.smithi089.stdout:1296: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:40:08.295 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:08.295 INFO:tasks.rados.rados.0.smithi089.stdout:1282: finishing write tid 4 to smithi08937929-3 2024-04-22T18:40:08.295 INFO:tasks.rados.rados.0.smithi089.stdout:1282: finishing write tid 5 to smithi08937929-3 2024-04-22T18:40:08.295 INFO:tasks.rados.rados.0.smithi089.stdout:1282: finishing write tid 6 to smithi08937929-3 2024-04-22T18:40:08.295 INFO:tasks.rados.rados.0.smithi089.stdout:1282: oid 3 updating version 0 to 56 2024-04-22T18:40:08.295 INFO:tasks.rados.rados.0.smithi089.stdout:1282: oid 3 version 56 is already newer than 54 2024-04-22T18:40:08.296 INFO:tasks.rados.rados.0.smithi089.stdout:1282: oid 3 updating version 56 to 58 2024-04-22T18:40:08.296 INFO:tasks.rados.rados.0.smithi089.stdout:1282: oid 3 version 58 is already newer than 57 2024-04-22T18:40:08.296 INFO:tasks.rados.rados.0.smithi089.stdout:1282: oid 3 version 58 is already newer than 55 2024-04-22T18:40:08.296 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 58 (ObjNum 237 snap 0 seq_num 237) dirty exists 2024-04-22T18:40:08.296 INFO:tasks.rados.rados.0.smithi089.stdout:1282: left oid 3 (ObjNum 237 snap 0 seq_num 237) 2024-04-22T18:40:08.296 INFO:tasks.rados.rados.0.smithi089.stdout:1285: finishing set_chunk to oid 23 2024-04-22T18:40:08.296 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 23 set_chunk returned error code -95 offset: 1676288 length: 337408 2024-04-22T18:40:08.296 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 0 (ObjNum 158 snap 0 seq_num 158) dirty exists 2024-04-22T18:40:08.296 INFO:tasks.rados.rados.0.smithi089.stdout:1286: finishing write tid 1 to smithi08937929-37 2024-04-22T18:40:08.296 INFO:tasks.rados.rados.0.smithi089.stdout:1286: finishing write tid 2 to smithi08937929-37 2024-04-22T18:40:08.296 INFO:tasks.rados.rados.0.smithi089.stdout:1275: finishing write tid 3 to smithi08937929-13 2024-04-22T18:40:08.296 INFO:tasks.rados.rados.0.smithi089.stdout:1275: finishing write tid 4 to smithi08937929-13 2024-04-22T18:40:08.296 INFO:tasks.rados.rados.0.smithi089.stdout:1275: finishing write tid 5 to smithi08937929-13 2024-04-22T18:40:08.296 INFO:tasks.rados.rados.0.smithi089.stdout:1275: oid 13 updating version 0 to 59 2024-04-22T18:40:08.296 INFO:tasks.rados.rados.0.smithi089.stdout:1275: oid 13 version 59 is already newer than 57 2024-04-22T18:40:08.297 INFO:tasks.rados.rados.0.smithi089.stdout:1275: oid 13 version 59 is already newer than 58 2024-04-22T18:40:08.297 INFO:tasks.rados.rados.0.smithi089.stdout:1275: oid 13 version 59 is already newer than 56 2024-04-22T18:40:08.297 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 59 (ObjNum 234 snap 0 seq_num 234) dirty exists 2024-04-22T18:40:08.297 INFO:tasks.rados.rados.0.smithi089.stdout:1275: left oid 13 (ObjNum 234 snap 0 seq_num 234) 2024-04-22T18:40:08.297 INFO:tasks.rados.rados.0.smithi089.stdout:1278: finishing write tid 1 to smithi08937929-47 2024-04-22T18:40:08.297 INFO:tasks.rados.rados.0.smithi089.stdout:1278: finishing write tid 2 to smithi08937929-47 2024-04-22T18:40:08.297 INFO:tasks.rados.rados.0.smithi089.stdout:1275: done (15 left) 2024-04-22T18:40:08.297 INFO:tasks.rados.rados.0.smithi089.stdout:1282: done (14 left) 2024-04-22T18:40:08.297 INFO:tasks.rados.rados.0.smithi089.stdout:1285: done (13 left) 2024-04-22T18:40:08.297 INFO:tasks.rados.rados.0.smithi089.stdout:1297: write oid 28 current snap is 0 2024-04-22T18:40:08.297 INFO:tasks.rados.rados.0.smithi089.stdout:1278: finishing write tid 3 to smithi08937929-47 2024-04-22T18:40:08.297 INFO:tasks.rados.rados.0.smithi089.stdout:1286: finishing write tid 3 to smithi08937929-37 2024-04-22T18:40:08.297 INFO:tasks.rados.rados.0.smithi089.stdout:1284: got (0) Success 2024-04-22T18:40:08.297 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 0 (ObjNum 209 snap 0 seq_num 209) dirty exists 2024-04-22T18:40:08.298 INFO:tasks.rados.rados.0.smithi089.stdout:1286: finishing write tid 4 to smithi08937929-37 2024-04-22T18:40:08.298 INFO:tasks.rados.rados.0.smithi089.stdout:1286: oid 37 updating version 0 to 93 2024-04-22T18:40:08.298 INFO:tasks.rados.rados.0.smithi089.stdout:1286: oid 37 updating version 93 to 94 2024-04-22T18:40:08.298 INFO:tasks.rados.rados.0.smithi089.stdout:1286: oid 37 version 94 is already newer than 92 2024-04-22T18:40:08.298 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 94 (ObjNum 238 snap 0 seq_num 238) dirty exists 2024-04-22T18:40:08.298 INFO:tasks.rados.rados.0.smithi089.stdout:1286: left oid 37 (ObjNum 238 snap 0 seq_num 238) 2024-04-22T18:40:08.298 INFO:tasks.rados.rados.0.smithi089.stdout:1278: finishing write tid 4 to smithi08937929-47 2024-04-22T18:40:08.298 INFO:tasks.rados.rados.0.smithi089.stdout:1289: expect (ObjNum 231 snap 0 seq_num 231) 2024-04-22T18:40:08.303 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:08.303 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:08.303 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:08.303 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:08.303 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:08.312 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:08.396 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:08.397+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:08.425 INFO:tasks.rados.rados.0.smithi089.stdout:1278: finishing write tid 5 to smithi08937929-47 2024-04-22T18:40:08.425 INFO:tasks.rados.rados.0.smithi089.stdout:1278: oid 47 updating version 0 to 62 2024-04-22T18:40:08.425 INFO:tasks.rados.rados.0.smithi089.stdout:1278: oid 47 version 62 is already newer than 61 2024-04-22T18:40:08.425 INFO:tasks.rados.rados.0.smithi089.stdout:1278: oid 47 version 62 is already newer than 60 2024-04-22T18:40:08.425 INFO:tasks.rados.rados.0.smithi089.stdout:1278: oid 47 updating version 62 to 63 2024-04-22T18:40:08.425 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 63 (ObjNum 236 snap 0 seq_num 236) dirty exists 2024-04-22T18:40:08.425 INFO:tasks.rados.rados.0.smithi089.stdout:1278: left oid 47 (ObjNum 236 snap 0 seq_num 236) 2024-04-22T18:40:08.425 INFO:tasks.rados.rados.0.smithi089.stdout:1287: got (0) Success 2024-04-22T18:40:08.425 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 51 (ObjNum 208 snap 0 seq_num 208) dirty exists 2024-04-22T18:40:08.425 INFO:tasks.rados.rados.0.smithi089.stdout:1288: finishing write tid 1 to smithi08937929-22 2024-04-22T18:40:08.425 INFO:tasks.rados.rados.0.smithi089.stdout:1297: seq_num 241 ranges {450487=541952,1636278=414209} 2024-04-22T18:40:08.430 INFO:tasks.rados.rados.0.smithi089.stdout:1297: writing smithi08937929-28 from 450487 to 992439 tid 1 2024-04-22T18:40:08.436 INFO:tasks.rados.rados.0.smithi089.stdout:1297: writing smithi08937929-28 from 1636278 to 2050487 tid 2 2024-04-22T18:40:08.436 INFO:tasks.rados.rados.0.smithi089.stdout:1278: done (13 left) 2024-04-22T18:40:08.437 INFO:tasks.rados.rados.0.smithi089.stdout:1284: done (12 left) 2024-04-22T18:40:08.437 INFO:tasks.rados.rados.0.smithi089.stdout:1286: done (11 left) 2024-04-22T18:40:08.437 INFO:tasks.rados.rados.0.smithi089.stdout:1287: done (10 left) 2024-04-22T18:40:08.437 INFO:tasks.rados.rados.0.smithi089.stdout:1289: done (9 left) 2024-04-22T18:40:08.437 INFO:tasks.rados.rados.0.smithi089.stdout:1298: read oid 45 snap -1 2024-04-22T18:40:08.437 INFO:tasks.rados.rados.0.smithi089.stdout:1298: expect (ObjNum 166 snap 0 seq_num 166) 2024-04-22T18:40:08.437 INFO:tasks.rados.rados.0.smithi089.stdout:1299: write (excl) oid 9 current snap is 0 2024-04-22T18:40:08.437 INFO:tasks.rados.rados.0.smithi089.stdout:1288: finishing write tid 2 to smithi08937929-22 2024-04-22T18:40:08.437 INFO:tasks.rados.rados.0.smithi089.stdout:1288: finishing write tid 3 to smithi08937929-22 2024-04-22T18:40:08.437 INFO:tasks.rados.rados.0.smithi089.stdout:1288: finishing write tid 4 to smithi08937929-22 2024-04-22T18:40:08.437 INFO:tasks.rados.rados.0.smithi089.stdout:1288: oid 22 updating version 0 to 88 2024-04-22T18:40:08.437 INFO:tasks.rados.rados.0.smithi089.stdout:1288: oid 22 updating version 88 to 89 2024-04-22T18:40:08.438 INFO:tasks.rados.rados.0.smithi089.stdout:1288: oid 22 updating version 89 to 90 2024-04-22T18:40:08.438 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 90 (ObjNum 239 snap 0 seq_num 239) dirty exists 2024-04-22T18:40:08.438 INFO:tasks.rados.rados.0.smithi089.stdout:1288: left oid 22 (ObjNum 239 snap 0 seq_num 239) 2024-04-22T18:40:08.438 INFO:tasks.rados.rados.0.smithi089.stdout:1291: finishing set_chunk to oid 19 2024-04-22T18:40:08.438 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 19 set_chunk returned error code -95 offset: 409600 length: 2280448 2024-04-22T18:40:08.438 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 0 (ObjNum 195 snap 0 seq_num 195) dirty exists 2024-04-22T18:40:08.438 INFO:tasks.rados.rados.0.smithi089.stdout:1292: finishing set_chunk to oid 40 2024-04-22T18:40:08.438 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 55 (ObjNum 227 snap 0 seq_num 227) dirty exists 2024-04-22T18:40:08.438 INFO:tasks.rados.rados.0.smithi089.stdout:1294: expect (ObjNum 212 snap 0 seq_num 212) 2024-04-22T18:40:08.461 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:08.461 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:08.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:08.461 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:08.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.462 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:08.462 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:08.462 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.006229230000002, 2024-04-22T18:40:08.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:40:08.462 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.462 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.462 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:08.462 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:08.462 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:08.462 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:40:08.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.463 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:08.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:08.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:40:08.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:08.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:40:08.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:40:08.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:40:08.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:40:08.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:40:08.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:40:08.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:40:08.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:40:08.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:40:08.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:40:08.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:08.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:40:08.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:40:08.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:08.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:40:08.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:40:08.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:40:08.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:40:08.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:08.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:40:08.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:40:08.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:40:08.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:40:08.468 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.468 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.468 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.468 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:315 4.b 4:d923883e:::smithi08937929-16:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:08.469 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:40:08.469 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 70.469766449999995, 2024-04-22T18:40:08.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40309447300000001, 2024-04-22T18:40:08.469 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.469 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.469 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:08.469 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:08.469 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:08.469 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 315 2024-04-22T18:40:08.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.469 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:40:08.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:40:08.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991817+0000", 2024-04-22T18:40:08.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:08.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991825+0000", 2024-04-22T18:40:08.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7279999999999995e-06 2024-04-22T18:40:08.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991834+0000", 2024-04-22T18:40:08.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1980000000000007e-06 2024-04-22T18:40:08.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991841+0000", 2024-04-22T18:40:08.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0740000000000004e-06 2024-04-22T18:40:08.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991881+0000", 2024-04-22T18:40:08.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-04-22T18:40:08.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991940+0000", 2024-04-22T18:40:08.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9382000000000002e-05 2024-04-22T18:40:08.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:08.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394250+0000", 2024-04-22T18:40:08.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40230947700000003 2024-04-22T18:40:08.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:08.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394752+0000", 2024-04-22T18:40:08.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050269699999999997 2024-04-22T18:40:08.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394865+0000", 2024-04-22T18:40:08.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011267700000000001 2024-04-22T18:40:08.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:08.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394892+0000", 2024-04-22T18:40:08.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7535999999999999e-05 2024-04-22T18:40:08.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394914+0000", 2024-04-22T18:40:08.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1398000000000001e-05 2024-04-22T18:40:08.475 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.475 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.475 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.475 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:774 4.b 4:defe92fb:::smithi08937929-44:head [write 2778819~95145] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:08.475 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:40:08.475 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.313256773999996, 2024-04-22T18:40:08.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40475725000000001, 2024-04-22T18:40:08.476 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.476 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.476 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:08.476 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:08.476 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:08.476 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 774 2024-04-22T18:40:08.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.476 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:40:08.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:40:08.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148328+0000", 2024-04-22T18:40:08.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:08.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148392+0000", 2024-04-22T18:40:08.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3323000000000003e-05 2024-04-22T18:40:08.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148398+0000", 2024-04-22T18:40:08.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4420000000000003e-06 2024-04-22T18:40:08.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148409+0000", 2024-04-22T18:40:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509999999999999e-05 2024-04-22T18:40:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251407+0000", 2024-04-22T18:40:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102998305 2024-04-22T18:40:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251500+0000", 2024-04-22T18:40:08.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3607e-05 2024-04-22T18:40:08.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:08.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251648+0000", 2024-04-22T18:40:08.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014788099999999999 2024-04-22T18:40:08.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:08.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256149+0000", 2024-04-22T18:40:08.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045010980000000003 2024-04-22T18:40:08.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553016+0000", 2024-04-22T18:40:08.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29686620800000002 2024-04-22T18:40:08.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:08.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553054+0000", 2024-04-22T18:40:08.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8414999999999997e-05 2024-04-22T18:40:08.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553086+0000", 2024-04-22T18:40:08.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2236000000000001e-05 2024-04-22T18:40:08.482 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.482 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.482 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.482 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:08.483 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:08.483 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.313171982, 2024-04-22T18:40:08.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:40:08.483 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.483 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:08.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:08.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:08.483 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:40:08.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.483 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:40:08.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:40:08.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:40:08.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:40:08.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:40:08.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:40:08.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:40:08.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:40:08.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:40:08.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:40:08.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:40:08.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:08.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:40:08.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:40:08.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:08.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:40:08.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:40:08.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:40:08.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:40:08.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:08.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:40:08.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:40:08.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:40:08.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:40:08.489 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.489 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.489 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.490 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:08.490 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:08.490 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.313153506000006, 2024-04-22T18:40:08.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:40:08.490 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.490 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:08.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:08.491 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:08.491 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:08.491 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:40:08.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.491 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:08.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:08.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:08.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:40:08.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:40:08.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:40:08.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:40:08.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:40:08.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:40:08.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:40:08.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:40:08.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:08.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:40:08.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:40:08.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:40:08.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:40:08.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:40:08.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:40:08.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:40:08.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:40:08.496 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.496 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.496 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.496 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1242 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-22T18:40:08.496 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:40:08.496 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.719907956999997, 2024-04-22T18:40:08.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40662891899999998, 2024-04-22T18:40:08.496 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.497 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:08.497 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:08.497 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:08.497 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:08.497 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1242 2024-04-22T18:40:08.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.497 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:40:08.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:40:08.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741677+0000", 2024-04-22T18:40:08.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741680+0000", 2024-04-22T18:40:08.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8059999999999999e-06 2024-04-22T18:40:08.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741685+0000", 2024-04-22T18:40:08.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0780000000000003e-06 2024-04-22T18:40:08.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741690+0000", 2024-04-22T18:40:08.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4020000000000004e-06 2024-04-22T18:40:08.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741732+0000", 2024-04-22T18:40:08.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2110000000000002e-05 2024-04-22T18:40:08.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741798+0000", 2024-04-22T18:40:08.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6432000000000004e-05 2024-04-22T18:40:08.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.148307+0000", 2024-04-22T18:40:08.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40650862100000001 2024-04-22T18:40:08.501 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.501 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.501 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.502 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1408 4.6 4:630bd8e5:::smithi08937929-45:head [write 2816389~693495] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:40:08.502 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:40:08.502 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.299938150000003, 2024-04-22T18:40:08.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408959452, 2024-04-22T18:40:08.502 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.502 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.502 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:08.502 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:08.502 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:08.502 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1408 2024-04-22T18:40:08.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.502 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:40:08.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:40:08.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161647+0000", 2024-04-22T18:40:08.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161951+0000", 2024-04-22T18:40:08.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030389799999999998 2024-04-22T18:40:08.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161955+0000", 2024-04-22T18:40:08.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1989999999999999e-06 2024-04-22T18:40:08.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161958+0000", 2024-04-22T18:40:08.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2260000000000001e-06 2024-04-22T18:40:08.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.265867+0000", 2024-04-22T18:40:08.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103908927 2024-04-22T18:40:08.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.265960+0000", 2024-04-22T18:40:08.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3016999999999994e-05 2024-04-22T18:40:08.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:08.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.266124+0000", 2024-04-22T18:40:08.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016347700000000001 2024-04-22T18:40:08.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570491+0000", 2024-04-22T18:40:08.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30436694199999997 2024-04-22T18:40:08.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:08.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570502+0000", 2024-04-22T18:40:08.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1107999999999999e-05 2024-04-22T18:40:08.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:08.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570556+0000", 2024-04-22T18:40:08.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4388000000000002e-05 2024-04-22T18:40:08.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570607+0000", 2024-04-22T18:40:08.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0782999999999998e-05 2024-04-22T18:40:08.508 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.508 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.508 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.509 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1409 4.6 4:630bd8e5:::smithi08937929-45:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:40:08.509 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:08.509 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.299624446000003, 2024-04-22T18:40:08.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40880186200000002, 2024-04-22T18:40:08.509 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.509 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.509 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:08.509 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:08.509 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:08.509 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1409 2024-04-22T18:40:08.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.510 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:08.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:08.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:08.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161964+0000", 2024-04-22T18:40:08.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5909999999999998e-06 2024-04-22T18:40:08.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161966+0000", 2024-04-22T18:40:08.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5979999999999999e-06 2024-04-22T18:40:08.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161969+0000", 2024-04-22T18:40:08.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7049999999999999e-06 2024-04-22T18:40:08.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269205+0000", 2024-04-22T18:40:08.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107236228 2024-04-22T18:40:08.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269320+0000", 2024-04-22T18:40:08.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114571 2024-04-22T18:40:08.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:08.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269476+0000", 2024-04-22T18:40:08.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155928 2024-04-22T18:40:08.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:08.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570619+0000", 2024-04-22T18:40:08.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30114322700000001 2024-04-22T18:40:08.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570672+0000", 2024-04-22T18:40:08.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3232000000000001e-05 2024-04-22T18:40:08.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:08.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570724+0000", 2024-04-22T18:40:08.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2258e-05 2024-04-22T18:40:08.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570763+0000", 2024-04-22T18:40:08.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8927999999999997e-05 2024-04-22T18:40:08.515 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.515 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.515 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.516 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1410 4.6 4:630bd8e5:::smithi08937929-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:40:08.516 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:08.516 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.299614929000001, 2024-04-22T18:40:08.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40900504500000001, 2024-04-22T18:40:08.516 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.516 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:08.516 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:08.516 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:08.516 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:08.516 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1410 2024-04-22T18:40:08.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.516 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:08.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:08.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:08.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161972+0000", 2024-04-22T18:40:08.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6419999999999999e-06 2024-04-22T18:40:08.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161976+0000", 2024-04-22T18:40:08.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9260000000000002e-06 2024-04-22T18:40:08.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161979+0000", 2024-04-22T18:40:08.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9359999999999999e-06 2024-04-22T18:40:08.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570790+0000", 2024-04-22T18:40:08.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40881099199999998 2024-04-22T18:40:08.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570879+0000", 2024-04-22T18:40:08.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8895999999999997e-05 2024-04-22T18:40:08.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570976+0000", 2024-04-22T18:40:08.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6923999999999996e-05 2024-04-22T18:40:08.520 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.520 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.520 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.521 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:08.521 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:08.521 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.550105193999997, 2024-04-22T18:40:08.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:40:08.521 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.521 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.521 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:08.521 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:08.521 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:08.521 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:40:08.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.522 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:08.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:08.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:40:08.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:08.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:40:08.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:40:08.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:40:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:40:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:40:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:40:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:40:08.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:40:08.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:40:08.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:40:08.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:08.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:40:08.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:40:08.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:40:08.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:40:08.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:08.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:40:08.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:40:08.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:08.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:40:08.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:40:08.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:40:08.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:40:08.527 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.527 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.527 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.528 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:08.528 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:08.528 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.020945867999998, 2024-04-22T18:40:08.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:40:08.528 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.528 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:08.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:08.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:08.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:08.528 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:40:08.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.529 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:08.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:08.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:08.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:40:08.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:40:08.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:40:08.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:40:08.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:40:08.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:40:08.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:40:08.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:40:08.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:08.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:40:08.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:40:08.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:40:08.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:40:08.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:40:08.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:40:08.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:40:08.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:40:08.533 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.533 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.533 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.534 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:08.534 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:08.534 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.843238473, 2024-04-22T18:40:08.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:40:08.534 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.534 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.534 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:08.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:08.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:40:08.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:08.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:40:08.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:40:08.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:40:08.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:40:08.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:40:08.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:40:08.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:40:08.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:40:08.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:40:08.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:40:08.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:08.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:40:08.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:40:08.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:08.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:40:08.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:40:08.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:40:08.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:40:08.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:08.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:40:08.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:40:08.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:40:08.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:40:08.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.540 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.540 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:08.540 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:08.540 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.843170201, 2024-04-22T18:40:08.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:40:08.541 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.541 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.541 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:08.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:08.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:40:08.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:40:08.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:40:08.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:40:08.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:40:08.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:40:08.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:40:08.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:40:08.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:40:08.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:40:08.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:40:08.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:08.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:40:08.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:40:08.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:08.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:40:08.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:40:08.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:40:08.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:40:08.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:08.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:40:08.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:40:08.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:40:08.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:08.546 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.546 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.546 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.547 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:08.547 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:08.547 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.84300236, 2024-04-22T18:40:08.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:40:08.547 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.547 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.547 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:08.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:08.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:08.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:40:08.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:40:08.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:40:08.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:40:08.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:40:08.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:40:08.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:40:08.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:40:08.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:40:08.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:40:08.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:08.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:40:08.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:40:08.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:08.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:40:08.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:40:08.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:40:08.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:40:08.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:08.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:40:08.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:40:08.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:40:08.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:40:08.553 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.553 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.553 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.553 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:08.553 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:08.553 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.877676885, 2024-04-22T18:40:08.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:40:08.553 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.553 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.554 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:08.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:08.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:40:08.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:08.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:40:08.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:40:08.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:40:08.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:40:08.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:40:08.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:40:08.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:40:08.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:40:08.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:40:08.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:40:08.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:08.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:40:08.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:40:08.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:08.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:40:08.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:40:08.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:40:08.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:40:08.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:08.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:40:08.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:40:08.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:40:08.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:40:08.559 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.559 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.559 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.559 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:08.560 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:08.560 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.777524698000001, 2024-04-22T18:40:08.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:40:08.560 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.560 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.560 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:08.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:08.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:40:08.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:08.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:40:08.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:40:08.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:40:08.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:40:08.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:40:08.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:40:08.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:40:08.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:40:08.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:40:08.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:40:08.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:08.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:40:08.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:40:08.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:08.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:40:08.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:40:08.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:40:08.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:40:08.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:08.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:40:08.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:40:08.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:40:08.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:40:08.565 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.565 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.565 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.566 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2385 4.13 4:c93c842e:::smithi08937929-25:head [write 1937856~527061] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:40:08.566 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.287418+0000", 2024-04-22T18:40:08.566 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.17416764, 2024-04-22T18:40:08.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39986265300000001, 2024-04-22T18:40:08.566 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.566 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.566 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:08.566 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:08.566 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:08.566 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2385 2024-04-22T18:40:08.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.567 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287418+0000", 2024-04-22T18:40:08.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287418+0000", 2024-04-22T18:40:08.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287417+0000", 2024-04-22T18:40:08.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:08.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287636+0000", 2024-04-22T18:40:08.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000219089 2024-04-22T18:40:08.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287654+0000", 2024-04-22T18:40:08.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8294000000000001e-05 2024-04-22T18:40:08.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287661+0000", 2024-04-22T18:40:08.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7780000000000002e-06 2024-04-22T18:40:08.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.483858+0000", 2024-04-22T18:40:08.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19619666199999999 2024-04-22T18:40:08.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.483917+0000", 2024-04-22T18:40:08.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9454999999999999e-05 2024-04-22T18:40:08.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:08.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.483999+0000", 2024-04-22T18:40:08.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1359999999999994e-05 2024-04-22T18:40:08.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:08.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586037+0000", 2024-04-22T18:40:08.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102038217 2024-04-22T18:40:08.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687214+0000", 2024-04-22T18:40:08.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101177195 2024-04-22T18:40:08.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:08.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687253+0000", 2024-04-22T18:40:08.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9409999999999997e-05 2024-04-22T18:40:08.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687281+0000", 2024-04-22T18:40:08.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7478e-05 2024-04-22T18:40:08.572 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.572 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.572 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.572 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2386 4.13 4:c93c842e:::smithi08937929-25:head [write 3252761~175867] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:40:08.573 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.292661+0000", 2024-04-22T18:40:08.573 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.168924992000001, 2024-04-22T18:40:08.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39468563099999998, 2024-04-22T18:40:08.573 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.573 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:08.573 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:08.573 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:08.573 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:08.573 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2386 2024-04-22T18:40:08.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.573 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.292661+0000", 2024-04-22T18:40:08.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.292661+0000", 2024-04-22T18:40:08.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.292659+0000", 2024-04-22T18:40:08.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:08.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.292788+0000", 2024-04-22T18:40:08.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012955 2024-04-22T18:40:08.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.292806+0000", 2024-04-22T18:40:08.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7357999999999999e-05 2024-04-22T18:40:08.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392900+0000", 2024-04-22T18:40:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100093799 2024-04-22T18:40:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.485125+0000", 2024-04-22T18:40:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.092225336000000005 2024-04-22T18:40:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.485172+0000", 2024-04-22T18:40:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7216000000000003e-05 2024-04-22T18:40:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.485244+0000", 2024-04-22T18:40:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1888e-05 2024-04-22T18:40:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.587565+0000", 2024-04-22T18:40:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102321 2024-04-22T18:40:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687303+0000", 2024-04-22T18:40:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099737563000000001 2024-04-22T18:40:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687324+0000", 2024-04-22T18:40:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0868000000000001e-05 2024-04-22T18:40:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687346+0000", 2024-04-22T18:40:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2915999999999999e-05 2024-04-22T18:40:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:40:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.068660177999998, 2024-04-22T18:40:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:40:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:40:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:40:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:40:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:40:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:40:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:40:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:40:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:40:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:40:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:40:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:40:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:40:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:40:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:40:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:40:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:40:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:40:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:40:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.172621806, 2024-04-22T18:40:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:40:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:40:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:40:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:40:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:40:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:40:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:40:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:40:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:40:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:40:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:40:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:40:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:40:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:40:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:08.590 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:08.592 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:08.593 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:08.593+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:08.622 INFO:tasks.rados.rados.0.smithi089.stdout:1295: finishing set_chunk to oid 46 2024-04-22T18:40:08.623 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 46 set_chunk returned error code -95 offset: 74240 length: 1376256 2024-04-22T18:40:08.623 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-22T18:40:08.623 INFO:tasks.rados.rados.0.smithi089.stdout:1296: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:40:08.658 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:08.659+0000 7f7f42baf640 -1 osd.4 77 handle_osd_map injecting map crc failure 2024-04-22T18:40:08.660 INFO:teuthology.orchestra.run.smithi089.stderr:set 4.7 pg_upmap_items mapping to [4->1,7->3] 2024-04-22T18:40:08.715 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:08.716+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:08.729 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:08.729 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:08.730 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:08.730 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:08.739 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:08.762 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:08.763+0000 7f309102d640 -1 osd.1 77 handle_osd_map injecting map crc failure 2024-04-22T18:40:08.769 INFO:tasks.rados.rados.0.smithi089.stdout:1298: expect (ObjNum 166 snap 0 seq_num 166) 2024-04-22T18:40:08.793 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:08.795+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:08.864 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:08.864 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:08.864 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:08.864 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:08.864 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:08.873 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:08.915 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:08.916+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:08.959 INFO:tasks.rados.rados.0.smithi089.stdout:1290: finishing write tid 1 to smithi08937929-33 2024-04-22T18:40:08.959 INFO:tasks.rados.rados.0.smithi089.stdout:1293: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:40:08.996 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:08.997+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:09.020 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:09.020 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:09.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:09.021 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:09.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.021 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:9 4.f 4:f2c48a0e:::smithi08937929-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:09.021 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:09.021 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.902951186999999, 2024-04-22T18:40:09.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49290714600000002, 2024-04-22T18:40:09.021 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.021 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.021 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.021 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.021 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.021 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 9 2024-04-22T18:40:09.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.021 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:09.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:09.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116773+0000", 2024-04-22T18:40:09.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:09.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117024+0000", 2024-04-22T18:40:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025191299999999999 2024-04-22T18:40:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117036+0000", 2024-04-22T18:40:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1595e-05 2024-04-22T18:40:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117046+0000", 2024-04-22T18:40:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0456e-05 2024-04-22T18:40:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414324+0000", 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297277603 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414404+0000", 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9738000000000001e-05 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414494+0000", 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9733000000000003e-05 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518194+0000", 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10370061 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609618+0000", 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091423360999999995 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609655+0000", 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7580000000000003e-05 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609681+0000", 2024-04-22T18:40:09.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5908e-05 2024-04-22T18:40:09.026 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.026 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.026 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.026 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:10 4.f 4:f2c48a0e:::smithi08937929-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:09.026 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:09.026 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.897098927000002, 2024-04-22T18:40:09.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48712681400000002, 2024-04-22T18:40:09.026 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.026 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.026 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.026 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.026 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.027 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 10 2024-04-22T18:40:09.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.027 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:09.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:09.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122625+0000", 2024-04-22T18:40:09.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:09.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122828+0000", 2024-04-22T18:40:09.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020359499999999999 2024-04-22T18:40:09.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122840+0000", 2024-04-22T18:40:09.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1239e-05 2024-04-22T18:40:09.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122845+0000", 2024-04-22T18:40:09.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2719999999999997e-06 2024-04-22T18:40:09.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415105+0000", 2024-04-22T18:40:09.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29225972099999997 2024-04-22T18:40:09.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415169+0000", 2024-04-22T18:40:09.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4170000000000004e-05 2024-04-22T18:40:09.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:09.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415257+0000", 2024-04-22T18:40:09.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8331999999999994e-05 2024-04-22T18:40:09.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:09.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518199+0000", 2024-04-22T18:40:09.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10294174 2024-04-22T18:40:09.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:09.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609707+0000", 2024-04-22T18:40:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091508408999999999 2024-04-22T18:40:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609729+0000", 2024-04-22T18:40:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1917e-05 2024-04-22T18:40:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609753+0000", 2024-04-22T18:40:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3776000000000001e-05 2024-04-22T18:40:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:11 4.f 4:f2c48a0e:::smithi08937929-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.896875554000005, 2024-04-22T18:40:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49452214500000002, 2024-04-22T18:40:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 11 2024-04-22T18:40:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122849+0000", 2024-04-22T18:40:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122853+0000", 2024-04-22T18:40:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9029999999999997e-06 2024-04-22T18:40:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122856+0000", 2024-04-22T18:40:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.01e-06 2024-04-22T18:40:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122859+0000", 2024-04-22T18:40:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415755+0000", 2024-04-22T18:40:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29289606800000001 2024-04-22T18:40:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415808+0000", 2024-04-22T18:40:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3415999999999998e-05 2024-04-22T18:40:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415888+0000", 2024-04-22T18:40:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9061999999999996e-05 2024-04-22T18:40:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518203+0000", 2024-04-22T18:40:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102315533 2024-04-22T18:40:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617164+0000", 2024-04-22T18:40:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098960548999999995 2024-04-22T18:40:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617241+0000", 2024-04-22T18:40:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7292000000000005e-05 2024-04-22T18:40:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617372+0000", 2024-04-22T18:40:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013075399999999999 2024-04-22T18:40:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:12 4.f 4:f2c48a0e:::smithi08937929-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.896863690000004, 2024-04-22T18:40:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49473910999999998, 2024-04-22T18:40:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 12 2024-04-22T18:40:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122863+0000", 2024-04-22T18:40:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.863e-06 2024-04-22T18:40:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122866+0000", 2024-04-22T18:40:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:40:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122869+0000", 2024-04-22T18:40:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5950000000000001e-06 2024-04-22T18:40:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416008+0000", 2024-04-22T18:40:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29313975199999998 2024-04-22T18:40:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416048+0000", 2024-04-22T18:40:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9888000000000002e-05 2024-04-22T18:40:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617447+0000", 2024-04-22T18:40:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20139881400000001 2024-04-22T18:40:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617537+0000", 2024-04-22T18:40:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9606999999999998e-05 2024-04-22T18:40:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617600+0000", 2024-04-22T18:40:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3884e-05 2024-04-22T18:40:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:33 4.f 4:f0ed3d49:::smithi08937929-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.832923488999995, 2024-04-22T18:40:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62359692099999997, 2024-04-22T18:40:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 33 2024-04-22T18:40:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186799+0000", 2024-04-22T18:40:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187103+0000", 2024-04-22T18:40:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030354900000000002 2024-04-22T18:40:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187117+0000", 2024-04-22T18:40:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4578e-05 2024-04-22T18:40:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187129+0000", 2024-04-22T18:40:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1421e-05 2024-04-22T18:40:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516134+0000", 2024-04-22T18:40:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32900553900000001 2024-04-22T18:40:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516215+0000", 2024-04-22T18:40:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0537999999999994e-05 2024-04-22T18:40:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516302+0000", 2024-04-22T18:40:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6601000000000007e-05 2024-04-22T18:40:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518207+0000", 2024-04-22T18:40:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019058370000000001 2024-04-22T18:40:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810302+0000", 2024-04-22T18:40:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29209509700000003 2024-04-22T18:40:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810348+0000", 2024-04-22T18:40:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5992999999999998e-05 2024-04-22T18:40:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810398+0000", 2024-04-22T18:40:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0046000000000002e-05 2024-04-22T18:40:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:34 4.f 4:f0ed3d49:::smithi08937929-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.828321477000003, 2024-04-22T18:40:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.61954655800000002, 2024-04-22T18:40:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34 2024-04-22T18:40:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191401+0000", 2024-04-22T18:40:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191503+0000", 2024-04-22T18:40:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010190400000000001 2024-04-22T18:40:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191517+0000", 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3267e-05 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191523+0000", 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517062+0000", 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32553894300000003 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517141+0000", 2024-04-22T18:40:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9667999999999997e-05 2024-04-22T18:40:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517267+0000", 2024-04-22T18:40:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000125494 2024-04-22T18:40:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518213+0000", 2024-04-22T18:40:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000946275 2024-04-22T18:40:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810901+0000", 2024-04-22T18:40:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29268821699999997 2024-04-22T18:40:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810929+0000", 2024-04-22T18:40:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7645999999999999e-05 2024-04-22T18:40:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810950+0000", 2024-04-22T18:40:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1333999999999999e-05 2024-04-22T18:40:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.828198509000003, 2024-04-22T18:40:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:40:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:40:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:40:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:40:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:40:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:40:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:40:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:40:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:40:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:40:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:40:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:40:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:40:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:40:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:40:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:40:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:40:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:40:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:40:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:40:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:40:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:40:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.828187434, 2024-04-22T18:40:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:40:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:40:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:40:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:40:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:40:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:40:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:40:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:40:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:40:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:40:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:40:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:40:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:40:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.226817940000004, 2024-04-22T18:40:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:40:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:40:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:40:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:40:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:40:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:40:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:40:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:40:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:40:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:40:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:40:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:40:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:40:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:40:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:40:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:40:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:40:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:40:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:40:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:40:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:40:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:40:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:40:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1262 4.7 4:e117a562:::smithi08937929-33:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:40:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.566674763999998, 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40488836700000003, 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1262 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:40:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453048+0000", 2024-04-22T18:40:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453058+0000", 2024-04-22T18:40:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:40:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453069+0000", 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1117999999999999e-05 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453075+0000", 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0909999999999998e-06 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453115+0000", 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0119e-05 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:40:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453119+0000", 2024-04-22T18:40:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3610000000000003e-06 2024-04-22T18:40:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454814+0000", 2024-04-22T18:40:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00169436 2024-04-22T18:40:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454861+0000", 2024-04-22T18:40:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7184000000000001e-05 2024-04-22T18:40:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756925+0000", 2024-04-22T18:40:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30206361799999998 2024-04-22T18:40:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757577+0000", 2024-04-22T18:40:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065199599999999996 2024-04-22T18:40:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857846+0000", 2024-04-22T18:40:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100269369 2024-04-22T18:40:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857896+0000", 2024-04-22T18:40:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9607000000000001e-05 2024-04-22T18:40:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857939+0000", 2024-04-22T18:40:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3149999999999999e-05 2024-04-22T18:40:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2205 4.2 4:4fb26dd8:::smithi08937929-10:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:40:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.629129663000001, 2024-04-22T18:40:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.413791136, 2024-04-22T18:40:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2205 2024-04-22T18:40:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:40:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:40:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:40:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390598+0000", 2024-04-22T18:40:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7630000000000001e-06 2024-04-22T18:40:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390603+0000", 2024-04-22T18:40:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.293e-06 2024-04-22T18:40:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390608+0000", 2024-04-22T18:40:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8790000000000002e-06 2024-04-22T18:40:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390653+0000", 2024-04-22T18:40:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5661999999999998e-05 2024-04-22T18:40:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:40:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390657+0000", 2024-04-22T18:40:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9709999999999998e-06 2024-04-22T18:40:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.592250+0000", 2024-04-22T18:40:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20159280600000001 2024-04-22T18:40:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.592322+0000", 2024-04-22T18:40:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2171999999999999e-05 2024-04-22T18:40:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.804373+0000", 2024-04-22T18:40:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.212050297 2024-04-22T18:40:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.804387+0000", 2024-04-22T18:40:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3859999999999999e-05 2024-04-22T18:40:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:7 3.0 3:1554f608:::d614c1f2f74bb29087660bd6e888eafe46b1a7ed3cbdede85b36de771c7f0d66:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.847540414, 2024-04-22T18:40:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48116529899999999, 2024-04-22T18:40:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172184+0000", 2024-04-22T18:40:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172349+0000", 2024-04-22T18:40:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000165074 2024-04-22T18:40:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172355+0000", 2024-04-22T18:40:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8900000000000004e-06 2024-04-22T18:40:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172359+0000", 2024-04-22T18:40:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.569e-06 2024-04-22T18:40:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271561+0000", 2024-04-22T18:40:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099202569000000004 2024-04-22T18:40:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271659+0000", 2024-04-22T18:40:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8171000000000006e-05 2024-04-22T18:40:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271775+0000", 2024-04-22T18:40:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000116137 2024-04-22T18:40:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.272327+0000", 2024-04-22T18:40:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00055106999999999997 2024-04-22T18:40:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653290+0000", 2024-04-22T18:40:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.38096334500000001 2024-04-22T18:40:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653326+0000", 2024-04-22T18:40:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6168000000000003e-05 2024-04-22T18:40:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653350+0000", 2024-04-22T18:40:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3921999999999999e-05 2024-04-22T18:40:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2708 4.2 4:409e6456:::smithi08937929-11:head [read 0~3294172,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.553783+0000", 2024-04-22T18:40:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.4659420169999997, 2024-04-22T18:40:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.403846977, 2024-04-22T18:40:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2708 2024-04-22T18:40:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553783+0000", 2024-04-22T18:40:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553783+0000", 2024-04-22T18:40:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553783+0000", 2024-04-22T18:40:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553786+0000", 2024-04-22T18:40:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8809999999999998e-06 2024-04-22T18:40:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553791+0000", 2024-04-22T18:40:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7849999999999999e-06 2024-04-22T18:40:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553797+0000", 2024-04-22T18:40:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8629999999999996e-06 2024-04-22T18:40:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655434+0000", 2024-04-22T18:40:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10163662 2024-04-22T18:40:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655539+0000", 2024-04-22T18:40:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105431 2024-04-22T18:40:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.956537+0000", 2024-04-22T18:40:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30099759100000001 2024-04-22T18:40:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.956612+0000", 2024-04-22T18:40:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5407000000000005e-05 2024-04-22T18:40:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.957630+0000", 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001017869 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2709 4.2 4:409e6456:::smithi08937929-11:head [sparse-read 0~2843709] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.553801+0000", 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.4659236890000003, 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40417458899999997, 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2709 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553801+0000", 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553801+0000", 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553801+0000", 2024-04-22T18:40:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553803+0000", 2024-04-22T18:40:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4899999999999999e-06 2024-04-22T18:40:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553808+0000", 2024-04-22T18:40:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9169999999999997e-06 2024-04-22T18:40:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553812+0000", 2024-04-22T18:40:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1470000000000003e-06 2024-04-22T18:40:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655577+0000", 2024-04-22T18:40:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10176484299999999 2024-04-22T18:40:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655684+0000", 2024-04-22T18:40:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107221 2024-04-22T18:40:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.957688+0000", 2024-04-22T18:40:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30200392199999998 2024-04-22T18:40:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.957737+0000", 2024-04-22T18:40:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8362999999999998e-05 2024-04-22T18:40:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.957976+0000", 2024-04-22T18:40:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000239211 2024-04-22T18:40:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2710 4.2 4:409e6456:::smithi08937929-11:head [read 0~3294172,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.553816+0000", 2024-04-22T18:40:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.4659090780000001, 2024-04-22T18:40:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40483363500000002, 2024-04-22T18:40:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2710 2024-04-22T18:40:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553816+0000", 2024-04-22T18:40:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553816+0000", 2024-04-22T18:40:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553815+0000", 2024-04-22T18:40:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553819+0000", 2024-04-22T18:40:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3220000000000001e-06 2024-04-22T18:40:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553824+0000", 2024-04-22T18:40:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8879999999999996e-06 2024-04-22T18:40:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553828+0000", 2024-04-22T18:40:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.245e-06 2024-04-22T18:40:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655717+0000", 2024-04-22T18:40:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10188926500000001 2024-04-22T18:40:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655784+0000", 2024-04-22T18:40:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6308000000000003e-05 2024-04-22T18:40:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.957997+0000", 2024-04-22T18:40:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.302213385 2024-04-22T18:40:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.958037+0000", 2024-04-22T18:40:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470999999999999e-05 2024-04-22T18:40:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.958650+0000", 2024-04-22T18:40:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061223400000000002 2024-04-22T18:40:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:166 3.0 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.317111+0000", 2024-04-22T18:40:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.7026138199999998, 2024-04-22T18:40:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.379585277, 2024-04-22T18:40:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317111+0000", 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317111+0000", 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317111+0000", 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317206+0000", 2024-04-22T18:40:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4988999999999995e-05 2024-04-22T18:40:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317213+0000", 2024-04-22T18:40:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.391e-06 2024-04-22T18:40:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317219+0000", 2024-04-22T18:40:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6330000000000001e-06 2024-04-22T18:40:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.346889+0000", 2024-04-22T18:40:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.02967036 2024-04-22T18:40:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.346967+0000", 2024-04-22T18:40:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8582000000000001e-05 2024-04-22T18:40:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.347098+0000", 2024-04-22T18:40:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013041699999999999 2024-04-22T18:40:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.420231+0000", 2024-04-22T18:40:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.073133190000000001 2024-04-22T18:40:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696579+0000", 2024-04-22T18:40:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.27634776999999999 2024-04-22T18:40:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696649+0000", 2024-04-22T18:40:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9830000000000001e-05 2024-04-22T18:40:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696697+0000", 2024-04-22T18:40:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7825999999999999e-05 2024-04-22T18:40:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:167 3.0 3:384c1ce2:::81dc8c58ebe106ae6fa9feadc06291c645d9079ad84b78214d4e6440fbc5def8:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.317224+0000", 2024-04-22T18:40:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.7025010040000002, 2024-04-22T18:40:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37960258600000002, 2024-04-22T18:40:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317224+0000", 2024-04-22T18:40:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317224+0000", 2024-04-22T18:40:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317223+0000", 2024-04-22T18:40:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317273+0000", 2024-04-22T18:40:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9548000000000001e-05 2024-04-22T18:40:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317280+0000", 2024-04-22T18:40:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:40:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317286+0000", 2024-04-22T18:40:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1240000000000002e-06 2024-04-22T18:40:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.347517+0000", 2024-04-22T18:40:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.030231428000000001 2024-04-22T18:40:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.347584+0000", 2024-04-22T18:40:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6669999999999997e-05 2024-04-22T18:40:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.347722+0000", 2024-04-22T18:40:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013773100000000001 2024-04-22T18:40:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.420237+0000", 2024-04-22T18:40:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.072515653999999999 2024-04-22T18:40:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696747+0000", 2024-04-22T18:40:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.27650990199999997 2024-04-22T18:40:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696792+0000", 2024-04-22T18:40:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4591e-05 2024-04-22T18:40:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696827+0000", 2024-04-22T18:40:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4876000000000003e-05 2024-04-22T18:40:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:172 3.0 3:1e450458:::73c5b8dc943cb866ee2d52255e62488506020b89464b8097642f92f95390407a:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.317290+0000", 2024-04-22T18:40:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.7024349729999999, 2024-04-22T18:40:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37964102999999999, 2024-04-22T18:40:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317290+0000", 2024-04-22T18:40:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317290+0000", 2024-04-22T18:40:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317289+0000", 2024-04-22T18:40:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317359+0000", 2024-04-22T18:40:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9906000000000005e-05 2024-04-22T18:40:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317367+0000", 2024-04-22T18:40:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4370000000000001e-06 2024-04-22T18:40:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317376+0000", 2024-04-22T18:40:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3179999999999996e-06 2024-04-22T18:40:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.419267+0000", 2024-04-22T18:40:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10189088 2024-04-22T18:40:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.419371+0000", 2024-04-22T18:40:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010370300000000001 2024-04-22T18:40:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.419590+0000", 2024-04-22T18:40:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021932700000000001 2024-04-22T18:40:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.420825+0000", 2024-04-22T18:40:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012349679999999999 2024-04-22T18:40:09.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:09.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696860+0000", 2024-04-22T18:40:09.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.27603527900000002 2024-04-22T18:40:09.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696901+0000", 2024-04-22T18:40:09.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0515000000000001e-05 2024-04-22T18:40:09.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696931+0000", 2024-04-22T18:40:09.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0300000000000001e-05 2024-04-22T18:40:09.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.118 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.118 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2845 4.7 4:e117a562:::smithi08937929-33:head [setxattr _header (56) in=7b,truncate 2033680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-04-22T18:40:09.118 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:08.157995+0000", 2024-04-22T18:40:09.118 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.86173036599999997, 2024-04-22T18:40:09.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.416467168, 2024-04-22T18:40:09.118 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.118 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.118 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.118 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.119 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.119 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2845 2024-04-22T18:40:09.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.119 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.157995+0000", 2024-04-22T18:40:09.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.157995+0000", 2024-04-22T18:40:09.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.157994+0000", 2024-04-22T18:40:09.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.157998+0000", 2024-04-22T18:40:09.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.631e-06 2024-04-22T18:40:09.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158001+0000", 2024-04-22T18:40:09.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:40:09.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158004+0000", 2024-04-22T18:40:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9610000000000001e-06 2024-04-22T18:40:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.354298+0000", 2024-04-22T18:40:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19629424500000001 2024-04-22T18:40:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.354367+0000", 2024-04-22T18:40:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9016999999999993e-05 2024-04-22T18:40:09.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:09.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.354470+0000", 2024-04-22T18:40:09.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103365 2024-04-22T18:40:09.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:09.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.455161+0000", 2024-04-22T18:40:09.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100690442 2024-04-22T18:40:09.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:09.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574379+0000", 2024-04-22T18:40:09.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.119217818 2024-04-22T18:40:09.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574406+0000", 2024-04-22T18:40:09.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7712000000000002e-05 2024-04-22T18:40:09.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574462+0000", 2024-04-22T18:40:09.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5405000000000002e-05 2024-04-22T18:40:09.123 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.123 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.123 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.124 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2846 4.7 4:e117a562:::smithi08937929-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-04-22T18:40:09.124 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:08.158020+0000", 2024-04-22T18:40:09.124 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.86170540100000004, 2024-04-22T18:40:09.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41656018099999997, 2024-04-22T18:40:09.124 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.124 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:09.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:09.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:09.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:09.124 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2846 2024-04-22T18:40:09.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.124 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158020+0000", 2024-04-22T18:40:09.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158020+0000", 2024-04-22T18:40:09.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158019+0000", 2024-04-22T18:40:09.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158021+0000", 2024-04-22T18:40:09.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.928e-06 2024-04-22T18:40:09.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158026+0000", 2024-04-22T18:40:09.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5029999999999999e-06 2024-04-22T18:40:09.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158029+0000", 2024-04-22T18:40:09.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5889999999999999e-06 2024-04-22T18:40:09.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.455080+0000", 2024-04-22T18:40:09.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29705077499999999 2024-04-22T18:40:09.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:09.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.455142+0000", 2024-04-22T18:40:09.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1982000000000004e-05 2024-04-22T18:40:09.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574482+0000", 2024-04-22T18:40:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.119340448 2024-04-22T18:40:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574529+0000", 2024-04-22T18:40:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6570999999999998e-05 2024-04-22T18:40:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574580+0000", 2024-04-22T18:40:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0787e-05 2024-04-22T18:40:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.128 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.129 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.129 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:09.130 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:09.130 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:09.131+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:09.167 INFO:tasks.rados.rados.0.smithi089.stdout:1290: finishing write tid 2 to smithi08937929-33 2024-04-22T18:40:09.167 INFO:tasks.rados.rados.0.smithi089.stdout:1297: finishing write tid 1 to smithi08937929-28 2024-04-22T18:40:09.168 INFO:tasks.rados.rados.0.smithi089.stdout:1297: finishing write tid 2 to smithi08937929-28 2024-04-22T18:40:09.168 INFO:tasks.rados.rados.0.smithi089.stdout:1290: finishing write tid 3 to smithi08937929-33 2024-04-22T18:40:09.168 INFO:tasks.rados.rados.0.smithi089.stdout:1299: seq_num 242 ranges {467294=617201,1578407=488887} 2024-04-22T18:40:09.173 INFO:tasks.rados.rados.0.smithi089.stdout:1299: writing smithi08937929-9 from 467294 to 1084495 tid 1 2024-04-22T18:40:09.182 INFO:tasks.rados.rados.0.smithi089.stdout:1299: writing smithi08937929-9 from 1578407 to 2067294 tid 2 2024-04-22T18:40:09.182 INFO:tasks.rados.rados.0.smithi089.stdout:1288: done (10 left) 2024-04-22T18:40:09.182 INFO:tasks.rados.rados.0.smithi089.stdout:1291: done (9 left) 2024-04-22T18:40:09.182 INFO:tasks.rados.rados.0.smithi089.stdout:1292: done (8 left) 2024-04-22T18:40:09.182 INFO:tasks.rados.rados.0.smithi089.stdout:1293: done (7 left) 2024-04-22T18:40:09.182 INFO:tasks.rados.rados.0.smithi089.stdout:1294: done (6 left) 2024-04-22T18:40:09.183 INFO:tasks.rados.rados.0.smithi089.stdout:1295: done (5 left) 2024-04-22T18:40:09.183 INFO:tasks.rados.rados.0.smithi089.stdout:1296: done (4 left) 2024-04-22T18:40:09.183 INFO:tasks.rados.rados.0.smithi089.stdout:1298: done (3 left) 2024-04-22T18:40:09.183 INFO:tasks.rados.rados.0.smithi089.stdout:1300: tier_flush oid 29 2024-04-22T18:40:09.183 INFO:tasks.rados.rados.0.smithi089.stdout:1300: tier_flush oid 29 snap -1 2024-04-22T18:40:09.183 INFO:tasks.rados.rados.0.smithi089.stdout:1290: finishing write tid 4 to smithi08937929-33 2024-04-22T18:40:09.183 INFO:tasks.rados.rados.0.smithi089.stdout:1290: oid 33 updating version 0 to 42 2024-04-22T18:40:09.183 INFO:tasks.rados.rados.0.smithi089.stdout:1290: oid 33 version 42 is already newer than 40 2024-04-22T18:40:09.183 INFO:tasks.rados.rados.0.smithi089.stdout:1290: oid 33 version 42 is already newer than 41 2024-04-22T18:40:09.183 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 42 (ObjNum 240 snap 0 seq_num 240) dirty exists 2024-04-22T18:40:09.183 INFO:tasks.rados.rados.0.smithi089.stdout:1290: left oid 33 (ObjNum 240 snap 0 seq_num 240) 2024-04-22T18:40:09.183 INFO:tasks.rados.rados.0.smithi089.stdout:1297: finishing write tid 3 to smithi08937929-28 2024-04-22T18:40:09.183 INFO:tasks.rados.rados.0.smithi089.stdout:1290: done (3 left) 2024-04-22T18:40:09.183 INFO:tasks.rados.rados.0.smithi089.stdout:1301: write (excl) oid 30 current snap is 0 2024-04-22T18:40:09.184 INFO:tasks.rados.rados.0.smithi089.stdout:1297: finishing write tid 4 to smithi08937929-28 2024-04-22T18:40:09.184 INFO:tasks.rados.rados.0.smithi089.stdout:1297: oid 28 updating version 0 to 123 2024-04-22T18:40:09.184 INFO:tasks.rados.rados.0.smithi089.stdout:1297: oid 28 updating version 123 to 124 2024-04-22T18:40:09.184 INFO:tasks.rados.rados.0.smithi089.stdout:1297: oid 28 version 124 is already newer than 122 2024-04-22T18:40:09.184 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 124 (ObjNum 241 snap 0 seq_num 241) dirty exists 2024-04-22T18:40:09.184 INFO:tasks.rados.rados.0.smithi089.stdout:1297: left oid 28 (ObjNum 241 snap 0 seq_num 241) 2024-04-22T18:40:09.184 INFO:tasks.rados.rados.0.smithi089.stdout:1301: seq_num 243 ranges {484101=692450,1636889=447212} 2024-04-22T18:40:09.185 INFO:tasks.rados.rados.0.smithi089.stdout:1301: writing smithi08937929-30 from 484101 to 1176551 tid 1 2024-04-22T18:40:09.190 INFO:tasks.rados.rados.0.smithi089.stdout:1301: writing smithi08937929-30 from 1636889 to 2084101 tid 2 2024-04-22T18:40:09.190 INFO:tasks.rados.rados.0.smithi089.stdout:1297: done (3 left) 2024-04-22T18:40:09.190 INFO:tasks.rados.rados.0.smithi089.stdout:1302: tier_flush oid 37 2024-04-22T18:40:09.190 INFO:tasks.rados.rados.0.smithi089.stdout:1302: tier_flush oid 37 snap -1 2024-04-22T18:40:09.190 INFO:tasks.rados.rados.0.smithi089.stdout:1303: write (excl) oid 33 current snap is 0 2024-04-22T18:40:09.191 INFO:tasks.rados.rados.0.smithi089.stdout:1303: seq_num 244 ranges {500908=484052,1411724=689184} 2024-04-22T18:40:09.195 INFO:tasks.rados.rados.0.smithi089.stdout:1303: writing smithi08937929-33 from 500908 to 984960 tid 1 2024-04-22T18:40:09.202 INFO:tasks.rados.rados.0.smithi089.stdout:1303: writing smithi08937929-33 from 1411724 to 2100908 tid 2 2024-04-22T18:40:09.202 INFO:tasks.rados.rados.0.smithi089.stdout:1304: read oid 8 snap -1 2024-04-22T18:40:09.202 INFO:tasks.rados.rados.0.smithi089.stdout:1304: expect (ObjNum 219 snap 0 seq_num 219) 2024-04-22T18:40:09.202 INFO:tasks.rados.rados.0.smithi089.stdout:1305: read oid 35 snap -1 2024-04-22T18:40:09.202 INFO:tasks.rados.rados.0.smithi089.stdout:1305: expect (ObjNum 212 snap 0 seq_num 212) 2024-04-22T18:40:09.202 INFO:tasks.rados.rados.0.smithi089.stdout:1306: set_chunk oid 17 target oid 2024-04-22T18:40:09.230 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:09.232+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:09.271 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:09.272 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:09.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.272 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2877 4.d e78/63 4:bc165c76:::smithi08937929-30:head v 78'41, mlcod=66'39)", 2024-04-22T18:40:09.272 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.188785+0000", 2024-04-22T18:40:09.272 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.084733181000000005, 2024-04-22T18:40:09.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.084751971999999995, 2024-04-22T18:40:09.272 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.272 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:09.272 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.188785+0000", 2024-04-22T18:40:09.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.188785+0000", 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.188784+0000", 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.189629+0000", 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084526200000000005 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.189648+0000", 2024-04-22T18:40:09.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8399000000000001e-05 2024-04-22T18:40:09.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.189658+0000", 2024-04-22T18:40:09.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9990000000000003e-06 2024-04-22T18:40:09.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.193736+0000", 2024-04-22T18:40:09.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040777970000000002 2024-04-22T18:40:09.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.193761+0000", 2024-04-22T18:40:09.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5035999999999999e-05 2024-04-22T18:40:09.275 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.275 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.275 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.275 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2878 4.d e78/63 4:bc165c76:::smithi08937929-30:head v 78'42, mlcod=66'39)", 2024-04-22T18:40:09.275 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.193258+0000", 2024-04-22T18:40:09.275 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.080259393999999998, 2024-04-22T18:40:09.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.080311564000000002, 2024-04-22T18:40:09.276 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.276 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:09.276 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.193258+0000", 2024-04-22T18:40:09.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.193258+0000", 2024-04-22T18:40:09.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.193257+0000", 2024-04-22T18:40:09.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:09.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.193667+0000", 2024-04-22T18:40:09.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041062999999999999 2024-04-22T18:40:09.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.193679+0000", 2024-04-22T18:40:09.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1548e-05 2024-04-22T18:40:09.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.193686+0000", 2024-04-22T18:40:09.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5490000000000003e-06 2024-04-22T18:40:09.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.195170+0000", 2024-04-22T18:40:09.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0014845870000000001 2024-04-22T18:40:09.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.195198+0000", 2024-04-22T18:40:09.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8235999999999999e-05 2024-04-22T18:40:09.278 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.279 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.279 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.279 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2879 4.d e78/63)", 2024-04-22T18:40:09.279 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.194542+0000", 2024-04-22T18:40:09.279 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.078975285000000006, 2024-04-22T18:40:09.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.079054466000000004, 2024-04-22T18:40:09.279 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.279 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:09.279 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.194542+0000", 2024-04-22T18:40:09.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.194542+0000", 2024-04-22T18:40:09.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.194541+0000", 2024-04-22T18:40:09.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:09.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.194550+0000", 2024-04-22T18:40:09.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4449999999999996e-06 2024-04-22T18:40:09.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.194557+0000", 2024-04-22T18:40:09.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7880000000000007e-06 2024-04-22T18:40:09.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.194564+0000", 2024-04-22T18:40:09.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2999999999999998e-06 2024-04-22T18:40:09.281 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.282 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.282 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.282 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.282 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:09.282 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 3 2024-04-22T18:40:09.282 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:09.283 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:09.336 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:40:09.367 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:09.368+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:09.427 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:09.427 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:09.427 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:09.427 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:09.427 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:09.433 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:09.434+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:09.433 INFO:tasks.rados.rados.0.smithi089.stdout:1306: set_chunk oid 17 offset: 1440256 length: 1546752 target oid 2109e7e4526cbe26b2963cdd11634231ee7f4d35dabe9962c44b62fafb130ddf offset: 0 snap -1 2024-04-22T18:40:09.433 INFO:tasks.rados.rados.0.smithi089.stdout:1307: write (excl) oid 12 current snap is 0 2024-04-22T18:40:09.433 INFO:tasks.rados.rados.0.smithi089.stdout:1307: seq_num 245 ranges {517715=559301,1870206=247509} 2024-04-22T18:40:09.438 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:09.439 INFO:tasks.rados.rados.0.smithi089.stdout:1307: writing smithi08937929-12 from 517715 to 1077016 tid 1 2024-04-22T18:40:09.445 INFO:tasks.rados.rados.0.smithi089.stdout:1307: writing smithi08937929-12 from 1870206 to 2117715 tid 2 2024-04-22T18:40:09.445 INFO:tasks.rados.rados.0.smithi089.stdout:1308: read oid 49 snap -1 2024-04-22T18:40:09.445 INFO:tasks.rados.rados.0.smithi089.stdout:1308: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-22T18:40:09.445 INFO:tasks.rados.rados.0.smithi089.stdout:1309: write oid 48 current snap is 0 2024-04-22T18:40:09.445 INFO:tasks.rados.rados.0.smithi089.stdout:1309: seq_num 246 ranges {534522=634550,1812335=322187} 2024-04-22T18:40:09.451 INFO:tasks.rados.rados.0.smithi089.stdout:1309: writing smithi08937929-48 from 534522 to 1169072 tid 1 2024-04-22T18:40:09.457 INFO:tasks.rados.rados.0.smithi089.stdout:1309: writing smithi08937929-48 from 1812335 to 2134522 tid 2 2024-04-22T18:40:09.457 INFO:tasks.rados.rados.0.smithi089.stdout:1310: read oid 20 snap -1 2024-04-22T18:40:09.458 INFO:tasks.rados.rados.0.smithi089.stdout:1310: expect (ObjNum 201 snap 0 seq_num 201) 2024-04-22T18:40:09.458 INFO:tasks.rados.rados.0.smithi089.stdout:1311: write (excl) oid 46 current snap is 0 2024-04-22T18:40:09.458 INFO:tasks.rados.rados.0.smithi089.stdout:1299: finishing write tid 1 to smithi08937929-9 2024-04-22T18:40:09.458 INFO:tasks.rados.rados.0.smithi089.stdout:1299: finishing write tid 2 to smithi08937929-9 2024-04-22T18:40:09.458 INFO:tasks.rados.rados.0.smithi089.stdout:1299: finishing write tid 3 to smithi08937929-9 2024-04-22T18:40:09.458 INFO:tasks.rados.rados.0.smithi089.stdout:1299: finishing write tid 4 to smithi08937929-9 2024-04-22T18:40:09.458 INFO:tasks.rados.rados.0.smithi089.stdout:1299: oid 9 updating version 0 to 60 2024-04-22T18:40:09.458 INFO:tasks.rados.rados.0.smithi089.stdout:1299: oid 9 updating version 60 to 61 2024-04-22T18:40:09.458 INFO:tasks.rados.rados.0.smithi089.stdout:1299: oid 9 version 61 is already newer than 59 2024-04-22T18:40:09.458 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 61 (ObjNum 242 snap 0 seq_num 242) dirty exists 2024-04-22T18:40:09.458 INFO:tasks.rados.rados.0.smithi089.stdout:1299: left oid 9 (ObjNum 242 snap 0 seq_num 242) 2024-04-22T18:40:09.458 INFO:tasks.rados.rados.0.smithi089.stdout:1311: seq_num 247 ranges {551329=709799,1870817=280512} 2024-04-22T18:40:09.464 INFO:tasks.rados.rados.0.smithi089.stdout:1311: writing smithi08937929-46 from 551329 to 1261128 tid 1 2024-04-22T18:40:09.470 INFO:tasks.rados.rados.0.smithi089.stdout:1311: writing smithi08937929-46 from 1870817 to 2151329 tid 2 2024-04-22T18:40:09.470 INFO:tasks.rados.rados.0.smithi089.stdout:1299: done (12 left) 2024-04-22T18:40:09.470 INFO:tasks.rados.rados.0.smithi089.stdout:1312: tier_promote oid 25 2024-04-22T18:40:09.470 INFO:tasks.rados.rados.0.smithi089.stdout:1313: tier_evict oid 16 2024-04-22T18:40:09.470 INFO:tasks.rados.rados.0.smithi089.stdout:1313: tier_evict oid 16 snap -1 2024-04-22T18:40:09.470 INFO:tasks.rados.rados.0.smithi089.stdout:1314: set_chunk oid 21 target oid 2024-04-22T18:40:09.470 INFO:tasks.rados.rados.0.smithi089.stdout:1314: set_chunk oid 21 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:40:09.470 INFO:tasks.rados.rados.0.smithi089.stdout:1315: write oid 24 current snap is 0 2024-04-22T18:40:09.470 INFO:tasks.rados.rados.0.smithi089.stdout:1315: seq_num 248 ranges {568136=785048,1929299=238837} 2024-04-22T18:40:09.476 INFO:tasks.rados.rados.0.smithi089.stdout:1315: writing smithi08937929-24 from 568136 to 1353184 tid 1 2024-04-22T18:40:09.481 INFO:tasks.rados.rados.0.smithi089.stdout:1315: writing smithi08937929-24 from 1929299 to 2168136 tid 2 2024-04-22T18:40:09.481 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:09.482 INFO:tasks.rados.rados.0.smithi089.stdout:1314: finishing set_chunk to oid 21 2024-04-22T18:40:09.482 INFO:tasks.rados.rados.0.smithi089.stdout:1314: got expected ENOENT (src dne) 2024-04-22T18:40:09.482 INFO:tasks.rados.rados.0.smithi089.stdout:1314: done (15 left) 2024-04-22T18:40:09.482 INFO:tasks.rados.rados.0.smithi089.stdout:1316: read oid 6 snap -1 2024-04-22T18:40:09.482 INFO:tasks.rados.rados.0.smithi089.stdout:1316: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:40:09.482 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:09.533 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:09.535+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:09.539 INFO:tasks.rados.rados.0.smithi089.stdout:1303: finishing write tid 1 to smithi08937929-33 2024-04-22T18:40:09.539 INFO:tasks.rados.rados.0.smithi089.stdout:1303: finishing write tid 2 to smithi08937929-33 2024-04-22T18:40:09.539 INFO:tasks.rados.rados.0.smithi089.stdout:1303: finishing write tid 3 to smithi08937929-33 2024-04-22T18:40:09.539 INFO:tasks.rados.rados.0.smithi089.stdout:1303: finishing write tid 4 to smithi08937929-33 2024-04-22T18:40:09.539 INFO:tasks.rados.rados.0.smithi089.stdout:1303: oid 33 updating version 0 to 45 2024-04-22T18:40:09.539 INFO:tasks.rados.rados.0.smithi089.stdout:1303: oid 33 version 45 is already newer than 43 2024-04-22T18:40:09.539 INFO:tasks.rados.rados.0.smithi089.stdout:1303: oid 33 version 45 is already newer than 44 2024-04-22T18:40:09.539 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 45 (ObjNum 244 snap 0 seq_num 244) dirty exists 2024-04-22T18:40:09.539 INFO:tasks.rados.rados.0.smithi089.stdout:1303: left oid 33 (ObjNum 244 snap 0 seq_num 244) 2024-04-22T18:40:09.539 INFO:tasks.rados.rados.0.smithi089.stdout:1303: done (15 left) 2024-04-22T18:40:09.540 INFO:tasks.rados.rados.0.smithi089.stdout:1317: read oid 11 snap -1 2024-04-22T18:40:09.540 INFO:tasks.rados.rados.0.smithi089.stdout:1317: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:40:09.540 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:09.546 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:73 3.0 e34/19 3:061a7a63:::392f44ec26aa22e85be5f08273a7e45656f3bf4a4854038391a88e880ec01077:head v 34'31, mlcod=31'29)", 2024-04-22T18:40:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.088644363, 2024-04-22T18:40:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103503311, 2024-04-22T18:40:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458193+0000", 2024-04-22T18:40:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026628900000000001 2024-04-22T18:40:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458208+0000", 2024-04-22T18:40:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4780000000000001e-05 2024-04-22T18:40:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458217+0000", 2024-04-22T18:40:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4909999999999994e-06 2024-04-22T18:40:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459162+0000", 2024-04-22T18:40:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094440500000000005 2024-04-22T18:40:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459185+0000", 2024-04-22T18:40:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3685000000000001e-05 2024-04-22T18:40:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561394+0000", 2024-04-22T18:40:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10220874100000001 2024-04-22T18:40:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561430+0000", 2024-04-22T18:40:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6304999999999999e-05 2024-04-22T18:40:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:75 3.0 e34/19 3:18418e0f:::b7c35f165c374d5e075dde9ac2607e47ace3b0a1e2343ec6efb171380f725eea:head v 34'32, mlcod=31'29)", 2024-04-22T18:40:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.087631090000002, 2024-04-22T18:40:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104798078, 2024-04-22T18:40:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458939+0000", 2024-04-22T18:40:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459092+0000", 2024-04-22T18:40:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153838 2024-04-22T18:40:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459107+0000", 2024-04-22T18:40:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4703e-05 2024-04-22T18:40:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459115+0000", 2024-04-22T18:40:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-04-22T18:40:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560093+0000", 2024-04-22T18:40:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10097840499999999 2024-04-22T18:40:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560109+0000", 2024-04-22T18:40:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5631e-05 2024-04-22T18:40:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563706+0000", 2024-04-22T18:40:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035972059999999999 2024-04-22T18:40:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563738+0000", 2024-04-22T18:40:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2184000000000002e-05 2024-04-22T18:40:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:77 3.0 e34/19 3:28387c9f:::b6d7840f02d561da04e398fbf94d91b85e9066b426e48baf3b11dd4a4f6096c0:head v 34'33, mlcod=31'29)", 2024-04-22T18:40:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.086681010000007, 2024-04-22T18:40:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104186079, 2024-04-22T18:40:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459889+0000", 2024-04-22T18:40:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459963+0000", 2024-04-22T18:40:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3732999999999994e-05 2024-04-22T18:40:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459973+0000", 2024-04-22T18:40:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0285999999999999e-05 2024-04-22T18:40:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459987+0000", 2024-04-22T18:40:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3675999999999999e-05 2024-04-22T18:40:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560515+0000", 2024-04-22T18:40:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10052781299999999 2024-04-22T18:40:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560538+0000", 2024-04-22T18:40:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3861000000000001e-05 2024-04-22T18:40:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564044+0000", 2024-04-22T18:40:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035052009999999999 2024-04-22T18:40:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564076+0000", 2024-04-22T18:40:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2737000000000002e-05 2024-04-22T18:40:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:78 3.0 e34/19 3:3a5582ac:::bdd92f7deaa2d50e204891f64f9043469ba687d78b218ed518f2483ba3486a99:head v 34'34, mlcod=31'29)", 2024-04-22T18:40:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.086014891999994, 2024-04-22T18:40:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103547263, 2024-04-22T18:40:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460555+0000", 2024-04-22T18:40:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460861+0000", 2024-04-22T18:40:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000305958 2024-04-22T18:40:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460874+0000", 2024-04-22T18:40:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2639e-05 2024-04-22T18:40:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460881+0000", 2024-04-22T18:40:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2570000000000001e-06 2024-04-22T18:40:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560990+0000", 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010883199999999 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561011+0000", 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1319e-05 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564079+0000", 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00306815 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564104+0000", 2024-04-22T18:40:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4172000000000002e-05 2024-04-22T18:40:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:40:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.085399597999995, 2024-04-22T18:40:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:40:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:40:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:40:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:40:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:40:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:40:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:40:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:40:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:40:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:40:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:40:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:40:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:40:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:40:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:40:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:40:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:40:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.208725776000001, 2024-04-22T18:40:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:40:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:40:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:40:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:40:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:40:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:40:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:40:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:40:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:40:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:40:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1689 3.0 e47/19 3:1d4bcadf:::e7eb8db0531d9f456e91fe87bc79e47ebd80459f1783dc2a5e53d4822cd34f49:head v 47'85, mlcod=47'83)", 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.837683318000003, 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102152033, 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:40:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708885+0000", 2024-04-22T18:40:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709414+0000", 2024-04-22T18:40:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000528459 2024-04-22T18:40:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709428+0000", 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4745e-05 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709436+0000", 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0830000000000004e-06 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.809587+0000", 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100150641 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.809603+0000", 2024-04-22T18:40:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6061000000000001e-05 2024-04-22T18:40:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.811013+0000", 2024-04-22T18:40:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0014094680000000001 2024-04-22T18:40:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.811040+0000", 2024-04-22T18:40:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7474000000000002e-05 2024-04-22T18:40:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1729 3.0 e47/19 3:27b110a1:::34336650543f8f8bff225091f9c8c14678dc47baabcb4d8371890fc935dbc3e4:head v 47'89, mlcod=47'87)", 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.068629547999997, 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102111412, 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:40:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477940+0000", 2024-04-22T18:40:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478305+0000", 2024-04-22T18:40:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036512200000000001 2024-04-22T18:40:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478320+0000", 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5475999999999999e-05 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478326+0000", 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.302e-06 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.578536+0000", 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100209436 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.578575+0000", 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9223e-05 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.580023+0000", 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001447463 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.580053+0000", 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0534999999999998e-05 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(3.0 65/63 [PushOp(3:0f5f80b3:::e65ec2deb1a27f55813533c3e73a682460e829b5dd6586a33a43649649fa77f6:head, version: 53'111, data_included: [0~589824,917504~221696], data_size: 811520, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:0f5f80b3:::e65ec2deb1a27f55813533c3e73a682460e829b5dd6586a33a43649649fa77f6:head@53'111, size: 1139200, copy_subset: [0~1139200], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1139200, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:54.953508+0000", 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.593063179, 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101969016, 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.953508+0000", 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.953508+0000", 2024-04-22T18:40:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.953506+0000", 2024-04-22T18:40:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.953962+0000", 2024-04-22T18:40:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00045575900000000003 2024-04-22T18:40:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.953984+0000", 2024-04-22T18:40:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1579000000000001e-05 2024-04-22T18:40:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.054112+0000", 2024-04-22T18:40:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100128071 2024-04-22T18:40:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.054190+0000", 2024-04-22T18:40:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8047999999999995e-05 2024-04-22T18:40:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.054205+0000", 2024-04-22T18:40:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5273000000000001e-05 2024-04-22T18:40:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.055477+0000", 2024-04-22T18:40:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001272053 2024-04-22T18:40:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2699 3.0 e73/63 3:20ef4e39:::73f6cd42eb17b7fbc29bb083f5443f1d5c65f51bcb8aa3d092b3b80aa1d4c657:head v 73'144, mlcod=71'142)", 2024-04-22T18:40:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.350683+0000", 2024-04-22T18:40:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.1958879160000002, 2024-04-22T18:40:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10232055599999999, 2024-04-22T18:40:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.350683+0000", 2024-04-22T18:40:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.350683+0000", 2024-04-22T18:40:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.350680+0000", 2024-04-22T18:40:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351512+0000", 2024-04-22T18:40:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00083224099999999995 2024-04-22T18:40:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351524+0000", 2024-04-22T18:40:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1763000000000001e-05 2024-04-22T18:40:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351531+0000", 2024-04-22T18:40:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0670000000000003e-06 2024-04-22T18:40:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351596+0000", 2024-04-22T18:40:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4659999999999994e-05 2024-04-22T18:40:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351616+0000", 2024-04-22T18:40:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0596000000000001e-05 2024-04-22T18:40:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.452977+0000", 2024-04-22T18:40:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101360332 2024-04-22T18:40:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.453004+0000", 2024-04-22T18:40:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7127e-05 2024-04-22T18:40:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.7.13:5 3.0 e73/63 3:0460d980:::f535329b03f52d4e4d8e7478ca44bc24de62c1ca53882f92d791340fd034cd80:head v 73'145, mlcod=71'142)", 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.351538+0000", 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.1950330109999996, 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101716578, 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351538+0000", 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351538+0000", 2024-04-22T18:40:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351538+0000", 2024-04-22T18:40:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351558+0000", 2024-04-22T18:40:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0599e-05 2024-04-22T18:40:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351562+0000", 2024-04-22T18:40:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.512e-06 2024-04-22T18:40:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351564+0000", 2024-04-22T18:40:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3829999999999999e-06 2024-04-22T18:40:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.452770+0000", 2024-04-22T18:40:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10120549500000001 2024-04-22T18:40:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.452789+0000", 2024-04-22T18:40:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9624999999999999e-05 2024-04-22T18:40:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.453234+0000", 2024-04-22T18:40:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044458600000000002 2024-04-22T18:40:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.453255+0000", 2024-04-22T18:40:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0812999999999999e-05 2024-04-22T18:40:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.7.13:7 3.0 e73/63 3:1554f608:::d614c1f2f74bb29087660bd6e888eafe46b1a7ed3cbdede85b36de771c7f0d66:head v 73'146, mlcod=71'142)", 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.351566+0000", 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.1950049360000001, 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102004704, 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351566+0000", 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351566+0000", 2024-04-22T18:40:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351566+0000", 2024-04-22T18:40:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:40:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351570+0000", 2024-04-22T18:40:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1649999999999999e-06 2024-04-22T18:40:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351573+0000", 2024-04-22T18:40:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2189999999999998e-06 2024-04-22T18:40:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351576+0000", 2024-04-22T18:40:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7689999999999998e-06 2024-04-22T18:40:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.453030+0000", 2024-04-22T18:40:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101453997 2024-04-22T18:40:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.453046+0000", 2024-04-22T18:40:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5634e-05 2024-04-22T18:40:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.453542+0000", 2024-04-22T18:40:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049580499999999999 2024-04-22T18:40:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.453571+0000", 2024-04-22T18:40:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9306e-05 2024-04-22T18:40:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:40:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.0039222529999998, 2024-04-22T18:40:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:40:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:40:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:40:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:40:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:40:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:40:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:40:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:40:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:40:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:40:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:40:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2745 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'28, mlcod=74'25)", 2024-04-22T18:40:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.643080+0000", 2024-04-22T18:40:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.9034914919999997, 2024-04-22T18:40:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10246031899999999, 2024-04-22T18:40:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643080+0000", 2024-04-22T18:40:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643080+0000", 2024-04-22T18:40:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643078+0000", 2024-04-22T18:40:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643090+0000", 2024-04-22T18:40:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1531e-05 2024-04-22T18:40:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643099+0000", 2024-04-22T18:40:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0650000000000005e-06 2024-04-22T18:40:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643105+0000", 2024-04-22T18:40:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.993e-06 2024-04-22T18:40:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.744552+0000", 2024-04-22T18:40:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101447464 2024-04-22T18:40:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.744577+0000", 2024-04-22T18:40:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4539e-05 2024-04-22T18:40:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745515+0000", 2024-04-22T18:40:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093760499999999999 2024-04-22T18:40:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745540+0000", 2024-04-22T18:40:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5582999999999999e-05 2024-04-22T18:40:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:160 3.0 e76/63 3:18d95008:::60ff66c0337e5d444612de0bca253cc28d83704a27a7c526a661c5877482b5f3:head v 76'156, mlcod=76'154)", 2024-04-22T18:40:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391289+0000", 2024-04-22T18:40:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 2.1552825360000001, 2024-04-22T18:40:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101821024, 2024-04-22T18:40:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391289+0000", 2024-04-22T18:40:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391289+0000", 2024-04-22T18:40:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391287+0000", 2024-04-22T18:40:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391329+0000", 2024-04-22T18:40:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2197999999999998e-05 2024-04-22T18:40:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391345+0000", 2024-04-22T18:40:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6608000000000001e-05 2024-04-22T18:40:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391355+0000", 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0134e-05 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391417+0000", 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1278999999999994e-05 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391440+0000", 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2758000000000001e-05 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493059+0000", 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101619691 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493110+0000", 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0556000000000002e-05 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:161 3.0 e76/63 3:3638311b:::f282598a146af3da1c05c8882972fba387afd249b39f6eeb6e6052a5de37fd2d:head v 76'157, mlcod=76'154)", 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 2.1552028229999998, 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104845172, 2024-04-22T18:40:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391440+0000", 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2045999999999994e-05 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391448+0000", 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8189999999999992e-06 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391453+0000", 2024-04-22T18:40:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8379999999999998e-06 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391839+0000", 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000385325 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391854+0000", 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5006e-05 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496191+0000", 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104336792 2024-04-22T18:40:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496214+0000", 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3079999999999999e-05 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:165 3.0 e76/63 3:1e2f7d15:::93d9bab0d1f539405a95a28c02bcf2dcbdfa6f3e449e5e5e3f27f774df3c508a:head v 76'158, mlcod=76'154)", 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 2.1551086399999999, 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104788752, 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391462+0000", 2024-04-22T18:40:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391522+0000", 2024-04-22T18:40:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0148000000000001e-05 2024-04-22T18:40:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391531+0000", 2024-04-22T18:40:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5709999999999998e-06 2024-04-22T18:40:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391536+0000", 2024-04-22T18:40:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6180000000000003e-06 2024-04-22T18:40:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392218+0000", 2024-04-22T18:40:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000681326 2024-04-22T18:40:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392237+0000", 2024-04-22T18:40:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9726999999999999e-05 2024-04-22T18:40:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496231+0000", 2024-04-22T18:40:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103993672 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496251+0000", 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0327999999999999e-05 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:166 3.0 e76/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 76'159, mlcod=76'154)", 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 2.1550307420000001, 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104724628, 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391540+0000", 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391627+0000", 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7330000000000006e-05 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391635+0000", 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6709999999999999e-06 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391640+0000", 2024-04-22T18:40:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3199999999999999e-06 2024-04-22T18:40:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492541+0000", 2024-04-22T18:40:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100900873 2024-04-22T18:40:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492560+0000", 2024-04-22T18:40:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8502999999999999e-05 2024-04-22T18:40:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496254+0000", 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036945849999999998 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496265+0000", 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0869999999999999e-05 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:167 3.0 e76/63 3:384c1ce2:::81dc8c58ebe106ae6fa9feadc06291c645d9079ad84b78214d4e6440fbc5def8:head v 76'160, mlcod=76'154)", 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 2.1549158670000002, 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10487859400000001, 2024-04-22T18:40:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391707+0000", 2024-04-22T18:40:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2343999999999999e-05 2024-04-22T18:40:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391714+0000", 2024-04-22T18:40:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.596e-06 2024-04-22T18:40:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391719+0000", 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9130000000000002e-06 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493154+0000", 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101435256 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493177+0000", 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2796e-05 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496509+0000", 2024-04-22T18:40:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033320139999999999 2024-04-22T18:40:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496534+0000", 2024-04-22T18:40:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5265999999999998e-05 2024-04-22T18:40:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:103 3.0 e78/63 3:0f5f80b3:::e65ec2deb1a27f55813533c3e73a682460e829b5dd6586a33a43649649fa77f6:head v 78'163, mlcod=76'161)", 2024-04-22T18:40:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.283291+0000", 2024-04-22T18:40:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.26328054299999998, 2024-04-22T18:40:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101953407, 2024-04-22T18:40:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.283291+0000", 2024-04-22T18:40:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.283291+0000", 2024-04-22T18:40:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.283290+0000", 2024-04-22T18:40:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.283296+0000", 2024-04-22T18:40:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6930000000000004e-06 2024-04-22T18:40:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.283309+0000", 2024-04-22T18:40:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3308e-05 2024-04-22T18:40:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.383415+0000", 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010583000000001 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.383479+0000", 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4289999999999996e-05 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.383509+0000", 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9609999999999999e-05 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.385202+0000", 2024-04-22T18:40:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0016931520000000001 2024-04-22T18:40:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.385244+0000", 2024-04-22T18:40:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1891000000000003e-05 2024-04-22T18:40:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.629 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:09.630 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:09.631 INFO:tasks.rados.rados.0.smithi089.stdout:1316: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:40:09.634 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:09.635+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:09.654 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:09.654 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:09.670 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:09.671+0000 7f3e7be88640 -1 osd.0 78 handle_osd_map injecting map crc failure 2024-04-22T18:40:09.673 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:09.674 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:09.674 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:40:09.691 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:09.692 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:09.735 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:09.737+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:09.742 INFO:tasks.rados.rados.0.smithi089.stdout:1305: expect (ObjNum 212 snap 0 seq_num 212) 2024-04-22T18:40:09.763 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:09.763 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:09.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.763 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2895 4.c e78/20)", 2024-04-22T18:40:09.763 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.737594+0000", 2024-04-22T18:40:09.763 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.027181607, 2024-04-22T18:40:09.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027200146000000001, 2024-04-22T18:40:09.763 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.763 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:09.763 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.737594+0000", 2024-04-22T18:40:09.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.737594+0000", 2024-04-22T18:40:09.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.737594+0000", 2024-04-22T18:40:09.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.738005+0000", 2024-04-22T18:40:09.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041153000000000001 2024-04-22T18:40:09.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.738013+0000", 2024-04-22T18:40:09.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8730000000000007e-06 2024-04-22T18:40:09.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.738020+0000", 2024-04-22T18:40:09.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.43e-06 2024-04-22T18:40:09.765 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.766 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.766 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.766 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2896 4.c e78/20)", 2024-04-22T18:40:09.766 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.738031+0000", 2024-04-22T18:40:09.766 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.026744440000000001, 2024-04-22T18:40:09.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026791041000000002, 2024-04-22T18:40:09.766 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.766 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:09.766 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.738031+0000", 2024-04-22T18:40:09.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.738031+0000", 2024-04-22T18:40:09.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.738031+0000", 2024-04-22T18:40:09.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.738131+0000", 2024-04-22T18:40:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000100225 2024-04-22T18:40:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.738138+0000", 2024-04-22T18:40:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4819999999999999e-06 2024-04-22T18:40:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.738142+0000", 2024-04-22T18:40:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9189999999999998e-06 2024-04-22T18:40:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.768 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.769 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2897 4.c e78/20)", 2024-04-22T18:40:09.769 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.738167+0000", 2024-04-22T18:40:09.769 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.026608552000000001, 2024-04-22T18:40:09.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026678668999999999, 2024-04-22T18:40:09.769 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:09.769 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "no flag points reached", 2024-04-22T18:40:09.769 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:09.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:09.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.738167+0000", 2024-04-22T18:40:09.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:09.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.738167+0000", 2024-04-22T18:40:09.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:09.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:09.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.738167+0000", 2024-04-22T18:40:09.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:09.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:09.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.738171+0000", 2024-04-22T18:40:09.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6700000000000002e-06 2024-04-22T18:40:09.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:09.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:09.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:09.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.738177+0000", 2024-04-22T18:40:09.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3380000000000004e-06 2024-04-22T18:40:09.771 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.771 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:09.771 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.771 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:09.771 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:09.771 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 3 2024-04-22T18:40:09.771 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:09.772 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:09.856 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:09.856 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:09.857 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:09.857+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:09.873 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:09.873 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:09.901 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:09.901 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:09.919 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:09.919 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:09.928 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:09.936 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:09.938+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:09.943 INFO:tasks.rados.rados.0.smithi089.stdout:1304: expect (ObjNum 219 snap 0 seq_num 219) 2024-04-22T18:40:10.028 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:10.028 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:10.037 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:10.038+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:10.047 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:10.047 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:10.050 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:10.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:10.079 INFO:tasks.rados.rados.0.smithi089.stdout:1301: finishing write tid 1 to smithi08937929-30 2024-04-22T18:40:10.079 INFO:tasks.rados.rados.0.smithi089.stdout:1301: finishing write tid 2 to smithi08937929-30 2024-04-22T18:40:10.079 INFO:tasks.rados.rados.0.smithi089.stdout:1301: finishing write tid 3 to smithi08937929-30 2024-04-22T18:40:10.080 INFO:tasks.rados.rados.0.smithi089.stdout:1301: finishing write tid 4 to smithi08937929-30 2024-04-22T18:40:10.080 INFO:tasks.rados.rados.0.smithi089.stdout:1301: oid 30 updating version 0 to 41 2024-04-22T18:40:10.080 INFO:tasks.rados.rados.0.smithi089.stdout:1301: oid 30 updating version 41 to 42 2024-04-22T18:40:10.080 INFO:tasks.rados.rados.0.smithi089.stdout:1301: oid 30 updating version 42 to 43 2024-04-22T18:40:10.080 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 43 (ObjNum 243 snap 0 seq_num 243) dirty exists 2024-04-22T18:40:10.080 INFO:tasks.rados.rados.0.smithi089.stdout:1301: left oid 30 (ObjNum 243 snap 0 seq_num 243) 2024-04-22T18:40:10.080 INFO:tasks.rados.rados.0.smithi089.stdout:1301: done (15 left) 2024-04-22T18:40:10.080 INFO:tasks.rados.rados.0.smithi089.stdout:1304: done (14 left) 2024-04-22T18:40:10.080 INFO:tasks.rados.rados.0.smithi089.stdout:1305: done (13 left) 2024-04-22T18:40:10.080 INFO:tasks.rados.rados.0.smithi089.stdout:1316: done (12 left) 2024-04-22T18:40:10.080 INFO:tasks.rados.rados.0.smithi089.stdout:1318: write (excl) oid 4 current snap is 0 2024-04-22T18:40:10.080 INFO:tasks.rados.rados.0.smithi089.stdout:1308: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-22T18:40:10.138 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:10.138+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:10.163 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:10.163 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:10.169 INFO:tasks.rados.rados.0.smithi089.stdout:1317: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:40:10.171 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:10.256 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:10.258+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:10.313 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:10.313 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:10.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:10.339 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:10.340+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:10.373 INFO:tasks.rados.rados.0.smithi089.stdout:1311: finishing write tid 1 to smithi08937929-46 2024-04-22T18:40:10.373 INFO:tasks.rados.rados.0.smithi089.stdout:1311: finishing write tid 2 to smithi08937929-46 2024-04-22T18:40:10.373 INFO:tasks.rados.rados.0.smithi089.stdout:1311: finishing write tid 3 to smithi08937929-46 2024-04-22T18:40:10.373 INFO:tasks.rados.rados.0.smithi089.stdout:1311: finishing write tid 4 to smithi08937929-46 2024-04-22T18:40:10.373 INFO:tasks.rados.rados.0.smithi089.stdout:1311: oid 46 updating version 0 to 19 2024-04-22T18:40:10.373 INFO:tasks.rados.rados.0.smithi089.stdout:1311: oid 46 version 19 is already newer than 17 2024-04-22T18:40:10.373 INFO:tasks.rados.rados.0.smithi089.stdout:1311: oid 46 version 19 is already newer than 18 2024-04-22T18:40:10.373 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 19 (ObjNum 247 snap 0 seq_num 247) dirty exists 2024-04-22T18:40:10.374 INFO:tasks.rados.rados.0.smithi089.stdout:1311: left oid 46 (ObjNum 247 snap 0 seq_num 247) 2024-04-22T18:40:10.374 INFO:tasks.rados.rados.0.smithi089.stdout:1318: seq_num 249 ranges {584943=460297,1587781=597162} 2024-04-22T18:40:10.378 INFO:tasks.rados.rados.0.smithi089.stdout:1318: writing smithi08937929-4 from 584943 to 1045240 tid 1 2024-04-22T18:40:10.385 INFO:tasks.rados.rados.0.smithi089.stdout:1318: writing smithi08937929-4 from 1587781 to 2184943 tid 2 2024-04-22T18:40:10.385 INFO:tasks.rados.rados.0.smithi089.stdout:1308: done (12 left) 2024-04-22T18:40:10.385 INFO:tasks.rados.rados.0.smithi089.stdout:1311: done (11 left) 2024-04-22T18:40:10.385 INFO:tasks.rados.rados.0.smithi089.stdout:1317: done (10 left) 2024-04-22T18:40:10.386 INFO:tasks.rados.rados.0.smithi089.stdout:1319: read oid 13 snap -1 2024-04-22T18:40:10.386 INFO:tasks.rados.rados.0.smithi089.stdout:1319: expect (ObjNum 234 snap 0 seq_num 234) 2024-04-22T18:40:10.386 INFO:tasks.rados.rados.0.smithi089.stdout:1320: read oid 2 snap -1 2024-04-22T18:40:10.386 INFO:tasks.rados.rados.0.smithi089.stdout:1320: expect (ObjNum 214 snap 0 seq_num 214) 2024-04-22T18:40:10.386 INFO:tasks.rados.rados.0.smithi089.stdout:1321: read oid 32 snap -1 2024-04-22T18:40:10.386 INFO:tasks.rados.rados.0.smithi089.stdout:1321: expect (ObjNum 209 snap 0 seq_num 209) 2024-04-22T18:40:10.386 INFO:tasks.rados.rados.0.smithi089.stdout:1313: got (22) Invalid argument 2024-04-22T18:40:10.386 INFO:tasks.rados.rados.0.smithi089.stdout:1309: finishing write tid 1 to smithi08937929-48 2024-04-22T18:40:10.386 INFO:tasks.rados.rados.0.smithi089.stdout:1310: expect (ObjNum 201 snap 0 seq_num 201) 2024-04-22T18:40:10.440 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:10.442+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:10.464 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:10.464 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:10.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:10.557 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:10.559+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:10.572 INFO:tasks.rados.rados.0.smithi089.stdout:1309: finishing write tid 2 to smithi08937929-48 2024-04-22T18:40:10.573 INFO:tasks.rados.rados.0.smithi089.stdout:1306: finishing set_chunk to oid 17 2024-04-22T18:40:10.573 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 67 (ObjNum 192 snap 0 seq_num 192) dirty exists 2024-04-22T18:40:10.573 INFO:tasks.rados.rados.0.smithi089.stdout:1307: finishing write tid 1 to smithi08937929-12 2024-04-22T18:40:10.573 INFO:tasks.rados.rados.0.smithi089.stdout:1307: finishing write tid 2 to smithi08937929-12 2024-04-22T18:40:10.573 INFO:tasks.rados.rados.0.smithi089.stdout:1315: finishing write tid 1 to smithi08937929-24 2024-04-22T18:40:10.573 INFO:tasks.rados.rados.0.smithi089.stdout:1315: finishing write tid 2 to smithi08937929-24 2024-04-22T18:40:10.573 INFO:tasks.rados.rados.0.smithi089.stdout:1309: finishing write tid 3 to smithi08937929-48 2024-04-22T18:40:10.573 INFO:tasks.rados.rados.0.smithi089.stdout:1309: finishing write tid 4 to smithi08937929-48 2024-04-22T18:40:10.573 INFO:tasks.rados.rados.0.smithi089.stdout:1309: oid 48 updating version 0 to 65 2024-04-22T18:40:10.573 INFO:tasks.rados.rados.0.smithi089.stdout:1309: oid 48 updating version 65 to 66 2024-04-22T18:40:10.573 INFO:tasks.rados.rados.0.smithi089.stdout:1309: oid 48 updating version 66 to 67 2024-04-22T18:40:10.573 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 67 (ObjNum 246 snap 0 seq_num 246) dirty exists 2024-04-22T18:40:10.573 INFO:tasks.rados.rados.0.smithi089.stdout:1309: left oid 48 (ObjNum 246 snap 0 seq_num 246) 2024-04-22T18:40:10.573 INFO:tasks.rados.rados.0.smithi089.stdout:1307: finishing write tid 3 to smithi08937929-12 2024-04-22T18:40:10.574 INFO:tasks.rados.rados.0.smithi089.stdout:1322: read oid 28 snap -1 2024-04-22T18:40:10.574 INFO:tasks.rados.rados.0.smithi089.stdout:1322: expect (ObjNum 241 snap 0 seq_num 241) 2024-04-22T18:40:10.574 INFO:tasks.rados.rados.0.smithi089.stdout:1306: done (13 left) 2024-04-22T18:40:10.574 INFO:tasks.rados.rados.0.smithi089.stdout:1309: done (12 left) 2024-04-22T18:40:10.574 INFO:tasks.rados.rados.0.smithi089.stdout:1310: done (11 left) 2024-04-22T18:40:10.574 INFO:tasks.rados.rados.0.smithi089.stdout:1313: done (10 left) 2024-04-22T18:40:10.574 INFO:tasks.rados.rados.0.smithi089.stdout:1323: read oid 46 snap -1 2024-04-22T18:40:10.574 INFO:tasks.rados.rados.0.smithi089.stdout:1323: expect (ObjNum 247 snap 0 seq_num 247) 2024-04-22T18:40:10.574 INFO:tasks.rados.rados.0.smithi089.stdout:1324: delete oid 1 current snap is 0 2024-04-22T18:40:10.574 INFO:tasks.rados.rados.0.smithi089.stdout:1307: finishing write tid 4 to smithi08937929-12 2024-04-22T18:40:10.574 INFO:tasks.rados.rados.0.smithi089.stdout:1307: oid 12 updating version 0 to 92 2024-04-22T18:40:10.574 INFO:tasks.rados.rados.0.smithi089.stdout:1307: oid 12 updating version 92 to 93 2024-04-22T18:40:10.574 INFO:tasks.rados.rados.0.smithi089.stdout:1307: oid 12 version 93 is already newer than 91 2024-04-22T18:40:10.574 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 93 (ObjNum 245 snap 0 seq_num 245) dirty exists 2024-04-22T18:40:10.575 INFO:tasks.rados.rados.0.smithi089.stdout:1307: left oid 12 (ObjNum 245 snap 0 seq_num 245) 2024-04-22T18:40:10.575 INFO:tasks.rados.rados.0.smithi089.stdout:1315: finishing write tid 3 to smithi08937929-24 2024-04-22T18:40:10.575 INFO:tasks.rados.rados.0.smithi089.stdout:1315: finishing write tid 4 to smithi08937929-24 2024-04-22T18:40:10.575 INFO:tasks.rados.rados.0.smithi089.stdout:1315: oid 24 updating version 0 to 126 2024-04-22T18:40:10.575 INFO:tasks.rados.rados.0.smithi089.stdout:1315: oid 24 version 126 is already newer than 125 2024-04-22T18:40:10.575 INFO:tasks.rados.rados.0.smithi089.stdout:1315: oid 24 updating version 126 to 127 2024-04-22T18:40:10.575 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 127 (ObjNum 248 snap 0 seq_num 248) dirty exists 2024-04-22T18:40:10.575 INFO:tasks.rados.rados.0.smithi089.stdout:1315: left oid 24 (ObjNum 248 snap 0 seq_num 248) 2024-04-22T18:40:10.575 INFO:tasks.rados.rados.0.smithi089.stdout:1312: got (0) Success 2024-04-22T18:40:10.575 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 73 (ObjNum 232 snap 0 seq_num 232) dirty exists 2024-04-22T18:40:10.575 INFO:tasks.rados.rados.0.smithi089.stdout:1319: expect (ObjNum 234 snap 0 seq_num 234) 2024-04-22T18:40:10.618 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:10.618 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:10.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:10.668 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:10.670+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:10.673 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:40:10.676 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:10.678+0000 7fa766d7d640 -1 osd.5 79 handle_osd_map injecting map crc failure 2024-04-22T18:40:10.743 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:10.744+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:10.759 INFO:tasks.rados.rados.0.smithi089.stdout:1320: expect (ObjNum 214 snap 0 seq_num 214) 2024-04-22T18:40:10.769 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:10.769 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:10.778 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:10.843 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:10.845+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:10.920 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:10.920 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:10.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:10.973 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:10.975+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:10.999 INFO:tasks.rados.rados.0.smithi089.stdout:1321: expect (ObjNum 209 snap 0 seq_num 209) 2024-04-22T18:40:11.031 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:11.031 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:11.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:11.081 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:11.083+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:11.109 INFO:tasks.rados.rados.0.smithi089.stdout:1322: expect (ObjNum 241 snap 0 seq_num 241) 2024-04-22T18:40:11.145 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:11.147+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:11.175 INFO:tasks.rados.rados.0.smithi089.stdout:1318: finishing write tid 1 to smithi08937929-4 2024-04-22T18:40:11.175 INFO:tasks.rados.rados.0.smithi089.stdout:1318: finishing write tid 2 to smithi08937929-4 2024-04-22T18:40:11.175 INFO:tasks.rados.rados.0.smithi089.stdout:1318: finishing write tid 3 to smithi08937929-4 2024-04-22T18:40:11.176 INFO:tasks.rados.rados.0.smithi089.stdout:1318: finishing write tid 4 to smithi08937929-4 2024-04-22T18:40:11.176 INFO:tasks.rados.rados.0.smithi089.stdout:1318: oid 4 updating version 0 to 94 2024-04-22T18:40:11.176 INFO:tasks.rados.rados.0.smithi089.stdout:1318: oid 4 version 94 is already newer than 92 2024-04-22T18:40:11.176 INFO:tasks.rados.rados.0.smithi089.stdout:1318: oid 4 version 94 is already newer than 93 2024-04-22T18:40:11.176 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 94 (ObjNum 249 snap 0 seq_num 249) dirty exists 2024-04-22T18:40:11.176 INFO:tasks.rados.rados.0.smithi089.stdout:1318: left oid 4 (ObjNum 249 snap 0 seq_num 249) 2024-04-22T18:40:11.176 INFO:tasks.rados.rados.0.smithi089.stdout:1323: expect (ObjNum 247 snap 0 seq_num 247) 2024-04-22T18:40:11.182 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:11.182 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:11.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:11.246 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:11.248+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:11.265 INFO:tasks.rados.rados.0.smithi089.stdout:1300: got (0) Success 2024-04-22T18:40:11.265 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 0 (ObjNum 149 snap 0 seq_num 149) dirty exists 2024-04-22T18:40:11.265 INFO:tasks.rados.rados.0.smithi089.stdout:1300: done (11 left) 2024-04-22T18:40:11.266 INFO:tasks.rados.rados.0.smithi089.stdout:1307: done (10 left) 2024-04-22T18:40:11.266 INFO:tasks.rados.rados.0.smithi089.stdout:1312: done (9 left) 2024-04-22T18:40:11.266 INFO:tasks.rados.rados.0.smithi089.stdout:1315: done (8 left) 2024-04-22T18:40:11.266 INFO:tasks.rados.rados.0.smithi089.stdout:1318: done (7 left) 2024-04-22T18:40:11.266 INFO:tasks.rados.rados.0.smithi089.stdout:1319: done (6 left) 2024-04-22T18:40:11.266 INFO:tasks.rados.rados.0.smithi089.stdout:1320: done (5 left) 2024-04-22T18:40:11.266 INFO:tasks.rados.rados.0.smithi089.stdout:1321: done (4 left) 2024-04-22T18:40:11.266 INFO:tasks.rados.rados.0.smithi089.stdout:1322: done (3 left) 2024-04-22T18:40:11.266 INFO:tasks.rados.rados.0.smithi089.stdout:1323: done (2 left) 2024-04-22T18:40:11.266 INFO:tasks.rados.rados.0.smithi089.stdout:1324: done (1 left) 2024-04-22T18:40:11.266 INFO:tasks.rados.rados.0.smithi089.stdout:1325: set_chunk oid 14 target oid 2024-04-22T18:40:11.293 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:11.294 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:11.303 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:11.347 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:11.349+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:11.447 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:11.447 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:11.448 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:11.450+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:11.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:11.550 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:11.551+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:11.556 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:11.556 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:11.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:11.589 INFO:tasks.rados.rados.0.smithi089.stdout:1325: set_chunk oid 14 offset: 355328 length: 780800 target oid 544a73847a5dfbfe56fdc2fd95b196cb5edc542e55aaf81145cfa77f5b6c8ff4 offset: 0 snap -1 2024-04-22T18:40:11.589 INFO:tasks.rados.rados.0.smithi089.stdout:1326: read oid 10 snap -1 2024-04-22T18:40:11.589 INFO:tasks.rados.rados.0.smithi089.stdout:1326: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:40:11.589 INFO:tasks.rados.rados.0.smithi089.stdout:1327: write oid 2 current snap is 0 2024-04-22T18:40:11.589 INFO:tasks.rados.rados.0.smithi089.stdout:1327: seq_num 251 ranges {618557=727148,2104745=113812} 2024-04-22T18:40:11.596 INFO:tasks.rados.rados.0.smithi089.stdout:1327: writing smithi08937929-2 from 618557 to 1345705 tid 1 2024-04-22T18:40:11.601 INFO:tasks.rados.rados.0.smithi089.stdout:1327: writing smithi08937929-2 from 2104745 to 2218557 tid 2 2024-04-22T18:40:11.601 INFO:tasks.rados.rados.0.smithi089.stdout:1328: read oid 44 snap -1 2024-04-22T18:40:11.601 INFO:tasks.rados.rados.0.smithi089.stdout:1328: expect (ObjNum 187 snap 0 seq_num 187) 2024-04-22T18:40:11.601 INFO:tasks.rados.rados.0.smithi089.stdout:1329: set_chunk oid 48 target oid 2024-04-22T18:40:11.650 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:11.652+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:11.679 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:11.681+0000 7f3cfee88640 -1 osd.6 80 handle_osd_map injecting map crc failure 2024-04-22T18:40:11.710 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:11.710 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:11.719 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:11.760 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:11.762+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:11.824 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:11.824 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:11.833 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:11.840 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:11.842+0000 7fa766d7d640 -1 osd.5 80 handle_osd_map injecting map crc failure 2024-04-22T18:40:11.853 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:11.855+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:11.907 INFO:tasks.rados.rados.0.smithi089.stdout:1329: set_chunk oid 48 offset: 550912 length: 568832 target oid c6c461e26a7380070e4ab40bccec53e16ed2663aa012784ab8250e270af0fa0f offset: 0 snap -1 2024-04-22T18:40:11.907 INFO:tasks.rados.rados.0.smithi089.stdout:1330: read oid 15 snap -1 2024-04-22T18:40:11.907 INFO:tasks.rados.rados.0.smithi089.stdout:1330: expect deleted 2024-04-22T18:40:11.907 INFO:tasks.rados.rados.0.smithi089.stdout:1331: read oid 23 snap -1 2024-04-22T18:40:11.907 INFO:tasks.rados.rados.0.smithi089.stdout:1331: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:40:11.907 INFO:tasks.rados.rados.0.smithi089.stdout:1332: read oid 33 snap -1 2024-04-22T18:40:11.907 INFO:tasks.rados.rados.0.smithi089.stdout:1332: expect (ObjNum 244 snap 0 seq_num 244) 2024-04-22T18:40:11.907 INFO:tasks.rados.rados.0.smithi089.stdout:1333: set_chunk oid 28 target oid 2024-04-22T18:40:11.936 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:11.936 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:11.945 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:11.954 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:11.956+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:12.017 INFO:tasks.rados.rados.0.smithi089.stdout:1333: set_chunk oid 28 offset: 1823232 length: 159232 target oid 26fd829b5520e4d41090c9f48ff5cb4742a3b7e5f9f676b82241435488ace2da offset: 0 snap -1 2024-04-22T18:40:12.017 INFO:tasks.rados.rados.0.smithi089.stdout:1334: write oid 26 current snap is 0 2024-04-22T18:40:12.017 INFO:tasks.rados.rados.0.smithi089.stdout:1302: got (0) Success 2024-04-22T18:40:12.017 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 0 (ObjNum 238 snap 0 seq_num 238) dirty exists 2024-04-22T18:40:12.017 INFO:tasks.rados.rados.0.smithi089.stdout:1334: seq_num 252 ranges {635364=402397,1763227=472137} 2024-04-22T18:40:12.022 INFO:tasks.rados.rados.0.smithi089.stdout:1334: writing smithi08937929-26 from 635364 to 1037761 tid 1 2024-04-22T18:40:12.029 INFO:tasks.rados.rados.0.smithi089.stdout:1334: writing smithi08937929-26 from 1763227 to 2235364 tid 2 2024-04-22T18:40:12.030 INFO:tasks.rados.rados.0.smithi089.stdout:1302: done (10 left) 2024-04-22T18:40:12.030 INFO:tasks.rados.rados.0.smithi089.stdout:1335: read oid 29 snap -1 2024-04-22T18:40:12.030 INFO:tasks.rados.rados.0.smithi089.stdout:1335: expect (ObjNum 149 snap 0 seq_num 149) 2024-04-22T18:40:12.030 INFO:tasks.rados.rados.0.smithi089.stdout:1336: read oid 4 snap -1 2024-04-22T18:40:12.030 INFO:tasks.rados.rados.0.smithi089.stdout:1336: expect (ObjNum 249 snap 0 seq_num 249) 2024-04-22T18:40:12.030 INFO:tasks.rados.rados.0.smithi089.stdout:1337: write oid 8 current snap is 0 2024-04-22T18:40:12.030 INFO:tasks.rados.rados.0.smithi089.stdout:1337: seq_num 253 ranges {652171=477646,1821709=430462} 2024-04-22T18:40:12.035 INFO:tasks.rados.rados.0.smithi089.stdout:1337: writing smithi08937929-8 from 652171 to 1129817 tid 1 2024-04-22T18:40:12.042 INFO:tasks.rados.rados.0.smithi089.stdout:1337: writing smithi08937929-8 from 1821709 to 2252171 tid 2 2024-04-22T18:40:12.042 INFO:tasks.rados.rados.0.smithi089.stdout:1338: write oid 5 current snap is 0 2024-04-22T18:40:12.042 INFO:tasks.rados.rados.0.smithi089.stdout:1338: seq_num 254 ranges {668978=552895,1763838=505140} 2024-04-22T18:40:12.048 INFO:tasks.rados.rados.0.smithi089.stdout:1338: writing smithi08937929-5 from 668978 to 1221873 tid 1 2024-04-22T18:40:12.052 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:12.053 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:12.054 INFO:tasks.rados.rados.0.smithi089.stdout:1338: writing smithi08937929-5 from 1763838 to 2268978 tid 2 2024-04-22T18:40:12.054 INFO:tasks.rados.rados.0.smithi089.stdout:1339: read oid 9 snap -1 2024-04-22T18:40:12.054 INFO:tasks.rados.rados.0.smithi089.stdout:1339: expect (ObjNum 242 snap 0 seq_num 242) 2024-04-22T18:40:12.054 INFO:tasks.rados.rados.0.smithi089.stdout:1340: read oid 7 snap -1 2024-04-22T18:40:12.054 INFO:tasks.rados.rados.0.smithi089.stdout:1340: expect (ObjNum 215 snap 0 seq_num 215) 2024-04-22T18:40:12.055 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:12.055 INFO:tasks.rados.rados.0.smithi089.stdout:1328: expect (ObjNum 187 snap 0 seq_num 187) 2024-04-22T18:40:12.056 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:12.057+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:12.062 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:40:12.065 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:12.157 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:12.158+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:12.182 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:12.182 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:12.199 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:12.199 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:12.212 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:12.213 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:12.214 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:12.215 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:12.222 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:12.231 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:12.231 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:12.239 INFO:tasks.rados.rados.0.smithi089.stdout:1325: finishing set_chunk to oid 14 2024-04-22T18:40:12.239 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 14 set_chunk returned error code -95 offset: 355328 length: 780800 2024-04-22T18:40:12.239 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 0 (ObjNum 163 snap 0 seq_num 163) dirty exists 2024-04-22T18:40:12.240 INFO:tasks.rados.rados.0.smithi089.stdout:1326: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:40:12.254 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:12.254 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:12.257 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:12.259+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:12.271 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:12.271 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:12.287 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:12.287 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:12.304 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:12.304 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:12.309 INFO:tasks.rados.rados.0.smithi089.stdout:1327: finishing write tid 1 to smithi08937929-2 2024-04-22T18:40:12.309 INFO:tasks.rados.rados.0.smithi089.stdout:1327: finishing write tid 2 to smithi08937929-2 2024-04-22T18:40:12.309 INFO:tasks.rados.rados.0.smithi089.stdout:1327: finishing write tid 3 to smithi08937929-2 2024-04-22T18:40:12.309 INFO:tasks.rados.rados.0.smithi089.stdout:1327: finishing write tid 4 to smithi08937929-2 2024-04-22T18:40:12.309 INFO:tasks.rados.rados.0.smithi089.stdout:1327: oid 2 updating version 0 to 62 2024-04-22T18:40:12.309 INFO:tasks.rados.rados.0.smithi089.stdout:1327: oid 2 updating version 62 to 63 2024-04-22T18:40:12.309 INFO:tasks.rados.rados.0.smithi089.stdout:1327: oid 2 updating version 63 to 64 2024-04-22T18:40:12.309 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 64 (ObjNum 251 snap 0 seq_num 251) dirty exists 2024-04-22T18:40:12.309 INFO:tasks.rados.rados.0.smithi089.stdout:1327: left oid 2 (ObjNum 251 snap 0 seq_num 251) 2024-04-22T18:40:12.309 INFO:tasks.rados.rados.0.smithi089.stdout:1332: expect (ObjNum 244 snap 0 seq_num 244) 2024-04-22T18:40:12.334 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:12.334 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:12.334 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:12.334 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:12.334 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:12.343 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:12.385 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:12.386+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:12.419 INFO:tasks.rados.rados.0.smithi089.stdout:1331: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:40:12.458 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:12.460+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:12.488 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:12.488 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:12.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:12.488 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:12.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.488 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:9 4.f 4:f2c48a0e:::smithi08937929-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:12.488 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:12.488 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.371165116, 2024-04-22T18:40:12.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49290714600000002, 2024-04-22T18:40:12.489 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.489 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:12.489 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.489 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.489 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.489 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 9 2024-04-22T18:40:12.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.489 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:12.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:12.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116773+0000", 2024-04-22T18:40:12.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:12.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117024+0000", 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025191299999999999 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117036+0000", 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1595e-05 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117046+0000", 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0456e-05 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414324+0000", 2024-04-22T18:40:12.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297277603 2024-04-22T18:40:12.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414404+0000", 2024-04-22T18:40:12.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9738000000000001e-05 2024-04-22T18:40:12.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:12.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414494+0000", 2024-04-22T18:40:12.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9733000000000003e-05 2024-04-22T18:40:12.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:12.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518194+0000", 2024-04-22T18:40:12.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10370061 2024-04-22T18:40:12.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:12.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609618+0000", 2024-04-22T18:40:12.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091423360999999995 2024-04-22T18:40:12.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:12.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609655+0000", 2024-04-22T18:40:12.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7580000000000003e-05 2024-04-22T18:40:12.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609681+0000", 2024-04-22T18:40:12.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5908e-05 2024-04-22T18:40:12.494 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.494 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.494 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.494 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:10 4.f 4:f2c48a0e:::smithi08937929-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:12.494 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:12.494 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.365312856000003, 2024-04-22T18:40:12.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48712681400000002, 2024-04-22T18:40:12.495 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.495 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:12.495 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.495 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.495 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.495 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 10 2024-04-22T18:40:12.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.495 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:12.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:12.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122625+0000", 2024-04-22T18:40:12.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:12.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122828+0000", 2024-04-22T18:40:12.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020359499999999999 2024-04-22T18:40:12.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122840+0000", 2024-04-22T18:40:12.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1239e-05 2024-04-22T18:40:12.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122845+0000", 2024-04-22T18:40:12.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2719999999999997e-06 2024-04-22T18:40:12.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415105+0000", 2024-04-22T18:40:12.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29225972099999997 2024-04-22T18:40:12.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415169+0000", 2024-04-22T18:40:12.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4170000000000004e-05 2024-04-22T18:40:12.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:12.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415257+0000", 2024-04-22T18:40:12.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8331999999999994e-05 2024-04-22T18:40:12.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:12.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518199+0000", 2024-04-22T18:40:12.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10294174 2024-04-22T18:40:12.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:12.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609707+0000", 2024-04-22T18:40:12.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091508408999999999 2024-04-22T18:40:12.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:12.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609729+0000", 2024-04-22T18:40:12.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1917e-05 2024-04-22T18:40:12.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609753+0000", 2024-04-22T18:40:12.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3776000000000001e-05 2024-04-22T18:40:12.500 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.500 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.500 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.500 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:11 4.f 4:f2c48a0e:::smithi08937929-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:12.500 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:12.500 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.365089483000006, 2024-04-22T18:40:12.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49452214500000002, 2024-04-22T18:40:12.500 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.501 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:12.501 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.501 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.501 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.501 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 11 2024-04-22T18:40:12.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.501 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:12.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:12.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122849+0000", 2024-04-22T18:40:12.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:12.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122853+0000", 2024-04-22T18:40:12.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9029999999999997e-06 2024-04-22T18:40:12.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122856+0000", 2024-04-22T18:40:12.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.01e-06 2024-04-22T18:40:12.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122859+0000", 2024-04-22T18:40:12.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:12.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415755+0000", 2024-04-22T18:40:12.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29289606800000001 2024-04-22T18:40:12.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415808+0000", 2024-04-22T18:40:12.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3415999999999998e-05 2024-04-22T18:40:12.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:12.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415888+0000", 2024-04-22T18:40:12.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9061999999999996e-05 2024-04-22T18:40:12.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:12.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518203+0000", 2024-04-22T18:40:12.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102315533 2024-04-22T18:40:12.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:12.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617164+0000", 2024-04-22T18:40:12.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098960548999999995 2024-04-22T18:40:12.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:12.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617241+0000", 2024-04-22T18:40:12.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7292000000000005e-05 2024-04-22T18:40:12.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617372+0000", 2024-04-22T18:40:12.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013075399999999999 2024-04-22T18:40:12.506 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.506 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.506 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.506 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:12 4.f 4:f2c48a0e:::smithi08937929-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:12.506 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:12.506 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.365077619000004, 2024-04-22T18:40:12.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49473910999999998, 2024-04-22T18:40:12.506 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.506 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:12.506 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.506 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.506 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.507 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 12 2024-04-22T18:40:12.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.507 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:12.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:12.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:12.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:12.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122863+0000", 2024-04-22T18:40:12.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.863e-06 2024-04-22T18:40:12.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122866+0000", 2024-04-22T18:40:12.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:40:12.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122869+0000", 2024-04-22T18:40:12.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5950000000000001e-06 2024-04-22T18:40:12.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416008+0000", 2024-04-22T18:40:12.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29313975199999998 2024-04-22T18:40:12.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:12.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416048+0000", 2024-04-22T18:40:12.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9888000000000002e-05 2024-04-22T18:40:12.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617447+0000", 2024-04-22T18:40:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20139881400000001 2024-04-22T18:40:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617537+0000", 2024-04-22T18:40:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9606999999999998e-05 2024-04-22T18:40:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617600+0000", 2024-04-22T18:40:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3884e-05 2024-04-22T18:40:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.510 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.511 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.511 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:33 4.f 4:f0ed3d49:::smithi08937929-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:12.511 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:12.511 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.301137417999996, 2024-04-22T18:40:12.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62359692099999997, 2024-04-22T18:40:12.511 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.511 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:12.511 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.511 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.511 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.511 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 33 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186799+0000", 2024-04-22T18:40:12.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187103+0000", 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030354900000000002 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187117+0000", 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4578e-05 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187129+0000", 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1421e-05 2024-04-22T18:40:12.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516134+0000", 2024-04-22T18:40:12.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32900553900000001 2024-04-22T18:40:12.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516215+0000", 2024-04-22T18:40:12.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0537999999999994e-05 2024-04-22T18:40:12.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:12.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516302+0000", 2024-04-22T18:40:12.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6601000000000007e-05 2024-04-22T18:40:12.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518207+0000", 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019058370000000001 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810302+0000", 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29209509700000003 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810348+0000", 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5992999999999998e-05 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810398+0000", 2024-04-22T18:40:12.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0046000000000002e-05 2024-04-22T18:40:12.516 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.516 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.516 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.516 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:34 4.f 4:f0ed3d49:::smithi08937929-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:12.516 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:12.516 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.296535406000004, 2024-04-22T18:40:12.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.61954655800000002, 2024-04-22T18:40:12.516 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.516 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:12.516 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191401+0000", 2024-04-22T18:40:12.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:12.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191503+0000", 2024-04-22T18:40:12.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010190400000000001 2024-04-22T18:40:12.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191517+0000", 2024-04-22T18:40:12.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3267e-05 2024-04-22T18:40:12.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191523+0000", 2024-04-22T18:40:12.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-04-22T18:40:12.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517062+0000", 2024-04-22T18:40:12.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32553894300000003 2024-04-22T18:40:12.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517141+0000", 2024-04-22T18:40:12.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9667999999999997e-05 2024-04-22T18:40:12.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:12.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517267+0000", 2024-04-22T18:40:12.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000125494 2024-04-22T18:40:12.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:12.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518213+0000", 2024-04-22T18:40:12.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000946275 2024-04-22T18:40:12.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:12.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810901+0000", 2024-04-22T18:40:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29268821699999997 2024-04-22T18:40:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810929+0000", 2024-04-22T18:40:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7645999999999999e-05 2024-04-22T18:40:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810950+0000", 2024-04-22T18:40:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1333999999999999e-05 2024-04-22T18:40:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.522 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:12.522 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:12.522 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.296412438000004, 2024-04-22T18:40:12.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:40:12.522 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.522 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:12.522 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.522 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.522 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.522 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:40:12.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.522 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:12.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:12.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:40:12.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:12.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:40:12.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:40:12.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:40:12.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:40:12.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:12.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:40:12.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:40:12.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:12.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:40:12.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:40:12.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:12.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:40:12.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:40:12.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:12.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:40:12.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:40:12.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:40:12.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:40:12.527 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.527 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.527 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.527 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:12.527 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:12.527 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.296401363000001, 2024-04-22T18:40:12.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:40:12.528 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.528 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:12.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.528 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:40:12.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.528 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:12.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:12.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:40:12.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:12.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:40:12.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:40:12.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:40:12.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:40:12.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:40:12.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:40:12.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:40:12.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:40:12.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:12.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:40:12.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:40:12.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:40:12.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:40:12.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.695031869000005, 2024-04-22T18:40:12.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:40:12.533 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.533 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:12.533 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.533 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.533 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.533 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:40:12.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.533 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:12.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:12.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:40:12.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:12.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:40:12.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:40:12.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:40:12.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:40:12.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:40:12.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:40:12.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:40:12.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:40:12.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:40:12.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:40:12.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:12.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:40:12.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:40:12.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:12.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:40:12.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:40:12.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:12.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:40:12.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:40:12.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:12.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:40:12.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:40:12.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:40:12.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:40:12.538 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.538 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.538 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.539 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1262 4.7 4:e117a562:::smithi08937929-33:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:12.539 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:40:12.539 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.034888692999999, 2024-04-22T18:40:12.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40488836700000003, 2024-04-22T18:40:12.539 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.539 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:12.539 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.539 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.539 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.539 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1262 2024-04-22T18:40:12.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.539 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453048+0000", 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453058+0000", 2024-04-22T18:40:12.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453069+0000", 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1117999999999999e-05 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453075+0000", 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0909999999999998e-06 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453115+0000", 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0119e-05 2024-04-22T18:40:12.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:40:12.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453119+0000", 2024-04-22T18:40:12.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3610000000000003e-06 2024-04-22T18:40:12.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454814+0000", 2024-04-22T18:40:12.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00169436 2024-04-22T18:40:12.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454861+0000", 2024-04-22T18:40:12.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7184000000000001e-05 2024-04-22T18:40:12.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:12.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756925+0000", 2024-04-22T18:40:12.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30206361799999998 2024-04-22T18:40:12.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:12.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757577+0000", 2024-04-22T18:40:12.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065199599999999996 2024-04-22T18:40:12.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857846+0000", 2024-04-22T18:40:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100269369 2024-04-22T18:40:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857896+0000", 2024-04-22T18:40:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9607000000000001e-05 2024-04-22T18:40:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857939+0000", 2024-04-22T18:40:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3149999999999999e-05 2024-04-22T18:40:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2205 4.2 4:4fb26dd8:::smithi08937929-10:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:40:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.097343592000001, 2024-04-22T18:40:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.413791136, 2024-04-22T18:40:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2205 2024-04-22T18:40:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:40:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:40:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:40:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390598+0000", 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7630000000000001e-06 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390603+0000", 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.293e-06 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390608+0000", 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8790000000000002e-06 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390653+0000", 2024-04-22T18:40:12.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5661999999999998e-05 2024-04-22T18:40:12.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:40:12.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390657+0000", 2024-04-22T18:40:12.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9709999999999998e-06 2024-04-22T18:40:12.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.592250+0000", 2024-04-22T18:40:12.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20159280600000001 2024-04-22T18:40:12.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.592322+0000", 2024-04-22T18:40:12.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2171999999999999e-05 2024-04-22T18:40:12.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.804373+0000", 2024-04-22T18:40:12.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.212050297 2024-04-22T18:40:12.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.804387+0000", 2024-04-22T18:40:12.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3859999999999999e-05 2024-04-22T18:40:12.550 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.550 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.550 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.550 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:7 3.0 3:1554f608:::d614c1f2f74bb29087660bd6e888eafe46b1a7ed3cbdede85b36de771c7f0d66:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:12.550 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:12.550 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.315754343, 2024-04-22T18:40:12.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48116529899999999, 2024-04-22T18:40:12.550 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.550 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:12.550 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172184+0000", 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172349+0000", 2024-04-22T18:40:12.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000165074 2024-04-22T18:40:12.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172355+0000", 2024-04-22T18:40:12.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8900000000000004e-06 2024-04-22T18:40:12.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172359+0000", 2024-04-22T18:40:12.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.569e-06 2024-04-22T18:40:12.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271561+0000", 2024-04-22T18:40:12.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099202569000000004 2024-04-22T18:40:12.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271659+0000", 2024-04-22T18:40:12.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8171000000000006e-05 2024-04-22T18:40:12.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:12.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271775+0000", 2024-04-22T18:40:12.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000116137 2024-04-22T18:40:12.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:12.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.272327+0000", 2024-04-22T18:40:12.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00055106999999999997 2024-04-22T18:40:12.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:12.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653290+0000", 2024-04-22T18:40:12.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.38096334500000001 2024-04-22T18:40:12.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:12.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653326+0000", 2024-04-22T18:40:12.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6168000000000003e-05 2024-04-22T18:40:12.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653350+0000", 2024-04-22T18:40:12.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3921999999999999e-05 2024-04-22T18:40:12.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.555 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.555 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2708 4.2 4:409e6456:::smithi08937929-11:head [read 0~3294172,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:12.556 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.553783+0000", 2024-04-22T18:40:12.556 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.9341559460000006, 2024-04-22T18:40:12.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.403846977, 2024-04-22T18:40:12.556 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.556 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:12.556 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.556 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.556 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.556 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2708 2024-04-22T18:40:12.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.556 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553783+0000", 2024-04-22T18:40:12.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553783+0000", 2024-04-22T18:40:12.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553783+0000", 2024-04-22T18:40:12.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:12.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553786+0000", 2024-04-22T18:40:12.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8809999999999998e-06 2024-04-22T18:40:12.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553791+0000", 2024-04-22T18:40:12.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7849999999999999e-06 2024-04-22T18:40:12.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553797+0000", 2024-04-22T18:40:12.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8629999999999996e-06 2024-04-22T18:40:12.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655434+0000", 2024-04-22T18:40:12.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10163662 2024-04-22T18:40:12.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:12.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655539+0000", 2024-04-22T18:40:12.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105431 2024-04-22T18:40:12.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.956537+0000", 2024-04-22T18:40:12.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30099759100000001 2024-04-22T18:40:12.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.956612+0000", 2024-04-22T18:40:12.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5407000000000005e-05 2024-04-22T18:40:12.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.957630+0000", 2024-04-22T18:40:12.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001017869 2024-04-22T18:40:12.560 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.560 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.560 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2709 4.2 4:409e6456:::smithi08937929-11:head [sparse-read 0~2843709] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.553801+0000", 2024-04-22T18:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.9341376179999994, 2024-04-22T18:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40417458899999997, 2024-04-22T18:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2709 2024-04-22T18:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553801+0000", 2024-04-22T18:40:12.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553801+0000", 2024-04-22T18:40:12.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553801+0000", 2024-04-22T18:40:12.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:12.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553803+0000", 2024-04-22T18:40:12.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4899999999999999e-06 2024-04-22T18:40:12.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553808+0000", 2024-04-22T18:40:12.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9169999999999997e-06 2024-04-22T18:40:12.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553812+0000", 2024-04-22T18:40:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1470000000000003e-06 2024-04-22T18:40:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655577+0000", 2024-04-22T18:40:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10176484299999999 2024-04-22T18:40:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655684+0000", 2024-04-22T18:40:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107221 2024-04-22T18:40:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.957688+0000", 2024-04-22T18:40:12.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30200392199999998 2024-04-22T18:40:12.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.957737+0000", 2024-04-22T18:40:12.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8362999999999998e-05 2024-04-22T18:40:12.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.957976+0000", 2024-04-22T18:40:12.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000239211 2024-04-22T18:40:12.566 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.566 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.566 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.566 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2710 4.2 4:409e6456:::smithi08937929-11:head [read 0~3294172,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:12.566 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.553816+0000", 2024-04-22T18:40:12.566 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.9341230070000002, 2024-04-22T18:40:12.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40483363500000002, 2024-04-22T18:40:12.566 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.566 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:12.566 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.566 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.567 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.567 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2710 2024-04-22T18:40:12.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.567 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553816+0000", 2024-04-22T18:40:12.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553816+0000", 2024-04-22T18:40:12.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553815+0000", 2024-04-22T18:40:12.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:12.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553819+0000", 2024-04-22T18:40:12.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3220000000000001e-06 2024-04-22T18:40:12.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553824+0000", 2024-04-22T18:40:12.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8879999999999996e-06 2024-04-22T18:40:12.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553828+0000", 2024-04-22T18:40:12.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.245e-06 2024-04-22T18:40:12.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655717+0000", 2024-04-22T18:40:12.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10188926500000001 2024-04-22T18:40:12.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:12.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655784+0000", 2024-04-22T18:40:12.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6308000000000003e-05 2024-04-22T18:40:12.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.957997+0000", 2024-04-22T18:40:12.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.302213385 2024-04-22T18:40:12.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.958037+0000", 2024-04-22T18:40:12.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470999999999999e-05 2024-04-22T18:40:12.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.958650+0000", 2024-04-22T18:40:12.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061223400000000002 2024-04-22T18:40:12.571 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.571 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.571 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.571 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:166 3.0 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:12.571 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.317111+0000", 2024-04-22T18:40:12.571 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1708277489999999, 2024-04-22T18:40:12.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.379585277, 2024-04-22T18:40:12.571 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.572 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:12.572 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317111+0000", 2024-04-22T18:40:12.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317111+0000", 2024-04-22T18:40:12.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317111+0000", 2024-04-22T18:40:12.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:12.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317206+0000", 2024-04-22T18:40:12.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4988999999999995e-05 2024-04-22T18:40:12.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317213+0000", 2024-04-22T18:40:12.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.391e-06 2024-04-22T18:40:12.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317219+0000", 2024-04-22T18:40:12.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6330000000000001e-06 2024-04-22T18:40:12.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.346889+0000", 2024-04-22T18:40:12.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.02967036 2024-04-22T18:40:12.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.346967+0000", 2024-04-22T18:40:12.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8582000000000001e-05 2024-04-22T18:40:12.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:12.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.347098+0000", 2024-04-22T18:40:12.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013041699999999999 2024-04-22T18:40:12.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:12.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.420231+0000", 2024-04-22T18:40:12.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.073133190000000001 2024-04-22T18:40:12.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:12.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696579+0000", 2024-04-22T18:40:12.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.27634776999999999 2024-04-22T18:40:12.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:12.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696649+0000", 2024-04-22T18:40:12.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9830000000000001e-05 2024-04-22T18:40:12.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696697+0000", 2024-04-22T18:40:12.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7825999999999999e-05 2024-04-22T18:40:12.576 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.576 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.576 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.577 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:167 3.0 3:384c1ce2:::81dc8c58ebe106ae6fa9feadc06291c645d9079ad84b78214d4e6440fbc5def8:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:12.577 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.317224+0000", 2024-04-22T18:40:12.577 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1707149330000002, 2024-04-22T18:40:12.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37960258600000002, 2024-04-22T18:40:12.577 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.577 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:12.577 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317224+0000", 2024-04-22T18:40:12.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317224+0000", 2024-04-22T18:40:12.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317223+0000", 2024-04-22T18:40:12.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:12.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317273+0000", 2024-04-22T18:40:12.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9548000000000001e-05 2024-04-22T18:40:12.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317280+0000", 2024-04-22T18:40:12.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:40:12.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317286+0000", 2024-04-22T18:40:12.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1240000000000002e-06 2024-04-22T18:40:12.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.347517+0000", 2024-04-22T18:40:12.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.030231428000000001 2024-04-22T18:40:12.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.347584+0000", 2024-04-22T18:40:12.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6669999999999997e-05 2024-04-22T18:40:12.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:12.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.347722+0000", 2024-04-22T18:40:12.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013773100000000001 2024-04-22T18:40:12.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:12.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.420237+0000", 2024-04-22T18:40:12.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.072515653999999999 2024-04-22T18:40:12.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:12.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696747+0000", 2024-04-22T18:40:12.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.27650990199999997 2024-04-22T18:40:12.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:12.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696792+0000", 2024-04-22T18:40:12.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4591e-05 2024-04-22T18:40:12.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696827+0000", 2024-04-22T18:40:12.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4876000000000003e-05 2024-04-22T18:40:12.582 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.582 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.582 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.583 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:172 3.0 3:1e450458:::73c5b8dc943cb866ee2d52255e62488506020b89464b8097642f92f95390407a:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:12.583 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.317290+0000", 2024-04-22T18:40:12.583 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1706489019999999, 2024-04-22T18:40:12.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.37964102999999999, 2024-04-22T18:40:12.583 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.583 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:12.583 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317290+0000", 2024-04-22T18:40:12.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317290+0000", 2024-04-22T18:40:12.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317289+0000", 2024-04-22T18:40:12.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:12.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317359+0000", 2024-04-22T18:40:12.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9906000000000005e-05 2024-04-22T18:40:12.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317367+0000", 2024-04-22T18:40:12.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4370000000000001e-06 2024-04-22T18:40:12.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.317376+0000", 2024-04-22T18:40:12.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3179999999999996e-06 2024-04-22T18:40:12.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.419267+0000", 2024-04-22T18:40:12.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10189088 2024-04-22T18:40:12.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.419371+0000", 2024-04-22T18:40:12.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010370300000000001 2024-04-22T18:40:12.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:12.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.419590+0000", 2024-04-22T18:40:12.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021932700000000001 2024-04-22T18:40:12.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:12.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.420825+0000", 2024-04-22T18:40:12.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012349679999999999 2024-04-22T18:40:12.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:12.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696860+0000", 2024-04-22T18:40:12.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.27603527900000002 2024-04-22T18:40:12.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:12.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696901+0000", 2024-04-22T18:40:12.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0515000000000001e-05 2024-04-22T18:40:12.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.696931+0000", 2024-04-22T18:40:12.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0300000000000001e-05 2024-04-22T18:40:12.588 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.588 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.588 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.588 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2845 4.7 4:e117a562:::smithi08937929-33:head [setxattr _header (56) in=7b,truncate 2033680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-04-22T18:40:12.588 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:08.157995+0000", 2024-04-22T18:40:12.588 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.3299442949999998, 2024-04-22T18:40:12.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.416467168, 2024-04-22T18:40:12.589 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.589 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:12.589 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.589 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.589 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.589 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2845 2024-04-22T18:40:12.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.589 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.157995+0000", 2024-04-22T18:40:12.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.157995+0000", 2024-04-22T18:40:12.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.157994+0000", 2024-04-22T18:40:12.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:12.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.157998+0000", 2024-04-22T18:40:12.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.631e-06 2024-04-22T18:40:12.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158001+0000", 2024-04-22T18:40:12.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:40:12.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158004+0000", 2024-04-22T18:40:12.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9610000000000001e-06 2024-04-22T18:40:12.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.354298+0000", 2024-04-22T18:40:12.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19629424500000001 2024-04-22T18:40:12.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.354367+0000", 2024-04-22T18:40:12.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9016999999999993e-05 2024-04-22T18:40:12.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:12.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.354470+0000", 2024-04-22T18:40:12.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103365 2024-04-22T18:40:12.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:12.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.455161+0000", 2024-04-22T18:40:12.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100690442 2024-04-22T18:40:12.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:12.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574379+0000", 2024-04-22T18:40:12.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.119217818 2024-04-22T18:40:12.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:12.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574406+0000", 2024-04-22T18:40:12.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7712000000000002e-05 2024-04-22T18:40:12.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574462+0000", 2024-04-22T18:40:12.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5405000000000002e-05 2024-04-22T18:40:12.594 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.594 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.594 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.594 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2846 4.7 4:e117a562:::smithi08937929-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-04-22T18:40:12.594 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:08.158020+0000", 2024-04-22T18:40:12.594 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.3299193300000001, 2024-04-22T18:40:12.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41656018099999997, 2024-04-22T18:40:12.595 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:12.595 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:12.595 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:12.595 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:12.595 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:12.595 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2846 2024-04-22T18:40:12.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.595 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:12.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:12.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158020+0000", 2024-04-22T18:40:12.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:12.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158020+0000", 2024-04-22T18:40:12.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:12.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:12.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158019+0000", 2024-04-22T18:40:12.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:12.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158021+0000", 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.928e-06 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158026+0000", 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5029999999999999e-06 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158029+0000", 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5889999999999999e-06 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.455080+0000", 2024-04-22T18:40:12.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29705077499999999 2024-04-22T18:40:12.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:12.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.455142+0000", 2024-04-22T18:40:12.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1982000000000004e-05 2024-04-22T18:40:12.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:12.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574482+0000", 2024-04-22T18:40:12.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.119340448 2024-04-22T18:40:12.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:12.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574529+0000", 2024-04-22T18:40:12.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6570999999999998e-05 2024-04-22T18:40:12.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:12.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:12.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:12.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574580+0000", 2024-04-22T18:40:12.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0787e-05 2024-04-22T18:40:12.599 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.599 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.599 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.599 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:12.599 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:12.599 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:12.601 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:12.601 INFO:tasks.rados.rados.0.smithi089.stdout:1334: finishing write tid 1 to smithi08937929-26 2024-04-22T18:40:12.602 INFO:tasks.rados.rados.0.smithi089.stdout:1339: expect (ObjNum 242 snap 0 seq_num 242) 2024-04-22T18:40:12.602 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:12.602+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:12.670 INFO:tasks.rados.rados.0.smithi089.stdout:1340: expect (ObjNum 215 snap 0 seq_num 215) 2024-04-22T18:40:12.687 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:40:12.701 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:12.703+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:12.746 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:12.746 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:12.746 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:12.747 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:12.756 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:12.802 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:12.803+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:12.882 INFO:tasks.rados.rados.0.smithi089.stdout:1335: expect (ObjNum 149 snap 0 seq_num 149) 2024-04-22T18:40:12.889 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:12.890 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:12.899 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:12.941 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:12.942+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:13.004 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:13.005+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:13.032 INFO:tasks.rados.rados.0.smithi089.stdout:1333: finishing set_chunk to oid 28 2024-04-22T18:40:13.032 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 124 (ObjNum 241 snap 0 seq_num 241) dirty exists 2024-04-22T18:40:13.032 INFO:tasks.rados.rados.0.smithi089.stdout:1336: expect (ObjNum 249 snap 0 seq_num 249) 2024-04-22T18:40:13.044 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:73 3.0 e34/19 3:061a7a63:::392f44ec26aa22e85be5f08273a7e45656f3bf4a4854038391a88e880ec01077:head v 34'31, mlcod=31'29)", 2024-04-22T18:40:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.587037214999995, 2024-04-22T18:40:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103503311, 2024-04-22T18:40:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458193+0000", 2024-04-22T18:40:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026628900000000001 2024-04-22T18:40:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458208+0000", 2024-04-22T18:40:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4780000000000001e-05 2024-04-22T18:40:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458217+0000", 2024-04-22T18:40:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4909999999999994e-06 2024-04-22T18:40:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459162+0000", 2024-04-22T18:40:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094440500000000005 2024-04-22T18:40:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459185+0000", 2024-04-22T18:40:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3685000000000001e-05 2024-04-22T18:40:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561394+0000", 2024-04-22T18:40:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10220874100000001 2024-04-22T18:40:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561430+0000", 2024-04-22T18:40:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6304999999999999e-05 2024-04-22T18:40:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:75 3.0 e34/19 3:18418e0f:::b7c35f165c374d5e075dde9ac2607e47ace3b0a1e2343ec6efb171380f725eea:head v 34'32, mlcod=31'29)", 2024-04-22T18:40:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.586023941999997, 2024-04-22T18:40:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104798078, 2024-04-22T18:40:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458939+0000", 2024-04-22T18:40:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459092+0000", 2024-04-22T18:40:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153838 2024-04-22T18:40:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459107+0000", 2024-04-22T18:40:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4703e-05 2024-04-22T18:40:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459115+0000", 2024-04-22T18:40:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-04-22T18:40:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560093+0000", 2024-04-22T18:40:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10097840499999999 2024-04-22T18:40:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560109+0000", 2024-04-22T18:40:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5631e-05 2024-04-22T18:40:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563706+0000", 2024-04-22T18:40:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035972059999999999 2024-04-22T18:40:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563738+0000", 2024-04-22T18:40:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2184000000000002e-05 2024-04-22T18:40:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:77 3.0 e34/19 3:28387c9f:::b6d7840f02d561da04e398fbf94d91b85e9066b426e48baf3b11dd4a4f6096c0:head v 34'33, mlcod=31'29)", 2024-04-22T18:40:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.585073862000002, 2024-04-22T18:40:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104186079, 2024-04-22T18:40:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459889+0000", 2024-04-22T18:40:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459963+0000", 2024-04-22T18:40:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3732999999999994e-05 2024-04-22T18:40:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459973+0000", 2024-04-22T18:40:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0285999999999999e-05 2024-04-22T18:40:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459987+0000", 2024-04-22T18:40:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3675999999999999e-05 2024-04-22T18:40:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560515+0000", 2024-04-22T18:40:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10052781299999999 2024-04-22T18:40:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560538+0000", 2024-04-22T18:40:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3861000000000001e-05 2024-04-22T18:40:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564044+0000", 2024-04-22T18:40:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035052009999999999 2024-04-22T18:40:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564076+0000", 2024-04-22T18:40:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2737000000000002e-05 2024-04-22T18:40:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:78 3.0 e34/19 3:3a5582ac:::bdd92f7deaa2d50e204891f64f9043469ba687d78b218ed518f2483ba3486a99:head v 34'34, mlcod=31'29)", 2024-04-22T18:40:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.584407744000004, 2024-04-22T18:40:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103547263, 2024-04-22T18:40:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460555+0000", 2024-04-22T18:40:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460861+0000", 2024-04-22T18:40:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000305958 2024-04-22T18:40:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460874+0000", 2024-04-22T18:40:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2639e-05 2024-04-22T18:40:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460881+0000", 2024-04-22T18:40:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2570000000000001e-06 2024-04-22T18:40:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560990+0000", 2024-04-22T18:40:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010883199999999 2024-04-22T18:40:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561011+0000", 2024-04-22T18:40:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1319e-05 2024-04-22T18:40:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564079+0000", 2024-04-22T18:40:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00306815 2024-04-22T18:40:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564104+0000", 2024-04-22T18:40:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4172000000000002e-05 2024-04-22T18:40:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:40:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.583792450000004, 2024-04-22T18:40:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:40:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:40:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:40:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:40:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:40:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:40:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:40:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:40:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:40:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:40:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:40:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:40:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:40:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:40:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:40:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:40:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:40:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 64.707118628000003, 2024-04-22T18:40:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:40:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:40:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:40:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:40:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:40:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:40:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:40:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:40:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:40:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:40:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:40:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:40:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:40:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:40:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:40:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:40:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1689 3.0 e47/19 3:1d4bcadf:::e7eb8db0531d9f456e91fe87bc79e47ebd80459f1783dc2a5e53d4822cd34f49:head v 47'85, mlcod=47'83)", 2024-04-22T18:40:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:40:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.336076169999998, 2024-04-22T18:40:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102152033, 2024-04-22T18:40:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:40:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708888+0000", 2024-04-22T18:40:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.708885+0000", 2024-04-22T18:40:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709414+0000", 2024-04-22T18:40:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000528459 2024-04-22T18:40:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709428+0000", 2024-04-22T18:40:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4745e-05 2024-04-22T18:40:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.709436+0000", 2024-04-22T18:40:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0830000000000004e-06 2024-04-22T18:40:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.809587+0000", 2024-04-22T18:40:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100150641 2024-04-22T18:40:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.809603+0000", 2024-04-22T18:40:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6061000000000001e-05 2024-04-22T18:40:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.811013+0000", 2024-04-22T18:40:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0014094680000000001 2024-04-22T18:40:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.811040+0000", 2024-04-22T18:40:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7474000000000002e-05 2024-04-22T18:40:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:1729 3.0 e47/19 3:27b110a1:::34336650543f8f8bff225091f9c8c14678dc47baabcb4d8371890fc935dbc3e4:head v 47'89, mlcod=47'87)", 2024-04-22T18:40:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:40:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.567022399999999, 2024-04-22T18:40:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102111412, 2024-04-22T18:40:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:40:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477942+0000", 2024-04-22T18:40:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.477940+0000", 2024-04-22T18:40:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478305+0000", 2024-04-22T18:40:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036512200000000001 2024-04-22T18:40:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478320+0000", 2024-04-22T18:40:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5475999999999999e-05 2024-04-22T18:40:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.478326+0000", 2024-04-22T18:40:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.302e-06 2024-04-22T18:40:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.578536+0000", 2024-04-22T18:40:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100209436 2024-04-22T18:40:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.578575+0000", 2024-04-22T18:40:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9223e-05 2024-04-22T18:40:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.580023+0000", 2024-04-22T18:40:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001447463 2024-04-22T18:40:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.580053+0000", 2024-04-22T18:40:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0534999999999998e-05 2024-04-22T18:40:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(3.0 65/63 [PushOp(3:0f5f80b3:::e65ec2deb1a27f55813533c3e73a682460e829b5dd6586a33a43649649fa77f6:head, version: 53'111, data_included: [0~589824,917504~221696], data_size: 811520, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:0f5f80b3:::e65ec2deb1a27f55813533c3e73a682460e829b5dd6586a33a43649649fa77f6:head@53'111, size: 1139200, copy_subset: [0~1139200], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1139200, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-22T18:40:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:54.953508+0000", 2024-04-22T18:40:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.091456031, 2024-04-22T18:40:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101969016, 2024-04-22T18:40:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.953508+0000", 2024-04-22T18:40:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.953508+0000", 2024-04-22T18:40:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.953506+0000", 2024-04-22T18:40:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.953962+0000", 2024-04-22T18:40:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00045575900000000003 2024-04-22T18:40:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:54.953984+0000", 2024-04-22T18:40:13.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1579000000000001e-05 2024-04-22T18:40:13.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.054112+0000", 2024-04-22T18:40:13.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100128071 2024-04-22T18:40:13.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.054190+0000", 2024-04-22T18:40:13.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8047999999999995e-05 2024-04-22T18:40:13.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.054205+0000", 2024-04-22T18:40:13.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5273000000000001e-05 2024-04-22T18:40:13.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:55.055477+0000", 2024-04-22T18:40:13.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001272053 2024-04-22T18:40:13.086 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.086 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.087 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2699 3.0 e73/63 3:20ef4e39:::73f6cd42eb17b7fbc29bb083f5443f1d5c65f51bcb8aa3d092b3b80aa1d4c657:head v 73'144, mlcod=71'142)", 2024-04-22T18:40:13.087 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.350683+0000", 2024-04-22T18:40:13.087 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.6942807680000005, 2024-04-22T18:40:13.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10232055599999999, 2024-04-22T18:40:13.087 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.087 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.087 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.350683+0000", 2024-04-22T18:40:13.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.350683+0000", 2024-04-22T18:40:13.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.350680+0000", 2024-04-22T18:40:13.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:13.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351512+0000", 2024-04-22T18:40:13.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00083224099999999995 2024-04-22T18:40:13.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351524+0000", 2024-04-22T18:40:13.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1763000000000001e-05 2024-04-22T18:40:13.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351531+0000", 2024-04-22T18:40:13.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0670000000000003e-06 2024-04-22T18:40:13.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351596+0000", 2024-04-22T18:40:13.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4659999999999994e-05 2024-04-22T18:40:13.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351616+0000", 2024-04-22T18:40:13.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0596000000000001e-05 2024-04-22T18:40:13.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.452977+0000", 2024-04-22T18:40:13.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101360332 2024-04-22T18:40:13.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.453004+0000", 2024-04-22T18:40:13.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7127e-05 2024-04-22T18:40:13.091 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.091 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.091 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.091 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.7.13:7 3.0 e73/63 3:1554f608:::d614c1f2f74bb29087660bd6e888eafe46b1a7ed3cbdede85b36de771c7f0d66:head v 73'146, mlcod=71'142)", 2024-04-22T18:40:13.092 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.351566+0000", 2024-04-22T18:40:13.092 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.6933977880000004, 2024-04-22T18:40:13.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102004704, 2024-04-22T18:40:13.092 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.092 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.092 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351566+0000", 2024-04-22T18:40:13.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351566+0000", 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351566+0000", 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351570+0000", 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1649999999999999e-06 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351573+0000", 2024-04-22T18:40:13.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2189999999999998e-06 2024-04-22T18:40:13.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.351576+0000", 2024-04-22T18:40:13.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7689999999999998e-06 2024-04-22T18:40:13.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.453030+0000", 2024-04-22T18:40:13.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101453997 2024-04-22T18:40:13.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.453046+0000", 2024-04-22T18:40:13.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5634e-05 2024-04-22T18:40:13.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.453542+0000", 2024-04-22T18:40:13.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049580499999999999 2024-04-22T18:40:13.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.453571+0000", 2024-04-22T18:40:13.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9306e-05 2024-04-22T18:40:13.095 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.096 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.096 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.096 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:40:13.096 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:13.096 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5023151049999992, 2024-04-22T18:40:13.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:40:13.096 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.096 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.096 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:40:13.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:40:13.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:40:13.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:40:13.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:40:13.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:40:13.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:40:13.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:40:13.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:40:13.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:40:13.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:40:13.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:40:13.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:40:13.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:40:13.100 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.100 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.100 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.100 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2745 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'28, mlcod=74'25)", 2024-04-22T18:40:13.100 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.643080+0000", 2024-04-22T18:40:13.100 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.4018843440000008, 2024-04-22T18:40:13.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10246031899999999, 2024-04-22T18:40:13.100 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.100 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.100 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643080+0000", 2024-04-22T18:40:13.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643080+0000", 2024-04-22T18:40:13.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643078+0000", 2024-04-22T18:40:13.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:13.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643090+0000", 2024-04-22T18:40:13.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1531e-05 2024-04-22T18:40:13.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643099+0000", 2024-04-22T18:40:13.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0650000000000005e-06 2024-04-22T18:40:13.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643105+0000", 2024-04-22T18:40:13.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.993e-06 2024-04-22T18:40:13.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.744552+0000", 2024-04-22T18:40:13.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101447464 2024-04-22T18:40:13.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.744577+0000", 2024-04-22T18:40:13.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4539e-05 2024-04-22T18:40:13.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745515+0000", 2024-04-22T18:40:13.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093760499999999999 2024-04-22T18:40:13.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745540+0000", 2024-04-22T18:40:13.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5582999999999999e-05 2024-04-22T18:40:13.104 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.104 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.104 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.105 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:160 3.0 e76/63 3:18d95008:::60ff66c0337e5d444612de0bca253cc28d83704a27a7c526a661c5877482b5f3:head v 76'156, mlcod=76'154)", 2024-04-22T18:40:13.105 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391289+0000", 2024-04-22T18:40:13.105 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.6536753879999999, 2024-04-22T18:40:13.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101821024, 2024-04-22T18:40:13.105 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.105 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.105 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391289+0000", 2024-04-22T18:40:13.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391289+0000", 2024-04-22T18:40:13.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391287+0000", 2024-04-22T18:40:13.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:13.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391329+0000", 2024-04-22T18:40:13.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2197999999999998e-05 2024-04-22T18:40:13.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391345+0000", 2024-04-22T18:40:13.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6608000000000001e-05 2024-04-22T18:40:13.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391355+0000", 2024-04-22T18:40:13.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0134e-05 2024-04-22T18:40:13.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391417+0000", 2024-04-22T18:40:13.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1278999999999994e-05 2024-04-22T18:40:13.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391440+0000", 2024-04-22T18:40:13.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2758000000000001e-05 2024-04-22T18:40:13.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493059+0000", 2024-04-22T18:40:13.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101619691 2024-04-22T18:40:13.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493110+0000", 2024-04-22T18:40:13.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0556000000000002e-05 2024-04-22T18:40:13.109 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.109 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.109 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.109 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:161 3.0 e76/63 3:3638311b:::f282598a146af3da1c05c8882972fba387afd249b39f6eeb6e6052a5de37fd2d:head v 76'157, mlcod=76'154)", 2024-04-22T18:40:13.109 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:13.109 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.653595675, 2024-04-22T18:40:13.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104845172, 2024-04-22T18:40:13.110 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.110 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.110 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:13.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:13.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:13.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:13.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391440+0000", 2024-04-22T18:40:13.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2045999999999994e-05 2024-04-22T18:40:13.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391448+0000", 2024-04-22T18:40:13.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8189999999999992e-06 2024-04-22T18:40:13.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391453+0000", 2024-04-22T18:40:13.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8379999999999998e-06 2024-04-22T18:40:13.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391839+0000", 2024-04-22T18:40:13.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000385325 2024-04-22T18:40:13.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391854+0000", 2024-04-22T18:40:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5006e-05 2024-04-22T18:40:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496191+0000", 2024-04-22T18:40:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104336792 2024-04-22T18:40:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496214+0000", 2024-04-22T18:40:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3079999999999999e-05 2024-04-22T18:40:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.113 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.114 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.114 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:165 3.0 e76/63 3:1e2f7d15:::93d9bab0d1f539405a95a28c02bcf2dcbdfa6f3e449e5e5e3f27f774df3c508a:head v 76'158, mlcod=76'154)", 2024-04-22T18:40:13.114 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:13.114 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.6535014920000002, 2024-04-22T18:40:13.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104788752, 2024-04-22T18:40:13.114 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.114 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.114 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:13.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391462+0000", 2024-04-22T18:40:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391522+0000", 2024-04-22T18:40:13.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0148000000000001e-05 2024-04-22T18:40:13.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391531+0000", 2024-04-22T18:40:13.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5709999999999998e-06 2024-04-22T18:40:13.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391536+0000", 2024-04-22T18:40:13.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6180000000000003e-06 2024-04-22T18:40:13.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392218+0000", 2024-04-22T18:40:13.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000681326 2024-04-22T18:40:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392237+0000", 2024-04-22T18:40:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9726999999999999e-05 2024-04-22T18:40:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496231+0000", 2024-04-22T18:40:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103993672 2024-04-22T18:40:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496251+0000", 2024-04-22T18:40:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0327999999999999e-05 2024-04-22T18:40:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:166 3.0 e76/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 76'159, mlcod=76'154)", 2024-04-22T18:40:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.6534235940000004, 2024-04-22T18:40:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104724628, 2024-04-22T18:40:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391540+0000", 2024-04-22T18:40:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391627+0000", 2024-04-22T18:40:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7330000000000006e-05 2024-04-22T18:40:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391635+0000", 2024-04-22T18:40:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6709999999999999e-06 2024-04-22T18:40:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391640+0000", 2024-04-22T18:40:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3199999999999999e-06 2024-04-22T18:40:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492541+0000", 2024-04-22T18:40:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100900873 2024-04-22T18:40:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492560+0000", 2024-04-22T18:40:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8502999999999999e-05 2024-04-22T18:40:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496254+0000", 2024-04-22T18:40:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036945849999999998 2024-04-22T18:40:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496265+0000", 2024-04-22T18:40:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0869999999999999e-05 2024-04-22T18:40:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:167 3.0 e76/63 3:384c1ce2:::81dc8c58ebe106ae6fa9feadc06291c645d9079ad84b78214d4e6440fbc5def8:head v 76'160, mlcod=76'154)", 2024-04-22T18:40:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.653308719, 2024-04-22T18:40:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10487859400000001, 2024-04-22T18:40:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:13.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:13.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391707+0000", 2024-04-22T18:40:13.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2343999999999999e-05 2024-04-22T18:40:13.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391714+0000", 2024-04-22T18:40:13.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.596e-06 2024-04-22T18:40:13.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391719+0000", 2024-04-22T18:40:13.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9130000000000002e-06 2024-04-22T18:40:13.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493154+0000", 2024-04-22T18:40:13.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101435256 2024-04-22T18:40:13.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493177+0000", 2024-04-22T18:40:13.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2796e-05 2024-04-22T18:40:13.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496509+0000", 2024-04-22T18:40:13.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033320139999999999 2024-04-22T18:40:13.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496534+0000", 2024-04-22T18:40:13.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5265999999999998e-05 2024-04-22T18:40:13.127 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.127 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.127 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.127 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:103 3.0 e78/63 3:0f5f80b3:::e65ec2deb1a27f55813533c3e73a682460e829b5dd6586a33a43649649fa77f6:head v 78'163, mlcod=76'161)", 2024-04-22T18:40:13.127 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.283291+0000", 2024-04-22T18:40:13.127 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.7616733949999999, 2024-04-22T18:40:13.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101953407, 2024-04-22T18:40:13.127 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.127 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.127 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.283291+0000", 2024-04-22T18:40:13.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.283291+0000", 2024-04-22T18:40:13.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.283290+0000", 2024-04-22T18:40:13.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.283296+0000", 2024-04-22T18:40:13.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6930000000000004e-06 2024-04-22T18:40:13.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.283309+0000", 2024-04-22T18:40:13.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3308e-05 2024-04-22T18:40:13.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.383415+0000", 2024-04-22T18:40:13.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010583000000001 2024-04-22T18:40:13.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.383479+0000", 2024-04-22T18:40:13.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4289999999999996e-05 2024-04-22T18:40:13.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.383509+0000", 2024-04-22T18:40:13.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9609999999999999e-05 2024-04-22T18:40:13.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.385202+0000", 2024-04-22T18:40:13.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0016931520000000001 2024-04-22T18:40:13.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.385244+0000", 2024-04-22T18:40:13.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1891000000000003e-05 2024-04-22T18:40:13.131 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.131 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.131 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.132 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:197 3.0 e78/63 3:19ca42e0:::2109e7e4526cbe26b2963cdd11634231ee7f4d35dabe9962c44b62fafb130ddf:head v 78'167, mlcod=78'165)", 2024-04-22T18:40:13.132 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.540869+0000", 2024-04-22T18:40:13.132 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.5040950669999997, 2024-04-22T18:40:13.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10287555800000001, 2024-04-22T18:40:13.132 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.132 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.132 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540869+0000", 2024-04-22T18:40:13.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540869+0000", 2024-04-22T18:40:13.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540867+0000", 2024-04-22T18:40:13.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:13.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540879+0000", 2024-04-22T18:40:13.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1316e-05 2024-04-22T18:40:13.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540895+0000", 2024-04-22T18:40:13.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6072000000000001e-05 2024-04-22T18:40:13.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540904+0000", 2024-04-22T18:40:13.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8969999999999997e-06 2024-04-22T18:40:13.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.641023+0000", 2024-04-22T18:40:13.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10011914 2024-04-22T18:40:13.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.641049+0000", 2024-04-22T18:40:13.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6455e-05 2024-04-22T18:40:13.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.643687+0000", 2024-04-22T18:40:13.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026375499999999998 2024-04-22T18:40:13.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.643745+0000", 2024-04-22T18:40:13.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7725e-05 2024-04-22T18:40:13.136 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.136 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.136 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.136 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.136 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.136 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:13.137 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:13.138 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:13.139+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:13.153 INFO:tasks.rados.rados.0.smithi089.stdout:1334: finishing write tid 2 to smithi08937929-26 2024-04-22T18:40:13.153 INFO:tasks.rados.rados.0.smithi089.stdout:1334: finishing write tid 3 to smithi08937929-26 2024-04-22T18:40:13.153 INFO:tasks.rados.rados.0.smithi089.stdout:1334: finishing write tid 4 to smithi08937929-26 2024-04-22T18:40:13.153 INFO:tasks.rados.rados.0.smithi089.stdout:1334: oid 26 updating version 0 to 96 2024-04-22T18:40:13.153 INFO:tasks.rados.rados.0.smithi089.stdout:1334: oid 26 updating version 96 to 97 2024-04-22T18:40:13.153 INFO:tasks.rados.rados.0.smithi089.stdout:1334: oid 26 version 97 is already newer than 95 2024-04-22T18:40:13.153 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 97 (ObjNum 252 snap 0 seq_num 252) dirty exists 2024-04-22T18:40:13.153 INFO:tasks.rados.rados.0.smithi089.stdout:1334: left oid 26 (ObjNum 252 snap 0 seq_num 252) 2024-04-22T18:40:13.153 INFO:tasks.rados.rados.0.smithi089.stdout:1329: finishing set_chunk to oid 48 2024-04-22T18:40:13.153 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 67 (ObjNum 246 snap 0 seq_num 246) dirty exists 2024-04-22T18:40:13.153 INFO:tasks.rados.rados.0.smithi089.stdout:1337: finishing write tid 1 to smithi08937929-8 2024-04-22T18:40:13.154 INFO:tasks.rados.rados.0.smithi089.stdout:1325: done (15 left) 2024-04-22T18:40:13.154 INFO:tasks.rados.rados.0.smithi089.stdout:1326: done (14 left) 2024-04-22T18:40:13.154 INFO:tasks.rados.rados.0.smithi089.stdout:1327: done (13 left) 2024-04-22T18:40:13.154 INFO:tasks.rados.rados.0.smithi089.stdout:1328: done (12 left) 2024-04-22T18:40:13.154 INFO:tasks.rados.rados.0.smithi089.stdout:1329: done (11 left) 2024-04-22T18:40:13.154 INFO:tasks.rados.rados.0.smithi089.stdout:1330: done (10 left) 2024-04-22T18:40:13.154 INFO:tasks.rados.rados.0.smithi089.stdout:1331: done (9 left) 2024-04-22T18:40:13.154 INFO:tasks.rados.rados.0.smithi089.stdout:1332: done (8 left) 2024-04-22T18:40:13.154 INFO:tasks.rados.rados.0.smithi089.stdout:1333: done (7 left) 2024-04-22T18:40:13.154 INFO:tasks.rados.rados.0.smithi089.stdout:1334: done (6 left) 2024-04-22T18:40:13.154 INFO:tasks.rados.rados.0.smithi089.stdout:1335: done (5 left) 2024-04-22T18:40:13.154 INFO:tasks.rados.rados.0.smithi089.stdout:1336: done (4 left) 2024-04-22T18:40:13.154 INFO:tasks.rados.rados.0.smithi089.stdout:1339: done (3 left) 2024-04-22T18:40:13.154 INFO:tasks.rados.rados.0.smithi089.stdout:1340: done (2 left) 2024-04-22T18:40:13.155 INFO:tasks.rados.rados.0.smithi089.stdout:1341: read oid 48 snap -1 2024-04-22T18:40:13.155 INFO:tasks.rados.rados.0.smithi089.stdout:1341: expect (ObjNum 246 snap 0 seq_num 246) 2024-04-22T18:40:13.155 INFO:tasks.rados.rados.0.smithi089.stdout:1342: set_chunk oid 22 target oid 2024-04-22T18:40:13.238 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:13.240+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:13.257 INFO:tasks.rados.rados.0.smithi089.stdout:1342: set_chunk oid 22 offset: 159744 length: 160256 target oid 7c9326fc5b2f7ae0fa9228f57b956bf697046c4a9ac75ec365fc814f6f4e2c0b offset: 0 snap -1 2024-04-22T18:40:13.257 INFO:tasks.rados.rados.0.smithi089.stdout:1343: set_chunk oid 2 target oid 2024-04-22T18:40:13.257 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:13.257 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:13.257 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:13.257 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:13.267 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:13.338 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:13.340+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:13.369 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:13.369 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:13.369 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:13.369 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:13.369 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:13.378 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:13.439 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:13.440+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:13.467 INFO:tasks.rados.rados.0.smithi089.stdout:1343: set_chunk oid 2 offset: 654336 length: 905728 target oid a33d83b94e02a3a4ac12a025f9845af4da3af842eac7011455c4e584d2f5a649 offset: 0 snap -1 2024-04-22T18:40:13.467 INFO:tasks.rados.rados.0.smithi089.stdout:1344: read oid 12 snap -1 2024-04-22T18:40:13.467 INFO:tasks.rados.rados.0.smithi089.stdout:1344: expect (ObjNum 245 snap 0 seq_num 245) 2024-04-22T18:40:13.467 INFO:tasks.rados.rados.0.smithi089.stdout:1337: finishing write tid 2 to smithi08937929-8 2024-04-22T18:40:13.468 INFO:tasks.rados.rados.0.smithi089.stdout:1337: finishing write tid 3 to smithi08937929-8 2024-04-22T18:40:13.468 INFO:tasks.rados.rados.0.smithi089.stdout:1345: read oid 30 snap -1 2024-04-22T18:40:13.468 INFO:tasks.rados.rados.0.smithi089.stdout:1345: expect (ObjNum 243 snap 0 seq_num 243) 2024-04-22T18:40:13.468 INFO:tasks.rados.rados.0.smithi089.stdout:1346: read oid 21 snap -1 2024-04-22T18:40:13.468 INFO:tasks.rados.rados.0.smithi089.stdout:1346: expect deleted 2024-04-22T18:40:13.468 INFO:tasks.rados.rados.0.smithi089.stdout:1347: read oid 33 snap -1 2024-04-22T18:40:13.468 INFO:tasks.rados.rados.0.smithi089.stdout:1347: expect (ObjNum 244 snap 0 seq_num 244) 2024-04-22T18:40:13.468 INFO:tasks.rados.rados.0.smithi089.stdout:1348: write (excl) oid 16 current snap is 0 2024-04-22T18:40:13.468 INFO:tasks.rados.rados.0.smithi089.stdout:1337: finishing write tid 4 to smithi08937929-8 2024-04-22T18:40:13.468 INFO:tasks.rados.rados.0.smithi089.stdout:1337: oid 8 updating version 0 to 22 2024-04-22T18:40:13.468 INFO:tasks.rados.rados.0.smithi089.stdout:1337: oid 8 version 22 is already newer than 21 2024-04-22T18:40:13.468 INFO:tasks.rados.rados.0.smithi089.stdout:1337: oid 8 version 22 is already newer than 20 2024-04-22T18:40:13.468 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 22 (ObjNum 253 snap 0 seq_num 253) dirty exists 2024-04-22T18:40:13.469 INFO:tasks.rados.rados.0.smithi089.stdout:1337: left oid 8 (ObjNum 253 snap 0 seq_num 253) 2024-04-22T18:40:13.469 INFO:tasks.rados.rados.0.smithi089.stdout:1338: finishing write tid 1 to smithi08937929-5 2024-04-22T18:40:13.469 INFO:tasks.rados.rados.0.smithi089.stdout:1338: finishing write tid 2 to smithi08937929-5 2024-04-22T18:40:13.469 INFO:tasks.rados.rados.0.smithi089.stdout:1338: finishing write tid 3 to smithi08937929-5 2024-04-22T18:40:13.469 INFO:tasks.rados.rados.0.smithi089.stdout:1338: finishing write tid 4 to smithi08937929-5 2024-04-22T18:40:13.469 INFO:tasks.rados.rados.0.smithi089.stdout:1338: oid 5 updating version 0 to 130 2024-04-22T18:40:13.469 INFO:tasks.rados.rados.0.smithi089.stdout:1338: oid 5 updating version 130 to 131 2024-04-22T18:40:13.469 INFO:tasks.rados.rados.0.smithi089.stdout:1338: oid 5 version 131 is already newer than 129 2024-04-22T18:40:13.469 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 131 (ObjNum 254 snap 0 seq_num 254) dirty exists 2024-04-22T18:40:13.469 INFO:tasks.rados.rados.0.smithi089.stdout:1338: left oid 5 (ObjNum 254 snap 0 seq_num 254) 2024-04-22T18:40:13.469 INFO:tasks.rados.rados.0.smithi089.stdout:1341: expect (ObjNum 246 snap 0 seq_num 246) 2024-04-22T18:40:13.482 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:78 4.0 4:01baf5a9:::smithi08937929-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.969351454000005, 2024-04-22T18:40:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.398522192, 2024-04-22T18:40:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 78 2024-04-22T18:40:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511934+0000", 2024-04-22T18:40:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512012+0000", 2024-04-22T18:40:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8329000000000002e-05 2024-04-22T18:40:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512023+0000", 2024-04-22T18:40:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0755e-05 2024-04-22T18:40:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512028+0000", 2024-04-22T18:40:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6470000000000003e-06 2024-04-22T18:40:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512068+0000", 2024-04-22T18:40:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9721999999999997e-05 2024-04-22T18:40:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512125+0000", 2024-04-22T18:40:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6564999999999998e-05 2024-04-22T18:40:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512200+0000", 2024-04-22T18:40:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5708000000000001e-05 2024-04-22T18:40:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513131+0000", 2024-04-22T18:40:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093063 2024-04-22T18:40:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910394+0000", 2024-04-22T18:40:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.397263388 2024-04-22T18:40:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910430+0000", 2024-04-22T18:40:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5485999999999999e-05 2024-04-22T18:40:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910457+0000", 2024-04-22T18:40:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7086000000000001e-05 2024-04-22T18:40:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:79 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.969250798000004, 2024-04-22T18:40:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39855419399999997, 2024-04-22T18:40:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 79 2024-04-22T18:40:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512035+0000", 2024-04-22T18:40:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512045+0000", 2024-04-22T18:40:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0379e-05 2024-04-22T18:40:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512051+0000", 2024-04-22T18:40:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6030000000000004e-06 2024-04-22T18:40:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512057+0000", 2024-04-22T18:40:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7270000000000004e-06 2024-04-22T18:40:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512756+0000", 2024-04-22T18:40:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069954000000000001 2024-04-22T18:40:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512806+0000", 2024-04-22T18:40:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9922000000000001e-05 2024-04-22T18:40:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512927+0000", 2024-04-22T18:40:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012059 2024-04-22T18:40:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.515883+0000", 2024-04-22T18:40:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029563739999999999 2024-04-22T18:40:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910510+0000", 2024-04-22T18:40:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39462745199999999 2024-04-22T18:40:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910542+0000", 2024-04-22T18:40:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1225e-05 2024-04-22T18:40:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910590+0000", 2024-04-22T18:40:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8052e-05 2024-04-22T18:40:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:80 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.969225696999999, 2024-04-22T18:40:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39866376999999997, 2024-04-22T18:40:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 80 2024-04-22T18:40:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512060+0000", 2024-04-22T18:40:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512063+0000", 2024-04-22T18:40:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.999e-06 2024-04-22T18:40:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512069+0000", 2024-04-22T18:40:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.891e-06 2024-04-22T18:40:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512073+0000", 2024-04-22T18:40:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4429999999999996e-06 2024-04-22T18:40:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513084+0000", 2024-04-22T18:40:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010102609999999999 2024-04-22T18:40:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513123+0000", 2024-04-22T18:40:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9237999999999999e-05 2024-04-22T18:40:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910618+0000", 2024-04-22T18:40:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39749549299999998 2024-04-22T18:40:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910670+0000", 2024-04-22T18:40:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1221000000000003e-05 2024-04-22T18:40:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910724+0000", 2024-04-22T18:40:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4777000000000001e-05 2024-04-22T18:40:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:419 4.0 4:01baf5a9:::smithi08937929-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:40:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:40:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.079064677000005, 2024-04-22T18:40:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30324478199999999, 2024-04-22T18:40:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 419 2024-04-22T18:40:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:40:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:40:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402219+0000", 2024-04-22T18:40:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402229+0000", 2024-04-22T18:40:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2550000000000003e-06 2024-04-22T18:40:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402242+0000", 2024-04-22T18:40:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2989999999999999e-05 2024-04-22T18:40:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402250+0000", 2024-04-22T18:40:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7900000000000005e-06 2024-04-22T18:40:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404085+0000", 2024-04-22T18:40:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001834397 2024-04-22T18:40:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:40:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404092+0000", 2024-04-22T18:40:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7889999999999997e-06 2024-04-22T18:40:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502534+0000", 2024-04-22T18:40:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098442331999999994 2024-04-22T18:40:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502631+0000", 2024-04-22T18:40:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7617999999999999e-05 2024-04-22T18:40:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604414+0000", 2024-04-22T18:40:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101782847 2024-04-22T18:40:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604888+0000", 2024-04-22T18:40:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047325999999999998 2024-04-22T18:40:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705376+0000", 2024-04-22T18:40:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100488623 2024-04-22T18:40:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705424+0000", 2024-04-22T18:40:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7964999999999999e-05 2024-04-22T18:40:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705466+0000", 2024-04-22T18:40:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2274999999999998e-05 2024-04-22T18:40:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:229 3.2 e51/29 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head v 51'117, mlcod=51'115)", 2024-04-22T18:40:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.670243638999999, 2024-04-22T18:40:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31548799399999999, 2024-04-22T18:40:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811041+0000", 2024-04-22T18:40:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811054+0000", 2024-04-22T18:40:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3188e-05 2024-04-22T18:40:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811068+0000", 2024-04-22T18:40:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4239999999999999e-05 2024-04-22T18:40:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.911141+0000", 2024-04-22T18:40:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100072851 2024-04-22T18:40:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025053+0000", 2024-04-22T18:40:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.11391237 2024-04-22T18:40:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025083+0000", 2024-04-22T18:40:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9703999999999999e-05 2024-04-22T18:40:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126504+0000", 2024-04-22T18:40:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10142053500000001 2024-04-22T18:40:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126531+0000", 2024-04-22T18:40:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7172999999999999e-05 2024-04-22T18:40:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2347 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 3378207] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:40:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:40:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.293465414, 2024-04-22T18:40:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30520418599999999, 2024-04-22T18:40:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2347 2024-04-22T18:40:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187820+0000", 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187823+0000", 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6299999999999998e-06 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187827+0000", 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3739999999999998e-06 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187829+0000", 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6290000000000002e-06 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291848+0000", 2024-04-22T18:40:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104018812 2024-04-22T18:40:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291924+0000", 2024-04-22T18:40:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6421000000000006e-05 2024-04-22T18:40:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292050+0000", 2024-04-22T18:40:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012571500000000001 2024-04-22T18:40:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292407+0000", 2024-04-22T18:40:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035709199999999999 2024-04-22T18:40:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.492962+0000", 2024-04-22T18:40:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20055479200000001 2024-04-22T18:40:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.492993+0000", 2024-04-22T18:40:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1041000000000003e-05 2024-04-22T18:40:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493025+0000", 2024-04-22T18:40:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2092999999999999e-05 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2348 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.293454098000002, 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30532242999999998, 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2348 2024-04-22T18:40:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187833+0000", 2024-04-22T18:40:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.437e-06 2024-04-22T18:40:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187836+0000", 2024-04-22T18:40:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.897e-06 2024-04-22T18:40:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187839+0000", 2024-04-22T18:40:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7250000000000002e-06 2024-04-22T18:40:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292213+0000", 2024-04-22T18:40:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104373761 2024-04-22T18:40:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292256+0000", 2024-04-22T18:40:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2896999999999999e-05 2024-04-22T18:40:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493044+0000", 2024-04-22T18:40:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20078818700000001 2024-04-22T18:40:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493099+0000", 2024-04-22T18:40:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5009000000000001e-05 2024-04-22T18:40:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493155+0000", 2024-04-22T18:40:13.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5807000000000003e-05 2024-04-22T18:40:13.522 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.522 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.522 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.522 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:289 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:13.522 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:13.522 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.840561387999999, 2024-04-22T18:40:13.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9176801189999999, 2024-04-22T18:40:13.522 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.523 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.523 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:13.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:13.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640723+0000", 2024-04-22T18:40:13.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:13.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640732+0000", 2024-04-22T18:40:13.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5809999999999997e-06 2024-04-22T18:40:13.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640744+0000", 2024-04-22T18:40:13.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.261e-05 2024-04-22T18:40:13.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640753+0000", 2024-04-22T18:40:13.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7460000000000006e-06 2024-04-22T18:40:13.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642189+0000", 2024-04-22T18:40:13.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001435551 2024-04-22T18:40:13.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:13.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642200+0000", 2024-04-22T18:40:13.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1389999999999999e-05 2024-04-22T18:40:13.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.847015+0000", 2024-04-22T18:40:13.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048149160000001 2024-04-22T18:40:13.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:13.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.848217+0000", 2024-04-22T18:40:13.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012020659999999999 2024-04-22T18:40:13.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351823+0000", 2024-04-22T18:40:13.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50360565099999999 2024-04-22T18:40:13.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351907+0000", 2024-04-22T18:40:13.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4141999999999995e-05 2024-04-22T18:40:13.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:13.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352051+0000", 2024-04-22T18:40:13.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014386600000000001 2024-04-22T18:40:13.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453617+0000", 2024-04-22T18:40:13.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156649500000001 2024-04-22T18:40:13.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558297+0000", 2024-04-22T18:40:13.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10468000399999999 2024-04-22T18:40:13.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558331+0000", 2024-04-22T18:40:13.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3518e-05 2024-04-22T18:40:13.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558405+0000", 2024-04-22T18:40:13.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4221000000000006e-05 2024-04-22T18:40:13.529 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.529 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.529 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.529 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:1 3.2 3:6d90b903:::64b5942574fb36ca1b9a2be186c6b740edf58e989056a2f88b83cd280451a438:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:13.529 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:13.529 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.309260428, 2024-04-22T18:40:13.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.285534422, 2024-04-22T18:40:13.529 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.529 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.530 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:13.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:13.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172024+0000", 2024-04-22T18:40:13.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:13.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173091+0000", 2024-04-22T18:40:13.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010677569999999999 2024-04-22T18:40:13.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173103+0000", 2024-04-22T18:40:13.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1241e-05 2024-04-22T18:40:13.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173109+0000", 2024-04-22T18:40:13.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4300000000000003e-06 2024-04-22T18:40:13.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173168+0000", 2024-04-22T18:40:13.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9011000000000001e-05 2024-04-22T18:40:13.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:13.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173177+0000", 2024-04-22T18:40:13.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-04-22T18:40:13.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948501+0000", 2024-04-22T18:40:13.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77532387199999997 2024-04-22T18:40:13.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:13.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948912+0000", 2024-04-22T18:40:13.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041183000000000002 2024-04-22T18:40:13.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950438+0000", 2024-04-22T18:40:13.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015253199999999999 2024-04-22T18:40:13.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950488+0000", 2024-04-22T18:40:13.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0285999999999999e-05 2024-04-22T18:40:13.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:13.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950601+0000", 2024-04-22T18:40:13.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011244899999999999 2024-04-22T18:40:13.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954048+0000", 2024-04-22T18:40:13.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003447313 2024-04-22T18:40:13.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457373+0000", 2024-04-22T18:40:13.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50332465299999996 2024-04-22T18:40:13.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457430+0000", 2024-04-22T18:40:13.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7574999999999997e-05 2024-04-22T18:40:13.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457560+0000", 2024-04-22T18:40:13.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013023900000000001 2024-04-22T18:40:13.536 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.536 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.536 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.536 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:8 3.2 3:5f64590d:::7c47933f331d40689081b2bd8f9d7f7bdae1be1fdccb52d696047a3969e902b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:13.536 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:13.536 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.308168059, 2024-04-22T18:40:13.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3854200569999999, 2024-04-22T18:40:13.536 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.536 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.536 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173117+0000", 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173393+0000", 2024-04-22T18:40:13.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000275278 2024-04-22T18:40:13.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173399+0000", 2024-04-22T18:40:13.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4590000000000003e-06 2024-04-22T18:40:13.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173406+0000", 2024-04-22T18:40:13.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2450000000000001e-06 2024-04-22T18:40:13.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173795+0000", 2024-04-22T18:40:13.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038912400000000002 2024-04-22T18:40:13.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:13.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173804+0000", 2024-04-22T18:40:13.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-04-22T18:40:13.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.949506+0000", 2024-04-22T18:40:13.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77570185700000005 2024-04-22T18:40:13.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:13.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950166+0000", 2024-04-22T18:40:13.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066022800000000001 2024-04-22T18:40:13.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352843+0000", 2024-04-22T18:40:13.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40267661799999999 2024-04-22T18:40:13.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352911+0000", 2024-04-22T18:40:13.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8126000000000005e-05 2024-04-22T18:40:13.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:13.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.353031+0000", 2024-04-22T18:40:13.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119919 2024-04-22T18:40:13.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453622+0000", 2024-04-22T18:40:13.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100590793 2024-04-22T18:40:13.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558446+0000", 2024-04-22T18:40:13.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104824073 2024-04-22T18:40:13.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558479+0000", 2024-04-22T18:40:13.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3362999999999999e-05 2024-04-22T18:40:13.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558538+0000", 2024-04-22T18:40:13.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9046999999999999e-05 2024-04-22T18:40:13.543 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.543 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.543 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.543 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:10 3.2 3:518c37f9:::851eca84dafbf2f780c4f07291110c453349bdc9c623c17bdcd26e27badb30ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:13.543 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:13.543 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.307874465999999, 2024-04-22T18:40:13.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3852527729999999, 2024-04-22T18:40:13.543 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.543 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.544 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:13.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:13.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173411+0000", 2024-04-22T18:40:13.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:13.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173749+0000", 2024-04-22T18:40:13.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000338029 2024-04-22T18:40:13.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173755+0000", 2024-04-22T18:40:13.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0530000000000003e-06 2024-04-22T18:40:13.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173759+0000", 2024-04-22T18:40:13.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8190000000000002e-06 2024-04-22T18:40:13.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173855+0000", 2024-04-22T18:40:13.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5927999999999994e-05 2024-04-22T18:40:13.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:13.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173862+0000", 2024-04-22T18:40:13.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4449999999999999e-06 2024-04-22T18:40:13.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950835+0000", 2024-04-22T18:40:13.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77697248299999999 2024-04-22T18:40:13.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:13.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951632+0000", 2024-04-22T18:40:13.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079752299999999996 2024-04-22T18:40:13.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355407+0000", 2024-04-22T18:40:13.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40377480500000001 2024-04-22T18:40:13.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355462+0000", 2024-04-22T18:40:13.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4577999999999998e-05 2024-04-22T18:40:13.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:13.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355571+0000", 2024-04-22T18:40:13.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108964 2024-04-22T18:40:13.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453625+0000", 2024-04-22T18:40:13.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098054397000000001 2024-04-22T18:40:13.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558574+0000", 2024-04-22T18:40:13.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104948571 2024-04-22T18:40:13.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558602+0000", 2024-04-22T18:40:13.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7966999999999998e-05 2024-04-22T18:40:13.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558665+0000", 2024-04-22T18:40:13.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3032000000000002e-05 2024-04-22T18:40:13.550 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.550 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.550 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.550 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:11 3.2 3:718aa8aa:::9603c4f8e269a436b2f0bf376c5374e77bdd7cf58fccae60e4e65ac4696806d4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:13.550 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:13.550 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.307522848, 2024-04-22T18:40:13.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4852604440000001, 2024-04-22T18:40:13.551 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.551 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.551 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:13.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:13.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:13.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173813+0000", 2024-04-22T18:40:13.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0655999999999997e-05 2024-04-22T18:40:13.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173819+0000", 2024-04-22T18:40:13.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.468e-06 2024-04-22T18:40:13.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173822+0000", 2024-04-22T18:40:13.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4879999999999999e-06 2024-04-22T18:40:13.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173983+0000", 2024-04-22T18:40:13.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160519 2024-04-22T18:40:13.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:13.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173991+0000", 2024-04-22T18:40:13.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6490000000000008e-06 2024-04-22T18:40:13.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951688+0000", 2024-04-22T18:40:13.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77769743800000002 2024-04-22T18:40:13.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:13.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952419+0000", 2024-04-22T18:40:13.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073067499999999997 2024-04-22T18:40:13.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355901+0000", 2024-04-22T18:40:13.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40348237599999998 2024-04-22T18:40:13.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355953+0000", 2024-04-22T18:40:13.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1477000000000002e-05 2024-04-22T18:40:13.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:13.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356060+0000", 2024-04-22T18:40:13.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107114 2024-04-22T18:40:13.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453629+0000", 2024-04-22T18:40:13.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097569127000000005 2024-04-22T18:40:13.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658807+0000", 2024-04-22T18:40:13.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20517829800000001 2024-04-22T18:40:13.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658876+0000", 2024-04-22T18:40:13.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8454000000000006e-05 2024-04-22T18:40:13.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659024+0000", 2024-04-22T18:40:13.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148338 2024-04-22T18:40:13.557 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.557 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.557 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.557 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:12 3.2 3:475e1cd9:::efaab2bd13543cdbb143e869e20eb0ec424f7f5202707859b241ed91d4674936:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:13.557 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:13.557 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.307459904, 2024-04-22T18:40:13.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4853405259999999, 2024-04-22T18:40:13.558 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.558 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.558 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:13.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:13.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:13.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173940+0000", 2024-04-22T18:40:13.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011465500000000001 2024-04-22T18:40:13.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173946+0000", 2024-04-22T18:40:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5289999999999999e-06 2024-04-22T18:40:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173950+0000", 2024-04-22T18:40:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8290000000000001e-06 2024-04-22T18:40:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174277+0000", 2024-04-22T18:40:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032761299999999998 2024-04-22T18:40:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:13.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174285+0000", 2024-04-22T18:40:13.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.52e-06 2024-04-22T18:40:13.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952455+0000", 2024-04-22T18:40:13.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77817000700000005 2024-04-22T18:40:13.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:13.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954035+0000", 2024-04-22T18:40:13.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001580078 2024-04-22T18:40:13.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356379+0000", 2024-04-22T18:40:13.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40234440399999999 2024-04-22T18:40:13.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356428+0000", 2024-04-22T18:40:13.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8832999999999998e-05 2024-04-22T18:40:13.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:13.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356532+0000", 2024-04-22T18:40:13.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103235 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453632+0000", 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097100346000000004 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659069+0000", 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20543763800000001 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659105+0000", 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5240000000000001e-05 2024-04-22T18:40:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659167+0000", 2024-04-22T18:40:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2207999999999998e-05 2024-04-22T18:40:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:16 3.2 3:47c76cc8:::c00d18458eb20c0c0aa9c612cf9793a68445ab0ea7b9e650ecd0072c60a96c0f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.307331945, 2024-04-22T18:40:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.485347744, 2024-04-22T18:40:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174234+0000", 2024-04-22T18:40:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028047599999999999 2024-04-22T18:40:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174240+0000", 2024-04-22T18:40:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2079999999999997e-06 2024-04-22T18:40:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174244+0000", 2024-04-22T18:40:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5810000000000001e-06 2024-04-22T18:40:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174670+0000", 2024-04-22T18:40:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042640000000000001 2024-04-22T18:40:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174675+0000", 2024-04-22T18:40:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8709999999999996e-06 2024-04-22T18:40:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054168+0000", 2024-04-22T18:40:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87949257300000006 2024-04-22T18:40:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054877+0000", 2024-04-22T18:40:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070884300000000002 2024-04-22T18:40:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356836+0000", 2024-04-22T18:40:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30195926000000001 2024-04-22T18:40:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356887+0000", 2024-04-22T18:40:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0767999999999999e-05 2024-04-22T18:40:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356991+0000", 2024-04-22T18:40:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104072 2024-04-22T18:40:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453636+0000", 2024-04-22T18:40:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096644968999999997 2024-04-22T18:40:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659207+0000", 2024-04-22T18:40:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20557154699999999 2024-04-22T18:40:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659240+0000", 2024-04-22T18:40:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2665e-05 2024-04-22T18:40:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659302+0000", 2024-04-22T18:40:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2234000000000001e-05 2024-04-22T18:40:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:17 3.2 3:6ef31daa:::1cf4ff496c4480e58510f607481d976b47d84868e2c54016273c38baf590f3b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.307037664999999, 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4851703729999999, 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174248+0000", 2024-04-22T18:40:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174626+0000", 2024-04-22T18:40:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037839699999999999 2024-04-22T18:40:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174633+0000", 2024-04-22T18:40:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0990000000000004e-06 2024-04-22T18:40:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174636+0000", 2024-04-22T18:40:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5540000000000002e-06 2024-04-22T18:40:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174790+0000", 2024-04-22T18:40:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153807 2024-04-22T18:40:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174793+0000", 2024-04-22T18:40:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4970000000000001e-06 2024-04-22T18:40:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054902+0000", 2024-04-22T18:40:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88010886300000002 2024-04-22T18:40:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055366+0000", 2024-04-22T18:40:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046336199999999999 2024-04-22T18:40:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.452973+0000", 2024-04-22T18:40:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39760783999999999 2024-04-22T18:40:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453046+0000", 2024-04-22T18:40:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2374999999999997e-05 2024-04-22T18:40:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453193+0000", 2024-04-22T18:40:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014722499999999999 2024-04-22T18:40:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554698+0000", 2024-04-22T18:40:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101505096 2024-04-22T18:40:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659337+0000", 2024-04-22T18:40:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104639236 2024-04-22T18:40:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659365+0000", 2024-04-22T18:40:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7594e-05 2024-04-22T18:40:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659419+0000", 2024-04-22T18:40:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4058999999999999e-05 2024-04-22T18:40:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:18 3.2 3:792a4ed0:::e1c814961b3d1531fb5ed475b3d496a0405c3a819b68a92ff9b2391fe912370f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.306641408999999, 2024-04-22T18:40:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.585036954, 2024-04-22T18:40:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174640+0000", 2024-04-22T18:40:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-04-22T18:40:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174758+0000", 2024-04-22T18:40:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011819099999999999 2024-04-22T18:40:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174764+0000", 2024-04-22T18:40:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6729999999999997e-06 2024-04-22T18:40:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174767+0000", 2024-04-22T18:40:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3830000000000001e-06 2024-04-22T18:40:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373669+0000", 2024-04-22T18:40:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198901989 2024-04-22T18:40:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373686+0000", 2024-04-22T18:40:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7286e-05 2024-04-22T18:40:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055408+0000", 2024-04-22T18:40:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68172133400000001 2024-04-22T18:40:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055938+0000", 2024-04-22T18:40:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053016599999999999 2024-04-22T18:40:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453668+0000", 2024-04-22T18:40:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39773030799999998 2024-04-22T18:40:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453717+0000", 2024-04-22T18:40:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8884000000000002e-05 2024-04-22T18:40:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453823+0000", 2024-04-22T18:40:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105791 2024-04-22T18:40:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554715+0000", 2024-04-22T18:40:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100891628 2024-04-22T18:40:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759439+0000", 2024-04-22T18:40:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20472459400000001 2024-04-22T18:40:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759525+0000", 2024-04-22T18:40:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5653000000000001e-05 2024-04-22T18:40:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759682+0000", 2024-04-22T18:40:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015711800000000001 2024-04-22T18:40:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:158 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.301956+0000", 2024-04-22T18:40:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.1793305539999999, 2024-04-22T18:40:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30491982099999998, 2024-04-22T18:40:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.301956+0000", 2024-04-22T18:40:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.301956+0000", 2024-04-22T18:40:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.301953+0000", 2024-04-22T18:40:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302335+0000", 2024-04-22T18:40:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038256800000000001 2024-04-22T18:40:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302351+0000", 2024-04-22T18:40:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5313000000000001e-05 2024-04-22T18:40:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302360+0000", 2024-04-22T18:40:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:40:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302441+0000", 2024-04-22T18:40:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1242000000000006e-05 2024-04-22T18:40:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302851+0000", 2024-04-22T18:40:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041006400000000002 2024-04-22T18:40:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.505313+0000", 2024-04-22T18:40:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.202461317 2024-04-22T18:40:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.505387+0000", 2024-04-22T18:40:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4689000000000003e-05 2024-04-22T18:40:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.505549+0000", 2024-04-22T18:40:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016139799999999999 2024-04-22T18:40:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.605526+0000", 2024-04-22T18:40:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099977658999999997 2024-04-22T18:40:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606733+0000", 2024-04-22T18:40:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012065279999999999 2024-04-22T18:40:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606779+0000", 2024-04-22T18:40:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6516e-05 2024-04-22T18:40:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606876+0000", 2024-04-22T18:40:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6181999999999995e-05 2024-04-22T18:40:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:171 3.2 3:723389a0:::8853427863ea44e2e1f1b69134450b6d67dacc49ab40862e8e5106679d802392:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.303144+0000", 2024-04-22T18:40:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.1781425609999996, 2024-04-22T18:40:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30331065200000001, 2024-04-22T18:40:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303144+0000", 2024-04-22T18:40:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303144+0000", 2024-04-22T18:40:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303143+0000", 2024-04-22T18:40:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303401+0000", 2024-04-22T18:40:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000257581 2024-04-22T18:40:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303408+0000", 2024-04-22T18:40:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7730000000000002e-06 2024-04-22T18:40:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303413+0000", 2024-04-22T18:40:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6559999999999997e-06 2024-04-22T18:40:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.307294+0000", 2024-04-22T18:40:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038801370000000001 2024-04-22T18:40:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.307501+0000", 2024-04-22T18:40:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020771200000000001 2024-04-22T18:40:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.307795+0000", 2024-04-22T18:40:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029333000000000001 2024-04-22T18:40:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404962+0000", 2024-04-22T18:40:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097167373000000001 2024-04-22T18:40:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606305+0000", 2024-04-22T18:40:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20134328500000001 2024-04-22T18:40:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606352+0000", 2024-04-22T18:40:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6774000000000003e-05 2024-04-22T18:40:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606454+0000", 2024-04-22T18:40:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102413 2024-04-22T18:40:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:174 3.2 3:594d3ad2:::b481f33bf204d13d836f094b25e4b106d427bebe44d81ba5fbb696a319414e75:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.303419+0000", 2024-04-22T18:40:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.1778674039999997, 2024-04-22T18:40:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.303235174, 2024-04-22T18:40:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303419+0000", 2024-04-22T18:40:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303419+0000", 2024-04-22T18:40:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303418+0000", 2024-04-22T18:40:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303527+0000", 2024-04-22T18:40:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108925 2024-04-22T18:40:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303535+0000", 2024-04-22T18:40:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2930000000000001e-06 2024-04-22T18:40:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303544+0000", 2024-04-22T18:40:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5159999999999994e-06 2024-04-22T18:40:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404063+0000", 2024-04-22T18:40:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100518801 2024-04-22T18:40:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404178+0000", 2024-04-22T18:40:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114764 2024-04-22T18:40:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404352+0000", 2024-04-22T18:40:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017431200000000001 2024-04-22T18:40:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.405276+0000", 2024-04-22T18:40:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00092411499999999998 2024-04-22T18:40:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606498+0000", 2024-04-22T18:40:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20122152200000001 2024-04-22T18:40:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606545+0000", 2024-04-22T18:40:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7206000000000002e-05 2024-04-22T18:40:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606654+0000", 2024-04-22T18:40:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109379 2024-04-22T18:40:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:177 3.2 3:6231f5c4:::2915c267dc4ac87f18561ed5fed8de54cbaef6bdd9b80bf698b516d3f8bdd3ee:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e78)", 2024-04-22T18:40:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.334408+0000", 2024-04-22T18:40:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.146878557, 2024-04-22T18:40:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30399278099999999, 2024-04-22T18:40:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334408+0000", 2024-04-22T18:40:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334408+0000", 2024-04-22T18:40:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334404+0000", 2024-04-22T18:40:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999962 2024-04-22T18:40:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334583+0000", 2024-04-22T18:40:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000178655 2024-04-22T18:40:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334603+0000", 2024-04-22T18:40:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0251e-05 2024-04-22T18:40:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.434698+0000", 2024-04-22T18:40:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100094581 2024-04-22T18:40:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.434779+0000", 2024-04-22T18:40:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1528999999999998e-05 2024-04-22T18:40:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.434949+0000", 2024-04-22T18:40:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000170113 2024-04-22T18:40:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.435216+0000", 2024-04-22T18:40:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026647900000000002 2024-04-22T18:40:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.437050+0000", 2024-04-22T18:40:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018344450000000001 2024-04-22T18:40:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.638245+0000", 2024-04-22T18:40:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20119524499999999 2024-04-22T18:40:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.638295+0000", 2024-04-22T18:40:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9994000000000003e-05 2024-04-22T18:40:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.638401+0000", 2024-04-22T18:40:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105145 2024-04-22T18:40:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:13.608 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:13.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:13.611 INFO:tasks.rados.rados.0.smithi089.stdout:1342: finishing set_chunk to oid 22 2024-04-22T18:40:13.611 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 90 (ObjNum 239 snap 0 seq_num 239) dirty exists 2024-04-22T18:40:13.611 INFO:tasks.rados.rados.0.smithi089.stdout:1348: seq_num 255 ranges {685785=628144,1822320=463465} 2024-04-22T18:40:13.611 INFO:tasks.rados.rados.0.smithi089.stdout:1348: writing smithi08937929-16 from 685785 to 1313929 tid 1 2024-04-22T18:40:13.611 INFO:tasks.rados.rados.0.smithi089.stdout:1348: writing smithi08937929-16 from 1822320 to 2285785 tid 2 2024-04-22T18:40:13.611 INFO:tasks.rados.rados.0.smithi089.stdout:1337: done (9 left) 2024-04-22T18:40:13.611 INFO:tasks.rados.rados.0.smithi089.stdout:1338: done (8 left) 2024-04-22T18:40:13.611 INFO:tasks.rados.rados.0.smithi089.stdout:1341: done (7 left) 2024-04-22T18:40:13.612 INFO:tasks.rados.rados.0.smithi089.stdout:1342: done (6 left) 2024-04-22T18:40:13.612 INFO:tasks.rados.rados.0.smithi089.stdout:1349: delete oid 37 current snap is 0 2024-04-22T18:40:13.612 INFO:tasks.rados.rados.0.smithi089.stdout:1344: expect (ObjNum 245 snap 0 seq_num 245) 2024-04-22T18:40:13.612 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:13.610+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:13.676 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 6, 7, 2] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:40:13.676 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:40:13.676 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-04-22T18:40:13.676 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-04-22T18:40:13.676 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:40:13.697 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:40:13.710 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:13.712+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:13.718 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:13.718 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:13.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:13.748 INFO:tasks.rados.rados.0.smithi089.stdout:1345: expect (ObjNum 243 snap 0 seq_num 243) 2024-04-22T18:40:13.811 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:13.812+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:13.869 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:13.869 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:13.870 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:13.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:13.879 INFO:tasks.rados.rados.0.smithi089.stdout:1347: expect (ObjNum 244 snap 0 seq_num 244) 2024-04-22T18:40:13.922 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:13.924+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:13.961 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:40:13.961 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:40:13.972 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-04-22T18:40:13.696383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":239287436,"num_objects":181,"num_object_clones":0,"num_object_copies":362,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":2950,"num_read_kb":2470632,"num_write":1903,"num_write_kb":459958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":79825161,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":30,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2408,"ondisk_log_size":2408,"up":78,"acting":78,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":80,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":514128,"kb_used_data":292216,"kb_used_omap":10,"kb_used_meta":209525,"kb_avail":660088752,"statfs":{"total":676457349120,"available":675930882048,"internally_reserved":0,"allocated":299229184,"data_stored":303735267,"data_compressed":7433532,"data_compressed_allocated":7929856,"data_compressed_original":17157318,"omap_allocated":11150,"internal_metadata":214553714},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,1,0,1,6,4,7],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6271330,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":1,"num_objects_misplaced":-9,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":143,"num_read_kb":132639,"num_write":161,"num_write_kb":26955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":7371630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.024402"},"pg_stats":[{"pgid":"4.19","version":"76'87","reported_seq":305,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:10.001341+0000","last_change":"2024-04-22T18:39:57.558463+0000","last_active":"2024-04-22T18:40:10.001341+0000","last_peered":"2024-04-22T18:40:10.001341+0000","last_clean":"2024-04-22T18:40:10.001341+0000","last_became_active":"2024-04-22T18:39:47.010533+0000","last_became_peered":"2024-04-22T18:39:47.010533+0000","last_unstale":"2024-04-22T18:40:10.001341+0000","last_undegraded":"2024-04-22T18:40:10.001341+0000","last_fullsized":"2024-04-22T18:40:10.001341+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'83","last_scrub_stamp":"2024-04-22T18:39:57.558302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.558302+0000","objects_scrubbed":2,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:14.300038+0000","scrub_duration":0.31429868900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6638765,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"77'56","reported_seq":209,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T18:40:08.660134+0000","last_change":"2024-04-22T18:40:06.800906+0000","last_active":"2024-04-22T18:40:08.660134+0000","last_peered":"2024-04-22T18:40:08.660134+0000","last_clean":"2024-04-22T18:40:08.660134+0000","last_became_active":"2024-04-22T18:40:02.645235+0000","last_became_peered":"2024-04-22T18:40:02.645235+0000","last_unstale":"2024-04-22T18:40:08.660134+0000","last_undegraded":"2024-04-22T18:40:08.660134+0000","last_fullsized":"2024-04-22T18:40:08.660134+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2058858,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":100483,"num_write":49,"num_write_kb":11981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2528838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"52'33","reported_seq":158,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.674462+0000","last_change":"2024-04-22T18:40:02.646106+0000","last_active":"2024-04-22T18:40:09.674462+0000","last_peered":"2024-04-22T18:40:09.674462+0000","last_clean":"2024-04-22T18:40:09.674462+0000","last_became_active":"2024-04-22T18:40:02.645801+0000","last_became_peered":"2024-04-22T18:40:02.645801+0000","last_unstale":"2024-04-22T18:40:09.674462+0000","last_undegraded":"2024-04-22T18:40:09.674462+0000","last_fullsized":"2024-04-22T18:40:09.674462+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042051,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":49,"num_read_kb":52611,"num_write":15,"num_write_kb":3979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"78'67","reported_seq":214,"reported_epoch":79,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T18:40:10.590106+0000","last_change":"2024-04-22T18:40:10.366148+0000","last_active":"2024-04-22T18:40:10.590106+0000","last_peered":"2024-04-22T18:40:10.590106+0000","last_clean":"2024-04-22T18:40:10.590106+0000","last_became_active":"2024-04-22T18:39:47.104695+0000","last_became_peered":"2024-04-22T18:39:47.104695+0000","last_unstale":"2024-04-22T18:40:10.590106+0000","last_undegraded":"2024-04-22T18:40:10.590106+0000","last_fullsized":"2024-04-22T18:40:10.590106+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2453855,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":85787,"num_write":42,"num_write_kb":11971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"54'29","reported_seq":156,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.675112+0000","last_change":"2024-04-22T18:39:56.614792+0000","last_active":"2024-04-22T18:40:09.675112+0000","last_peered":"2024-04-22T18:40:09.675112+0000","last_clean":"2024-04-22T18:40:09.675112+0000","last_became_active":"2024-04-22T18:39:47.007675+0000","last_became_peered":"2024-04-22T18:39:47.007675+0000","last_unstale":"2024-04-22T18:40:09.675112+0000","last_undegraded":"2024-04-22T18:40:09.675112+0000","last_fullsized":"2024-04-22T18:40:09.675112+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'29","last_scrub_stamp":"2024-04-22T18:39:56.614637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:56.614637+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:22.964555+0000","scrub_duration":0.37681601799999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"51'19","reported_seq":10,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-22T18:40:11.079649+0000","last_change":"2024-04-22T18:40:11.079224+0000","last_active":"2024-04-22T18:40:11.079649+0000","last_peered":"2024-04-22T18:40:11.079649+0000","last_clean":"2024-04-22T18:40:11.079649+0000","last_became_active":"2024-04-22T18:40:10.978840+0000","last_became_peered":"2024-04-22T18:40:10.978840+0000","last_unstale":"2024-04-22T18:40:11.079649+0000","last_undegraded":"2024-04-22T18:40:11.079649+0000","last_fullsized":"2024-04-22T18:40:11.079649+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1882368,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":22802,"num_write":10,"num_write_kb":3047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1349842,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"79'75","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-22T18:40:11.267815+0000","last_change":"2024-04-22T18:40:10.277811+0000","last_active":"2024-04-22T18:40:11.267815+0000","last_peered":"2024-04-22T18:40:11.267815+0000","last_clean":"2024-04-22T18:40:11.267815+0000","last_became_active":"2024-04-22T18:40:09.027450+0000","last_became_peered":"2024-04-22T18:40:09.027450+0000","last_unstale":"2024-04-22T18:40:11.267815+0000","last_undegraded":"2024-04-22T18:40:11.267815+0000","last_fullsized":"2024-04-22T18:40:11.267815+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6605151,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":118941,"num_write":43,"num_write_kb":12157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7825098,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"58'33","reported_seq":162,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.674185+0000","last_change":"2024-04-22T18:40:03.451290+0000","last_active":"2024-04-22T18:40:09.674185+0000","last_peered":"2024-04-22T18:40:09.674185+0000","last_clean":"2024-04-22T18:40:09.674185+0000","last_became_active":"2024-04-22T18:40:02.844371+0000","last_became_peered":"2024-04-22T18:40:02.844371+0000","last_unstale":"2024-04-22T18:40:09.674185+0000","last_undegraded":"2024-04-22T18:40:09.674185+0000","last_fullsized":"2024-04-22T18:40:09.674185+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2445419,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":67666,"num_write":16,"num_write_kb":4115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":970119,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"73'113","reported_seq":323,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.776615+0000","last_change":"2024-04-22T18:40:04.645761+0000","last_active":"2024-04-22T18:40:09.776615+0000","last_peered":"2024-04-22T18:40:09.776615+0000","last_clean":"2024-04-22T18:40:09.776615+0000","last_became_active":"2024-04-22T18:40:04.645418+0000","last_became_peered":"2024-04-22T18:40:04.645418+0000","last_unstale":"2024-04-22T18:40:09.776615+0000","last_undegraded":"2024-04-22T18:40:09.776615+0000","last_fullsized":"2024-04-22T18:40:09.776615+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10102482,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":155,"num_read_kb":162433,"num_write":62,"num_write_kb":17851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1101926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"31'5","reported_seq":114,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:10.212897+0000","last_change":"2024-04-22T18:40:10.212897+0000","last_active":"2024-04-22T18:40:10.212897+0000","last_peered":"2024-04-22T18:40:10.212897+0000","last_clean":"2024-04-22T18:40:10.212897+0000","last_became_active":"2024-04-22T18:39:59.265189+0000","last_became_peered":"2024-04-22T18:39:59.265189+0000","last_unstale":"2024-04-22T18:40:10.212897+0000","last_undegraded":"2024-04-22T18:40:10.212897+0000","last_fullsized":"2024-04-22T18:40:10.212897+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":5,"last_scrub":"31'5","last_scrub_stamp":"2024-04-22T18:40:10.212815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.212815+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:17.620115+0000","scrub_duration":0.0015068670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":72,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.675167+0000","last_change":"2024-04-22T18:38:39.026078+0000","last_active":"2024-04-22T18:40:09.675167+0000","last_peered":"2024-04-22T18:40:09.675167+0000","last_clean":"2024-04-22T18:40:09.675167+0000","last_became_active":"2024-04-22T18:38:39.025741+0000","last_became_peered":"2024-04-22T18:38:39.025741+0000","last_unstale":"2024-04-22T18:40:09.675167+0000","last_undegraded":"2024-04-22T18:40:09.675167+0000","last_fullsized":"2024-04-22T18:40:09.675167+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"79'127","reported_seq":382,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:10.578299+0000","last_change":"2024-04-22T18:40:04.048901+0000","last_active":"2024-04-22T18:40:10.578299+0000","last_peered":"2024-04-22T18:40:10.578299+0000","last_clean":"2024-04-22T18:40:10.578299+0000","last_became_active":"2024-04-22T18:40:02.858745+0000","last_became_peered":"2024-04-22T18:40:02.858745+0000","last_unstale":"2024-04-22T18:40:10.578299+0000","last_undegraded":"2024-04-22T18:40:10.578299+0000","last_fullsized":"2024-04-22T18:40:10.578299+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:07.943249+0000","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13748192,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":86,"num_read_kb":106236,"num_write":19,"num_write_kb":5368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9086896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":72,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.874555+0000","last_change":"2024-04-22T18:38:39.110499+0000","last_active":"2024-04-22T18:40:09.874555+0000","last_peered":"2024-04-22T18:40:09.874555+0000","last_clean":"2024-04-22T18:40:09.874555+0000","last_became_active":"2024-04-22T18:38:39.010173+0000","last_became_peered":"2024-04-22T18:38:39.010173+0000","last_unstale":"2024-04-22T18:40:09.874555+0000","last_undegraded":"2024-04-22T18:40:09.874555+0000","last_fullsized":"2024-04-22T18:40:09.874555+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"71'14","reported_seq":124,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.776579+0000","last_change":"2024-04-22T18:39:54.378188+0000","last_active":"2024-04-22T18:40:09.776579+0000","last_peered":"2024-04-22T18:40:09.776579+0000","last_clean":"2024-04-22T18:40:09.776579+0000","last_became_active":"2024-04-22T18:39:46.805569+0000","last_became_peered":"2024-04-22T18:39:46.805569+0000","last_unstale":"2024-04-22T18:40:09.776579+0000","last_undegraded":"2024-04-22T18:40:09.776579+0000","last_fullsized":"2024-04-22T18:40:09.776579+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'13","last_scrub_stamp":"2024-04-22T18:39:54.377980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.377980+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:00.921987+0000","scrub_duration":0.30260881099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3378207,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":35426,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":80,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.538708+0000","last_change":"2024-04-22T18:40:09.538708+0000","last_active":"2024-04-22T18:40:09.538708+0000","last_peered":"2024-04-22T18:40:09.538708+0000","last_clean":"2024-04-22T18:40:09.538708+0000","last_became_active":"2024-04-22T18:38:39.210049+0000","last_became_peered":"2024-04-22T18:38:39.210049+0000","last_unstale":"2024-04-22T18:40:09.538708+0000","last_undegraded":"2024-04-22T18:40:09.538708+0000","last_fullsized":"2024-04-22T18:40:09.538708+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:09.538542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:09.538542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:31.805660+0000","scrub_duration":0.35201753200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"78'70","reported_seq":241,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.941392+0000","last_change":"2024-04-22T18:40:03.763662+0000","last_active":"2024-04-22T18:40:09.941392+0000","last_peered":"2024-04-22T18:40:09.941392+0000","last_clean":"2024-04-22T18:40:09.941392+0000","last_became_active":"2024-04-22T18:40:02.739213+0000","last_became_peered":"2024-04-22T18:40:02.739213+0000","last_unstale":"2024-04-22T18:40:09.941392+0000","last_undegraded":"2024-04-22T18:40:09.941392+0000","last_fullsized":"2024-04-22T18:40:09.941392+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'67","last_scrub_stamp":"2024-04-22T18:39:53.916646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.916646+0000","objects_scrubbed":2,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:06.252157+0000","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269011,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":16271,"num_write":1,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3912084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":72,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.674349+0000","last_change":"2024-04-22T18:38:39.307735+0000","last_active":"2024-04-22T18:40:09.674349+0000","last_peered":"2024-04-22T18:40:09.674349+0000","last_clean":"2024-04-22T18:40:09.674349+0000","last_became_active":"2024-04-22T18:38:39.307459+0000","last_became_peered":"2024-04-22T18:38:39.307459+0000","last_unstale":"2024-04-22T18:40:09.674349+0000","last_undegraded":"2024-04-22T18:40:09.674349+0000","last_fullsized":"2024-04-22T18:40:09.674349+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"64'38","reported_seq":169,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.674336+0000","last_change":"2024-04-22T18:40:04.057863+0000","last_active":"2024-04-22T18:40:09.674336+0000","last_peered":"2024-04-22T18:40:09.674336+0000","last_clean":"2024-04-22T18:40:09.674336+0000","last_became_active":"2024-04-22T18:40:02.844289+0000","last_became_peered":"2024-04-22T18:40:02.844289+0000","last_unstale":"2024-04-22T18:40:09.674336+0000","last_undegraded":"2024-04-22T18:40:09.674336+0000","last_fullsized":"2024-04-22T18:40:09.674336+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2932821,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":107389,"num_write":20,"num_write_kb":5595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2843709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"80'247","reported_seq":324,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-22T18:40:10.847169+0000","last_change":"2024-04-22T18:38:47.173101+0000","last_active":"2024-04-22T18:40:10.847169+0000","last_peered":"2024-04-22T18:40:10.847169+0000","last_clean":"2024-04-22T18:40:10.847169+0000","last_became_active":"2024-04-22T18:38:47.172816+0000","last_became_peered":"2024-04-22T18:38:47.172816+0000","last_unstale":"2024-04-22T18:40:10.847169+0000","last_undegraded":"2024-04-22T18:40:10.847169+0000","last_fullsized":"2024-04-22T18:40:10.847169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:12.128375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25409973,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":212,"num_read_kb":3202,"num_write":297,"num_write_kb":78010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":79,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T18:40:10.483840+0000","last_change":"2024-04-22T18:40:10.183115+0000","last_active":"2024-04-22T18:40:10.483840+0000","last_peered":"2024-04-22T18:40:10.483840+0000","last_clean":"2024-04-22T18:40:10.483840+0000","last_became_active":"2024-04-22T18:40:02.945025+0000","last_became_peered":"2024-04-22T18:40:02.945025+0000","last_unstale":"2024-04-22T18:40:10.483840+0000","last_undegraded":"2024-04-22T18:40:10.483840+0000","last_fullsized":"2024-04-22T18:40:10.483840+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":154,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.674285+0000","last_change":"2024-04-22T18:39:52.361604+0000","last_active":"2024-04-22T18:40:09.674285+0000","last_peered":"2024-04-22T18:40:09.674285+0000","last_clean":"2024-04-22T18:40:09.674285+0000","last_became_active":"2024-04-22T18:39:51.943071+0000","last_became_peered":"2024-04-22T18:39:51.943071+0000","last_unstale":"2024-04-22T18:40:09.674285+0000","last_undegraded":"2024-04-22T18:40:09.674285+0000","last_fullsized":"2024-04-22T18:40:09.674285+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1949628,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":55801,"num_write":19,"num_write_kb":6188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":3820686,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"79'168","reported_seq":449,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-22T18:40:10.745588+0000","last_change":"2024-04-22T18:39:56.824261+0000","last_active":"2024-04-22T18:40:10.745588+0000","last_peered":"2024-04-22T18:40:10.745588+0000","last_clean":"2024-04-22T18:40:10.745588+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:40:10.745588+0000","last_undegraded":"2024-04-22T18:40:10.745588+0000","last_fullsized":"2024-04-22T18:40:10.745588+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23924416,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":139,"num_read_kb":1003,"num_write":209,"num_write_kb":52707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":39,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:10.403338+0000","last_change":"2024-04-22T18:40:10.403338+0000","last_active":"2024-04-22T18:40:10.403338+0000","last_peered":"2024-04-22T18:40:10.403338+0000","last_clean":"2024-04-22T18:40:10.403338+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:40:10.403338+0000","last_undegraded":"2024-04-22T18:40:10.403338+0000","last_fullsized":"2024-04-22T18:40:10.403338+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:10.403189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.403189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:34.232014+0000","scrub_duration":0.201433005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"78'45","reported_seq":202,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.774627+0000","last_change":"2024-04-22T18:40:03.361557+0000","last_active":"2024-04-22T18:40:09.774627+0000","last_peered":"2024-04-22T18:40:09.774627+0000","last_clean":"2024-04-22T18:40:09.774627+0000","last_became_active":"2024-04-22T18:40:02.744089+0000","last_became_peered":"2024-04-22T18:40:02.744089+0000","last_unstale":"2024-04-22T18:40:09.774627+0000","last_undegraded":"2024-04-22T18:40:09.774627+0000","last_fullsized":"2024-04-22T18:40:09.774627+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:16.501156+0000","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2100908,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":33874,"num_write":8,"num_write_kb":2049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2812934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"80'198","reported_seq":279,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-22T18:40:10.774834+0000","last_change":"2024-04-22T18:38:47.173337+0000","last_active":"2024-04-22T18:40:10.774834+0000","last_peered":"2024-04-22T18:40:10.774834+0000","last_clean":"2024-04-22T18:40:10.774834+0000","last_became_active":"2024-04-22T18:38:47.173065+0000","last_became_peered":"2024-04-22T18:38:47.173065+0000","last_unstale":"2024-04-22T18:40:10.774834+0000","last_undegraded":"2024-04-22T18:40:10.774834+0000","last_fullsized":"2024-04-22T18:40:10.774834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22319376,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":177,"num_read_kb":5112,"num_write":243,"num_write_kb":60299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":69,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T18:40:08.660149+0000","last_change":"2024-04-22T18:38:39.125885+0000","last_active":"2024-04-22T18:40:08.660149+0000","last_peered":"2024-04-22T18:40:08.660149+0000","last_clean":"2024-04-22T18:40:08.660149+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:40:08.660149+0000","last_undegraded":"2024-04-22T18:40:08.660149+0000","last_fullsized":"2024-04-22T18:40:08.660149+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"77'66","reported_seq":234,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:10.491476+0000","last_change":"2024-04-22T18:40:08.763711+0000","last_active":"2024-04-22T18:40:10.491476+0000","last_peered":"2024-04-22T18:40:10.491476+0000","last_clean":"2024-04-22T18:40:10.491476+0000","last_became_active":"2024-04-22T18:39:46.809886+0000","last_became_peered":"2024-04-22T18:39:46.809886+0000","last_unstale":"2024-04-22T18:40:10.491476+0000","last_undegraded":"2024-04-22T18:40:10.491476+0000","last_fullsized":"2024-04-22T18:40:10.491476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":2,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:37.433066+0000","scrub_duration":0.22461577699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6302625,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":15426,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"78'196","reported_seq":440,"reported_epoch":80,"state":"active+recovering+degraded","last_fresh":"2024-04-22T18:40:11.085986+0000","last_change":"2024-04-22T18:40:11.085600+0000","last_active":"2024-04-22T18:40:11.085986+0000","last_peered":"2024-04-22T18:40:11.085986+0000","last_clean":"2024-04-22T18:40:09.737634+0000","last_became_active":"2024-04-22T18:40:10.784978+0000","last_became_peered":"2024-04-22T18:40:10.784978+0000","last_unstale":"2024-04-22T18:40:11.085986+0000","last_undegraded":"2024-04-22T18:40:10.677896+0000","last_fullsized":"2024-04-22T18:40:11.085986+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:13.768039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23257629,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":183,"num_read_kb":4078,"num_write":250,"num_write_kb":54597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":13295755,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3","objects":1},{"shards":"3,6","objects":38}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":94,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.842878+0000","last_change":"2024-04-22T18:40:07.420765+0000","last_active":"2024-04-22T18:40:09.842878+0000","last_peered":"2024-04-22T18:40:09.842878+0000","last_clean":"2024-04-22T18:40:09.842878+0000","last_became_active":"2024-04-22T18:40:02.741862+0000","last_became_peered":"2024-04-22T18:40:02.741862+0000","last_unstale":"2024-04-22T18:40:09.842878+0000","last_undegraded":"2024-04-22T18:40:09.842878+0000","last_fullsized":"2024-04-22T18:40:09.842878+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:07.420618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:07.420618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.642486+0000","scrub_duration":0.30164867299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":231,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-22T18:40:08.660195+0000","last_change":"2024-04-22T18:39:53.346906+0000","last_active":"2024-04-22T18:40:08.660195+0000","last_peered":"2024-04-22T18:40:08.660195+0000","last_clean":"2024-04-22T18:40:08.660195+0000","last_became_active":"2024-04-22T18:38:38.108568+0000","last_became_peered":"2024-04-22T18:38:38.108568+0000","last_unstale":"2024-04-22T18:40:08.660195+0000","last_undegraded":"2024-04-22T18:40:08.660195+0000","last_fullsized":"2024-04-22T18:40:08.660195+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:39:53.346787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.346787+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:17.891739+0000","scrub_duration":0.20273052499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"77'56","reported_seq":194,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.675131+0000","last_change":"2024-04-22T18:39:55.746188+0000","last_active":"2024-04-22T18:40:09.675131+0000","last_peered":"2024-04-22T18:40:09.675131+0000","last_clean":"2024-04-22T18:40:09.675131+0000","last_became_active":"2024-04-22T18:39:47.108040+0000","last_became_peered":"2024-04-22T18:39:47.108040+0000","last_unstale":"2024-04-22T18:40:09.675131+0000","last_undegraded":"2024-04-22T18:40:09.675131+0000","last_fullsized":"2024-04-22T18:40:09.675131+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'38","last_scrub_stamp":"2024-04-22T18:39:55.746067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:55.746067+0000","objects_scrubbed":1,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:04.875560+0000","scrub_duration":0.60877772299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7613571,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":17966,"num_write":21,"num_write_kb":6447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"79'52","reported_seq":186,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.746379+0000","last_change":"2024-04-22T18:40:06.087720+0000","last_active":"2024-04-22T18:40:09.746379+0000","last_peered":"2024-04-22T18:40:09.746379+0000","last_clean":"2024-04-22T18:40:09.746379+0000","last_became_active":"2024-04-22T18:40:02.645205+0000","last_became_peered":"2024-04-22T18:40:02.645205+0000","last_unstale":"2024-04-22T18:40:09.746379+0000","last_undegraded":"2024-04-22T18:40:09.746379+0000","last_fullsized":"2024-04-22T18:40:09.746379+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1436998,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":92005,"num_write":30,"num_write_kb":8835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2522610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"79'94","reported_seq":315,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:10.589764+0000","last_change":"2024-04-22T18:40:05.576934+0000","last_active":"2024-04-22T18:40:10.589764+0000","last_peered":"2024-04-22T18:40:10.589764+0000","last_clean":"2024-04-22T18:40:10.589764+0000","last_became_active":"2024-04-22T18:39:46.807956+0000","last_became_peered":"2024-04-22T18:39:46.807956+0000","last_unstale":"2024-04-22T18:40:10.589764+0000","last_undegraded":"2024-04-22T18:40:10.589764+0000","last_fullsized":"2024-04-22T18:40:10.589764+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:11.922250+0000","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8386726,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":4,"num_write_kb":1034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"76'63","reported_seq":215,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:10.392168+0000","last_change":"2024-04-22T18:38:48.372952+0000","last_active":"2024-04-22T18:40:10.392168+0000","last_peered":"2024-04-22T18:40:10.392168+0000","last_clean":"2024-04-22T18:40:10.392168+0000","last_became_active":"2024-04-22T18:38:48.277922+0000","last_became_peered":"2024-04-22T18:38:48.277922+0000","last_unstale":"2024-04-22T18:40:10.392168+0000","last_undegraded":"2024-04-22T18:40:10.392168+0000","last_fullsized":"2024-04-22T18:40:10.392168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:15.471951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11243883,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":203,"num_read_kb":248131,"num_write":62,"num_write_kb":16900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"80'96","reported_seq":303,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-22T18:40:11.087782+0000","last_change":"2024-04-22T18:38:48.269362+0000","last_active":"2024-04-22T18:40:11.087782+0000","last_peered":"2024-04-22T18:40:11.087782+0000","last_clean":"2024-04-22T18:40:11.087782+0000","last_became_active":"2024-04-22T18:38:48.269066+0000","last_became_peered":"2024-04-22T18:38:48.269066+0000","last_unstale":"2024-04-22T18:40:11.087782+0000","last_undegraded":"2024-04-22T18:40:11.087782+0000","last_fullsized":"2024-04-22T18:40:11.087782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10453954,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":321,"num_read_kb":350414,"num_write":95,"num_write_kb":28108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"78'93","reported_seq":294,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:10.075957+0000","last_change":"2024-04-22T18:38:48.274538+0000","last_active":"2024-04-22T18:40:10.075957+0000","last_peered":"2024-04-22T18:40:10.075957+0000","last_clean":"2024-04-22T18:40:10.075957+0000","last_became_active":"2024-04-22T18:38:48.174204+0000","last_became_peered":"2024-04-22T18:38:48.174204+0000","last_unstale":"2024-04-22T18:40:10.075957+0000","last_undegraded":"2024-04-22T18:40:10.075957+0000","last_fullsized":"2024-04-22T18:40:10.075957+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:40:11.823691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11563282,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":337,"num_read_kb":398016,"num_write":96,"num_write_kb":27211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"78'43","reported_seq":194,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:09.743319+0000","last_change":"2024-04-22T18:39:52.257129+0000","last_active":"2024-04-22T18:40:09.743319+0000","last_peered":"2024-04-22T18:40:09.743319+0000","last_clean":"2024-04-22T18:40:09.743319+0000","last_became_active":"2024-04-22T18:39:51.934575+0000","last_became_peered":"2024-04-22T18:39:51.934575+0000","last_unstale":"2024-04-22T18:40:09.743319+0000","last_undegraded":"2024-04-22T18:40:09.743319+0000","last_fullsized":"2024-04-22T18:40:09.743319+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5697606,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":134,"num_read_kb":135578,"num_write":45,"num_write_kb":13377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":6895340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"78'19","reported_seq":119,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:10.674998+0000","last_change":"2024-04-22T18:38:48.075222+0000","last_active":"2024-04-22T18:40:10.674998+0000","last_peered":"2024-04-22T18:40:10.674998+0000","last_clean":"2024-04-22T18:40:10.674998+0000","last_became_active":"2024-04-22T18:38:48.074901+0000","last_became_peered":"2024-04-22T18:38:48.074901+0000","last_unstale":"2024-04-22T18:40:10.674998+0000","last_undegraded":"2024-04-22T18:40:10.674998+0000","last_fullsized":"2024-04-22T18:40:10.674998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832062,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":93441,"num_write":20,"num_write_kb":6518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"78'61","reported_seq":192,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-22T18:40:10.489542+0000","last_change":"2024-04-22T18:38:48.173640+0000","last_active":"2024-04-22T18:40:10.489542+0000","last_peered":"2024-04-22T18:40:10.489542+0000","last_clean":"2024-04-22T18:40:10.489542+0000","last_became_active":"2024-04-22T18:38:48.173369+0000","last_became_peered":"2024-04-22T18:38:48.173369+0000","last_unstale":"2024-04-22T18:40:10.489542+0000","last_undegraded":"2024-04-22T18:40:10.489542+0000","last_fullsized":"2024-04-22T18:40:10.489542+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9647251,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":147,"num_read_kb":140437,"num_write":63,"num_write_kb":17122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":143785674,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2123,"num_read_kb":2457137,"num_write":745,"num_write_kb":211929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":45670082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":30,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208154624,"data_stored":207594606,"data_compressed":567111,"data_compressed_allocated":598016,"data_compressed_original":1096902,"omap_allocated":0,"internal_metadata":0},"log_size":1482,"ondisk_log_size":1482,"up":52,"acting":52,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":94911394,"num_objects":132,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":711,"num_read_kb":13395,"num_write":999,"num_write_kb":245613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":34155079,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":111878144,"data_stored":119268095,"data_compressed":6860942,"data_compressed_allocated":7307264,"data_compressed_original":15097856,"omap_allocated":0,"internal_metadata":0},"log_size":809,"ondisk_log_size":809,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51292,"kb_used_data":20176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94320548,"statfs":{"total":96636764160,"available":96584241152,"internally_reserved":0,"allocated":20660224,"data_stored":20641542,"data_compressed":189598,"data_compressed_allocated":208896,"data_compressed_original":774144,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":4.508,"5min":4.508,"15min":4.508},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":85.542000000000002,"5min":85.542000000000002,"15min":85.542000000000002},"last":0.36099999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.8650000000000002,"5min":4.8650000000000002,"15min":4.8650000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":79.927999999999997,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":13.048,"5min":13.048,"15min":13.048},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":99.388999999999996,"5min":99.388999999999996,"15min":99.388999999999996},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.085,"5min":4.085,"15min":4.085},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":77.320999999999998,"5min":77.320999999999998,"15min":77.320999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.59599999999999997}]},{"osd":3,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.1509999999999998,"5min":4.1509999999999998,"15min":4.1509999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":77.361000000000004,"5min":77.361000000000004,"15min":77.361000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.54300000000000004}]},{"osd":4,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.4589999999999996,"5min":4.4589999999999996,"15min":4.4589999999999996},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":85.509,"5min":85.509,"15min":85.509},"last":0.49299999999999999},{"interface":"front","average":{"1min":22.872,"5min":22.872,"15min":22.872},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":327.77300000000002,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.308}]},{"osd":5,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":3.0710000000000002,"5min":3.0710000000000002,"15min":3.0710000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":55.116999999999997,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.39500000000000002},{"interface":"front","average":{"1min":7.1619999999999999,"5min":7.1619999999999999,"15min":7.1619999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":85.475999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.53100000000000003}]},{"osd":6,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":13.276,"5min":13.276,"15min":13.276},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":99.620999999999995,"5min":99.620999999999995,"15min":99.620999999999995},"last":50.284999999999997},{"interface":"front","average":{"1min":4.133,"5min":4.133,"15min":4.133},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":77.405000000000001,"5min":77.405000000000001,"15min":77.405000000000001},"last":0.38400000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71536,"kb_used_data":42036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94300304,"statfs":{"total":96636764160,"available":96563511296,"internally_reserved":0,"allocated":43044864,"data_stored":43384776,"data_compressed":996606,"data_compressed_allocated":1056768,"data_compressed_original":2113536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.794,"15min":1.794},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":32.097999999999999,"5min":32.097999999999999,"15min":32.097999999999999},"last":0.378},{"interface":"front","average":{"1min":25.808,"5min":25.808,"15min":25.808},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":166.85499999999999,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.26100000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":37.161999999999999,"5min":37.161999999999999,"15min":37.161999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":10.092000000000001,"5min":10.092000000000001,"15min":10.092000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":222.55199999999999,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.35199999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.32000000000000001},{"interface":"front","average":{"1min":18.803999999999998,"5min":18.803999999999998,"15min":18.803999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":166.81800000000001,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.38700000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":21.518999999999998,"5min":21.518999999999998,"15min":21.518999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":171.99100000000001,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.438},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.36599999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":7.6029999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":81.697000000000003,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.28000000000000003},{"interface":"front","average":{"1min":19.327999999999999,"5min":19.327999999999999,"15min":19.327999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":166.733,"5min":166.733,"15min":166.733},"last":27.693000000000001}]},{"osd":5,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.41799999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":18.850999999999999,"5min":18.850999999999999,"15min":18.850999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":166.79400000000001,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.40600000000000003}]}]},{"osd":5,"up_from":0,"seq":55834574869,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110192,"kb_used_data":72932,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94261648,"statfs":{"total":96636764160,"available":96523927552,"internally_reserved":0,"allocated":74682368,"data_stored":75844949,"data_compressed":1517149,"data_compressed_allocated":1617920,"data_compressed_original":3544675,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1,0,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":16.648,"5min":16.648,"15min":16.648},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":86.712999999999994,"5min":86.712999999999994,"15min":86.712999999999994},"last":252.96199999999999},{"interface":"front","average":{"1min":8.5679999999999996,"5min":8.5679999999999996,"15min":8.5679999999999996},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":78.158000000000001,"5min":78.158000000000001,"15min":78.158000000000001},"last":148.95500000000001}]},{"osd":1,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":90.787999999999997},{"interface":"front","average":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":78.866,"5min":78.866,"15min":78.866},"last":91.873999999999995}]},{"osd":2,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":8.5700000000000003,"5min":8.5700000000000003,"15min":8.5700000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":76.724999999999994,"5min":76.724999999999994,"15min":76.724999999999994},"last":148.89400000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":90.772000000000006}]},{"osd":3,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":4.0220000000000002,"5min":4.0220000000000002,"15min":4.0220000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":78.912000000000006,"5min":78.912000000000006,"15min":78.912000000000006},"last":252.91900000000001},{"interface":"front","average":{"1min":20.968,"5min":20.968,"15min":20.968},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":227.41200000000001,"5min":227.41200000000001,"15min":227.41200000000001},"last":148.928}]},{"osd":5,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":90.688999999999993},{"interface":"front","average":{"1min":4.0380000000000003,"5min":4.0380000000000003,"15min":4.0380000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":78.811999999999998,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.31900000000000001}]},{"osd":6,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":13.16,"5min":13.16,"15min":13.16},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":124.303,"5min":124.303,"15min":124.303},"last":148.84899999999999},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":90.727999999999994}]},{"osd":7,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":8.5600000000000005,"5min":8.5600000000000005,"15min":8.5600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":76.697000000000003,"5min":76.697000000000003,"15min":76.697000000000003},"last":148.80099999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":90.754000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65212,"kb_used_data":32388,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94306628,"statfs":{"total":96636764160,"available":96569987072,"internally_reserved":0,"allocated":33165312,"data_stored":33411885,"data_compressed":828850,"data_compressed_allocated":880640,"data_compressed_original":1785856,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":12.199999999999999,"5min":12.199999999999999,"15min":12.199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":98.957999999999998,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.26900000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":5.0449999999999999,"5min":5.0449999999999999,"15min":5.0449999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":94.063000000000002,"5min":94.063000000000002,"15min":94.063000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":12.901999999999999,"5min":12.901999999999999,"15min":12.901999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.312}]},{"osd":4,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":4.1079999999999997,"5min":4.1079999999999997,"15min":4.1079999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":71.998000000000005,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":4.1379999999999999,"5min":4.1379999999999999,"15min":4.1379999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":72.227000000000004,"5min":72.227000000000004,"15min":72.227000000000004},"last":0.48699999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":13.94,"5min":13.94,"15min":13.94},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":13.965,"5min":13.965,"15min":13.965},"last":0.45700000000000002}]},{"osd":6,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.44400000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.46999999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48588,"kb_used_data":21564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94323252,"statfs":{"total":96636764160,"available":96587010048,"internally_reserved":0,"allocated":22081536,"data_stored":22713344,"data_compressed":1111203,"data_compressed_allocated":1183744,"data_compressed_original":2342912,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":149.03,"5min":149.03,"15min":149.03},"last":0.27700000000000002},{"interface":"front","average":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":31.866,"5min":31.866,"15min":31.866},"last":0.22900000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":13.868,"5min":13.868,"15min":13.868},"last":0.248},{"interface":"front","average":{"1min":14.885,"5min":14.885,"15min":14.885},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":160.37100000000001,"5min":160.37100000000001,"15min":160.37100000000001},"last":0.26300000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.316}]},{"osd":4,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":18.771999999999998,"5min":18.771999999999998,"15min":18.771999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":160.584,"5min":160.584,"15min":160.584},"last":0.39700000000000002},{"interface":"front","average":{"1min":2.448,"5min":2.448,"15min":2.448},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":36.841000000000001,"5min":36.841000000000001,"15min":36.841000000000001},"last":0.36099999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":3.052,"5min":3.052,"15min":3.052},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":48.631,"5min":48.631,"15min":48.631},"last":0.48699999999999999},{"interface":"front","average":{"1min":3.476,"5min":3.476,"15min":3.476},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":56.18,"5min":56.18,"15min":56.18},"last":0.42999999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.45600000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.374},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.41799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90388,"kb_used_data":57316,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94281452,"statfs":{"total":96636764160,"available":96544206848,"internally_reserved":0,"allocated":58691584,"data_stored":59175541,"data_compressed":1200174,"data_compressed_allocated":1277952,"data_compressed_original":2532963,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":7.859,"5min":7.859,"15min":7.859},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":88.171000000000006,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.32100000000000001},{"interface":"front","average":{"1min":7.8630000000000004,"5min":7.8630000000000004,"15min":7.8630000000000004},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":88.156000000000006,"5min":88.156000000000006,"15min":88.156000000000006},"last":0.23999999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":8.5999999999999996,"5min":8.5999999999999996,"15min":8.5999999999999996},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":183.04400000000001,"5min":183.04400000000001,"15min":183.04400000000001},"last":0.252},{"interface":"front","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.28299999999999997}]},{"osd":3,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.40699999999999997},{"interface":"front","average":{"1min":14.492000000000001,"5min":14.492000000000001,"15min":14.492000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":231.864,"5min":231.864,"15min":231.864},"last":0.30099999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":12.984,"5min":12.984,"15min":12.984},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":192.42099999999999,"5min":192.42099999999999,"15min":192.42099999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.47699999999999998}]},{"osd":5,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":14.071,"5min":14.071,"15min":14.071},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":138.089,"5min":138.089,"15min":138.089},"last":0.52100000000000002},{"interface":"front","average":{"1min":20.077000000000002,"5min":20.077000000000002,"15min":20.077000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":183.08199999999999,"5min":183.08199999999999,"15min":183.08199999999999},"last":0.49399999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":51.070999999999998,"5min":51.070999999999998,"15min":51.070999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":16.532,"5min":16.532,"15min":16.532},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":183.10900000000001,"5min":183.10900000000001,"15min":183.10900000000001},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":3.0390000000000001,"5min":3.0390000000000001,"15min":3.0390000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":51.097000000000001,"5min":51.097000000000001,"15min":51.097000000000001},"last":0.36799999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76920,"kb_used_data":45804,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94294920,"statfs":{"total":96636764160,"available":96557998080,"internally_reserved":0,"allocated":46903296,"data_stored":48563230,"data_compressed":1589952,"data_compressed_allocated":1703936,"data_compressed_original":4063232,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,0,0,6,2,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":6.7599999999999998,"5min":6.7599999999999998,"15min":6.7599999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":133.09899999999999,"5min":133.09899999999999,"15min":133.09899999999999},"last":99.296000000000006},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":25.827999999999999,"5min":25.827999999999999,"15min":25.827999999999999},"last":0.20899999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":25.873000000000001,"5min":25.873000000000001,"15min":25.873000000000001},"last":0.185},{"interface":"front","average":{"1min":0.318,"5min":0.318,"15min":0.318},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":94.820999999999998}]},{"osd":3,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":99.332999999999998},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":25.898,"5min":25.898,"15min":25.898},"last":297.82100000000003}]},{"osd":4,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":94.599999999999994},{"interface":"front","average":{"1min":9.9120000000000008,"5min":9.9120000000000008,"15min":9.9120000000000008},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":133.036,"5min":133.036,"15min":133.036},"last":160.02199999999999}]},{"osd":5,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":26.16,"5min":26.16,"15min":26.16},"last":0.253},{"interface":"front","average":{"1min":13.955,"5min":13.955,"15min":13.955},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":282.197,"5min":282.197,"15min":282.197},"last":94.626000000000005}]},{"osd":6,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":3.198,"5min":3.198,"15min":3.198},"last":99.355000000000004},{"interface":"front","average":{"1min":1.748,"5min":1.748,"15min":1.748},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":26.190000000000001,"5min":26.190000000000001,"15min":26.190000000000001},"last":0.311}]},{"osd":7,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":4.6779999999999999,"5min":4.6779999999999999,"15min":4.6779999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":87.209999999999994,"5min":87.209999999999994,"15min":87.209999999999994},"last":94.662999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.2440000000000002,"5min":3.2440000000000002,"15min":3.2440000000000002},"last":99.379999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29335552,"data_stored":31070534,"data_compressed":1490418,"data_compressed_allocated":1589248,"data_compressed_original":3420160,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14815232,"data_stored":15936841,"data_compressed":1111203,"data_compressed_allocated":1183744,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14815232,"data_stored":15936841,"data_compressed":1111203,"data_compressed_allocated":1183744,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11698176,"data_stored":12538012,"data_compressed":828850,"data_compressed_allocated":880640,"data_compressed_original":1785856,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":29335552,"data_stored":31070534,"data_compressed":1490418,"data_compressed_allocated":1589248,"data_compressed_original":3420160,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":223399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":11649024,"data_stored":12491934,"data_compressed":828850,"data_compressed_allocated":880640,"data_compressed_original":1785856,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16621568,"data_stored":16580352,"data_compressed":69966,"data_compressed_allocated":73728,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42983424,"data_stored":42855095,"data_compressed":88971,"data_compressed_allocated":94208,"data_compressed_original":190051,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6430720,"data_stored":6397277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20602880,"data_stored":20495368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":44511232,"data_stored":44395678,"data_compressed":26731,"data_compressed_allocated":28672,"data_compressed_original":124515,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33280000,"data_stored":33118894,"data_compressed":53657,"data_compressed_allocated":57344,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":30601216,"data_stored":30601671,"data_compressed":167756,"data_compressed_allocated":176128,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":13123584,"data_stored":13150271,"data_compressed":160030,"data_compressed_allocated":167936,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:40:13.975 INFO:tasks.ceph.ceph_manager.ceph:decrease pool size by 10 2024-04-22T18:40:13.975 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 16 2024-04-22T18:40:13.981 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:13.981 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:13.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:13.982 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:13.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.982 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:13.982 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:13.982 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.819250018000005, 2024-04-22T18:40:13.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:40:13.982 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:13.982 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.982 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:13.982 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:13.982 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:13.982 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:40:13.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.983 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:13.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:13.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:13.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:13.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:13.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:13.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:13.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:13.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:40:13.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:13.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:13.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:40:13.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:40:13.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:13.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:40:13.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:40:13.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:40:13.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:40:13.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:13.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:40:13.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:40:13.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:13.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:40:13.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:40:13.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:13.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:40:13.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:40:13.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:13.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:40:13.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:40:13.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:40:13.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:40:13.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:13.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:40:13.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:40:13.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:13.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:40:13.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:40:13.987 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:13.987 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:13.987 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:13.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.987 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:13.987 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:13.987 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.717691196999994, 2024-04-22T18:40:13.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:40:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:40:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:13.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:13.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:13.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:13.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:40:13.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:13.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:13.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:40:13.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:40:13.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:13.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:40:13.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:40:13.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:40:13.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:40:13.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:13.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:40:13.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:40:13.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:13.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:40:13.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:40:13.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:13.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:40:13.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:40:13.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:13.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:40:13.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:40:13.993 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:13.993 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:13.993 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:13.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.993 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:13.993 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:13.993 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.717664795000005, 2024-04-22T18:40:13.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:40:13.993 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:13.993 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:13.993 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:13.993 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:13.994 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:13.994 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:40:13.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.994 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:13.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:13.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:13.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:13.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:13.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:13.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:13.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:13.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:40:13.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:13.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:13.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:40:13.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:40:13.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:13.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:40:13.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:40:13.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:13.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:40:13.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:40:13.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:13.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:40:13.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:40:13.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:13.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:40:13.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:40:13.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:13.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:40:13.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:40:13.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:13.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:40:13.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:40:13.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:13.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:40:13.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:40:13.998 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:13.998 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:13.998 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:13.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.998 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:13.998 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:13.998 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.319330465999997, 2024-04-22T18:40:13.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:40:13.999 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:13.999 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:13.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:13.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:13.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:13.999 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:40:13.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:13.999 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:13.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:13.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:13.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:13.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:13.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:14.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:14.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:14.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:40:14.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:40:14.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:40:14.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:40:14.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:40:14.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:40:14.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:40:14.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:40:14.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:14.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:40:14.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:40:14.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:14.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:40:14.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:40:14.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:14.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:40:14.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:40:14.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:14.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:40:14.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:40:14.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:14.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:40:14.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:40:14.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:40:14.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:40:14.004 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.004 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.004 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.004 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:14.004 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:14.004 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.218494168000007, 2024-04-22T18:40:14.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:40:14.004 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.005 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:14.005 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.005 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.005 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.005 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:40:14.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.005 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:14.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:14.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:40:14.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:14.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:40:14.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:40:14.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:40:14.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:40:14.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:40:14.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:40:14.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:40:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:40:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:40:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:40:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:40:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:40:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:40:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:40:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:40:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:40:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:40:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:40:14.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:40:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:40:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.218336898000004, 2024-04-22T18:40:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:40:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:40:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:40:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:40:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:40:14.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:40:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:40:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:40:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:40:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:40:14.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:40:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:40:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:40:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:40:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:40:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:40:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:40:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:40:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:40:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:40:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:40:14.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:40:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:40:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.218322289, 2024-04-22T18:40:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:40:14.016 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:40:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:40:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:40:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:40:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:40:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:40:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:40:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:40:14.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:40:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:40:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:40:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:40:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:40:14.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:40:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:40:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:40:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.218311904000004, 2024-04-22T18:40:14.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:40:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:40:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:40:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:40:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:40:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:40:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:40:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:40:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:40:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:40:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:40:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:40:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:40:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:40:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:40:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:40:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:40:14.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:40:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:40:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:40:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:40:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:40:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:40:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.217381974000006, 2024-04-22T18:40:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:40:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:14.027 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:40:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:14.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:40:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:40:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:40:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:40:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:40:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:40:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:40:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:40:14.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:40:14.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:40:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:40:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:40:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:40:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:40:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:40:14.032 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.216919849000007, 2024-04-22T18:40:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:40:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.033 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:40:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:40:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:40:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:40:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:40:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:40:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:40:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:14.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:40:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:40:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:40:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:40:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:40:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:40:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:40:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:40:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:40:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:40:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.216907044999999, 2024-04-22T18:40:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:40:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:40:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.039 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:40:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:40:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:40:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:40:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:40:14.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:40:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:40:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:40:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:40:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:40:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:40:14.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:40:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:40:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:40:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:40:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:40:14.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:40:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 75.889134588999994, 2024-04-22T18:40:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:40:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.044 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:40:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:40:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:40:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:40:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:40:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:40:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:40:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:40:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:40:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:40:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:14.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:40:14.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:40:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:40:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:40:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:40:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:40:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:40:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 74.880561477000001, 2024-04-22T18:40:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:40:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:40:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.050 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:40:14.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:40:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:40:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:40:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:40:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:40:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:40:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:40:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:40:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:40:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:40:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:14.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:40:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:40:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:40:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:40:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:40:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:40:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:40:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:40:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:40:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:40:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:40:14.055 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.845913961999997, 2024-04-22T18:40:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:40:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:40:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:40:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:40:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:40:14.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:40:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:40:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:40:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:40:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:40:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:40:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:40:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:40:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:40:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:40:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.059 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1684 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 43.376429999000003, 2024-04-22T18:40:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987885199999998, 2024-04-22T18:40:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1684 2024-04-22T18:40:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605278+0000", 2024-04-22T18:40:14.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605280+0000", 2024-04-22T18:40:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1770000000000001e-06 2024-04-22T18:40:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605285+0000", 2024-04-22T18:40:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-04-22T18:40:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605289+0000", 2024-04-22T18:40:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.687e-06 2024-04-22T18:40:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705352+0000", 2024-04-22T18:40:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006322199999999 2024-04-22T18:40:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705423+0000", 2024-04-22T18:40:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1136999999999993e-05 2024-04-22T18:40:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105017+0000", 2024-04-22T18:40:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.399593268 2024-04-22T18:40:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105081+0000", 2024-04-22T18:40:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4703999999999996e-05 2024-04-22T18:40:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105157+0000", 2024-04-22T18:40:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6205999999999995e-05 2024-04-22T18:40:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2191 4.1 4.fdaa93c1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-04-22T18:40:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.431837749, 2024-04-22T18:40:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51064384399999996, 2024-04-22T18:40:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2191 2024-04-22T18:40:14.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549872+0000", 2024-04-22T18:40:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.387e-06 2024-04-22T18:40:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549875+0000", 2024-04-22T18:40:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549880+0000", 2024-04-22T18:40:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2529999999999998e-06 2024-04-22T18:40:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060496+0000", 2024-04-22T18:40:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51061597999999997 2024-04-22T18:40:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060515+0000", 2024-04-22T18:40:14.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9130000000000001e-05 2024-04-22T18:40:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2198 4.d 4:bc165c76:::smithi08937929-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.430570048, 2024-04-22T18:40:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50502289899999997, 2024-04-22T18:40:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.069 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2198 2024-04-22T18:40:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551138+0000", 2024-04-22T18:40:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551143+0000", 2024-04-22T18:40:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8949999999999997e-06 2024-04-22T18:40:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551151+0000", 2024-04-22T18:40:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8729999999999996e-06 2024-04-22T18:40:14.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551158+0000", 2024-04-22T18:40:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3099999999999997e-06 2024-04-22T18:40:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551180+0000", 2024-04-22T18:40:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2197000000000001e-05 2024-04-22T18:40:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551185+0000", 2024-04-22T18:40:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7010000000000004e-06 2024-04-22T18:40:14.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657133+0000", 2024-04-22T18:40:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105948498 2024-04-22T18:40:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657183+0000", 2024-04-22T18:40:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0124000000000003e-05 2024-04-22T18:40:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.055924+0000", 2024-04-22T18:40:14.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39874079299999998 2024-04-22T18:40:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056018+0000", 2024-04-22T18:40:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4032999999999999e-05 2024-04-22T18:40:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056161+0000", 2024-04-22T18:40:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143385 2024-04-22T18:40:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2212 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.430500406, 2024-04-22T18:40:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50989936400000002, 2024-04-22T18:40:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2212 2024-04-22T18:40:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551211+0000", 2024-04-22T18:40:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6280000000000001e-06 2024-04-22T18:40:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551214+0000", 2024-04-22T18:40:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1209999999999998e-06 2024-04-22T18:40:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551217+0000", 2024-04-22T18:40:14.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8490000000000002e-06 2024-04-22T18:40:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061086+0000", 2024-04-22T18:40:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50986975099999998 2024-04-22T18:40:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061108+0000", 2024-04-22T18:40:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1319e-05 2024-04-22T18:40:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2213 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.430489989000002, 2024-04-22T18:40:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50998486099999996, 2024-04-22T18:40:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2213 2024-04-22T18:40:14.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551218+0000", 2024-04-22T18:40:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551220+0000", 2024-04-22T18:40:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.319e-06 2024-04-22T18:40:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551223+0000", 2024-04-22T18:40:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9660000000000001e-06 2024-04-22T18:40:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551226+0000", 2024-04-22T18:40:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9050000000000001e-06 2024-04-22T18:40:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061162+0000", 2024-04-22T18:40:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.509936838 2024-04-22T18:40:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061204+0000", 2024-04-22T18:40:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1094999999999999e-05 2024-04-22T18:40:14.082 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2214 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.430481, 2024-04-22T18:40:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50997458100000004, 2024-04-22T18:40:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2214 2024-04-22T18:40:14.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.084 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:14.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:14.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:14.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:14.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:14.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:14.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:14.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551227+0000", 2024-04-22T18:40:14.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551229+0000", 2024-04-22T18:40:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8649999999999999e-06 2024-04-22T18:40:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551232+0000", 2024-04-22T18:40:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:14.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551235+0000", 2024-04-22T18:40:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:40:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061194+0000", 2024-04-22T18:40:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50995819099999995 2024-04-22T18:40:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061202+0000", 2024-04-22T18:40:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.541e-06 2024-04-22T18:40:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.086 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:14.087 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:14.087 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:14.088 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:13.982+0000 7f1e2dd22640 -1 osd.2 82 handle_osd_map injecting map crc failure 2024-04-22T18:40:14.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:14.091 INFO:tasks.rados.rados.0.smithi089.stdout:1343: finishing set_chunk to oid 2 2024-04-22T18:40:14.092 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 64 (ObjNum 251 snap 0 seq_num 251) dirty exists 2024-04-22T18:40:14.092 INFO:tasks.rados.rados.0.smithi089.stdout:1348: finishing write tid 1 to smithi08937929-16 2024-04-22T18:40:14.092 INFO:tasks.rados.rados.0.smithi089.stdout:1343: done (6 left) 2024-04-22T18:40:14.092 INFO:tasks.rados.rados.0.smithi089.stdout:1344: done (5 left) 2024-04-22T18:40:14.092 INFO:tasks.rados.rados.0.smithi089.stdout:1345: done (4 left) 2024-04-22T18:40:14.092 INFO:tasks.rados.rados.0.smithi089.stdout:1346: done (3 left) 2024-04-22T18:40:14.092 INFO:tasks.rados.rados.0.smithi089.stdout:1347: done (2 left) 2024-04-22T18:40:14.092 INFO:tasks.rados.rados.0.smithi089.stdout:1349: done (1 left) 2024-04-22T18:40:14.092 INFO:tasks.rados.rados.0.smithi089.stdout:1350: read oid 14 snap -1 2024-04-22T18:40:14.092 INFO:tasks.rados.rados.0.smithi089.stdout:1350: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:40:14.092 INFO:tasks.rados.rados.0.smithi089.stdout:1351: write (excl) oid 37 current snap is 0 2024-04-22T18:40:14.092 INFO:tasks.rados.rados.0.smithi089.stdout:1348: finishing write tid 2 to smithi08937929-16 2024-04-22T18:40:14.092 INFO:tasks.rados.rados.0.smithi089.stdout:1348: finishing write tid 3 to smithi08937929-16 2024-04-22T18:40:14.092 INFO:tasks.rados.rados.0.smithi089.stdout:1351: seq_num 257 ranges {719399=778642,1939284=380115} 2024-04-22T18:40:14.092 INFO:tasks.rados.rados.0.smithi089.stdout:1351: writing smithi08937929-37 from 719399 to 1498041 tid 1 2024-04-22T18:40:14.093 INFO:tasks.rados.rados.0.smithi089.stdout:1351: writing smithi08937929-37 from 1939284 to 2319399 tid 2 2024-04-22T18:40:14.093 INFO:tasks.rados.rados.0.smithi089.stdout:1352: tier_evict oid 43 2024-04-22T18:40:14.093 INFO:tasks.rados.rados.0.smithi089.stdout:1352: tier_evict oid 43 snap -1 2024-04-22T18:40:14.093 INFO:tasks.rados.rados.0.smithi089.stdout:1348: finishing write tid 4 to smithi08937929-16 2024-04-22T18:40:14.093 INFO:tasks.rados.rados.0.smithi089.stdout:1348: oid 16 updating version 0 to 98 2024-04-22T18:40:14.093 INFO:tasks.rados.rados.0.smithi089.stdout:1348: oid 16 updating version 98 to 99 2024-04-22T18:40:14.093 INFO:tasks.rados.rados.0.smithi089.stdout:1348: oid 16 version 99 is already newer than 97 2024-04-22T18:40:14.093 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 99 (ObjNum 255 snap 0 seq_num 255) dirty exists 2024-04-22T18:40:14.093 INFO:tasks.rados.rados.0.smithi089.stdout:1348: left oid 16 (ObjNum 255 snap 0 seq_num 255) 2024-04-22T18:40:14.093 INFO:tasks.rados.rados.0.smithi089.stdout:1348: done (3 left) 2024-04-22T18:40:14.093 INFO:tasks.rados.rados.0.smithi089.stdout:1353: read oid 50 snap -1 2024-04-22T18:40:14.093 INFO:tasks.rados.rados.0.smithi089.stdout:1353: expect (ObjNum 189 snap 0 seq_num 189) 2024-04-22T18:40:14.093 INFO:tasks.rados.rados.0.smithi089.stdout:1354: tier_promote oid 24 2024-04-22T18:40:14.093 INFO:tasks.rados.rados.0.smithi089.stdout:1355: write oid 49 current snap is 0 2024-04-22T18:40:14.093 INFO:tasks.rados.rados.0.smithi089.stdout:1355: seq_num 258 ranges {736206=453891,1881413=454793} 2024-04-22T18:40:14.094 INFO:tasks.rados.rados.0.smithi089.stdout:1355: writing smithi08937929-49 from 736206 to 1190097 tid 1 2024-04-22T18:40:14.094 INFO:tasks.rados.rados.0.smithi089.stdout:1355: writing smithi08937929-49 from 1881413 to 2336206 tid 2 2024-04-22T18:40:14.094 INFO:tasks.rados.rados.0.smithi089.stdout:1356: tier_promote oid 31 2024-04-22T18:40:14.094 INFO:tasks.rados.rados.0.smithi089.stdout:1352: got (22) Invalid argument 2024-04-22T18:40:14.094 INFO:tasks.rados.rados.0.smithi089.stdout:1352: done (6 left) 2024-04-22T18:40:14.094 INFO:tasks.rados.rados.0.smithi089.stdout:1357: write oid 3 current snap is 0 2024-04-22T18:40:14.094 INFO:tasks.rados.rados.0.smithi089.stdout:1357: seq_num 259 ranges {753013=645493,2056248=296765} 2024-04-22T18:40:14.094 INFO:tasks.rados.rados.0.smithi089.stdout:1357: writing smithi08937929-3 from 753013 to 1398506 tid 1 2024-04-22T18:40:14.094 INFO:tasks.rados.rados.0.smithi089.stdout:1357: writing smithi08937929-3 from 2056248 to 2353013 tid 2 2024-04-22T18:40:14.094 INFO:tasks.rados.rados.0.smithi089.stdout:1358: delete oid 46 current snap is 0 2024-04-22T18:40:14.094 INFO:tasks.rados.rados.0.smithi089.stdout:1353: expect (ObjNum 189 snap 0 seq_num 189) 2024-04-22T18:40:14.094 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:14.089+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:14.188 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:14.190+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:14.233 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:14.233 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:14.233 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:14.233 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:14.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:14.250 INFO:tasks.rados.rados.0.smithi089.stdout:1356: got (2) No such file or directory 2024-04-22T18:40:14.250 INFO:tasks.rados.rados.0.smithi089.stdout:1356: got expected ENOENT (src dne) 2024-04-22T18:40:14.250 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 31 v 45 (ObjNum 129 snap 0 seq_num 2413417912) clean dne 2024-04-22T18:40:14.250 INFO:tasks.rados.rados.0.smithi089.stdout:1357: finishing write tid 1 to smithi08937929-3 2024-04-22T18:40:14.250 INFO:tasks.rados.rados.0.smithi089.stdout:1357: finishing write tid 2 to smithi08937929-3 2024-04-22T18:40:14.250 INFO:tasks.rados.rados.0.smithi089.stdout:1357: finishing write tid 3 to smithi08937929-3 2024-04-22T18:40:14.250 INFO:tasks.rados.rados.0.smithi089.stdout:1353: done (7 left) 2024-04-22T18:40:14.250 INFO:tasks.rados.rados.0.smithi089.stdout:1356: done (6 left) 2024-04-22T18:40:14.250 INFO:tasks.rados.rados.0.smithi089.stdout:1358: done (5 left) 2024-04-22T18:40:14.250 INFO:tasks.rados.rados.0.smithi089.stdout:1359: read oid 1 snap -1 2024-04-22T18:40:14.250 INFO:tasks.rados.rados.0.smithi089.stdout:1359: expect deleted 2024-04-22T18:40:14.250 INFO:tasks.rados.rados.0.smithi089.stdout:1360: write (excl) oid 13 current snap is 0 2024-04-22T18:40:14.250 INFO:tasks.rados.rados.0.smithi089.stdout:1360: seq_num 261 ranges {786627=795991,2173212=213415} 2024-04-22T18:40:14.259 INFO:tasks.rados.rados.0.smithi089.stdout:1360: writing smithi08937929-13 from 786627 to 1582618 tid 1 2024-04-22T18:40:14.265 INFO:tasks.rados.rados.0.smithi089.stdout:1360: writing smithi08937929-13 from 2173212 to 2386627 tid 2 2024-04-22T18:40:14.265 INFO:tasks.rados.rados.0.smithi089.stdout:1361: read oid 2 snap -1 2024-04-22T18:40:14.266 INFO:tasks.rados.rados.0.smithi089.stdout:1361: expect (ObjNum 251 snap 0 seq_num 251) 2024-04-22T18:40:14.266 INFO:tasks.rados.rados.0.smithi089.stdout:1362: read oid 36 snap -1 2024-04-22T18:40:14.266 INFO:tasks.rados.rados.0.smithi089.stdout:1362: expect (ObjNum 208 snap 0 seq_num 208) 2024-04-22T18:40:14.266 INFO:tasks.rados.rados.0.smithi089.stdout:1363: write (excl) oid 33 current snap is 0 2024-04-22T18:40:14.266 INFO:tasks.rados.rados.0.smithi089.stdout:1363: seq_num 262 ranges {403434=471240,1315341=400756} 2024-04-22T18:40:14.271 INFO:tasks.rados.rados.0.smithi089.stdout:1363: writing smithi08937929-33 from 403434 to 874674 tid 1 2024-04-22T18:40:14.277 INFO:tasks.rados.rados.0.smithi089.stdout:1363: writing smithi08937929-33 from 1315341 to 1716097 tid 2 2024-04-22T18:40:14.277 INFO:tasks.rados.rados.0.smithi089.stdout:1364: read oid 34 snap -1 2024-04-22T18:40:14.277 INFO:tasks.rados.rados.0.smithi089.stdout:1364: expect (ObjNum 235 snap 0 seq_num 235) 2024-04-22T18:40:14.277 INFO:tasks.rados.rados.0.smithi089.stdout:1365: read oid 25 snap -1 2024-04-22T18:40:14.277 INFO:tasks.rados.rados.0.smithi089.stdout:1365: expect (ObjNum 232 snap 0 seq_num 232) 2024-04-22T18:40:14.277 INFO:tasks.rados.rados.0.smithi089.stdout:1366: write oid 12 current snap is 0 2024-04-22T18:40:14.277 INFO:tasks.rados.rados.0.smithi089.stdout:1366: seq_num 263 ranges {420241=546489,1373823=544414} 2024-04-22T18:40:14.283 INFO:tasks.rados.rados.0.smithi089.stdout:1366: writing smithi08937929-12 from 420241 to 966730 tid 1 2024-04-22T18:40:14.289 INFO:tasks.rados.rados.0.smithi089.stdout:1366: writing smithi08937929-12 from 1373823 to 1918237 tid 2 2024-04-22T18:40:14.289 INFO:tasks.rados.rados.0.smithi089.stdout:1367: write (excl) oid 5 current snap is 0 2024-04-22T18:40:14.289 INFO:tasks.rados.rados.0.smithi089.stdout:1362: expect (ObjNum 208 snap 0 seq_num 208) 2024-04-22T18:40:14.316 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:40:14.328 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:14.330+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:14.389 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:14.389 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:14.389 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:14.389 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:14.389 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:14.390 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:14.392+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:14.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:14.438 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:14.438 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:14.455 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:14.455 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:14.470 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:14.470 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:14.488 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:14.488 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:14.489 INFO:tasks.rados.rados.0.smithi089.stdout:1350: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:40:14.491 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:14.493+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:14.507 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:14.507 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:14.524 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:14.524 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:14.540 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:14.540 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:14.542 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:14.543 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:14.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:14.556 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:14.557 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:14.593 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:14.595+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:14.618 INFO:tasks.rados.rados.0.smithi089.stdout:1357: finishing write tid 4 to smithi08937929-3 2024-04-22T18:40:14.618 INFO:tasks.rados.rados.0.smithi089.stdout:1357: oid 3 updating version 0 to 68 2024-04-22T18:40:14.619 INFO:tasks.rados.rados.0.smithi089.stdout:1357: oid 3 version 68 is already newer than 66 2024-04-22T18:40:14.619 INFO:tasks.rados.rados.0.smithi089.stdout:1357: oid 3 version 68 is already newer than 67 2024-04-22T18:40:14.619 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 68 (ObjNum 259 snap 0 seq_num 259) dirty exists 2024-04-22T18:40:14.619 INFO:tasks.rados.rados.0.smithi089.stdout:1357: left oid 3 (ObjNum 259 snap 0 seq_num 259) 2024-04-22T18:40:14.619 INFO:tasks.rados.rados.0.smithi089.stdout:1361: expect (ObjNum 251 snap 0 seq_num 251) 2024-04-22T18:40:14.693 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:14.694+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:14.693 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:14.693 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:14.697 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:14.699+0000 7fec57bee640 -1 osd.7 83 handle_osd_map injecting map crc failure 2024-04-22T18:40:14.699 INFO:teuthology.orchestra.run.smithi089.stderr:set pool 4 pg_num to 16 2024-04-22T18:40:14.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:14.793 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:14.795+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:14.805 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:14.805 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:14.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:14.816 INFO:tasks.rados.rados.0.smithi089.stdout:1351: finishing write tid 1 to smithi08937929-37 2024-04-22T18:40:14.816 INFO:tasks.rados.rados.0.smithi089.stdout:1351: finishing write tid 2 to smithi08937929-37 2024-04-22T18:40:14.816 INFO:tasks.rados.rados.0.smithi089.stdout:1354: got (0) Success 2024-04-22T18:40:14.817 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 127 (ObjNum 248 snap 0 seq_num 248) dirty exists 2024-04-22T18:40:14.817 INFO:tasks.rados.rados.0.smithi089.stdout:1364: expect (ObjNum 235 snap 0 seq_num 235) 2024-04-22T18:40:14.894 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:14.896+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:14.914 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:14.914 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:14.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:14.994 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:14.996+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:15.065 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:15.065 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:15.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:15.077 INFO:tasks.rados.rados.0.smithi089.stdout:1360: finishing write tid 1 to smithi08937929-13 2024-04-22T18:40:15.078 INFO:tasks.rados.rados.0.smithi089.stdout:1360: finishing write tid 2 to smithi08937929-13 2024-04-22T18:40:15.078 INFO:tasks.rados.rados.0.smithi089.stdout:1351: finishing write tid 3 to smithi08937929-37 2024-04-22T18:40:15.078 INFO:tasks.rados.rados.0.smithi089.stdout:1351: finishing write tid 4 to smithi08937929-37 2024-04-22T18:40:15.078 INFO:tasks.rados.rados.0.smithi089.stdout:1351: oid 37 updating version 0 to 102 2024-04-22T18:40:15.078 INFO:tasks.rados.rados.0.smithi089.stdout:1351: oid 37 updating version 102 to 103 2024-04-22T18:40:15.078 INFO:tasks.rados.rados.0.smithi089.stdout:1351: oid 37 version 103 is already newer than 101 2024-04-22T18:40:15.078 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 103 (ObjNum 257 snap 0 seq_num 257) dirty exists 2024-04-22T18:40:15.078 INFO:tasks.rados.rados.0.smithi089.stdout:1351: left oid 37 (ObjNum 257 snap 0 seq_num 257) 2024-04-22T18:40:15.078 INFO:tasks.rados.rados.0.smithi089.stdout:1355: finishing write tid 1 to smithi08937929-49 2024-04-22T18:40:15.078 INFO:tasks.rados.rados.0.smithi089.stdout:1367: seq_num 264 ranges {437048=621738,1832305=404425} 2024-04-22T18:40:15.083 INFO:tasks.rados.rados.0.smithi089.stdout:1367: writing smithi08937929-5 from 437048 to 1058786 tid 1 2024-04-22T18:40:15.089 INFO:tasks.rados.rados.0.smithi089.stdout:1367: writing smithi08937929-5 from 1832305 to 2236730 tid 2 2024-04-22T18:40:15.090 INFO:tasks.rados.rados.0.smithi089.stdout:1350: done (13 left) 2024-04-22T18:40:15.090 INFO:tasks.rados.rados.0.smithi089.stdout:1351: done (12 left) 2024-04-22T18:40:15.090 INFO:tasks.rados.rados.0.smithi089.stdout:1354: done (11 left) 2024-04-22T18:40:15.090 INFO:tasks.rados.rados.0.smithi089.stdout:1357: done (10 left) 2024-04-22T18:40:15.090 INFO:tasks.rados.rados.0.smithi089.stdout:1359: done (9 left) 2024-04-22T18:40:15.090 INFO:tasks.rados.rados.0.smithi089.stdout:1361: done (8 left) 2024-04-22T18:40:15.090 INFO:tasks.rados.rados.0.smithi089.stdout:1362: done (7 left) 2024-04-22T18:40:15.090 INFO:tasks.rados.rados.0.smithi089.stdout:1364: done (6 left) 2024-04-22T18:40:15.090 INFO:tasks.rados.rados.0.smithi089.stdout:1368: set_chunk oid 27 target oid 2024-04-22T18:40:15.117 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:15.119+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:15.176 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:15.176 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:15.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:15.192 INFO:tasks.rados.rados.0.smithi089.stdout:1368: set_chunk oid 27 offset: 1841664 length: 267264 target oid c99fd512f21dd45484294408b0eb7a45641b2f7a45a6819d19baf188d271693c offset: 0 snap -1 2024-04-22T18:40:15.192 INFO:tasks.rados.rados.0.smithi089.stdout:1369: set_chunk oid 40 target oid 2024-04-22T18:40:15.196 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:15.198+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:15.297 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:15.298+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:15.328 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:15.328 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:15.337 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:15.398 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:15.400+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:15.405 INFO:tasks.rados.rados.0.smithi089.stdout:1369: set_chunk oid 40 offset: 239616 length: 1459200 target oid 3df93dbb121e2efb81eaabed8b4ed54116de76d98eb71c493d6087f45d8d5485 offset: 0 snap -1 2024-04-22T18:40:15.405 INFO:tasks.rados.rados.0.smithi089.stdout:1370: read oid 14 snap -1 2024-04-22T18:40:15.406 INFO:tasks.rados.rados.0.smithi089.stdout:1370: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:40:15.406 INFO:tasks.rados.rados.0.smithi089.stdout:1355: finishing write tid 2 to smithi08937929-49 2024-04-22T18:40:15.406 INFO:tasks.rados.rados.0.smithi089.stdout:1363: finishing write tid 1 to smithi08937929-33 2024-04-22T18:40:15.406 INFO:tasks.rados.rados.0.smithi089.stdout:1363: finishing write tid 2 to smithi08937929-33 2024-04-22T18:40:15.406 INFO:tasks.rados.rados.0.smithi089.stdout:1371: read oid 23 snap -1 2024-04-22T18:40:15.406 INFO:tasks.rados.rados.0.smithi089.stdout:1371: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:40:15.406 INFO:tasks.rados.rados.0.smithi089.stdout:1372: read oid 35 snap -1 2024-04-22T18:40:15.406 INFO:tasks.rados.rados.0.smithi089.stdout:1372: expect (ObjNum 212 snap 0 seq_num 212) 2024-04-22T18:40:15.406 INFO:tasks.rados.rados.0.smithi089.stdout:1373: write (excl) oid 28 current snap is 0 2024-04-22T18:40:15.406 INFO:tasks.rados.rados.0.smithi089.stdout:1363: finishing write tid 3 to smithi08937929-33 2024-04-22T18:40:15.406 INFO:tasks.rados.rados.0.smithi089.stdout:1363: finishing write tid 4 to smithi08937929-33 2024-04-22T18:40:15.406 INFO:tasks.rados.rados.0.smithi089.stdout:1363: oid 33 updating version 0 to 47 2024-04-22T18:40:15.407 INFO:tasks.rados.rados.0.smithi089.stdout:1363: oid 33 updating version 47 to 48 2024-04-22T18:40:15.407 INFO:tasks.rados.rados.0.smithi089.stdout:1363: oid 33 updating version 48 to 49 2024-04-22T18:40:15.407 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 49 (ObjNum 262 snap 0 seq_num 262) dirty exists 2024-04-22T18:40:15.407 INFO:tasks.rados.rados.0.smithi089.stdout:1363: left oid 33 (ObjNum 262 snap 0 seq_num 262) 2024-04-22T18:40:15.407 INFO:tasks.rados.rados.0.smithi089.stdout:1355: finishing write tid 3 to smithi08937929-49 2024-04-22T18:40:15.407 INFO:tasks.rados.rados.0.smithi089.stdout:1355: finishing write tid 4 to smithi08937929-49 2024-04-22T18:40:15.407 INFO:tasks.rados.rados.0.smithi089.stdout:1355: oid 49 updating version 0 to 105 2024-04-22T18:40:15.407 INFO:tasks.rados.rados.0.smithi089.stdout:1355: oid 49 updating version 105 to 106 2024-04-22T18:40:15.407 INFO:tasks.rados.rados.0.smithi089.stdout:1355: oid 49 version 106 is already newer than 104 2024-04-22T18:40:15.407 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 106 (ObjNum 258 snap 0 seq_num 258) dirty exists 2024-04-22T18:40:15.407 INFO:tasks.rados.rados.0.smithi089.stdout:1355: left oid 49 (ObjNum 258 snap 0 seq_num 258) 2024-04-22T18:40:15.407 INFO:tasks.rados.rados.0.smithi089.stdout:1373: seq_num 265 ranges {453855=696987,1890787=548083} 2024-04-22T18:40:15.409 INFO:tasks.rados.rados.0.smithi089.stdout:1373: writing smithi08937929-28 from 453855 to 1150842 tid 1 2024-04-22T18:40:15.416 INFO:tasks.rados.rados.0.smithi089.stdout:1373: writing smithi08937929-28 from 1890787 to 2438870 tid 2 2024-04-22T18:40:15.417 INFO:tasks.rados.rados.0.smithi089.stdout:1355: done (11 left) 2024-04-22T18:40:15.417 INFO:tasks.rados.rados.0.smithi089.stdout:1363: done (10 left) 2024-04-22T18:40:15.417 INFO:tasks.rados.rados.0.smithi089.stdout:1374: tier_evict oid 17 2024-04-22T18:40:15.417 INFO:tasks.rados.rados.0.smithi089.stdout:1374: tier_evict oid 17 snap -1 2024-04-22T18:40:15.417 INFO:tasks.rados.rados.0.smithi089.stdout:1365: expect (ObjNum 232 snap 0 seq_num 232) 2024-04-22T18:40:15.460 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:15.460 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:15.469 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:15.512 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:15.514+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:15.571 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:15.572 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:15.581 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:15.599 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:15.601+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:15.700 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:15.701+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:15.711 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:40:15.725 INFO:tasks.rados.rados.0.smithi089.stdout:1360: finishing write tid 3 to smithi08937929-13 2024-04-22T18:40:15.725 INFO:tasks.rados.rados.0.smithi089.stdout:1366: finishing write tid 1 to smithi08937929-12 2024-04-22T18:40:15.725 INFO:tasks.rados.rados.0.smithi089.stdout:1360: finishing write tid 4 to smithi08937929-13 2024-04-22T18:40:15.725 INFO:tasks.rados.rados.0.smithi089.stdout:1360: oid 13 updating version 0 to 65 2024-04-22T18:40:15.725 INFO:tasks.rados.rados.0.smithi089.stdout:1360: oid 13 version 65 is already newer than 64 2024-04-22T18:40:15.725 INFO:tasks.rados.rados.0.smithi089.stdout:1360: oid 13 updating version 65 to 66 2024-04-22T18:40:15.725 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 66 (ObjNum 261 snap 0 seq_num 261) dirty exists 2024-04-22T18:40:15.726 INFO:tasks.rados.rados.0.smithi089.stdout:1360: left oid 13 (ObjNum 261 snap 0 seq_num 261) 2024-04-22T18:40:15.726 INFO:tasks.rados.rados.0.smithi089.stdout:1366: finishing write tid 2 to smithi08937929-12 2024-04-22T18:40:15.726 INFO:tasks.rados.rados.0.smithi089.stdout:1360: done (10 left) 2024-04-22T18:40:15.726 INFO:tasks.rados.rados.0.smithi089.stdout:1365: done (9 left) 2024-04-22T18:40:15.726 INFO:tasks.rados.rados.0.smithi089.stdout:1375: write oid 45 current snap is 0 2024-04-22T18:40:15.726 INFO:tasks.rados.rados.0.smithi089.stdout:1366: finishing write tid 3 to smithi08937929-12 2024-04-22T18:40:15.726 INFO:tasks.rados.rados.0.smithi089.stdout:1366: finishing write tid 4 to smithi08937929-12 2024-04-22T18:40:15.726 INFO:tasks.rados.rados.0.smithi089.stdout:1366: oid 12 updating version 0 to 95 2024-04-22T18:40:15.726 INFO:tasks.rados.rados.0.smithi089.stdout:1366: oid 12 updating version 95 to 96 2024-04-22T18:40:15.726 INFO:tasks.rados.rados.0.smithi089.stdout:1366: oid 12 updating version 96 to 97 2024-04-22T18:40:15.726 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 97 (ObjNum 263 snap 0 seq_num 263) dirty exists 2024-04-22T18:40:15.726 INFO:tasks.rados.rados.0.smithi089.stdout:1366: left oid 12 (ObjNum 263 snap 0 seq_num 263) 2024-04-22T18:40:15.726 INFO:tasks.rados.rados.0.smithi089.stdout:1367: finishing write tid 1 to smithi08937929-5 2024-04-22T18:40:15.727 INFO:tasks.rados.rados.0.smithi089.stdout:1367: finishing write tid 2 to smithi08937929-5 2024-04-22T18:40:15.727 INFO:tasks.rados.rados.0.smithi089.stdout:1367: finishing write tid 3 to smithi08937929-5 2024-04-22T18:40:15.727 INFO:tasks.rados.rados.0.smithi089.stdout:1367: finishing write tid 4 to smithi08937929-5 2024-04-22T18:40:15.727 INFO:tasks.rados.rados.0.smithi089.stdout:1367: oid 5 updating version 0 to 136 2024-04-22T18:40:15.727 INFO:tasks.rados.rados.0.smithi089.stdout:1367: oid 5 version 136 is already newer than 134 2024-04-22T18:40:15.727 INFO:tasks.rados.rados.0.smithi089.stdout:1367: oid 5 version 136 is already newer than 135 2024-04-22T18:40:15.727 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 136 (ObjNum 264 snap 0 seq_num 264) dirty exists 2024-04-22T18:40:15.727 INFO:tasks.rados.rados.0.smithi089.stdout:1367: left oid 5 (ObjNum 264 snap 0 seq_num 264) 2024-04-22T18:40:15.727 INFO:tasks.rados.rados.0.smithi089.stdout:1368: finishing set_chunk to oid 27 2024-04-22T18:40:15.727 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 72 (ObjNum 160 snap 0 seq_num 160) dirty exists 2024-04-22T18:40:15.727 INFO:tasks.rados.rados.0.smithi089.stdout:1375: seq_num 266 ranges {470662=772236,1832916=408094} 2024-04-22T18:40:15.727 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:15.728 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:15.730 INFO:tasks.rados.rados.0.smithi089.stdout:1375: writing smithi08937929-45 from 470662 to 1242898 tid 1 2024-04-22T18:40:15.735 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:15.736 INFO:tasks.rados.rados.0.smithi089.stdout:1375: writing smithi08937929-45 from 1832916 to 2241010 tid 2 2024-04-22T18:40:15.736 INFO:tasks.rados.rados.0.smithi089.stdout:1366: done (9 left) 2024-04-22T18:40:15.736 INFO:tasks.rados.rados.0.smithi089.stdout:1367: done (8 left) 2024-04-22T18:40:15.736 INFO:tasks.rados.rados.0.smithi089.stdout:1368: done (7 left) 2024-04-22T18:40:15.736 INFO:tasks.rados.rados.0.smithi089.stdout:1376: tier_evict oid 16 2024-04-22T18:40:15.736 INFO:tasks.rados.rados.0.smithi089.stdout:1376: tier_evict oid 16 snap -1 2024-04-22T18:40:15.737 INFO:tasks.rados.rados.0.smithi089.stdout:1369: finishing set_chunk to oid 40 2024-04-22T18:40:15.737 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 40 set_chunk returned error code -95 offset: 239616 length: 1459200 2024-04-22T18:40:15.737 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 0 (ObjNum 227 snap 0 seq_num 227) dirty exists 2024-04-22T18:40:15.737 INFO:tasks.rados.rados.0.smithi089.stdout:1370: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:40:15.801 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:15.802+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:15.841 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:15.841 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:15.850 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:15.889 INFO:tasks.rados.rados.0.smithi089.stdout:1374: got (22) Invalid argument 2024-04-22T18:40:15.889 INFO:tasks.rados.rados.0.smithi089.stdout:1372: expect (ObjNum 212 snap 0 seq_num 212) 2024-04-22T18:40:15.901 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:15.903+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:15.953 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:15.953 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:15.962 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:16.004 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:16.005+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:16.080 INFO:tasks.rados.rados.0.smithi089.stdout:1371: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:40:16.103 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:16.105+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:16.105 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:16.105 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:16.114 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:16.204 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:16.205+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:16.231 INFO:tasks.rados.rados.0.smithi089.stdout:1373: finishing write tid 1 to smithi08937929-28 2024-04-22T18:40:16.231 INFO:tasks.rados.rados.0.smithi089.stdout:1373: finishing write tid 2 to smithi08937929-28 2024-04-22T18:40:16.231 INFO:tasks.rados.rados.0.smithi089.stdout:1373: finishing write tid 3 to smithi08937929-28 2024-04-22T18:40:16.231 INFO:tasks.rados.rados.0.smithi089.stdout:1373: finishing write tid 4 to smithi08937929-28 2024-04-22T18:40:16.231 INFO:tasks.rados.rados.0.smithi089.stdout:1373: oid 28 updating version 0 to 139 2024-04-22T18:40:16.231 INFO:tasks.rados.rados.0.smithi089.stdout:1373: oid 28 version 139 is already newer than 138 2024-04-22T18:40:16.231 INFO:tasks.rados.rados.0.smithi089.stdout:1373: oid 28 version 139 is already newer than 137 2024-04-22T18:40:16.231 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 139 (ObjNum 265 snap 0 seq_num 265) dirty exists 2024-04-22T18:40:16.231 INFO:tasks.rados.rados.0.smithi089.stdout:1373: left oid 28 (ObjNum 265 snap 0 seq_num 265) 2024-04-22T18:40:16.232 INFO:tasks.rados.rados.0.smithi089.stdout:1376: got (22) Invalid argument 2024-04-22T18:40:16.232 INFO:tasks.rados.rados.0.smithi089.stdout:1375: finishing write tid 1 to smithi08937929-45 2024-04-22T18:40:16.232 INFO:tasks.rados.rados.0.smithi089.stdout:1375: finishing write tid 2 to smithi08937929-45 2024-04-22T18:40:16.232 INFO:tasks.rados.rados.0.smithi089.stdout:1369: done (7 left) 2024-04-22T18:40:16.232 INFO:tasks.rados.rados.0.smithi089.stdout:1370: done (6 left) 2024-04-22T18:40:16.232 INFO:tasks.rados.rados.0.smithi089.stdout:1371: done (5 left) 2024-04-22T18:40:16.232 INFO:tasks.rados.rados.0.smithi089.stdout:1372: done (4 left) 2024-04-22T18:40:16.232 INFO:tasks.rados.rados.0.smithi089.stdout:1373: done (3 left) 2024-04-22T18:40:16.232 INFO:tasks.rados.rados.0.smithi089.stdout:1374: done (2 left) 2024-04-22T18:40:16.232 INFO:tasks.rados.rados.0.smithi089.stdout:1376: done (1 left) 2024-04-22T18:40:16.232 INFO:tasks.rados.rados.0.smithi089.stdout:1377: read oid 36 snap -1 2024-04-22T18:40:16.232 INFO:tasks.rados.rados.0.smithi089.stdout:1377: expect (ObjNum 208 snap 0 seq_num 208) 2024-04-22T18:40:16.232 INFO:tasks.rados.rados.0.smithi089.stdout:1378: read oid 5 snap -1 2024-04-22T18:40:16.233 INFO:tasks.rados.rados.0.smithi089.stdout:1378: expect (ObjNum 264 snap 0 seq_num 264) 2024-04-22T18:40:16.233 INFO:tasks.rados.rados.0.smithi089.stdout:1379: read oid 43 snap -1 2024-04-22T18:40:16.233 INFO:tasks.rados.rados.0.smithi089.stdout:1379: expect (ObjNum 226 snap 0 seq_num 226) 2024-04-22T18:40:16.233 INFO:tasks.rados.rados.0.smithi089.stdout:1380: write (excl) oid 37 current snap is 0 2024-04-22T18:40:16.233 INFO:tasks.rados.rados.0.smithi089.stdout:1375: finishing write tid 3 to smithi08937929-45 2024-04-22T18:40:16.233 INFO:tasks.rados.rados.0.smithi089.stdout:1375: finishing write tid 4 to smithi08937929-45 2024-04-22T18:40:16.233 INFO:tasks.rados.rados.0.smithi089.stdout:1375: oid 45 updating version 0 to 69 2024-04-22T18:40:16.233 INFO:tasks.rados.rados.0.smithi089.stdout:1375: oid 45 version 69 is already newer than 68 2024-04-22T18:40:16.233 INFO:tasks.rados.rados.0.smithi089.stdout:1375: oid 45 version 69 is already newer than 67 2024-04-22T18:40:16.233 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 69 (ObjNum 266 snap 0 seq_num 266) dirty exists 2024-04-22T18:40:16.233 INFO:tasks.rados.rados.0.smithi089.stdout:1375: left oid 45 (ObjNum 266 snap 0 seq_num 266) 2024-04-22T18:40:16.233 INFO:tasks.rados.rados.0.smithi089.stdout:1380: seq_num 267 ranges {487469=563838,1607751=551752} 2024-04-22T18:40:16.235 INFO:tasks.rados.rados.0.smithi089.stdout:1380: writing smithi08937929-37 from 487469 to 1051307 tid 1 2024-04-22T18:40:16.242 INFO:tasks.rados.rados.0.smithi089.stdout:1380: writing smithi08937929-37 from 1607751 to 2159503 tid 2 2024-04-22T18:40:16.242 INFO:tasks.rados.rados.0.smithi089.stdout:1375: done (4 left) 2024-04-22T18:40:16.242 INFO:tasks.rados.rados.0.smithi089.stdout:1381: tier_evict oid 41 2024-04-22T18:40:16.242 INFO:tasks.rados.rados.0.smithi089.stdout:1381: tier_evict oid 41 snap -1 2024-04-22T18:40:16.242 INFO:tasks.rados.rados.0.smithi089.stdout:1382: set_chunk oid 32 target oid 2024-04-22T18:40:16.251 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:16.252+0000 7f8605f32640 -1 osd.3 84 handle_osd_map injecting map crc failure 2024-04-22T18:40:16.266 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:16.266 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:16.275 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:16.318 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:16.320+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:16.377 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:16.377 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:16.386 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:16.427 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:16.428+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:16.507 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:16.508+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:16.528 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:16.528 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:16.536 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:16.568 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:40:16.608 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:16.609+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:16.639 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:40:16.640 INFO:tasks.rados.rados.0.smithi089.stdout:1382: set_chunk oid 32 offset: 177152 length: 87552 target oid 3ae296b6ae39e02f816ae3fcf288cc5d539aa4fa82171c7013427a340ea76c27 offset: 0 snap -1 2024-04-22T18:40:16.640 INFO:tasks.rados.rados.0.smithi089.stdout:1383: write (excl) oid 39 current snap is 0 2024-04-22T18:40:16.640 INFO:tasks.rados.rados.0.smithi089.stdout:1383: seq_num 268 ranges {504276=639087,1666233=695410} 2024-04-22T18:40:16.643 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:16.643 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:16.648 INFO:tasks.rados.rados.0.smithi089.stdout:1383: writing smithi08937929-39 from 504276 to 1143363 tid 1 2024-04-22T18:40:16.655 INFO:tasks.rados.rados.0.smithi089.stdout:1383: writing smithi08937929-39 from 1666233 to 2361643 tid 2 2024-04-22T18:40:16.655 INFO:tasks.rados.rados.0.smithi089.stdout:1384: read oid 15 snap -1 2024-04-22T18:40:16.656 INFO:tasks.rados.rados.0.smithi089.stdout:1384: expect deleted 2024-04-22T18:40:16.656 INFO:tasks.rados.rados.0.smithi089.stdout:1385: set_chunk oid 26 target oid 2024-04-22T18:40:16.657 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:16.700 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:16.701 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:16.709 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:16.710+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:16.718 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:16.718 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:16.734 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:16.734 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:16.750 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:16.751 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:16.768 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:16.768 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:16.785 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:16.785 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:16.802 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:16.802 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:16.802 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:16.802 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:16.802 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:16.803 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:16.803 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:16.810 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:16.811+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:16.811 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:16.820 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:16.820 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:16.910 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:16.912+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:16.923 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:16.923 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:16.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:16.923 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:16.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.923 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:78 4.0 4:01baf5a9:::smithi08937929-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:16.924 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:16.924 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.411262364999999, 2024-04-22T18:40:16.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.398522192, 2024-04-22T18:40:16.924 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:16.924 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:16.924 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:16.924 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:16.924 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:16.924 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 78 2024-04-22T18:40:16.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.924 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:16.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:16.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:16.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:16.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:16.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:16.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511934+0000", 2024-04-22T18:40:16.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:16.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:16.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512012+0000", 2024-04-22T18:40:16.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8329000000000002e-05 2024-04-22T18:40:16.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:16.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512023+0000", 2024-04-22T18:40:16.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0755e-05 2024-04-22T18:40:16.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:16.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512028+0000", 2024-04-22T18:40:16.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6470000000000003e-06 2024-04-22T18:40:16.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512068+0000", 2024-04-22T18:40:16.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9721999999999997e-05 2024-04-22T18:40:16.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:16.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512125+0000", 2024-04-22T18:40:16.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6564999999999998e-05 2024-04-22T18:40:16.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:16.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512200+0000", 2024-04-22T18:40:16.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5708000000000001e-05 2024-04-22T18:40:16.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:16.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513131+0000", 2024-04-22T18:40:16.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093063 2024-04-22T18:40:16.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:16.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910394+0000", 2024-04-22T18:40:16.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.397263388 2024-04-22T18:40:16.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:16.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910430+0000", 2024-04-22T18:40:16.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5485999999999999e-05 2024-04-22T18:40:16.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910457+0000", 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7086000000000001e-05 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:79 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.411161708999998, 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39855419399999997, 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:16.929 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:16.930 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 79 2024-04-22T18:40:16.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.930 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:16.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:16.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:16.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:16.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:16.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:16.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512035+0000", 2024-04-22T18:40:16.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:16.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:16.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512045+0000", 2024-04-22T18:40:16.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0379e-05 2024-04-22T18:40:16.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:16.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512051+0000", 2024-04-22T18:40:16.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6030000000000004e-06 2024-04-22T18:40:16.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512057+0000", 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7270000000000004e-06 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512756+0000", 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069954000000000001 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512806+0000", 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9922000000000001e-05 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512927+0000", 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012059 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.515883+0000", 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029563739999999999 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910510+0000", 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39462745199999999 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:16.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910542+0000", 2024-04-22T18:40:16.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1225e-05 2024-04-22T18:40:16.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:16.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910590+0000", 2024-04-22T18:40:16.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8052e-05 2024-04-22T18:40:16.934 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.934 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:16.934 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.934 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:80 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:16.934 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:16.934 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.411136608000007, 2024-04-22T18:40:16.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39866376999999997, 2024-04-22T18:40:16.935 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:16.935 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:16.935 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:16.935 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:16.935 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:16.935 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 80 2024-04-22T18:40:16.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.935 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:16.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:16.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:16.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:16.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:16.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:16.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512060+0000", 2024-04-22T18:40:16.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:16.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:16.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512063+0000", 2024-04-22T18:40:16.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.999e-06 2024-04-22T18:40:16.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512069+0000", 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.891e-06 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512073+0000", 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4429999999999996e-06 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513084+0000", 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010102609999999999 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:16.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513123+0000", 2024-04-22T18:40:16.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9237999999999999e-05 2024-04-22T18:40:16.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910618+0000", 2024-04-22T18:40:16.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39749549299999998 2024-04-22T18:40:16.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:16.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910670+0000", 2024-04-22T18:40:16.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1221000000000003e-05 2024-04-22T18:40:16.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910724+0000", 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4777000000000001e-05 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:419 4.0 4:01baf5a9:::smithi08937929-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 77.520975587999999, 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30324478199999999, 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:16.939 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 419 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402219+0000", 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402229+0000", 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2550000000000003e-06 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402242+0000", 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2989999999999999e-05 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:16.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402250+0000", 2024-04-22T18:40:16.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7900000000000005e-06 2024-04-22T18:40:16.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404085+0000", 2024-04-22T18:40:16.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001834397 2024-04-22T18:40:16.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:40:16.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404092+0000", 2024-04-22T18:40:16.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7889999999999997e-06 2024-04-22T18:40:16.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502534+0000", 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098442331999999994 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502631+0000", 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7617999999999999e-05 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604414+0000", 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101782847 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:16.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604888+0000", 2024-04-22T18:40:16.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047325999999999998 2024-04-22T18:40:16.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:16.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705376+0000", 2024-04-22T18:40:16.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100488623 2024-04-22T18:40:16.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:16.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705424+0000", 2024-04-22T18:40:16.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7964999999999999e-05 2024-04-22T18:40:16.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705466+0000", 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2274999999999998e-05 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:229 3.2 e51/29 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head v 51'117, mlcod=51'115)", 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.11215455, 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31548799399999999, 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:16.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:16.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:16.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:16.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:16.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:16.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811041+0000", 2024-04-22T18:40:16.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:16.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811054+0000", 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3188e-05 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811068+0000", 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4239999999999999e-05 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.911141+0000", 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100072851 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025053+0000", 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.11391237 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025083+0000", 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9703999999999999e-05 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126504+0000", 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10142053500000001 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:16.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126531+0000", 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7172999999999999e-05 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2347 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 3378207] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.735376325000001, 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30520418599999999, 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:16.949 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2347 2024-04-22T18:40:16.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.950 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:16.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:16.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:40:16.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:16.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:40:16.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187820+0000", 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187823+0000", 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6299999999999998e-06 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187827+0000", 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3739999999999998e-06 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:16.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187829+0000", 2024-04-22T18:40:16.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6290000000000002e-06 2024-04-22T18:40:16.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291848+0000", 2024-04-22T18:40:16.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104018812 2024-04-22T18:40:16.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:16.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291924+0000", 2024-04-22T18:40:16.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6421000000000006e-05 2024-04-22T18:40:16.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:16.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292050+0000", 2024-04-22T18:40:16.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012571500000000001 2024-04-22T18:40:16.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:16.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292407+0000", 2024-04-22T18:40:16.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035709199999999999 2024-04-22T18:40:16.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:16.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.492962+0000", 2024-04-22T18:40:16.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20055479200000001 2024-04-22T18:40:16.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:16.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.492993+0000", 2024-04-22T18:40:16.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1041000000000003e-05 2024-04-22T18:40:16.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:16.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493025+0000", 2024-04-22T18:40:16.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2092999999999999e-05 2024-04-22T18:40:16.954 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.954 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:16.954 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.954 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2348 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:40:16.954 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:16.955 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.735365008999999, 2024-04-22T18:40:16.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30532242999999998, 2024-04-22T18:40:16.955 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:16.955 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:16.955 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:16.955 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:16.955 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:16.955 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2348 2024-04-22T18:40:16.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.955 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:16.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:16.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:16.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187833+0000", 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.437e-06 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187836+0000", 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.897e-06 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187839+0000", 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7250000000000002e-06 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292213+0000", 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104373761 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:16.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292256+0000", 2024-04-22T18:40:16.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2896999999999999e-05 2024-04-22T18:40:16.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493044+0000", 2024-04-22T18:40:16.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20078818700000001 2024-04-22T18:40:16.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:16.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493099+0000", 2024-04-22T18:40:16.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5009000000000001e-05 2024-04-22T18:40:16.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493155+0000", 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5807000000000003e-05 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:289 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.282472299, 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9176801189999999, 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:16.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640723+0000", 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640732+0000", 2024-04-22T18:40:16.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5809999999999997e-06 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640744+0000", 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.261e-05 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640753+0000", 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7460000000000006e-06 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642189+0000", 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001435551 2024-04-22T18:40:16.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:16.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642200+0000", 2024-04-22T18:40:16.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1389999999999999e-05 2024-04-22T18:40:16.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.847015+0000", 2024-04-22T18:40:16.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048149160000001 2024-04-22T18:40:16.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:16.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.848217+0000", 2024-04-22T18:40:16.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012020659999999999 2024-04-22T18:40:16.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351823+0000", 2024-04-22T18:40:16.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50360565099999999 2024-04-22T18:40:16.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:16.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351907+0000", 2024-04-22T18:40:16.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4141999999999995e-05 2024-04-22T18:40:16.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:16.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352051+0000", 2024-04-22T18:40:16.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014386600000000001 2024-04-22T18:40:16.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453617+0000", 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156649500000001 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558297+0000", 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10468000399999999 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558331+0000", 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3518e-05 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:16.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558405+0000", 2024-04-22T18:40:16.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4221000000000006e-05 2024-04-22T18:40:16.965 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.965 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:16.965 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.965 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:1 3.2 3:6d90b903:::64b5942574fb36ca1b9a2be186c6b740edf58e989056a2f88b83cd280451a438:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:16.965 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:16.965 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.751171339000001, 2024-04-22T18:40:16.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.285534422, 2024-04-22T18:40:16.965 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:16.965 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:16.965 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:16.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:16.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:16.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:16.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:16.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:16.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172024+0000", 2024-04-22T18:40:16.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:16.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:16.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173091+0000", 2024-04-22T18:40:16.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010677569999999999 2024-04-22T18:40:16.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:16.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173103+0000", 2024-04-22T18:40:16.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1241e-05 2024-04-22T18:40:16.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:16.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173109+0000", 2024-04-22T18:40:16.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4300000000000003e-06 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173168+0000", 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9011000000000001e-05 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173177+0000", 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948501+0000", 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77532387199999997 2024-04-22T18:40:16.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948912+0000", 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041183000000000002 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950438+0000", 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015253199999999999 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950488+0000", 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0285999999999999e-05 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:16.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950601+0000", 2024-04-22T18:40:16.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011244899999999999 2024-04-22T18:40:16.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:16.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954048+0000", 2024-04-22T18:40:16.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003447313 2024-04-22T18:40:16.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:16.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457373+0000", 2024-04-22T18:40:16.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50332465299999996 2024-04-22T18:40:16.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:16.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457430+0000", 2024-04-22T18:40:16.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7574999999999997e-05 2024-04-22T18:40:16.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:16.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457560+0000", 2024-04-22T18:40:16.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013023900000000001 2024-04-22T18:40:16.971 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.971 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:16.971 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.971 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:8 3.2 3:5f64590d:::7c47933f331d40689081b2bd8f9d7f7bdae1be1fdccb52d696047a3969e902b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:16.971 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:16.972 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.750078970000001, 2024-04-22T18:40:16.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3854200569999999, 2024-04-22T18:40:16.972 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:16.972 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:16.972 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:16.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:16.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:16.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:16.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:16.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:16.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173117+0000", 2024-04-22T18:40:16.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:16.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:16.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173393+0000", 2024-04-22T18:40:16.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000275278 2024-04-22T18:40:16.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:16.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173399+0000", 2024-04-22T18:40:16.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4590000000000003e-06 2024-04-22T18:40:16.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:16.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173406+0000", 2024-04-22T18:40:16.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2450000000000001e-06 2024-04-22T18:40:16.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173795+0000", 2024-04-22T18:40:16.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038912400000000002 2024-04-22T18:40:16.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:16.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173804+0000", 2024-04-22T18:40:16.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-04-22T18:40:16.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.949506+0000", 2024-04-22T18:40:16.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77570185700000005 2024-04-22T18:40:16.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:16.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950166+0000", 2024-04-22T18:40:16.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066022800000000001 2024-04-22T18:40:16.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352843+0000", 2024-04-22T18:40:16.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40267661799999999 2024-04-22T18:40:16.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:16.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352911+0000", 2024-04-22T18:40:16.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8126000000000005e-05 2024-04-22T18:40:16.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:16.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.353031+0000", 2024-04-22T18:40:16.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119919 2024-04-22T18:40:16.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:16.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453622+0000", 2024-04-22T18:40:16.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100590793 2024-04-22T18:40:16.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:16.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558446+0000", 2024-04-22T18:40:16.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104824073 2024-04-22T18:40:16.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:16.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558479+0000", 2024-04-22T18:40:16.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3362999999999999e-05 2024-04-22T18:40:16.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:16.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558538+0000", 2024-04-22T18:40:16.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9046999999999999e-05 2024-04-22T18:40:16.978 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.978 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:16.978 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.978 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:10 3.2 3:518c37f9:::851eca84dafbf2f780c4f07291110c453349bdc9c623c17bdcd26e27badb30ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:16.978 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:16.978 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.749785377, 2024-04-22T18:40:16.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3852527729999999, 2024-04-22T18:40:16.978 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:16.978 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:16.978 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:16.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:16.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:16.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:16.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:16.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:16.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173411+0000", 2024-04-22T18:40:16.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:16.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:16.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173749+0000", 2024-04-22T18:40:16.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000338029 2024-04-22T18:40:16.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:16.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173755+0000", 2024-04-22T18:40:16.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0530000000000003e-06 2024-04-22T18:40:16.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:16.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173759+0000", 2024-04-22T18:40:16.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8190000000000002e-06 2024-04-22T18:40:16.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173855+0000", 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5927999999999994e-05 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173862+0000", 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4449999999999999e-06 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950835+0000", 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77697248299999999 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951632+0000", 2024-04-22T18:40:16.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079752299999999996 2024-04-22T18:40:16.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355407+0000", 2024-04-22T18:40:16.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40377480500000001 2024-04-22T18:40:16.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:16.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355462+0000", 2024-04-22T18:40:16.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4577999999999998e-05 2024-04-22T18:40:16.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355571+0000", 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108964 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453625+0000", 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098054397000000001 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558574+0000", 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104948571 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558602+0000", 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7966999999999998e-05 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558665+0000", 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3032000000000002e-05 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:11 3.2 3:718aa8aa:::9603c4f8e269a436b2f0bf376c5374e77bdd7cf58fccae60e4e65ac4696806d4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.749433759, 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4852604440000001, 2024-04-22T18:40:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173813+0000", 2024-04-22T18:40:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0655999999999997e-05 2024-04-22T18:40:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173819+0000", 2024-04-22T18:40:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.468e-06 2024-04-22T18:40:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173822+0000", 2024-04-22T18:40:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4879999999999999e-06 2024-04-22T18:40:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173983+0000", 2024-04-22T18:40:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160519 2024-04-22T18:40:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173991+0000", 2024-04-22T18:40:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6490000000000008e-06 2024-04-22T18:40:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951688+0000", 2024-04-22T18:40:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77769743800000002 2024-04-22T18:40:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952419+0000", 2024-04-22T18:40:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073067499999999997 2024-04-22T18:40:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355901+0000", 2024-04-22T18:40:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40348237599999998 2024-04-22T18:40:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355953+0000", 2024-04-22T18:40:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1477000000000002e-05 2024-04-22T18:40:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356060+0000", 2024-04-22T18:40:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107114 2024-04-22T18:40:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453629+0000", 2024-04-22T18:40:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097569127000000005 2024-04-22T18:40:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658807+0000", 2024-04-22T18:40:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20517829800000001 2024-04-22T18:40:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658876+0000", 2024-04-22T18:40:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8454000000000006e-05 2024-04-22T18:40:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659024+0000", 2024-04-22T18:40:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148338 2024-04-22T18:40:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:12 3.2 3:475e1cd9:::efaab2bd13543cdbb143e869e20eb0ec424f7f5202707859b241ed91d4674936:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.749370815000001, 2024-04-22T18:40:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4853405259999999, 2024-04-22T18:40:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173940+0000", 2024-04-22T18:40:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011465500000000001 2024-04-22T18:40:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173946+0000", 2024-04-22T18:40:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5289999999999999e-06 2024-04-22T18:40:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173950+0000", 2024-04-22T18:40:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8290000000000001e-06 2024-04-22T18:40:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174277+0000", 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032761299999999998 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174285+0000", 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.52e-06 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952455+0000", 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77817000700000005 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954035+0000", 2024-04-22T18:40:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001580078 2024-04-22T18:40:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356379+0000", 2024-04-22T18:40:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40234440399999999 2024-04-22T18:40:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356428+0000", 2024-04-22T18:40:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8832999999999998e-05 2024-04-22T18:40:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356532+0000", 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103235 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453632+0000", 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097100346000000004 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659069+0000", 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20543763800000001 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659105+0000", 2024-04-22T18:40:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5240000000000001e-05 2024-04-22T18:40:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659167+0000", 2024-04-22T18:40:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2207999999999998e-05 2024-04-22T18:40:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:16 3.2 3:47c76cc8:::c00d18458eb20c0c0aa9c612cf9793a68445ab0ea7b9e650ecd0072c60a96c0f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.749242856, 2024-04-22T18:40:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.485347744, 2024-04-22T18:40:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174234+0000", 2024-04-22T18:40:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028047599999999999 2024-04-22T18:40:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174240+0000", 2024-04-22T18:40:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2079999999999997e-06 2024-04-22T18:40:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174244+0000", 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5810000000000001e-06 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174670+0000", 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042640000000000001 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174675+0000", 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8709999999999996e-06 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054168+0000", 2024-04-22T18:40:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87949257300000006 2024-04-22T18:40:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054877+0000", 2024-04-22T18:40:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070884300000000002 2024-04-22T18:40:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356836+0000", 2024-04-22T18:40:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30195926000000001 2024-04-22T18:40:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356887+0000", 2024-04-22T18:40:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0767999999999999e-05 2024-04-22T18:40:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356991+0000", 2024-04-22T18:40:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104072 2024-04-22T18:40:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453636+0000", 2024-04-22T18:40:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096644968999999997 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659207+0000", 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20557154699999999 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659240+0000", 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2665e-05 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659302+0000", 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2234000000000001e-05 2024-04-22T18:40:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:17 3.2 3:6ef31daa:::1cf4ff496c4480e58510f607481d976b47d84868e2c54016273c38baf590f3b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.748948576, 2024-04-22T18:40:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4851703729999999, 2024-04-22T18:40:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174248+0000", 2024-04-22T18:40:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174626+0000", 2024-04-22T18:40:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037839699999999999 2024-04-22T18:40:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174633+0000", 2024-04-22T18:40:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0990000000000004e-06 2024-04-22T18:40:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174636+0000", 2024-04-22T18:40:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5540000000000002e-06 2024-04-22T18:40:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174790+0000", 2024-04-22T18:40:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153807 2024-04-22T18:40:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174793+0000", 2024-04-22T18:40:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4970000000000001e-06 2024-04-22T18:40:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054902+0000", 2024-04-22T18:40:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88010886300000002 2024-04-22T18:40:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055366+0000", 2024-04-22T18:40:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046336199999999999 2024-04-22T18:40:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.452973+0000", 2024-04-22T18:40:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39760783999999999 2024-04-22T18:40:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453046+0000", 2024-04-22T18:40:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2374999999999997e-05 2024-04-22T18:40:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453193+0000", 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014722499999999999 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554698+0000", 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101505096 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659337+0000", 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104639236 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659365+0000", 2024-04-22T18:40:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7594e-05 2024-04-22T18:40:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659419+0000", 2024-04-22T18:40:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4058999999999999e-05 2024-04-22T18:40:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:18 3.2 3:792a4ed0:::e1c814961b3d1531fb5ed475b3d496a0405c3a819b68a92ff9b2391fe912370f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.74855232, 2024-04-22T18:40:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.585036954, 2024-04-22T18:40:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174640+0000", 2024-04-22T18:40:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-04-22T18:40:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174758+0000", 2024-04-22T18:40:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011819099999999999 2024-04-22T18:40:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174764+0000", 2024-04-22T18:40:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6729999999999997e-06 2024-04-22T18:40:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174767+0000", 2024-04-22T18:40:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3830000000000001e-06 2024-04-22T18:40:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373669+0000", 2024-04-22T18:40:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198901989 2024-04-22T18:40:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373686+0000", 2024-04-22T18:40:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7286e-05 2024-04-22T18:40:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055408+0000", 2024-04-22T18:40:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68172133400000001 2024-04-22T18:40:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055938+0000", 2024-04-22T18:40:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053016599999999999 2024-04-22T18:40:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453668+0000", 2024-04-22T18:40:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39773030799999998 2024-04-22T18:40:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453717+0000", 2024-04-22T18:40:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8884000000000002e-05 2024-04-22T18:40:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453823+0000", 2024-04-22T18:40:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105791 2024-04-22T18:40:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554715+0000", 2024-04-22T18:40:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100891628 2024-04-22T18:40:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759439+0000", 2024-04-22T18:40:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20472459400000001 2024-04-22T18:40:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759525+0000", 2024-04-22T18:40:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5653000000000001e-05 2024-04-22T18:40:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759682+0000", 2024-04-22T18:40:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015711800000000001 2024-04-22T18:40:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:158 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.301956+0000", 2024-04-22T18:40:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.6212414650000007, 2024-04-22T18:40:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30491982099999998, 2024-04-22T18:40:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.301956+0000", 2024-04-22T18:40:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.301956+0000", 2024-04-22T18:40:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.301953+0000", 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302335+0000", 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038256800000000001 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302351+0000", 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5313000000000001e-05 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302360+0000", 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302441+0000", 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1242000000000006e-05 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302851+0000", 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041006400000000002 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.505313+0000", 2024-04-22T18:40:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.202461317 2024-04-22T18:40:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.505387+0000", 2024-04-22T18:40:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4689000000000003e-05 2024-04-22T18:40:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.505549+0000", 2024-04-22T18:40:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016139799999999999 2024-04-22T18:40:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.605526+0000", 2024-04-22T18:40:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099977658999999997 2024-04-22T18:40:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606733+0000", 2024-04-22T18:40:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012065279999999999 2024-04-22T18:40:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606779+0000", 2024-04-22T18:40:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6516e-05 2024-04-22T18:40:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606876+0000", 2024-04-22T18:40:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6181999999999995e-05 2024-04-22T18:40:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:171 3.2 3:723389a0:::8853427863ea44e2e1f1b69134450b6d67dacc49ab40862e8e5106679d802392:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.303144+0000", 2024-04-22T18:40:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.6200534720000004, 2024-04-22T18:40:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30331065200000001, 2024-04-22T18:40:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303144+0000", 2024-04-22T18:40:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303144+0000", 2024-04-22T18:40:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303143+0000", 2024-04-22T18:40:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303401+0000", 2024-04-22T18:40:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000257581 2024-04-22T18:40:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303408+0000", 2024-04-22T18:40:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7730000000000002e-06 2024-04-22T18:40:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303413+0000", 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6559999999999997e-06 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.307294+0000", 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038801370000000001 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.307501+0000", 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020771200000000001 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.307795+0000", 2024-04-22T18:40:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029333000000000001 2024-04-22T18:40:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404962+0000", 2024-04-22T18:40:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097167373000000001 2024-04-22T18:40:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606305+0000", 2024-04-22T18:40:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20134328500000001 2024-04-22T18:40:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606352+0000", 2024-04-22T18:40:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6774000000000003e-05 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606454+0000", 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102413 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:174 3.2 3:594d3ad2:::b481f33bf204d13d836f094b25e4b106d427bebe44d81ba5fbb696a319414e75:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.303419+0000", 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.6197783149999996, 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.303235174, 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303419+0000", 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303419+0000", 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303418+0000", 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303527+0000", 2024-04-22T18:40:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108925 2024-04-22T18:40:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303535+0000", 2024-04-22T18:40:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2930000000000001e-06 2024-04-22T18:40:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303544+0000", 2024-04-22T18:40:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5159999999999994e-06 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404063+0000", 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100518801 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404178+0000", 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114764 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404352+0000", 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017431200000000001 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.405276+0000", 2024-04-22T18:40:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00092411499999999998 2024-04-22T18:40:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606498+0000", 2024-04-22T18:40:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20122152200000001 2024-04-22T18:40:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606545+0000", 2024-04-22T18:40:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7206000000000002e-05 2024-04-22T18:40:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606654+0000", 2024-04-22T18:40:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109379 2024-04-22T18:40:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:177 3.2 3:6231f5c4:::2915c267dc4ac87f18561ed5fed8de54cbaef6bdd9b80bf698b516d3f8bdd3ee:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e78)", 2024-04-22T18:40:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.334408+0000", 2024-04-22T18:40:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.5887894679999999, 2024-04-22T18:40:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30399278099999999, 2024-04-22T18:40:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334408+0000", 2024-04-22T18:40:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334408+0000", 2024-04-22T18:40:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334404+0000", 2024-04-22T18:40:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999962 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334583+0000", 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000178655 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334603+0000", 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0251e-05 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.434698+0000", 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100094581 2024-04-22T18:40:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.434779+0000", 2024-04-22T18:40:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1528999999999998e-05 2024-04-22T18:40:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.434949+0000", 2024-04-22T18:40:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000170113 2024-04-22T18:40:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.435216+0000", 2024-04-22T18:40:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026647900000000002 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.437050+0000", 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018344450000000001 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.638245+0000", 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20119524499999999 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.638295+0000", 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9994000000000003e-05 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.638401+0000", 2024-04-22T18:40:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105145 2024-04-22T18:40:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:17.038 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:17.039 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:17.040 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:17.040+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:17.140 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:17.141+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:17.152 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:17.152 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:17.152 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:17.152 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:17.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:17.241 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:17.242+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:17.263 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:17.263 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:17.263 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:17.263 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:17.263 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:17.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:17.342 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:17.343+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:17.416 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.253542660999997, 2024-04-22T18:40:17.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:40:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:40:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:40:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:40:17.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:40:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:40:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:40:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:40:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:40:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:40:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:40:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:40:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:40:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:40:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:40:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:40:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:40:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:40:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:40:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:40:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:40:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:40:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:40:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:17.422 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.15198384, 2024-04-22T18:40:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:40:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:40:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:17.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:40:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:40:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:40:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:40:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:40:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:40:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:40:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:40:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:40:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:40:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:40:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:40:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:40:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:40:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:40:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:40:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:40:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:40:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:40:17.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:40:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:40:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.151957437999997, 2024-04-22T18:40:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:40:17.428 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:40:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:40:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:40:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:40:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:40:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:40:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:40:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:40:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:40:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:40:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:40:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:40:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:40:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:40:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:40:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:40:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:40:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:40:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 82.753623109000003, 2024-04-22T18:40:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:40:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.433 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:40:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:40:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:40:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:40:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:40:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:40:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:40:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:40:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:40:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:40:17.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:40:17.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:17.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:40:17.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:40:17.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:40:17.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:40:17.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:40:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:40:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:40:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:40:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:40:17.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:40:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 82.652786810999999, 2024-04-22T18:40:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:40:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.439 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:40:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:40:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:40:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:40:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:40:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:40:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:40:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:40:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:40:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:40:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:40:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:40:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:40:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:40:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:40:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:40:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:40:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:40:17.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:40:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:40:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:40:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:40:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 82.652629540999996, 2024-04-22T18:40:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:40:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:40:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:40:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:40:17.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:40:17.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:40:17.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:40:17.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:40:17.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:40:17.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:40:17.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:40:17.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:40:17.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:40:17.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:17.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:40:17.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:40:17.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:40:17.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:40:17.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:40:17.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:40:17.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:40:17.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:40:17.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:40:17.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:40:17.450 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.450 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.450 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.450 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:17.450 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:17.450 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 82.652614932000006, 2024-04-22T18:40:17.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:40:17.451 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.451 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:17.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.451 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:40:17.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.451 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:17.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:17.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:17.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:40:17.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:40:17.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:40:17.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:17.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:40:17.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:40:17.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:40:17.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:40:17.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:17.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:40:17.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:40:17.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:40:17.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:40:17.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:40:17.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:40:17.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:40:17.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:40:17.455 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.455 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.455 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.455 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:17.455 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:17.455 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 82.652604546999996, 2024-04-22T18:40:17.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:40:17.456 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.456 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.456 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.456 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.456 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.456 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:40:17.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.456 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:17.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:17.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:40:17.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:40:17.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:40:17.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:40:17.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:40:17.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:40:17.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:40:17.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:40:17.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:40:17.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:40:17.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:40:17.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:17.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:40:17.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:40:17.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:40:17.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:40:17.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:40:17.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:40:17.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:40:17.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:40:17.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:40:17.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:40:17.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.461 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.461 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:17.461 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:17.461 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 82.651674616999998, 2024-04-22T18:40:17.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:40:17.461 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.461 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.461 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.461 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.462 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:40:17.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.462 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:17.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:17.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:40:17.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:17.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:40:17.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:40:17.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:40:17.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:40:17.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:17.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:40:17.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:40:17.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:40:17.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:40:17.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:40:17.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:40:17.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:40:17.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:40:17.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:40:17.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:40:17.466 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.466 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.466 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.467 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:17.467 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:17.467 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 82.651212491999999, 2024-04-22T18:40:17.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:40:17.467 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.467 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.467 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.467 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.467 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.467 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:40:17.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.467 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:17.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:17.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:40:17.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:17.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:40:17.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:40:17.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:40:17.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:40:17.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:40:17.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:40:17.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:40:17.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:40:17.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:40:17.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:40:17.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:17.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:40:17.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:40:17.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:40:17.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:40:17.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:40:17.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:40:17.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:40:17.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:40:17.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:40:17.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:40:17.472 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.472 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.472 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.472 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:17.472 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:17.472 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 82.651199688000005, 2024-04-22T18:40:17.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:40:17.473 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.473 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:17.473 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.473 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.473 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.473 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:40:17.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.473 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:17.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:17.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:40:17.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:40:17.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:40:17.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:40:17.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:40:17.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:40:17.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:40:17.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:40:17.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:40:17.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:17.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:40:17.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:40:17.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:40:17.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:40:17.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:40:17.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:40:17.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:40:17.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:40:17.477 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.477 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.477 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.477 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:17.478 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:17.478 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.323427232, 2024-04-22T18:40:17.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:40:17.478 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.478 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.478 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.478 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.478 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.478 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:40:17.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.478 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:17.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:17.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:40:17.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:17.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:40:17.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:40:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:40:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:40:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:40:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:40:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:40:17.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:40:17.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:40:17.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:40:17.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:40:17.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:40:17.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:40:17.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:40:17.483 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.483 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.483 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.483 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:40:17.483 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:17.483 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 78.314854120000007, 2024-04-22T18:40:17.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:40:17.483 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.483 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.483 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.483 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.483 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.483 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:40:17.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.484 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:17.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:17.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:40:17.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:17.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:40:17.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:40:17.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:40:17.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:40:17.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:40:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:40:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:40:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:40:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:40:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:40:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:17.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:40:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:40:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:40:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:40:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:40:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:40:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:40:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:40:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:40:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:40:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:40:17.488 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.280206604999997, 2024-04-22T18:40:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:40:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:40:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:40:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:40:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:40:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:40:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:40:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:40:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:40:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:40:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:40:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:40:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:40:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:40:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:40:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.492 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1684 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.810722642000002, 2024-04-22T18:40:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987885199999998, 2024-04-22T18:40:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1684 2024-04-22T18:40:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.493 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605278+0000", 2024-04-22T18:40:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605280+0000", 2024-04-22T18:40:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1770000000000001e-06 2024-04-22T18:40:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605285+0000", 2024-04-22T18:40:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-04-22T18:40:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605289+0000", 2024-04-22T18:40:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.687e-06 2024-04-22T18:40:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705352+0000", 2024-04-22T18:40:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006322199999999 2024-04-22T18:40:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705423+0000", 2024-04-22T18:40:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1136999999999993e-05 2024-04-22T18:40:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105017+0000", 2024-04-22T18:40:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.399593268 2024-04-22T18:40:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105081+0000", 2024-04-22T18:40:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4703999999999996e-05 2024-04-22T18:40:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105157+0000", 2024-04-22T18:40:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6205999999999995e-05 2024-04-22T18:40:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.497 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2191 4.1 4.fdaa93c1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-04-22T18:40:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 31.866130391999999, 2024-04-22T18:40:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51064384399999996, 2024-04-22T18:40:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2191 2024-04-22T18:40:17.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549872+0000", 2024-04-22T18:40:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.387e-06 2024-04-22T18:40:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549875+0000", 2024-04-22T18:40:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549880+0000", 2024-04-22T18:40:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2529999999999998e-06 2024-04-22T18:40:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060496+0000", 2024-04-22T18:40:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51061597999999997 2024-04-22T18:40:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060515+0000", 2024-04-22T18:40:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9130000000000001e-05 2024-04-22T18:40:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.501 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2198 4.d 4:bc165c76:::smithi08937929-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 31.864862690999999, 2024-04-22T18:40:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50502289899999997, 2024-04-22T18:40:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.502 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2198 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551138+0000", 2024-04-22T18:40:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551143+0000", 2024-04-22T18:40:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8949999999999997e-06 2024-04-22T18:40:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551151+0000", 2024-04-22T18:40:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8729999999999996e-06 2024-04-22T18:40:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551158+0000", 2024-04-22T18:40:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3099999999999997e-06 2024-04-22T18:40:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551180+0000", 2024-04-22T18:40:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2197000000000001e-05 2024-04-22T18:40:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551185+0000", 2024-04-22T18:40:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7010000000000004e-06 2024-04-22T18:40:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657133+0000", 2024-04-22T18:40:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105948498 2024-04-22T18:40:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657183+0000", 2024-04-22T18:40:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0124000000000003e-05 2024-04-22T18:40:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.055924+0000", 2024-04-22T18:40:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39874079299999998 2024-04-22T18:40:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056018+0000", 2024-04-22T18:40:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4032999999999999e-05 2024-04-22T18:40:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056161+0000", 2024-04-22T18:40:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143385 2024-04-22T18:40:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2212 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:17.507 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 31.864793048999999, 2024-04-22T18:40:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50989936400000002, 2024-04-22T18:40:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2212 2024-04-22T18:40:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551211+0000", 2024-04-22T18:40:17.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6280000000000001e-06 2024-04-22T18:40:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551214+0000", 2024-04-22T18:40:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1209999999999998e-06 2024-04-22T18:40:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551217+0000", 2024-04-22T18:40:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8490000000000002e-06 2024-04-22T18:40:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061086+0000", 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50986975099999998 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061108+0000", 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1319e-05 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2213 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 31.864782632000001, 2024-04-22T18:40:17.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50998486099999996, 2024-04-22T18:40:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2213 2024-04-22T18:40:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551218+0000", 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551220+0000", 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.319e-06 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551223+0000", 2024-04-22T18:40:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9660000000000001e-06 2024-04-22T18:40:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551226+0000", 2024-04-22T18:40:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9050000000000001e-06 2024-04-22T18:40:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061162+0000", 2024-04-22T18:40:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.509936838 2024-04-22T18:40:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061204+0000", 2024-04-22T18:40:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1094999999999999e-05 2024-04-22T18:40:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2214 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 31.864773642999999, 2024-04-22T18:40:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50997458100000004, 2024-04-22T18:40:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:17.515 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2214 2024-04-22T18:40:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:17.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551227+0000", 2024-04-22T18:40:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551229+0000", 2024-04-22T18:40:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8649999999999999e-06 2024-04-22T18:40:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551232+0000", 2024-04-22T18:40:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551235+0000", 2024-04-22T18:40:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:40:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061194+0000", 2024-04-22T18:40:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50995819099999995 2024-04-22T18:40:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061202+0000", 2024-04-22T18:40:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.541e-06 2024-04-22T18:40:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.519 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.519 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:17.520 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:17.521 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:17.522+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:17.621 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:17.623+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:17.622 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:17.622 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:17.622 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:17.622 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:17.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:17.721 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:17.722+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:17.735 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:17.735 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:17.735 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:17.735 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:17.735 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:17.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:17.822 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:17.824+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:17.887 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:17.888 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:17.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:17.888 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:17.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.888 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:40:17.888 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:17.888 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.793417722000001, 2024-04-22T18:40:17.888 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:40:17.889 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.889 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.889 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.889 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.889 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.889 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:40:17.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.889 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.889 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.889 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:17.889 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:17.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:40:17.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:17.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.890 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.890 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:40:17.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:40:17.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:40:17.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:40:17.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:40:17.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:40:17.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:40:17.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:40:17.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:40:17.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:40:17.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:17.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:40:17.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:40:17.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:40:17.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:40:17.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:40:17.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:40:17.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:40:17.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:40:17.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:40:17.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:40:17.894 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.894 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.894 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.894 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:17.894 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:17.894 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.893919847000006, 2024-04-22T18:40:17.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:40:17.895 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.895 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.895 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:17.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:17.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:40:17.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:17.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:40:17.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:40:17.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:40:17.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:40:17.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:40:17.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:40:17.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:40:17.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:40:17.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:17.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:40:17.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:40:17.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:40:17.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:40:17.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:40:17.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:40:17.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:17.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:40:17.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:40:17.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:40:17.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:40:17.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:40:17.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:40:17.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:40:17.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:40:17.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:40:17.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:40:17.900 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.900 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.900 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.901 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:40:17.901 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:17.901 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 78.785620055999999, 2024-04-22T18:40:17.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:40:17.901 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.901 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.901 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.901 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.901 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.901 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:40:17.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.901 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:17.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:17.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:40:17.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:17.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:40:17.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:40:17.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:40:17.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:40:17.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:40:17.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:40:17.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:40:17.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:40:17.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:40:17.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:40:17.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:17.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:40:17.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:40:17.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:40:17.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:40:17.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:40:17.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:40:17.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:40:17.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:40:17.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:40:17.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:40:17.906 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.906 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.907 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.907 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:17.907 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:17.907 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.454076427999993, 2024-04-22T18:40:17.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:40:17.907 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.907 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.907 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:17.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:17.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:40:17.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:40:17.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:40:17.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:40:17.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:40:17.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:40:17.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:40:17.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:40:17.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:40:17.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:40:17.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:40:17.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:17.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:40:17.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:40:17.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:40:17.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:40:17.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:40:17.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:40:17.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:40:17.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:40:17.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:40:17.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:40:17.912 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.912 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.912 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.912 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:17.912 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:17.912 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.362990620000005, 2024-04-22T18:40:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:40:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:40:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:17.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:40:17.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:17.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:40:17.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:40:17.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:40:17.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:40:17.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:40:17.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:40:17.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:40:17.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:40:17.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:40:17.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:40:17.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:17.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:40:17.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:40:17.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:40:17.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:40:17.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:40:17.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:40:17.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:40:17.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:40:17.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:40:17.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:40:17.918 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.918 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.918 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.918 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:17.918 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:17.918 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.357190360000004, 2024-04-22T18:40:17.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:40:17.919 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.919 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.919 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.919 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.919 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.919 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:40:17.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.919 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:17.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:17.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:40:17.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:17.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:40:17.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:40:17.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:40:17.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:40:17.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:40:17.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:40:17.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:40:17.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:40:17.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:40:17.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:40:17.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:17.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:40:17.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:40:17.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:40:17.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:40:17.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:40:17.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:40:17.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:40:17.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:40:17.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:40:17.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:40:17.924 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.924 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.924 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.924 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:17.924 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:17.925 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.352450016999995, 2024-04-22T18:40:17.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:40:17.925 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.925 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.925 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.925 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.925 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.925 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:40:17.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.925 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:17.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:17.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:40:17.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:17.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:40:17.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:40:17.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:40:17.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:40:17.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:40:17.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:40:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:40:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:40:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:40:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:40:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:40:17.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:40:17.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:40:17.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:40:17.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:40:17.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:40:17.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:40:17.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:40:17.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:40:17.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:40:17.930 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.930 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.930 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.931 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:17.931 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:17.931 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.351977118999997, 2024-04-22T18:40:17.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:40:17.931 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.931 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.931 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:40:17.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.931 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:17.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:17.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:40:17.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:17.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:40:17.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:40:17.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:40:17.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:40:17.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:40:17.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:40:17.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:40:17.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:40:17.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:40:17.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:40:17.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:17.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:40:17.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:40:17.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:40:17.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:40:17.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:40:17.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:40:17.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:40:17.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:40:17.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:40:17.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:40:17.937 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.937 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.937 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.937 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:17.937 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:17.937 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.351957197999994, 2024-04-22T18:40:17.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:40:17.937 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.938 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:17.938 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.938 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.938 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.938 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:40:17.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.938 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:17.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:17.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:17.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:40:17.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:40:17.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:17.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:40:17.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:40:17.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:40:17.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:40:17.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:40:17.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:40:17.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:40:17.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:40:17.942 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.942 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.942 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.942 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:17.942 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:17.942 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.042633432999999, 2024-04-22T18:40:17.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:40:17.942 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.943 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.943 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:17.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:17.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:40:17.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:17.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:40:17.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:40:17.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:40:17.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:40:17.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:40:17.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:40:17.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:40:17.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:40:17.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:40:17.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:40:17.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:17.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:40:17.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:40:17.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:40:17.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:40:17.947 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.042425897000001, 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:17.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:40:17.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:17.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:40:17.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:40:17.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:40:17.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:40:17.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:40:17.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:40:17.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:40:17.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:40:17.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:40:17.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:40:17.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:17.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:40:17.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:40:17.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:40:17.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:40:17.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:40:17.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:40:17.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:40:17.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:40:17.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:40:17.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:40:17.953 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.953 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.953 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.954 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:17.954 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:17.954 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.176776164000003, 2024-04-22T18:40:17.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:40:17.954 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.954 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.954 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:17.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:17.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:40:17.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:17.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:40:17.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:40:17.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:40:17.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:40:17.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:40:17.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:40:17.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:40:17.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:40:17.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:40:17.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:40:17.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:17.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:40:17.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:40:17.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:40:17.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:40:17.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:40:17.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:40:17.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:40:17.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:40:17.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:40:17.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:40:17.960 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.960 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.960 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.960 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:17.960 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:17.960 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.962937779000001, 2024-04-22T18:40:17.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:40:17.960 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.960 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.960 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:17.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:17.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:40:17.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:17.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:40:17.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:40:17.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:40:17.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:40:17.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:40:17.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:40:17.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:40:17.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:40:17.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:40:17.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:40:17.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:17.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:40:17.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:40:17.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:40:17.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:40:17.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:40:17.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:40:17.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:40:17.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:40:17.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:40:17.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:40:17.965 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.965 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.965 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.965 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:17.965 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:17.965 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.962904369, 2024-04-22T18:40:17.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:40:17.966 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.966 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.966 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:17.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:17.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:17.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:40:17.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:40:17.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:40:17.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:40:17.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:40:17.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:40:17.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:40:17.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:40:17.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:40:17.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:40:17.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:17.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:40:17.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:40:17.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:40:17.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:40:17.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:40:17.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:40:17.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:40:17.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:40:17.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:40:17.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:40:17.971 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.971 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.971 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.971 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:17.971 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:17.971 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.904021714999999, 2024-04-22T18:40:17.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:40:17.971 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.971 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.971 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:17.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:17.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:40:17.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:17.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:40:17.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:40:17.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:40:17.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:40:17.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:40:17.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:40:17.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:40:17.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:40:17.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:40:17.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:40:17.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:17.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:40:17.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:40:17.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:40:17.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:40:17.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:40:17.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:40:17.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:40:17.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:40:17.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:40:17.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:40:17.976 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.976 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.976 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.976 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:17.976 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:17.976 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.903883098999998, 2024-04-22T18:40:17.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:40:17.977 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.977 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.977 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:17.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:17.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:17.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:40:17.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:40:17.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:40:17.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:40:17.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:40:17.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:40:17.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:40:17.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:40:17.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:40:17.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:40:17.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:17.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:40:17.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:40:17.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:40:17.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:40:17.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:40:17.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:40:17.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:40:17.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:40:17.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:40:17.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:40:17.982 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.982 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.982 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.982 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:17.982 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:17.982 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.903864096, 2024-04-22T18:40:17.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:40:17.982 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.982 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.983 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:17.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:17.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:17.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:17.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:40:17.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:40:17.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:40:17.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:40:17.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:40:17.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:40:17.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:40:17.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:40:17.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:40:17.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:40:17.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:17.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:40:17.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:40:17.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:40:17.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:40:17.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:40:17.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:40:17.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:40:17.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:40:17.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:40:17.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:40:17.987 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.987 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.987 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.282851563000001, 2024-04-22T18:40:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:40:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:40:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.988 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:40:17.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:40:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:40:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:40:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:40:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:40:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:40:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:40:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:40:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:40:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:40:17.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:40:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:40:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:40:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:40:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:40:17.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:40:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:40:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:40:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:40:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:40:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.282331972000001, 2024-04-22T18:40:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:40:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:40:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:17.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:40:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:40:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:40:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:40:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:40:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:40:17.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:40:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:40:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:40:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:40:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:40:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:40:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:40:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:40:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:40:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:40:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:40:17.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:40:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:40:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:40:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:40:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:17.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.182179089000002, 2024-04-22T18:40:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:40:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:40:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:40:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:40:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:40:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:40:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:40:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:40:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:40:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:40:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:40:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:40:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:40:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:40:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:40:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:40:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:40:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:40:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:40:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.004 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.005 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.005 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:18.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:18.007 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:18.007+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:18.106 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:18.108+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:18.148 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:18.148 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:18.148 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:18.148 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:18.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:18.207 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:18.208+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:18.257 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:18.258 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:18.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:18.307 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:18.309+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:18.408 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:18.409+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:18.409 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:18.409 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:18.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:18.409 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:18.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.410 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:22 4.a 4:5aa97f94:::smithi08937929-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:18.410 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:40:18.410 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.252099760999997, 2024-04-22T18:40:18.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30948398900000001, 2024-04-22T18:40:18.410 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.410 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.410 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.410 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 22 2024-04-22T18:40:18.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.411 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:40:18.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:40:18.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157259+0000", 2024-04-22T18:40:18.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:18.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157318+0000", 2024-04-22T18:40:18.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8906000000000001e-05 2024-04-22T18:40:18.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157327+0000", 2024-04-22T18:40:18.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:40:18.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157332+0000", 2024-04-22T18:40:18.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4689999999999996e-06 2024-04-22T18:40:18.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257479+0000", 2024-04-22T18:40:18.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014700999999999 2024-04-22T18:40:18.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:18.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257492+0000", 2024-04-22T18:40:18.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2574e-05 2024-04-22T18:40:18.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259582+0000", 2024-04-22T18:40:18.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020904019999999999 2024-04-22T18:40:18.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259668+0000", 2024-04-22T18:40:18.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5826000000000001e-05 2024-04-22T18:40:18.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:18.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259772+0000", 2024-04-22T18:40:18.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104048 2024-04-22T18:40:18.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363453+0000", 2024-04-22T18:40:18.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103680183 2024-04-22T18:40:18.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466692+0000", 2024-04-22T18:40:18.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10323948199999999 2024-04-22T18:40:18.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466720+0000", 2024-04-22T18:40:18.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7886e-05 2024-04-22T18:40:18.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466744+0000", 2024-04-22T18:40:18.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4049e-05 2024-04-22T18:40:18.417 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.417 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.417 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.417 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:23 4.a 4:5aa97f94:::smithi08937929-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:18.417 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:40:18.417 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.252022073999996, 2024-04-22T18:40:18.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39003820300000003, 2024-04-22T18:40:18.417 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.417 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.418 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.418 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.418 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.418 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 23 2024-04-22T18:40:18.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.418 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:40:18.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:40:18.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157337+0000", 2024-04-22T18:40:18.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157340+0000", 2024-04-22T18:40:18.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8609999999999998e-06 2024-04-22T18:40:18.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157343+0000", 2024-04-22T18:40:18.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9859999999999999e-06 2024-04-22T18:40:18.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157346+0000", 2024-04-22T18:40:18.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8119999999999999e-06 2024-04-22T18:40:18.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257563+0000", 2024-04-22T18:40:18.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100217497 2024-04-22T18:40:18.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:18.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257574+0000", 2024-04-22T18:40:18.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0777e-05 2024-04-22T18:40:18.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260112+0000", 2024-04-22T18:40:18.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002537681 2024-04-22T18:40:18.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260191+0000", 2024-04-22T18:40:18.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9320999999999995e-05 2024-04-22T18:40:18.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:18.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260333+0000", 2024-04-22T18:40:18.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014177699999999999 2024-04-22T18:40:18.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363792+0000", 2024-04-22T18:40:18.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103458862 2024-04-22T18:40:18.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547303+0000", 2024-04-22T18:40:18.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18351162100000001 2024-04-22T18:40:18.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547337+0000", 2024-04-22T18:40:18.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3640999999999999e-05 2024-04-22T18:40:18.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547376+0000", 2024-04-22T18:40:18.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8915000000000003e-05 2024-04-22T18:40:18.423 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.423 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.423 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.424 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:24 4.a 4:5aa97f94:::smithi08937929-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:18.424 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:18.424 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.251995643000001, 2024-04-22T18:40:18.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39017823499999998, 2024-04-22T18:40:18.424 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.424 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:18.424 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.424 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.424 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.424 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 24 2024-04-22T18:40:18.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.425 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:18.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:18.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:18.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157366+0000", 2024-04-22T18:40:18.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3120000000000001e-06 2024-04-22T18:40:18.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157370+0000", 2024-04-22T18:40:18.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1300000000000003e-06 2024-04-22T18:40:18.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157373+0000", 2024-04-22T18:40:18.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-04-22T18:40:18.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260553+0000", 2024-04-22T18:40:18.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103179996 2024-04-22T18:40:18.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:18.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260625+0000", 2024-04-22T18:40:18.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2236000000000004e-05 2024-04-22T18:40:18.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547414+0000", 2024-04-22T18:40:18.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.286789181 2024-04-22T18:40:18.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547471+0000", 2024-04-22T18:40:18.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6192e-05 2024-04-22T18:40:18.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547542+0000", 2024-04-22T18:40:18.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1622e-05 2024-04-22T18:40:18.429 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.429 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.429 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.429 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:49 4.a 4:5d2aee44:::smithi08937929-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:18.429 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:40:18.429 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.175196984999999, 2024-04-22T18:40:18.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31344442500000003, 2024-04-22T18:40:18.429 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.429 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.429 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.429 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.430 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.430 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 49 2024-04-22T18:40:18.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.430 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:40:18.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:40:18.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234158+0000", 2024-04-22T18:40:18.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999957 2024-04-22T18:40:18.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234901+0000", 2024-04-22T18:40:18.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074250800000000001 2024-04-22T18:40:18.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234914+0000", 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3244999999999999e-05 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234922+0000", 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4590000000000001e-06 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260665+0000", 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025743423000000001 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260779+0000", 2024-04-22T18:40:18.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114063 2024-04-22T18:40:18.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:18.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260913+0000", 2024-04-22T18:40:18.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013355499999999999 2024-04-22T18:40:18.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363798+0000", 2024-04-22T18:40:18.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102885003 2024-04-22T18:40:18.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547567+0000", 2024-04-22T18:40:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.183768971 2024-04-22T18:40:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547589+0000", 2024-04-22T18:40:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2507999999999999e-05 2024-04-22T18:40:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547607+0000", 2024-04-22T18:40:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8009999999999999e-05 2024-04-22T18:40:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.435 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:50 4.a 4:5d2aee44:::smithi08937929-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:18.435 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:18.435 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.169750805999996, 2024-04-22T18:40:18.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40856295399999998, 2024-04-22T18:40:18.435 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.435 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.435 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.435 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.435 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.435 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 50 2024-04-22T18:40:18.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.435 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:18.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:18.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239607+0000", 2024-04-22T18:40:18.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:18.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240309+0000", 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070201599999999997 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240320+0000", 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1024e-05 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240327+0000", 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0049999999999998e-06 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362065+0000", 2024-04-22T18:40:18.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.121737812 2024-04-22T18:40:18.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362121+0000", 2024-04-22T18:40:18.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5868000000000001e-05 2024-04-22T18:40:18.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:18.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362210+0000", 2024-04-22T18:40:18.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8533000000000001e-05 2024-04-22T18:40:18.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365322+0000", 2024-04-22T18:40:18.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031121220000000002 2024-04-22T18:40:18.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648054+0000", 2024-04-22T18:40:18.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.282732549 2024-04-22T18:40:18.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648105+0000", 2024-04-22T18:40:18.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1276999999999997e-05 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648172+0000", 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6398999999999999e-05 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:51 4.a 4:5d2aee44:::smithi08937929-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.169028272000006, 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40793057199999999, 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.440 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.441 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.441 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.441 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.441 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 51 2024-04-22T18:40:18.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.441 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:18.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:18.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:18.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240336+0000", 2024-04-22T18:40:18.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:40:18.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240339+0000", 2024-04-22T18:40:18.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9009999999999998e-06 2024-04-22T18:40:18.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240342+0000", 2024-04-22T18:40:18.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:40:18.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363080+0000", 2024-04-22T18:40:18.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122738075 2024-04-22T18:40:18.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363164+0000", 2024-04-22T18:40:18.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3722999999999998e-05 2024-04-22T18:40:18.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:18.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363287+0000", 2024-04-22T18:40:18.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012301099999999999 2024-04-22T18:40:18.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466753+0000", 2024-04-22T18:40:18.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103466217 2024-04-22T18:40:18.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648205+0000", 2024-04-22T18:40:18.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.181451788 2024-04-22T18:40:18.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648231+0000", 2024-04-22T18:40:18.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6597999999999998e-05 2024-04-22T18:40:18.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.446 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648262+0000", 2024-04-22T18:40:18.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0871000000000003e-05 2024-04-22T18:40:18.446 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.446 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.446 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.446 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:52 4.a 4:5d2aee44:::smithi08937929-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:18.446 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:18.446 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.169015813000001, 2024-04-22T18:40:18.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40812731200000002, 2024-04-22T18:40:18.446 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.446 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:18.446 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.447 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.447 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.447 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 52 2024-04-22T18:40:18.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.447 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:18.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:18.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:18.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240345+0000", 2024-04-22T18:40:18.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4300000000000001e-06 2024-04-22T18:40:18.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240349+0000", 2024-04-22T18:40:18.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2570000000000001e-06 2024-04-22T18:40:18.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240358+0000", 2024-04-22T18:40:18.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-22T18:40:18.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363697+0000", 2024-04-22T18:40:18.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12333965299999999 2024-04-22T18:40:18.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:18.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363777+0000", 2024-04-22T18:40:18.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9883999999999996e-05 2024-04-22T18:40:18.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648296+0000", 2024-04-22T18:40:18.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451861299999998 2024-04-22T18:40:18.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648379+0000", 2024-04-22T18:40:18.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3302999999999998e-05 2024-04-22T18:40:18.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648471+0000", 2024-04-22T18:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2029999999999998e-05 2024-04-22T18:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.149712477999998, 2024-04-22T18:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.452 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.452 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:40:18.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.452 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:18.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:18.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:40:18.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:18.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:40:18.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:40:18.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:40:18.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:40:18.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:40:18.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:40:18.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:40:18.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:40:18.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:40:18.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:40:18.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:18.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:40:18.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:40:18.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:40:18.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:40:18.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:40:18.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:40:18.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:40:18.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:40:18.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:40:18.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:40:18.456 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.456 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.456 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.457 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:18.457 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:18.457 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.144930048000006, 2024-04-22T18:40:18.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:40:18.457 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.457 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.457 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.457 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.457 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.457 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:40:18.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.457 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:18.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:18.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:40:18.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:18.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:40:18.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:40:18.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:40:18.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:40:18.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:40:18.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:40:18.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:40:18.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:40:18.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:40:18.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:40:18.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:18.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:40:18.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:40:18.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:40:18.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:40:18.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:40:18.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:40:18.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:40:18.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:40:18.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:40:18.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:40:18.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.462 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.463 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:18.463 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:18.463 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.144835845000003, 2024-04-22T18:40:18.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:40:18.463 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.463 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.463 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.463 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.463 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.463 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:40:18.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.463 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:18.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:18.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:40:18.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:18.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:40:18.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:40:18.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:40:18.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:40:18.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:40:18.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:40:18.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:40:18.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:40:18.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:40:18.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:40:18.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:18.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:40:18.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:40:18.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:40:18.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:40:18.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:40:18.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:40:18.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:40:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:40:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:40:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:40:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.144814956999994, 2024-04-22T18:40:18.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:40:18.469 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.469 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:18.469 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.469 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.469 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.469 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:40:18.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.469 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:18.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:18.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:40:18.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:18.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:40:18.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:40:18.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:40:18.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:40:18.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:40:18.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:40:18.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:40:18.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:40:18.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:40:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:40:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:40:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:40:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:40:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:40:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:40:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:40:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.361807294000002, 2024-04-22T18:40:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:40:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:40:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:40:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:40:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:40:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:40:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:40:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:40:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:40:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:40:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:40:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:40:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:40:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:40:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:40:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:40:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:40:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:40:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:40:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:40:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:40:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:40:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:40:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.353975887999994, 2024-04-22T18:40:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:40:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:40:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:40:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:40:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:40:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:40:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:40:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:40:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:40:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:40:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:40:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:40:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:40:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:40:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:40:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:40:18.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:40:18.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:40:18.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:40:18.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:40:18.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:40:18.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:40:18.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:40:18.485 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.485 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.485 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.485 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2052 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:18.485 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:18.485 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.831864107000001, 2024-04-22T18:40:18.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43058958600000002, 2024-04-22T18:40:18.485 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.486 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.486 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.486 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.486 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.486 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2052 2024-04-22T18:40:18.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.486 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:18.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:18.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577494+0000", 2024-04-22T18:40:18.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:18.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577500+0000", 2024-04-22T18:40:18.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3279999999999996e-06 2024-04-22T18:40:18.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577510+0000", 2024-04-22T18:40:18.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9450000000000005e-06 2024-04-22T18:40:18.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577515+0000", 2024-04-22T18:40:18.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7799999999999997e-06 2024-04-22T18:40:18.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677622+0000", 2024-04-22T18:40:18.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010654099999999 2024-04-22T18:40:18.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677710+0000", 2024-04-22T18:40:18.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8472000000000003e-05 2024-04-22T18:40:18.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:18.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.906619+0000", 2024-04-22T18:40:18.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.228908519 2024-04-22T18:40:18.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907343+0000", 2024-04-22T18:40:18.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072442600000000002 2024-04-22T18:40:18.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.007972+0000", 2024-04-22T18:40:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100628781 2024-04-22T18:40:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008022+0000", 2024-04-22T18:40:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9843999999999999e-05 2024-04-22T18:40:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008085+0000", 2024-04-22T18:40:18.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3256999999999994e-05 2024-04-22T18:40:18.491 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.491 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.491 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.491 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2087 4.a 4:537c89f8:::smithi08937929-47:head [assert-version v37,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-04-22T18:40:18.491 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:40:18.491 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.501947346999998, 2024-04-22T18:40:18.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38904324299999998, 2024-04-22T18:40:18.491 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.491 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.491 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.491 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.491 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.492 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2087 2024-04-22T18:40:18.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.492 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:40:18.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:40:18.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907410+0000", 2024-04-22T18:40:18.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:18.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907417+0000", 2024-04-22T18:40:18.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.306e-06 2024-04-22T18:40:18.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907427+0000", 2024-04-22T18:40:18.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4930000000000004e-06 2024-04-22T18:40:18.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907432+0000", 2024-04-22T18:40:18.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5400000000000003e-06 2024-04-22T18:40:18.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907477+0000", 2024-04-22T18:40:18.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4851999999999997e-05 2024-04-22T18:40:18.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907537+0000", 2024-04-22T18:40:18.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9460999999999999e-05 2024-04-22T18:40:18.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:18.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.295587+0000", 2024-04-22T18:40:18.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38805073200000001 2024-04-22T18:40:18.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296281+0000", 2024-04-22T18:40:18.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00069370800000000002 2024-04-22T18:40:18.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296400+0000", 2024-04-22T18:40:18.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118579 2024-04-22T18:40:18.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296431+0000", 2024-04-22T18:40:18.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1627000000000001e-05 2024-04-22T18:40:18.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296456+0000", 2024-04-22T18:40:18.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4331000000000001e-05 2024-04-22T18:40:18.496 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.496 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.497 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.497 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:18.497 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:18.497 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.202103258000001, 2024-04-22T18:40:18.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:40:18.497 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.497 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.497 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.497 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.497 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.497 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:40:18.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.497 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:18.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:18.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:40:18.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:18.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:40:18.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:40:18.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:40:18.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:40:18.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:40:18.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:40:18.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:40:18.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:40:18.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:40:18.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:40:18.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:18.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:40:18.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:40:18.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:40:18.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:40:18.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:40:18.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:40:18.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:40:18.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:40:18.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:40:18.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:40:18.502 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.502 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.502 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.502 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2505 4.5 4:a6d237d2:::smithi08937929-43:head [setxattr _header (56) in=7b,truncate 3529470] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:18.502 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:18.503 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.201710562999999, 2024-04-22T18:40:18.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43706744400000003, 2024-04-22T18:40:18.503 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.503 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.503 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.503 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.503 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.503 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2505 2024-04-22T18:40:18.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.503 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:18.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:18.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207648+0000", 2024-04-22T18:40:18.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:18.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207653+0000", 2024-04-22T18:40:18.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2989999999999998e-06 2024-04-22T18:40:18.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207656+0000", 2024-04-22T18:40:18.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0199999999999999e-06 2024-04-22T18:40:18.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207659+0000", 2024-04-22T18:40:18.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2270000000000002e-06 2024-04-22T18:40:18.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243213+0000", 2024-04-22T18:40:18.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035553653999999997 2024-04-22T18:40:18.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243273+0000", 2024-04-22T18:40:18.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0300000000000002e-05 2024-04-22T18:40:18.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:18.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243359+0000", 2024-04-22T18:40:18.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6275999999999999e-05 2024-04-22T18:40:18.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297443+0000", 2024-04-22T18:40:18.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.054083930000000002 2024-04-22T18:40:18.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644651+0000", 2024-04-22T18:40:18.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.347208181 2024-04-22T18:40:18.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644684+0000", 2024-04-22T18:40:18.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3080999999999997e-05 2024-04-22T18:40:18.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644717+0000", 2024-04-22T18:40:18.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2264000000000001e-05 2024-04-22T18:40:18.508 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.508 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.508 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.508 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:18.508 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:18.508 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.201698350000001, 2024-04-22T18:40:18.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:40:18.509 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.509 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:18.509 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.509 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.509 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.509 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:40:18.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.509 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:18.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:18.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:18.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:40:18.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:40:18.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:40:18.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:40:18.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:40:18.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:40:18.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:40:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:40:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:40:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:40:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:40:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:40:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2615 4.5 4:a6d237d2:::smithi08937929-43:head [write 463926~462166] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:58.352901+0000", 2024-04-22T18:40:18.513 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.056458901999999, 2024-04-22T18:40:18.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.307052514, 2024-04-22T18:40:18.514 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.514 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.514 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.514 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.514 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.514 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2615 2024-04-22T18:40:18.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.514 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.352901+0000", 2024-04-22T18:40:18.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.352901+0000", 2024-04-22T18:40:18.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.352897+0000", 2024-04-22T18:40:18.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999962 2024-04-22T18:40:18.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.353421+0000", 2024-04-22T18:40:18.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052391999999999998 2024-04-22T18:40:18.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.353433+0000", 2024-04-22T18:40:18.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2128e-05 2024-04-22T18:40:18.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.353460+0000", 2024-04-22T18:40:18.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7211000000000001e-05 2024-04-22T18:40:18.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.453670+0000", 2024-04-22T18:40:18.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10020929100000001 2024-04-22T18:40:18.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.453809+0000", 2024-04-22T18:40:18.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013967800000000001 2024-04-22T18:40:18.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:18.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.454021+0000", 2024-04-22T18:40:18.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021190900000000001 2024-04-22T18:40:18.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659805+0000", 2024-04-22T18:40:18.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20578357999999999 2024-04-22T18:40:18.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659886+0000", 2024-04-22T18:40:18.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1310000000000006e-05 2024-04-22T18:40:18.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659925+0000", 2024-04-22T18:40:18.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9233000000000002e-05 2024-04-22T18:40:18.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659953+0000", 2024-04-22T18:40:18.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7894000000000001e-05 2024-04-22T18:40:18.519 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.519 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.519 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.519 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2715 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:18.519 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:03.692698+0000", 2024-04-22T18:40:18.519 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.716662008, 2024-04-22T18:40:18.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40343488, 2024-04-22T18:40:18.519 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.520 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:18.520 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.520 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:18.520 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:18.520 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2715 2024-04-22T18:40:18.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.520 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.692698+0000", 2024-04-22T18:40:18.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.692698+0000", 2024-04-22T18:40:18.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.692697+0000", 2024-04-22T18:40:18.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.692700+0000", 2024-04-22T18:40:18.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3410000000000001e-06 2024-04-22T18:40:18.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.692703+0000", 2024-04-22T18:40:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7230000000000002e-06 2024-04-22T18:40:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.792867+0000", 2024-04-22T18:40:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10016380499999999 2024-04-22T18:40:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.793431+0000", 2024-04-22T18:40:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056344299999999995 2024-04-22T18:40:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.793496+0000", 2024-04-22T18:40:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5415999999999998e-05 2024-04-22T18:40:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:04.095990+0000", 2024-04-22T18:40:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30249418099999997 2024-04-22T18:40:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:04.096045+0000", 2024-04-22T18:40:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4486e-05 2024-04-22T18:40:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:04.096133+0000", 2024-04-22T18:40:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7956999999999998e-05 2024-04-22T18:40:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.524 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:18.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:18.526 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:18.527+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:18.627 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:18.628+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:18.655 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:40:18.667 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:18.667 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:18.667 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:18.667 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:18.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:18.720 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:18.722+0000 7f309102d640 -1 osd.1 87 handle_osd_map injecting map crc failure 2024-04-22T18:40:18.727 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:18.729+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:18.781 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:18.781 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:18.782 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:18.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:18.831 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:40:18.833 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:18.834+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:18.893 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 99.151416878000006, 2024-04-22T18:40:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:40:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.894 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:40:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:40:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:40:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:40:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.896 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:40:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:40:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:40:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:40:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:40:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:40:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:40:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:40:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:40:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:40:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:40:18.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:40:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:40:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:40:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:40:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:40:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:40:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:40:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 99.151377186999994, 2024-04-22T18:40:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:40:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.900 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:40:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:40:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:40:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:40:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:40:18.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:40:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:40:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:40:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:40:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:40:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:40:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:40:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:40:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:40:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:40:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:40:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:40:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:40:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:40:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:40:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:40:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:40:18.905 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:40:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.907196139999996, 2024-04-22T18:40:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402419109, 2024-04-22T18:40:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.906 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.907 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 68 2024-04-22T18:40:18.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.907 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:40:18.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:40:18.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986204+0000", 2024-04-22T18:40:18.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986208+0000", 2024-04-22T18:40:18.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8149999999999999e-06 2024-04-22T18:40:18.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986214+0000", 2024-04-22T18:40:18.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.604e-06 2024-04-22T18:40:18.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986218+0000", 2024-04-22T18:40:18.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6689999999999999e-06 2024-04-22T18:40:18.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986833+0000", 2024-04-22T18:40:18.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000615648 2024-04-22T18:40:18.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986891+0000", 2024-04-22T18:40:18.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7915999999999999e-05 2024-04-22T18:40:18.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986981+0000", 2024-04-22T18:40:18.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759999999999994e-05 2024-04-22T18:40:18.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.987584+0000", 2024-04-22T18:40:18.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060270199999999999 2024-04-22T18:40:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388527+0000", 2024-04-22T18:40:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40094366999999997 2024-04-22T18:40:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388583+0000", 2024-04-22T18:40:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5383000000000002e-05 2024-04-22T18:40:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388624+0000", 2024-04-22T18:40:18.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501999999999997e-05 2024-04-22T18:40:18.912 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.912 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.912 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.912 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:18.912 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:18.912 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.907180350999994, 2024-04-22T18:40:18.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:40:18.912 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.913 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.913 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.913 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.913 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.913 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:40:18.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.913 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:18.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:18.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:40:18.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:40:18.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:40:18.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:40:18.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:40:18.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:40:18.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:40:18.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:40:18.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:40:18.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:40:18.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:40:18.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:40:18.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:40:18.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:40:18.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:40:18.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:40:18.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:40:18.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:40:18.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:40:18.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.917 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.917 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:40:18.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:40:18.917 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.918 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.918 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.918 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:18.918 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:40:18.918 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 95.790787342000002, 2024-04-22T18:40:18.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305563733, 2024-04-22T18:40:18.918 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.918 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.919 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.919 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.919 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.919 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 87 2024-04-22T18:40:18.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.919 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:40:18.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:40:18.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102613+0000", 2024-04-22T18:40:18.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:18.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102668+0000", 2024-04-22T18:40:18.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4669000000000001e-05 2024-04-22T18:40:18.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102680+0000", 2024-04-22T18:40:18.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2026e-05 2024-04-22T18:40:18.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102687+0000", 2024-04-22T18:40:18.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8339999999999999e-06 2024-04-22T18:40:18.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405644+0000", 2024-04-22T18:40:18.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30295623199999999 2024-04-22T18:40:18.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405743+0000", 2024-04-22T18:40:18.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.98e-05 2024-04-22T18:40:18.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405868+0000", 2024-04-22T18:40:18.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012457899999999999 2024-04-22T18:40:18.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407047+0000", 2024-04-22T18:40:18.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011791449999999999 2024-04-22T18:40:18.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408088+0000", 2024-04-22T18:40:18.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010411680000000001 2024-04-22T18:40:18.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408135+0000", 2024-04-22T18:40:18.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6255999999999999e-05 2024-04-22T18:40:18.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408178+0000", 2024-04-22T18:40:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2877000000000003e-05 2024-04-22T18:40:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:40:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 95.790707413000007, 2024-04-22T18:40:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30560089000000001, 2024-04-22T18:40:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.924 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 88 2024-04-22T18:40:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:40:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:40:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102693+0000", 2024-04-22T18:40:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102698+0000", 2024-04-22T18:40:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9699999999999998e-06 2024-04-22T18:40:18.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102708+0000", 2024-04-22T18:40:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.006e-05 2024-04-22T18:40:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102715+0000", 2024-04-22T18:40:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6599999999999995e-06 2024-04-22T18:40:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406637+0000", 2024-04-22T18:40:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392130099999998 2024-04-22T18:40:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406696+0000", 2024-04-22T18:40:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9066e-05 2024-04-22T18:40:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406784+0000", 2024-04-22T18:40:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8194000000000003e-05 2024-04-22T18:40:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.928 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407447+0000", 2024-04-22T18:40:18.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066295700000000004 2024-04-22T18:40:18.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408237+0000", 2024-04-22T18:40:18.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079032600000000005 2024-04-22T18:40:18.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408270+0000", 2024-04-22T18:40:18.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3157000000000001e-05 2024-04-22T18:40:18.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408295+0000", 2024-04-22T18:40:18.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4151999999999998e-05 2024-04-22T18:40:18.930 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.930 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.930 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.930 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:18.930 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:40:18.930 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.039475288999995, 2024-04-22T18:40:18.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30533819600000001, 2024-04-22T18:40:18.930 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.930 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.931 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 106 2024-04-22T18:40:18.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.931 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:40:18.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:40:18.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853925+0000", 2024-04-22T18:40:18.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853958+0000", 2024-04-22T18:40:18.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2536000000000002e-05 2024-04-22T18:40:18.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853964+0000", 2024-04-22T18:40:18.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6239999999999999e-06 2024-04-22T18:40:18.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853967+0000", 2024-04-22T18:40:18.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9700000000000001e-06 2024-04-22T18:40:18.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855144+0000", 2024-04-22T18:40:18.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001176774 2024-04-22T18:40:18.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855225+0000", 2024-04-22T18:40:18.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0697000000000003e-05 2024-04-22T18:40:18.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855353+0000", 2024-04-22T18:40:18.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012788 2024-04-22T18:40:18.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058003+0000", 2024-04-22T18:40:18.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202649939 2024-04-22T18:40:18.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159184+0000", 2024-04-22T18:40:18.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101181058 2024-04-22T18:40:18.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159238+0000", 2024-04-22T18:40:18.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3967e-05 2024-04-22T18:40:18.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159264+0000", 2024-04-22T18:40:18.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6237999999999998e-05 2024-04-22T18:40:18.936 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.936 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.936 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.936 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:18.936 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:40:18.936 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.039428439000005, 2024-04-22T18:40:18.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30544294500000002, 2024-04-22T18:40:18.936 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.936 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.936 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.937 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 107 2024-04-22T18:40:18.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.937 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:40:18.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:40:18.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853972+0000", 2024-04-22T18:40:18.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853975+0000", 2024-04-22T18:40:18.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0939999999999999e-06 2024-04-22T18:40:18.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853978+0000", 2024-04-22T18:40:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:40:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853981+0000", 2024-04-22T18:40:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.858e-06 2024-04-22T18:40:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855831+0000", 2024-04-22T18:40:18.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018503689999999999 2024-04-22T18:40:18.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855932+0000", 2024-04-22T18:40:18.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010071 2024-04-22T18:40:18.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856029+0000", 2024-04-22T18:40:18.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6751999999999998e-05 2024-04-22T18:40:18.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058013+0000", 2024-04-22T18:40:18.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20198412600000001 2024-04-22T18:40:18.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159326+0000", 2024-04-22T18:40:18.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101312969 2024-04-22T18:40:18.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159376+0000", 2024-04-22T18:40:18.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9968e-05 2024-04-22T18:40:18.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159416+0000", 2024-04-22T18:40:18.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9641999999999998e-05 2024-04-22T18:40:18.942 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.942 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.942 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.942 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:18.942 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:18.942 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.039340042999996, 2024-04-22T18:40:18.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30548257499999998, 2024-04-22T18:40:18.942 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.942 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.943 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.943 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.943 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.943 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 108 2024-04-22T18:40:18.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.943 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:18.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:18.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:18.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854064+0000", 2024-04-22T18:40:18.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5820000000000002e-06 2024-04-22T18:40:18.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854080+0000", 2024-04-22T18:40:18.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5379999999999998e-05 2024-04-22T18:40:18.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854084+0000", 2024-04-22T18:40:18.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.963e-06 2024-04-22T18:40:18.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856413+0000", 2024-04-22T18:40:18.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023298070000000001 2024-04-22T18:40:18.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856494+0000", 2024-04-22T18:40:18.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0863000000000002e-05 2024-04-22T18:40:18.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856624+0000", 2024-04-22T18:40:18.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129799 2024-04-22T18:40:18.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058947+0000", 2024-04-22T18:40:18.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20232328899999999 2024-04-22T18:40:18.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159468+0000", 2024-04-22T18:40:18.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100520597 2024-04-22T18:40:18.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159508+0000", 2024-04-22T18:40:18.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9830000000000003e-05 2024-04-22T18:40:18.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159544+0000", 2024-04-22T18:40:18.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5944e-05 2024-04-22T18:40:18.947 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.948 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.948 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.948 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:18.948 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:40:18.948 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.039295550999995, 2024-04-22T18:40:18.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30556502499999999, 2024-04-22T18:40:18.948 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.948 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.948 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.948 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.948 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.948 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:40:18.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.949 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:40:18.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:40:18.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854105+0000", 2024-04-22T18:40:18.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854175+0000", 2024-04-22T18:40:18.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0178000000000005e-05 2024-04-22T18:40:18.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854191+0000", 2024-04-22T18:40:18.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5965999999999999e-05 2024-04-22T18:40:18.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854196+0000", 2024-04-22T18:40:18.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.42e-06 2024-04-22T18:40:18.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956526+0000", 2024-04-22T18:40:18.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10233094299999999 2024-04-22T18:40:18.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956634+0000", 2024-04-22T18:40:18.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107398 2024-04-22T18:40:18.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956730+0000", 2024-04-22T18:40:18.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6217000000000004e-05 2024-04-22T18:40:18.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058955+0000", 2024-04-22T18:40:18.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102224749 2024-04-22T18:40:18.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159596+0000", 2024-04-22T18:40:18.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100641297 2024-04-22T18:40:18.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159637+0000", 2024-04-22T18:40:18.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874000000000002e-05 2024-04-22T18:40:18.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159671+0000", 2024-04-22T18:40:18.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3603999999999999e-05 2024-04-22T18:40:18.953 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.953 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.954 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.954 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:18.954 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:18.954 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.039201704000007, 2024-04-22T18:40:18.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30683048800000001, 2024-04-22T18:40:18.954 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.954 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.954 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.954 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.954 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.954 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110 2024-04-22T18:40:18.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.955 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:18.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:18.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:18.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854202+0000", 2024-04-22T18:40:18.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:40:18.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854206+0000", 2024-04-22T18:40:18.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5870000000000001e-06 2024-04-22T18:40:18.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854208+0000", 2024-04-22T18:40:18.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7199999999999998e-06 2024-04-22T18:40:18.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057285+0000", 2024-04-22T18:40:18.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20307620800000001 2024-04-22T18:40:18.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057414+0000", 2024-04-22T18:40:18.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012896899999999999 2024-04-22T18:40:18.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057588+0000", 2024-04-22T18:40:18.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000174802 2024-04-22T18:40:18.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159429+0000", 2024-04-22T18:40:18.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101840783 2024-04-22T18:40:18.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160947+0000", 2024-04-22T18:40:18.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001518152 2024-04-22T18:40:18.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160992+0000", 2024-04-22T18:40:18.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4725000000000003e-05 2024-04-22T18:40:18.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161030+0000", 2024-04-22T18:40:18.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7789e-05 2024-04-22T18:40:18.959 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.959 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.959 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.960 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:18.960 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:18.960 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.039190523000002, 2024-04-22T18:40:18.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:40:18.960 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.960 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.960 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.960 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.960 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.960 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:40:18.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.960 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:18.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:18.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:40:18.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:40:18.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:40:18.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:40:18.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:40:18.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:40:18.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:40:18.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:40:18.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:40:18.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:40:18.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:40:18.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:40:18.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:40:18.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:40:18.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:40:18.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:40:18.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:40:18.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:40:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:40:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:40:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:40:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.966 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:18.966 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:18.966 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.039174865000007, 2024-04-22T18:40:18.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:40:18.966 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.966 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.966 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.966 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.966 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.966 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:40:18.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.966 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:18.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:18.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:18.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:40:18.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:18.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:40:18.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:40:18.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:40:18.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:40:18.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:40:18.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:40:18.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:40:18.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:40:18.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:40:18.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:40:18.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:40:18.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:40:18.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:40:18.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:40:18.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:40:18.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:40:18.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:40:18.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:40:18.971 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.971 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.971 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.971 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:18.971 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:18.971 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.039123703000001, 2024-04-22T18:40:18.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:40:18.972 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.972 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.972 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.972 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.972 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.972 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:40:18.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.972 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:18.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:18.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:18.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:40:18.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:40:18.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:40:18.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:40:18.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:40:18.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:40:18.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:40:18.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:40:18.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:40:18.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:40:18.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:40:18.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:40:18.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:40:18.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:40:18.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:40:18.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:40:18.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:40:18.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:40:18.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:40:18.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:40:18.977 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.977 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.977 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.977 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:18.977 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:18.977 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.039108479000006, 2024-04-22T18:40:18.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:40:18.977 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.977 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.978 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.978 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.978 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.978 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:40:18.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.978 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:18.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:18.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:40:18.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:40:18.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:40:18.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:40:18.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:40:18.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:40:18.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:40:18.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:40:18.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:40:18.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:40:18.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:40:18.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:40:18.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:40:18.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:40:18.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:40:18.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:40:18.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:40:18.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:40:18.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:40:18.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:40:18.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:40:18.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.983 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.983 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:18.983 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:18.983 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.039048902000005, 2024-04-22T18:40:18.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:40:18.983 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.983 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.984 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.984 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.984 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.984 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:40:18.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.984 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:18.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:18.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:40:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:40:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:40:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:40:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:40:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:40:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:40:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:40:18.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:40:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:40:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:40:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:40:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:40:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:40:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:40:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:40:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:40:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:40:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:40:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:40:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.038966586000001, 2024-04-22T18:40:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:40:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.989 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:40:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:40:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:40:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:40:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:40:18.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:40:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:40:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:40:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:40:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:40:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:40:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:40:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:40:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:40:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:40:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:40:18.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:40:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:40:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:40:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:40:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:18.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.038956429999999, 2024-04-22T18:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:40:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:18.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:40:18.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:18.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:18.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:40:18.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:40:18.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:18.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:40:18.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:40:18.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:18.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:40:18.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:40:18.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:18.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:40:18.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:40:18.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:18.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:40:18.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:40:18.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:18.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:40:18.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:40:18.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:18.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:40:18.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:40:18.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:18.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:18.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:18.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:40:18.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:40:19.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:19.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:40:19.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:40:19.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:19.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:40:19.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:40:19.000 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:19.000 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:19.000 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:19.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.001 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:161 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:40:19.001 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:40:19.001 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 90.438462138999995, 2024-04-22T18:40:19.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40239169000000002, 2024-04-22T18:40:19.001 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:19.001 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:19.001 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:19.001 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:19.001 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:19.001 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 161 2024-04-22T18:40:19.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.001 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:19.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:19.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:40:19.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:19.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:19.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:40:19.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:19.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:19.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454938+0000", 2024-04-22T18:40:19.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:19.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:19.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454982+0000", 2024-04-22T18:40:19.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3470999999999998e-05 2024-04-22T18:40:19.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:19.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454988+0000", 2024-04-22T18:40:19.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7640000000000002e-06 2024-04-22T18:40:19.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:19.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555054+0000", 2024-04-22T18:40:19.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006607300000001 2024-04-22T18:40:19.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:19.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555607+0000", 2024-04-22T18:40:19.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055319500000000003 2024-04-22T18:40:19.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:19.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555689+0000", 2024-04-22T18:40:19.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1817999999999995e-05 2024-04-22T18:40:19.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:19.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555804+0000", 2024-04-22T18:40:19.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115346 2024-04-22T18:40:19.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:19.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556554+0000", 2024-04-22T18:40:19.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000750102 2024-04-22T18:40:19.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:19.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857265+0000", 2024-04-22T18:40:19.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30071122700000003 2024-04-22T18:40:19.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:19.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857302+0000", 2024-04-22T18:40:19.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7069e-05 2024-04-22T18:40:19.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:19.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857331+0000", 2024-04-22T18:40:19.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8291999999999999e-05 2024-04-22T18:40:19.006 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:19.006 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:19.006 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:19.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.006 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2835 4.18 4:187adac1:::smithi08937929-36:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:19.006 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:19.006 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 11.277032039, 2024-04-22T18:40:19.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.403020987, 2024-04-22T18:40:19.007 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:19.007 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:19.007 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:19.007 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:19.007 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:19.007 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2835 2024-04-22T18:40:19.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.007 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:19.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:19.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:19.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:19.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:19.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:19.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:19.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:19.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616367+0000", 2024-04-22T18:40:19.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:19.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:19.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616377+0000", 2024-04-22T18:40:19.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0618999999999999e-05 2024-04-22T18:40:19.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:19.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616410+0000", 2024-04-22T18:40:19.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3021000000000001e-05 2024-04-22T18:40:19.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:19.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616437+0000", 2024-04-22T18:40:19.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7158e-05 2024-04-22T18:40:19.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:19.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.716588+0000", 2024-04-22T18:40:19.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100150674 2024-04-22T18:40:19.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:19.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.716679+0000", 2024-04-22T18:40:19.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1231999999999997e-05 2024-04-22T18:40:19.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:19.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:19.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:19.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:08.019390+0000", 2024-04-22T18:40:19.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30271076299999999 2024-04-22T18:40:19.010 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:19.010 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:19.011 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:19.011 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:19.011 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:19.011 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:19.012 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:19.012 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:19.013 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:19.013 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:19.013 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:19.013+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:19.014 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:19.014 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:19.015 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:19.015 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:19.015 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:19.016 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:19.016 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:19.032 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:19.032 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:19.050 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:19.050 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:19.066 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:19.066 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:19.119 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:19.120+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:19.121 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:19.121 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:19.130 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:19.151 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:19.152+0000 7fa766d7d640 -1 osd.5 86 handle_osd_map injecting map crc failure 2024-04-22T18:40:19.214 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:19.216+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:19.272 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:19.272 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:19.281 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:19.323 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:19.324+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:19.387 INFO:tasks.rados.rados.0.smithi089.stdout:1385: set_chunk oid 26 offset: 1226752 length: 231424 target oid 75fa82bb58fb45384f1ca5f733f541de64e88f10e3ca4a901c5c901c8a7989b9 offset: 0 snap -1 2024-04-22T18:40:19.387 INFO:tasks.rados.rados.0.smithi089.stdout:1386: set_chunk oid 19 target oid 2024-04-22T18:40:19.415 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:19.416+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:19.423 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:19.423 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:19.432 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:19.516 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:19.516+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:19.534 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:19.534 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:19.543 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:19.616 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:19.617+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:19.644 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:19.644 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:19.653 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:19.695 INFO:tasks.rados.rados.0.smithi089.stdout:1386: set_chunk oid 19 offset: 404992 length: 1408512 target oid 7439175aa076f549fd203f3dd0659c93e12885d9ad463dffe8eb2643d45f8dd9 offset: 0 snap -1 2024-04-22T18:40:19.695 INFO:tasks.rados.rados.0.smithi089.stdout:1387: write oid 4 current snap is 0 2024-04-22T18:40:19.695 INFO:tasks.rados.rados.0.smithi089.stdout:1387: seq_num 269 ranges {521083=714336,1724715=555421} 2024-04-22T18:40:19.702 INFO:tasks.rados.rados.0.smithi089.stdout:1387: writing smithi08937929-4 from 521083 to 1235419 tid 1 2024-04-22T18:40:19.708 INFO:tasks.rados.rados.0.smithi089.stdout:1387: writing smithi08937929-4 from 1724715 to 2280136 tid 2 2024-04-22T18:40:19.708 INFO:tasks.rados.rados.0.smithi089.stdout:1388: set_chunk oid 49 target oid 2024-04-22T18:40:19.713 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 6, 7, 2] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:40:19.713 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:40:19.713 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-22T18:40:19.713 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-22T18:40:19.717 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:19.718+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:19.729 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:19.730+0000 7fec57bee640 -1 osd.7 88 handle_osd_map injecting map crc failure 2024-04-22T18:40:19.731 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:40:19.760 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:19.761 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:19.770 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:19.817 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:19.818+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:19.818 INFO:tasks.rados.rados.0.smithi089.stdout:1388: set_chunk oid 49 offset: 641024 length: 1139712 target oid c3d872f78a4afbe7cac11b6c7e2d755572d0ff6d3a5a1636ffce4e0d9c68bff1 offset: 0 snap -1 2024-04-22T18:40:19.819 INFO:tasks.rados.rados.0.smithi089.stdout:1389: read oid 22 snap -1 2024-04-22T18:40:19.819 INFO:tasks.rados.rados.0.smithi089.stdout:1389: expect (ObjNum 239 snap 0 seq_num 239) 2024-04-22T18:40:19.819 INFO:tasks.rados.rados.0.smithi089.stdout:1390: read oid 6 snap -1 2024-04-22T18:40:19.819 INFO:tasks.rados.rados.0.smithi089.stdout:1390: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:40:19.819 INFO:tasks.rados.rados.0.smithi089.stdout:1391: read oid 8 snap -1 2024-04-22T18:40:19.819 INFO:tasks.rados.rados.0.smithi089.stdout:1391: expect (ObjNum 253 snap 0 seq_num 253) 2024-04-22T18:40:19.819 INFO:tasks.rados.rados.0.smithi089.stdout:1392: write oid 35 current snap is 0 2024-04-22T18:40:19.819 INFO:tasks.rados.rados.0.smithi089.stdout:1392: seq_num 270 ranges {537890=789585,1783197=699079} 2024-04-22T18:40:19.826 INFO:tasks.rados.rados.0.smithi089.stdout:1392: writing smithi08937929-35 from 537890 to 1327475 tid 1 2024-04-22T18:40:19.828 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:19.830+0000 7f8605f32640 -1 osd.3 88 handle_osd_map injecting map crc failure 2024-04-22T18:40:19.833 INFO:tasks.rados.rados.0.smithi089.stdout:1392: writing smithi08937929-35 from 1783197 to 2482276 tid 2 2024-04-22T18:40:19.833 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:19.833 INFO:tasks.rados.rados.0.smithi089.stdout:1379: expect (ObjNum 226 snap 0 seq_num 226) 2024-04-22T18:40:19.918 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:19.920+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:19.921 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:19.921 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:19.930 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:20.020 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:40:20.084 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:20.084 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 89, 2024-04-22T18:40:20.084 INFO:teuthology.orchestra.run.smithi089.stdout: "fsid": "9e25744c-eff1-4a4a-8d47-0dda406b2c35", 2024-04-22T18:40:20.084 INFO:teuthology.orchestra.run.smithi089.stdout: "created": "2024-04-22T18:38:26.841687+0000", 2024-04-22T18:40:20.084 INFO:teuthology.orchestra.run.smithi089.stdout: "modified": "2024-04-22T18:40:19.720002+0000", 2024-04-22T18:40:20.084 INFO:teuthology.orchestra.run.smithi089.stdout: "last_up_change": "2024-04-22T18:38:35.971820+0000", 2024-04-22T18:40:20.084 INFO:teuthology.orchestra.run.smithi089.stdout: "last_in_change": "2024-04-22T18:40:01.634193+0000", 2024-04-22T18:40:20.084 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-22T18:40:20.085 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_num": 5799936, 2024-04-22T18:40:20.085 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_set": [ 2024-04-22T18:40:20.085 INFO:teuthology.orchestra.run.smithi089.stdout: "pglog_hardlimit", 2024-04-22T18:40:20.085 INFO:teuthology.orchestra.run.smithi089.stdout: "purged_snapdirs", 2024-04-22T18:40:20.085 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_deletes", 2024-04-22T18:40:20.085 INFO:teuthology.orchestra.run.smithi089.stdout: "sortbitwise" 2024-04-22T18:40:20.085 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:20.085 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_version": 5, 2024-04-22T18:40:20.085 INFO:teuthology.orchestra.run.smithi089.stdout: "full_ratio": 0.94999998807907104, 2024-04-22T18:40:20.085 INFO:teuthology.orchestra.run.smithi089.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-22T18:40:20.085 INFO:teuthology.orchestra.run.smithi089.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-22T18:40:20.085 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_snapshot": "", 2024-04-22T18:40:20.085 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_max": 4, 2024-04-22T18:40:20.086 INFO:teuthology.orchestra.run.smithi089.stdout: "max_osd": 8, 2024-04-22T18:40:20.086 INFO:teuthology.orchestra.run.smithi089.stdout: "require_min_compat_client": "luminous", 2024-04-22T18:40:20.086 INFO:teuthology.orchestra.run.smithi089.stdout: "min_compat_client": "jewel", 2024-04-22T18:40:20.086 INFO:teuthology.orchestra.run.smithi089.stdout: "require_osd_release": "reef", 2024-04-22T18:40:20.086 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_crimson": false, 2024-04-22T18:40:20.086 INFO:teuthology.orchestra.run.smithi089.stdout: "pools": [ 2024-04-22T18:40:20.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.086 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 1, 2024-04-22T18:40:20.086 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": ".mgr", 2024-04-22T18:40:20.087 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:36.991610+0000", 2024-04-22T18:40:20.087 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:40:20.087 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:40:20.087 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:40:20.087 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:40:20.087 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:40:20.087 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:40:20.087 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:40:20.087 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:40:20.087 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:40:20.087 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:40:20.088 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:40:20.088 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:40:20.088 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 1, 2024-04-22T18:40:20.088 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 1, 2024-04-22T18:40:20.088 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 1, 2024-04-22T18:40:20.088 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 1, 2024-04-22T18:40:20.088 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 1, 2024-04-22T18:40:20.088 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:40:20.088 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:40:20.088 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:40:20.088 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:40:20.088 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:40:20.089 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:40:20.089 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:40:20.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.089 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "16", 2024-04-22T18:40:20.089 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:40:20.089 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:40:20.089 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-04-22T18:40:20.089 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:40:20.089 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:40:20.089 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:40:20.089 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:40:20.090 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:40:20.090 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:40:20.090 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:40:20.090 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:40:20.090 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:40:20.090 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:40:20.090 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:40:20.090 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:40:20.090 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:40:20.090 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:40:20.091 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:40:20.091 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:40:20.091 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:40:20.091 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:40:20.091 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:40:20.091 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:40:20.091 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:40:20.091 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:40:20.091 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:40:20.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.091 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:40:20.091 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:40:20.091 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:40:20.091 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:40:20.092 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:40:20.092 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:40:20.092 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:40:20.092 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:40:20.092 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:40:20.092 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:40:20.092 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:40:20.092 INFO:teuthology.orchestra.run.smithi089.stdout: "options": { 2024-04-22T18:40:20.092 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_max": 32, 2024-04-22T18:40:20.092 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_min": 1 2024-04-22T18:40:20.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.092 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-04-22T18:40:20.093 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": {} 2024-04-22T18:40:20.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.093 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:40:20.093 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 8, 2024-04-22T18:40:20.093 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 8, 2024-04-22T18:40:20.093 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.25, 2024-04-22T18:40:20.093 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2, 2024-04-22T18:40:20.093 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2, 2024-04-22T18:40:20.093 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.98099899291992188, 2024-04-22T18:40:20.093 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-22T18:40:20.093 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-22T18:40:20.093 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.093 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 2, 2024-04-22T18:40:20.094 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "rbd", 2024-04-22T18:40:20.094 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:37.622620+0000", 2024-04-22T18:40:20.094 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:40:20.094 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:40:20.094 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:40:20.094 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:40:20.094 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:40:20.094 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:40:20.094 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:40:20.094 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:40:20.094 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:40:20.094 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:40:20.094 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:40:20.095 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:40:20.095 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 8, 2024-04-22T18:40:20.095 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 8, 2024-04-22T18:40:20.095 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 8, 2024-04-22T18:40:20.095 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 8, 2024-04-22T18:40:20.095 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 8, 2024-04-22T18:40:20.095 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:40:20.095 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:40:20.095 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:40:20.095 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:40:20.095 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:40:20.095 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:40:20.095 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:40:20.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.096 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "17", 2024-04-22T18:40:20.096 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:40:20.096 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:40:20.096 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-04-22T18:40:20.096 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:40:20.096 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:40:20.096 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:40:20.096 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:40:20.096 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:40:20.096 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:40:20.097 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:40:20.097 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:40:20.097 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:40:20.097 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:40:20.097 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:40:20.097 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:40:20.097 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:40:20.097 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:40:20.097 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:40:20.097 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:40:20.097 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:40:20.098 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:40:20.098 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:40:20.098 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:40:20.098 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:40:20.098 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:40:20.098 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:40:20.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.098 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:40:20.098 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:40:20.098 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:40:20.098 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:40:20.098 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:40:20.098 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:40:20.099 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:40:20.099 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:40:20.099 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:40:20.099 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:40:20.099 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:40:20.099 INFO:teuthology.orchestra.run.smithi089.stdout: "options": {}, 2024-04-22T18:40:20.099 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-04-22T18:40:20.099 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd": {} 2024-04-22T18:40:20.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.099 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:40:20.099 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 3, 2024-04-22T18:40:20.099 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 3, 2024-04-22T18:40:20.099 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.75, 2024-04-22T18:40:20.099 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2.25, 2024-04-22T18:40:20.099 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2.25, 2024-04-22T18:40:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.98099899291992188, 2024-04-22T18:40:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-22T18:40:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-22T18:40:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 3, 2024-04-22T18:40:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "low_tier", 2024-04-22T18:40:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:46.038005+0000", 2024-04-22T18:40:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:40:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:40:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:40:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:40:20.100 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:40:20.101 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:40:20.101 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:40:20.101 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:40:20.101 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:40:20.101 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:40:20.101 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:40:20.101 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:40:20.101 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 4, 2024-04-22T18:40:20.101 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 4, 2024-04-22T18:40:20.101 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 4, 2024-04-22T18:40:20.101 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 4, 2024-04-22T18:40:20.101 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 4, 2024-04-22T18:40:20.101 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:40:20.101 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:40:20.102 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:40:20.102 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:40:20.102 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:40:20.102 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:40:20.102 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:40:20.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.102 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "21", 2024-04-22T18:40:20.102 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:40:20.102 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:40:20.102 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-04-22T18:40:20.102 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:40:20.102 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:40:20.102 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:40:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:40:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:40:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:40:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:40:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:40:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:40:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:40:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:40:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:40:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:40:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:40:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:40:20.103 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:40:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:40:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:40:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:40:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:40:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:40:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:40:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:40:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:40:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:40:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:40:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:40:20.104 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:40:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:40:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:40:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:40:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:40:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:40:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:40:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: "options": {}, 2024-04-22T18:40:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": {}, 2024-04-22T18:40:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:40:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 2, 2024-04-22T18:40:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 2, 2024-04-22T18:40:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.75, 2024-04-22T18:40:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 1.5, 2024-04-22T18:40:20.105 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 1.5, 2024-04-22T18:40:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.98099899291992188, 2024-04-22T18:40:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-22T18:40:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-22T18:40:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 4, 2024-04-22T18:40:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "unique_pool_0", 2024-04-22T18:40:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:46.483369+0000", 2024-04-22T18:40:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:40:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:40:20.106 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:40:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:40:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:40:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:40:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:40:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:40:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:40:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:40:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:40:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:40:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 25, 2024-04-22T18:40:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 22, 2024-04-22T18:40:20.107 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 16, 2024-04-22T18:40:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 16, 2024-04-22T18:40:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 24, 2024-04-22T18:40:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:40:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "4.19", 2024-04-22T18:40:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 87, 2024-04-22T18:40:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 86, 2024-04-22T18:40:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 86, 2024-04-22T18:40:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "84'88", 2024-04-22T18:40:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "81'97" 2024-04-22T18:40:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "89", 2024-04-22T18:40:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:40:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "89", 2024-04-22T18:40:20.108 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "89", 2024-04-22T18:40:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:40:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:40:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:40:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:40:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:40:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:40:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:40:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:40:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:40:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:40:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:40:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:40:20.109 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:40:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:40:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:40:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:40:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:40:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:40:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:40:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:40:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:40:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:40:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:40:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:40:20.110 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:40:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:40:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:40:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:40:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:40:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:40:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:40:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:40:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:40:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:40:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: "options": { 2024-04-22T18:40:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: "dedup_cdc_chunk_size": 131072, 2024-04-22T18:40:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: "dedup_chunk_algorithm": "fastcdc", 2024-04-22T18:40:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: "dedup_tier": 3, 2024-04-22T18:40:20.111 INFO:teuthology.orchestra.run.smithi089.stdout: "fingerprint_algorithm": "sha256" 2024-04-22T18:40:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-04-22T18:40:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: "rados": {} 2024-04-22T18:40:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:40:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 1.9099999666213989, 2024-04-22T18:40:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 1.9099999666213989, 2024-04-22T18:40:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.87999999523162842, 2024-04-22T18:40:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 1.6799999475479126, 2024-04-22T18:40:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 1.6799999475479126, 2024-04-22T18:40:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.98099899291992188, 2024-04-22T18:40:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.98000001907348633, 2024-04-22T18:40:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-04-22T18:40:20.112 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: "osds": [ 2024-04-22T18:40:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-04-22T18:40:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "bbf33cfb-0f07-4df6-b90f-83e9edb039a6", 2024-04-22T18:40:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:40:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:40:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:40:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 72, 2024-04-22T18:40:20.113 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6800", 2024-04-22T18:40:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:40:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.114 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6801", 2024-04-22T18:40:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:40:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6803", 2024-04-22T18:40:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:40:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6802", 2024-04-22T18:40:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:40:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6800/31890", 2024-04-22T18:40:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6801/31890", 2024-04-22T18:40:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6803/31890", 2024-04-22T18:40:20.116 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6802/31890", 2024-04-22T18:40:20.117 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:20.117 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:20.117 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:20.117 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.117 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-04-22T18:40:20.117 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "30415a8e-9f47-416f-9fd0-716c2cb3f562", 2024-04-22T18:40:20.117 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:20.117 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:40:20.117 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:40:20.117 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:40:20.118 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:20.118 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:20.118 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:20.118 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 72, 2024-04-22T18:40:20.118 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:20.118 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:20.118 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:20.118 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.118 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.118 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6804", 2024-04-22T18:40:20.118 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:40:20.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.119 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.119 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:20.119 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.119 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.119 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6805", 2024-04-22T18:40:20.119 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:40:20.119 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.119 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.119 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:20.119 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.120 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.120 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6807", 2024-04-22T18:40:20.120 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:40:20.120 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.120 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.120 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:20.120 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.120 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.120 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6806", 2024-04-22T18:40:20.120 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:40:20.120 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.120 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.121 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6804/31899", 2024-04-22T18:40:20.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6805/31899", 2024-04-22T18:40:20.121 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6807/31899", 2024-04-22T18:40:20.121 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6806/31899", 2024-04-22T18:40:20.121 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:20.121 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:20.121 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:20.121 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.121 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-04-22T18:40:20.121 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "b8da6b88-8355-4225-930d-bfed881089b7", 2024-04-22T18:40:20.121 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:20.122 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:40:20.122 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:40:20.122 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:40:20.122 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:20.122 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:20.122 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:20.122 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 72, 2024-04-22T18:40:20.122 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:20.122 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:20.122 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:20.122 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.123 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.123 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6812", 2024-04-22T18:40:20.123 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:40:20.123 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.123 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.123 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:20.123 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.123 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.123 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6813", 2024-04-22T18:40:20.123 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:40:20.124 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.124 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.124 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:20.124 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.124 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.124 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6815", 2024-04-22T18:40:20.124 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:40:20.124 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.124 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.124 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:20.124 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.125 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.125 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6814", 2024-04-22T18:40:20.125 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:40:20.125 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.125 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.125 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6812/31903", 2024-04-22T18:40:20.125 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6813/31903", 2024-04-22T18:40:20.125 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6815/31903", 2024-04-22T18:40:20.125 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6814/31903", 2024-04-22T18:40:20.125 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:20.125 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:20.125 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:20.125 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.126 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-04-22T18:40:20.126 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "3eaec913-9d7d-4b03-a28f-0c334e7c39f1", 2024-04-22T18:40:20.126 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:20.126 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:40:20.126 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:40:20.126 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 0.847991943359375, 2024-04-22T18:40:20.126 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:20.126 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:20.126 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:20.126 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 79, 2024-04-22T18:40:20.127 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:20.127 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:20.127 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:20.127 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.127 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.127 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6808", 2024-04-22T18:40:20.127 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:40:20.127 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.127 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.127 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:20.127 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.128 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.128 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6809", 2024-04-22T18:40:20.128 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:40:20.128 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.128 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.128 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:20.128 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.128 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.128 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6811", 2024-04-22T18:40:20.128 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:40:20.129 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.129 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.129 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:20.129 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.129 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.129 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6810", 2024-04-22T18:40:20.129 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:40:20.129 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.129 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.129 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6808/31913", 2024-04-22T18:40:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6809/31913", 2024-04-22T18:40:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6811/31913", 2024-04-22T18:40:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6810/31913", 2024-04-22T18:40:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-04-22T18:40:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "887acc3b-6f6b-49eb-a039-13050a2c7d64", 2024-04-22T18:40:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:20.130 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:40:20.131 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:40:20.131 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:40:20.131 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:20.131 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:20.131 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:20.131 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 87, 2024-04-22T18:40:20.131 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:20.131 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:20.131 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:20.131 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.131 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.131 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6808", 2024-04-22T18:40:20.131 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:40:20.132 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.132 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.132 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:20.132 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.132 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.132 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6809", 2024-04-22T18:40:20.132 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:40:20.132 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.132 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.132 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:20.132 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.133 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.133 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6811", 2024-04-22T18:40:20.133 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:40:20.133 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.133 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.133 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:20.133 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.133 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.133 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6810", 2024-04-22T18:40:20.133 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:40:20.134 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.134 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.134 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6808/31922", 2024-04-22T18:40:20.134 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6809/31922", 2024-04-22T18:40:20.134 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6811/31922", 2024-04-22T18:40:20.134 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6810/31922", 2024-04-22T18:40:20.134 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:20.134 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:20.134 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:20.134 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.134 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-04-22T18:40:20.135 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "dcaddfd0-f8d1-4c28-bda2-d17eed606b60", 2024-04-22T18:40:20.135 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:20.135 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 0, 2024-04-22T18:40:20.135 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 0, 2024-04-22T18:40:20.135 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:40:20.135 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:20.135 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:20.135 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:20.135 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 63, 2024-04-22T18:40:20.135 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:20.135 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:20.135 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:20.135 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.136 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.136 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6804", 2024-04-22T18:40:20.136 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:40:20.136 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.136 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.136 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:20.136 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.136 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.136 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6805", 2024-04-22T18:40:20.136 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:40:20.136 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.136 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.137 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:20.137 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.137 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.137 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6807", 2024-04-22T18:40:20.137 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:40:20.137 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.137 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.137 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:20.137 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.138 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.138 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6806", 2024-04-22T18:40:20.138 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:40:20.138 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.138 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.138 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6804/31921", 2024-04-22T18:40:20.138 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6805/31921", 2024-04-22T18:40:20.138 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6807/31921", 2024-04-22T18:40:20.138 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6806/31921", 2024-04-22T18:40:20.138 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:20.138 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:20.139 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:20.139 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.139 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-04-22T18:40:20.139 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "acf0527b-3020-4b0a-b64c-5068d21a5781", 2024-04-22T18:40:20.139 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:20.139 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:40:20.139 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:40:20.139 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:40:20.139 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:20.139 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:20.139 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:20.140 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 79, 2024-04-22T18:40:20.140 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:20.140 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:20.140 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:20.140 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.140 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.140 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6800", 2024-04-22T18:40:20.140 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:40:20.140 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.140 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.140 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:20.140 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.141 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.141 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6801", 2024-04-22T18:40:20.141 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:40:20.141 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.141 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.141 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:20.141 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.141 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.141 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6803", 2024-04-22T18:40:20.141 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:40:20.141 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.142 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.142 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:20.142 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.142 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.142 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6802", 2024-04-22T18:40:20.142 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:40:20.142 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.142 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.142 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6800/31920", 2024-04-22T18:40:20.142 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6801/31920", 2024-04-22T18:40:20.142 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6803/31920", 2024-04-22T18:40:20.143 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6802/31920", 2024-04-22T18:40:20.143 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:20.143 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:20.143 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:20.143 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.143 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-04-22T18:40:20.143 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "305394bc-be3d-420b-88d7-d273670540b1", 2024-04-22T18:40:20.143 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:40:20.143 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:40:20.143 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:40:20.143 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:40:20.143 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:40:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:40:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:40:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 81, 2024-04-22T18:40:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:40:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:40:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:40:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6812", 2024-04-22T18:40:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:40:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:40:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6813", 2024-04-22T18:40:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:40:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:40:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6815", 2024-04-22T18:40:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:40:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:40:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:40:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:40:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6814", 2024-04-22T18:40:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:40:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.146 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6812/31939", 2024-04-22T18:40:20.147 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6813/31939", 2024-04-22T18:40:20.147 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6815/31939", 2024-04-22T18:40:20.147 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6814/31939", 2024-04-22T18:40:20.147 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:40:20.147 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:40:20.147 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:40:20.147 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:20.147 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.147 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:20.147 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_xinfo": [ 2024-04-22T18:40:20.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.147 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-04-22T18:40:20.147 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:20.147 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:20.147 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:20.148 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:20.148 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:40:20.148 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:48.032183+0000", 2024-04-22T18:40:20.148 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:20.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.148 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-04-22T18:40:20.148 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:20.148 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:20.148 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:20.148 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:20.148 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:40:20.148 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:48.108964+0000", 2024-04-22T18:40:20.148 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:20.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.149 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-04-22T18:40:20.149 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:20.149 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:20.149 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:20.149 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:20.149 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:40:20.149 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:50.436792+0000", 2024-04-22T18:40:20.149 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:20.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.149 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-04-22T18:40:20.149 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:20.150 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:20.150 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:20.150 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:20.150 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:40:20.150 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:52.033545+0000", 2024-04-22T18:40:20.150 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:20.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.150 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-04-22T18:40:20.150 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:20.150 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:20.150 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:20.150 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:20.151 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:40:20.151 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:39.194178+0000", 2024-04-22T18:40:20.151 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:20.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.151 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-04-22T18:40:20.151 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:20.151 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:20.151 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:20.151 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:20.151 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 65536, 2024-04-22T18:40:20.151 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:40.812449+0000", 2024-04-22T18:40:20.152 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:20.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.152 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-04-22T18:40:20.152 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:20.152 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:20.152 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:20.152 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:20.152 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:40:20.152 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:44.814263+0000", 2024-04-22T18:40:20.152 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:20.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:20.153 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-04-22T18:40:20.153 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:40:20.153 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:40:20.153 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:40:20.153 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:40:20.153 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:40:20.153 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:39:42.667515+0000", 2024-04-22T18:40:20.153 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:40:20.153 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.153 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:20.153 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap": [], 2024-04-22T18:40:20.153 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_items": [], 2024-04-22T18:40:20.153 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_primaries": [], 2024-04-22T18:40:20.154 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_temp": [], 2024-04-22T18:40:20.154 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_temp": [], 2024-04-22T18:40:20.154 INFO:teuthology.orchestra.run.smithi089.stdout: "blocklist": {}, 2024-04-22T18:40:20.154 INFO:teuthology.orchestra.run.smithi089.stdout: "range_blocklist": {}, 2024-04-22T18:40:20.154 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profiles": { 2024-04-22T18:40:20.154 INFO:teuthology.orchestra.run.smithi089.stdout: "default": { 2024-04-22T18:40:20.154 INFO:teuthology.orchestra.run.smithi089.stdout: "crush-failure-domain": "osd", 2024-04-22T18:40:20.154 INFO:teuthology.orchestra.run.smithi089.stdout: "k": "2", 2024-04-22T18:40:20.154 INFO:teuthology.orchestra.run.smithi089.stdout: "m": "1", 2024-04-22T18:40:20.154 INFO:teuthology.orchestra.run.smithi089.stdout: "plugin": "jerasure", 2024-04-22T18:40:20.154 INFO:teuthology.orchestra.run.smithi089.stdout: "technique": "reed_sol_van" 2024-04-22T18:40:20.154 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:20.155 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps_queue": [], 2024-04-22T18:40:20.155 INFO:teuthology.orchestra.run.smithi089.stdout: "new_removed_snaps": [], 2024-04-22T18:40:20.155 INFO:teuthology.orchestra.run.smithi089.stdout: "new_purged_snaps": [], 2024-04-22T18:40:20.155 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_node_flags": {}, 2024-04-22T18:40:20.155 INFO:teuthology.orchestra.run.smithi089.stdout: "device_class_flags": {}, 2024-04-22T18:40:20.155 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode": { 2024-04-22T18:40:20.155 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_enabled": false, 2024-04-22T18:40:20.155 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_bucket_count": 0, 2024-04-22T18:40:20.155 INFO:teuthology.orchestra.run.smithi089.stdout: "degraded_stretch_mode": 0, 2024-04-22T18:40:20.155 INFO:teuthology.orchestra.run.smithi089.stdout: "recovering_stretch_mode": 0, 2024-04-22T18:40:20.155 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_bucket": 0 2024-04-22T18:40:20.156 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:20.156 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:20.156 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:20.021+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:20.158 INFO:tasks.thrashosds.thrasher:j is {'epoch': 89, 'fsid': '9e25744c-eff1-4a4a-8d47-0dda406b2c35', 'created': '2024-04-22T18:38:26.841687+0000', 'modified': '2024-04-22T18:40:19.720002+0000', 'last_up_change': '2024-04-22T18:38:35.971820+0000', 'last_in_change': '2024-04-22T18:40:01.634193+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T18:38:36.991610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9809989929199219, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-22T18:38:37.622620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9809989929199219, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-04-22T18:38:46.038005+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9809989929199219, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-22T18:38:46.483369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 25, 'pg_placement_num': 22, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 24, 'last_pg_merge_meta': {'source_pgid': '4.19', 'ready_epoch': 87, 'last_epoch_started': 86, 'last_epoch_clean': 86, 'source_version': "84'88", 'target_version': "81'97"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '89', 'last_force_op_resend_preluminous': '89', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'dedup_cdc_chunk_size': 131072, 'dedup_chunk_algorithm': 'fastcdc', 'dedup_tier': 3, 'fingerprint_algorithm': 'sha256'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.6799999475479126, 'raw_score_stable': 1.6799999475479126, 'primary_affinity_weighted': 0.9809989929199219, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}], 'osds': [{'osd': 0, 'uuid': 'bbf33cfb-0f07-4df6-b90f-83e9edb039a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6800', 'nonce': 31890}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6801', 'nonce': 31890}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6803', 'nonce': 31890}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6802', 'nonce': 31890}]}, 'public_addr': '172.21.15.89:6800/31890', 'cluster_addr': '172.21.15.89:6801/31890', 'heartbeat_back_addr': '172.21.15.89:6803/31890', 'heartbeat_front_addr': '172.21.15.89:6802/31890', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '30415a8e-9f47-416f-9fd0-716c2cb3f562', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6804', 'nonce': 31899}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6805', 'nonce': 31899}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6807', 'nonce': 31899}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6806', 'nonce': 31899}]}, 'public_addr': '172.21.15.89:6804/31899', 'cluster_addr': '172.21.15.89:6805/31899', 'heartbeat_back_addr': '172.21.15.89:6807/31899', 'heartbeat_front_addr': '172.21.15.89:6806/31899', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b8da6b88-8355-4225-930d-bfed881089b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6812', 'nonce': 31903}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6813', 'nonce': 31903}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6815', 'nonce': 31903}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6814', 'nonce': 31903}]}, 'public_addr': '172.21.15.89:6812/31903', 'cluster_addr': '172.21.15.89:6813/31903', 'heartbeat_back_addr': '172.21.15.89:6815/31903', 'heartbeat_front_addr': '172.21.15.89:6814/31903', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3eaec913-9d7d-4b03-a28f-0c334e7c39f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.847991943359375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 79, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6808', 'nonce': 31913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6809', 'nonce': 31913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6811', 'nonce': 31913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6810', 'nonce': 31913}]}, 'public_addr': '172.21.15.89:6808/31913', 'cluster_addr': '172.21.15.89:6809/31913', 'heartbeat_back_addr': '172.21.15.89:6811/31913', 'heartbeat_front_addr': '172.21.15.89:6810/31913', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '887acc3b-6f6b-49eb-a039-13050a2c7d64', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 87, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 31922}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 31922}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 31922}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 31922}]}, 'public_addr': '172.21.15.121:6808/31922', 'cluster_addr': '172.21.15.121:6809/31922', 'heartbeat_back_addr': '172.21.15.121:6811/31922', 'heartbeat_front_addr': '172.21.15.121:6810/31922', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'dcaddfd0-f8d1-4c28-bda2-d17eed606b60', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 63, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 31921}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 31921}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 31921}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 31921}]}, 'public_addr': '172.21.15.121:6804/31921', 'cluster_addr': '172.21.15.121:6805/31921', 'heartbeat_back_addr': '172.21.15.121:6807/31921', 'heartbeat_front_addr': '172.21.15.121:6806/31921', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'acf0527b-3020-4b0a-b64c-5068d21a5781', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 79, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 31920}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 31920}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 31920}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 31920}]}, 'public_addr': '172.21.15.121:6800/31920', 'cluster_addr': '172.21.15.121:6801/31920', 'heartbeat_back_addr': '172.21.15.121:6803/31920', 'heartbeat_front_addr': '172.21.15.121:6802/31920', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '305394bc-be3d-420b-88d7-d273670540b1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 31939}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 31939}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 31939}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 31939}]}, 'public_addr': '172.21.15.121:6812/31939', 'cluster_addr': '172.21.15.121:6813/31939', 'heartbeat_back_addr': '172.21.15.121:6815/31939', 'heartbeat_front_addr': '172.21.15.121:6814/31939', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:39:48.032183+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:39:48.108964+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:39:50.436792+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:39:52.033545+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:39:39.194178+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-22T18:39:40.812449+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:39:44.814263+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:39:42.667515+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-22T18:40:20.158 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:40:20.159 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:20.159 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:20.160 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:20.160 INFO:tasks.rados.rados.0.smithi089.stdout:1377: expect (ObjNum 208 snap 0 seq_num 208) 2024-04-22T18:40:20.161 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:20.158+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:20.258 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:20.258+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:20.302 INFO:tasks.rados.rados.0.smithi089.stdout:1378: expect (ObjNum 264 snap 0 seq_num 264) 2024-04-22T18:40:20.304 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:20.304 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:20.314 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:20.358 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:20.359+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:20.419 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:20.419 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:20.428 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:20.448 INFO:tasks.rados.rados.0.smithi089.stdout:1380: finishing write tid 1 to smithi08937929-37 2024-04-22T18:40:20.449 INFO:tasks.rados.rados.0.smithi089.stdout:1381: got (22) Invalid argument 2024-04-22T18:40:20.449 INFO:tasks.rados.rados.0.smithi089.stdout:1380: finishing write tid 2 to smithi08937929-37 2024-04-22T18:40:20.449 INFO:tasks.rados.rados.0.smithi089.stdout:1377: done (15 left) 2024-04-22T18:40:20.449 INFO:tasks.rados.rados.0.smithi089.stdout:1378: done (14 left) 2024-04-22T18:40:20.449 INFO:tasks.rados.rados.0.smithi089.stdout:1379: done (13 left) 2024-04-22T18:40:20.449 INFO:tasks.rados.rados.0.smithi089.stdout:1381: done (12 left) 2024-04-22T18:40:20.449 INFO:tasks.rados.rados.0.smithi089.stdout:1393: write (excl) oid 5 current snap is 0 2024-04-22T18:40:20.449 INFO:tasks.rados.rados.0.smithi089.stdout:1393: seq_num 271 ranges {554697=464834,1725326=559090} 2024-04-22T18:40:20.454 INFO:tasks.rados.rados.0.smithi089.stdout:1393: writing smithi08937929-5 from 554697 to 1019531 tid 1 2024-04-22T18:40:20.459 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:20.460+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:20.462 INFO:tasks.rados.rados.0.smithi089.stdout:1393: writing smithi08937929-5 from 1725326 to 2284416 tid 2 2024-04-22T18:40:20.462 INFO:tasks.rados.rados.0.smithi089.stdout:1394: tier_promote oid 18 2024-04-22T18:40:20.462 INFO:tasks.rados.rados.0.smithi089.stdout:1380: finishing write tid 3 to smithi08937929-37 2024-04-22T18:40:20.462 INFO:tasks.rados.rados.0.smithi089.stdout:1380: finishing write tid 4 to smithi08937929-37 2024-04-22T18:40:20.462 INFO:tasks.rados.rados.0.smithi089.stdout:1380: oid 37 updating version 0 to 108 2024-04-22T18:40:20.462 INFO:tasks.rados.rados.0.smithi089.stdout:1380: oid 37 updating version 108 to 109 2024-04-22T18:40:20.462 INFO:tasks.rados.rados.0.smithi089.stdout:1380: oid 37 version 109 is already newer than 107 2024-04-22T18:40:20.463 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 109 (ObjNum 267 snap 0 seq_num 267) dirty exists 2024-04-22T18:40:20.463 INFO:tasks.rados.rados.0.smithi089.stdout:1380: left oid 37 (ObjNum 267 snap 0 seq_num 267) 2024-04-22T18:40:20.463 INFO:tasks.rados.rados.0.smithi089.stdout:1382: finishing set_chunk to oid 32 2024-04-22T18:40:20.463 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 32 set_chunk returned error code -95 offset: 177152 length: 87552 2024-04-22T18:40:20.463 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 0 (ObjNum 209 snap 0 seq_num 209) dirty exists 2024-04-22T18:40:20.463 INFO:tasks.rados.rados.0.smithi089.stdout:1383: finishing write tid 1 to smithi08937929-39 2024-04-22T18:40:20.463 INFO:tasks.rados.rados.0.smithi089.stdout:1383: finishing write tid 2 to smithi08937929-39 2024-04-22T18:40:20.463 INFO:tasks.rados.rados.0.smithi089.stdout:1383: finishing write tid 3 to smithi08937929-39 2024-04-22T18:40:20.463 INFO:tasks.rados.rados.0.smithi089.stdout:1383: finishing write tid 4 to smithi08937929-39 2024-04-22T18:40:20.463 INFO:tasks.rados.rados.0.smithi089.stdout:1383: oid 39 updating version 0 to 100 2024-04-22T18:40:20.463 INFO:tasks.rados.rados.0.smithi089.stdout:1383: oid 39 version 100 is already newer than 99 2024-04-22T18:40:20.463 INFO:tasks.rados.rados.0.smithi089.stdout:1383: oid 39 version 100 is already newer than 98 2024-04-22T18:40:20.464 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 100 (ObjNum 268 snap 0 seq_num 268) dirty exists 2024-04-22T18:40:20.464 INFO:tasks.rados.rados.0.smithi089.stdout:1383: left oid 39 (ObjNum 268 snap 0 seq_num 268) 2024-04-22T18:40:20.464 INFO:tasks.rados.rados.0.smithi089.stdout:1385: finishing set_chunk to oid 26 2024-04-22T18:40:20.464 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 97 (ObjNum 252 snap 0 seq_num 252) dirty exists 2024-04-22T18:40:20.464 INFO:tasks.rados.rados.0.smithi089.stdout:1387: finishing write tid 1 to smithi08937929-4 2024-04-22T18:40:20.464 INFO:tasks.rados.rados.0.smithi089.stdout:1386: finishing set_chunk to oid 19 2024-04-22T18:40:20.464 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 19 set_chunk returned error code -95 offset: 404992 length: 1408512 2024-04-22T18:40:20.464 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 0 (ObjNum 195 snap 0 seq_num 195) dirty exists 2024-04-22T18:40:20.464 INFO:tasks.rados.rados.0.smithi089.stdout:1389: expect (ObjNum 239 snap 0 seq_num 239) 2024-04-22T18:40:20.499 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:40:20.500 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:40:20.514 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-04-22T18:40:19.734502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":226449778,"num_objects":171,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":3229,"num_read_kb":2745808,"num_write":1999,"num_write_kb":480168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":79964431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":29,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2402,"ondisk_log_size":2402,"up":76,"acting":76,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":497536,"kb_used_data":277096,"kb_used_omap":10,"kb_used_meta":209525,"kb_avail":660105344,"statfs":{"total":676457349120,"available":675947872256,"internally_reserved":0,"allocated":283746304,"data_stored":287663377,"data_compressed":7222214,"data_compressed_allocated":7700480,"data_compressed_original":16387270,"omap_allocated":11150,"internal_metadata":214553714},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12837658,"num_objects":-10,"num_object_clones":0,"num_object_copies":-20,"num_objects_missing_on_primary":-1,"num_objects_missing":-1,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":0,"num_read":279,"num_read_kb":275176,"num_write":96,"num_write_kb":20210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":139270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.025549"},"pg_stats":[{"pgid":"4.18","version":"77'56","reported_seq":218,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:14.802872+0000","last_change":"2024-04-22T18:40:06.800906+0000","last_active":"2024-04-22T18:40:14.802872+0000","last_peered":"2024-04-22T18:40:14.802872+0000","last_clean":"2024-04-22T18:40:14.802872+0000","last_became_active":"2024-04-22T18:40:02.645235+0000","last_became_peered":"2024-04-22T18:40:02.645235+0000","last_unstale":"2024-04-22T18:40:14.802872+0000","last_undegraded":"2024-04-22T18:40:14.802872+0000","last_fullsized":"2024-04-22T18:40:14.802872+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2058858,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":89,"num_read_kb":116725,"num_write":49,"num_write_kb":11981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2528838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"52'33","reported_seq":163,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.297775+0000","last_change":"2024-04-22T18:40:02.646106+0000","last_active":"2024-04-22T18:40:15.297775+0000","last_peered":"2024-04-22T18:40:15.297775+0000","last_clean":"2024-04-22T18:40:15.297775+0000","last_became_active":"2024-04-22T18:40:02.645801+0000","last_became_peered":"2024-04-22T18:40:02.645801+0000","last_unstale":"2024-04-22T18:40:15.297775+0000","last_undegraded":"2024-04-22T18:40:15.297775+0000","last_fullsized":"2024-04-22T18:40:15.297775+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042051,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":49,"num_read_kb":52611,"num_write":15,"num_write_kb":3979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"81'68","reported_seq":224,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:14.701254+0000","last_change":"2024-04-22T18:40:11.604262+0000","last_active":"2024-04-22T18:40:14.701254+0000","last_peered":"2024-04-22T18:40:14.701254+0000","last_clean":"2024-04-22T18:40:14.701254+0000","last_became_active":"2024-04-22T18:39:47.104695+0000","last_became_peered":"2024-04-22T18:39:47.104695+0000","last_unstale":"2024-04-22T18:40:14.701254+0000","last_undegraded":"2024-04-22T18:40:14.701254+0000","last_fullsized":"2024-04-22T18:40:14.701254+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2453855,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":88,"num_read_kb":89869,"num_write":42,"num_write_kb":11971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"54'29","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.853705+0000","last_change":"2024-04-22T18:40:13.280829+0000","last_active":"2024-04-22T18:40:15.853705+0000","last_peered":"2024-04-22T18:40:15.853705+0000","last_clean":"2024-04-22T18:40:15.853705+0000","last_became_active":"2024-04-22T18:40:13.180144+0000","last_became_peered":"2024-04-22T18:40:13.180144+0000","last_unstale":"2024-04-22T18:40:15.853705+0000","last_undegraded":"2024-04-22T18:40:15.853705+0000","last_fullsized":"2024-04-22T18:40:15.853705+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'29","last_scrub_stamp":"2024-04-22T18:39:56.614637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:56.614637+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2915998,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":45460,"num_write":12,"num_write_kb":3102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"51'19","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-22T18:40:13.986338+0000","last_change":"2024-04-22T18:40:11.079224+0000","last_active":"2024-04-22T18:40:13.986338+0000","last_peered":"2024-04-22T18:40:13.986338+0000","last_clean":"2024-04-22T18:40:13.986338+0000","last_became_active":"2024-04-22T18:40:10.978840+0000","last_became_peered":"2024-04-22T18:40:10.978840+0000","last_unstale":"2024-04-22T18:40:13.986338+0000","last_undegraded":"2024-04-22T18:40:13.986338+0000","last_fullsized":"2024-04-22T18:40:13.986338+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1882368,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":22802,"num_write":10,"num_write_kb":3047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1349842,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"79'75","reported_seq":37,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.407644+0000","last_change":"2024-04-22T18:40:10.277811+0000","last_active":"2024-04-22T18:40:15.407644+0000","last_peered":"2024-04-22T18:40:15.407644+0000","last_clean":"2024-04-22T18:40:15.407644+0000","last_became_active":"2024-04-22T18:40:09.027450+0000","last_became_peered":"2024-04-22T18:40:09.027450+0000","last_unstale":"2024-04-22T18:40:15.407644+0000","last_undegraded":"2024-04-22T18:40:15.407644+0000","last_fullsized":"2024-04-22T18:40:15.407644+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6605151,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":147897,"num_write":43,"num_write_kb":12157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7825098,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"58'33","reported_seq":170,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.297630+0000","last_change":"2024-04-22T18:40:03.451290+0000","last_active":"2024-04-22T18:40:15.297630+0000","last_peered":"2024-04-22T18:40:15.297630+0000","last_clean":"2024-04-22T18:40:15.297630+0000","last_became_active":"2024-04-22T18:40:02.844371+0000","last_became_peered":"2024-04-22T18:40:02.844371+0000","last_unstale":"2024-04-22T18:40:15.297630+0000","last_undegraded":"2024-04-22T18:40:15.297630+0000","last_fullsized":"2024-04-22T18:40:15.297630+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2445419,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":76762,"num_write":16,"num_write_kb":4115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":970119,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"73'113","reported_seq":327,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-22T18:40:13.986333+0000","last_change":"2024-04-22T18:40:04.645761+0000","last_active":"2024-04-22T18:40:13.986333+0000","last_peered":"2024-04-22T18:40:13.986333+0000","last_clean":"2024-04-22T18:40:13.986333+0000","last_became_active":"2024-04-22T18:40:04.645418+0000","last_became_peered":"2024-04-22T18:40:04.645418+0000","last_unstale":"2024-04-22T18:40:13.986333+0000","last_undegraded":"2024-04-22T18:40:13.986333+0000","last_fullsized":"2024-04-22T18:40:13.986333+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10102482,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":155,"num_read_kb":162433,"num_write":62,"num_write_kb":17851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1101926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"31'5","reported_seq":118,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-22T18:40:14.086377+0000","last_change":"2024-04-22T18:40:10.212897+0000","last_active":"2024-04-22T18:40:14.086377+0000","last_peered":"2024-04-22T18:40:14.086377+0000","last_clean":"2024-04-22T18:40:14.086377+0000","last_became_active":"2024-04-22T18:39:59.265189+0000","last_became_peered":"2024-04-22T18:39:59.265189+0000","last_unstale":"2024-04-22T18:40:14.086377+0000","last_undegraded":"2024-04-22T18:40:14.086377+0000","last_fullsized":"2024-04-22T18:40:14.086377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":5,"last_scrub":"31'5","last_scrub_stamp":"2024-04-22T18:40:10.212815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.212815+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:17.620115+0000","scrub_duration":0.0015068670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":77,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.087781+0000","last_change":"2024-04-22T18:38:39.026078+0000","last_active":"2024-04-22T18:40:15.087781+0000","last_peered":"2024-04-22T18:40:15.087781+0000","last_clean":"2024-04-22T18:40:15.087781+0000","last_became_active":"2024-04-22T18:38:39.025741+0000","last_became_peered":"2024-04-22T18:38:39.025741+0000","last_unstale":"2024-04-22T18:40:15.087781+0000","last_undegraded":"2024-04-22T18:40:15.087781+0000","last_fullsized":"2024-04-22T18:40:15.087781+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"84'139","reported_seq":415,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T18:40:16.433171+0000","last_change":"2024-04-22T18:40:04.048901+0000","last_active":"2024-04-22T18:40:16.433171+0000","last_peered":"2024-04-22T18:40:16.433171+0000","last_clean":"2024-04-22T18:40:16.433171+0000","last_became_active":"2024-04-22T18:40:02.858745+0000","last_became_peered":"2024-04-22T18:40:02.858745+0000","last_unstale":"2024-04-22T18:40:16.433171+0000","last_undegraded":"2024-04-22T18:40:16.433171+0000","last_fullsized":"2024-04-22T18:40:16.433171+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:07.943249+0000","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12891068,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":126,"num_read_kb":149386,"num_write":32,"num_write_kb":9134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9086896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":77,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.297766+0000","last_change":"2024-04-22T18:38:39.110499+0000","last_active":"2024-04-22T18:40:15.297766+0000","last_peered":"2024-04-22T18:40:15.297766+0000","last_clean":"2024-04-22T18:40:15.297766+0000","last_became_active":"2024-04-22T18:38:39.010173+0000","last_became_peered":"2024-04-22T18:38:39.010173+0000","last_unstale":"2024-04-22T18:40:15.297766+0000","last_undegraded":"2024-04-22T18:40:15.297766+0000","last_fullsized":"2024-04-22T18:40:15.297766+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"71'14","reported_seq":128,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-22T18:40:13.986345+0000","last_change":"2024-04-22T18:39:54.378188+0000","last_active":"2024-04-22T18:40:13.986345+0000","last_peered":"2024-04-22T18:40:13.986345+0000","last_clean":"2024-04-22T18:40:13.986345+0000","last_became_active":"2024-04-22T18:39:46.805569+0000","last_became_peered":"2024-04-22T18:39:46.805569+0000","last_unstale":"2024-04-22T18:40:13.986345+0000","last_undegraded":"2024-04-22T18:40:13.986345+0000","last_fullsized":"2024-04-22T18:40:13.986345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'13","last_scrub_stamp":"2024-04-22T18:39:54.377980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.377980+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:00.921987+0000","scrub_duration":0.30260881099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3378207,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":35426,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":86,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:14.802825+0000","last_change":"2024-04-22T18:40:09.538708+0000","last_active":"2024-04-22T18:40:14.802825+0000","last_peered":"2024-04-22T18:40:14.802825+0000","last_clean":"2024-04-22T18:40:14.802825+0000","last_became_active":"2024-04-22T18:38:39.210049+0000","last_became_peered":"2024-04-22T18:38:39.210049+0000","last_unstale":"2024-04-22T18:40:14.802825+0000","last_undegraded":"2024-04-22T18:40:14.802825+0000","last_fullsized":"2024-04-22T18:40:14.802825+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:09.538542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:09.538542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:31.805660+0000","scrub_duration":0.35201753200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"78'70","reported_seq":246,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:14.701265+0000","last_change":"2024-04-22T18:40:03.763662+0000","last_active":"2024-04-22T18:40:14.701265+0000","last_peered":"2024-04-22T18:40:14.701265+0000","last_clean":"2024-04-22T18:40:14.701265+0000","last_became_active":"2024-04-22T18:40:02.739213+0000","last_became_peered":"2024-04-22T18:40:02.739213+0000","last_unstale":"2024-04-22T18:40:14.701265+0000","last_undegraded":"2024-04-22T18:40:14.701265+0000","last_fullsized":"2024-04-22T18:40:14.701265+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'67","last_scrub_stamp":"2024-04-22T18:39:53.916646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.916646+0000","objects_scrubbed":2,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:06.252157+0000","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269011,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":16271,"num_write":1,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3912084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":77,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.297736+0000","last_change":"2024-04-22T18:38:39.307735+0000","last_active":"2024-04-22T18:40:15.297736+0000","last_peered":"2024-04-22T18:40:15.297736+0000","last_clean":"2024-04-22T18:40:15.297736+0000","last_became_active":"2024-04-22T18:38:39.307459+0000","last_became_peered":"2024-04-22T18:38:39.307459+0000","last_unstale":"2024-04-22T18:40:15.297736+0000","last_undegraded":"2024-04-22T18:40:15.297736+0000","last_fullsized":"2024-04-22T18:40:15.297736+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"64'38","reported_seq":174,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.497842+0000","last_change":"2024-04-22T18:40:04.057863+0000","last_active":"2024-04-22T18:40:15.497842+0000","last_peered":"2024-04-22T18:40:15.497842+0000","last_clean":"2024-04-22T18:40:15.497842+0000","last_became_active":"2024-04-22T18:40:02.844289+0000","last_became_peered":"2024-04-22T18:40:02.844289+0000","last_unstale":"2024-04-22T18:40:15.497842+0000","last_undegraded":"2024-04-22T18:40:15.497842+0000","last_fullsized":"2024-04-22T18:40:15.497842+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2932821,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":107389,"num_write":20,"num_write_kb":5595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2843709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"84'257","reported_seq":339,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.528565+0000","last_change":"2024-04-22T18:38:47.173101+0000","last_active":"2024-04-22T18:40:15.528565+0000","last_peered":"2024-04-22T18:40:15.528565+0000","last_clean":"2024-04-22T18:40:15.528565+0000","last_became_active":"2024-04-22T18:38:47.172816+0000","last_became_peered":"2024-04-22T18:38:47.172816+0000","last_unstale":"2024-04-22T18:40:15.528565+0000","last_undegraded":"2024-04-22T18:40:15.528565+0000","last_fullsized":"2024-04-22T18:40:15.528565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24302843,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":220,"num_read_kb":3208,"num_write":307,"num_write_kb":78323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.297725+0000","last_change":"2024-04-22T18:40:11.679870+0000","last_active":"2024-04-22T18:40:15.297725+0000","last_peered":"2024-04-22T18:40:15.297725+0000","last_clean":"2024-04-22T18:40:15.297725+0000","last_became_active":"2024-04-22T18:40:02.945025+0000","last_became_peered":"2024-04-22T18:40:02.945025+0000","last_unstale":"2024-04-22T18:40:15.297725+0000","last_undegraded":"2024-04-22T18:40:15.297725+0000","last_fullsized":"2024-04-22T18:40:15.297725+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":159,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.297723+0000","last_change":"2024-04-22T18:39:52.361604+0000","last_active":"2024-04-22T18:40:15.297723+0000","last_peered":"2024-04-22T18:40:15.297723+0000","last_clean":"2024-04-22T18:40:15.297723+0000","last_became_active":"2024-04-22T18:39:51.943071+0000","last_became_peered":"2024-04-22T18:39:51.943071+0000","last_unstale":"2024-04-22T18:40:15.297723+0000","last_undegraded":"2024-04-22T18:40:15.297723+0000","last_fullsized":"2024-04-22T18:40:15.297723+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1949628,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":55801,"num_write":19,"num_write_kb":6188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":3820686,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"83'171","reported_seq":457,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.297629+0000","last_change":"2024-04-22T18:39:56.824261+0000","last_active":"2024-04-22T18:40:15.297629+0000","last_peered":"2024-04-22T18:40:15.297629+0000","last_clean":"2024-04-22T18:40:15.297629+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:40:15.297629+0000","last_undegraded":"2024-04-22T18:40:15.297629+0000","last_fullsized":"2024-04-22T18:40:15.297629+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24285815,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":141,"num_read_kb":1004,"num_write":212,"num_write_kb":53263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.253299+0000","last_change":"2024-04-22T18:40:10.403338+0000","last_active":"2024-04-22T18:40:15.253299+0000","last_peered":"2024-04-22T18:40:15.253299+0000","last_clean":"2024-04-22T18:40:15.253299+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:40:15.253299+0000","last_undegraded":"2024-04-22T18:40:15.253299+0000","last_fullsized":"2024-04-22T18:40:15.253299+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:10.403189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.403189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:34.232014+0000","scrub_duration":0.201433005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"83'49","reported_seq":218,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.297868+0000","last_change":"2024-04-22T18:40:03.361557+0000","last_active":"2024-04-22T18:40:15.297868+0000","last_peered":"2024-04-22T18:40:15.297868+0000","last_clean":"2024-04-22T18:40:15.297868+0000","last_became_active":"2024-04-22T18:40:02.744089+0000","last_became_peered":"2024-04-22T18:40:02.744089+0000","last_unstale":"2024-04-22T18:40:15.297868+0000","last_undegraded":"2024-04-22T18:40:15.297868+0000","last_fullsized":"2024-04-22T18:40:15.297868+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:16.501156+0000","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403434,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":50806,"num_write":12,"num_write_kb":2902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2812934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"84'209","reported_seq":295,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.405139+0000","last_change":"2024-04-22T18:38:47.173337+0000","last_active":"2024-04-22T18:40:15.405139+0000","last_peered":"2024-04-22T18:40:15.405139+0000","last_clean":"2024-04-22T18:40:15.405139+0000","last_became_active":"2024-04-22T18:38:47.173065+0000","last_became_peered":"2024-04-22T18:38:47.173065+0000","last_unstale":"2024-04-22T18:40:15.405139+0000","last_undegraded":"2024-04-22T18:40:15.405139+0000","last_fullsized":"2024-04-22T18:40:15.405139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19725183,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":186,"num_read_kb":5120,"num_write":254,"num_write_kb":62609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":75,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:14.702777+0000","last_change":"2024-04-22T18:38:39.125885+0000","last_active":"2024-04-22T18:40:14.702777+0000","last_peered":"2024-04-22T18:40:14.702777+0000","last_clean":"2024-04-22T18:40:14.702777+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:40:14.702777+0000","last_undegraded":"2024-04-22T18:40:14.702777+0000","last_fullsized":"2024-04-22T18:40:14.702777+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"84'69","reported_seq":244,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T18:40:16.139254+0000","last_change":"2024-04-22T18:40:08.763711+0000","last_active":"2024-04-22T18:40:16.139254+0000","last_peered":"2024-04-22T18:40:16.139254+0000","last_clean":"2024-04-22T18:40:16.139254+0000","last_became_active":"2024-04-22T18:39:46.809886+0000","last_became_peered":"2024-04-22T18:39:46.809886+0000","last_unstale":"2024-04-22T18:40:16.139254+0000","last_undegraded":"2024-04-22T18:40:16.139254+0000","last_fullsized":"2024-04-22T18:40:16.139254+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":2,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:37.433066+0000","scrub_duration":0.22461577699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5983325,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":15513,"num_write":4,"num_write_kb":1154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"84'201","reported_seq":457,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.196136+0000","last_change":"2024-04-22T18:40:11.390249+0000","last_active":"2024-04-22T18:40:15.196136+0000","last_peered":"2024-04-22T18:40:15.196136+0000","last_clean":"2024-04-22T18:40:15.196136+0000","last_became_active":"2024-04-22T18:40:10.784978+0000","last_became_peered":"2024-04-22T18:40:10.784978+0000","last_unstale":"2024-04-22T18:40:15.196136+0000","last_undegraded":"2024-04-22T18:40:15.196136+0000","last_fullsized":"2024-04-22T18:40:15.196136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24171551,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":187,"num_read_kb":4592,"num_write":255,"num_write_kb":55621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":13435025,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":99,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.253299+0000","last_change":"2024-04-22T18:40:07.420765+0000","last_active":"2024-04-22T18:40:15.253299+0000","last_peered":"2024-04-22T18:40:15.253299+0000","last_clean":"2024-04-22T18:40:15.253299+0000","last_became_active":"2024-04-22T18:40:02.741862+0000","last_became_peered":"2024-04-22T18:40:02.741862+0000","last_unstale":"2024-04-22T18:40:15.253299+0000","last_undegraded":"2024-04-22T18:40:15.253299+0000","last_fullsized":"2024-04-22T18:40:15.253299+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:07.420618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:07.420618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.642486+0000","scrub_duration":0.30164867299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":237,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:14.702824+0000","last_change":"2024-04-22T18:39:53.346906+0000","last_active":"2024-04-22T18:40:14.702824+0000","last_peered":"2024-04-22T18:40:14.702824+0000","last_clean":"2024-04-22T18:40:14.702824+0000","last_became_active":"2024-04-22T18:38:38.108568+0000","last_became_peered":"2024-04-22T18:38:38.108568+0000","last_unstale":"2024-04-22T18:40:14.702824+0000","last_undegraded":"2024-04-22T18:40:14.702824+0000","last_fullsized":"2024-04-22T18:40:14.702824+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:39:53.346787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.346787+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:17.891739+0000","scrub_duration":0.20273052499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"77'56","reported_seq":203,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T18:40:16.239469+0000","last_change":"2024-04-22T18:39:55.746188+0000","last_active":"2024-04-22T18:40:16.239469+0000","last_peered":"2024-04-22T18:40:16.239469+0000","last_clean":"2024-04-22T18:40:16.239469+0000","last_became_active":"2024-04-22T18:39:47.108040+0000","last_became_peered":"2024-04-22T18:39:47.108040+0000","last_unstale":"2024-04-22T18:40:16.239469+0000","last_undegraded":"2024-04-22T18:40:16.239469+0000","last_fullsized":"2024-04-22T18:40:16.239469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'38","last_scrub_stamp":"2024-04-22T18:39:55.746067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:55.746067+0000","objects_scrubbed":1,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:04.875560+0000","scrub_duration":0.60877772299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7613571,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":37300,"num_write":21,"num_write_kb":6447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"79'52","reported_seq":192,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:14.702693+0000","last_change":"2024-04-22T18:40:06.087720+0000","last_active":"2024-04-22T18:40:14.702693+0000","last_peered":"2024-04-22T18:40:14.702693+0000","last_clean":"2024-04-22T18:40:14.702693+0000","last_became_active":"2024-04-22T18:40:02.645205+0000","last_became_peered":"2024-04-22T18:40:02.645205+0000","last_unstale":"2024-04-22T18:40:14.702693+0000","last_undegraded":"2024-04-22T18:40:14.702693+0000","last_fullsized":"2024-04-22T18:40:14.702693+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1436998,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":92005,"num_write":30,"num_write_kb":8835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2522610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"81'97","reported_seq":328,"reported_epoch":85,"state":"peering","last_fresh":"2024-04-22T18:40:15.709537+0000","last_change":"2024-04-22T18:40:15.709755+0000","last_active":"2024-04-22T18:40:15.709537+0000","last_peered":"2024-04-22T18:40:14.701657+0000","last_clean":"2024-04-22T18:40:14.701657+0000","last_became_active":"2024-04-22T18:39:46.807956+0000","last_became_peered":"2024-04-22T18:39:46.807956+0000","last_unstale":"2024-04-22T18:40:15.709537+0000","last_undegraded":"2024-04-22T18:40:15.709537+0000","last_fullsized":"2024-04-22T18:40:15.709537+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:34.318564+0000","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8269110,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":7323,"num_write":8,"num_write_kb":1889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"83'66","reported_seq":224,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.088055+0000","last_change":"2024-04-22T18:38:48.372952+0000","last_active":"2024-04-22T18:40:15.088055+0000","last_peered":"2024-04-22T18:40:15.088055+0000","last_clean":"2024-04-22T18:40:15.088055+0000","last_became_active":"2024-04-22T18:38:48.277922+0000","last_became_peered":"2024-04-22T18:38:48.277922+0000","last_unstale":"2024-04-22T18:40:15.088055+0000","last_undegraded":"2024-04-22T18:40:15.088055+0000","last_fullsized":"2024-04-22T18:40:15.088055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9697672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":205,"num_read_kb":248132,"num_write":66,"num_write_kb":17887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"85'107","reported_seq":328,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.811111+0000","last_change":"2024-04-22T18:38:48.269362+0000","last_active":"2024-04-22T18:40:15.811111+0000","last_peered":"2024-04-22T18:40:15.811111+0000","last_clean":"2024-04-22T18:40:15.811111+0000","last_became_active":"2024-04-22T18:38:48.269066+0000","last_became_peered":"2024-04-22T18:38:48.269066+0000","last_unstale":"2024-04-22T18:40:15.811111+0000","last_undegraded":"2024-04-22T18:40:15.811111+0000","last_fullsized":"2024-04-22T18:40:15.811111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12588542,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":344,"num_read_kb":371776,"num_write":109,"num_write_kb":31748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"83'97","reported_seq":311,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.408916+0000","last_change":"2024-04-22T18:38:48.274538+0000","last_active":"2024-04-22T18:40:15.408916+0000","last_peered":"2024-04-22T18:40:15.408916+0000","last_clean":"2024-04-22T18:40:15.408916+0000","last_became_active":"2024-04-22T18:38:48.174204+0000","last_became_peered":"2024-04-22T18:38:48.174204+0000","last_unstale":"2024-04-22T18:40:15.408916+0000","last_undegraded":"2024-04-22T18:40:15.408916+0000","last_fullsized":"2024-04-22T18:40:15.408916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11865808,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":358,"num_read_kb":427281,"num_write":100,"num_write_kb":28277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"78'43","reported_seq":205,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:14.801811+0000","last_change":"2024-04-22T18:39:52.257129+0000","last_active":"2024-04-22T18:40:14.801811+0000","last_peered":"2024-04-22T18:40:14.801811+0000","last_clean":"2024-04-22T18:40:14.801811+0000","last_became_active":"2024-04-22T18:39:51.934575+0000","last_became_peered":"2024-04-22T18:39:51.934575+0000","last_unstale":"2024-04-22T18:40:14.801811+0000","last_undegraded":"2024-04-22T18:40:14.801811+0000","last_fullsized":"2024-04-22T18:40:14.801811+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5697606,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":150,"num_read_kb":153878,"num_write":45,"num_write_kb":13377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":6895340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"83'23","reported_seq":129,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:14.701605+0000","last_change":"2024-04-22T18:38:48.075222+0000","last_active":"2024-04-22T18:40:14.701605+0000","last_peered":"2024-04-22T18:40:14.701605+0000","last_clean":"2024-04-22T18:40:14.701605+0000","last_became_active":"2024-04-22T18:38:48.074901+0000","last_became_peered":"2024-04-22T18:38:48.074901+0000","last_unstale":"2024-04-22T18:40:14.701605+0000","last_undegraded":"2024-04-22T18:40:14.701605+0000","last_fullsized":"2024-04-22T18:40:14.701605+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252171,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":93442,"num_write":25,"num_write_kb":7406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"83'68","reported_seq":213,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-22T18:40:15.297811+0000","last_change":"2024-04-22T18:38:48.173640+0000","last_active":"2024-04-22T18:40:15.297811+0000","last_peered":"2024-04-22T18:40:15.297811+0000","last_clean":"2024-04-22T18:40:15.297811+0000","last_became_active":"2024-04-22T18:38:48.173369+0000","last_became_peered":"2024-04-22T18:38:48.173369+0000","last_unstale":"2024-04-22T18:40:15.297811+0000","last_undegraded":"2024-04-22T18:40:15.297811+0000","last_fullsized":"2024-04-22T18:40:15.297811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6638864,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":166,"num_read_kb":155496,"num_write":71,"num_write_kb":18866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":25,"stat_sum":{"num_bytes":133374018,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2379,"num_read_kb":2731784,"num_write":812,"num_write_kb":227936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":45670082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":29,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202104832,"data_stored":201456637,"data_compressed":462883,"data_compressed_allocated":491520,"data_compressed_original":834758,"omap_allocated":0,"internal_metadata":0},"log_size":1447,"ondisk_log_size":1447,"up":50,"acting":50,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":92485392,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":734,"num_read_kb":13924,"num_write":1028,"num_write_kb":249816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":34294349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":108863488,"data_stored":115919225,"data_compressed":6753852,"data_compressed_allocated":7184384,"data_compressed_original":14589952,"omap_allocated":0,"internal_metadata":0},"log_size":838,"ondisk_log_size":838,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59136,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44892,"kb_used_data":13768,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94326948,"statfs":{"total":96636764160,"available":96590794752,"internally_reserved":0,"allocated":14098432,"data_stored":14083466,"data_compressed":189598,"data_compressed_allocated":208896,"data_compressed_original":774144,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.36299999999999999},{"interface":"front","average":{"1min":4.508,"5min":4.508,"15min":4.508},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":85.542000000000002,"5min":85.542000000000002,"15min":85.542000000000002},"last":0.38100000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.8650000000000002,"5min":4.8650000000000002,"15min":4.8650000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":79.927999999999997,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":13.048,"5min":13.048,"15min":13.048},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":99.388999999999996,"5min":99.388999999999996,"15min":99.388999999999996},"last":0.46200000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.085,"5min":4.085,"15min":4.085},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":77.320999999999998,"5min":77.320999999999998,"15min":77.320999999999998},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.48199999999999998}]},{"osd":3,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.1509999999999998,"5min":4.1509999999999998,"15min":4.1509999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":77.361000000000004,"5min":77.361000000000004,"15min":77.361000000000004},"last":0.437},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.44700000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.4589999999999996,"5min":4.4589999999999996,"15min":4.4589999999999996},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":85.509,"5min":85.509,"15min":85.509},"last":0.27400000000000002},{"interface":"front","average":{"1min":22.872,"5min":22.872,"15min":22.872},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":327.77300000000002,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.32100000000000001}]},{"osd":5,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":3.0710000000000002,"5min":3.0710000000000002,"15min":3.0710000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":55.116999999999997,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.28999999999999998},{"interface":"front","average":{"1min":7.1619999999999999,"5min":7.1619999999999999,"15min":7.1619999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":85.475999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.42199999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":13.276,"5min":13.276,"15min":13.276},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":99.620999999999995,"5min":99.620999999999995,"15min":99.620999999999995},"last":0.34599999999999997},{"interface":"front","average":{"1min":4.133,"5min":4.133,"15min":4.133},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":77.405000000000001,"5min":77.405000000000001,"15min":77.405000000000001},"last":0.47299999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67980,"kb_used_data":40280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94303860,"statfs":{"total":96636764160,"available":96567152640,"internally_reserved":0,"allocated":41246720,"data_stored":41531708,"data_compressed":988880,"data_compressed_allocated":1048576,"data_compressed_original":2048000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.794,"15min":1.794},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":32.097999999999999,"5min":32.097999999999999,"15min":32.097999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":25.808,"5min":25.808,"15min":25.808},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":166.85499999999999,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.46600000000000003}]},{"osd":1,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":37.161999999999999,"5min":37.161999999999999,"15min":37.161999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":10.092000000000001,"5min":10.092000000000001,"15min":10.092000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":222.55199999999999,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.32200000000000001},{"interface":"front","average":{"1min":18.803999999999998,"5min":18.803999999999998,"15min":18.803999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":166.81800000000001,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.48799999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":21.518999999999998,"5min":21.518999999999998,"15min":21.518999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":171.99100000000001,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.42599999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":7.6029999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":81.697000000000003,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.39100000000000001},{"interface":"front","average":{"1min":19.327999999999999,"5min":19.327999999999999,"15min":19.327999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":166.733,"5min":166.733,"15min":166.733},"last":0.41199999999999998}]},{"osd":5,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.434}]},{"osd":7,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":18.850999999999999,"5min":18.850999999999999,"15min":18.850999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":166.79400000000001,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.505},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.39800000000000002}]}]},{"osd":5,"up_from":0,"seq":55834574870,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100332,"kb_used_data":68532,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94271508,"statfs":{"total":96636764160,"available":96534024192,"internally_reserved":0,"allocated":70176768,"data_stored":71105029,"data_compressed":1455878,"data_compressed_allocated":1548288,"data_compressed_original":3225187,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":16.648,"5min":16.648,"15min":16.648},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":86.712999999999994,"5min":86.712999999999994,"15min":86.712999999999994},"last":0.312},{"interface":"front","average":{"1min":8.5679999999999996,"5min":8.5679999999999996,"15min":8.5679999999999996},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":78.158000000000001,"5min":78.158000000000001,"15min":78.158000000000001},"last":0.40799999999999997}]},{"osd":1,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.34999999999999998},{"interface":"front","average":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":78.866,"5min":78.866,"15min":78.866},"last":0.379}]},{"osd":2,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":8.5700000000000003,"5min":8.5700000000000003,"15min":8.5700000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":76.724999999999994,"5min":76.724999999999994,"15min":76.724999999999994},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.47199999999999998}]},{"osd":3,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":4.0220000000000002,"5min":4.0220000000000002,"15min":4.0220000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":78.912000000000006,"5min":78.912000000000006,"15min":78.912000000000006},"last":0.378},{"interface":"front","average":{"1min":20.968,"5min":20.968,"15min":20.968},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":227.41200000000001,"5min":227.41200000000001,"15min":227.41200000000001},"last":20.917999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.503},{"interface":"front","average":{"1min":4.0380000000000003,"5min":4.0380000000000003,"15min":4.0380000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":78.811999999999998,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.439}]},{"osd":6,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":13.16,"5min":13.16,"15min":13.16},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":124.303,"5min":124.303,"15min":124.303},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.42599999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":8.5600000000000005,"5min":8.5600000000000005,"15min":8.5600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":76.697000000000003,"5min":76.697000000000003,"15min":76.697000000000003},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.39700000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64980,"kb_used_data":31076,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94306860,"statfs":{"total":96636764160,"available":96570224640,"internally_reserved":0,"allocated":31821824,"data_stored":32058096,"data_compressed":828850,"data_compressed_allocated":880640,"data_compressed_original":1785856,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":12.199999999999999,"5min":12.199999999999999,"15min":12.199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":98.957999999999998,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.29699999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":5.0449999999999999,"5min":5.0449999999999999,"15min":5.0449999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":94.063000000000002,"5min":94.063000000000002,"15min":94.063000000000002},"last":0.254},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":16.388999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":12.901999999999999,"5min":12.901999999999999,"15min":12.901999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.27800000000000002}]},{"osd":4,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":4.1079999999999997,"5min":4.1079999999999997,"15min":4.1079999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":71.998000000000005,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":4.1379999999999999,"5min":4.1379999999999999,"15min":4.1379999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":72.227000000000004,"5min":72.227000000000004,"15min":72.227000000000004},"last":0.46800000000000003}]},{"osd":5,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":13.94,"5min":13.94,"15min":13.94},"last":0.372},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":13.965,"5min":13.965,"15min":13.965},"last":0.40799999999999997}]},{"osd":6,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.44500000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.51000000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48984,"kb_used_data":21956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94322856,"statfs":{"total":96636764160,"available":96586604544,"internally_reserved":0,"allocated":22482944,"data_stored":23095836,"data_compressed":1111203,"data_compressed_allocated":1183744,"data_compressed_original":2342912,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":149.03,"5min":149.03,"15min":149.03},"last":0.25900000000000001},{"interface":"front","average":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":31.866,"5min":31.866,"15min":31.866},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":13.868,"5min":13.868,"15min":13.868},"last":0.53800000000000003},{"interface":"front","average":{"1min":14.885,"5min":14.885,"15min":14.885},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":160.37100000000001,"5min":160.37100000000001,"15min":160.37100000000001},"last":0.29199999999999998}]},{"osd":3,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.37}]},{"osd":4,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":18.771999999999998,"5min":18.771999999999998,"15min":18.771999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":160.584,"5min":160.584,"15min":160.584},"last":0.32300000000000001},{"interface":"front","average":{"1min":2.448,"5min":2.448,"15min":2.448},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":36.841000000000001,"5min":36.841000000000001,"15min":36.841000000000001},"last":0.48799999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":3.052,"5min":3.052,"15min":3.052},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":48.631,"5min":48.631,"15min":48.631},"last":0.40200000000000002},{"interface":"front","average":{"1min":3.476,"5min":3.476,"15min":3.476},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":56.18,"5min":56.18,"15min":56.18},"last":0.41699999999999998}]},{"osd":6,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42899999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.51300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93624,"kb_used_data":56392,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94278216,"statfs":{"total":96636764160,"available":96540893184,"internally_reserved":0,"allocated":57745408,"data_stored":58186668,"data_compressed":1155786,"data_compressed_allocated":1232896,"data_compressed_original":2467427,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":7.859,"5min":7.859,"15min":7.859},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":88.171000000000006,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.28799999999999998},{"interface":"front","average":{"1min":7.8630000000000004,"5min":7.8630000000000004,"15min":7.8630000000000004},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":88.156000000000006,"5min":88.156000000000006,"15min":88.156000000000006},"last":0.36499999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":8.5999999999999996,"5min":8.5999999999999996,"15min":8.5999999999999996},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":183.04400000000001,"5min":183.04400000000001,"15min":183.04400000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.25600000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.216},{"interface":"front","average":{"1min":14.492000000000001,"5min":14.492000000000001,"15min":14.492000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":231.864,"5min":231.864,"15min":231.864},"last":0.32800000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":12.984,"5min":12.984,"15min":12.984},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":192.42099999999999,"5min":192.42099999999999,"15min":192.42099999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.45700000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":14.071,"5min":14.071,"15min":14.071},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":138.089,"5min":138.089,"15min":138.089},"last":0.44800000000000001},{"interface":"front","average":{"1min":20.077000000000002,"5min":20.077000000000002,"15min":20.077000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":183.08199999999999,"5min":183.08199999999999,"15min":183.08199999999999},"last":0.436}]},{"osd":6,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":51.070999999999998,"5min":51.070999999999998,"15min":51.070999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":16.532,"5min":16.532,"15min":16.532},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":183.10900000000001,"5min":183.10900000000001,"15min":183.10900000000001},"last":0.41599999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.48999999999999999},{"interface":"front","average":{"1min":3.0390000000000001,"5min":3.0390000000000001,"15min":3.0390000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":51.097000000000001,"5min":51.097000000000001,"15min":51.097000000000001},"last":0.42399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76744,"kb_used_data":45092,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94295096,"statfs":{"total":96636764160,"available":96558178304,"internally_reserved":0,"allocated":46174208,"data_stored":47602574,"data_compressed":1492019,"data_compressed_allocated":1597440,"data_compressed_original":3743744,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":6.7599999999999998,"5min":6.7599999999999998,"15min":6.7599999999999998},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":133.09899999999999,"5min":133.09899999999999,"15min":133.09899999999999},"last":59.875},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":25.827999999999999,"5min":25.827999999999999,"15min":25.827999999999999},"last":0.23200000000000001}]},{"osd":2,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":25.873000000000001,"5min":25.873000000000001,"15min":25.873000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.318,"5min":0.318,"15min":0.318},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.33800000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":0.25},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":25.898,"5min":25.898,"15min":25.898},"last":0.371}]},{"osd":4,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":9.9120000000000008,"5min":9.9120000000000008,"15min":9.9120000000000008},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":133.036,"5min":133.036,"15min":133.036},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":26.16,"5min":26.16,"15min":26.16},"last":0.45500000000000002},{"interface":"front","average":{"1min":13.955,"5min":13.955,"15min":13.955},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":282.197,"5min":282.197,"15min":282.197},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":3.198,"5min":3.198,"15min":3.198},"last":0.44},{"interface":"front","average":{"1min":1.748,"5min":1.748,"15min":1.748},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":26.190000000000001,"5min":26.190000000000001,"15min":26.190000000000001},"last":0.42599999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":4.6779999999999999,"5min":4.6779999999999999,"15min":4.6779999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":87.209999999999994,"5min":87.209999999999994,"15min":87.209999999999994},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.2440000000000002,"5min":3.2440000000000002,"15min":3.2440000000000002},"last":0.47499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26791936,"data_stored":28359883,"data_compressed":1436873,"data_compressed_allocated":1527808,"data_compressed_original":3166208,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15175680,"data_stored":16298240,"data_compressed":1111203,"data_compressed_allocated":1183744,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15175680,"data_stored":16298240,"data_compressed":1111203,"data_compressed_allocated":1183744,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12349440,"data_stored":13189790,"data_compressed":828850,"data_compressed_allocated":880640,"data_compressed_original":1785856,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":26791936,"data_stored":28359883,"data_compressed":1436873,"data_compressed_allocated":1527808,"data_compressed_original":3166208,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":223399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12349440,"data_stored":13189790,"data_compressed":828850,"data_compressed_allocated":880640,"data_compressed_original":1785856,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18374656,"data_stored":18302657,"data_compressed":25578,"data_compressed_allocated":28672,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41615360,"data_stored":41477133,"data_compressed":44583,"data_compressed_allocated":49152,"data_compressed_original":124515,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6430720,"data_stored":6397277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":18829312,"data_stored":18735468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":42487808,"data_stored":42338719,"data_compressed":19005,"data_compressed_allocated":20480,"data_compressed_original":58979,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33280000,"data_stored":33118894,"data_compressed":53657,"data_compressed_allocated":57344,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27963392,"data_stored":27936218,"data_compressed":160030,"data_compressed_allocated":167936,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":13123584,"data_stored":13150271,"data_compressed":160030,"data_compressed_allocated":167936,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:40:20.517 INFO:tasks.thrashosds.thrasher:Setting 4.11 to [2, 7, 3, 1] 2024-04-22T18:40:20.517 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.11', '2', '7', '3', '1'] 2024-04-22T18:40:20.518 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 4.11 2 7 3 1 2024-04-22T18:40:20.560 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:20.561+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:20.560 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:20.561 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:20.570 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:20.597 INFO:tasks.rados.rados.0.smithi089.stdout:1391: expect (ObjNum 253 snap 0 seq_num 253) 2024-04-22T18:40:20.661 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:20.662+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:20.702 INFO:tasks.rados.rados.0.smithi089.stdout:1392: finishing write tid 1 to smithi08937929-35 2024-04-22T18:40:20.702 INFO:tasks.rados.rados.0.smithi089.stdout:1390: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:40:20.702 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:20.702 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:20.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:20.762 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:20.763+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:20.827 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:20.827 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:20.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:20.849 INFO:tasks.rados.rados.0.smithi089.stdout:1392: finishing write tid 2 to smithi08937929-35 2024-04-22T18:40:20.850 INFO:tasks.rados.rados.0.smithi089.stdout:1392: finishing write tid 3 to smithi08937929-35 2024-04-22T18:40:20.850 INFO:tasks.rados.rados.0.smithi089.stdout:1392: finishing write tid 4 to smithi08937929-35 2024-04-22T18:40:20.850 INFO:tasks.rados.rados.0.smithi089.stdout:1392: oid 35 updating version 0 to 99 2024-04-22T18:40:20.850 INFO:tasks.rados.rados.0.smithi089.stdout:1392: oid 35 updating version 99 to 100 2024-04-22T18:40:20.850 INFO:tasks.rados.rados.0.smithi089.stdout:1392: oid 35 version 100 is already newer than 98 2024-04-22T18:40:20.850 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 100 (ObjNum 270 snap 0 seq_num 270) dirty exists 2024-04-22T18:40:20.850 INFO:tasks.rados.rados.0.smithi089.stdout:1392: left oid 35 (ObjNum 270 snap 0 seq_num 270) 2024-04-22T18:40:20.850 INFO:tasks.rados.rados.0.smithi089.stdout:1387: finishing write tid 2 to smithi08937929-4 2024-04-22T18:40:20.850 INFO:tasks.rados.rados.0.smithi089.stdout:1388: finishing set_chunk to oid 49 2024-04-22T18:40:20.850 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 106 (ObjNum 258 snap 0 seq_num 258) dirty exists 2024-04-22T18:40:20.850 INFO:tasks.rados.rados.0.smithi089.stdout:1387: finishing write tid 3 to smithi08937929-4 2024-04-22T18:40:20.851 INFO:tasks.rados.rados.0.smithi089.stdout:1380: done (13 left) 2024-04-22T18:40:20.851 INFO:tasks.rados.rados.0.smithi089.stdout:1382: done (12 left) 2024-04-22T18:40:20.851 INFO:tasks.rados.rados.0.smithi089.stdout:1383: done (11 left) 2024-04-22T18:40:20.851 INFO:tasks.rados.rados.0.smithi089.stdout:1384: done (10 left) 2024-04-22T18:40:20.851 INFO:tasks.rados.rados.0.smithi089.stdout:1385: done (9 left) 2024-04-22T18:40:20.851 INFO:tasks.rados.rados.0.smithi089.stdout:1386: done (8 left) 2024-04-22T18:40:20.851 INFO:tasks.rados.rados.0.smithi089.stdout:1388: done (7 left) 2024-04-22T18:40:20.851 INFO:tasks.rados.rados.0.smithi089.stdout:1389: done (6 left) 2024-04-22T18:40:20.851 INFO:tasks.rados.rados.0.smithi089.stdout:1390: done (5 left) 2024-04-22T18:40:20.851 INFO:tasks.rados.rados.0.smithi089.stdout:1391: done (4 left) 2024-04-22T18:40:20.851 INFO:tasks.rados.rados.0.smithi089.stdout:1392: done (3 left) 2024-04-22T18:40:20.851 INFO:tasks.rados.rados.0.smithi089.stdout:1395: read oid 8 snap -1 2024-04-22T18:40:20.851 INFO:tasks.rados.rados.0.smithi089.stdout:1395: expect (ObjNum 253 snap 0 seq_num 253) 2024-04-22T18:40:20.852 INFO:tasks.rados.rados.0.smithi089.stdout:1396: read oid 13 snap -1 2024-04-22T18:40:20.852 INFO:tasks.rados.rados.0.smithi089.stdout:1396: expect (ObjNum 261 snap 0 seq_num 261) 2024-04-22T18:40:20.852 INFO:tasks.rados.rados.0.smithi089.stdout:1397: read oid 36 snap -1 2024-04-22T18:40:20.852 INFO:tasks.rados.rados.0.smithi089.stdout:1397: expect (ObjNum 208 snap 0 seq_num 208) 2024-04-22T18:40:20.852 INFO:tasks.rados.rados.0.smithi089.stdout:1398: write oid 16 current snap is 0 2024-04-22T18:40:20.852 INFO:tasks.rados.rados.0.smithi089.stdout:1387: finishing write tid 4 to smithi08937929-4 2024-04-22T18:40:20.852 INFO:tasks.rados.rados.0.smithi089.stdout:1387: oid 4 updating version 0 to 101 2024-04-22T18:40:20.852 INFO:tasks.rados.rados.0.smithi089.stdout:1387: oid 4 updating version 101 to 103 2024-04-22T18:40:20.852 INFO:tasks.rados.rados.0.smithi089.stdout:1387: oid 4 version 103 is already newer than 102 2024-04-22T18:40:20.852 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 103 (ObjNum 269 snap 0 seq_num 269) dirty exists 2024-04-22T18:40:20.852 INFO:tasks.rados.rados.0.smithi089.stdout:1387: left oid 4 (ObjNum 269 snap 0 seq_num 269) 2024-04-22T18:40:20.852 INFO:tasks.rados.rados.0.smithi089.stdout:1398: seq_num 272 ranges {571504=540083,1783808=702748} 2024-04-22T18:40:20.854 INFO:tasks.rados.rados.0.smithi089.stdout:1398: writing smithi08937929-16 from 571504 to 1111587 tid 1 2024-04-22T18:40:20.863 INFO:tasks.rados.rados.0.smithi089.stdout:1398: writing smithi08937929-16 from 1783808 to 2486556 tid 2 2024-04-22T18:40:20.863 INFO:tasks.rados.rados.0.smithi089.stdout:1387: done (6 left) 2024-04-22T18:40:20.863 INFO:tasks.rados.rados.0.smithi089.stdout:1399: read oid 11 snap -1 2024-04-22T18:40:20.863 INFO:tasks.rados.rados.0.smithi089.stdout:1399: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:40:20.863 INFO:tasks.rados.rados.0.smithi089.stdout:1393: finishing write tid 1 to smithi08937929-5 2024-04-22T18:40:20.863 INFO:tasks.rados.rados.0.smithi089.stdout:1400: read oid 28 snap -1 2024-04-22T18:40:20.863 INFO:tasks.rados.rados.0.smithi089.stdout:1400: expect (ObjNum 265 snap 0 seq_num 265) 2024-04-22T18:40:20.863 INFO:tasks.rados.rados.0.smithi089.stdout:1401: set_chunk oid 32 target oid 2024-04-22T18:40:20.864 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:20.864+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:20.938 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:20.938 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:20.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:20.964 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:20.965+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:21.051 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:21.051 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:21.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:21.078 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:40:21.103 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:21.104+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:21.162 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:21.162 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:21.164 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:21.165+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:21.171 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:21.196 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:21.197 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:21.213 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:21.214 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:21.229 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:21.229 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:21.245 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:21.245 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:21.262 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:21.262 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:21.265 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:21.266+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:21.278 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:21.278 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:21.314 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:21.314 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:21.314 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:21.314 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:21.314 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:21.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:21.358 INFO:tasks.rados.rados.0.smithi089.stdout:1401: set_chunk oid 32 offset: 901632 length: 2593792 target oid bb7b99a5e8d8163793d64d588df4e35b93bb9664659ea19ac0daf22339214ef4 offset: 0 snap -1 2024-04-22T18:40:21.358 INFO:tasks.rados.rados.0.smithi089.stdout:1402: read oid 38 snap -1 2024-04-22T18:40:21.358 INFO:tasks.rados.rados.0.smithi089.stdout:1402: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:40:21.358 INFO:tasks.rados.rados.0.smithi089.stdout:1403: tier_flush oid 43 2024-04-22T18:40:21.358 INFO:tasks.rados.rados.0.smithi089.stdout:1403: tier_flush oid 43 snap -1 2024-04-22T18:40:21.358 INFO:tasks.rados.rados.0.smithi089.stdout:1394: got (0) Success 2024-04-22T18:40:21.358 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 63 (ObjNum 181 snap 0 seq_num 181) dirty exists 2024-04-22T18:40:21.358 INFO:tasks.rados.rados.0.smithi089.stdout:1393: finishing write tid 2 to smithi08937929-5 2024-04-22T18:40:21.359 INFO:tasks.rados.rados.0.smithi089.stdout:1393: finishing write tid 3 to smithi08937929-5 2024-04-22T18:40:21.359 INFO:tasks.rados.rados.0.smithi089.stdout:1393: finishing write tid 4 to smithi08937929-5 2024-04-22T18:40:21.359 INFO:tasks.rados.rados.0.smithi089.stdout:1393: oid 5 updating version 0 to 140 2024-04-22T18:40:21.359 INFO:tasks.rados.rados.0.smithi089.stdout:1393: oid 5 updating version 140 to 142 2024-04-22T18:40:21.359 INFO:tasks.rados.rados.0.smithi089.stdout:1393: oid 5 version 142 is already newer than 141 2024-04-22T18:40:21.359 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 142 (ObjNum 271 snap 0 seq_num 271) dirty exists 2024-04-22T18:40:21.359 INFO:tasks.rados.rados.0.smithi089.stdout:1393: left oid 5 (ObjNum 271 snap 0 seq_num 271) 2024-04-22T18:40:21.359 INFO:tasks.rados.rados.0.smithi089.stdout:1400: expect (ObjNum 265 snap 0 seq_num 265) 2024-04-22T18:40:21.359 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:21.360 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:21.366 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:21.367+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:21.379 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:21.467 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:21.468+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:21.470 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:40:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.375987734000006, 2024-04-22T18:40:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:40:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:21.471 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:21.471 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:21.471 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:40:21.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.471 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:21.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:21.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:40:21.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:21.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:40:21.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:40:21.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:40:21.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:40:21.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:40:21.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:40:21.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:40:21.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:40:21.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:40:21.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:40:21.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:21.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:40:21.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:40:21.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:40:21.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:40:21.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:40:21.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:40:21.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:40:21.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:40:21.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.476489858999997, 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.476 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:40:21.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:40:21.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:40:21.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:40:21.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:40:21.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:40:21.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:40:21.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:40:21.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:21.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:40:21.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:40:21.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:40:21.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:40:21.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:40:21.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:40:21.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:21.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:40:21.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:40:21.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:40:21.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:40:21.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:40:21.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:40:21.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:40:21.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:40:21.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:40:21.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:40:21.482 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.482 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.482 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.482 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:40:21.482 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:21.482 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 82.368190068000004, 2024-04-22T18:40:21.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:40:21.483 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.483 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.483 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:21.483 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:21.483 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:21.483 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:40:21.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.483 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:21.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:21.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:40:21.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:21.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:40:21.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:40:21.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:40:21.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:40:21.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:40:21.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:40:21.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:40:21.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:40:21.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:40:21.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:40:21.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:40:21.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:40:21.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:40:21.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:40:21.488 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.488 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.488 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.488 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:21.488 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:21.488 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.036646439999998, 2024-04-22T18:40:21.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:40:21.488 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.488 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.488 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:21.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:21.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:40:21.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:21.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:40:21.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:40:21.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:40:21.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:40:21.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:40:21.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:40:21.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:40:21.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:40:21.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:40:21.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:40:21.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:21.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:40:21.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:40:21.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:40:21.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:40:21.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:40:21.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:40:21.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:40:21.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:40:21.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:40:21.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:40:21.493 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.493 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.493 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.493 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:21.493 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:21.493 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 68.945560631999996, 2024-04-22T18:40:21.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:40:21.493 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.493 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.494 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:21.494 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:21.494 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:21.494 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:40:21.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.494 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:21.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:40:21.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:40:21.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:40:21.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:40:21.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:40:21.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:40:21.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:40:21.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:40:21.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:21.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:40:21.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:40:21.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:40:21.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:40:21.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:40:21.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:40:21.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:40:21.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:40:21.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:40:21.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:40:21.499 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.499 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.499 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.499 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:21.499 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:21.499 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 68.939760371999995, 2024-04-22T18:40:21.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:40:21.499 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.499 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.499 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:21.499 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:21.499 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:21.500 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:40:21.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.500 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:21.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:21.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:40:21.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:21.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:40:21.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:40:21.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:40:21.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:40:21.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:40:21.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:40:21.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:40:21.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:40:21.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:40:21.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:40:21.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:21.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:40:21.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:40:21.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:40:21.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:40:21.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:40:21.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:40:21.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:40:21.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:40:21.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:40:21.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:40:21.504 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.504 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.504 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.505 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:21.505 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:21.505 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 68.935020029, 2024-04-22T18:40:21.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:40:21.505 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.505 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.505 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:21.505 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:21.505 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:21.505 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:40:21.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.505 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:21.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:21.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:40:21.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:21.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:40:21.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:40:21.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:40:21.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:40:21.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:40:21.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:40:21.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:40:21.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:40:21.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:40:21.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:40:21.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:21.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:40:21.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:40:21.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:40:21.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:40:21.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:40:21.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:40:21.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:40:21.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:40:21.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:40:21.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:40:21.510 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.510 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.510 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.510 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:21.510 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:21.510 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 68.934547131000002, 2024-04-22T18:40:21.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:40:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:40:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:21.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:40:21.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:21.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:40:21.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:40:21.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:40:21.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:40:21.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:40:21.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:40:21.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:40:21.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:40:21.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:40:21.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:40:21.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:21.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:40:21.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:40:21.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:40:21.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:40:21.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:40:21.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:40:21.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:40:21.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:40:21.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:40:21.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:40:21.516 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.516 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.516 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.516 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:21.516 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:21.516 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 68.934527209999999, 2024-04-22T18:40:21.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:40:21.516 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.516 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:21.516 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:21.516 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:21.516 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:21.517 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:40:21.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.517 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:21.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:21.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:21.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:21.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:40:21.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:40:21.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:40:21.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:40:21.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:40:21.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:40:21.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:40:21.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:40:21.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:21.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:40:21.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:40:21.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:40:21.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:40:21.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:40:21.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:40:21.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:40:21.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:40:21.521 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.521 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.521 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.521 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:21.521 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:21.521 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 63.625203444999997, 2024-04-22T18:40:21.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:40:21.521 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.521 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.521 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:21.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:21.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:40:21.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:21.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:40:21.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:40:21.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:40:21.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:40:21.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:40:21.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:40:21.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:40:21.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:40:21.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:40:21.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:40:21.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:21.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:40:21.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:40:21.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:40:21.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:40:21.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:40:21.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:40:21.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:40:21.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:40:21.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:40:21.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:40:21.526 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.526 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.526 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.526 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:21.526 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:21.526 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 63.624995908999999, 2024-04-22T18:40:21.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:40:21.527 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.527 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.527 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:21.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:21.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:40:21.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:21.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:40:21.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:40:21.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:40:21.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:40:21.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:40:21.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:40:21.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:40:21.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:40:21.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:40:21.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:40:21.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:21.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:40:21.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:40:21.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:40:21.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:40:21.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:40:21.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:40:21.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:40:21.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:40:21.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:40:21.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:40:21.531 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.531 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.531 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.532 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:21.532 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:21.532 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 44.759346176000001, 2024-04-22T18:40:21.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:40:21.532 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.532 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.532 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:21.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:21.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:40:21.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:21.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:40:21.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:40:21.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:40:21.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:40:21.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:40:21.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:40:21.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:40:21.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:40:21.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:40:21.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:40:21.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:21.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:40:21.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:40:21.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:40:21.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:40:21.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:40:21.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:40:21.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:40:21.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:40:21.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:40:21.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:40:21.536 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.536 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.537 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.537 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:21.537 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:21.537 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 44.545507790999999, 2024-04-22T18:40:21.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:40:21.537 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.537 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.537 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:21.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:21.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:40:21.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:21.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:40:21.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:40:21.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:40:21.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:40:21.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:40:21.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:40:21.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:40:21.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:40:21.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:40:21.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:40:21.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:21.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:40:21.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:40:21.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:40:21.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:40:21.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:40:21.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:40:21.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:40:21.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:40:21.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:40:21.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:40:21.542 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.542 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.542 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.542 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:21.542 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:21.542 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 44.545474380999998, 2024-04-22T18:40:21.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:40:21.543 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.543 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.543 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:21.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:21.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:21.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:21.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:40:21.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:40:21.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:40:21.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:40:21.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:40:21.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:40:21.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:40:21.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:40:21.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:40:21.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:40:21.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:21.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:40:21.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:40:21.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:40:21.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:40:21.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:40:21.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:40:21.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:40:21.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:40:21.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:40:21.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:40:21.548 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.548 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.548 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.548 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:21.548 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:21.548 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 44.486591726999997, 2024-04-22T18:40:21.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:40:21.548 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.549 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.549 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:21.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:21.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:40:21.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:21.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:40:21.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:40:21.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:40:21.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:40:21.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:40:21.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:40:21.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:40:21.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:40:21.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:40:21.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:40:21.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:21.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:40:21.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:40:21.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:40:21.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:40:21.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:40:21.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:40:21.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:40:21.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:40:21.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:40:21.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:40:21.553 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.553 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.553 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.554 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:21.554 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:21.554 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 44.486453111000003, 2024-04-22T18:40:21.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:40:21.554 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.554 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.554 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:21.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:21.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:21.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:21.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:40:21.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:40:21.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:40:21.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:40:21.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:40:21.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:40:21.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:40:21.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:40:21.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:40:21.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:40:21.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:21.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:40:21.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:40:21.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:40:21.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:40:21.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:40:21.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:40:21.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:40:21.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:40:21.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:40:21.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:40:21.559 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.559 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.559 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.559 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:21.559 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:21.560 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 44.486434107999997, 2024-04-22T18:40:21.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:40:21.560 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.560 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.560 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:21.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:21.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:21.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:21.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:40:21.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:40:21.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:40:21.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:40:21.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:40:21.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:40:21.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:40:21.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:40:21.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:40:21.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:40:21.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:21.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:40:21.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:40:21.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:40:21.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:40:21.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:40:21.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:40:21.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:40:21.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:40:21.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:40:21.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:40:21.565 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.565 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.565 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.565 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:21.565 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:21.565 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.865421574999999, 2024-04-22T18:40:21.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:40:21.565 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.565 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.565 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:21.565 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:21.565 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:21.566 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:40:21.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.566 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:21.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:21.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:40:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:40:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:40:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:40:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:40:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:40:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:40:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:40:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:40:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:40:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:40:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:40:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:40:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:40:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:40:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:40:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:40:21.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:40:21.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:40:21.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:40:21.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:40:21.570 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.570 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.570 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.571 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:21.571 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:21.571 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.864901983999999, 2024-04-22T18:40:21.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:40:21.571 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.571 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.571 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:21.571 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:21.571 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:21.571 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:40:21.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.571 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:40:21.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:40:21.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:40:21.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:40:21.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:40:21.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:40:21.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:40:21.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:40:21.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:40:21.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:40:21.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:21.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:40:21.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:40:21.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:40:21.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:40:21.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:40:21.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:40:21.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:40:21.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:40:21.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:40:21.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:40:21.576 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.576 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.576 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.576 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:21.577 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:21.577 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.764749101, 2024-04-22T18:40:21.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:40:21.577 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.577 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:21.577 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:21.577 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:21.577 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:21.577 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:40:21.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.577 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:21.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:21.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:40:21.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:21.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:40:21.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:40:21.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:40:21.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:40:21.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:40:21.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:40:21.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:40:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:40:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:40:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:40:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:40:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:40:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:40:21.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:40:21.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:40:21.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:40:21.581 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.581 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.581 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.581 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.581 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.581 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:21.583 INFO:tasks.rados.rados.0.smithi089.stdout:1398: finishing write tid 1 to smithi08937929-16 2024-04-22T18:40:21.583 INFO:tasks.rados.rados.0.smithi089.stdout:1398: finishing write tid 2 to smithi08937929-16 2024-04-22T18:40:21.583 INFO:tasks.rados.rados.0.smithi089.stdout:1398: finishing write tid 3 to smithi08937929-16 2024-04-22T18:40:21.583 INFO:tasks.rados.rados.0.smithi089.stdout:1395: expect (ObjNum 253 snap 0 seq_num 253) 2024-04-22T18:40:21.583 INFO:tasks.rados.rados.0.smithi089.stdout:1398: finishing write tid 4 to smithi08937929-16 2024-04-22T18:40:21.583 INFO:tasks.rados.rados.0.smithi089.stdout:1398: oid 16 updating version 0 to 113 2024-04-22T18:40:21.583 INFO:tasks.rados.rados.0.smithi089.stdout:1398: oid 16 version 113 is already newer than 111 2024-04-22T18:40:21.583 INFO:tasks.rados.rados.0.smithi089.stdout:1398: oid 16 version 113 is already newer than 112 2024-04-22T18:40:21.583 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 113 (ObjNum 272 snap 0 seq_num 272) dirty exists 2024-04-22T18:40:21.583 INFO:tasks.rados.rados.0.smithi089.stdout:1398: left oid 16 (ObjNum 272 snap 0 seq_num 272) 2024-04-22T18:40:21.583 INFO:tasks.rados.rados.0.smithi089.stdout:1396: expect (ObjNum 261 snap 0 seq_num 261) 2024-04-22T18:40:21.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:21.584 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:21.583+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:21.683 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:21.684+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:21.694 INFO:tasks.rados.rados.0.smithi089.stdout:1399: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:40:21.725 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:21.726 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:21.726 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:21.726 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:21.735 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:21.736 INFO:teuthology.orchestra.run.smithi089.stderr:set 4.11 pg_upmap_items mapping to [2->7,3->1] 2024-04-22T18:40:21.746 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:40:21.790 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:21.791+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:21.880 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:21.880 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:21.880 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:21.880 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:21.880 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:21.885 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:21.887+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:21.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:21.904 INFO:tasks.rados.rados.0.smithi089.stdout:1401: finishing set_chunk to oid 32 2024-04-22T18:40:21.904 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 32 set_chunk returned error code -95 offset: 901632 length: 2593792 2024-04-22T18:40:21.904 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 0 (ObjNum 209 snap 0 seq_num 209) dirty exists 2024-04-22T18:40:21.904 INFO:tasks.rados.rados.0.smithi089.stdout:1402: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:40:21.986 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:21.988+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:21.991 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:21.991 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:21.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:21.991 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:21.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.991 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:22 4.a 4:5aa97f94:::smithi08937929-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:40:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.834214680000002, 2024-04-22T18:40:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30948398900000001, 2024-04-22T18:40:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 22 2024-04-22T18:40:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:40:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157260+0000", 2024-04-22T18:40:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:21.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157259+0000", 2024-04-22T18:40:21.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:21.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:21.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157318+0000", 2024-04-22T18:40:21.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8906000000000001e-05 2024-04-22T18:40:21.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:21.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157327+0000", 2024-04-22T18:40:21.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:40:21.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:21.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157332+0000", 2024-04-22T18:40:21.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4689999999999996e-06 2024-04-22T18:40:21.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257479+0000", 2024-04-22T18:40:21.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014700999999999 2024-04-22T18:40:21.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:21.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257492+0000", 2024-04-22T18:40:21.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2574e-05 2024-04-22T18:40:21.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:21.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259582+0000", 2024-04-22T18:40:21.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0020904019999999999 2024-04-22T18:40:21.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:21.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259668+0000", 2024-04-22T18:40:21.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.5826000000000001e-05 2024-04-22T18:40:21.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:21.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259772+0000", 2024-04-22T18:40:21.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000104048 2024-04-22T18:40:21.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:21.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363453+0000", 2024-04-22T18:40:21.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103680183 2024-04-22T18:40:21.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:21.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466692+0000", 2024-04-22T18:40:21.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10323948199999999 2024-04-22T18:40:21.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:21.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466720+0000", 2024-04-22T18:40:21.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7886e-05 2024-04-22T18:40:21.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:21.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466744+0000", 2024-04-22T18:40:21.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4049e-05 2024-04-22T18:40:21.998 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.999 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:21.999 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:21.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:21.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:21.999 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:23 4.a 4:5aa97f94:::smithi08937929-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:21.999 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:40:21.999 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.834136993000001, 2024-04-22T18:40:21.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39003820300000003, 2024-04-22T18:40:21.999 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:21.999 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:21.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:21.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:21.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:21.999 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 23 2024-04-22T18:40:21.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.000 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:40:22.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157338+0000", 2024-04-22T18:40:22.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157337+0000", 2024-04-22T18:40:22.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157340+0000", 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8609999999999998e-06 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157343+0000", 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9859999999999999e-06 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157346+0000", 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8119999999999999e-06 2024-04-22T18:40:22.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257563+0000", 2024-04-22T18:40:22.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100217497 2024-04-22T18:40:22.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:22.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.257574+0000", 2024-04-22T18:40:22.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0777e-05 2024-04-22T18:40:22.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260112+0000", 2024-04-22T18:40:22.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002537681 2024-04-22T18:40:22.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260191+0000", 2024-04-22T18:40:22.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9320999999999995e-05 2024-04-22T18:40:22.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:22.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260333+0000", 2024-04-22T18:40:22.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014177699999999999 2024-04-22T18:40:22.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363792+0000", 2024-04-22T18:40:22.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103458862 2024-04-22T18:40:22.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547303+0000", 2024-04-22T18:40:22.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18351162100000001 2024-04-22T18:40:22.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547337+0000", 2024-04-22T18:40:22.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3640999999999999e-05 2024-04-22T18:40:22.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547376+0000", 2024-04-22T18:40:22.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8915000000000003e-05 2024-04-22T18:40:22.005 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.005 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.005 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.005 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:24 4.a 4:5aa97f94:::smithi08937929-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:22.005 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:22.005 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.834110562000006, 2024-04-22T18:40:22.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39017823499999998, 2024-04-22T18:40:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 24 2024-04-22T18:40:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157364+0000", 2024-04-22T18:40:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157366+0000", 2024-04-22T18:40:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3120000000000001e-06 2024-04-22T18:40:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157370+0000", 2024-04-22T18:40:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1300000000000003e-06 2024-04-22T18:40:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.157373+0000", 2024-04-22T18:40:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9960000000000002e-06 2024-04-22T18:40:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260553+0000", 2024-04-22T18:40:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103179996 2024-04-22T18:40:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260625+0000", 2024-04-22T18:40:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2236000000000004e-05 2024-04-22T18:40:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547414+0000", 2024-04-22T18:40:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.286789181 2024-04-22T18:40:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547471+0000", 2024-04-22T18:40:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6192e-05 2024-04-22T18:40:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547542+0000", 2024-04-22T18:40:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1622e-05 2024-04-22T18:40:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:49 4.a 4:5d2aee44:::smithi08937929-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:40:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.757311904000005, 2024-04-22T18:40:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31344442500000003, 2024-04-22T18:40:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 49 2024-04-22T18:40:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:40:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234163+0000", 2024-04-22T18:40:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234158+0000", 2024-04-22T18:40:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999957 2024-04-22T18:40:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234901+0000", 2024-04-22T18:40:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074250800000000001 2024-04-22T18:40:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234914+0000", 2024-04-22T18:40:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3244999999999999e-05 2024-04-22T18:40:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.234922+0000", 2024-04-22T18:40:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4590000000000001e-06 2024-04-22T18:40:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260665+0000", 2024-04-22T18:40:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025743423000000001 2024-04-22T18:40:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260779+0000", 2024-04-22T18:40:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114063 2024-04-22T18:40:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260913+0000", 2024-04-22T18:40:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013355499999999999 2024-04-22T18:40:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363798+0000", 2024-04-22T18:40:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102885003 2024-04-22T18:40:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547567+0000", 2024-04-22T18:40:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.183768971 2024-04-22T18:40:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547589+0000", 2024-04-22T18:40:22.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2507999999999999e-05 2024-04-22T18:40:22.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.547607+0000", 2024-04-22T18:40:22.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8009999999999999e-05 2024-04-22T18:40:22.016 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.016 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.016 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.017 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:50 4.a 4:5d2aee44:::smithi08937929-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:22.017 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:22.017 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.751865725000002, 2024-04-22T18:40:22.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40856295399999998, 2024-04-22T18:40:22.017 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.017 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.017 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 50 2024-04-22T18:40:22.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.017 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:22.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:22.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239607+0000", 2024-04-22T18:40:22.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:22.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240309+0000", 2024-04-22T18:40:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070201599999999997 2024-04-22T18:40:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240320+0000", 2024-04-22T18:40:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1024e-05 2024-04-22T18:40:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240327+0000", 2024-04-22T18:40:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0049999999999998e-06 2024-04-22T18:40:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362065+0000", 2024-04-22T18:40:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.121737812 2024-04-22T18:40:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362121+0000", 2024-04-22T18:40:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5868000000000001e-05 2024-04-22T18:40:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362210+0000", 2024-04-22T18:40:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8533000000000001e-05 2024-04-22T18:40:22.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365322+0000", 2024-04-22T18:40:22.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031121220000000002 2024-04-22T18:40:22.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648054+0000", 2024-04-22T18:40:22.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.282732549 2024-04-22T18:40:22.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648105+0000", 2024-04-22T18:40:22.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1276999999999997e-05 2024-04-22T18:40:22.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648172+0000", 2024-04-22T18:40:22.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6398999999999999e-05 2024-04-22T18:40:22.022 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.022 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.022 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.022 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:51 4.a 4:5d2aee44:::smithi08937929-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:22.023 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:22.023 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.751143190999997, 2024-04-22T18:40:22.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40793057199999999, 2024-04-22T18:40:22.023 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.023 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.023 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 51 2024-04-22T18:40:22.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.023 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:22.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:22.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:22.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240336+0000", 2024-04-22T18:40:22.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:40:22.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240339+0000", 2024-04-22T18:40:22.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9009999999999998e-06 2024-04-22T18:40:22.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240342+0000", 2024-04-22T18:40:22.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:40:22.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363080+0000", 2024-04-22T18:40:22.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122738075 2024-04-22T18:40:22.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363164+0000", 2024-04-22T18:40:22.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3722999999999998e-05 2024-04-22T18:40:22.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:22.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363287+0000", 2024-04-22T18:40:22.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012301099999999999 2024-04-22T18:40:22.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466753+0000", 2024-04-22T18:40:22.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103466217 2024-04-22T18:40:22.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648205+0000", 2024-04-22T18:40:22.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.181451788 2024-04-22T18:40:22.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648231+0000", 2024-04-22T18:40:22.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6597999999999998e-05 2024-04-22T18:40:22.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648262+0000", 2024-04-22T18:40:22.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0871000000000003e-05 2024-04-22T18:40:22.028 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.028 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.028 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:52 4.a 4:5d2aee44:::smithi08937929-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.751130732000007, 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40812731200000002, 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 52 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:22.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240345+0000", 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4300000000000001e-06 2024-04-22T18:40:22.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240349+0000", 2024-04-22T18:40:22.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2570000000000001e-06 2024-04-22T18:40:22.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240358+0000", 2024-04-22T18:40:22.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-22T18:40:22.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363697+0000", 2024-04-22T18:40:22.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12333965299999999 2024-04-22T18:40:22.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:22.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363777+0000", 2024-04-22T18:40:22.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9883999999999996e-05 2024-04-22T18:40:22.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648296+0000", 2024-04-22T18:40:22.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451861299999998 2024-04-22T18:40:22.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648379+0000", 2024-04-22T18:40:22.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3302999999999998e-05 2024-04-22T18:40:22.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648471+0000", 2024-04-22T18:40:22.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2029999999999998e-05 2024-04-22T18:40:22.033 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.033 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.033 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.033 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:22.033 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:22.033 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.731827397000004, 2024-04-22T18:40:22.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:40:22.034 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.034 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.034 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.034 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.034 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.034 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:40:22.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.034 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:22.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:22.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:40:22.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:22.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:40:22.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:40:22.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:40:22.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:40:22.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:40:22.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:40:22.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:40:22.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:40:22.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:40:22.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:40:22.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:22.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:40:22.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:40:22.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:40:22.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:40:22.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:40:22.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:40:22.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:40:22.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:40:22.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:40:22.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:40:22.039 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.039 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.039 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.039 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:22.039 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:22.040 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.727044966999998, 2024-04-22T18:40:22.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:40:22.040 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.040 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.040 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.040 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.040 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.040 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:40:22.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.040 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:22.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:22.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:40:22.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:22.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:40:22.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:40:22.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:40:22.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:40:22.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:40:22.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:40:22.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:40:22.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:40:22.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:40:22.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:40:22.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:22.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:40:22.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:40:22.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:40:22.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:40:22.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:40:22.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:40:22.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:40:22.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:40:22.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:40:22.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:40:22.045 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.045 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.045 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.045 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:22.045 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:22.045 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.726950763999994, 2024-04-22T18:40:22.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:40:22.045 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.046 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.046 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.046 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.046 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.046 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:40:22.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.046 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:22.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:22.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:40:22.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:22.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:40:22.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:40:22.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:40:22.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:40:22.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:40:22.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:40:22.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:40:22.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:40:22.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:40:22.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:40:22.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:22.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:40:22.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:40:22.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:40:22.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:40:22.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:40:22.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:40:22.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:40:22.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:40:22.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:40:22.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:40:22.051 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.051 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.051 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.051 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:22.051 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:22.051 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.726929876, 2024-04-22T18:40:22.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:40:22.051 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.051 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:22.052 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.052 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.052 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.052 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:40:22.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.052 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:22.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:22.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:40:22.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:22.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:40:22.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:40:22.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:40:22.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:40:22.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:40:22.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:40:22.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:40:22.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:40:22.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:22.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:40:22.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:40:22.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:40:22.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:40:22.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:40:22.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:40:22.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:40:22.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:40:22.056 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.056 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.056 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.056 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:22.056 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:22.056 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 86.943922212999993, 2024-04-22T18:40:22.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:40:22.056 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.056 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:40:22.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:22.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:40:22.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:40:22.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:40:22.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:40:22.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:40:22.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:40:22.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:40:22.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:40:22.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:40:22.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:40:22.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:22.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:40:22.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:40:22.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:40:22.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:40:22.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:40:22.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:40:22.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:40:22.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:40:22.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:40:22.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:40:22.061 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.061 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.061 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 86.936090806999999, 2024-04-22T18:40:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:40:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:40:22.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.063 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:22.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:22.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:40:22.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:22.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:40:22.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:40:22.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:40:22.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:40:22.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:40:22.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:40:22.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:40:22.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:40:22.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:40:22.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:40:22.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:22.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:40:22.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:40:22.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:40:22.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:40:22.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:40:22.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:40:22.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:40:22.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:40:22.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:40:22.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:40:22.067 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.067 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.067 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.068 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2052 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:22.068 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:22.068 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.413979026, 2024-04-22T18:40:22.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43058958600000002, 2024-04-22T18:40:22.068 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.068 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.068 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.068 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.068 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.068 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2052 2024-04-22T18:40:22.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.068 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:22.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:22.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577494+0000", 2024-04-22T18:40:22.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:22.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577500+0000", 2024-04-22T18:40:22.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3279999999999996e-06 2024-04-22T18:40:22.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577510+0000", 2024-04-22T18:40:22.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9450000000000005e-06 2024-04-22T18:40:22.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577515+0000", 2024-04-22T18:40:22.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7799999999999997e-06 2024-04-22T18:40:22.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677622+0000", 2024-04-22T18:40:22.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010654099999999 2024-04-22T18:40:22.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677710+0000", 2024-04-22T18:40:22.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8472000000000003e-05 2024-04-22T18:40:22.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:22.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.906619+0000", 2024-04-22T18:40:22.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.228908519 2024-04-22T18:40:22.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907343+0000", 2024-04-22T18:40:22.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072442600000000002 2024-04-22T18:40:22.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.007972+0000", 2024-04-22T18:40:22.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100628781 2024-04-22T18:40:22.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008022+0000", 2024-04-22T18:40:22.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9843999999999999e-05 2024-04-22T18:40:22.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008085+0000", 2024-04-22T18:40:22.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3256999999999994e-05 2024-04-22T18:40:22.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.073 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.074 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2087 4.a 4:537c89f8:::smithi08937929-47:head [assert-version v37,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-04-22T18:40:22.074 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:40:22.074 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.084062265999997, 2024-04-22T18:40:22.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38904324299999998, 2024-04-22T18:40:22.074 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.074 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.074 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.074 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.074 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.074 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2087 2024-04-22T18:40:22.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.074 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:40:22.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907412+0000", 2024-04-22T18:40:22.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907410+0000", 2024-04-22T18:40:22.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:22.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907417+0000", 2024-04-22T18:40:22.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.306e-06 2024-04-22T18:40:22.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907427+0000", 2024-04-22T18:40:22.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4930000000000004e-06 2024-04-22T18:40:22.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907432+0000", 2024-04-22T18:40:22.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5400000000000003e-06 2024-04-22T18:40:22.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907477+0000", 2024-04-22T18:40:22.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4851999999999997e-05 2024-04-22T18:40:22.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907537+0000", 2024-04-22T18:40:22.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9460999999999999e-05 2024-04-22T18:40:22.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:22.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.295587+0000", 2024-04-22T18:40:22.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38805073200000001 2024-04-22T18:40:22.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296281+0000", 2024-04-22T18:40:22.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00069370800000000002 2024-04-22T18:40:22.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296400+0000", 2024-04-22T18:40:22.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118579 2024-04-22T18:40:22.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296431+0000", 2024-04-22T18:40:22.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1627000000000001e-05 2024-04-22T18:40:22.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.296456+0000", 2024-04-22T18:40:22.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4331000000000001e-05 2024-04-22T18:40:22.080 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.080 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.080 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.080 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:22.080 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:22.081 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.784218177, 2024-04-22T18:40:22.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:40:22.081 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.081 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.081 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.081 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.081 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.081 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:40:22.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.081 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:22.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:22.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:40:22.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:22.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:40:22.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:40:22.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:40:22.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:40:22.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:40:22.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:40:22.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:40:22.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:40:22.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:40:22.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:40:22.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:22.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:40:22.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:40:22.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:40:22.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:40:22.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:40:22.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:40:22.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:40:22.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:40:22.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:40:22.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:40:22.086 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.086 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.086 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.086 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2505 4.5 4:a6d237d2:::smithi08937929-43:head [setxattr _header (56) in=7b,truncate 3529470] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:22.086 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:22.087 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.783825482000001, 2024-04-22T18:40:22.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43706744400000003, 2024-04-22T18:40:22.087 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.087 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.087 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.087 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.087 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.087 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2505 2024-04-22T18:40:22.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.087 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:22.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:22.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207648+0000", 2024-04-22T18:40:22.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:22.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207653+0000", 2024-04-22T18:40:22.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2989999999999998e-06 2024-04-22T18:40:22.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207656+0000", 2024-04-22T18:40:22.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0199999999999999e-06 2024-04-22T18:40:22.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207659+0000", 2024-04-22T18:40:22.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2270000000000002e-06 2024-04-22T18:40:22.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243213+0000", 2024-04-22T18:40:22.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035553653999999997 2024-04-22T18:40:22.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243273+0000", 2024-04-22T18:40:22.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0300000000000002e-05 2024-04-22T18:40:22.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:22.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243359+0000", 2024-04-22T18:40:22.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6275999999999999e-05 2024-04-22T18:40:22.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297443+0000", 2024-04-22T18:40:22.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.054083930000000002 2024-04-22T18:40:22.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644651+0000", 2024-04-22T18:40:22.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.347208181 2024-04-22T18:40:22.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644684+0000", 2024-04-22T18:40:22.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3080999999999997e-05 2024-04-22T18:40:22.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644717+0000", 2024-04-22T18:40:22.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2264000000000001e-05 2024-04-22T18:40:22.092 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.092 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.092 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.092 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:22.093 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:22.093 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 26.783813268999999, 2024-04-22T18:40:22.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:40:22.093 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.093 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:22.093 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.093 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.093 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.093 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:40:22.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.094 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:22.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:22.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:22.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:40:22.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:40:22.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:40:22.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:40:22.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:40:22.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:40:22.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:40:22.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:40:22.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:22.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:40:22.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:40:22.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:40:22.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:40:22.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:40:22.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:40:22.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:40:22.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:40:22.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.098 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.098 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2615 4.5 4:a6d237d2:::smithi08937929-43:head [write 463926~462166] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-22T18:40:22.098 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:58.352901+0000", 2024-04-22T18:40:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 23.638573821000001, 2024-04-22T18:40:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.307052514, 2024-04-22T18:40:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2615 2024-04-22T18:40:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.352901+0000", 2024-04-22T18:40:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.352901+0000", 2024-04-22T18:40:22.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.352897+0000", 2024-04-22T18:40:22.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999962 2024-04-22T18:40:22.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.353421+0000", 2024-04-22T18:40:22.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052391999999999998 2024-04-22T18:40:22.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.353433+0000", 2024-04-22T18:40:22.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2128e-05 2024-04-22T18:40:22.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.353460+0000", 2024-04-22T18:40:22.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7211000000000001e-05 2024-04-22T18:40:22.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.453670+0000", 2024-04-22T18:40:22.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10020929100000001 2024-04-22T18:40:22.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.453809+0000", 2024-04-22T18:40:22.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013967800000000001 2024-04-22T18:40:22.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:22.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.454021+0000", 2024-04-22T18:40:22.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00021190900000000001 2024-04-22T18:40:22.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659805+0000", 2024-04-22T18:40:22.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20578357999999999 2024-04-22T18:40:22.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659886+0000", 2024-04-22T18:40:22.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1310000000000006e-05 2024-04-22T18:40:22.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659925+0000", 2024-04-22T18:40:22.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9233000000000002e-05 2024-04-22T18:40:22.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:58.659953+0000", 2024-04-22T18:40:22.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7894000000000001e-05 2024-04-22T18:40:22.104 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.104 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.104 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.104 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2715 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:22.104 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:03.692698+0000", 2024-04-22T18:40:22.104 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.298776926999999, 2024-04-22T18:40:22.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40343488, 2024-04-22T18:40:22.104 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.104 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:22.105 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.105 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.105 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.105 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2715 2024-04-22T18:40:22.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.105 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.692698+0000", 2024-04-22T18:40:22.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.692698+0000", 2024-04-22T18:40:22.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.692697+0000", 2024-04-22T18:40:22.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.692700+0000", 2024-04-22T18:40:22.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3410000000000001e-06 2024-04-22T18:40:22.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.692703+0000", 2024-04-22T18:40:22.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7230000000000002e-06 2024-04-22T18:40:22.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.792867+0000", 2024-04-22T18:40:22.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10016380499999999 2024-04-22T18:40:22.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.793431+0000", 2024-04-22T18:40:22.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056344299999999995 2024-04-22T18:40:22.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:22.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:03.793496+0000", 2024-04-22T18:40:22.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5415999999999998e-05 2024-04-22T18:40:22.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:04.095990+0000", 2024-04-22T18:40:22.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30249418099999997 2024-04-22T18:40:22.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:04.096045+0000", 2024-04-22T18:40:22.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4486e-05 2024-04-22T18:40:22.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:04.096133+0000", 2024-04-22T18:40:22.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7956999999999998e-05 2024-04-22T18:40:22.109 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.109 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.109 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.109 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.109 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.109 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:22.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:22.113 INFO:tasks.rados.rados.0.smithi089.stdout:1403: got (0) Success 2024-04-22T18:40:22.113 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 0 (ObjNum 226 snap 0 seq_num 226) dirty exists 2024-04-22T18:40:22.113 INFO:tasks.rados.rados.0.smithi089.stdout:1393: done (10 left) 2024-04-22T18:40:22.113 INFO:tasks.rados.rados.0.smithi089.stdout:1394: done (9 left) 2024-04-22T18:40:22.113 INFO:tasks.rados.rados.0.smithi089.stdout:1395: done (8 left) 2024-04-22T18:40:22.113 INFO:tasks.rados.rados.0.smithi089.stdout:1396: done (7 left) 2024-04-22T18:40:22.113 INFO:tasks.rados.rados.0.smithi089.stdout:1398: done (6 left) 2024-04-22T18:40:22.113 INFO:tasks.rados.rados.0.smithi089.stdout:1399: done (5 left) 2024-04-22T18:40:22.113 INFO:tasks.rados.rados.0.smithi089.stdout:1400: done (4 left) 2024-04-22T18:40:22.113 INFO:tasks.rados.rados.0.smithi089.stdout:1401: done (3 left) 2024-04-22T18:40:22.113 INFO:tasks.rados.rados.0.smithi089.stdout:1402: done (2 left) 2024-04-22T18:40:22.113 INFO:tasks.rados.rados.0.smithi089.stdout:1403: done (1 left) 2024-04-22T18:40:22.113 INFO:tasks.rados.rados.0.smithi089.stdout:1404: tier_evict oid 21 2024-04-22T18:40:22.113 INFO:tasks.rados.rados.0.smithi089.stdout:1404: tier_evict oid 21 snap -1 2024-04-22T18:40:22.114 INFO:tasks.rados.rados.0.smithi089.stdout:1405: read oid 1 snap -1 2024-04-22T18:40:22.114 INFO:tasks.rados.rados.0.smithi089.stdout:1405: expect deleted 2024-04-22T18:40:22.114 INFO:tasks.rados.rados.0.smithi089.stdout:1406: read oid 40 snap -1 2024-04-22T18:40:22.114 INFO:tasks.rados.rados.0.smithi089.stdout:1406: expect (ObjNum 227 snap 0 seq_num 227) 2024-04-22T18:40:22.114 INFO:tasks.rados.rados.0.smithi089.stdout:1407: read oid 24 snap -1 2024-04-22T18:40:22.114 INFO:tasks.rados.rados.0.smithi089.stdout:1407: expect (ObjNum 248 snap 0 seq_num 248) 2024-04-22T18:40:22.114 INFO:tasks.rados.rados.0.smithi089.stdout:1408: write (excl) oid 19 current snap is 0 2024-04-22T18:40:22.114 INFO:tasks.rados.rados.0.smithi089.stdout:1408: seq_num 273 ranges {588311=615332,1842290=562759} 2024-04-22T18:40:22.114 INFO:tasks.rados.rados.0.smithi089.stdout:1408: writing smithi08937929-19 from 588311 to 1203643 tid 1 2024-04-22T18:40:22.114 INFO:tasks.rados.rados.0.smithi089.stdout:1408: writing smithi08937929-19 from 1842290 to 2405049 tid 2 2024-04-22T18:40:22.114 INFO:tasks.rados.rados.0.smithi089.stdout:1409: read oid 16 snap -1 2024-04-22T18:40:22.115 INFO:tasks.rados.rados.0.smithi089.stdout:1409: expect (ObjNum 272 snap 0 seq_num 272) 2024-04-22T18:40:22.115 INFO:tasks.rados.rados.0.smithi089.stdout:1410: write (excl) oid 42 current snap is 0 2024-04-22T18:40:22.115 INFO:tasks.rados.rados.0.smithi089.stdout:1410: seq_num 274 ranges {605118=406934,1617125=706417} 2024-04-22T18:40:22.115 INFO:tasks.rados.rados.0.smithi089.stdout:1410: writing smithi08937929-42 from 605118 to 1012052 tid 1 2024-04-22T18:40:22.115 INFO:tasks.rados.rados.0.smithi089.stdout:1410: writing smithi08937929-42 from 1617125 to 2323542 tid 2 2024-04-22T18:40:22.115 INFO:tasks.rados.rados.0.smithi089.stdout:1411: set_chunk oid 4 target oid 2024-04-22T18:40:22.115 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:22.112+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:22.211 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:22.212+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:22.218 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3163 4.8 4.835b5e18 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:40:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.38188207299999999, 2024-04-22T18:40:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38190201000000001, 2024-04-22T18:40:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:40:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3163 2024-04-22T18:40:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:40:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:40:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837455+0000", 2024-04-22T18:40:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837464+0000", 2024-04-22T18:40:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5319999999999995e-06 2024-04-22T18:40:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837476+0000", 2024-04-22T18:40:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1759e-05 2024-04-22T18:40:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837488+0000", 2024-04-22T18:40:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1666e-05 2024-04-22T18:40:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837541+0000", 2024-04-22T18:40:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2939999999999998e-05 2024-04-22T18:40:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837547+0000", 2024-04-22T18:40:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6409999999999996e-06 2024-04-22T18:40:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3164 4.8 4.835b5e18 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.38184270399999998, 2024-04-22T18:40:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38190489999999999, 2024-04-22T18:40:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:40:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3164 2024-04-22T18:40:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837499+0000", 2024-04-22T18:40:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.994e-06 2024-04-22T18:40:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837503+0000", 2024-04-22T18:40:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7249999999999999e-06 2024-04-22T18:40:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837506+0000", 2024-04-22T18:40:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5980000000000001e-06 2024-04-22T18:40:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937641+0000", 2024-04-22T18:40:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10013467700000001 2024-04-22T18:40:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937650+0000", 2024-04-22T18:40:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3489999999999998e-06 2024-04-22T18:40:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3165 4.8 4.835b5e18 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.38182393599999997, 2024-04-22T18:40:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38192069200000001, 2024-04-22T18:40:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:40:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3165 2024-04-22T18:40:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837517+0000", 2024-04-22T18:40:22.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3369999999999998e-06 2024-04-22T18:40:22.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837523+0000", 2024-04-22T18:40:22.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5450000000000003e-06 2024-04-22T18:40:22.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837526+0000", 2024-04-22T18:40:22.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.354e-06 2024-04-22T18:40:22.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937682+0000", 2024-04-22T18:40:22.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100155917 2024-04-22T18:40:22.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:22.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937688+0000", 2024-04-22T18:40:22.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:40:22.233 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.233 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.233 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.233 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3183 4.8 4.516623d8 (undecoded) ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:22.233 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:40:22.233 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.183420738, 2024-04-22T18:40:22.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.183551137, 2024-04-22T18:40:22.233 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.234 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:40:22.234 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.234 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.234 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.234 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3183 2024-04-22T18:40:22.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.234 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:40:22.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:40:22.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035913+0000", 2024-04-22T18:40:22.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999943 2024-04-22T18:40:22.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035935+0000", 2024-04-22T18:40:22.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2232999999999999e-05 2024-04-22T18:40:22.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035969+0000", 2024-04-22T18:40:22.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3969e-05 2024-04-22T18:40:22.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035987+0000", 2024-04-22T18:40:22.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8397e-05 2024-04-22T18:40:22.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:22.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2916000000000003e-05 2024-04-22T18:40:22.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:22.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:40:22.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0103e-05 2024-04-22T18:40:22.237 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.237 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.238 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.238 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3184 4.8 4.641f9b48 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:22.238 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:40:22.238 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.183332779, 2024-04-22T18:40:22.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.183497033, 2024-04-22T18:40:22.238 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.238 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:40:22.239 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.239 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.239 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.239 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3184 2024-04-22T18:40:22.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.239 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:40:22.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:40:22.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036005+0000", 2024-04-22T18:40:22.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:22.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036026+0000", 2024-04-22T18:40:22.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1092999999999999e-05 2024-04-22T18:40:22.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036030+0000", 2024-04-22T18:40:22.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.467e-06 2024-04-22T18:40:22.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036033+0000", 2024-04-22T18:40:22.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9780000000000001e-06 2024-04-22T18:40:22.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036120+0000", 2024-04-22T18:40:22.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7495000000000002e-05 2024-04-22T18:40:22.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:22.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036125+0000", 2024-04-22T18:40:22.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8720000000000001e-06 2024-04-22T18:40:22.243 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.243 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.243 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.243 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3185 4.8 4.641f9b48 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:22.243 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:22.243 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.18327886700000001, 2024-04-22T18:40:22.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18348683900000001, 2024-04-22T18:40:22.243 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.244 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:40:22.244 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.244 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.244 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.244 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3185 2024-04-22T18:40:22.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.244 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:22.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:22.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:22.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036062+0000", 2024-04-22T18:40:22.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9099999999999999e-06 2024-04-22T18:40:22.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036066+0000", 2024-04-22T18:40:22.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1699999999999999e-06 2024-04-22T18:40:22.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036077+0000", 2024-04-22T18:40:22.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0637e-05 2024-04-22T18:40:22.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036147+0000", 2024-04-22T18:40:22.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0025999999999997e-05 2024-04-22T18:40:22.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:22.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036151+0000", 2024-04-22T18:40:22.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:40:22.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.248 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.248 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3186 4.8 4.641f9b48 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:22.248 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:22.248 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.18325931100000001, 2024-04-22T18:40:22.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.18350213800000001, 2024-04-22T18:40:22.248 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.248 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:40:22.249 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.249 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.249 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.249 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3186 2024-04-22T18:40:22.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.249 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:22.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:22.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036079+0000", 2024-04-22T18:40:22.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036081+0000", 2024-04-22T18:40:22.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6449999999999999e-06 2024-04-22T18:40:22.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036084+0000", 2024-04-22T18:40:22.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9399999999999998e-06 2024-04-22T18:40:22.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:40:22.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.879e-06 2024-04-22T18:40:22.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036168+0000", 2024-04-22T18:40:22.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7657999999999994e-05 2024-04-22T18:40:22.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:22.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036171+0000", 2024-04-22T18:40:22.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7639999999999999e-06 2024-04-22T18:40:22.252 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.252 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.253 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.253 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.253 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:40:22.253 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 7 2024-04-22T18:40:22.253 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:22.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:22.338 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:22.339+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:22.396 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:22.397 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:22.397 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:22.397 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:22.397 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:22.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:22.446 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:22.447+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:22.514 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:22.515+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:22.547 INFO:tasks.rados.rados.0.smithi089.stdout:1411: set_chunk oid 4 offset: 992256 length: 802304 target oid 3e4e73f5d4a0122957865f0faeac2483204ee98d7d497ef3b54afe2a02b39a3d offset: 0 snap -1 2024-04-22T18:40:22.547 INFO:tasks.rados.rados.0.smithi089.stdout:1412: write (excl) oid 18 current snap is 0 2024-04-22T18:40:22.547 INFO:tasks.rados.rados.0.smithi089.stdout:1412: seq_num 275 ranges {621925=482183,1559254=566428} 2024-04-22T18:40:22.549 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:22.549 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:22.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:22.549 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:22.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.549 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:22.549 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:22.550 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 102.80706882200001, 2024-04-22T18:40:22.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:40:22.550 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.550 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.550 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.550 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.550 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.550 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:40:22.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.550 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:22.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:22.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:40:22.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:22.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:40:22.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:40:22.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:40:22.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:40:22.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:40:22.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:40:22.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:40:22.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:40:22.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:40:22.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:40:22.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:40:22.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:40:22.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:40:22.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:40:22.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:40:22.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:40:22.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:40:22.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:40:22.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:40:22.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:40:22.555 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.555 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.555 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.556 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:22.556 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:22.556 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 102.80702913099999, 2024-04-22T18:40:22.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:40:22.556 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.556 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.556 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.556 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.556 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.556 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:40:22.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.556 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:22.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:22.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:40:22.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:22.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:40:22.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:40:22.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:40:22.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:40:22.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:40:22.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:40:22.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:40:22.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:40:22.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:40:22.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:40:22.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:40:22.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:40:22.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:40:22.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:40:22.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:40:22.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:40:22.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:40:22.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:40:22.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:40:22.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:40:22.561 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.561 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.561 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.562 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:22.562 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:40:22.562 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 100.562848084, 2024-04-22T18:40:22.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402419109, 2024-04-22T18:40:22.562 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.562 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.562 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.562 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.562 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.562 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 68 2024-04-22T18:40:22.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.562 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:40:22.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:40:22.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986204+0000", 2024-04-22T18:40:22.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986208+0000", 2024-04-22T18:40:22.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8149999999999999e-06 2024-04-22T18:40:22.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986214+0000", 2024-04-22T18:40:22.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.604e-06 2024-04-22T18:40:22.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986218+0000", 2024-04-22T18:40:22.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6689999999999999e-06 2024-04-22T18:40:22.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986833+0000", 2024-04-22T18:40:22.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000615648 2024-04-22T18:40:22.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986891+0000", 2024-04-22T18:40:22.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7915999999999999e-05 2024-04-22T18:40:22.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986981+0000", 2024-04-22T18:40:22.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759999999999994e-05 2024-04-22T18:40:22.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.987584+0000", 2024-04-22T18:40:22.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060270199999999999 2024-04-22T18:40:22.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388527+0000", 2024-04-22T18:40:22.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40094366999999997 2024-04-22T18:40:22.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388583+0000", 2024-04-22T18:40:22.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5383000000000002e-05 2024-04-22T18:40:22.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388624+0000", 2024-04-22T18:40:22.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501999999999997e-05 2024-04-22T18:40:22.567 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.567 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.568 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.568 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:22.568 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:22.568 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 100.56283229500001, 2024-04-22T18:40:22.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:40:22.568 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.568 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.568 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.568 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.568 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.568 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:40:22.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.568 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:22.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:22.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:40:22.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:40:22.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:40:22.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:40:22.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:40:22.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:40:22.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:40:22.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:40:22.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:40:22.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:40:22.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:40:22.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:40:22.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:40:22.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:40:22.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:40:22.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:40:22.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:40:22.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:40:22.573 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 99.446439286, 2024-04-22T18:40:22.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305563733, 2024-04-22T18:40:22.574 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.574 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.574 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.574 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.574 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.574 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 87 2024-04-22T18:40:22.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.574 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:40:22.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102614+0000", 2024-04-22T18:40:22.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102613+0000", 2024-04-22T18:40:22.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:22.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102668+0000", 2024-04-22T18:40:22.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4669000000000001e-05 2024-04-22T18:40:22.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102680+0000", 2024-04-22T18:40:22.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2026e-05 2024-04-22T18:40:22.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102687+0000", 2024-04-22T18:40:22.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8339999999999999e-06 2024-04-22T18:40:22.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405644+0000", 2024-04-22T18:40:22.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30295623199999999 2024-04-22T18:40:22.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405743+0000", 2024-04-22T18:40:22.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.98e-05 2024-04-22T18:40:22.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.405868+0000", 2024-04-22T18:40:22.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012457899999999999 2024-04-22T18:40:22.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407047+0000", 2024-04-22T18:40:22.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0011791449999999999 2024-04-22T18:40:22.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408088+0000", 2024-04-22T18:40:22.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010411680000000001 2024-04-22T18:40:22.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408135+0000", 2024-04-22T18:40:22.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6255999999999999e-05 2024-04-22T18:40:22.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408178+0000", 2024-04-22T18:40:22.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2877000000000003e-05 2024-04-22T18:40:22.579 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.579 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.579 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.579 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:22.579 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:40:22.579 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 99.446359357000006, 2024-04-22T18:40:22.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30560089000000001, 2024-04-22T18:40:22.580 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.580 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.580 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.580 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.580 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.580 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 88 2024-04-22T18:40:22.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.580 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:40:22.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102694+0000", 2024-04-22T18:40:22.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102693+0000", 2024-04-22T18:40:22.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:22.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102698+0000", 2024-04-22T18:40:22.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9699999999999998e-06 2024-04-22T18:40:22.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102708+0000", 2024-04-22T18:40:22.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.006e-05 2024-04-22T18:40:22.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.102715+0000", 2024-04-22T18:40:22.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6599999999999995e-06 2024-04-22T18:40:22.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406637+0000", 2024-04-22T18:40:22.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392130099999998 2024-04-22T18:40:22.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406696+0000", 2024-04-22T18:40:22.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9066e-05 2024-04-22T18:40:22.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.406784+0000", 2024-04-22T18:40:22.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8194000000000003e-05 2024-04-22T18:40:22.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.407447+0000", 2024-04-22T18:40:22.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00066295700000000004 2024-04-22T18:40:22.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408237+0000", 2024-04-22T18:40:22.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00079032600000000005 2024-04-22T18:40:22.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408270+0000", 2024-04-22T18:40:22.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3157000000000001e-05 2024-04-22T18:40:22.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:43.408295+0000", 2024-04-22T18:40:22.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4151999999999998e-05 2024-04-22T18:40:22.585 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.585 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.585 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.585 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:106 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:22.585 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:40:22.585 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.695127232999994, 2024-04-22T18:40:22.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30533819600000001, 2024-04-22T18:40:22.585 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.585 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.585 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.585 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.585 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.586 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 106 2024-04-22T18:40:22.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.586 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:40:22.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853926+0000", 2024-04-22T18:40:22.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853925+0000", 2024-04-22T18:40:22.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853958+0000", 2024-04-22T18:40:22.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2536000000000002e-05 2024-04-22T18:40:22.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853964+0000", 2024-04-22T18:40:22.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6239999999999999e-06 2024-04-22T18:40:22.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853967+0000", 2024-04-22T18:40:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9700000000000001e-06 2024-04-22T18:40:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855144+0000", 2024-04-22T18:40:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001176774 2024-04-22T18:40:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855225+0000", 2024-04-22T18:40:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0697000000000003e-05 2024-04-22T18:40:22.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855353+0000", 2024-04-22T18:40:22.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012788 2024-04-22T18:40:22.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058003+0000", 2024-04-22T18:40:22.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202649939 2024-04-22T18:40:22.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159184+0000", 2024-04-22T18:40:22.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101181058 2024-04-22T18:40:22.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159238+0000", 2024-04-22T18:40:22.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3967e-05 2024-04-22T18:40:22.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159264+0000", 2024-04-22T18:40:22.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6237999999999998e-05 2024-04-22T18:40:22.590 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.590 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.590 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.591 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:22.591 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:40:22.591 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.695080383000004, 2024-04-22T18:40:22.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30544294500000002, 2024-04-22T18:40:22.591 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.591 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.591 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.591 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.591 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.591 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 107 2024-04-22T18:40:22.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.591 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:40:22.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853973+0000", 2024-04-22T18:40:22.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853972+0000", 2024-04-22T18:40:22.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853975+0000", 2024-04-22T18:40:22.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0939999999999999e-06 2024-04-22T18:40:22.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853978+0000", 2024-04-22T18:40:22.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:40:22.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.853981+0000", 2024-04-22T18:40:22.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.858e-06 2024-04-22T18:40:22.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855831+0000", 2024-04-22T18:40:22.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018503689999999999 2024-04-22T18:40:22.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.855932+0000", 2024-04-22T18:40:22.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010071 2024-04-22T18:40:22.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856029+0000", 2024-04-22T18:40:22.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6751999999999998e-05 2024-04-22T18:40:22.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058013+0000", 2024-04-22T18:40:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20198412600000001 2024-04-22T18:40:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159326+0000", 2024-04-22T18:40:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101312969 2024-04-22T18:40:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159376+0000", 2024-04-22T18:40:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9968e-05 2024-04-22T18:40:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159416+0000", 2024-04-22T18:40:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9641999999999998e-05 2024-04-22T18:40:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.694991986999995, 2024-04-22T18:40:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30548257499999998, 2024-04-22T18:40:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 108 2024-04-22T18:40:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854061+0000", 2024-04-22T18:40:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854064+0000", 2024-04-22T18:40:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5820000000000002e-06 2024-04-22T18:40:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854080+0000", 2024-04-22T18:40:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5379999999999998e-05 2024-04-22T18:40:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854084+0000", 2024-04-22T18:40:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.963e-06 2024-04-22T18:40:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856413+0000", 2024-04-22T18:40:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023298070000000001 2024-04-22T18:40:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856494+0000", 2024-04-22T18:40:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0863000000000002e-05 2024-04-22T18:40:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.856624+0000", 2024-04-22T18:40:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000129799 2024-04-22T18:40:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058947+0000", 2024-04-22T18:40:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20232328899999999 2024-04-22T18:40:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159468+0000", 2024-04-22T18:40:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100520597 2024-04-22T18:40:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159508+0000", 2024-04-22T18:40:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9830000000000003e-05 2024-04-22T18:40:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159544+0000", 2024-04-22T18:40:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5944e-05 2024-04-22T18:40:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:40:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.694947494999994, 2024-04-22T18:40:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30556502499999999, 2024-04-22T18:40:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 109 2024-04-22T18:40:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:40:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854106+0000", 2024-04-22T18:40:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854105+0000", 2024-04-22T18:40:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854175+0000", 2024-04-22T18:40:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0178000000000005e-05 2024-04-22T18:40:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854191+0000", 2024-04-22T18:40:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5965999999999999e-05 2024-04-22T18:40:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854196+0000", 2024-04-22T18:40:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.42e-06 2024-04-22T18:40:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956526+0000", 2024-04-22T18:40:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10233094299999999 2024-04-22T18:40:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956634+0000", 2024-04-22T18:40:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000107398 2024-04-22T18:40:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.956730+0000", 2024-04-22T18:40:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6217000000000004e-05 2024-04-22T18:40:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058955+0000", 2024-04-22T18:40:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102224749 2024-04-22T18:40:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159596+0000", 2024-04-22T18:40:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100641297 2024-04-22T18:40:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159637+0000", 2024-04-22T18:40:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874000000000002e-05 2024-04-22T18:40:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159671+0000", 2024-04-22T18:40:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3603999999999999e-05 2024-04-22T18:40:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.694853648000006, 2024-04-22T18:40:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30683048800000001, 2024-04-22T18:40:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 110 2024-04-22T18:40:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854199+0000", 2024-04-22T18:40:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854202+0000", 2024-04-22T18:40:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:40:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854206+0000", 2024-04-22T18:40:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5870000000000001e-06 2024-04-22T18:40:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854208+0000", 2024-04-22T18:40:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7199999999999998e-06 2024-04-22T18:40:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057285+0000", 2024-04-22T18:40:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20307620800000001 2024-04-22T18:40:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057414+0000", 2024-04-22T18:40:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012896899999999999 2024-04-22T18:40:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.057588+0000", 2024-04-22T18:40:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000174802 2024-04-22T18:40:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159429+0000", 2024-04-22T18:40:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101840783 2024-04-22T18:40:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160947+0000", 2024-04-22T18:40:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001518152 2024-04-22T18:40:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160992+0000", 2024-04-22T18:40:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4725000000000003e-05 2024-04-22T18:40:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161030+0000", 2024-04-22T18:40:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7789e-05 2024-04-22T18:40:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.694842467000001, 2024-04-22T18:40:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:40:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:40:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:40:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:40:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:40:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:40:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:40:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:40:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:40:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:40:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:40:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:40:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:40:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:40:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:40:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:40:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:40:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:40:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:40:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:40:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:40:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:40:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:40:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.694826809000006, 2024-04-22T18:40:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:40:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:40:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:40:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:40:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:40:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:40:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:40:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:40:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:40:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.694775647, 2024-04-22T18:40:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:40:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:40:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:40:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:40:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:40:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:40:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:40:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:40:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:40:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:40:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:40:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:40:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:40:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:40:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:40:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:40:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:40:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:40:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:40:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:40:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:40:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:40:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.694760423000005, 2024-04-22T18:40:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:40:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:40:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:40:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:40:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:40:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:40:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:40:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:40:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:40:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:40:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:40:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:40:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:40:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:40:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:40:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:40:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:40:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:40:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:40:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:40:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:40:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:40:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:40:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.694700846000003, 2024-04-22T18:40:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:40:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:40:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:40:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:40:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:40:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:40:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:40:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:40:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:40:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:40:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:40:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:40:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:40:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:40:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:40:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:40:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:40:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:40:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:40:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:40:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:40:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:40:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.69461853, 2024-04-22T18:40:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:40:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:40:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:40:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:40:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:40:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:40:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:40:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:40:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:40:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:40:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:40:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:40:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:40:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:40:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:40:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:40:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:40:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:40:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:40:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:40:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:40:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:40:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:40:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.694608373999998, 2024-04-22T18:40:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:40:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:40:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:40:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:40:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:40:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:40:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:40:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:40:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:40:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:40:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:40:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:40:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:40:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:40:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:40:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:40:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:40:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:40:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:40:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:40:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:40:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:40:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:40:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:161 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:40:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:40:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.094114082999994, 2024-04-22T18:40:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40239169000000002, 2024-04-22T18:40:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 161 2024-04-22T18:40:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:40:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:40:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454938+0000", 2024-04-22T18:40:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454982+0000", 2024-04-22T18:40:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3470999999999998e-05 2024-04-22T18:40:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454988+0000", 2024-04-22T18:40:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7640000000000002e-06 2024-04-22T18:40:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555054+0000", 2024-04-22T18:40:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006607300000001 2024-04-22T18:40:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555607+0000", 2024-04-22T18:40:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055319500000000003 2024-04-22T18:40:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555689+0000", 2024-04-22T18:40:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1817999999999995e-05 2024-04-22T18:40:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555804+0000", 2024-04-22T18:40:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115346 2024-04-22T18:40:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556554+0000", 2024-04-22T18:40:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000750102 2024-04-22T18:40:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857265+0000", 2024-04-22T18:40:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30071122700000003 2024-04-22T18:40:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857302+0000", 2024-04-22T18:40:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7069e-05 2024-04-22T18:40:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857331+0000", 2024-04-22T18:40:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8291999999999999e-05 2024-04-22T18:40:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2835 4.18 4:187adac1:::smithi08937929-36:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.932683983, 2024-04-22T18:40:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.403020987, 2024-04-22T18:40:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2835 2024-04-22T18:40:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616367+0000", 2024-04-22T18:40:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616377+0000", 2024-04-22T18:40:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0618999999999999e-05 2024-04-22T18:40:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616410+0000", 2024-04-22T18:40:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3021000000000001e-05 2024-04-22T18:40:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616437+0000", 2024-04-22T18:40:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7158e-05 2024-04-22T18:40:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.716588+0000", 2024-04-22T18:40:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100150674 2024-04-22T18:40:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.716679+0000", 2024-04-22T18:40:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1231999999999997e-05 2024-04-22T18:40:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:08.019390+0000", 2024-04-22T18:40:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30271076299999999 2024-04-22T18:40:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:22.667 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:22.670 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:22.669+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:22.670 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:22.671 INFO:tasks.rados.rados.0.smithi089.stdout:1412: writing smithi08937929-18 from 621925 to 1104108 tid 1 2024-04-22T18:40:22.672 INFO:tasks.rados.rados.0.smithi089.stdout:1412: writing smithi08937929-18 from 1559254 to 2125682 tid 2 2024-04-22T18:40:22.672 INFO:tasks.rados.rados.0.smithi089.stdout:1413: set_chunk oid 13 target oid 2024-04-22T18:40:22.672 INFO:tasks.rados.rados.0.smithi089.stdout:1413: set_chunk oid 13 offset: 370688 length: 879104 target oid 7687265870c4370be322f750dced442ba67bab1cab39691bdddf2fb650823189 offset: 0 snap -1 2024-04-22T18:40:22.672 INFO:tasks.rados.rados.0.smithi089.stdout:1414: read oid 45 snap -1 2024-04-22T18:40:22.672 INFO:tasks.rados.rados.0.smithi089.stdout:1414: expect (ObjNum 266 snap 0 seq_num 266) 2024-04-22T18:40:22.672 INFO:tasks.rados.rados.0.smithi089.stdout:1415: write (excl) oid 15 current snap is 0 2024-04-22T18:40:22.672 INFO:tasks.rados.rados.0.smithi089.stdout:1415: seq_num 276 ranges {638732=557432,1617736=710086} 2024-04-22T18:40:22.672 INFO:tasks.rados.rados.0.smithi089.stdout:1415: writing smithi08937929-15 from 638732 to 1196164 tid 1 2024-04-22T18:40:22.672 INFO:tasks.rados.rados.0.smithi089.stdout:1415: writing smithi08937929-15 from 1617736 to 2327822 tid 2 2024-04-22T18:40:22.672 INFO:tasks.rados.rados.0.smithi089.stdout:1416: write (excl) oid 34 current snap is 0 2024-04-22T18:40:22.672 INFO:tasks.rados.rados.0.smithi089.stdout:1416: seq_num 277 ranges {655539=632681,2076218=570097} 2024-04-22T18:40:22.672 INFO:tasks.rados.rados.0.smithi089.stdout:1416: writing smithi08937929-34 from 655539 to 1288220 tid 1 2024-04-22T18:40:22.673 INFO:tasks.rados.rados.0.smithi089.stdout:1416: writing smithi08937929-34 from 2076218 to 2646315 tid 2 2024-04-22T18:40:22.673 INFO:tasks.rados.rados.0.smithi089.stdout:1417: write (excl) oid 46 current snap is 0 2024-04-22T18:40:22.673 INFO:tasks.rados.rados.0.smithi089.stdout:1406: expect (ObjNum 227 snap 0 seq_num 227) 2024-04-22T18:40:22.718 INFO:tasks.rados.rados.0.smithi089.stdout:1409: expect (ObjNum 272 snap 0 seq_num 272) 2024-04-22T18:40:22.734 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:22.735+0000 7f7f44bb3640 -1 osd.4 91 handle_osd_map injecting map crc failure 2024-04-22T18:40:22.737 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:40:22.769 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:22.770+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:22.779 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:22.780 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:22.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:22.780 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3216 4.6 4.a71bd0c6 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:22.780 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.668671+0000", 2024-04-22T18:40:22.780 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.112787029, 2024-04-22T18:40:22.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.112810917, 2024-04-22T18:40:22.780 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:22.780 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:22.780 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:22.780 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:22.780 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:22.780 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3216 2024-04-22T18:40:22.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:22.780 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:22.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:22.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:22.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.668671+0000", 2024-04-22T18:40:22.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:22.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:22.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:22.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:22.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.668671+0000", 2024-04-22T18:40:22.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:22.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:22.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:22.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:22.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.668670+0000", 2024-04-22T18:40:22.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:22.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:22.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:22.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:22.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.668676+0000", 2024-04-22T18:40:22.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3929999999999996e-06 2024-04-22T18:40:22.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:22.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:22.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:22.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.668704+0000", 2024-04-22T18:40:22.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8316000000000001e-05 2024-04-22T18:40:22.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:22.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:22.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.768773+0000", 2024-04-22T18:40:22.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100069087 2024-04-22T18:40:22.782 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:22.783 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:22.783 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:22.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:22.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:22.783 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:55 3.1 3.bd2eceed (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:22.783 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.769687+0000", 2024-04-22T18:40:22.783 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.011771106999999999, 2024-04-22T18:40:22.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.01185171, 2024-04-22T18:40:22.783 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:22.783 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:22.783 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:22.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:22.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:22.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.769687+0000", 2024-04-22T18:40:22.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:22.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:22.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:22.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:22.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.769687+0000", 2024-04-22T18:40:22.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:22.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:22.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:22.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:22.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.769686+0000", 2024-04-22T18:40:22.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:22.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:22.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:22.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:22.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.769694+0000", 2024-04-22T18:40:22.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5050000000000007e-06 2024-04-22T18:40:22.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:22.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:22.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:22.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.769707+0000", 2024-04-22T18:40:22.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2784e-05 2024-04-22T18:40:22.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:22.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:22.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:22.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.769717+0000", 2024-04-22T18:40:22.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0319999999999999e-05 2024-04-22T18:40:22.786 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:22.786 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:22.786 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:22.786 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:22.786 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:22.786 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 2 2024-04-22T18:40:22.786 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:22.789 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:22.842 INFO:tasks.rados.rados.0.smithi089.stdout:1407: expect (ObjNum 248 snap 0 seq_num 248) 2024-04-22T18:40:22.869 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:22.870+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:22.931 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:22.931 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:22.931 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:22.931 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:22.931 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:22.941 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:22.969 INFO:tasks.rados.rados.0.smithi089.stdout:1410: finishing write tid 1 to smithi08937929-42 2024-04-22T18:40:22.969 INFO:tasks.rados.rados.0.smithi089.stdout:1410: finishing write tid 2 to smithi08937929-42 2024-04-22T18:40:22.969 INFO:tasks.rados.rados.0.smithi089.stdout:1410: finishing write tid 3 to smithi08937929-42 2024-04-22T18:40:22.970 INFO:tasks.rados.rados.0.smithi089.stdout:1410: finishing write tid 4 to smithi08937929-42 2024-04-22T18:40:22.970 INFO:tasks.rados.rados.0.smithi089.stdout:1410: oid 42 updating version 0 to 30 2024-04-22T18:40:22.970 INFO:tasks.rados.rados.0.smithi089.stdout:1410: oid 42 updating version 30 to 31 2024-04-22T18:40:22.970 INFO:tasks.rados.rados.0.smithi089.stdout:1410: oid 42 version 31 is already newer than 29 2024-04-22T18:40:22.970 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 31 (ObjNum 274 snap 0 seq_num 274) dirty exists 2024-04-22T18:40:22.970 INFO:tasks.rados.rados.0.smithi089.stdout:1410: left oid 42 (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:22.970 INFO:tasks.rados.rados.0.smithi089.stdout:1416: finishing write tid 1 to smithi08937929-34 2024-04-22T18:40:22.970 INFO:tasks.rados.rados.0.smithi089.stdout:1417: seq_num 278 ranges {672346=707930,2134700=537646} 2024-04-22T18:40:22.971 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:22.972+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:22.976 INFO:tasks.rados.rados.0.smithi089.stdout:1417: writing smithi08937929-46 from 672346 to 1380276 tid 1 2024-04-22T18:40:22.984 INFO:tasks.rados.rados.0.smithi089.stdout:1417: writing smithi08937929-46 from 2134700 to 2672346 tid 2 2024-04-22T18:40:22.984 INFO:tasks.rados.rados.0.smithi089.stdout:1406: done (14 left) 2024-04-22T18:40:22.984 INFO:tasks.rados.rados.0.smithi089.stdout:1407: done (13 left) 2024-04-22T18:40:22.984 INFO:tasks.rados.rados.0.smithi089.stdout:1409: done (12 left) 2024-04-22T18:40:22.984 INFO:tasks.rados.rados.0.smithi089.stdout:1410: done (11 left) 2024-04-22T18:40:22.984 INFO:tasks.rados.rados.0.smithi089.stdout:1418: set_chunk oid 2 target oid 2024-04-22T18:40:23.073 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:23.073 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:23.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:23.073 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:23.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.074 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:23.074 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:23.074 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 88.617962308000003, 2024-04-22T18:40:23.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:40:23.074 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.074 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:23.074 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:23.074 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:23.074 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:23.074 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:40:23.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.074 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:23.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:23.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:40:23.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:23.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:40:23.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:40:23.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:40:23.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:40:23.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:40:23.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:40:23.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:40:23.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:40:23.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:40:23.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:40:23.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:23.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:40:23.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:40:23.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:23.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:40:23.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:40:23.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:23.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:40:23.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:40:23.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:23.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:40:23.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:40:23.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:40:23.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:40:23.079 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.079 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.079 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.080 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:315 4.b 4:d923883e:::smithi08937929-16:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:23.080 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:40:23.080 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 85.081499527999995, 2024-04-22T18:40:23.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40309447300000001, 2024-04-22T18:40:23.080 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.080 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:23.080 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:23.080 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:23.080 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:23.080 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 315 2024-04-22T18:40:23.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.081 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:40:23.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:40:23.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991817+0000", 2024-04-22T18:40:23.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:23.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991825+0000", 2024-04-22T18:40:23.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7279999999999995e-06 2024-04-22T18:40:23.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991834+0000", 2024-04-22T18:40:23.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1980000000000007e-06 2024-04-22T18:40:23.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991841+0000", 2024-04-22T18:40:23.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0740000000000004e-06 2024-04-22T18:40:23.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991881+0000", 2024-04-22T18:40:23.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-04-22T18:40:23.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991940+0000", 2024-04-22T18:40:23.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9382000000000002e-05 2024-04-22T18:40:23.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:23.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394250+0000", 2024-04-22T18:40:23.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40230947700000003 2024-04-22T18:40:23.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:23.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394752+0000", 2024-04-22T18:40:23.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050269699999999997 2024-04-22T18:40:23.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:23.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394865+0000", 2024-04-22T18:40:23.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011267700000000001 2024-04-22T18:40:23.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:23.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394892+0000", 2024-04-22T18:40:23.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7535999999999999e-05 2024-04-22T18:40:23.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394914+0000", 2024-04-22T18:40:23.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1398000000000001e-05 2024-04-22T18:40:23.086 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.086 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.086 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.086 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:774 4.b 4:defe92fb:::smithi08937929-44:head [write 2778819~95145] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:23.086 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:40:23.086 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 79.924989851999996, 2024-04-22T18:40:23.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40475725000000001, 2024-04-22T18:40:23.087 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.087 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:23.087 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:23.087 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:23.087 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:23.087 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 774 2024-04-22T18:40:23.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.087 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:40:23.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:40:23.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148328+0000", 2024-04-22T18:40:23.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:23.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148392+0000", 2024-04-22T18:40:23.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3323000000000003e-05 2024-04-22T18:40:23.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148398+0000", 2024-04-22T18:40:23.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4420000000000003e-06 2024-04-22T18:40:23.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148409+0000", 2024-04-22T18:40:23.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509999999999999e-05 2024-04-22T18:40:23.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251407+0000", 2024-04-22T18:40:23.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102998305 2024-04-22T18:40:23.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251500+0000", 2024-04-22T18:40:23.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3607e-05 2024-04-22T18:40:23.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:23.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251648+0000", 2024-04-22T18:40:23.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014788099999999999 2024-04-22T18:40:23.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:23.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256149+0000", 2024-04-22T18:40:23.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045010980000000003 2024-04-22T18:40:23.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:23.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553016+0000", 2024-04-22T18:40:23.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29686620800000002 2024-04-22T18:40:23.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:23.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553054+0000", 2024-04-22T18:40:23.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8414999999999997e-05 2024-04-22T18:40:23.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553086+0000", 2024-04-22T18:40:23.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2236000000000001e-05 2024-04-22T18:40:23.092 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.092 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.092 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.092 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:23.093 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:23.093 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 79.92490506, 2024-04-22T18:40:23.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:40:23.093 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.093 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:23.093 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:23.093 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:23.093 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:23.093 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:40:23.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.093 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:23.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:23.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:40:23.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:23.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:40:23.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:40:23.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:40:23.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:40:23.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:40:23.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:40:23.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:40:23.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:40:23.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:40:23.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:40:23.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:23.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:40:23.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:40:23.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:40:23.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:40:23.098 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.098 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.098 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.098 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:23.098 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:23.098 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 79.924886584000006, 2024-04-22T18:40:23.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:40:23.099 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.099 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:23.099 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:23.099 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:23.099 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:23.099 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:40:23.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.099 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:23.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:23.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:23.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:23.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:40:23.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:40:23.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:40:23.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:40:23.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:40:23.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:40:23.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:40:23.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:40:23.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:23.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:40:23.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:40:23.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:40:23.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:40:23.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:40:23.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:40:23.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:40:23.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:40:23.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.103 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.103 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1242 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-22T18:40:23.104 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:40:23.104 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 66.331641035000004, 2024-04-22T18:40:23.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40662891899999998, 2024-04-22T18:40:23.104 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.104 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:23.104 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:23.104 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:23.104 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:23.104 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1242 2024-04-22T18:40:23.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.104 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:40:23.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:40:23.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741677+0000", 2024-04-22T18:40:23.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:23.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741680+0000", 2024-04-22T18:40:23.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8059999999999999e-06 2024-04-22T18:40:23.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741685+0000", 2024-04-22T18:40:23.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0780000000000003e-06 2024-04-22T18:40:23.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741690+0000", 2024-04-22T18:40:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4020000000000004e-06 2024-04-22T18:40:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741732+0000", 2024-04-22T18:40:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2110000000000002e-05 2024-04-22T18:40:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741798+0000", 2024-04-22T18:40:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6432000000000004e-05 2024-04-22T18:40:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.148307+0000", 2024-04-22T18:40:23.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40650862100000001 2024-04-22T18:40:23.108 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.108 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.108 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.108 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1408 4.6 4:630bd8e5:::smithi08937929-45:head [write 2816389~693495] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:40:23.108 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:40:23.108 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.911671228000003, 2024-04-22T18:40:23.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408959452, 2024-04-22T18:40:23.108 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.108 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:23.108 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:23.108 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:23.109 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:23.109 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1408 2024-04-22T18:40:23.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.109 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:40:23.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:40:23.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161647+0000", 2024-04-22T18:40:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161951+0000", 2024-04-22T18:40:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030389799999999998 2024-04-22T18:40:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161955+0000", 2024-04-22T18:40:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1989999999999999e-06 2024-04-22T18:40:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161958+0000", 2024-04-22T18:40:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2260000000000001e-06 2024-04-22T18:40:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.265867+0000", 2024-04-22T18:40:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103908927 2024-04-22T18:40:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.265960+0000", 2024-04-22T18:40:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3016999999999994e-05 2024-04-22T18:40:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:23.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.266124+0000", 2024-04-22T18:40:23.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016347700000000001 2024-04-22T18:40:23.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:23.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570491+0000", 2024-04-22T18:40:23.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30436694199999997 2024-04-22T18:40:23.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:23.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570502+0000", 2024-04-22T18:40:23.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1107999999999999e-05 2024-04-22T18:40:23.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570556+0000", 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4388000000000002e-05 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570607+0000", 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0782999999999998e-05 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1409 4.6 4:630bd8e5:::smithi08937929-45:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:40:23.113 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:23.114 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.911357524000003, 2024-04-22T18:40:23.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40880186200000002, 2024-04-22T18:40:23.114 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.114 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:23.114 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:23.114 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:23.114 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:23.114 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1409 2024-04-22T18:40:23.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.114 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:23.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:23.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:23.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:23.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161964+0000", 2024-04-22T18:40:23.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5909999999999998e-06 2024-04-22T18:40:23.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161966+0000", 2024-04-22T18:40:23.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5979999999999999e-06 2024-04-22T18:40:23.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161969+0000", 2024-04-22T18:40:23.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7049999999999999e-06 2024-04-22T18:40:23.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269205+0000", 2024-04-22T18:40:23.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107236228 2024-04-22T18:40:23.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269320+0000", 2024-04-22T18:40:23.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114571 2024-04-22T18:40:23.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:23.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269476+0000", 2024-04-22T18:40:23.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155928 2024-04-22T18:40:23.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:23.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570619+0000", 2024-04-22T18:40:23.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30114322700000001 2024-04-22T18:40:23.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:23.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570672+0000", 2024-04-22T18:40:23.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3232000000000001e-05 2024-04-22T18:40:23.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:23.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570724+0000", 2024-04-22T18:40:23.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2258e-05 2024-04-22T18:40:23.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570763+0000", 2024-04-22T18:40:23.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8927999999999997e-05 2024-04-22T18:40:23.119 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.119 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.119 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.119 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1410 4.6 4:630bd8e5:::smithi08937929-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:40:23.119 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:23.119 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.911348007000001, 2024-04-22T18:40:23.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40900504500000001, 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1410 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:23.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:23.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:23.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161972+0000", 2024-04-22T18:40:23.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6419999999999999e-06 2024-04-22T18:40:23.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161976+0000", 2024-04-22T18:40:23.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9260000000000002e-06 2024-04-22T18:40:23.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161979+0000", 2024-04-22T18:40:23.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9359999999999999e-06 2024-04-22T18:40:23.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570790+0000", 2024-04-22T18:40:23.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40881099199999998 2024-04-22T18:40:23.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570879+0000", 2024-04-22T18:40:23.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8895999999999997e-05 2024-04-22T18:40:23.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570976+0000", 2024-04-22T18:40:23.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6923999999999996e-05 2024-04-22T18:40:23.123 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.123 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.123 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.123 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:23.124 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:23.124 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.161838271999997, 2024-04-22T18:40:23.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:40:23.124 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.124 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:23.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:23.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:23.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:23.124 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:40:23.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.124 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:23.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:23.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:40:23.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:23.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:40:23.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:40:23.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:40:23.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:40:23.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:40:23.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:40:23.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:40:23.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:40:23.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:40:23.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:40:23.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:23.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:40:23.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:40:23.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:23.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:40:23.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:40:23.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:23.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:40:23.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:40:23.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:23.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:40:23.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:40:23.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:40:23.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:40:23.129 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.129 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.129 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.130 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:23.130 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:23.130 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.632678945999999, 2024-04-22T18:40:23.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:40:23.130 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.130 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:23.130 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:23.130 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:23.130 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:23.130 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:40:23.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.131 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:23.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:23.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:23.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:23.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:40:23.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:40:23.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:40:23.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:40:23.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:40:23.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:40:23.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:40:23.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:40:23.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:23.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:40:23.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:40:23.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:40:23.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:40:23.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:40:23.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:40:23.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:40:23.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:40:23.135 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.135 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.135 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.135 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:23.135 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:23.135 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.454971551, 2024-04-22T18:40:23.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:40:23.135 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.135 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:23.135 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:23.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:23.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:40:23.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:23.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:40:23.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:40:23.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:40:23.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:40:23.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:40:23.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:40:23.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:40:23.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:40:23.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:40:23.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:40:23.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:23.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:40:23.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:40:23.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:23.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:40:23.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:40:23.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:23.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:40:23.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:40:23.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:23.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:40:23.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:40:23.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:40:23.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:40:23.140 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.140 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.140 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.141 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:23.141 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:23.141 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.454903279, 2024-04-22T18:40:23.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:40:23.141 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.141 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:23.141 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:23.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:23.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:40:23.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:23.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:40:23.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:40:23.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:40:23.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:40:23.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:40:23.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:40:23.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:40:23.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:40:23.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:40:23.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:40:23.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:23.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:40:23.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:40:23.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:23.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:40:23.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:40:23.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:23.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:40:23.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:40:23.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:23.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:40:23.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:40:23.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:40:23.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:23.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.146 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.146 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:23.146 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:23.146 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.454735438, 2024-04-22T18:40:23.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:40:23.147 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.147 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:23.147 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:23.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:23.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:23.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:23.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:40:23.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:40:23.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:40:23.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:40:23.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:40:23.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:40:23.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:40:23.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:40:23.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:40:23.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:40:23.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:23.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:40:23.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:40:23.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:23.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:40:23.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:40:23.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:23.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:40:23.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:40:23.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:23.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:40:23.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:40:23.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:40:23.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:40:23.151 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.489409963, 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:23.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:40:23.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:23.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:40:23.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:40:23.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:40:23.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:40:23.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:40:23.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:40:23.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:40:23.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:40:23.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:40:23.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:40:23.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:23.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:40:23.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:40:23.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:23.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:40:23.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:40:23.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:23.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:40:23.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:40:23.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:23.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:40:23.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:40:23.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:40:23.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:40:23.157 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.157 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.157 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.157 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:23.157 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:23.158 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.389257776000001, 2024-04-22T18:40:23.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:40:23.158 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.158 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:23.158 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:23.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:23.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:40:23.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:23.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:40:23.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:40:23.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:40:23.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:40:23.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:23.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:40:23.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:40:23.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:23.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:40:23.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:40:23.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:23.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:40:23.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:40:23.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:23.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:40:23.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:40:23.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:40:23.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:40:23.162 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.163 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.163 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.163 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2385 4.13 4:c93c842e:::smithi08937929-25:head [write 1937856~527061] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:40:23.163 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.287418+0000", 2024-04-22T18:40:23.163 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.785900718000001, 2024-04-22T18:40:23.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39986265300000001, 2024-04-22T18:40:23.163 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.163 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:23.163 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:23.163 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:23.164 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:23.164 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2385 2024-04-22T18:40:23.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.164 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287418+0000", 2024-04-22T18:40:23.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287418+0000", 2024-04-22T18:40:23.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287417+0000", 2024-04-22T18:40:23.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:23.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287636+0000", 2024-04-22T18:40:23.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000219089 2024-04-22T18:40:23.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287654+0000", 2024-04-22T18:40:23.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8294000000000001e-05 2024-04-22T18:40:23.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287661+0000", 2024-04-22T18:40:23.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7780000000000002e-06 2024-04-22T18:40:23.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.483858+0000", 2024-04-22T18:40:23.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19619666199999999 2024-04-22T18:40:23.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.483917+0000", 2024-04-22T18:40:23.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9454999999999999e-05 2024-04-22T18:40:23.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.483999+0000", 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1359999999999994e-05 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586037+0000", 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102038217 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687214+0000", 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101177195 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687253+0000", 2024-04-22T18:40:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9409999999999997e-05 2024-04-22T18:40:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687281+0000", 2024-04-22T18:40:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7478e-05 2024-04-22T18:40:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:40:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.680393256000002, 2024-04-22T18:40:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:40:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:40:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:23.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:40:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:40:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:40:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:40:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:40:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:40:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:40:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:40:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:40:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:40:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:40:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:40:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:40:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:40:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:40:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:40:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:40:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:23.173 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 15.784354884000001, 2024-04-22T18:40:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:40:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:40:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:40:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:40:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:40:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:40:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:40:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:40:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:40:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:40:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:40:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:40:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:40:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:40:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:203 3.1 3:81736bc9:::096453d42e9dfceb4e0d7d7f6f19be374b0d28e8be156d4462bac4cc8cafbecd:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e83)", 2024-04-22T18:40:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:13.982753+0000", 2024-04-22T18:40:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.0905656310000005, 2024-04-22T18:40:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40124371800000003, 2024-04-22T18:40:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982753+0000", 2024-04-22T18:40:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982753+0000", 2024-04-22T18:40:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982751+0000", 2024-04-22T18:40:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982760+0000", 2024-04-22T18:40:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6840000000000002e-06 2024-04-22T18:40:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982775+0000", 2024-04-22T18:40:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4831000000000001e-05 2024-04-22T18:40:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982783+0000", 2024-04-22T18:40:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0609999999999996e-06 2024-04-22T18:40:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982838+0000", 2024-04-22T18:40:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4922e-05 2024-04-22T18:40:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982940+0000", 2024-04-22T18:40:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102082 2024-04-22T18:40:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.983117+0000", 2024-04-22T18:40:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017719900000000001 2024-04-22T18:40:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.983739+0000", 2024-04-22T18:40:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00062191700000000004 2024-04-22T18:40:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:14.383901+0000", 2024-04-22T18:40:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40016206100000001 2024-04-22T18:40:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:14.383954+0000", 2024-04-22T18:40:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2855000000000001e-05 2024-04-22T18:40:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:14.383997+0000", 2024-04-22T18:40:23.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3034000000000002e-05 2024-04-22T18:40:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:23.183 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:23.183 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:23.183 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:23.074+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:23.184 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:23.193 INFO:tasks.rados.rados.0.smithi089.stdout:1418: set_chunk oid 2 offset: 1249280 length: 926720 target oid 428e7a313f51966e06ef4d9653fd5b09e17c1df0b67c11ff4c9324c339b4be83 offset: 0 snap -1 2024-04-22T18:40:23.193 INFO:tasks.rados.rados.0.smithi089.stdout:1419: read oid 35 snap -1 2024-04-22T18:40:23.193 INFO:tasks.rados.rados.0.smithi089.stdout:1419: expect (ObjNum 270 snap 0 seq_num 270) 2024-04-22T18:40:23.193 INFO:tasks.rados.rados.0.smithi089.stdout:1420: write oid 16 current snap is 0 2024-04-22T18:40:23.193 INFO:tasks.rados.rados.0.smithi089.stdout:1416: finishing write tid 2 to smithi08937929-34 2024-04-22T18:40:23.193 INFO:tasks.rados.rados.0.smithi089.stdout:1416: finishing write tid 3 to smithi08937929-34 2024-04-22T18:40:23.193 INFO:tasks.rados.rados.0.smithi089.stdout:1416: finishing write tid 4 to smithi08937929-34 2024-04-22T18:40:23.193 INFO:tasks.rados.rados.0.smithi089.stdout:1416: oid 34 updating version 0 to 105 2024-04-22T18:40:23.193 INFO:tasks.rados.rados.0.smithi089.stdout:1416: oid 34 version 105 is already newer than 104 2024-04-22T18:40:23.193 INFO:tasks.rados.rados.0.smithi089.stdout:1416: oid 34 updating version 105 to 106 2024-04-22T18:40:23.194 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 106 (ObjNum 277 snap 0 seq_num 277) dirty exists 2024-04-22T18:40:23.194 INFO:tasks.rados.rados.0.smithi089.stdout:1416: left oid 34 (ObjNum 277 snap 0 seq_num 277) 2024-04-22T18:40:23.194 INFO:tasks.rados.rados.0.smithi089.stdout:1412: finishing write tid 1 to smithi08937929-18 2024-04-22T18:40:23.194 INFO:tasks.rados.rados.0.smithi089.stdout:1412: finishing write tid 2 to smithi08937929-18 2024-04-22T18:40:23.194 INFO:tasks.rados.rados.0.smithi089.stdout:1412: finishing write tid 3 to smithi08937929-18 2024-04-22T18:40:23.194 INFO:tasks.rados.rados.0.smithi089.stdout:1412: finishing write tid 4 to smithi08937929-18 2024-04-22T18:40:23.194 INFO:tasks.rados.rados.0.smithi089.stdout:1412: oid 18 updating version 0 to 73 2024-04-22T18:40:23.194 INFO:tasks.rados.rados.0.smithi089.stdout:1412: oid 18 updating version 73 to 74 2024-04-22T18:40:23.194 INFO:tasks.rados.rados.0.smithi089.stdout:1412: oid 18 updating version 74 to 75 2024-04-22T18:40:23.194 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 75 (ObjNum 275 snap 0 seq_num 275) dirty exists 2024-04-22T18:40:23.194 INFO:tasks.rados.rados.0.smithi089.stdout:1412: left oid 18 (ObjNum 275 snap 0 seq_num 275) 2024-04-22T18:40:23.194 INFO:tasks.rados.rados.0.smithi089.stdout:1411: finishing set_chunk to oid 4 2024-04-22T18:40:23.195 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 103 (ObjNum 269 snap 0 seq_num 269) dirty exists 2024-04-22T18:40:23.195 INFO:tasks.rados.rados.0.smithi089.stdout:1414: expect (ObjNum 266 snap 0 seq_num 266) 2024-04-22T18:40:23.195 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:23.193+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:23.287 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:23.288+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:23.298 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:23.298 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:23.298 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:23.299 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:23.307 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:23.363 INFO:tasks.rados.rados.0.smithi089.stdout:1415: finishing write tid 1 to smithi08937929-15 2024-04-22T18:40:23.363 INFO:tasks.rados.rados.0.smithi089.stdout:1420: seq_num 279 ranges {689153=783179,2076829=457413} 2024-04-22T18:40:23.372 INFO:tasks.rados.rados.0.smithi089.stdout:1420: writing smithi08937929-16 from 689153 to 1472332 tid 1 2024-04-22T18:40:23.379 INFO:tasks.rados.rados.0.smithi089.stdout:1420: writing smithi08937929-16 from 2076829 to 2534242 tid 2 2024-04-22T18:40:23.379 INFO:tasks.rados.rados.0.smithi089.stdout:1411: done (13 left) 2024-04-22T18:40:23.379 INFO:tasks.rados.rados.0.smithi089.stdout:1412: done (12 left) 2024-04-22T18:40:23.379 INFO:tasks.rados.rados.0.smithi089.stdout:1414: done (11 left) 2024-04-22T18:40:23.379 INFO:tasks.rados.rados.0.smithi089.stdout:1416: done (10 left) 2024-04-22T18:40:23.379 INFO:tasks.rados.rados.0.smithi089.stdout:1421: tier_promote oid 6 2024-04-22T18:40:23.380 INFO:tasks.rados.rados.0.smithi089.stdout:1397: expect (ObjNum 208 snap 0 seq_num 208) 2024-04-22T18:40:23.387 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:23.389+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:23.390 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:40:23.416 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:23.416 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:23.416 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:23.416 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:23.416 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:23.425 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:23.509 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:23.510+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:23.521 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:23.538 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:23.538 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:23.554 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:23.554 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:23.569 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:23.569 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:23.570 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:23.570 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:23.579 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:23.586 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:23.587 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:23.588 INFO:tasks.rados.rados.0.smithi089.stdout:1404: got (2) No such file or directory 2024-04-22T18:40:23.589 INFO:tasks.rados.rados.0.smithi089.stdout:1404: got expected ENOENT (src dne) 2024-04-22T18:40:23.589 INFO:tasks.rados.rados.0.smithi089.stdout:1415: finishing write tid 2 to smithi08937929-15 2024-04-22T18:40:23.589 INFO:tasks.rados.rados.0.smithi089.stdout:1415: finishing write tid 3 to smithi08937929-15 2024-04-22T18:40:23.589 INFO:tasks.rados.rados.0.smithi089.stdout:1397: done (10 left) 2024-04-22T18:40:23.589 INFO:tasks.rados.rados.0.smithi089.stdout:1404: done (9 left) 2024-04-22T18:40:23.589 INFO:tasks.rados.rados.0.smithi089.stdout:1422: read oid 10 snap -1 2024-04-22T18:40:23.589 INFO:tasks.rados.rados.0.smithi089.stdout:1422: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:40:23.589 INFO:tasks.rados.rados.0.smithi089.stdout:1423: read oid 32 snap -1 2024-04-22T18:40:23.589 INFO:tasks.rados.rados.0.smithi089.stdout:1423: expect (ObjNum 209 snap 0 seq_num 209) 2024-04-22T18:40:23.589 INFO:tasks.rados.rados.0.smithi089.stdout:1424: write (excl) oid 24 current snap is 0 2024-04-22T18:40:23.589 INFO:tasks.rados.rados.0.smithi089.stdout:1415: finishing write tid 4 to smithi08937929-15 2024-04-22T18:40:23.589 INFO:tasks.rados.rados.0.smithi089.stdout:1415: oid 15 updating version 0 to 69 2024-04-22T18:40:23.589 INFO:tasks.rados.rados.0.smithi089.stdout:1415: oid 15 version 69 is already newer than 67 2024-04-22T18:40:23.590 INFO:tasks.rados.rados.0.smithi089.stdout:1415: oid 15 version 69 is already newer than 68 2024-04-22T18:40:23.590 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 69 (ObjNum 276 snap 0 seq_num 276) dirty exists 2024-04-22T18:40:23.590 INFO:tasks.rados.rados.0.smithi089.stdout:1415: left oid 15 (ObjNum 276 snap 0 seq_num 276) 2024-04-22T18:40:23.590 INFO:tasks.rados.rados.0.smithi089.stdout:1417: finishing write tid 1 to smithi08937929-46 2024-04-22T18:40:23.590 INFO:tasks.rados.rados.0.smithi089.stdout:1417: finishing write tid 2 to smithi08937929-46 2024-04-22T18:40:23.590 INFO:tasks.rados.rados.0.smithi089.stdout:1417: finishing write tid 3 to smithi08937929-46 2024-04-22T18:40:23.590 INFO:tasks.rados.rados.0.smithi089.stdout:1417: finishing write tid 4 to smithi08937929-46 2024-04-22T18:40:23.590 INFO:tasks.rados.rados.0.smithi089.stdout:1417: oid 46 updating version 0 to 24 2024-04-22T18:40:23.590 INFO:tasks.rados.rados.0.smithi089.stdout:1417: oid 46 updating version 24 to 26 2024-04-22T18:40:23.590 INFO:tasks.rados.rados.0.smithi089.stdout:1417: oid 46 version 26 is already newer than 25 2024-04-22T18:40:23.590 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 26 (ObjNum 278 snap 0 seq_num 278) dirty exists 2024-04-22T18:40:23.590 INFO:tasks.rados.rados.0.smithi089.stdout:1417: left oid 46 (ObjNum 278 snap 0 seq_num 278) 2024-04-22T18:40:23.591 INFO:tasks.rados.rados.0.smithi089.stdout:1418: finishing set_chunk to oid 2 2024-04-22T18:40:23.591 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk returned error code -95 offset: 1249280 length: 926720 2024-04-22T18:40:23.591 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 251 snap 0 seq_num 251) dirty exists 2024-04-22T18:40:23.591 INFO:tasks.rados.rados.0.smithi089.stdout:1419: expect (ObjNum 270 snap 0 seq_num 270) 2024-04-22T18:40:23.592 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:23.593+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:23.605 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:23.605 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:23.623 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:23.623 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:23.640 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:23.640 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:23.692 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:23.693 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:23.693 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:23.694+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:23.702 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:23.756 INFO:tasks.rados.rados.0.smithi089.stdout:1413: finishing set_chunk to oid 13 2024-04-22T18:40:23.756 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 66 (ObjNum 261 snap 0 seq_num 261) dirty exists 2024-04-22T18:40:23.756 INFO:tasks.rados.rados.0.smithi089.stdout:1420: finishing write tid 1 to smithi08937929-16 2024-04-22T18:40:23.756 INFO:tasks.rados.rados.0.smithi089.stdout:1420: finishing write tid 2 to smithi08937929-16 2024-04-22T18:40:23.756 INFO:tasks.rados.rados.0.smithi089.stdout:1420: finishing write tid 3 to smithi08937929-16 2024-04-22T18:40:23.756 INFO:tasks.rados.rados.0.smithi089.stdout:1420: finishing write tid 4 to smithi08937929-16 2024-04-22T18:40:23.756 INFO:tasks.rados.rados.0.smithi089.stdout:1420: oid 16 updating version 0 to 115 2024-04-22T18:40:23.756 INFO:tasks.rados.rados.0.smithi089.stdout:1420: oid 16 updating version 115 to 116 2024-04-22T18:40:23.756 INFO:tasks.rados.rados.0.smithi089.stdout:1420: oid 16 version 116 is already newer than 114 2024-04-22T18:40:23.757 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 116 (ObjNum 279 snap 0 seq_num 279) dirty exists 2024-04-22T18:40:23.757 INFO:tasks.rados.rados.0.smithi089.stdout:1420: left oid 16 (ObjNum 279 snap 0 seq_num 279) 2024-04-22T18:40:23.757 INFO:tasks.rados.rados.0.smithi089.stdout:1421: got (0) Success 2024-04-22T18:40:23.757 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 51 (ObjNum 199 snap 0 seq_num 199) dirty exists 2024-04-22T18:40:23.757 INFO:tasks.rados.rados.0.smithi089.stdout:1423: expect (ObjNum 209 snap 0 seq_num 209) 2024-04-22T18:40:23.794 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:23.795+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:23.807 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:23.807 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:23.816 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:23.872 INFO:tasks.rados.rados.0.smithi089.stdout:1422: expect (ObjNum 124 snap 0 seq_num 124) 2024-04-22T18:40:23.895 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:23.896+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:23.922 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:23.922 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:23.931 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:23.942 INFO:tasks.rados.rados.0.smithi089.stdout:1424: seq_num 280 ranges {705960=458428,1735311=717424} 2024-04-22T18:40:23.949 INFO:tasks.rados.rados.0.smithi089.stdout:1424: writing smithi08937929-24 from 705960 to 1164388 tid 1 2024-04-22T18:40:23.957 INFO:tasks.rados.rados.0.smithi089.stdout:1424: writing smithi08937929-24 from 1735311 to 2452735 tid 2 2024-04-22T18:40:23.957 INFO:tasks.rados.rados.0.smithi089.stdout:1405: done (11 left) 2024-04-22T18:40:23.957 INFO:tasks.rados.rados.0.smithi089.stdout:1413: done (10 left) 2024-04-22T18:40:23.957 INFO:tasks.rados.rados.0.smithi089.stdout:1415: done (9 left) 2024-04-22T18:40:23.958 INFO:tasks.rados.rados.0.smithi089.stdout:1417: done (8 left) 2024-04-22T18:40:23.958 INFO:tasks.rados.rados.0.smithi089.stdout:1418: done (7 left) 2024-04-22T18:40:23.958 INFO:tasks.rados.rados.0.smithi089.stdout:1419: done (6 left) 2024-04-22T18:40:23.958 INFO:tasks.rados.rados.0.smithi089.stdout:1420: done (5 left) 2024-04-22T18:40:23.958 INFO:tasks.rados.rados.0.smithi089.stdout:1421: done (4 left) 2024-04-22T18:40:23.958 INFO:tasks.rados.rados.0.smithi089.stdout:1422: done (3 left) 2024-04-22T18:40:23.958 INFO:tasks.rados.rados.0.smithi089.stdout:1423: done (2 left) 2024-04-22T18:40:23.958 INFO:tasks.rados.rados.0.smithi089.stdout:1425: tier_promote oid 39 2024-04-22T18:40:23.958 INFO:tasks.rados.rados.0.smithi089.stdout:1424: finishing write tid 1 to smithi08937929-24 2024-04-22T18:40:23.958 INFO:tasks.rados.rados.0.smithi089.stdout:1426: read oid 31 snap -1 2024-04-22T18:40:23.958 INFO:tasks.rados.rados.0.smithi089.stdout:1426: expect deleted 2024-04-22T18:40:23.958 INFO:tasks.rados.rados.0.smithi089.stdout:1427: read oid 4 snap -1 2024-04-22T18:40:23.958 INFO:tasks.rados.rados.0.smithi089.stdout:1427: expect (ObjNum 269 snap 0 seq_num 269) 2024-04-22T18:40:23.958 INFO:tasks.rados.rados.0.smithi089.stdout:1428: tier_promote oid 8 2024-04-22T18:40:23.959 INFO:tasks.rados.rados.0.smithi089.stdout:1429: read oid 32 snap -1 2024-04-22T18:40:23.959 INFO:tasks.rados.rados.0.smithi089.stdout:1429: expect (ObjNum 209 snap 0 seq_num 209) 2024-04-22T18:40:23.959 INFO:tasks.rados.rados.0.smithi089.stdout:1430: write oid 47 current snap is 0 2024-04-22T18:40:23.959 INFO:tasks.rados.rados.0.smithi089.stdout:1430: seq_num 281 ranges {722767=533677,1793793=461082} 2024-04-22T18:40:23.963 INFO:tasks.rados.rados.0.smithi089.stdout:1430: writing smithi08937929-47 from 722767 to 1256444 tid 1 2024-04-22T18:40:23.969 INFO:tasks.rados.rados.0.smithi089.stdout:1430: writing smithi08937929-47 from 1793793 to 2254875 tid 2 2024-04-22T18:40:23.969 INFO:tasks.rados.rados.0.smithi089.stdout:1431: read oid 33 snap -1 2024-04-22T18:40:23.969 INFO:tasks.rados.rados.0.smithi089.stdout:1431: expect (ObjNum 262 snap 0 seq_num 262) 2024-04-22T18:40:23.969 INFO:tasks.rados.rados.0.smithi089.stdout:1432: set_chunk oid 1 target oid 2024-04-22T18:40:23.969 INFO:tasks.rados.rados.0.smithi089.stdout:1432: set_chunk oid 1 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:40:23.969 INFO:tasks.rados.rados.0.smithi089.stdout:1433: read oid 17 snap -1 2024-04-22T18:40:23.969 INFO:tasks.rados.rados.0.smithi089.stdout:1433: expect (ObjNum 192 snap 0 seq_num 192) 2024-04-22T18:40:23.969 INFO:tasks.rados.rados.0.smithi089.stdout:1434: set_chunk oid 48 target oid 2024-04-22T18:40:23.995 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:23.996+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:24.033 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:24.033 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:24.042 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:24.125 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:24.127+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:24.187 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:24.187 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:24.196 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:24.197 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:24.198+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:24.268 INFO:tasks.rados.rados.0.smithi089.stdout:1434: set_chunk oid 48 offset: 855552 length: 722432 target oid ff5dd6d50a07f52f5c18f586c42a4f1cd93f755b198cb05b1453fc6ed2555f85 offset: 0 snap -1 2024-04-22T18:40:24.268 INFO:tasks.rados.rados.0.smithi089.stdout:1435: read oid 23 snap -1 2024-04-22T18:40:24.268 INFO:tasks.rados.rados.0.smithi089.stdout:1435: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:40:24.268 INFO:tasks.rados.rados.0.smithi089.stdout:1436: set_chunk oid 11 target oid 2024-04-22T18:40:24.298 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:24.298+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:24.340 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:24.340 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:24.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:24.398 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:24.400+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:24.461 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:24.462 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:24.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:24.499 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:24.501+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:24.574 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:24.574 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:24.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:24.626 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:24.627+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:24.664 INFO:tasks.rados.rados.0.smithi089.stdout:1436: set_chunk oid 11 offset: 1923072 length: 995840 target oid 91a662067350c2527fb28d04562862f3c169a91cf4b1ba4dc3a2f90ea4f2c8c2 offset: 0 snap -1 2024-04-22T18:40:24.664 INFO:tasks.rados.rados.0.smithi089.stdout:1437: write (excl) oid 22 current snap is 0 2024-04-22T18:40:24.664 INFO:tasks.rados.rados.0.smithi089.stdout:1408: finishing write tid 1 to smithi08937929-19 2024-04-22T18:40:24.664 INFO:tasks.rados.rados.0.smithi089.stdout:1437: seq_num 282 ranges {739574=725279,1968628=721093} 2024-04-22T18:40:24.672 INFO:tasks.rados.rados.0.smithi089.stdout:1437: writing smithi08937929-22 from 739574 to 1464853 tid 1 2024-04-22T18:40:24.679 INFO:tasks.rados.rados.0.smithi089.stdout:1437: writing smithi08937929-22 from 1968628 to 2689721 tid 2 2024-04-22T18:40:24.679 INFO:tasks.rados.rados.0.smithi089.stdout:1438: read oid 16 snap -1 2024-04-22T18:40:24.679 INFO:tasks.rados.rados.0.smithi089.stdout:1438: expect (ObjNum 279 snap 0 seq_num 279) 2024-04-22T18:40:24.679 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:24.679 INFO:tasks.rados.rados.0.smithi089.stdout:1408: finishing write tid 2 to smithi08937929-19 2024-04-22T18:40:24.680 INFO:tasks.rados.rados.0.smithi089.stdout:1408: finishing write tid 3 to smithi08937929-19 2024-04-22T18:40:24.680 INFO:tasks.rados.rados.0.smithi089.stdout:1408: finishing write tid 4 to smithi08937929-19 2024-04-22T18:40:24.680 INFO:tasks.rados.rados.0.smithi089.stdout:1408: oid 19 updating version 0 to 114 2024-04-22T18:40:24.680 INFO:tasks.rados.rados.0.smithi089.stdout:1408: oid 19 updating version 114 to 115 2024-04-22T18:40:24.680 INFO:tasks.rados.rados.0.smithi089.stdout:1408: oid 19 updating version 115 to 116 2024-04-22T18:40:24.680 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 116 (ObjNum 273 snap 0 seq_num 273) dirty exists 2024-04-22T18:40:24.680 INFO:tasks.rados.rados.0.smithi089.stdout:1408: left oid 19 (ObjNum 273 snap 0 seq_num 273) 2024-04-22T18:40:24.680 INFO:tasks.rados.rados.0.smithi089.stdout:1432: finishing set_chunk to oid 1 2024-04-22T18:40:24.680 INFO:tasks.rados.rados.0.smithi089.stdout:1432: got expected ENOENT (src dne) 2024-04-22T18:40:24.680 INFO:tasks.rados.rados.0.smithi089.stdout:1408: done (15 left) 2024-04-22T18:40:24.680 INFO:tasks.rados.rados.0.smithi089.stdout:1432: done (14 left) 2024-04-22T18:40:24.680 INFO:tasks.rados.rados.0.smithi089.stdout:1439: read oid 29 snap -1 2024-04-22T18:40:24.680 INFO:tasks.rados.rados.0.smithi089.stdout:1439: expect (ObjNum 149 snap 0 seq_num 149) 2024-04-22T18:40:24.680 INFO:tasks.rados.rados.0.smithi089.stdout:1440: set_chunk oid 13 target oid 2024-04-22T18:40:24.686 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:24.686 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:24.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:24.701 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:24.702+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:24.747 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:24.748+0000 7fec57bee640 -1 osd.7 93 handle_osd_map injecting map crc failure 2024-04-22T18:40:24.749 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:24.750+0000 7f3093031640 -1 osd.1 93 handle_osd_map injecting map crc failure 2024-04-22T18:40:24.753 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:40:24.802 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:24.803+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:24.838 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:24.838 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:24.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:24.902 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:24.904+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:24.985 INFO:tasks.rados.rados.0.smithi089.stdout:1440: set_chunk oid 13 offset: 187392 length: 1030144 target oid 88f5dcc6382512d967c31e9ff351c76732da7aebba828b2f554e518334424da5 offset: 0 snap -1 2024-04-22T18:40:24.986 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:24.986 INFO:tasks.rados.rados.0.smithi089.stdout:1430: finishing write tid 1 to smithi08937929-47 2024-04-22T18:40:24.986 INFO:tasks.rados.rados.0.smithi089.stdout:1430: finishing write tid 2 to smithi08937929-47 2024-04-22T18:40:24.986 INFO:tasks.rados.rados.0.smithi089.stdout:1430: finishing write tid 3 to smithi08937929-47 2024-04-22T18:40:24.986 INFO:tasks.rados.rados.0.smithi089.stdout:1428: got (0) Success 2024-04-22T18:40:24.986 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 22 (ObjNum 253 snap 0 seq_num 253) dirty exists 2024-04-22T18:40:24.986 INFO:tasks.rados.rados.0.smithi089.stdout:1433: expect (ObjNum 192 snap 0 seq_num 192) 2024-04-22T18:40:24.988 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:24.988 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:24.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:25.040 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:25.041+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:25.098 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:25.098 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:25.104 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:25.105+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:25.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:25.121 INFO:tasks.rados.rados.0.smithi089.stdout:1429: expect (ObjNum 209 snap 0 seq_num 209) 2024-04-22T18:40:25.205 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:25.206+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:25.231 INFO:tasks.rados.rados.0.smithi089.stdout:1424: finishing write tid 2 to smithi08937929-24 2024-04-22T18:40:25.231 INFO:tasks.rados.rados.0.smithi089.stdout:1427: expect (ObjNum 269 snap 0 seq_num 269) 2024-04-22T18:40:25.249 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:25.249 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:25.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:25.342 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:25.343+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:25.394 INFO:tasks.rados.rados.0.smithi089.stdout:1424: finishing write tid 3 to smithi08937929-24 2024-04-22T18:40:25.395 INFO:tasks.rados.rados.0.smithi089.stdout:1424: finishing write tid 4 to smithi08937929-24 2024-04-22T18:40:25.395 INFO:tasks.rados.rados.0.smithi089.stdout:1424: oid 24 updating version 0 to 144 2024-04-22T18:40:25.395 INFO:tasks.rados.rados.0.smithi089.stdout:1424: oid 24 updating version 144 to 145 2024-04-22T18:40:25.395 INFO:tasks.rados.rados.0.smithi089.stdout:1424: oid 24 version 145 is already newer than 143 2024-04-22T18:40:25.395 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 145 (ObjNum 280 snap 0 seq_num 280) dirty exists 2024-04-22T18:40:25.395 INFO:tasks.rados.rados.0.smithi089.stdout:1424: left oid 24 (ObjNum 280 snap 0 seq_num 280) 2024-04-22T18:40:25.395 INFO:tasks.rados.rados.0.smithi089.stdout:1434: finishing set_chunk to oid 48 2024-04-22T18:40:25.395 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 48 set_chunk returned error code -95 offset: 855552 length: 722432 2024-04-22T18:40:25.395 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 0 (ObjNum 246 snap 0 seq_num 246) dirty exists 2024-04-22T18:40:25.395 INFO:tasks.rados.rados.0.smithi089.stdout:1430: finishing write tid 4 to smithi08937929-47 2024-04-22T18:40:25.395 INFO:tasks.rados.rados.0.smithi089.stdout:1430: oid 47 updating version 0 to 75 2024-04-22T18:40:25.395 INFO:tasks.rados.rados.0.smithi089.stdout:1430: oid 47 version 75 is already newer than 74 2024-04-22T18:40:25.395 INFO:tasks.rados.rados.0.smithi089.stdout:1430: oid 47 version 75 is already newer than 73 2024-04-22T18:40:25.395 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 75 (ObjNum 281 snap 0 seq_num 281) dirty exists 2024-04-22T18:40:25.396 INFO:tasks.rados.rados.0.smithi089.stdout:1430: left oid 47 (ObjNum 281 snap 0 seq_num 281) 2024-04-22T18:40:25.396 INFO:tasks.rados.rados.0.smithi089.stdout:1435: expect (ObjNum 158 snap 0 seq_num 158) 2024-04-22T18:40:25.399 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:25.399 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:25.407 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:25.408+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:25.408 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:25.508 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:25.509+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:25.538 INFO:tasks.rados.rados.0.smithi089.stdout:1425: got (0) Success 2024-04-22T18:40:25.539 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 100 (ObjNum 268 snap 0 seq_num 268) dirty exists 2024-04-22T18:40:25.539 INFO:tasks.rados.rados.0.smithi089.stdout:1436: finishing set_chunk to oid 11 2024-04-22T18:40:25.539 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 11 set_chunk returned error code -95 offset: 1923072 length: 995840 2024-04-22T18:40:25.539 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 0 (ObjNum 196 snap 0 seq_num 196) dirty exists 2024-04-22T18:40:25.539 INFO:tasks.rados.rados.0.smithi089.stdout:1437: finishing write tid 1 to smithi08937929-22 2024-04-22T18:40:25.539 INFO:tasks.rados.rados.0.smithi089.stdout:1424: done (15 left) 2024-04-22T18:40:25.539 INFO:tasks.rados.rados.0.smithi089.stdout:1425: done (14 left) 2024-04-22T18:40:25.539 INFO:tasks.rados.rados.0.smithi089.stdout:1427: done (13 left) 2024-04-22T18:40:25.539 INFO:tasks.rados.rados.0.smithi089.stdout:1428: done (12 left) 2024-04-22T18:40:25.539 INFO:tasks.rados.rados.0.smithi089.stdout:1429: done (11 left) 2024-04-22T18:40:25.539 INFO:tasks.rados.rados.0.smithi089.stdout:1430: done (10 left) 2024-04-22T18:40:25.539 INFO:tasks.rados.rados.0.smithi089.stdout:1433: done (9 left) 2024-04-22T18:40:25.539 INFO:tasks.rados.rados.0.smithi089.stdout:1434: done (8 left) 2024-04-22T18:40:25.540 INFO:tasks.rados.rados.0.smithi089.stdout:1435: done (7 left) 2024-04-22T18:40:25.540 INFO:tasks.rados.rados.0.smithi089.stdout:1436: done (6 left) 2024-04-22T18:40:25.540 INFO:tasks.rados.rados.0.smithi089.stdout:1441: tier_evict oid 18 2024-04-22T18:40:25.540 INFO:tasks.rados.rados.0.smithi089.stdout:1441: tier_evict oid 18 snap -1 2024-04-22T18:40:25.540 INFO:tasks.rados.rados.0.smithi089.stdout:1437: finishing write tid 2 to smithi08937929-22 2024-04-22T18:40:25.540 INFO:tasks.rados.rados.0.smithi089.stdout:1437: finishing write tid 3 to smithi08937929-22 2024-04-22T18:40:25.540 INFO:tasks.rados.rados.0.smithi089.stdout:1437: finishing write tid 4 to smithi08937929-22 2024-04-22T18:40:25.540 INFO:tasks.rados.rados.0.smithi089.stdout:1437: oid 22 updating version 0 to 103 2024-04-22T18:40:25.540 INFO:tasks.rados.rados.0.smithi089.stdout:1437: oid 22 version 103 is already newer than 102 2024-04-22T18:40:25.540 INFO:tasks.rados.rados.0.smithi089.stdout:1437: oid 22 version 103 is already newer than 101 2024-04-22T18:40:25.540 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 103 (ObjNum 282 snap 0 seq_num 282) dirty exists 2024-04-22T18:40:25.540 INFO:tasks.rados.rados.0.smithi089.stdout:1437: left oid 22 (ObjNum 282 snap 0 seq_num 282) 2024-04-22T18:40:25.540 INFO:tasks.rados.rados.0.smithi089.stdout:1438: expect (ObjNum 279 snap 0 seq_num 279) 2024-04-22T18:40:25.550 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:25.550 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:25.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:25.609 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:25.611+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:25.652 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:40:25.699 INFO:tasks.rados.rados.0.smithi089.stdout:1439: expect (ObjNum 149 snap 0 seq_num 149) 2024-04-22T18:40:25.700 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:25.700 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:25.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:25.709 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:25.710+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:25.763 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:40:25.810 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:25.811+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:25.810 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:25.811 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:25.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:25.850 INFO:tasks.rados.rados.0.smithi089.stdout:1440: finishing set_chunk to oid 13 2024-04-22T18:40:25.850 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 13 set_chunk returned error code -95 offset: 187392 length: 1030144 2024-04-22T18:40:25.851 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 0 (ObjNum 261 snap 0 seq_num 261) dirty exists 2024-04-22T18:40:25.851 INFO:tasks.rados.rados.0.smithi089.stdout:1441: got (22) Invalid argument 2024-04-22T18:40:25.851 INFO:tasks.rados.rados.0.smithi089.stdout:1437: done (6 left) 2024-04-22T18:40:25.851 INFO:tasks.rados.rados.0.smithi089.stdout:1438: done (5 left) 2024-04-22T18:40:25.851 INFO:tasks.rados.rados.0.smithi089.stdout:1439: done (4 left) 2024-04-22T18:40:25.851 INFO:tasks.rados.rados.0.smithi089.stdout:1440: done (3 left) 2024-04-22T18:40:25.851 INFO:tasks.rados.rados.0.smithi089.stdout:1441: done (2 left) 2024-04-22T18:40:25.851 INFO:tasks.rados.rados.0.smithi089.stdout:1442: set_chunk oid 13 target oid 2024-04-22T18:40:25.865 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:25.865 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:25.886 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:25.886 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:25.902 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:25.902 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:25.911 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:25.912+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:25.917 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:25.918 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:25.919 INFO:tasks.rados.rados.0.smithi089.stdout:1442: set_chunk oid 13 offset: 797696 length: 431104 target oid c67bb8d19b8aa01689196ce5540896599585d6db42fbb2125b5c866602359554 offset: 0 snap -1 2024-04-22T18:40:25.919 INFO:tasks.rados.rados.0.smithi089.stdout:1443: read oid 6 snap -1 2024-04-22T18:40:25.919 INFO:tasks.rados.rados.0.smithi089.stdout:1443: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:40:25.919 INFO:tasks.rados.rados.0.smithi089.stdout:1444: tier_promote oid 3 2024-04-22T18:40:25.919 INFO:tasks.rados.rados.0.smithi089.stdout:1445: read oid 17 snap -1 2024-04-22T18:40:25.919 INFO:tasks.rados.rados.0.smithi089.stdout:1445: expect (ObjNum 192 snap 0 seq_num 192) 2024-04-22T18:40:25.919 INFO:tasks.rados.rados.0.smithi089.stdout:1446: set_chunk oid 24 target oid 2024-04-22T18:40:25.936 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:25.936 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:25.952 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:25.953 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:25.962 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:25.963 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:25.969 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:25.969 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:25.972 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:25.986 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:25.987 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:26.012 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:26.013+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:26.077 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:26.077 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:26.077 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:26.077 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:26.086 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:26.127 INFO:tasks.rados.rados.0.smithi089.stdout:1446: set_chunk oid 24 offset: 1501696 length: 909824 target oid 1396ea6ada26bf96e33eac446369a898460615bd851d1e8b2aedde8ec740fe1c offset: 0 snap -1 2024-04-22T18:40:26.127 INFO:tasks.rados.rados.0.smithi089.stdout:1447: set_chunk oid 4 target oid 2024-04-22T18:40:26.128 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:26.129+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:26.188 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:26.188 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:26.188 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:26.188 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:26.188 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:26.197 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:26.238 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:26.239+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:26.315 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:26.315+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:26.341 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 91.885976526999997, 2024-04-22T18:40:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:40:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:40:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:40:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:40:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:40:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:40:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:40:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:40:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:40:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:40:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:40:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:40:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:40:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:40:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:40:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:40:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:40:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:40:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:40:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:40:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:40:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:40:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:40:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:315 4.b 4:d923883e:::smithi08937929-16:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:40:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 88.349513747000003, 2024-04-22T18:40:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40309447300000001, 2024-04-22T18:40:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 315 2024-04-22T18:40:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:40:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991819+0000", 2024-04-22T18:40:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991817+0000", 2024-04-22T18:40:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991825+0000", 2024-04-22T18:40:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7279999999999995e-06 2024-04-22T18:40:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991834+0000", 2024-04-22T18:40:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1980000000000007e-06 2024-04-22T18:40:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991841+0000", 2024-04-22T18:40:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0740000000000004e-06 2024-04-22T18:40:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991881+0000", 2024-04-22T18:40:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-04-22T18:40:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.991940+0000", 2024-04-22T18:40:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9382000000000002e-05 2024-04-22T18:40:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394250+0000", 2024-04-22T18:40:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40230947700000003 2024-04-22T18:40:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394752+0000", 2024-04-22T18:40:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050269699999999997 2024-04-22T18:40:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394865+0000", 2024-04-22T18:40:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011267700000000001 2024-04-22T18:40:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394892+0000", 2024-04-22T18:40:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7535999999999999e-05 2024-04-22T18:40:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.394914+0000", 2024-04-22T18:40:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1398000000000001e-05 2024-04-22T18:40:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:774 4.b 4:defe92fb:::smithi08937929-44:head [write 2778819~95145] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:40:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 83.193004071000004, 2024-04-22T18:40:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40475725000000001, 2024-04-22T18:40:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 774 2024-04-22T18:40:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:40:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148329+0000", 2024-04-22T18:40:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148328+0000", 2024-04-22T18:40:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148392+0000", 2024-04-22T18:40:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3323000000000003e-05 2024-04-22T18:40:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148398+0000", 2024-04-22T18:40:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4420000000000003e-06 2024-04-22T18:40:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148409+0000", 2024-04-22T18:40:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509999999999999e-05 2024-04-22T18:40:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251407+0000", 2024-04-22T18:40:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102998305 2024-04-22T18:40:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251500+0000", 2024-04-22T18:40:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3607e-05 2024-04-22T18:40:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.251648+0000", 2024-04-22T18:40:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014788099999999999 2024-04-22T18:40:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256149+0000", 2024-04-22T18:40:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045010980000000003 2024-04-22T18:40:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553016+0000", 2024-04-22T18:40:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29686620800000002 2024-04-22T18:40:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553054+0000", 2024-04-22T18:40:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8414999999999997e-05 2024-04-22T18:40:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.553086+0000", 2024-04-22T18:40:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2236000000000001e-05 2024-04-22T18:40:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 83.192919278999995, 2024-04-22T18:40:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:40:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:40:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:40:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:40:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:40:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:40:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:40:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:40:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:40:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:40:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:40:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:40:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:40:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:40:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:40:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:40:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:40:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:40:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:40:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:40:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:40:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:40:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:40:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 83.192900803000001, 2024-04-22T18:40:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:40:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:40:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:40:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:40:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:40:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:40:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:40:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:40:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:40:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:40:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:40:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:40:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:40:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:40:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:40:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:40:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:40:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:40:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1242 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-22T18:40:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:40:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.599655253999998, 2024-04-22T18:40:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40662891899999998, 2024-04-22T18:40:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1242 2024-04-22T18:40:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:40:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741678+0000", 2024-04-22T18:40:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741677+0000", 2024-04-22T18:40:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741680+0000", 2024-04-22T18:40:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8059999999999999e-06 2024-04-22T18:40:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741685+0000", 2024-04-22T18:40:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0780000000000003e-06 2024-04-22T18:40:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741690+0000", 2024-04-22T18:40:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4020000000000004e-06 2024-04-22T18:40:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741732+0000", 2024-04-22T18:40:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2110000000000002e-05 2024-04-22T18:40:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:16.741798+0000", 2024-04-22T18:40:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6432000000000004e-05 2024-04-22T18:40:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.148307+0000", 2024-04-22T18:40:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40650862100000001 2024-04-22T18:40:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1408 4.6 4:630bd8e5:::smithi08937929-45:head [write 2816389~693495] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:40:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:40:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 64.179685446999997, 2024-04-22T18:40:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408959452, 2024-04-22T18:40:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1408 2024-04-22T18:40:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:40:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161648+0000", 2024-04-22T18:40:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161647+0000", 2024-04-22T18:40:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161951+0000", 2024-04-22T18:40:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030389799999999998 2024-04-22T18:40:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161955+0000", 2024-04-22T18:40:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1989999999999999e-06 2024-04-22T18:40:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161958+0000", 2024-04-22T18:40:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2260000000000001e-06 2024-04-22T18:40:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.265867+0000", 2024-04-22T18:40:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103908927 2024-04-22T18:40:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.265960+0000", 2024-04-22T18:40:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3016999999999994e-05 2024-04-22T18:40:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.266124+0000", 2024-04-22T18:40:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016347700000000001 2024-04-22T18:40:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570491+0000", 2024-04-22T18:40:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30436694199999997 2024-04-22T18:40:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570502+0000", 2024-04-22T18:40:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1107999999999999e-05 2024-04-22T18:40:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570556+0000", 2024-04-22T18:40:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4388000000000002e-05 2024-04-22T18:40:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570607+0000", 2024-04-22T18:40:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0782999999999998e-05 2024-04-22T18:40:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1409 4.6 4:630bd8e5:::smithi08937929-45:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:40:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 64.179371743000004, 2024-04-22T18:40:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40880186200000002, 2024-04-22T18:40:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1409 2024-04-22T18:40:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161961+0000", 2024-04-22T18:40:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161964+0000", 2024-04-22T18:40:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5909999999999998e-06 2024-04-22T18:40:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161966+0000", 2024-04-22T18:40:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5979999999999999e-06 2024-04-22T18:40:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161969+0000", 2024-04-22T18:40:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7049999999999999e-06 2024-04-22T18:40:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269205+0000", 2024-04-22T18:40:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107236228 2024-04-22T18:40:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269320+0000", 2024-04-22T18:40:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114571 2024-04-22T18:40:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.269476+0000", 2024-04-22T18:40:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155928 2024-04-22T18:40:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570619+0000", 2024-04-22T18:40:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30114322700000001 2024-04-22T18:40:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570672+0000", 2024-04-22T18:40:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3232000000000001e-05 2024-04-22T18:40:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570724+0000", 2024-04-22T18:40:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2258e-05 2024-04-22T18:40:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570763+0000", 2024-04-22T18:40:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8927999999999997e-05 2024-04-22T18:40:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.387 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.387 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1410 4.6 4:630bd8e5:::smithi08937929-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-04-22T18:40:26.387 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:26.387 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 64.179362225999995, 2024-04-22T18:40:26.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40900504500000001, 2024-04-22T18:40:26.387 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.387 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:26.387 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.387 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.387 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.387 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1410 2024-04-22T18:40:26.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.388 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:26.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:26.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161971+0000", 2024-04-22T18:40:26.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161972+0000", 2024-04-22T18:40:26.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6419999999999999e-06 2024-04-22T18:40:26.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161976+0000", 2024-04-22T18:40:26.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9260000000000002e-06 2024-04-22T18:40:26.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.161979+0000", 2024-04-22T18:40:26.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9359999999999999e-06 2024-04-22T18:40:26.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570790+0000", 2024-04-22T18:40:26.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40881099199999998 2024-04-22T18:40:26.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570879+0000", 2024-04-22T18:40:26.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8895999999999997e-05 2024-04-22T18:40:26.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:22.570976+0000", 2024-04-22T18:40:26.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6923999999999996e-05 2024-04-22T18:40:26.391 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.391 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.391 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.391 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:26.391 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:26.391 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.429852490999998, 2024-04-22T18:40:26.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:40:26.391 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.392 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.392 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.392 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.392 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.392 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:40:26.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.392 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:26.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:26.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:40:26.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:26.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:40:26.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:40:26.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:40:26.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:40:26.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:40:26.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:40:26.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:40:26.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:40:26.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:40:26.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:40:26.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:26.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:40:26.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:40:26.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:40:26.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:40:26.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:40:26.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:40:26.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:40:26.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:40:26.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:40:26.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:40:26.397 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.397 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.397 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.397 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:26.397 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:26.397 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 49.900693165, 2024-04-22T18:40:26.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:40:26.397 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.397 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:26.397 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.397 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.397 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.397 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:40:26.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.398 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:26.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:26.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:26.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:40:26.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:40:26.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:40:26.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:40:26.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:40:26.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:40:26.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:40:26.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:40:26.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:26.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:40:26.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:40:26.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:40:26.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:40:26.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:40:26.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:40:26.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:40:26.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:40:26.402 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.402 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.402 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.402 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:26.402 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:26.402 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 49.722985770000001, 2024-04-22T18:40:26.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:40:26.402 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.402 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.402 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:26.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:26.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:40:26.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:26.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:40:26.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:40:26.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:40:26.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:40:26.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:40:26.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:40:26.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:40:26.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:40:26.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:40:26.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:40:26.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:26.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:40:26.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:40:26.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:40:26.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:40:26.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:40:26.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:40:26.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:40:26.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:40:26.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:40:26.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:40:26.407 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.407 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.407 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.407 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:26.408 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:26.408 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 49.722917498000001, 2024-04-22T18:40:26.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:40:26.408 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.408 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.408 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:26.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:26.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:40:26.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:40:26.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:40:26.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:40:26.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:40:26.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:40:26.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:40:26.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:40:26.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:40:26.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:40:26.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:40:26.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:26.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:40:26.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:40:26.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:40:26.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:40:26.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:40:26.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:40:26.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:40:26.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:40:26.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:40:26.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:26.412 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.412 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.413 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.413 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:26.413 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:26.413 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 49.722749657000001, 2024-04-22T18:40:26.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:40:26.413 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.413 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.413 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:26.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:26.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:26.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:40:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:40:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:40:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:40:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:40:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:40:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:40:26.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:40:26.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:40:26.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:40:26.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:26.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:40:26.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:40:26.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:40:26.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:40:26.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:40:26.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:40:26.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:40:26.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:40:26.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:40:26.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:40:26.418 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.418 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.418 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.418 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:26.418 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:26.418 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.757424182000001, 2024-04-22T18:40:26.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:40:26.418 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.418 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.419 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:26.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:26.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:40:26.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:26.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:40:26.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:40:26.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:40:26.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:40:26.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:40:26.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:40:26.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:40:26.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:40:26.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:40:26.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:40:26.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:26.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:40:26.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:40:26.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:40:26.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:40:26.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:40:26.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:40:26.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:40:26.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:40:26.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:40:26.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:40:26.423 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.423 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.423 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.423 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:26.424 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:26.424 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.657271995000002, 2024-04-22T18:40:26.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:40:26.424 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.424 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.424 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:26.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:26.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:40:26.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:26.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:40:26.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:40:26.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:40:26.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:40:26.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:40:26.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:40:26.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:40:26.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:40:26.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:40:26.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:40:26.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:26.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:40:26.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:40:26.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:40:26.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:40:26.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:40:26.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:40:26.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:40:26.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:40:26.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:40:26.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:40:26.428 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.429 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.429 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.429 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2385 4.13 4:c93c842e:::smithi08937929-25:head [write 1937856~527061] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:40:26.429 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.287418+0000", 2024-04-22T18:40:26.429 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.053914937000002, 2024-04-22T18:40:26.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39986265300000001, 2024-04-22T18:40:26.429 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.429 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.429 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.429 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.429 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.429 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2385 2024-04-22T18:40:26.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.430 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287418+0000", 2024-04-22T18:40:26.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287418+0000", 2024-04-22T18:40:26.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287417+0000", 2024-04-22T18:40:26.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:26.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287636+0000", 2024-04-22T18:40:26.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000219089 2024-04-22T18:40:26.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287654+0000", 2024-04-22T18:40:26.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8294000000000001e-05 2024-04-22T18:40:26.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.287661+0000", 2024-04-22T18:40:26.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7780000000000002e-06 2024-04-22T18:40:26.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.483858+0000", 2024-04-22T18:40:26.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19619666199999999 2024-04-22T18:40:26.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.483917+0000", 2024-04-22T18:40:26.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9454999999999999e-05 2024-04-22T18:40:26.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:26.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.483999+0000", 2024-04-22T18:40:26.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1359999999999994e-05 2024-04-22T18:40:26.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586037+0000", 2024-04-22T18:40:26.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102038217 2024-04-22T18:40:26.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687214+0000", 2024-04-22T18:40:26.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101177195 2024-04-22T18:40:26.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687253+0000", 2024-04-22T18:40:26.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9409999999999997e-05 2024-04-22T18:40:26.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.687281+0000", 2024-04-22T18:40:26.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7478e-05 2024-04-22T18:40:26.434 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.434 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.434 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.434 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:40:26.434 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:26.434 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.948407475000003, 2024-04-22T18:40:26.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:40:26.435 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.435 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:26.435 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.435 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.435 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.435 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:40:26.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.435 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:26.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:26.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:40:26.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:40:26.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:40:26.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:40:26.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:40:26.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:40:26.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:40:26.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:40:26.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:40:26.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:26.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:40:26.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:40:26.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:40:26.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:40:26.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:40:26.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:40:26.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:40:26.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:40:26.439 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.439 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.439 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.439 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:26.439 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:26.439 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.052369103, 2024-04-22T18:40:26.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:40:26.440 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.440 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:26.440 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.440 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.440 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.440 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:40:26.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.440 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:26.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:26.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:26.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:40:26.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:40:26.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:40:26.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:40:26.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:40:26.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:40:26.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:40:26.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:40:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:40:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:40:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:40:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:40:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.444 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:203 3.1 3:81736bc9:::096453d42e9dfceb4e0d7d7f6f19be374b0d28e8be156d4462bac4cc8cafbecd:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e83)", 2024-04-22T18:40:26.444 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:13.982753+0000", 2024-04-22T18:40:26.444 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.35857985, 2024-04-22T18:40:26.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40124371800000003, 2024-04-22T18:40:26.444 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.444 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.444 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982753+0000", 2024-04-22T18:40:26.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982753+0000", 2024-04-22T18:40:26.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982751+0000", 2024-04-22T18:40:26.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:26.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982760+0000", 2024-04-22T18:40:26.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6840000000000002e-06 2024-04-22T18:40:26.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982775+0000", 2024-04-22T18:40:26.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4831000000000001e-05 2024-04-22T18:40:26.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982783+0000", 2024-04-22T18:40:26.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0609999999999996e-06 2024-04-22T18:40:26.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982838+0000", 2024-04-22T18:40:26.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4922e-05 2024-04-22T18:40:26.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.982940+0000", 2024-04-22T18:40:26.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102082 2024-04-22T18:40:26.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:26.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.983117+0000", 2024-04-22T18:40:26.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017719900000000001 2024-04-22T18:40:26.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:13.983739+0000", 2024-04-22T18:40:26.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00062191700000000004 2024-04-22T18:40:26.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:14.383901+0000", 2024-04-22T18:40:26.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40016206100000001 2024-04-22T18:40:26.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:14.383954+0000", 2024-04-22T18:40:26.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2855000000000001e-05 2024-04-22T18:40:26.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:14.383997+0000", 2024-04-22T18:40:26.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3034000000000002e-05 2024-04-22T18:40:26.448 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.449 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.449 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.449 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.449 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.449 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:26.450 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:26.451 INFO:tasks.rados.rados.0.smithi089.stdout:1447: set_chunk oid 4 offset: 147968 length: 166912 target oid 9c07de0d206043a7f519e612e85f16ef41f44da502af032b2d5df5ab93a5a301 offset: 0 snap -1 2024-04-22T18:40:26.452 INFO:tasks.rados.rados.0.smithi089.stdout:1448: write oid 9 current snap is 0 2024-04-22T18:40:26.452 INFO:tasks.rados.rados.0.smithi089.stdout:1442: finishing set_chunk to oid 13 2024-04-22T18:40:26.452 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 13 set_chunk returned error code -95 offset: 797696 length: 431104 2024-04-22T18:40:26.452 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 0 (ObjNum 261 snap 0 seq_num 261) dirty exists 2024-04-22T18:40:26.452 INFO:tasks.rados.rados.0.smithi089.stdout:1448: seq_num 283 ranges {756381=400528,1910757=464751} 2024-04-22T18:40:26.452 INFO:tasks.rados.rados.0.smithi089.stdout:1448: writing smithi08937929-9 from 756381 to 1156909 tid 1 2024-04-22T18:40:26.452 INFO:tasks.rados.rados.0.smithi089.stdout:1448: writing smithi08937929-9 from 1910757 to 2375508 tid 2 2024-04-22T18:40:26.452 INFO:tasks.rados.rados.0.smithi089.stdout:1442: done (8 left) 2024-04-22T18:40:26.452 INFO:tasks.rados.rados.0.smithi089.stdout:1449: write (excl) oid 41 current snap is 0 2024-04-22T18:40:26.452 INFO:tasks.rados.rados.0.smithi089.stdout:1449: seq_num 284 ranges {773188=475777,1969239=724762} 2024-04-22T18:40:26.452 INFO:tasks.rados.rados.0.smithi089.stdout:1449: writing smithi08937929-41 from 773188 to 1248965 tid 1 2024-04-22T18:40:26.452 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:26.450+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:26.458 INFO:tasks.rados.rados.0.smithi089.stdout:1449: writing smithi08937929-41 from 1969239 to 2694001 tid 2 2024-04-22T18:40:26.458 INFO:tasks.rados.rados.0.smithi089.stdout:1450: tier_flush oid 13 2024-04-22T18:40:26.458 INFO:tasks.rados.rados.0.smithi089.stdout:1450: tier_flush oid 13 snap -1 2024-04-22T18:40:26.458 INFO:tasks.rados.rados.0.smithi089.stdout:1451: write (excl) oid 30 current snap is 0 2024-04-22T18:40:26.458 INFO:tasks.rados.rados.0.smithi089.stdout:1445: expect (ObjNum 192 snap 0 seq_num 192) 2024-04-22T18:40:26.551 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:26.552+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:26.564 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:26.564 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:26.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.564 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3316 4.11 4.c4de1ef1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:26.564 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.539840+0000", 2024-04-22T18:40:26.564 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.025013094, 2024-04-22T18:40:26.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.025028107000000001, 2024-04-22T18:40:26.564 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.564 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for peered", 2024-04-22T18:40:26.564 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.564 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.564 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.564 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3316 2024-04-22T18:40:26.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.565 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539840+0000", 2024-04-22T18:40:26.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539840+0000", 2024-04-22T18:40:26.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539839+0000", 2024-04-22T18:40:26.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540217+0000", 2024-04-22T18:40:26.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037753899999999998 2024-04-22T18:40:26.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540225+0000", 2024-04-22T18:40:26.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4630000000000001e-06 2024-04-22T18:40:26.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540231+0000", 2024-04-22T18:40:26.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0440000000000001e-06 2024-04-22T18:40:26.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540273+0000", 2024-04-22T18:40:26.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1480000000000003e-05 2024-04-22T18:40:26.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:26.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540282+0000", 2024-04-22T18:40:26.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.935e-06 2024-04-22T18:40:26.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.567 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.568 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3317 4.11 4.c4de1ef1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:26.568 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.540240+0000", 2024-04-22T18:40:26.568 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.024613236, 2024-04-22T18:40:26.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.024669712999999999, 2024-04-22T18:40:26.568 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.568 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for peered", 2024-04-22T18:40:26.568 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.568 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.568 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.568 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3317 2024-04-22T18:40:26.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.568 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540240+0000", 2024-04-22T18:40:26.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540240+0000", 2024-04-22T18:40:26.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540239+0000", 2024-04-22T18:40:26.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:26.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540243+0000", 2024-04-22T18:40:26.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4569999999999998e-06 2024-04-22T18:40:26.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540250+0000", 2024-04-22T18:40:26.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6440000000000003e-06 2024-04-22T18:40:26.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540256+0000", 2024-04-22T18:40:26.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8100000000000003e-06 2024-04-22T18:40:26.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540302+0000", 2024-04-22T18:40:26.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5859999999999998e-05 2024-04-22T18:40:26.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:26.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540309+0000", 2024-04-22T18:40:26.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3950000000000003e-06 2024-04-22T18:40:26.571 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.571 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.571 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.571 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3318 4.11 4.c4de1ef1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:26.572 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:26.572 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.024583043999999998, 2024-04-22T18:40:26.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.024673774999999998, 2024-04-22T18:40:26.572 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.572 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for peered", 2024-04-22T18:40:26.572 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.572 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.572 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.572 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3318 2024-04-22T18:40:26.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.572 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:26.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:26.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:26.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540276+0000", 2024-04-22T18:40:26.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.049e-06 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540282+0000", 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7050000000000002e-06 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540288+0000", 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2900000000000002e-06 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540326+0000", 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8818e-05 2024-04-22T18:40:26.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:26.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540330+0000", 2024-04-22T18:40:26.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6260000000000001e-06 2024-04-22T18:40:26.575 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.575 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.575 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.575 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:62 3.0 3.f0028d38 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:26.575 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.540344+0000", 2024-04-22T18:40:26.575 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.024509383999999999, 2024-04-22T18:40:26.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.024633946, 2024-04-22T18:40:26.576 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.576 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:26.576 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540344+0000", 2024-04-22T18:40:26.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540344+0000", 2024-04-22T18:40:26.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540343+0000", 2024-04-22T18:40:26.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540404+0000", 2024-04-22T18:40:26.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.067e-05 2024-04-22T18:40:26.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540413+0000", 2024-04-22T18:40:26.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0950000000000002e-06 2024-04-22T18:40:26.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540419+0000", 2024-04-22T18:40:26.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3790000000000003e-06 2024-04-22T18:40:26.578 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.578 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.578 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.578 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:66 3.0 3.fed3c168 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:26.578 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.540427+0000", 2024-04-22T18:40:26.578 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.024426388, 2024-04-22T18:40:26.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.024576348000000001, 2024-04-22T18:40:26.578 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540427+0000", 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540427+0000", 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540426+0000", 2024-04-22T18:40:26.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:26.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540576+0000", 2024-04-22T18:40:26.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015033399999999999 2024-04-22T18:40:26.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540583+0000", 2024-04-22T18:40:26.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4819999999999999e-06 2024-04-22T18:40:26.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540587+0000", 2024-04-22T18:40:26.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:328 3.0 3:0a121103:::9c07de0d206043a7f519e612e85f16ef41f44da502af032b2d5df5ab93a5a301:head [call cas.chunk_get_ref in=77b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.432877+0000", 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.13197642400000001, 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13215655100000001, 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.432877+0000", 2024-04-22T18:40:26.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.432877+0000", 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.432875+0000", 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.432882+0000", 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0119999999999999e-06 2024-04-22T18:40:26.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.432892+0000", 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7780000000000002e-06 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.432899+0000", 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.148e-06 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.432990+0000", 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0785000000000006e-05 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.433056+0000", 2024-04-22T18:40:26.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6285000000000007e-05 2024-04-22T18:40:26.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:26.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.433175+0000", 2024-04-22T18:40:26.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011879099999999999 2024-04-22T18:40:26.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.433639+0000", 2024-04-22T18:40:26.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046396600000000001 2024-04-22T18:40:26.584 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.584 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.585 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.585 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop_reply(osd.4.13:328 3.0 e95/63)", 2024-04-22T18:40:26.585 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.534083+0000", 2024-04-22T18:40:26.585 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.030769636999999999, 2024-04-22T18:40:26.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.030984332999999999, 2024-04-22T18:40:26.585 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.585 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:26.585 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.534083+0000", 2024-04-22T18:40:26.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.534083+0000", 2024-04-22T18:40:26.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.534081+0000", 2024-04-22T18:40:26.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:26.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.534092+0000", 2024-04-22T18:40:26.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0400999999999999e-05 2024-04-22T18:40:26.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.534108+0000", 2024-04-22T18:40:26.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6509000000000001e-05 2024-04-22T18:40:26.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.534118+0000", 2024-04-22T18:40:26.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0231000000000001e-05 2024-04-22T18:40:26.587 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.587 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.587 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.588 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3312 4.f 4:f0ed3d49:::smithi08937929-9:head [write 1910757~464751 in=464751b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:26.588 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.539349+0000", 2024-04-22T18:40:26.588 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.025503582, 2024-04-22T18:40:26.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.025745275000000001, 2024-04-22T18:40:26.588 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.588 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:40:26.588 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.588 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.588 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.588 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3312 2024-04-22T18:40:26.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.588 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539349+0000", 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539349+0000", 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539348+0000", 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539566+0000", 2024-04-22T18:40:26.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021830499999999999 2024-04-22T18:40:26.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539574+0000", 2024-04-22T18:40:26.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.221e-06 2024-04-22T18:40:26.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539578+0000", 2024-04-22T18:40:26.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6979999999999997e-06 2024-04-22T18:40:26.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540944+0000", 2024-04-22T18:40:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013658679999999999 2024-04-22T18:40:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.541041+0000", 2024-04-22T18:40:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6403000000000005e-05 2024-04-22T18:40:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.541180+0000", 2024-04-22T18:40:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000139648 2024-04-22T18:40:26.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.544462+0000", 2024-04-22T18:40:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032814620000000002 2024-04-22T18:40:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3313 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=62b,truncate 2756381] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.539583+0000", 2024-04-22T18:40:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.025270371, 2024-04-22T18:40:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.025553533999999999, 2024-04-22T18:40:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.592 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:40:26.593 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.593 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.593 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.593 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3313 2024-04-22T18:40:26.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.593 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539583+0000", 2024-04-22T18:40:26.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539583+0000", 2024-04-22T18:40:26.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539582+0000", 2024-04-22T18:40:26.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539586+0000", 2024-04-22T18:40:26.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3840000000000001e-06 2024-04-22T18:40:26.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539589+0000", 2024-04-22T18:40:26.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5559999999999999e-06 2024-04-22T18:40:26.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539592+0000", 2024-04-22T18:40:26.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:40:26.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.542281+0000", 2024-04-22T18:40:26.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026895690000000002 2024-04-22T18:40:26.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.542379+0000", 2024-04-22T18:40:26.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7141999999999999e-05 2024-04-22T18:40:26.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:26.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.542481+0000", 2024-04-22T18:40:26.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101902 2024-04-22T18:40:26.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.544659+0000", 2024-04-22T18:40:26.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002178583 2024-04-22T18:40:26.596 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.596 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.597 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.597 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3314 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:26.597 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.539595+0000", 2024-04-22T18:40:26.597 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.025257667000000001, 2024-04-22T18:40:26.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.025580289999999999, 2024-04-22T18:40:26.597 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.597 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for rw locks", 2024-04-22T18:40:26.597 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.597 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.597 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.597 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3314 2024-04-22T18:40:26.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.598 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539595+0000", 2024-04-22T18:40:26.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539595+0000", 2024-04-22T18:40:26.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539595+0000", 2024-04-22T18:40:26.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539597+0000", 2024-04-22T18:40:26.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.579e-06 2024-04-22T18:40:26.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539600+0000", 2024-04-22T18:40:26.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9450000000000002e-06 2024-04-22T18:40:26.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539603+0000", 2024-04-22T18:40:26.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9679999999999998e-06 2024-04-22T18:40:26.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.542710+0000", 2024-04-22T18:40:26.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031076630000000001 2024-04-22T18:40:26.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:26.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.542785+0000", 2024-04-22T18:40:26.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5012999999999995e-05 2024-04-22T18:40:26.600 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.600 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.600 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.601 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3315 4.11 4.c4de1ef1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:26.601 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.539605+0000", 2024-04-22T18:40:26.601 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.025248202000000001, 2024-04-22T18:40:26.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.025603135999999999, 2024-04-22T18:40:26.601 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.601 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for peered", 2024-04-22T18:40:26.601 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.601 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.601 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.601 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3315 2024-04-22T18:40:26.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.601 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539605+0000", 2024-04-22T18:40:26.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539605+0000", 2024-04-22T18:40:26.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539605+0000", 2024-04-22T18:40:26.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539823+0000", 2024-04-22T18:40:26.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021812000000000001 2024-04-22T18:40:26.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539828+0000", 2024-04-22T18:40:26.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1869999999999996e-06 2024-04-22T18:40:26.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539832+0000", 2024-04-22T18:40:26.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8130000000000002e-06 2024-04-22T18:40:26.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539886+0000", 2024-04-22T18:40:26.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.448e-05 2024-04-22T18:40:26.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:26.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539897+0000", 2024-04-22T18:40:26.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1032e-05 2024-04-22T18:40:26.604 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.604 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.604 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.604 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.604 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:26.604 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 11 2024-04-22T18:40:26.605 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:26.606 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:26.606 INFO:tasks.rados.rados.0.smithi089.stdout:1443: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:40:26.652 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:26.653+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:26.712 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:26.712 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:26.712 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:26.712 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:26.712 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:26.721 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:26.748 INFO:tasks.rados.rados.0.smithi089.stdout:1444: got (0) Success 2024-04-22T18:40:26.748 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 68 (ObjNum 259 snap 0 seq_num 259) dirty exists 2024-04-22T18:40:26.749 INFO:tasks.rados.rados.0.smithi089.stdout:1446: finishing set_chunk to oid 24 2024-04-22T18:40:26.749 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 145 (ObjNum 280 snap 0 seq_num 280) dirty exists 2024-04-22T18:40:26.749 INFO:tasks.rados.rados.0.smithi089.stdout:1448: finishing write tid 1 to smithi08937929-9 2024-04-22T18:40:26.749 INFO:tasks.rados.rados.0.smithi089.stdout:1448: finishing write tid 2 to smithi08937929-9 2024-04-22T18:40:26.749 INFO:tasks.rados.rados.0.smithi089.stdout:1451: seq_num 285 ranges {789995=551026,2027721=468420} 2024-04-22T18:40:26.750 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 6, 7, 2] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:40:26.750 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:40:26.750 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 3, 4, 6, 7, 2] 2024-04-22T18:40:26.750 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-04-22T18:40:26.752 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:26.753+0000 7f3cfee88640 -1 osd.6 95 handle_osd_map injecting map crc failure 2024-04-22T18:40:26.753 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:26.753+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:26.756 INFO:tasks.rados.rados.0.smithi089.stdout:1451: writing smithi08937929-30 from 789995 to 1341021 tid 1 2024-04-22T18:40:26.763 INFO:tasks.rados.rados.0.smithi089.stdout:1451: writing smithi08937929-30 from 2027721 to 2496141 tid 2 2024-04-22T18:40:26.763 INFO:tasks.rados.rados.0.smithi089.stdout:1443: done (10 left) 2024-04-22T18:40:26.763 INFO:tasks.rados.rados.0.smithi089.stdout:1444: done (9 left) 2024-04-22T18:40:26.763 INFO:tasks.rados.rados.0.smithi089.stdout:1445: done (8 left) 2024-04-22T18:40:26.763 INFO:tasks.rados.rados.0.smithi089.stdout:1446: done (7 left) 2024-04-22T18:40:26.763 INFO:tasks.rados.rados.0.smithi089.stdout:1452: set_chunk oid 14 target oid 2024-04-22T18:40:26.829 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:9 4.f 4:f2c48a0e:::smithi08937929-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.713084309999999, 2024-04-22T18:40:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49290714600000002, 2024-04-22T18:40:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.830 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 9 2024-04-22T18:40:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116773+0000", 2024-04-22T18:40:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117024+0000", 2024-04-22T18:40:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025191299999999999 2024-04-22T18:40:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117036+0000", 2024-04-22T18:40:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1595e-05 2024-04-22T18:40:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117046+0000", 2024-04-22T18:40:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0456e-05 2024-04-22T18:40:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414324+0000", 2024-04-22T18:40:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297277603 2024-04-22T18:40:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414404+0000", 2024-04-22T18:40:26.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9738000000000001e-05 2024-04-22T18:40:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414494+0000", 2024-04-22T18:40:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9733000000000003e-05 2024-04-22T18:40:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518194+0000", 2024-04-22T18:40:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10370061 2024-04-22T18:40:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609618+0000", 2024-04-22T18:40:26.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091423360999999995 2024-04-22T18:40:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609655+0000", 2024-04-22T18:40:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7580000000000003e-05 2024-04-22T18:40:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609681+0000", 2024-04-22T18:40:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5908e-05 2024-04-22T18:40:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:10 4.f 4:f2c48a0e:::smithi08937929-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.707232050000002, 2024-04-22T18:40:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48712681400000002, 2024-04-22T18:40:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 10 2024-04-22T18:40:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122625+0000", 2024-04-22T18:40:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122828+0000", 2024-04-22T18:40:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020359499999999999 2024-04-22T18:40:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122840+0000", 2024-04-22T18:40:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1239e-05 2024-04-22T18:40:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122845+0000", 2024-04-22T18:40:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2719999999999997e-06 2024-04-22T18:40:26.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415105+0000", 2024-04-22T18:40:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29225972099999997 2024-04-22T18:40:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415169+0000", 2024-04-22T18:40:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4170000000000004e-05 2024-04-22T18:40:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415257+0000", 2024-04-22T18:40:26.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8331999999999994e-05 2024-04-22T18:40:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518199+0000", 2024-04-22T18:40:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10294174 2024-04-22T18:40:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609707+0000", 2024-04-22T18:40:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091508408999999999 2024-04-22T18:40:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609729+0000", 2024-04-22T18:40:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1917e-05 2024-04-22T18:40:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609753+0000", 2024-04-22T18:40:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3776000000000001e-05 2024-04-22T18:40:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:11 4.f 4:f2c48a0e:::smithi08937929-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:26.841 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.707008677000005, 2024-04-22T18:40:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49452214500000002, 2024-04-22T18:40:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 11 2024-04-22T18:40:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:26.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122849+0000", 2024-04-22T18:40:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122853+0000", 2024-04-22T18:40:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9029999999999997e-06 2024-04-22T18:40:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122856+0000", 2024-04-22T18:40:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.01e-06 2024-04-22T18:40:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122859+0000", 2024-04-22T18:40:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415755+0000", 2024-04-22T18:40:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29289606800000001 2024-04-22T18:40:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415808+0000", 2024-04-22T18:40:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3415999999999998e-05 2024-04-22T18:40:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415888+0000", 2024-04-22T18:40:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9061999999999996e-05 2024-04-22T18:40:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518203+0000", 2024-04-22T18:40:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102315533 2024-04-22T18:40:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617164+0000", 2024-04-22T18:40:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098960548999999995 2024-04-22T18:40:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617241+0000", 2024-04-22T18:40:26.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7292000000000005e-05 2024-04-22T18:40:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617372+0000", 2024-04-22T18:40:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013075399999999999 2024-04-22T18:40:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:12 4.f 4:f2c48a0e:::smithi08937929-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.706996813000003, 2024-04-22T18:40:26.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49473910999999998, 2024-04-22T18:40:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 12 2024-04-22T18:40:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122863+0000", 2024-04-22T18:40:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.863e-06 2024-04-22T18:40:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122866+0000", 2024-04-22T18:40:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:40:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122869+0000", 2024-04-22T18:40:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5950000000000001e-06 2024-04-22T18:40:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416008+0000", 2024-04-22T18:40:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29313975199999998 2024-04-22T18:40:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416048+0000", 2024-04-22T18:40:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9888000000000002e-05 2024-04-22T18:40:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617447+0000", 2024-04-22T18:40:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20139881400000001 2024-04-22T18:40:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617537+0000", 2024-04-22T18:40:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9606999999999998e-05 2024-04-22T18:40:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617600+0000", 2024-04-22T18:40:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3884e-05 2024-04-22T18:40:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:33 4.f 4:f0ed3d49:::smithi08937929-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.643056611999995, 2024-04-22T18:40:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62359692099999997, 2024-04-22T18:40:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.852 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 33 2024-04-22T18:40:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:26.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186799+0000", 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187103+0000", 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030354900000000002 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187117+0000", 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4578e-05 2024-04-22T18:40:26.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187129+0000", 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1421e-05 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516134+0000", 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32900553900000001 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516215+0000", 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0537999999999994e-05 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516302+0000", 2024-04-22T18:40:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6601000000000007e-05 2024-04-22T18:40:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518207+0000", 2024-04-22T18:40:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019058370000000001 2024-04-22T18:40:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810302+0000", 2024-04-22T18:40:26.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29209509700000003 2024-04-22T18:40:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810348+0000", 2024-04-22T18:40:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5992999999999998e-05 2024-04-22T18:40:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810398+0000", 2024-04-22T18:40:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0046000000000002e-05 2024-04-22T18:40:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:34 4.f 4:f0ed3d49:::smithi08937929-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.638454600000003, 2024-04-22T18:40:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.61954655800000002, 2024-04-22T18:40:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34 2024-04-22T18:40:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191401+0000", 2024-04-22T18:40:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191503+0000", 2024-04-22T18:40:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010190400000000001 2024-04-22T18:40:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191517+0000", 2024-04-22T18:40:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3267e-05 2024-04-22T18:40:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191523+0000", 2024-04-22T18:40:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-04-22T18:40:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517062+0000", 2024-04-22T18:40:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32553894300000003 2024-04-22T18:40:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517141+0000", 2024-04-22T18:40:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9667999999999997e-05 2024-04-22T18:40:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517267+0000", 2024-04-22T18:40:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000125494 2024-04-22T18:40:26.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518213+0000", 2024-04-22T18:40:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000946275 2024-04-22T18:40:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810901+0000", 2024-04-22T18:40:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29268821699999997 2024-04-22T18:40:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810929+0000", 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7645999999999999e-05 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810950+0000", 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1333999999999999e-05 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:26.863 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.638331632000003, 2024-04-22T18:40:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:40:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:40:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:40:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:40:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:40:26.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:40:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:40:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:40:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:40:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:40:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:40:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:40:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:40:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:40:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:40:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:40:26.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.868 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.638320557, 2024-04-22T18:40:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:40:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:40:26.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:40:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:40:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:40:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:40:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:40:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:40:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:40:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:40:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:40:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:40:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:40:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:40:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:40:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:40:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:40:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:40:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:40:26.873 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 89.036951063000004, 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.874 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:40:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:40:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:40:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:40:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:40:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:40:26.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:40:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:40:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:40:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:40:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1262 4.7 4:e117a562:::smithi08937929-33:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.376807886999998, 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40488836700000003, 2024-04-22T18:40:26.879 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1262 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453050+0000", 2024-04-22T18:40:26.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453048+0000", 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453058+0000", 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3940000000000002e-06 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453069+0000", 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1117999999999999e-05 2024-04-22T18:40:26.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453075+0000", 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0909999999999998e-06 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453115+0000", 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0119e-05 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.453119+0000", 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3610000000000003e-06 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454814+0000", 2024-04-22T18:40:26.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00169436 2024-04-22T18:40:26.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.454861+0000", 2024-04-22T18:40:26.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7184000000000001e-05 2024-04-22T18:40:26.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:26.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.756925+0000", 2024-04-22T18:40:26.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30206361799999998 2024-04-22T18:40:26.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.757577+0000", 2024-04-22T18:40:26.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065199599999999996 2024-04-22T18:40:26.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857846+0000", 2024-04-22T18:40:26.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100269369 2024-04-22T18:40:26.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857896+0000", 2024-04-22T18:40:26.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9607000000000001e-05 2024-04-22T18:40:26.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:17.857939+0000", 2024-04-22T18:40:26.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3149999999999999e-05 2024-04-22T18:40:26.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.885 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.885 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2205 4.2 4:4fb26dd8:::smithi08937929-10:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:26.885 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:40:26.885 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.439262786, 2024-04-22T18:40:26.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.413791136, 2024-04-22T18:40:26.885 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.886 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:26.886 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.886 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.886 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.886 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2205 2024-04-22T18:40:26.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.886 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:40:26.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:40:26.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390595+0000", 2024-04-22T18:40:26.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390598+0000", 2024-04-22T18:40:26.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7630000000000001e-06 2024-04-22T18:40:26.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390603+0000", 2024-04-22T18:40:26.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.293e-06 2024-04-22T18:40:26.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390608+0000", 2024-04-22T18:40:26.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8790000000000002e-06 2024-04-22T18:40:26.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390653+0000", 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5661999999999998e-05 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.390657+0000", 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9709999999999998e-06 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.592250+0000", 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20159280600000001 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.592322+0000", 2024-04-22T18:40:26.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2171999999999999e-05 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.804373+0000", 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.212050297 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:45.804387+0000", 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3859999999999999e-05 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.890 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:7 3.0 3:1554f608:::d614c1f2f74bb29087660bd6e888eafe46b1a7ed3cbdede85b36de771c7f0d66:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.657673537000001, 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48116529899999999, 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172184+0000", 2024-04-22T18:40:26.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172349+0000", 2024-04-22T18:40:26.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000165074 2024-04-22T18:40:26.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172355+0000", 2024-04-22T18:40:26.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8900000000000004e-06 2024-04-22T18:40:26.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172359+0000", 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.569e-06 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271561+0000", 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099202569000000004 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271659+0000", 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8171000000000006e-05 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271775+0000", 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000116137 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.272327+0000", 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00055106999999999997 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653290+0000", 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.38096334500000001 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653326+0000", 2024-04-22T18:40:26.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6168000000000003e-05 2024-04-22T18:40:26.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653350+0000", 2024-04-22T18:40:26.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3921999999999999e-05 2024-04-22T18:40:26.895 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.895 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.895 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.895 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2709 4.2 4:409e6456:::smithi08937929-11:head [sparse-read 0~2843709] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:26.895 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.553801+0000", 2024-04-22T18:40:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.276056812, 2024-04-22T18:40:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40417458899999997, 2024-04-22T18:40:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2709 2024-04-22T18:40:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553801+0000", 2024-04-22T18:40:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553801+0000", 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553801+0000", 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553803+0000", 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4899999999999999e-06 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553808+0000", 2024-04-22T18:40:26.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9169999999999997e-06 2024-04-22T18:40:26.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553812+0000", 2024-04-22T18:40:26.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1470000000000003e-06 2024-04-22T18:40:26.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655577+0000", 2024-04-22T18:40:26.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10176484299999999 2024-04-22T18:40:26.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655684+0000", 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107221 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.957688+0000", 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30200392199999998 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.957737+0000", 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8362999999999998e-05 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.957976+0000", 2024-04-22T18:40:26.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000239211 2024-04-22T18:40:26.900 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.900 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.900 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.900 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2710 4.2 4:409e6456:::smithi08937929-11:head [read 0~3294172,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:26.900 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.553816+0000", 2024-04-22T18:40:26.900 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.276042200999999, 2024-04-22T18:40:26.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40483363500000002, 2024-04-22T18:40:26.900 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.900 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2710 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553816+0000", 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553816+0000", 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553815+0000", 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553819+0000", 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3220000000000001e-06 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553824+0000", 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8879999999999996e-06 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.553828+0000", 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.245e-06 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655717+0000", 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10188926500000001 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.655784+0000", 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6308000000000003e-05 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.957997+0000", 2024-04-22T18:40:26.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.302213385 2024-04-22T18:40:26.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.958037+0000", 2024-04-22T18:40:26.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470999999999999e-05 2024-04-22T18:40:26.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.958650+0000", 2024-04-22T18:40:26.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061223400000000002 2024-04-22T18:40:26.904 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.904 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.904 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2845 4.7 4:e117a562:::smithi08937929-33:head [setxattr _header (56) in=7b,truncate 2033680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:08.157995+0000", 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.671863489, 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.416467168, 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2845 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.157995+0000", 2024-04-22T18:40:26.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.157995+0000", 2024-04-22T18:40:26.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.157994+0000", 2024-04-22T18:40:26.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.157998+0000", 2024-04-22T18:40:26.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.631e-06 2024-04-22T18:40:26.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158001+0000", 2024-04-22T18:40:26.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:40:26.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158004+0000", 2024-04-22T18:40:26.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9610000000000001e-06 2024-04-22T18:40:26.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.354298+0000", 2024-04-22T18:40:26.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19629424500000001 2024-04-22T18:40:26.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.354367+0000", 2024-04-22T18:40:26.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9016999999999993e-05 2024-04-22T18:40:26.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:26.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.354470+0000", 2024-04-22T18:40:26.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103365 2024-04-22T18:40:26.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:26.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.455161+0000", 2024-04-22T18:40:26.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100690442 2024-04-22T18:40:26.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:26.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574379+0000", 2024-04-22T18:40:26.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.119217818 2024-04-22T18:40:26.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:26.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574406+0000", 2024-04-22T18:40:26.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7712000000000002e-05 2024-04-22T18:40:26.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574462+0000", 2024-04-22T18:40:26.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5405000000000002e-05 2024-04-22T18:40:26.910 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.910 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.910 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.910 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2846 4.7 4:e117a562:::smithi08937929-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:08.158020+0000", 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.671838524000002, 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41656018099999997, 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2846 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158020+0000", 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158020+0000", 2024-04-22T18:40:26.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158019+0000", 2024-04-22T18:40:26.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158021+0000", 2024-04-22T18:40:26.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.928e-06 2024-04-22T18:40:26.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158026+0000", 2024-04-22T18:40:26.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5029999999999999e-06 2024-04-22T18:40:26.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.158029+0000", 2024-04-22T18:40:26.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5889999999999999e-06 2024-04-22T18:40:26.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.455080+0000", 2024-04-22T18:40:26.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29705077499999999 2024-04-22T18:40:26.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:26.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.455142+0000", 2024-04-22T18:40:26.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1982000000000004e-05 2024-04-22T18:40:26.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574482+0000", 2024-04-22T18:40:26.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.119340448 2024-04-22T18:40:26.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:26.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574529+0000", 2024-04-22T18:40:26.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6570999999999998e-05 2024-04-22T18:40:26.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:08.574580+0000", 2024-04-22T18:40:26.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0787e-05 2024-04-22T18:40:26.915 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.915 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.915 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.915 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3193 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:26.915 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:40:26.915 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.7871122679999996, 2024-04-22T18:40:26.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69912102700000001, 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3193 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042744+0000", 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043013+0000", 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026906 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043028+0000", 2024-04-22T18:40:26.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5359999999999999e-05 2024-04-22T18:40:26.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043036+0000", 2024-04-22T18:40:26.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8420000000000005e-06 2024-04-22T18:40:26.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043081+0000", 2024-04-22T18:40:26.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5019999999999999e-05 2024-04-22T18:40:26.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:26.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043087+0000", 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5420000000000002e-06 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741856+0000", 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69876864500000002 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741867+0000", 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.084e-05 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3194 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.7798622880000003, 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69190137100000004, 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3194 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:40:26.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049991+0000", 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999957 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050284+0000", 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029296699999999999 2024-04-22T18:40:26.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050304+0000", 2024-04-22T18:40:26.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.969e-05 2024-04-22T18:40:26.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050314+0000", 2024-04-22T18:40:26.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0268999999999999e-05 2024-04-22T18:40:26.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150531+0000", 2024-04-22T18:40:26.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100216868 2024-04-22T18:40:26.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:26.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150545+0000", 2024-04-22T18:40:26.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4063000000000001e-05 2024-04-22T18:40:26.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741890+0000", 2024-04-22T18:40:26.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59134479100000004 2024-04-22T18:40:26.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741897+0000", 2024-04-22T18:40:26.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1189999999999999e-06 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3195 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.7795298480000001, 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69159474200000004, 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3195 2024-04-22T18:40:26.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.925 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:26.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:26.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:26.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:26.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050333+0000", 2024-04-22T18:40:26.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0289999999999996e-06 2024-04-22T18:40:26.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050341+0000", 2024-04-22T18:40:26.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0050000000000004e-06 2024-04-22T18:40:26.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050346+0000", 2024-04-22T18:40:26.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8329999999999999e-06 2024-04-22T18:40:26.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150628+0000", 2024-04-22T18:40:26.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100281548 2024-04-22T18:40:26.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:26.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150672+0000", 2024-04-22T18:40:26.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3875999999999998e-05 2024-04-22T18:40:26.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741916+0000", 2024-04-22T18:40:26.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59124448799999996 2024-04-22T18:40:26.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741923+0000", 2024-04-22T18:40:26.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7689999999999999e-06 2024-04-22T18:40:26.928 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.928 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.928 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.928 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3196 4.11 4.9f8e4db1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:26.928 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:26.928 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.7794997290000003, 2024-04-22T18:40:26.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69158947400000004, 2024-04-22T18:40:26.929 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:26.929 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:26.929 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:26.929 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:26.929 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:26.929 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3196 2024-04-22T18:40:26.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.929 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:26.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:26.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:26.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:26.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:26.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:26.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:26.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:26.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:26.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:26.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050361+0000", 2024-04-22T18:40:26.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2040000000000002e-06 2024-04-22T18:40:26.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:26.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050367+0000", 2024-04-22T18:40:26.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3119999999999999e-06 2024-04-22T18:40:26.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:26.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050372+0000", 2024-04-22T18:40:26.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0869999999999997e-06 2024-04-22T18:40:26.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150688+0000", 2024-04-22T18:40:26.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100315381 2024-04-22T18:40:26.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:26.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150691+0000", 2024-04-22T18:40:26.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8899999999999999e-06 2024-04-22T18:40:26.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:26.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741941+0000", 2024-04-22T18:40:26.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59125044900000001 2024-04-22T18:40:26.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:26.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:26.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:26.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741948+0000", 2024-04-22T18:40:26.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.827e-06 2024-04-22T18:40:26.932 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.932 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.933 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.933 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:26.933 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:26.933 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:26.934 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:26.934 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:26.935+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:27.023 INFO:tasks.rados.rados.0.smithi089.stdout:1452: set_chunk oid 14 offset: 1235456 length: 314880 target oid ef2800a47005b9c97fa7413799b93f3cbf5d4cfe7c1e1d6a698b85eefad05754 offset: 0 snap -1 2024-04-22T18:40:27.023 INFO:tasks.rados.rados.0.smithi089.stdout:1453: read oid 37 snap -1 2024-04-22T18:40:27.023 INFO:tasks.rados.rados.0.smithi089.stdout:1453: expect (ObjNum 267 snap 0 seq_num 267) 2024-04-22T18:40:27.023 INFO:tasks.rados.rados.0.smithi089.stdout:1454: tier_promote oid 49 2024-04-22T18:40:27.023 INFO:tasks.rados.rados.0.smithi089.stdout:1448: finishing write tid 3 to smithi08937929-9 2024-04-22T18:40:27.023 INFO:tasks.rados.rados.0.smithi089.stdout:1448: finishing write tid 4 to smithi08937929-9 2024-04-22T18:40:27.023 INFO:tasks.rados.rados.0.smithi089.stdout:1448: oid 9 updating version 0 to 71 2024-04-22T18:40:27.023 INFO:tasks.rados.rados.0.smithi089.stdout:1448: oid 9 version 71 is already newer than 70 2024-04-22T18:40:27.024 INFO:tasks.rados.rados.0.smithi089.stdout:1448: oid 9 updating version 71 to 72 2024-04-22T18:40:27.024 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 72 (ObjNum 283 snap 0 seq_num 283) dirty exists 2024-04-22T18:40:27.024 INFO:tasks.rados.rados.0.smithi089.stdout:1448: left oid 9 (ObjNum 283 snap 0 seq_num 283) 2024-04-22T18:40:27.024 INFO:tasks.rados.rados.0.smithi089.stdout:1447: finishing set_chunk to oid 4 2024-04-22T18:40:27.024 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 103 (ObjNum 269 snap 0 seq_num 269) dirty exists 2024-04-22T18:40:27.024 INFO:tasks.rados.rados.0.smithi089.stdout:1451: finishing write tid 1 to smithi08937929-30 2024-04-22T18:40:27.024 INFO:tasks.rados.rados.0.smithi089.stdout:1451: finishing write tid 2 to smithi08937929-30 2024-04-22T18:40:27.024 INFO:tasks.rados.rados.0.smithi089.stdout:1451: finishing write tid 3 to smithi08937929-30 2024-04-22T18:40:27.024 INFO:tasks.rados.rados.0.smithi089.stdout:1451: finishing write tid 4 to smithi08937929-30 2024-04-22T18:40:27.024 INFO:tasks.rados.rados.0.smithi089.stdout:1451: oid 30 updating version 0 to 45 2024-04-22T18:40:27.024 INFO:tasks.rados.rados.0.smithi089.stdout:1451: oid 30 version 45 is already newer than 44 2024-04-22T18:40:27.024 INFO:tasks.rados.rados.0.smithi089.stdout:1451: oid 30 updating version 45 to 46 2024-04-22T18:40:27.024 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 46 (ObjNum 285 snap 0 seq_num 285) dirty exists 2024-04-22T18:40:27.024 INFO:tasks.rados.rados.0.smithi089.stdout:1451: left oid 30 (ObjNum 285 snap 0 seq_num 285) 2024-04-22T18:40:27.024 INFO:tasks.rados.rados.0.smithi089.stdout:1450: got (0) Success 2024-04-22T18:40:27.025 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 0 (ObjNum 261 snap 0 seq_num 261) dirty exists 2024-04-22T18:40:27.025 INFO:tasks.rados.rados.0.smithi089.stdout:1447: done (9 left) 2024-04-22T18:40:27.025 INFO:tasks.rados.rados.0.smithi089.stdout:1448: done (8 left) 2024-04-22T18:40:27.025 INFO:tasks.rados.rados.0.smithi089.stdout:1450: done (7 left) 2024-04-22T18:40:27.025 INFO:tasks.rados.rados.0.smithi089.stdout:1451: done (6 left) 2024-04-22T18:40:27.025 INFO:tasks.rados.rados.0.smithi089.stdout:1455: read oid 43 snap -1 2024-04-22T18:40:27.025 INFO:tasks.rados.rados.0.smithi089.stdout:1455: expect (ObjNum 226 snap 0 seq_num 226) 2024-04-22T18:40:27.025 INFO:tasks.rados.rados.0.smithi089.stdout:1456: read oid 42 snap -1 2024-04-22T18:40:27.025 INFO:tasks.rados.rados.0.smithi089.stdout:1456: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:27.025 INFO:tasks.rados.rados.0.smithi089.stdout:1457: tier_flush oid 19 2024-04-22T18:40:27.025 INFO:tasks.rados.rados.0.smithi089.stdout:1457: tier_flush oid 19 snap -1 2024-04-22T18:40:27.025 INFO:tasks.rados.rados.0.smithi089.stdout:1458: read oid 22 snap -1 2024-04-22T18:40:27.025 INFO:tasks.rados.rados.0.smithi089.stdout:1458: expect (ObjNum 282 snap 0 seq_num 282) 2024-04-22T18:40:27.025 INFO:tasks.rados.rados.0.smithi089.stdout:1459: write oid 48 current snap is 0 2024-04-22T18:40:27.025 INFO:tasks.rados.rados.0.smithi089.stdout:1459: seq_num 286 ranges {406802=626275,1686203=728431} 2024-04-22T18:40:27.028 INFO:tasks.rados.rados.0.smithi089.stdout:1459: writing smithi08937929-48 from 406802 to 1033077 tid 1 2024-04-22T18:40:27.034 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:27.035+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:27.036 INFO:tasks.rados.rados.0.smithi089.stdout:1459: writing smithi08937929-48 from 1686203 to 2414634 tid 2 2024-04-22T18:40:27.036 INFO:tasks.rados.rados.0.smithi089.stdout:1460: set_chunk oid 11 target oid 2024-04-22T18:40:27.051 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-22T18:40:27.051+0000 7f9b4bbc8640 -1 mon.a@0(leader).osd e96 definitely_dead 0 2024-04-22T18:40:27.080 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:27.080 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:27.080 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:27.081 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:27.089 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:27.135 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:27.135+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:27.193 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:27.194 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:27.194 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:27.194 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:27.194 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:27.202 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:27.244 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:27.244+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:27.336 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:27.337+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:27.346 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:27.346 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:27.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:27.346 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:27.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.347 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:73 3.0 e34/19 3:061a7a63:::392f44ec26aa22e85be5f08273a7e45656f3bf4a4854038391a88e880ec01077:head v 34'31, mlcod=31'29)", 2024-04-22T18:40:27.347 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:27.347 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.889080509999999, 2024-04-22T18:40:27.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103503311, 2024-04-22T18:40:27.347 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.347 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.347 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:27.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:27.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:27.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458193+0000", 2024-04-22T18:40:27.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026628900000000001 2024-04-22T18:40:27.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458208+0000", 2024-04-22T18:40:27.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4780000000000001e-05 2024-04-22T18:40:27.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458217+0000", 2024-04-22T18:40:27.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4909999999999994e-06 2024-04-22T18:40:27.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459162+0000", 2024-04-22T18:40:27.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094440500000000005 2024-04-22T18:40:27.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459185+0000", 2024-04-22T18:40:27.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3685000000000001e-05 2024-04-22T18:40:27.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561394+0000", 2024-04-22T18:40:27.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10220874100000001 2024-04-22T18:40:27.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561430+0000", 2024-04-22T18:40:27.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6304999999999999e-05 2024-04-22T18:40:27.351 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.351 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.351 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.351 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:75 3.0 e34/19 3:18418e0f:::b7c35f165c374d5e075dde9ac2607e47ace3b0a1e2343ec6efb171380f725eea:head v 34'32, mlcod=31'29)", 2024-04-22T18:40:27.351 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:27.351 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.888067237000001, 2024-04-22T18:40:27.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104798078, 2024-04-22T18:40:27.351 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.351 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.351 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:27.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:27.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458939+0000", 2024-04-22T18:40:27.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:27.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459092+0000", 2024-04-22T18:40:27.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153838 2024-04-22T18:40:27.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459107+0000", 2024-04-22T18:40:27.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4703e-05 2024-04-22T18:40:27.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459115+0000", 2024-04-22T18:40:27.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-04-22T18:40:27.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560093+0000", 2024-04-22T18:40:27.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10097840499999999 2024-04-22T18:40:27.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560109+0000", 2024-04-22T18:40:27.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5631e-05 2024-04-22T18:40:27.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563706+0000", 2024-04-22T18:40:27.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035972059999999999 2024-04-22T18:40:27.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563738+0000", 2024-04-22T18:40:27.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2184000000000002e-05 2024-04-22T18:40:27.355 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.355 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.355 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.355 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:77 3.0 e34/19 3:28387c9f:::b6d7840f02d561da04e398fbf94d91b85e9066b426e48baf3b11dd4a4f6096c0:head v 34'33, mlcod=31'29)", 2024-04-22T18:40:27.355 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:27.356 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.887117157000006, 2024-04-22T18:40:27.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104186079, 2024-04-22T18:40:27.356 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.356 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.356 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:27.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:27.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459889+0000", 2024-04-22T18:40:27.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:27.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459963+0000", 2024-04-22T18:40:27.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3732999999999994e-05 2024-04-22T18:40:27.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459973+0000", 2024-04-22T18:40:27.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0285999999999999e-05 2024-04-22T18:40:27.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459987+0000", 2024-04-22T18:40:27.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3675999999999999e-05 2024-04-22T18:40:27.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560515+0000", 2024-04-22T18:40:27.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10052781299999999 2024-04-22T18:40:27.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560538+0000", 2024-04-22T18:40:27.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3861000000000001e-05 2024-04-22T18:40:27.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564044+0000", 2024-04-22T18:40:27.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035052009999999999 2024-04-22T18:40:27.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564076+0000", 2024-04-22T18:40:27.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2737000000000002e-05 2024-04-22T18:40:27.359 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.359 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.359 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.360 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:78 3.0 e34/19 3:3a5582ac:::bdd92f7deaa2d50e204891f64f9043469ba687d78b218ed518f2483ba3486a99:head v 34'34, mlcod=31'29)", 2024-04-22T18:40:27.360 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:27.360 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.886451038999994, 2024-04-22T18:40:27.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103547263, 2024-04-22T18:40:27.360 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.360 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.360 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:27.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:27.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460555+0000", 2024-04-22T18:40:27.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:27.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460861+0000", 2024-04-22T18:40:27.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000305958 2024-04-22T18:40:27.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460874+0000", 2024-04-22T18:40:27.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2639e-05 2024-04-22T18:40:27.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460881+0000", 2024-04-22T18:40:27.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2570000000000001e-06 2024-04-22T18:40:27.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560990+0000", 2024-04-22T18:40:27.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010883199999999 2024-04-22T18:40:27.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561011+0000", 2024-04-22T18:40:27.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1319e-05 2024-04-22T18:40:27.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564079+0000", 2024-04-22T18:40:27.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00306815 2024-04-22T18:40:27.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564104+0000", 2024-04-22T18:40:27.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4172000000000002e-05 2024-04-22T18:40:27.364 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.364 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.364 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.364 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:40:27.364 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:27.364 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.885835744999994, 2024-04-22T18:40:27.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:40:27.364 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.364 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.364 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:27.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:27.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:40:27.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:27.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:40:27.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:40:27.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:40:27.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:40:27.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:40:27.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:40:27.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:40:27.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:40:27.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:40:27.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:40:27.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:40:27.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:40:27.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:40:27.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:40:27.368 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.368 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.368 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.368 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:40:27.368 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:27.368 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 79.009161922999994, 2024-04-22T18:40:27.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:40:27.369 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.369 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.369 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:27.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:27.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:40:27.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:40:27.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:40:27.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:40:27.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:40:27.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:40:27.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:40:27.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:40:27.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:40:27.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:40:27.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:40:27.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:40:27.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:40:27.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:40:27.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:40:27.372 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.373 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.373 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.373 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:40:27.373 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:27.373 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.804358400000002, 2024-04-22T18:40:27.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:40:27.373 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.373 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.373 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:27.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:27.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:40:27.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:40:27.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:40:27.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:40:27.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:40:27.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:40:27.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:40:27.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:40:27.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:40:27.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:40:27.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:40:27.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:40:27.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:40:27.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:40:27.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:40:27.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:40:27.377 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.377 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.377 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.377 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2745 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'28, mlcod=74'25)", 2024-04-22T18:40:27.377 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.643080+0000", 2024-04-22T18:40:27.377 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.703927639, 2024-04-22T18:40:27.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10246031899999999, 2024-04-22T18:40:27.377 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.377 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.377 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643080+0000", 2024-04-22T18:40:27.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643080+0000", 2024-04-22T18:40:27.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643078+0000", 2024-04-22T18:40:27.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:27.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643090+0000", 2024-04-22T18:40:27.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1531e-05 2024-04-22T18:40:27.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643099+0000", 2024-04-22T18:40:27.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0650000000000005e-06 2024-04-22T18:40:27.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643105+0000", 2024-04-22T18:40:27.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.993e-06 2024-04-22T18:40:27.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.744552+0000", 2024-04-22T18:40:27.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101447464 2024-04-22T18:40:27.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.744577+0000", 2024-04-22T18:40:27.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4539e-05 2024-04-22T18:40:27.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745515+0000", 2024-04-22T18:40:27.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093760499999999999 2024-04-22T18:40:27.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745540+0000", 2024-04-22T18:40:27.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5582999999999999e-05 2024-04-22T18:40:27.381 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.381 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.381 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.381 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:161 3.0 e76/63 3:3638311b:::f282598a146af3da1c05c8882972fba387afd249b39f6eeb6e6052a5de37fd2d:head v 76'157, mlcod=76'154)", 2024-04-22T18:40:27.381 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:27.381 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.955638969999999, 2024-04-22T18:40:27.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104845172, 2024-04-22T18:40:27.382 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.382 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.382 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:27.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:27.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:27.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:27.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391440+0000", 2024-04-22T18:40:27.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2045999999999994e-05 2024-04-22T18:40:27.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391448+0000", 2024-04-22T18:40:27.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8189999999999992e-06 2024-04-22T18:40:27.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391453+0000", 2024-04-22T18:40:27.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8379999999999998e-06 2024-04-22T18:40:27.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391839+0000", 2024-04-22T18:40:27.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000385325 2024-04-22T18:40:27.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391854+0000", 2024-04-22T18:40:27.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5006e-05 2024-04-22T18:40:27.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496191+0000", 2024-04-22T18:40:27.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104336792 2024-04-22T18:40:27.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496214+0000", 2024-04-22T18:40:27.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3079999999999999e-05 2024-04-22T18:40:27.385 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.385 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.386 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.386 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:165 3.0 e76/63 3:1e2f7d15:::93d9bab0d1f539405a95a28c02bcf2dcbdfa6f3e449e5e5e3f27f774df3c508a:head v 76'158, mlcod=76'154)", 2024-04-22T18:40:27.386 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:27.386 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.955544787000001, 2024-04-22T18:40:27.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104788752, 2024-04-22T18:40:27.386 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.386 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.386 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:27.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:27.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391462+0000", 2024-04-22T18:40:27.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391522+0000", 2024-04-22T18:40:27.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0148000000000001e-05 2024-04-22T18:40:27.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391531+0000", 2024-04-22T18:40:27.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5709999999999998e-06 2024-04-22T18:40:27.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391536+0000", 2024-04-22T18:40:27.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6180000000000003e-06 2024-04-22T18:40:27.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392218+0000", 2024-04-22T18:40:27.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000681326 2024-04-22T18:40:27.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392237+0000", 2024-04-22T18:40:27.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9726999999999999e-05 2024-04-22T18:40:27.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496231+0000", 2024-04-22T18:40:27.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103993672 2024-04-22T18:40:27.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496251+0000", 2024-04-22T18:40:27.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0327999999999999e-05 2024-04-22T18:40:27.390 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.390 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.390 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.390 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:166 3.0 e76/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 76'159, mlcod=76'154)", 2024-04-22T18:40:27.390 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:27.390 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.955466889, 2024-04-22T18:40:27.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104724628, 2024-04-22T18:40:27.390 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.390 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.390 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:27.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:27.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391540+0000", 2024-04-22T18:40:27.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391627+0000", 2024-04-22T18:40:27.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7330000000000006e-05 2024-04-22T18:40:27.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391635+0000", 2024-04-22T18:40:27.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6709999999999999e-06 2024-04-22T18:40:27.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391640+0000", 2024-04-22T18:40:27.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3199999999999999e-06 2024-04-22T18:40:27.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492541+0000", 2024-04-22T18:40:27.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100900873 2024-04-22T18:40:27.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492560+0000", 2024-04-22T18:40:27.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8502999999999999e-05 2024-04-22T18:40:27.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496254+0000", 2024-04-22T18:40:27.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036945849999999998 2024-04-22T18:40:27.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496265+0000", 2024-04-22T18:40:27.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0869999999999999e-05 2024-04-22T18:40:27.394 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.394 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.394 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.394 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:167 3.0 e76/63 3:384c1ce2:::81dc8c58ebe106ae6fa9feadc06291c645d9079ad84b78214d4e6440fbc5def8:head v 76'160, mlcod=76'154)", 2024-04-22T18:40:27.394 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:27.394 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.955352013999999, 2024-04-22T18:40:27.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10487859400000001, 2024-04-22T18:40:27.395 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.395 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.395 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:27.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:27.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:27.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391707+0000", 2024-04-22T18:40:27.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2343999999999999e-05 2024-04-22T18:40:27.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391714+0000", 2024-04-22T18:40:27.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.596e-06 2024-04-22T18:40:27.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391719+0000", 2024-04-22T18:40:27.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9130000000000002e-06 2024-04-22T18:40:27.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493154+0000", 2024-04-22T18:40:27.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101435256 2024-04-22T18:40:27.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493177+0000", 2024-04-22T18:40:27.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2796e-05 2024-04-22T18:40:27.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496509+0000", 2024-04-22T18:40:27.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033320139999999999 2024-04-22T18:40:27.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496534+0000", 2024-04-22T18:40:27.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5265999999999998e-05 2024-04-22T18:40:27.398 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.398 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.398 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.399 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:197 3.0 e78/63 3:19ca42e0:::2109e7e4526cbe26b2963cdd11634231ee7f4d35dabe9962c44b62fafb130ddf:head v 78'167, mlcod=78'165)", 2024-04-22T18:40:27.399 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.540869+0000", 2024-04-22T18:40:27.399 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.806138361999999, 2024-04-22T18:40:27.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10287555800000001, 2024-04-22T18:40:27.399 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.399 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.399 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540869+0000", 2024-04-22T18:40:27.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540869+0000", 2024-04-22T18:40:27.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540867+0000", 2024-04-22T18:40:27.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:27.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540879+0000", 2024-04-22T18:40:27.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1316e-05 2024-04-22T18:40:27.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540895+0000", 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6072000000000001e-05 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540904+0000", 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8969999999999997e-06 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.641023+0000", 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10011914 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.641049+0000", 2024-04-22T18:40:27.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6455e-05 2024-04-22T18:40:27.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.643687+0000", 2024-04-22T18:40:27.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026375499999999998 2024-04-22T18:40:27.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.643745+0000", 2024-04-22T18:40:27.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7725e-05 2024-04-22T18:40:27.403 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.403 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.403 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.403 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:39 3.0 e90/63 3:26983a86:::2d4b6112bcf431e3da007c45a018afa7fe010a4f62df8d2cdc4bdc7924d18240:head v 90'177, mlcod=89'175)", 2024-04-22T18:40:27.403 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.526733+0000", 2024-04-22T18:40:27.403 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.820274575, 2024-04-22T18:40:27.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10253753, 2024-04-22T18:40:27.403 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.403 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.403 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526733+0000", 2024-04-22T18:40:27.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526733+0000", 2024-04-22T18:40:27.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526732+0000", 2024-04-22T18:40:27.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526806+0000", 2024-04-22T18:40:27.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3538e-05 2024-04-22T18:40:27.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526811+0000", 2024-04-22T18:40:27.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6020000000000002e-06 2024-04-22T18:40:27.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526815+0000", 2024-04-22T18:40:27.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0319999999999997e-06 2024-04-22T18:40:27.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526859+0000", 2024-04-22T18:40:27.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3934000000000003e-05 2024-04-22T18:40:27.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526881+0000", 2024-04-22T18:40:27.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2245000000000001e-05 2024-04-22T18:40:27.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629213+0000", 2024-04-22T18:40:27.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10233270699999999 2024-04-22T18:40:27.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629270+0000", 2024-04-22T18:40:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6910999999999998e-05 2024-04-22T18:40:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:40 3.0 e90/63 3:394d11e1:::4fe26d62e8e61f933e14e15995651b4040179bacb8e375345fa3999664858986:head v 90'178, mlcod=89'175)", 2024-04-22T18:40:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.8201864560000001, 2024-04-22T18:40:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102485589, 2024-04-22T18:40:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526928+0000", 2024-04-22T18:40:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107627 2024-04-22T18:40:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526932+0000", 2024-04-22T18:40:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5250000000000001e-06 2024-04-22T18:40:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526935+0000", 2024-04-22T18:40:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9359999999999999e-06 2024-04-22T18:40:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527379+0000", 2024-04-22T18:40:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044412 2024-04-22T18:40:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527400+0000", 2024-04-22T18:40:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0726000000000001e-05 2024-04-22T18:40:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629290+0000", 2024-04-22T18:40:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101890194 2024-04-22T18:40:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629307+0000", 2024-04-22T18:40:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6855999999999999e-05 2024-04-22T18:40:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:42 3.0 e90/63 3:284d7fe5:::e600b8844e4992cc361b6d4a2be033ebaef2fe54ce8cba0a930e4c0b20ff2b84:head v 90'179, mlcod=89'175)", 2024-04-22T18:40:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.526939+0000", 2024-04-22T18:40:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.8200679839999996, 2024-04-22T18:40:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102385087, 2024-04-22T18:40:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526939+0000", 2024-04-22T18:40:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526939+0000", 2024-04-22T18:40:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526939+0000", 2024-04-22T18:40:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527005+0000", 2024-04-22T18:40:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5889999999999994e-05 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527009+0000", 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1620000000000001e-06 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527012+0000", 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7989999999999998e-06 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527758+0000", 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00074576799999999995 2024-04-22T18:40:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527776+0000", 2024-04-22T18:40:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8026999999999999e-05 2024-04-22T18:40:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629309+0000", 2024-04-22T18:40:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101533442 2024-04-22T18:40:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629325+0000", 2024-04-22T18:40:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5274999999999999e-05 2024-04-22T18:40:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:44 3.0 e90/63 3:01335a1f:::33072ed8a3aad555f2749220abc0fe98eb0cb77a7ced650c470f3b78f33ab55e:head v 90'180, mlcod=89'175)", 2024-04-22T18:40:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.527014+0000", 2024-04-22T18:40:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.8199932390000004, 2024-04-22T18:40:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10232838599999999, 2024-04-22T18:40:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527014+0000", 2024-04-22T18:40:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527014+0000", 2024-04-22T18:40:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527014+0000", 2024-04-22T18:40:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:40:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527071+0000", 2024-04-22T18:40:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6737999999999998e-05 2024-04-22T18:40:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527074+0000", 2024-04-22T18:40:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9780000000000001e-06 2024-04-22T18:40:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527078+0000", 2024-04-22T18:40:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5410000000000002e-06 2024-04-22T18:40:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.628286+0000", 2024-04-22T18:40:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10120736900000001 2024-04-22T18:40:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.628301+0000", 2024-04-22T18:40:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5818999999999999e-05 2024-04-22T18:40:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629326+0000", 2024-04-22T18:40:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010242490000000001 2024-04-22T18:40:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629343+0000", 2024-04-22T18:40:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6926999999999999e-05 2024-04-22T18:40:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:47 3.0 e90/63 3:0e62034f:::069f05989298bc0f714a6a846e6234d7fa060b1f5513fa9a7b2ddcab658ece07:head v 90'181, mlcod=89'175)", 2024-04-22T18:40:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.8199271369999996, 2024-04-22T18:40:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10267794, 2024-04-22T18:40:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:40:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527136+0000", 2024-04-22T18:40:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6178000000000003e-05 2024-04-22T18:40:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527141+0000", 2024-04-22T18:40:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4889999999999997e-06 2024-04-22T18:40:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527144+0000", 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0259999999999999e-06 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.628693+0000", 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101549033 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.628713+0000", 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9932000000000002e-05 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629733+0000", 2024-04-22T18:40:27.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010206200000000001 2024-04-22T18:40:27.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629758+0000", 2024-04-22T18:40:27.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4861000000000001e-05 2024-04-22T18:40:27.424 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.424 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.424 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.424 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:3201 4.4 e91/63 4:24717d05:::smithi08937929-42:head v 91'30, mlcod=74'27)", 2024-04-22T18:40:27.424 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.156471+0000", 2024-04-22T18:40:27.424 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1905363270000002, 2024-04-22T18:40:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102886424, 2024-04-22T18:40:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156471+0000", 2024-04-22T18:40:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156471+0000", 2024-04-22T18:40:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156470+0000", 2024-04-22T18:40:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156869+0000", 2024-04-22T18:40:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00039887499999999998 2024-04-22T18:40:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156879+0000", 2024-04-22T18:40:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9399999999999997e-06 2024-04-22T18:40:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.257011+0000", 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100131962 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.257129+0000", 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000118644 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.257144+0000", 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5211000000000001e-05 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.259333+0000", 2024-04-22T18:40:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021890569999999999 2024-04-22T18:40:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.259358+0000", 2024-04-22T18:40:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4039000000000001e-05 2024-04-22T18:40:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:66 3.0 e95/63 3:1683cb7f:::f03bfe2533cb949a19fa6d0ef489d3c38ae0c5777f73a45f60cfab23d5b5f5a4:head v 95'186, mlcod=95'183)", 2024-04-22T18:40:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.635704+0000", 2024-04-22T18:40:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.71130325299999997, 2024-04-22T18:40:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10283379300000001, 2024-04-22T18:40:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635704+0000", 2024-04-22T18:40:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635704+0000", 2024-04-22T18:40:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635703+0000", 2024-04-22T18:40:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635809+0000", 2024-04-22T18:40:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010657 2024-04-22T18:40:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635822+0000", 2024-04-22T18:40:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2500000000000001e-05 2024-04-22T18:40:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635830+0000", 2024-04-22T18:40:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5469999999999999e-06 2024-04-22T18:40:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.735454+0000", 2024-04-22T18:40:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099624303999999997 2024-04-22T18:40:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.735473+0000", 2024-04-22T18:40:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8709999999999999e-05 2024-04-22T18:40:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.738511+0000", 2024-04-22T18:40:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030386689999999999 2024-04-22T18:40:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.738538+0000", 2024-04-22T18:40:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6767999999999999e-05 2024-04-22T18:40:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.433 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:27.434 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:27.435 INFO:tasks.rados.rados.0.smithi089.stdout:1460: set_chunk oid 11 offset: 1257472 length: 1537024 target oid f719318c32cadc4731005313c5f7be78ccd205fcd72beded1a7f5d38a56ca73c offset: 0 snap -1 2024-04-22T18:40:27.435 INFO:tasks.rados.rados.0.smithi089.stdout:1461: write oid 27 current snap is 0 2024-04-22T18:40:27.435 INFO:tasks.rados.rados.0.smithi089.stdout:1461: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-04-22T18:40:27.435 INFO:tasks.rados.rados.0.smithi089.stdout:1461: writing smithi08937929-27 from 423609 to 1125133 tid 1 2024-04-22T18:40:27.435 INFO:tasks.rados.rados.0.smithi089.stdout:1461: writing smithi08937929-27 from 1628332 to 2100421 tid 2 2024-04-22T18:40:27.435 INFO:tasks.rados.rados.0.smithi089.stdout:1461: writing smithi08937929-27 from 2648987 to 2823609 tid 3 2024-04-22T18:40:27.435 INFO:tasks.rados.rados.0.smithi089.stdout:1462: set_chunk oid 10 target oid 2024-04-22T18:40:27.438 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:27.438+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:27.537 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:27.537+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:27.538 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:27.538 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:27.538 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:27.538 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:27.546 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:27.571 INFO:tasks.rados.rados.0.smithi089.stdout:1462: set_chunk oid 10 offset: 1788416 length: 157184 target oid ee8ca23a7050e771c77dff761e6ea5a11ed57796f4e8a17cbb5804a7fdc937b9 offset: 0 snap -1 2024-04-22T18:40:27.571 INFO:tasks.rados.rados.0.smithi089.stdout:1463: write oid 15 current snap is 0 2024-04-22T18:40:27.571 INFO:tasks.rados.rados.0.smithi089.stdout:1463: seq_num 288 ranges {440416=776773,1686814=732100} 2024-04-22T18:40:27.577 INFO:tasks.rados.rados.0.smithi089.stdout:1463: writing smithi08937929-15 from 440416 to 1217189 tid 1 2024-04-22T18:40:27.584 INFO:tasks.rados.rados.0.smithi089.stdout:1463: writing smithi08937929-15 from 1686814 to 2418914 tid 2 2024-04-22T18:40:27.584 INFO:tasks.rados.rados.0.smithi089.stdout:1464: read oid 35 snap -1 2024-04-22T18:40:27.584 INFO:tasks.rados.rados.0.smithi089.stdout:1464: expect (ObjNum 270 snap 0 seq_num 270) 2024-04-22T18:40:27.584 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:27.584 INFO:tasks.rados.rados.0.smithi089.stdout:1453: expect (ObjNum 267 snap 0 seq_num 267) 2024-04-22T18:40:27.638 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:27.638+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:27.672 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:27.673 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:27.673 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:27.673 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:27.673 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:27.682 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:27.698 INFO:tasks.rados.rados.0.smithi089.stdout:1458: expect (ObjNum 282 snap 0 seq_num 282) 2024-04-22T18:40:27.739 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:27.740+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:27.768 INFO:teuthology.orchestra.run.smithi089.stderr:marked out osd.4. 2024-04-22T18:40:27.778 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:40:27.809 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:78 4.0 4:01baf5a9:::smithi08937929-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 93.297267598999994, 2024-04-22T18:40:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.398522192, 2024-04-22T18:40:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 78 2024-04-22T18:40:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511934+0000", 2024-04-22T18:40:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512012+0000", 2024-04-22T18:40:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8329000000000002e-05 2024-04-22T18:40:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512023+0000", 2024-04-22T18:40:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0755e-05 2024-04-22T18:40:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512028+0000", 2024-04-22T18:40:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6470000000000003e-06 2024-04-22T18:40:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512068+0000", 2024-04-22T18:40:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9721999999999997e-05 2024-04-22T18:40:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512125+0000", 2024-04-22T18:40:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6564999999999998e-05 2024-04-22T18:40:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512200+0000", 2024-04-22T18:40:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5708000000000001e-05 2024-04-22T18:40:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513131+0000", 2024-04-22T18:40:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093063 2024-04-22T18:40:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910394+0000", 2024-04-22T18:40:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.397263388 2024-04-22T18:40:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910430+0000", 2024-04-22T18:40:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5485999999999999e-05 2024-04-22T18:40:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910457+0000", 2024-04-22T18:40:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7086000000000001e-05 2024-04-22T18:40:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:79 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 93.297166942999993, 2024-04-22T18:40:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39855419399999997, 2024-04-22T18:40:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 79 2024-04-22T18:40:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512035+0000", 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512045+0000", 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0379e-05 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512051+0000", 2024-04-22T18:40:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6030000000000004e-06 2024-04-22T18:40:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512057+0000", 2024-04-22T18:40:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7270000000000004e-06 2024-04-22T18:40:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512756+0000", 2024-04-22T18:40:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069954000000000001 2024-04-22T18:40:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512806+0000", 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9922000000000001e-05 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512927+0000", 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012059 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.515883+0000", 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029563739999999999 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910510+0000", 2024-04-22T18:40:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39462745199999999 2024-04-22T18:40:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910542+0000", 2024-04-22T18:40:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1225e-05 2024-04-22T18:40:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910590+0000", 2024-04-22T18:40:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8052e-05 2024-04-22T18:40:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:80 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 93.297141842000002, 2024-04-22T18:40:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39866376999999997, 2024-04-22T18:40:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 80 2024-04-22T18:40:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512060+0000", 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512063+0000", 2024-04-22T18:40:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.999e-06 2024-04-22T18:40:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512069+0000", 2024-04-22T18:40:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.891e-06 2024-04-22T18:40:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512073+0000", 2024-04-22T18:40:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4429999999999996e-06 2024-04-22T18:40:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513084+0000", 2024-04-22T18:40:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010102609999999999 2024-04-22T18:40:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513123+0000", 2024-04-22T18:40:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9237999999999999e-05 2024-04-22T18:40:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910618+0000", 2024-04-22T18:40:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39749549299999998 2024-04-22T18:40:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910670+0000", 2024-04-22T18:40:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1221000000000003e-05 2024-04-22T18:40:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910724+0000", 2024-04-22T18:40:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4777000000000001e-05 2024-04-22T18:40:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:419 4.0 4:01baf5a9:::smithi08937929-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:40:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:40:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 88.406980821999994, 2024-04-22T18:40:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30324478199999999, 2024-04-22T18:40:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 419 2024-04-22T18:40:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:40:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:40:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402219+0000", 2024-04-22T18:40:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402229+0000", 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2550000000000003e-06 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402242+0000", 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2989999999999999e-05 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402250+0000", 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7900000000000005e-06 2024-04-22T18:40:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404085+0000", 2024-04-22T18:40:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001834397 2024-04-22T18:40:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:40:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404092+0000", 2024-04-22T18:40:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7889999999999997e-06 2024-04-22T18:40:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502534+0000", 2024-04-22T18:40:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098442331999999994 2024-04-22T18:40:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502631+0000", 2024-04-22T18:40:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7617999999999999e-05 2024-04-22T18:40:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604414+0000", 2024-04-22T18:40:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101782847 2024-04-22T18:40:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604888+0000", 2024-04-22T18:40:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047325999999999998 2024-04-22T18:40:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705376+0000", 2024-04-22T18:40:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100488623 2024-04-22T18:40:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705424+0000", 2024-04-22T18:40:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7964999999999999e-05 2024-04-22T18:40:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705466+0000", 2024-04-22T18:40:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2274999999999998e-05 2024-04-22T18:40:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:229 3.2 e51/29 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head v 51'117, mlcod=51'115)", 2024-04-22T18:40:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.998159784000002, 2024-04-22T18:40:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31548799399999999, 2024-04-22T18:40:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811041+0000", 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811054+0000", 2024-04-22T18:40:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3188e-05 2024-04-22T18:40:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811068+0000", 2024-04-22T18:40:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4239999999999999e-05 2024-04-22T18:40:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.911141+0000", 2024-04-22T18:40:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100072851 2024-04-22T18:40:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025053+0000", 2024-04-22T18:40:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.11391237 2024-04-22T18:40:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025083+0000", 2024-04-22T18:40:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9703999999999999e-05 2024-04-22T18:40:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126504+0000", 2024-04-22T18:40:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10142053500000001 2024-04-22T18:40:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126531+0000", 2024-04-22T18:40:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7172999999999999e-05 2024-04-22T18:40:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2347 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 3378207] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:40:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:40:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.621381559, 2024-04-22T18:40:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30520418599999999, 2024-04-22T18:40:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2347 2024-04-22T18:40:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:40:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:40:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187820+0000", 2024-04-22T18:40:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187823+0000", 2024-04-22T18:40:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6299999999999998e-06 2024-04-22T18:40:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187827+0000", 2024-04-22T18:40:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3739999999999998e-06 2024-04-22T18:40:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187829+0000", 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6290000000000002e-06 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291848+0000", 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104018812 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291924+0000", 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6421000000000006e-05 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292050+0000", 2024-04-22T18:40:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012571500000000001 2024-04-22T18:40:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292407+0000", 2024-04-22T18:40:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035709199999999999 2024-04-22T18:40:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.492962+0000", 2024-04-22T18:40:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20055479200000001 2024-04-22T18:40:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.492993+0000", 2024-04-22T18:40:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1041000000000003e-05 2024-04-22T18:40:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493025+0000", 2024-04-22T18:40:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2092999999999999e-05 2024-04-22T18:40:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2348 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:40:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.621370243000001, 2024-04-22T18:40:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30532242999999998, 2024-04-22T18:40:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2348 2024-04-22T18:40:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187833+0000", 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.437e-06 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187836+0000", 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.897e-06 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187839+0000", 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7250000000000002e-06 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292213+0000", 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104373761 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292256+0000", 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2896999999999999e-05 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493044+0000", 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20078818700000001 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493099+0000", 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5009000000000001e-05 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493155+0000", 2024-04-22T18:40:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5807000000000003e-05 2024-04-22T18:40:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:289 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 25.168477533000001, 2024-04-22T18:40:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9176801189999999, 2024-04-22T18:40:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640723+0000", 2024-04-22T18:40:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640732+0000", 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5809999999999997e-06 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640744+0000", 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.261e-05 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640753+0000", 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7460000000000006e-06 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642189+0000", 2024-04-22T18:40:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001435551 2024-04-22T18:40:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642200+0000", 2024-04-22T18:40:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1389999999999999e-05 2024-04-22T18:40:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.847015+0000", 2024-04-22T18:40:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048149160000001 2024-04-22T18:40:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.848217+0000", 2024-04-22T18:40:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012020659999999999 2024-04-22T18:40:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351823+0000", 2024-04-22T18:40:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50360565099999999 2024-04-22T18:40:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351907+0000", 2024-04-22T18:40:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4141999999999995e-05 2024-04-22T18:40:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352051+0000", 2024-04-22T18:40:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014386600000000001 2024-04-22T18:40:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453617+0000", 2024-04-22T18:40:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156649500000001 2024-04-22T18:40:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558297+0000", 2024-04-22T18:40:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10468000399999999 2024-04-22T18:40:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558331+0000", 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3518e-05 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558405+0000", 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4221000000000006e-05 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:1 3.2 3:6d90b903:::64b5942574fb36ca1b9a2be186c6b740edf58e989056a2f88b83cd280451a438:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.637176573000001, 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.285534422, 2024-04-22T18:40:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172024+0000", 2024-04-22T18:40:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173091+0000", 2024-04-22T18:40:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010677569999999999 2024-04-22T18:40:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173103+0000", 2024-04-22T18:40:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1241e-05 2024-04-22T18:40:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173109+0000", 2024-04-22T18:40:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4300000000000003e-06 2024-04-22T18:40:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173168+0000", 2024-04-22T18:40:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9011000000000001e-05 2024-04-22T18:40:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173177+0000", 2024-04-22T18:40:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-04-22T18:40:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948501+0000", 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77532387199999997 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948912+0000", 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041183000000000002 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950438+0000", 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015253199999999999 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950488+0000", 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0285999999999999e-05 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950601+0000", 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011244899999999999 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954048+0000", 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003447313 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457373+0000", 2024-04-22T18:40:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50332465299999996 2024-04-22T18:40:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457430+0000", 2024-04-22T18:40:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7574999999999997e-05 2024-04-22T18:40:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457560+0000", 2024-04-22T18:40:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013023900000000001 2024-04-22T18:40:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:8 3.2 3:5f64590d:::7c47933f331d40689081b2bd8f9d7f7bdae1be1fdccb52d696047a3969e902b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.636084203999999, 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3854200569999999, 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173117+0000", 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173393+0000", 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000275278 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173399+0000", 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4590000000000003e-06 2024-04-22T18:40:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173406+0000", 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2450000000000001e-06 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173795+0000", 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038912400000000002 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173804+0000", 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.949506+0000", 2024-04-22T18:40:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77570185700000005 2024-04-22T18:40:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950166+0000", 2024-04-22T18:40:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066022800000000001 2024-04-22T18:40:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352843+0000", 2024-04-22T18:40:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40267661799999999 2024-04-22T18:40:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352911+0000", 2024-04-22T18:40:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8126000000000005e-05 2024-04-22T18:40:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.353031+0000", 2024-04-22T18:40:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119919 2024-04-22T18:40:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453622+0000", 2024-04-22T18:40:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100590793 2024-04-22T18:40:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558446+0000", 2024-04-22T18:40:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104824073 2024-04-22T18:40:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558479+0000", 2024-04-22T18:40:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3362999999999999e-05 2024-04-22T18:40:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558538+0000", 2024-04-22T18:40:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9046999999999999e-05 2024-04-22T18:40:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:10 3.2 3:518c37f9:::851eca84dafbf2f780c4f07291110c453349bdc9c623c17bdcd26e27badb30ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.635790611000001, 2024-04-22T18:40:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3852527729999999, 2024-04-22T18:40:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173411+0000", 2024-04-22T18:40:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173749+0000", 2024-04-22T18:40:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000338029 2024-04-22T18:40:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173755+0000", 2024-04-22T18:40:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0530000000000003e-06 2024-04-22T18:40:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173759+0000", 2024-04-22T18:40:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8190000000000002e-06 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173855+0000", 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5927999999999994e-05 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173862+0000", 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4449999999999999e-06 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950835+0000", 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77697248299999999 2024-04-22T18:40:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951632+0000", 2024-04-22T18:40:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079752299999999996 2024-04-22T18:40:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355407+0000", 2024-04-22T18:40:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40377480500000001 2024-04-22T18:40:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355462+0000", 2024-04-22T18:40:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4577999999999998e-05 2024-04-22T18:40:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355571+0000", 2024-04-22T18:40:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108964 2024-04-22T18:40:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453625+0000", 2024-04-22T18:40:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098054397000000001 2024-04-22T18:40:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558574+0000", 2024-04-22T18:40:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104948571 2024-04-22T18:40:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558602+0000", 2024-04-22T18:40:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7966999999999998e-05 2024-04-22T18:40:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558665+0000", 2024-04-22T18:40:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3032000000000002e-05 2024-04-22T18:40:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:11 3.2 3:718aa8aa:::9603c4f8e269a436b2f0bf376c5374e77bdd7cf58fccae60e4e65ac4696806d4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.635438993000001, 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4852604440000001, 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173813+0000", 2024-04-22T18:40:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0655999999999997e-05 2024-04-22T18:40:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173819+0000", 2024-04-22T18:40:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.468e-06 2024-04-22T18:40:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173822+0000", 2024-04-22T18:40:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4879999999999999e-06 2024-04-22T18:40:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173983+0000", 2024-04-22T18:40:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160519 2024-04-22T18:40:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173991+0000", 2024-04-22T18:40:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6490000000000008e-06 2024-04-22T18:40:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951688+0000", 2024-04-22T18:40:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77769743800000002 2024-04-22T18:40:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952419+0000", 2024-04-22T18:40:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073067499999999997 2024-04-22T18:40:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355901+0000", 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40348237599999998 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355953+0000", 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1477000000000002e-05 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356060+0000", 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107114 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453629+0000", 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097569127000000005 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658807+0000", 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20517829800000001 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658876+0000", 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8454000000000006e-05 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659024+0000", 2024-04-22T18:40:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148338 2024-04-22T18:40:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:12 3.2 3:475e1cd9:::efaab2bd13543cdbb143e869e20eb0ec424f7f5202707859b241ed91d4674936:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.635376049000001, 2024-04-22T18:40:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4853405259999999, 2024-04-22T18:40:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173940+0000", 2024-04-22T18:40:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011465500000000001 2024-04-22T18:40:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173946+0000", 2024-04-22T18:40:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5289999999999999e-06 2024-04-22T18:40:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173950+0000", 2024-04-22T18:40:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8290000000000001e-06 2024-04-22T18:40:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174277+0000", 2024-04-22T18:40:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032761299999999998 2024-04-22T18:40:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174285+0000", 2024-04-22T18:40:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.52e-06 2024-04-22T18:40:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952455+0000", 2024-04-22T18:40:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77817000700000005 2024-04-22T18:40:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954035+0000", 2024-04-22T18:40:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001580078 2024-04-22T18:40:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356379+0000", 2024-04-22T18:40:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40234440399999999 2024-04-22T18:40:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356428+0000", 2024-04-22T18:40:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8832999999999998e-05 2024-04-22T18:40:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356532+0000", 2024-04-22T18:40:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103235 2024-04-22T18:40:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453632+0000", 2024-04-22T18:40:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097100346000000004 2024-04-22T18:40:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659069+0000", 2024-04-22T18:40:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20543763800000001 2024-04-22T18:40:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659105+0000", 2024-04-22T18:40:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5240000000000001e-05 2024-04-22T18:40:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659167+0000", 2024-04-22T18:40:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2207999999999998e-05 2024-04-22T18:40:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:16 3.2 3:47c76cc8:::c00d18458eb20c0c0aa9c612cf9793a68445ab0ea7b9e650ecd0072c60a96c0f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.635248090000001, 2024-04-22T18:40:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.485347744, 2024-04-22T18:40:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174234+0000", 2024-04-22T18:40:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028047599999999999 2024-04-22T18:40:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174240+0000", 2024-04-22T18:40:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2079999999999997e-06 2024-04-22T18:40:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174244+0000", 2024-04-22T18:40:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5810000000000001e-06 2024-04-22T18:40:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174670+0000", 2024-04-22T18:40:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042640000000000001 2024-04-22T18:40:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174675+0000", 2024-04-22T18:40:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8709999999999996e-06 2024-04-22T18:40:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054168+0000", 2024-04-22T18:40:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87949257300000006 2024-04-22T18:40:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054877+0000", 2024-04-22T18:40:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070884300000000002 2024-04-22T18:40:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356836+0000", 2024-04-22T18:40:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30195926000000001 2024-04-22T18:40:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356887+0000", 2024-04-22T18:40:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0767999999999999e-05 2024-04-22T18:40:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356991+0000", 2024-04-22T18:40:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104072 2024-04-22T18:40:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453636+0000", 2024-04-22T18:40:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096644968999999997 2024-04-22T18:40:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659207+0000", 2024-04-22T18:40:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20557154699999999 2024-04-22T18:40:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659240+0000", 2024-04-22T18:40:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2665e-05 2024-04-22T18:40:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659302+0000", 2024-04-22T18:40:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2234000000000001e-05 2024-04-22T18:40:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:17 3.2 3:6ef31daa:::1cf4ff496c4480e58510f607481d976b47d84868e2c54016273c38baf590f3b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.634953809999999, 2024-04-22T18:40:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4851703729999999, 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174248+0000", 2024-04-22T18:40:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174626+0000", 2024-04-22T18:40:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037839699999999999 2024-04-22T18:40:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174633+0000", 2024-04-22T18:40:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0990000000000004e-06 2024-04-22T18:40:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174636+0000", 2024-04-22T18:40:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5540000000000002e-06 2024-04-22T18:40:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174790+0000", 2024-04-22T18:40:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153807 2024-04-22T18:40:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174793+0000", 2024-04-22T18:40:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4970000000000001e-06 2024-04-22T18:40:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054902+0000", 2024-04-22T18:40:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88010886300000002 2024-04-22T18:40:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055366+0000", 2024-04-22T18:40:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046336199999999999 2024-04-22T18:40:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.452973+0000", 2024-04-22T18:40:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39760783999999999 2024-04-22T18:40:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453046+0000", 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2374999999999997e-05 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453193+0000", 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014722499999999999 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554698+0000", 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101505096 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659337+0000", 2024-04-22T18:40:27.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104639236 2024-04-22T18:40:27.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659365+0000", 2024-04-22T18:40:27.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7594e-05 2024-04-22T18:40:27.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659419+0000", 2024-04-22T18:40:27.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4058999999999999e-05 2024-04-22T18:40:27.897 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.897 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.898 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.898 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:18 3.2 3:792a4ed0:::e1c814961b3d1531fb5ed475b3d496a0405c3a819b68a92ff9b2391fe912370f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:27.898 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:27.898 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.634557554000001, 2024-04-22T18:40:27.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.585036954, 2024-04-22T18:40:27.898 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.898 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.898 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:27.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:27.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174640+0000", 2024-04-22T18:40:27.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-04-22T18:40:27.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174758+0000", 2024-04-22T18:40:27.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011819099999999999 2024-04-22T18:40:27.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174764+0000", 2024-04-22T18:40:27.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6729999999999997e-06 2024-04-22T18:40:27.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174767+0000", 2024-04-22T18:40:27.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3830000000000001e-06 2024-04-22T18:40:27.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373669+0000", 2024-04-22T18:40:27.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198901989 2024-04-22T18:40:27.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:27.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373686+0000", 2024-04-22T18:40:27.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7286e-05 2024-04-22T18:40:27.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055408+0000", 2024-04-22T18:40:27.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68172133400000001 2024-04-22T18:40:27.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:27.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055938+0000", 2024-04-22T18:40:27.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053016599999999999 2024-04-22T18:40:27.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453668+0000", 2024-04-22T18:40:27.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39773030799999998 2024-04-22T18:40:27.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453717+0000", 2024-04-22T18:40:27.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8884000000000002e-05 2024-04-22T18:40:27.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453823+0000", 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105791 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554715+0000", 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100891628 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759439+0000", 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20472459400000001 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759525+0000", 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5653000000000001e-05 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759682+0000", 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015711800000000001 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:158 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.301956+0000", 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.507246699, 2024-04-22T18:40:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30491982099999998, 2024-04-22T18:40:27.905 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.905 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.905 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.301956+0000", 2024-04-22T18:40:27.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.301956+0000", 2024-04-22T18:40:27.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.301953+0000", 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302335+0000", 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038256800000000001 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302351+0000", 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5313000000000001e-05 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302360+0000", 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302441+0000", 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1242000000000006e-05 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302851+0000", 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041006400000000002 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.505313+0000", 2024-04-22T18:40:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.202461317 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.505387+0000", 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4689000000000003e-05 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.505549+0000", 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016139799999999999 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.605526+0000", 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099977658999999997 2024-04-22T18:40:27.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606733+0000", 2024-04-22T18:40:27.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012065279999999999 2024-04-22T18:40:27.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606779+0000", 2024-04-22T18:40:27.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6516e-05 2024-04-22T18:40:27.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606876+0000", 2024-04-22T18:40:27.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6181999999999995e-05 2024-04-22T18:40:27.909 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.910 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.910 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.910 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:171 3.2 3:723389a0:::8853427863ea44e2e1f1b69134450b6d67dacc49ab40862e8e5106679d802392:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:27.910 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.303144+0000", 2024-04-22T18:40:27.910 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.506058706000001, 2024-04-22T18:40:27.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30331065200000001, 2024-04-22T18:40:27.910 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.910 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.910 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303144+0000", 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303144+0000", 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303143+0000", 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303401+0000", 2024-04-22T18:40:27.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000257581 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303408+0000", 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7730000000000002e-06 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303413+0000", 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6559999999999997e-06 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.307294+0000", 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038801370000000001 2024-04-22T18:40:27.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.307501+0000", 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020771200000000001 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.307795+0000", 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029333000000000001 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404962+0000", 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097167373000000001 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606305+0000", 2024-04-22T18:40:27.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20134328500000001 2024-04-22T18:40:27.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606352+0000", 2024-04-22T18:40:27.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6774000000000003e-05 2024-04-22T18:40:27.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606454+0000", 2024-04-22T18:40:27.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102413 2024-04-22T18:40:27.914 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.915 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.915 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.915 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:174 3.2 3:594d3ad2:::b481f33bf204d13d836f094b25e4b106d427bebe44d81ba5fbb696a319414e75:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:27.915 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.303419+0000", 2024-04-22T18:40:27.915 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.505783549, 2024-04-22T18:40:27.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.303235174, 2024-04-22T18:40:27.915 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.915 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.915 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303419+0000", 2024-04-22T18:40:27.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303419+0000", 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303418+0000", 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303527+0000", 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108925 2024-04-22T18:40:27.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303535+0000", 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2930000000000001e-06 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303544+0000", 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5159999999999994e-06 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404063+0000", 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100518801 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404178+0000", 2024-04-22T18:40:27.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114764 2024-04-22T18:40:27.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:27.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404352+0000", 2024-04-22T18:40:27.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017431200000000001 2024-04-22T18:40:27.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.405276+0000", 2024-04-22T18:40:27.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00092411499999999998 2024-04-22T18:40:27.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606498+0000", 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20122152200000001 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606545+0000", 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7206000000000002e-05 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606654+0000", 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109379 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.919 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:177 3.2 3:6231f5c4:::2915c267dc4ac87f18561ed5fed8de54cbaef6bdd9b80bf698b516d3f8bdd3ee:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e78)", 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.334408+0000", 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.474794702000001, 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30399278099999999, 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334408+0000", 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334408+0000", 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334404+0000", 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999962 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334583+0000", 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000178655 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334603+0000", 2024-04-22T18:40:27.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0251e-05 2024-04-22T18:40:27.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:27.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.434698+0000", 2024-04-22T18:40:27.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100094581 2024-04-22T18:40:27.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:27.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.434779+0000", 2024-04-22T18:40:27.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1528999999999998e-05 2024-04-22T18:40:27.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:27.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.434949+0000", 2024-04-22T18:40:27.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000170113 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.435216+0000", 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026647900000000002 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.437050+0000", 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018344450000000001 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.638245+0000", 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20119524499999999 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:27.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.638295+0000", 2024-04-22T18:40:27.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9994000000000003e-05 2024-04-22T18:40:27.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:27.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:27.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:27.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.638401+0000", 2024-04-22T18:40:27.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105145 2024-04-22T18:40:27.924 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.924 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.924 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.924 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:27.924 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:27.924 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:27.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:27.926 INFO:tasks.rados.rados.0.smithi089.stdout:1449: finishing write tid 1 to smithi08937929-41 2024-04-22T18:40:27.926 INFO:tasks.rados.rados.0.smithi089.stdout:1453: done (15 left) 2024-04-22T18:40:27.926 INFO:tasks.rados.rados.0.smithi089.stdout:1458: done (14 left) 2024-04-22T18:40:27.927 INFO:tasks.rados.rados.0.smithi089.stdout:1465: set_chunk oid 22 target oid 2024-04-22T18:40:27.927 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:27.926+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:27.999 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:40:28.026 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:28.027+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:28.037 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:28.038 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:28.038 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:28.038 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:28.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:28.116 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:28.117 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:28.128 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:28.128+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:28.133 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:28.133 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:28.149 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:28.149 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:28.165 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:28.165 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:28.181 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:28.181 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:28.190 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:28.190 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:28.198 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:28.198 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:28.199 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:28.214 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:28.214 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:28.231 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:28.231 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:28.242 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:28.243+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:28.301 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:28.301 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:28.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:28.329 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:28.330+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:28.430 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:28.431+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:28.460 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:28.532 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:28.532+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:28.562 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:28.562 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:28.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:28.633 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:28.633+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:28.672 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:28.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:28.734 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:28.735+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:28.775 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:40:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:28.785 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:28.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:28.835 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:28.836+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:28.936 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:28.936+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:28.938 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:28.938 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:28.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:29.036 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:29.037+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:29.049 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:29.049 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:29.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:29.137 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:29.138+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:29.161 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:29.161 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:29.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:29.239 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:29.239+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:29.272 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:29.272 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:29.281 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:29.339 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:29.340+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:29.384 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:29.384 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:29.394 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:29.439 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:29.441+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:29.485 INFO:tasks.rados.rados.0.smithi089.stdout:1465: set_chunk oid 22 offset: 485888 length: 553984 target oid caecc2483978192f902232f10c417afab8151b96b28d8501df8f4fdd88a64b9b offset: 0 snap -1 2024-04-22T18:40:29.485 INFO:tasks.rados.rados.0.smithi089.stdout:1466: tier_evict oid 1 2024-04-22T18:40:29.485 INFO:tasks.rados.rados.0.smithi089.stdout:1466: tier_evict oid 1 snap -1 2024-04-22T18:40:29.485 INFO:tasks.rados.rados.0.smithi089.stdout:1431: expect (ObjNum 262 snap 0 seq_num 262) 2024-04-22T18:40:29.500 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:29.500 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:29.510 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:29.541 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:29.542+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:29.612 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:29.612 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:29.621 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:29.632 INFO:tasks.rados.rados.0.smithi089.stdout:1449: finishing write tid 2 to smithi08937929-41 2024-04-22T18:40:29.632 INFO:tasks.rados.rados.0.smithi089.stdout:1455: expect (ObjNum 226 snap 0 seq_num 226) 2024-04-22T18:40:29.642 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:29.643+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:29.724 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:29.724 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:29.733 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:29.766 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:29.767+0000 7f7f44bb3640 -1 osd.4 98 handle_osd_map injecting map crc failure 2024-04-22T18:40:29.775 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:29.775+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:29.844 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:29.844+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:29.878 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:29.878 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:29.887 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:29.903 INFO:tasks.rados.rados.0.smithi089.stdout:1456: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:29.945 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:29.946+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:30.005 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:30.005 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:30.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:30.061 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:30.062+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:30.122 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:30.123 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:30.131 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:30.133 INFO:tasks.rados.rados.0.smithi089.stdout:1449: finishing write tid 3 to smithi08937929-41 2024-04-22T18:40:30.133 INFO:tasks.rados.rados.0.smithi089.stdout:1449: finishing write tid 4 to smithi08937929-41 2024-04-22T18:40:30.133 INFO:tasks.rados.rados.0.smithi089.stdout:1449: oid 41 updating version 0 to 117 2024-04-22T18:40:30.133 INFO:tasks.rados.rados.0.smithi089.stdout:1449: oid 41 updating version 117 to 119 2024-04-22T18:40:30.133 INFO:tasks.rados.rados.0.smithi089.stdout:1449: oid 41 version 119 is already newer than 118 2024-04-22T18:40:30.133 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 119 (ObjNum 284 snap 0 seq_num 284) dirty exists 2024-04-22T18:40:30.133 INFO:tasks.rados.rados.0.smithi089.stdout:1449: left oid 41 (ObjNum 284 snap 0 seq_num 284) 2024-04-22T18:40:30.133 INFO:tasks.rados.rados.0.smithi089.stdout:1452: finishing set_chunk to oid 14 2024-04-22T18:40:30.134 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 51 (ObjNum 163 snap 0 seq_num 163) dirty exists 2024-04-22T18:40:30.134 INFO:tasks.rados.rados.0.smithi089.stdout:1459: finishing write tid 1 to smithi08937929-48 2024-04-22T18:40:30.134 INFO:tasks.rados.rados.0.smithi089.stdout:1426: done (15 left) 2024-04-22T18:40:30.134 INFO:tasks.rados.rados.0.smithi089.stdout:1431: done (14 left) 2024-04-22T18:40:30.134 INFO:tasks.rados.rados.0.smithi089.stdout:1449: done (13 left) 2024-04-22T18:40:30.134 INFO:tasks.rados.rados.0.smithi089.stdout:1452: done (12 left) 2024-04-22T18:40:30.134 INFO:tasks.rados.rados.0.smithi089.stdout:1455: done (11 left) 2024-04-22T18:40:30.134 INFO:tasks.rados.rados.0.smithi089.stdout:1456: done (10 left) 2024-04-22T18:40:30.134 INFO:tasks.rados.rados.0.smithi089.stdout:1467: write (excl) oid 26 current snap is 0 2024-04-22T18:40:30.134 INFO:tasks.rados.rados.0.smithi089.stdout:1459: finishing write tid 2 to smithi08937929-48 2024-04-22T18:40:30.134 INFO:tasks.rados.rados.0.smithi089.stdout:1467: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2024-04-22T18:40:30.138 INFO:tasks.rados.rados.0.smithi089.stdout:1467: writing smithi08937929-26 from 457223 to 1025598 tid 1 2024-04-22T18:40:30.143 INFO:tasks.rados.rados.0.smithi089.stdout:1467: writing smithi08937929-26 from 1461649 to 1937407 tid 2 2024-04-22T18:40:30.147 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:30.148+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:30.149 INFO:tasks.rados.rados.0.smithi089.stdout:1467: writing smithi08937929-26 from 2420186 to 2857223 tid 3 2024-04-22T18:40:30.149 INFO:tasks.rados.rados.0.smithi089.stdout:1460: finishing set_chunk to oid 11 2024-04-22T18:40:30.150 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 11 set_chunk returned error code -95 offset: 1257472 length: 1537024 2024-04-22T18:40:30.150 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 0 (ObjNum 196 snap 0 seq_num 196) dirty exists 2024-04-22T18:40:30.150 INFO:tasks.rados.rados.0.smithi089.stdout:1461: finishing write tid 1 to smithi08937929-27 2024-04-22T18:40:30.150 INFO:tasks.rados.rados.0.smithi089.stdout:1459: finishing write tid 3 to smithi08937929-48 2024-04-22T18:40:30.150 INFO:tasks.rados.rados.0.smithi089.stdout:1468: read oid 28 snap -1 2024-04-22T18:40:30.150 INFO:tasks.rados.rados.0.smithi089.stdout:1468: expect (ObjNum 265 snap 0 seq_num 265) 2024-04-22T18:40:30.150 INFO:tasks.rados.rados.0.smithi089.stdout:1460: done (11 left) 2024-04-22T18:40:30.150 INFO:tasks.rados.rados.0.smithi089.stdout:1469: tier_promote oid 6 2024-04-22T18:40:30.150 INFO:tasks.rados.rados.0.smithi089.stdout:1459: finishing write tid 4 to smithi08937929-48 2024-04-22T18:40:30.150 INFO:tasks.rados.rados.0.smithi089.stdout:1459: oid 48 updating version 0 to 70 2024-04-22T18:40:30.150 INFO:tasks.rados.rados.0.smithi089.stdout:1459: oid 48 updating version 70 to 71 2024-04-22T18:40:30.150 INFO:tasks.rados.rados.0.smithi089.stdout:1459: oid 48 version 71 is already newer than 69 2024-04-22T18:40:30.151 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 71 (ObjNum 286 snap 0 seq_num 286) dirty exists 2024-04-22T18:40:30.151 INFO:tasks.rados.rados.0.smithi089.stdout:1459: left oid 48 (ObjNum 286 snap 0 seq_num 286) 2024-04-22T18:40:30.151 INFO:tasks.rados.rados.0.smithi089.stdout:1454: got (0) Success 2024-04-22T18:40:30.151 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 106 (ObjNum 258 snap 0 seq_num 258) dirty exists 2024-04-22T18:40:30.151 INFO:tasks.rados.rados.0.smithi089.stdout:1461: finishing write tid 2 to smithi08937929-27 2024-04-22T18:40:30.151 INFO:tasks.rados.rados.0.smithi089.stdout:1461: finishing write tid 3 to smithi08937929-27 2024-04-22T18:40:30.151 INFO:tasks.rados.rados.0.smithi089.stdout:1461: finishing write tid 4 to smithi08937929-27 2024-04-22T18:40:30.151 INFO:tasks.rados.rados.0.smithi089.stdout:1461: finishing write tid 5 to smithi08937929-27 2024-04-22T18:40:30.151 INFO:tasks.rados.rados.0.smithi089.stdout:1461: oid 27 updating version 0 to 107 2024-04-22T18:40:30.151 INFO:tasks.rados.rados.0.smithi089.stdout:1461: oid 27 updating version 107 to 109 2024-04-22T18:40:30.151 INFO:tasks.rados.rados.0.smithi089.stdout:1461: oid 27 version 109 is already newer than 108 2024-04-22T18:40:30.151 INFO:tasks.rados.rados.0.smithi089.stdout:1461: oid 27 updating version 109 to 110 2024-04-22T18:40:30.151 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 110 (ObjNum 287 snap 0 seq_num 287) dirty exists 2024-04-22T18:40:30.152 INFO:tasks.rados.rados.0.smithi089.stdout:1461: left oid 27 (ObjNum 287 snap 0 seq_num 287) 2024-04-22T18:40:30.152 INFO:tasks.rados.rados.0.smithi089.stdout:1457: got (0) Success 2024-04-22T18:40:30.152 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 0 (ObjNum 273 snap 0 seq_num 273) dirty exists 2024-04-22T18:40:30.152 INFO:tasks.rados.rados.0.smithi089.stdout:1463: finishing write tid 1 to smithi08937929-15 2024-04-22T18:40:30.152 INFO:tasks.rados.rados.0.smithi089.stdout:1464: expect (ObjNum 270 snap 0 seq_num 270) 2024-04-22T18:40:30.238 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:30.239 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:30.242 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:40:30.247 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:30.248 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:30.247+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:30.316 INFO:tasks.rados.rados.0.smithi089.stdout:1463: finishing write tid 2 to smithi08937929-15 2024-04-22T18:40:30.316 INFO:tasks.rados.rados.0.smithi089.stdout:1463: finishing write tid 3 to smithi08937929-15 2024-04-22T18:40:30.316 INFO:tasks.rados.rados.0.smithi089.stdout:1454: done (11 left) 2024-04-22T18:40:30.316 INFO:tasks.rados.rados.0.smithi089.stdout:1457: done (10 left) 2024-04-22T18:40:30.316 INFO:tasks.rados.rados.0.smithi089.stdout:1459: done (9 left) 2024-04-22T18:40:30.316 INFO:tasks.rados.rados.0.smithi089.stdout:1461: done (8 left) 2024-04-22T18:40:30.316 INFO:tasks.rados.rados.0.smithi089.stdout:1464: done (7 left) 2024-04-22T18:40:30.316 INFO:tasks.rados.rados.0.smithi089.stdout:1470: delete oid 35 current snap is 0 2024-04-22T18:40:30.317 INFO:tasks.rados.rados.0.smithi089.stdout:1462: finishing set_chunk to oid 10 2024-04-22T18:40:30.317 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 21 (ObjNum 124 snap 0 seq_num 124) dirty exists 2024-04-22T18:40:30.317 INFO:tasks.rados.rados.0.smithi089.stdout:1463: finishing write tid 4 to smithi08937929-15 2024-04-22T18:40:30.317 INFO:tasks.rados.rados.0.smithi089.stdout:1463: oid 15 updating version 0 to 78 2024-04-22T18:40:30.317 INFO:tasks.rados.rados.0.smithi089.stdout:1463: oid 15 version 78 is already newer than 77 2024-04-22T18:40:30.317 INFO:tasks.rados.rados.0.smithi089.stdout:1463: oid 15 updating version 78 to 79 2024-04-22T18:40:30.317 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 79 (ObjNum 288 snap 0 seq_num 288) dirty exists 2024-04-22T18:40:30.317 INFO:tasks.rados.rados.0.smithi089.stdout:1463: left oid 15 (ObjNum 288 snap 0 seq_num 288) 2024-04-22T18:40:30.317 INFO:tasks.rados.rados.0.smithi089.stdout:1466: got (2) No such file or directory 2024-04-22T18:40:30.317 INFO:tasks.rados.rados.0.smithi089.stdout:1466: got expected ENOENT (src dne) 2024-04-22T18:40:30.317 INFO:tasks.rados.rados.0.smithi089.stdout:1465: finishing set_chunk to oid 22 2024-04-22T18:40:30.317 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 103 (ObjNum 282 snap 0 seq_num 282) dirty exists 2024-04-22T18:40:30.317 INFO:tasks.rados.rados.0.smithi089.stdout:1468: expect (ObjNum 265 snap 0 seq_num 265) 2024-04-22T18:40:30.348 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:30.349+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:30.371 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:30.371 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:30.390 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:30.390 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:30.395 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:30.395 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:30.404 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:30.406 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:30.407 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:30.424 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:30.424 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:30.434 INFO:tasks.rados.rados.0.smithi089.stdout:1467: finishing write tid 1 to smithi08937929-26 2024-04-22T18:40:30.434 INFO:tasks.rados.rados.0.smithi089.stdout:1467: finishing write tid 2 to smithi08937929-26 2024-04-22T18:40:30.434 INFO:tasks.rados.rados.0.smithi089.stdout:1467: finishing write tid 3 to smithi08937929-26 2024-04-22T18:40:30.434 INFO:tasks.rados.rados.0.smithi089.stdout:1467: finishing write tid 4 to smithi08937929-26 2024-04-22T18:40:30.434 INFO:tasks.rados.rados.0.smithi089.stdout:1467: finishing write tid 5 to smithi08937929-26 2024-04-22T18:40:30.434 INFO:tasks.rados.rados.0.smithi089.stdout:1467: oid 26 updating version 0 to 112 2024-04-22T18:40:30.434 INFO:tasks.rados.rados.0.smithi089.stdout:1467: oid 26 version 112 is already newer than 111 2024-04-22T18:40:30.434 INFO:tasks.rados.rados.0.smithi089.stdout:1467: oid 26 updating version 112 to 114 2024-04-22T18:40:30.435 INFO:tasks.rados.rados.0.smithi089.stdout:1467: oid 26 version 114 is already newer than 113 2024-04-22T18:40:30.435 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 114 (ObjNum 289 snap 0 seq_num 289) dirty exists 2024-04-22T18:40:30.435 INFO:tasks.rados.rados.0.smithi089.stdout:1467: left oid 26 (ObjNum 289 snap 0 seq_num 289) 2024-04-22T18:40:30.435 INFO:tasks.rados.rados.0.smithi089.stdout:1462: done (7 left) 2024-04-22T18:40:30.435 INFO:tasks.rados.rados.0.smithi089.stdout:1463: done (6 left) 2024-04-22T18:40:30.435 INFO:tasks.rados.rados.0.smithi089.stdout:1465: done (5 left) 2024-04-22T18:40:30.435 INFO:tasks.rados.rados.0.smithi089.stdout:1466: done (4 left) 2024-04-22T18:40:30.435 INFO:tasks.rados.rados.0.smithi089.stdout:1467: done (3 left) 2024-04-22T18:40:30.435 INFO:tasks.rados.rados.0.smithi089.stdout:1468: done (2 left) 2024-04-22T18:40:30.435 INFO:tasks.rados.rados.0.smithi089.stdout:1470: done (1 left) 2024-04-22T18:40:30.435 INFO:tasks.rados.rados.0.smithi089.stdout:1471: set_chunk oid 32 target oid 2024-04-22T18:40:30.441 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:30.441 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:30.450 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:30.451+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:30.458 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:30.458 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:30.475 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:30.475 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:30.492 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:30.492 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:30.513 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:30.513 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:30.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:30.514 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:30.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.514 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:73 3.0 e34/19 3:061a7a63:::392f44ec26aa22e85be5f08273a7e45656f3bf4a4854038391a88e880ec01077:head v 34'31, mlcod=31'29)", 2024-04-22T18:40:30.514 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:30.514 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.056070058000003, 2024-04-22T18:40:30.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103503311, 2024-04-22T18:40:30.514 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.514 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.514 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:30.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458193+0000", 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026628900000000001 2024-04-22T18:40:30.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458208+0000", 2024-04-22T18:40:30.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4780000000000001e-05 2024-04-22T18:40:30.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458217+0000", 2024-04-22T18:40:30.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4909999999999994e-06 2024-04-22T18:40:30.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459162+0000", 2024-04-22T18:40:30.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094440500000000005 2024-04-22T18:40:30.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459185+0000", 2024-04-22T18:40:30.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3685000000000001e-05 2024-04-22T18:40:30.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561394+0000", 2024-04-22T18:40:30.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10220874100000001 2024-04-22T18:40:30.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561430+0000", 2024-04-22T18:40:30.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6304999999999999e-05 2024-04-22T18:40:30.518 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.518 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.518 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.518 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:75 3.0 e34/19 3:18418e0f:::b7c35f165c374d5e075dde9ac2607e47ace3b0a1e2343ec6efb171380f725eea:head v 34'32, mlcod=31'29)", 2024-04-22T18:40:30.518 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:30.518 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.055056785000005, 2024-04-22T18:40:30.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104798078, 2024-04-22T18:40:30.518 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.518 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.518 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458939+0000", 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459092+0000", 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153838 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459107+0000", 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4703e-05 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459115+0000", 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560093+0000", 2024-04-22T18:40:30.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10097840499999999 2024-04-22T18:40:30.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560109+0000", 2024-04-22T18:40:30.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5631e-05 2024-04-22T18:40:30.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563706+0000", 2024-04-22T18:40:30.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035972059999999999 2024-04-22T18:40:30.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563738+0000", 2024-04-22T18:40:30.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2184000000000002e-05 2024-04-22T18:40:30.522 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.522 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.522 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.522 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:77 3.0 e34/19 3:28387c9f:::b6d7840f02d561da04e398fbf94d91b85e9066b426e48baf3b11dd4a4f6096c0:head v 34'33, mlcod=31'29)", 2024-04-22T18:40:30.522 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:30.522 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.054106704999995, 2024-04-22T18:40:30.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104186079, 2024-04-22T18:40:30.522 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.522 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.522 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:30.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:30.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459889+0000", 2024-04-22T18:40:30.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:30.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459963+0000", 2024-04-22T18:40:30.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3732999999999994e-05 2024-04-22T18:40:30.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459973+0000", 2024-04-22T18:40:30.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0285999999999999e-05 2024-04-22T18:40:30.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459987+0000", 2024-04-22T18:40:30.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3675999999999999e-05 2024-04-22T18:40:30.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560515+0000", 2024-04-22T18:40:30.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10052781299999999 2024-04-22T18:40:30.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560538+0000", 2024-04-22T18:40:30.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3861000000000001e-05 2024-04-22T18:40:30.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564044+0000", 2024-04-22T18:40:30.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035052009999999999 2024-04-22T18:40:30.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564076+0000", 2024-04-22T18:40:30.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2737000000000002e-05 2024-04-22T18:40:30.526 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.526 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.526 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.526 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:78 3.0 e34/19 3:3a5582ac:::bdd92f7deaa2d50e204891f64f9043469ba687d78b218ed518f2483ba3486a99:head v 34'34, mlcod=31'29)", 2024-04-22T18:40:30.526 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:30.526 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.053440586999997, 2024-04-22T18:40:30.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103547263, 2024-04-22T18:40:30.526 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.527 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.527 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:30.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:30.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460555+0000", 2024-04-22T18:40:30.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:30.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460861+0000", 2024-04-22T18:40:30.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000305958 2024-04-22T18:40:30.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460874+0000", 2024-04-22T18:40:30.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2639e-05 2024-04-22T18:40:30.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460881+0000", 2024-04-22T18:40:30.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2570000000000001e-06 2024-04-22T18:40:30.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560990+0000", 2024-04-22T18:40:30.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010883199999999 2024-04-22T18:40:30.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561011+0000", 2024-04-22T18:40:30.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1319e-05 2024-04-22T18:40:30.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564079+0000", 2024-04-22T18:40:30.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00306815 2024-04-22T18:40:30.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564104+0000", 2024-04-22T18:40:30.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4172000000000002e-05 2024-04-22T18:40:30.530 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.530 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.530 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.531 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:40:30.531 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:30.531 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.052825292999998, 2024-04-22T18:40:30.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:40:30.531 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.531 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.531 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:30.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:30.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:40:30.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:30.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:40:30.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:40:30.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:40:30.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:40:30.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:40:30.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:40:30.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:40:30.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:40:30.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:40:30.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:40:30.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:40:30.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:40:30.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:40:30.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:40:30.535 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.535 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.535 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.535 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:40:30.535 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:30.535 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.176151470999997, 2024-04-22T18:40:30.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:40:30.536 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.536 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.536 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:30.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:30.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:40:30.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:30.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:40:30.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:40:30.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:40:30.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:40:30.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:40:30.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:40:30.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:40:30.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:40:30.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:40:30.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:40:30.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:40:30.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:40:30.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:40:30.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:40:30.539 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.539 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.539 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.540 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:40:30.540 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:30.540 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 25.971347948000002, 2024-04-22T18:40:30.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:40:30.540 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.540 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.540 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:30.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:30.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:40:30.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:40:30.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:40:30.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:40:30.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:40:30.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:40:30.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:40:30.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:40:30.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2745 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'28, mlcod=74'25)", 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.643080+0000", 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 25.870917187, 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10246031899999999, 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643080+0000", 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643080+0000", 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643078+0000", 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643090+0000", 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1531e-05 2024-04-22T18:40:30.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643099+0000", 2024-04-22T18:40:30.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0650000000000005e-06 2024-04-22T18:40:30.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643105+0000", 2024-04-22T18:40:30.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.993e-06 2024-04-22T18:40:30.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.744552+0000", 2024-04-22T18:40:30.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101447464 2024-04-22T18:40:30.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.744577+0000", 2024-04-22T18:40:30.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4539e-05 2024-04-22T18:40:30.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745515+0000", 2024-04-22T18:40:30.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093760499999999999 2024-04-22T18:40:30.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745540+0000", 2024-04-22T18:40:30.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5582999999999999e-05 2024-04-22T18:40:30.548 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.548 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.548 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.548 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:161 3.0 e76/63 3:3638311b:::f282598a146af3da1c05c8882972fba387afd249b39f6eeb6e6052a5de37fd2d:head v 76'157, mlcod=76'154)", 2024-04-22T18:40:30.548 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:30.548 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.122628517999999, 2024-04-22T18:40:30.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104845172, 2024-04-22T18:40:30.548 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.548 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.548 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:30.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:30.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:30.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:30.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391440+0000", 2024-04-22T18:40:30.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2045999999999994e-05 2024-04-22T18:40:30.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391448+0000", 2024-04-22T18:40:30.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8189999999999992e-06 2024-04-22T18:40:30.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391453+0000", 2024-04-22T18:40:30.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8379999999999998e-06 2024-04-22T18:40:30.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391839+0000", 2024-04-22T18:40:30.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000385325 2024-04-22T18:40:30.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391854+0000", 2024-04-22T18:40:30.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5006e-05 2024-04-22T18:40:30.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496191+0000", 2024-04-22T18:40:30.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104336792 2024-04-22T18:40:30.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496214+0000", 2024-04-22T18:40:30.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3079999999999999e-05 2024-04-22T18:40:30.552 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.552 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.552 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.552 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:165 3.0 e76/63 3:1e2f7d15:::93d9bab0d1f539405a95a28c02bcf2dcbdfa6f3e449e5e5e3f27f774df3c508a:head v 76'158, mlcod=76'154)", 2024-04-22T18:40:30.553 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:30.553 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.122534335000001, 2024-04-22T18:40:30.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104788752, 2024-04-22T18:40:30.553 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.553 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.553 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:30.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391462+0000", 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391522+0000", 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0148000000000001e-05 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391531+0000", 2024-04-22T18:40:30.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5709999999999998e-06 2024-04-22T18:40:30.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391536+0000", 2024-04-22T18:40:30.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6180000000000003e-06 2024-04-22T18:40:30.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392218+0000", 2024-04-22T18:40:30.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000681326 2024-04-22T18:40:30.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392237+0000", 2024-04-22T18:40:30.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9726999999999999e-05 2024-04-22T18:40:30.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496231+0000", 2024-04-22T18:40:30.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103993672 2024-04-22T18:40:30.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496251+0000", 2024-04-22T18:40:30.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0327999999999999e-05 2024-04-22T18:40:30.556 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.556 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.556 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.557 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:166 3.0 e76/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 76'159, mlcod=76'154)", 2024-04-22T18:40:30.557 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:30.557 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.122456437, 2024-04-22T18:40:30.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104724628, 2024-04-22T18:40:30.557 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.557 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.557 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:30.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:30.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391540+0000", 2024-04-22T18:40:30.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:30.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391627+0000", 2024-04-22T18:40:30.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7330000000000006e-05 2024-04-22T18:40:30.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391635+0000", 2024-04-22T18:40:30.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6709999999999999e-06 2024-04-22T18:40:30.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391640+0000", 2024-04-22T18:40:30.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3199999999999999e-06 2024-04-22T18:40:30.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492541+0000", 2024-04-22T18:40:30.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100900873 2024-04-22T18:40:30.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492560+0000", 2024-04-22T18:40:30.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8502999999999999e-05 2024-04-22T18:40:30.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496254+0000", 2024-04-22T18:40:30.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036945849999999998 2024-04-22T18:40:30.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496265+0000", 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0869999999999999e-05 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:167 3.0 e76/63 3:384c1ce2:::81dc8c58ebe106ae6fa9feadc06291c645d9079ad84b78214d4e6440fbc5def8:head v 76'160, mlcod=76'154)", 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.122341561999999, 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10487859400000001, 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:30.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:30.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:30.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:30.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391707+0000", 2024-04-22T18:40:30.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2343999999999999e-05 2024-04-22T18:40:30.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391714+0000", 2024-04-22T18:40:30.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.596e-06 2024-04-22T18:40:30.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391719+0000", 2024-04-22T18:40:30.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9130000000000002e-06 2024-04-22T18:40:30.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493154+0000", 2024-04-22T18:40:30.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101435256 2024-04-22T18:40:30.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493177+0000", 2024-04-22T18:40:30.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2796e-05 2024-04-22T18:40:30.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496509+0000", 2024-04-22T18:40:30.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033320139999999999 2024-04-22T18:40:30.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496534+0000", 2024-04-22T18:40:30.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5265999999999998e-05 2024-04-22T18:40:30.565 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.565 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.565 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.565 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:197 3.0 e78/63 3:19ca42e0:::2109e7e4526cbe26b2963cdd11634231ee7f4d35dabe9962c44b62fafb130ddf:head v 78'167, mlcod=78'165)", 2024-04-22T18:40:30.565 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.540869+0000", 2024-04-22T18:40:30.565 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.973127909999999, 2024-04-22T18:40:30.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10287555800000001, 2024-04-22T18:40:30.565 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.566 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.566 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540869+0000", 2024-04-22T18:40:30.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540869+0000", 2024-04-22T18:40:30.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540867+0000", 2024-04-22T18:40:30.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:30.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540879+0000", 2024-04-22T18:40:30.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1316e-05 2024-04-22T18:40:30.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540895+0000", 2024-04-22T18:40:30.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6072000000000001e-05 2024-04-22T18:40:30.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540904+0000", 2024-04-22T18:40:30.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8969999999999997e-06 2024-04-22T18:40:30.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.641023+0000", 2024-04-22T18:40:30.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10011914 2024-04-22T18:40:30.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.641049+0000", 2024-04-22T18:40:30.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6455e-05 2024-04-22T18:40:30.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.643687+0000", 2024-04-22T18:40:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026375499999999998 2024-04-22T18:40:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.643745+0000", 2024-04-22T18:40:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7725e-05 2024-04-22T18:40:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.569 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:39 3.0 e90/63 3:26983a86:::2d4b6112bcf431e3da007c45a018afa7fe010a4f62df8d2cdc4bdc7924d18240:head v 90'177, mlcod=89'175)", 2024-04-22T18:40:30.570 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.526733+0000", 2024-04-22T18:40:30.570 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.9872641229999992, 2024-04-22T18:40:30.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10253753, 2024-04-22T18:40:30.570 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.570 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.570 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526733+0000", 2024-04-22T18:40:30.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526733+0000", 2024-04-22T18:40:30.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526732+0000", 2024-04-22T18:40:30.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:30.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526806+0000", 2024-04-22T18:40:30.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3538e-05 2024-04-22T18:40:30.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526811+0000", 2024-04-22T18:40:30.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6020000000000002e-06 2024-04-22T18:40:30.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526815+0000", 2024-04-22T18:40:30.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0319999999999997e-06 2024-04-22T18:40:30.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526859+0000", 2024-04-22T18:40:30.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3934000000000003e-05 2024-04-22T18:40:30.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526881+0000", 2024-04-22T18:40:30.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2245000000000001e-05 2024-04-22T18:40:30.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629213+0000", 2024-04-22T18:40:30.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10233270699999999 2024-04-22T18:40:30.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629270+0000", 2024-04-22T18:40:30.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6910999999999998e-05 2024-04-22T18:40:30.573 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.573 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.573 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.574 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:40 3.0 e90/63 3:394d11e1:::4fe26d62e8e61f933e14e15995651b4040179bacb8e375345fa3999664858986:head v 90'178, mlcod=89'175)", 2024-04-22T18:40:30.574 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:30.574 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.9871760040000002, 2024-04-22T18:40:30.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102485589, 2024-04-22T18:40:30.574 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.574 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.574 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:30.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526928+0000", 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107627 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526932+0000", 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5250000000000001e-06 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526935+0000", 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9359999999999999e-06 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527379+0000", 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044412 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527400+0000", 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0726000000000001e-05 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629290+0000", 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101890194 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629307+0000", 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6855999999999999e-05 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:42 3.0 e90/63 3:284d7fe5:::e600b8844e4992cc361b6d4a2be033ebaef2fe54ce8cba0a930e4c0b20ff2b84:head v 90'179, mlcod=89'175)", 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.526939+0000", 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.9870575319999997, 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102385087, 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526939+0000", 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526939+0000", 2024-04-22T18:40:30.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526939+0000", 2024-04-22T18:40:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527005+0000", 2024-04-22T18:40:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5889999999999994e-05 2024-04-22T18:40:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527009+0000", 2024-04-22T18:40:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1620000000000001e-06 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527012+0000", 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7989999999999998e-06 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527758+0000", 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00074576799999999995 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527776+0000", 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8026999999999999e-05 2024-04-22T18:40:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629309+0000", 2024-04-22T18:40:30.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101533442 2024-04-22T18:40:30.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629325+0000", 2024-04-22T18:40:30.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5274999999999999e-05 2024-04-22T18:40:30.581 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.581 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.581 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.582 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:44 3.0 e90/63 3:01335a1f:::33072ed8a3aad555f2749220abc0fe98eb0cb77a7ced650c470f3b78f33ab55e:head v 90'180, mlcod=89'175)", 2024-04-22T18:40:30.582 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.527014+0000", 2024-04-22T18:40:30.582 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.9869827870000005, 2024-04-22T18:40:30.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10232838599999999, 2024-04-22T18:40:30.582 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.582 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.582 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527014+0000", 2024-04-22T18:40:30.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527014+0000", 2024-04-22T18:40:30.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527014+0000", 2024-04-22T18:40:30.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:40:30.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527071+0000", 2024-04-22T18:40:30.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6737999999999998e-05 2024-04-22T18:40:30.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527074+0000", 2024-04-22T18:40:30.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9780000000000001e-06 2024-04-22T18:40:30.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527078+0000", 2024-04-22T18:40:30.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5410000000000002e-06 2024-04-22T18:40:30.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.628286+0000", 2024-04-22T18:40:30.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10120736900000001 2024-04-22T18:40:30.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.628301+0000", 2024-04-22T18:40:30.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5818999999999999e-05 2024-04-22T18:40:30.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629326+0000", 2024-04-22T18:40:30.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010242490000000001 2024-04-22T18:40:30.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629343+0000", 2024-04-22T18:40:30.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6926999999999999e-05 2024-04-22T18:40:30.586 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.586 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.586 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.586 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:47 3.0 e90/63 3:0e62034f:::069f05989298bc0f714a6a846e6234d7fa060b1f5513fa9a7b2ddcab658ece07:head v 90'181, mlcod=89'175)", 2024-04-22T18:40:30.586 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:30.586 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.9869166850000006, 2024-04-22T18:40:30.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10267794, 2024-04-22T18:40:30.586 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.586 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.586 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:30.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:30.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:30.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:40:30.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527136+0000", 2024-04-22T18:40:30.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6178000000000003e-05 2024-04-22T18:40:30.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527141+0000", 2024-04-22T18:40:30.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4889999999999997e-06 2024-04-22T18:40:30.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527144+0000", 2024-04-22T18:40:30.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0259999999999999e-06 2024-04-22T18:40:30.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.628693+0000", 2024-04-22T18:40:30.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101549033 2024-04-22T18:40:30.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.628713+0000", 2024-04-22T18:40:30.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9932000000000002e-05 2024-04-22T18:40:30.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629733+0000", 2024-04-22T18:40:30.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010206200000000001 2024-04-22T18:40:30.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629758+0000", 2024-04-22T18:40:30.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4861000000000001e-05 2024-04-22T18:40:30.590 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.590 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.590 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.590 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:3201 4.4 e91/63 4:24717d05:::smithi08937929-42:head v 91'30, mlcod=74'27)", 2024-04-22T18:40:30.591 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.156471+0000", 2024-04-22T18:40:30.591 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.3575258750000003, 2024-04-22T18:40:30.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102886424, 2024-04-22T18:40:30.591 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.591 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.591 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156471+0000", 2024-04-22T18:40:30.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156471+0000", 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156470+0000", 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156869+0000", 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00039887499999999998 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156879+0000", 2024-04-22T18:40:30.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9399999999999997e-06 2024-04-22T18:40:30.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.257011+0000", 2024-04-22T18:40:30.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100131962 2024-04-22T18:40:30.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.257129+0000", 2024-04-22T18:40:30.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000118644 2024-04-22T18:40:30.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.257144+0000", 2024-04-22T18:40:30.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5211000000000001e-05 2024-04-22T18:40:30.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.259333+0000", 2024-04-22T18:40:30.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021890569999999999 2024-04-22T18:40:30.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.259358+0000", 2024-04-22T18:40:30.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4039000000000001e-05 2024-04-22T18:40:30.594 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.594 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.595 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.595 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:66 3.0 e95/63 3:1683cb7f:::f03bfe2533cb949a19fa6d0ef489d3c38ae0c5777f73a45f60cfab23d5b5f5a4:head v 95'186, mlcod=95'183)", 2024-04-22T18:40:30.595 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.635704+0000", 2024-04-22T18:40:30.595 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.8782928009999997, 2024-04-22T18:40:30.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10283379300000001, 2024-04-22T18:40:30.595 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.595 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.595 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635704+0000", 2024-04-22T18:40:30.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635704+0000", 2024-04-22T18:40:30.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635703+0000", 2024-04-22T18:40:30.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:30.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635809+0000", 2024-04-22T18:40:30.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010657 2024-04-22T18:40:30.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635822+0000", 2024-04-22T18:40:30.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2500000000000001e-05 2024-04-22T18:40:30.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635830+0000", 2024-04-22T18:40:30.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5469999999999999e-06 2024-04-22T18:40:30.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.735454+0000", 2024-04-22T18:40:30.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099624303999999997 2024-04-22T18:40:30.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.735473+0000", 2024-04-22T18:40:30.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8709999999999999e-05 2024-04-22T18:40:30.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.738511+0000", 2024-04-22T18:40:30.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030386689999999999 2024-04-22T18:40:30.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.738538+0000", 2024-04-22T18:40:30.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6767999999999999e-05 2024-04-22T18:40:30.599 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.599 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.599 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.599 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.599 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.599 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:30.600 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:30.601 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:30.600+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:30.701 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:30.702+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:30.744 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:30.744 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:30.744 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:30.745 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:30.753 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:30.768 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:30.769+0000 7f7f44bb3640 -1 osd.4 99 handle_osd_map injecting map crc failure 2024-04-22T18:40:30.791 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:40:30.802 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:30.803+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:30.860 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:30.860 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:30.860 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:30.860 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:30.860 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:30.869 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:30.912 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:30.913+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:30.976 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:30.976 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:30.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:30.976 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:30.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.976 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:78 4.0 4:01baf5a9:::smithi08937929-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:30.976 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:30.976 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.46388211, 2024-04-22T18:40:30.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.398522192, 2024-04-22T18:40:30.976 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.977 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.977 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:30.977 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:30.977 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:30.977 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 78 2024-04-22T18:40:30.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.977 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:30.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:30.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511934+0000", 2024-04-22T18:40:30.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:30.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512012+0000", 2024-04-22T18:40:30.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8329000000000002e-05 2024-04-22T18:40:30.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512023+0000", 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0755e-05 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512028+0000", 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6470000000000003e-06 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512068+0000", 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9721999999999997e-05 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512125+0000", 2024-04-22T18:40:30.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6564999999999998e-05 2024-04-22T18:40:30.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:30.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512200+0000", 2024-04-22T18:40:30.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5708000000000001e-05 2024-04-22T18:40:30.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:30.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513131+0000", 2024-04-22T18:40:30.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093063 2024-04-22T18:40:30.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910394+0000", 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.397263388 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910430+0000", 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5485999999999999e-05 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910457+0000", 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7086000000000001e-05 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.981 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.982 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:79 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:30.982 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:30.982 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.463781453999999, 2024-04-22T18:40:30.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39855419399999997, 2024-04-22T18:40:30.982 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.982 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.982 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:30.982 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:30.982 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:30.982 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 79 2024-04-22T18:40:30.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.982 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:30.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:30.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512035+0000", 2024-04-22T18:40:30.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:30.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512045+0000", 2024-04-22T18:40:30.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0379e-05 2024-04-22T18:40:30.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512051+0000", 2024-04-22T18:40:30.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6030000000000004e-06 2024-04-22T18:40:30.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512057+0000", 2024-04-22T18:40:30.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7270000000000004e-06 2024-04-22T18:40:30.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512756+0000", 2024-04-22T18:40:30.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069954000000000001 2024-04-22T18:40:30.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512806+0000", 2024-04-22T18:40:30.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9922000000000001e-05 2024-04-22T18:40:30.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:30.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512927+0000", 2024-04-22T18:40:30.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012059 2024-04-22T18:40:30.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:30.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.515883+0000", 2024-04-22T18:40:30.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029563739999999999 2024-04-22T18:40:30.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:30.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910510+0000", 2024-04-22T18:40:30.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39462745199999999 2024-04-22T18:40:30.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910542+0000", 2024-04-22T18:40:30.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1225e-05 2024-04-22T18:40:30.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910590+0000", 2024-04-22T18:40:30.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8052e-05 2024-04-22T18:40:30.987 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.987 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.987 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.987 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:80 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:30.987 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:30.987 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.463756352999994, 2024-04-22T18:40:30.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39866376999999997, 2024-04-22T18:40:30.988 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.988 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:30.988 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:30.988 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:30.988 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:30.988 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 80 2024-04-22T18:40:30.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.988 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:30.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:30.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512060+0000", 2024-04-22T18:40:30.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:30.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512063+0000", 2024-04-22T18:40:30.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.999e-06 2024-04-22T18:40:30.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512069+0000", 2024-04-22T18:40:30.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.891e-06 2024-04-22T18:40:30.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512073+0000", 2024-04-22T18:40:30.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4429999999999996e-06 2024-04-22T18:40:30.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513084+0000", 2024-04-22T18:40:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010102609999999999 2024-04-22T18:40:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513123+0000", 2024-04-22T18:40:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9237999999999999e-05 2024-04-22T18:40:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910618+0000", 2024-04-22T18:40:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39749549299999998 2024-04-22T18:40:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910670+0000", 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1221000000000003e-05 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910724+0000", 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4777000000000001e-05 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:419 4.0 4:01baf5a9:::smithi08937929-20:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e31)", 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 91.573595333, 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30324478199999999, 2024-04-22T18:40:30.992 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 419 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402222+0000", 2024-04-22T18:40:30.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:30.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402219+0000", 2024-04-22T18:40:30.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:30.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:30.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402229+0000", 2024-04-22T18:40:30.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2550000000000003e-06 2024-04-22T18:40:30.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:30.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402242+0000", 2024-04-22T18:40:30.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2989999999999999e-05 2024-04-22T18:40:30.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:30.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.402250+0000", 2024-04-22T18:40:30.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7900000000000005e-06 2024-04-22T18:40:30.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404085+0000", 2024-04-22T18:40:30.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001834397 2024-04-22T18:40:30.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:40:30.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.404092+0000", 2024-04-22T18:40:30.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7889999999999997e-06 2024-04-22T18:40:30.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:30.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502534+0000", 2024-04-22T18:40:30.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098442331999999994 2024-04-22T18:40:30.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:30.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.502631+0000", 2024-04-22T18:40:30.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7617999999999999e-05 2024-04-22T18:40:30.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604414+0000", 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101782847 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.604888+0000", 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047325999999999998 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705376+0000", 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100488623 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:30.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705424+0000", 2024-04-22T18:40:30.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7964999999999999e-05 2024-04-22T18:40:30.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:30.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:59.705466+0000", 2024-04-22T18:40:30.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2274999999999998e-05 2024-04-22T18:40:30.998 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.998 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:30.998 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:30.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.998 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:229 3.2 e51/29 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head v 51'117, mlcod=51'115)", 2024-04-22T18:40:30.998 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:30.998 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 54.164774295000001, 2024-04-22T18:40:30.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31548799399999999, 2024-04-22T18:40:30.999 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:30.999 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:30.999 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:30.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:30.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:30.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:30.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:30.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:30.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:30.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:30.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811041+0000", 2024-04-22T18:40:31.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:31.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811054+0000", 2024-04-22T18:40:31.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3188e-05 2024-04-22T18:40:31.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811068+0000", 2024-04-22T18:40:31.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4239999999999999e-05 2024-04-22T18:40:31.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.911141+0000", 2024-04-22T18:40:31.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100072851 2024-04-22T18:40:31.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025053+0000", 2024-04-22T18:40:31.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.11391237 2024-04-22T18:40:31.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025083+0000", 2024-04-22T18:40:31.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9703999999999999e-05 2024-04-22T18:40:31.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:31.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126504+0000", 2024-04-22T18:40:31.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10142053500000001 2024-04-22T18:40:31.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126531+0000", 2024-04-22T18:40:31.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7172999999999999e-05 2024-04-22T18:40:31.002 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.002 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.002 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.002 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2347 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 3378207] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:40:31.003 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:40:31.003 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.787996069999998, 2024-04-22T18:40:31.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30520418599999999, 2024-04-22T18:40:31.003 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:31.003 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.003 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:31.003 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:31.003 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.003 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2347 2024-04-22T18:40:31.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.003 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:31.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:31.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:40:31.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:31.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187821+0000", 2024-04-22T18:40:31.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187820+0000", 2024-04-22T18:40:31.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187823+0000", 2024-04-22T18:40:31.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6299999999999998e-06 2024-04-22T18:40:31.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187827+0000", 2024-04-22T18:40:31.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3739999999999998e-06 2024-04-22T18:40:31.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187829+0000", 2024-04-22T18:40:31.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6290000000000002e-06 2024-04-22T18:40:31.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291848+0000", 2024-04-22T18:40:31.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104018812 2024-04-22T18:40:31.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.291924+0000", 2024-04-22T18:40:31.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6421000000000006e-05 2024-04-22T18:40:31.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:31.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292050+0000", 2024-04-22T18:40:31.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012571500000000001 2024-04-22T18:40:31.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:31.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292407+0000", 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035709199999999999 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.492962+0000", 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20055479200000001 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.492993+0000", 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1041000000000003e-05 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493025+0000", 2024-04-22T18:40:31.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2092999999999999e-05 2024-04-22T18:40:31.008 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.008 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.008 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.008 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2348 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:40:31.008 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:31.008 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.787984754, 2024-04-22T18:40:31.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30532242999999998, 2024-04-22T18:40:31.008 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:31.008 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:31.008 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:31.008 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:31.008 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.009 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2348 2024-04-22T18:40:31.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.009 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:31.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:31.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:31.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:31.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:31.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:31.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187833+0000", 2024-04-22T18:40:31.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.437e-06 2024-04-22T18:40:31.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187836+0000", 2024-04-22T18:40:31.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.897e-06 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187839+0000", 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7250000000000002e-06 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292213+0000", 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104373761 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292256+0000", 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2896999999999999e-05 2024-04-22T18:40:31.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493044+0000", 2024-04-22T18:40:31.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20078818700000001 2024-04-22T18:40:31.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493099+0000", 2024-04-22T18:40:31.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5009000000000001e-05 2024-04-22T18:40:31.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493155+0000", 2024-04-22T18:40:31.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5807000000000003e-05 2024-04-22T18:40:31.012 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.013 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.013 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.013 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:289 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:31.013 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:31.013 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.335092044, 2024-04-22T18:40:31.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9176801189999999, 2024-04-22T18:40:31.013 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:31.013 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.013 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:31.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:31.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:31.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640723+0000", 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640732+0000", 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5809999999999997e-06 2024-04-22T18:40:31.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640744+0000", 2024-04-22T18:40:31.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.261e-05 2024-04-22T18:40:31.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640753+0000", 2024-04-22T18:40:31.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7460000000000006e-06 2024-04-22T18:40:31.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642189+0000", 2024-04-22T18:40:31.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001435551 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642200+0000", 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1389999999999999e-05 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.847015+0000", 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048149160000001 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.848217+0000", 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012020659999999999 2024-04-22T18:40:31.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351823+0000", 2024-04-22T18:40:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50360565099999999 2024-04-22T18:40:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351907+0000", 2024-04-22T18:40:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4141999999999995e-05 2024-04-22T18:40:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352051+0000", 2024-04-22T18:40:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014386600000000001 2024-04-22T18:40:31.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:31.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453617+0000", 2024-04-22T18:40:31.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156649500000001 2024-04-22T18:40:31.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558297+0000", 2024-04-22T18:40:31.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10468000399999999 2024-04-22T18:40:31.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:31.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558331+0000", 2024-04-22T18:40:31.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3518e-05 2024-04-22T18:40:31.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558405+0000", 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4221000000000006e-05 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:1 3.2 3:6d90b903:::64b5942574fb36ca1b9a2be186c6b740edf58e989056a2f88b83cd280451a438:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.803791084, 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.285534422, 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.019 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:31.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:31.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:31.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:31.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:31.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172024+0000", 2024-04-22T18:40:31.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:31.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173091+0000", 2024-04-22T18:40:31.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010677569999999999 2024-04-22T18:40:31.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173103+0000", 2024-04-22T18:40:31.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1241e-05 2024-04-22T18:40:31.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173109+0000", 2024-04-22T18:40:31.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4300000000000003e-06 2024-04-22T18:40:31.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173168+0000", 2024-04-22T18:40:31.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9011000000000001e-05 2024-04-22T18:40:31.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:31.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173177+0000", 2024-04-22T18:40:31.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-04-22T18:40:31.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948501+0000", 2024-04-22T18:40:31.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77532387199999997 2024-04-22T18:40:31.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948912+0000", 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041183000000000002 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950438+0000", 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015253199999999999 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950488+0000", 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0285999999999999e-05 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:31.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950601+0000", 2024-04-22T18:40:31.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011244899999999999 2024-04-22T18:40:31.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:31.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954048+0000", 2024-04-22T18:40:31.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003447313 2024-04-22T18:40:31.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457373+0000", 2024-04-22T18:40:31.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50332465299999996 2024-04-22T18:40:31.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:31.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457430+0000", 2024-04-22T18:40:31.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7574999999999997e-05 2024-04-22T18:40:31.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457560+0000", 2024-04-22T18:40:31.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013023900000000001 2024-04-22T18:40:31.025 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.025 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.025 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.025 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:8 3.2 3:5f64590d:::7c47933f331d40689081b2bd8f9d7f7bdae1be1fdccb52d696047a3969e902b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:31.025 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:31.026 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.802698714999998, 2024-04-22T18:40:31.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3854200569999999, 2024-04-22T18:40:31.026 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:31.026 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.026 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:31.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:31.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:31.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:31.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:31.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173117+0000", 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173393+0000", 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000275278 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173399+0000", 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4590000000000003e-06 2024-04-22T18:40:31.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173406+0000", 2024-04-22T18:40:31.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2450000000000001e-06 2024-04-22T18:40:31.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173795+0000", 2024-04-22T18:40:31.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038912400000000002 2024-04-22T18:40:31.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:31.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173804+0000", 2024-04-22T18:40:31.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-04-22T18:40:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.949506+0000", 2024-04-22T18:40:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77570185700000005 2024-04-22T18:40:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950166+0000", 2024-04-22T18:40:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066022800000000001 2024-04-22T18:40:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352843+0000", 2024-04-22T18:40:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40267661799999999 2024-04-22T18:40:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352911+0000", 2024-04-22T18:40:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8126000000000005e-05 2024-04-22T18:40:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.353031+0000", 2024-04-22T18:40:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119919 2024-04-22T18:40:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:31.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453622+0000", 2024-04-22T18:40:31.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100590793 2024-04-22T18:40:31.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558446+0000", 2024-04-22T18:40:31.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104824073 2024-04-22T18:40:31.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:31.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558479+0000", 2024-04-22T18:40:31.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3362999999999999e-05 2024-04-22T18:40:31.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558538+0000", 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9046999999999999e-05 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:10 3.2 3:518c37f9:::851eca84dafbf2f780c4f07291110c453349bdc9c623c17bdcd26e27badb30ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.802405122, 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3852527729999999, 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:31.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173411+0000", 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173749+0000", 2024-04-22T18:40:31.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000338029 2024-04-22T18:40:31.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173755+0000", 2024-04-22T18:40:31.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0530000000000003e-06 2024-04-22T18:40:31.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173759+0000", 2024-04-22T18:40:31.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8190000000000002e-06 2024-04-22T18:40:31.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173855+0000", 2024-04-22T18:40:31.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5927999999999994e-05 2024-04-22T18:40:31.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:31.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173862+0000", 2024-04-22T18:40:31.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4449999999999999e-06 2024-04-22T18:40:31.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950835+0000", 2024-04-22T18:40:31.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77697248299999999 2024-04-22T18:40:31.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951632+0000", 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079752299999999996 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355407+0000", 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40377480500000001 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355462+0000", 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4577999999999998e-05 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:31.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355571+0000", 2024-04-22T18:40:31.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108964 2024-04-22T18:40:31.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:31.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453625+0000", 2024-04-22T18:40:31.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098054397000000001 2024-04-22T18:40:31.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558574+0000", 2024-04-22T18:40:31.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104948571 2024-04-22T18:40:31.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:31.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558602+0000", 2024-04-22T18:40:31.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7966999999999998e-05 2024-04-22T18:40:31.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558665+0000", 2024-04-22T18:40:31.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3032000000000002e-05 2024-04-22T18:40:31.038 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.038 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.038 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.038 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:11 3.2 3:718aa8aa:::9603c4f8e269a436b2f0bf376c5374e77bdd7cf58fccae60e4e65ac4696806d4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:31.039 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:31.039 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.802053504, 2024-04-22T18:40:31.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4852604440000001, 2024-04-22T18:40:31.039 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:31.039 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.039 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:31.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:31.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:31.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:31.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:31.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:31.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173813+0000", 2024-04-22T18:40:31.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0655999999999997e-05 2024-04-22T18:40:31.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173819+0000", 2024-04-22T18:40:31.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.468e-06 2024-04-22T18:40:31.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173822+0000", 2024-04-22T18:40:31.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4879999999999999e-06 2024-04-22T18:40:31.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173983+0000", 2024-04-22T18:40:31.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160519 2024-04-22T18:40:31.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:31.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173991+0000", 2024-04-22T18:40:31.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6490000000000008e-06 2024-04-22T18:40:31.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951688+0000", 2024-04-22T18:40:31.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77769743800000002 2024-04-22T18:40:31.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:31.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952419+0000", 2024-04-22T18:40:31.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073067499999999997 2024-04-22T18:40:31.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355901+0000", 2024-04-22T18:40:31.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40348237599999998 2024-04-22T18:40:31.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355953+0000", 2024-04-22T18:40:31.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1477000000000002e-05 2024-04-22T18:40:31.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:31.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356060+0000", 2024-04-22T18:40:31.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107114 2024-04-22T18:40:31.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:31.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453629+0000", 2024-04-22T18:40:31.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097569127000000005 2024-04-22T18:40:31.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658807+0000", 2024-04-22T18:40:31.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20517829800000001 2024-04-22T18:40:31.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:31.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658876+0000", 2024-04-22T18:40:31.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8454000000000006e-05 2024-04-22T18:40:31.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659024+0000", 2024-04-22T18:40:31.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148338 2024-04-22T18:40:31.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.046 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.047 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:12 3.2 3:475e1cd9:::efaab2bd13543cdbb143e869e20eb0ec424f7f5202707859b241ed91d4674936:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:31.047 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:31.047 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.80199056, 2024-04-22T18:40:31.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4853405259999999, 2024-04-22T18:40:31.047 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:31.047 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.047 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:31.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:31.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:31.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:31.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:31.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:31.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173940+0000", 2024-04-22T18:40:31.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011465500000000001 2024-04-22T18:40:31.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173946+0000", 2024-04-22T18:40:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5289999999999999e-06 2024-04-22T18:40:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173950+0000", 2024-04-22T18:40:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8290000000000001e-06 2024-04-22T18:40:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174277+0000", 2024-04-22T18:40:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032761299999999998 2024-04-22T18:40:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174285+0000", 2024-04-22T18:40:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.52e-06 2024-04-22T18:40:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952455+0000", 2024-04-22T18:40:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77817000700000005 2024-04-22T18:40:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954035+0000", 2024-04-22T18:40:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001580078 2024-04-22T18:40:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356379+0000", 2024-04-22T18:40:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40234440399999999 2024-04-22T18:40:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356428+0000", 2024-04-22T18:40:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8832999999999998e-05 2024-04-22T18:40:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356532+0000", 2024-04-22T18:40:31.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103235 2024-04-22T18:40:31.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:31.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453632+0000", 2024-04-22T18:40:31.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097100346000000004 2024-04-22T18:40:31.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659069+0000", 2024-04-22T18:40:31.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20543763800000001 2024-04-22T18:40:31.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:31.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659105+0000", 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5240000000000001e-05 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659167+0000", 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2207999999999998e-05 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:16 3.2 3:47c76cc8:::c00d18458eb20c0c0aa9c612cf9793a68445ab0ea7b9e650ecd0072c60a96c0f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.801862601, 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.485347744, 2024-04-22T18:40:31.053 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:31.054 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.054 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:31.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:31.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:31.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:31.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:31.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:31.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:31.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174234+0000", 2024-04-22T18:40:31.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028047599999999999 2024-04-22T18:40:31.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174240+0000", 2024-04-22T18:40:31.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2079999999999997e-06 2024-04-22T18:40:31.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174244+0000", 2024-04-22T18:40:31.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5810000000000001e-06 2024-04-22T18:40:31.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174670+0000", 2024-04-22T18:40:31.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042640000000000001 2024-04-22T18:40:31.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:31.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174675+0000", 2024-04-22T18:40:31.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8709999999999996e-06 2024-04-22T18:40:31.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054168+0000", 2024-04-22T18:40:31.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87949257300000006 2024-04-22T18:40:31.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:31.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054877+0000", 2024-04-22T18:40:31.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070884300000000002 2024-04-22T18:40:31.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356836+0000", 2024-04-22T18:40:31.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30195926000000001 2024-04-22T18:40:31.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356887+0000", 2024-04-22T18:40:31.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0767999999999999e-05 2024-04-22T18:40:31.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:31.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356991+0000", 2024-04-22T18:40:31.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104072 2024-04-22T18:40:31.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:31.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453636+0000", 2024-04-22T18:40:31.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096644968999999997 2024-04-22T18:40:31.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659207+0000", 2024-04-22T18:40:31.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20557154699999999 2024-04-22T18:40:31.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:31.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659240+0000", 2024-04-22T18:40:31.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2665e-05 2024-04-22T18:40:31.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659302+0000", 2024-04-22T18:40:31.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2234000000000001e-05 2024-04-22T18:40:31.059 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.059 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.059 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.060 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:17 3.2 3:6ef31daa:::1cf4ff496c4480e58510f607481d976b47d84868e2c54016273c38baf590f3b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:31.060 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:31.060 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.801568321000001, 2024-04-22T18:40:31.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4851703729999999, 2024-04-22T18:40:31.060 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:31.060 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.060 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:31.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:31.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:31.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:31.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:31.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174248+0000", 2024-04-22T18:40:31.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174626+0000", 2024-04-22T18:40:31.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037839699999999999 2024-04-22T18:40:31.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174633+0000", 2024-04-22T18:40:31.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0990000000000004e-06 2024-04-22T18:40:31.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174636+0000", 2024-04-22T18:40:31.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5540000000000002e-06 2024-04-22T18:40:31.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174790+0000", 2024-04-22T18:40:31.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153807 2024-04-22T18:40:31.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:31.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174793+0000", 2024-04-22T18:40:31.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4970000000000001e-06 2024-04-22T18:40:31.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054902+0000", 2024-04-22T18:40:31.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88010886300000002 2024-04-22T18:40:31.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:31.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055366+0000", 2024-04-22T18:40:31.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046336199999999999 2024-04-22T18:40:31.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.452973+0000", 2024-04-22T18:40:31.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39760783999999999 2024-04-22T18:40:31.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453046+0000", 2024-04-22T18:40:31.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2374999999999997e-05 2024-04-22T18:40:31.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:31.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453193+0000", 2024-04-22T18:40:31.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014722499999999999 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554698+0000", 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101505096 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659337+0000", 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104639236 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659365+0000", 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7594e-05 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659419+0000", 2024-04-22T18:40:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4058999999999999e-05 2024-04-22T18:40:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:18 3.2 3:792a4ed0:::e1c814961b3d1531fb5ed475b3d496a0405c3a819b68a92ff9b2391fe912370f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.801172064999999, 2024-04-22T18:40:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.585036954, 2024-04-22T18:40:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174640+0000", 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174758+0000", 2024-04-22T18:40:31.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011819099999999999 2024-04-22T18:40:31.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174764+0000", 2024-04-22T18:40:31.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6729999999999997e-06 2024-04-22T18:40:31.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174767+0000", 2024-04-22T18:40:31.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3830000000000001e-06 2024-04-22T18:40:31.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373669+0000", 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198901989 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373686+0000", 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7286e-05 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055408+0000", 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68172133400000001 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:31.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055938+0000", 2024-04-22T18:40:31.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053016599999999999 2024-04-22T18:40:31.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453668+0000", 2024-04-22T18:40:31.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39773030799999998 2024-04-22T18:40:31.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453717+0000", 2024-04-22T18:40:31.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8884000000000002e-05 2024-04-22T18:40:31.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453823+0000", 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105791 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554715+0000", 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100891628 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759439+0000", 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20472459400000001 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:31.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759525+0000", 2024-04-22T18:40:31.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5653000000000001e-05 2024-04-22T18:40:31.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759682+0000", 2024-04-22T18:40:31.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015711800000000001 2024-04-22T18:40:31.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.072 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.072 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:158 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:31.072 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.301956+0000", 2024-04-22T18:40:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.673861209999998, 2024-04-22T18:40:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30491982099999998, 2024-04-22T18:40:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.301956+0000", 2024-04-22T18:40:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.301956+0000", 2024-04-22T18:40:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.301953+0000", 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302335+0000", 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038256800000000001 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302351+0000", 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5313000000000001e-05 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302360+0000", 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302441+0000", 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1242000000000006e-05 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.302851+0000", 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041006400000000002 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.505313+0000", 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.202461317 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.505387+0000", 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4689000000000003e-05 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.505549+0000", 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016139799999999999 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.605526+0000", 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099977658999999997 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606733+0000", 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012065279999999999 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606779+0000", 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6516e-05 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606876+0000", 2024-04-22T18:40:31.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6181999999999995e-05 2024-04-22T18:40:31.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.078 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.078 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:171 3.2 3:723389a0:::8853427863ea44e2e1f1b69134450b6d67dacc49ab40862e8e5106679d802392:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:31.078 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.303144+0000", 2024-04-22T18:40:31.078 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.672673217, 2024-04-22T18:40:31.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30331065200000001, 2024-04-22T18:40:31.078 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:31.078 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.078 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:31.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303144+0000", 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303144+0000", 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303143+0000", 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303401+0000", 2024-04-22T18:40:31.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000257581 2024-04-22T18:40:31.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303408+0000", 2024-04-22T18:40:31.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7730000000000002e-06 2024-04-22T18:40:31.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303413+0000", 2024-04-22T18:40:31.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6559999999999997e-06 2024-04-22T18:40:31.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.307294+0000", 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038801370000000001 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.307501+0000", 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020771200000000001 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.307795+0000", 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029333000000000001 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404962+0000", 2024-04-22T18:40:31.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097167373000000001 2024-04-22T18:40:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606305+0000", 2024-04-22T18:40:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20134328500000001 2024-04-22T18:40:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606352+0000", 2024-04-22T18:40:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6774000000000003e-05 2024-04-22T18:40:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606454+0000", 2024-04-22T18:40:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102413 2024-04-22T18:40:31.083 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.083 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.083 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.083 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:174 3.2 3:594d3ad2:::b481f33bf204d13d836f094b25e4b106d427bebe44d81ba5fbb696a319414e75:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:31.083 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.303419+0000", 2024-04-22T18:40:31.083 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.672398059999999, 2024-04-22T18:40:31.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.303235174, 2024-04-22T18:40:31.083 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:31.083 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.083 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:31.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:31.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303419+0000", 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303419+0000", 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303418+0000", 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303527+0000", 2024-04-22T18:40:31.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108925 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303535+0000", 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2930000000000001e-06 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.303544+0000", 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5159999999999994e-06 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404063+0000", 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100518801 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404178+0000", 2024-04-22T18:40:31.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114764 2024-04-22T18:40:31.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:31.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.404352+0000", 2024-04-22T18:40:31.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017431200000000001 2024-04-22T18:40:31.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:31.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.405276+0000", 2024-04-22T18:40:31.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00092411499999999998 2024-04-22T18:40:31.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606498+0000", 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20122152200000001 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606545+0000", 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7206000000000002e-05 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.606654+0000", 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109379 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.088 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.0.13:177 3.2 3:6231f5c4:::2915c267dc4ac87f18561ed5fed8de54cbaef6bdd9b80bf698b516d3f8bdd3ee:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e78)", 2024-04-22T18:40:31.088 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.334408+0000", 2024-04-22T18:40:31.088 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.641409212999999, 2024-04-22T18:40:31.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30399278099999999, 2024-04-22T18:40:31.088 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:31.088 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.088 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:31.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:31.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334408+0000", 2024-04-22T18:40:31.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:31.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334408+0000", 2024-04-22T18:40:31.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:31.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:31.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334404+0000", 2024-04-22T18:40:31.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999962 2024-04-22T18:40:31.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:31.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334583+0000", 2024-04-22T18:40:31.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000178655 2024-04-22T18:40:31.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:31.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.334603+0000", 2024-04-22T18:40:31.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0251e-05 2024-04-22T18:40:31.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.434698+0000", 2024-04-22T18:40:31.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100094581 2024-04-22T18:40:31.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:31.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.434779+0000", 2024-04-22T18:40:31.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1528999999999998e-05 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.434949+0000", 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000170113 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.435216+0000", 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026647900000000002 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.437050+0000", 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018344450000000001 2024-04-22T18:40:31.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.638245+0000", 2024-04-22T18:40:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20119524499999999 2024-04-22T18:40:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.638295+0000", 2024-04-22T18:40:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9994000000000003e-05 2024-04-22T18:40:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:31.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.638401+0000", 2024-04-22T18:40:31.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105145 2024-04-22T18:40:31.093 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.093 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.093 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.093 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:31.093 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:31.093 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:31.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:31.096 INFO:tasks.rados.rados.0.smithi089.stdout:1471: set_chunk oid 32 offset: 1812992 length: 307712 target oid 3958b3d9bf78c479df3ed02107a05b14ae5b5125819e73a125cdd7901a6a94e8 offset: 0 snap -1 2024-04-22T18:40:31.096 INFO:tasks.rados.rados.0.smithi089.stdout:1472: write oid 5 current snap is 0 2024-04-22T18:40:31.096 INFO:tasks.rados.rados.0.smithi089.stdout:1469: got (0) Success 2024-04-22T18:40:31.096 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 51 (ObjNum 199 snap 0 seq_num 199) dirty exists 2024-04-22T18:40:31.096 INFO:tasks.rados.rados.0.smithi089.stdout:1472: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-04-22T18:40:31.096 INFO:tasks.rados.rados.0.smithi089.stdout:1472: writing smithi08937929-5 from 490837 to 1209710 tid 1 2024-04-22T18:40:31.096 INFO:tasks.rados.rados.0.smithi089.stdout:1472: writing smithi08937929-5 from 1862260 to 2341687 tid 2 2024-04-22T18:40:31.096 INFO:tasks.rados.rados.0.smithi089.stdout:1472: writing smithi08937929-5 from 2875032 to 2890837 tid 3 2024-04-22T18:40:31.096 INFO:tasks.rados.rados.0.smithi089.stdout:1469: done (2 left) 2024-04-22T18:40:31.096 INFO:tasks.rados.rados.0.smithi089.stdout:1473: read oid 19 snap -1 2024-04-22T18:40:31.096 INFO:tasks.rados.rados.0.smithi089.stdout:1473: expect (ObjNum 273 snap 0 seq_num 273) 2024-04-22T18:40:31.096 INFO:tasks.rados.rados.0.smithi089.stdout:1474: set_chunk oid 34 target oid 2024-04-22T18:40:31.097 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:31.094+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:31.110 INFO:tasks.rados.rados.0.smithi089.stdout:1474: set_chunk oid 34 offset: 524800 length: 1333760 target oid baa1d4681fbe2b611fd4f7e18f5aa75fdc8a31a37797a9b3b160fb875bc5d3eb offset: 0 snap -1 2024-04-22T18:40:31.110 INFO:tasks.rados.rados.0.smithi089.stdout:1475: read oid 24 snap -1 2024-04-22T18:40:31.110 INFO:tasks.rados.rados.0.smithi089.stdout:1475: expect (ObjNum 280 snap 0 seq_num 280) 2024-04-22T18:40:31.110 INFO:tasks.rados.rados.0.smithi089.stdout:1476: read oid 17 snap -1 2024-04-22T18:40:31.110 INFO:tasks.rados.rados.0.smithi089.stdout:1476: expect (ObjNum 192 snap 0 seq_num 192) 2024-04-22T18:40:31.110 INFO:tasks.rados.rados.0.smithi089.stdout:1477: read oid 9 snap -1 2024-04-22T18:40:31.110 INFO:tasks.rados.rados.0.smithi089.stdout:1477: expect (ObjNum 283 snap 0 seq_num 283) 2024-04-22T18:40:31.110 INFO:tasks.rados.rados.0.smithi089.stdout:1478: read oid 29 snap -1 2024-04-22T18:40:31.110 INFO:tasks.rados.rados.0.smithi089.stdout:1478: expect (ObjNum 149 snap 0 seq_num 149) 2024-04-22T18:40:31.110 INFO:tasks.rados.rados.0.smithi089.stdout:1479: tier_evict oid 37 2024-04-22T18:40:31.111 INFO:tasks.rados.rados.0.smithi089.stdout:1479: tier_evict oid 37 snap -1 2024-04-22T18:40:31.111 INFO:tasks.rados.rados.0.smithi089.stdout:1480: tier_flush oid 27 2024-04-22T18:40:31.111 INFO:tasks.rados.rados.0.smithi089.stdout:1480: tier_flush oid 27 snap -1 2024-04-22T18:40:31.111 INFO:tasks.rados.rados.0.smithi089.stdout:1481: write (excl) oid 10 current snap is 0 2024-04-22T18:40:31.111 INFO:tasks.rados.rados.0.smithi089.stdout:1481: seq_num 292 ranges {507644=794122,1920742=623085} 2024-04-22T18:40:31.116 INFO:tasks.rados.rados.0.smithi089.stdout:1481: writing smithi08937929-10 from 507644 to 1301766 tid 1 2024-04-22T18:40:31.124 INFO:tasks.rados.rados.0.smithi089.stdout:1481: writing smithi08937929-10 from 1920742 to 2543827 tid 2 2024-04-22T18:40:31.124 INFO:tasks.rados.rados.0.smithi089.stdout:1482: set_chunk oid 21 target oid 2024-04-22T18:40:31.124 INFO:tasks.rados.rados.0.smithi089.stdout:1482: set_chunk oid 21 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:40:31.124 INFO:tasks.rados.rados.0.smithi089.stdout:1483: write oid 15 current snap is 0 2024-04-22T18:40:31.124 INFO:tasks.rados.rados.0.smithi089.stdout:1483: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2024-04-22T18:40:31.129 INFO:tasks.rados.rados.0.smithi089.stdout:1483: writing smithi08937929-15 from 524451 to 993822 tid 1 2024-04-22T18:40:31.135 INFO:tasks.rados.rados.0.smithi089.stdout:1483: writing smithi08937929-15 from 1579224 to 2062320 tid 2 2024-04-22T18:40:31.140 INFO:tasks.rados.rados.0.smithi089.stdout:1483: writing smithi08937929-15 from 2529878 to 2924451 tid 3 2024-04-22T18:40:31.140 INFO:tasks.rados.rados.0.smithi089.stdout:1484: set_chunk oid 8 target oid 2024-04-22T18:40:31.195 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:31.196+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:31.212 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:31.212 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:31.212 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:31.212 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:31.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:31.296 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:31.297+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:31.329 INFO:tasks.rados.rados.0.smithi089.stdout:1484: set_chunk oid 8 offset: 365056 length: 539136 target oid e47bad38b207220e0e564b2958c2a1d50304b43f598897ae1edf90a970696c54 offset: 0 snap -1 2024-04-22T18:40:31.329 INFO:tasks.rados.rados.0.smithi089.stdout:1485: read oid 6 snap -1 2024-04-22T18:40:31.329 INFO:tasks.rados.rados.0.smithi089.stdout:1485: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:40:31.329 INFO:tasks.rados.rados.0.smithi089.stdout:1486: read oid 30 snap -1 2024-04-22T18:40:31.329 INFO:tasks.rados.rados.0.smithi089.stdout:1486: expect (ObjNum 285 snap 0 seq_num 285) 2024-04-22T18:40:31.329 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:31.330 INFO:tasks.rados.rados.0.smithi089.stdout:1471: finishing set_chunk to oid 32 2024-04-22T18:40:31.330 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 32 set_chunk returned error code -95 offset: 1812992 length: 307712 2024-04-22T18:40:31.330 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 0 (ObjNum 209 snap 0 seq_num 209) dirty exists 2024-04-22T18:40:31.330 INFO:tasks.rados.rados.0.smithi089.stdout:1471: done (15 left) 2024-04-22T18:40:31.330 INFO:tasks.rados.rados.0.smithi089.stdout:1487: read oid 50 snap -1 2024-04-22T18:40:31.330 INFO:tasks.rados.rados.0.smithi089.stdout:1487: expect (ObjNum 189 snap 0 seq_num 189) 2024-04-22T18:40:31.330 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:31.330 INFO:tasks.rados.rados.0.smithi089.stdout:1473: expect (ObjNum 273 snap 0 seq_num 273) 2024-04-22T18:40:31.364 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:31.364 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:31.365 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:31.365 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:31.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:31.397 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:31.398+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:31.451 INFO:tasks.rados.rados.0.smithi089.stdout:1482: finishing set_chunk to oid 21 2024-04-22T18:40:31.451 INFO:tasks.rados.rados.0.smithi089.stdout:1482: got expected ENOENT (src dne) 2024-04-22T18:40:31.451 INFO:tasks.rados.rados.0.smithi089.stdout:1477: expect (ObjNum 283 snap 0 seq_num 283) 2024-04-22T18:40:31.478 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:31.478 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:31.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:31.478 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:31.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.478 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:31.479 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:31.479 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.315916060999996, 2024-04-22T18:40:31.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:40:31.479 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.479 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.479 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.479 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:40:31.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.479 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:31.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:31.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:40:31.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:31.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:40:31.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:40:31.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:40:31.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:40:31.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:40:31.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:40:31.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:40:31.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:40:31.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:40:31.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:40:31.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:31.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:40:31.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:40:31.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:31.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:40:31.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:40:31.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:40:31.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:40:31.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:31.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:40:31.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:40:31.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:40:31.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:40:31.484 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.484 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.484 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.484 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:31.484 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:31.484 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.214357239999998, 2024-04-22T18:40:31.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:40:31.484 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.484 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.484 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.484 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.484 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.485 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:40:31.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.485 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:31.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:31.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:40:31.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:31.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:40:31.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:40:31.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:40:31.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:40:31.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:40:31.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:40:31.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:40:31.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:40:31.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:40:31.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:40:31.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:31.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:40:31.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:40:31.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:31.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:40:31.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:40:31.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:40:31.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.489 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:31.490 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:31.490 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.214330837999995, 2024-04-22T18:40:31.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:40:31.490 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.490 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:31.490 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.490 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.490 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.490 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:40:31.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.490 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:31.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:31.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:40:31.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:40:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:40:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:40:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:40:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:40:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:40:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:40:31.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:40:31.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:31.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:40:31.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:40:31.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:40:31.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:40:31.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:40:31.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:40:31.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:40:31.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:40:31.494 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.494 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.494 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.494 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:31.495 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:31.495 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.815996509000001, 2024-04-22T18:40:31.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:40:31.495 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.495 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.495 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.495 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.495 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.495 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:40:31.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.495 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:31.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:31.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:31.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:40:31.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:40:31.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:40:31.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:40:31.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:31.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:40:31.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:40:31.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:31.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:40:31.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:40:31.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:40:31.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:40:31.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:31.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:40:31.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:40:31.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:40:31.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:40:31.500 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.500 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.500 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.500 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:31.500 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:31.500 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.715160210999997, 2024-04-22T18:40:31.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:40:31.500 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.500 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.500 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.500 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:40:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:40:31.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:31.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:40:31.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:40:31.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:40:31.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:40:31.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:40:31.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:40:31.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:40:31.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:40:31.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:40:31.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:40:31.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:31.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:40:31.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:40:31.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:31.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:40:31.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:40:31.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:40:31.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:40:31.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:31.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:40:31.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:40:31.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:40:31.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:40:31.505 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.505 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.505 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.506 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:31.506 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:31.506 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.715002940999995, 2024-04-22T18:40:31.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:40:31.506 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.506 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.506 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.506 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.506 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.506 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:40:31.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.506 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:40:31.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:40:31.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:40:31.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:40:31.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:40:31.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:40:31.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:40:31.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:40:31.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:40:31.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:40:31.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:31.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:40:31.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:40:31.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:31.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:40:31.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:40:31.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:40:31.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:40:31.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:31.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:40:31.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:40:31.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:40:31.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:40:31.511 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.511 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.511 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.511 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:31.511 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:31.511 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.714988332000004, 2024-04-22T18:40:31.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:40:31.511 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.511 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:31.512 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.512 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.512 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.512 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:40:31.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.512 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:31.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:31.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:31.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:40:31.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:40:31.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:40:31.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:31.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:40:31.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:40:31.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:40:31.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:40:31.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:31.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:40:31.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:40:31.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:40:31.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:40:31.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:40:31.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:40:31.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:40:31.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:40:31.516 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.516 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.516 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.516 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:31.516 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:31.516 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.714977946999994, 2024-04-22T18:40:31.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:31.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:40:31.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:40:31.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:40:31.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:40:31.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:40:31.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:40:31.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:40:31.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:40:31.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:40:31.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:40:31.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:40:31.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:31.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:40:31.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:40:31.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:31.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:40:31.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:40:31.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:40:31.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:40:31.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:31.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:40:31.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:40:31.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:40:31.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:40:31.522 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.522 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.522 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.522 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:31.522 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:31.522 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.714048016999996, 2024-04-22T18:40:31.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:40:31.522 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.522 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.522 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.523 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.523 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.523 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:40:31.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.523 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:31.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:31.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:40:31.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:31.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:40:31.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:40:31.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:40:31.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:40:31.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:40:31.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:40:31.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:40:31.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:40:31.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:40:31.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:40:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:40:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:40:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:40:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:40:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:40:31.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:40:31.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:31.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:40:31.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:40:31.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:40:31.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:40:31.527 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.527 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.527 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.528 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:31.528 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:31.528 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.713585891999998, 2024-04-22T18:40:31.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:40:31.528 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.528 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.528 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.528 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.528 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.528 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:40:31.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.528 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:31.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:31.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:40:31.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:31.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:40:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:40:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:40:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:40:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:40:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:40:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:40:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:40:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:40:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:40:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:40:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:40:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:40:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:40:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:40:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:40:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:40:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:40:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:40:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:40:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.713573088000004, 2024-04-22T18:40:31.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:40:31.534 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.534 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:31.534 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.534 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.534 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.534 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:40:31.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.534 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:31.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:40:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:40:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:40:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:40:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:40:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:40:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:40:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:40:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:40:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:40:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:40:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 93.385800631999999, 2024-04-22T18:40:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:40:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:40:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:40:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:40:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:40:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:40:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:40:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:40:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:40:31.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:40:31.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:31.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:40:31.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:40:31.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:31.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:40:31.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:40:31.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:40:31.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:40:31.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:31.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:40:31.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:40:31.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:40:31.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 92.377227520000005, 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:40:31.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.545 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:31.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:31.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:40:31.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:31.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:40:31.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:40:31.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:40:31.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:40:31.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:40:31.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:40:31.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:40:31.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:40:31.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:40:31.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:40:31.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:31.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:40:31.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:40:31.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:31.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:40:31.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:40:31.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:31.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:40:31.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:40:31.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:31.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:40:31.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:40:31.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:40:31.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:40:31.549 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.549 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.549 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.549 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:40:31.550 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:31.550 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.342580005000002, 2024-04-22T18:40:31.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:40:31.550 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.550 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:31.550 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.550 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.550 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.550 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:40:31.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.550 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:31.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:31.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:40:31.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:40:31.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:40:31.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:40:31.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:40:31.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:40:31.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:40:31.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:40:31.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:40:31.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:40:31.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:40:31.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:40:31.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:40:31.553 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.554 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.554 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.554 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1684 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:31.554 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:31.554 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.873096042, 2024-04-22T18:40:31.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987885199999998, 2024-04-22T18:40:31.554 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.554 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:31.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.554 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1684 2024-04-22T18:40:31.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605278+0000", 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605280+0000", 2024-04-22T18:40:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1770000000000001e-06 2024-04-22T18:40:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605285+0000", 2024-04-22T18:40:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-04-22T18:40:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605289+0000", 2024-04-22T18:40:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.687e-06 2024-04-22T18:40:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705352+0000", 2024-04-22T18:40:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006322199999999 2024-04-22T18:40:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705423+0000", 2024-04-22T18:40:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1136999999999993e-05 2024-04-22T18:40:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105017+0000", 2024-04-22T18:40:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.399593268 2024-04-22T18:40:31.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105081+0000", 2024-04-22T18:40:31.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4703999999999996e-05 2024-04-22T18:40:31.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105157+0000", 2024-04-22T18:40:31.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6205999999999995e-05 2024-04-22T18:40:31.558 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.558 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.558 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2191 4.1 4.fdaa93c1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-04-22T18:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 45.928503792000001, 2024-04-22T18:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51064384399999996, 2024-04-22T18:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2191 2024-04-22T18:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:31.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:31.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:31.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549872+0000", 2024-04-22T18:40:31.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.387e-06 2024-04-22T18:40:31.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549875+0000", 2024-04-22T18:40:31.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:31.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549880+0000", 2024-04-22T18:40:31.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2529999999999998e-06 2024-04-22T18:40:31.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060496+0000", 2024-04-22T18:40:31.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51061597999999997 2024-04-22T18:40:31.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060515+0000", 2024-04-22T18:40:31.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9130000000000001e-05 2024-04-22T18:40:31.562 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.562 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.562 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.562 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2198 4.d 4:bc165c76:::smithi08937929-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:31.563 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:31.563 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 45.927236090999997, 2024-04-22T18:40:31.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50502289899999997, 2024-04-22T18:40:31.563 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.563 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:31.563 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.563 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.563 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.563 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2198 2024-04-22T18:40:31.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.563 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551138+0000", 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551143+0000", 2024-04-22T18:40:31.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8949999999999997e-06 2024-04-22T18:40:31.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551151+0000", 2024-04-22T18:40:31.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8729999999999996e-06 2024-04-22T18:40:31.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551158+0000", 2024-04-22T18:40:31.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3099999999999997e-06 2024-04-22T18:40:31.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551180+0000", 2024-04-22T18:40:31.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2197000000000001e-05 2024-04-22T18:40:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551185+0000", 2024-04-22T18:40:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7010000000000004e-06 2024-04-22T18:40:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657133+0000", 2024-04-22T18:40:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105948498 2024-04-22T18:40:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657183+0000", 2024-04-22T18:40:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0124000000000003e-05 2024-04-22T18:40:31.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.055924+0000", 2024-04-22T18:40:31.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39874079299999998 2024-04-22T18:40:31.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:31.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056018+0000", 2024-04-22T18:40:31.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4032999999999999e-05 2024-04-22T18:40:31.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056161+0000", 2024-04-22T18:40:31.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143385 2024-04-22T18:40:31.568 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.568 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.568 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.568 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2212 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:31.568 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:31.568 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 45.927166448999998, 2024-04-22T18:40:31.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50989936400000002, 2024-04-22T18:40:31.568 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.568 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:31.568 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.568 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.568 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.569 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2212 2024-04-22T18:40:31.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.569 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:31.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:31.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:31.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551211+0000", 2024-04-22T18:40:31.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6280000000000001e-06 2024-04-22T18:40:31.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551214+0000", 2024-04-22T18:40:31.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1209999999999998e-06 2024-04-22T18:40:31.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551217+0000", 2024-04-22T18:40:31.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8490000000000002e-06 2024-04-22T18:40:31.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061086+0000", 2024-04-22T18:40:31.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50986975099999998 2024-04-22T18:40:31.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061108+0000", 2024-04-22T18:40:31.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1319e-05 2024-04-22T18:40:31.572 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.572 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.572 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.572 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2213 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:31.572 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:31.572 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 45.927156031999999, 2024-04-22T18:40:31.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50998486099999996, 2024-04-22T18:40:31.572 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.572 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:31.572 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.572 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.572 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.573 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2213 2024-04-22T18:40:31.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.573 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:31.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:31.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551218+0000", 2024-04-22T18:40:31.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551220+0000", 2024-04-22T18:40:31.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.319e-06 2024-04-22T18:40:31.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551223+0000", 2024-04-22T18:40:31.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9660000000000001e-06 2024-04-22T18:40:31.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551226+0000", 2024-04-22T18:40:31.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9050000000000001e-06 2024-04-22T18:40:31.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061162+0000", 2024-04-22T18:40:31.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.509936838 2024-04-22T18:40:31.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061204+0000", 2024-04-22T18:40:31.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1094999999999999e-05 2024-04-22T18:40:31.575 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.575 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2214 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 45.927147042999998, 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50997458100000004, 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2214 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551227+0000", 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:31.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551229+0000", 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8649999999999999e-06 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551232+0000", 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551235+0000", 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061194+0000", 2024-04-22T18:40:31.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50995819099999995 2024-04-22T18:40:31.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:31.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:31.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:31.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061202+0000", 2024-04-22T18:40:31.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.541e-06 2024-04-22T18:40:31.579 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.579 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.579 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.579 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:31.579 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:31.579 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:31.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:31.581 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:31.581+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:31.611 INFO:tasks.rados.rados.0.smithi089.stdout:1481: finishing write tid 1 to smithi08937929-10 2024-04-22T18:40:31.611 INFO:tasks.rados.rados.0.smithi089.stdout:1483: finishing write tid 1 to smithi08937929-15 2024-04-22T18:40:31.611 INFO:tasks.rados.rados.0.smithi089.stdout:1476: expect (ObjNum 192 snap 0 seq_num 192) 2024-04-22T18:40:31.681 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:31.682+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:31.724 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:31.724 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:31.724 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:31.724 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:31.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:31.746 INFO:tasks.rados.rados.0.smithi089.stdout:1478: expect (ObjNum 149 snap 0 seq_num 149) 2024-04-22T18:40:31.786 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:40:31.818 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:31.819+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:31.876 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:31.876 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:31.883 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:31.884+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:31.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:31.893 INFO:tasks.rados.rados.0.smithi089.stdout:1479: got (22) Invalid argument 2024-04-22T18:40:31.893 INFO:tasks.rados.rados.0.smithi089.stdout:1472: finishing write tid 1 to smithi08937929-5 2024-04-22T18:40:31.893 INFO:tasks.rados.rados.0.smithi089.stdout:1472: finishing write tid 2 to smithi08937929-5 2024-04-22T18:40:31.893 INFO:tasks.rados.rados.0.smithi089.stdout:1472: finishing write tid 3 to smithi08937929-5 2024-04-22T18:40:31.893 INFO:tasks.rados.rados.0.smithi089.stdout:1473: done (15 left) 2024-04-22T18:40:31.893 INFO:tasks.rados.rados.0.smithi089.stdout:1476: done (14 left) 2024-04-22T18:40:31.894 INFO:tasks.rados.rados.0.smithi089.stdout:1477: done (13 left) 2024-04-22T18:40:31.894 INFO:tasks.rados.rados.0.smithi089.stdout:1478: done (12 left) 2024-04-22T18:40:31.894 INFO:tasks.rados.rados.0.smithi089.stdout:1479: done (11 left) 2024-04-22T18:40:31.894 INFO:tasks.rados.rados.0.smithi089.stdout:1482: done (10 left) 2024-04-22T18:40:31.894 INFO:tasks.rados.rados.0.smithi089.stdout:1488: write (excl) oid 22 current snap is 0 2024-04-22T18:40:31.894 INFO:tasks.rados.rados.0.smithi089.stdout:1488: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-04-22T18:40:31.899 INFO:tasks.rados.rados.0.smithi089.stdout:1488: writing smithi08937929-22 from 541258 to 1085878 tid 1 2024-04-22T18:40:31.905 INFO:tasks.rados.rados.0.smithi089.stdout:1488: writing smithi08937929-22 from 1637706 to 2264460 tid 2 2024-04-22T18:40:31.909 INFO:tasks.rados.rados.0.smithi089.stdout:1488: writing smithi08937929-22 from 2840948 to 2941258 tid 3 2024-04-22T18:40:31.909 INFO:tasks.rados.rados.0.smithi089.stdout:1489: read oid 48 snap -1 2024-04-22T18:40:31.909 INFO:tasks.rados.rados.0.smithi089.stdout:1489: expect (ObjNum 286 snap 0 seq_num 286) 2024-04-22T18:40:31.910 INFO:tasks.rados.rados.0.smithi089.stdout:1490: read oid 39 snap -1 2024-04-22T18:40:31.910 INFO:tasks.rados.rados.0.smithi089.stdout:1490: expect (ObjNum 268 snap 0 seq_num 268) 2024-04-22T18:40:31.910 INFO:tasks.rados.rados.0.smithi089.stdout:1491: read oid 2 snap -1 2024-04-22T18:40:31.910 INFO:tasks.rados.rados.0.smithi089.stdout:1491: expect (ObjNum 251 snap 0 seq_num 251) 2024-04-22T18:40:31.910 INFO:tasks.rados.rados.0.smithi089.stdout:1492: write oid 29 current snap is 0 2024-04-22T18:40:31.910 INFO:tasks.rados.rados.0.smithi089.stdout:1492: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-04-22T18:40:31.915 INFO:tasks.rados.rados.0.smithi089.stdout:1492: writing smithi08937929-29 from 558065 to 1177934 tid 1 2024-04-22T18:40:31.920 INFO:tasks.rados.rados.0.smithi089.stdout:1492: writing smithi08937929-29 from 1579835 to 2066600 tid 2 2024-04-22T18:40:31.925 INFO:tasks.rados.rados.0.smithi089.stdout:1492: writing smithi08937929-29 from 2468371 to 2958065 tid 3 2024-04-22T18:40:31.925 INFO:tasks.rados.rados.0.smithi089.stdout:1493: read oid 33 snap -1 2024-04-22T18:40:31.925 INFO:tasks.rados.rados.0.smithi089.stdout:1493: expect (ObjNum 262 snap 0 seq_num 262) 2024-04-22T18:40:31.925 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:31.925 INFO:tasks.rados.rados.0.smithi089.stdout:1474: finishing set_chunk to oid 34 2024-04-22T18:40:31.926 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 106 (ObjNum 277 snap 0 seq_num 277) dirty exists 2024-04-22T18:40:31.926 INFO:tasks.rados.rados.0.smithi089.stdout:1474: done (15 left) 2024-04-22T18:40:31.926 INFO:tasks.rados.rados.0.smithi089.stdout:1494: read oid 7 snap -1 2024-04-22T18:40:31.926 INFO:tasks.rados.rados.0.smithi089.stdout:1494: expect (ObjNum 215 snap 0 seq_num 215) 2024-04-22T18:40:31.926 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:31.926 INFO:tasks.rados.rados.0.smithi089.stdout:1481: finishing write tid 2 to smithi08937929-10 2024-04-22T18:40:31.926 INFO:tasks.rados.rados.0.smithi089.stdout:1486: expect (ObjNum 285 snap 0 seq_num 285) 2024-04-22T18:40:31.984 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:31.984+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:32.030 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:32.030 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:32.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:32.030 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:32.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.030 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:40:32.030 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:32.030 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.936085453999993, 2024-04-22T18:40:32.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:40:32.030 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.030 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.031 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.031 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.031 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.031 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:40:32.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.031 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:32.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:32.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:40:32.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:32.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:40:32.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:40:32.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:40:32.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:40:32.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:40:32.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:40:32.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:40:32.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:40:32.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:40:32.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:40:32.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:32.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:40:32.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:40:32.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:40:32.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:40:32.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:40:32.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:40:32.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:40:32.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:40:32.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:40:32.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:40:32.035 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.035 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.036 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:32.036 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:32.036 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.036587578999999, 2024-04-22T18:40:32.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:40:32.036 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.036 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.036 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:32.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:32.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:40:32.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:32.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:40:32.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:40:32.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:40:32.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:40:32.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:40:32.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:40:32.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:40:32.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:40:32.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:32.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:40:32.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:40:32.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:40:32.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:40:32.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:40:32.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:40:32.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:32.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:40:32.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:40:32.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:40:32.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:40:32.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:40:32.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:40:32.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:40:32.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:40:32.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:40:32.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:40:32.042 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.042 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.042 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.042 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:40:32.042 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:32.042 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 92.928287788000006, 2024-04-22T18:40:32.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:40:32.042 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.042 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.042 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.042 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:40:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:40:32.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:32.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:40:32.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:40:32.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:40:32.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:40:32.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:40:32.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:40:32.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:40:32.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:40:32.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:40:32.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:40:32.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:32.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:40:32.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:40:32.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:40:32.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:40:32.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:40:32.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:40:32.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:40:32.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:40:32.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:40:32.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:40:32.047 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.047 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.047 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.048 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:32.048 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:32.048 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.59674416, 2024-04-22T18:40:32.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:40:32.048 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.048 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.048 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:32.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:32.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:40:32.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:32.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:40:32.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:40:32.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:40:32.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:40:32.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:40:32.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:40:32.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:40:32.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:40:32.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:40:32.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:40:32.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:32.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:40:32.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:40:32.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:40:32.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:40:32.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:40:32.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:40:32.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:40:32.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:40:32.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:40:32.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:40:32.053 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.053 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.053 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.053 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:32.053 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:32.053 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.505658351999998, 2024-04-22T18:40:32.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:40:32.053 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.053 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.054 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:40:32.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.054 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:32.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:32.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:40:32.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:32.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:40:32.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:40:32.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:40:32.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:40:32.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:40:32.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:40:32.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:40:32.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:40:32.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:40:32.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:40:32.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:32.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:40:32.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:40:32.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:40:32.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:40:32.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:40:32.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:40:32.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:40:32.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:40:32.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:40:32.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:40:32.058 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.059 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.059 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.059 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:32.059 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:32.059 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.499858091999997, 2024-04-22T18:40:32.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:40:32.059 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.059 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.059 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.059 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.059 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.059 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:40:32.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:32.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:40:32.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:40:32.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:40:32.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:40:32.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:40:32.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:40:32.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:40:32.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:40:32.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:40:32.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:40:32.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:32.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:40:32.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:40:32.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:40:32.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:40:32.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:40:32.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:40:32.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:40:32.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:40:32.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:40:32.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:40:32.064 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.064 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.064 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.064 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:32.064 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:32.064 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.495117749000002, 2024-04-22T18:40:32.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:40:32.065 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.065 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.065 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.065 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.065 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.065 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:40:32.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.065 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:32.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:32.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:40:32.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:32.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:40:32.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:40:32.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:40:32.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:40:32.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:40:32.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:40:32.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:40:32.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:40:32.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:40:32.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:40:32.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:32.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:40:32.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:40:32.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:40:32.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:40:32.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:40:32.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:40:32.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:40:32.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:40:32.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:40:32.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:40:32.070 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.070 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.070 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.070 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:32.070 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:32.070 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.494644851000004, 2024-04-22T18:40:32.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:40:32.070 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.070 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.070 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.070 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.070 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.071 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:40:32.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.071 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:32.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:32.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:40:32.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:32.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:40:32.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:40:32.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:40:32.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:40:32.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:40:32.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:40:32.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:40:32.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:40:32.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:40:32.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:40:32.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:32.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:40:32.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:40:32.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:40:32.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:40:32.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:40:32.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:40:32.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:40:32.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:40:32.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:40:32.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:40:32.075 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.075 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.075 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.076 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:32.076 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:32.076 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.494624930000001, 2024-04-22T18:40:32.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:40:32.076 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.076 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:32.076 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.076 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.076 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.076 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:40:32.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.076 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:32.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:32.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:32.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:32.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:40:32.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:40:32.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:40:32.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:40:32.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:40:32.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:40:32.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:40:32.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:40:32.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:40:32.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:40:32.080 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.080 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.080 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.081 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:32.081 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:32.081 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 74.185301164999998, 2024-04-22T18:40:32.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:40:32.081 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.081 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.081 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:32.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:32.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:40:32.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:32.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:40:32.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:40:32.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:40:32.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:40:32.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:40:32.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:40:32.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:40:32.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:40:32.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:40:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:40:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:40:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:40:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:40:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:40:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:40:32.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:40:32.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:40:32.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:40:32.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:40:32.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:40:32.085 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.086 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.086 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.086 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:32.086 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:32.086 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 74.185093628999994, 2024-04-22T18:40:32.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:40:32.086 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.086 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.086 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:32.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:40:32.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:40:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:40:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:40:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:40:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:40:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:40:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:40:32.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:40:32.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:40:32.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:40:32.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:40:32.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:40:32.090 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.319443896000003, 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:40:32.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:40:32.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:32.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:40:32.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:40:32.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:40:32.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:40:32.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:40:32.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:40:32.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:40:32.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:40:32.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:40:32.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:40:32.095 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.095 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.095 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.096 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:32.096 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:32.096 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.105605511, 2024-04-22T18:40:32.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:40:32.096 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.096 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.096 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:32.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:32.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:40:32.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:32.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:40:32.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:40:32.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:40:32.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:40:32.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:40:32.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:40:32.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:40:32.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:40:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:40:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:40:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:40:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:40:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:40:32.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:40:32.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:40:32.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:40:32.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:40:32.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:40:32.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:40:32.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.105572101, 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:32.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:32.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:32.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:32.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:40:32.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:40:32.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:40:32.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:40:32.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:40:32.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:40:32.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:40:32.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:40:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:40:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:40:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:40:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:40:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:40:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:40:32.106 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.106 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.106 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.106 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:32.106 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:32.106 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.046689446999999, 2024-04-22T18:40:32.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:40:32.106 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.106 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.106 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:32.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:32.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:40:32.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:32.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:40:32.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:40:32.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:40:32.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:40:32.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:40:32.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:40:32.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:40:32.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:40:32.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:40:32.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:40:32.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:32.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:40:32.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:40:32.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:40:32.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:40:32.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:40:32.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:40:32.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:40:32.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:40:32.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:40:32.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:40:32.111 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.111 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.111 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.111 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:32.111 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:32.111 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.046550830999998, 2024-04-22T18:40:32.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:40:32.111 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:32.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:32.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:40:32.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:40:32.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:40:32.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:40:32.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:40:32.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:40:32.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:40:32.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:40:32.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:40:32.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:40:32.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:32.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:40:32.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:40:32.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:40:32.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:40:32.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:40:32.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:40:32.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.046531827999999, 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:40:32.116 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.117 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.117 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:32.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:32.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:32.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:32.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:40:32.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:40:32.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:40:32.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:40:32.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:40:32.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:40:32.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:40:32.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:40:32.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:40:32.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:40:32.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:32.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:40:32.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:40:32.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:40:32.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:40:32.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:40:32.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:40:32.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:40:32.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:40:32.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:40:32.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:40:32.121 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.121 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.121 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.121 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:32.121 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:32.121 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.425519295000001, 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:32.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:40:32.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:32.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:40:32.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:40:32.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:40:32.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:40:32.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:40:32.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:40:32.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:40:32.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:40:32.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:40:32.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:40:32.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:40:32.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:40:32.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:40:32.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:40:32.127 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.127 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.127 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.127 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:32.127 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:32.127 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.424999704000001, 2024-04-22T18:40:32.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:40:32.127 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.127 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.127 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.127 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.127 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.128 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:40:32.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.128 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:32.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:32.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:40:32.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:32.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:40:32.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:40:32.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:40:32.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:40:32.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:40:32.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:40:32.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:40:32.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:40:32.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:40:32.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:40:32.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:32.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:40:32.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:40:32.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:40:32.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:40:32.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:40:32.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:40:32.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:40:32.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:40:32.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:40:32.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:40:32.132 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.132 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.132 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.133 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:32.133 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:32.133 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.324846821000001, 2024-04-22T18:40:32.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:40:32.133 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.133 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:32.133 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.133 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.133 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.133 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:40:32.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.133 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:32.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:32.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:40:32.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:32.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:40:32.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:40:32.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:40:32.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:40:32.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:40:32.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:40:32.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:40:32.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:40:32.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:32.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:40:32.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:40:32.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:40:32.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:40:32.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:40:32.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:40:32.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:40:32.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:40:32.137 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.137 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.137 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.137 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.138 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.138 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:32.139 INFO:tasks.rados.rados.0.smithi089.stdout:1472: finishing write tid 4 to smithi08937929-5 2024-04-22T18:40:32.139 INFO:tasks.rados.rados.0.smithi089.stdout:1472: finishing write tid 5 to smithi08937929-5 2024-04-22T18:40:32.139 INFO:tasks.rados.rados.0.smithi089.stdout:1472: oid 5 updating version 0 to 150 2024-04-22T18:40:32.139 INFO:tasks.rados.rados.0.smithi089.stdout:1472: oid 5 version 150 is already newer than 149 2024-04-22T18:40:32.139 INFO:tasks.rados.rados.0.smithi089.stdout:1472: oid 5 version 150 is already newer than 147 2024-04-22T18:40:32.139 INFO:tasks.rados.rados.0.smithi089.stdout:1472: oid 5 version 150 is already newer than 148 2024-04-22T18:40:32.139 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 150 (ObjNum 291 snap 0 seq_num 291) dirty exists 2024-04-22T18:40:32.139 INFO:tasks.rados.rados.0.smithi089.stdout:1472: left oid 5 (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:40:32.139 INFO:tasks.rados.rados.0.smithi089.stdout:1475: expect (ObjNum 280 snap 0 seq_num 280) 2024-04-22T18:40:32.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:32.140 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:32.140+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:32.239 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:32.240+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:32.270 INFO:tasks.rados.rados.0.smithi089.stdout:1487: expect (ObjNum 189 snap 0 seq_num 189) 2024-04-22T18:40:32.281 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:32.281 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:32.281 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:32.281 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:32.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:32.340 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:32.340+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:32.391 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:32.391 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:32.391 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:32.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:32.444 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:32.445+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:32.485 INFO:tasks.rados.rados.0.smithi089.stdout:1481: finishing write tid 3 to smithi08937929-10 2024-04-22T18:40:32.485 INFO:tasks.rados.rados.0.smithi089.stdout:1481: finishing write tid 4 to smithi08937929-10 2024-04-22T18:40:32.485 INFO:tasks.rados.rados.0.smithi089.stdout:1481: oid 10 updating version 0 to 36 2024-04-22T18:40:32.485 INFO:tasks.rados.rados.0.smithi089.stdout:1481: oid 10 version 36 is already newer than 35 2024-04-22T18:40:32.485 INFO:tasks.rados.rados.0.smithi089.stdout:1481: oid 10 updating version 36 to 37 2024-04-22T18:40:32.486 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 37 (ObjNum 292 snap 0 seq_num 292) dirty exists 2024-04-22T18:40:32.486 INFO:tasks.rados.rados.0.smithi089.stdout:1481: left oid 10 (ObjNum 292 snap 0 seq_num 292) 2024-04-22T18:40:32.486 INFO:tasks.rados.rados.0.smithi089.stdout:1485: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:40:32.503 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:32.503 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:32.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:32.503 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:50 4.a 4:5d2aee44:::smithi08937929-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.264019348000005, 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40856295399999998, 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 50 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:32.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:32.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239607+0000", 2024-04-22T18:40:32.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:32.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240309+0000", 2024-04-22T18:40:32.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070201599999999997 2024-04-22T18:40:32.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240320+0000", 2024-04-22T18:40:32.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1024e-05 2024-04-22T18:40:32.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240327+0000", 2024-04-22T18:40:32.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0049999999999998e-06 2024-04-22T18:40:32.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362065+0000", 2024-04-22T18:40:32.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.121737812 2024-04-22T18:40:32.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362121+0000", 2024-04-22T18:40:32.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5868000000000001e-05 2024-04-22T18:40:32.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:32.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362210+0000", 2024-04-22T18:40:32.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8533000000000001e-05 2024-04-22T18:40:32.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365322+0000", 2024-04-22T18:40:32.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031121220000000002 2024-04-22T18:40:32.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648054+0000", 2024-04-22T18:40:32.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.282732549 2024-04-22T18:40:32.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648105+0000", 2024-04-22T18:40:32.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1276999999999997e-05 2024-04-22T18:40:32.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648172+0000", 2024-04-22T18:40:32.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6398999999999999e-05 2024-04-22T18:40:32.509 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.509 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.509 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.509 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:51 4.a 4:5d2aee44:::smithi08937929-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:32.509 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:32.509 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.263296814, 2024-04-22T18:40:32.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40793057199999999, 2024-04-22T18:40:32.509 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.510 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.510 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.510 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.510 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.510 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 51 2024-04-22T18:40:32.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.510 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:32.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:32.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:32.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:32.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240336+0000", 2024-04-22T18:40:32.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:40:32.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240339+0000", 2024-04-22T18:40:32.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9009999999999998e-06 2024-04-22T18:40:32.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240342+0000", 2024-04-22T18:40:32.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:40:32.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363080+0000", 2024-04-22T18:40:32.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122738075 2024-04-22T18:40:32.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363164+0000", 2024-04-22T18:40:32.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3722999999999998e-05 2024-04-22T18:40:32.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:32.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363287+0000", 2024-04-22T18:40:32.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012301099999999999 2024-04-22T18:40:32.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466753+0000", 2024-04-22T18:40:32.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103466217 2024-04-22T18:40:32.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648205+0000", 2024-04-22T18:40:32.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.181451788 2024-04-22T18:40:32.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648231+0000", 2024-04-22T18:40:32.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6597999999999998e-05 2024-04-22T18:40:32.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648262+0000", 2024-04-22T18:40:32.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0871000000000003e-05 2024-04-22T18:40:32.518 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.518 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.518 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.518 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:52 4.a 4:5d2aee44:::smithi08937929-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:32.518 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:32.518 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.263284354999996, 2024-04-22T18:40:32.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40812731200000002, 2024-04-22T18:40:32.518 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.518 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:32.519 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.519 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.519 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.519 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 52 2024-04-22T18:40:32.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.519 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:32.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:32.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:32.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:32.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240345+0000", 2024-04-22T18:40:32.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4300000000000001e-06 2024-04-22T18:40:32.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240349+0000", 2024-04-22T18:40:32.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2570000000000001e-06 2024-04-22T18:40:32.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240358+0000", 2024-04-22T18:40:32.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-22T18:40:32.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363697+0000", 2024-04-22T18:40:32.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12333965299999999 2024-04-22T18:40:32.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:32.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363777+0000", 2024-04-22T18:40:32.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9883999999999996e-05 2024-04-22T18:40:32.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648296+0000", 2024-04-22T18:40:32.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451861299999998 2024-04-22T18:40:32.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648379+0000", 2024-04-22T18:40:32.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3302999999999998e-05 2024-04-22T18:40:32.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648471+0000", 2024-04-22T18:40:32.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2029999999999998e-05 2024-04-22T18:40:32.523 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.523 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.523 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.523 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:32.523 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:32.523 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.243981020000007, 2024-04-22T18:40:32.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:40:32.523 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.523 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.524 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.524 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.524 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.524 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:40:32.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.524 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:32.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:32.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:40:32.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:32.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:40:32.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:40:32.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:40:32.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:40:32.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:40:32.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:40:32.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:40:32.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:40:32.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:40:32.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:40:32.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:32.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:40:32.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:40:32.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:40:32.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:40:32.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:40:32.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:40:32.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:40:32.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:40:32.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:40:32.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:40:32.528 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.528 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.528 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.529 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:32.529 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:32.529 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.239198590000001, 2024-04-22T18:40:32.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:40:32.529 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.529 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.529 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.529 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.529 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.529 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:40:32.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.529 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:32.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:32.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:40:32.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:32.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:40:32.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:40:32.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:40:32.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:40:32.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:40:32.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:40:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:40:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:40:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:40:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:40:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:40:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:40:32.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:40:32.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:40:32.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:40:32.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:40:32.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:40:32.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:40:32.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:40:32.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:40:32.534 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.534 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.534 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.534 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:32.534 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:32.535 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.239104386999998, 2024-04-22T18:40:32.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:40:32.535 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.535 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.535 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.535 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.535 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.535 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:40:32.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.535 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:32.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:32.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:40:32.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:32.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:40:32.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:40:32.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:40:32.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:40:32.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:40:32.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:40:32.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:40:32.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:40:32.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:40:32.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:40:32.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:32.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:40:32.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:40:32.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:40:32.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:40:32.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:40:32.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:40:32.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:40:32.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:40:32.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:40:32.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:40:32.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.540 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.540 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:32.540 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:32.540 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.239083499000003, 2024-04-22T18:40:32.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:40:32.540 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.541 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:32.541 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.541 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.541 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.541 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:40:32.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.541 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:32.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:32.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:40:32.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:32.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:40:32.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:40:32.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:40:32.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:40:32.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:40:32.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:40:32.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:40:32.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:40:32.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:32.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:40:32.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:40:32.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:40:32.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:40:32.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:40:32.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:40:32.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:40:32.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:40:32.545 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.545 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.545 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.545 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:32.545 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:32.545 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.456075835999997, 2024-04-22T18:40:32.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:40:32.545 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.546 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.546 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:40:32.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.546 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:32.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:32.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:40:32.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:32.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:40:32.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:40:32.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:40:32.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:40:32.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:40:32.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:40:32.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:40:32.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:40:32.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:40:32.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:40:32.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:32.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:40:32.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:40:32.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:40:32.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:40:32.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:40:32.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:40:32.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:40:32.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:40:32.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:40:32.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:40:32.550 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.551 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.551 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.551 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:32.551 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:32.551 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.448244430000003, 2024-04-22T18:40:32.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:40:32.551 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.551 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.551 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.551 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.551 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.552 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:40:32.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.552 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:32.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:32.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:40:32.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:32.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:40:32.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:40:32.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:40:32.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:40:32.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:40:32.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:40:32.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:40:32.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:40:32.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:40:32.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:40:32.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:32.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:40:32.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:40:32.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:40:32.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:40:32.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:40:32.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:40:32.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:40:32.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:40:32.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:40:32.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:40:32.556 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.556 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.556 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.557 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2052 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:32.557 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:32.557 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.926132649000003, 2024-04-22T18:40:32.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43058958600000002, 2024-04-22T18:40:32.557 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.557 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.557 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.557 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.557 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.557 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2052 2024-04-22T18:40:32.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.557 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:32.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:32.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577494+0000", 2024-04-22T18:40:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577500+0000", 2024-04-22T18:40:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3279999999999996e-06 2024-04-22T18:40:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577510+0000", 2024-04-22T18:40:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9450000000000005e-06 2024-04-22T18:40:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577515+0000", 2024-04-22T18:40:32.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7799999999999997e-06 2024-04-22T18:40:32.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677622+0000", 2024-04-22T18:40:32.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010654099999999 2024-04-22T18:40:32.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677710+0000", 2024-04-22T18:40:32.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8472000000000003e-05 2024-04-22T18:40:32.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:32.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.906619+0000", 2024-04-22T18:40:32.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.228908519 2024-04-22T18:40:32.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907343+0000", 2024-04-22T18:40:32.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072442600000000002 2024-04-22T18:40:32.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.007972+0000", 2024-04-22T18:40:32.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100628781 2024-04-22T18:40:32.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008022+0000", 2024-04-22T18:40:32.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9843999999999999e-05 2024-04-22T18:40:32.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008085+0000", 2024-04-22T18:40:32.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3256999999999994e-05 2024-04-22T18:40:32.562 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.562 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.562 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.562 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:32.562 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:32.563 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.296371800000003, 2024-04-22T18:40:32.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:40:32.563 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.563 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.563 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.563 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.563 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.563 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:40:32.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.563 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:32.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:32.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:40:32.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:32.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:40:32.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:40:32.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:40:32.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:40:32.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:40:32.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:40:32.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:40:32.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:40:32.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:40:32.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:40:32.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:32.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:40:32.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:40:32.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:40:32.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:40:32.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:40:32.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:40:32.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:40:32.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:40:32.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:40:32.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:40:32.568 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.568 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.568 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.568 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2505 4.5 4:a6d237d2:::smithi08937929-43:head [setxattr _header (56) in=7b,truncate 3529470] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:32.568 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:32.568 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.295979105000001, 2024-04-22T18:40:32.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43706744400000003, 2024-04-22T18:40:32.568 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.568 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.568 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.568 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2505 2024-04-22T18:40:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207648+0000", 2024-04-22T18:40:32.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:32.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207653+0000", 2024-04-22T18:40:32.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2989999999999998e-06 2024-04-22T18:40:32.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207656+0000", 2024-04-22T18:40:32.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0199999999999999e-06 2024-04-22T18:40:32.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207659+0000", 2024-04-22T18:40:32.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2270000000000002e-06 2024-04-22T18:40:32.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243213+0000", 2024-04-22T18:40:32.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035553653999999997 2024-04-22T18:40:32.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243273+0000", 2024-04-22T18:40:32.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0300000000000002e-05 2024-04-22T18:40:32.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:32.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243359+0000", 2024-04-22T18:40:32.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6275999999999999e-05 2024-04-22T18:40:32.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297443+0000", 2024-04-22T18:40:32.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.054083930000000002 2024-04-22T18:40:32.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644651+0000", 2024-04-22T18:40:32.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.347208181 2024-04-22T18:40:32.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644684+0000", 2024-04-22T18:40:32.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3080999999999997e-05 2024-04-22T18:40:32.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644717+0000", 2024-04-22T18:40:32.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2264000000000001e-05 2024-04-22T18:40:32.573 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.573 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.573 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.574 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:32.574 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:32.574 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.295966892000003, 2024-04-22T18:40:32.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:40:32.574 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.574 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:32.574 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.574 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.574 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.574 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:40:32.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.574 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:32.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:32.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:32.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:32.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:40:32.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:40:32.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:40:32.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:40:32.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:40:32.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:40:32.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:40:32.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:40:32.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:32.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:40:32.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:40:32.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3213 4.a 4:5d2aee44:::smithi08937929-13:head [assert-version v66,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.9350670040000004, 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69928701400000004, 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3213 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:40:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:40:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568559+0000", 2024-04-22T18:40:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568567+0000", 2024-04-22T18:40:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-04-22T18:40:32.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568579+0000", 2024-04-22T18:40:32.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-04-22T18:40:32.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.668711+0000", 2024-04-22T18:40:32.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100131866 2024-04-22T18:40:32.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.768945+0000", 2024-04-22T18:40:32.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100234252 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769002+0000", 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7092999999999997e-05 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.166404+0000", 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39740241999999998 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.191686+0000", 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025281739000000001 2024-04-22T18:40:32.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267758+0000", 2024-04-22T18:40:32.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076071606999999999 2024-04-22T18:40:32.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267812+0000", 2024-04-22T18:40:32.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4478999999999998e-05 2024-04-22T18:40:32.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267848+0000", 2024-04-22T18:40:32.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6062999999999997e-05 2024-04-22T18:40:32.583 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.584 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.584 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.584 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3218 4.a 4:519c08f7:::smithi08937929-15:head [write 1617736~710086] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:32.584 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.669206+0000", 2024-04-22T18:40:32.584 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.8344220129999993, 2024-04-22T18:40:32.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40462619, 2024-04-22T18:40:32.584 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.584 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.584 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.584 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.584 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.584 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3218 2024-04-22T18:40:32.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.585 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669206+0000", 2024-04-22T18:40:32.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669206+0000", 2024-04-22T18:40:32.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669206+0000", 2024-04-22T18:40:32.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:32.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669812+0000", 2024-04-22T18:40:32.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060657099999999998 2024-04-22T18:40:32.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669820+0000", 2024-04-22T18:40:32.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8560000000000007e-06 2024-04-22T18:40:32.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669831+0000", 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1131000000000001e-05 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.770325+0000", 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10049406499999999 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.770405+0000", 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9487000000000007e-05 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.770532+0000", 2024-04-22T18:40:32.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000126749 2024-04-22T18:40:32.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.772387+0000", 2024-04-22T18:40:32.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001855385 2024-04-22T18:40:32.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.073766+0000", 2024-04-22T18:40:32.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30137940299999999 2024-04-22T18:40:32.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.073808+0000", 2024-04-22T18:40:32.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1186000000000002e-05 2024-04-22T18:40:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.073832+0000", 2024-04-22T18:40:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4882999999999999e-05 2024-04-22T18:40:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3220 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.669853+0000", 2024-04-22T18:40:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.8337754099999994, 2024-04-22T18:40:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40421206599999998, 2024-04-22T18:40:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3220 2024-04-22T18:40:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669853+0000", 2024-04-22T18:40:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669853+0000", 2024-04-22T18:40:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669852+0000", 2024-04-22T18:40:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669854+0000", 2024-04-22T18:40:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.866e-06 2024-04-22T18:40:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669857+0000", 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769953+0000", 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009555000000001 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.772113+0000", 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002159771 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.772183+0000", 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0486000000000002e-05 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.073963+0000", 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30178019099999998 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.074009+0000", 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5794999999999998e-05 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.074065+0000", 2024-04-22T18:40:32.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5655000000000002e-05 2024-04-22T18:40:32.594 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.594 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.594 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.594 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3319 4.a 4:5d2aee44:::smithi08937929-13:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:32.594 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:40:32.594 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.0433192529999999, 2024-04-22T18:40:32.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476623401, 2024-04-22T18:40:32.594 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.594 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:32.594 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.594 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.594 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3319 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460306+0000", 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460318+0000", 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2896e-05 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460345+0000", 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6650000000000001e-05 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460360+0000", 2024-04-22T18:40:32.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4729000000000001e-05 2024-04-22T18:40:32.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460502+0000", 2024-04-22T18:40:32.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141729 2024-04-22T18:40:32.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460644+0000", 2024-04-22T18:40:32.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142755 2024-04-22T18:40:32.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.936932+0000", 2024-04-22T18:40:32.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476288078 2024-04-22T18:40:32.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.598 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.598 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3412 4.a 4:519c08f7:::smithi08937929-15:head [write 2529878~394573] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:32.598 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:40:32.598 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.3614846919999999, 2024-04-22T18:40:32.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48918094899999998, 2024-04-22T18:40:32.598 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.598 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.598 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.598 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.598 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.598 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3412 2024-04-22T18:40:32.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.599 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:40:32.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:40:32.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142141+0000", 2024-04-22T18:40:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142496+0000", 2024-04-22T18:40:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035444699999999998 2024-04-22T18:40:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142510+0000", 2024-04-22T18:40:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3841e-05 2024-04-22T18:40:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142518+0000", 2024-04-22T18:40:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8089999999999993e-06 2024-04-22T18:40:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142560+0000", 2024-04-22T18:40:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2097e-05 2024-04-22T18:40:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142637+0000", 2024-04-22T18:40:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6965000000000006e-05 2024-04-22T18:40:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142715+0000", 2024-04-22T18:40:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8694000000000002e-05 2024-04-22T18:40:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144179+0000", 2024-04-22T18:40:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146342 2024-04-22T18:40:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631229+0000", 2024-04-22T18:40:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48705018500000002 2024-04-22T18:40:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631281+0000", 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2216999999999998e-05 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631324+0000", 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3344999999999999e-05 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3413 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2924451] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.3611035519999999, 2024-04-22T18:40:32.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58904086899999997, 2024-04-22T18:40:32.604 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.604 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:32.604 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.604 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.604 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.604 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3413 2024-04-22T18:40:32.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.604 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:40:32.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:40:32.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142523+0000", 2024-04-22T18:40:32.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:32.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142529+0000", 2024-04-22T18:40:32.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6049999999999997e-06 2024-04-22T18:40:32.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142535+0000", 2024-04-22T18:40:32.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4740000000000002e-06 2024-04-22T18:40:32.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142541+0000", 2024-04-22T18:40:32.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1839999999999998e-06 2024-04-22T18:40:32.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143553+0000", 2024-04-22T18:40:32.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010126689999999999 2024-04-22T18:40:32.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143659+0000", 2024-04-22T18:40:32.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105393 2024-04-22T18:40:32.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:32.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143825+0000", 2024-04-22T18:40:32.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165809 2024-04-22T18:40:32.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:32.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144329+0000", 2024-04-22T18:40:32.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050483799999999997 2024-04-22T18:40:32.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:32.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731480+0000", 2024-04-22T18:40:32.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58715100600000003 2024-04-22T18:40:32.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:32.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731522+0000", 2024-04-22T18:40:32.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2098000000000002e-05 2024-04-22T18:40:32.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731566+0000", 2024-04-22T18:40:32.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3066999999999999e-05 2024-04-22T18:40:32.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.608 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.609 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.609 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3414 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:32.609 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:40:32.609 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 1.361074948, 2024-04-22T18:40:32.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58926020199999996, 2024-04-22T18:40:32.609 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:32.609 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:32.609 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:32.609 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:32.609 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:32.609 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3414 2024-04-22T18:40:32.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.609 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:32.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:32.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:40:32.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:32.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:40:32.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:32.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:32.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142552+0000", 2024-04-22T18:40:32.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:32.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142556+0000", 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0289999999999999e-06 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142574+0000", 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7266999999999999e-05 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142581+0000", 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144121+0000", 2024-04-22T18:40:32.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015397480000000001 2024-04-22T18:40:32.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:32.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144170+0000", 2024-04-22T18:40:32.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8803e-05 2024-04-22T18:40:32.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:32.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731624+0000", 2024-04-22T18:40:32.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58745365800000005 2024-04-22T18:40:32.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:32.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731712+0000", 2024-04-22T18:40:32.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8016999999999994e-05 2024-04-22T18:40:32.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:32.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:32.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:32.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731813+0000", 2024-04-22T18:40:32.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101817 2024-04-22T18:40:32.613 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.613 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.613 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.613 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:32.613 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:32.613 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:32.614 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:40:32.616 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:32.616 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:32.616+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:32.627 INFO:tasks.rados.rados.0.smithi089.stdout:1483: finishing write tid 2 to smithi08937929-15 2024-04-22T18:40:32.627 INFO:tasks.rados.rados.0.smithi089.stdout:1480: got (0) Success 2024-04-22T18:40:32.627 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 0 (ObjNum 287 snap 0 seq_num 287) dirty exists 2024-04-22T18:40:32.627 INFO:tasks.rados.rados.0.smithi089.stdout:1484: finishing set_chunk to oid 8 2024-04-22T18:40:32.628 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 22 (ObjNum 253 snap 0 seq_num 253) dirty exists 2024-04-22T18:40:32.628 INFO:tasks.rados.rados.0.smithi089.stdout:1483: finishing write tid 3 to smithi08937929-15 2024-04-22T18:40:32.628 INFO:tasks.rados.rados.0.smithi089.stdout:1472: done (15 left) 2024-04-22T18:40:32.628 INFO:tasks.rados.rados.0.smithi089.stdout:1475: done (14 left) 2024-04-22T18:40:32.628 INFO:tasks.rados.rados.0.smithi089.stdout:1480: done (13 left) 2024-04-22T18:40:32.628 INFO:tasks.rados.rados.0.smithi089.stdout:1481: done (12 left) 2024-04-22T18:40:32.628 INFO:tasks.rados.rados.0.smithi089.stdout:1484: done (11 left) 2024-04-22T18:40:32.628 INFO:tasks.rados.rados.0.smithi089.stdout:1485: done (10 left) 2024-04-22T18:40:32.628 INFO:tasks.rados.rados.0.smithi089.stdout:1486: done (9 left) 2024-04-22T18:40:32.628 INFO:tasks.rados.rados.0.smithi089.stdout:1487: done (8 left) 2024-04-22T18:40:32.628 INFO:tasks.rados.rados.0.smithi089.stdout:1495: read oid 42 snap -1 2024-04-22T18:40:32.628 INFO:tasks.rados.rados.0.smithi089.stdout:1495: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:32.628 INFO:tasks.rados.rados.0.smithi089.stdout:1496: write oid 32 current snap is 0 2024-04-22T18:40:32.628 INFO:tasks.rados.rados.0.smithi089.stdout:1483: finishing write tid 4 to smithi08937929-15 2024-04-22T18:40:32.628 INFO:tasks.rados.rados.0.smithi089.stdout:1483: finishing write tid 5 to smithi08937929-15 2024-04-22T18:40:32.629 INFO:tasks.rados.rados.0.smithi089.stdout:1483: oid 15 updating version 0 to 84 2024-04-22T18:40:32.629 INFO:tasks.rados.rados.0.smithi089.stdout:1483: oid 15 updating version 84 to 85 2024-04-22T18:40:32.629 INFO:tasks.rados.rados.0.smithi089.stdout:1483: oid 15 version 85 is already newer than 83 2024-04-22T18:40:32.629 INFO:tasks.rados.rados.0.smithi089.stdout:1483: oid 15 version 85 is already newer than 82 2024-04-22T18:40:32.629 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 85 (ObjNum 293 snap 0 seq_num 293) dirty exists 2024-04-22T18:40:32.629 INFO:tasks.rados.rados.0.smithi089.stdout:1483: left oid 15 (ObjNum 293 snap 0 seq_num 293) 2024-04-22T18:40:32.629 INFO:tasks.rados.rados.0.smithi089.stdout:1496: seq_num 296 ranges {574872=695118,2038317=630423} 2024-04-22T18:40:32.633 INFO:tasks.rados.rados.0.smithi089.stdout:1496: writing smithi08937929-32 from 574872 to 1269990 tid 1 2024-04-22T18:40:32.641 INFO:tasks.rados.rados.0.smithi089.stdout:1496: writing smithi08937929-32 from 2038317 to 2668740 tid 2 2024-04-22T18:40:32.641 INFO:tasks.rados.rados.0.smithi089.stdout:1483: done (9 left) 2024-04-22T18:40:32.641 INFO:tasks.rados.rados.0.smithi089.stdout:1497: read oid 20 snap -1 2024-04-22T18:40:32.641 INFO:tasks.rados.rados.0.smithi089.stdout:1497: expect (ObjNum 201 snap 0 seq_num 201) 2024-04-22T18:40:32.642 INFO:tasks.rados.rados.0.smithi089.stdout:1498: set_chunk oid 10 target oid 2024-04-22T18:40:32.716 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:32.717+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:32.752 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:32.753 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:32.760 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:32.760 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:32.760 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:32.760 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:32.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:32.770 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:32.770 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:32.785 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 6, 7, 2] out_osds: [5, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:40:32.785 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:40:32.785 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 3, 6, 7, 2] 2024-04-22T18:40:32.785 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2024-04-22T18:40:32.786 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:32.787 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:32.786+0000 7fa768d81640 -1 osd.5 101 handle_osd_map injecting map crc failure 2024-04-22T18:40:32.787 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:32.789 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:32.789+0000 7f8605f32640 -1 osd.3 101 handle_osd_map injecting map crc failure 2024-04-22T18:40:32.803 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:32.803 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:32.816 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:32.816+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:32.820 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:32.820 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:32.839 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:32.839+0000 7f1e2dd22640 -1 osd.2 101 handle_osd_map injecting map crc failure 2024-04-22T18:40:32.889 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:32.889 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:32.889 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:32.889 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:32.889 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:32.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:32.904 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:32.905 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:32.921 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:32.921 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:32.937 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:32.938 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:32.941 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:32.942+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:32.999 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:32.999 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:33.008 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:33.049 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:33.050+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:33.094 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-22T18:40:33.094+0000 7f9b4bbc8640 -1 mon.a@0(leader).osd e102 definitely_dead 0 2024-04-22T18:40:33.112 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:33.112 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:33.117 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:33.118+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:33.121 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:33.218 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:33.218+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:33.263 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:33.263 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:33.272 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:33.343 INFO:tasks.rados.rados.0.smithi089.stdout:1498: set_chunk oid 10 offset: 1258496 length: 513024 target oid 7a0ac4c3b70f870122151f53ca9cd62898d26d5c871013e8b0d7846ed36f71bc offset: 0 snap -1 2024-04-22T18:40:33.343 INFO:tasks.rados.rados.0.smithi089.stdout:1499: read oid 18 snap -1 2024-04-22T18:40:33.343 INFO:tasks.rados.rados.0.smithi089.stdout:1499: expect (ObjNum 275 snap 0 seq_num 275) 2024-04-22T18:40:33.343 INFO:tasks.rados.rados.0.smithi089.stdout:1500: set_chunk oid 31 target oid 2024-04-22T18:40:33.343 INFO:tasks.rados.rados.0.smithi089.stdout:1500: set_chunk oid 31 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:40:33.343 INFO:tasks.rados.rados.0.smithi089.stdout:1501: write (excl) oid 41 current snap is 0 2024-04-22T18:40:33.343 INFO:tasks.rados.rados.0.smithi089.stdout:1489: expect (ObjNum 286 snap 0 seq_num 286) 2024-04-22T18:40:33.343 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:33.343+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:33.415 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:33.415 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:33.419 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:33.420+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:33.424 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:33.426 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:33.427+0000 7f7f44bb3640 -1 osd.4 101 handle_osd_map injecting map crc failure 2024-04-22T18:40:33.498 INFO:tasks.rados.rados.0.smithi089.stdout:1488: finishing write tid 1 to smithi08937929-22 2024-04-22T18:40:33.499 INFO:tasks.rados.rados.0.smithi089.stdout:1494: expect (ObjNum 215 snap 0 seq_num 215) 2024-04-22T18:40:33.521 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:33.522+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:33.568 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:33.568 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:33.578 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:33.621 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:33.622+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:33.709 INFO:tasks.rados.rados.0.smithi089.stdout:1488: finishing write tid 2 to smithi08937929-22 2024-04-22T18:40:33.710 INFO:tasks.rados.rados.0.smithi089.stdout:1488: finishing write tid 3 to smithi08937929-22 2024-04-22T18:40:33.710 INFO:tasks.rados.rados.0.smithi089.stdout:1488: finishing write tid 4 to smithi08937929-22 2024-04-22T18:40:33.710 INFO:tasks.rados.rados.0.smithi089.stdout:1488: finishing write tid 5 to smithi08937929-22 2024-04-22T18:40:33.710 INFO:tasks.rados.rados.0.smithi089.stdout:1488: oid 22 updating version 0 to 107 2024-04-22T18:40:33.710 INFO:tasks.rados.rados.0.smithi089.stdout:1488: oid 22 updating version 107 to 108 2024-04-22T18:40:33.710 INFO:tasks.rados.rados.0.smithi089.stdout:1488: oid 22 version 108 is already newer than 106 2024-04-22T18:40:33.710 INFO:tasks.rados.rados.0.smithi089.stdout:1488: oid 22 updating version 108 to 109 2024-04-22T18:40:33.710 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 109 (ObjNum 294 snap 0 seq_num 294) dirty exists 2024-04-22T18:40:33.710 INFO:tasks.rados.rados.0.smithi089.stdout:1488: left oid 22 (ObjNum 294 snap 0 seq_num 294) 2024-04-22T18:40:33.710 INFO:tasks.rados.rados.0.smithi089.stdout:1491: expect (ObjNum 251 snap 0 seq_num 251) 2024-04-22T18:40:33.720 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:33.720 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:33.722 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:33.723+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:33.729 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:33.796 INFO:teuthology.orchestra.run.smithi089.stderr:marked out osd.0. 2024-04-22T18:40:33.799 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:40:33.823 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:33.824+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:33.838 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:33.838 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:33.847 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:33.916 INFO:tasks.rados.rados.0.smithi089.stdout:1493: expect (ObjNum 262 snap 0 seq_num 262) 2024-04-22T18:40:33.923 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:33.924+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:33.993 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:33.993 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:34.003 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:34.025 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:34.026+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:34.064 INFO:tasks.rados.rados.0.smithi089.stdout:1492: finishing write tid 1 to smithi08937929-29 2024-04-22T18:40:34.064 INFO:tasks.rados.rados.0.smithi089.stdout:1492: finishing write tid 2 to smithi08937929-29 2024-04-22T18:40:34.064 INFO:tasks.rados.rados.0.smithi089.stdout:1492: finishing write tid 3 to smithi08937929-29 2024-04-22T18:40:34.064 INFO:tasks.rados.rados.0.smithi089.stdout:1492: finishing write tid 4 to smithi08937929-29 2024-04-22T18:40:34.064 INFO:tasks.rados.rados.0.smithi089.stdout:1492: finishing write tid 5 to smithi08937929-29 2024-04-22T18:40:34.064 INFO:tasks.rados.rados.0.smithi089.stdout:1492: oid 29 updating version 0 to 121 2024-04-22T18:40:34.064 INFO:tasks.rados.rados.0.smithi089.stdout:1492: oid 29 updating version 121 to 122 2024-04-22T18:40:34.065 INFO:tasks.rados.rados.0.smithi089.stdout:1492: oid 29 version 122 is already newer than 120 2024-04-22T18:40:34.065 INFO:tasks.rados.rados.0.smithi089.stdout:1492: oid 29 version 122 is already newer than 119 2024-04-22T18:40:34.065 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 122 (ObjNum 295 snap 0 seq_num 295) dirty exists 2024-04-22T18:40:34.065 INFO:tasks.rados.rados.0.smithi089.stdout:1492: left oid 29 (ObjNum 295 snap 0 seq_num 295) 2024-04-22T18:40:34.065 INFO:tasks.rados.rados.0.smithi089.stdout:1501: seq_num 297 ranges {591679=486720,1813152=490434} 2024-04-22T18:40:34.070 INFO:tasks.rados.rados.0.smithi089.stdout:1501: writing smithi08937929-41 from 591679 to 1078399 tid 1 2024-04-22T18:40:34.077 INFO:tasks.rados.rados.0.smithi089.stdout:1501: writing smithi08937929-41 from 1813152 to 2303586 tid 2 2024-04-22T18:40:34.077 INFO:tasks.rados.rados.0.smithi089.stdout:1488: done (13 left) 2024-04-22T18:40:34.077 INFO:tasks.rados.rados.0.smithi089.stdout:1489: done (12 left) 2024-04-22T18:40:34.077 INFO:tasks.rados.rados.0.smithi089.stdout:1491: done (11 left) 2024-04-22T18:40:34.077 INFO:tasks.rados.rados.0.smithi089.stdout:1492: done (10 left) 2024-04-22T18:40:34.077 INFO:tasks.rados.rados.0.smithi089.stdout:1493: done (9 left) 2024-04-22T18:40:34.077 INFO:tasks.rados.rados.0.smithi089.stdout:1494: done (8 left) 2024-04-22T18:40:34.077 INFO:tasks.rados.rados.0.smithi089.stdout:1502: read oid 4 snap -1 2024-04-22T18:40:34.078 INFO:tasks.rados.rados.0.smithi089.stdout:1502: expect (ObjNum 269 snap 0 seq_num 269) 2024-04-22T18:40:34.078 INFO:tasks.rados.rados.0.smithi089.stdout:1503: write (excl) oid 47 current snap is 0 2024-04-22T18:40:34.078 INFO:tasks.rados.rados.0.smithi089.stdout:1497: expect (ObjNum 201 snap 0 seq_num 201) 2024-04-22T18:40:34.126 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:34.127+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:34.148 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:34.148 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:34.157 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:34.241 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:34.242+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:34.270 INFO:tasks.rados.rados.0.smithi089.stdout:1495: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:34.301 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:34.301 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:34.310 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:34.351 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:34.351+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:34.429 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:34.429+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:34.453 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:34.453 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:34.462 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:34.497 INFO:tasks.rados.rados.0.smithi089.stdout:1496: finishing write tid 1 to smithi08937929-32 2024-04-22T18:40:34.497 INFO:tasks.rados.rados.0.smithi089.stdout:1496: finishing write tid 2 to smithi08937929-32 2024-04-22T18:40:34.497 INFO:tasks.rados.rados.0.smithi089.stdout:1503: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2024-04-22T18:40:34.503 INFO:tasks.rados.rados.0.smithi089.stdout:1503: writing smithi08937929-47 from 608486 to 1170455 tid 1 2024-04-22T18:40:34.511 INFO:tasks.rados.rados.0.smithi089.stdout:1503: writing smithi08937929-47 from 1871634 to 2505726 tid 2 2024-04-22T18:40:34.514 INFO:tasks.rados.rados.0.smithi089.stdout:1503: writing smithi08937929-47 from 2950640 to 3008486 tid 3 2024-04-22T18:40:34.514 INFO:tasks.rados.rados.0.smithi089.stdout:1495: done (9 left) 2024-04-22T18:40:34.514 INFO:tasks.rados.rados.0.smithi089.stdout:1497: done (8 left) 2024-04-22T18:40:34.514 INFO:tasks.rados.rados.0.smithi089.stdout:1504: write (excl) oid 49 current snap is 0 2024-04-22T18:40:34.514 INFO:tasks.rados.rados.0.smithi089.stdout:1496: finishing write tid 3 to smithi08937929-32 2024-04-22T18:40:34.514 INFO:tasks.rados.rados.0.smithi089.stdout:1504: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2024-04-22T18:40:34.520 INFO:tasks.rados.rados.0.smithi089.stdout:1504: writing smithi08937929-49 from 625293 to 1262511 tid 1 2024-04-22T18:40:34.526 INFO:tasks.rados.rados.0.smithi089.stdout:1504: writing smithi08937929-49 from 1813763 to 2307866 tid 2 2024-04-22T18:40:34.530 INFO:tasks.rados.rados.0.smithi089.stdout:1504: writing smithi08937929-49 from 2978063 to 3025293 tid 3 2024-04-22T18:40:34.530 INFO:tasks.rados.rados.0.smithi089.stdout:1505: read oid 21 snap -1 2024-04-22T18:40:34.531 INFO:tasks.rados.rados.0.smithi089.stdout:1505: expect deleted 2024-04-22T18:40:34.531 INFO:tasks.rados.rados.0.smithi089.stdout:1506: tier_promote oid 40 2024-04-22T18:40:34.531 INFO:tasks.rados.rados.0.smithi089.stdout:1496: finishing write tid 4 to smithi08937929-32 2024-04-22T18:40:34.531 INFO:tasks.rados.rados.0.smithi089.stdout:1496: oid 32 updating version 0 to 72 2024-04-22T18:40:34.531 INFO:tasks.rados.rados.0.smithi089.stdout:1496: oid 32 updating version 72 to 74 2024-04-22T18:40:34.531 INFO:tasks.rados.rados.0.smithi089.stdout:1496: oid 32 version 74 is already newer than 73 2024-04-22T18:40:34.531 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 74 (ObjNum 296 snap 0 seq_num 296) dirty exists 2024-04-22T18:40:34.531 INFO:tasks.rados.rados.0.smithi089.stdout:1496: left oid 32 (ObjNum 296 snap 0 seq_num 296) 2024-04-22T18:40:34.531 INFO:tasks.rados.rados.0.smithi089.stdout:1490: expect (ObjNum 268 snap 0 seq_num 268) 2024-04-22T18:40:34.531 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:34.530+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:34.599 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:34.599 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:34.609 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:34.650 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:34.650+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:34.696 INFO:tasks.rados.rados.0.smithi089.stdout:1500: finishing set_chunk to oid 31 2024-04-22T18:40:34.696 INFO:tasks.rados.rados.0.smithi089.stdout:1500: got expected ENOENT (src dne) 2024-04-22T18:40:34.697 INFO:tasks.rados.rados.0.smithi089.stdout:1499: expect (ObjNum 275 snap 0 seq_num 275) 2024-04-22T18:40:34.733 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:34.734+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:34.751 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:34.751 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:34.761 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:34.802 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:34.802+0000 7f3e7be88640 -1 osd.0 103 handle_osd_map injecting map crc failure 2024-04-22T18:40:34.803 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:40:34.803 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:34.803+0000 7f3093031640 -1 osd.1 103 handle_osd_map injecting map crc failure 2024-04-22T18:40:34.834 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:34.834+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:34.862 INFO:tasks.rados.rados.0.smithi089.stdout:1498: finishing set_chunk to oid 10 2024-04-22T18:40:34.862 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 37 (ObjNum 292 snap 0 seq_num 292) dirty exists 2024-04-22T18:40:34.862 INFO:tasks.rados.rados.0.smithi089.stdout:1490: done (10 left) 2024-04-22T18:40:34.863 INFO:tasks.rados.rados.0.smithi089.stdout:1496: done (9 left) 2024-04-22T18:40:34.863 INFO:tasks.rados.rados.0.smithi089.stdout:1498: done (8 left) 2024-04-22T18:40:34.863 INFO:tasks.rados.rados.0.smithi089.stdout:1499: done (7 left) 2024-04-22T18:40:34.863 INFO:tasks.rados.rados.0.smithi089.stdout:1500: done (6 left) 2024-04-22T18:40:34.863 INFO:tasks.rados.rados.0.smithi089.stdout:1507: read oid 5 snap -1 2024-04-22T18:40:34.863 INFO:tasks.rados.rados.0.smithi089.stdout:1507: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:40:34.863 INFO:tasks.rados.rados.0.smithi089.stdout:1508: delete oid 13 current snap is 0 2024-04-22T18:40:34.863 INFO:tasks.rados.rados.0.smithi089.stdout:1501: finishing write tid 1 to smithi08937929-41 2024-04-22T18:40:34.863 INFO:tasks.rados.rados.0.smithi089.stdout:1501: finishing write tid 2 to smithi08937929-41 2024-04-22T18:40:34.863 INFO:tasks.rados.rados.0.smithi089.stdout:1501: finishing write tid 3 to smithi08937929-41 2024-04-22T18:40:34.863 INFO:tasks.rados.rados.0.smithi089.stdout:1501: finishing write tid 4 to smithi08937929-41 2024-04-22T18:40:34.863 INFO:tasks.rados.rados.0.smithi089.stdout:1501: oid 41 updating version 0 to 122 2024-04-22T18:40:34.863 INFO:tasks.rados.rados.0.smithi089.stdout:1501: oid 41 version 122 is already newer than 121 2024-04-22T18:40:34.863 INFO:tasks.rados.rados.0.smithi089.stdout:1501: oid 41 updating version 122 to 123 2024-04-22T18:40:34.864 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 123 (ObjNum 297 snap 0 seq_num 297) dirty exists 2024-04-22T18:40:34.864 INFO:tasks.rados.rados.0.smithi089.stdout:1501: left oid 41 (ObjNum 297 snap 0 seq_num 297) 2024-04-22T18:40:34.864 INFO:tasks.rados.rados.0.smithi089.stdout:1502: expect (ObjNum 269 snap 0 seq_num 269) 2024-04-22T18:40:34.875 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:34.876 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:34.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:34.934 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:34.935+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:34.953 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:40:34.989 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:34.989 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:34.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:35.032 INFO:tasks.rados.rados.0.smithi089.stdout:1503: finishing write tid 1 to smithi08937929-47 2024-04-22T18:40:35.032 INFO:tasks.rados.rados.0.smithi089.stdout:1503: finishing write tid 2 to smithi08937929-47 2024-04-22T18:40:35.032 INFO:tasks.rados.rados.0.smithi089.stdout:1503: finishing write tid 3 to smithi08937929-47 2024-04-22T18:40:35.032 INFO:tasks.rados.rados.0.smithi089.stdout:1503: finishing write tid 4 to smithi08937929-47 2024-04-22T18:40:35.032 INFO:tasks.rados.rados.0.smithi089.stdout:1503: finishing write tid 5 to smithi08937929-47 2024-04-22T18:40:35.032 INFO:tasks.rados.rados.0.smithi089.stdout:1503: oid 47 updating version 0 to 86 2024-04-22T18:40:35.032 INFO:tasks.rados.rados.0.smithi089.stdout:1503: oid 47 updating version 86 to 88 2024-04-22T18:40:35.032 INFO:tasks.rados.rados.0.smithi089.stdout:1503: oid 47 version 88 is already newer than 87 2024-04-22T18:40:35.032 INFO:tasks.rados.rados.0.smithi089.stdout:1503: oid 47 updating version 88 to 89 2024-04-22T18:40:35.032 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 89 (ObjNum 298 snap 0 seq_num 298) dirty exists 2024-04-22T18:40:35.032 INFO:tasks.rados.rados.0.smithi089.stdout:1503: left oid 47 (ObjNum 298 snap 0 seq_num 298) 2024-04-22T18:40:35.039 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:35.040+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:35.069 INFO:tasks.rados.rados.0.smithi089.stdout:1507: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:40:35.073 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:35.073 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:35.089 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:35.090 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:35.105 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:35.105 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:35.123 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:35.123 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:35.136 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:35.136+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:35.139 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:35.139 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:35.144 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:35.144 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:35.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:35.144 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:35.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.144 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:35.144 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:35.145 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 100.982003267, 2024-04-22T18:40:35.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:40:35.145 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.145 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.145 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.145 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.145 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.145 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:40:35.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.145 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:35.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:35.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:40:35.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:35.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:40:35.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:40:35.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:40:35.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:40:35.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:40:35.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:40:35.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:40:35.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:40:35.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:40:35.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:40:35.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:35.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:40:35.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:40:35.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:40:35.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:40:35.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:40:35.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:40:35.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:40:35.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:40:35.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:40:35.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:40:35.150 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.150 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.150 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.150 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:35.150 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:35.150 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 100.880444446, 2024-04-22T18:40:35.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:40:35.150 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.150 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.151 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.151 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.151 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.151 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:40:35.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.151 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:35.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:40:35.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:40:35.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:40:35.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:40:35.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:40:35.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:40:35.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:40:35.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:40:35.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:35.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:40:35.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:40:35.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:40:35.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:40:35.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:40:35.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:40:35.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:40:35.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:40:35.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:40:35.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:40:35.155 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.155 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.156 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.156 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:35.156 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:35.156 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 100.880418044, 2024-04-22T18:40:35.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:40:35.156 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.156 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:35.156 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.156 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.156 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.156 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:40:35.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.156 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:35.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:35.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:40:35.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:40:35.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:40:35.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:40:35.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:40:35.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:40:35.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:40:35.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:40:35.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:40:35.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:35.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:40:35.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:40:35.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:40:35.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:40:35.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:40:35.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:40:35.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:40:35.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:40:35.160 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.161 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.161 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.161 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:35.161 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:35.161 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 100.482083715, 2024-04-22T18:40:35.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:40:35.161 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.161 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.161 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.161 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.161 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.161 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:40:35.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.162 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:35.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:35.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:40:35.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:40:35.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:40:35.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:40:35.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:40:35.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:40:35.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:35.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:40:35.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:40:35.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:40:35.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:40:35.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:40:35.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:40:35.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:40:35.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:40:35.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:40:35.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:40:35.166 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.166 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.166 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.166 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:35.166 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:35.166 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 100.381247417, 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:35.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:40:35.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:35.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:40:35.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:40:35.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:40:35.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:40:35.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:40:35.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:40:35.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:40:35.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:40:35.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:40:35.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:40:35.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:35.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:40:35.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:40:35.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:40:35.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:40:35.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:40:35.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:40:35.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:40:35.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:40:35.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:40:35.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:40:35.172 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.172 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.172 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.172 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:35.172 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:35.172 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 100.38109014699999, 2024-04-22T18:40:35.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:40:35.172 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.172 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.172 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.172 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.173 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.173 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:40:35.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.173 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:35.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:35.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:40:35.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:40:35.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:40:35.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:40:35.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:40:35.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:40:35.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:40:35.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:40:35.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:40:35.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:40:35.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:40:35.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:35.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:40:35.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:40:35.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:40:35.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:40:35.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:40:35.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:40:35.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:40:35.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:40:35.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:40:35.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:40:35.177 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.177 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.177 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.178 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:35.178 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:35.178 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 100.381075538, 2024-04-22T18:40:35.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:40:35.178 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.178 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:35.178 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.178 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.178 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.178 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:40:35.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.178 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:35.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:35.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:35.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:40:35.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:40:35.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:35.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:40:35.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:40:35.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:40:35.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:40:35.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:40:35.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:40:35.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:40:35.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:40:35.182 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.182 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.182 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.182 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:35.183 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:35.183 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 100.38106515299999, 2024-04-22T18:40:35.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:40:35.183 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.183 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.183 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.183 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.183 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.183 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:40:35.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.183 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:40:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:40:35.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:40:35.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:40:35.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:40:35.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:40:35.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:40:35.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:40:35.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:40:35.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:40:35.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:40:35.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:35.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:40:35.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:40:35.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:40:35.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:40:35.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:40:35.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:40:35.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:40:35.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:40:35.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:40:35.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:40:35.188 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.188 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.188 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.188 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:35.188 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:35.188 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 100.380135223, 2024-04-22T18:40:35.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:40:35.189 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.189 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.189 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.189 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.189 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.189 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:40:35.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.189 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:35.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:35.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:40:35.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:35.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:40:35.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:40:35.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:40:35.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:40:35.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:40:35.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:40:35.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:40:35.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:40:35.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:40:35.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:40:35.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:35.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:40:35.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:40:35.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:40:35.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:40:35.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:40:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:40:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:40:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:40:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:40:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:40:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.193 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 100.379673098, 2024-04-22T18:40:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:40:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:40:35.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.195 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:35.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:35.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:40:35.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:35.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:40:35.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:40:35.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:40:35.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:40:35.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:40:35.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:40:35.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:40:35.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:40:35.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:40:35.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:40:35.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:35.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:40:35.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:40:35.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:40:35.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:40:35.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:40:35.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:40:35.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:40:35.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:40:35.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:40:35.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:40:35.199 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.199 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.199 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.199 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:35.199 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:35.200 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 100.379660294, 2024-04-22T18:40:35.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:40:35.200 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.200 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:35.200 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.200 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.200 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.200 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:40:35.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.200 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:35.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:35.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:40:35.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:40:35.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:40:35.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:40:35.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:40:35.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:40:35.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:40:35.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:40:35.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.051887837999999, 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.204 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.205 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.205 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.205 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.205 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:40:35.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.205 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:35.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:35.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:40:35.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:35.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:40:35.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:40:35.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:40:35.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:40:35.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:40:35.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:40:35.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:40:35.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:40:35.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.210 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:40:35.210 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:35.210 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.043314726000006, 2024-04-22T18:40:35.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:40:35.210 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.210 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.210 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.210 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.210 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.210 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:40:35.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.210 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:35.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:35.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:40:35.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:35.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:40:35.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:40:35.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:40:35.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:40:35.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:40:35.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:40:35.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:40:35.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:40:35.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:40:35.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:40:35.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:40:35.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:40:35.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:40:35.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:40:35.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.215 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.215 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:40:35.215 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:35.215 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.008667211000002, 2024-04-22T18:40:35.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:40:35.215 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.215 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:35.215 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.215 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.216 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:40:35.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.216 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:35.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:35.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:40:35.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:40:35.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:40:35.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:40:35.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:40:35.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:40:35.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:40:35.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:40:35.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:40:35.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:40:35.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:40:35.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:40:35.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:40:35.219 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.219 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.219 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.219 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1684 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:35.219 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:35.219 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.539183248000001, 2024-04-22T18:40:35.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987885199999998, 2024-04-22T18:40:35.219 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.220 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:35.220 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.220 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.220 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.220 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1684 2024-04-22T18:40:35.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.220 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:35.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:35.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605278+0000", 2024-04-22T18:40:35.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605280+0000", 2024-04-22T18:40:35.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1770000000000001e-06 2024-04-22T18:40:35.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605285+0000", 2024-04-22T18:40:35.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-04-22T18:40:35.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605289+0000", 2024-04-22T18:40:35.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.687e-06 2024-04-22T18:40:35.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705352+0000", 2024-04-22T18:40:35.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006322199999999 2024-04-22T18:40:35.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:35.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705423+0000", 2024-04-22T18:40:35.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1136999999999993e-05 2024-04-22T18:40:35.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105017+0000", 2024-04-22T18:40:35.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.399593268 2024-04-22T18:40:35.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105081+0000", 2024-04-22T18:40:35.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4703999999999996e-05 2024-04-22T18:40:35.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105157+0000", 2024-04-22T18:40:35.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6205999999999995e-05 2024-04-22T18:40:35.224 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.224 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.224 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.224 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2191 4.1 4.fdaa93c1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-04-22T18:40:35.224 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:35.224 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 49.594590998000001, 2024-04-22T18:40:35.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51064384399999996, 2024-04-22T18:40:35.224 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.224 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:35.224 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.224 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.225 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.225 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2191 2024-04-22T18:40:35.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.225 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:35.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:35.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:35.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549872+0000", 2024-04-22T18:40:35.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.387e-06 2024-04-22T18:40:35.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549875+0000", 2024-04-22T18:40:35.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:35.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549880+0000", 2024-04-22T18:40:35.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2529999999999998e-06 2024-04-22T18:40:35.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060496+0000", 2024-04-22T18:40:35.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51061597999999997 2024-04-22T18:40:35.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060515+0000", 2024-04-22T18:40:35.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9130000000000001e-05 2024-04-22T18:40:35.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.228 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.228 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2198 4.d 4:bc165c76:::smithi08937929-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:35.228 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:35.228 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 49.593323296999998, 2024-04-22T18:40:35.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50502289899999997, 2024-04-22T18:40:35.228 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.228 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:35.228 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.228 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.229 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.229 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2198 2024-04-22T18:40:35.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.229 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:35.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:35.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551138+0000", 2024-04-22T18:40:35.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:35.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551143+0000", 2024-04-22T18:40:35.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8949999999999997e-06 2024-04-22T18:40:35.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551151+0000", 2024-04-22T18:40:35.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8729999999999996e-06 2024-04-22T18:40:35.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551158+0000", 2024-04-22T18:40:35.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3099999999999997e-06 2024-04-22T18:40:35.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551180+0000", 2024-04-22T18:40:35.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2197000000000001e-05 2024-04-22T18:40:35.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:35.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551185+0000", 2024-04-22T18:40:35.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7010000000000004e-06 2024-04-22T18:40:35.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657133+0000", 2024-04-22T18:40:35.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105948498 2024-04-22T18:40:35.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:35.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657183+0000", 2024-04-22T18:40:35.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0124000000000003e-05 2024-04-22T18:40:35.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.055924+0000", 2024-04-22T18:40:35.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39874079299999998 2024-04-22T18:40:35.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056018+0000", 2024-04-22T18:40:35.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4032999999999999e-05 2024-04-22T18:40:35.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056161+0000", 2024-04-22T18:40:35.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143385 2024-04-22T18:40:35.233 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.233 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.233 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.234 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2212 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:35.234 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:35.234 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 49.593253654999998, 2024-04-22T18:40:35.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50989936400000002, 2024-04-22T18:40:35.234 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.234 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:35.234 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.234 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.234 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.234 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2212 2024-04-22T18:40:35.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.234 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:35.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:35.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:35.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551211+0000", 2024-04-22T18:40:35.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6280000000000001e-06 2024-04-22T18:40:35.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551214+0000", 2024-04-22T18:40:35.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1209999999999998e-06 2024-04-22T18:40:35.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551217+0000", 2024-04-22T18:40:35.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8490000000000002e-06 2024-04-22T18:40:35.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061086+0000", 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50986975099999998 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061108+0000", 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1319e-05 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2213 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:35.237 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:35.238 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 49.593243237999999, 2024-04-22T18:40:35.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50998486099999996, 2024-04-22T18:40:35.238 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.238 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:35.238 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.238 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.238 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.238 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2213 2024-04-22T18:40:35.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.238 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:35.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:35.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551218+0000", 2024-04-22T18:40:35.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551220+0000", 2024-04-22T18:40:35.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.319e-06 2024-04-22T18:40:35.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551223+0000", 2024-04-22T18:40:35.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9660000000000001e-06 2024-04-22T18:40:35.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551226+0000", 2024-04-22T18:40:35.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9050000000000001e-06 2024-04-22T18:40:35.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061162+0000", 2024-04-22T18:40:35.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.509936838 2024-04-22T18:40:35.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061204+0000", 2024-04-22T18:40:35.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1094999999999999e-05 2024-04-22T18:40:35.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.241 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.241 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2214 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:35.241 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:35.241 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 49.593234248999998, 2024-04-22T18:40:35.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50997458100000004, 2024-04-22T18:40:35.242 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.242 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:35.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.242 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2214 2024-04-22T18:40:35.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.242 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:35.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:35.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551227+0000", 2024-04-22T18:40:35.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551229+0000", 2024-04-22T18:40:35.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8649999999999999e-06 2024-04-22T18:40:35.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551232+0000", 2024-04-22T18:40:35.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:35.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551235+0000", 2024-04-22T18:40:35.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:40:35.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061194+0000", 2024-04-22T18:40:35.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50995819099999995 2024-04-22T18:40:35.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061202+0000", 2024-04-22T18:40:35.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.541e-06 2024-04-22T18:40:35.245 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.245 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.245 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.245 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.245 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.245 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:35.246 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:35.246 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:35.247 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:35.247 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:35.248 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:35.248 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:35.247+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:35.309 INFO:tasks.rados.rados.0.smithi089.stdout:1501: done (7 left) 2024-04-22T18:40:35.309 INFO:tasks.rados.rados.0.smithi089.stdout:1502: done (6 left) 2024-04-22T18:40:35.309 INFO:tasks.rados.rados.0.smithi089.stdout:1503: done (5 left) 2024-04-22T18:40:35.309 INFO:tasks.rados.rados.0.smithi089.stdout:1505: done (4 left) 2024-04-22T18:40:35.309 INFO:tasks.rados.rados.0.smithi089.stdout:1507: done (3 left) 2024-04-22T18:40:35.309 INFO:tasks.rados.rados.0.smithi089.stdout:1508: done (2 left) 2024-04-22T18:40:35.309 INFO:tasks.rados.rados.0.smithi089.stdout:1509: read oid 28 snap -1 2024-04-22T18:40:35.309 INFO:tasks.rados.rados.0.smithi089.stdout:1509: expect (ObjNum 265 snap 0 seq_num 265) 2024-04-22T18:40:35.309 INFO:tasks.rados.rados.0.smithi089.stdout:1510: read oid 26 snap -1 2024-04-22T18:40:35.309 INFO:tasks.rados.rados.0.smithi089.stdout:1510: expect (ObjNum 289 snap 0 seq_num 289) 2024-04-22T18:40:35.309 INFO:tasks.rados.rados.0.smithi089.stdout:1511: read oid 42 snap -1 2024-04-22T18:40:35.309 INFO:tasks.rados.rados.0.smithi089.stdout:1511: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:35.309 INFO:tasks.rados.rados.0.smithi089.stdout:1512: set_chunk oid 1 target oid 2024-04-22T18:40:35.310 INFO:tasks.rados.rados.0.smithi089.stdout:1512: set_chunk oid 1 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:40:35.310 INFO:tasks.rados.rados.0.smithi089.stdout:1513: tier_promote oid 4 2024-04-22T18:40:35.310 INFO:tasks.rados.rados.0.smithi089.stdout:1514: write (excl) oid 15 current snap is 0 2024-04-22T18:40:35.310 INFO:tasks.rados.rados.0.smithi089.stdout:1514: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2024-04-22T18:40:35.316 INFO:tasks.rados.rados.0.smithi089.stdout:1514: writing smithi08937929-15 from 658907 to 1446623 tid 1 2024-04-22T18:40:35.321 INFO:tasks.rados.rados.0.smithi089.stdout:1514: writing smithi08937929-15 from 1930727 to 2428499 tid 2 2024-04-22T18:40:35.325 INFO:tasks.rados.rados.0.smithi089.stdout:1514: writing smithi08937929-15 from 3032909 to 3058907 tid 3 2024-04-22T18:40:35.325 INFO:tasks.rados.rados.0.smithi089.stdout:1515: delete oid 7 current snap is 0 2024-04-22T18:40:35.327 INFO:tasks.rados.rados.0.smithi089.stdout:1515: done (8 left) 2024-04-22T18:40:35.327 INFO:tasks.rados.rados.0.smithi089.stdout:1516: read oid 2 snap -1 2024-04-22T18:40:35.327 INFO:tasks.rados.rados.0.smithi089.stdout:1516: expect (ObjNum 251 snap 0 seq_num 251) 2024-04-22T18:40:35.327 INFO:tasks.rados.rados.0.smithi089.stdout:1517: tier_promote oid 18 2024-04-22T18:40:35.327 INFO:tasks.rados.rados.0.smithi089.stdout:1518: read oid 5 snap -1 2024-04-22T18:40:35.327 INFO:tasks.rados.rados.0.smithi089.stdout:1518: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:40:35.327 INFO:tasks.rados.rados.0.smithi089.stdout:1519: read oid 46 snap -1 2024-04-22T18:40:35.327 INFO:tasks.rados.rados.0.smithi089.stdout:1519: expect (ObjNum 278 snap 0 seq_num 278) 2024-04-22T18:40:35.328 INFO:tasks.rados.rados.0.smithi089.stdout:1520: set_chunk oid 45 target oid 2024-04-22T18:40:35.346 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:35.347+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:35.351 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:35.352 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:35.352 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:35.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:35.430 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:35.430 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:35.482 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:35.482+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:35.506 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:35.506 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:35.506 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:35.506 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:35.506 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:35.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:35.549 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:35.550+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:35.618 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:35.618 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.524808769, 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:35.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:35.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:40:35.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:35.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:40:35.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:40:35.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:40:35.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:40:35.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:35.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:40:35.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:40:35.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:40:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:40:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:40:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:40:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:40:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:40:35.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.625310893999995, 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.624 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:35.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:35.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:40:35.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:35.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:40:35.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:40:35.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:40:35.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:40:35.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:40:35.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:40:35.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:40:35.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:40:35.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:35.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:40:35.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:40:35.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:40:35.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:40:35.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.629 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.630 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.630 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:40:35.630 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:35.630 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 96.517011103000002, 2024-04-22T18:40:35.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:40:35.630 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.630 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.630 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:40:35.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.631 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:35.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:35.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:40:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:40:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:40:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:40:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:40:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:40:35.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:40:35.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:40:35.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:40:35.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:40:35.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:40:35.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:35.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:40:35.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:40:35.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:40:35.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:40:35.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:40:35.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:40:35.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:40:35.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:40:35.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:40:35.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:40:35.635 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.635 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.635 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.635 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:35.635 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:35.636 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.185467474999996, 2024-04-22T18:40:35.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:40:35.636 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.636 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.636 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:35.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:35.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:40:35.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:40:35.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:40:35.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:40:35.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:40:35.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:40:35.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:40:35.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:40:35.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:40:35.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:40:35.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:40:35.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:40:35.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:40:35.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:40:35.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:40:35.640 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.640 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.640 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.641 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:35.641 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:35.641 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.094381666999993, 2024-04-22T18:40:35.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:40:35.641 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.641 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.641 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.641 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.641 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.641 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:40:35.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.641 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:35.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:35.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:40:35.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:35.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:40:35.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:40:35.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:40:35.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:40:35.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:40:35.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:40:35.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:40:35.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:40:35.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:40:35.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:40:35.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:35.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:40:35.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:40:35.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:40:35.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:40:35.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:40:35.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:40:35.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:40:35.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:40:35.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:40:35.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:40:35.646 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.646 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.646 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.646 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:35.646 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:35.647 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.088581407000007, 2024-04-22T18:40:35.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:40:35.647 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.647 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.647 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.647 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.647 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.647 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:40:35.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.647 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:35.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:40:35.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:40:35.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:40:35.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:40:35.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:40:35.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:40:35.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:40:35.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:40:35.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:40:35.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:35.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:40:35.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:40:35.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:40:35.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:40:35.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:40:35.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:40:35.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:40:35.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:40:35.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:40:35.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:40:35.652 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.652 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.652 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.652 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:35.652 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:35.652 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.083841063999998, 2024-04-22T18:40:35.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:40:35.652 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.652 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.652 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.652 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.652 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.652 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:40:35.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.653 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:35.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:35.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:40:35.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:35.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:40:35.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:40:35.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:40:35.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:40:35.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:40:35.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:40:35.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:40:35.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:40:35.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:40:35.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:40:35.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:35.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:40:35.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:40:35.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:40:35.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:40:35.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:40:35.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:40:35.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:40:35.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:40:35.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:40:35.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:40:35.657 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.657 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.657 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.658 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:35.658 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:35.658 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.083368166, 2024-04-22T18:40:35.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:40:35.658 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.658 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.658 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.658 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.658 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.658 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:40:35.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.658 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:35.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:35.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:40:35.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:35.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:40:35.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:40:35.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:40:35.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:40:35.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:40:35.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:40:35.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:40:35.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:40:35.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:40:35.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:40:35.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:40:35.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:40:35.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:40:35.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:40:35.663 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.663 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.663 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.663 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:35.663 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:35.663 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.083348244999996, 2024-04-22T18:40:35.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:40:35.663 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.663 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:35.664 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.664 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.664 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.664 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:40:35.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.664 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:35.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:40:35.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:40:35.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:40:35.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:40:35.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:40:35.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:40:35.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 77.774024479999994, 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:35.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:35.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:40:35.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:35.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:40:35.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:40:35.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:40:35.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:40:35.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:40:35.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:40:35.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:40:35.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:40:35.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:40:35.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:40:35.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:35.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:40:35.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:40:35.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:40:35.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:40:35.673 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.673 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.673 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.673 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:35.673 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:35.673 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 77.773816944000004, 2024-04-22T18:40:35.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:40:35.673 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.673 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.673 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:35.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:35.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:40:35.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:35.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:40:35.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:40:35.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:40:35.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:40:35.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:40:35.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:40:35.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:40:35.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:40:35.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:40:35.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:40:35.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:35.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:40:35.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:40:35.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:40:35.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:40:35.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:40:35.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:40:35.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:40:35.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:40:35.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:40:35.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:40:35.678 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.678 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.678 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.678 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:35.679 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:35.679 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 58.908167210999999, 2024-04-22T18:40:35.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:40:35.679 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.679 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.679 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:35.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:35.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:40:35.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:35.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:40:35.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:40:35.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:40:35.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:40:35.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:35.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:40:35.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:40:35.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:40:35.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:40:35.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:40:35.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:40:35.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:40:35.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:40:35.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:40:35.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:40:35.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.683 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.684 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.684 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:35.684 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:35.684 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 58.694328826000003, 2024-04-22T18:40:35.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:40:35.684 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.684 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.684 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:35.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:35.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:40:35.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:35.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:40:35.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:40:35.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:40:35.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:40:35.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:40:35.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:40:35.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:40:35.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:40:35.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:40:35.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:40:35.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:35.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:40:35.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:40:35.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:40:35.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:40:35.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:40:35.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:40:35.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:40:35.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:40:35.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:40:35.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:40:35.689 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.689 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.689 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.689 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:35.689 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:35.689 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 58.694295416000003, 2024-04-22T18:40:35.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:40:35.690 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.690 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.690 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:35.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:35.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:35.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:40:35.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:40:35.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:40:35.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:40:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:40:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:40:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:40:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:40:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:40:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:40:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:40:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:40:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:40:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:40:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:40:35.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:40:35.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:40:35.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:40:35.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:40:35.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:40:35.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.694 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 58.635412762000001, 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:35.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:40:35.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:35.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:40:35.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:40:35.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:40:35.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:40:35.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:35.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:40:35.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:40:35.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:40:35.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:40:35.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:40:35.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:40:35.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:40:35.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:40:35.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:40:35.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:40:35.700 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.700 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.700 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.700 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:35.700 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:35.700 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 58.635274146, 2024-04-22T18:40:35.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:40:35.700 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.700 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.700 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:35.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:35.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:35.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:40:35.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:40:35.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:40:35.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:40:35.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:40:35.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:40:35.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:40:35.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:40:35.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:40:35.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:40:35.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:35.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:40:35.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:40:35.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:40:35.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:40:35.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:40:35.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:40:35.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:40:35.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:40:35.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:40:35.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:40:35.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.705 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.705 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:35.705 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:35.705 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 58.635255143000002, 2024-04-22T18:40:35.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:40:35.705 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.706 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.706 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:35.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:35.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:35.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:40:35.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:40:35.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:40:35.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:40:35.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:40:35.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:40:35.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:40:35.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:40:35.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:40:35.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:40:35.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:35.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:40:35.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:40:35.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:40:35.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:40:35.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:40:35.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:40:35.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:40:35.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:40:35.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:40:35.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:40:35.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.710 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.711 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:35.711 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:35.711 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.014242609999997, 2024-04-22T18:40:35.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:40:35.711 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.711 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.711 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.711 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.711 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.711 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:40:35.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.711 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:35.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:35.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:40:35.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:35.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:40:35.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:40:35.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:40:35.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:40:35.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:40:35.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:40:35.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:40:35.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:40:35.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:40:35.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:40:35.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:35.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:40:35.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:40:35.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:40:35.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:40:35.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:40:35.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:40:35.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:40:35.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:40:35.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:40:35.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:40:35.716 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.716 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.716 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.716 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:35.716 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:35.716 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.013723018999997, 2024-04-22T18:40:35.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:40:35.717 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.717 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:35.717 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.717 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.717 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.717 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:40:35.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.717 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:35.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:35.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:40:35.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:35.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:40:35.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:40:35.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:40:35.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:40:35.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:40:35.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:40:35.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:40:35.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:40:35.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:40:35.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:40:35.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:35.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:40:35.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:40:35.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:35.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:40:35.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:40:35.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:35.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:40:35.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:40:35.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:35.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:40:35.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:40:35.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:40:35.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:40:35.721 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.722 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.722 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.722 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:35.722 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:35.722 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 54.913570135999997, 2024-04-22T18:40:35.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:40:35.722 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.722 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:35.722 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:35.722 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:35.722 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:35.722 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:40:35.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.723 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:35.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:35.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:40:35.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:35.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:40:35.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:40:35.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:40:35.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:40:35.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:40:35.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:40:35.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:40:35.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:40:35.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:35.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:40:35.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:40:35.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:40:35.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:40:35.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:35.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:40:35.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:40:35.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:35.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:40:35.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:40:35.726 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.727 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.727 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.727 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.727 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.727 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:35.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:35.729 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:35.729+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:35.829 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:35.830+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:35.834 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:35.834 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:77 3.3 3:dcc8aeec:::3590d5741efdda41a26b62a0982727439225b7b4b51097b313f30d8dc10bf6f6:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:40:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.58875758300000003, 2024-04-22T18:40:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58879992699999995, 2024-04-22T18:40:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for degraded object", 2024-04-22T18:40:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:40:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:40:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247571+0000", 2024-04-22T18:40:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247574+0000", 2024-04-22T18:40:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:40:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247577+0000", 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3069999999999998e-06 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247580+0000", 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8870000000000001e-06 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.340928+0000", 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093348678000000004 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:40:35.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341122+0000", 2024-04-22T18:40:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000193227 2024-04-22T18:40:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618335+0000", 2024-04-22T18:40:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.277213775 2024-04-22T18:40:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618422+0000", 2024-04-22T18:40:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6537000000000002e-05 2024-04-22T18:40:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:78 3.3 3:df7e90d2:::ab0657e24bd96704775de2a378746bc22309c5a7366a5eaf9b4e1f1f2bef0a4c:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:40:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.58874238599999995, 2024-04-22T18:40:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58882839499999995, 2024-04-22T18:40:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for degraded object", 2024-04-22T18:40:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:40:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:40:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247586+0000", 2024-04-22T18:40:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247589+0000", 2024-04-22T18:40:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.181e-06 2024-04-22T18:40:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247592+0000", 2024-04-22T18:40:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0230000000000001e-06 2024-04-22T18:40:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247595+0000", 2024-04-22T18:40:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.428e-06 2024-04-22T18:40:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341160+0000", 2024-04-22T18:40:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093564571999999999 2024-04-22T18:40:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:40:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341264+0000", 2024-04-22T18:40:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010391999999999999 2024-04-22T18:40:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619269+0000", 2024-04-22T18:40:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27800575100000002 2024-04-22T18:40:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619346+0000", 2024-04-22T18:40:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6459999999999996e-05 2024-04-22T18:40:35.842 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:135 3.3 3:f89134e5:::3e4e73f5d4a0122957865f0faeac2483204ee98d7d497ef3b54afe2a02b39a3d:head [read 0~802304 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:40:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.22088822999999999, 2024-04-22T18:40:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22101263199999999, 2024-04-22T18:40:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for degraded object", 2024-04-22T18:40:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:40:35.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:40:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615438+0000", 2024-04-22T18:40:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615449+0000", 2024-04-22T18:40:35.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1219999999999999e-05 2024-04-22T18:40:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615460+0000", 2024-04-22T18:40:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0403e-05 2024-04-22T18:40:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.715588+0000", 2024-04-22T18:40:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100128881 2024-04-22T18:40:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722533+0000", 2024-04-22T18:40:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069444010000000002 2024-04-22T18:40:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:40:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722658+0000", 2024-04-22T18:40:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012560699999999999 2024-04-22T18:40:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826437+0000", 2024-04-22T18:40:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103778194 2024-04-22T18:40:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826532+0000", 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5111000000000005e-05 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:80 3.3 3:f90f47f2:::e5603f596d699ee3e27e255225eeb90a9f0c1b4a4d099d1c586b19019220aa3f:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.58870330900000001, 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58886542900000005, 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for degraded object", 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247625+0000", 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:35.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247628+0000", 2024-04-22T18:40:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6129999999999998e-06 2024-04-22T18:40:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247631+0000", 2024-04-22T18:40:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3869999999999999e-06 2024-04-22T18:40:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247634+0000", 2024-04-22T18:40:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9919999999999999e-06 2024-04-22T18:40:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341550+0000", 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093915584999999996 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341700+0000", 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015000300000000001 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620744+0000", 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27904399200000002 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:35.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620794+0000", 2024-04-22T18:40:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9579000000000001e-05 2024-04-22T18:40:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:40:35.851 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 4 2024-04-22T18:40:35.851 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:35.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:35.916 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:35.917+0000 7fec57bee640 -1 osd.7 104 handle_osd_map injecting map crc failure 2024-04-22T18:40:35.930 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:35.931+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:35.996 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:35.996 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:35.996 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:35.997 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:35.997 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:36.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:36.046 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:36.047+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:36.119 INFO:tasks.rados.rados.0.smithi089.stdout:1520: set_chunk oid 45 offset: 155136 length: 2074112 target oid 4c4255203fe86747cad8a2c543f4cba74fbabbe1751b2eefefc7be15d567a83d offset: 0 snap -1 2024-04-22T18:40:36.120 INFO:tasks.rados.rados.0.smithi089.stdout:1521: write oid 22 current snap is 0 2024-04-22T18:40:36.120 INFO:tasks.rados.rados.0.smithi089.stdout:1512: finishing set_chunk to oid 1 2024-04-22T18:40:36.120 INFO:tasks.rados.rados.0.smithi089.stdout:1512: got expected ENOENT (src dne) 2024-04-22T18:40:36.120 INFO:tasks.rados.rados.0.smithi089.stdout:1521: seq_num 303 ranges {692521=538214,1931338=785088} 2024-04-22T18:40:36.126 INFO:tasks.rados.rados.0.smithi089.stdout:1521: writing smithi08937929-22 from 692521 to 1230735 tid 1 2024-04-22T18:40:36.132 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:36.133+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:36.135 INFO:tasks.rados.rados.0.smithi089.stdout:1521: writing smithi08937929-22 from 1931338 to 2716426 tid 2 2024-04-22T18:40:36.135 INFO:tasks.rados.rados.0.smithi089.stdout:1512: done (13 left) 2024-04-22T18:40:36.135 INFO:tasks.rados.rados.0.smithi089.stdout:1522: read oid 32 snap -1 2024-04-22T18:40:36.135 INFO:tasks.rados.rados.0.smithi089.stdout:1522: expect (ObjNum 296 snap 0 seq_num 296) 2024-04-22T18:40:36.135 INFO:tasks.rados.rados.0.smithi089.stdout:1523: tier_flush oid 43 2024-04-22T18:40:36.135 INFO:tasks.rados.rados.0.smithi089.stdout:1523: tier_flush oid 43 snap -1 2024-04-22T18:40:36.135 INFO:tasks.rados.rados.0.smithi089.stdout:1509: expect (ObjNum 265 snap 0 seq_num 265) 2024-04-22T18:40:36.150 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:36.150 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:36.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:36.151 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:36.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.151 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:50 4.a 4:5d2aee44:::smithi08937929-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:36.151 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:36.151 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.911058406, 2024-04-22T18:40:36.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40856295399999998, 2024-04-22T18:40:36.151 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.151 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.151 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.151 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.151 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.151 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 50 2024-04-22T18:40:36.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.151 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:36.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239609+0000", 2024-04-22T18:40:36.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.239607+0000", 2024-04-22T18:40:36.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240309+0000", 2024-04-22T18:40:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00070201599999999997 2024-04-22T18:40:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240320+0000", 2024-04-22T18:40:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1024e-05 2024-04-22T18:40:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240327+0000", 2024-04-22T18:40:36.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0049999999999998e-06 2024-04-22T18:40:36.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362065+0000", 2024-04-22T18:40:36.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.121737812 2024-04-22T18:40:36.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362121+0000", 2024-04-22T18:40:36.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5868000000000001e-05 2024-04-22T18:40:36.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:36.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.362210+0000", 2024-04-22T18:40:36.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8533000000000001e-05 2024-04-22T18:40:36.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.365322+0000", 2024-04-22T18:40:36.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0031121220000000002 2024-04-22T18:40:36.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648054+0000", 2024-04-22T18:40:36.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.282732549 2024-04-22T18:40:36.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648105+0000", 2024-04-22T18:40:36.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1276999999999997e-05 2024-04-22T18:40:36.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648172+0000", 2024-04-22T18:40:36.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6398999999999999e-05 2024-04-22T18:40:36.156 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.156 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.156 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.156 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:51 4.a 4:5d2aee44:::smithi08937929-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:36.156 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:36.156 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.910335872, 2024-04-22T18:40:36.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40793057199999999, 2024-04-22T18:40:36.157 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.157 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.157 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.157 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.157 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.157 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 51 2024-04-22T18:40:36.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.157 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:36.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:36.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240331+0000", 2024-04-22T18:40:36.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240336+0000", 2024-04-22T18:40:36.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1070000000000004e-06 2024-04-22T18:40:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240339+0000", 2024-04-22T18:40:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9009999999999998e-06 2024-04-22T18:40:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240342+0000", 2024-04-22T18:40:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:40:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363080+0000", 2024-04-22T18:40:36.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.122738075 2024-04-22T18:40:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363164+0000", 2024-04-22T18:40:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3722999999999998e-05 2024-04-22T18:40:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363287+0000", 2024-04-22T18:40:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012301099999999999 2024-04-22T18:40:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466753+0000", 2024-04-22T18:40:36.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103466217 2024-04-22T18:40:36.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648205+0000", 2024-04-22T18:40:36.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.181451788 2024-04-22T18:40:36.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648231+0000", 2024-04-22T18:40:36.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6597999999999998e-05 2024-04-22T18:40:36.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648262+0000", 2024-04-22T18:40:36.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0871000000000003e-05 2024-04-22T18:40:36.162 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.162 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.162 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.162 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:52 4.a 4:5d2aee44:::smithi08937929-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:36.162 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:36.162 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.910323413, 2024-04-22T18:40:36.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40812731200000002, 2024-04-22T18:40:36.162 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.162 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:36.162 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.162 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.163 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.163 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 52 2024-04-22T18:40:36.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.163 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:36.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:36.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240344+0000", 2024-04-22T18:40:36.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240345+0000", 2024-04-22T18:40:36.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4300000000000001e-06 2024-04-22T18:40:36.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240349+0000", 2024-04-22T18:40:36.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2570000000000001e-06 2024-04-22T18:40:36.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.240358+0000", 2024-04-22T18:40:36.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.4430000000000003e-06 2024-04-22T18:40:36.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363697+0000", 2024-04-22T18:40:36.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12333965299999999 2024-04-22T18:40:36.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:36.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363777+0000", 2024-04-22T18:40:36.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9883999999999996e-05 2024-04-22T18:40:36.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648296+0000", 2024-04-22T18:40:36.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28451861299999998 2024-04-22T18:40:36.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648379+0000", 2024-04-22T18:40:36.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3302999999999998e-05 2024-04-22T18:40:36.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.648471+0000", 2024-04-22T18:40:36.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2029999999999998e-05 2024-04-22T18:40:36.167 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.167 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.167 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.167 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:36.167 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:36.167 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.891020078, 2024-04-22T18:40:36.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:40:36.167 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.167 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.167 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.167 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.167 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.167 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:40:36.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.168 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:36.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:36.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:40:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:40:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:40:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:40:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:40:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:40:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:40:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:40:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:40:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:40:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:40:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:36.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:40:36.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:40:36.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:40:36.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:40:36.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:40:36.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:40:36.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:40:36.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:40:36.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:40:36.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:40:36.172 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.172 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.172 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.172 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:36.172 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:36.173 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.88623764800001, 2024-04-22T18:40:36.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:40:36.173 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.173 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.173 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.173 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.173 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.173 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:40:36.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.173 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:36.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:36.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:40:36.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:36.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:40:36.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:40:36.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:40:36.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:40:36.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:40:36.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:40:36.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:40:36.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:40:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:40:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:40:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:40:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:40:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:40:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:40:36.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:40:36.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:40:36.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:40:36.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:40:36.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:40:36.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:40:36.178 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.178 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.178 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.178 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:36.178 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:36.178 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.886143445, 2024-04-22T18:40:36.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:40:36.178 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.178 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.178 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.178 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.178 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.178 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:40:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:40:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:40:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:40:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:40:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:40:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:40:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:40:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:40:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:40:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:40:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:40:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:36.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:40:36.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:40:36.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:40:36.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:40:36.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:40:36.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:40:36.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:40:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:40:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:40:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:40:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:36.183 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.88612255699999, 2024-04-22T18:40:36.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:40:36.184 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.184 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:36.184 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.184 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.184 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.184 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:40:36.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.184 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:36.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:40:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:40:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:40:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:40:36.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:40:36.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:40:36.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:40:36.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:40:36.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:40:36.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:36.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:40:36.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:40:36.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:40:36.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:40:36.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:40:36.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:40:36.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:40:36.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:40:36.188 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.188 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.188 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.188 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:36.188 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:36.188 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.103114894, 2024-04-22T18:40:36.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:40:36.188 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:36.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:40:36.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:36.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:40:36.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:40:36.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:40:36.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:40:36.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:40:36.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:40:36.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:40:36.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:40:36.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:40:36.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:40:36.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:36.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:40:36.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:40:36.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:40:36.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:40:36.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:40:36.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:40:36.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:40:36.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:40:36.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:40:36.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:40:36.193 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.193 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.193 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.194 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:36.194 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:36.194 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.09528348800001, 2024-04-22T18:40:36.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:40:36.194 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.194 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.194 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.194 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.194 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.194 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:40:36.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.194 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:36.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:36.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:40:36.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:36.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:40:36.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:40:36.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:40:36.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:40:36.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:40:36.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:40:36.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:40:36.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:40:36.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:40:36.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:40:36.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:36.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:40:36.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:40:36.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:40:36.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:40:36.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:40:36.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:40:36.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:40:36.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:40:36.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:40:36.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:40:36.199 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.199 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.199 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.199 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2052 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v19,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:36.199 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:36.199 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.573171707, 2024-04-22T18:40:36.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43058958600000002, 2024-04-22T18:40:36.200 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.200 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.200 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.200 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.200 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.200 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2052 2024-04-22T18:40:36.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.200 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:36.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577496+0000", 2024-04-22T18:40:36.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577494+0000", 2024-04-22T18:40:36.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:36.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577500+0000", 2024-04-22T18:40:36.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3279999999999996e-06 2024-04-22T18:40:36.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577510+0000", 2024-04-22T18:40:36.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9450000000000005e-06 2024-04-22T18:40:36.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.577515+0000", 2024-04-22T18:40:36.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7799999999999997e-06 2024-04-22T18:40:36.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677622+0000", 2024-04-22T18:40:36.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010654099999999 2024-04-22T18:40:36.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.677710+0000", 2024-04-22T18:40:36.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8472000000000003e-05 2024-04-22T18:40:36.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:36.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.906619+0000", 2024-04-22T18:40:36.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.228908519 2024-04-22T18:40:36.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.907343+0000", 2024-04-22T18:40:36.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072442600000000002 2024-04-22T18:40:36.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.007972+0000", 2024-04-22T18:40:36.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100628781 2024-04-22T18:40:36.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008022+0000", 2024-04-22T18:40:36.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9843999999999999e-05 2024-04-22T18:40:36.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.008085+0000", 2024-04-22T18:40:36.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3256999999999994e-05 2024-04-22T18:40:36.204 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.204 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.205 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.205 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:36.205 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:36.205 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.943410858, 2024-04-22T18:40:36.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:40:36.205 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.205 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.205 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.205 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.205 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.205 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:40:36.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.206 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:36.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:36.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:40:36.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:36.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:40:36.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:40:36.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:40:36.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:40:36.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:40:36.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:40:36.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:40:36.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:40:36.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:40:36.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:40:36.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:36.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:40:36.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:40:36.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:40:36.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:40:36.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:40:36.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:40:36.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:40:36.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:40:36.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:40:36.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:40:36.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.210 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.210 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2505 4.5 4:a6d237d2:::smithi08937929-43:head [setxattr _header (56) in=7b,truncate 3529470] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:36.210 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:36.210 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.943018162999998, 2024-04-22T18:40:36.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43706744400000003, 2024-04-22T18:40:36.211 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.211 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.211 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.211 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.211 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.211 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2505 2024-04-22T18:40:36.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.211 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:36.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207649+0000", 2024-04-22T18:40:36.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207648+0000", 2024-04-22T18:40:36.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:36.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207653+0000", 2024-04-22T18:40:36.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2989999999999998e-06 2024-04-22T18:40:36.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207656+0000", 2024-04-22T18:40:36.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0199999999999999e-06 2024-04-22T18:40:36.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207659+0000", 2024-04-22T18:40:36.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2270000000000002e-06 2024-04-22T18:40:36.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243213+0000", 2024-04-22T18:40:36.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035553653999999997 2024-04-22T18:40:36.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243273+0000", 2024-04-22T18:40:36.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0300000000000002e-05 2024-04-22T18:40:36.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:36.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243359+0000", 2024-04-22T18:40:36.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6275999999999999e-05 2024-04-22T18:40:36.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297443+0000", 2024-04-22T18:40:36.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.054083930000000002 2024-04-22T18:40:36.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644651+0000", 2024-04-22T18:40:36.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.347208181 2024-04-22T18:40:36.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644684+0000", 2024-04-22T18:40:36.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3080999999999997e-05 2024-04-22T18:40:36.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644717+0000", 2024-04-22T18:40:36.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2264000000000001e-05 2024-04-22T18:40:36.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.215 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.216 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.216 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:36.216 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:36.216 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 40.94300595, 2024-04-22T18:40:36.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:40:36.216 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.216 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:36.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.216 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.216 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:40:36.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:40:36.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:40:36.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:40:36.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:40:36.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:40:36.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:40:36.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:40:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:40:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:40:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:40:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3213 4.a 4:5d2aee44:::smithi08937929-13:head [assert-version v66,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:36.221 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:40:36.221 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.582106061999999, 2024-04-22T18:40:36.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69928701400000004, 2024-04-22T18:40:36.221 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.221 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.221 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.221 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.221 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.221 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3213 2024-04-22T18:40:36.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.221 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:40:36.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:40:36.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568559+0000", 2024-04-22T18:40:36.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:36.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568567+0000", 2024-04-22T18:40:36.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-04-22T18:40:36.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568579+0000", 2024-04-22T18:40:36.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-04-22T18:40:36.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.668711+0000", 2024-04-22T18:40:36.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100131866 2024-04-22T18:40:36.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.768945+0000", 2024-04-22T18:40:36.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100234252 2024-04-22T18:40:36.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769002+0000", 2024-04-22T18:40:36.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7092999999999997e-05 2024-04-22T18:40:36.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:36.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.166404+0000", 2024-04-22T18:40:36.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39740241999999998 2024-04-22T18:40:36.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.191686+0000", 2024-04-22T18:40:36.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025281739000000001 2024-04-22T18:40:36.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267758+0000", 2024-04-22T18:40:36.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076071606999999999 2024-04-22T18:40:36.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267812+0000", 2024-04-22T18:40:36.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4478999999999998e-05 2024-04-22T18:40:36.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267848+0000", 2024-04-22T18:40:36.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6062999999999997e-05 2024-04-22T18:40:36.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.226 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.226 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3218 4.a 4:519c08f7:::smithi08937929-15:head [write 1617736~710086] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:36.226 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.669206+0000", 2024-04-22T18:40:36.226 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.481461071, 2024-04-22T18:40:36.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40462619, 2024-04-22T18:40:36.226 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.226 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.227 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.227 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.227 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.227 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3218 2024-04-22T18:40:36.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.227 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669206+0000", 2024-04-22T18:40:36.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669206+0000", 2024-04-22T18:40:36.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669206+0000", 2024-04-22T18:40:36.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669812+0000", 2024-04-22T18:40:36.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060657099999999998 2024-04-22T18:40:36.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669820+0000", 2024-04-22T18:40:36.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8560000000000007e-06 2024-04-22T18:40:36.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669831+0000", 2024-04-22T18:40:36.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1131000000000001e-05 2024-04-22T18:40:36.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.770325+0000", 2024-04-22T18:40:36.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10049406499999999 2024-04-22T18:40:36.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.770405+0000", 2024-04-22T18:40:36.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9487000000000007e-05 2024-04-22T18:40:36.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:36.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.770532+0000", 2024-04-22T18:40:36.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000126749 2024-04-22T18:40:36.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.772387+0000", 2024-04-22T18:40:36.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001855385 2024-04-22T18:40:36.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.073766+0000", 2024-04-22T18:40:36.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30137940299999999 2024-04-22T18:40:36.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.073808+0000", 2024-04-22T18:40:36.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1186000000000002e-05 2024-04-22T18:40:36.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.073832+0000", 2024-04-22T18:40:36.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4882999999999999e-05 2024-04-22T18:40:36.231 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.231 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.231 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.232 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3220 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:36.232 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.669853+0000", 2024-04-22T18:40:36.232 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.480814468, 2024-04-22T18:40:36.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40421206599999998, 2024-04-22T18:40:36.232 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.232 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:36.232 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.232 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.232 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.232 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3220 2024-04-22T18:40:36.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.232 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669853+0000", 2024-04-22T18:40:36.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669853+0000", 2024-04-22T18:40:36.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669852+0000", 2024-04-22T18:40:36.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669854+0000", 2024-04-22T18:40:36.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.866e-06 2024-04-22T18:40:36.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.669857+0000", 2024-04-22T18:40:36.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:40:36.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769953+0000", 2024-04-22T18:40:36.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009555000000001 2024-04-22T18:40:36.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.772113+0000", 2024-04-22T18:40:36.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002159771 2024-04-22T18:40:36.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:36.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.772183+0000", 2024-04-22T18:40:36.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0486000000000002e-05 2024-04-22T18:40:36.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.073963+0000", 2024-04-22T18:40:36.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30178019099999998 2024-04-22T18:40:36.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.074009+0000", 2024-04-22T18:40:36.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5794999999999998e-05 2024-04-22T18:40:36.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.074065+0000", 2024-04-22T18:40:36.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5655000000000002e-05 2024-04-22T18:40:36.236 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.236 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.236 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.236 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3319 4.a 4:5d2aee44:::smithi08937929-13:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:36.237 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:40:36.237 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.6903583110000007, 2024-04-22T18:40:36.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476623401, 2024-04-22T18:40:36.237 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.237 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:36.237 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.237 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.237 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.237 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3319 2024-04-22T18:40:36.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.237 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:40:36.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:40:36.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460306+0000", 2024-04-22T18:40:36.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:36.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460318+0000", 2024-04-22T18:40:36.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2896e-05 2024-04-22T18:40:36.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460345+0000", 2024-04-22T18:40:36.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6650000000000001e-05 2024-04-22T18:40:36.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460360+0000", 2024-04-22T18:40:36.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4729000000000001e-05 2024-04-22T18:40:36.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460502+0000", 2024-04-22T18:40:36.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141729 2024-04-22T18:40:36.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460644+0000", 2024-04-22T18:40:36.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142755 2024-04-22T18:40:36.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.936932+0000", 2024-04-22T18:40:36.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476288078 2024-04-22T18:40:36.240 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.240 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.240 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3412 4.a 4:519c08f7:::smithi08937929-15:head [write 2529878~394573] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:40:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.0085237500000002, 2024-04-22T18:40:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48918094899999998, 2024-04-22T18:40:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3412 2024-04-22T18:40:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:40:36.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:40:36.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142141+0000", 2024-04-22T18:40:36.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:36.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142496+0000", 2024-04-22T18:40:36.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035444699999999998 2024-04-22T18:40:36.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142510+0000", 2024-04-22T18:40:36.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3841e-05 2024-04-22T18:40:36.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142518+0000", 2024-04-22T18:40:36.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8089999999999993e-06 2024-04-22T18:40:36.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142560+0000", 2024-04-22T18:40:36.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2097e-05 2024-04-22T18:40:36.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142637+0000", 2024-04-22T18:40:36.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6965000000000006e-05 2024-04-22T18:40:36.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:36.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142715+0000", 2024-04-22T18:40:36.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8694000000000002e-05 2024-04-22T18:40:36.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144179+0000", 2024-04-22T18:40:36.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146342 2024-04-22T18:40:36.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631229+0000", 2024-04-22T18:40:36.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48705018500000002 2024-04-22T18:40:36.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631281+0000", 2024-04-22T18:40:36.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2216999999999998e-05 2024-04-22T18:40:36.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631324+0000", 2024-04-22T18:40:36.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3344999999999999e-05 2024-04-22T18:40:36.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.246 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.246 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3413 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2924451] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:36.246 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:40:36.246 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.0081426100000002, 2024-04-22T18:40:36.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58904086899999997, 2024-04-22T18:40:36.246 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3413 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:40:36.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142523+0000", 2024-04-22T18:40:36.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:36.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142529+0000", 2024-04-22T18:40:36.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6049999999999997e-06 2024-04-22T18:40:36.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142535+0000", 2024-04-22T18:40:36.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4740000000000002e-06 2024-04-22T18:40:36.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142541+0000", 2024-04-22T18:40:36.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1839999999999998e-06 2024-04-22T18:40:36.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143553+0000", 2024-04-22T18:40:36.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010126689999999999 2024-04-22T18:40:36.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143659+0000", 2024-04-22T18:40:36.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105393 2024-04-22T18:40:36.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:36.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143825+0000", 2024-04-22T18:40:36.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165809 2024-04-22T18:40:36.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144329+0000", 2024-04-22T18:40:36.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050483799999999997 2024-04-22T18:40:36.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731480+0000", 2024-04-22T18:40:36.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58715100600000003 2024-04-22T18:40:36.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731522+0000", 2024-04-22T18:40:36.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2098000000000002e-05 2024-04-22T18:40:36.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731566+0000", 2024-04-22T18:40:36.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3066999999999999e-05 2024-04-22T18:40:36.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.251 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.252 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3414 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:36.252 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:40:36.252 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.0081140059999996, 2024-04-22T18:40:36.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58926020199999996, 2024-04-22T18:40:36.252 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.252 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:36.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.252 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3414 2024-04-22T18:40:36.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.252 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142552+0000", 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142556+0000", 2024-04-22T18:40:36.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0289999999999999e-06 2024-04-22T18:40:36.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142574+0000", 2024-04-22T18:40:36.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7266999999999999e-05 2024-04-22T18:40:36.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142581+0000", 2024-04-22T18:40:36.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-04-22T18:40:36.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144121+0000", 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015397480000000001 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144170+0000", 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8803e-05 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731624+0000", 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58745365800000005 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731712+0000", 2024-04-22T18:40:36.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8016999999999994e-05 2024-04-22T18:40:36.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731813+0000", 2024-04-22T18:40:36.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101817 2024-04-22T18:40:36.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.256 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.256 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.256 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:36.257 INFO:tasks.rados.rados.0.smithi089.stdout:1514: finishing write tid 1 to smithi08937929-15 2024-04-22T18:40:36.257 INFO:tasks.rados.rados.0.smithi089.stdout:1514: finishing write tid 2 to smithi08937929-15 2024-04-22T18:40:36.257 INFO:tasks.rados.rados.0.smithi089.stdout:1514: finishing write tid 3 to smithi08937929-15 2024-04-22T18:40:36.257 INFO:tasks.rados.rados.0.smithi089.stdout:1519: expect (ObjNum 278 snap 0 seq_num 278) 2024-04-22T18:40:36.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:36.259 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:36.258+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:36.333 INFO:tasks.rados.rados.0.smithi089.stdout:1518: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:40:36.358 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:36.358+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:36.401 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:36.401 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:36.401 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:36.401 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:36.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:36.459 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:36.460+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:36.512 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:36.512 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:36.512 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:36.512 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:36.512 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:36.520 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:36.560 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:36.560+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:36.565 INFO:tasks.rados.rados.0.smithi089.stdout:1514: finishing write tid 4 to smithi08937929-15 2024-04-22T18:40:36.565 INFO:tasks.rados.rados.0.smithi089.stdout:1514: finishing write tid 5 to smithi08937929-15 2024-04-22T18:40:36.565 INFO:tasks.rados.rados.0.smithi089.stdout:1514: oid 15 updating version 0 to 91 2024-04-22T18:40:36.565 INFO:tasks.rados.rados.0.smithi089.stdout:1514: oid 15 updating version 91 to 93 2024-04-22T18:40:36.565 INFO:tasks.rados.rados.0.smithi089.stdout:1514: oid 15 version 93 is already newer than 92 2024-04-22T18:40:36.565 INFO:tasks.rados.rados.0.smithi089.stdout:1514: oid 15 updating version 93 to 94 2024-04-22T18:40:36.565 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 94 (ObjNum 301 snap 0 seq_num 301) dirty exists 2024-04-22T18:40:36.565 INFO:tasks.rados.rados.0.smithi089.stdout:1514: left oid 15 (ObjNum 301 snap 0 seq_num 301) 2024-04-22T18:40:36.565 INFO:tasks.rados.rados.0.smithi089.stdout:1506: got (0) Success 2024-04-22T18:40:36.566 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 55 (ObjNum 227 snap 0 seq_num 227) dirty exists 2024-04-22T18:40:36.566 INFO:tasks.rados.rados.0.smithi089.stdout:1511: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:36.661 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:36.662+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:36.665 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 116.923718581, 2024-04-22T18:40:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:40:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:40:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:40:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:40:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:40:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:40:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:40:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:40:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:40:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:40:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:40:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:40:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:40:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:40:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 116.92367889000001, 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:40:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:40:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:40:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:40:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:40:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:40:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:40:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:40:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:40:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:40:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:40:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:40:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:40:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:40:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:40:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:40:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:40:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:40:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:40:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:68 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:40:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 114.67949784300001, 2024-04-22T18:40:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402419109, 2024-04-22T18:40:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 68 2024-04-22T18:40:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:40:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986205+0000", 2024-04-22T18:40:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986204+0000", 2024-04-22T18:40:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986208+0000", 2024-04-22T18:40:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8149999999999999e-06 2024-04-22T18:40:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986214+0000", 2024-04-22T18:40:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.604e-06 2024-04-22T18:40:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986218+0000", 2024-04-22T18:40:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6689999999999999e-06 2024-04-22T18:40:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986833+0000", 2024-04-22T18:40:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000615648 2024-04-22T18:40:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986891+0000", 2024-04-22T18:40:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7915999999999999e-05 2024-04-22T18:40:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986981+0000", 2024-04-22T18:40:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9759999999999994e-05 2024-04-22T18:40:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.987584+0000", 2024-04-22T18:40:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00060270199999999999 2024-04-22T18:40:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388527+0000", 2024-04-22T18:40:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40094366999999997 2024-04-22T18:40:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388583+0000", 2024-04-22T18:40:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5383000000000002e-05 2024-04-22T18:40:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388624+0000", 2024-04-22T18:40:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1501999999999997e-05 2024-04-22T18:40:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 114.679482054, 2024-04-22T18:40:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:40:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:40:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:40:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:40:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:40:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:40:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:40:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:40:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:40:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:40:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:40:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:40:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:40:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:40:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:40:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:40:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:40:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:40:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:40:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:40:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:40:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:40:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:40:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.811492226, 2024-04-22T18:40:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:40:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:40:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:40:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:40:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:40:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:40:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:40:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:40:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:40:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:40:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:40:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:40:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:40:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:40:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:40:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:40:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:40:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:40:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.811476568, 2024-04-22T18:40:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:40:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:40:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:40:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:40:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:40:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:40:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:40:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:40:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:40:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:40:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:40:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:40:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:40:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:40:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.811425406, 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:40:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.700 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:36.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:36.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:40:36.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:40:36.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.811410182, 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.704 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:40:36.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:40:36.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.811350605, 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.709 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:40:36.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:40:36.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:40:36.714 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.714 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.714 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.714 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:36.714 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:36.714 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.811268289, 2024-04-22T18:40:36.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:40:36.714 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.714 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.714 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:40:36.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:40:36.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:40:36.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:40:36.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:40:36.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:40:36.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:36.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:40:36.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:40:36.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:40:36.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:40:36.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:40:36.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:40:36.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:40:36.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:40:36.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:40:36.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:40:36.719 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.719 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.719 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.719 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:36.719 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.811258133, 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:36.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:40:36.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:40:36.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:40:36.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:40:36.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:40:36.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:40:36.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:40:36.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:40:36.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:40:36.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:40:36.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:40:36.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:36.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:40:36.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:40:36.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:40:36.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:40:36.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:40:36.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:40:36.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:40:36.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:40:36.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:40:36.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:40:36.724 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.725 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.725 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.725 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:161 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-22T18:40:36.725 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:40:36.725 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 108.21076384200001, 2024-04-22T18:40:36.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40239169000000002, 2024-04-22T18:40:36.725 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.725 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:36.725 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.725 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:36.725 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:36.725 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 161 2024-04-22T18:40:36.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.726 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:40:36.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454939+0000", 2024-04-22T18:40:36.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454938+0000", 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454982+0000", 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3470999999999998e-05 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.454988+0000", 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7640000000000002e-06 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555054+0000", 2024-04-22T18:40:36.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006607300000001 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555607+0000", 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00055319500000000003 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555689+0000", 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1817999999999995e-05 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.555804+0000", 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115346 2024-04-22T18:40:36.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:36.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.556554+0000", 2024-04-22T18:40:36.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000750102 2024-04-22T18:40:36.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:36.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857265+0000", 2024-04-22T18:40:36.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30071122700000003 2024-04-22T18:40:36.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:36.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857302+0000", 2024-04-22T18:40:36.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7069e-05 2024-04-22T18:40:36.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:48.857331+0000", 2024-04-22T18:40:36.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8291999999999999e-05 2024-04-22T18:40:36.730 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.730 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.730 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.730 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2835 4.18 4:187adac1:::smithi08937929-36:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:36.730 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:36.730 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.049333742000002, 2024-04-22T18:40:36.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.403020987, 2024-04-22T18:40:36.731 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.731 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:36.731 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.731 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.731 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.731 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2835 2024-04-22T18:40:36.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.731 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:36.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616367+0000", 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616377+0000", 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0618999999999999e-05 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616410+0000", 2024-04-22T18:40:36.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3021000000000001e-05 2024-04-22T18:40:36.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616437+0000", 2024-04-22T18:40:36.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7158e-05 2024-04-22T18:40:36.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.716588+0000", 2024-04-22T18:40:36.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100150674 2024-04-22T18:40:36.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.716679+0000", 2024-04-22T18:40:36.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1231999999999997e-05 2024-04-22T18:40:36.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:08.019390+0000", 2024-04-22T18:40:36.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30271076299999999 2024-04-22T18:40:36.734 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.734 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.734 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.734 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3163 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:36.734 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:40:36.735 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.828245695, 2024-04-22T18:40:36.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1009801299999999, 2024-04-22T18:40:36.735 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.735 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:36.735 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.735 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.735 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.735 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3163 2024-04-22T18:40:36.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.735 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:40:36.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837455+0000", 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837464+0000", 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5319999999999995e-06 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837476+0000", 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1759e-05 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837488+0000", 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1666e-05 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837541+0000", 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2939999999999998e-05 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:36.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837547+0000", 2024-04-22T18:40:36.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6409999999999996e-06 2024-04-22T18:40:36.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937423+0000", 2024-04-22T18:40:36.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.099875545 2024-04-22T18:40:36.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937567+0000", 2024-04-22T18:40:36.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014363099999999999 2024-04-22T18:40:36.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938437+0000", 2024-04-22T18:40:36.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087060399999999997 2024-04-22T18:40:36.739 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.739 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.739 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.739 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3164 4.8 4:187adac1:::smithi08937929-36:head [sparse-read 0~2643888] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:36.739 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:36.739 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.828206326, 2024-04-22T18:40:36.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1013670310000001, 2024-04-22T18:40:36.739 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.739 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:36.739 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.739 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.740 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.740 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3164 2024-04-22T18:40:36.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.740 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:36.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:36.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:36.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837499+0000", 2024-04-22T18:40:36.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.994e-06 2024-04-22T18:40:36.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837503+0000", 2024-04-22T18:40:36.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7249999999999999e-06 2024-04-22T18:40:36.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837506+0000", 2024-04-22T18:40:36.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5980000000000001e-06 2024-04-22T18:40:36.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937641+0000", 2024-04-22T18:40:36.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10013467700000001 2024-04-22T18:40:36.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:36.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937650+0000", 2024-04-22T18:40:36.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3489999999999998e-06 2024-04-22T18:40:36.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938468+0000", 2024-04-22T18:40:36.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0008176710000001 2024-04-22T18:40:36.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938582+0000", 2024-04-22T18:40:36.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114287 2024-04-22T18:40:36.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938864+0000", 2024-04-22T18:40:36.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028120899999999998 2024-04-22T18:40:36.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.743 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.744 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.744 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3165 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:36.744 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:36.744 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.828187558, 2024-04-22T18:40:36.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2022754499999999, 2024-04-22T18:40:36.744 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.744 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:36.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.744 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3165 2024-04-22T18:40:36.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.744 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837517+0000", 2024-04-22T18:40:36.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3369999999999998e-06 2024-04-22T18:40:36.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837523+0000", 2024-04-22T18:40:36.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5450000000000003e-06 2024-04-22T18:40:36.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837526+0000", 2024-04-22T18:40:36.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.354e-06 2024-04-22T18:40:36.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937682+0000", 2024-04-22T18:40:36.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100155917 2024-04-22T18:40:36.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:36.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937688+0000", 2024-04-22T18:40:36.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:40:36.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.038953+0000", 2024-04-22T18:40:36.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012648110000001 2024-04-22T18:40:36.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:36.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039034+0000", 2024-04-22T18:40:36.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0969000000000004e-05 2024-04-22T18:40:36.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039791+0000", 2024-04-22T18:40:36.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075672900000000004 2024-04-22T18:40:36.748 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.748 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.748 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.748 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3183 4.8 4:1bc4668a:::smithi08937929-21:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.62978436, 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0040478260000001, 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3183 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:40:36.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035913+0000", 2024-04-22T18:40:36.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999943 2024-04-22T18:40:36.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035935+0000", 2024-04-22T18:40:36.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2232999999999999e-05 2024-04-22T18:40:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035969+0000", 2024-04-22T18:40:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3969e-05 2024-04-22T18:40:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035987+0000", 2024-04-22T18:40:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8397e-05 2024-04-22T18:40:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:36.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2916000000000003e-05 2024-04-22T18:40:36.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:36.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:40:36.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0103e-05 2024-04-22T18:40:36.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039841+0000", 2024-04-22T18:40:36.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0037508959999999 2024-04-22T18:40:36.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039966+0000", 2024-04-22T18:40:36.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012514500000000001 2024-04-22T18:40:36.753 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.753 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.753 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.753 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3184 4.8 4:12d9f826:::smithi08937929-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:36.753 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:40:36.753 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.629696401, 2024-04-22T18:40:36.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.00402868, 2024-04-22T18:40:36.753 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.753 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:36.753 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.753 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.753 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.753 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3184 2024-04-22T18:40:36.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.754 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:40:36.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:40:36.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036005+0000", 2024-04-22T18:40:36.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:36.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036026+0000", 2024-04-22T18:40:36.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1092999999999999e-05 2024-04-22T18:40:36.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036030+0000", 2024-04-22T18:40:36.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.467e-06 2024-04-22T18:40:36.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036033+0000", 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9780000000000001e-06 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036120+0000", 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7495000000000002e-05 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036125+0000", 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8720000000000001e-06 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039982+0000", 2024-04-22T18:40:36.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0038571919999999 2024-04-22T18:40:36.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040035+0000", 2024-04-22T18:40:36.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2992999999999999e-05 2024-04-22T18:40:36.757 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.757 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.757 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.757 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3185 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:36.757 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:36.757 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.629642489, 2024-04-22T18:40:36.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0041404039999999, 2024-04-22T18:40:36.758 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.758 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:36.758 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.758 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.758 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.758 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3185 2024-04-22T18:40:36.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.758 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:36.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:36.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:36.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036062+0000", 2024-04-22T18:40:36.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9099999999999999e-06 2024-04-22T18:40:36.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036066+0000", 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1699999999999999e-06 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036077+0000", 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0637e-05 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036147+0000", 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0025999999999997e-05 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:36.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036151+0000", 2024-04-22T18:40:36.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:40:36.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040068+0000", 2024-04-22T18:40:36.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.003916604 2024-04-22T18:40:36.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040201+0000", 2024-04-22T18:40:36.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013275400000000001 2024-04-22T18:40:36.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.761 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.762 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.762 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3186 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:36.762 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:36.762 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.629622933, 2024-04-22T18:40:36.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2044167510000001, 2024-04-22T18:40:36.762 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:36.762 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:36.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:36.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:36.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:36.762 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3186 2024-04-22T18:40:36.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.763 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:36.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:36.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:36.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:36.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:36.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:36.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:36.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036079+0000", 2024-04-22T18:40:36.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:36.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:36.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036081+0000", 2024-04-22T18:40:36.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6449999999999999e-06 2024-04-22T18:40:36.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:36.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036084+0000", 2024-04-22T18:40:36.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9399999999999998e-06 2024-04-22T18:40:36.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:36.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:40:36.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.879e-06 2024-04-22T18:40:36.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036168+0000", 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7657999999999994e-05 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036171+0000", 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7639999999999999e-06 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240345+0000", 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2041734260000001 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:36.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:36.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:36.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240497+0000", 2024-04-22T18:40:36.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015182800000000001 2024-04-22T18:40:36.768 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.768 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.768 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.768 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:36.768 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:36.768 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:36.769 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:36.770+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:36.770 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:36.784 INFO:tasks.rados.rados.0.smithi089.stdout:1516: expect (ObjNum 251 snap 0 seq_num 251) 2024-04-22T18:40:36.819 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:40:36.870 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:36.871+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:36.875 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:36.875 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:36.875 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:36.875 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:36.884 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:36.971 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:36.972+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:36.988 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:36.989 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:36.998 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:36.999 INFO:tasks.rados.rados.0.smithi089.stdout:1510: expect (ObjNum 289 snap 0 seq_num 289) 2024-04-22T18:40:37.073 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:37.074+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:37.104 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 102.648622827, 2024-04-22T18:40:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:40:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:37.104 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:40:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:40:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:40:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:40:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:40:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:40:37.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 93.955565578999995, 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:40:37.109 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:37.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:40:37.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:40:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:40:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:40:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:40:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:40:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:40:37.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:40:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:40:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:40:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:40:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:40:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:40:37.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:40:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:40:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:40:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:40:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.114 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 93.955547103000001, 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 66.192498791000006, 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:40:37.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:40:37.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:40:37.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:40:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:40:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.663339465, 2024-04-22T18:40:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:40:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:37.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:40:37.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.128 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.485632070000001, 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:37.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.485563798000001, 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.133 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:40:37.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:40:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:40:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:40:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:40:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:40:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:40:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.485395957000001, 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:40:37.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:40:37.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.142 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.520070482000001, 2024-04-22T18:40:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:40:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:37.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:40:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:40:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:40:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:40:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:40:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:40:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.419918295000002, 2024-04-22T18:40:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:40:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:40:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:40:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:40:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:40:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:40:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:40:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:40:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:40:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:40:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:37.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.711053775000003, 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:37.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:40:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:40:37.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:40:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:40:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:40:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:40:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:40:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:40:37.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:40:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:40:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:40:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:40:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:40:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:40:37.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 29.815015403, 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.157 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:40:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:40:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:40:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:40:37.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3330 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.079937117, 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43151673600000001, 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:37.161 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3330 2024-04-22T18:40:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:40:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:40:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024041+0000", 2024-04-22T18:40:37.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:37.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024045+0000", 2024-04-22T18:40:37.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3950000000000001e-06 2024-04-22T18:40:37.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024051+0000", 2024-04-22T18:40:37.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0260000000000004e-06 2024-04-22T18:40:37.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024056+0000", 2024-04-22T18:40:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4489999999999998e-06 2024-04-22T18:40:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028383+0000", 2024-04-22T18:40:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004326557 2024-04-22T18:40:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028471+0000", 2024-04-22T18:40:37.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8387999999999995e-05 2024-04-22T18:40:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.455559+0000", 2024-04-22T18:40:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.42708770699999998 2024-04-22T18:40:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3377 4.6 4:6700eab5:::smithi08937929-32:head [read 1812992~307712] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-04-22T18:40:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:40:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.6676117609999999, 2024-04-22T18:40:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.52001745799999999, 2024-04-22T18:40:37.165 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3377 2024-04-22T18:40:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:40:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436364+0000", 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436376+0000", 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2121e-05 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436394+0000", 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8094e-05 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436405+0000", 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0189e-05 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436471+0000", 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6817999999999996e-05 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436481+0000", 2024-04-22T18:40:37.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2720000000000003e-06 2024-04-22T18:40:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956084+0000", 2024-04-22T18:40:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51960320800000004 2024-04-22T18:40:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956232+0000", 2024-04-22T18:40:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014778600000000001 2024-04-22T18:40:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956385+0000", 2024-04-22T18:40:37.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153394 2024-04-22T18:40:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:131 3.1 3:a0a19997:::ee8ca23a7050e771c77dff761e6ea5a11ed57796f4e8a17cbb5804a7fdc937b9:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:40:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.7747421650000001, 2024-04-22T18:40:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51449564999999997, 2024-04-22T18:40:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:37.170 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:40:37.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:40:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329235+0000", 2024-04-22T18:40:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329244+0000", 2024-04-22T18:40:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6819999999999992e-06 2024-04-22T18:40:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329259+0000", 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698000000000001e-05 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329265+0000", 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1510000000000001e-06 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329311+0000", 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6096e-05 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.332269+0000", 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029581910000000002 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630603+0000", 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29833418 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630685+0000", 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1217999999999994e-05 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:37.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630813+0000", 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128706 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.631447+0000", 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063427000000000004 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843586+0000", 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21213873 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843623+0000", 2024-04-22T18:40:37.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6848999999999999e-05 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843733+0000", 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010982200000000001 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3441 4.b 4:d3068ee3:::smithi08937929-29:head [write 558065~619869] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1862661039999995, 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50272020399999995, 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.178 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3441 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917711+0000", 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917838+0000", 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012734899999999999 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917854+0000", 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6130999999999999e-05 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917863+0000", 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5969999999999992e-06 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992845+0000", 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.074982193000000003 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992919+0000", 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.993019+0000", 2024-04-22T18:40:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9718999999999999e-05 2024-04-22T18:40:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095554+0000", 2024-04-22T18:40:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102535193 2024-04-22T18:40:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420183+0000", 2024-04-22T18:40:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32462853600000002 2024-04-22T18:40:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420401+0000", 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021881400000000001 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420433+0000", 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079000000000002e-05 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3442 4.b 4:d3068ee3:::smithi08937929-29:head [write 1579835~486765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1815793929999998, 2024-04-22T18:40:37.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49822799099999998, 2024-04-22T18:40:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3442 2024-04-22T18:40:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:40:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922399+0000", 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922502+0000", 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102784 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922513+0000", 2024-04-22T18:40:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1051e-05 2024-04-22T18:40:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922518+0000", 2024-04-22T18:40:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8529999999999997e-06 2024-04-22T18:40:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093158+0000", 2024-04-22T18:40:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.17063958800000001 2024-04-22T18:40:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093230+0000", 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1803000000000004e-05 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093326+0000", 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6454999999999997e-05 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095557+0000", 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022306399999999999 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420484+0000", 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32492717700000001 2024-04-22T18:40:37.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420606+0000", 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012225800000000001 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420628+0000", 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1596000000000001e-05 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3443 4.b 4:d3068ee3:::smithi08937929-29:head [write 2468371~489694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:37.188 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1769773460000001, 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49372487199999998, 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3443 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:40:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927001+0000", 2024-04-22T18:40:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927117+0000", 2024-04-22T18:40:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011637 2024-04-22T18:40:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927128+0000", 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0696000000000001e-05 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927134+0000", 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8640000000000001e-06 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094288+0000", 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16715412900000001 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094359+0000", 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1639999999999998e-05 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094451+0000", 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1252e-05 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095560+0000", 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011095530000000001 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:37.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420661+0000", 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32510105500000003 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420710+0000", 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9018999999999999e-05 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420727+0000", 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6722e-05 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3444 4.b 4:d3068ee3:::smithi08937929-29:head [setxattr _header (56) in=7b,truncate 2958065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:40:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1768416290000001, 2024-04-22T18:40:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49366935299999998, 2024-04-22T18:40:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3444 2024-04-22T18:40:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927137+0000", 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927141+0000", 2024-04-22T18:40:37.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3270000000000001e-06 2024-04-22T18:40:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927144+0000", 2024-04-22T18:40:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9730000000000002e-06 2024-04-22T18:40:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927147+0000", 2024-04-22T18:40:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.959e-06 2024-04-22T18:40:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095118+0000", 2024-04-22T18:40:37.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16797115500000001 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095169+0000", 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1316000000000001e-05 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095260+0000", 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1184e-05 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095999+0000", 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073868200000000005 2024-04-22T18:40:37.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420757+0000", 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32475783499999999 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420781+0000", 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4022000000000001e-05 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420807+0000", 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6366000000000001e-05 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.198 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3445 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1768305349999997, 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.493780049, 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3445 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:37.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927150+0000", 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.542e-06 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927153+0000", 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0520000000000002e-06 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027218+0000", 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100064604 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095501+0000", 2024-04-22T18:40:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.068283431000000006 2024-04-22T18:40:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095545+0000", 2024-04-22T18:40:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4295999999999997e-05 2024-04-22T18:40:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420833+0000", 2024-04-22T18:40:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32528718400000001 2024-04-22T18:40:37.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420877+0000", 2024-04-22T18:40:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4403999999999997e-05 2024-04-22T18:40:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420929+0000", 2024-04-22T18:40:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1848999999999998e-05 2024-04-22T18:40:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.203 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.203 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:37.205 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:37.206 INFO:tasks.rados.rados.0.smithi089.stdout:1504: finishing write tid 1 to smithi08937929-49 2024-04-22T18:40:37.206 INFO:tasks.rados.rados.0.smithi089.stdout:1504: finishing write tid 2 to smithi08937929-49 2024-04-22T18:40:37.206 INFO:tasks.rados.rados.0.smithi089.stdout:1504: finishing write tid 3 to smithi08937929-49 2024-04-22T18:40:37.206 INFO:tasks.rados.rados.0.smithi089.stdout:1504: finishing write tid 4 to smithi08937929-49 2024-04-22T18:40:37.206 INFO:tasks.rados.rados.0.smithi089.stdout:1504: finishing write tid 5 to smithi08937929-49 2024-04-22T18:40:37.206 INFO:tasks.rados.rados.0.smithi089.stdout:1504: oid 49 updating version 0 to 124 2024-04-22T18:40:37.206 INFO:tasks.rados.rados.0.smithi089.stdout:1504: oid 49 version 124 is already newer than 123 2024-04-22T18:40:37.206 INFO:tasks.rados.rados.0.smithi089.stdout:1504: oid 49 updating version 124 to 126 2024-04-22T18:40:37.206 INFO:tasks.rados.rados.0.smithi089.stdout:1504: oid 49 version 126 is already newer than 125 2024-04-22T18:40:37.206 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 126 (ObjNum 299 snap 0 seq_num 299) dirty exists 2024-04-22T18:40:37.207 INFO:tasks.rados.rados.0.smithi089.stdout:1504: left oid 49 (ObjNum 299 snap 0 seq_num 299) 2024-04-22T18:40:37.207 INFO:tasks.rados.rados.0.smithi089.stdout:1504: done (14 left) 2024-04-22T18:40:37.207 INFO:tasks.rados.rados.0.smithi089.stdout:1506: done (13 left) 2024-04-22T18:40:37.207 INFO:tasks.rados.rados.0.smithi089.stdout:1509: done (12 left) 2024-04-22T18:40:37.207 INFO:tasks.rados.rados.0.smithi089.stdout:1510: done (11 left) 2024-04-22T18:40:37.207 INFO:tasks.rados.rados.0.smithi089.stdout:1511: done (10 left) 2024-04-22T18:40:37.207 INFO:tasks.rados.rados.0.smithi089.stdout:1514: done (9 left) 2024-04-22T18:40:37.207 INFO:tasks.rados.rados.0.smithi089.stdout:1516: done (8 left) 2024-04-22T18:40:37.207 INFO:tasks.rados.rados.0.smithi089.stdout:1518: done (7 left) 2024-04-22T18:40:37.207 INFO:tasks.rados.rados.0.smithi089.stdout:1519: done (6 left) 2024-04-22T18:40:37.207 INFO:tasks.rados.rados.0.smithi089.stdout:1524: tier_evict oid 8 2024-04-22T18:40:37.207 INFO:tasks.rados.rados.0.smithi089.stdout:1524: tier_evict oid 8 snap -1 2024-04-22T18:40:37.207 INFO:tasks.rados.rados.0.smithi089.stdout:1520: finishing set_chunk to oid 45 2024-04-22T18:40:37.207 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 45 set_chunk returned error code -95 offset: 155136 length: 2074112 2024-04-22T18:40:37.207 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 0 (ObjNum 266 snap 0 seq_num 266) dirty exists 2024-04-22T18:40:37.208 INFO:tasks.rados.rados.0.smithi089.stdout:1523: got (0) Success 2024-04-22T18:40:37.208 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 51 (ObjNum 226 snap 0 seq_num 226) dirty exists 2024-04-22T18:40:37.208 INFO:tasks.rados.rados.0.smithi089.stdout:1517: got (0) Success 2024-04-22T18:40:37.208 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 75 (ObjNum 275 snap 0 seq_num 275) dirty exists 2024-04-22T18:40:37.208 INFO:tasks.rados.rados.0.smithi089.stdout:1522: expect (ObjNum 296 snap 0 seq_num 296) 2024-04-22T18:40:37.208 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:37.205+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:37.305 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:37.306+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:37.309 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGRecoveryDelete(3.3 e106,103 3:efc8bf17:::299002303f2c559260d98e2aff0aa463bf66227aa186de94b269e72f47b45a85:head,59'192)", 2024-04-22T18:40:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:37.291279+0000", 2024-04-22T18:40:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.019325063, 2024-04-22T18:40:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019342247, 2024-04-22T18:40:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "no flag points reached", 2024-04-22T18:40:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:37.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:37.291279+0000", 2024-04-22T18:40:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:37.291279+0000", 2024-04-22T18:40:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:37.291277+0000", 2024-04-22T18:40:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:37.291288+0000", 2024-04-22T18:40:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.145e-05 2024-04-22T18:40:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:37.291307+0000", 2024-04-22T18:40:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8508999999999998e-05 2024-04-22T18:40:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:37.311 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 1 2024-04-22T18:40:37.311 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:37.312 INFO:tasks.rados.rados.0.smithi089.stdout:1513: got (0) Success 2024-04-22T18:40:37.312 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 103 (ObjNum 269 snap 0 seq_num 269) dirty exists 2024-04-22T18:40:37.312 INFO:tasks.rados.rados.0.smithi089.stdout:1521: finishing write tid 1 to smithi08937929-22 2024-04-22T18:40:37.313 INFO:tasks.rados.rados.0.smithi089.stdout:1521: finishing write tid 2 to smithi08937929-22 2024-04-22T18:40:37.313 INFO:tasks.rados.rados.0.smithi089.stdout:1513: done (6 left) 2024-04-22T18:40:37.313 INFO:tasks.rados.rados.0.smithi089.stdout:1517: done (5 left) 2024-04-22T18:40:37.313 INFO:tasks.rados.rados.0.smithi089.stdout:1520: done (4 left) 2024-04-22T18:40:37.313 INFO:tasks.rados.rados.0.smithi089.stdout:1522: done (3 left) 2024-04-22T18:40:37.313 INFO:tasks.rados.rados.0.smithi089.stdout:1523: done (2 left) 2024-04-22T18:40:37.313 INFO:tasks.rados.rados.0.smithi089.stdout:1525: tier_flush oid 13 2024-04-22T18:40:37.313 INFO:tasks.rados.rados.0.smithi089.stdout:1525: tier_flush oid 13 snap -1 2024-04-22T18:40:37.313 INFO:tasks.rados.rados.0.smithi089.stdout:1521: finishing write tid 3 to smithi08937929-22 2024-04-22T18:40:37.313 INFO:tasks.rados.rados.0.smithi089.stdout:1521: finishing write tid 4 to smithi08937929-22 2024-04-22T18:40:37.313 INFO:tasks.rados.rados.0.smithi089.stdout:1521: oid 22 updating version 0 to 111 2024-04-22T18:40:37.313 INFO:tasks.rados.rados.0.smithi089.stdout:1521: oid 22 version 111 is already newer than 110 2024-04-22T18:40:37.313 INFO:tasks.rados.rados.0.smithi089.stdout:1521: oid 22 updating version 111 to 112 2024-04-22T18:40:37.313 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 112 (ObjNum 303 snap 0 seq_num 303) dirty exists 2024-04-22T18:40:37.313 INFO:tasks.rados.rados.0.smithi089.stdout:1521: left oid 22 (ObjNum 303 snap 0 seq_num 303) 2024-04-22T18:40:37.314 INFO:tasks.rados.rados.0.smithi089.stdout:1524: got (22) Invalid argument 2024-04-22T18:40:37.314 INFO:tasks.rados.rados.0.smithi089.stdout:1521: done (2 left) 2024-04-22T18:40:37.314 INFO:tasks.rados.rados.0.smithi089.stdout:1524: done (1 left) 2024-04-22T18:40:37.314 INFO:tasks.rados.rados.0.smithi089.stdout:1526: set_chunk oid 34 target oid 2024-04-22T18:40:37.317 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:37.406 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:37.407+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:37.442 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:40:37.460 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:37.460 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:37.460 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:37.460 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:37.460 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:37.470 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:37.507 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:37.508+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:37.553 INFO:tasks.rados.rados.0.smithi089.stdout:1526: set_chunk oid 34 offset: 365056 length: 1889792 target oid dac21edf01ad44295778a5686720a781b2950a92372d95246547c64133f664ca offset: 0 snap -1 2024-04-22T18:40:37.553 INFO:tasks.rados.rados.0.smithi089.stdout:1527: read oid 14 snap -1 2024-04-22T18:40:37.553 INFO:tasks.rados.rados.0.smithi089.stdout:1527: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:40:37.553 INFO:tasks.rados.rados.0.smithi089.stdout:1528: write oid 35 current snap is 0 2024-04-22T18:40:37.553 INFO:tasks.rados.rados.0.smithi089.stdout:1525: got (2) No such file or directory 2024-04-22T18:40:37.553 INFO:tasks.rados.rados.0.smithi089.stdout:1525: got expected ENOENT (src dne) 2024-04-22T18:40:37.553 INFO:tasks.rados.rados.0.smithi089.stdout:1528: seq_num 304 ranges {709328=613463,1989820=645099} 2024-04-22T18:40:37.559 INFO:tasks.rados.rados.0.smithi089.stdout:1528: writing smithi08937929-35 from 709328 to 1322791 tid 1 2024-04-22T18:40:37.561 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:37.561 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:37.569 INFO:tasks.rados.rados.0.smithi089.stdout:1528: writing smithi08937929-35 from 1989820 to 2634919 tid 2 2024-04-22T18:40:37.569 INFO:tasks.rados.rados.0.smithi089.stdout:1525: done (3 left) 2024-04-22T18:40:37.569 INFO:tasks.rados.rados.0.smithi089.stdout:1529: tier_promote oid 28 2024-04-22T18:40:37.569 INFO:tasks.rados.rados.0.smithi089.stdout:1526: finishing set_chunk to oid 34 2024-04-22T18:40:37.569 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 34 set_chunk returned error code -95 offset: 365056 length: 1889792 2024-04-22T18:40:37.569 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 0 (ObjNum 277 snap 0 seq_num 277) dirty exists 2024-04-22T18:40:37.569 INFO:tasks.rados.rados.0.smithi089.stdout:1528: finishing write tid 1 to smithi08937929-35 2024-04-22T18:40:37.569 INFO:tasks.rados.rados.0.smithi089.stdout:1526: done (3 left) 2024-04-22T18:40:37.569 INFO:tasks.rados.rados.0.smithi089.stdout:1530: read oid 44 snap -1 2024-04-22T18:40:37.570 INFO:tasks.rados.rados.0.smithi089.stdout:1530: expect (ObjNum 187 snap 0 seq_num 187) 2024-04-22T18:40:37.570 INFO:tasks.rados.rados.0.smithi089.stdout:1531: read oid 16 snap -1 2024-04-22T18:40:37.570 INFO:tasks.rados.rados.0.smithi089.stdout:1531: expect (ObjNum 279 snap 0 seq_num 279) 2024-04-22T18:40:37.570 INFO:tasks.rados.rados.0.smithi089.stdout:1532: read oid 1 snap -1 2024-04-22T18:40:37.570 INFO:tasks.rados.rados.0.smithi089.stdout:1532: expect deleted 2024-04-22T18:40:37.570 INFO:tasks.rados.rados.0.smithi089.stdout:1533: set_chunk oid 3 target oid 2024-04-22T18:40:37.582 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:37.582 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:37.598 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:37.599 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:37.608 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:37.608+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:37.617 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:37.617 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:37.626 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:37.640 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:37.641+0000 7fec57bee640 -1 osd.7 106 handle_osd_map injecting map crc failure 2024-04-22T18:40:37.642 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:40:37.688 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:37.688 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:37.704 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:37.705 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:37.709 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:37.709+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:37.721 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:37.721 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:37.738 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:37.738 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:37.748 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:37.748 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:37.757 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:37.760 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:37.809 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:37.810+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:37.867 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:37.867 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:37.876 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:37.880 INFO:tasks.rados.rados.0.smithi089.stdout:1533: set_chunk oid 3 offset: 609280 length: 834560 target oid 778761ccec52b6c97750fc6126369c0d249b1e61a50c3b0711427d3d73d7f426 offset: 0 snap -1 2024-04-22T18:40:37.881 INFO:tasks.rados.rados.0.smithi089.stdout:1534: read oid 7 snap -1 2024-04-22T18:40:37.881 INFO:tasks.rados.rados.0.smithi089.stdout:1534: expect deleted 2024-04-22T18:40:37.881 INFO:tasks.rados.rados.0.smithi089.stdout:1535: read oid 32 snap -1 2024-04-22T18:40:37.881 INFO:tasks.rados.rados.0.smithi089.stdout:1535: expect (ObjNum 296 snap 0 seq_num 296) 2024-04-22T18:40:37.881 INFO:tasks.rados.rados.0.smithi089.stdout:1529: got (0) Success 2024-04-22T18:40:37.881 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 139 (ObjNum 265 snap 0 seq_num 265) dirty exists 2024-04-22T18:40:37.881 INFO:tasks.rados.rados.0.smithi089.stdout:1527: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:40:37.910 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:37.911+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:37.983 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:37.983 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:37.993 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:38.012 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:38.012+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:38.015 INFO:tasks.rados.rados.0.smithi089.stdout:1528: finishing write tid 2 to smithi08937929-35 2024-04-22T18:40:38.015 INFO:tasks.rados.rados.0.smithi089.stdout:1528: finishing write tid 3 to smithi08937929-35 2024-04-22T18:40:38.015 INFO:tasks.rados.rados.0.smithi089.stdout:1528: finishing write tid 4 to smithi08937929-35 2024-04-22T18:40:38.015 INFO:tasks.rados.rados.0.smithi089.stdout:1528: oid 35 updating version 0 to 114 2024-04-22T18:40:38.015 INFO:tasks.rados.rados.0.smithi089.stdout:1528: oid 35 version 114 is already newer than 113 2024-04-22T18:40:38.015 INFO:tasks.rados.rados.0.smithi089.stdout:1528: oid 35 updating version 114 to 115 2024-04-22T18:40:38.015 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 115 (ObjNum 304 snap 0 seq_num 304) dirty exists 2024-04-22T18:40:38.015 INFO:tasks.rados.rados.0.smithi089.stdout:1528: left oid 35 (ObjNum 304 snap 0 seq_num 304) 2024-04-22T18:40:38.015 INFO:tasks.rados.rados.0.smithi089.stdout:1530: expect (ObjNum 187 snap 0 seq_num 187) 2024-04-22T18:40:38.112 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:38.112+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:38.118 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:38.118 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:38.127 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:38.197 INFO:tasks.rados.rados.0.smithi089.stdout:1531: expect (ObjNum 279 snap 0 seq_num 279) 2024-04-22T18:40:38.213 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:38.214+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:38.270 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:38.270 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:38.279 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:38.313 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:38.314+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:38.364 INFO:tasks.rados.rados.0.smithi089.stdout:1535: expect (ObjNum 296 snap 0 seq_num 296) 2024-04-22T18:40:38.414 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:38.415+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:38.423 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:38.424 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:38.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:38.505 INFO:tasks.rados.rados.0.smithi089.stdout:1533: finishing set_chunk to oid 3 2024-04-22T18:40:38.505 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 68 (ObjNum 259 snap 0 seq_num 259) dirty exists 2024-04-22T18:40:38.505 INFO:tasks.rados.rados.0.smithi089.stdout:1527: done (8 left) 2024-04-22T18:40:38.505 INFO:tasks.rados.rados.0.smithi089.stdout:1528: done (7 left) 2024-04-22T18:40:38.505 INFO:tasks.rados.rados.0.smithi089.stdout:1529: done (6 left) 2024-04-22T18:40:38.505 INFO:tasks.rados.rados.0.smithi089.stdout:1530: done (5 left) 2024-04-22T18:40:38.505 INFO:tasks.rados.rados.0.smithi089.stdout:1531: done (4 left) 2024-04-22T18:40:38.505 INFO:tasks.rados.rados.0.smithi089.stdout:1532: done (3 left) 2024-04-22T18:40:38.505 INFO:tasks.rados.rados.0.smithi089.stdout:1533: done (2 left) 2024-04-22T18:40:38.505 INFO:tasks.rados.rados.0.smithi089.stdout:1534: done (1 left) 2024-04-22T18:40:38.506 INFO:tasks.rados.rados.0.smithi089.stdout:1535: done (0 left) 2024-04-22T18:40:38.506 INFO:tasks.rados.rados.0.smithi089.stdout:1536: write oid 48 current snap is 0 2024-04-22T18:40:38.506 INFO:tasks.rados.rados.0.smithi089.stdout:1536: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2024-04-22T18:40:38.510 INFO:tasks.rados.rados.0.smithi089.stdout:1536: writing smithi08937929-48 from 726135 to 1131200 tid 1 2024-04-22T18:40:38.515 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:38.516+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:38.518 INFO:tasks.rados.rados.0.smithi089.stdout:1536: writing smithi08937929-48 from 1764655 to 2553412 tid 2 2024-04-22T18:40:38.522 INFO:tasks.rados.rados.0.smithi089.stdout:1536: writing smithi08937929-48 from 3026248 to 3126135 tid 3 2024-04-22T18:40:38.522 INFO:tasks.rados.rados.0.smithi089.stdout:1537: set_chunk oid 1 target oid 2024-04-22T18:40:38.522 INFO:tasks.rados.rados.0.smithi089.stdout:1537: set_chunk oid 1 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:40:38.522 INFO:tasks.rados.rados.0.smithi089.stdout:1538: read oid 38 snap -1 2024-04-22T18:40:38.522 INFO:tasks.rados.rados.0.smithi089.stdout:1538: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:40:38.522 INFO:tasks.rados.rados.0.smithi089.stdout:1539: set_chunk oid 46 target oid 2024-04-22T18:40:38.542 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:38.542 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:38.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:38.616 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:38.617+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:38.653 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:38.653 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:38.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:38.717 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:38.718+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:38.728 INFO:tasks.rados.rados.0.smithi089.stdout:1539: set_chunk oid 46 offset: 700416 length: 1454592 target oid 641b583a056f8d1df08e653529e56953049def8b2692711bbc6853c4b0eaae4a offset: 0 snap -1 2024-04-22T18:40:38.728 INFO:tasks.rados.rados.0.smithi089.stdout:1540: write (excl) oid 33 current snap is 0 2024-04-22T18:40:38.729 INFO:tasks.rados.rados.0.smithi089.stdout:1538: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:40:38.764 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:38.765 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:38.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:38.812 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 2] out_osds: [5, 4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:40:38.812 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:40:38.812 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-04-22T18:40:38.812 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:40:38.819 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:38.820+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:38.856 INFO:tasks.rados.rados.0.smithi089.stdout:1536: finishing write tid 1 to smithi08937929-48 2024-04-22T18:40:38.856 INFO:tasks.rados.rados.0.smithi089.stdout:1537: finishing set_chunk to oid 1 2024-04-22T18:40:38.856 INFO:tasks.rados.rados.0.smithi089.stdout:1537: got expected ENOENT (src dne) 2024-04-22T18:40:38.856 INFO:tasks.rados.rados.0.smithi089.stdout:1536: finishing write tid 2 to smithi08937929-48 2024-04-22T18:40:38.856 INFO:tasks.rados.rados.0.smithi089.stdout:1536: finishing write tid 3 to smithi08937929-48 2024-04-22T18:40:38.856 INFO:tasks.rados.rados.0.smithi089.stdout:1536: finishing write tid 4 to smithi08937929-48 2024-04-22T18:40:38.856 INFO:tasks.rados.rados.0.smithi089.stdout:1536: finishing write tid 5 to smithi08937929-48 2024-04-22T18:40:38.857 INFO:tasks.rados.rados.0.smithi089.stdout:1536: oid 48 updating version 0 to 75 2024-04-22T18:40:38.857 INFO:tasks.rados.rados.0.smithi089.stdout:1536: oid 48 updating version 75 to 76 2024-04-22T18:40:38.857 INFO:tasks.rados.rados.0.smithi089.stdout:1536: oid 48 updating version 76 to 78 2024-04-22T18:40:38.857 INFO:tasks.rados.rados.0.smithi089.stdout:1536: oid 48 version 78 is already newer than 77 2024-04-22T18:40:38.857 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 78 (ObjNum 305 snap 0 seq_num 305) dirty exists 2024-04-22T18:40:38.857 INFO:tasks.rados.rados.0.smithi089.stdout:1536: left oid 48 (ObjNum 305 snap 0 seq_num 305) 2024-04-22T18:40:38.857 INFO:tasks.rados.rados.0.smithi089.stdout:1540: seq_num 306 ranges {742942=480314,1823137=648768} 2024-04-22T18:40:38.862 INFO:tasks.rados.rados.0.smithi089.stdout:1540: writing smithi08937929-33 from 742942 to 1223256 tid 1 2024-04-22T18:40:38.870 INFO:tasks.rados.rados.0.smithi089.stdout:1540: writing smithi08937929-33 from 1823137 to 2471905 tid 2 2024-04-22T18:40:38.870 INFO:tasks.rados.rados.0.smithi089.stdout:1536: done (4 left) 2024-04-22T18:40:38.870 INFO:tasks.rados.rados.0.smithi089.stdout:1537: done (3 left) 2024-04-22T18:40:38.871 INFO:tasks.rados.rados.0.smithi089.stdout:1538: done (2 left) 2024-04-22T18:40:38.871 INFO:tasks.rados.rados.0.smithi089.stdout:1541: tier_evict oid 13 2024-04-22T18:40:38.871 INFO:tasks.rados.rados.0.smithi089.stdout:1541: tier_evict oid 13 snap -1 2024-04-22T18:40:38.871 INFO:tasks.rados.rados.0.smithi089.stdout:1540: finishing write tid 1 to smithi08937929-33 2024-04-22T18:40:38.871 INFO:tasks.rados.rados.0.smithi089.stdout:1542: read oid 14 snap -1 2024-04-22T18:40:38.871 INFO:tasks.rados.rados.0.smithi089.stdout:1542: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:40:38.871 INFO:tasks.rados.rados.0.smithi089.stdout:1543: read oid 9 snap -1 2024-04-22T18:40:38.871 INFO:tasks.rados.rados.0.smithi089.stdout:1543: expect (ObjNum 283 snap 0 seq_num 283) 2024-04-22T18:40:38.871 INFO:tasks.rados.rados.0.smithi089.stdout:1544: write (excl) oid 18 current snap is 0 2024-04-22T18:40:38.871 INFO:tasks.rados.rados.0.smithi089.stdout:1544: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2024-04-22T18:40:38.877 INFO:tasks.rados.rados.0.smithi089.stdout:1544: writing smithi08937929-18 from 759749 to 1315312 tid 1 2024-04-22T18:40:38.877 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:38.877 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:38.883 INFO:tasks.rados.rados.0.smithi089.stdout:1544: writing smithi08937929-18 from 1765266 to 2557692 tid 2 2024-04-22T18:40:38.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:38.888 INFO:tasks.rados.rados.0.smithi089.stdout:1544: writing smithi08937929-18 from 2964741 to 3159749 tid 3 2024-04-22T18:40:38.888 INFO:tasks.rados.rados.0.smithi089.stdout:1545: read oid 22 snap -1 2024-04-22T18:40:38.888 INFO:tasks.rados.rados.0.smithi089.stdout:1545: expect (ObjNum 303 snap 0 seq_num 303) 2024-04-22T18:40:38.888 INFO:tasks.rados.rados.0.smithi089.stdout:1546: read oid 37 snap -1 2024-04-22T18:40:38.888 INFO:tasks.rados.rados.0.smithi089.stdout:1546: expect (ObjNum 267 snap 0 seq_num 267) 2024-04-22T18:40:38.888 INFO:tasks.rados.rados.0.smithi089.stdout:1547: read oid 28 snap -1 2024-04-22T18:40:38.888 INFO:tasks.rados.rados.0.smithi089.stdout:1547: expect (ObjNum 265 snap 0 seq_num 265) 2024-04-22T18:40:38.888 INFO:tasks.rados.rados.0.smithi089.stdout:1548: delete oid 10 current snap is 0 2024-04-22T18:40:38.928 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:38.929+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:39.020 INFO:tasks.rados.rados.0.smithi089.stdout:1539: finishing set_chunk to oid 46 2024-04-22T18:40:39.020 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 26 (ObjNum 278 snap 0 seq_num 278) dirty exists 2024-04-22T18:40:39.020 INFO:tasks.rados.rados.0.smithi089.stdout:1541: got (22) Invalid argument 2024-04-22T18:40:39.020 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:39.021+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:39.027 INFO:tasks.rados.rados.0.smithi089.stdout:1547: expect (ObjNum 265 snap 0 seq_num 265) 2024-04-22T18:40:39.029 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:39.029 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:39.038 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:39.102 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:40:39.103 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:40:39.114 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-04-22T18:40:38.981792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":244627059,"num_objects":201,"num_object_clones":0,"num_object_copies":402,"num_objects_missing_on_primary":101,"num_objects_missing":101,"num_objects_degraded":0,"num_objects_misplaced":114,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":3468,"num_read_kb":2897211,"num_write":2172,"num_write_kb":505994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":331,"num_bytes_recovered":160662127,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":31,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2251,"ondisk_log_size":2251,"up":70,"acting":70,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":65,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":403156,"kb_used_data":241792,"kb_used_omap":7,"kb_used_meta":151416,"kb_avail":471456044,"statfs":{"total":483183820800,"available":482770989056,"internally_reserved":0,"allocated":247595008,"data_stored":250584783,"data_compressed":5920473,"data_compressed_allocated":6303744,"data_compressed_original":13496320,"omap_allocated":7962,"internal_metadata":155050214},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,1,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12774129,"num_objects":-34,"num_object_clones":0,"num_object_copies":-68,"num_objects_missing_on_primary":101,"num_objects_missing":101,"num_objects_degraded":0,"num_objects_misplaced":43,"num_objects_unfound":0,"num_objects_dirty":-32,"num_whiteouts":0,"num_read":189,"num_read_kb":182964,"num_write":78,"num_write_kb":12270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":41153855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.177966"},"pg_stats":[{"pgid":"4.15","version":"54'29","reported_seq":34,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:34.805000+0000","last_change":"2024-04-22T18:40:33.895403+0000","last_active":"2024-04-22T18:40:34.805000+0000","last_peered":"2024-04-22T18:40:34.805000+0000","last_clean":"2024-04-22T18:40:34.805000+0000","last_became_active":"2024-04-22T18:40:13.180144+0000","last_became_peered":"2024-04-22T18:40:13.180144+0000","last_unstale":"2024-04-22T18:40:34.805000+0000","last_undegraded":"2024-04-22T18:40:34.805000+0000","last_fullsized":"2024-04-22T18:40:34.805000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'29","last_scrub_stamp":"2024-04-22T18:39:56.614637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:56.614637+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2915998,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":45460,"num_write":12,"num_write_kb":3102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"51'19","reported_seq":34,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:34.906806+0000","last_change":"2024-04-22T18:40:11.079224+0000","last_active":"2024-04-22T18:40:34.906806+0000","last_peered":"2024-04-22T18:40:34.906806+0000","last_clean":"2024-04-22T18:40:34.906806+0000","last_became_active":"2024-04-22T18:40:10.978840+0000","last_became_peered":"2024-04-22T18:40:10.978840+0000","last_unstale":"2024-04-22T18:40:34.906806+0000","last_undegraded":"2024-04-22T18:40:34.906806+0000","last_fullsized":"2024-04-22T18:40:34.906806+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1882368,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":22802,"num_write":10,"num_write_kb":3047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1349842,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"96'76","reported_seq":84,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:34.912124+0000","last_change":"2024-04-22T18:40:29.100526+0000","last_active":"2024-04-22T18:40:34.912124+0000","last_peered":"2024-04-22T18:40:34.912124+0000","last_clean":"2024-04-22T18:40:34.912124+0000","last_became_active":"2024-04-22T18:40:28.877828+0000","last_became_peered":"2024-04-22T18:40:28.877828+0000","last_unstale":"2024-04-22T18:40:34.912124+0000","last_undegraded":"2024-04-22T18:40:34.912124+0000","last_fullsized":"2024-04-22T18:40:34.912124+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'75","last_scrub_stamp":"2024-04-22T18:40:22.670296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.670296+0000","objects_scrubbed":2,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:44.049940+0000","scrub_duration":0.20951747300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6638765,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4062110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"102'38","reported_seq":201,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:34.808938+0000","last_change":"2024-04-22T18:40:03.451290+0000","last_active":"2024-04-22T18:40:34.808938+0000","last_peered":"2024-04-22T18:40:34.808938+0000","last_clean":"2024-04-22T18:40:34.808938+0000","last_became_active":"2024-04-22T18:40:02.844371+0000","last_became_peered":"2024-04-22T18:40:02.844371+0000","last_unstale":"2024-04-22T18:40:34.808938+0000","last_undegraded":"2024-04-22T18:40:34.808938+0000","last_fullsized":"2024-04-22T18:40:34.808938+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3268995,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":92,"num_read_kb":83390,"num_write":20,"num_write_kb":5500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":970119,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"103'123","reported_seq":386,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:34.808865+0000","last_change":"2024-04-22T18:40:26.844287+0000","last_active":"2024-04-22T18:40:34.808865+0000","last_peered":"2024-04-22T18:40:34.808865+0000","last_clean":"2024-04-22T18:40:34.808865+0000","last_became_active":"2024-04-22T18:40:26.742351+0000","last_became_peered":"2024-04-22T18:40:26.742351+0000","last_unstale":"2024-04-22T18:40:34.808865+0000","last_undegraded":"2024-04-22T18:40:34.808865+0000","last_fullsized":"2024-04-22T18:40:34.808865+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'113","last_scrub_stamp":"2024-04-22T18:40:20.291988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.291988+0000","objects_scrubbed":2,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:22.684852+0000","scrub_duration":0.10270805500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5579990,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":8049,"num_write":12,"num_write_kb":3279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3940523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"31'5","reported_seq":138,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:34.806582+0000","last_change":"2024-04-22T18:40:10.212897+0000","last_active":"2024-04-22T18:40:34.806582+0000","last_peered":"2024-04-22T18:40:34.806582+0000","last_clean":"2024-04-22T18:40:34.806582+0000","last_became_active":"2024-04-22T18:39:59.265189+0000","last_became_peered":"2024-04-22T18:39:59.265189+0000","last_unstale":"2024-04-22T18:40:34.806582+0000","last_undegraded":"2024-04-22T18:40:34.806582+0000","last_fullsized":"2024-04-22T18:40:34.806582+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":5,"last_scrub":"31'5","last_scrub_stamp":"2024-04-22T18:40:10.212815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.212815+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:17.620115+0000","scrub_duration":0.0015068670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":107,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:34.908365+0000","last_change":"2024-04-22T18:40:19.137046+0000","last_active":"2024-04-22T18:40:34.908365+0000","last_peered":"2024-04-22T18:40:34.908365+0000","last_clean":"2024-04-22T18:40:34.908365+0000","last_became_active":"2024-04-22T18:38:39.025741+0000","last_became_peered":"2024-04-22T18:38:39.025741+0000","last_unstale":"2024-04-22T18:40:34.908365+0000","last_undegraded":"2024-04-22T18:40:34.908365+0000","last_fullsized":"2024-04-22T18:40:34.908365+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:19.136933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:19.136933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:21.035233+0000","scrub_duration":0.0023924060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"100'150","reported_seq":515,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-22T18:40:36.329312+0000","last_change":"2024-04-22T18:40:36.329312+0000","last_active":"2024-04-22T18:40:36.329312+0000","last_peered":"2024-04-22T18:40:36.329312+0000","last_clean":"2024-04-22T18:40:36.329312+0000","last_became_active":"2024-04-22T18:40:34.914043+0000","last_became_peered":"2024-04-22T18:40:34.914043+0000","last_unstale":"2024-04-22T18:40:36.329312+0000","last_undegraded":"2024-04-22T18:40:36.329312+0000","last_fullsized":"2024-04-22T18:40:36.329312+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:06.180858+0000","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13882681,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":207,"num_read_kb":239346,"num_write":45,"num_write_kb":12471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":26002234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":97,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:34.809882+0000","last_change":"2024-04-22T18:38:39.110499+0000","last_active":"2024-04-22T18:40:34.809882+0000","last_peered":"2024-04-22T18:40:34.809882+0000","last_clean":"2024-04-22T18:40:34.809882+0000","last_became_active":"2024-04-22T18:38:39.010173+0000","last_became_peered":"2024-04-22T18:38:39.010173+0000","last_unstale":"2024-04-22T18:40:34.809882+0000","last_undegraded":"2024-04-22T18:40:34.809882+0000","last_fullsized":"2024-04-22T18:40:34.809882+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"71'14","reported_seq":151,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:34.806604+0000","last_change":"2024-04-22T18:39:54.378188+0000","last_active":"2024-04-22T18:40:34.806604+0000","last_peered":"2024-04-22T18:40:34.806604+0000","last_clean":"2024-04-22T18:40:34.806604+0000","last_became_active":"2024-04-22T18:39:46.805569+0000","last_became_peered":"2024-04-22T18:39:46.805569+0000","last_unstale":"2024-04-22T18:40:34.806604+0000","last_undegraded":"2024-04-22T18:40:34.806604+0000","last_fullsized":"2024-04-22T18:40:34.806604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'13","last_scrub_stamp":"2024-04-22T18:39:54.377980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.377980+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:00.921987+0000","scrub_duration":0.30260881099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3378207,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":46866,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":114,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:35.008146+0000","last_change":"2024-04-22T18:40:35.007077+0000","last_active":"2024-04-22T18:40:35.008146+0000","last_peered":"2024-04-22T18:40:35.008146+0000","last_clean":"2024-04-22T18:40:35.008146+0000","last_became_active":"2024-04-22T18:40:35.006777+0000","last_became_peered":"2024-04-22T18:40:35.006777+0000","last_unstale":"2024-04-22T18:40:35.008146+0000","last_undegraded":"2024-04-22T18:40:35.008146+0000","last_fullsized":"2024-04-22T18:40:35.008146+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:09.538542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:09.538542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:18.970457+0000","scrub_duration":0.35201753200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"105'77","reported_seq":271,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-22T18:40:35.964734+0000","last_change":"2024-04-22T18:40:35.964284+0000","last_active":"2024-04-22T18:40:35.964734+0000","last_peered":"2024-04-22T18:40:35.964734+0000","last_clean":"2024-04-22T18:40:35.964734+0000","last_became_active":"2024-04-22T18:40:34.910015+0000","last_became_peered":"2024-04-22T18:40:34.910015+0000","last_unstale":"2024-04-22T18:40:35.964734+0000","last_undegraded":"2024-04-22T18:40:35.964734+0000","last_fullsized":"2024-04-22T18:40:35.964734+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'67","last_scrub_stamp":"2024-04-22T18:39:53.916646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.916646+0000","objects_scrubbed":2,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:06.623573+0000","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5848869,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":16271,"num_write":7,"num_write_kb":1109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7812071,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":105,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:35.016591+0000","last_change":"2024-04-22T18:40:35.014535+0000","last_active":"2024-04-22T18:40:35.016591+0000","last_peered":"2024-04-22T18:40:35.016591+0000","last_clean":"2024-04-22T18:40:35.016591+0000","last_became_active":"2024-04-22T18:40:35.014154+0000","last_became_peered":"2024-04-22T18:40:35.014154+0000","last_unstale":"2024-04-22T18:40:35.016591+0000","last_undegraded":"2024-04-22T18:40:35.016591+0000","last_fullsized":"2024-04-22T18:40:35.016591+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"64'38","reported_seq":223,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:35.557017+0000","last_change":"2024-04-22T18:40:35.556480+0000","last_active":"2024-04-22T18:40:35.557017+0000","last_peered":"2024-04-22T18:40:35.557017+0000","last_clean":"2024-04-22T18:40:35.557017+0000","last_became_active":"2024-04-22T18:40:34.911555+0000","last_became_peered":"2024-04-22T18:40:34.911555+0000","last_unstale":"2024-04-22T18:40:35.557017+0000","last_undegraded":"2024-04-22T18:40:35.557017+0000","last_fullsized":"2024-04-22T18:40:35.557017+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2932821,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":88,"num_read_kb":124012,"num_write":20,"num_write_kb":5595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":8531127,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"105'285","reported_seq":502,"reported_epoch":105,"state":"active+recovering","last_fresh":"2024-04-22T18:40:36.359937+0000","last_change":"2024-04-22T18:40:35.517639+0000","last_active":"2024-04-22T18:40:36.359937+0000","last_peered":"2024-04-22T18:40:36.359937+0000","last_clean":"2024-04-22T18:40:26.741597+0000","last_became_active":"2024-04-22T18:40:35.239662+0000","last_became_peered":"2024-04-22T18:40:35.239662+0000","last_unstale":"2024-04-22T18:40:36.359937+0000","last_undegraded":"2024-04-22T18:40:36.359937+0000","last_fullsized":"2024-04-22T18:40:36.359937+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27298068,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":52,"num_objects_missing":52,"num_objects_degraded":0,"num_objects_misplaced":55,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":243,"num_read_kb":4001,"num_write":346,"num_write_kb":85059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":11081486,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":20},{"shards":"0,4,6","objects":11},{"shards":"0,6","objects":4},{"shards":"1,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":48,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:34.809583+0000","last_change":"2024-04-22T18:40:33.899901+0000","last_active":"2024-04-22T18:40:34.809583+0000","last_peered":"2024-04-22T18:40:34.809583+0000","last_clean":"2024-04-22T18:40:34.809583+0000","last_became_active":"2024-04-22T18:40:02.945025+0000","last_became_peered":"2024-04-22T18:40:02.945025+0000","last_unstale":"2024-04-22T18:40:34.809583+0000","last_undegraded":"2024-04-22T18:40:34.809583+0000","last_fullsized":"2024-04-22T18:40:34.809583+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"91'31","reported_seq":205,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:35.513424+0000","last_change":"2024-04-22T18:40:21.426510+0000","last_active":"2024-04-22T18:40:35.513424+0000","last_peered":"2024-04-22T18:40:35.513424+0000","last_clean":"2024-04-22T18:40:35.513424+0000","last_became_active":"2024-04-22T18:39:51.943071+0000","last_became_peered":"2024-04-22T18:39:51.943071+0000","last_unstale":"2024-04-22T18:40:35.513424+0000","last_undegraded":"2024-04-22T18:40:35.513424+0000","last_fullsized":"2024-04-22T18:40:35.513424+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-04-22T18:40:21.426350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.426350+0000","objects_scrubbed":1,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.678685+0000","scrub_duration":0.30334366400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2605118,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":29878,"num_write":4,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"104'204","reported_seq":525,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-22T18:40:35.918130+0000","last_change":"2024-04-22T18:40:20.465005+0000","last_active":"2024-04-22T18:40:35.918130+0000","last_peered":"2024-04-22T18:40:35.918130+0000","last_clean":"2024-04-22T18:40:35.918130+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:40:35.918130+0000","last_undegraded":"2024-04-22T18:40:35.918130+0000","last_fullsized":"2024-04-22T18:40:35.918130+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'176","last_scrub_stamp":"2024-04-22T18:40:20.464799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.464799+0000","objects_scrubbed":31,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:45.914514+0000","scrub_duration":0.30629975799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27202147,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":173,"num_read_kb":2350,"num_write":257,"num_write_kb":59936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":64,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:34.812023+0000","last_change":"2024-04-22T18:40:10.403338+0000","last_active":"2024-04-22T18:40:34.812023+0000","last_peered":"2024-04-22T18:40:34.812023+0000","last_clean":"2024-04-22T18:40:34.812023+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:40:34.812023+0000","last_undegraded":"2024-04-22T18:40:34.812023+0000","last_fullsized":"2024-04-22T18:40:34.812023+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:10.403189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.403189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:34.232014+0000","scrub_duration":0.201433005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"83'49","reported_seq":278,"reported_epoch":105,"state":"active+clean+remapped","last_fresh":"2024-04-22T18:40:36.232036+0000","last_change":"2024-04-22T18:40:36.232036+0000","last_active":"2024-04-22T18:40:36.232036+0000","last_peered":"2024-04-22T18:40:36.232036+0000","last_clean":"2024-04-22T18:40:36.232036+0000","last_became_active":"2024-04-22T18:40:35.921933+0000","last_became_peered":"2024-04-22T18:40:35.921933+0000","last_unstale":"2024-04-22T18:40:36.232036+0000","last_undegraded":"2024-04-22T18:40:36.232036+0000","last_fullsized":"2024-04-22T18:40:36.232036+0000","mapping_epoch":104,"log_start":"83'49","ondisk_log_start":"83'49","created":20,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":49,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5445485,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":133537,"num_write":27,"num_write_kb":6881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4509531,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,0],"avail_no_missing":["1","0","7"],"object_location_counts":[{"shards":"0,1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"105'246","reported_seq":466,"reported_epoch":105,"state":"active+recovering","last_fresh":"2024-04-22T18:40:36.129540+0000","last_change":"2024-04-22T18:40:35.139872+0000","last_active":"2024-04-22T18:40:36.129540+0000","last_peered":"2024-04-22T18:40:36.129540+0000","last_clean":"2024-04-22T18:40:27.674861+0000","last_became_active":"2024-04-22T18:40:35.038135+0000","last_became_peered":"2024-04-22T18:40:35.038135+0000","last_unstale":"2024-04-22T18:40:36.129540+0000","last_undegraded":"2024-04-22T18:40:36.129540+0000","last_fullsized":"2024-04-22T18:40:36.129540+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'209","last_scrub_stamp":"2024-04-22T18:40:18.683442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:18.683442+0000","objects_scrubbed":24,"log_size":246,"log_dups_size":0,"ondisk_log_size":246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:21.496831+0000","scrub_duration":0.204344365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25126812,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":49,"num_objects_missing":49,"num_objects_degraded":0,"num_objects_misplaced":56,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":217,"num_read_kb":5181,"num_write":306,"num_write_kb":71825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":7992874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,3","objects":3},{"shards":"0,3,4","objects":11},{"shards":"0,4","objects":21},{"shards":"3,7","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":106,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:34.804862+0000","last_change":"2024-04-22T18:40:22.521828+0000","last_active":"2024-04-22T18:40:34.804862+0000","last_peered":"2024-04-22T18:40:34.804862+0000","last_clean":"2024-04-22T18:40:34.804862+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:40:34.804862+0000","last_undegraded":"2024-04-22T18:40:34.804862+0000","last_fullsized":"2024-04-22T18:40:34.804862+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.521638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:43.407910+0000","scrub_duration":0.30190717299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"101'74","reported_seq":308,"reported_epoch":105,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:40:36.240973+0000","last_change":"2024-04-22T18:40:36.123466+0000","last_active":"2024-04-22T18:40:36.240973+0000","last_peered":"2024-04-22T18:40:36.240973+0000","last_clean":"2024-04-22T18:40:32.014508+0000","last_became_active":"2024-04-22T18:40:35.920106+0000","last_became_peered":"2024-04-22T18:40:35.920106+0000","last_unstale":"2024-04-22T18:40:36.240973+0000","last_undegraded":"2024-04-22T18:40:36.240973+0000","last_fullsized":"2024-04-22T18:40:36.240973+0000","mapping_epoch":104,"log_start":"96'71","ondisk_log_start":"96'71","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":81,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:11.654569+0000","scrub_duration":0.22461577699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8252336,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":57,"num_read_kb":79546,"num_write":12,"num_write_kb":3773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"104'235","reported_seq":524,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-22T18:40:35.956931+0000","last_change":"2024-04-22T18:40:21.871081+0000","last_active":"2024-04-22T18:40:35.956931+0000","last_peered":"2024-04-22T18:40:35.956931+0000","last_clean":"2024-04-22T18:40:35.956931+0000","last_became_active":"2024-04-22T18:40:10.784978+0000","last_became_peered":"2024-04-22T18:40:10.784978+0000","last_unstale":"2024-04-22T18:40:35.956931+0000","last_undegraded":"2024-04-22T18:40:35.956931+0000","last_fullsized":"2024-04-22T18:40:35.956931+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'212","last_scrub_stamp":"2024-04-22T18:40:21.870855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.870855+0000","objects_scrubbed":46,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:28.179796+0000","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29322427,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":219,"num_read_kb":4762,"num_write":301,"num_write_kb":63032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":13435025,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":119,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:34.812006+0000","last_change":"2024-04-22T18:40:07.420765+0000","last_active":"2024-04-22T18:40:34.812006+0000","last_peered":"2024-04-22T18:40:34.812006+0000","last_clean":"2024-04-22T18:40:34.812006+0000","last_became_active":"2024-04-22T18:40:02.741862+0000","last_became_peered":"2024-04-22T18:40:02.741862+0000","last_unstale":"2024-04-22T18:40:34.812006+0000","last_undegraded":"2024-04-22T18:40:34.812006+0000","last_fullsized":"2024-04-22T18:40:34.812006+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:07.420618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:07.420618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.642486+0000","scrub_duration":0.30164867299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":271,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:35.216919+0000","last_change":"2024-04-22T18:40:35.216423+0000","last_active":"2024-04-22T18:40:35.216919+0000","last_peered":"2024-04-22T18:40:35.216919+0000","last_clean":"2024-04-22T18:40:35.216919+0000","last_became_active":"2024-04-22T18:40:34.907666+0000","last_became_peered":"2024-04-22T18:40:34.907666+0000","last_unstale":"2024-04-22T18:40:35.216919+0000","last_undegraded":"2024-04-22T18:40:35.216919+0000","last_fullsized":"2024-04-22T18:40:35.216919+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:39:53.346787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.346787+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:15.525107+0000","scrub_duration":0.20273052499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"104'59","reported_seq":246,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-22T18:40:35.956968+0000","last_change":"2024-04-22T18:40:29.696372+0000","last_active":"2024-04-22T18:40:35.956968+0000","last_peered":"2024-04-22T18:40:35.956968+0000","last_clean":"2024-04-22T18:40:35.956968+0000","last_became_active":"2024-04-22T18:40:28.989193+0000","last_became_peered":"2024-04-22T18:40:28.989193+0000","last_unstale":"2024-04-22T18:40:35.956968+0000","last_undegraded":"2024-04-22T18:40:35.956968+0000","last_fullsized":"2024-04-22T18:40:35.956968+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'38","last_scrub_stamp":"2024-04-22T18:39:55.746067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:55.746067+0000","objects_scrubbed":1,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:20.212013+0000","scrub_duration":0.60877772299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7613571,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":73214,"num_write":22,"num_write_kb":6499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4764796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"79'52","reported_seq":231,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:34.804808+0000","last_change":"2024-04-22T18:40:22.839561+0000","last_active":"2024-04-22T18:40:34.804808+0000","last_peered":"2024-04-22T18:40:34.804808+0000","last_clean":"2024-04-22T18:40:34.804808+0000","last_became_active":"2024-04-22T18:40:22.839321+0000","last_became_peered":"2024-04-22T18:40:22.839321+0000","last_unstale":"2024-04-22T18:40:34.804808+0000","last_undegraded":"2024-04-22T18:40:34.804808+0000","last_fullsized":"2024-04-22T18:40:34.804808+0000","mapping_epoch":72,"log_start":"79'52","ondisk_log_start":"79'52","created":20,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":61,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3495856,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":178,"num_read_kb":241214,"num_write":79,"num_write_kb":20816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5051448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"105'114","reported_seq":439,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-22T18:40:36.362412+0000","last_change":"2024-04-22T18:40:32.995107+0000","last_active":"2024-04-22T18:40:36.362412+0000","last_peered":"2024-04-22T18:40:36.362412+0000","last_clean":"2024-04-22T18:40:36.362412+0000","last_became_active":"2024-04-22T18:40:32.992029+0000","last_became_peered":"2024-04-22T18:40:32.992029+0000","last_unstale":"2024-04-22T18:40:36.362412+0000","last_undegraded":"2024-04-22T18:40:36.362412+0000","last_fullsized":"2024-04-22T18:40:36.362412+0000","mapping_epoch":101,"log_start":"84'88","ondisk_log_start":"84'88","created":20,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":26,"log_dups_size":106,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:11.825208+0000","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13361730,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":71,"num_read_kb":72816,"num_write":38,"num_write_kb":11469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9980228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"104'94","reported_seq":303,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:35.521886+0000","last_change":"2024-04-22T18:40:22.140818+0000","last_active":"2024-04-22T18:40:35.521886+0000","last_peered":"2024-04-22T18:40:35.521886+0000","last_clean":"2024-04-22T18:40:35.521886+0000","last_became_active":"2024-04-22T18:38:48.277922+0000","last_became_peered":"2024-04-22T18:38:48.277922+0000","last_unstale":"2024-04-22T18:40:35.521886+0000","last_undegraded":"2024-04-22T18:40:35.521886+0000","last_fullsized":"2024-04-22T18:40:35.521886+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:39.617182+0000","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9411986,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":250,"num_read_kb":295077,"num_write":96,"num_write_kb":25457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"104'126","reported_seq":413,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-22T18:40:36.165987+0000","last_change":"2024-04-22T18:40:36.165101+0000","last_active":"2024-04-22T18:40:36.165987+0000","last_peered":"2024-04-22T18:40:36.165987+0000","last_clean":"2024-04-22T18:40:36.165987+0000","last_became_active":"2024-04-22T18:40:35.110751+0000","last_became_peered":"2024-04-22T18:40:35.110751+0000","last_unstale":"2024-04-22T18:40:36.165987+0000","last_undegraded":"2024-04-22T18:40:36.165987+0000","last_fullsized":"2024-04-22T18:40:36.165987+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'109","last_scrub_stamp":"2024-04-22T18:40:17.658363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:17.658363+0000","objects_scrubbed":5,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:37.958968+0000","scrub_duration":0.202176033,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14302889,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":394,"num_read_kb":426512,"num_write":131,"num_write_kb":38541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8880696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"101'109","reported_seq":368,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:34.806765+0000","last_change":"2024-04-22T18:40:29.506777+0000","last_active":"2024-04-22T18:40:34.806765+0000","last_peered":"2024-04-22T18:40:34.806765+0000","last_clean":"2024-04-22T18:40:34.806765+0000","last_became_active":"2024-04-22T18:40:28.975797+0000","last_became_peered":"2024-04-22T18:40:28.975797+0000","last_unstale":"2024-04-22T18:40:34.806765+0000","last_undegraded":"2024-04-22T18:40:34.806765+0000","last_fullsized":"2024-04-22T18:40:34.806765+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9227109,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":387,"num_read_kb":454521,"num_write":114,"num_write_kb":32389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7572471,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"104'47","reported_seq":258,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:35.328903+0000","last_change":"2024-04-22T18:40:34.755224+0000","last_active":"2024-04-22T18:40:35.328903+0000","last_peered":"2024-04-22T18:40:35.328903+0000","last_clean":"2024-04-22T18:40:35.328903+0000","last_became_active":"2024-04-22T18:40:28.778850+0000","last_became_peered":"2024-04-22T18:40:28.778850+0000","last_unstale":"2024-04-22T18:40:35.328903+0000","last_undegraded":"2024-04-22T18:40:35.328903+0000","last_fullsized":"2024-04-22T18:40:35.328903+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'46","last_scrub_stamp":"2024-04-22T18:40:34.755106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:34.755106+0000","objects_scrubbed":2,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:52.199053+0000","scrub_duration":0.40192567099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2789995,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":170,"num_read_kb":182885,"num_write":50,"num_write_kb":14374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":10972843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"100'28","reported_seq":181,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:35.513886+0000","last_change":"2024-04-22T18:40:29.090636+0000","last_active":"2024-04-22T18:40:35.513886+0000","last_peered":"2024-04-22T18:40:35.513886+0000","last_clean":"2024-04-22T18:40:35.513886+0000","last_became_active":"2024-04-22T18:40:28.778111+0000","last_became_peered":"2024-04-22T18:40:28.778111+0000","last_unstale":"2024-04-22T18:40:35.513886+0000","last_undegraded":"2024-04-22T18:40:35.513886+0000","last_fullsized":"2024-04-22T18:40:35.513886+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'23","last_scrub_stamp":"2024-04-22T18:40:20.587606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.587606+0000","objects_scrubbed":1,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.710668+0000","scrub_duration":0.10225366599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4924517,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":121421,"num_write":29,"num_write_kb":8624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2364451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"95'72","reported_seq":248,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-22T18:40:35.611593+0000","last_change":"2024-04-22T18:38:48.173640+0000","last_active":"2024-04-22T18:40:35.611593+0000","last_peered":"2024-04-22T18:40:35.611593+0000","last_clean":"2024-04-22T18:40:35.611593+0000","last_became_active":"2024-04-22T18:38:48.173369+0000","last_became_peered":"2024-04-22T18:38:48.173369+0000","last_unstale":"2024-04-22T18:40:35.611593+0000","last_undegraded":"2024-04-22T18:40:35.611593+0000","last_fullsized":"2024-04-22T18:40:35.611593+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7327951,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":193,"num_read_kb":184000,"num_write":75,"num_write_kb":19712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":135087237,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":2500,"num_read_kb":2880817,"num_write":803,"num_write_kb":223726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":106764490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":31,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210722816,"data_stored":210020678,"data_compressed":531589,"data_compressed_allocated":557056,"data_compressed_original":847872,"omap_allocated":0,"internal_metadata":0},"log_size":1164,"ondisk_log_size":1164,"up":44,"acting":44,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":108949454,"num_objects":152,"num_object_clones":0,"num_object_copies":304,"num_objects_missing_on_primary":101,"num_objects_missing":101,"num_objects_degraded":0,"num_objects_misplaced":111,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":852,"num_read_kb":16294,"num_write":1210,"num_write_kb":279852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":53368709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":154923008,"data_stored":164281391,"data_compressed":9120847,"data_compressed_allocated":9707520,"data_compressed_original":19587072,"omap_allocated":0,"internal_metadata":0},"log_size":970,"ondisk_log_size":970,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1586784,"data_compressed":90130,"data_compressed_allocated":122880,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55688,"kb_used_data":23620,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94316152,"statfs":{"total":96636764160,"available":96579739648,"internally_reserved":0,"allocated":24186880,"data_stored":24024242,"data_compressed":189598,"data_compressed_allocated":208896,"data_compressed_original":774144,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.49199999999999999},{"interface":"front","average":{"1min":4.508,"5min":4.508,"15min":4.508},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":85.542000000000002,"5min":85.542000000000002,"15min":85.542000000000002},"last":0.254}]},{"osd":1,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.8650000000000002,"5min":4.8650000000000002,"15min":4.8650000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":79.927999999999997,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.41099999999999998},{"interface":"front","average":{"1min":13.048,"5min":13.048,"15min":13.048},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":99.388999999999996,"5min":99.388999999999996,"15min":99.388999999999996},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.085,"5min":4.085,"15min":4.085},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":77.320999999999998,"5min":77.320999999999998,"15min":77.320999999999998},"last":0.434},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.52300000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.1509999999999998,"5min":4.1509999999999998,"15min":4.1509999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":77.361000000000004,"5min":77.361000000000004,"15min":77.361000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.55000000000000004}]},{"osd":4,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":4.4589999999999996,"5min":4.4589999999999996,"15min":4.4589999999999996},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":85.509,"5min":85.509,"15min":85.509},"last":0.32500000000000001},{"interface":"front","average":{"1min":22.872,"5min":22.872,"15min":22.872},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":327.77300000000002,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.50600000000000001}]},{"osd":5,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":3.0710000000000002,"5min":3.0710000000000002,"15min":3.0710000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":55.116999999999997,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.57699999999999996},{"interface":"front","average":{"1min":7.1619999999999999,"5min":7.1619999999999999,"15min":7.1619999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":85.475999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.44800000000000001}]},{"osd":6,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":13.276,"5min":13.276,"15min":13.276},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":99.620999999999995,"5min":99.620999999999995,"15min":99.620999999999995},"last":0.34699999999999998},{"interface":"front","average":{"1min":4.133,"5min":4.133,"15min":4.133},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":77.405000000000001,"5min":77.405000000000001,"15min":77.405000000000001},"last":0.59999999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95688,"kb_used_data":61148,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94276152,"statfs":{"total":96636764160,"available":96538779648,"internally_reserved":0,"allocated":62615552,"data_stored":63971627,"data_compressed":1721294,"data_compressed_allocated":1835008,"data_compressed_original":4116480,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.794,"15min":1.794},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":32.097999999999999,"5min":32.097999999999999,"15min":32.097999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":25.808,"5min":25.808,"15min":25.808},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":166.85499999999999,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":37.161999999999999,"5min":37.161999999999999,"15min":37.161999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":10.092000000000001,"5min":10.092000000000001,"15min":10.092000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":222.55199999999999,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.56599999999999995},{"interface":"front","average":{"1min":18.803999999999998,"5min":18.803999999999998,"15min":18.803999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":166.81800000000001,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.44400000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":21.518999999999998,"5min":21.518999999999998,"15min":21.518999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":171.99100000000001,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.46999999999999997}]},{"osd":4,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":7.6029999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":81.697000000000003,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.52400000000000002},{"interface":"front","average":{"1min":19.327999999999999,"5min":19.327999999999999,"15min":19.327999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":166.733,"5min":166.733,"15min":166.733},"last":0.58799999999999997}]},{"osd":5,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.627}]},{"osd":7,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":18.850999999999999,"5min":18.850999999999999,"15min":18.850999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":166.79400000000001,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.53500000000000003}]}]},{"osd":5,"up_from":0,"seq":55834574874,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":55834574874,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86892,"kb_used_data":54732,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94284948,"statfs":{"total":96636764160,"available":96547786752,"internally_reserved":0,"allocated":56045568,"data_stored":56806876,"data_compressed":1536402,"data_compressed_allocated":1630208,"data_compressed_original":3264512,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":12.199999999999999,"5min":12.199999999999999,"15min":12.199999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":98.957999999999998,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.40600000000000003}]},{"osd":1,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":5.0449999999999999,"5min":5.0449999999999999,"15min":5.0449999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":94.063000000000002,"5min":94.063000000000002,"15min":94.063000000000002},"last":197.02099999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.375}]},{"osd":2,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":12.901999999999999,"5min":12.901999999999999,"15min":12.901999999999999},"last":25.492000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.42399999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":4.1079999999999997,"5min":4.1079999999999997,"15min":4.1079999999999997},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":71.998000000000005,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.442},{"interface":"front","average":{"1min":4.1379999999999999,"5min":4.1379999999999999,"15min":4.1379999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":72.227000000000004,"5min":72.227000000000004,"15min":72.227000000000004},"last":25.559000000000001}]},{"osd":5,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":13.94,"5min":13.94,"15min":13.94},"last":25.52},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":13.965,"5min":13.965,"15min":13.965},"last":0.48099999999999998}]},{"osd":6,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":96.140000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.58199999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51744,"kb_used_data":24720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94320096,"statfs":{"total":96636764160,"available":96583778304,"internally_reserved":0,"allocated":25313280,"data_stored":25962241,"data_compressed":1195783,"data_compressed_allocated":1273856,"data_compressed_original":2605056,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":8.1270000000000007,"5min":8.1270000000000007,"15min":8.1270000000000007},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":149.03,"5min":149.03,"15min":149.03},"last":0.20799999999999999},{"interface":"front","average":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":31.866,"5min":31.866,"15min":31.866},"last":0.218}]},{"osd":1,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":13.868,"5min":13.868,"15min":13.868},"last":0.193},{"interface":"front","average":{"1min":14.885,"5min":14.885,"15min":14.885},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":160.37100000000001,"5min":160.37100000000001,"15min":160.37100000000001},"last":393.42399999999998}]},{"osd":3,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.252},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.23300000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":18.771999999999998,"5min":18.771999999999998,"15min":18.771999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":160.584,"5min":160.584,"15min":160.584},"last":0.36299999999999999},{"interface":"front","average":{"1min":2.448,"5min":2.448,"15min":2.448},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":36.841000000000001,"5min":36.841000000000001,"15min":36.841000000000001},"last":0.32200000000000001}]},{"osd":5,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":3.052,"5min":3.052,"15min":3.052},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":48.631,"5min":48.631,"15min":48.631},"last":0.33300000000000002},{"interface":"front","average":{"1min":3.476,"5min":3.476,"15min":3.476},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":56.18,"5min":56.18,"15min":56.18},"last":0.28499999999999998}]},{"osd":6,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.35199999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":7.0759999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.38800000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113144,"kb_used_data":77572,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94258696,"statfs":{"total":96636764160,"available":96520904704,"internally_reserved":0,"allocated":79433728,"data_stored":79819797,"data_compressed":1277396,"data_compressed_allocated":1355776,"data_compressed_original":2736128,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":7.859,"5min":7.859,"15min":7.859},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":88.171000000000006,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.38200000000000001},{"interface":"front","average":{"1min":7.8630000000000004,"5min":7.8630000000000004,"15min":7.8630000000000004},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":88.156000000000006,"5min":88.156000000000006,"15min":88.156000000000006},"last":0.35199999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":8.5999999999999996,"5min":8.5999999999999996,"15min":8.5999999999999996},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":183.04400000000001,"5min":183.04400000000001,"15min":183.04400000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.307}]},{"osd":3,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.439},{"interface":"front","average":{"1min":14.492000000000001,"5min":14.492000000000001,"15min":14.492000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":231.864,"5min":231.864,"15min":231.864},"last":0.26700000000000002}]},{"osd":4,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":12.984,"5min":12.984,"15min":12.984},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":192.42099999999999,"5min":192.42099999999999,"15min":192.42099999999999},"last":0.498},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":14.071,"5min":14.071,"15min":14.071},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":138.089,"5min":138.089,"15min":138.089},"last":0.55500000000000005},{"interface":"front","average":{"1min":20.077000000000002,"5min":20.077000000000002,"15min":20.077000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":183.08199999999999,"5min":183.08199999999999,"15min":183.08199999999999},"last":0.40000000000000002}]},{"osd":6,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":51.070999999999998,"5min":51.070999999999998,"15min":51.070999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":16.532,"5min":16.532,"15min":16.532},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":183.10900000000001,"5min":183.10900000000001,"15min":183.10900000000001},"last":0.46999999999999997}]},{"osd":7,"last update":"Mon Apr 22 18:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.48099999999999998},{"interface":"front","average":{"1min":3.0390000000000001,"5min":3.0390000000000001,"15min":3.0390000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":51.097000000000001,"5min":51.097000000000001,"15min":51.097000000000001},"last":0.51900000000000002}]}]},{"osd":0,"up_from":0,"seq":55834574874,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33095680,"data_stored":34790173,"data_compressed":1758468,"data_compressed_allocated":1871872,"data_compressed_original":3694592,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17854464,"data_stored":19162348,"data_compressed":1180819,"data_compressed_allocated":1257472,"data_compressed_original":2605056,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":17252352,"data_stored":18491116,"data_compressed":1121173,"data_compressed_allocated":1196032,"data_compressed_original":2473984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25542656,"data_stored":27126156,"data_compressed":1624594,"data_compressed_allocated":1728512,"data_compressed_original":3395584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":34144256,"data_stored":36043524,"data_compressed":1815904,"data_compressed_allocated":1929216,"data_compressed_original":3956736,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":223399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":25038848,"data_stored":26632707,"data_compressed":1482695,"data_compressed_allocated":1576960,"data_compressed_original":3264512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1765376,"data_stored":1811968,"data_compressed":137194,"data_compressed_allocated":147456,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7086080,"data_stored":7045793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60231680,"data_stored":60028069,"data_compressed":108870,"data_compressed_allocated":114688,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6791168,"data_stored":6761888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32043008,"data_stored":31904304,"data_compressed":49002,"data_compressed_allocated":49152,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":10027008,"data_stored":9980228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33280000,"data_stored":33118894,"data_compressed":53657,"data_compressed_allocated":57344,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36175872,"data_stored":36103080,"data_compressed":160030,"data_compressed_allocated":167936,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":25088000,"data_stored":25078422,"data_compressed":160030,"data_compressed_allocated":167936,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:40:39.117 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-04-22T18:40:39.117 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-04-22T18:40:39.144 INFO:tasks.rados.rados.0.smithi089.stdout:1542: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:40:39.145 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:39.145+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:39.180 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:39.180 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:39.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:39.233 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:39.234+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:39.279 INFO:tasks.rados.rados.0.smithi089.stdout:1545: expect (ObjNum 303 snap 0 seq_num 303) 2024-04-22T18:40:39.292 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:39.293 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:39.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:39.322 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:39.323+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:39.424 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:39.425+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:39.445 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:39.445 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:39.454 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:39.532 INFO:tasks.rados.rados.0.smithi089.stdout:1539: done (9 left) 2024-04-22T18:40:39.532 INFO:tasks.rados.rados.0.smithi089.stdout:1541: done (8 left) 2024-04-22T18:40:39.532 INFO:tasks.rados.rados.0.smithi089.stdout:1542: done (7 left) 2024-04-22T18:40:39.532 INFO:tasks.rados.rados.0.smithi089.stdout:1545: done (6 left) 2024-04-22T18:40:39.532 INFO:tasks.rados.rados.0.smithi089.stdout:1547: done (5 left) 2024-04-22T18:40:39.532 INFO:tasks.rados.rados.0.smithi089.stdout:1548: done (4 left) 2024-04-22T18:40:39.532 INFO:tasks.rados.rados.0.smithi089.stdout:1549: read oid 42 snap -1 2024-04-22T18:40:39.532 INFO:tasks.rados.rados.0.smithi089.stdout:1549: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:39.532 INFO:tasks.rados.rados.0.smithi089.stdout:1550: read oid 11 snap -1 2024-04-22T18:40:39.532 INFO:tasks.rados.rados.0.smithi089.stdout:1550: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:40:39.533 INFO:tasks.rados.rados.0.smithi089.stdout:1551: read oid 48 snap -1 2024-04-22T18:40:39.533 INFO:tasks.rados.rados.0.smithi089.stdout:1551: expect (ObjNum 305 snap 0 seq_num 305) 2024-04-22T18:40:39.533 INFO:tasks.rados.rados.0.smithi089.stdout:1552: read oid 20 snap -1 2024-04-22T18:40:39.533 INFO:tasks.rados.rados.0.smithi089.stdout:1552: expect (ObjNum 201 snap 0 seq_num 201) 2024-04-22T18:40:39.533 INFO:tasks.rados.rados.0.smithi089.stdout:1553: read oid 26 snap -1 2024-04-22T18:40:39.533 INFO:tasks.rados.rados.0.smithi089.stdout:1553: expect (ObjNum 289 snap 0 seq_num 289) 2024-04-22T18:40:39.533 INFO:tasks.rados.rados.0.smithi089.stdout:1554: set_chunk oid 31 target oid 2024-04-22T18:40:39.533 INFO:tasks.rados.rados.0.smithi089.stdout:1554: set_chunk oid 31 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:40:39.533 INFO:tasks.rados.rados.0.smithi089.stdout:1555: read oid 29 snap -1 2024-04-22T18:40:39.533 INFO:tasks.rados.rados.0.smithi089.stdout:1555: expect (ObjNum 295 snap 0 seq_num 295) 2024-04-22T18:40:39.533 INFO:tasks.rados.rados.0.smithi089.stdout:1556: read oid 2 snap -1 2024-04-22T18:40:39.533 INFO:tasks.rados.rados.0.smithi089.stdout:1556: expect (ObjNum 251 snap 0 seq_num 251) 2024-04-22T18:40:39.533 INFO:tasks.rados.rados.0.smithi089.stdout:1557: write (excl) oid 7 current snap is 0 2024-04-22T18:40:39.533 INFO:tasks.rados.rados.0.smithi089.stdout:1557: seq_num 309 ranges {793363=706061,2282230=796095} 2024-04-22T18:40:39.539 INFO:tasks.rados.rados.0.smithi089.stdout:1557: writing smithi08937929-7 from 793363 to 1499424 tid 1 2024-04-22T18:40:39.539 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:39.540+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:39.547 INFO:tasks.rados.rados.0.smithi089.stdout:1557: writing smithi08937929-7 from 2282230 to 3078325 tid 2 2024-04-22T18:40:39.547 INFO:tasks.rados.rados.0.smithi089.stdout:1558: tier_evict oid 39 2024-04-22T18:40:39.548 INFO:tasks.rados.rados.0.smithi089.stdout:1558: tier_evict oid 39 snap -1 2024-04-22T18:40:39.548 INFO:tasks.rados.rados.0.smithi089.stdout:1543: expect (ObjNum 283 snap 0 seq_num 283) 2024-04-22T18:40:39.596 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:39.596 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:39.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:39.625 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:39.625+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:39.656 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:40:39.713 INFO:tasks.rados.rados.0.smithi089.stdout:1546: expect (ObjNum 267 snap 0 seq_num 267) 2024-04-22T18:40:39.726 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:39.726+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:39.748 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:39.748 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:39.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:39.773 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:40:39.822 INFO:teuthology.orchestra.run.smithi089.stderr:set pool 4 pg_num to 26 2024-04-22T18:40:39.827 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:39.827+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:39.831 INFO:tasks.rados.rados.0.smithi089.stdout:1540: finishing write tid 2 to smithi08937929-33 2024-04-22T18:40:39.831 INFO:tasks.rados.rados.0.smithi089.stdout:1540: finishing write tid 3 to smithi08937929-33 2024-04-22T18:40:39.831 INFO:tasks.rados.rados.0.smithi089.stdout:1540: finishing write tid 4 to smithi08937929-33 2024-04-22T18:40:39.831 INFO:tasks.rados.rados.0.smithi089.stdout:1540: oid 33 updating version 0 to 50 2024-04-22T18:40:39.831 INFO:tasks.rados.rados.0.smithi089.stdout:1540: oid 33 updating version 50 to 51 2024-04-22T18:40:39.831 INFO:tasks.rados.rados.0.smithi089.stdout:1540: oid 33 updating version 51 to 52 2024-04-22T18:40:39.832 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 52 (ObjNum 306 snap 0 seq_num 306) dirty exists 2024-04-22T18:40:39.832 INFO:tasks.rados.rados.0.smithi089.stdout:1540: left oid 33 (ObjNum 306 snap 0 seq_num 306) 2024-04-22T18:40:39.832 INFO:tasks.rados.rados.0.smithi089.stdout:1544: finishing write tid 1 to smithi08937929-18 2024-04-22T18:40:39.832 INFO:tasks.rados.rados.0.smithi089.stdout:1540: done (13 left) 2024-04-22T18:40:39.832 INFO:tasks.rados.rados.0.smithi089.stdout:1543: done (12 left) 2024-04-22T18:40:39.832 INFO:tasks.rados.rados.0.smithi089.stdout:1546: done (11 left) 2024-04-22T18:40:39.832 INFO:tasks.rados.rados.0.smithi089.stdout:1559: read oid 8 snap -1 2024-04-22T18:40:39.832 INFO:tasks.rados.rados.0.smithi089.stdout:1559: expect (ObjNum 253 snap 0 seq_num 253) 2024-04-22T18:40:39.832 INFO:tasks.rados.rados.0.smithi089.stdout:1560: read oid 13 snap -1 2024-04-22T18:40:39.832 INFO:tasks.rados.rados.0.smithi089.stdout:1560: expect deleted 2024-04-22T18:40:39.832 INFO:tasks.rados.rados.0.smithi089.stdout:1561: set_chunk oid 50 target oid 2024-04-22T18:40:39.898 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:39.898 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:39.900 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:39.900 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:39.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:39.915 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:39.915 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:39.919 INFO:tasks.rados.rados.0.smithi089.stdout:1561: set_chunk oid 50 offset: 2541568 length: 283648 target oid a2c4b9118a88f232faad1432aafb3d870906f5a8debf55263b2d559b6b66a5f2 offset: 0 snap -1 2024-04-22T18:40:39.919 INFO:tasks.rados.rados.0.smithi089.stdout:1562: read oid 14 snap -1 2024-04-22T18:40:39.919 INFO:tasks.rados.rados.0.smithi089.stdout:1562: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:40:39.919 INFO:tasks.rados.rados.0.smithi089.stdout:1563: write oid 45 current snap is 0 2024-04-22T18:40:39.919 INFO:tasks.rados.rados.0.smithi089.stdout:1544: finishing write tid 2 to smithi08937929-18 2024-04-22T18:40:39.919 INFO:tasks.rados.rados.0.smithi089.stdout:1544: finishing write tid 3 to smithi08937929-18 2024-04-22T18:40:39.919 INFO:tasks.rados.rados.0.smithi089.stdout:1544: finishing write tid 4 to smithi08937929-18 2024-04-22T18:40:39.920 INFO:tasks.rados.rados.0.smithi089.stdout:1544: finishing write tid 5 to smithi08937929-18 2024-04-22T18:40:39.920 INFO:tasks.rados.rados.0.smithi089.stdout:1544: oid 18 updating version 0 to 80 2024-04-22T18:40:39.920 INFO:tasks.rados.rados.0.smithi089.stdout:1544: oid 18 updating version 80 to 81 2024-04-22T18:40:39.920 INFO:tasks.rados.rados.0.smithi089.stdout:1544: oid 18 version 81 is already newer than 78 2024-04-22T18:40:39.920 INFO:tasks.rados.rados.0.smithi089.stdout:1544: oid 18 version 81 is already newer than 79 2024-04-22T18:40:39.920 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 81 (ObjNum 307 snap 0 seq_num 307) dirty exists 2024-04-22T18:40:39.920 INFO:tasks.rados.rados.0.smithi089.stdout:1544: left oid 18 (ObjNum 307 snap 0 seq_num 307) 2024-04-22T18:40:39.920 INFO:tasks.rados.rados.0.smithi089.stdout:1563: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2024-04-22T18:40:39.926 INFO:tasks.rados.rados.0.smithi089.stdout:1563: writing smithi08937929-45 from 410170 to 1191480 tid 1 2024-04-22T18:40:39.927 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:39.928+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:39.934 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:39.934 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:39.934 INFO:tasks.rados.rados.0.smithi089.stdout:1563: writing smithi08937929-45 from 1940712 to 2596818 tid 2 2024-04-22T18:40:39.937 INFO:tasks.rados.rados.0.smithi089.stdout:1563: writing smithi08937929-45 from 3163363 to 3210170 tid 3 2024-04-22T18:40:39.938 INFO:tasks.rados.rados.0.smithi089.stdout:1544: done (15 left) 2024-04-22T18:40:39.938 INFO:tasks.rados.rados.0.smithi089.stdout:1564: read oid 40 snap -1 2024-04-22T18:40:39.938 INFO:tasks.rados.rados.0.smithi089.stdout:1564: expect (ObjNum 227 snap 0 seq_num 227) 2024-04-22T18:40:39.938 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:39.938 INFO:tasks.rados.rados.0.smithi089.stdout:1552: expect (ObjNum 201 snap 0 seq_num 201) 2024-04-22T18:40:39.952 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:39.953 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:39.969 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:39.969 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:39.985 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:39.985 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:40.011 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:40.011 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:40.020 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:40.028 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:40.029+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:40.124 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:40.124 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:40.124 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:40.124 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:40.129 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:40.130+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:40.134 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:40.134 INFO:tasks.rados.rados.0.smithi089.stdout:1557: finishing write tid 1 to smithi08937929-7 2024-04-22T18:40:40.134 INFO:tasks.rados.rados.0.smithi089.stdout:1554: finishing set_chunk to oid 31 2024-04-22T18:40:40.135 INFO:tasks.rados.rados.0.smithi089.stdout:1554: got expected ENOENT (src dne) 2024-04-22T18:40:40.135 INFO:tasks.rados.rados.0.smithi089.stdout:1550: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:40:40.138 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:40.138 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:40.156 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:40.156 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:40.230 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:40.230+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:40.237 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:40.237 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:40.238 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:40.238 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:40.238 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:40.247 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:40.331 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:40.331+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:40.344 INFO:tasks.rados.rados.0.smithi089.stdout:1549: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:40.392 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:40.392 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:40.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:40.393 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:40.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.393 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:40.393 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:40.393 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 105.93726241100001, 2024-04-22T18:40:40.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:40:40.393 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.393 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.393 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.393 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.393 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.393 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:40:40.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.393 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:40.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:40.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:40:40.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:40.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:40:40.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:40:40.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:40:40.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:40:40.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:40.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:40:40.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:40:40.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:40:40.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:40:40.398 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.398 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.398 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.398 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:40.398 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:40.398 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 97.244205163000004, 2024-04-22T18:40:40.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:40:40.398 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.398 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.398 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.398 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.399 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.399 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:40:40.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.399 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:40.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:40.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:40:40.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:40.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:40:40.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:40:40.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:40:40.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:40:40.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:40.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:40:40.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:40:40.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:40:40.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:40:40.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 97.244186686999996, 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:40.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:40.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:40.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:40.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:40:40.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:40:40.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:40:40.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:40:40.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:40:40.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:40:40.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:40:40.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:40:40.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:40.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:40:40.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:40:40.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:40:40.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:40:40.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.481138375, 2024-04-22T18:40:40.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:40:40.409 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.409 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.409 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.409 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.409 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.409 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:40:40.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.409 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:40.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:40.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:40:40.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:40.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:40:40.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:40:40.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:40:40.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:40:40.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:40.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:40:40.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:40:40.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:40:40.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:40:40.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:40:40.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:40:40.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:40:40.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:40:40.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:40:40.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:40:40.413 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.413 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.414 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.414 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:40.414 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:40.414 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 63.951979049000002, 2024-04-22T18:40:40.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:40:40.414 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.414 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:40.414 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.414 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.414 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.414 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:40:40.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.414 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:40.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:40:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:40:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:40:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:40:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:40:40.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:40:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:40:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:40:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:40:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:40:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:40:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:40:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:40:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:40:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:40:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:40:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.418 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 63.774271654000003, 2024-04-22T18:40:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:40:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:40:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:40:40.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:40:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:40:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:40:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:40:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:40:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:40:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.423 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 63.774203382000003, 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:40:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:40:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:40:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:40:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:40:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:40:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:40:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:40:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:40:40.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:40:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:40:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:40:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:40:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:40:40.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:40.428 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 63.774035541000003, 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:40:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:40:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:40:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:40:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:40:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:40:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:40:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:40:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:40:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:40:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:40:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:40:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.433 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.808710066000003, 2024-04-22T18:40:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:40:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:40.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:40:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:40:40.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:40:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:40:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:40:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:40:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:40:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:40:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:40:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:40:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:40:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:40:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:40:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:40:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:40:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:40:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:40:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:40:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:40:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.708557878999997, 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:40.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:40:40.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:40:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:40:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:40:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:40:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:40:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:40:40.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:40:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:40:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:40:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:40:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:40:40.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 49.999693358999998, 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:40:40.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:40:40.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:40:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:40:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:40:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:40:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:40:40.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:40:40.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:40.448 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.103654986999999, 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:40:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:40:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:40:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:40:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:40:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:40:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3330 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.368576701, 2024-04-22T18:40:40.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43151673600000001, 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3330 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024041+0000", 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024045+0000", 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3950000000000001e-06 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024051+0000", 2024-04-22T18:40:40.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0260000000000004e-06 2024-04-22T18:40:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024056+0000", 2024-04-22T18:40:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4489999999999998e-06 2024-04-22T18:40:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028383+0000", 2024-04-22T18:40:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004326557 2024-04-22T18:40:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028471+0000", 2024-04-22T18:40:40.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8387999999999995e-05 2024-04-22T18:40:40.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.455559+0000", 2024-04-22T18:40:40.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.42708770699999998 2024-04-22T18:40:40.456 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.456 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.456 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.456 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3377 4.6 4:6700eab5:::smithi08937929-32:head [read 1812992~307712] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-04-22T18:40:40.456 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:40:40.456 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.9562513450000001, 2024-04-22T18:40:40.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.52001745799999999, 2024-04-22T18:40:40.456 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.457 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:40.457 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.457 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.457 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.457 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3377 2024-04-22T18:40:40.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.457 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:40:40.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436364+0000", 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436376+0000", 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2121e-05 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436394+0000", 2024-04-22T18:40:40.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8094e-05 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436405+0000", 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0189e-05 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436471+0000", 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6817999999999996e-05 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436481+0000", 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2720000000000003e-06 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956084+0000", 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51960320800000004 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956232+0000", 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014778600000000001 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956385+0000", 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153394 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.460 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.461 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:131 3.1 3:a0a19997:::ee8ca23a7050e771c77dff761e6ea5a11ed57796f4e8a17cbb5804a7fdc937b9:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:40.461 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:40:40.461 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.0633817489999995, 2024-04-22T18:40:40.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51449564999999997, 2024-04-22T18:40:40.461 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.461 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.461 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:40:40.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:40:40.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329235+0000", 2024-04-22T18:40:40.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:40.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329244+0000", 2024-04-22T18:40:40.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6819999999999992e-06 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329259+0000", 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698000000000001e-05 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329265+0000", 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1510000000000001e-06 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329311+0000", 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6096e-05 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.332269+0000", 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029581910000000002 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630603+0000", 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29833418 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630685+0000", 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1217999999999994e-05 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:40.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630813+0000", 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128706 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.631447+0000", 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063427000000000004 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843586+0000", 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21213873 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843623+0000", 2024-04-22T18:40:40.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6848999999999999e-05 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843733+0000", 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010982200000000001 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3441 4.b 4:d3068ee3:::smithi08937929-29:head [write 558065~619869] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.4749056879999998, 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50272020399999995, 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.467 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.467 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.467 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.467 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3441 2024-04-22T18:40:40.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.467 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:40:40.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:40:40.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917711+0000", 2024-04-22T18:40:40.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:40.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917838+0000", 2024-04-22T18:40:40.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012734899999999999 2024-04-22T18:40:40.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917854+0000", 2024-04-22T18:40:40.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6130999999999999e-05 2024-04-22T18:40:40.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917863+0000", 2024-04-22T18:40:40.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5969999999999992e-06 2024-04-22T18:40:40.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992845+0000", 2024-04-22T18:40:40.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.074982193000000003 2024-04-22T18:40:40.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992919+0000", 2024-04-22T18:40:40.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:40:40.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:40.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.993019+0000", 2024-04-22T18:40:40.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9718999999999999e-05 2024-04-22T18:40:40.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095554+0000", 2024-04-22T18:40:40.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102535193 2024-04-22T18:40:40.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420183+0000", 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32462853600000002 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420401+0000", 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021881400000000001 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420433+0000", 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079000000000002e-05 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.471 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.472 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.472 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3442 4.b 4:d3068ee3:::smithi08937929-29:head [write 1579835~486765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:40.472 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:40:40.472 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.470218977, 2024-04-22T18:40:40.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49822799099999998, 2024-04-22T18:40:40.472 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.472 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.472 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.472 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.472 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.472 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3442 2024-04-22T18:40:40.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.473 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:40:40.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:40:40.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922399+0000", 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922502+0000", 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102784 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922513+0000", 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1051e-05 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922518+0000", 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8529999999999997e-06 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093158+0000", 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.17063958800000001 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093230+0000", 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1803000000000004e-05 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093326+0000", 2024-04-22T18:40:40.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6454999999999997e-05 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095557+0000", 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022306399999999999 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420484+0000", 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32492717700000001 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420606+0000", 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012225800000000001 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420628+0000", 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1596000000000001e-05 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3443 4.b 4:d3068ee3:::smithi08937929-29:head [write 2468371~489694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.4656169299999995, 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49372487199999998, 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.477 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.478 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.478 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3443 2024-04-22T18:40:40.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.478 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:40:40.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:40:40.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927001+0000", 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927117+0000", 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011637 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927128+0000", 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0696000000000001e-05 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927134+0000", 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8640000000000001e-06 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094288+0000", 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16715412900000001 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094359+0000", 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1639999999999998e-05 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:40.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094451+0000", 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1252e-05 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095560+0000", 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011095530000000001 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420661+0000", 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32510105500000003 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420710+0000", 2024-04-22T18:40:40.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9018999999999999e-05 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420727+0000", 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6722e-05 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3444 4.b 4:d3068ee3:::smithi08937929-29:head [setxattr _header (56) in=7b,truncate 2958065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.4654812130000003, 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49366935299999998, 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.482 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.483 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3444 2024-04-22T18:40:40.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.483 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:40:40.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:40:40.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927137+0000", 2024-04-22T18:40:40.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:40.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927141+0000", 2024-04-22T18:40:40.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3270000000000001e-06 2024-04-22T18:40:40.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927144+0000", 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9730000000000002e-06 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927147+0000", 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.959e-06 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095118+0000", 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16797115500000001 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095169+0000", 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1316000000000001e-05 2024-04-22T18:40:40.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:40.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095260+0000", 2024-04-22T18:40:40.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1184e-05 2024-04-22T18:40:40.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095999+0000", 2024-04-22T18:40:40.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073868200000000005 2024-04-22T18:40:40.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420757+0000", 2024-04-22T18:40:40.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32475783499999999 2024-04-22T18:40:40.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420781+0000", 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4022000000000001e-05 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420807+0000", 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6366000000000001e-05 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3445 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:40.487 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.4654701190000008, 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.493780049, 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3445 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:40.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:40.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:40.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927150+0000", 2024-04-22T18:40:40.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.542e-06 2024-04-22T18:40:40.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927153+0000", 2024-04-22T18:40:40.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0520000000000002e-06 2024-04-22T18:40:40.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027218+0000", 2024-04-22T18:40:40.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100064604 2024-04-22T18:40:40.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095501+0000", 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.068283431000000006 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095545+0000", 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4295999999999997e-05 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420833+0000", 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32528718400000001 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420877+0000", 2024-04-22T18:40:40.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4403999999999997e-05 2024-04-22T18:40:40.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420929+0000", 2024-04-22T18:40:40.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1848999999999998e-05 2024-04-22T18:40:40.492 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.492 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.492 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.492 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.492 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.492 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:40.494 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:40.494 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:40.494+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:40.569 INFO:tasks.rados.rados.0.smithi089.stdout:1553: expect (ObjNum 289 snap 0 seq_num 289) 2024-04-22T18:40:40.594 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:40.595+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:40.636 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:40.637 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:40.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.637 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGRecoveryDelete(3.3 e108,103 3:f3044891:::cd6b3221845642dabc017338ad15f593b823ed4d8d570a00a6ceba3eb3bddd61:head,79'244)", 2024-04-22T18:40:40.637 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:40.538826+0000", 2024-04-22T18:40:40.637 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.099653290000000005, 2024-04-22T18:40:40.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099674756000000003, 2024-04-22T18:40:40.637 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.637 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:40.637 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:40.538826+0000", 2024-04-22T18:40:40.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:40.538826+0000", 2024-04-22T18:40:40.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:40.538824+0000", 2024-04-22T18:40:40.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:40.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:40.538833+0000", 2024-04-22T18:40:40.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.551e-06 2024-04-22T18:40:40.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:40.538849+0000", 2024-04-22T18:40:40.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5659e-05 2024-04-22T18:40:40.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:40.538858+0000", 2024-04-22T18:40:40.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6190000000000006e-06 2024-04-22T18:40:40.639 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.639 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.639 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.640 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGRecoveryDelete(3.3 e108,103 3:dd688cb4:::9fabdac50edc2413c988979759e360e365446b2b39a24e037e3fb8023fbd09f8:head,79'245)", 2024-04-22T18:40:40.640 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:40.539069+0000", 2024-04-22T18:40:40.640 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.099410154000000001, 2024-04-22T18:40:40.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099461899000000006, 2024-04-22T18:40:40.640 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.640 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:40.640 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:40.539069+0000", 2024-04-22T18:40:40.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:40.539069+0000", 2024-04-22T18:40:40.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:40.539068+0000", 2024-04-22T18:40:40.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:40.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:40.539074+0000", 2024-04-22T18:40:40.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2360000000000004e-06 2024-04-22T18:40:40.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:40.539084+0000", 2024-04-22T18:40:40.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0266e-05 2024-04-22T18:40:40.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:40.539092+0000", 2024-04-22T18:40:40.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2649999999999996e-06 2024-04-22T18:40:40.642 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.642 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.642 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.642 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.643 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:40.643 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 2 2024-04-22T18:40:40.643 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:40.646 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:40.729 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:40.730+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:40.742 INFO:tasks.rados.rados.0.smithi089.stdout:1555: expect (ObjNum 295 snap 0 seq_num 295) 2024-04-22T18:40:40.788 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:40.788 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:40.788 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:40.788 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:40.789 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:40.796 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:40.796+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:40.797 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:40.823 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:40:40.897 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:40.897+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:40.908 INFO:tasks.rados.rados.0.smithi089.stdout:1557: finishing write tid 2 to smithi08937929-7 2024-04-22T18:40:40.908 INFO:tasks.rados.rados.0.smithi089.stdout:1551: expect (ObjNum 305 snap 0 seq_num 305) 2024-04-22T18:40:40.928 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:40.928 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:40.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:40.928 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:40.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.928 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:9 4.f 4:f2c48a0e:::smithi08937929-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:40.928 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:40.928 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 106.81126143500001, 2024-04-22T18:40:40.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49290714600000002, 2024-04-22T18:40:40.928 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.929 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.929 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.929 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.929 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.929 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 9 2024-04-22T18:40:40.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.929 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:40.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:40.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116773+0000", 2024-04-22T18:40:40.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:40.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117024+0000", 2024-04-22T18:40:40.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025191299999999999 2024-04-22T18:40:40.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117036+0000", 2024-04-22T18:40:40.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1595e-05 2024-04-22T18:40:40.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117046+0000", 2024-04-22T18:40:40.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0456e-05 2024-04-22T18:40:40.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414324+0000", 2024-04-22T18:40:40.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297277603 2024-04-22T18:40:40.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414404+0000", 2024-04-22T18:40:40.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9738000000000001e-05 2024-04-22T18:40:40.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:40.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414494+0000", 2024-04-22T18:40:40.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9733000000000003e-05 2024-04-22T18:40:40.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518194+0000", 2024-04-22T18:40:40.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10370061 2024-04-22T18:40:40.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609618+0000", 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091423360999999995 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609655+0000", 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7580000000000003e-05 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609681+0000", 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5908e-05 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.933 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.934 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.934 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:10 4.f 4:f2c48a0e:::smithi08937929-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:40.934 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:40.934 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 106.80540917499999, 2024-04-22T18:40:40.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48712681400000002, 2024-04-22T18:40:40.934 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.934 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.934 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.934 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.934 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.934 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 10 2024-04-22T18:40:40.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122625+0000", 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122828+0000", 2024-04-22T18:40:40.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020359499999999999 2024-04-22T18:40:40.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122840+0000", 2024-04-22T18:40:40.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1239e-05 2024-04-22T18:40:40.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122845+0000", 2024-04-22T18:40:40.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2719999999999997e-06 2024-04-22T18:40:40.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415105+0000", 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29225972099999997 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415169+0000", 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4170000000000004e-05 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415257+0000", 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8331999999999994e-05 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518199+0000", 2024-04-22T18:40:40.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10294174 2024-04-22T18:40:40.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609707+0000", 2024-04-22T18:40:40.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091508408999999999 2024-04-22T18:40:40.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609729+0000", 2024-04-22T18:40:40.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1917e-05 2024-04-22T18:40:40.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609753+0000", 2024-04-22T18:40:40.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3776000000000001e-05 2024-04-22T18:40:40.939 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.939 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.939 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.939 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:11 4.f 4:f2c48a0e:::smithi08937929-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:40.939 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:40.939 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 106.805185802, 2024-04-22T18:40:40.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49452214500000002, 2024-04-22T18:40:40.939 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.939 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.939 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.940 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.940 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.940 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 11 2024-04-22T18:40:40.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.940 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:40.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:40.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122849+0000", 2024-04-22T18:40:40.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:40.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122853+0000", 2024-04-22T18:40:40.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9029999999999997e-06 2024-04-22T18:40:40.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122856+0000", 2024-04-22T18:40:40.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.01e-06 2024-04-22T18:40:40.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122859+0000", 2024-04-22T18:40:40.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:40.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415755+0000", 2024-04-22T18:40:40.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29289606800000001 2024-04-22T18:40:40.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415808+0000", 2024-04-22T18:40:40.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3415999999999998e-05 2024-04-22T18:40:40.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:40.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415888+0000", 2024-04-22T18:40:40.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9061999999999996e-05 2024-04-22T18:40:40.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518203+0000", 2024-04-22T18:40:40.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102315533 2024-04-22T18:40:40.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617164+0000", 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098960548999999995 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617241+0000", 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7292000000000005e-05 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617372+0000", 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013075399999999999 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.944 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.945 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.945 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:12 4.f 4:f2c48a0e:::smithi08937929-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:40.945 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:40.945 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 106.805173938, 2024-04-22T18:40:40.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49473910999999998, 2024-04-22T18:40:40.945 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.945 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:40.945 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.945 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.945 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.945 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 12 2024-04-22T18:40:40.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.946 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:40.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:40.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122863+0000", 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.863e-06 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122866+0000", 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122869+0000", 2024-04-22T18:40:40.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5950000000000001e-06 2024-04-22T18:40:40.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416008+0000", 2024-04-22T18:40:40.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29313975199999998 2024-04-22T18:40:40.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:40.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416048+0000", 2024-04-22T18:40:40.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9888000000000002e-05 2024-04-22T18:40:40.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617447+0000", 2024-04-22T18:40:40.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20139881400000001 2024-04-22T18:40:40.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617537+0000", 2024-04-22T18:40:40.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9606999999999998e-05 2024-04-22T18:40:40.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617600+0000", 2024-04-22T18:40:40.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3884e-05 2024-04-22T18:40:40.949 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.949 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.949 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.950 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:33 4.f 4:f0ed3d49:::smithi08937929-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:40.950 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:40.950 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 106.741233737, 2024-04-22T18:40:40.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62359692099999997, 2024-04-22T18:40:40.950 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.950 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.950 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.950 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.950 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.950 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 33 2024-04-22T18:40:40.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.950 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:40.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:40.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186799+0000", 2024-04-22T18:40:40.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187103+0000", 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030354900000000002 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187117+0000", 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4578e-05 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187129+0000", 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1421e-05 2024-04-22T18:40:40.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516134+0000", 2024-04-22T18:40:40.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32900553900000001 2024-04-22T18:40:40.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516215+0000", 2024-04-22T18:40:40.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0537999999999994e-05 2024-04-22T18:40:40.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:40.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516302+0000", 2024-04-22T18:40:40.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6601000000000007e-05 2024-04-22T18:40:40.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518207+0000", 2024-04-22T18:40:40.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019058370000000001 2024-04-22T18:40:40.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810302+0000", 2024-04-22T18:40:40.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29209509700000003 2024-04-22T18:40:40.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810348+0000", 2024-04-22T18:40:40.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5992999999999998e-05 2024-04-22T18:40:40.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810398+0000", 2024-04-22T18:40:40.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0046000000000002e-05 2024-04-22T18:40:40.955 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.955 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.955 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.955 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:34 4.f 4:f0ed3d49:::smithi08937929-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:40.955 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:40.955 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 106.736631725, 2024-04-22T18:40:40.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.61954655800000002, 2024-04-22T18:40:40.955 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191401+0000", 2024-04-22T18:40:40.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:40.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191503+0000", 2024-04-22T18:40:40.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010190400000000001 2024-04-22T18:40:40.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191517+0000", 2024-04-22T18:40:40.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3267e-05 2024-04-22T18:40:40.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191523+0000", 2024-04-22T18:40:40.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-04-22T18:40:40.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517062+0000", 2024-04-22T18:40:40.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32553894300000003 2024-04-22T18:40:40.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517141+0000", 2024-04-22T18:40:40.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9667999999999997e-05 2024-04-22T18:40:40.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:40.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517267+0000", 2024-04-22T18:40:40.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000125494 2024-04-22T18:40:40.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518213+0000", 2024-04-22T18:40:40.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000946275 2024-04-22T18:40:40.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810901+0000", 2024-04-22T18:40:40.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29268821699999997 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810929+0000", 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7645999999999999e-05 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810950+0000", 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1333999999999999e-05 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.960 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 106.736508757, 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:40.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:40:40.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:40.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:40:40.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:40:40.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:40:40.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:40:40.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:40.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:40:40.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:40:40.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:40:40.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:40:40.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.965 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 106.73649768200001, 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:40.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:40.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:40:40.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:40.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:40:40.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:40:40.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:40:40.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:40:40.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:40:40.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:40:40.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:40:40.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:40:40.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:40.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:40:40.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:40:40.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:40:40.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:40:40.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:40:40.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:40:40.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:40:40.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:40:40.970 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.970 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.970 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.971 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:40.971 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:40.971 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 103.135128188, 2024-04-22T18:40:40.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:40:40.971 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.971 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.971 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.971 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.971 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.971 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:40:40.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.971 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:40:40.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:40:40.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:40:40.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:40:40.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:40:40.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:40:40.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:40:40.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:40:40.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:40:40.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:40:40.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:40.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:40:40.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:40:40.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:40:40.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:40:40.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:40:40.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:40:40.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:40:40.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:40:40.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:40:40.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:40:40.976 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.976 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.976 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.976 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:7 3.0 3:1554f608:::d614c1f2f74bb29087660bd6e888eafe46b1a7ed3cbdede85b36de771c7f0d66:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:40.976 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:40.976 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.755850662, 2024-04-22T18:40:40.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48116529899999999, 2024-04-22T18:40:40.976 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.976 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.976 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:40.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:40.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172184+0000", 2024-04-22T18:40:40.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:40.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172349+0000", 2024-04-22T18:40:40.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000165074 2024-04-22T18:40:40.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172355+0000", 2024-04-22T18:40:40.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8900000000000004e-06 2024-04-22T18:40:40.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172359+0000", 2024-04-22T18:40:40.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.569e-06 2024-04-22T18:40:40.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271561+0000", 2024-04-22T18:40:40.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099202569000000004 2024-04-22T18:40:40.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:40.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271659+0000", 2024-04-22T18:40:40.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8171000000000006e-05 2024-04-22T18:40:40.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:40.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271775+0000", 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000116137 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.272327+0000", 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00055106999999999997 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653290+0000", 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.38096334500000001 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653326+0000", 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6168000000000003e-05 2024-04-22T18:40:40.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653350+0000", 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3921999999999999e-05 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3193 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.885289393000001, 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69912102700000001, 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:40.981 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.982 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.982 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.982 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3193 2024-04-22T18:40:40.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.982 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:40:40.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:40:40.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042744+0000", 2024-04-22T18:40:40.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:40.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043013+0000", 2024-04-22T18:40:40.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026906 2024-04-22T18:40:40.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043028+0000", 2024-04-22T18:40:40.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5359999999999999e-05 2024-04-22T18:40:40.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043036+0000", 2024-04-22T18:40:40.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8420000000000005e-06 2024-04-22T18:40:40.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043081+0000", 2024-04-22T18:40:40.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5019999999999999e-05 2024-04-22T18:40:40.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:40.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043087+0000", 2024-04-22T18:40:40.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5420000000000002e-06 2024-04-22T18:40:40.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741856+0000", 2024-04-22T18:40:40.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69876864500000002 2024-04-22T18:40:40.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741867+0000", 2024-04-22T18:40:40.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.084e-05 2024-04-22T18:40:40.985 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.985 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.985 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.986 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3194 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:40.986 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:40:40.986 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.878039413, 2024-04-22T18:40:40.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69190137100000004, 2024-04-22T18:40:40.986 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.986 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:40.986 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.986 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.986 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.986 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3194 2024-04-22T18:40:40.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.986 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049991+0000", 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999957 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050284+0000", 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029296699999999999 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050304+0000", 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.969e-05 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050314+0000", 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0268999999999999e-05 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150531+0000", 2024-04-22T18:40:40.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100216868 2024-04-22T18:40:40.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:40.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150545+0000", 2024-04-22T18:40:40.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4063000000000001e-05 2024-04-22T18:40:40.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741890+0000", 2024-04-22T18:40:40.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59134479100000004 2024-04-22T18:40:40.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741897+0000", 2024-04-22T18:40:40.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1189999999999999e-06 2024-04-22T18:40:40.990 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.990 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.990 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.990 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3195 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:40.990 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:40.990 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.877706972999999, 2024-04-22T18:40:40.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69159474200000004, 2024-04-22T18:40:40.990 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.990 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:40.990 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.990 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.990 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3195 2024-04-22T18:40:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050333+0000", 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0289999999999996e-06 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050341+0000", 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0050000000000004e-06 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050346+0000", 2024-04-22T18:40:40.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8329999999999999e-06 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150628+0000", 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100281548 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150672+0000", 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3875999999999998e-05 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741916+0000", 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59124448799999996 2024-04-22T18:40:40.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741923+0000", 2024-04-22T18:40:40.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7689999999999999e-06 2024-04-22T18:40:40.994 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.994 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.994 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.994 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3196 4.11 4.9f8e4db1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:40.994 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:40.994 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.877676854000001, 2024-04-22T18:40:40.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69158947400000004, 2024-04-22T18:40:40.994 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3196 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:40.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050361+0000", 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2040000000000002e-06 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050367+0000", 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3119999999999999e-06 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:40.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050372+0000", 2024-04-22T18:40:40.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0869999999999997e-06 2024-04-22T18:40:40.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150688+0000", 2024-04-22T18:40:40.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100315381 2024-04-22T18:40:40.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:40.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150691+0000", 2024-04-22T18:40:40.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8899999999999999e-06 2024-04-22T18:40:40.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:40.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741941+0000", 2024-04-22T18:40:40.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59125044900000001 2024-04-22T18:40:40.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:40.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741948+0000", 2024-04-22T18:40:40.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.827e-06 2024-04-22T18:40:40.998 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.998 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:40.998 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:40.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.999 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3316 4.11 4:8f787b23:::smithi08937929-41:head [write 1969239~724762] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:40.999 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.539840+0000", 2024-04-22T18:40:40.999 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.388195331, 2024-04-22T18:40:40.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50791082700000001, 2024-04-22T18:40:40.999 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:40.999 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:40.999 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:40.999 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:40.999 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:40.999 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3316 2024-04-22T18:40:40.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:40.999 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:40.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:40.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:40.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539840+0000", 2024-04-22T18:40:41.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539840+0000", 2024-04-22T18:40:41.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539839+0000", 2024-04-22T18:40:41.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540217+0000", 2024-04-22T18:40:41.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037753899999999998 2024-04-22T18:40:41.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540225+0000", 2024-04-22T18:40:41.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4630000000000001e-06 2024-04-22T18:40:41.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540231+0000", 2024-04-22T18:40:41.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0440000000000001e-06 2024-04-22T18:40:41.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540273+0000", 2024-04-22T18:40:41.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1480000000000003e-05 2024-04-22T18:40:41.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:41.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540282+0000", 2024-04-22T18:40:41.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.935e-06 2024-04-22T18:40:41.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.844412+0000", 2024-04-22T18:40:41.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30413055500000002 2024-04-22T18:40:41.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.844463+0000", 2024-04-22T18:40:41.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0918000000000003e-05 2024-04-22T18:40:41.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:41.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.844537+0000", 2024-04-22T18:40:41.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3983e-05 2024-04-22T18:40:41.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:41.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.948265+0000", 2024-04-22T18:40:41.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103727471 2024-04-22T18:40:41.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:41.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.047658+0000", 2024-04-22T18:40:41.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099393426000000007 2024-04-22T18:40:41.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.047707+0000", 2024-04-22T18:40:41.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8733000000000002e-05 2024-04-22T18:40:41.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.047751+0000", 2024-04-22T18:40:41.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3847000000000002e-05 2024-04-22T18:40:41.004 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.005 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.005 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.005 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3317 4.11 4:8f787b23:::smithi08937929-41:head [setxattr _header (56) in=7b,truncate 2773188] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:41.005 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.540240+0000", 2024-04-22T18:40:41.005 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.387795473000001, 2024-04-22T18:40:41.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62115056400000002, 2024-04-22T18:40:41.005 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.005 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.005 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:41.005 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:41.005 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:41.005 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3317 2024-04-22T18:40:41.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.006 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540240+0000", 2024-04-22T18:40:41.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540240+0000", 2024-04-22T18:40:41.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540239+0000", 2024-04-22T18:40:41.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540243+0000", 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4569999999999998e-06 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540250+0000", 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6440000000000003e-06 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540256+0000", 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8100000000000003e-06 2024-04-22T18:40:41.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540302+0000", 2024-04-22T18:40:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5859999999999998e-05 2024-04-22T18:40:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540309+0000", 2024-04-22T18:40:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3950000000000003e-06 2024-04-22T18:40:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.947528+0000", 2024-04-22T18:40:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40721847999999999 2024-04-22T18:40:41.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.947588+0000", 2024-04-22T18:40:41.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0364e-05 2024-04-22T18:40:41.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:41.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.947687+0000", 2024-04-22T18:40:41.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8733999999999994e-05 2024-04-22T18:40:41.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:41.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.948414+0000", 2024-04-22T18:40:41.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072766900000000002 2024-04-22T18:40:41.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:41.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161295+0000", 2024-04-22T18:40:41.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.212880818 2024-04-22T18:40:41.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161342+0000", 2024-04-22T18:40:41.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6894000000000002e-05 2024-04-22T18:40:41.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161390+0000", 2024-04-22T18:40:41.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8245999999999999e-05 2024-04-22T18:40:41.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.011 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.011 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3318 4.11 4:8f787b23:::smithi08937929-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:41.011 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:41.011 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.387765281, 2024-04-22T18:40:41.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62124934899999995, 2024-04-22T18:40:41.011 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.011 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3318 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540276+0000", 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.049e-06 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540282+0000", 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7050000000000002e-06 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540288+0000", 2024-04-22T18:40:41.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2900000000000002e-06 2024-04-22T18:40:41.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540326+0000", 2024-04-22T18:40:41.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8818e-05 2024-04-22T18:40:41.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:41.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540330+0000", 2024-04-22T18:40:41.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6260000000000001e-06 2024-04-22T18:40:41.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.948024+0000", 2024-04-22T18:40:41.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40769397600000001 2024-04-22T18:40:41.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:41.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.948063+0000", 2024-04-22T18:40:41.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9127e-05 2024-04-22T18:40:41.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161420+0000", 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.213356876 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161463+0000", 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3219000000000001e-05 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161519+0000", 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6155e-05 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.017 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3337 4.11 4:8db271f9:::smithi08937929-19:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e96)", 2024-04-22T18:40:41.017 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:27.149915+0000", 2024-04-22T18:40:41.017 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.778120271000001, 2024-04-22T18:40:41.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.42102647599999998, 2024-04-22T18:40:41.017 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.017 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:41.017 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:41.017 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:41.017 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:41.017 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3337 2024-04-22T18:40:41.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.017 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149915+0000", 2024-04-22T18:40:41.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149915+0000", 2024-04-22T18:40:41.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149915+0000", 2024-04-22T18:40:41.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149918+0000", 2024-04-22T18:40:41.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1030000000000002e-06 2024-04-22T18:40:41.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149921+0000", 2024-04-22T18:40:41.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0309999999999999e-06 2024-04-22T18:40:41.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149924+0000", 2024-04-22T18:40:41.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9510000000000002e-06 2024-04-22T18:40:41.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149991+0000", 2024-04-22T18:40:41.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6773999999999995e-05 2024-04-22T18:40:41.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.150065+0000", 2024-04-22T18:40:41.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:40:41.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.570942+0000", 2024-04-22T18:40:41.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.420876947 2024-04-22T18:40:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3404 4.9 4:9ec4daef:::smithi08937929-27:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.111428+0000", 2024-04-22T18:40:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.8166077010000006, 2024-04-22T18:40:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41963582999999999, 2024-04-22T18:40:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:41.021 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3404 2024-04-22T18:40:41.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.022 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.111428+0000", 2024-04-22T18:40:41.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.111428+0000", 2024-04-22T18:40:41.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.111427+0000", 2024-04-22T18:40:41.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.111430+0000", 2024-04-22T18:40:41.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9450000000000002e-06 2024-04-22T18:40:41.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.111435+0000", 2024-04-22T18:40:41.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2449999999999998e-06 2024-04-22T18:40:41.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.111440+0000", 2024-04-22T18:40:41.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0259999999999998e-06 2024-04-22T18:40:41.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.111509+0000", 2024-04-22T18:40:41.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8773e-05 2024-04-22T18:40:41.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.111624+0000", 2024-04-22T18:40:41.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115198 2024-04-22T18:40:41.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.531063+0000", 2024-04-22T18:40:41.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41943929800000002 2024-04-22T18:40:41.025 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.025 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.025 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.025 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3504 4.9 4:96737e19:::smithi08937929-4:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:41.025 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:40:41.025 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.5171736779999998, 2024-04-22T18:40:41.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.951755291, 2024-04-22T18:40:41.025 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.025 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:41.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:41.026 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:41.026 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:41.026 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3504 2024-04-22T18:40:41.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.026 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:40:41.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:40:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410861+0000", 2024-04-22T18:40:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410863+0000", 2024-04-22T18:40:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.635e-06 2024-04-22T18:40:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410866+0000", 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8499999999999998e-06 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410868+0000", 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.497e-06 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615137+0000", 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20426903499999999 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615218+0000", 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0860999999999997e-05 2024-04-22T18:40:41.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.362617+0000", 2024-04-22T18:40:41.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.74739882700000004 2024-04-22T18:40:41.029 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.029 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.029 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.029 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.029 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.029 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:41.031 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:41.031 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:41.031+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:41.043 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:41.043+0000 7f8605f32640 -1 osd.3 108 handle_osd_map injecting map crc failure 2024-04-22T18:40:41.120 INFO:tasks.rados.rados.0.smithi089.stdout:1557: finishing write tid 3 to smithi08937929-7 2024-04-22T18:40:41.120 INFO:tasks.rados.rados.0.smithi089.stdout:1557: finishing write tid 4 to smithi08937929-7 2024-04-22T18:40:41.120 INFO:tasks.rados.rados.0.smithi089.stdout:1557: oid 7 updating version 0 to 48 2024-04-22T18:40:41.120 INFO:tasks.rados.rados.0.smithi089.stdout:1557: oid 7 updating version 48 to 49 2024-04-22T18:40:41.121 INFO:tasks.rados.rados.0.smithi089.stdout:1557: oid 7 updating version 49 to 50 2024-04-22T18:40:41.121 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 50 (ObjNum 309 snap 0 seq_num 309) dirty exists 2024-04-22T18:40:41.121 INFO:tasks.rados.rados.0.smithi089.stdout:1557: left oid 7 (ObjNum 309 snap 0 seq_num 309) 2024-04-22T18:40:41.121 INFO:tasks.rados.rados.0.smithi089.stdout:1556: expect (ObjNum 251 snap 0 seq_num 251) 2024-04-22T18:40:41.131 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:41.131+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:41.145 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:41.146 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:41.146 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:41.146 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:41.155 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:41.232 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:41.233+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:41.298 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:41.298 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:41.298 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:41.299 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:41.299 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:41.307 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:41.323 INFO:tasks.rados.rados.0.smithi089.stdout:1558: got (22) Invalid argument 2024-04-22T18:40:41.323 INFO:tasks.rados.rados.0.smithi089.stdout:1559: expect (ObjNum 253 snap 0 seq_num 253) 2024-04-22T18:40:41.333 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:41.333+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:41.412 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:41.412 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:41.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:41.412 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:41.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.412 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:73 3.0 e34/19 3:061a7a63:::392f44ec26aa22e85be5f08273a7e45656f3bf4a4854038391a88e880ec01077:head v 34'31, mlcod=31'29)", 2024-04-22T18:40:41.412 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:41.413 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.954069740999998, 2024-04-22T18:40:41.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103503311, 2024-04-22T18:40:41.413 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.413 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.413 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:41.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:41.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458193+0000", 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026628900000000001 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458208+0000", 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4780000000000001e-05 2024-04-22T18:40:41.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458217+0000", 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4909999999999994e-06 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459162+0000", 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094440500000000005 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459185+0000", 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3685000000000001e-05 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561394+0000", 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10220874100000001 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561430+0000", 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6304999999999999e-05 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:75 3.0 e34/19 3:18418e0f:::b7c35f165c374d5e075dde9ac2607e47ace3b0a1e2343ec6efb171380f725eea:head v 34'32, mlcod=31'29)", 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.953056468, 2024-04-22T18:40:41.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104798078, 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458939+0000", 2024-04-22T18:40:41.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:41.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459092+0000", 2024-04-22T18:40:41.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153838 2024-04-22T18:40:41.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459107+0000", 2024-04-22T18:40:41.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4703e-05 2024-04-22T18:40:41.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459115+0000", 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560093+0000", 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10097840499999999 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560109+0000", 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5631e-05 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563706+0000", 2024-04-22T18:40:41.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035972059999999999 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563738+0000", 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2184000000000002e-05 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:77 3.0 e34/19 3:28387c9f:::b6d7840f02d561da04e398fbf94d91b85e9066b426e48baf3b11dd4a4f6096c0:head v 34'33, mlcod=31'29)", 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.952106388000004, 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104186079, 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.420 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459889+0000", 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459963+0000", 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3732999999999994e-05 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459973+0000", 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0285999999999999e-05 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459987+0000", 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3675999999999999e-05 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560515+0000", 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10052781299999999 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560538+0000", 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3861000000000001e-05 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564044+0000", 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035052009999999999 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564076+0000", 2024-04-22T18:40:41.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2737000000000002e-05 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:78 3.0 e34/19 3:3a5582ac:::bdd92f7deaa2d50e204891f64f9043469ba687d78b218ed518f2483ba3486a99:head v 34'34, mlcod=31'29)", 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.951440270000006, 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103547263, 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:41.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460555+0000", 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460861+0000", 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000305958 2024-04-22T18:40:41.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460874+0000", 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2639e-05 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460881+0000", 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2570000000000001e-06 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560990+0000", 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010883199999999 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561011+0000", 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1319e-05 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564079+0000", 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00306815 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564104+0000", 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4172000000000002e-05 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.427 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.950824976000007, 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:40:41.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:40:41.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:40:41.431 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 93.074151154000006, 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:40:41.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.869347630999997, 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.435 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:41.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:41.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:40:41.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:40:41.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:40:41.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:40:41.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:40:41.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:40:41.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:40:41.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:161 3.0 e76/63 3:3638311b:::f282598a146af3da1c05c8882972fba387afd249b39f6eeb6e6052a5de37fd2d:head v 76'157, mlcod=76'154)", 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 34.020628201000001, 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104845172, 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391440+0000", 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2045999999999994e-05 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391448+0000", 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8189999999999992e-06 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391453+0000", 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8379999999999998e-06 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391839+0000", 2024-04-22T18:40:41.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000385325 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391854+0000", 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5006e-05 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496191+0000", 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104336792 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496214+0000", 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3079999999999999e-05 2024-04-22T18:40:41.442 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.443 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.443 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.443 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:165 3.0 e76/63 3:1e2f7d15:::93d9bab0d1f539405a95a28c02bcf2dcbdfa6f3e449e5e5e3f27f774df3c508a:head v 76'158, mlcod=76'154)", 2024-04-22T18:40:41.443 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:41.443 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 34.020534017999999, 2024-04-22T18:40:41.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104788752, 2024-04-22T18:40:41.443 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.443 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.443 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391462+0000", 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391522+0000", 2024-04-22T18:40:41.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0148000000000001e-05 2024-04-22T18:40:41.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391531+0000", 2024-04-22T18:40:41.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5709999999999998e-06 2024-04-22T18:40:41.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391536+0000", 2024-04-22T18:40:41.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6180000000000003e-06 2024-04-22T18:40:41.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392218+0000", 2024-04-22T18:40:41.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000681326 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392237+0000", 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9726999999999999e-05 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496231+0000", 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103993672 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496251+0000", 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0327999999999999e-05 2024-04-22T18:40:41.446 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:166 3.0 e76/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 76'159, mlcod=76'154)", 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 34.020456119999999, 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104724628, 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391540+0000", 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391627+0000", 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7330000000000006e-05 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391635+0000", 2024-04-22T18:40:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6709999999999999e-06 2024-04-22T18:40:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391640+0000", 2024-04-22T18:40:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3199999999999999e-06 2024-04-22T18:40:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492541+0000", 2024-04-22T18:40:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100900873 2024-04-22T18:40:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492560+0000", 2024-04-22T18:40:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8502999999999999e-05 2024-04-22T18:40:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496254+0000", 2024-04-22T18:40:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036945849999999998 2024-04-22T18:40:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496265+0000", 2024-04-22T18:40:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0869999999999999e-05 2024-04-22T18:40:41.450 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:167 3.0 e76/63 3:384c1ce2:::81dc8c58ebe106ae6fa9feadc06291c645d9079ad84b78214d4e6440fbc5def8:head v 76'160, mlcod=76'154)", 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 34.020341244999997, 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10487859400000001, 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391707+0000", 2024-04-22T18:40:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2343999999999999e-05 2024-04-22T18:40:41.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391714+0000", 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.596e-06 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391719+0000", 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9130000000000002e-06 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493154+0000", 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101435256 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493177+0000", 2024-04-22T18:40:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2796e-05 2024-04-22T18:40:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496509+0000", 2024-04-22T18:40:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033320139999999999 2024-04-22T18:40:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496534+0000", 2024-04-22T18:40:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5265999999999998e-05 2024-04-22T18:40:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.454 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:197 3.0 e78/63 3:19ca42e0:::2109e7e4526cbe26b2963cdd11634231ee7f4d35dabe9962c44b62fafb130ddf:head v 78'167, mlcod=78'165)", 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.540869+0000", 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.871127593000001, 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10287555800000001, 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540869+0000", 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540869+0000", 2024-04-22T18:40:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540867+0000", 2024-04-22T18:40:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540879+0000", 2024-04-22T18:40:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1316e-05 2024-04-22T18:40:41.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540895+0000", 2024-04-22T18:40:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6072000000000001e-05 2024-04-22T18:40:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540904+0000", 2024-04-22T18:40:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8969999999999997e-06 2024-04-22T18:40:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.641023+0000", 2024-04-22T18:40:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10011914 2024-04-22T18:40:41.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.641049+0000", 2024-04-22T18:40:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6455e-05 2024-04-22T18:40:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.643687+0000", 2024-04-22T18:40:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026375499999999998 2024-04-22T18:40:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.643745+0000", 2024-04-22T18:40:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7725e-05 2024-04-22T18:40:41.458 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:39 3.0 e90/63 3:26983a86:::2d4b6112bcf431e3da007c45a018afa7fe010a4f62df8d2cdc4bdc7924d18240:head v 90'177, mlcod=89'175)", 2024-04-22T18:40:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.526733+0000", 2024-04-22T18:40:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.885263806000001, 2024-04-22T18:40:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10253753, 2024-04-22T18:40:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526733+0000", 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526733+0000", 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526732+0000", 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526806+0000", 2024-04-22T18:40:41.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3538e-05 2024-04-22T18:40:41.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526811+0000", 2024-04-22T18:40:41.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6020000000000002e-06 2024-04-22T18:40:41.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526815+0000", 2024-04-22T18:40:41.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0319999999999997e-06 2024-04-22T18:40:41.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526859+0000", 2024-04-22T18:40:41.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3934000000000003e-05 2024-04-22T18:40:41.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526881+0000", 2024-04-22T18:40:41.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2245000000000001e-05 2024-04-22T18:40:41.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629213+0000", 2024-04-22T18:40:41.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10233270699999999 2024-04-22T18:40:41.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629270+0000", 2024-04-22T18:40:41.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6910999999999998e-05 2024-04-22T18:40:41.463 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.463 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.463 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.463 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:40 3.0 e90/63 3:394d11e1:::4fe26d62e8e61f933e14e15995651b4040179bacb8e375345fa3999664858986:head v 90'178, mlcod=89'175)", 2024-04-22T18:40:41.463 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:41.463 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.885175687, 2024-04-22T18:40:41.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102485589, 2024-04-22T18:40:41.463 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.463 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526928+0000", 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107627 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526932+0000", 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5250000000000001e-06 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526935+0000", 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9359999999999999e-06 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527379+0000", 2024-04-22T18:40:41.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044412 2024-04-22T18:40:41.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527400+0000", 2024-04-22T18:40:41.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0726000000000001e-05 2024-04-22T18:40:41.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629290+0000", 2024-04-22T18:40:41.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101890194 2024-04-22T18:40:41.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629307+0000", 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6855999999999999e-05 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:47 3.0 e90/63 3:0e62034f:::069f05989298bc0f714a6a846e6234d7fa060b1f5513fa9a7b2ddcab658ece07:head v 90'181, mlcod=89'175)", 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.884916367999999, 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10267794, 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527136+0000", 2024-04-22T18:40:41.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6178000000000003e-05 2024-04-22T18:40:41.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527141+0000", 2024-04-22T18:40:41.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4889999999999997e-06 2024-04-22T18:40:41.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527144+0000", 2024-04-22T18:40:41.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0259999999999999e-06 2024-04-22T18:40:41.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.628693+0000", 2024-04-22T18:40:41.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101549033 2024-04-22T18:40:41.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.628713+0000", 2024-04-22T18:40:41.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9932000000000002e-05 2024-04-22T18:40:41.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629733+0000", 2024-04-22T18:40:41.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010206200000000001 2024-04-22T18:40:41.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629758+0000", 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4861000000000001e-05 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:3201 4.4 e91/63 4:24717d05:::smithi08937929-42:head v 91'30, mlcod=74'27)", 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.156471+0000", 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.255525557999999, 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102886424, 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156471+0000", 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156471+0000", 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156470+0000", 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156869+0000", 2024-04-22T18:40:41.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00039887499999999998 2024-04-22T18:40:41.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156879+0000", 2024-04-22T18:40:41.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9399999999999997e-06 2024-04-22T18:40:41.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.257011+0000", 2024-04-22T18:40:41.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100131962 2024-04-22T18:40:41.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.257129+0000", 2024-04-22T18:40:41.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000118644 2024-04-22T18:40:41.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.257144+0000", 2024-04-22T18:40:41.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5211000000000001e-05 2024-04-22T18:40:41.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.259333+0000", 2024-04-22T18:40:41.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021890569999999999 2024-04-22T18:40:41.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.259358+0000", 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4039000000000001e-05 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:66 3.0 e95/63 3:1683cb7f:::f03bfe2533cb949a19fa6d0ef489d3c38ae0c5777f73a45f60cfab23d5b5f5a4:head v 95'186, mlcod=95'183)", 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.635704+0000", 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.776292484000001, 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10283379300000001, 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635704+0000", 2024-04-22T18:40:41.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635704+0000", 2024-04-22T18:40:41.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635703+0000", 2024-04-22T18:40:41.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:41.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635809+0000", 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010657 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635822+0000", 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2500000000000001e-05 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635830+0000", 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5469999999999999e-06 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.735454+0000", 2024-04-22T18:40:41.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099624303999999997 2024-04-22T18:40:41.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.735473+0000", 2024-04-22T18:40:41.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8709999999999999e-05 2024-04-22T18:40:41.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.738511+0000", 2024-04-22T18:40:41.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030386689999999999 2024-04-22T18:40:41.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.738538+0000", 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6767999999999999e-05 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3449 4.d 4:bee1184f:::smithi08937929-7:head [read 0~3613505,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927107+0000", 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4848893079999996, 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103315584, 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:41.479 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3449 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927107+0000", 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927107+0000", 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927105+0000", 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927116+0000", 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1436e-05 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927131+0000", 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5172e-05 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927139+0000", 2024-04-22T18:40:41.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3540000000000002e-06 2024-04-22T18:40:41.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927192+0000", 2024-04-22T18:40:41.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3349e-05 2024-04-22T18:40:41.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:40:41.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927200+0000", 2024-04-22T18:40:41.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:40:41.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027549+0000", 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100349691 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027602+0000", 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2986999999999999e-05 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.030423+0000", 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028205130000000002 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3450 4.d 4:bee1184f:::smithi08937929-7:head [sparse-read 0~2105792] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4848481180000004, 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103826771, 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3450 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:41.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927148+0000", 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927151+0000", 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7e-06 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927156+0000", 2024-04-22T18:40:41.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6699999999999999e-06 2024-04-22T18:40:41.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927161+0000", 2024-04-22T18:40:41.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.933e-06 2024-04-22T18:40:41.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927255+0000", 2024-04-22T18:40:41.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4195000000000002e-05 2024-04-22T18:40:41.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927262+0000", 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9480000000000002e-06 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.030462+0000", 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103199402 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.030553+0000", 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1080000000000002e-05 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.030975+0000", 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000422426 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3451 4.d 4:bee1184f:::smithi08937929-7:head [read 0~3613505,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4848173039999999, 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20164568099999999, 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:41.488 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3451 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:41.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927183+0000", 2024-04-22T18:40:41.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1110000000000001e-06 2024-04-22T18:40:41.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927188+0000", 2024-04-22T18:40:41.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1569999999999999e-06 2024-04-22T18:40:41.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927192+0000", 2024-04-22T18:40:41.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4710000000000001e-06 2024-04-22T18:40:41.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027382+0000", 2024-04-22T18:40:41.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100189341 2024-04-22T18:40:41.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:41.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027387+0000", 2024-04-22T18:40:41.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8400000000000002e-06 2024-04-22T18:40:41.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127638+0000", 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10025102499999999 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127740+0000", 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102749 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.128825+0000", 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001084593 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.492 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:41.494 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:41.494 INFO:tasks.rados.rados.0.smithi089.stdout:1562: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:40:41.494 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:41.495+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:41.554 INFO:tasks.rados.rados.0.smithi089.stdout:1564: expect (ObjNum 227 snap 0 seq_num 227) 2024-04-22T18:40:41.594 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:41.595+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:41.637 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:41.637 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:41.637 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:41.637 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:41.648 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:41.686 INFO:tasks.rados.rados.0.smithi089.stdout:1561: finishing set_chunk to oid 50 2024-04-22T18:40:41.686 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 107 (ObjNum 189 snap 0 seq_num 189) dirty exists 2024-04-22T18:40:41.686 INFO:tasks.rados.rados.0.smithi089.stdout:1563: finishing write tid 1 to smithi08937929-45 2024-04-22T18:40:41.686 INFO:tasks.rados.rados.0.smithi089.stdout:1563: finishing write tid 2 to smithi08937929-45 2024-04-22T18:40:41.686 INFO:tasks.rados.rados.0.smithi089.stdout:1563: finishing write tid 3 to smithi08937929-45 2024-04-22T18:40:41.686 INFO:tasks.rados.rados.0.smithi089.stdout:1563: finishing write tid 4 to smithi08937929-45 2024-04-22T18:40:41.687 INFO:tasks.rados.rados.0.smithi089.stdout:1563: finishing write tid 5 to smithi08937929-45 2024-04-22T18:40:41.687 INFO:tasks.rados.rados.0.smithi089.stdout:1563: oid 45 updating version 0 to 82 2024-04-22T18:40:41.687 INFO:tasks.rados.rados.0.smithi089.stdout:1563: oid 45 version 82 is already newer than 81 2024-04-22T18:40:41.687 INFO:tasks.rados.rados.0.smithi089.stdout:1563: oid 45 version 82 is already newer than 79 2024-04-22T18:40:41.687 INFO:tasks.rados.rados.0.smithi089.stdout:1563: oid 45 version 82 is already newer than 80 2024-04-22T18:40:41.687 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 82 (ObjNum 310 snap 0 seq_num 310) dirty exists 2024-04-22T18:40:41.687 INFO:tasks.rados.rados.0.smithi089.stdout:1563: left oid 45 (ObjNum 310 snap 0 seq_num 310) 2024-04-22T18:40:41.687 INFO:tasks.rados.rados.0.smithi089.stdout:1549: done (15 left) 2024-04-22T18:40:41.687 INFO:tasks.rados.rados.0.smithi089.stdout:1550: done (14 left) 2024-04-22T18:40:41.687 INFO:tasks.rados.rados.0.smithi089.stdout:1551: done (13 left) 2024-04-22T18:40:41.687 INFO:tasks.rados.rados.0.smithi089.stdout:1552: done (12 left) 2024-04-22T18:40:41.687 INFO:tasks.rados.rados.0.smithi089.stdout:1553: done (11 left) 2024-04-22T18:40:41.687 INFO:tasks.rados.rados.0.smithi089.stdout:1554: done (10 left) 2024-04-22T18:40:41.687 INFO:tasks.rados.rados.0.smithi089.stdout:1555: done (9 left) 2024-04-22T18:40:41.687 INFO:tasks.rados.rados.0.smithi089.stdout:1556: done (8 left) 2024-04-22T18:40:41.688 INFO:tasks.rados.rados.0.smithi089.stdout:1557: done (7 left) 2024-04-22T18:40:41.688 INFO:tasks.rados.rados.0.smithi089.stdout:1558: done (6 left) 2024-04-22T18:40:41.688 INFO:tasks.rados.rados.0.smithi089.stdout:1559: done (5 left) 2024-04-22T18:40:41.688 INFO:tasks.rados.rados.0.smithi089.stdout:1560: done (4 left) 2024-04-22T18:40:41.688 INFO:tasks.rados.rados.0.smithi089.stdout:1561: done (3 left) 2024-04-22T18:40:41.688 INFO:tasks.rados.rados.0.smithi089.stdout:1562: done (2 left) 2024-04-22T18:40:41.688 INFO:tasks.rados.rados.0.smithi089.stdout:1563: done (1 left) 2024-04-22T18:40:41.688 INFO:tasks.rados.rados.0.smithi089.stdout:1564: done (0 left) 2024-04-22T18:40:41.688 INFO:tasks.rados.rados.0.smithi089.stdout:1565: write (excl) oid 8 current snap is 0 2024-04-22T18:40:41.688 INFO:tasks.rados.rados.0.smithi089.stdout:1565: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-04-22T18:40:41.690 INFO:tasks.rados.rados.0.smithi089.stdout:1565: writing smithi08937929-8 from 426977 to 883536 tid 1 2024-04-22T18:40:41.694 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:41.695+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:41.698 INFO:tasks.rados.rados.0.smithi089.stdout:1565: writing smithi08937929-8 from 1482841 to 2282605 tid 2 2024-04-22T18:40:41.704 INFO:tasks.rados.rados.0.smithi089.stdout:1565: writing smithi08937929-8 from 2958080 to 3226977 tid 3 2024-04-22T18:40:41.704 INFO:tasks.rados.rados.0.smithi089.stdout:1566: tier_evict oid 43 2024-04-22T18:40:41.704 INFO:tasks.rados.rados.0.smithi089.stdout:1566: tier_evict oid 43 snap -1 2024-04-22T18:40:41.704 INFO:tasks.rados.rados.0.smithi089.stdout:1567: read oid 36 snap -1 2024-04-22T18:40:41.704 INFO:tasks.rados.rados.0.smithi089.stdout:1567: expect (ObjNum 208 snap 0 seq_num 208) 2024-04-22T18:40:41.705 INFO:tasks.rados.rados.0.smithi089.stdout:1568: read oid 20 snap -1 2024-04-22T18:40:41.705 INFO:tasks.rados.rados.0.smithi089.stdout:1568: expect (ObjNum 201 snap 0 seq_num 201) 2024-04-22T18:40:41.705 INFO:tasks.rados.rados.0.smithi089.stdout:1569: read oid 34 snap -1 2024-04-22T18:40:41.705 INFO:tasks.rados.rados.0.smithi089.stdout:1569: expect (ObjNum 277 snap 0 seq_num 277) 2024-04-22T18:40:41.705 INFO:tasks.rados.rados.0.smithi089.stdout:1570: read oid 27 snap -1 2024-04-22T18:40:41.705 INFO:tasks.rados.rados.0.smithi089.stdout:1570: expect (ObjNum 287 snap 0 seq_num 287) 2024-04-22T18:40:41.705 INFO:tasks.rados.rados.0.smithi089.stdout:1571: read oid 42 snap -1 2024-04-22T18:40:41.705 INFO:tasks.rados.rados.0.smithi089.stdout:1571: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:41.705 INFO:tasks.rados.rados.0.smithi089.stdout:1572: delete oid 24 current snap is 0 2024-04-22T18:40:41.791 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:41.791 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:41.791 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:41.792 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:41.792 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:41.795 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:41.796+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:41.800 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:41.805 INFO:tasks.rados.rados.0.smithi089.stdout:1566: got (22) Invalid argument 2024-04-22T18:40:41.805 INFO:tasks.rados.rados.0.smithi089.stdout:1565: finishing write tid 1 to smithi08937929-8 2024-04-22T18:40:41.805 INFO:tasks.rados.rados.0.smithi089.stdout:1565: finishing write tid 2 to smithi08937929-8 2024-04-22T18:40:41.807 INFO:tasks.rados.rados.0.smithi089.stdout:1571: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:41.896 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:41.896+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:41.912 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:41.912 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:41.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:41.912 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:41.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:78 4.0 4:01baf5a9:::smithi08937929-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.400265748, 2024-04-22T18:40:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.398522192, 2024-04-22T18:40:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 78 2024-04-22T18:40:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:41.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:41.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511934+0000", 2024-04-22T18:40:41.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:41.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512012+0000", 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8329000000000002e-05 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512023+0000", 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0755e-05 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512028+0000", 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6470000000000003e-06 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512068+0000", 2024-04-22T18:40:41.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9721999999999997e-05 2024-04-22T18:40:41.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512125+0000", 2024-04-22T18:40:41.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6564999999999998e-05 2024-04-22T18:40:41.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:41.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512200+0000", 2024-04-22T18:40:41.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5708000000000001e-05 2024-04-22T18:40:41.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:41.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513131+0000", 2024-04-22T18:40:41.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093063 2024-04-22T18:40:41.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:41.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910394+0000", 2024-04-22T18:40:41.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.397263388 2024-04-22T18:40:41.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910430+0000", 2024-04-22T18:40:41.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5485999999999999e-05 2024-04-22T18:40:41.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910457+0000", 2024-04-22T18:40:41.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7086000000000001e-05 2024-04-22T18:40:41.918 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.918 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.918 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.918 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:79 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:41.918 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:41.918 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.40016509199999, 2024-04-22T18:40:41.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39855419399999997, 2024-04-22T18:40:41.918 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.918 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.918 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 79 2024-04-22T18:40:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512035+0000", 2024-04-22T18:40:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512045+0000", 2024-04-22T18:40:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0379e-05 2024-04-22T18:40:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512051+0000", 2024-04-22T18:40:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6030000000000004e-06 2024-04-22T18:40:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512057+0000", 2024-04-22T18:40:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7270000000000004e-06 2024-04-22T18:40:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512756+0000", 2024-04-22T18:40:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069954000000000001 2024-04-22T18:40:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512806+0000", 2024-04-22T18:40:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9922000000000001e-05 2024-04-22T18:40:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512927+0000", 2024-04-22T18:40:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012059 2024-04-22T18:40:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.515883+0000", 2024-04-22T18:40:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029563739999999999 2024-04-22T18:40:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910510+0000", 2024-04-22T18:40:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39462745199999999 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910542+0000", 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1225e-05 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910590+0000", 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8052e-05 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.923 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:80 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.400139991, 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39866376999999997, 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 80 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512060+0000", 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512063+0000", 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.999e-06 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512069+0000", 2024-04-22T18:40:41.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.891e-06 2024-04-22T18:40:41.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512073+0000", 2024-04-22T18:40:41.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4429999999999996e-06 2024-04-22T18:40:41.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513084+0000", 2024-04-22T18:40:41.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010102609999999999 2024-04-22T18:40:41.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513123+0000", 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9237999999999999e-05 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910618+0000", 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39749549299999998 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910670+0000", 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1221000000000003e-05 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910724+0000", 2024-04-22T18:40:41.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4777000000000001e-05 2024-04-22T18:40:41.928 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.928 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.928 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.928 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:229 3.2 e51/29 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head v 51'117, mlcod=51'115)", 2024-04-22T18:40:41.928 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:41.928 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.101157932999996, 2024-04-22T18:40:41.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31548799399999999, 2024-04-22T18:40:41.928 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.928 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.928 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811041+0000", 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811054+0000", 2024-04-22T18:40:41.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3188e-05 2024-04-22T18:40:41.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811068+0000", 2024-04-22T18:40:41.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4239999999999999e-05 2024-04-22T18:40:41.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.911141+0000", 2024-04-22T18:40:41.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100072851 2024-04-22T18:40:41.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025053+0000", 2024-04-22T18:40:41.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.11391237 2024-04-22T18:40:41.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025083+0000", 2024-04-22T18:40:41.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9703999999999999e-05 2024-04-22T18:40:41.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126504+0000", 2024-04-22T18:40:41.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10142053500000001 2024-04-22T18:40:41.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126531+0000", 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7172999999999999e-05 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2348 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.724368392000002, 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30532242999999998, 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:41.932 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2348 2024-04-22T18:40:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187833+0000", 2024-04-22T18:40:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.437e-06 2024-04-22T18:40:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187836+0000", 2024-04-22T18:40:41.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.897e-06 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187839+0000", 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7250000000000002e-06 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292213+0000", 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104373761 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292256+0000", 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2896999999999999e-05 2024-04-22T18:40:41.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493044+0000", 2024-04-22T18:40:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20078818700000001 2024-04-22T18:40:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493099+0000", 2024-04-22T18:40:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5009000000000001e-05 2024-04-22T18:40:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493155+0000", 2024-04-22T18:40:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5807000000000003e-05 2024-04-22T18:40:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.937 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.937 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.937 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:289 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:41.937 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:41.937 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.271475682000002, 2024-04-22T18:40:41.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9176801189999999, 2024-04-22T18:40:41.937 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.937 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.937 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:41.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:41.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640723+0000", 2024-04-22T18:40:41.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:41.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640732+0000", 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5809999999999997e-06 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640744+0000", 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.261e-05 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640753+0000", 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7460000000000006e-06 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642189+0000", 2024-04-22T18:40:41.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001435551 2024-04-22T18:40:41.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:41.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642200+0000", 2024-04-22T18:40:41.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1389999999999999e-05 2024-04-22T18:40:41.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.847015+0000", 2024-04-22T18:40:41.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048149160000001 2024-04-22T18:40:41.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:41.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.848217+0000", 2024-04-22T18:40:41.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012020659999999999 2024-04-22T18:40:41.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351823+0000", 2024-04-22T18:40:41.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50360565099999999 2024-04-22T18:40:41.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351907+0000", 2024-04-22T18:40:41.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4141999999999995e-05 2024-04-22T18:40:41.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:41.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352051+0000", 2024-04-22T18:40:41.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014386600000000001 2024-04-22T18:40:41.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:41.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453617+0000", 2024-04-22T18:40:41.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156649500000001 2024-04-22T18:40:41.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:41.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558297+0000", 2024-04-22T18:40:41.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10468000399999999 2024-04-22T18:40:41.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558331+0000", 2024-04-22T18:40:41.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3518e-05 2024-04-22T18:40:41.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558405+0000", 2024-04-22T18:40:41.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4221000000000006e-05 2024-04-22T18:40:41.943 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.943 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.943 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.943 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:1 3.2 3:6d90b903:::64b5942574fb36ca1b9a2be186c6b740edf58e989056a2f88b83cd280451a438:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.740174721999999, 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.285534422, 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172024+0000", 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173091+0000", 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010677569999999999 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173103+0000", 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1241e-05 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173109+0000", 2024-04-22T18:40:41.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4300000000000003e-06 2024-04-22T18:40:41.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173168+0000", 2024-04-22T18:40:41.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9011000000000001e-05 2024-04-22T18:40:41.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:41.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173177+0000", 2024-04-22T18:40:41.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-04-22T18:40:41.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948501+0000", 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77532387199999997 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948912+0000", 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041183000000000002 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950438+0000", 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015253199999999999 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950488+0000", 2024-04-22T18:40:41.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0285999999999999e-05 2024-04-22T18:40:41.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:41.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950601+0000", 2024-04-22T18:40:41.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011244899999999999 2024-04-22T18:40:41.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:41.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954048+0000", 2024-04-22T18:40:41.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003447313 2024-04-22T18:40:41.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:41.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457373+0000", 2024-04-22T18:40:41.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50332465299999996 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457430+0000", 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7574999999999997e-05 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457560+0000", 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013023900000000001 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:8 3.2 3:5f64590d:::7c47933f331d40689081b2bd8f9d7f7bdae1be1fdccb52d696047a3969e902b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:41.949 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:41.950 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.739082353000001, 2024-04-22T18:40:41.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3854200569999999, 2024-04-22T18:40:41.950 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.950 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.950 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:41.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173117+0000", 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173393+0000", 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000275278 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173399+0000", 2024-04-22T18:40:41.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4590000000000003e-06 2024-04-22T18:40:41.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173406+0000", 2024-04-22T18:40:41.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2450000000000001e-06 2024-04-22T18:40:41.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173795+0000", 2024-04-22T18:40:41.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038912400000000002 2024-04-22T18:40:41.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:41.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173804+0000", 2024-04-22T18:40:41.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-04-22T18:40:41.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.949506+0000", 2024-04-22T18:40:41.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77570185700000005 2024-04-22T18:40:41.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:41.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950166+0000", 2024-04-22T18:40:41.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066022800000000001 2024-04-22T18:40:41.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352843+0000", 2024-04-22T18:40:41.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40267661799999999 2024-04-22T18:40:41.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352911+0000", 2024-04-22T18:40:41.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8126000000000005e-05 2024-04-22T18:40:41.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:41.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.353031+0000", 2024-04-22T18:40:41.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119919 2024-04-22T18:40:41.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:41.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453622+0000", 2024-04-22T18:40:41.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100590793 2024-04-22T18:40:41.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:41.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558446+0000", 2024-04-22T18:40:41.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104824073 2024-04-22T18:40:41.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558479+0000", 2024-04-22T18:40:41.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3362999999999999e-05 2024-04-22T18:40:41.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558538+0000", 2024-04-22T18:40:41.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9046999999999999e-05 2024-04-22T18:40:41.956 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.956 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.956 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.956 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:10 3.2 3:518c37f9:::851eca84dafbf2f780c4f07291110c453349bdc9c623c17bdcd26e27badb30ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:41.956 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:41.956 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.738788759999998, 2024-04-22T18:40:41.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3852527729999999, 2024-04-22T18:40:41.956 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.956 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.956 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:41.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:41.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173411+0000", 2024-04-22T18:40:41.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:41.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173749+0000", 2024-04-22T18:40:41.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000338029 2024-04-22T18:40:41.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173755+0000", 2024-04-22T18:40:41.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0530000000000003e-06 2024-04-22T18:40:41.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173759+0000", 2024-04-22T18:40:41.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8190000000000002e-06 2024-04-22T18:40:41.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173855+0000", 2024-04-22T18:40:41.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5927999999999994e-05 2024-04-22T18:40:41.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:41.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173862+0000", 2024-04-22T18:40:41.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4449999999999999e-06 2024-04-22T18:40:41.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950835+0000", 2024-04-22T18:40:41.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77697248299999999 2024-04-22T18:40:41.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:41.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951632+0000", 2024-04-22T18:40:41.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079752299999999996 2024-04-22T18:40:41.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355407+0000", 2024-04-22T18:40:41.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40377480500000001 2024-04-22T18:40:41.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355462+0000", 2024-04-22T18:40:41.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4577999999999998e-05 2024-04-22T18:40:41.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:41.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355571+0000", 2024-04-22T18:40:41.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108964 2024-04-22T18:40:41.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:41.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453625+0000", 2024-04-22T18:40:41.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098054397000000001 2024-04-22T18:40:41.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:41.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558574+0000", 2024-04-22T18:40:41.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104948571 2024-04-22T18:40:41.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558602+0000", 2024-04-22T18:40:41.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7966999999999998e-05 2024-04-22T18:40:41.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558665+0000", 2024-04-22T18:40:41.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3032000000000002e-05 2024-04-22T18:40:41.962 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.962 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.962 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.963 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:11 3.2 3:718aa8aa:::9603c4f8e269a436b2f0bf376c5374e77bdd7cf58fccae60e4e65ac4696806d4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:41.963 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:41.963 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.738437142000002, 2024-04-22T18:40:41.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4852604440000001, 2024-04-22T18:40:41.963 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.963 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.963 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:41.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:41.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:41.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173813+0000", 2024-04-22T18:40:41.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0655999999999997e-05 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173819+0000", 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.468e-06 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173822+0000", 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4879999999999999e-06 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173983+0000", 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160519 2024-04-22T18:40:41.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173991+0000", 2024-04-22T18:40:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6490000000000008e-06 2024-04-22T18:40:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951688+0000", 2024-04-22T18:40:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77769743800000002 2024-04-22T18:40:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952419+0000", 2024-04-22T18:40:41.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073067499999999997 2024-04-22T18:40:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355901+0000", 2024-04-22T18:40:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40348237599999998 2024-04-22T18:40:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355953+0000", 2024-04-22T18:40:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1477000000000002e-05 2024-04-22T18:40:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:41.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356060+0000", 2024-04-22T18:40:41.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107114 2024-04-22T18:40:41.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:41.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453629+0000", 2024-04-22T18:40:41.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097569127000000005 2024-04-22T18:40:41.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:41.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658807+0000", 2024-04-22T18:40:41.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20517829800000001 2024-04-22T18:40:41.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658876+0000", 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8454000000000006e-05 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659024+0000", 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148338 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:12 3.2 3:475e1cd9:::efaab2bd13543cdbb143e869e20eb0ec424f7f5202707859b241ed91d4674936:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.738374198000002, 2024-04-22T18:40:41.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4853405259999999, 2024-04-22T18:40:41.970 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.970 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.970 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:41.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:41.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:41.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173940+0000", 2024-04-22T18:40:41.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011465500000000001 2024-04-22T18:40:41.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173946+0000", 2024-04-22T18:40:41.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5289999999999999e-06 2024-04-22T18:40:41.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173950+0000", 2024-04-22T18:40:41.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8290000000000001e-06 2024-04-22T18:40:41.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174277+0000", 2024-04-22T18:40:41.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032761299999999998 2024-04-22T18:40:41.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:41.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174285+0000", 2024-04-22T18:40:41.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.52e-06 2024-04-22T18:40:41.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952455+0000", 2024-04-22T18:40:41.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77817000700000005 2024-04-22T18:40:41.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:41.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954035+0000", 2024-04-22T18:40:41.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001580078 2024-04-22T18:40:41.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356379+0000", 2024-04-22T18:40:41.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40234440399999999 2024-04-22T18:40:41.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356428+0000", 2024-04-22T18:40:41.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8832999999999998e-05 2024-04-22T18:40:41.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:41.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356532+0000", 2024-04-22T18:40:41.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103235 2024-04-22T18:40:41.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:41.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453632+0000", 2024-04-22T18:40:41.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097100346000000004 2024-04-22T18:40:41.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:41.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659069+0000", 2024-04-22T18:40:41.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20543763800000001 2024-04-22T18:40:41.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659105+0000", 2024-04-22T18:40:41.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5240000000000001e-05 2024-04-22T18:40:41.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659167+0000", 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2207999999999998e-05 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:16 3.2 3:47c76cc8:::c00d18458eb20c0c0aa9c612cf9793a68445ab0ea7b9e650ecd0072c60a96c0f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.738246238999999, 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.485347744, 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:41.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:41.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:41.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:41.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174234+0000", 2024-04-22T18:40:41.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028047599999999999 2024-04-22T18:40:41.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174240+0000", 2024-04-22T18:40:41.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2079999999999997e-06 2024-04-22T18:40:41.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174244+0000", 2024-04-22T18:40:41.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5810000000000001e-06 2024-04-22T18:40:41.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174670+0000", 2024-04-22T18:40:41.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042640000000000001 2024-04-22T18:40:41.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:41.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174675+0000", 2024-04-22T18:40:41.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8709999999999996e-06 2024-04-22T18:40:41.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054168+0000", 2024-04-22T18:40:41.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87949257300000006 2024-04-22T18:40:41.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:41.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054877+0000", 2024-04-22T18:40:41.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070884300000000002 2024-04-22T18:40:41.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356836+0000", 2024-04-22T18:40:41.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30195926000000001 2024-04-22T18:40:41.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356887+0000", 2024-04-22T18:40:41.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0767999999999999e-05 2024-04-22T18:40:41.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:41.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356991+0000", 2024-04-22T18:40:41.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104072 2024-04-22T18:40:41.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:41.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453636+0000", 2024-04-22T18:40:41.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096644968999999997 2024-04-22T18:40:41.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:41.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659207+0000", 2024-04-22T18:40:41.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20557154699999999 2024-04-22T18:40:41.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659240+0000", 2024-04-22T18:40:41.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2665e-05 2024-04-22T18:40:41.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659302+0000", 2024-04-22T18:40:41.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2234000000000001e-05 2024-04-22T18:40:41.982 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.982 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.982 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.983 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:17 3.2 3:6ef31daa:::1cf4ff496c4480e58510f607481d976b47d84868e2c54016273c38baf590f3b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:41.983 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:41.983 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.737951959, 2024-04-22T18:40:41.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4851703729999999, 2024-04-22T18:40:41.983 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.983 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.983 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:41.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:41.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174248+0000", 2024-04-22T18:40:41.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:41.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174626+0000", 2024-04-22T18:40:41.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037839699999999999 2024-04-22T18:40:41.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174633+0000", 2024-04-22T18:40:41.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0990000000000004e-06 2024-04-22T18:40:41.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174636+0000", 2024-04-22T18:40:41.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5540000000000002e-06 2024-04-22T18:40:41.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174790+0000", 2024-04-22T18:40:41.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153807 2024-04-22T18:40:41.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:41.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174793+0000", 2024-04-22T18:40:41.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4970000000000001e-06 2024-04-22T18:40:41.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054902+0000", 2024-04-22T18:40:41.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88010886300000002 2024-04-22T18:40:41.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:41.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055366+0000", 2024-04-22T18:40:41.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046336199999999999 2024-04-22T18:40:41.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.452973+0000", 2024-04-22T18:40:41.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39760783999999999 2024-04-22T18:40:41.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453046+0000", 2024-04-22T18:40:41.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2374999999999997e-05 2024-04-22T18:40:41.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:41.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453193+0000", 2024-04-22T18:40:41.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014722499999999999 2024-04-22T18:40:41.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554698+0000", 2024-04-22T18:40:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101505096 2024-04-22T18:40:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659337+0000", 2024-04-22T18:40:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104639236 2024-04-22T18:40:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659365+0000", 2024-04-22T18:40:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7594e-05 2024-04-22T18:40:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659419+0000", 2024-04-22T18:40:41.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4058999999999999e-05 2024-04-22T18:40:41.989 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.989 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.989 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.989 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:18 3.2 3:792a4ed0:::e1c814961b3d1531fb5ed475b3d496a0405c3a819b68a92ff9b2391fe912370f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:41.989 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:41.989 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.737555702999998, 2024-04-22T18:40:41.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.585036954, 2024-04-22T18:40:41.989 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.990 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.990 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:41.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:41.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174640+0000", 2024-04-22T18:40:41.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-04-22T18:40:41.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174758+0000", 2024-04-22T18:40:41.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011819099999999999 2024-04-22T18:40:41.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174764+0000", 2024-04-22T18:40:41.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6729999999999997e-06 2024-04-22T18:40:41.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174767+0000", 2024-04-22T18:40:41.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3830000000000001e-06 2024-04-22T18:40:41.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373669+0000", 2024-04-22T18:40:41.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198901989 2024-04-22T18:40:41.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:41.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373686+0000", 2024-04-22T18:40:41.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7286e-05 2024-04-22T18:40:41.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055408+0000", 2024-04-22T18:40:41.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68172133400000001 2024-04-22T18:40:41.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:41.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055938+0000", 2024-04-22T18:40:41.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053016599999999999 2024-04-22T18:40:41.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453668+0000", 2024-04-22T18:40:41.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39773030799999998 2024-04-22T18:40:41.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:41.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453717+0000", 2024-04-22T18:40:41.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8884000000000002e-05 2024-04-22T18:40:41.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:41.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453823+0000", 2024-04-22T18:40:41.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105791 2024-04-22T18:40:41.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:41.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554715+0000", 2024-04-22T18:40:41.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100891628 2024-04-22T18:40:41.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:41.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759439+0000", 2024-04-22T18:40:41.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20472459400000001 2024-04-22T18:40:41.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:41.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759525+0000", 2024-04-22T18:40:41.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5653000000000001e-05 2024-04-22T18:40:41.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:41.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759682+0000", 2024-04-22T18:40:41.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015711800000000001 2024-04-22T18:40:41.995 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.996 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:41.996 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:41.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.996 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:75 3.1 3:ab43a94d:::f38e6e97cb24e2b90da964fee5d2d6d8674386870fc552915e0c3fdbefd4b126:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:41.996 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:40:41.996 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.6646791859999999, 2024-04-22T18:40:41.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60149928399999997, 2024-04-22T18:40:41.996 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:41.996 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:41.996 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:41.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:41.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:40:41.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247520+0000", 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247527+0000", 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1670000000000002e-06 2024-04-22T18:40:41.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:41.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247539+0000", 2024-04-22T18:40:41.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503e-05 2024-04-22T18:40:41.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:41.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247549+0000", 2024-04-22T18:40:41.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0000000000000001e-05 2024-04-22T18:40:41.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247796+0000", 2024-04-22T18:40:41.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024712699999999998 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247955+0000", 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158534 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.343985+0000", 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096030344000000004 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.344035+0000", 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9969999999999998e-05 2024-04-22T18:40:41.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.745806+0000", 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.401771029 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746040+0000", 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023339300000000001 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746198+0000", 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015793299999999999 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848737+0000", 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10253898 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848751+0000", 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4694e-05 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848825+0000", 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3443999999999997e-05 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:42.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849021+0000", 2024-04-22T18:40:42.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000196047 2024-04-22T18:40:42.002 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:42.002 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:42.002 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:42.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.002 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:76 3.1 3:b44829a0:::3788286a255d538503614e5f04049b0bb54cacaa317ab102b5dee8f5b640b296:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:42.002 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:40:42.002 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.6646401199999996, 2024-04-22T18:40:42.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60168681599999996, 2024-04-22T18:40:42.002 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:42.002 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:42.002 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:42.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:42.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247564+0000", 2024-04-22T18:40:42.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1150000000000004e-06 2024-04-22T18:40:42.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:42.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247570+0000", 2024-04-22T18:40:42.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3940000000000001e-06 2024-04-22T18:40:42.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247576+0000", 2024-04-22T18:40:42.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3619999999999997e-06 2024-04-22T18:40:42.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:42.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.340845+0000", 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.093269199999999997 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.341043+0000", 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019834400000000001 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345106+0000", 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040629780000000001 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345160+0000", 2024-04-22T18:40:42.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4246999999999997e-05 2024-04-22T18:40:42.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:42.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747047+0000", 2024-04-22T18:40:42.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40188678300000003 2024-04-22T18:40:42.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:42.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747162+0000", 2024-04-22T18:40:42.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011483400000000001 2024-04-22T18:40:42.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:42.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747429+0000", 2024-04-22T18:40:42.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026700299999999998 2024-04-22T18:40:42.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:42.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849032+0000", 2024-04-22T18:40:42.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101603203 2024-04-22T18:40:42.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:42.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849095+0000", 2024-04-22T18:40:42.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2597999999999999e-05 2024-04-22T18:40:42.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:42.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849136+0000", 2024-04-22T18:40:42.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1251000000000002e-05 2024-04-22T18:40:42.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:42.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849247+0000", 2024-04-22T18:40:42.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011127 2024-04-22T18:40:42.008 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:42.008 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:42.008 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:42.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.008 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3524 4.c 4:314c2fec:::smithi08937929-22:head [write 692521~538214] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:40:42.008 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.128717+0000", 2024-04-22T18:40:42.008 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.7834840249999999, 2024-04-22T18:40:42.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40275804199999998, 2024-04-22T18:40:42.009 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:42.009 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:42.009 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:42.009 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:42.009 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.009 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3524 2024-04-22T18:40:42.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.009 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:42.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:42.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128717+0000", 2024-04-22T18:40:42.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:42.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:42.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128717+0000", 2024-04-22T18:40:42.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:42.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:42.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128715+0000", 2024-04-22T18:40:42.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:42.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:42.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128877+0000", 2024-04-22T18:40:42.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016187999999999999 2024-04-22T18:40:42.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128886+0000", 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8780000000000004e-06 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.228979+0000", 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092734 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229043+0000", 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4036999999999997e-05 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:42.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229146+0000", 2024-04-22T18:40:42.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010301700000000001 2024-04-22T18:40:42.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:42.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229286+0000", 2024-04-22T18:40:42.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014017000000000001 2024-04-22T18:40:42.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:42.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.232704+0000", 2024-04-22T18:40:42.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0034182169999999999 2024-04-22T18:40:42.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.531413+0000", 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29870898200000001 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.531445+0000", 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1751999999999997e-05 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.531475+0000", 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9906000000000001e-05 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:42.013 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:42.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.014 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3525 4.c 4:314c2fec:::smithi08937929-22:head [write 1931338~785088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:40:42.014 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.229011+0000", 2024-04-22T18:40:42.014 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.6831894270000003, 2024-04-22T18:40:42.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40428670300000003, 2024-04-22T18:40:42.014 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:42.014 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:42.014 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:42.014 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:42.014 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.014 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3525 2024-04-22T18:40:42.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.014 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229011+0000", 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229011+0000", 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229010+0000", 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:42.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229182+0000", 2024-04-22T18:40:42.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000171645 2024-04-22T18:40:42.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:42.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229188+0000", 2024-04-22T18:40:42.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.674e-06 2024-04-22T18:40:42.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229193+0000", 2024-04-22T18:40:42.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8890000000000001e-06 2024-04-22T18:40:42.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:42.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.230814+0000", 2024-04-22T18:40:42.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00162068 2024-04-22T18:40:42.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:42.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.230865+0000", 2024-04-22T18:40:42.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1700000000000003e-05 2024-04-22T18:40:42.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:42.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.230967+0000", 2024-04-22T18:40:42.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101481 2024-04-22T18:40:42.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.234144+0000", 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031771849999999999 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.633226+0000", 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39908168799999999 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.633257+0000", 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1096999999999997e-05 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:42.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.633298+0000", 2024-04-22T18:40:42.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0960000000000001e-05 2024-04-22T18:40:42.019 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:42.019 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:42.019 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:42.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.019 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3526 4.c 4:314c2fec:::smithi08937929-22:head [setxattr _header (56) in=7b,truncate 3092521] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:40:42.019 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:40:42.019 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.68300334, 2024-04-22T18:40:42.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60463626000000004, 2024-04-22T18:40:42.019 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:42.019 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:42.019 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:42.020 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:42.020 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.020 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3526 2024-04-22T18:40:42.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.020 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:42.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:42.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:40:42.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:42.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:42.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:40:42.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229200+0000", 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2320000000000001e-06 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229203+0000", 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:40:42.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329292+0000", 2024-04-22T18:40:42.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100088891 2024-04-22T18:40:42.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:42.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329357+0000", 2024-04-22T18:40:42.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5115000000000003e-05 2024-04-22T18:40:42.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:42.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329453+0000", 2024-04-22T18:40:42.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5802000000000002e-05 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329594+0000", 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014157500000000001 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.330172+0000", 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057782799999999996 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833712+0000", 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50353975299999998 2024-04-22T18:40:42.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:42.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833774+0000", 2024-04-22T18:40:42.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1832e-05 2024-04-22T18:40:42.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:42.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833834+0000", 2024-04-22T18:40:42.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9809999999999998e-05 2024-04-22T18:40:42.024 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:42.024 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:42.024 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:42.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.024 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3527 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.329322+0000", 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.582878268, 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50479107700000003, 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3527 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329322+0000", 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329322+0000", 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329321+0000", 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329337+0000", 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5820000000000001e-05 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:42.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329345+0000", 2024-04-22T18:40:42.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4610000000000008e-06 2024-04-22T18:40:42.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.429437+0000", 2024-04-22T18:40:42.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092323 2024-04-22T18:40:42.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:42.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.429503+0000", 2024-04-22T18:40:42.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5446999999999999e-05 2024-04-22T18:40:42.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.429626+0000", 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012354600000000001 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833904+0000", 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.404277901 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.834004+0000", 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9444999999999994e-05 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.834113+0000", 2024-04-22T18:40:42.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010976099999999999 2024-04-22T18:40:42.029 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:42.029 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:42.029 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:42.029 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:42.029 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:42.029 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:42.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:42.030 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:42.031+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:42.033 INFO:tasks.rados.rados.0.smithi089.stdout:1565: finishing write tid 3 to smithi08937929-8 2024-04-22T18:40:42.033 INFO:tasks.rados.rados.0.smithi089.stdout:1568: expect (ObjNum 201 snap 0 seq_num 201) 2024-04-22T18:40:42.130 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:42.131+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:42.144 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:42.144 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:42.144 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:42.144 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:42.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:42.167 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:40:42.223 INFO:tasks.rados.rados.0.smithi089.stdout:1565: finishing write tid 4 to smithi08937929-8 2024-04-22T18:40:42.223 INFO:tasks.rados.rados.0.smithi089.stdout:1565: finishing write tid 5 to smithi08937929-8 2024-04-22T18:40:42.223 INFO:tasks.rados.rados.0.smithi089.stdout:1565: oid 8 updating version 0 to 32 2024-04-22T18:40:42.223 INFO:tasks.rados.rados.0.smithi089.stdout:1565: oid 8 version 32 is already newer than 31 2024-04-22T18:40:42.223 INFO:tasks.rados.rados.0.smithi089.stdout:1565: oid 8 updating version 32 to 33 2024-04-22T18:40:42.223 INFO:tasks.rados.rados.0.smithi089.stdout:1565: oid 8 version 33 is already newer than 30 2024-04-22T18:40:42.223 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 33 (ObjNum 311 snap 0 seq_num 311) dirty exists 2024-04-22T18:40:42.223 INFO:tasks.rados.rados.0.smithi089.stdout:1565: left oid 8 (ObjNum 311 snap 0 seq_num 311) 2024-04-22T18:40:42.223 INFO:tasks.rados.rados.0.smithi089.stdout:1565: done (7 left) 2024-04-22T18:40:42.223 INFO:tasks.rados.rados.0.smithi089.stdout:1566: done (6 left) 2024-04-22T18:40:42.224 INFO:tasks.rados.rados.0.smithi089.stdout:1568: done (5 left) 2024-04-22T18:40:42.224 INFO:tasks.rados.rados.0.smithi089.stdout:1571: done (4 left) 2024-04-22T18:40:42.224 INFO:tasks.rados.rados.0.smithi089.stdout:1572: done (3 left) 2024-04-22T18:40:42.224 INFO:tasks.rados.rados.0.smithi089.stdout:1573: write (excl) oid 41 current snap is 0 2024-04-22T18:40:42.224 INFO:tasks.rados.rados.0.smithi089.stdout:1573: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2024-04-22T18:40:42.229 INFO:tasks.rados.rados.0.smithi089.stdout:1573: writing smithi08937929-41 from 460591 to 1184001 tid 1 2024-04-22T18:40:42.231 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:42.231+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:42.234 INFO:tasks.rados.rados.0.smithi089.stdout:1573: writing smithi08937929-41 from 1716158 to 2119591 tid 2 2024-04-22T18:40:42.241 INFO:tasks.rados.rados.0.smithi089.stdout:1573: writing smithi08937929-41 from 2729279 to 3169419 tid 3 2024-04-22T18:40:42.241 INFO:tasks.rados.rados.0.smithi089.stdout:1574: read oid 4 snap -1 2024-04-22T18:40:42.241 INFO:tasks.rados.rados.0.smithi089.stdout:1574: expect (ObjNum 269 snap 0 seq_num 269) 2024-04-22T18:40:42.242 INFO:tasks.rados.rados.0.smithi089.stdout:1575: tier_evict oid 35 2024-04-22T18:40:42.242 INFO:tasks.rados.rados.0.smithi089.stdout:1575: tier_evict oid 35 snap -1 2024-04-22T18:40:42.242 INFO:tasks.rados.rados.0.smithi089.stdout:1576: tier_flush oid 9 2024-04-22T18:40:42.242 INFO:tasks.rados.rados.0.smithi089.stdout:1576: tier_flush oid 9 snap -1 2024-04-22T18:40:42.242 INFO:tasks.rados.rados.0.smithi089.stdout:1577: read oid 15 snap -1 2024-04-22T18:40:42.242 INFO:tasks.rados.rados.0.smithi089.stdout:1577: expect (ObjNum 301 snap 0 seq_num 301) 2024-04-22T18:40:42.242 INFO:tasks.rados.rados.0.smithi089.stdout:1578: read oid 48 snap -1 2024-04-22T18:40:42.242 INFO:tasks.rados.rados.0.smithi089.stdout:1578: expect (ObjNum 305 snap 0 seq_num 305) 2024-04-22T18:40:42.242 INFO:tasks.rados.rados.0.smithi089.stdout:1579: read oid 14 snap -1 2024-04-22T18:40:42.242 INFO:tasks.rados.rados.0.smithi089.stdout:1579: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:40:42.242 INFO:tasks.rados.rados.0.smithi089.stdout:1580: read oid 46 snap -1 2024-04-22T18:40:42.242 INFO:tasks.rados.rados.0.smithi089.stdout:1580: expect (ObjNum 278 snap 0 seq_num 278) 2024-04-22T18:40:42.242 INFO:tasks.rados.rados.0.smithi089.stdout:1581: read oid 45 snap -1 2024-04-22T18:40:42.243 INFO:tasks.rados.rados.0.smithi089.stdout:1581: expect (ObjNum 310 snap 0 seq_num 310) 2024-04-22T18:40:42.243 INFO:tasks.rados.rados.0.smithi089.stdout:1582: read oid 50 snap -1 2024-04-22T18:40:42.243 INFO:tasks.rados.rados.0.smithi089.stdout:1582: expect (ObjNum 189 snap 0 seq_num 189) 2024-04-22T18:40:42.243 INFO:tasks.rados.rados.0.smithi089.stdout:1583: read oid 16 snap -1 2024-04-22T18:40:42.243 INFO:tasks.rados.rados.0.smithi089.stdout:1583: expect (ObjNum 279 snap 0 seq_num 279) 2024-04-22T18:40:42.243 INFO:tasks.rados.rados.0.smithi089.stdout:1584: read oid 20 snap -1 2024-04-22T18:40:42.243 INFO:tasks.rados.rados.0.smithi089.stdout:1584: expect (ObjNum 201 snap 0 seq_num 201) 2024-04-22T18:40:42.259 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:42.259 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:42.259 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:42.259 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:42.259 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:42.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:42.286 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:42.286 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:42.302 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:42.302 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:42.317 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:42.318 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:42.332 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:42.333+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:42.337 INFO:tasks.rados.rados.0.smithi089.stdout:1585: read oid 32 snap -1 2024-04-22T18:40:42.337 INFO:tasks.rados.rados.0.smithi089.stdout:1585: expect (ObjNum 296 snap 0 seq_num 296) 2024-04-22T18:40:42.337 INFO:tasks.rados.rados.0.smithi089.stdout:1573: finishing write tid 1 to smithi08937929-41 2024-04-22T18:40:42.360 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:42.360 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:42.374 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:42.375 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:42.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:42.375 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:42.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.375 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:42.375 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:42.375 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 108.21217363300001, 2024-04-22T18:40:42.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:40:42.375 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.375 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:42.375 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.375 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.376 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.376 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:40:42.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.376 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:42.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:42.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:40:42.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:42.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:40:42.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:40:42.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:40:42.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:40:42.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:40:42.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:40:42.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:40:42.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:40:42.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:40:42.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:40:42.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:42.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:40:42.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:40:42.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:42.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:40:42.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:40:42.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:42.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:40:42.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:40:42.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:42.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:40:42.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:40:42.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:40:42.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:40:42.380 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.380 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.381 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.381 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:42.381 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:42.381 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 108.11061481199999, 2024-04-22T18:40:42.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:40:42.381 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.381 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:42.381 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.381 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.381 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.381 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:40:42.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.382 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:42.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:42.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:40:42.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:42.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:40:42.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:40:42.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:40:42.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:40:42.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:40:42.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:40:42.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:40:42.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:40:42.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:40:42.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:40:42.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:42.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:40:42.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:40:42.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:42.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:40:42.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:40:42.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:42.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:40:42.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:40:42.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:42.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:40:42.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:40:42.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:40:42.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:40:42.386 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.386 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.386 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.386 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:42.387 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:42.387 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 108.11058841000001, 2024-04-22T18:40:42.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:40:42.387 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.387 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:42.387 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.387 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.387 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.387 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:40:42.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.387 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:42.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:42.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.388 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:40:42.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:42.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:40:42.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:40:42.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:40:42.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:40:42.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:40:42.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:40:42.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:40:42.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:40:42.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:42.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:40:42.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:40:42.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:40:42.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:40:42.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:40:42.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:40:42.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:40:42.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:40:42.391 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 107.712254081, 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.392 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:42.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:42.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:42.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:42.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:40:42.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:40:42.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:40:42.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:40:42.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:40:42.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:40:42.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:40:42.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:40:42.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:40:42.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:40:42.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:42.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:40:42.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:40:42.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:40:42.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:40:42.397 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.397 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.397 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.397 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:42.397 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:42.397 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 107.61141778299999, 2024-04-22T18:40:42.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:40:42.397 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.398 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:42.398 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.398 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.398 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.398 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:40:42.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.398 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:42.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:42.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:40:42.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:42.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:40:42.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:40:42.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.400 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:40:42.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:40:42.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:42.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:40:42.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:40:42.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:42.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:40:42.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:40:42.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:42.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:40:42.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:40:42.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:42.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:40:42.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:40:42.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:40:42.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:40:42.402 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 107.611260513, 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.403 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:42.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:42.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:40:42.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:42.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:40:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:40:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:40:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:40:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:40:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:40:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:40:42.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:40:42.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:40:42.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:40:42.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:42.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:40:42.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:40:42.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:40:42.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:40:42.408 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.408 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.408 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.408 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:42.408 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:42.408 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 107.611245904, 2024-04-22T18:40:42.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:40:42.408 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.408 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:42.409 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.409 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.409 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.409 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:40:42.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.409 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:42.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:42.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:42.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:42.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:40:42.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:40:42.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:40:42.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:42.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:40:42.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:40:42.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:40:42.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:40:42.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:42.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:40:42.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:40:42.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:40:42.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:40:42.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:40:42.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:40:42.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:40:42.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:40:42.413 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.413 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.413 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.413 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:42.413 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:42.413 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 107.611235519, 2024-04-22T18:40:42.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:40:42.413 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.413 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:42.414 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.414 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.414 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.414 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:40:42.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.414 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:42.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:40:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:40:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:40:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:40:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:40:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:40:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:40:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:40:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:40:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:40:42.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:40:42.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:42.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:40:42.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:40:42.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.417 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:42.417 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:40:42.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:40:42.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:42.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:40:42.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:40:42.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:42.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:40:42.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:40:42.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:40:42.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:40:42.419 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.419 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.419 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.419 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:42.419 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:42.419 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 107.61030558900001, 2024-04-22T18:40:42.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:40:42.419 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.419 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:42.419 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.419 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.419 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.420 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:40:42.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.420 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:42.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:42.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:40:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:40:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:40:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:40:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:40:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:40:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:40:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:40:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:40:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:40:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:40:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:40:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:40:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:40:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:40:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:40:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:40:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:42.424 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 107.60984346399999, 2024-04-22T18:40:42.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:40:42.425 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.425 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:42.425 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.425 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.425 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.425 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:40:42.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.425 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:42.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:42.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:40:42.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:42.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:40:42.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:40:42.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:40:42.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:40:42.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:40:42.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:40:42.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:40:42.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:40:42.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:40:42.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:40:42.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:42.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:40:42.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:40:42.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.428 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:42.428 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:40:42.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:40:42.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:42.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:40:42.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:40:42.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:42.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:40:42.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:40:42.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:40:42.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:40:42.430 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.430 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.430 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.430 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:42.430 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:42.430 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 107.60983066, 2024-04-22T18:40:42.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:40:42.430 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.430 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:42.430 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.430 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.431 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.431 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:40:42.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.431 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:42.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:42.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:40:42.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:42.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:40:42.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:40:42.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:40:42.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:40:42.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:40:42.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:40:42.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:40:42.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:40:42.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:42.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:40:42.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:40:42.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:40:42.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:40:42.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:40:42.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:40:42.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.28205820399999, 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.435 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:40:42.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:40:42.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:40:42.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:40:42.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:40:42.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:40:42.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:42.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:40:42.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:40:42.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:42.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:40:42.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:40:42.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:42.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:40:42.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:40:42.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:42.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:40:42.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:40:42.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:40:42.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:40:42.440 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.440 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.440 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.440 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:40:42.440 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:42.440 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 103.273485092, 2024-04-22T18:40:42.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:40:42.441 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.441 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:42.441 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.441 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.441 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.441 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:40:42.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.441 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:42.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:42.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:40:42.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:42.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:40:42.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:40:42.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:40:42.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:40:42.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:40:42.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:40:42.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:40:42.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:40:42.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:42.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:40:42.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:40:42.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:42.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:40:42.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:40:42.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:40:42.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:40:42.446 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.446 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.446 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.446 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:40:42.446 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:42.446 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.238837576999998, 2024-04-22T18:40:42.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:40:42.446 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.446 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:42.446 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:40:42.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:42.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:40:42.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:40:42.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:40:42.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:40:42.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:40:42.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:40:42.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:40:42.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:40:42.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:40:42.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:40:42.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:40:42.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:40:42.450 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.450 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.450 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.450 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1684 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:42.450 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:42.450 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 71.769353613999996, 2024-04-22T18:40:42.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987885199999998, 2024-04-22T18:40:42.450 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.450 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:42.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.451 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1684 2024-04-22T18:40:42.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.451 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:42.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.451 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.451 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:42.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605278+0000", 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605280+0000", 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1770000000000001e-06 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605285+0000", 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-04-22T18:40:42.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605289+0000", 2024-04-22T18:40:42.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.687e-06 2024-04-22T18:40:42.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705352+0000", 2024-04-22T18:40:42.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006322199999999 2024-04-22T18:40:42.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:42.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705423+0000", 2024-04-22T18:40:42.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1136999999999993e-05 2024-04-22T18:40:42.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105017+0000", 2024-04-22T18:40:42.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.399593268 2024-04-22T18:40:42.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105081+0000", 2024-04-22T18:40:42.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4703999999999996e-05 2024-04-22T18:40:42.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105157+0000", 2024-04-22T18:40:42.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6205999999999995e-05 2024-04-22T18:40:42.455 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.455 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.455 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.455 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2191 4.1 4.fdaa93c1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-04-22T18:40:42.455 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:42.455 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.824761363999997, 2024-04-22T18:40:42.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51064384399999996, 2024-04-22T18:40:42.455 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.455 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:42.455 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.455 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.456 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.456 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2191 2024-04-22T18:40:42.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.456 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:42.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:42.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:42.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:42.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549872+0000", 2024-04-22T18:40:42.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.387e-06 2024-04-22T18:40:42.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549875+0000", 2024-04-22T18:40:42.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:42.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549880+0000", 2024-04-22T18:40:42.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2529999999999998e-06 2024-04-22T18:40:42.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060496+0000", 2024-04-22T18:40:42.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51061597999999997 2024-04-22T18:40:42.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060515+0000", 2024-04-22T18:40:42.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9130000000000001e-05 2024-04-22T18:40:42.459 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.459 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.459 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.459 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2198 4.d 4:bc165c76:::smithi08937929-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:42.459 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:42.459 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.823493663000001, 2024-04-22T18:40:42.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50502289899999997, 2024-04-22T18:40:42.459 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.459 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:42.460 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.460 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.460 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.460 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2198 2024-04-22T18:40:42.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.460 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:42.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:42.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551138+0000", 2024-04-22T18:40:42.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:42.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551143+0000", 2024-04-22T18:40:42.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8949999999999997e-06 2024-04-22T18:40:42.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551151+0000", 2024-04-22T18:40:42.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8729999999999996e-06 2024-04-22T18:40:42.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551158+0000", 2024-04-22T18:40:42.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3099999999999997e-06 2024-04-22T18:40:42.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551180+0000", 2024-04-22T18:40:42.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2197000000000001e-05 2024-04-22T18:40:42.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:42.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551185+0000", 2024-04-22T18:40:42.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7010000000000004e-06 2024-04-22T18:40:42.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657133+0000", 2024-04-22T18:40:42.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105948498 2024-04-22T18:40:42.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:42.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657183+0000", 2024-04-22T18:40:42.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0124000000000003e-05 2024-04-22T18:40:42.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.055924+0000", 2024-04-22T18:40:42.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39874079299999998 2024-04-22T18:40:42.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:42.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056018+0000", 2024-04-22T18:40:42.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4032999999999999e-05 2024-04-22T18:40:42.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056161+0000", 2024-04-22T18:40:42.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143385 2024-04-22T18:40:42.465 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.465 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.465 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.465 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2212 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:42.465 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:42.465 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.823424021000001, 2024-04-22T18:40:42.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50989936400000002, 2024-04-22T18:40:42.465 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.465 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:42.465 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.465 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.465 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.465 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2212 2024-04-22T18:40:42.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.466 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:42.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:42.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:42.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:42.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551211+0000", 2024-04-22T18:40:42.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6280000000000001e-06 2024-04-22T18:40:42.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551214+0000", 2024-04-22T18:40:42.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1209999999999998e-06 2024-04-22T18:40:42.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551217+0000", 2024-04-22T18:40:42.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8490000000000002e-06 2024-04-22T18:40:42.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061086+0000", 2024-04-22T18:40:42.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50986975099999998 2024-04-22T18:40:42.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061108+0000", 2024-04-22T18:40:42.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1319e-05 2024-04-22T18:40:42.469 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.469 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.469 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.469 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2213 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:42.469 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:42.469 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.823413604000002, 2024-04-22T18:40:42.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50998486099999996, 2024-04-22T18:40:42.469 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.469 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:42.469 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.469 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.469 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.470 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2213 2024-04-22T18:40:42.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.470 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:42.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:42.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551218+0000", 2024-04-22T18:40:42.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:42.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551220+0000", 2024-04-22T18:40:42.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.319e-06 2024-04-22T18:40:42.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551223+0000", 2024-04-22T18:40:42.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9660000000000001e-06 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551226+0000", 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9050000000000001e-06 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061162+0000", 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.509936838 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061204+0000", 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1094999999999999e-05 2024-04-22T18:40:42.472 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.473 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.473 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.473 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2214 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:42.473 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:42.473 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.823404615000001, 2024-04-22T18:40:42.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50997458100000004, 2024-04-22T18:40:42.473 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:42.473 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:42.473 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:42.473 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:42.473 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:42.473 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2214 2024-04-22T18:40:42.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.474 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:42.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:42.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:42.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:42.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:42.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:42.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:42.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551227+0000", 2024-04-22T18:40:42.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:42.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:42.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551229+0000", 2024-04-22T18:40:42.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8649999999999999e-06 2024-04-22T18:40:42.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:42.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551232+0000", 2024-04-22T18:40:42.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:42.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:42.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551235+0000", 2024-04-22T18:40:42.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:40:42.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:42.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061194+0000", 2024-04-22T18:40:42.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50995819099999995 2024-04-22T18:40:42.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:42.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:42.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:42.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061202+0000", 2024-04-22T18:40:42.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.541e-06 2024-04-22T18:40:42.479 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.479 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.479 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.479 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:42.479 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:42.479 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:42.480 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:42.480 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:42.481 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:42.481 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:42.482 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:42.482 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:42.423+0000 7fa766d7d640 -1 osd.5 109 handle_osd_map injecting map crc failure 2024-04-22T18:40:42.482 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:42.481+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:42.483 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:42.483 INFO:tasks.rados.rados.0.smithi089.stdout:1575: got (22) Invalid argument 2024-04-22T18:40:42.483 INFO:tasks.rados.rados.0.smithi089.stdout:1575: done (15 left) 2024-04-22T18:40:42.483 INFO:tasks.rados.rados.0.smithi089.stdout:1586: read oid 35 snap -1 2024-04-22T18:40:42.483 INFO:tasks.rados.rados.0.smithi089.stdout:1586: expect (ObjNum 304 snap 0 seq_num 304) 2024-04-22T18:40:42.483 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:42.483 INFO:tasks.rados.rados.0.smithi089.stdout:1579: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:40:42.484 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:42.484 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:42.484 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:42.484 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:42.580 INFO:tasks.rados.rados.0.smithi089.stdout:1584: expect (ObjNum 201 snap 0 seq_num 201) 2024-04-22T18:40:42.580 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:42.581+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:42.623 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:42.624 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:42.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:42.644 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:42.645+0000 7f309102d640 -1 osd.1 110 handle_osd_map injecting map crc failure 2024-04-22T18:40:42.645 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:42.645+0000 7fec57bee640 -1 osd.7 110 handle_osd_map injecting map crc failure 2024-04-22T18:40:42.688 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:42.689+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:42.745 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:42.745 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:42.755 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:42.767 INFO:tasks.rados.rados.0.smithi089.stdout:1577: expect (ObjNum 301 snap 0 seq_num 301) 2024-04-22T18:40:42.782 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:42.784+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:42.840 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:40:42.883 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:42.885+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:42.897 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:42.897 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:42.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:42.912 INFO:tasks.rados.rados.0.smithi089.stdout:1586: expect (ObjNum 304 snap 0 seq_num 304) 2024-04-22T18:40:42.984 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:42.985+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:43.009 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:43.009 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:43.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:43.019 INFO:tasks.rados.rados.0.smithi089.stdout:1582: expect (ObjNum 189 snap 0 seq_num 189) 2024-04-22T18:40:43.085 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:43.086+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:43.120 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:43.120 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:43.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:43.213 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:43.214+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:43.240 INFO:tasks.rados.rados.0.smithi089.stdout:1580: expect (ObjNum 278 snap 0 seq_num 278) 2024-04-22T18:40:43.270 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:43.270 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:43.279 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:43.287 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:43.288+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:43.326 INFO:tasks.rados.rados.0.smithi089.stdout:1573: finishing write tid 2 to smithi08937929-41 2024-04-22T18:40:43.326 INFO:tasks.rados.rados.0.smithi089.stdout:1573: finishing write tid 3 to smithi08937929-41 2024-04-22T18:40:43.326 INFO:tasks.rados.rados.0.smithi089.stdout:1573: finishing write tid 4 to smithi08937929-41 2024-04-22T18:40:43.326 INFO:tasks.rados.rados.0.smithi089.stdout:1573: finishing write tid 5 to smithi08937929-41 2024-04-22T18:40:43.326 INFO:tasks.rados.rados.0.smithi089.stdout:1573: oid 41 updating version 0 to 126 2024-04-22T18:40:43.326 INFO:tasks.rados.rados.0.smithi089.stdout:1573: oid 41 version 126 is already newer than 124 2024-04-22T18:40:43.326 INFO:tasks.rados.rados.0.smithi089.stdout:1573: oid 41 version 126 is already newer than 125 2024-04-22T18:40:43.326 INFO:tasks.rados.rados.0.smithi089.stdout:1573: oid 41 updating version 126 to 127 2024-04-22T18:40:43.326 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 127 (ObjNum 313 snap 0 seq_num 313) dirty exists 2024-04-22T18:40:43.326 INFO:tasks.rados.rados.0.smithi089.stdout:1573: left oid 41 (ObjNum 313 snap 0 seq_num 313) 2024-04-22T18:40:43.326 INFO:tasks.rados.rados.0.smithi089.stdout:1583: expect (ObjNum 279 snap 0 seq_num 279) 2024-04-22T18:40:43.387 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:43.389+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:43.422 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:43.423 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:43.431 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:43.487 INFO:tasks.rados.rados.0.smithi089.stdout:1576: got (0) Success 2024-04-22T18:40:43.487 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 0 (ObjNum 283 snap 0 seq_num 283) dirty exists 2024-04-22T18:40:43.487 INFO:tasks.rados.rados.0.smithi089.stdout:1581: expect (ObjNum 310 snap 0 seq_num 310) 2024-04-22T18:40:43.488 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:43.489+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:43.533 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:43.533 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:43.542 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:43.626 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:43.627+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:43.645 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:40:43.652 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:43.653+0000 7fec55bea640 -1 osd.7 111 handle_osd_map injecting map crc failure 2024-04-22T18:40:43.656 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:43.657+0000 7f3cfee88640 -1 osd.6 111 handle_osd_map injecting map crc failure 2024-04-22T18:40:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:43.689 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:43.690+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:43.694 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:43.703 INFO:tasks.rados.rados.0.smithi089.stdout:1567: expect (ObjNum 208 snap 0 seq_num 208) 2024-04-22T18:40:43.790 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:43.791+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:43.805 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:43.805 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:43.814 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:43.891 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:43.893+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:43.900 INFO:tasks.rados.rados.0.smithi089.stdout:1569: expect (ObjNum 277 snap 0 seq_num 277) 2024-04-22T18:40:43.919 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:43.919 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:43.928 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:44.011 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:44.013+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:44.069 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:44.078 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:44.093 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:44.094+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:44.094 INFO:tasks.rados.rados.0.smithi089.stdout:1574: expect (ObjNum 269 snap 0 seq_num 269) 2024-04-22T18:40:44.194 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:44.195+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:44.196 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:44.196 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:44.205 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:44.265 INFO:tasks.rados.rados.0.smithi089.stdout:1585: expect (ObjNum 296 snap 0 seq_num 296) 2024-04-22T18:40:44.294 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:44.296+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:44.307 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:44.307 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:44.316 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:44.395 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:44.396+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:44.408 INFO:tasks.rados.rados.0.smithi089.stdout:1570: expect (ObjNum 287 snap 0 seq_num 287) 2024-04-22T18:40:44.460 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:44.460 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:44.470 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:44.494 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:40:44.495 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:44.496+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:44.575 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:44.576 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:44.580 INFO:tasks.rados.rados.0.smithi089.stdout:1567: done (15 left) 2024-04-22T18:40:44.580 INFO:tasks.rados.rados.0.smithi089.stdout:1569: done (14 left) 2024-04-22T18:40:44.580 INFO:tasks.rados.rados.0.smithi089.stdout:1570: done (13 left) 2024-04-22T18:40:44.580 INFO:tasks.rados.rados.0.smithi089.stdout:1573: done (12 left) 2024-04-22T18:40:44.580 INFO:tasks.rados.rados.0.smithi089.stdout:1574: done (11 left) 2024-04-22T18:40:44.581 INFO:tasks.rados.rados.0.smithi089.stdout:1576: done (10 left) 2024-04-22T18:40:44.581 INFO:tasks.rados.rados.0.smithi089.stdout:1577: done (9 left) 2024-04-22T18:40:44.581 INFO:tasks.rados.rados.0.smithi089.stdout:1579: done (8 left) 2024-04-22T18:40:44.581 INFO:tasks.rados.rados.0.smithi089.stdout:1580: done (7 left) 2024-04-22T18:40:44.581 INFO:tasks.rados.rados.0.smithi089.stdout:1581: done (6 left) 2024-04-22T18:40:44.581 INFO:tasks.rados.rados.0.smithi089.stdout:1582: done (5 left) 2024-04-22T18:40:44.581 INFO:tasks.rados.rados.0.smithi089.stdout:1583: done (4 left) 2024-04-22T18:40:44.581 INFO:tasks.rados.rados.0.smithi089.stdout:1584: done (3 left) 2024-04-22T18:40:44.581 INFO:tasks.rados.rados.0.smithi089.stdout:1585: done (2 left) 2024-04-22T18:40:44.581 INFO:tasks.rados.rados.0.smithi089.stdout:1586: done (1 left) 2024-04-22T18:40:44.581 INFO:tasks.rados.rados.0.smithi089.stdout:1587: set_chunk oid 38 target oid 2024-04-22T18:40:44.586 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:44.630 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:44.631+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:44.634 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:44.634 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:44.654 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:44.670 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:44.670 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:44.684 INFO:tasks.rados.rados.0.smithi089.stdout:1587: set_chunk oid 38 offset: 2790400 length: 678912 target oid 6ee4f85166c958bbc48cdbabd3e431d17179c988792b63645b4892d4a30dfdda offset: 0 snap -1 2024-04-22T18:40:44.684 INFO:tasks.rados.rados.0.smithi089.stdout:1588: tier_promote oid 46 2024-04-22T18:40:44.684 INFO:tasks.rados.rados.0.smithi089.stdout:1589: set_chunk oid 19 target oid 2024-04-22T18:40:44.686 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:44.686 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:44.692 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:44.693 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:44.693 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:44.697 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:44.698+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:44.702 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:44.703 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:44.704 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:44.721 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:44.721 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:44.738 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:44.738 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:44.755 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:44.756 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:44.798 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:44.799+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:44.814 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:44.815 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:44.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:44.815 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:44.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.815 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:73 3.0 e34/19 3:061a7a63:::392f44ec26aa22e85be5f08273a7e45656f3bf4a4854038391a88e880ec01077:head v 34'31, mlcod=31'29)", 2024-04-22T18:40:44.815 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:44.815 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 100.357194087, 2024-04-22T18:40:44.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103503311, 2024-04-22T18:40:44.815 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.815 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.815 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:44.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:44.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.457927+0000", 2024-04-22T18:40:44.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:44.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458193+0000", 2024-04-22T18:40:44.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026628900000000001 2024-04-22T18:40:44.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458208+0000", 2024-04-22T18:40:44.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4780000000000001e-05 2024-04-22T18:40:44.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458217+0000", 2024-04-22T18:40:44.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4909999999999994e-06 2024-04-22T18:40:44.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459162+0000", 2024-04-22T18:40:44.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094440500000000005 2024-04-22T18:40:44.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459185+0000", 2024-04-22T18:40:44.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3685000000000001e-05 2024-04-22T18:40:44.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561394+0000", 2024-04-22T18:40:44.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10220874100000001 2024-04-22T18:40:44.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561430+0000", 2024-04-22T18:40:44.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6304999999999999e-05 2024-04-22T18:40:44.819 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.819 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.819 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.819 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:75 3.0 e34/19 3:18418e0f:::b7c35f165c374d5e075dde9ac2607e47ace3b0a1e2343ec6efb171380f725eea:head v 34'32, mlcod=31'29)", 2024-04-22T18:40:44.819 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:44.820 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 100.356180814, 2024-04-22T18:40:44.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104798078, 2024-04-22T18:40:44.820 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.820 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.820 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:44.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:44.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458939+0000", 2024-04-22T18:40:44.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:44.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459092+0000", 2024-04-22T18:40:44.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153838 2024-04-22T18:40:44.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459107+0000", 2024-04-22T18:40:44.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4703e-05 2024-04-22T18:40:44.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459115+0000", 2024-04-22T18:40:44.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-04-22T18:40:44.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560093+0000", 2024-04-22T18:40:44.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10097840499999999 2024-04-22T18:40:44.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560109+0000", 2024-04-22T18:40:44.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5631e-05 2024-04-22T18:40:44.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563706+0000", 2024-04-22T18:40:44.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035972059999999999 2024-04-22T18:40:44.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563738+0000", 2024-04-22T18:40:44.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2184000000000002e-05 2024-04-22T18:40:44.823 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.823 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.824 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.824 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:77 3.0 e34/19 3:28387c9f:::b6d7840f02d561da04e398fbf94d91b85e9066b426e48baf3b11dd4a4f6096c0:head v 34'33, mlcod=31'29)", 2024-04-22T18:40:44.824 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:44.824 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 100.355230734, 2024-04-22T18:40:44.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104186079, 2024-04-22T18:40:44.824 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.824 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.824 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:44.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:44.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459889+0000", 2024-04-22T18:40:44.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:44.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459963+0000", 2024-04-22T18:40:44.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3732999999999994e-05 2024-04-22T18:40:44.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459973+0000", 2024-04-22T18:40:44.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0285999999999999e-05 2024-04-22T18:40:44.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459987+0000", 2024-04-22T18:40:44.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3675999999999999e-05 2024-04-22T18:40:44.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560515+0000", 2024-04-22T18:40:44.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10052781299999999 2024-04-22T18:40:44.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560538+0000", 2024-04-22T18:40:44.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3861000000000001e-05 2024-04-22T18:40:44.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564044+0000", 2024-04-22T18:40:44.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035052009999999999 2024-04-22T18:40:44.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564076+0000", 2024-04-22T18:40:44.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2737000000000002e-05 2024-04-22T18:40:44.828 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.828 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.828 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.828 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:78 3.0 e34/19 3:3a5582ac:::bdd92f7deaa2d50e204891f64f9043469ba687d78b218ed518f2483ba3486a99:head v 34'34, mlcod=31'29)", 2024-04-22T18:40:44.828 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:44.828 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 100.354564616, 2024-04-22T18:40:44.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103547263, 2024-04-22T18:40:44.828 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.828 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.828 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:44.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460556+0000", 2024-04-22T18:40:44.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460555+0000", 2024-04-22T18:40:44.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:44.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460861+0000", 2024-04-22T18:40:44.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000305958 2024-04-22T18:40:44.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460874+0000", 2024-04-22T18:40:44.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2639e-05 2024-04-22T18:40:44.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.460881+0000", 2024-04-22T18:40:44.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2570000000000001e-06 2024-04-22T18:40:44.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560990+0000", 2024-04-22T18:40:44.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010883199999999 2024-04-22T18:40:44.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.561011+0000", 2024-04-22T18:40:44.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1319e-05 2024-04-22T18:40:44.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564079+0000", 2024-04-22T18:40:44.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00306815 2024-04-22T18:40:44.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564104+0000", 2024-04-22T18:40:44.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4172000000000002e-05 2024-04-22T18:40:44.832 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.832 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.832 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.832 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:40:44.832 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:44.832 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 100.35394932200001, 2024-04-22T18:40:44.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:40:44.833 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.833 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.833 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:44.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:44.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:40:44.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:44.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:40:44.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:40:44.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:40:44.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:40:44.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:40:44.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:40:44.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:40:44.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:40:44.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:40:44.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:40:44.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:40:44.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:40:44.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:40:44.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:40:44.836 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.836 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.836 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.837 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:40:44.837 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:44.837 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.477275500000005, 2024-04-22T18:40:44.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:40:44.837 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.837 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.837 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:44.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:44.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:40:44.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:44.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:40:44.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:40:44.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:40:44.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:40:44.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:40:44.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:40:44.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:40:44.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:40:44.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:40:44.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:40:44.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:40:44.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:40:44.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:40:44.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:40:44.840 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.841 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.841 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.841 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:40:44.841 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:44.841 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.272471977000002, 2024-04-22T18:40:44.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:40:44.841 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.841 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.841 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:44.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:44.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:40:44.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:40:44.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:40:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:40:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:40:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:40:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:40:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:40:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:40:44.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:40:44.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:40:44.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:40:44.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:40:44.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:40:44.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:40:44.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:40:44.845 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.845 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.845 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.845 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:161 3.0 e76/63 3:3638311b:::f282598a146af3da1c05c8882972fba387afd249b39f6eeb6e6052a5de37fd2d:head v 76'157, mlcod=76'154)", 2024-04-22T18:40:44.845 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:44.845 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.423752546999999, 2024-04-22T18:40:44.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104845172, 2024-04-22T18:40:44.845 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.845 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.845 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:44.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:44.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:44.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:44.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391440+0000", 2024-04-22T18:40:44.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2045999999999994e-05 2024-04-22T18:40:44.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391448+0000", 2024-04-22T18:40:44.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8189999999999992e-06 2024-04-22T18:40:44.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391453+0000", 2024-04-22T18:40:44.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8379999999999998e-06 2024-04-22T18:40:44.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391839+0000", 2024-04-22T18:40:44.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000385325 2024-04-22T18:40:44.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391854+0000", 2024-04-22T18:40:44.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5006e-05 2024-04-22T18:40:44.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496191+0000", 2024-04-22T18:40:44.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104336792 2024-04-22T18:40:44.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496214+0000", 2024-04-22T18:40:44.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3079999999999999e-05 2024-04-22T18:40:44.849 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.849 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.849 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.849 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:165 3.0 e76/63 3:1e2f7d15:::93d9bab0d1f539405a95a28c02bcf2dcbdfa6f3e449e5e5e3f27f774df3c508a:head v 76'158, mlcod=76'154)", 2024-04-22T18:40:44.849 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:44.849 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.423658363999998, 2024-04-22T18:40:44.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104788752, 2024-04-22T18:40:44.850 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.850 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.850 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:44.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:44.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391462+0000", 2024-04-22T18:40:44.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:44.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391522+0000", 2024-04-22T18:40:44.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0148000000000001e-05 2024-04-22T18:40:44.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391531+0000", 2024-04-22T18:40:44.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5709999999999998e-06 2024-04-22T18:40:44.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391536+0000", 2024-04-22T18:40:44.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6180000000000003e-06 2024-04-22T18:40:44.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392218+0000", 2024-04-22T18:40:44.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000681326 2024-04-22T18:40:44.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392237+0000", 2024-04-22T18:40:44.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9726999999999999e-05 2024-04-22T18:40:44.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496231+0000", 2024-04-22T18:40:44.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103993672 2024-04-22T18:40:44.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496251+0000", 2024-04-22T18:40:44.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0327999999999999e-05 2024-04-22T18:40:44.853 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.853 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.853 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.853 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:166 3.0 e76/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 76'159, mlcod=76'154)", 2024-04-22T18:40:44.854 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:44.854 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.423580465999997, 2024-04-22T18:40:44.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104724628, 2024-04-22T18:40:44.854 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.854 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.854 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:44.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:44.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391540+0000", 2024-04-22T18:40:44.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:44.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391627+0000", 2024-04-22T18:40:44.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7330000000000006e-05 2024-04-22T18:40:44.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391635+0000", 2024-04-22T18:40:44.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6709999999999999e-06 2024-04-22T18:40:44.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391640+0000", 2024-04-22T18:40:44.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3199999999999999e-06 2024-04-22T18:40:44.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492541+0000", 2024-04-22T18:40:44.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100900873 2024-04-22T18:40:44.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492560+0000", 2024-04-22T18:40:44.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8502999999999999e-05 2024-04-22T18:40:44.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496254+0000", 2024-04-22T18:40:44.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036945849999999998 2024-04-22T18:40:44.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496265+0000", 2024-04-22T18:40:44.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0869999999999999e-05 2024-04-22T18:40:44.857 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.857 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.857 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.858 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:167 3.0 e76/63 3:384c1ce2:::81dc8c58ebe106ae6fa9feadc06291c645d9079ad84b78214d4e6440fbc5def8:head v 76'160, mlcod=76'154)", 2024-04-22T18:40:44.858 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:44.858 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.423465591000003, 2024-04-22T18:40:44.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10487859400000001, 2024-04-22T18:40:44.858 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.858 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.858 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:44.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:44.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:44.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:44.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391707+0000", 2024-04-22T18:40:44.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2343999999999999e-05 2024-04-22T18:40:44.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391714+0000", 2024-04-22T18:40:44.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.596e-06 2024-04-22T18:40:44.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391719+0000", 2024-04-22T18:40:44.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9130000000000002e-06 2024-04-22T18:40:44.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493154+0000", 2024-04-22T18:40:44.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101435256 2024-04-22T18:40:44.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493177+0000", 2024-04-22T18:40:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2796e-05 2024-04-22T18:40:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496509+0000", 2024-04-22T18:40:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033320139999999999 2024-04-22T18:40:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496534+0000", 2024-04-22T18:40:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5265999999999998e-05 2024-04-22T18:40:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.862 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.862 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.862 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:197 3.0 e78/63 3:19ca42e0:::2109e7e4526cbe26b2963cdd11634231ee7f4d35dabe9962c44b62fafb130ddf:head v 78'167, mlcod=78'165)", 2024-04-22T18:40:44.862 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:09.540869+0000", 2024-04-22T18:40:44.862 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.274251939000003, 2024-04-22T18:40:44.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10287555800000001, 2024-04-22T18:40:44.862 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.862 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.862 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540869+0000", 2024-04-22T18:40:44.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540869+0000", 2024-04-22T18:40:44.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540867+0000", 2024-04-22T18:40:44.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:44.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540879+0000", 2024-04-22T18:40:44.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1316e-05 2024-04-22T18:40:44.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540895+0000", 2024-04-22T18:40:44.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6072000000000001e-05 2024-04-22T18:40:44.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.540904+0000", 2024-04-22T18:40:44.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8969999999999997e-06 2024-04-22T18:40:44.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.641023+0000", 2024-04-22T18:40:44.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10011914 2024-04-22T18:40:44.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.641049+0000", 2024-04-22T18:40:44.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6455e-05 2024-04-22T18:40:44.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.643687+0000", 2024-04-22T18:40:44.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026375499999999998 2024-04-22T18:40:44.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:09.643745+0000", 2024-04-22T18:40:44.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7725e-05 2024-04-22T18:40:44.866 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.866 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.866 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.866 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:39 3.0 e90/63 3:26983a86:::2d4b6112bcf431e3da007c45a018afa7fe010a4f62df8d2cdc4bdc7924d18240:head v 90'177, mlcod=89'175)", 2024-04-22T18:40:44.866 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.526733+0000", 2024-04-22T18:40:44.866 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.288388152, 2024-04-22T18:40:44.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10253753, 2024-04-22T18:40:44.867 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.867 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.867 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526733+0000", 2024-04-22T18:40:44.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526733+0000", 2024-04-22T18:40:44.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526732+0000", 2024-04-22T18:40:44.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:44.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526806+0000", 2024-04-22T18:40:44.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3538e-05 2024-04-22T18:40:44.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526811+0000", 2024-04-22T18:40:44.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6020000000000002e-06 2024-04-22T18:40:44.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526815+0000", 2024-04-22T18:40:44.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0319999999999997e-06 2024-04-22T18:40:44.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526859+0000", 2024-04-22T18:40:44.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3934000000000003e-05 2024-04-22T18:40:44.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526881+0000", 2024-04-22T18:40:44.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2245000000000001e-05 2024-04-22T18:40:44.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629213+0000", 2024-04-22T18:40:44.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10233270699999999 2024-04-22T18:40:44.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629270+0000", 2024-04-22T18:40:44.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6910999999999998e-05 2024-04-22T18:40:44.870 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.870 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.870 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.871 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:40 3.0 e90/63 3:394d11e1:::4fe26d62e8e61f933e14e15995651b4040179bacb8e375345fa3999664858986:head v 90'178, mlcod=89'175)", 2024-04-22T18:40:44.871 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:44.871 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.288300032999999, 2024-04-22T18:40:44.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102485589, 2024-04-22T18:40:44.871 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.871 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.871 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:44.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:44.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526821+0000", 2024-04-22T18:40:44.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:44.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526928+0000", 2024-04-22T18:40:44.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107627 2024-04-22T18:40:44.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526932+0000", 2024-04-22T18:40:44.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5250000000000001e-06 2024-04-22T18:40:44.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.526935+0000", 2024-04-22T18:40:44.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9359999999999999e-06 2024-04-22T18:40:44.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527379+0000", 2024-04-22T18:40:44.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00044412 2024-04-22T18:40:44.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527400+0000", 2024-04-22T18:40:44.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0726000000000001e-05 2024-04-22T18:40:44.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629290+0000", 2024-04-22T18:40:44.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101890194 2024-04-22T18:40:44.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629307+0000", 2024-04-22T18:40:44.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6855999999999999e-05 2024-04-22T18:40:44.874 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.875 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.875 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.875 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:47 3.0 e90/63 3:0e62034f:::069f05989298bc0f714a6a846e6234d7fa060b1f5513fa9a7b2ddcab658ece07:head v 90'181, mlcod=89'175)", 2024-04-22T18:40:44.875 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:44.875 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.288040714000001, 2024-04-22T18:40:44.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10267794, 2024-04-22T18:40:44.875 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.875 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.875 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:44.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:44.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527080+0000", 2024-04-22T18:40:44.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-04-22T18:40:44.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527136+0000", 2024-04-22T18:40:44.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6178000000000003e-05 2024-04-22T18:40:44.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527141+0000", 2024-04-22T18:40:44.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4889999999999997e-06 2024-04-22T18:40:44.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.527144+0000", 2024-04-22T18:40:44.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0259999999999999e-06 2024-04-22T18:40:44.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.628693+0000", 2024-04-22T18:40:44.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101549033 2024-04-22T18:40:44.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.628713+0000", 2024-04-22T18:40:44.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9932000000000002e-05 2024-04-22T18:40:44.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629733+0000", 2024-04-22T18:40:44.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010206200000000001 2024-04-22T18:40:44.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:21.629758+0000", 2024-04-22T18:40:44.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4861000000000001e-05 2024-04-22T18:40:44.879 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.879 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.879 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.879 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:3201 4.4 e91/63 4:24717d05:::smithi08937929-42:head v 91'30, mlcod=74'27)", 2024-04-22T18:40:44.879 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.156471+0000", 2024-04-22T18:40:44.879 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.658649904000001, 2024-04-22T18:40:44.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102886424, 2024-04-22T18:40:44.880 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.880 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.880 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156471+0000", 2024-04-22T18:40:44.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156471+0000", 2024-04-22T18:40:44.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156470+0000", 2024-04-22T18:40:44.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:44.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156869+0000", 2024-04-22T18:40:44.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00039887499999999998 2024-04-22T18:40:44.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.156879+0000", 2024-04-22T18:40:44.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9399999999999997e-06 2024-04-22T18:40:44.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.257011+0000", 2024-04-22T18:40:44.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100131962 2024-04-22T18:40:44.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.257129+0000", 2024-04-22T18:40:44.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000118644 2024-04-22T18:40:44.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.257144+0000", 2024-04-22T18:40:44.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5211000000000001e-05 2024-04-22T18:40:44.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.259333+0000", 2024-04-22T18:40:44.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021890569999999999 2024-04-22T18:40:44.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.259358+0000", 2024-04-22T18:40:44.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4039000000000001e-05 2024-04-22T18:40:44.883 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.883 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.883 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.884 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.6.13:66 3.0 e95/63 3:1683cb7f:::f03bfe2533cb949a19fa6d0ef489d3c38ae0c5777f73a45f60cfab23d5b5f5a4:head v 95'186, mlcod=95'183)", 2024-04-22T18:40:44.884 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.635704+0000", 2024-04-22T18:40:44.884 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.179416830000001, 2024-04-22T18:40:44.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10283379300000001, 2024-04-22T18:40:44.884 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.884 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:44.884 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635704+0000", 2024-04-22T18:40:44.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635704+0000", 2024-04-22T18:40:44.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635703+0000", 2024-04-22T18:40:44.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:44.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635809+0000", 2024-04-22T18:40:44.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010657 2024-04-22T18:40:44.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635822+0000", 2024-04-22T18:40:44.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2500000000000001e-05 2024-04-22T18:40:44.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.635830+0000", 2024-04-22T18:40:44.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5469999999999999e-06 2024-04-22T18:40:44.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.735454+0000", 2024-04-22T18:40:44.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099624303999999997 2024-04-22T18:40:44.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.735473+0000", 2024-04-22T18:40:44.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8709999999999999e-05 2024-04-22T18:40:44.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:44.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.738511+0000", 2024-04-22T18:40:44.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030386689999999999 2024-04-22T18:40:44.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.738538+0000", 2024-04-22T18:40:44.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6767999999999999e-05 2024-04-22T18:40:44.888 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.888 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.888 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.888 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3449 4.d 4:bee1184f:::smithi08937929-7:head [read 0~3613505,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:44.888 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927107+0000", 2024-04-22T18:40:44.888 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.888013654, 2024-04-22T18:40:44.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103315584, 2024-04-22T18:40:44.889 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.889 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:44.889 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:44.889 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:44.889 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:44.889 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3449 2024-04-22T18:40:44.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.889 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927107+0000", 2024-04-22T18:40:44.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927107+0000", 2024-04-22T18:40:44.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927105+0000", 2024-04-22T18:40:44.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:44.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927116+0000", 2024-04-22T18:40:44.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1436e-05 2024-04-22T18:40:44.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927131+0000", 2024-04-22T18:40:44.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5172e-05 2024-04-22T18:40:44.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927139+0000", 2024-04-22T18:40:44.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3540000000000002e-06 2024-04-22T18:40:44.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927192+0000", 2024-04-22T18:40:44.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3349e-05 2024-04-22T18:40:44.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:40:44.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927200+0000", 2024-04-22T18:40:44.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:40:44.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027549+0000", 2024-04-22T18:40:44.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100349691 2024-04-22T18:40:44.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027602+0000", 2024-04-22T18:40:44.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2986999999999999e-05 2024-04-22T18:40:44.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.030423+0000", 2024-04-22T18:40:44.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028205130000000002 2024-04-22T18:40:44.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.893 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.893 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3450 4.d 4:bee1184f:::smithi08937929-7:head [sparse-read 0~2105792] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:44.893 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:44.893 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.887972464000001, 2024-04-22T18:40:44.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103826771, 2024-04-22T18:40:44.893 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.894 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:44.894 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:44.894 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:44.894 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:44.894 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3450 2024-04-22T18:40:44.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.894 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:44.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:44.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927148+0000", 2024-04-22T18:40:44.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:44.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927151+0000", 2024-04-22T18:40:44.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7e-06 2024-04-22T18:40:44.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927156+0000", 2024-04-22T18:40:44.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6699999999999999e-06 2024-04-22T18:40:44.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927161+0000", 2024-04-22T18:40:44.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.933e-06 2024-04-22T18:40:44.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927255+0000", 2024-04-22T18:40:44.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4195000000000002e-05 2024-04-22T18:40:44.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927262+0000", 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9480000000000002e-06 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.030462+0000", 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103199402 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.030553+0000", 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1080000000000002e-05 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.030975+0000", 2024-04-22T18:40:44.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000422426 2024-04-22T18:40:44.898 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.898 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.898 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.898 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3451 4.d 4:bee1184f:::smithi08937929-7:head [read 0~3613505,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:44.898 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:44.898 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.88794165, 2024-04-22T18:40:44.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20164568099999999, 2024-04-22T18:40:44.898 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:44.898 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:44.899 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:44.899 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:44.899 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:44.899 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3451 2024-04-22T18:40:44.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.899 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:44.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:44.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:44.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:44.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:44.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:44.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:44.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:44.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:44.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:44.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927183+0000", 2024-04-22T18:40:44.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1110000000000001e-06 2024-04-22T18:40:44.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:44.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927188+0000", 2024-04-22T18:40:44.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1569999999999999e-06 2024-04-22T18:40:44.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:44.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927192+0000", 2024-04-22T18:40:44.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4710000000000001e-06 2024-04-22T18:40:44.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027382+0000", 2024-04-22T18:40:44.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100189341 2024-04-22T18:40:44.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:44.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027387+0000", 2024-04-22T18:40:44.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8400000000000002e-06 2024-04-22T18:40:44.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:44.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127638+0000", 2024-04-22T18:40:44.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10025102499999999 2024-04-22T18:40:44.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:44.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127740+0000", 2024-04-22T18:40:44.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102749 2024-04-22T18:40:44.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:44.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:44.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:44.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.128825+0000", 2024-04-22T18:40:44.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001084593 2024-04-22T18:40:44.903 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.903 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.903 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.903 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:44.903 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:44.903 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:44.903 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 2] out_osds: [5, 4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:40:44.903 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:40:44.903 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-04-22T18:40:44.904 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-22T18:40:44.904 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-22T18:40:44.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-04-22T18:40:44.905 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:44.906 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:44.853+0000 7f8605f32640 -1 osd.3 112 handle_osd_map injecting map crc failure 2024-04-22T18:40:44.907 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:44.906+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:44.957 INFO:tasks.rados.rados.0.smithi089.stdout:1589: set_chunk oid 19 offset: 1056256 length: 693248 target oid d9421dbc8d0e6569f7a9824a2b16ea315066d61027451fefb676355d5be86288 offset: 0 snap -1 2024-04-22T18:40:44.957 INFO:tasks.rados.rados.0.smithi089.stdout:1590: read oid 18 snap -1 2024-04-22T18:40:44.957 INFO:tasks.rados.rados.0.smithi089.stdout:1590: expect (ObjNum 307 snap 0 seq_num 307) 2024-04-22T18:40:44.957 INFO:tasks.rados.rados.0.smithi089.stdout:1591: write oid 21 current snap is 0 2024-04-22T18:40:44.957 INFO:tasks.rados.rados.0.smithi089.stdout:1591: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2024-04-22T18:40:44.964 INFO:tasks.rados.rados.0.smithi089.stdout:1591: writing smithi08937929-21 from 477398 to 1276057 tid 1 2024-04-22T18:40:44.970 INFO:tasks.rados.rados.0.smithi089.stdout:1591: writing smithi08937929-21 from 1774640 to 2438084 tid 2 2024-04-22T18:40:44.975 INFO:tasks.rados.rados.0.smithi089.stdout:1591: writing smithi08937929-21 from 2873055 to 3277398 tid 3 2024-04-22T18:40:44.975 INFO:tasks.rados.rados.0.smithi089.stdout:1592: write (excl) oid 16 current snap is 0 2024-04-22T18:40:44.975 INFO:tasks.rados.rados.0.smithi089.stdout:1592: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2024-04-22T18:40:44.980 INFO:tasks.rados.rados.0.smithi089.stdout:1592: writing smithi08937929-16 from 494205 to 968113 tid 1 2024-04-22T18:40:44.985 INFO:tasks.rados.rados.0.smithi089.stdout:1592: writing smithi08937929-16 from 1433122 to 1840224 tid 2 2024-04-22T18:40:44.991 INFO:tasks.rados.rados.0.smithi089.stdout:1592: writing smithi08937929-16 from 2384125 to 2846809 tid 3 2024-04-22T18:40:44.991 INFO:tasks.rados.rados.0.smithi089.stdout:1593: write (excl) oid 49 current snap is 0 2024-04-22T18:40:44.991 INFO:tasks.rados.rados.0.smithi089.stdout:1593: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2024-04-22T18:40:44.996 INFO:tasks.rados.rados.0.smithi089.stdout:1593: writing smithi08937929-49 from 511012 to 1060169 tid 1 2024-04-22T18:40:45.003 INFO:tasks.rados.rados.0.smithi089.stdout:1593: writing smithi08937929-49 from 1775251 to 2326011 tid 2 2024-04-22T18:40:45.006 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:45.007+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:45.007 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:45.008 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-22T18:40:45.008 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:45.009 INFO:tasks.rados.rados.0.smithi089.stdout:1593: writing smithi08937929-49 from 3095195 to 3311012 tid 3 2024-04-22T18:40:45.009 INFO:tasks.rados.rados.0.smithi089.stdout:1594: write oid 20 current snap is 0 2024-04-22T18:40:45.009 INFO:tasks.rados.rados.0.smithi089.stdout:1594: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2024-04-22T18:40:45.013 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:45.013 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:45.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.013 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPushReply(3.1 112/103 [PushReplyOp(3:ba271266:::caa2fee493ffa27d6b1c17aa88694e5c9f90816bfa77695547fbd2bab978c2d4:head)])", 2024-04-22T18:40:45.013 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:44.883202+0000", 2024-04-22T18:40:45.013 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.13149525300000001, 2024-04-22T18:40:45.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.13151610999999999, 2024-04-22T18:40:45.013 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.013 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:45.013 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:44.883202+0000", 2024-04-22T18:40:45.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:44.883202+0000", 2024-04-22T18:40:45.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:44.883201+0000", 2024-04-22T18:40:45.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:45.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:44.883206+0000", 2024-04-22T18:40:45.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9749999999999998e-06 2024-04-22T18:40:45.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:44.883216+0000", 2024-04-22T18:40:45.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8379999999999997e-06 2024-04-22T18:40:45.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:44.883224+0000", 2024-04-22T18:40:45.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0399999999999993e-06 2024-04-22T18:40:45.015 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.016 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.016 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.016 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:3726 4.b e113/103)", 2024-04-22T18:40:45.016 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:44.987988+0000", 2024-04-22T18:40:45.016 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.026708852000000002, 2024-04-22T18:40:45.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026758595999999999, 2024-04-22T18:40:45.016 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.016 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "no flag points reached", 2024-04-22T18:40:45.016 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:44.987988+0000", 2024-04-22T18:40:45.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:44.987988+0000", 2024-04-22T18:40:45.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:44.987987+0000", 2024-04-22T18:40:45.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:45.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:44.988077+0000", 2024-04-22T18:40:45.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0394000000000002e-05 2024-04-22T18:40:45.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:44.988088+0000", 2024-04-22T18:40:45.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0117e-05 2024-04-22T18:40:45.018 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.018 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.018 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.018 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.018 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:45.018 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 2 2024-04-22T18:40:45.018 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:45.019 INFO:tasks.rados.rados.0.smithi089.stdout:1594: writing smithi08937929-20 from 527819 to 1152225 tid 1 2024-04-22T18:40:45.021 INFO:tasks.rados.rados.0.smithi089.stdout:1594: writing smithi08937929-20 from 1833733 to 2244504 tid 2 2024-04-22T18:40:45.022 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:45.027 INFO:tasks.rados.rados.0.smithi089.stdout:1594: writing smithi08937929-20 from 2722618 to 3324199 tid 3 2024-04-22T18:40:45.027 INFO:tasks.rados.rados.0.smithi089.stdout:1595: read oid 47 snap -1 2024-04-22T18:40:45.027 INFO:tasks.rados.rados.0.smithi089.stdout:1595: expect (ObjNum 298 snap 0 seq_num 298) 2024-04-22T18:40:45.027 INFO:tasks.rados.rados.0.smithi089.stdout:1596: read oid 32 snap -1 2024-04-22T18:40:45.027 INFO:tasks.rados.rados.0.smithi089.stdout:1596: expect (ObjNum 296 snap 0 seq_num 296) 2024-04-22T18:40:45.027 INFO:tasks.rados.rados.0.smithi089.stdout:1597: write oid 45 current snap is 0 2024-04-22T18:40:45.028 INFO:tasks.rados.rados.0.smithi089.stdout:1578: expect (ObjNum 305 snap 0 seq_num 305) 2024-04-22T18:40:45.107 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:45.108+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:45.130 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:45.130 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:45.130 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:45.130 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:45.130 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:45.137 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:45.220 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:45.222+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:45.243 INFO:tasks.rados.rados.0.smithi089.stdout:1589: finishing set_chunk to oid 19 2024-04-22T18:40:45.243 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 19 set_chunk returned error code -95 offset: 1056256 length: 693248 2024-04-22T18:40:45.243 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 0 (ObjNum 273 snap 0 seq_num 273) dirty exists 2024-04-22T18:40:45.243 INFO:tasks.rados.rados.0.smithi089.stdout:1592: finishing write tid 1 to smithi08937929-16 2024-04-22T18:40:45.244 INFO:tasks.rados.rados.0.smithi089.stdout:1597: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-04-22T18:40:45.250 INFO:tasks.rados.rados.0.smithi089.stdout:1597: writing smithi08937929-45 from 544626 to 1244281 tid 1 2024-04-22T18:40:45.256 INFO:tasks.rados.rados.0.smithi089.stdout:1597: writing smithi08937929-45 from 1892215 to 2446644 tid 2 2024-04-22T18:40:45.261 INFO:tasks.rados.rados.0.smithi089.stdout:1597: writing smithi08937929-45 from 3150041 to 3344626 tid 3 2024-04-22T18:40:45.261 INFO:tasks.rados.rados.0.smithi089.stdout:1578: done (11 left) 2024-04-22T18:40:45.261 INFO:tasks.rados.rados.0.smithi089.stdout:1589: done (10 left) 2024-04-22T18:40:45.262 INFO:tasks.rados.rados.0.smithi089.stdout:1598: set_chunk oid 4 target oid 2024-04-22T18:40:45.284 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:45.284 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:45.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:45.284 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:45.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.284 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:78 4.0 4:01baf5a9:::smithi08937929-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:45.284 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:45.284 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 110.771882702, 2024-04-22T18:40:45.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.398522192, 2024-04-22T18:40:45.284 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.284 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.285 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:45.285 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:45.285 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:45.285 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 78 2024-04-22T18:40:45.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.285 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:45.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511935+0000", 2024-04-22T18:40:45.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.511934+0000", 2024-04-22T18:40:45.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:45.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512012+0000", 2024-04-22T18:40:45.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8329000000000002e-05 2024-04-22T18:40:45.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512023+0000", 2024-04-22T18:40:45.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0755e-05 2024-04-22T18:40:45.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512028+0000", 2024-04-22T18:40:45.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6470000000000003e-06 2024-04-22T18:40:45.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512068+0000", 2024-04-22T18:40:45.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9721999999999997e-05 2024-04-22T18:40:45.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512125+0000", 2024-04-22T18:40:45.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6564999999999998e-05 2024-04-22T18:40:45.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:45.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512200+0000", 2024-04-22T18:40:45.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5708000000000001e-05 2024-04-22T18:40:45.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513131+0000", 2024-04-22T18:40:45.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093063 2024-04-22T18:40:45.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910394+0000", 2024-04-22T18:40:45.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.397263388 2024-04-22T18:40:45.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910430+0000", 2024-04-22T18:40:45.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5485999999999999e-05 2024-04-22T18:40:45.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910457+0000", 2024-04-22T18:40:45.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7086000000000001e-05 2024-04-22T18:40:45.289 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.290 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.290 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.290 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:79 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:45.290 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:45.290 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 110.771782046, 2024-04-22T18:40:45.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39855419399999997, 2024-04-22T18:40:45.290 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.290 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.290 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:45.290 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:45.290 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 79 2024-04-22T18:40:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512036+0000", 2024-04-22T18:40:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512035+0000", 2024-04-22T18:40:45.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:45.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512045+0000", 2024-04-22T18:40:45.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0379e-05 2024-04-22T18:40:45.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512051+0000", 2024-04-22T18:40:45.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6030000000000004e-06 2024-04-22T18:40:45.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512057+0000", 2024-04-22T18:40:45.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7270000000000004e-06 2024-04-22T18:40:45.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512756+0000", 2024-04-22T18:40:45.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069954000000000001 2024-04-22T18:40:45.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512806+0000", 2024-04-22T18:40:45.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9922000000000001e-05 2024-04-22T18:40:45.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:45.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512927+0000", 2024-04-22T18:40:45.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012059 2024-04-22T18:40:45.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.515883+0000", 2024-04-22T18:40:45.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029563739999999999 2024-04-22T18:40:45.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910510+0000", 2024-04-22T18:40:45.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39462745199999999 2024-04-22T18:40:45.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910542+0000", 2024-04-22T18:40:45.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1225e-05 2024-04-22T18:40:45.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910590+0000", 2024-04-22T18:40:45.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8052e-05 2024-04-22T18:40:45.295 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.295 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.295 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.296 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:80 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:45.296 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:45.296 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 110.77175694500001, 2024-04-22T18:40:45.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39866376999999997, 2024-04-22T18:40:45.296 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.296 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:45.296 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:45.296 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:45.296 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:45.296 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 80 2024-04-22T18:40:45.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.296 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:45.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512061+0000", 2024-04-22T18:40:45.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512060+0000", 2024-04-22T18:40:45.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:45.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512063+0000", 2024-04-22T18:40:45.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.999e-06 2024-04-22T18:40:45.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512069+0000", 2024-04-22T18:40:45.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.891e-06 2024-04-22T18:40:45.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.512073+0000", 2024-04-22T18:40:45.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4429999999999996e-06 2024-04-22T18:40:45.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513084+0000", 2024-04-22T18:40:45.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010102609999999999 2024-04-22T18:40:45.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:45.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.513123+0000", 2024-04-22T18:40:45.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9237999999999999e-05 2024-04-22T18:40:45.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910618+0000", 2024-04-22T18:40:45.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39749549299999998 2024-04-22T18:40:45.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910670+0000", 2024-04-22T18:40:45.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1221000000000003e-05 2024-04-22T18:40:45.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.910724+0000", 2024-04-22T18:40:45.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4777000000000001e-05 2024-04-22T18:40:45.300 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.300 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.300 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.301 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:229 3.2 e51/29 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head v 51'117, mlcod=51'115)", 2024-04-22T18:40:45.301 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:45.301 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.472774887, 2024-04-22T18:40:45.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31548799399999999, 2024-04-22T18:40:45.301 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.301 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.301 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:45.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811043+0000", 2024-04-22T18:40:45.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811041+0000", 2024-04-22T18:40:45.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:45.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811054+0000", 2024-04-22T18:40:45.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3188e-05 2024-04-22T18:40:45.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.811068+0000", 2024-04-22T18:40:45.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4239999999999999e-05 2024-04-22T18:40:45.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.911141+0000", 2024-04-22T18:40:45.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100072851 2024-04-22T18:40:45.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025053+0000", 2024-04-22T18:40:45.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.11391237 2024-04-22T18:40:45.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.025083+0000", 2024-04-22T18:40:45.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9703999999999999e-05 2024-04-22T18:40:45.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126504+0000", 2024-04-22T18:40:45.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10142053500000001 2024-04-22T18:40:45.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.126531+0000", 2024-04-22T18:40:45.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7172999999999999e-05 2024-04-22T18:40:45.304 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.304 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.305 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.305 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2348 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-22T18:40:45.305 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:45.305 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.095985345999999, 2024-04-22T18:40:45.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30532242999999998, 2024-04-22T18:40:45.305 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.305 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:45.305 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:45.305 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:45.305 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:45.305 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2348 2024-04-22T18:40:45.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.305 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:45.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:45.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187832+0000", 2024-04-22T18:40:45.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:45.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187833+0000", 2024-04-22T18:40:45.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.437e-06 2024-04-22T18:40:45.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187836+0000", 2024-04-22T18:40:45.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.897e-06 2024-04-22T18:40:45.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.187839+0000", 2024-04-22T18:40:45.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7250000000000002e-06 2024-04-22T18:40:45.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292213+0000", 2024-04-22T18:40:45.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104373761 2024-04-22T18:40:45.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:45.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.292256+0000", 2024-04-22T18:40:45.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2896999999999999e-05 2024-04-22T18:40:45.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493044+0000", 2024-04-22T18:40:45.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20078818700000001 2024-04-22T18:40:45.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493099+0000", 2024-04-22T18:40:45.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5009000000000001e-05 2024-04-22T18:40:45.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:49.493155+0000", 2024-04-22T18:40:45.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5807000000000003e-05 2024-04-22T18:40:45.309 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.309 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.309 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.310 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:289 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:45.310 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:45.310 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.643092635999999, 2024-04-22T18:40:45.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9176801189999999, 2024-04-22T18:40:45.310 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.310 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.310 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:45.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:40:45.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640723+0000", 2024-04-22T18:40:45.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:45.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640732+0000", 2024-04-22T18:40:45.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5809999999999997e-06 2024-04-22T18:40:45.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640744+0000", 2024-04-22T18:40:45.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.261e-05 2024-04-22T18:40:45.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640753+0000", 2024-04-22T18:40:45.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7460000000000006e-06 2024-04-22T18:40:45.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642189+0000", 2024-04-22T18:40:45.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001435551 2024-04-22T18:40:45.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642200+0000", 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1389999999999999e-05 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.847015+0000", 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048149160000001 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.848217+0000", 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012020659999999999 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351823+0000", 2024-04-22T18:40:45.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50360565099999999 2024-04-22T18:40:45.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351907+0000", 2024-04-22T18:40:45.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4141999999999995e-05 2024-04-22T18:40:45.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:45.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352051+0000", 2024-04-22T18:40:45.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014386600000000001 2024-04-22T18:40:45.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453617+0000", 2024-04-22T18:40:45.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156649500000001 2024-04-22T18:40:45.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558297+0000", 2024-04-22T18:40:45.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10468000399999999 2024-04-22T18:40:45.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558331+0000", 2024-04-22T18:40:45.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3518e-05 2024-04-22T18:40:45.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558405+0000", 2024-04-22T18:40:45.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4221000000000006e-05 2024-04-22T18:40:45.316 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.316 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.316 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.316 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:1 3.2 3:6d90b903:::64b5942574fb36ca1b9a2be186c6b740edf58e989056a2f88b83cd280451a438:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:45.316 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:45.316 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.111791676000003, 2024-04-22T18:40:45.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.285534422, 2024-04-22T18:40:45.316 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.317 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.317 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:45.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:40:45.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172024+0000", 2024-04-22T18:40:45.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:45.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173091+0000", 2024-04-22T18:40:45.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010677569999999999 2024-04-22T18:40:45.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173103+0000", 2024-04-22T18:40:45.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1241e-05 2024-04-22T18:40:45.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173109+0000", 2024-04-22T18:40:45.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4300000000000003e-06 2024-04-22T18:40:45.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173168+0000", 2024-04-22T18:40:45.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9011000000000001e-05 2024-04-22T18:40:45.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:45.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173177+0000", 2024-04-22T18:40:45.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-04-22T18:40:45.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948501+0000", 2024-04-22T18:40:45.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77532387199999997 2024-04-22T18:40:45.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:45.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948912+0000", 2024-04-22T18:40:45.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041183000000000002 2024-04-22T18:40:45.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950438+0000", 2024-04-22T18:40:45.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015253199999999999 2024-04-22T18:40:45.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950488+0000", 2024-04-22T18:40:45.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0285999999999999e-05 2024-04-22T18:40:45.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:45.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950601+0000", 2024-04-22T18:40:45.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011244899999999999 2024-04-22T18:40:45.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954048+0000", 2024-04-22T18:40:45.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003447313 2024-04-22T18:40:45.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457373+0000", 2024-04-22T18:40:45.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50332465299999996 2024-04-22T18:40:45.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457430+0000", 2024-04-22T18:40:45.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7574999999999997e-05 2024-04-22T18:40:45.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457560+0000", 2024-04-22T18:40:45.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013023900000000001 2024-04-22T18:40:45.322 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.323 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.323 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.323 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:8 3.2 3:5f64590d:::7c47933f331d40689081b2bd8f9d7f7bdae1be1fdccb52d696047a3969e902b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:45.323 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:45.323 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.110699306999997, 2024-04-22T18:40:45.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3854200569999999, 2024-04-22T18:40:45.323 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.323 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.323 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:45.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:40:45.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173117+0000", 2024-04-22T18:40:45.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:45.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173393+0000", 2024-04-22T18:40:45.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000275278 2024-04-22T18:40:45.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173399+0000", 2024-04-22T18:40:45.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4590000000000003e-06 2024-04-22T18:40:45.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173406+0000", 2024-04-22T18:40:45.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2450000000000001e-06 2024-04-22T18:40:45.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173795+0000", 2024-04-22T18:40:45.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038912400000000002 2024-04-22T18:40:45.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:45.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173804+0000", 2024-04-22T18:40:45.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-04-22T18:40:45.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.949506+0000", 2024-04-22T18:40:45.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77570185700000005 2024-04-22T18:40:45.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:45.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950166+0000", 2024-04-22T18:40:45.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066022800000000001 2024-04-22T18:40:45.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352843+0000", 2024-04-22T18:40:45.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40267661799999999 2024-04-22T18:40:45.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352911+0000", 2024-04-22T18:40:45.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8126000000000005e-05 2024-04-22T18:40:45.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:45.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.353031+0000", 2024-04-22T18:40:45.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119919 2024-04-22T18:40:45.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453622+0000", 2024-04-22T18:40:45.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100590793 2024-04-22T18:40:45.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558446+0000", 2024-04-22T18:40:45.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104824073 2024-04-22T18:40:45.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558479+0000", 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3362999999999999e-05 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558538+0000", 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9046999999999999e-05 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:10 3.2 3:518c37f9:::851eca84dafbf2f780c4f07291110c453349bdc9c623c17bdcd26e27badb30ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:45.329 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.110405714000002, 2024-04-22T18:40:45.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3852527729999999, 2024-04-22T18:40:45.330 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.330 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.330 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:45.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:40:45.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173411+0000", 2024-04-22T18:40:45.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:45.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173749+0000", 2024-04-22T18:40:45.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000338029 2024-04-22T18:40:45.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173755+0000", 2024-04-22T18:40:45.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0530000000000003e-06 2024-04-22T18:40:45.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173759+0000", 2024-04-22T18:40:45.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8190000000000002e-06 2024-04-22T18:40:45.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173855+0000", 2024-04-22T18:40:45.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5927999999999994e-05 2024-04-22T18:40:45.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:45.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173862+0000", 2024-04-22T18:40:45.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4449999999999999e-06 2024-04-22T18:40:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950835+0000", 2024-04-22T18:40:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77697248299999999 2024-04-22T18:40:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951632+0000", 2024-04-22T18:40:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079752299999999996 2024-04-22T18:40:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355407+0000", 2024-04-22T18:40:45.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40377480500000001 2024-04-22T18:40:45.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355462+0000", 2024-04-22T18:40:45.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4577999999999998e-05 2024-04-22T18:40:45.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:45.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355571+0000", 2024-04-22T18:40:45.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108964 2024-04-22T18:40:45.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453625+0000", 2024-04-22T18:40:45.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098054397000000001 2024-04-22T18:40:45.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558574+0000", 2024-04-22T18:40:45.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104948571 2024-04-22T18:40:45.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558602+0000", 2024-04-22T18:40:45.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7966999999999998e-05 2024-04-22T18:40:45.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558665+0000", 2024-04-22T18:40:45.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3032000000000002e-05 2024-04-22T18:40:45.336 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.336 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.336 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.336 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:11 3.2 3:718aa8aa:::9603c4f8e269a436b2f0bf376c5374e77bdd7cf58fccae60e4e65ac4696806d4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:45.336 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:45.336 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.110054095999999, 2024-04-22T18:40:45.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4852604440000001, 2024-04-22T18:40:45.336 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.336 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.336 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:45.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:45.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:40:45.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:45.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173813+0000", 2024-04-22T18:40:45.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0655999999999997e-05 2024-04-22T18:40:45.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173819+0000", 2024-04-22T18:40:45.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.468e-06 2024-04-22T18:40:45.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173822+0000", 2024-04-22T18:40:45.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4879999999999999e-06 2024-04-22T18:40:45.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173983+0000", 2024-04-22T18:40:45.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160519 2024-04-22T18:40:45.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:45.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173991+0000", 2024-04-22T18:40:45.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6490000000000008e-06 2024-04-22T18:40:45.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951688+0000", 2024-04-22T18:40:45.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77769743800000002 2024-04-22T18:40:45.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:45.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952419+0000", 2024-04-22T18:40:45.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073067499999999997 2024-04-22T18:40:45.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355901+0000", 2024-04-22T18:40:45.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40348237599999998 2024-04-22T18:40:45.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355953+0000", 2024-04-22T18:40:45.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1477000000000002e-05 2024-04-22T18:40:45.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:45.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356060+0000", 2024-04-22T18:40:45.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107114 2024-04-22T18:40:45.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453629+0000", 2024-04-22T18:40:45.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097569127000000005 2024-04-22T18:40:45.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658807+0000", 2024-04-22T18:40:45.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20517829800000001 2024-04-22T18:40:45.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658876+0000", 2024-04-22T18:40:45.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8454000000000006e-05 2024-04-22T18:40:45.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659024+0000", 2024-04-22T18:40:45.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148338 2024-04-22T18:40:45.342 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.343 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.343 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.343 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:12 3.2 3:475e1cd9:::efaab2bd13543cdbb143e869e20eb0ec424f7f5202707859b241ed91d4674936:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:45.343 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:45.343 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.109991151999999, 2024-04-22T18:40:45.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4853405259999999, 2024-04-22T18:40:45.343 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.343 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.343 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:45.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:45.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:40:45.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:45.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173940+0000", 2024-04-22T18:40:45.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011465500000000001 2024-04-22T18:40:45.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173946+0000", 2024-04-22T18:40:45.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5289999999999999e-06 2024-04-22T18:40:45.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173950+0000", 2024-04-22T18:40:45.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8290000000000001e-06 2024-04-22T18:40:45.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174277+0000", 2024-04-22T18:40:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032761299999999998 2024-04-22T18:40:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174285+0000", 2024-04-22T18:40:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.52e-06 2024-04-22T18:40:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952455+0000", 2024-04-22T18:40:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77817000700000005 2024-04-22T18:40:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954035+0000", 2024-04-22T18:40:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001580078 2024-04-22T18:40:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356379+0000", 2024-04-22T18:40:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40234440399999999 2024-04-22T18:40:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356428+0000", 2024-04-22T18:40:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8832999999999998e-05 2024-04-22T18:40:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356532+0000", 2024-04-22T18:40:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103235 2024-04-22T18:40:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453632+0000", 2024-04-22T18:40:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097100346000000004 2024-04-22T18:40:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659069+0000", 2024-04-22T18:40:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20543763800000001 2024-04-22T18:40:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659105+0000", 2024-04-22T18:40:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5240000000000001e-05 2024-04-22T18:40:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659167+0000", 2024-04-22T18:40:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2207999999999998e-05 2024-04-22T18:40:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.349 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:16 3.2 3:47c76cc8:::c00d18458eb20c0c0aa9c612cf9793a68445ab0ea7b9e650ecd0072c60a96c0f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.109863193000002, 2024-04-22T18:40:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.485347744, 2024-04-22T18:40:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:40:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174234+0000", 2024-04-22T18:40:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028047599999999999 2024-04-22T18:40:45.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174240+0000", 2024-04-22T18:40:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2079999999999997e-06 2024-04-22T18:40:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174244+0000", 2024-04-22T18:40:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5810000000000001e-06 2024-04-22T18:40:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174670+0000", 2024-04-22T18:40:45.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042640000000000001 2024-04-22T18:40:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174675+0000", 2024-04-22T18:40:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8709999999999996e-06 2024-04-22T18:40:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054168+0000", 2024-04-22T18:40:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87949257300000006 2024-04-22T18:40:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:45.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054877+0000", 2024-04-22T18:40:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070884300000000002 2024-04-22T18:40:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356836+0000", 2024-04-22T18:40:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30195926000000001 2024-04-22T18:40:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356887+0000", 2024-04-22T18:40:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0767999999999999e-05 2024-04-22T18:40:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356991+0000", 2024-04-22T18:40:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104072 2024-04-22T18:40:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453636+0000", 2024-04-22T18:40:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096644968999999997 2024-04-22T18:40:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659207+0000", 2024-04-22T18:40:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20557154699999999 2024-04-22T18:40:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659240+0000", 2024-04-22T18:40:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2665e-05 2024-04-22T18:40:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659302+0000", 2024-04-22T18:40:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2234000000000001e-05 2024-04-22T18:40:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:17 3.2 3:6ef31daa:::1cf4ff496c4480e58510f607481d976b47d84868e2c54016273c38baf590f3b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:45.356 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.109568912999997, 2024-04-22T18:40:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4851703729999999, 2024-04-22T18:40:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:40:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174248+0000", 2024-04-22T18:40:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174626+0000", 2024-04-22T18:40:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037839699999999999 2024-04-22T18:40:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174633+0000", 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0990000000000004e-06 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174636+0000", 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5540000000000002e-06 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174790+0000", 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153807 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174793+0000", 2024-04-22T18:40:45.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4970000000000001e-06 2024-04-22T18:40:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054902+0000", 2024-04-22T18:40:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88010886300000002 2024-04-22T18:40:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055366+0000", 2024-04-22T18:40:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046336199999999999 2024-04-22T18:40:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.452973+0000", 2024-04-22T18:40:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39760783999999999 2024-04-22T18:40:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453046+0000", 2024-04-22T18:40:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2374999999999997e-05 2024-04-22T18:40:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453193+0000", 2024-04-22T18:40:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014722499999999999 2024-04-22T18:40:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554698+0000", 2024-04-22T18:40:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101505096 2024-04-22T18:40:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659337+0000", 2024-04-22T18:40:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104639236 2024-04-22T18:40:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659365+0000", 2024-04-22T18:40:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7594e-05 2024-04-22T18:40:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659419+0000", 2024-04-22T18:40:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4058999999999999e-05 2024-04-22T18:40:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:18 3.2 3:792a4ed0:::e1c814961b3d1531fb5ed475b3d496a0405c3a819b68a92ff9b2391fe912370f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.109172657000002, 2024-04-22T18:40:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.585036954, 2024-04-22T18:40:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174640+0000", 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174758+0000", 2024-04-22T18:40:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011819099999999999 2024-04-22T18:40:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174764+0000", 2024-04-22T18:40:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6729999999999997e-06 2024-04-22T18:40:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174767+0000", 2024-04-22T18:40:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3830000000000001e-06 2024-04-22T18:40:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373669+0000", 2024-04-22T18:40:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198901989 2024-04-22T18:40:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373686+0000", 2024-04-22T18:40:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7286e-05 2024-04-22T18:40:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055408+0000", 2024-04-22T18:40:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68172133400000001 2024-04-22T18:40:45.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055938+0000", 2024-04-22T18:40:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053016599999999999 2024-04-22T18:40:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453668+0000", 2024-04-22T18:40:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39773030799999998 2024-04-22T18:40:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453717+0000", 2024-04-22T18:40:45.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8884000000000002e-05 2024-04-22T18:40:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453823+0000", 2024-04-22T18:40:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105791 2024-04-22T18:40:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554715+0000", 2024-04-22T18:40:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100891628 2024-04-22T18:40:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759439+0000", 2024-04-22T18:40:45.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20472459400000001 2024-04-22T18:40:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759525+0000", 2024-04-22T18:40:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5653000000000001e-05 2024-04-22T18:40:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759682+0000", 2024-04-22T18:40:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015711800000000001 2024-04-22T18:40:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:75 3.1 3:ab43a94d:::f38e6e97cb24e2b90da964fee5d2d6d8674386870fc552915e0c3fdbefd4b126:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:40:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.036296139999999, 2024-04-22T18:40:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60149928399999997, 2024-04-22T18:40:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:40:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:40:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247520+0000", 2024-04-22T18:40:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247527+0000", 2024-04-22T18:40:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1670000000000002e-06 2024-04-22T18:40:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247539+0000", 2024-04-22T18:40:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503e-05 2024-04-22T18:40:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247549+0000", 2024-04-22T18:40:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0000000000000001e-05 2024-04-22T18:40:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247796+0000", 2024-04-22T18:40:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024712699999999998 2024-04-22T18:40:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:40:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247955+0000", 2024-04-22T18:40:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158534 2024-04-22T18:40:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.343985+0000", 2024-04-22T18:40:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096030344000000004 2024-04-22T18:40:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.344035+0000", 2024-04-22T18:40:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9969999999999998e-05 2024-04-22T18:40:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.745806+0000", 2024-04-22T18:40:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.401771029 2024-04-22T18:40:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746040+0000", 2024-04-22T18:40:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023339300000000001 2024-04-22T18:40:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746198+0000", 2024-04-22T18:40:45.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015793299999999999 2024-04-22T18:40:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848737+0000", 2024-04-22T18:40:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10253898 2024-04-22T18:40:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848751+0000", 2024-04-22T18:40:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4694e-05 2024-04-22T18:40:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848825+0000", 2024-04-22T18:40:45.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3443999999999997e-05 2024-04-22T18:40:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849021+0000", 2024-04-22T18:40:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000196047 2024-04-22T18:40:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:76 3.1 3:b44829a0:::3788286a255d538503614e5f04049b0bb54cacaa317ab102b5dee8f5b640b296:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:40:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.036257074, 2024-04-22T18:40:45.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60168681599999996, 2024-04-22T18:40:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:40:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:40:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:40:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247564+0000", 2024-04-22T18:40:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1150000000000004e-06 2024-04-22T18:40:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247570+0000", 2024-04-22T18:40:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3940000000000001e-06 2024-04-22T18:40:45.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247576+0000", 2024-04-22T18:40:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3619999999999997e-06 2024-04-22T18:40:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.340845+0000", 2024-04-22T18:40:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.093269199999999997 2024-04-22T18:40:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:40:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.341043+0000", 2024-04-22T18:40:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019834400000000001 2024-04-22T18:40:45.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345106+0000", 2024-04-22T18:40:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040629780000000001 2024-04-22T18:40:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345160+0000", 2024-04-22T18:40:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4246999999999997e-05 2024-04-22T18:40:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747047+0000", 2024-04-22T18:40:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40188678300000003 2024-04-22T18:40:45.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747162+0000", 2024-04-22T18:40:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011483400000000001 2024-04-22T18:40:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747429+0000", 2024-04-22T18:40:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026700299999999998 2024-04-22T18:40:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849032+0000", 2024-04-22T18:40:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101603203 2024-04-22T18:40:45.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849095+0000", 2024-04-22T18:40:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2597999999999999e-05 2024-04-22T18:40:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849136+0000", 2024-04-22T18:40:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1251000000000002e-05 2024-04-22T18:40:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849247+0000", 2024-04-22T18:40:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011127 2024-04-22T18:40:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3524 4.c 4:314c2fec:::smithi08937929-22:head [write 692521~538214] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:40:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.128717+0000", 2024-04-22T18:40:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.1551009790000002, 2024-04-22T18:40:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40275804199999998, 2024-04-22T18:40:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:45.383 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3524 2024-04-22T18:40:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128717+0000", 2024-04-22T18:40:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128717+0000", 2024-04-22T18:40:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128715+0000", 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128877+0000", 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016187999999999999 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128886+0000", 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8780000000000004e-06 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.228979+0000", 2024-04-22T18:40:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092734 2024-04-22T18:40:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229043+0000", 2024-04-22T18:40:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4036999999999997e-05 2024-04-22T18:40:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229146+0000", 2024-04-22T18:40:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010301700000000001 2024-04-22T18:40:45.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229286+0000", 2024-04-22T18:40:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014017000000000001 2024-04-22T18:40:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.232704+0000", 2024-04-22T18:40:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0034182169999999999 2024-04-22T18:40:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.531413+0000", 2024-04-22T18:40:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29870898200000001 2024-04-22T18:40:45.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.531445+0000", 2024-04-22T18:40:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1751999999999997e-05 2024-04-22T18:40:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.531475+0000", 2024-04-22T18:40:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9906000000000001e-05 2024-04-22T18:40:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3525 4.c 4:314c2fec:::smithi08937929-22:head [write 1931338~785088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:40:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.229011+0000", 2024-04-22T18:40:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.0548063810000006, 2024-04-22T18:40:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40428670300000003, 2024-04-22T18:40:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3525 2024-04-22T18:40:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229011+0000", 2024-04-22T18:40:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229011+0000", 2024-04-22T18:40:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229010+0000", 2024-04-22T18:40:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229182+0000", 2024-04-22T18:40:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000171645 2024-04-22T18:40:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229188+0000", 2024-04-22T18:40:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.674e-06 2024-04-22T18:40:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229193+0000", 2024-04-22T18:40:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8890000000000001e-06 2024-04-22T18:40:45.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.230814+0000", 2024-04-22T18:40:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00162068 2024-04-22T18:40:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.230865+0000", 2024-04-22T18:40:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1700000000000003e-05 2024-04-22T18:40:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.230967+0000", 2024-04-22T18:40:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101481 2024-04-22T18:40:45.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.234144+0000", 2024-04-22T18:40:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031771849999999999 2024-04-22T18:40:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.633226+0000", 2024-04-22T18:40:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39908168799999999 2024-04-22T18:40:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.633257+0000", 2024-04-22T18:40:45.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1096999999999997e-05 2024-04-22T18:40:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.633298+0000", 2024-04-22T18:40:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0960000000000001e-05 2024-04-22T18:40:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3526 4.c 4:314c2fec:::smithi08937929-22:head [setxattr _header (56) in=7b,truncate 3092521] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:40:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:40:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.0546202939999993, 2024-04-22T18:40:45.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60463626000000004, 2024-04-22T18:40:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3526 2024-04-22T18:40:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:40:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:40:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:40:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229200+0000", 2024-04-22T18:40:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2320000000000001e-06 2024-04-22T18:40:45.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229203+0000", 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329292+0000", 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100088891 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329357+0000", 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5115000000000003e-05 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329453+0000", 2024-04-22T18:40:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5802000000000002e-05 2024-04-22T18:40:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329594+0000", 2024-04-22T18:40:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014157500000000001 2024-04-22T18:40:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.330172+0000", 2024-04-22T18:40:45.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057782799999999996 2024-04-22T18:40:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833712+0000", 2024-04-22T18:40:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50353975299999998 2024-04-22T18:40:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833774+0000", 2024-04-22T18:40:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1832e-05 2024-04-22T18:40:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833834+0000", 2024-04-22T18:40:45.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9809999999999998e-05 2024-04-22T18:40:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3527 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:40:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.329322+0000", 2024-04-22T18:40:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.9544952220000003, 2024-04-22T18:40:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50479107700000003, 2024-04-22T18:40:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:45.400 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3527 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329322+0000", 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329322+0000", 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329321+0000", 2024-04-22T18:40:45.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329337+0000", 2024-04-22T18:40:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5820000000000001e-05 2024-04-22T18:40:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329345+0000", 2024-04-22T18:40:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4610000000000008e-06 2024-04-22T18:40:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.429437+0000", 2024-04-22T18:40:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092323 2024-04-22T18:40:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.429503+0000", 2024-04-22T18:40:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5446999999999999e-05 2024-04-22T18:40:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.429626+0000", 2024-04-22T18:40:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012354600000000001 2024-04-22T18:40:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833904+0000", 2024-04-22T18:40:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.404277901 2024-04-22T18:40:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.834004+0000", 2024-04-22T18:40:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9444999999999994e-05 2024-04-22T18:40:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.834113+0000", 2024-04-22T18:40:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010976099999999999 2024-04-22T18:40:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.404 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.405 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:45.405 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:45.405 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:45.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:45.407 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:45.407+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:45.506 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:45.508+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:45.509 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:45.509 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:45.509 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:45.510 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:45.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:45.608 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:45.609+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:45.908 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:40:45.911 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:45.911 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:45.911 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:45.911 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:45.911 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:45.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:45.912 INFO:tasks.rados.rados.0.smithi089.stdout:1598: set_chunk oid 4 offset: 1286144 length: 13312 target oid 4cfe7a731b9f028ff1eeea11ba689e1dac630ed9130b02a6267a7522e048ff73 offset: 0 snap -1 2024-04-22T18:40:45.912 INFO:tasks.rados.rados.0.smithi089.stdout:1599: write (excl) oid 33 current snap is 0 2024-04-22T18:40:45.912 INFO:tasks.rados.rados.0.smithi089.stdout:1599: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2024-04-22T18:40:45.912 INFO:tasks.rados.rados.0.smithi089.stdout:1599: writing smithi08937929-33 from 561433 to 1336337 tid 1 2024-04-22T18:40:45.913 INFO:tasks.rados.rados.0.smithi089.stdout:1599: writing smithi08937929-33 from 1950697 to 2365137 tid 2 2024-04-22T18:40:45.913 INFO:tasks.rados.rados.0.smithi089.stdout:1599: writing smithi08937929-33 from 2777464 to 3361433 tid 3 2024-04-22T18:40:45.913 INFO:tasks.rados.rados.0.smithi089.stdout:1600: read oid 37 snap -1 2024-04-22T18:40:45.913 INFO:tasks.rados.rados.0.smithi089.stdout:1600: expect (ObjNum 267 snap 0 seq_num 267) 2024-04-22T18:40:45.913 INFO:tasks.rados.rados.0.smithi089.stdout:1601: read oid 30 snap -1 2024-04-22T18:40:45.913 INFO:tasks.rados.rados.0.smithi089.stdout:1601: expect (ObjNum 285 snap 0 seq_num 285) 2024-04-22T18:40:45.913 INFO:tasks.rados.rados.0.smithi089.stdout:1602: read oid 15 snap -1 2024-04-22T18:40:45.913 INFO:tasks.rados.rados.0.smithi089.stdout:1602: expect (ObjNum 301 snap 0 seq_num 301) 2024-04-22T18:40:45.913 INFO:tasks.rados.rados.0.smithi089.stdout:1603: delete oid 10 current snap is 0 2024-04-22T18:40:45.913 INFO:tasks.rados.rados.0.smithi089.stdout:1595: expect (ObjNum 298 snap 0 seq_num 298) 2024-04-22T18:40:45.913 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:45.848+0000 7f309102d640 -1 osd.1 113 handle_osd_map injecting map crc failure 2024-04-22T18:40:45.914 INFO:tasks.rados.rados.0.smithi089.stdout:1588: got (0) Success 2024-04-22T18:40:45.914 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 26 (ObjNum 278 snap 0 seq_num 278) dirty exists 2024-04-22T18:40:45.914 INFO:tasks.rados.rados.0.smithi089.stdout:1590: expect (ObjNum 307 snap 0 seq_num 307) 2024-04-22T18:40:45.915 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:45.911+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:46.010 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:46.012+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:46.017 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.85494054999999, 2024-04-22T18:40:46.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:40:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:40:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:40:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:40:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:40:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:40:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:40:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:40:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:40:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:40:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:40:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:40:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:40:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:40:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:40:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:40:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:40:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:40:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:40:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:40:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.753381729, 2024-04-22T18:40:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:40:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:40:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.023 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:40:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:46.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:40:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:40:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:40:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:40:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:40:46.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:40:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:40:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:40:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:40:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:40:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:46.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:40:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:40:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:40:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:40:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:40:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:40:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:40:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:40:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:40:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:40:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:46.028 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.75335532699999, 2024-04-22T18:40:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:40:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:40:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:40:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:40:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:40:46.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:40:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:40:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:40:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:40:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:40:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:40:46.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:40:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:40:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:40:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:40:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:40:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:40:46.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:40:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:40:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.355020998, 2024-04-22T18:40:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:40:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.033 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:40:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:40:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:40:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:40:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:40:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:40:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:40:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:40:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:40:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:40:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:40:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:40:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:40:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:40:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:40:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:40:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:40:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:40:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:40:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:40:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:40:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.038 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.2541847, 2024-04-22T18:40:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:40:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:40:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.039 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:40:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:46.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:40:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:40:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:40:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:40:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:40:46.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:40:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:40:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:40:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:40:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:40:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:40:46.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:40:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:40:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:40:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:40:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:40:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:40:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:40:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:40:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:40:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.25402742999999, 2024-04-22T18:40:46.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:40:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:40:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:40:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:40:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:40:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:40:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:40:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:40:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:40:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:40:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:40:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:40:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:40:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:40:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:40:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:40:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:40:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:40:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:40:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:40:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:40:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.049 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.254012821, 2024-04-22T18:40:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:40:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:40:46.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:46.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:40:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:40:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:40:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:40:46.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:40:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:40:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:40:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:40:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:40:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:40:46.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:40:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:40:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:40:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:40:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:40:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.25400243599999, 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:40:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:40:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:40:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:40:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:40:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:40:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:40:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:40:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:40:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:40:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:40:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:40:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:40:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:40:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:40:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:40:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:40:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:40:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:40:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:40:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:40:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.253072506, 2024-04-22T18:40:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:40:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.060 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:40:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:46.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:40:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:40:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:40:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:40:46.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:40:46.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:40:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:40:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:40:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:40:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:40:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:40:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:40:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:40:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:40:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:40:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.252610381, 2024-04-22T18:40:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:40:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:40:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:40:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:40:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:40:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:40:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:40:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:40:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:40:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:40:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:40:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:40:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:40:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:40:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:40:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:40:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:40:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:40:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:40:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:40:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:40:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:40:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:40:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.252597577, 2024-04-22T18:40:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:40:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.071 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:40:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:40:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:40:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:40:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:40:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:40:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:40:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:40:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:40:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:40:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:40:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:40:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:40:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.075 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 107.924825121, 2024-04-22T18:40:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:40:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:40:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.076 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:40:46.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:40:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:40:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:40:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:40:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:40:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:40:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:40:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:40:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:40:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:40:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:40:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:40:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:40:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:40:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:40:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:40:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:40:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:40:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:40:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:40:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:40:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:46.081 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 106.916252009, 2024-04-22T18:40:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:40:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:40:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:46.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:40:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:40:46.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:40:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:40:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:40:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:40:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:40:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:40:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:40:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:40:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:40:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:40:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:40:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:40:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:40:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:40:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:40:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:40:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:40:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:40:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:40:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:40:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.881604494000001, 2024-04-22T18:40:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:40:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.087 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:40:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:40:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:40:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:40:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:40:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:40:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:40:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:40:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:40:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:40:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:40:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:40:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:40:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:40:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.091 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1684 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 75.412120530999999, 2024-04-22T18:40:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987885199999998, 2024-04-22T18:40:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1684 2024-04-22T18:40:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:46.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605278+0000", 2024-04-22T18:40:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605280+0000", 2024-04-22T18:40:46.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1770000000000001e-06 2024-04-22T18:40:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605285+0000", 2024-04-22T18:40:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-04-22T18:40:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605289+0000", 2024-04-22T18:40:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.687e-06 2024-04-22T18:40:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705352+0000", 2024-04-22T18:40:46.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006322199999999 2024-04-22T18:40:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705423+0000", 2024-04-22T18:40:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1136999999999993e-05 2024-04-22T18:40:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105017+0000", 2024-04-22T18:40:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.399593268 2024-04-22T18:40:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105081+0000", 2024-04-22T18:40:46.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4703999999999996e-05 2024-04-22T18:40:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105157+0000", 2024-04-22T18:40:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6205999999999995e-05 2024-04-22T18:40:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2191 4.1 4.fdaa93c1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-04-22T18:40:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.467528281, 2024-04-22T18:40:46.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51064384399999996, 2024-04-22T18:40:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2191 2024-04-22T18:40:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549872+0000", 2024-04-22T18:40:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.387e-06 2024-04-22T18:40:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549875+0000", 2024-04-22T18:40:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549880+0000", 2024-04-22T18:40:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2529999999999998e-06 2024-04-22T18:40:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060496+0000", 2024-04-22T18:40:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51061597999999997 2024-04-22T18:40:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060515+0000", 2024-04-22T18:40:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9130000000000001e-05 2024-04-22T18:40:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2198 4.d 4:bc165c76:::smithi08937929-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.466260579999997, 2024-04-22T18:40:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50502289899999997, 2024-04-22T18:40:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.100 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2198 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551138+0000", 2024-04-22T18:40:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551143+0000", 2024-04-22T18:40:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8949999999999997e-06 2024-04-22T18:40:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551151+0000", 2024-04-22T18:40:46.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8729999999999996e-06 2024-04-22T18:40:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551158+0000", 2024-04-22T18:40:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3099999999999997e-06 2024-04-22T18:40:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551180+0000", 2024-04-22T18:40:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2197000000000001e-05 2024-04-22T18:40:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551185+0000", 2024-04-22T18:40:46.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7010000000000004e-06 2024-04-22T18:40:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657133+0000", 2024-04-22T18:40:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105948498 2024-04-22T18:40:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657183+0000", 2024-04-22T18:40:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0124000000000003e-05 2024-04-22T18:40:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.055924+0000", 2024-04-22T18:40:46.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39874079299999998 2024-04-22T18:40:46.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056018+0000", 2024-04-22T18:40:46.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4032999999999999e-05 2024-04-22T18:40:46.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056161+0000", 2024-04-22T18:40:46.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143385 2024-04-22T18:40:46.105 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.105 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.105 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.106 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2212 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:46.106 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:46.106 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.466190937999997, 2024-04-22T18:40:46.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50989936400000002, 2024-04-22T18:40:46.106 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.106 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:46.106 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.106 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.106 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.106 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2212 2024-04-22T18:40:46.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.106 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551211+0000", 2024-04-22T18:40:46.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6280000000000001e-06 2024-04-22T18:40:46.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551214+0000", 2024-04-22T18:40:46.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1209999999999998e-06 2024-04-22T18:40:46.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551217+0000", 2024-04-22T18:40:46.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8490000000000002e-06 2024-04-22T18:40:46.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061086+0000", 2024-04-22T18:40:46.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50986975099999998 2024-04-22T18:40:46.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061108+0000", 2024-04-22T18:40:46.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1319e-05 2024-04-22T18:40:46.109 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.109 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.109 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.109 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2213 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:46.109 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:46.109 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.466180520999998, 2024-04-22T18:40:46.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50998486099999996, 2024-04-22T18:40:46.110 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.110 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:46.110 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.110 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.110 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.110 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2213 2024-04-22T18:40:46.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.110 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:46.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:46.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551218+0000", 2024-04-22T18:40:46.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551220+0000", 2024-04-22T18:40:46.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.319e-06 2024-04-22T18:40:46.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551223+0000", 2024-04-22T18:40:46.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9660000000000001e-06 2024-04-22T18:40:46.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551226+0000", 2024-04-22T18:40:46.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9050000000000001e-06 2024-04-22T18:40:46.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061162+0000", 2024-04-22T18:40:46.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.509936838 2024-04-22T18:40:46.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061204+0000", 2024-04-22T18:40:46.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1094999999999999e-05 2024-04-22T18:40:46.113 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.113 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.113 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.113 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2214 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:46.113 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:46.113 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.466171531999997, 2024-04-22T18:40:46.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50997458100000004, 2024-04-22T18:40:46.113 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.113 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:46.113 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.114 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.114 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.114 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2214 2024-04-22T18:40:46.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.114 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:46.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:46.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551227+0000", 2024-04-22T18:40:46.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551229+0000", 2024-04-22T18:40:46.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8649999999999999e-06 2024-04-22T18:40:46.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551232+0000", 2024-04-22T18:40:46.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551235+0000", 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061194+0000", 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50995819099999995 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061202+0000", 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.541e-06 2024-04-22T18:40:46.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.117 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.117 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.117 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.117 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.117 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:46.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:46.119 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:46.119+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:46.120 INFO:tasks.rados.rados.0.smithi089.stdout:1587: finishing set_chunk to oid 38 2024-04-22T18:40:46.120 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 38 v 86 (ObjNum 230 snap 0 seq_num 230) dirty exists 2024-04-22T18:40:46.120 INFO:tasks.rados.rados.0.smithi089.stdout:1591: finishing write tid 1 to smithi08937929-21 2024-04-22T18:40:46.120 INFO:tasks.rados.rados.0.smithi089.stdout:1591: finishing write tid 2 to smithi08937929-21 2024-04-22T18:40:46.120 INFO:tasks.rados.rados.0.smithi089.stdout:1587: done (15 left) 2024-04-22T18:40:46.120 INFO:tasks.rados.rados.0.smithi089.stdout:1588: done (14 left) 2024-04-22T18:40:46.120 INFO:tasks.rados.rados.0.smithi089.stdout:1590: done (13 left) 2024-04-22T18:40:46.120 INFO:tasks.rados.rados.0.smithi089.stdout:1595: done (12 left) 2024-04-22T18:40:46.120 INFO:tasks.rados.rados.0.smithi089.stdout:1603: done (11 left) 2024-04-22T18:40:46.120 INFO:tasks.rados.rados.0.smithi089.stdout:1604: read oid 10 snap -1 2024-04-22T18:40:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:1604: expect deleted 2024-04-22T18:40:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:1605: write (excl) oid 17 current snap is 0 2024-04-22T18:40:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:1591: finishing write tid 3 to smithi08937929-21 2024-04-22T18:40:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:1591: finishing write tid 4 to smithi08937929-21 2024-04-22T18:40:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:1592: finishing write tid 2 to smithi08937929-16 2024-04-22T18:40:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:1592: finishing write tid 3 to smithi08937929-16 2024-04-22T18:40:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:1592: finishing write tid 4 to smithi08937929-16 2024-04-22T18:40:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:1605: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-04-22T18:40:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:1605: writing smithi08937929-17 from 595047 to 1236802 tid 1 2024-04-22T18:40:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:1605: writing smithi08937929-17 from 1667661 to 2085770 tid 2 2024-04-22T18:40:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:1605: writing smithi08937929-17 from 2548663 to 3311685 tid 3 2024-04-22T18:40:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:1606: tier_promote oid 13 2024-04-22T18:40:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:1592: finishing write tid 5 to smithi08937929-16 2024-04-22T18:40:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:1592: oid 16 updating version 0 to 127 2024-04-22T18:40:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:1592: oid 16 updating version 127 to 129 2024-04-22T18:40:46.122 INFO:tasks.rados.rados.0.smithi089.stdout:1592: oid 16 version 129 is already newer than 128 2024-04-22T18:40:46.122 INFO:tasks.rados.rados.0.smithi089.stdout:1592: oid 16 updating version 129 to 130 2024-04-22T18:40:46.122 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 130 (ObjNum 315 snap 0 seq_num 315) dirty exists 2024-04-22T18:40:46.122 INFO:tasks.rados.rados.0.smithi089.stdout:1592: left oid 16 (ObjNum 315 snap 0 seq_num 315) 2024-04-22T18:40:46.122 INFO:tasks.rados.rados.0.smithi089.stdout:1596: expect (ObjNum 296 snap 0 seq_num 296) 2024-04-22T18:40:46.213 INFO:tasks.rados.rados.0.smithi089.stdout:1593: finishing write tid 1 to smithi08937929-49 2024-04-22T18:40:46.213 INFO:tasks.rados.rados.0.smithi089.stdout:1593: finishing write tid 2 to smithi08937929-49 2024-04-22T18:40:46.213 INFO:tasks.rados.rados.0.smithi089.stdout:1593: finishing write tid 3 to smithi08937929-49 2024-04-22T18:40:46.213 INFO:tasks.rados.rados.0.smithi089.stdout:1593: finishing write tid 4 to smithi08937929-49 2024-04-22T18:40:46.213 INFO:tasks.rados.rados.0.smithi089.stdout:1593: finishing write tid 5 to smithi08937929-49 2024-04-22T18:40:46.213 INFO:tasks.rados.rados.0.smithi089.stdout:1593: oid 49 updating version 0 to 131 2024-04-22T18:40:46.213 INFO:tasks.rados.rados.0.smithi089.stdout:1593: oid 49 updating version 131 to 134 2024-04-22T18:40:46.214 INFO:tasks.rados.rados.0.smithi089.stdout:1593: oid 49 version 134 is already newer than 133 2024-04-22T18:40:46.214 INFO:tasks.rados.rados.0.smithi089.stdout:1593: oid 49 version 134 is already newer than 132 2024-04-22T18:40:46.214 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 134 (ObjNum 316 snap 0 seq_num 316) dirty exists 2024-04-22T18:40:46.214 INFO:tasks.rados.rados.0.smithi089.stdout:1593: left oid 49 (ObjNum 316 snap 0 seq_num 316) 2024-04-22T18:40:46.214 INFO:tasks.rados.rados.0.smithi089.stdout:1594: finishing write tid 1 to smithi08937929-20 2024-04-22T18:40:46.214 INFO:tasks.rados.rados.0.smithi089.stdout:1592: done (13 left) 2024-04-22T18:40:46.214 INFO:tasks.rados.rados.0.smithi089.stdout:1593: done (12 left) 2024-04-22T18:40:46.214 INFO:tasks.rados.rados.0.smithi089.stdout:1596: done (11 left) 2024-04-22T18:40:46.214 INFO:tasks.rados.rados.0.smithi089.stdout:1607: delete oid 32 current snap is 0 2024-04-22T18:40:46.214 INFO:tasks.rados.rados.0.smithi089.stdout:1594: finishing write tid 2 to smithi08937929-20 2024-04-22T18:40:46.214 INFO:tasks.rados.rados.0.smithi089.stdout:1594: finishing write tid 3 to smithi08937929-20 2024-04-22T18:40:46.214 INFO:tasks.rados.rados.0.smithi089.stdout:1591: finishing write tid 5 to smithi08937929-21 2024-04-22T18:40:46.214 INFO:tasks.rados.rados.0.smithi089.stdout:1591: oid 21 updating version 0 to 55 2024-04-22T18:40:46.214 INFO:tasks.rados.rados.0.smithi089.stdout:1591: oid 21 version 55 is already newer than 53 2024-04-22T18:40:46.215 INFO:tasks.rados.rados.0.smithi089.stdout:1591: oid 21 updating version 55 to 56 2024-04-22T18:40:46.215 INFO:tasks.rados.rados.0.smithi089.stdout:1591: oid 21 version 56 is already newer than 54 2024-04-22T18:40:46.215 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 56 (ObjNum 314 snap 0 seq_num 314) dirty exists 2024-04-22T18:40:46.215 INFO:tasks.rados.rados.0.smithi089.stdout:1591: left oid 21 (ObjNum 314 snap 0 seq_num 314) 2024-04-22T18:40:46.215 INFO:tasks.rados.rados.0.smithi089.stdout:1597: finishing write tid 1 to smithi08937929-45 2024-04-22T18:40:46.215 INFO:tasks.rados.rados.0.smithi089.stdout:1597: finishing write tid 2 to smithi08937929-45 2024-04-22T18:40:46.215 INFO:tasks.rados.rados.0.smithi089.stdout:1597: finishing write tid 3 to smithi08937929-45 2024-04-22T18:40:46.215 INFO:tasks.rados.rados.0.smithi089.stdout:1597: finishing write tid 4 to smithi08937929-45 2024-04-22T18:40:46.215 INFO:tasks.rados.rados.0.smithi089.stdout:1597: finishing write tid 5 to smithi08937929-45 2024-04-22T18:40:46.215 INFO:tasks.rados.rados.0.smithi089.stdout:1597: oid 45 updating version 0 to 83 2024-04-22T18:40:46.215 INFO:tasks.rados.rados.0.smithi089.stdout:1597: oid 45 updating version 83 to 85 2024-04-22T18:40:46.215 INFO:tasks.rados.rados.0.smithi089.stdout:1597: oid 45 updating version 85 to 86 2024-04-22T18:40:46.215 INFO:tasks.rados.rados.0.smithi089.stdout:1597: oid 45 version 86 is already newer than 84 2024-04-22T18:40:46.215 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 86 (ObjNum 318 snap 0 seq_num 318) dirty exists 2024-04-22T18:40:46.215 INFO:tasks.rados.rados.0.smithi089.stdout:1597: left oid 45 (ObjNum 318 snap 0 seq_num 318) 2024-04-22T18:40:46.216 INFO:tasks.rados.rados.0.smithi089.stdout:1600: expect (ObjNum 267 snap 0 seq_num 267) 2024-04-22T18:40:46.219 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:46.220+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:46.261 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:46.261 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:46.270 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:46.320 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:46.321+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:46.323 INFO:tasks.rados.rados.0.smithi089.stdout:1594: finishing write tid 4 to smithi08937929-20 2024-04-22T18:40:46.323 INFO:tasks.rados.rados.0.smithi089.stdout:1594: finishing write tid 5 to smithi08937929-20 2024-04-22T18:40:46.323 INFO:tasks.rados.rados.0.smithi089.stdout:1594: oid 20 updating version 0 to 16 2024-04-22T18:40:46.323 INFO:tasks.rados.rados.0.smithi089.stdout:1594: oid 20 updating version 16 to 18 2024-04-22T18:40:46.323 INFO:tasks.rados.rados.0.smithi089.stdout:1594: oid 20 version 18 is already newer than 15 2024-04-22T18:40:46.323 INFO:tasks.rados.rados.0.smithi089.stdout:1594: oid 20 version 18 is already newer than 17 2024-04-22T18:40:46.323 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 18 (ObjNum 317 snap 0 seq_num 317) dirty exists 2024-04-22T18:40:46.323 INFO:tasks.rados.rados.0.smithi089.stdout:1594: left oid 20 (ObjNum 317 snap 0 seq_num 317) 2024-04-22T18:40:46.323 INFO:tasks.rados.rados.0.smithi089.stdout:1601: expect (ObjNum 285 snap 0 seq_num 285) 2024-04-22T18:40:46.371 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:46.371 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:46.371 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:46.371 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:46.371 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:46.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:46.421 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:46.422+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:46.491 INFO:tasks.rados.rados.0.smithi089.stdout:1602: expect (ObjNum 301 snap 0 seq_num 301) 2024-04-22T18:40:46.522 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:46.523+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:46.523 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:46.523 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:46.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:46.523 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:46.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.523 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:40:46.523 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:46.524 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 112.429756725, 2024-04-22T18:40:46.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:40:46.524 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.524 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.524 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.524 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.524 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.524 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:40:46.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.524 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:46.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:46.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:40:46.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:46.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:40:46.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:40:46.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:40:46.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:40:46.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:40:46.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:40:46.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:40:46.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:40:46.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:40:46.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:40:46.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:46.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:40:46.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:40:46.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:40:46.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:40:46.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:40:46.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:40:46.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:40:46.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:40:46.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:40:46.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:40:46.529 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.529 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.529 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.529 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:46.529 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:46.529 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 108.53025885, 2024-04-22T18:40:46.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:40:46.529 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.529 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.529 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:46.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:46.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:40:46.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:46.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:40:46.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:40:46.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:46.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:40:46.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:40:46.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:40:46.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:40:46.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:40:46.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:40:46.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:40:46.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:40:46.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:40:46.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:40:46.534 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.534 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.534 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.535 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:40:46.535 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:46.535 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 107.421959059, 2024-04-22T18:40:46.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:40:46.535 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.535 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.535 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.535 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.535 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.535 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:40:46.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.535 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:40:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:40:46.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:40:46.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:40:46.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:40:46.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:40:46.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:40:46.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:40:46.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:40:46.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:40:46.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:40:46.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:46.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:40:46.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:40:46.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:40:46.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:40:46.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:40:46.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:40:46.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.090415430999997, 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.540 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:40:46.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:40:46.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:40:46.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:40:46.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:40:46.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:40:46.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:40:46.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:40:46.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:40:46.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:40:46.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:46.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:40:46.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:40:46.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 93.999329622999994, 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.545 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:40:46.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:46.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:40:46.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:40:46.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:40:46.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:40:46.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:40:46.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:40:46.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:40:46.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:40:46.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:40:46.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:40:46.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:46.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:40:46.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:40:46.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:40:46.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:40:46.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:40:46.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:40:46.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:40:46.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:40:46.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:40:46.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:40:46.550 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.550 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.550 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.551 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:46.551 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:46.551 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 93.993529362999993, 2024-04-22T18:40:46.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:40:46.551 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.551 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.551 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.551 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.551 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.551 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:40:46.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.551 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:40:46.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:40:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:40:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:40:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:40:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:40:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:40:46.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:40:46.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:40:46.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:46.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:40:46.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:40:46.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:40:46.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:40:46.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:40:46.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:40:46.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:40:46.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:40:46.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:40:46.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:40:46.556 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.556 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.556 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.556 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:46.556 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:46.556 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 93.988789019999999, 2024-04-22T18:40:46.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:40:46.556 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.556 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:40:46.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:46.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:40:46.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:40:46.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:40:46.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:40:46.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:40:46.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:40:46.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:40:46.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:40:46.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:40:46.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:40:46.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:46.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:40:46.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:40:46.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:40:46.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:40:46.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:40:46.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:40:46.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:40:46.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:40:46.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:40:46.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:40:46.561 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.561 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.561 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.561 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 93.988316122000001, 2024-04-22T18:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:46.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:46.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:40:46.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:46.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:40:46.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:40:46.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:40:46.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:40:46.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:46.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:40:46.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:40:46.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:40:46.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:40:46.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.567 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.567 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:46.567 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:46.567 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 93.988296200999997, 2024-04-22T18:40:46.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:40:46.567 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.567 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:46.567 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.567 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.567 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.567 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:40:46.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.568 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:46.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:46.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:46.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:40:46.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:40:46.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:40:46.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:40:46.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:40:46.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:40:46.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:40:46.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:40:46.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:46.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:40:46.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:40:46.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:40:46.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:40:46.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:40:46.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:40:46.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:40:46.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:40:46.571 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.571 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.571 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.572 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:46.572 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:46.572 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 88.678972435999995, 2024-04-22T18:40:46.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:40:46.572 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.572 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.572 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:46.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:46.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:40:46.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:46.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:40:46.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:40:46.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:40:46.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:40:46.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:40:46.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:40:46.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:40:46.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:40:46.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:40:46.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:40:46.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:46.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:40:46.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:40:46.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:40:46.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:40:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:40:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:40:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:40:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:40:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:40:46.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:40:46.577 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.577 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.577 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.577 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:46.577 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:46.577 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 88.678764900000004, 2024-04-22T18:40:46.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:40:46.577 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.577 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.577 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:46.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:46.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:40:46.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:46.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:40:46.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:40:46.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:40:46.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:40:46.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:40:46.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:40:46.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:40:46.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:40:46.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:40:46.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:40:46.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:46.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:40:46.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:40:46.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:40:46.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:40:46.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:40:46.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:40:46.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:40:46.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:40:46.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:40:46.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:40:46.582 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.582 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.582 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.582 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:46.582 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:46.582 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.813115167000007, 2024-04-22T18:40:46.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:40:46.582 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.582 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.583 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:46.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:46.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:40:46.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:46.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:40:46.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:40:46.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:40:46.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:40:46.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:40:46.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:40:46.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:40:46.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:40:46.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:40:46.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:40:46.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:46.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:40:46.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:40:46.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:40:46.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:40:46.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:40:46.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:40:46.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:40:46.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:40:46.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:40:46.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:40:46.587 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.587 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.587 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.587 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:46.587 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:46.588 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.599276782000004, 2024-04-22T18:40:46.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:40:46.588 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.588 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.588 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:46.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:46.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:40:46.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:46.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:40:46.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:40:46.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:40:46.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:40:46.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:40:46.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:40:46.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:40:46.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:40:46.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:40:46.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:40:46.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:46.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:40:46.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:40:46.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:40:46.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:40:46.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:40:46.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:40:46.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:40:46.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:40:46.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:40:46.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:40:46.592 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.592 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.592 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.593 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:46.593 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:46.593 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.599243372000004, 2024-04-22T18:40:46.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:40:46.593 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.593 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.593 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:46.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:46.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:46.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:40:46.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:40:46.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:40:46.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:40:46.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:40:46.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:40:46.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:40:46.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:40:46.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:40:46.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:40:46.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:46.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:40:46.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:40:46.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:40:46.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:40:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:40:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:40:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:40:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:40:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:40:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:40:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.540360718000002, 2024-04-22T18:40:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:40:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:46.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:46.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:40:46.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:46.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:40:46.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:40:46.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:40:46.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:40:46.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:40:46.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:40:46.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:40:46.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:40:46.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:40:46.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:40:46.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:46.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:40:46.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:40:46.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:40:46.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:40:46.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:40:46.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:40:46.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:40:46.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:40:46.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:40:46.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:40:46.603 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.603 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.603 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.603 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:46.603 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:46.603 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.540222102000001, 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:46.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:40:46.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:40:46.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:40:46.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:40:46.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:40:46.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:40:46.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:40:46.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:40:46.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:40:46.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:40:46.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:46.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:40:46.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:40:46.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:40:46.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:40:46.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:40:46.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:40:46.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:40:46.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:40:46.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:40:46.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:40:46.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.608 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.608 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:46.609 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:46.609 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.540203098999996, 2024-04-22T18:40:46.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:40:46.609 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.609 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.609 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:46.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:46.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:46.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:40:46.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:40:46.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:40:46.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:40:46.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:40:46.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:40:46.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:40:46.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:40:46.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:40:46.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:40:46.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:46.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:40:46.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:40:46.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:40:46.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:40:46.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:40:46.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:40:46.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:40:46.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:40:46.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:40:46.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:40:46.613 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.613 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.614 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.614 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:46.614 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:46.614 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.919190565999997, 2024-04-22T18:40:46.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:40:46.614 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.614 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.614 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.614 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.614 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.614 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:40:46.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.615 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:46.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:46.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:40:46.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:46.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:40:46.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:40:46.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:40:46.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:40:46.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:40:46.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:40:46.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:40:46.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:40:46.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:40:46.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:40:46.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:46.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:40:46.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:40:46.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:40:46.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:40:46.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:40:46.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:40:46.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:40:46.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:40:46.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:40:46.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:40:46.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.619 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.619 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:46.619 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.918670974999998, 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:46.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:40:46.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:46.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:40:46.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:40:46.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:40:46.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:40:46.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:40:46.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:40:46.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:40:46.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:40:46.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:40:46.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:40:46.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:46.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:40:46.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:40:46.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:40:46.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:40:46.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:46.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:40:46.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:40:46.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:46.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:40:46.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:40:46.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:40:46.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:40:46.627 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.627 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.627 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.627 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:46.627 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:46.627 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.818518092000005, 2024-04-22T18:40:46.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:40:46.627 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.627 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:46.627 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.627 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.628 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.628 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:40:46.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.628 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:46.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:46.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:40:46.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:46.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:40:46.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:40:46.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:40:46.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:40:46.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:40:46.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:40:46.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:40:46.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:40:46.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:46.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:40:46.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:40:46.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:40:46.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:40:46.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:40:46.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:40:46.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:46.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:40:46.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:40:46.632 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.632 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.632 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.632 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.632 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.632 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:46.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:46.634 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:46.634+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:46.635 INFO:tasks.rados.rados.0.smithi089.stdout:1598: finishing set_chunk to oid 4 2024-04-22T18:40:46.635 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 4 set_chunk returned error code -95 offset: 1286144 length: 13312 2024-04-22T18:40:46.635 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 0 (ObjNum 269 snap 0 seq_num 269) dirty exists 2024-04-22T18:40:46.635 INFO:tasks.rados.rados.0.smithi089.stdout:1599: finishing write tid 1 to smithi08937929-33 2024-04-22T18:40:46.635 INFO:tasks.rados.rados.0.smithi089.stdout:1599: finishing write tid 2 to smithi08937929-33 2024-04-22T18:40:46.635 INFO:tasks.rados.rados.0.smithi089.stdout:1599: finishing write tid 3 to smithi08937929-33 2024-04-22T18:40:46.635 INFO:tasks.rados.rados.0.smithi089.stdout:1599: finishing write tid 4 to smithi08937929-33 2024-04-22T18:40:46.635 INFO:tasks.rados.rados.0.smithi089.stdout:1606: got (2) No such file or directory 2024-04-22T18:40:46.636 INFO:tasks.rados.rados.0.smithi089.stdout:1606: got expected ENOENT (src dne) 2024-04-22T18:40:46.636 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 94 (ObjNum 129 snap 0 seq_num 2414200864) clean dne 2024-04-22T18:40:46.636 INFO:tasks.rados.rados.0.smithi089.stdout:1599: finishing write tid 5 to smithi08937929-33 2024-04-22T18:40:46.636 INFO:tasks.rados.rados.0.smithi089.stdout:1599: oid 33 updating version 0 to 55 2024-04-22T18:40:46.636 INFO:tasks.rados.rados.0.smithi089.stdout:1599: oid 33 version 55 is already newer than 53 2024-04-22T18:40:46.636 INFO:tasks.rados.rados.0.smithi089.stdout:1599: oid 33 version 55 is already newer than 54 2024-04-22T18:40:46.636 INFO:tasks.rados.rados.0.smithi089.stdout:1599: oid 33 updating version 55 to 56 2024-04-22T18:40:46.636 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 56 (ObjNum 319 snap 0 seq_num 319) dirty exists 2024-04-22T18:40:46.636 INFO:tasks.rados.rados.0.smithi089.stdout:1599: left oid 33 (ObjNum 319 snap 0 seq_num 319) 2024-04-22T18:40:46.636 INFO:tasks.rados.rados.0.smithi089.stdout:1605: finishing write tid 1 to smithi08937929-17 2024-04-22T18:40:46.636 INFO:tasks.rados.rados.0.smithi089.stdout:1605: finishing write tid 2 to smithi08937929-17 2024-04-22T18:40:46.636 INFO:tasks.rados.rados.0.smithi089.stdout:1605: finishing write tid 3 to smithi08937929-17 2024-04-22T18:40:46.636 INFO:tasks.rados.rados.0.smithi089.stdout:1605: finishing write tid 4 to smithi08937929-17 2024-04-22T18:40:46.636 INFO:tasks.rados.rados.0.smithi089.stdout:1591: done (11 left) 2024-04-22T18:40:46.636 INFO:tasks.rados.rados.0.smithi089.stdout:1594: done (10 left) 2024-04-22T18:40:46.637 INFO:tasks.rados.rados.0.smithi089.stdout:1597: done (9 left) 2024-04-22T18:40:46.637 INFO:tasks.rados.rados.0.smithi089.stdout:1598: done (8 left) 2024-04-22T18:40:46.637 INFO:tasks.rados.rados.0.smithi089.stdout:1599: done (7 left) 2024-04-22T18:40:46.637 INFO:tasks.rados.rados.0.smithi089.stdout:1600: done (6 left) 2024-04-22T18:40:46.637 INFO:tasks.rados.rados.0.smithi089.stdout:1601: done (5 left) 2024-04-22T18:40:46.637 INFO:tasks.rados.rados.0.smithi089.stdout:1602: done (4 left) 2024-04-22T18:40:46.637 INFO:tasks.rados.rados.0.smithi089.stdout:1604: done (3 left) 2024-04-22T18:40:46.637 INFO:tasks.rados.rados.0.smithi089.stdout:1606: done (2 left) 2024-04-22T18:40:46.637 INFO:tasks.rados.rados.0.smithi089.stdout:1607: done (1 left) 2024-04-22T18:40:46.637 INFO:tasks.rados.rados.0.smithi089.stdout:1608: write (excl) oid 47 current snap is 0 2024-04-22T18:40:46.637 INFO:tasks.rados.rados.0.smithi089.stdout:1605: finishing write tid 5 to smithi08937929-17 2024-04-22T18:40:46.637 INFO:tasks.rados.rados.0.smithi089.stdout:1605: oid 17 updating version 0 to 85 2024-04-22T18:40:46.637 INFO:tasks.rados.rados.0.smithi089.stdout:1605: oid 17 version 85 is already newer than 84 2024-04-22T18:40:46.637 INFO:tasks.rados.rados.0.smithi089.stdout:1605: oid 17 version 85 is already newer than 82 2024-04-22T18:40:46.637 INFO:tasks.rados.rados.0.smithi089.stdout:1605: oid 17 version 85 is already newer than 83 2024-04-22T18:40:46.638 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 85 (ObjNum 321 snap 0 seq_num 321) dirty exists 2024-04-22T18:40:46.638 INFO:tasks.rados.rados.0.smithi089.stdout:1605: left oid 17 (ObjNum 321 snap 0 seq_num 321) 2024-04-22T18:40:46.638 INFO:tasks.rados.rados.0.smithi089.stdout:1608: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-04-22T18:40:46.640 INFO:tasks.rados.rados.0.smithi089.stdout:1608: writing smithi08937929-47 from 628661 to 1420914 tid 1 2024-04-22T18:40:46.647 INFO:tasks.rados.rados.0.smithi089.stdout:1608: writing smithi08937929-47 from 2184625 to 2606403 tid 2 2024-04-22T18:40:46.652 INFO:tasks.rados.rados.0.smithi089.stdout:1608: writing smithi08937929-47 from 3403509 to 3428661 tid 3 2024-04-22T18:40:46.653 INFO:tasks.rados.rados.0.smithi089.stdout:1605: done (1 left) 2024-04-22T18:40:46.653 INFO:tasks.rados.rados.0.smithi089.stdout:1609: read oid 18 snap -1 2024-04-22T18:40:46.653 INFO:tasks.rados.rados.0.smithi089.stdout:1609: expect (ObjNum 307 snap 0 seq_num 307) 2024-04-22T18:40:46.653 INFO:tasks.rados.rados.0.smithi089.stdout:1610: tier_evict oid 20 2024-04-22T18:40:46.653 INFO:tasks.rados.rados.0.smithi089.stdout:1610: tier_evict oid 20 snap -1 2024-04-22T18:40:46.653 INFO:tasks.rados.rados.0.smithi089.stdout:1611: tier_promote oid 22 2024-04-22T18:40:46.653 INFO:tasks.rados.rados.0.smithi089.stdout:1612: read oid 24 snap -1 2024-04-22T18:40:46.653 INFO:tasks.rados.rados.0.smithi089.stdout:1612: expect deleted 2024-04-22T18:40:46.653 INFO:tasks.rados.rados.0.smithi089.stdout:1613: read oid 44 snap -1 2024-04-22T18:40:46.653 INFO:tasks.rados.rados.0.smithi089.stdout:1613: expect (ObjNum 187 snap 0 seq_num 187) 2024-04-22T18:40:46.653 INFO:tasks.rados.rados.0.smithi089.stdout:1614: set_chunk oid 6 target oid 2024-04-22T18:40:46.733 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:46.734+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:46.771 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:40:46.779 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:46.779 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:46.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.779 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3781 4.a 4:537c89f8:::smithi08937929-47:head [write 3403509~25152 in=25152b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-04-22T18:40:46.779 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:46.750470+0000", 2024-04-22T18:40:46.779 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.030223904999999999, 2024-04-22T18:40:46.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.030242207, 2024-04-22T18:40:46.779 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.779 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:40:46.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3781 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750470+0000", 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750470+0000", 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750467+0000", 2024-04-22T18:40:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750509+0000", 2024-04-22T18:40:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1916999999999999e-05 2024-04-22T18:40:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750517+0000", 2024-04-22T18:40:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9170000000000006e-06 2024-04-22T18:40:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750529+0000", 2024-04-22T18:40:46.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1565e-05 2024-04-22T18:40:46.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.752317+0000", 2024-04-22T18:40:46.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001788651 2024-04-22T18:40:46.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.752369+0000", 2024-04-22T18:40:46.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1799000000000003e-05 2024-04-22T18:40:46.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.752461+0000", 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1996000000000005e-05 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.753364+0000", 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090294599999999996 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3782 4.a 4:537c89f8:::smithi08937929-47:head [setxattr _header (56) in=63b,truncate 3428661] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:46.750533+0000", 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.030160869999999999, 2024-04-22T18:40:46.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.030225568000000001, 2024-04-22T18:40:46.784 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.784 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:40:46.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.784 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3782 2024-04-22T18:40:46.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.784 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750533+0000", 2024-04-22T18:40:46.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750533+0000", 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750532+0000", 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750536+0000", 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5389999999999999e-06 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750539+0000", 2024-04-22T18:40:46.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.988e-06 2024-04-22T18:40:46.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750542+0000", 2024-04-22T18:40:46.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:40:46.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.752776+0000", 2024-04-22T18:40:46.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022336470000000001 2024-04-22T18:40:46.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.752818+0000", 2024-04-22T18:40:46.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2747000000000003e-05 2024-04-22T18:40:46.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:46.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.752885+0000", 2024-04-22T18:40:46.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6433999999999994e-05 2024-04-22T18:40:46.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.753374+0000", 2024-04-22T18:40:46.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00048894300000000004 2024-04-22T18:40:46.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.787 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.788 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.788 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3783 4.a 4:537c89f8:::smithi08937929-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e114)", 2024-04-22T18:40:46.788 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:46.750549+0000", 2024-04-22T18:40:46.788 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.030144691000000001, 2024-04-22T18:40:46.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.030249133000000001, 2024-04-22T18:40:46.788 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.788 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for rw locks", 2024-04-22T18:40:46.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.788 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3783 2024-04-22T18:40:46.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.789 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750549+0000", 2024-04-22T18:40:46.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750549+0000", 2024-04-22T18:40:46.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750549+0000", 2024-04-22T18:40:46.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750550+0000", 2024-04-22T18:40:46.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.781e-06 2024-04-22T18:40:46.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750554+0000", 2024-04-22T18:40:46.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2720000000000002e-06 2024-04-22T18:40:46.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750557+0000", 2024-04-22T18:40:46.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2370000000000001e-06 2024-04-22T18:40:46.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.753044+0000", 2024-04-22T18:40:46.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002486697 2024-04-22T18:40:46.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:46.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.753095+0000", 2024-04-22T18:40:46.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0954e-05 2024-04-22T18:40:46.791 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.791 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.791 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3795 4.a 4.29fe955a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e114)", 2024-04-22T18:40:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:46.750593+0000", 2024-04-22T18:40:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.030100702999999999, 2024-04-22T18:40:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.030237204, 2024-04-22T18:40:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3795 2024-04-22T18:40:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750593+0000", 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750593+0000", 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750593+0000", 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750594+0000", 2024-04-22T18:40:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5549999999999999e-06 2024-04-22T18:40:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750598+0000", 2024-04-22T18:40:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3699999999999999e-06 2024-04-22T18:40:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750600+0000", 2024-04-22T18:40:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7700000000000002e-06 2024-04-22T18:40:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGRecoveryDeleteReply(3.3 e114,103 3:dfd120ed:::b0cb569310f4fda0afeb047106fcd64b806324c8b0d4c8d1e9b4754fdaa8ceff:head,101'277)", 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:46.759703+0000", 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.020990753000000001, 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.021154613999999999, 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.759703+0000", 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.759703+0000", 2024-04-22T18:40:46.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.759702+0000", 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.759707+0000", 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6829999999999999e-06 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.759716+0000", 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.978e-06 2024-04-22T18:40:46.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.759722+0000", 2024-04-22T18:40:46.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7989999999999999e-06 2024-04-22T18:40:46.797 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.797 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.797 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.797 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3780 4.a 4:537c89f8:::smithi08937929-47:head [write 2184625~421778 in=421778b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-04-22T18:40:46.797 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:46.650021+0000", 2024-04-22T18:40:46.797 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.130672446, 2024-04-22T18:40:46.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.130861593, 2024-04-22T18:40:46.797 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:46.797 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:40:46.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:46.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:46.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:46.798 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3780 2024-04-22T18:40:46.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.798 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:46.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:46.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.650021+0000", 2024-04-22T18:40:46.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:46.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.650021+0000", 2024-04-22T18:40:46.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.650019+0000", 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.650335+0000", 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00031619599999999998 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.650349+0000", 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4035e-05 2024-04-22T18:40:46.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750451+0000", 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100101257 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750499+0000", 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7849999999999998e-05 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750601+0000", 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000102216 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:46.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.750758+0000", 2024-04-22T18:40:46.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015737300000000001 2024-04-22T18:40:46.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:46.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:46.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:46.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:46.753025+0000", 2024-04-22T18:40:46.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022666140000000001 2024-04-22T18:40:46.801 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.801 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:46.801 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.801 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:46.801 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:40:46.801 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 6 2024-04-22T18:40:46.801 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:46.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:46.834 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:46.836+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:40:46.893 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:46.894 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:46.910 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:46.910 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:46.926 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:46.926 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:46.935 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:46.936+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:46.941 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:46.942 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:46.943 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:46.943 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:46.943 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:46.943 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:46.944 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:46.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:46.958 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:46.958 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:46.974 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:46.975 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:46.991 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:46.991 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:47.008 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:47.008 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:47.037 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:47.038+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:47.047 INFO:tasks.rados.rados.0.smithi089.stdout:1614: set_chunk oid 6 offset: 2406912 length: 882176 target oid 54c5ecd640f68b96296af6c9e57f20adb69a8a8045a9f069a536a00b5ed4877c offset: 0 snap -1 2024-04-22T18:40:47.047 INFO:tasks.rados.rados.0.smithi089.stdout:1615: write oid 48 current snap is 0 2024-04-22T18:40:47.048 INFO:tasks.rados.rados.0.smithi089.stdout:1610: got (22) Invalid argument 2024-04-22T18:40:47.048 INFO:tasks.rados.rados.0.smithi089.stdout:1615: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2024-04-22T18:40:47.053 INFO:tasks.rados.rados.0.smithi089.stdout:1615: writing smithi08937929-48 from 645468 to 1112970 tid 1 2024-04-22T18:40:47.059 INFO:tasks.rados.rados.0.smithi089.stdout:1615: writing smithi08937929-48 from 1726754 to 2292190 tid 2 2024-04-22T18:40:47.066 INFO:tasks.rados.rados.0.smithi089.stdout:1615: writing smithi08937929-48 from 2798226 to 3395064 tid 3 2024-04-22T18:40:47.066 INFO:tasks.rados.rados.0.smithi089.stdout:1610: done (7 left) 2024-04-22T18:40:47.066 INFO:tasks.rados.rados.0.smithi089.stdout:1616: set_chunk oid 7 target oid 2024-04-22T18:40:47.095 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:47.095 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:47.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:47.146 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:47.147+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:47.152 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:47.153+0000 7fa766d7d640 -1 osd.5 114 handle_osd_map injecting map crc failure 2024-04-22T18:40:47.238 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:47.239+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:47.248 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:47.248 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:47.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:47.340 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:47.341+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:47.365 INFO:tasks.rados.rados.0.smithi089.stdout:1616: set_chunk oid 7 offset: 830464 length: 1515520 target oid 81ffa4316d87948045c84090422eacbfd050f1dff0a418409b403122ff7da426 offset: 0 snap -1 2024-04-22T18:40:47.365 INFO:tasks.rados.rados.0.smithi089.stdout:1617: write (excl) oid 19 current snap is 0 2024-04-22T18:40:47.365 INFO:tasks.rados.rados.0.smithi089.stdout:1617: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2024-04-22T18:40:47.371 INFO:tasks.rados.rados.0.smithi089.stdout:1617: writing smithi08937929-19 from 662275 to 1205026 tid 1 2024-04-22T18:40:47.377 INFO:tasks.rados.rados.0.smithi089.stdout:1617: writing smithi08937929-19 from 1785236 to 2210683 tid 2 2024-04-22T18:40:47.383 INFO:tasks.rados.rados.0.smithi089.stdout:1617: writing smithi08937929-19 from 2942002 to 3462275 tid 3 2024-04-22T18:40:47.383 INFO:tasks.rados.rados.0.smithi089.stdout:1618: write (excl) oid 25 current snap is 0 2024-04-22T18:40:47.384 INFO:tasks.rados.rados.0.smithi089.stdout:1618: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2024-04-22T18:40:47.390 INFO:tasks.rados.rados.0.smithi089.stdout:1618: writing smithi08937929-25 from 679082 to 1297082 tid 1 2024-04-22T18:40:47.396 INFO:tasks.rados.rados.0.smithi089.stdout:1618: writing smithi08937929-25 from 1843718 to 2412823 tid 2 2024-04-22T18:40:47.401 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:47.401 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:47.402 INFO:tasks.rados.rados.0.smithi089.stdout:1618: writing smithi08937929-25 from 2853072 to 3479082 tid 3 2024-04-22T18:40:47.402 INFO:tasks.rados.rados.0.smithi089.stdout:1619: read oid 45 snap -1 2024-04-22T18:40:47.402 INFO:tasks.rados.rados.0.smithi089.stdout:1619: expect (ObjNum 318 snap 0 seq_num 318) 2024-04-22T18:40:47.402 INFO:tasks.rados.rados.0.smithi089.stdout:1620: write (excl) oid 30 current snap is 0 2024-04-22T18:40:47.402 INFO:tasks.rados.rados.0.smithi089.stdout:1620: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2024-04-22T18:40:47.408 INFO:tasks.rados.rados.0.smithi089.stdout:1620: writing smithi08937929-30 from 695889 to 1105491 tid 1 2024-04-22T18:40:47.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:47.414 INFO:tasks.rados.rados.0.smithi089.stdout:1620: writing smithi08937929-30 from 1618553 to 2331316 tid 2 2024-04-22T18:40:47.421 INFO:tasks.rados.rados.0.smithi089.stdout:1620: writing smithi08937929-30 from 2996848 to 3495889 tid 3 2024-04-22T18:40:47.421 INFO:tasks.rados.rados.0.smithi089.stdout:1621: tier_evict oid 38 2024-04-22T18:40:47.421 INFO:tasks.rados.rados.0.smithi089.stdout:1621: tier_evict oid 38 snap -1 2024-04-22T18:40:47.421 INFO:tasks.rados.rados.0.smithi089.stdout:1608: finishing write tid 1 to smithi08937929-47 2024-04-22T18:40:47.421 INFO:tasks.rados.rados.0.smithi089.stdout:1611: got (0) Success 2024-04-22T18:40:47.422 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 112 (ObjNum 303 snap 0 seq_num 303) dirty exists 2024-04-22T18:40:47.422 INFO:tasks.rados.rados.0.smithi089.stdout:1613: expect (ObjNum 187 snap 0 seq_num 187) 2024-04-22T18:40:47.440 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:47.441+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:47.512 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:47.513 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:47.522 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:47.564 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:47.566+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:47.606 INFO:tasks.rados.rados.0.smithi089.stdout:1609: expect (ObjNum 307 snap 0 seq_num 307) 2024-04-22T18:40:47.627 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:47.627 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:47.636 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:47.641 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:47.642+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:47.742 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:47.743+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:47.776 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:47.777 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:47.785 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:47.810 INFO:tasks.rados.rados.0.smithi089.stdout:1608: finishing write tid 2 to smithi08937929-47 2024-04-22T18:40:47.810 INFO:tasks.rados.rados.0.smithi089.stdout:1608: finishing write tid 3 to smithi08937929-47 2024-04-22T18:40:47.811 INFO:tasks.rados.rados.0.smithi089.stdout:1608: finishing write tid 4 to smithi08937929-47 2024-04-22T18:40:47.811 INFO:tasks.rados.rados.0.smithi089.stdout:1608: finishing write tid 5 to smithi08937929-47 2024-04-22T18:40:47.811 INFO:tasks.rados.rados.0.smithi089.stdout:1608: oid 47 updating version 0 to 98 2024-04-22T18:40:47.811 INFO:tasks.rados.rados.0.smithi089.stdout:1608: oid 47 updating version 98 to 99 2024-04-22T18:40:47.811 INFO:tasks.rados.rados.0.smithi089.stdout:1608: oid 47 version 99 is already newer than 97 2024-04-22T18:40:47.811 INFO:tasks.rados.rados.0.smithi089.stdout:1608: oid 47 version 99 is already newer than 96 2024-04-22T18:40:47.811 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 99 (ObjNum 323 snap 0 seq_num 323) dirty exists 2024-04-22T18:40:47.811 INFO:tasks.rados.rados.0.smithi089.stdout:1608: left oid 47 (ObjNum 323 snap 0 seq_num 323) 2024-04-22T18:40:47.811 INFO:tasks.rados.rados.0.smithi089.stdout:1615: finishing write tid 1 to smithi08937929-48 2024-04-22T18:40:47.811 INFO:tasks.rados.rados.0.smithi089.stdout:1608: done (12 left) 2024-04-22T18:40:47.811 INFO:tasks.rados.rados.0.smithi089.stdout:1609: done (11 left) 2024-04-22T18:40:47.811 INFO:tasks.rados.rados.0.smithi089.stdout:1611: done (10 left) 2024-04-22T18:40:47.811 INFO:tasks.rados.rados.0.smithi089.stdout:1612: done (9 left) 2024-04-22T18:40:47.811 INFO:tasks.rados.rados.0.smithi089.stdout:1613: done (8 left) 2024-04-22T18:40:47.811 INFO:tasks.rados.rados.0.smithi089.stdout:1622: set_chunk oid 47 target oid 2024-04-22T18:40:47.842 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:47.844+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:47.913 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:47.913 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:47.922 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:47.943 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:47.944+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:48.023 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:48.023 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:48.032 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:48.044 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:48.045+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:48.123 INFO:tasks.rados.rados.0.smithi089.stdout:1622: set_chunk oid 47 offset: 1840128 length: 1524736 target oid 651c29bb878164358af51372469169f917dac8d29e40997ee3f6b0b35412f1ef offset: 0 snap -1 2024-04-22T18:40:48.123 INFO:tasks.rados.rados.0.smithi089.stdout:1623: read oid 5 snap -1 2024-04-22T18:40:48.123 INFO:tasks.rados.rados.0.smithi089.stdout:1623: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:40:48.123 INFO:tasks.rados.rados.0.smithi089.stdout:1624: delete oid 20 current snap is 0 2024-04-22T18:40:48.123 INFO:tasks.rados.rados.0.smithi089.stdout:1621: got (22) Invalid argument 2024-04-22T18:40:48.123 INFO:tasks.rados.rados.0.smithi089.stdout:1615: finishing write tid 2 to smithi08937929-48 2024-04-22T18:40:48.123 INFO:tasks.rados.rados.0.smithi089.stdout:1615: finishing write tid 3 to smithi08937929-48 2024-04-22T18:40:48.123 INFO:tasks.rados.rados.0.smithi089.stdout:1615: finishing write tid 4 to smithi08937929-48 2024-04-22T18:40:48.123 INFO:tasks.rados.rados.0.smithi089.stdout:1615: finishing write tid 5 to smithi08937929-48 2024-04-22T18:40:48.123 INFO:tasks.rados.rados.0.smithi089.stdout:1615: oid 48 updating version 0 to 86 2024-04-22T18:40:48.123 INFO:tasks.rados.rados.0.smithi089.stdout:1615: oid 48 version 86 is already newer than 84 2024-04-22T18:40:48.123 INFO:tasks.rados.rados.0.smithi089.stdout:1615: oid 48 version 86 is already newer than 83 2024-04-22T18:40:48.123 INFO:tasks.rados.rados.0.smithi089.stdout:1615: oid 48 version 86 is already newer than 85 2024-04-22T18:40:48.124 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 86 (ObjNum 324 snap 0 seq_num 324) dirty exists 2024-04-22T18:40:48.124 INFO:tasks.rados.rados.0.smithi089.stdout:1615: left oid 48 (ObjNum 324 snap 0 seq_num 324) 2024-04-22T18:40:48.124 INFO:tasks.rados.rados.0.smithi089.stdout:1617: finishing write tid 1 to smithi08937929-19 2024-04-22T18:40:48.124 INFO:tasks.rados.rados.0.smithi089.stdout:1617: finishing write tid 2 to smithi08937929-19 2024-04-22T18:40:48.124 INFO:tasks.rados.rados.0.smithi089.stdout:1617: finishing write tid 3 to smithi08937929-19 2024-04-22T18:40:48.124 INFO:tasks.rados.rados.0.smithi089.stdout:1617: finishing write tid 4 to smithi08937929-19 2024-04-22T18:40:48.124 INFO:tasks.rados.rados.0.smithi089.stdout:1619: expect (ObjNum 318 snap 0 seq_num 318) 2024-04-22T18:40:48.135 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:48.135 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:48.145 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:48.145 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:48.146+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:48.245 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:48.247+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:48.287 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:48.287 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:48.297 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:48.346 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:48.347+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:48.391 INFO:tasks.rados.rados.0.smithi089.stdout:1617: finishing write tid 5 to smithi08937929-19 2024-04-22T18:40:48.391 INFO:tasks.rados.rados.0.smithi089.stdout:1617: oid 19 updating version 0 to 129 2024-04-22T18:40:48.391 INFO:tasks.rados.rados.0.smithi089.stdout:1617: oid 19 updating version 129 to 131 2024-04-22T18:40:48.391 INFO:tasks.rados.rados.0.smithi089.stdout:1617: oid 19 version 131 is already newer than 128 2024-04-22T18:40:48.391 INFO:tasks.rados.rados.0.smithi089.stdout:1617: oid 19 version 131 is already newer than 130 2024-04-22T18:40:48.391 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 131 (ObjNum 325 snap 0 seq_num 325) dirty exists 2024-04-22T18:40:48.391 INFO:tasks.rados.rados.0.smithi089.stdout:1617: left oid 19 (ObjNum 325 snap 0 seq_num 325) 2024-04-22T18:40:48.391 INFO:tasks.rados.rados.0.smithi089.stdout:1614: finishing set_chunk to oid 6 2024-04-22T18:40:48.391 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 51 (ObjNum 199 snap 0 seq_num 199) dirty exists 2024-04-22T18:40:48.391 INFO:tasks.rados.rados.0.smithi089.stdout:1620: finishing write tid 1 to smithi08937929-30 2024-04-22T18:40:48.391 INFO:tasks.rados.rados.0.smithi089.stdout:1620: finishing write tid 2 to smithi08937929-30 2024-04-22T18:40:48.392 INFO:tasks.rados.rados.0.smithi089.stdout:1620: finishing write tid 3 to smithi08937929-30 2024-04-22T18:40:48.392 INFO:tasks.rados.rados.0.smithi089.stdout:1614: done (10 left) 2024-04-22T18:40:48.392 INFO:tasks.rados.rados.0.smithi089.stdout:1615: done (9 left) 2024-04-22T18:40:48.392 INFO:tasks.rados.rados.0.smithi089.stdout:1617: done (8 left) 2024-04-22T18:40:48.392 INFO:tasks.rados.rados.0.smithi089.stdout:1619: done (7 left) 2024-04-22T18:40:48.392 INFO:tasks.rados.rados.0.smithi089.stdout:1621: done (6 left) 2024-04-22T18:40:48.392 INFO:tasks.rados.rados.0.smithi089.stdout:1624: done (5 left) 2024-04-22T18:40:48.392 INFO:tasks.rados.rados.0.smithi089.stdout:1625: read oid 42 snap -1 2024-04-22T18:40:48.392 INFO:tasks.rados.rados.0.smithi089.stdout:1625: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:48.392 INFO:tasks.rados.rados.0.smithi089.stdout:1626: write oid 39 current snap is 0 2024-04-22T18:40:48.392 INFO:tasks.rados.rados.0.smithi089.stdout:1620: finishing write tid 4 to smithi08937929-30 2024-04-22T18:40:48.392 INFO:tasks.rados.rados.0.smithi089.stdout:1620: finishing write tid 5 to smithi08937929-30 2024-04-22T18:40:48.393 INFO:tasks.rados.rados.0.smithi089.stdout:1620: oid 30 updating version 0 to 53 2024-04-22T18:40:48.393 INFO:tasks.rados.rados.0.smithi089.stdout:1620: oid 30 version 53 is already newer than 52 2024-04-22T18:40:48.393 INFO:tasks.rados.rados.0.smithi089.stdout:1620: oid 30 updating version 53 to 54 2024-04-22T18:40:48.393 INFO:tasks.rados.rados.0.smithi089.stdout:1620: oid 30 version 54 is already newer than 51 2024-04-22T18:40:48.393 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 54 (ObjNum 327 snap 0 seq_num 327) dirty exists 2024-04-22T18:40:48.393 INFO:tasks.rados.rados.0.smithi089.stdout:1620: left oid 30 (ObjNum 327 snap 0 seq_num 327) 2024-04-22T18:40:48.393 INFO:tasks.rados.rados.0.smithi089.stdout:1616: finishing set_chunk to oid 7 2024-04-22T18:40:48.393 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 50 (ObjNum 309 snap 0 seq_num 309) dirty exists 2024-04-22T18:40:48.393 INFO:tasks.rados.rados.0.smithi089.stdout:1618: finishing write tid 1 to smithi08937929-25 2024-04-22T18:40:48.393 INFO:tasks.rados.rados.0.smithi089.stdout:1618: finishing write tid 2 to smithi08937929-25 2024-04-22T18:40:48.393 INFO:tasks.rados.rados.0.smithi089.stdout:1618: finishing write tid 3 to smithi08937929-25 2024-04-22T18:40:48.393 INFO:tasks.rados.rados.0.smithi089.stdout:1626: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2024-04-22T18:40:48.397 INFO:tasks.rados.rados.0.smithi089.stdout:1626: writing smithi08937929-39 from 729503 to 1289603 tid 1 2024-04-22T18:40:48.405 INFO:tasks.rados.rados.0.smithi089.stdout:1626: writing smithi08937929-39 from 2019164 to 2735596 tid 2 2024-04-22T18:40:48.409 INFO:tasks.rados.rados.0.smithi089.stdout:1626: writing smithi08937929-39 from 3335341 to 3529503 tid 3 2024-04-22T18:40:48.409 INFO:tasks.rados.rados.0.smithi089.stdout:1616: done (6 left) 2024-04-22T18:40:48.409 INFO:tasks.rados.rados.0.smithi089.stdout:1620: done (5 left) 2024-04-22T18:40:48.410 INFO:tasks.rados.rados.0.smithi089.stdout:1627: set_chunk oid 22 target oid 2024-04-22T18:40:48.441 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:48.441 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:48.447 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:48.448+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:48.450 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:48.548 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:48.549+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:48.552 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:48.552 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:48.561 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:48.649 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:48.651+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:48.703 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:48.704 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:48.704 INFO:tasks.rados.rados.0.smithi089.stdout:1627: set_chunk oid 22 offset: 315392 length: 742912 target oid 4a20cf2d05d5beef11be66e5e94200cf4516e4b5cd937fdd98aef7bdd3759bac offset: 0 snap -1 2024-04-22T18:40:48.704 INFO:tasks.rados.rados.0.smithi089.stdout:1628: set_chunk oid 30 target oid 2024-04-22T18:40:48.715 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:48.724 INFO:tasks.rados.rados.0.smithi089.stdout:1628: set_chunk oid 30 offset: 249344 length: 3165696 target oid b4060b13a742af6f9ae129fd4fd3d85095b2fad88d7df7d86078582a706d7b61 offset: 0 snap -1 2024-04-22T18:40:48.725 INFO:tasks.rados.rados.0.smithi089.stdout:1629: write oid 13 current snap is 0 2024-04-22T18:40:48.725 INFO:tasks.rados.rados.0.smithi089.stdout:1618: finishing write tid 4 to smithi08937929-25 2024-04-22T18:40:48.725 INFO:tasks.rados.rados.0.smithi089.stdout:1618: finishing write tid 5 to smithi08937929-25 2024-04-22T18:40:48.725 INFO:tasks.rados.rados.0.smithi089.stdout:1618: oid 25 updating version 0 to 78 2024-04-22T18:40:48.725 INFO:tasks.rados.rados.0.smithi089.stdout:1618: oid 25 version 78 is already newer than 77 2024-04-22T18:40:48.725 INFO:tasks.rados.rados.0.smithi089.stdout:1618: oid 25 updating version 78 to 80 2024-04-22T18:40:48.725 INFO:tasks.rados.rados.0.smithi089.stdout:1618: oid 25 version 80 is already newer than 79 2024-04-22T18:40:48.725 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 80 (ObjNum 326 snap 0 seq_num 326) dirty exists 2024-04-22T18:40:48.725 INFO:tasks.rados.rados.0.smithi089.stdout:1618: left oid 25 (ObjNum 326 snap 0 seq_num 326) 2024-04-22T18:40:48.726 INFO:tasks.rados.rados.0.smithi089.stdout:1623: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:40:48.750 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:48.751+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:48.818 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:48.818 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:48.827 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:48.850 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:48.851+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:48.862 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:40:48.951 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:48.952+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:48.966 INFO:tasks.rados.rados.0.smithi089.stdout:1622: finishing set_chunk to oid 47 2024-04-22T18:40:48.966 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 99 (ObjNum 323 snap 0 seq_num 323) dirty exists 2024-04-22T18:40:48.966 INFO:tasks.rados.rados.0.smithi089.stdout:1629: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2024-04-22T18:40:48.972 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:48.972 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:48.974 INFO:tasks.rados.rados.0.smithi089.stdout:1629: writing smithi08937929-13 from 746310 to 1381659 tid 1 2024-04-22T18:40:48.982 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:48.982 INFO:tasks.rados.rados.0.smithi089.stdout:1629: writing smithi08937929-13 from 2077646 to 2654089 tid 2 2024-04-22T18:40:48.988 INFO:tasks.rados.rados.0.smithi089.stdout:1629: writing smithi08937929-13 from 3362764 to 3546310 tid 3 2024-04-22T18:40:48.988 INFO:tasks.rados.rados.0.smithi089.stdout:1618: done (7 left) 2024-04-22T18:40:48.988 INFO:tasks.rados.rados.0.smithi089.stdout:1622: done (6 left) 2024-04-22T18:40:48.988 INFO:tasks.rados.rados.0.smithi089.stdout:1623: done (5 left) 2024-04-22T18:40:48.988 INFO:tasks.rados.rados.0.smithi089.stdout:1630: tier_evict oid 43 2024-04-22T18:40:48.988 INFO:tasks.rados.rados.0.smithi089.stdout:1630: tier_evict oid 43 snap -1 2024-04-22T18:40:48.988 INFO:tasks.rados.rados.0.smithi089.stdout:1625: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:49.020 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:40:49.073 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:49.075+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:49.091 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:49.092 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:49.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:49.150 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:49.151 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:49.153 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:49.154+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:49.167 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:49.167 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:49.185 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:49.185 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:49.201 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:49.201 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:49.214 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:49.214 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:49.217 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:49.217 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:49.218 INFO:tasks.rados.rados.0.smithi089.stdout:1626: finishing write tid 1 to smithi08937929-39 2024-04-22T18:40:49.218 INFO:tasks.rados.rados.0.smithi089.stdout:1626: finishing write tid 2 to smithi08937929-39 2024-04-22T18:40:49.218 INFO:tasks.rados.rados.0.smithi089.stdout:1625: done (5 left) 2024-04-22T18:40:49.218 INFO:tasks.rados.rados.0.smithi089.stdout:1631: write (excl) oid 19 current snap is 0 2024-04-22T18:40:49.218 INFO:tasks.rados.rados.0.smithi089.stdout:1631: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2024-04-22T18:40:49.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:49.226 INFO:tasks.rados.rados.0.smithi089.stdout:1631: writing smithi08937929-19 from 763117 to 1473715 tid 1 2024-04-22T18:40:49.233 INFO:tasks.rados.rados.0.smithi089.stdout:1631: writing smithi08937929-19 from 2136128 to 2856229 tid 2 2024-04-22T18:40:49.235 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:49.236 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:49.238 INFO:tasks.rados.rados.0.smithi089.stdout:1631: writing smithi08937929-19 from 3390187 to 3563117 tid 3 2024-04-22T18:40:49.238 INFO:tasks.rados.rados.0.smithi089.stdout:1632: read oid 46 snap -1 2024-04-22T18:40:49.238 INFO:tasks.rados.rados.0.smithi089.stdout:1632: expect (ObjNum 278 snap 0 seq_num 278) 2024-04-22T18:40:49.238 INFO:tasks.rados.rados.0.smithi089.stdout:1633: write (excl) oid 7 current snap is 0 2024-04-22T18:40:49.238 INFO:tasks.rados.rados.0.smithi089.stdout:1626: finishing write tid 3 to smithi08937929-39 2024-04-22T18:40:49.238 INFO:tasks.rados.rados.0.smithi089.stdout:1626: finishing write tid 4 to smithi08937929-39 2024-04-22T18:40:49.238 INFO:tasks.rados.rados.0.smithi089.stdout:1627: finishing set_chunk to oid 22 2024-04-22T18:40:49.238 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 22 set_chunk returned error code -95 offset: 315392 length: 742912 2024-04-22T18:40:49.238 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 0 (ObjNum 303 snap 0 seq_num 303) dirty exists 2024-04-22T18:40:49.238 INFO:tasks.rados.rados.0.smithi089.stdout:1626: finishing write tid 5 to smithi08937929-39 2024-04-22T18:40:49.239 INFO:tasks.rados.rados.0.smithi089.stdout:1626: oid 39 updating version 0 to 118 2024-04-22T18:40:49.239 INFO:tasks.rados.rados.0.smithi089.stdout:1626: oid 39 version 118 is already newer than 116 2024-04-22T18:40:49.239 INFO:tasks.rados.rados.0.smithi089.stdout:1626: oid 39 version 118 is already newer than 115 2024-04-22T18:40:49.239 INFO:tasks.rados.rados.0.smithi089.stdout:1626: oid 39 version 118 is already newer than 117 2024-04-22T18:40:49.239 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 118 (ObjNum 329 snap 0 seq_num 329) dirty exists 2024-04-22T18:40:49.239 INFO:tasks.rados.rados.0.smithi089.stdout:1626: left oid 39 (ObjNum 329 snap 0 seq_num 329) 2024-04-22T18:40:49.239 INFO:tasks.rados.rados.0.smithi089.stdout:1633: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-04-22T18:40:49.247 INFO:tasks.rados.rados.0.smithi089.stdout:1633: writing smithi08937929-7 from 779924 to 1565771 tid 1 2024-04-22T18:40:49.254 INFO:tasks.rados.rados.0.smithi089.stdout:1633: writing smithi08937929-7 from 2078257 to 2658369 tid 2 2024-04-22T18:40:49.261 INFO:tasks.rados.rados.0.smithi089.stdout:1633: writing smithi08937929-7 from 3301257 to 3579924 tid 3 2024-04-22T18:40:49.261 INFO:tasks.rados.rados.0.smithi089.stdout:1626: done (7 left) 2024-04-22T18:40:49.261 INFO:tasks.rados.rados.0.smithi089.stdout:1627: done (6 left) 2024-04-22T18:40:49.261 INFO:tasks.rados.rados.0.smithi089.stdout:1634: read oid 47 snap -1 2024-04-22T18:40:49.261 INFO:tasks.rados.rados.0.smithi089.stdout:1634: expect (ObjNum 323 snap 0 seq_num 323) 2024-04-22T18:40:49.261 INFO:tasks.rados.rados.0.smithi089.stdout:1635: write oid 16 current snap is 0 2024-04-22T18:40:49.261 INFO:tasks.rados.rados.0.smithi089.stdout:1630: got (22) Invalid argument 2024-04-22T18:40:49.261 INFO:tasks.rados.rados.0.smithi089.stdout:1628: finishing set_chunk to oid 30 2024-04-22T18:40:49.261 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 54 (ObjNum 327 snap 0 seq_num 327) dirty exists 2024-04-22T18:40:49.261 INFO:tasks.rados.rados.0.smithi089.stdout:1629: finishing write tid 1 to smithi08937929-13 2024-04-22T18:40:49.261 INFO:tasks.rados.rados.0.smithi089.stdout:1635: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2024-04-22T18:40:49.264 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:49.264 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:49.267 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:49.268+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:49.267 INFO:tasks.rados.rados.0.smithi089.stdout:1635: writing smithi08937929-16 from 796731 to 1257827 tid 1 2024-04-22T18:40:49.274 INFO:tasks.rados.rados.0.smithi089.stdout:1635: writing smithi08937929-16 from 1736739 to 2460509 tid 2 2024-04-22T18:40:49.280 INFO:tasks.rados.rados.0.smithi089.stdout:1635: writing smithi08937929-16 from 2928680 to 3596731 tid 3 2024-04-22T18:40:49.280 INFO:tasks.rados.rados.0.smithi089.stdout:1628: done (7 left) 2024-04-22T18:40:49.280 INFO:tasks.rados.rados.0.smithi089.stdout:1630: done (6 left) 2024-04-22T18:40:49.280 INFO:tasks.rados.rados.0.smithi089.stdout:1636: tier_promote oid 4 2024-04-22T18:40:49.280 INFO:tasks.rados.rados.0.smithi089.stdout:1629: finishing write tid 2 to smithi08937929-13 2024-04-22T18:40:49.280 INFO:tasks.rados.rados.0.smithi089.stdout:1633: finishing write tid 1 to smithi08937929-7 2024-04-22T18:40:49.280 INFO:tasks.rados.rados.0.smithi089.stdout:1633: finishing write tid 2 to smithi08937929-7 2024-04-22T18:40:49.280 INFO:tasks.rados.rados.0.smithi089.stdout:1637: write (excl) oid 48 current snap is 0 2024-04-22T18:40:49.281 INFO:tasks.rados.rados.0.smithi089.stdout:1635: finishing write tid 1 to smithi08937929-16 2024-04-22T18:40:49.281 INFO:tasks.rados.rados.0.smithi089.stdout:1637: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2024-04-22T18:40:49.285 INFO:tasks.rados.rados.0.smithi089.stdout:1637: writing smithi08937929-48 from 413538 to 949883 tid 1 2024-04-22T18:40:49.290 INFO:tasks.rados.rados.0.smithi089.stdout:1637: writing smithi08937929-48 from 1395221 to 1979002 tid 2 2024-04-22T18:40:49.297 INFO:tasks.rados.rados.0.smithi089.stdout:1637: writing smithi08937929-48 from 2556103 to 3214720 tid 3 2024-04-22T18:40:49.297 INFO:tasks.rados.rados.0.smithi089.stdout:1638: read oid 44 snap -1 2024-04-22T18:40:49.297 INFO:tasks.rados.rados.0.smithi089.stdout:1638: expect (ObjNum 187 snap 0 seq_num 187) 2024-04-22T18:40:49.297 INFO:tasks.rados.rados.0.smithi089.stdout:1639: read oid 1 snap -1 2024-04-22T18:40:49.297 INFO:tasks.rados.rados.0.smithi089.stdout:1639: expect deleted 2024-04-22T18:40:49.297 INFO:tasks.rados.rados.0.smithi089.stdout:1640: set_chunk oid 42 target oid 2024-04-22T18:40:49.325 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:49.325 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:49.335 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:49.345 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:49.345 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:49.376 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:49.377+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:49.455 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:49.456+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:49.479 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 115.317394886, 2024-04-22T18:40:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:40:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.480 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:40:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:40:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:40:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:40:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:40:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:40:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:40:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:40:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:40:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:40:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:40:49.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:40:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:40:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:40:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:40:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:49.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:40:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:40:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:40:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:40:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:40:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:40:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:40:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:40:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:49.486 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 115.215836065, 2024-04-22T18:40:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:40:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:40:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:40:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:40:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:40:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:40:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:40:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:40:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:40:49.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:40:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:40:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:40:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:40:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:40:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:40:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:40:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:40:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:40:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:49.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:40:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:40:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:40:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:40:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:40:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:40:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.492 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 115.215809663, 2024-04-22T18:40:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:40:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.493 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:40:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:40:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:40:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:40:49.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:40:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:40:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:40:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:40:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:40:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:40:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:40:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:40:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:40:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:40:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:40:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:40:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:40:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:40:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:40:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 114.81747533399999, 2024-04-22T18:40:49.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:40:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:40:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:49.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:40:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:40:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:40:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:40:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:40:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:40:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:40:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:40:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:40:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:40:49.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:40:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:40:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:40:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:40:49.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:40:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:40:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:40:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:40:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:40:49.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:40:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:40:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:49.506 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 114.716639036, 2024-04-22T18:40:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:40:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:40:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:40:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:40:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:40:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:40:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:40:49.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:40:49.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:40:49.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:40:49.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:40:49.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:40:49.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:40:49.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:40:49.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:49.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:40:49.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:40:49.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:49.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:40:49.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:40:49.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:49.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:40:49.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:40:49.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:49.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:40:49.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:40:49.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:40:49.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:40:49.514 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.514 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.514 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.514 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:49.514 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:49.514 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 114.716481766, 2024-04-22T18:40:49.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:40:49.515 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.515 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:49.515 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.515 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.515 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.515 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:40:49.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.515 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:49.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:40:49.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:40:49.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:49.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:40:49.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:40:49.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:40:49.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:40:49.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:40:49.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:40:49.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:40:49.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:40:49.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:40:49.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:40:49.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:49.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:40:49.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:40:49.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:49.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:40:49.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:40:49.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:49.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:40:49.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:40:49.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:49.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:40:49.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:40:49.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:40:49.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:40:49.521 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.521 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.521 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.521 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:49.521 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:49.521 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 114.716467157, 2024-04-22T18:40:49.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:40:49.521 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.521 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:49.522 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.522 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.522 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.522 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:40:49.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.522 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:49.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:49.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:40:49.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:49.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:40:49.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:40:49.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:40:49.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:49.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:40:49.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:40:49.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:40:49.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:40:49.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:49.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:40:49.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:40:49.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:40:49.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:40:49.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:40:49.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:40:49.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:40:49.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:40:49.526 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.527 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.527 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.527 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:49.527 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:49.527 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 114.716456772, 2024-04-22T18:40:49.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:40:49.527 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.527 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:49.527 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.527 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.528 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.528 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:40:49.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.528 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:49.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:40:49.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:40:49.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:49.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:40:49.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:40:49.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:40:49.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:40:49.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:40:49.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:40:49.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:40:49.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:40:49.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:40:49.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:40:49.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:49.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:40:49.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:40:49.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:49.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:40:49.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:40:49.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:49.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:40:49.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:40:49.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:49.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:40:49.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:40:49.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:40:49.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:40:49.533 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.533 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.533 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.534 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:49.534 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:49.534 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 114.715526842, 2024-04-22T18:40:49.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:40:49.534 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.534 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:49.534 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.534 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.534 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.534 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:40:49.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.535 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:49.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:40:49.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:40:49.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:49.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:40:49.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:40:49.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:40:49.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:40:49.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:40:49.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:40:49.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:40:49.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:40:49.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:40:49.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:40:49.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:49.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:40:49.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:40:49.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:49.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:40:49.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:40:49.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:49.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:40:49.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:40:49.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:49.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:40:49.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:40:49.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:40:49.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:40:49.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.540 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.541 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:49.541 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:49.541 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 114.715064717, 2024-04-22T18:40:49.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:40:49.541 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.541 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:49.541 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.541 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.541 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.542 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:40:49.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.542 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:49.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:40:49.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:40:49.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:49.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:40:49.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:40:49.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:40:49.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:40:49.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:40:49.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:40:49.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:40:49.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:40:49.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:40:49.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:40:49.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:49.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:40:49.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:40:49.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:49.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:40:49.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:40:49.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:49.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:40:49.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:40:49.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:49.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:40:49.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:40:49.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:40:49.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:40:49.547 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.547 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.547 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.547 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:49.547 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:49.548 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 114.715051913, 2024-04-22T18:40:49.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:40:49.548 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.548 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:49.548 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.548 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.548 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.548 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:40:49.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.548 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:49.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:40:49.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:40:49.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:49.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:40:49.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:40:49.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:40:49.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:40:49.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:40:49.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:40:49.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:40:49.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:40:49.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:49.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:40:49.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:40:49.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:40:49.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:40:49.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:40:49.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:40:49.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:40:49.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:40:49.553 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.553 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.553 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.553 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:49.553 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:49.553 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.38727945700001, 2024-04-22T18:40:49.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:40:49.554 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.554 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:49.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.554 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:40:49.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.554 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:49.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:40:49.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:40:49.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:49.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:40:49.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:40:49.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:40:49.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:40:49.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:40:49.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:40:49.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:40:49.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:40:49.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:40:49.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:40:49.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:49.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:40:49.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:40:49.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:49.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:40:49.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:40:49.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:49.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:40:49.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:40:49.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:49.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:40:49.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:40:49.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:40:49.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:40:49.559 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.559 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.559 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.560 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:40:49.560 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:49.560 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 110.378706345, 2024-04-22T18:40:49.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:40:49.560 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.560 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:49.560 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.560 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.560 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.560 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:40:49.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.560 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:49.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:40:49.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:40:49.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:49.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:40:49.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:40:49.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:40:49.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:40:49.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:40:49.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:40:49.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:40:49.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:40:49.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:40:49.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:40:49.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:49.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:40:49.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:40:49.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:49.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:40:49.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:40:49.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:49.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:40:49.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:40:49.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:49.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:40:49.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:40:49.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:40:49.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:40:49.566 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.566 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.566 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.566 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:40:49.566 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:49.566 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.344058829999994, 2024-04-22T18:40:49.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:40:49.566 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:40:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:40:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:40:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:40:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:40:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:40:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:40:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:40:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:40:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:40:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:40:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:40:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:40:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:40:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:40:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1684 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 78.874574867000007, 2024-04-22T18:40:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987885199999998, 2024-04-22T18:40:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1684 2024-04-22T18:40:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:40:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605278+0000", 2024-04-22T18:40:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605280+0000", 2024-04-22T18:40:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1770000000000001e-06 2024-04-22T18:40:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605285+0000", 2024-04-22T18:40:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-04-22T18:40:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605289+0000", 2024-04-22T18:40:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.687e-06 2024-04-22T18:40:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705352+0000", 2024-04-22T18:40:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006322199999999 2024-04-22T18:40:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705423+0000", 2024-04-22T18:40:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1136999999999993e-05 2024-04-22T18:40:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105017+0000", 2024-04-22T18:40:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.399593268 2024-04-22T18:40:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105081+0000", 2024-04-22T18:40:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4703999999999996e-05 2024-04-22T18:40:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105157+0000", 2024-04-22T18:40:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6205999999999995e-05 2024-04-22T18:40:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2191 4.1 4.fdaa93c1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-04-22T18:40:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 63.929982617, 2024-04-22T18:40:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51064384399999996, 2024-04-22T18:40:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2191 2024-04-22T18:40:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:40:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549872+0000", 2024-04-22T18:40:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.387e-06 2024-04-22T18:40:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549875+0000", 2024-04-22T18:40:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:40:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549880+0000", 2024-04-22T18:40:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2529999999999998e-06 2024-04-22T18:40:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060496+0000", 2024-04-22T18:40:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51061597999999997 2024-04-22T18:40:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060515+0000", 2024-04-22T18:40:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9130000000000001e-05 2024-04-22T18:40:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2198 4.d 4:bc165c76:::smithi08937929-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 63.928714915999997, 2024-04-22T18:40:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50502289899999997, 2024-04-22T18:40:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2198 2024-04-22T18:40:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:40:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551138+0000", 2024-04-22T18:40:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551143+0000", 2024-04-22T18:40:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8949999999999997e-06 2024-04-22T18:40:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551151+0000", 2024-04-22T18:40:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8729999999999996e-06 2024-04-22T18:40:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551158+0000", 2024-04-22T18:40:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3099999999999997e-06 2024-04-22T18:40:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551180+0000", 2024-04-22T18:40:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2197000000000001e-05 2024-04-22T18:40:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551185+0000", 2024-04-22T18:40:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7010000000000004e-06 2024-04-22T18:40:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657133+0000", 2024-04-22T18:40:49.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105948498 2024-04-22T18:40:49.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:49.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657183+0000", 2024-04-22T18:40:49.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0124000000000003e-05 2024-04-22T18:40:49.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.055924+0000", 2024-04-22T18:40:49.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39874079299999998 2024-04-22T18:40:49.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056018+0000", 2024-04-22T18:40:49.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4032999999999999e-05 2024-04-22T18:40:49.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056161+0000", 2024-04-22T18:40:49.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143385 2024-04-22T18:40:49.590 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.590 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.590 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.590 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2212 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:49.590 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:49.590 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 63.928645273999997, 2024-04-22T18:40:49.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50989936400000002, 2024-04-22T18:40:49.590 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.590 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:49.591 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.591 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.591 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.591 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2212 2024-04-22T18:40:49.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.591 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:49.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:49.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:40:49.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:49.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551211+0000", 2024-04-22T18:40:49.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6280000000000001e-06 2024-04-22T18:40:49.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551214+0000", 2024-04-22T18:40:49.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1209999999999998e-06 2024-04-22T18:40:49.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551217+0000", 2024-04-22T18:40:49.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8490000000000002e-06 2024-04-22T18:40:49.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061086+0000", 2024-04-22T18:40:49.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50986975099999998 2024-04-22T18:40:49.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061108+0000", 2024-04-22T18:40:49.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1319e-05 2024-04-22T18:40:49.594 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.594 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.594 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.594 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2213 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:49.594 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:49.594 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 63.928634856999999, 2024-04-22T18:40:49.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50998486099999996, 2024-04-22T18:40:49.595 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.595 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:49.595 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.595 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.595 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.595 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2213 2024-04-22T18:40:49.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.595 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:49.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:40:49.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551218+0000", 2024-04-22T18:40:49.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:49.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551220+0000", 2024-04-22T18:40:49.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.319e-06 2024-04-22T18:40:49.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551223+0000", 2024-04-22T18:40:49.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9660000000000001e-06 2024-04-22T18:40:49.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551226+0000", 2024-04-22T18:40:49.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9050000000000001e-06 2024-04-22T18:40:49.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061162+0000", 2024-04-22T18:40:49.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.509936838 2024-04-22T18:40:49.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061204+0000", 2024-04-22T18:40:49.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1094999999999999e-05 2024-04-22T18:40:49.599 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.599 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.599 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.599 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2214 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:40:49.600 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:49.600 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 63.928625867999997, 2024-04-22T18:40:49.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50997458100000004, 2024-04-22T18:40:49.600 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.600 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:49.600 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.600 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.600 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.600 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2214 2024-04-22T18:40:49.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.601 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:49.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:40:49.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551227+0000", 2024-04-22T18:40:49.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:49.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551229+0000", 2024-04-22T18:40:49.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8649999999999999e-06 2024-04-22T18:40:49.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551232+0000", 2024-04-22T18:40:49.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:49.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551235+0000", 2024-04-22T18:40:49.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:40:49.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061194+0000", 2024-04-22T18:40:49.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50995819099999995 2024-04-22T18:40:49.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061202+0000", 2024-04-22T18:40:49.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.541e-06 2024-04-22T18:40:49.604 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.604 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.604 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.604 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.605 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.605 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:49.606 INFO:tasks.rados.rados.0.smithi089.stdout:1640: set_chunk oid 42 offset: 536576 length: 1732608 target oid 8cd8a272eb5cb2608f046b97ca6a17d2d07574c336682df7c83db4eca4a66bae offset: 0 snap -1 2024-04-22T18:40:49.606 INFO:tasks.rados.rados.0.smithi089.stdout:1641: write oid 28 current snap is 0 2024-04-22T18:40:49.606 INFO:tasks.rados.rados.0.smithi089.stdout:1641: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-04-22T18:40:49.607 INFO:tasks.rados.rados.0.smithi089.stdout:1641: writing smithi08937929-28 from 430345 to 1158292 tid 1 2024-04-22T18:40:49.607 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:49.608 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:49.607+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:49.608 INFO:tasks.rados.rados.0.smithi089.stdout:1641: writing smithi08937929-28 from 1570056 to 2297495 tid 2 2024-04-22T18:40:49.613 INFO:tasks.rados.rados.0.smithi089.stdout:1641: writing smithi08937929-28 from 2699879 to 3169768 tid 3 2024-04-22T18:40:49.613 INFO:tasks.rados.rados.0.smithi089.stdout:1642: read oid 5 snap -1 2024-04-22T18:40:49.613 INFO:tasks.rados.rados.0.smithi089.stdout:1642: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:40:49.613 INFO:tasks.rados.rados.0.smithi089.stdout:1643: read oid 26 snap -1 2024-04-22T18:40:49.613 INFO:tasks.rados.rados.0.smithi089.stdout:1643: expect (ObjNum 289 snap 0 seq_num 289) 2024-04-22T18:40:49.613 INFO:tasks.rados.rados.0.smithi089.stdout:1644: read oid 11 snap -1 2024-04-22T18:40:49.613 INFO:tasks.rados.rados.0.smithi089.stdout:1644: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:40:49.613 INFO:tasks.rados.rados.0.smithi089.stdout:1645: read oid 21 snap -1 2024-04-22T18:40:49.613 INFO:tasks.rados.rados.0.smithi089.stdout:1645: expect (ObjNum 314 snap 0 seq_num 314) 2024-04-22T18:40:49.614 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:49.614 INFO:tasks.rados.rados.0.smithi089.stdout:1632: expect (ObjNum 278 snap 0 seq_num 278) 2024-04-22T18:40:49.698 INFO:tasks.rados.rados.0.smithi089.stdout:1629: finishing write tid 3 to smithi08937929-13 2024-04-22T18:40:49.698 INFO:tasks.rados.rados.0.smithi089.stdout:1629: finishing write tid 4 to smithi08937929-13 2024-04-22T18:40:49.698 INFO:tasks.rados.rados.0.smithi089.stdout:1629: finishing write tid 5 to smithi08937929-13 2024-04-22T18:40:49.698 INFO:tasks.rados.rados.0.smithi089.stdout:1629: oid 13 updating version 0 to 103 2024-04-22T18:40:49.698 INFO:tasks.rados.rados.0.smithi089.stdout:1629: oid 13 updating version 103 to 105 2024-04-22T18:40:49.698 INFO:tasks.rados.rados.0.smithi089.stdout:1629: oid 13 version 105 is already newer than 102 2024-04-22T18:40:49.698 INFO:tasks.rados.rados.0.smithi089.stdout:1629: oid 13 version 105 is already newer than 104 2024-04-22T18:40:49.698 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 105 (ObjNum 330 snap 0 seq_num 330) dirty exists 2024-04-22T18:40:49.699 INFO:tasks.rados.rados.0.smithi089.stdout:1629: left oid 13 (ObjNum 330 snap 0 seq_num 330) 2024-04-22T18:40:49.699 INFO:tasks.rados.rados.0.smithi089.stdout:1634: expect (ObjNum 323 snap 0 seq_num 323) 2024-04-22T18:40:49.707 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:49.708+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:49.749 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:49.750 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:49.750 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:49.750 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:49.758 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:49.808 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:49.809+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:49.856 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:49.857+0000 7f7f44bb3640 -1 osd.4 115 handle_osd_map injecting map crc failure 2024-04-22T18:40:49.859 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:40:49.861 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:49.862 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:49.862 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:49.862 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:49.862 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:49.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:49.915 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:49.916+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:49.935 INFO:tasks.rados.rados.0.smithi089.stdout:1633: finishing write tid 3 to smithi08937929-7 2024-04-22T18:40:49.935 INFO:tasks.rados.rados.0.smithi089.stdout:1633: finishing write tid 4 to smithi08937929-7 2024-04-22T18:40:49.935 INFO:tasks.rados.rados.0.smithi089.stdout:1633: finishing write tid 5 to smithi08937929-7 2024-04-22T18:40:49.935 INFO:tasks.rados.rados.0.smithi089.stdout:1633: oid 7 updating version 0 to 57 2024-04-22T18:40:49.935 INFO:tasks.rados.rados.0.smithi089.stdout:1633: oid 7 updating version 57 to 60 2024-04-22T18:40:49.935 INFO:tasks.rados.rados.0.smithi089.stdout:1633: oid 7 version 60 is already newer than 58 2024-04-22T18:40:49.936 INFO:tasks.rados.rados.0.smithi089.stdout:1633: oid 7 version 60 is already newer than 59 2024-04-22T18:40:49.936 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 60 (ObjNum 332 snap 0 seq_num 332) dirty exists 2024-04-22T18:40:49.936 INFO:tasks.rados.rados.0.smithi089.stdout:1633: left oid 7 (ObjNum 332 snap 0 seq_num 332) 2024-04-22T18:40:49.936 INFO:tasks.rados.rados.0.smithi089.stdout:1635: finishing write tid 2 to smithi08937929-16 2024-04-22T18:40:49.936 INFO:tasks.rados.rados.0.smithi089.stdout:1629: done (15 left) 2024-04-22T18:40:49.936 INFO:tasks.rados.rados.0.smithi089.stdout:1632: done (14 left) 2024-04-22T18:40:49.936 INFO:tasks.rados.rados.0.smithi089.stdout:1633: done (13 left) 2024-04-22T18:40:49.936 INFO:tasks.rados.rados.0.smithi089.stdout:1634: done (12 left) 2024-04-22T18:40:49.936 INFO:tasks.rados.rados.0.smithi089.stdout:1639: done (11 left) 2024-04-22T18:40:49.936 INFO:tasks.rados.rados.0.smithi089.stdout:1646: write (excl) oid 10 current snap is 0 2024-04-22T18:40:49.936 INFO:tasks.rados.rados.0.smithi089.stdout:1637: finishing write tid 1 to smithi08937929-48 2024-04-22T18:40:49.936 INFO:tasks.rados.rados.0.smithi089.stdout:1646: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-04-22T18:40:49.939 INFO:tasks.rados.rados.0.smithi089.stdout:1646: writing smithi08937929-10 from 447152 to 850348 tid 1 2024-04-22T18:40:49.946 INFO:tasks.rados.rados.0.smithi089.stdout:1646: writing smithi08937929-10 from 1512185 to 2099635 tid 2 2024-04-22T18:40:49.954 INFO:tasks.rados.rados.0.smithi089.stdout:1646: writing smithi08937929-10 from 2727302 to 3408463 tid 3 2024-04-22T18:40:49.954 INFO:tasks.rados.rados.0.smithi089.stdout:1647: delete oid 32 current snap is 0 2024-04-22T18:40:49.954 INFO:tasks.rados.rados.0.smithi089.stdout:1635: finishing write tid 3 to smithi08937929-16 2024-04-22T18:40:49.955 INFO:tasks.rados.rados.0.smithi089.stdout:1635: finishing write tid 4 to smithi08937929-16 2024-04-22T18:40:49.955 INFO:tasks.rados.rados.0.smithi089.stdout:1635: finishing write tid 5 to smithi08937929-16 2024-04-22T18:40:49.955 INFO:tasks.rados.rados.0.smithi089.stdout:1635: oid 16 updating version 0 to 136 2024-04-22T18:40:49.955 INFO:tasks.rados.rados.0.smithi089.stdout:1635: oid 16 version 136 is already newer than 135 2024-04-22T18:40:49.955 INFO:tasks.rados.rados.0.smithi089.stdout:1635: oid 16 updating version 136 to 138 2024-04-22T18:40:49.955 INFO:tasks.rados.rados.0.smithi089.stdout:1635: oid 16 version 138 is already newer than 137 2024-04-22T18:40:49.955 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 138 (ObjNum 333 snap 0 seq_num 333) dirty exists 2024-04-22T18:40:49.955 INFO:tasks.rados.rados.0.smithi089.stdout:1635: left oid 16 (ObjNum 333 snap 0 seq_num 333) 2024-04-22T18:40:49.955 INFO:tasks.rados.rados.0.smithi089.stdout:1637: finishing write tid 2 to smithi08937929-48 2024-04-22T18:40:49.955 INFO:tasks.rados.rados.0.smithi089.stdout:1637: finishing write tid 3 to smithi08937929-48 2024-04-22T18:40:49.955 INFO:tasks.rados.rados.0.smithi089.stdout:1631: finishing write tid 1 to smithi08937929-19 2024-04-22T18:40:49.955 INFO:tasks.rados.rados.0.smithi089.stdout:1631: finishing write tid 2 to smithi08937929-19 2024-04-22T18:40:49.955 INFO:tasks.rados.rados.0.smithi089.stdout:1631: finishing write tid 3 to smithi08937929-19 2024-04-22T18:40:49.956 INFO:tasks.rados.rados.0.smithi089.stdout:1631: finishing write tid 4 to smithi08937929-19 2024-04-22T18:40:49.956 INFO:tasks.rados.rados.0.smithi089.stdout:1631: finishing write tid 5 to smithi08937929-19 2024-04-22T18:40:49.956 INFO:tasks.rados.rados.0.smithi089.stdout:1631: oid 19 updating version 0 to 132 2024-04-22T18:40:49.956 INFO:tasks.rados.rados.0.smithi089.stdout:1631: oid 19 updating version 132 to 134 2024-04-22T18:40:49.956 INFO:tasks.rados.rados.0.smithi089.stdout:1631: oid 19 version 134 is already newer than 133 2024-04-22T18:40:49.956 INFO:tasks.rados.rados.0.smithi089.stdout:1631: oid 19 updating version 134 to 135 2024-04-22T18:40:49.956 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 135 (ObjNum 331 snap 0 seq_num 331) dirty exists 2024-04-22T18:40:49.956 INFO:tasks.rados.rados.0.smithi089.stdout:1631: left oid 19 (ObjNum 331 snap 0 seq_num 331) 2024-04-22T18:40:49.956 INFO:tasks.rados.rados.0.smithi089.stdout:1637: finishing write tid 4 to smithi08937929-48 2024-04-22T18:40:49.956 INFO:tasks.rados.rados.0.smithi089.stdout:1637: finishing write tid 5 to smithi08937929-48 2024-04-22T18:40:49.956 INFO:tasks.rados.rados.0.smithi089.stdout:1637: oid 48 updating version 0 to 90 2024-04-22T18:40:49.956 INFO:tasks.rados.rados.0.smithi089.stdout:1637: oid 48 version 90 is already newer than 87 2024-04-22T18:40:49.956 INFO:tasks.rados.rados.0.smithi089.stdout:1637: oid 48 version 90 is already newer than 89 2024-04-22T18:40:49.957 INFO:tasks.rados.rados.0.smithi089.stdout:1637: oid 48 version 90 is already newer than 88 2024-04-22T18:40:49.957 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 90 (ObjNum 334 snap 0 seq_num 334) dirty exists 2024-04-22T18:40:49.957 INFO:tasks.rados.rados.0.smithi089.stdout:1637: left oid 48 (ObjNum 334 snap 0 seq_num 334) 2024-04-22T18:40:49.957 INFO:tasks.rados.rados.0.smithi089.stdout:1645: expect (ObjNum 314 snap 0 seq_num 314) 2024-04-22T18:40:49.975 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:49.975 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:40:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 115.881864142, 2024-04-22T18:40:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:40:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.976 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:40:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:40:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:40:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:40:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:40:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:40:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:40:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:40:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:40:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:40:49.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:40:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:40:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:40:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:40:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:40:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:40:49.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:40:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:40:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:40:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:40:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:40:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:40:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:40:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.982366267, 2024-04-22T18:40:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:40:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:40:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:40:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:40:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:40:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:40:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:40:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:40:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:40:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:40:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:40:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:40:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:40:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:40:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:40:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:40:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:40:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:40:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:40:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:40:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:40:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:40:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:40:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:40:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:40:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:40:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:40:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:40:49.988 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 110.874066476, 2024-04-22T18:40:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:40:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:40:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:40:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:40:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:40:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:40:49.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:40:49.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:40:49.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:49.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:40:49.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:40:49.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:49.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:40:49.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:40:49.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:49.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:40:49.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:40:49.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:49.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:40:49.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:40:49.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:40:49.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.542522848000004, 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:49.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:49.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:40:49.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:49.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:49.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:40:49.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:49.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:49.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:40:49.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:49.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:40:49.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:40:49.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:49.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:40:49.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:40:49.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:49.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:49.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:49.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:40:49.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:40:49.999 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:49.999 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:49.999 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.451437040000002, 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:40:50.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:40:50.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:40:50.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:40:50.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:40:50.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:40:50.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:40:50.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:40:50.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:40:50.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:40:50.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:40:50.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:40:50.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:40:50.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:40:50.004 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.005 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.005 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.005 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:50.005 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:50.005 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.445636780000001, 2024-04-22T18:40:50.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:40:50.005 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.005 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.005 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.005 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.005 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.005 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:40:50.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.006 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:50.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:40:50.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:40:50.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:50.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:40:50.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:40:50.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:40:50.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:40:50.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:40:50.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:40:50.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.440896437000006, 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.010 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:40:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:40:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:40:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:40:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:40:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:40:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:40:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:40:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:40:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:40:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:40:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:40:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.440423538999994, 2024-04-22T18:40:50.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:40:50.016 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.016 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.016 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.016 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.016 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.016 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:40:50.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.016 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:50.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:40:50.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:40:50.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:50.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:40:50.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:40:50.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:40:50.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:40:50.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:40:50.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:40:50.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:40:50.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:40:50.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:40:50.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:40:50.020 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.020 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.020 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.021 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:40:50.021 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:50.021 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.440403618000005, 2024-04-22T18:40:50.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:40:50.021 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.021 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:50.021 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.021 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.021 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.021 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:40:50.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.021 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:50.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:50.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:40:50.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:50.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:40:50.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:40:50.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:40:50.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:40:50.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:40:50.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:40:50.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:40:50.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:40:50.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:50.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:40:50.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:40:50.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:40:50.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:40:50.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:40:50.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:40:50.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:40:50.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:40:50.027 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.027 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.027 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 92.131079853000003, 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:40:50.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:40:50.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:50.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:40:50.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:40:50.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:40:50.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:40:50.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:40:50.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:40:50.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:40:50.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:40:50.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:40:50.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:40:50.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:50.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:40:50.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:40:50.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:40:50.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:40:50.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:40:50.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:40:50.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:40:50.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:40:50.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:40:50.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 92.130872316999998, 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:50.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:40:50.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:40:50.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:50.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:40:50.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:40:50.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:40:50.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:40:50.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:50.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:40:50.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:40:50.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:40:50.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:40:50.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:40:50.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:40:50.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:40:50.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:40:50.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:40:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:40:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.265222584, 2024-04-22T18:40:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:40:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:40:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:40:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:40:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:40:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:40:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:40:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:40:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:40:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:40:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:40:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:40:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:40:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:40:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:40:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.051384198999997, 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:40:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:40:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:40:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:40:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:40:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:40:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:40:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:40:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:40:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:40:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:40:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:40:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:40:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:40:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:40:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:40:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:40:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:40:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.051350788999997, 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:40:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:40:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:40:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:40:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:40:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:40:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:40:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:40:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.992468134999996, 2024-04-22T18:40:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:40:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:40:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:40:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:40:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:40:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:40:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:40:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:40:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:40:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:40:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:40:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.992329518999995, 2024-04-22T18:40:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:40:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:50.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:50.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:40:50.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:50.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:40:50.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:40:50.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:40:50.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:40:50.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:50.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:40:50.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:40:50.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:40:50.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:40:50.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:40:50.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:40:50.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:40:50.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:40:50.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:40:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:40:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 72.992310516000003, 2024-04-22T18:40:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:40:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.063 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:40:50.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:40:50.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:40:50.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:40:50.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:50.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:40:50.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:40:50.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:40:50.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:40:50.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:40:50.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:40:50.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:40:50.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:40:50.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:40:50.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:40:50.068 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.068 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.068 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.068 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:50.068 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:50.068 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.371297983000005, 2024-04-22T18:40:50.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:40:50.068 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.068 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:40:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:40:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:40:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:40:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:40:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:40:50.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:40:50.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:40:50.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:40:50.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:40:50.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:40:50.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:50.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:40:50.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:40:50.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:40:50.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:40:50.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:40:50.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:40:50.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:50.073 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:50.074 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.370778392000005, 2024-04-22T18:40:50.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:40:50.074 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.074 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.074 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.074 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.074 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.074 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:40:50.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.074 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:50.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:40:50.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:40:50.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:50.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:40:50.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:40:50.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:40:50.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:40:50.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:40:50.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:40:50.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:40:50.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:40:50.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:40:50.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:40:50.078 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.078 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.079 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.079 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:50.079 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:50.079 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.270625508999998, 2024-04-22T18:40:50.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:40:50.079 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.079 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:50.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.079 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:40:50.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:50.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:40:50.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:40:50.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:50.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:40:50.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:40:50.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:40:50.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:40:50.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:40:50.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:40:50.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:40:50.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:40:50.083 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.083 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.083 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.083 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.083 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.083 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:50.084 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 6, 7, 2] out_osds: [5, 4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:40:50.084 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-22T18:40:50.085 INFO:tasks.rados.rados.0.smithi089.stdout:1636: got (0) Success 2024-04-22T18:40:50.085 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 103 (ObjNum 269 snap 0 seq_num 269) dirty exists 2024-04-22T18:40:50.085 INFO:tasks.rados.rados.0.smithi089.stdout:1638: expect (ObjNum 187 snap 0 seq_num 187) 2024-04-22T18:40:50.085 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:50.086+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:50.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:50.185 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:50.186+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:50.230 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:50.230 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:50.230 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:50.230 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:50.236 INFO:tasks.rados.rados.0.smithi089.stdout:1640: finishing set_chunk to oid 42 2024-04-22T18:40:50.236 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 31 (ObjNum 274 snap 0 seq_num 274) dirty exists 2024-04-22T18:40:50.236 INFO:tasks.rados.rados.0.smithi089.stdout:1643: expect (ObjNum 289 snap 0 seq_num 289) 2024-04-22T18:40:50.237 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:50.286 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:50.288+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:50.378 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:50.378 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:50.378 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:50.379 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:50.379 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:50.387 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:50.388 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:50.388+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:50.408 INFO:tasks.rados.rados.0.smithi089.stdout:1642: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:40:50.488 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:50.489+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:50.491 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:50.491 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:50.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:50.491 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:50.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.491 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:50.491 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:50.491 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 116.231506444, 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:50.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:40:50.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:50.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:40:50.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:40:50.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:40:50.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:40:50.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:40:50.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:40:50.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:40:50.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:40:50.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:40:50.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:40:50.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:50.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:40:50.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:40:50.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:40:50.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:40:50.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:40:50.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:40:50.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:40:50.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:40:50.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:40:50.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:40:50.497 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.497 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.497 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.497 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:50.497 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:50.497 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 116.226724014, 2024-04-22T18:40:50.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:40:50.497 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.497 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.497 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.497 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.497 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:40:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:40:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:40:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:40:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:40:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:40:50.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:50.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:40:50.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:40:50.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:40:50.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:40:50.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:40:50.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:40:50.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:40:50.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:40:50.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:40:50.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:40:50.502 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.502 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.502 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.502 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:50.503 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:50.503 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 116.226629811, 2024-04-22T18:40:50.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:40:50.503 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.503 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.503 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.503 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.503 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.503 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:40:50.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.503 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:50.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:50.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:40:50.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:50.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:40:50.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:40:50.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.507 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.508 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.508 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:50.508 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:50.508 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 116.226608923, 2024-04-22T18:40:50.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:40:50.508 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.508 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:50.508 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.508 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.508 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.508 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:40:50.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.508 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:40:50.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:40:50.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:40:50.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:40:50.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:40:50.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:40:50.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 115.44360125999999, 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:40:50.512 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:50.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:40:50.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:50.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:40:50.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:40:50.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:40:50.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:40:50.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:50.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:40:50.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:40:50.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:40:50.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:40:50.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:40:50.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:40:50.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:40:50.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:40:50.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:40:50.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:40:50.517 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.517 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.517 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.518 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:50.518 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:50.518 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 115.435769854, 2024-04-22T18:40:50.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:40:50.518 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.518 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.518 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.518 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.518 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.518 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:40:50.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.518 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:40:50.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:40:50.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:40:50.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:40:50.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:40:50.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:40:50.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:40:50.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:40:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:40:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:40:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:40:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:40:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:40:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.522 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.283897224, 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:40:50.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:40:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:40:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:40:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:40:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:40:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:40:50.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.527 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.528 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.528 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:50.528 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:50.528 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.283492316, 2024-04-22T18:40:50.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:40:50.528 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.528 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:50.528 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.528 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.528 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.528 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:40:50.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.528 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:40:50.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:40:50.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:40:50.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:40:50.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:40:50.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:40:50.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3213 4.a 4:5d2aee44:::smithi08937929-13:head [assert-version v66,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.922592428000002, 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69928701400000004, 2024-04-22T18:40:50.532 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.533 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.533 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.533 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.533 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.533 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3213 2024-04-22T18:40:50.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.533 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:40:50.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:40:50.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568559+0000", 2024-04-22T18:40:50.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:50.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568567+0000", 2024-04-22T18:40:50.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-04-22T18:40:50.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568579+0000", 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.668711+0000", 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100131866 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.768945+0000", 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100234252 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769002+0000", 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7092999999999997e-05 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.166404+0000", 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39740241999999998 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.191686+0000", 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025281739000000001 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267758+0000", 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076071606999999999 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267812+0000", 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4478999999999998e-05 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267848+0000", 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6062999999999997e-05 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.537 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3319 4.a 4:5d2aee44:::smithi08937929-13:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:50.538 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:40:50.538 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.030844677000001, 2024-04-22T18:40:50.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476623401, 2024-04-22T18:40:50.538 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.538 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:50.538 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.538 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.538 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.538 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3319 2024-04-22T18:40:50.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.538 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:40:50.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460306+0000", 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460318+0000", 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2896e-05 2024-04-22T18:40:50.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460345+0000", 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6650000000000001e-05 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460360+0000", 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4729000000000001e-05 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460502+0000", 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141729 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460644+0000", 2024-04-22T18:40:50.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142755 2024-04-22T18:40:50.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.936932+0000", 2024-04-22T18:40:50.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476288078 2024-04-22T18:40:50.541 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.541 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.541 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.541 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3412 4.a 4:519c08f7:::smithi08937929-15:head [write 2529878~394573] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:50.541 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:40:50.541 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.349010115999999, 2024-04-22T18:40:50.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48918094899999998, 2024-04-22T18:40:50.542 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.542 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.542 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.542 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.542 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.542 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3412 2024-04-22T18:40:50.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.542 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:40:50.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:40:50.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142141+0000", 2024-04-22T18:40:50.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:50.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142496+0000", 2024-04-22T18:40:50.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035444699999999998 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142510+0000", 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3841e-05 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142518+0000", 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8089999999999993e-06 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142560+0000", 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2097e-05 2024-04-22T18:40:50.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142637+0000", 2024-04-22T18:40:50.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6965000000000006e-05 2024-04-22T18:40:50.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:50.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142715+0000", 2024-04-22T18:40:50.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8694000000000002e-05 2024-04-22T18:40:50.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144179+0000", 2024-04-22T18:40:50.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146342 2024-04-22T18:40:50.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631229+0000", 2024-04-22T18:40:50.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48705018500000002 2024-04-22T18:40:50.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631281+0000", 2024-04-22T18:40:50.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2216999999999998e-05 2024-04-22T18:40:50.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631324+0000", 2024-04-22T18:40:50.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3344999999999999e-05 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3413 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2924451] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.348628976000001, 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58904086899999997, 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3413 2024-04-22T18:40:50.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142523+0000", 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142529+0000", 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6049999999999997e-06 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142535+0000", 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4740000000000002e-06 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142541+0000", 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1839999999999998e-06 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143553+0000", 2024-04-22T18:40:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010126689999999999 2024-04-22T18:40:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143659+0000", 2024-04-22T18:40:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105393 2024-04-22T18:40:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143825+0000", 2024-04-22T18:40:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165809 2024-04-22T18:40:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144329+0000", 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050483799999999997 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731480+0000", 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58715100600000003 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731522+0000", 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2098000000000002e-05 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731566+0000", 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3066999999999999e-05 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3414 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 19.348600372, 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58926020199999996, 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3414 2024-04-22T18:40:50.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.553 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:40:50.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:40:50.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142552+0000", 2024-04-22T18:40:50.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:50.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142556+0000", 2024-04-22T18:40:50.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0289999999999999e-06 2024-04-22T18:40:50.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142574+0000", 2024-04-22T18:40:50.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7266999999999999e-05 2024-04-22T18:40:50.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142581+0000", 2024-04-22T18:40:50.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144121+0000", 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015397480000000001 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144170+0000", 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8803e-05 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731624+0000", 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58745365800000005 2024-04-22T18:40:50.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731712+0000", 2024-04-22T18:40:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8016999999999994e-05 2024-04-22T18:40:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731813+0000", 2024-04-22T18:40:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101817 2024-04-22T18:40:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.556 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:77 3.3 3:dcc8aeec:::3590d5741efdda41a26b62a0982727439225b7b4b51097b313f30d8dc10bf6f6:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:50.557 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:40:50.557 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.243582050000001, 2024-04-22T18:40:50.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011812878, 2024-04-22T18:40:50.557 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.557 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.557 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:40:50.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:40:50.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247571+0000", 2024-04-22T18:40:50.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:50.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247574+0000", 2024-04-22T18:40:50.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:40:50.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247577+0000", 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3069999999999998e-06 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247580+0000", 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8870000000000001e-06 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.340928+0000", 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093348678000000004 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341122+0000", 2024-04-22T18:40:50.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000193227 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618335+0000", 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.277213775 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618422+0000", 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6537000000000002e-05 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954387+0000", 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33596509699999999 2024-04-22T18:40:50.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954548+0000", 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016042199999999999 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954858+0000", 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000310396 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157019+0000", 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20216071299999999 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259213+0000", 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102194734 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259262+0000", 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9052999999999999e-05 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259384+0000", 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122052 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:78 3.3 3:df7e90d2:::ab0657e24bd96704775de2a378746bc22309c5a7366a5eaf9b4e1f1f2bef0a4c:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.243566853000001, 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011974739, 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247586+0000", 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247589+0000", 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.181e-06 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247592+0000", 2024-04-22T18:40:50.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0230000000000001e-06 2024-04-22T18:40:50.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247595+0000", 2024-04-22T18:40:50.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.428e-06 2024-04-22T18:40:50.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341160+0000", 2024-04-22T18:40:50.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093564571999999999 2024-04-22T18:40:50.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:40:50.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341264+0000", 2024-04-22T18:40:50.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010391999999999999 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619269+0000", 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27800575100000002 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619346+0000", 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6459999999999996e-05 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.155957+0000", 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53661109299999998 2024-04-22T18:40:50.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156049+0000", 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2272e-05 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156236+0000", 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000186808 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.158088+0000", 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018522079999999999 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259421+0000", 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101332718 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259460+0000", 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8912000000000003e-05 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259562+0000", 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010175600000000001 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.568 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:80 3.3 3:f90f47f2:::e5603f596d699ee3e27e255225eeb90a9f0c1b4a4d099d1c586b19019220aa3f:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.243527776000001, 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0121096759999999, 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:40:50.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247625+0000", 2024-04-22T18:40:50.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:50.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247628+0000", 2024-04-22T18:40:50.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6129999999999998e-06 2024-04-22T18:40:50.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247631+0000", 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3869999999999999e-06 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247634+0000", 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9919999999999999e-06 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341550+0000", 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093915584999999996 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:40:50.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341700+0000", 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015000300000000001 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620744+0000", 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27904399200000002 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620794+0000", 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9579000000000001e-05 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157597+0000", 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53680343799999997 2024-04-22T18:40:50.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157678+0000", 2024-04-22T18:40:50.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0829999999999997e-05 2024-04-22T18:40:50.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:50.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157832+0000", 2024-04-22T18:40:50.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154184 2024-04-22T18:40:50.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259572+0000", 2024-04-22T18:40:50.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10174003600000001 2024-04-22T18:40:50.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259601+0000", 2024-04-22T18:40:50.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8841999999999999e-05 2024-04-22T18:40:50.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259638+0000", 2024-04-22T18:40:50.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7308999999999998e-05 2024-04-22T18:40:50.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259736+0000", 2024-04-22T18:40:50.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7283999999999999e-05 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:135 3.3 3:f89134e5:::3e4e73f5d4a0122957865f0faeac2483204ee98d7d497ef3b54afe2a02b39a3d:head [read 0~802304 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 14.875712697000001, 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.64370653499999997, 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615438+0000", 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615449+0000", 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1219999999999999e-05 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615460+0000", 2024-04-22T18:40:50.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0403e-05 2024-04-22T18:40:50.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.715588+0000", 2024-04-22T18:40:50.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100128881 2024-04-22T18:40:50.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722533+0000", 2024-04-22T18:40:50.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069444010000000002 2024-04-22T18:40:50.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:40:50.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722658+0000", 2024-04-22T18:40:50.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012560699999999999 2024-04-22T18:40:50.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826437+0000", 2024-04-22T18:40:50.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103778194 2024-04-22T18:40:50.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:50.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826532+0000", 2024-04-22T18:40:50.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5111000000000005e-05 2024-04-22T18:40:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258787+0000", 2024-04-22T18:40:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43225529099999999 2024-04-22T18:40:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258905+0000", 2024-04-22T18:40:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118348 2024-04-22T18:40:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259148+0000", 2024-04-22T18:40:50.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000242093 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3890 4.1 4:83c955bf:::smithi08937929-28:head [write 2699879~469889] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.87703785300000003, 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49042297000000001, 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3890 2024-04-22T18:40:50.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.581 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:40:50.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:40:50.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614115+0000", 2024-04-22T18:40:50.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:50.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614496+0000", 2024-04-22T18:40:50.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038181700000000001 2024-04-22T18:40:50.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614509+0000", 2024-04-22T18:40:50.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2381000000000001e-05 2024-04-22T18:40:50.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614517+0000", 2024-04-22T18:40:50.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0879999999999995e-06 2024-04-22T18:40:50.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615610+0000", 2024-04-22T18:40:50.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010933519999999999 2024-04-22T18:40:50.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615655+0000", 2024-04-22T18:40:50.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4966000000000003e-05 2024-04-22T18:40:50.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:50.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615769+0000", 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113986 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704882+0000", 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089112825000000007 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104488+0000", 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39960572999999999 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104517+0000", 2024-04-22T18:40:50.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9006e-05 2024-04-22T18:40:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104539+0000", 2024-04-22T18:40:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2073e-05 2024-04-22T18:40:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3891 4.1 4:83c955bf:::smithi08937929-28:head [setxattr _header (56) in=7b,truncate 3630345] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:40:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:40:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.87662962200000005, 2024-04-22T18:40:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59018588699999996, 2024-04-22T18:40:50.585 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.586 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.586 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.586 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.586 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.586 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3891 2024-04-22T18:40:50.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.586 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:40:50.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614523+0000", 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614528+0000", 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0379999999999999e-06 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614540+0000", 2024-04-22T18:40:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018000000000001e-05 2024-04-22T18:40:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614546+0000", 2024-04-22T18:40:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4990000000000002e-06 2024-04-22T18:40:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704438+0000", 2024-04-22T18:40:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089892044000000004 2024-04-22T18:40:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704504+0000", 2024-04-22T18:40:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6365000000000006e-05 2024-04-22T18:40:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704602+0000", 2024-04-22T18:40:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8048999999999996e-05 2024-04-22T18:40:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.806370+0000", 2024-04-22T18:40:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10176768 2024-04-22T18:40:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204629+0000", 2024-04-22T18:40:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39825917999999999 2024-04-22T18:40:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204673+0000", 2024-04-22T18:40:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3516999999999997e-05 2024-04-22T18:40:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204710+0000", 2024-04-22T18:40:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7391000000000002e-05 2024-04-22T18:40:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3892 4.1 4:83c955bf:::smithi08937929-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:40:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:40:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.87660304099999997, 2024-04-22T18:40:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59031281800000002, 2024-04-22T18:40:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3892 2024-04-22T18:40:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.591 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614550+0000", 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614553+0000", 2024-04-22T18:40:50.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0879999999999999e-06 2024-04-22T18:40:50.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614559+0000", 2024-04-22T18:40:50.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9259999999999997e-06 2024-04-22T18:40:50.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614565+0000", 2024-04-22T18:40:50.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7950000000000004e-06 2024-04-22T18:40:50.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704818+0000", 2024-04-22T18:40:50.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090252624000000004 2024-04-22T18:40:50.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:50.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704866+0000", 2024-04-22T18:40:50.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8667e-05 2024-04-22T18:40:50.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204742+0000", 2024-04-22T18:40:50.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987551899999999 2024-04-22T18:40:50.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204799+0000", 2024-04-22T18:40:50.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7299000000000001e-05 2024-04-22T18:40:50.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204863+0000", 2024-04-22T18:40:50.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4455000000000005e-05 2024-04-22T18:40:50.595 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.595 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.595 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.595 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.596 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.596 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:50.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:50.597 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:50.598+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:50.638 INFO:tasks.rados.rados.0.smithi089.stdout:1641: finishing write tid 1 to smithi08937929-28 2024-04-22T18:40:50.639 INFO:tasks.rados.rados.0.smithi089.stdout:1644: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:40:50.697 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:50.699+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:50.738 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:50.738 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:50.738 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:50.739 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:50.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:50.798 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:50.800+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:50.843 INFO:tasks.rados.rados.0.smithi089.stdout:1646: finishing write tid 1 to smithi08937929-10 2024-04-22T18:40:50.843 INFO:tasks.rados.rados.0.smithi089.stdout:1641: finishing write tid 2 to smithi08937929-28 2024-04-22T18:40:50.843 INFO:tasks.rados.rados.0.smithi089.stdout:1646: finishing write tid 2 to smithi08937929-10 2024-04-22T18:40:50.843 INFO:tasks.rados.rados.0.smithi089.stdout:1641: finishing write tid 3 to smithi08937929-28 2024-04-22T18:40:50.843 INFO:tasks.rados.rados.0.smithi089.stdout:1641: finishing write tid 4 to smithi08937929-28 2024-04-22T18:40:50.843 INFO:tasks.rados.rados.0.smithi089.stdout:1641: finishing write tid 5 to smithi08937929-28 2024-04-22T18:40:50.843 INFO:tasks.rados.rados.0.smithi089.stdout:1641: oid 28 updating version 0 to 154 2024-04-22T18:40:50.843 INFO:tasks.rados.rados.0.smithi089.stdout:1641: oid 28 updating version 154 to 157 2024-04-22T18:40:50.844 INFO:tasks.rados.rados.0.smithi089.stdout:1641: oid 28 version 157 is already newer than 156 2024-04-22T18:40:50.844 INFO:tasks.rados.rados.0.smithi089.stdout:1641: oid 28 version 157 is already newer than 155 2024-04-22T18:40:50.844 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 157 (ObjNum 335 snap 0 seq_num 335) dirty exists 2024-04-22T18:40:50.844 INFO:tasks.rados.rados.0.smithi089.stdout:1641: left oid 28 (ObjNum 335 snap 0 seq_num 335) 2024-04-22T18:40:50.844 INFO:tasks.rados.rados.0.smithi089.stdout:1646: finishing write tid 3 to smithi08937929-10 2024-04-22T18:40:50.844 INFO:tasks.rados.rados.0.smithi089.stdout:1646: finishing write tid 4 to smithi08937929-10 2024-04-22T18:40:50.844 INFO:tasks.rados.rados.0.smithi089.stdout:1646: finishing write tid 5 to smithi08937929-10 2024-04-22T18:40:50.844 INFO:tasks.rados.rados.0.smithi089.stdout:1646: oid 10 updating version 0 to 44 2024-04-22T18:40:50.844 INFO:tasks.rados.rados.0.smithi089.stdout:1646: oid 10 version 44 is already newer than 41 2024-04-22T18:40:50.844 INFO:tasks.rados.rados.0.smithi089.stdout:1646: oid 10 version 44 is already newer than 42 2024-04-22T18:40:50.844 INFO:tasks.rados.rados.0.smithi089.stdout:1646: oid 10 version 44 is already newer than 43 2024-04-22T18:40:50.844 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 44 (ObjNum 336 snap 0 seq_num 336) dirty exists 2024-04-22T18:40:50.844 INFO:tasks.rados.rados.0.smithi089.stdout:1646: left oid 10 (ObjNum 336 snap 0 seq_num 336) 2024-04-22T18:40:50.844 INFO:tasks.rados.rados.0.smithi089.stdout:1631: done (12 left) 2024-04-22T18:40:50.844 INFO:tasks.rados.rados.0.smithi089.stdout:1635: done (11 left) 2024-04-22T18:40:50.845 INFO:tasks.rados.rados.0.smithi089.stdout:1636: done (10 left) 2024-04-22T18:40:50.845 INFO:tasks.rados.rados.0.smithi089.stdout:1637: done (9 left) 2024-04-22T18:40:50.845 INFO:tasks.rados.rados.0.smithi089.stdout:1638: done (8 left) 2024-04-22T18:40:50.845 INFO:tasks.rados.rados.0.smithi089.stdout:1640: done (7 left) 2024-04-22T18:40:50.845 INFO:tasks.rados.rados.0.smithi089.stdout:1641: done (6 left) 2024-04-22T18:40:50.845 INFO:tasks.rados.rados.0.smithi089.stdout:1642: done (5 left) 2024-04-22T18:40:50.845 INFO:tasks.rados.rados.0.smithi089.stdout:1643: done (4 left) 2024-04-22T18:40:50.845 INFO:tasks.rados.rados.0.smithi089.stdout:1644: done (3 left) 2024-04-22T18:40:50.845 INFO:tasks.rados.rados.0.smithi089.stdout:1645: done (2 left) 2024-04-22T18:40:50.845 INFO:tasks.rados.rados.0.smithi089.stdout:1646: done (1 left) 2024-04-22T18:40:50.845 INFO:tasks.rados.rados.0.smithi089.stdout:1647: done (0 left) 2024-04-22T18:40:50.845 INFO:tasks.rados.rados.0.smithi089.stdout:1648: read oid 48 snap -1 2024-04-22T18:40:50.845 INFO:tasks.rados.rados.0.smithi089.stdout:1648: expect (ObjNum 334 snap 0 seq_num 334) 2024-04-22T18:40:50.845 INFO:tasks.rados.rados.0.smithi089.stdout:1649: tier_evict oid 3 2024-04-22T18:40:50.845 INFO:tasks.rados.rados.0.smithi089.stdout:1649: tier_evict oid 3 snap -1 2024-04-22T18:40:50.846 INFO:tasks.rados.rados.0.smithi089.stdout:1650: delete oid 26 current snap is 0 2024-04-22T18:40:50.846 INFO:tasks.rados.rados.0.smithi089.stdout:1649: got (22) Invalid argument 2024-04-22T18:40:50.846 INFO:tasks.rados.rados.0.smithi089.stdout:1648: expect (ObjNum 334 snap 0 seq_num 334) 2024-04-22T18:40:50.848 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:50.848 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:50.848 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:50.848 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:50.848 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:50.856 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:50.859 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.1 to 1 (10000) 2024-04-22T18:40:50.872 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-22T18:40:50.900 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:50.901+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:50.960 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 131.218163605, 2024-04-22T18:40:50.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:40:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:40:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:40:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:40:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:40:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:40:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:40:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:40:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:40:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:40:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:40:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 131.21812391399999, 2024-04-22T18:40:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:40:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:40:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:40:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:40:50.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:40:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:40:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:40:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:40:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:40:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:40:50.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:40:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 128.973927078, 2024-04-22T18:40:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:40:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:50.971 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:40:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:40:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:40:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:40:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:40:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:40:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:40:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:40:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:40:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:40:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:40:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:40:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.10593725, 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.976 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:40:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:40:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:40:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:40:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:40:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:40:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:40:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:40:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:40:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:40:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.981 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.105921592, 2024-04-22T18:40:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:40:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:40:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:50.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:40:50.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:40:50.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:40:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:40:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:40:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:40:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:40:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:40:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.986 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.10587043, 2024-04-22T18:40:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:40:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:40:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.987 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:40:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:40:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:40:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:40:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:40:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:40:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:40:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:40:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:40:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:40:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:40:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:40:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:40:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:40:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.105855206, 2024-04-22T18:40:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:40:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.992 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:40:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:40:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:40:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:40:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:40:50.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:40:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:40:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:40:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:40:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:40:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:40:50.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:40:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:40:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:40:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:40:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:40:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:40:50.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:40:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:40:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:40:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:40:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.105795629, 2024-04-22T18:40:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:40:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:40:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.998 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:50.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:40:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:40:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:40:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:40:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:40:51.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:40:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:40:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:40:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:40:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:40:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:40:51.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:40:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:40:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:40:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:40:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:40:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:40:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:40:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:40:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:40:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:51.003 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.105713313, 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:40:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:40:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:40:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:40:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:40:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:40:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:40:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:40:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:40:51.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:40:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:40:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:40:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:40:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:40:51.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:40:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:40:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:40:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:40:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:40:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:51.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:40:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:40:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.10570315699999, 2024-04-22T18:40:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:40:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:51.009 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:40:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:40:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:40:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:40:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:40:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:40:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:40:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:40:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:40:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:40:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:40:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:40:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:40:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:40:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:40:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:40:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2835 4.18 4:187adac1:::smithi08937929-36:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 43.343778766, 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.403020987, 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2835 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616367+0000", 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616377+0000", 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0618999999999999e-05 2024-04-22T18:40:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616410+0000", 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3021000000000001e-05 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616437+0000", 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7158e-05 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.716588+0000", 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100150674 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.716679+0000", 2024-04-22T18:40:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1231999999999997e-05 2024-04-22T18:40:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:08.019390+0000", 2024-04-22T18:40:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30271076299999999 2024-04-22T18:40:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3163 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:40:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.122690719000001, 2024-04-22T18:40:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1009801299999999, 2024-04-22T18:40:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3163 2024-04-22T18:40:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837455+0000", 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837464+0000", 2024-04-22T18:40:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5319999999999995e-06 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837476+0000", 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1759e-05 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837488+0000", 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1666e-05 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837541+0000", 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2939999999999998e-05 2024-04-22T18:40:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837547+0000", 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6409999999999996e-06 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937423+0000", 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.099875545 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937567+0000", 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014363099999999999 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938437+0000", 2024-04-22T18:40:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087060399999999997 2024-04-22T18:40:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3164 4.8 4:187adac1:::smithi08937929-36:head [sparse-read 0~2643888] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.122651350000002, 2024-04-22T18:40:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1013670310000001, 2024-04-22T18:40:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3164 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837499+0000", 2024-04-22T18:40:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.994e-06 2024-04-22T18:40:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837503+0000", 2024-04-22T18:40:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7249999999999999e-06 2024-04-22T18:40:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837506+0000", 2024-04-22T18:40:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5980000000000001e-06 2024-04-22T18:40:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937641+0000", 2024-04-22T18:40:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10013467700000001 2024-04-22T18:40:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937650+0000", 2024-04-22T18:40:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3489999999999998e-06 2024-04-22T18:40:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938468+0000", 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0008176710000001 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938582+0000", 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114287 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938864+0000", 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028120899999999998 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3165 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.122632582000001, 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2022754499999999, 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3165 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837517+0000", 2024-04-22T18:40:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3369999999999998e-06 2024-04-22T18:40:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837523+0000", 2024-04-22T18:40:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5450000000000003e-06 2024-04-22T18:40:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837526+0000", 2024-04-22T18:40:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.354e-06 2024-04-22T18:40:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937682+0000", 2024-04-22T18:40:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100155917 2024-04-22T18:40:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937688+0000", 2024-04-22T18:40:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:40:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.038953+0000", 2024-04-22T18:40:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012648110000001 2024-04-22T18:40:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039034+0000", 2024-04-22T18:40:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0969000000000004e-05 2024-04-22T18:40:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039791+0000", 2024-04-22T18:40:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075672900000000004 2024-04-22T18:40:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3183 4.8 4:1bc4668a:::smithi08937929-21:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:40:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.924229384, 2024-04-22T18:40:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0040478260000001, 2024-04-22T18:40:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3183 2024-04-22T18:40:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:40:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:40:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035913+0000", 2024-04-22T18:40:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999943 2024-04-22T18:40:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035935+0000", 2024-04-22T18:40:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2232999999999999e-05 2024-04-22T18:40:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035969+0000", 2024-04-22T18:40:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3969e-05 2024-04-22T18:40:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035987+0000", 2024-04-22T18:40:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8397e-05 2024-04-22T18:40:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2916000000000003e-05 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0103e-05 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039841+0000", 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0037508959999999 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039966+0000", 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012514500000000001 2024-04-22T18:40:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3184 4.8 4:12d9f826:::smithi08937929-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.924141424999998, 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.00402868, 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3184 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036005+0000", 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036026+0000", 2024-04-22T18:40:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1092999999999999e-05 2024-04-22T18:40:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036030+0000", 2024-04-22T18:40:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.467e-06 2024-04-22T18:40:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036033+0000", 2024-04-22T18:40:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9780000000000001e-06 2024-04-22T18:40:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036120+0000", 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7495000000000002e-05 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036125+0000", 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8720000000000001e-06 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039982+0000", 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0038571919999999 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040035+0000", 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2992999999999999e-05 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3185 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.924087513, 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0041404039999999, 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3185 2024-04-22T18:40:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036062+0000", 2024-04-22T18:40:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9099999999999999e-06 2024-04-22T18:40:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036066+0000", 2024-04-22T18:40:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1699999999999999e-06 2024-04-22T18:40:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036077+0000", 2024-04-22T18:40:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0637e-05 2024-04-22T18:40:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036147+0000", 2024-04-22T18:40:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0025999999999997e-05 2024-04-22T18:40:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036151+0000", 2024-04-22T18:40:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:40:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040068+0000", 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.003916604 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040201+0000", 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013275400000000001 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3186 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.924067956999998, 2024-04-22T18:40:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2044167510000001, 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3186 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036079+0000", 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036081+0000", 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6449999999999999e-06 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036084+0000", 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9399999999999998e-06 2024-04-22T18:40:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.879e-06 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036168+0000", 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7657999999999994e-05 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036171+0000", 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7639999999999999e-06 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240345+0000", 2024-04-22T18:40:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2041734260000001 2024-04-22T18:40:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240497+0000", 2024-04-22T18:40:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015182800000000001 2024-04-22T18:40:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3723 4.18 4:1bc4668a:::smithi08937929-21:head [setxattr _header (56) in=7b,truncate 3277398] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:40:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:45.060393+0000", 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.8997551279999998, 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40389026300000003, 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3723 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060393+0000", 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060393+0000", 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060392+0000", 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060396+0000", 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7100000000000001e-06 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060399+0000", 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0400000000000001e-06 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060402+0000", 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6350000000000002e-06 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165420+0000", 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10501865000000001 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165496+0000", 2024-04-22T18:40:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5768999999999999e-05 2024-04-22T18:40:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165616+0000", 2024-04-22T18:40:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120024 2024-04-22T18:40:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165906+0000", 2024-04-22T18:40:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028992200000000002 2024-04-22T18:40:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.464190+0000", 2024-04-22T18:40:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29828380599999998 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.464229+0000", 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9495e-05 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.464283+0000", 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3701e-05 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3724 4.18 4:1bc4668a:::smithi08937929-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:40:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:40:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.8997435810000001, 2024-04-22T18:40:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50413114699999995, 2024-04-22T18:40:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3724 2024-04-22T18:40:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:40:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:40:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:40:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060405+0000", 2024-04-22T18:40:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.395e-06 2024-04-22T18:40:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060408+0000", 2024-04-22T18:40:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:40:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060411+0000", 2024-04-22T18:40:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6060000000000001e-06 2024-04-22T18:40:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165799+0000", 2024-04-22T18:40:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105388369 2024-04-22T18:40:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165843+0000", 2024-04-22T18:40:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3891999999999999e-05 2024-04-22T18:40:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564408+0000", 2024-04-22T18:40:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39856466000000002 2024-04-22T18:40:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564461+0000", 2024-04-22T18:40:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3529999999999997e-05 2024-04-22T18:40:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564535+0000", 2024-04-22T18:40:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4147999999999995e-05 2024-04-22T18:40:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:51.062 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:51.063 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:51.064 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:51.065+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:51.109 INFO:tasks.rados.rados.0.smithi089.stdout:1648: done (2 left) 2024-04-22T18:40:51.109 INFO:tasks.rados.rados.0.smithi089.stdout:1649: done (1 left) 2024-04-22T18:40:51.109 INFO:tasks.rados.rados.0.smithi089.stdout:1650: done (0 left) 2024-04-22T18:40:51.109 INFO:tasks.rados.rados.0.smithi089.stdout:1651: write (excl) oid 23 current snap is 0 2024-04-22T18:40:51.109 INFO:tasks.rados.rados.0.smithi089.stdout:1651: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2024-04-22T18:40:51.116 INFO:tasks.rados.rados.0.smithi089.stdout:1651: writing smithi08937929-23 from 497573 to 1126516 tid 1 2024-04-22T18:40:51.125 INFO:tasks.rados.rados.0.smithi089.stdout:1651: writing smithi08937929-23 from 1687631 to 2422408 tid 2 2024-04-22T18:40:51.133 INFO:tasks.rados.rados.0.smithi089.stdout:1651: writing smithi08937929-23 from 3093218 to 3697573 tid 3 2024-04-22T18:40:51.134 INFO:tasks.rados.rados.0.smithi089.stdout:1652: read oid 17 snap -1 2024-04-22T18:40:51.134 INFO:tasks.rados.rados.0.smithi089.stdout:1652: expect (ObjNum 321 snap 0 seq_num 321) 2024-04-22T18:40:51.134 INFO:tasks.rados.rados.0.smithi089.stdout:1653: read oid 9 snap -1 2024-04-22T18:40:51.134 INFO:tasks.rados.rados.0.smithi089.stdout:1653: expect (ObjNum 283 snap 0 seq_num 283) 2024-04-22T18:40:51.134 INFO:tasks.rados.rados.0.smithi089.stdout:1654: read oid 10 snap -1 2024-04-22T18:40:51.134 INFO:tasks.rados.rados.0.smithi089.stdout:1654: expect (ObjNum 336 snap 0 seq_num 336) 2024-04-22T18:40:51.134 INFO:tasks.rados.rados.0.smithi089.stdout:1655: set_chunk oid 7 target oid 2024-04-22T18:40:51.149 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:51.150+0000 7f3d00e8c640 -1 osd.6 115 handle_osd_map injecting map crc failure 2024-04-22T18:40:51.164 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:51.165+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:51.167 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:51.168 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:51.168 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:51.168 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:51.176 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:51.265 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:51.266+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:51.318 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:51.318 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:51.318 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:51.319 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:51.319 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:51.327 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:51.338 INFO:tasks.rados.rados.0.smithi089.stdout:1655: set_chunk oid 7 offset: 1308160 length: 892928 target oid a13c5d28375df9492f4c5792e5ba50a86d2d30b6761d3431a19bb7ac7af485f2 offset: 0 snap -1 2024-04-22T18:40:51.338 INFO:tasks.rados.rados.0.smithi089.stdout:1656: read oid 33 snap -1 2024-04-22T18:40:51.338 INFO:tasks.rados.rados.0.smithi089.stdout:1656: expect (ObjNum 319 snap 0 seq_num 319) 2024-04-22T18:40:51.339 INFO:tasks.rados.rados.0.smithi089.stdout:1653: expect (ObjNum 283 snap 0 seq_num 283) 2024-04-22T18:40:51.356 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:40:51.399 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:51.401+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:51.438 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:51.438 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:51.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:51.438 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:51.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.438 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:51.438 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:51.438 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 116.982580462, 2024-04-22T18:40:51.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:40:51.438 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.439 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.439 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:51.439 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:51.439 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.439 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:40:51.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.439 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:51.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:51.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:40:51.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:51.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:40:51.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:40:51.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:40:51.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:40:51.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:40:51.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:40:51.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:40:51.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:40:51.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:40:51.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:40:51.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:51.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:40:51.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:40:51.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:51.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:40:51.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:40:51.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:40:51.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:40:51.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:51.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:40:51.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:40:51.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:40:51.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:40:51.444 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.444 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.444 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.444 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:51.444 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:51.444 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.289523214, 2024-04-22T18:40:51.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:40:51.444 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.445 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.445 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:51.445 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:51.445 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.445 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:40:51.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.445 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:51.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:51.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:40:51.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:40:51.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:40:51.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:40:51.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:40:51.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:40:51.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:40:51.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:40:51.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:40:51.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:40:51.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:40:51.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:51.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:40:51.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:40:51.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:51.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:40:51.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:40:51.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:40:51.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:40:51.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:51.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:40:51.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:40:51.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:40:51.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:40:51.450 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.450 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.450 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.450 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:51.450 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:51.450 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.28950473800001, 2024-04-22T18:40:51.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:40:51.450 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.450 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:51.450 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:51.451 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:51.451 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.451 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:40:51.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.451 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:51.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:51.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:51.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:40:51.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:40:51.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:40:51.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:40:51.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:40:51.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:40:51.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:40:51.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:40:51.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:40:51.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:40:51.457 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.457 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.457 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.457 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:51.458 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:51.458 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 80.526456425999996, 2024-04-22T18:40:51.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:40:51.458 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.458 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.458 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:51.458 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:51.458 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.458 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:40:51.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.458 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:51.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:51.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:40:51.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:51.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:40:51.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:40:51.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:40:51.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:40:51.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:40:51.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:40:51.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:40:51.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:40:51.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:40:51.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:40:51.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:51.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:40:51.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:40:51.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:40:51.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:40:51.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:51.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:40:51.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:40:51.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:51.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:40:51.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:40:51.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:40:51.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:40:51.463 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.463 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.463 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.463 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:51.463 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:51.464 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.997297099999997, 2024-04-22T18:40:51.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:40:51.464 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.464 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:51.464 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:51.464 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:51.464 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.464 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:40:51.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.464 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:51.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:51.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:51.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:40:51.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:40:51.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:40:51.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:40:51.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:40:51.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:40:51.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:40:51.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:40:51.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:51.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:40:51.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:40:51.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:40:51.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:40:51.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:40:51.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:40:51.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:40:51.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:40:51.468 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.468 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.468 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.469 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:51.469 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:51.469 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.819589704999999, 2024-04-22T18:40:51.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:40:51.469 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.469 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.469 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:51.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:40:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:40:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:40:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:40:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:40:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:40:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:40:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:40:51.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:40:51.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:40:51.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:40:51.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:51.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:40:51.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:40:51.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:51.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:40:51.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:40:51.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:40:51.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:40:51.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:51.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:40:51.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:40:51.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:40:51.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:40:51.474 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.474 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.474 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.474 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:51.474 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:51.474 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.819521433000006, 2024-04-22T18:40:51.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:40:51.474 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.475 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.475 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:51.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:51.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:40:51.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:40:51.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:40:51.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:40:51.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:40:51.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:40:51.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:40:51.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:40:51.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:40:51.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:40:51.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:40:51.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:51.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:40:51.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:40:51.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:51.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:40:51.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:40:51.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:40:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:40:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:40:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:40:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:40:51.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:51.480 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.480 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.480 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.480 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:51.480 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:51.480 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.819353591999999, 2024-04-22T18:40:51.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:40:51.480 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.480 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.480 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:51.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:51.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:51.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:40:51.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:40:51.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:40:51.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:40:51.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:40:51.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:40:51.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:40:51.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:40:51.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:40:51.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:40:51.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:51.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:40:51.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:40:51.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:51.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:40:51.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:40:51.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:40:51.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:40:51.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:51.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:40:51.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:40:51.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:40:51.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:40:51.485 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.485 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.485 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.485 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:51.485 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:51.486 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 70.854028116999999, 2024-04-22T18:40:51.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:40:51.486 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.486 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.486 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:51.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:51.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:40:51.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:51.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:40:51.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:40:51.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:40:51.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:40:51.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:40:51.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:40:51.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:40:51.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:40:51.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:40:51.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:40:51.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:51.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:40:51.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:40:51.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:51.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:40:51.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:40:51.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:40:51.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:40:51.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:51.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:40:51.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:40:51.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:40:51.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:40:51.491 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.491 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.491 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.491 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:51.491 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:51.491 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 70.753875930000007, 2024-04-22T18:40:51.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:40:51.491 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.491 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.491 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:51.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:51.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:40:51.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:51.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:40:51.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:40:51.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:40:51.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:40:51.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:40:51.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:40:51.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:40:51.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:40:51.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:40:51.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:40:51.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:51.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:40:51.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:40:51.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:51.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:40:51.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:40:51.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:40:51.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:40:51.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:51.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:40:51.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:40:51.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:40:51.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:40:51.496 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.496 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.496 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.497 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:40:51.497 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:51.497 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.045011410000001, 2024-04-22T18:40:51.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:40:51.497 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.497 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:51.497 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:51.497 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:51.497 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.497 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:40:51.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.497 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:51.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:51.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:40:51.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:40:51.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:40:51.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:40:51.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:40:51.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:40:51.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:40:51.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:40:51.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:40:51.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:51.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:40:51.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:40:51.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:40:51.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:40:51.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:40:51.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:40:51.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:40:51.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:40:51.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.502 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.502 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:51.502 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:51.502 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 44.148973038000001, 2024-04-22T18:40:51.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:40:51.502 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.502 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:51.502 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:51.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:51.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.503 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:40:51.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.503 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:51.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:51.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:51.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:40:51.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:40:51.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:40:51.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:40:51.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:40:51.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:40:51.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:40:51.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:40:51.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:40:51.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:40:51.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:40:51.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:40:51.506 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.506 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.507 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.507 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3330 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-04-22T18:40:51.507 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:40:51.507 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.413894752000001, 2024-04-22T18:40:51.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43151673600000001, 2024-04-22T18:40:51.507 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.507 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:51.507 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:51.507 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:51.508 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.508 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3330 2024-04-22T18:40:51.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.508 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:40:51.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:40:51.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024041+0000", 2024-04-22T18:40:51.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:51.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024045+0000", 2024-04-22T18:40:51.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3950000000000001e-06 2024-04-22T18:40:51.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024051+0000", 2024-04-22T18:40:51.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0260000000000004e-06 2024-04-22T18:40:51.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024056+0000", 2024-04-22T18:40:51.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4489999999999998e-06 2024-04-22T18:40:51.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028383+0000", 2024-04-22T18:40:51.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004326557 2024-04-22T18:40:51.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028471+0000", 2024-04-22T18:40:51.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8387999999999995e-05 2024-04-22T18:40:51.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.455559+0000", 2024-04-22T18:40:51.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.42708770699999998 2024-04-22T18:40:51.511 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.511 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.511 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.511 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3377 4.6 4:6700eab5:::smithi08937929-32:head [read 1812992~307712] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-04-22T18:40:51.511 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:40:51.511 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.001569396000001, 2024-04-22T18:40:51.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.52001745799999999, 2024-04-22T18:40:51.512 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.512 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:51.512 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:51.512 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:51.512 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.512 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3377 2024-04-22T18:40:51.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.512 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:40:51.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:40:51.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436364+0000", 2024-04-22T18:40:51.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:51.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436376+0000", 2024-04-22T18:40:51.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2121e-05 2024-04-22T18:40:51.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436394+0000", 2024-04-22T18:40:51.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8094e-05 2024-04-22T18:40:51.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436405+0000", 2024-04-22T18:40:51.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0189e-05 2024-04-22T18:40:51.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436471+0000", 2024-04-22T18:40:51.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6817999999999996e-05 2024-04-22T18:40:51.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:51.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436481+0000", 2024-04-22T18:40:51.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2720000000000003e-06 2024-04-22T18:40:51.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956084+0000", 2024-04-22T18:40:51.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51960320800000004 2024-04-22T18:40:51.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956232+0000", 2024-04-22T18:40:51.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014778600000000001 2024-04-22T18:40:51.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956385+0000", 2024-04-22T18:40:51.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153394 2024-04-22T18:40:51.516 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.516 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.516 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.517 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:131 3.1 3:a0a19997:::ee8ca23a7050e771c77dff761e6ea5a11ed57796f4e8a17cbb5804a7fdc937b9:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:51.517 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:40:51.517 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.1086998, 2024-04-22T18:40:51.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51449564999999997, 2024-04-22T18:40:51.517 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.517 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.517 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:40:51.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:40:51.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329235+0000", 2024-04-22T18:40:51.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:51.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329244+0000", 2024-04-22T18:40:51.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6819999999999992e-06 2024-04-22T18:40:51.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329259+0000", 2024-04-22T18:40:51.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698000000000001e-05 2024-04-22T18:40:51.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329265+0000", 2024-04-22T18:40:51.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1510000000000001e-06 2024-04-22T18:40:51.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329311+0000", 2024-04-22T18:40:51.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6096e-05 2024-04-22T18:40:51.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:51.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.332269+0000", 2024-04-22T18:40:51.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029581910000000002 2024-04-22T18:40:51.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630603+0000", 2024-04-22T18:40:51.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29833418 2024-04-22T18:40:51.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630685+0000", 2024-04-22T18:40:51.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1217999999999994e-05 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630813+0000", 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128706 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.631447+0000", 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063427000000000004 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843586+0000", 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21213873 2024-04-22T18:40:51.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:51.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843623+0000", 2024-04-22T18:40:51.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6848999999999999e-05 2024-04-22T18:40:51.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843733+0000", 2024-04-22T18:40:51.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010982200000000001 2024-04-22T18:40:51.522 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.522 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.522 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.522 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3441 4.b 4:d3068ee3:::smithi08937929-29:head [write 558065~619869] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:51.523 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:40:51.523 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.520223738999999, 2024-04-22T18:40:51.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50272020399999995, 2024-04-22T18:40:51.523 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.523 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.523 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:51.523 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:51.523 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.523 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3441 2024-04-22T18:40:51.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.523 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:40:51.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:40:51.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917711+0000", 2024-04-22T18:40:51.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:51.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917838+0000", 2024-04-22T18:40:51.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012734899999999999 2024-04-22T18:40:51.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917854+0000", 2024-04-22T18:40:51.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6130999999999999e-05 2024-04-22T18:40:51.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917863+0000", 2024-04-22T18:40:51.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5969999999999992e-06 2024-04-22T18:40:51.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992845+0000", 2024-04-22T18:40:51.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.074982193000000003 2024-04-22T18:40:51.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992919+0000", 2024-04-22T18:40:51.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:40:51.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:51.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.993019+0000", 2024-04-22T18:40:51.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9718999999999999e-05 2024-04-22T18:40:51.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:51.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095554+0000", 2024-04-22T18:40:51.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102535193 2024-04-22T18:40:51.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420183+0000", 2024-04-22T18:40:51.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32462853600000002 2024-04-22T18:40:51.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:51.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420401+0000", 2024-04-22T18:40:51.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021881400000000001 2024-04-22T18:40:51.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420433+0000", 2024-04-22T18:40:51.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079000000000002e-05 2024-04-22T18:40:51.528 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.528 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.528 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.528 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3442 4.b 4:d3068ee3:::smithi08937929-29:head [write 1579835~486765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:51.528 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:40:51.528 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.515537028000001, 2024-04-22T18:40:51.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49822799099999998, 2024-04-22T18:40:51.529 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.529 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.529 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:51.529 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:51.529 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.529 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3442 2024-04-22T18:40:51.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.529 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:40:51.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:40:51.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922399+0000", 2024-04-22T18:40:51.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:51.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922502+0000", 2024-04-22T18:40:51.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102784 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922513+0000", 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1051e-05 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922518+0000", 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8529999999999997e-06 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093158+0000", 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.17063958800000001 2024-04-22T18:40:51.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093230+0000", 2024-04-22T18:40:51.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1803000000000004e-05 2024-04-22T18:40:51.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:51.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093326+0000", 2024-04-22T18:40:51.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6454999999999997e-05 2024-04-22T18:40:51.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:51.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095557+0000", 2024-04-22T18:40:51.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022306399999999999 2024-04-22T18:40:51.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420484+0000", 2024-04-22T18:40:51.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32492717700000001 2024-04-22T18:40:51.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:51.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420606+0000", 2024-04-22T18:40:51.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012225800000000001 2024-04-22T18:40:51.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420628+0000", 2024-04-22T18:40:51.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1596000000000001e-05 2024-04-22T18:40:51.534 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.534 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.534 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.534 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3443 4.b 4:d3068ee3:::smithi08937929-29:head [write 2468371~489694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:51.534 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:40:51.534 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.510934980999998, 2024-04-22T18:40:51.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49372487199999998, 2024-04-22T18:40:51.534 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.534 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.534 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:51.534 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:51.535 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.535 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3443 2024-04-22T18:40:51.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.535 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:40:51.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:40:51.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927001+0000", 2024-04-22T18:40:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927117+0000", 2024-04-22T18:40:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011637 2024-04-22T18:40:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927128+0000", 2024-04-22T18:40:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0696000000000001e-05 2024-04-22T18:40:51.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927134+0000", 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8640000000000001e-06 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094288+0000", 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16715412900000001 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094359+0000", 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1639999999999998e-05 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:51.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094451+0000", 2024-04-22T18:40:51.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1252e-05 2024-04-22T18:40:51.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:51.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095560+0000", 2024-04-22T18:40:51.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011095530000000001 2024-04-22T18:40:51.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420661+0000", 2024-04-22T18:40:51.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32510105500000003 2024-04-22T18:40:51.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:51.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420710+0000", 2024-04-22T18:40:51.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9018999999999999e-05 2024-04-22T18:40:51.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420727+0000", 2024-04-22T18:40:51.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6722e-05 2024-04-22T18:40:51.539 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.539 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.539 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.540 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3444 4.b 4:d3068ee3:::smithi08937929-29:head [setxattr _header (56) in=7b,truncate 2958065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:51.540 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:40:51.540 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.510799263999999, 2024-04-22T18:40:51.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49366935299999998, 2024-04-22T18:40:51.540 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.540 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:51.540 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:51.540 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:51.540 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.540 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3444 2024-04-22T18:40:51.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.540 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:40:51.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:40:51.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927137+0000", 2024-04-22T18:40:51.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927141+0000", 2024-04-22T18:40:51.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3270000000000001e-06 2024-04-22T18:40:51.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927144+0000", 2024-04-22T18:40:51.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9730000000000002e-06 2024-04-22T18:40:51.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927147+0000", 2024-04-22T18:40:51.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.959e-06 2024-04-22T18:40:51.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095118+0000", 2024-04-22T18:40:51.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16797115500000001 2024-04-22T18:40:51.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095169+0000", 2024-04-22T18:40:51.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1316000000000001e-05 2024-04-22T18:40:51.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:51.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095260+0000", 2024-04-22T18:40:51.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1184e-05 2024-04-22T18:40:51.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:51.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095999+0000", 2024-04-22T18:40:51.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073868200000000005 2024-04-22T18:40:51.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:51.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420757+0000", 2024-04-22T18:40:51.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32475783499999999 2024-04-22T18:40:51.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:51.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420781+0000", 2024-04-22T18:40:51.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4022000000000001e-05 2024-04-22T18:40:51.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420807+0000", 2024-04-22T18:40:51.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6366000000000001e-05 2024-04-22T18:40:51.545 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.545 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.545 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.545 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3445 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:51.545 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:51.545 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.510788170000001, 2024-04-22T18:40:51.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.493780049, 2024-04-22T18:40:51.545 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:51.546 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:51.546 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:51.546 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:51.546 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:51.546 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3445 2024-04-22T18:40:51.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.546 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:51.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:51.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:51.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927150+0000", 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.542e-06 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:51.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927153+0000", 2024-04-22T18:40:51.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0520000000000002e-06 2024-04-22T18:40:51.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:51.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027218+0000", 2024-04-22T18:40:51.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100064604 2024-04-22T18:40:51.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095501+0000", 2024-04-22T18:40:51.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.068283431000000006 2024-04-22T18:40:51.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:51.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095545+0000", 2024-04-22T18:40:51.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4295999999999997e-05 2024-04-22T18:40:51.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:51.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420833+0000", 2024-04-22T18:40:51.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32528718400000001 2024-04-22T18:40:51.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:51.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420877+0000", 2024-04-22T18:40:51.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4403999999999997e-05 2024-04-22T18:40:51.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:51.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:51.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:51.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420929+0000", 2024-04-22T18:40:51.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1848999999999998e-05 2024-04-22T18:40:51.550 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.550 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.550 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.550 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:51.550 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:51.550 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:51.552 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:51.552 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:51.552 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:51.552 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:51.553 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:51.553 INFO:tasks.rados.rados.0.smithi089.stdout:1652: expect (ObjNum 321 snap 0 seq_num 321) 2024-04-22T18:40:51.553 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:51.554 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:51.554 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:51.554 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:51.555 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:51.555 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:51.555 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:51.552+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:51.557 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:51.558 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:51.574 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:51.574 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:51.591 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:51.591 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:51.645 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.3 to 1 (10000) 2024-04-22T18:40:51.652 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:51.653+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:51.659 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-22T18:40:51.664 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:51.664 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:51.673 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:51.703 INFO:tasks.rados.rados.0.smithi089.stdout:1654: expect (ObjNum 336 snap 0 seq_num 336) 2024-04-22T18:40:51.753 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:51.754+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:51.807 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:51.807 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:51.816 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:51.827 INFO:tasks.rados.rados.0.smithi089.stdout:1656: expect (ObjNum 319 snap 0 seq_num 319) 2024-04-22T18:40:51.852 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:51.853+0000 7fec55bea640 -1 osd.7 117 handle_osd_map injecting map crc failure 2024-04-22T18:40:51.853 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:51.854+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:51.872 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:40:51.922 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:51.922 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:51.932 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:51.976 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:51.977+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:52.036 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:52.036 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:52.037 INFO:tasks.rados.rados.0.smithi089.stdout:1651: finishing write tid 1 to smithi08937929-23 2024-04-22T18:40:52.037 INFO:tasks.rados.rados.0.smithi089.stdout:1651: finishing write tid 2 to smithi08937929-23 2024-04-22T18:40:52.037 INFO:tasks.rados.rados.0.smithi089.stdout:1651: finishing write tid 3 to smithi08937929-23 2024-04-22T18:40:52.038 INFO:tasks.rados.rados.0.smithi089.stdout:1655: finishing set_chunk to oid 7 2024-04-22T18:40:52.038 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 60 (ObjNum 332 snap 0 seq_num 332) dirty exists 2024-04-22T18:40:52.038 INFO:tasks.rados.rados.0.smithi089.stdout:1651: finishing write tid 4 to smithi08937929-23 2024-04-22T18:40:52.038 INFO:tasks.rados.rados.0.smithi089.stdout:1652: done (5 left) 2024-04-22T18:40:52.038 INFO:tasks.rados.rados.0.smithi089.stdout:1653: done (4 left) 2024-04-22T18:40:52.038 INFO:tasks.rados.rados.0.smithi089.stdout:1654: done (3 left) 2024-04-22T18:40:52.038 INFO:tasks.rados.rados.0.smithi089.stdout:1655: done (2 left) 2024-04-22T18:40:52.038 INFO:tasks.rados.rados.0.smithi089.stdout:1656: done (1 left) 2024-04-22T18:40:52.038 INFO:tasks.rados.rados.0.smithi089.stdout:1657: tier_promote oid 43 2024-04-22T18:40:52.038 INFO:tasks.rados.rados.0.smithi089.stdout:1651: finishing write tid 5 to smithi08937929-23 2024-04-22T18:40:52.038 INFO:tasks.rados.rados.0.smithi089.stdout:1651: oid 23 updating version 0 to 161 2024-04-22T18:40:52.038 INFO:tasks.rados.rados.0.smithi089.stdout:1651: oid 23 version 161 is already newer than 158 2024-04-22T18:40:52.038 INFO:tasks.rados.rados.0.smithi089.stdout:1651: oid 23 version 161 is already newer than 160 2024-04-22T18:40:52.038 INFO:tasks.rados.rados.0.smithi089.stdout:1651: oid 23 version 161 is already newer than 159 2024-04-22T18:40:52.039 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 161 (ObjNum 339 snap 0 seq_num 339) dirty exists 2024-04-22T18:40:52.039 INFO:tasks.rados.rados.0.smithi089.stdout:1651: left oid 23 (ObjNum 339 snap 0 seq_num 339) 2024-04-22T18:40:52.039 INFO:tasks.rados.rados.0.smithi089.stdout:1651: done (1 left) 2024-04-22T18:40:52.039 INFO:tasks.rados.rados.0.smithi089.stdout:1658: set_chunk oid 4 target oid 2024-04-22T18:40:52.046 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:52.055 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:52.056+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:52.148 INFO:tasks.rados.rados.0.smithi089.stdout:1658: set_chunk oid 4 offset: 1502720 length: 887296 target oid 93887f057dbfea9853671a9de1944d1a640436cc8887897af31174a58a4a75f0 offset: 0 snap -1 2024-04-22T18:40:52.148 INFO:tasks.rados.rados.0.smithi089.stdout:1659: read oid 25 snap -1 2024-04-22T18:40:52.148 INFO:tasks.rados.rados.0.smithi089.stdout:1659: expect (ObjNum 326 snap 0 seq_num 326) 2024-04-22T18:40:52.148 INFO:tasks.rados.rados.0.smithi089.stdout:1660: tier_evict oid 45 2024-04-22T18:40:52.148 INFO:tasks.rados.rados.0.smithi089.stdout:1660: tier_evict oid 45 snap -1 2024-04-22T18:40:52.148 INFO:tasks.rados.rados.0.smithi089.stdout:1661: write (excl) oid 39 current snap is 0 2024-04-22T18:40:52.148 INFO:tasks.rados.rados.0.smithi089.stdout:1661: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-04-22T18:40:52.154 INFO:tasks.rados.rados.0.smithi089.stdout:1661: writing smithi08937929-39 from 514380 to 1218572 tid 1 2024-04-22T18:40:52.156 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:52.157+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:52.158 INFO:tasks.rados.rados.0.smithi089.stdout:1661: writing smithi08937929-39 from 1629760 to 2108195 tid 2 2024-04-22T18:40:52.164 INFO:tasks.rados.rados.0.smithi089.stdout:1661: writing smithi08937929-39 from 2604288 to 3046890 tid 3 2024-04-22T18:40:52.164 INFO:tasks.rados.rados.0.smithi089.stdout:1662: write (excl) oid 21 current snap is 0 2024-04-22T18:40:52.164 INFO:tasks.rados.rados.0.smithi089.stdout:1662: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-04-22T18:40:52.170 INFO:tasks.rados.rados.0.smithi089.stdout:1662: writing smithi08937929-21 from 531187 to 1310628 tid 1 2024-04-22T18:40:52.179 INFO:tasks.rados.rados.0.smithi089.stdout:1662: writing smithi08937929-21 from 2088242 to 2826688 tid 2 2024-04-22T18:40:52.184 INFO:tasks.rados.rados.0.smithi089.stdout:1662: writing smithi08937929-21 from 3431711 to 3731187 tid 3 2024-04-22T18:40:52.184 INFO:tasks.rados.rados.0.smithi089.stdout:1663: read oid 30 snap -1 2024-04-22T18:40:52.184 INFO:tasks.rados.rados.0.smithi089.stdout:1663: expect (ObjNum 327 snap 0 seq_num 327) 2024-04-22T18:40:52.184 INFO:tasks.rados.rados.0.smithi089.stdout:1664: write (excl) oid 26 current snap is 0 2024-04-22T18:40:52.184 INFO:tasks.rados.rados.0.smithi089.stdout:1664: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-04-22T18:40:52.189 INFO:tasks.rados.rados.0.smithi089.stdout:1664: writing smithi08937929-26 from 547994 to 1002684 tid 1 2024-04-22T18:40:52.191 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:52.191 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:52.196 INFO:tasks.rados.rados.0.smithi089.stdout:1664: writing smithi08937929-26 from 1746724 to 2228828 tid 2 2024-04-22T18:40:52.201 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:52.201 INFO:tasks.rados.rados.0.smithi089.stdout:1664: writing smithi08937929-26 from 2659134 to 3124280 tid 3 2024-04-22T18:40:52.205 INFO:tasks.rados.rados.0.smithi089.stdout:1664: writing smithi08937929-26 from 3605976 to 3747994 tid 4 2024-04-22T18:40:52.205 INFO:tasks.rados.rados.0.smithi089.stdout:1665: read oid 3 snap -1 2024-04-22T18:40:52.205 INFO:tasks.rados.rados.0.smithi089.stdout:1665: expect (ObjNum 259 snap 0 seq_num 259) 2024-04-22T18:40:52.205 INFO:tasks.rados.rados.0.smithi089.stdout:1666: read oid 23 snap -1 2024-04-22T18:40:52.205 INFO:tasks.rados.rados.0.smithi089.stdout:1666: expect (ObjNum 339 snap 0 seq_num 339) 2024-04-22T18:40:52.205 INFO:tasks.rados.rados.0.smithi089.stdout:1667: tier_flush oid 31 2024-04-22T18:40:52.205 INFO:tasks.rados.rados.0.smithi089.stdout:1667: tier_flush oid 31 snap -1 2024-04-22T18:40:52.205 INFO:tasks.rados.rados.0.smithi089.stdout:1663: expect (ObjNum 327 snap 0 seq_num 327) 2024-04-22T18:40:52.257 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:52.258+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:52.303 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:52.303 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:52.313 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:52.398 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:52.399+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:52.446 INFO:tasks.rados.rados.0.smithi089.stdout:1662: finishing write tid 1 to smithi08937929-21 2024-04-22T18:40:52.446 INFO:tasks.rados.rados.0.smithi089.stdout:1662: finishing write tid 2 to smithi08937929-21 2024-04-22T18:40:52.446 INFO:tasks.rados.rados.0.smithi089.stdout:1662: finishing write tid 3 to smithi08937929-21 2024-04-22T18:40:52.446 INFO:tasks.rados.rados.0.smithi089.stdout:1663: done (10 left) 2024-04-22T18:40:52.447 INFO:tasks.rados.rados.0.smithi089.stdout:1668: tier_evict oid 22 2024-04-22T18:40:52.447 INFO:tasks.rados.rados.0.smithi089.stdout:1668: tier_evict oid 22 snap -1 2024-04-22T18:40:52.447 INFO:tasks.rados.rados.0.smithi089.stdout:1666: expect (ObjNum 339 snap 0 seq_num 339) 2024-04-22T18:40:52.458 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:52.459+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:52.459 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:52.459 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:52.468 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:52.559 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:52.560+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:52.613 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:52.613 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:52.622 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:52.660 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:52.661+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:52.671 INFO:tasks.rados.rados.0.smithi089.stdout:1662: finishing write tid 4 to smithi08937929-21 2024-04-22T18:40:52.673 INFO:tasks.rados.rados.0.smithi089.stdout:1662: finishing write tid 5 to smithi08937929-21 2024-04-22T18:40:52.674 INFO:tasks.rados.rados.0.smithi089.stdout:1662: oid 21 updating version 0 to 59 2024-04-22T18:40:52.674 INFO:tasks.rados.rados.0.smithi089.stdout:1662: oid 21 version 59 is already newer than 58 2024-04-22T18:40:52.674 INFO:tasks.rados.rados.0.smithi089.stdout:1662: oid 21 updating version 59 to 60 2024-04-22T18:40:52.674 INFO:tasks.rados.rados.0.smithi089.stdout:1662: oid 21 version 60 is already newer than 57 2024-04-22T18:40:52.674 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 60 (ObjNum 341 snap 0 seq_num 341) dirty exists 2024-04-22T18:40:52.674 INFO:tasks.rados.rados.0.smithi089.stdout:1662: left oid 21 (ObjNum 341 snap 0 seq_num 341) 2024-04-22T18:40:52.674 INFO:tasks.rados.rados.0.smithi089.stdout:1668: got (22) Invalid argument 2024-04-22T18:40:52.674 INFO:tasks.rados.rados.0.smithi089.stdout:1658: finishing set_chunk to oid 4 2024-04-22T18:40:52.674 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 4 set_chunk returned error code -95 offset: 1502720 length: 887296 2024-04-22T18:40:52.674 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 0 (ObjNum 269 snap 0 seq_num 269) dirty exists 2024-04-22T18:40:52.674 INFO:tasks.rados.rados.0.smithi089.stdout:1660: got (22) Invalid argument 2024-04-22T18:40:52.674 INFO:tasks.rados.rados.0.smithi089.stdout:1661: finishing write tid 1 to smithi08937929-39 2024-04-22T18:40:52.674 INFO:tasks.rados.rados.0.smithi089.stdout:1661: finishing write tid 2 to smithi08937929-39 2024-04-22T18:40:52.674 INFO:tasks.rados.rados.0.smithi089.stdout:1661: finishing write tid 3 to smithi08937929-39 2024-04-22T18:40:52.675 INFO:tasks.rados.rados.0.smithi089.stdout:1658: done (10 left) 2024-04-22T18:40:52.675 INFO:tasks.rados.rados.0.smithi089.stdout:1660: done (9 left) 2024-04-22T18:40:52.675 INFO:tasks.rados.rados.0.smithi089.stdout:1662: done (8 left) 2024-04-22T18:40:52.675 INFO:tasks.rados.rados.0.smithi089.stdout:1666: done (7 left) 2024-04-22T18:40:52.675 INFO:tasks.rados.rados.0.smithi089.stdout:1668: done (6 left) 2024-04-22T18:40:52.675 INFO:tasks.rados.rados.0.smithi089.stdout:1669: read oid 37 snap -1 2024-04-22T18:40:52.675 INFO:tasks.rados.rados.0.smithi089.stdout:1669: expect (ObjNum 267 snap 0 seq_num 267) 2024-04-22T18:40:52.675 INFO:tasks.rados.rados.0.smithi089.stdout:1670: read oid 33 snap -1 2024-04-22T18:40:52.675 INFO:tasks.rados.rados.0.smithi089.stdout:1670: expect (ObjNum 319 snap 0 seq_num 319) 2024-04-22T18:40:52.675 INFO:tasks.rados.rados.0.smithi089.stdout:1671: read oid 17 snap -1 2024-04-22T18:40:52.675 INFO:tasks.rados.rados.0.smithi089.stdout:1671: expect (ObjNum 321 snap 0 seq_num 321) 2024-04-22T18:40:52.675 INFO:tasks.rados.rados.0.smithi089.stdout:1672: set_chunk oid 11 target oid 2024-04-22T18:40:52.761 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:52.762+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:52.765 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:52.765 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:52.775 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:52.862 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:52.863+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:52.866 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:52.867+0000 7f3d00e8c640 -1 osd.6 118 handle_osd_map injecting map crc failure 2024-04-22T18:40:52.868 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.6 to 1 (10000) 2024-04-22T18:40:52.868 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:40:52.884 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-22T18:40:52.886 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:52.886 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:52.887 INFO:tasks.rados.rados.0.smithi089.stdout:1672: set_chunk oid 11 offset: 1109504 length: 1924096 target oid 9d9fc9b2c065bbd7b92d6791d0de09f4410857583cb983d0ed41c87330e430f2 offset: 0 snap -1 2024-04-22T18:40:52.887 INFO:tasks.rados.rados.0.smithi089.stdout:1673: read oid 2 snap -1 2024-04-22T18:40:52.887 INFO:tasks.rados.rados.0.smithi089.stdout:1673: expect (ObjNum 251 snap 0 seq_num 251) 2024-04-22T18:40:52.887 INFO:tasks.rados.rados.0.smithi089.stdout:1674: read oid 23 snap -1 2024-04-22T18:40:52.887 INFO:tasks.rados.rados.0.smithi089.stdout:1674: expect (ObjNum 339 snap 0 seq_num 339) 2024-04-22T18:40:52.887 INFO:tasks.rados.rados.0.smithi089.stdout:1675: read oid 42 snap -1 2024-04-22T18:40:52.887 INFO:tasks.rados.rados.0.smithi089.stdout:1675: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:52.887 INFO:tasks.rados.rados.0.smithi089.stdout:1676: write (excl) oid 46 current snap is 0 2024-04-22T18:40:52.887 INFO:tasks.rados.rados.0.smithi089.stdout:1659: expect (ObjNum 326 snap 0 seq_num 326) 2024-04-22T18:40:52.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:52.962 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:52.964+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:53.038 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:53.038 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:53.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:53.063 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:53.065+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:53.147 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:53.148 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:53.156 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:53.170 INFO:tasks.rados.rados.0.smithi089.stdout:1661: finishing write tid 4 to smithi08937929-39 2024-04-22T18:40:53.170 INFO:tasks.rados.rados.0.smithi089.stdout:1661: finishing write tid 5 to smithi08937929-39 2024-04-22T18:40:53.170 INFO:tasks.rados.rados.0.smithi089.stdout:1661: oid 39 updating version 0 to 122 2024-04-22T18:40:53.171 INFO:tasks.rados.rados.0.smithi089.stdout:1661: oid 39 updating version 122 to 123 2024-04-22T18:40:53.171 INFO:tasks.rados.rados.0.smithi089.stdout:1661: oid 39 updating version 123 to 125 2024-04-22T18:40:53.171 INFO:tasks.rados.rados.0.smithi089.stdout:1661: oid 39 version 125 is already newer than 124 2024-04-22T18:40:53.171 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 125 (ObjNum 340 snap 0 seq_num 340) dirty exists 2024-04-22T18:40:53.171 INFO:tasks.rados.rados.0.smithi089.stdout:1661: left oid 39 (ObjNum 340 snap 0 seq_num 340) 2024-04-22T18:40:53.171 INFO:tasks.rados.rados.0.smithi089.stdout:1667: got (2) No such file or directory 2024-04-22T18:40:53.171 INFO:tasks.rados.rados.0.smithi089.stdout:1667: got expected ENOENT (src dne) 2024-04-22T18:40:53.171 INFO:tasks.rados.rados.0.smithi089.stdout:1665: expect (ObjNum 259 snap 0 seq_num 259) 2024-04-22T18:40:53.200 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:53.201+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:53.245 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:53.245+0000 7f7f42baf640 -1 osd.4 118 handle_osd_map injecting map crc failure 2024-04-22T18:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:53.257 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:53.266 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:53.266+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:53.266 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:53.330 INFO:tasks.rados.rados.0.smithi089.stdout:1657: got (0) Success 2024-04-22T18:40:53.330 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 51 (ObjNum 226 snap 0 seq_num 226) dirty exists 2024-04-22T18:40:53.330 INFO:tasks.rados.rados.0.smithi089.stdout:1676: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2024-04-22T18:40:53.336 INFO:tasks.rados.rados.0.smithi089.stdout:1676: writing smithi08937929-46 from 564801 to 1211093 tid 1 2024-04-22T18:40:53.344 INFO:tasks.rados.rados.0.smithi089.stdout:1676: writing smithi08937929-46 from 1921559 to 2663674 tid 2 2024-04-22T18:40:53.350 INFO:tasks.rados.rados.0.smithi089.stdout:1676: writing smithi08937929-46 from 3202910 to 3764801 tid 3 2024-04-22T18:40:53.350 INFO:tasks.rados.rados.0.smithi089.stdout:1657: done (13 left) 2024-04-22T18:40:53.350 INFO:tasks.rados.rados.0.smithi089.stdout:1659: done (12 left) 2024-04-22T18:40:53.350 INFO:tasks.rados.rados.0.smithi089.stdout:1661: done (11 left) 2024-04-22T18:40:53.350 INFO:tasks.rados.rados.0.smithi089.stdout:1665: done (10 left) 2024-04-22T18:40:53.350 INFO:tasks.rados.rados.0.smithi089.stdout:1667: done (9 left) 2024-04-22T18:40:53.350 INFO:tasks.rados.rados.0.smithi089.stdout:1677: tier_promote oid 20 2024-04-22T18:40:53.350 INFO:tasks.rados.rados.0.smithi089.stdout:1671: expect (ObjNum 321 snap 0 seq_num 321) 2024-04-22T18:40:53.367 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:53.368+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:53.409 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:53.409 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:53.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:53.468 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:53.469+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:53.518 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:53.519 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:53.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:53.537 INFO:tasks.rados.rados.0.smithi089.stdout:1669: expect (ObjNum 267 snap 0 seq_num 267) 2024-04-22T18:40:53.571 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:53.572+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:53.602 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:40:53.630 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:53.630 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:53.639 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:53.647 INFO:tasks.rados.rados.0.smithi089.stdout:1664: finishing write tid 1 to smithi08937929-26 2024-04-22T18:40:53.647 INFO:tasks.rados.rados.0.smithi089.stdout:1670: expect (ObjNum 319 snap 0 seq_num 319) 2024-04-22T18:40:53.680 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:53.681+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:53.720 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:53.720 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:53.736 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:53.736 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:53.752 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:53.752 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:53.768 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:53.768 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:53.770 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:53.771+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:53.781 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:53.781 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:53.784 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:53.785 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:53.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:53.800 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:53.800 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:53.817 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:53.817 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:53.833 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:53.833 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:53.855 INFO:tasks.rados.rados.0.smithi089.stdout:1675: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:53.871 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:53.872+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:53.877 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.7 to 1 (10000) 2024-04-22T18:40:53.890 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-22T18:40:53.936 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:53.936 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:53.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 119.67694564, 2024-04-22T18:40:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:40:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:53.937 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:40:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:40:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:53.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:40:53.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:53.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:53.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:40:53.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:40:53.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:53.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:40:53.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:40:53.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:53.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:40:53.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:40:53.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:53.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:40:53.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:40:53.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:53.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:40:53.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:40:53.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:53.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:40:53.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:40:53.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:53.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:40:53.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:40:53.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:53.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:40:53.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:40:53.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:53.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:40:53.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:40:53.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:53.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:40:53.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:40:53.942 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.942 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:53.942 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.942 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:53.943 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:53.943 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 119.67216320999999, 2024-04-22T18:40:53.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:40:53.943 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:53.943 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:53.943 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:53.943 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:53.943 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:53.943 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:40:53.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.943 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:53.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:53.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:53.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:53.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:40:53.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:53.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:40:53.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:53.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:53.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:40:53.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:40:53.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:53.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:40:53.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:40:53.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:53.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:40:53.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:40:53.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:53.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:40:53.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:40:53.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:53.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:40:53.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:40:53.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:53.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:40:53.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:40:53.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:53.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:40:53.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:40:53.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:53.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:40:53.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:40:53.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:53.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:40:53.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:40:53.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:53.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:40:53.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:40:53.948 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.948 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:53.948 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.948 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:53.948 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:53.948 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 119.672069007, 2024-04-22T18:40:53.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:40:53.949 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:53.949 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:53.949 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:53.949 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:53.949 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:53.949 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:40:53.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.949 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:53.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:53.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:53.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:53.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:40:53.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:53.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:40:53.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:53.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:53.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:40:53.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:40:53.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:53.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:40:53.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:40:53.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:53.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:40:53.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:40:53.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:53.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:40:53.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:40:53.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:53.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:40:53.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:40:53.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:53.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:40:53.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:40:53.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:53.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:40:53.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:40:53.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:53.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:40:53.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:40:53.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:53.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:40:53.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:40:53.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:53.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:40:53.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:40:53.954 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.954 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:53.954 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.954 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:53.954 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:53.954 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 119.672048119, 2024-04-22T18:40:53.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:40:53.954 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:53.955 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:53.955 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:53.955 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:53.955 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:53.955 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:40:53.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.955 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:53.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:53.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:53.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:53.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:40:53.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:40:53.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:53.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:40:53.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:40:53.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:53.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:40:53.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:40:53.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:53.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:40:53.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:40:53.959 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.959 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:53.959 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.959 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:53.959 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:53.959 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 118.889040456, 2024-04-22T18:40:53.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:40:53.959 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:53.959 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:53.959 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:53.959 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:53.959 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:53.959 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:40:53.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.960 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:53.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:53.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:53.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:53.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:40:53.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:53.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:40:53.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:40:53.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:53.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:40:53.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:40:53.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:53.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:40:53.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:40:53.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:53.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:40:53.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:40:53.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:53.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:40:53.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:40:53.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:53.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:40:53.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:40:53.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:53.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:40:53.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:40:53.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:53.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:40:53.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:40:53.964 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.964 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:53.964 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.964 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:53.964 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:53.964 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 118.88120905, 2024-04-22T18:40:53.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:40:53.965 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:53.965 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:53.965 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:53.965 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:53.965 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:53.965 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:40:53.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.965 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:53.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:53.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:53.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:53.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:40:53.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:53.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:40:53.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:53.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:53.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:40:53.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:40:53.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:53.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:40:53.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:40:53.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:53.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:40:53.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:40:53.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:53.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:40:53.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:40:53.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:53.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:40:53.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:40:53.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:53.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:40:53.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:40:53.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:53.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:40:53.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:40:53.970 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.970 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:53.970 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.970 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:53.970 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:53.970 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 58.729336420000003, 2024-04-22T18:40:53.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:40:53.970 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:53.970 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:53.970 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:53.970 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:53.971 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:53.971 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:40:53.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.971 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:53.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:53.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:53.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:53.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:40:53.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:53.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:40:53.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:53.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:53.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:40:53.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:40:53.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:53.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:40:53.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:40:53.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:53.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:40:53.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:40:53.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:53.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:40:53.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:40:53.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:53.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:40:53.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:40:53.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:53.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:40:53.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:40:53.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:53.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:40:53.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:40:53.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:53.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:40:53.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:40:53.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:53.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:40:53.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:40:53.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:53.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:40:53.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:40:53.975 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.975 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:53.975 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.976 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:40:53.976 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:53.976 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 58.728931512000003, 2024-04-22T18:40:53.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:40:53.976 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:53.976 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:53.976 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:53.976 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:53.976 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:53.976 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:40:53.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.976 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:53.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:53.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:53.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:53.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:53.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:53.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:40:53.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:53.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:53.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:40:53.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:40:53.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:53.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:40:53.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:40:53.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:53.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:40:53.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:40:53.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:53.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:40:53.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:40:53.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:53.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:40:53.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:40:53.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:53.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:40:53.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:40:53.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3213 4.a 4:5d2aee44:::smithi08937929-13:head [assert-version v66,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:53.980 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:40:53.981 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 31.368031624, 2024-04-22T18:40:53.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69928701400000004, 2024-04-22T18:40:53.981 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:53.981 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:53.981 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:53.981 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:53.981 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:53.981 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3213 2024-04-22T18:40:53.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.981 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:53.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:53.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:40:53.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:53.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:40:53.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:53.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568559+0000", 2024-04-22T18:40:53.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:53.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:53.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568567+0000", 2024-04-22T18:40:53.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-04-22T18:40:53.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:53.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568579+0000", 2024-04-22T18:40:53.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-04-22T18:40:53.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:53.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.668711+0000", 2024-04-22T18:40:53.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100131866 2024-04-22T18:40:53.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:53.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.768945+0000", 2024-04-22T18:40:53.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100234252 2024-04-22T18:40:53.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:53.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769002+0000", 2024-04-22T18:40:53.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7092999999999997e-05 2024-04-22T18:40:53.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:53.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.166404+0000", 2024-04-22T18:40:53.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39740241999999998 2024-04-22T18:40:53.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:53.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.191686+0000", 2024-04-22T18:40:53.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025281739000000001 2024-04-22T18:40:53.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:53.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267758+0000", 2024-04-22T18:40:53.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076071606999999999 2024-04-22T18:40:53.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:53.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267812+0000", 2024-04-22T18:40:53.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4478999999999998e-05 2024-04-22T18:40:53.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.985 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:53.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267848+0000", 2024-04-22T18:40:53.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6062999999999997e-05 2024-04-22T18:40:53.986 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.986 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:53.986 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.986 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3319 4.a 4:5d2aee44:::smithi08937929-13:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:53.986 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:40:53.986 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.476283873, 2024-04-22T18:40:53.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476623401, 2024-04-22T18:40:53.986 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:53.986 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:53.987 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:53.987 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:53.987 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:53.987 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3319 2024-04-22T18:40:53.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.987 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:53.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:53.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:40:53.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:53.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:40:53.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:53.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460306+0000", 2024-04-22T18:40:53.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:53.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:53.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460318+0000", 2024-04-22T18:40:53.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2896e-05 2024-04-22T18:40:53.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460345+0000", 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6650000000000001e-05 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460360+0000", 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4729000000000001e-05 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460502+0000", 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141729 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:53.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460644+0000", 2024-04-22T18:40:53.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142755 2024-04-22T18:40:53.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:53.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.936932+0000", 2024-04-22T18:40:53.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476288078 2024-04-22T18:40:53.990 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.990 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:53.990 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.990 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3412 4.a 4:519c08f7:::smithi08937929-15:head [write 2529878~394573] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:53.990 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:40:53.990 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.794449312000001, 2024-04-22T18:40:53.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48918094899999998, 2024-04-22T18:40:53.991 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:53.991 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:53.991 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:53.991 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:53.991 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:53.991 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3412 2024-04-22T18:40:53.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.991 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:53.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:53.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:40:53.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:53.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:40:53.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:53.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142141+0000", 2024-04-22T18:40:53.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:53.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:53.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142496+0000", 2024-04-22T18:40:53.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035444699999999998 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142510+0000", 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3841e-05 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142518+0000", 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8089999999999993e-06 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142560+0000", 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2097e-05 2024-04-22T18:40:53.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:53.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142637+0000", 2024-04-22T18:40:53.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6965000000000006e-05 2024-04-22T18:40:53.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:53.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142715+0000", 2024-04-22T18:40:53.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8694000000000002e-05 2024-04-22T18:40:53.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:53.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144179+0000", 2024-04-22T18:40:53.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146342 2024-04-22T18:40:53.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:53.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631229+0000", 2024-04-22T18:40:53.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48705018500000002 2024-04-22T18:40:53.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:53.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631281+0000", 2024-04-22T18:40:53.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2216999999999998e-05 2024-04-22T18:40:53.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:53.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631324+0000", 2024-04-22T18:40:53.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3344999999999999e-05 2024-04-22T18:40:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3413 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2924451] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:40:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.794068171999999, 2024-04-22T18:40:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58904086899999997, 2024-04-22T18:40:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:53.996 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3413 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142523+0000", 2024-04-22T18:40:53.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:53.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:53.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142529+0000", 2024-04-22T18:40:53.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6049999999999997e-06 2024-04-22T18:40:53.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:53.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142535+0000", 2024-04-22T18:40:53.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4740000000000002e-06 2024-04-22T18:40:53.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:53.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142541+0000", 2024-04-22T18:40:53.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1839999999999998e-06 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143553+0000", 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010126689999999999 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143659+0000", 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105393 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143825+0000", 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165809 2024-04-22T18:40:53.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144329+0000", 2024-04-22T18:40:54.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050483799999999997 2024-04-22T18:40:54.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731480+0000", 2024-04-22T18:40:54.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58715100600000003 2024-04-22T18:40:54.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731522+0000", 2024-04-22T18:40:54.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2098000000000002e-05 2024-04-22T18:40:54.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731566+0000", 2024-04-22T18:40:54.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3066999999999999e-05 2024-04-22T18:40:54.001 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.001 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.001 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.001 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3414 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:54.001 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:40:54.001 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 22.794039567999999, 2024-04-22T18:40:54.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58926020199999996, 2024-04-22T18:40:54.001 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.002 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:54.002 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.002 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:54.002 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.002 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3414 2024-04-22T18:40:54.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.002 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:40:54.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:40:54.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142552+0000", 2024-04-22T18:40:54.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:54.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142556+0000", 2024-04-22T18:40:54.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0289999999999999e-06 2024-04-22T18:40:54.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142574+0000", 2024-04-22T18:40:54.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7266999999999999e-05 2024-04-22T18:40:54.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142581+0000", 2024-04-22T18:40:54.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-04-22T18:40:54.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144121+0000", 2024-04-22T18:40:54.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015397480000000001 2024-04-22T18:40:54.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:54.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144170+0000", 2024-04-22T18:40:54.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8803e-05 2024-04-22T18:40:54.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731624+0000", 2024-04-22T18:40:54.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58745365800000005 2024-04-22T18:40:54.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731712+0000", 2024-04-22T18:40:54.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8016999999999994e-05 2024-04-22T18:40:54.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.006 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731813+0000", 2024-04-22T18:40:54.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101817 2024-04-22T18:40:54.006 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.006 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.006 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.006 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:77 3.3 3:dcc8aeec:::3590d5741efdda41a26b62a0982727439225b7b4b51097b313f30d8dc10bf6f6:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:54.006 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:40:54.006 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.689021245999999, 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011812878, 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247571+0000", 2024-04-22T18:40:54.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247574+0000", 2024-04-22T18:40:54.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:40:54.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247577+0000", 2024-04-22T18:40:54.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3069999999999998e-06 2024-04-22T18:40:54.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247580+0000", 2024-04-22T18:40:54.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8870000000000001e-06 2024-04-22T18:40:54.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.340928+0000", 2024-04-22T18:40:54.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093348678000000004 2024-04-22T18:40:54.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:40:54.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341122+0000", 2024-04-22T18:40:54.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000193227 2024-04-22T18:40:54.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618335+0000", 2024-04-22T18:40:54.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.277213775 2024-04-22T18:40:54.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:54.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618422+0000", 2024-04-22T18:40:54.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6537000000000002e-05 2024-04-22T18:40:54.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954387+0000", 2024-04-22T18:40:54.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33596509699999999 2024-04-22T18:40:54.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954548+0000", 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016042199999999999 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954858+0000", 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000310396 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157019+0000", 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20216071299999999 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259213+0000", 2024-04-22T18:40:54.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102194734 2024-04-22T18:40:54.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259262+0000", 2024-04-22T18:40:54.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9052999999999999e-05 2024-04-22T18:40:54.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259384+0000", 2024-04-22T18:40:54.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122052 2024-04-22T18:40:54.012 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.012 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:78 3.3 3:df7e90d2:::ab0657e24bd96704775de2a378746bc22309c5a7366a5eaf9b4e1f1f2bef0a4c:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.689006049, 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011974739, 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:40:54.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247586+0000", 2024-04-22T18:40:54.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247589+0000", 2024-04-22T18:40:54.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.181e-06 2024-04-22T18:40:54.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247592+0000", 2024-04-22T18:40:54.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0230000000000001e-06 2024-04-22T18:40:54.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247595+0000", 2024-04-22T18:40:54.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.428e-06 2024-04-22T18:40:54.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341160+0000", 2024-04-22T18:40:54.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093564571999999999 2024-04-22T18:40:54.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:40:54.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341264+0000", 2024-04-22T18:40:54.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010391999999999999 2024-04-22T18:40:54.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619269+0000", 2024-04-22T18:40:54.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27800575100000002 2024-04-22T18:40:54.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:54.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619346+0000", 2024-04-22T18:40:54.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6459999999999996e-05 2024-04-22T18:40:54.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.155957+0000", 2024-04-22T18:40:54.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53661109299999998 2024-04-22T18:40:54.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156049+0000", 2024-04-22T18:40:54.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2272e-05 2024-04-22T18:40:54.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156236+0000", 2024-04-22T18:40:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000186808 2024-04-22T18:40:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.158088+0000", 2024-04-22T18:40:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018522079999999999 2024-04-22T18:40:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259421+0000", 2024-04-22T18:40:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101332718 2024-04-22T18:40:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259460+0000", 2024-04-22T18:40:54.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8912000000000003e-05 2024-04-22T18:40:54.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259562+0000", 2024-04-22T18:40:54.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010175600000000001 2024-04-22T18:40:54.019 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.019 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.019 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.019 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:80 3.3 3:f90f47f2:::e5603f596d699ee3e27e255225eeb90a9f0c1b4a4d099d1c586b19019220aa3f:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:54.019 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:40:54.020 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.688966971999999, 2024-04-22T18:40:54.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0121096759999999, 2024-04-22T18:40:54.020 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.020 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.020 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:40:54.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:40:54.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247625+0000", 2024-04-22T18:40:54.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247628+0000", 2024-04-22T18:40:54.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6129999999999998e-06 2024-04-22T18:40:54.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247631+0000", 2024-04-22T18:40:54.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3869999999999999e-06 2024-04-22T18:40:54.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247634+0000", 2024-04-22T18:40:54.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9919999999999999e-06 2024-04-22T18:40:54.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341550+0000", 2024-04-22T18:40:54.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093915584999999996 2024-04-22T18:40:54.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:40:54.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341700+0000", 2024-04-22T18:40:54.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015000300000000001 2024-04-22T18:40:54.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620744+0000", 2024-04-22T18:40:54.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27904399200000002 2024-04-22T18:40:54.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:54.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620794+0000", 2024-04-22T18:40:54.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9579000000000001e-05 2024-04-22T18:40:54.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157597+0000", 2024-04-22T18:40:54.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53680343799999997 2024-04-22T18:40:54.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157678+0000", 2024-04-22T18:40:54.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0829999999999997e-05 2024-04-22T18:40:54.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:54.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157832+0000", 2024-04-22T18:40:54.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154184 2024-04-22T18:40:54.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259572+0000", 2024-04-22T18:40:54.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10174003600000001 2024-04-22T18:40:54.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259601+0000", 2024-04-22T18:40:54.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8841999999999999e-05 2024-04-22T18:40:54.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259638+0000", 2024-04-22T18:40:54.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7308999999999998e-05 2024-04-22T18:40:54.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259736+0000", 2024-04-22T18:40:54.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7283999999999999e-05 2024-04-22T18:40:54.026 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.026 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.026 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.026 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:135 3.3 3:f89134e5:::3e4e73f5d4a0122957865f0faeac2483204ee98d7d497ef3b54afe2a02b39a3d:head [read 0~802304 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:54.026 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:40:54.026 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.321151893, 2024-04-22T18:40:54.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.64370653499999997, 2024-04-22T18:40:54.026 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.026 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:54.027 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:40:54.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:40:54.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615438+0000", 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615449+0000", 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1219999999999999e-05 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615460+0000", 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0403e-05 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.715588+0000", 2024-04-22T18:40:54.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100128881 2024-04-22T18:40:54.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722533+0000", 2024-04-22T18:40:54.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069444010000000002 2024-04-22T18:40:54.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:40:54.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722658+0000", 2024-04-22T18:40:54.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012560699999999999 2024-04-22T18:40:54.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826437+0000", 2024-04-22T18:40:54.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103778194 2024-04-22T18:40:54.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:54.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826532+0000", 2024-04-22T18:40:54.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5111000000000005e-05 2024-04-22T18:40:54.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258787+0000", 2024-04-22T18:40:54.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43225529099999999 2024-04-22T18:40:54.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258905+0000", 2024-04-22T18:40:54.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118348 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259148+0000", 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000242093 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3890 4.1 4:83c955bf:::smithi08937929-28:head [write 2699879~469889] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.3224770489999997, 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49042297000000001, 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.031 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3890 2024-04-22T18:40:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:40:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:40:54.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614115+0000", 2024-04-22T18:40:54.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:54.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614496+0000", 2024-04-22T18:40:54.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038181700000000001 2024-04-22T18:40:54.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614509+0000", 2024-04-22T18:40:54.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2381000000000001e-05 2024-04-22T18:40:54.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614517+0000", 2024-04-22T18:40:54.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0879999999999995e-06 2024-04-22T18:40:54.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615610+0000", 2024-04-22T18:40:54.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010933519999999999 2024-04-22T18:40:54.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615655+0000", 2024-04-22T18:40:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4966000000000003e-05 2024-04-22T18:40:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615769+0000", 2024-04-22T18:40:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113986 2024-04-22T18:40:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704882+0000", 2024-04-22T18:40:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089112825000000007 2024-04-22T18:40:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104488+0000", 2024-04-22T18:40:54.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39960572999999999 2024-04-22T18:40:54.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104517+0000", 2024-04-22T18:40:54.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9006e-05 2024-04-22T18:40:54.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104539+0000", 2024-04-22T18:40:54.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2073e-05 2024-04-22T18:40:54.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.036 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.037 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3891 4.1 4:83c955bf:::smithi08937929-28:head [setxattr _header (56) in=7b,truncate 3630345] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:40:54.037 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:40:54.037 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.322068818, 2024-04-22T18:40:54.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59018588699999996, 2024-04-22T18:40:54.037 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.037 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:54.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.037 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3891 2024-04-22T18:40:54.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.037 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:40:54.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:40:54.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614523+0000", 2024-04-22T18:40:54.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:54.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614528+0000", 2024-04-22T18:40:54.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0379999999999999e-06 2024-04-22T18:40:54.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614540+0000", 2024-04-22T18:40:54.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018000000000001e-05 2024-04-22T18:40:54.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614546+0000", 2024-04-22T18:40:54.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4990000000000002e-06 2024-04-22T18:40:54.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704438+0000", 2024-04-22T18:40:54.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089892044000000004 2024-04-22T18:40:54.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704504+0000", 2024-04-22T18:40:54.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6365000000000006e-05 2024-04-22T18:40:54.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:54.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704602+0000", 2024-04-22T18:40:54.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8048999999999996e-05 2024-04-22T18:40:54.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.806370+0000", 2024-04-22T18:40:54.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10176768 2024-04-22T18:40:54.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204629+0000", 2024-04-22T18:40:54.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39825917999999999 2024-04-22T18:40:54.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204673+0000", 2024-04-22T18:40:54.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3516999999999997e-05 2024-04-22T18:40:54.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204710+0000", 2024-04-22T18:40:54.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7391000000000002e-05 2024-04-22T18:40:54.042 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.042 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.042 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.042 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3892 4.1 4:83c955bf:::smithi08937929-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:40:54.042 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:40:54.042 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.3220422369999998, 2024-04-22T18:40:54.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59031281800000002, 2024-04-22T18:40:54.042 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.043 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:54.043 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.043 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:54.043 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.043 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3892 2024-04-22T18:40:54.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.043 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:40:54.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:40:54.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614550+0000", 2024-04-22T18:40:54.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614553+0000", 2024-04-22T18:40:54.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0879999999999999e-06 2024-04-22T18:40:54.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614559+0000", 2024-04-22T18:40:54.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9259999999999997e-06 2024-04-22T18:40:54.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614565+0000", 2024-04-22T18:40:54.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7950000000000004e-06 2024-04-22T18:40:54.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704818+0000", 2024-04-22T18:40:54.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090252624000000004 2024-04-22T18:40:54.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:54.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704866+0000", 2024-04-22T18:40:54.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8667e-05 2024-04-22T18:40:54.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204742+0000", 2024-04-22T18:40:54.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987551899999999 2024-04-22T18:40:54.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204799+0000", 2024-04-22T18:40:54.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7299000000000001e-05 2024-04-22T18:40:54.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204863+0000", 2024-04-22T18:40:54.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4455000000000005e-05 2024-04-22T18:40:54.047 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.047 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.047 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.047 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.047 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.047 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:54.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:54.049 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:54.049+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:54.083 INFO:tasks.rados.rados.0.smithi089.stdout:1673: expect (ObjNum 251 snap 0 seq_num 251) 2024-04-22T18:40:54.149 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:54.150+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:54.191 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:54.192 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:54.192 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:40:54.192 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:54.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:54.250 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:54.252+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:54.284 INFO:tasks.rados.rados.0.smithi089.stdout:1664: finishing write tid 2 to smithi08937929-26 2024-04-22T18:40:54.284 INFO:tasks.rados.rados.0.smithi089.stdout:1664: finishing write tid 3 to smithi08937929-26 2024-04-22T18:40:54.284 INFO:tasks.rados.rados.0.smithi089.stdout:1664: finishing write tid 4 to smithi08937929-26 2024-04-22T18:40:54.284 INFO:tasks.rados.rados.0.smithi089.stdout:1664: finishing write tid 5 to smithi08937929-26 2024-04-22T18:40:54.284 INFO:tasks.rados.rados.0.smithi089.stdout:1672: finishing set_chunk to oid 11 2024-04-22T18:40:54.284 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 11 set_chunk returned error code -95 offset: 1109504 length: 1924096 2024-04-22T18:40:54.284 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 0 (ObjNum 196 snap 0 seq_num 196) dirty exists 2024-04-22T18:40:54.284 INFO:tasks.rados.rados.0.smithi089.stdout:1674: expect (ObjNum 339 snap 0 seq_num 339) 2024-04-22T18:40:54.343 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:54.343 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:40:54.343 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:40:54.344 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:40:54.344 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:54.352 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:54.353+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:54.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:54.453 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:54.454+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:54.457 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:40:54.457 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:40:54.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:40:54.457 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:40:54.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.457 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:54.457 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:54.457 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.71513711899999, 2024-04-22T18:40:54.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:40:54.457 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.457 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.457 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.458 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:54.458 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:54.458 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:40:54.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.458 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:54.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:40:54.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:40:54.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:54.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:40:54.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:40:54.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:40:54.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:40:54.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:40:54.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:40:54.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:40:54.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:40:54.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:40:54.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:40:54.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:54.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:40:54.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:40:54.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:40:54.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:40:54.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:40:54.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:40:54.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:40:54.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:40:54.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:40:54.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:40:54.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.462 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.463 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.463 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:40:54.463 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:54.463 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.71509742800001, 2024-04-22T18:40:54.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:40:54.463 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.463 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.463 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.463 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:54.463 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:54.463 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:40:54.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.464 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:54.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:40:54.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:40:54.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:40:54.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:40:54.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:40:54.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:40:54.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:40:54.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:40:54.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:40:54.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:40:54.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:40:54.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:40:54.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:40:54.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:54.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:40:54.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:40:54.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:40:54.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:40:54.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:40:54.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:40:54.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:40:54.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:40:54.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:40:54.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:40:54.468 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.468 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.468 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.468 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:40:54.468 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:54.469 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 132.47090059199999, 2024-04-22T18:40:54.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:40:54.469 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.469 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.469 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.469 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:54.469 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:54.469 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:40:54.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.469 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:54.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:40:54.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:40:54.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:40:54.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:40:54.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:40:54.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:40:54.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:40:54.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:40:54.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:40:54.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:40:54.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:40:54.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:40:54.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:54.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:40:54.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:40:54.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:40:54.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:40:54.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:40:54.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:40:54.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:40:54.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:40:54.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:40:54.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:40:54.474 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.474 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.474 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.474 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:54.474 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:54.474 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.602910764, 2024-04-22T18:40:54.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:40:54.474 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.475 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.475 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.475 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:54.475 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:54.475 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:40:54.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.475 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:54.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:40:54.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:40:54.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:40:54.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:40:54.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:40:54.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:40:54.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:40:54.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:40:54.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:40:54.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:40:54.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:40:54.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:40:54.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:54.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:40:54.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:40:54.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:40:54.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:40:54.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:40:54.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:40:54.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:40:54.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:40:54.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:40:54.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:40:54.479 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.480 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.480 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.480 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:54.480 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:54.480 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.60289510600001, 2024-04-22T18:40:54.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:40:54.480 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.480 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.480 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.480 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:54.480 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:54.480 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:40:54.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.481 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:54.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:54.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.481 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.481 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:40:54.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:40:54.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:54.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:40:54.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:40:54.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:40:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:40:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:40:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:40:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:40:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:40:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:54.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:40:54.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:40:54.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:40:54.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:40:54.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:40:54.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:40:54.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:40:54.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:40:54.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:40:54.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:40:54.485 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.485 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.485 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.486 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:54.486 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:54.486 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.602843944, 2024-04-22T18:40:54.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:40:54.486 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.486 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.486 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.486 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:54.486 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:54.486 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:40:54.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.486 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:54.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:54.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.487 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:40:54.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:40:54.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:40:54.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:40:54.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:40:54.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:40:54.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:40:54.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:40:54.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:40:54.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:40:54.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:40:54.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:54.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:40:54.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:40:54.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:40:54.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:40:54.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:40:54.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:40:54.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:40:54.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:40:54.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:40:54.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:40:54.491 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.491 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.491 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.492 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:54.492 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:54.492 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.60282871999999, 2024-04-22T18:40:54.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:40:54.492 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.492 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.492 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.492 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:54.492 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:54.492 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:40:54.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.492 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:54.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:40:54.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:40:54.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:40:54.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:40:54.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:40:54.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:40:54.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:40:54.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:40:54.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:40:54.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:40:54.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:40:54.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:40:54.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:54.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:40:54.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:40:54.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:40:54.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:40:54.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:40:54.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:40:54.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:40:54.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:40:54.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:40:54.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:40:54.497 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.497 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.497 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.497 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:54.497 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:54.497 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.60276914299999, 2024-04-22T18:40:54.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:40:54.497 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.498 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.498 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.498 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:54.498 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:54.498 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:40:54.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.498 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:54.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:54.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:40:54.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:40:54.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:40:54.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:40:54.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:40:54.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:40:54.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:40:54.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:40:54.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:40:54.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:40:54.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:40:54.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:54.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:40:54.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:40:54.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:40:54.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:40:54.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:40:54.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:40:54.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:40:54.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:40:54.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:40:54.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:40:54.503 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.503 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.503 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.503 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:54.503 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:54.503 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.60268682700001, 2024-04-22T18:40:54.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:40:54.503 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.503 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.503 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.503 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:54.503 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:54.504 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:40:54.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.504 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:54.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:40:54.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:40:54.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:40:54.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:40:54.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:40:54.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:40:54.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:40:54.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:40:54.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:40:54.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:40:54.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:40:54.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:40:54.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:54.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:40:54.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:40:54.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:40:54.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:40:54.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:40:54.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:40:54.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:40:54.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:40:54.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:40:54.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:40:54.508 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.508 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.508 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.509 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:40:54.509 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:54.509 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.60267667100001, 2024-04-22T18:40:54.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:40:54.509 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.509 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.509 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.509 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:40:54.509 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:40:54.509 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:40:54.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.509 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:54.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:40:54.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:40:54.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:40:54.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:40:54.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:40:54.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:40:54.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:40:54.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:40:54.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:40:54.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:40:54.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:40:54.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:40:54.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:40:54.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:40:54.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:40:54.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:40:54.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:40:54.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:40:54.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:40:54.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:40:54.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:40:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:40:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:40:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2835 4.18 4:187adac1:::smithi08937929-36:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.840752279999997, 2024-04-22T18:40:54.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.403020987, 2024-04-22T18:40:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2835 2024-04-22T18:40:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:40:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616367+0000", 2024-04-22T18:40:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616377+0000", 2024-04-22T18:40:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0618999999999999e-05 2024-04-22T18:40:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616410+0000", 2024-04-22T18:40:54.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3021000000000001e-05 2024-04-22T18:40:54.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616437+0000", 2024-04-22T18:40:54.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7158e-05 2024-04-22T18:40:54.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.716588+0000", 2024-04-22T18:40:54.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100150674 2024-04-22T18:40:54.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.716679+0000", 2024-04-22T18:40:54.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1231999999999997e-05 2024-04-22T18:40:54.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:08.019390+0000", 2024-04-22T18:40:54.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30271076299999999 2024-04-22T18:40:54.518 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.518 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.518 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.518 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3163 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.619664233000002, 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1009801299999999, 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3163 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837455+0000", 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837464+0000", 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5319999999999995e-06 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837476+0000", 2024-04-22T18:40:54.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1759e-05 2024-04-22T18:40:54.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837488+0000", 2024-04-22T18:40:54.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1666e-05 2024-04-22T18:40:54.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837541+0000", 2024-04-22T18:40:54.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2939999999999998e-05 2024-04-22T18:40:54.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:54.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837547+0000", 2024-04-22T18:40:54.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6409999999999996e-06 2024-04-22T18:40:54.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937423+0000", 2024-04-22T18:40:54.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.099875545 2024-04-22T18:40:54.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937567+0000", 2024-04-22T18:40:54.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014363099999999999 2024-04-22T18:40:54.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938437+0000", 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087060399999999997 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3164 4.8 4:187adac1:::smithi08937929-36:head [sparse-read 0~2643888] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.619624864000002, 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1013670310000001, 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.523 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:54.524 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.524 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3164 2024-04-22T18:40:54.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.524 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:54.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:54.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837499+0000", 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.994e-06 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837503+0000", 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7249999999999999e-06 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837506+0000", 2024-04-22T18:40:54.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5980000000000001e-06 2024-04-22T18:40:54.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937641+0000", 2024-04-22T18:40:54.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10013467700000001 2024-04-22T18:40:54.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:54.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937650+0000", 2024-04-22T18:40:54.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3489999999999998e-06 2024-04-22T18:40:54.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938468+0000", 2024-04-22T18:40:54.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0008176710000001 2024-04-22T18:40:54.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938582+0000", 2024-04-22T18:40:54.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114287 2024-04-22T18:40:54.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938864+0000", 2024-04-22T18:40:54.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028120899999999998 2024-04-22T18:40:54.527 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.528 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.528 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.528 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3165 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:54.528 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:54.528 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.619606095999998, 2024-04-22T18:40:54.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2022754499999999, 2024-04-22T18:40:54.528 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.528 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:54.528 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.528 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:54.528 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.528 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3165 2024-04-22T18:40:54.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.529 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:54.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:54.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:40:54.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837517+0000", 2024-04-22T18:40:54.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3369999999999998e-06 2024-04-22T18:40:54.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837523+0000", 2024-04-22T18:40:54.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5450000000000003e-06 2024-04-22T18:40:54.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837526+0000", 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.354e-06 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937682+0000", 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100155917 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937688+0000", 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.038953+0000", 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012648110000001 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039034+0000", 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0969000000000004e-05 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039791+0000", 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075672900000000004 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3183 4.8 4:1bc4668a:::smithi08937929-21:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.421202897999997, 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0040478260000001, 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3183 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:40:54.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:40:54.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035913+0000", 2024-04-22T18:40:54.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999943 2024-04-22T18:40:54.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035935+0000", 2024-04-22T18:40:54.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2232999999999999e-05 2024-04-22T18:40:54.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035969+0000", 2024-04-22T18:40:54.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3969e-05 2024-04-22T18:40:54.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035987+0000", 2024-04-22T18:40:54.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8397e-05 2024-04-22T18:40:54.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:54.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2916000000000003e-05 2024-04-22T18:40:54.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:54.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:40:54.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0103e-05 2024-04-22T18:40:54.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039841+0000", 2024-04-22T18:40:54.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0037508959999999 2024-04-22T18:40:54.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039966+0000", 2024-04-22T18:40:54.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012514500000000001 2024-04-22T18:40:54.537 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.537 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.537 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.537 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3184 4.8 4:12d9f826:::smithi08937929-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:54.537 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:40:54.537 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.421114938999999, 2024-04-22T18:40:54.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.00402868, 2024-04-22T18:40:54.537 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.537 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:54.537 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.537 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:54.537 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.538 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3184 2024-04-22T18:40:54.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.538 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:40:54.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:40:54.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036005+0000", 2024-04-22T18:40:54.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:54.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036026+0000", 2024-04-22T18:40:54.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1092999999999999e-05 2024-04-22T18:40:54.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036030+0000", 2024-04-22T18:40:54.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.467e-06 2024-04-22T18:40:54.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036033+0000", 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9780000000000001e-06 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036120+0000", 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7495000000000002e-05 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036125+0000", 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8720000000000001e-06 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039982+0000", 2024-04-22T18:40:54.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0038571919999999 2024-04-22T18:40:54.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040035+0000", 2024-04-22T18:40:54.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2992999999999999e-05 2024-04-22T18:40:54.541 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.541 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.541 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.541 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3185 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:54.541 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:54.541 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.421061027, 2024-04-22T18:40:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0041404039999999, 2024-04-22T18:40:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3185 2024-04-22T18:40:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036062+0000", 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9099999999999999e-06 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036066+0000", 2024-04-22T18:40:54.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1699999999999999e-06 2024-04-22T18:40:54.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036077+0000", 2024-04-22T18:40:54.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0637e-05 2024-04-22T18:40:54.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036147+0000", 2024-04-22T18:40:54.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0025999999999997e-05 2024-04-22T18:40:54.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036151+0000", 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040068+0000", 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.003916604 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040201+0000", 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013275400000000001 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.546 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3186 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:54.546 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:54.546 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.421041471000002, 2024-04-22T18:40:54.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2044167510000001, 2024-04-22T18:40:54.546 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.546 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:40:54.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:54.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.546 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3186 2024-04-22T18:40:54.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.546 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036079+0000", 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036081+0000", 2024-04-22T18:40:54.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6449999999999999e-06 2024-04-22T18:40:54.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036084+0000", 2024-04-22T18:40:54.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9399999999999998e-06 2024-04-22T18:40:54.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:40:54.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.879e-06 2024-04-22T18:40:54.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036168+0000", 2024-04-22T18:40:54.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7657999999999994e-05 2024-04-22T18:40:54.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:40:54.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036171+0000", 2024-04-22T18:40:54.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7639999999999999e-06 2024-04-22T18:40:54.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240345+0000", 2024-04-22T18:40:54.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2041734260000001 2024-04-22T18:40:54.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240497+0000", 2024-04-22T18:40:54.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015182800000000001 2024-04-22T18:40:54.550 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.550 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.550 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.550 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3723 4.18 4:1bc4668a:::smithi08937929-21:head [setxattr _header (56) in=7b,truncate 3277398] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:40:54.550 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:45.060393+0000", 2024-04-22T18:40:54.550 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.3967286419999994, 2024-04-22T18:40:54.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40389026300000003, 2024-04-22T18:40:54.550 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.550 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.550 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.550 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:54.550 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3723 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060393+0000", 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060393+0000", 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060392+0000", 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060396+0000", 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7100000000000001e-06 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060399+0000", 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0400000000000001e-06 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060402+0000", 2024-04-22T18:40:54.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6350000000000002e-06 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165420+0000", 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10501865000000001 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165496+0000", 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5768999999999999e-05 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165616+0000", 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120024 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:40:54.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165906+0000", 2024-04-22T18:40:54.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028992200000000002 2024-04-22T18:40:54.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.464190+0000", 2024-04-22T18:40:54.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29828380599999998 2024-04-22T18:40:54.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:40:54.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.464229+0000", 2024-04-22T18:40:54.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9495e-05 2024-04-22T18:40:54.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.464283+0000", 2024-04-22T18:40:54.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3701e-05 2024-04-22T18:40:54.555 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.555 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.555 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.555 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3724 4.18 4:1bc4668a:::smithi08937929-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:40:54.555 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:40:54.555 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.3967170949999996, 2024-04-22T18:40:54.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50413114699999995, 2024-04-22T18:40:54.555 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:40:54.555 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:40:54.555 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3724 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060405+0000", 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.395e-06 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060408+0000", 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060411+0000", 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6060000000000001e-06 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165799+0000", 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105388369 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165843+0000", 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3891999999999999e-05 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:40:54.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564408+0000", 2024-04-22T18:40:54.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39856466000000002 2024-04-22T18:40:54.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:40:54.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564461+0000", 2024-04-22T18:40:54.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3529999999999997e-05 2024-04-22T18:40:54.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:40:54.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:40:54.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:40:54.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564535+0000", 2024-04-22T18:40:54.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4147999999999995e-05 2024-04-22T18:40:54.559 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.559 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.559 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.559 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:40:54.560 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:40:54.560 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:40:54.562 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:54.562+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:54.562 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:54.563 INFO:tasks.rados.rados.0.smithi089.stdout:1664: finishing write tid 6 to smithi08937929-26 2024-04-22T18:40:54.564 INFO:tasks.rados.rados.0.smithi089.stdout:1664: oid 26 updating version 0 to 129 2024-04-22T18:40:54.564 INFO:tasks.rados.rados.0.smithi089.stdout:1664: oid 26 updating version 129 to 130 2024-04-22T18:40:54.564 INFO:tasks.rados.rados.0.smithi089.stdout:1664: oid 26 version 130 is already newer than 127 2024-04-22T18:40:54.564 INFO:tasks.rados.rados.0.smithi089.stdout:1664: oid 26 version 130 is already newer than 126 2024-04-22T18:40:54.564 INFO:tasks.rados.rados.0.smithi089.stdout:1664: oid 26 version 130 is already newer than 128 2024-04-22T18:40:54.564 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 130 (ObjNum 342 snap 0 seq_num 342) dirty exists 2024-04-22T18:40:54.564 INFO:tasks.rados.rados.0.smithi089.stdout:1664: left oid 26 (ObjNum 342 snap 0 seq_num 342) 2024-04-22T18:40:54.564 INFO:tasks.rados.rados.0.smithi089.stdout:1664: done (9 left) 2024-04-22T18:40:54.564 INFO:tasks.rados.rados.0.smithi089.stdout:1669: done (8 left) 2024-04-22T18:40:54.564 INFO:tasks.rados.rados.0.smithi089.stdout:1670: done (7 left) 2024-04-22T18:40:54.564 INFO:tasks.rados.rados.0.smithi089.stdout:1671: done (6 left) 2024-04-22T18:40:54.564 INFO:tasks.rados.rados.0.smithi089.stdout:1672: done (5 left) 2024-04-22T18:40:54.564 INFO:tasks.rados.rados.0.smithi089.stdout:1673: done (4 left) 2024-04-22T18:40:54.564 INFO:tasks.rados.rados.0.smithi089.stdout:1674: done (3 left) 2024-04-22T18:40:54.564 INFO:tasks.rados.rados.0.smithi089.stdout:1675: done (2 left) 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:1678: read oid 25 snap -1 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:1678: expect (ObjNum 326 snap 0 seq_num 326) 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:1679: read oid 38 snap -1 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:1679: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:1680: write (excl) oid 45 current snap is 0 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:1677: got (2) No such file or directory 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:1677: got expected ENOENT (src dne) 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 19 (ObjNum 129 snap 0 seq_num 2415856336) clean dne 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:1676: finishing write tid 1 to smithi08937929-46 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:1676: finishing write tid 2 to smithi08937929-46 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:1680: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:1680: writing smithi08937929-45 from 581608 to 1303149 tid 1 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:1680: writing smithi08937929-45 from 1863688 to 2349461 tid 2 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:1680: writing smithi08937929-45 from 3113980 to 3718023 tid 3 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:1677: done (4 left) 2024-04-22T18:40:54.565 INFO:tasks.rados.rados.0.smithi089.stdout:1681: read oid 20 snap -1 2024-04-22T18:40:54.566 INFO:tasks.rados.rados.0.smithi089.stdout:1681: expect deleted 2024-04-22T18:40:54.566 INFO:tasks.rados.rados.0.smithi089.stdout:1682: write (excl) oid 33 current snap is 0 2024-04-22T18:40:54.566 INFO:tasks.rados.rados.0.smithi089.stdout:1676: finishing write tid 3 to smithi08937929-46 2024-04-22T18:40:54.566 INFO:tasks.rados.rados.0.smithi089.stdout:1676: finishing write tid 4 to smithi08937929-46 2024-04-22T18:40:54.566 INFO:tasks.rados.rados.0.smithi089.stdout:1676: finishing write tid 5 to smithi08937929-46 2024-04-22T18:40:54.566 INFO:tasks.rados.rados.0.smithi089.stdout:1676: oid 46 updating version 0 to 36 2024-04-22T18:40:54.566 INFO:tasks.rados.rados.0.smithi089.stdout:1676: oid 46 updating version 36 to 38 2024-04-22T18:40:54.566 INFO:tasks.rados.rados.0.smithi089.stdout:1676: oid 46 version 38 is already newer than 37 2024-04-22T18:40:54.566 INFO:tasks.rados.rados.0.smithi089.stdout:1676: oid 46 updating version 38 to 39 2024-04-22T18:40:54.566 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 39 (ObjNum 343 snap 0 seq_num 343) dirty exists 2024-04-22T18:40:54.566 INFO:tasks.rados.rados.0.smithi089.stdout:1676: left oid 46 (ObjNum 343 snap 0 seq_num 343) 2024-04-22T18:40:54.566 INFO:tasks.rados.rados.0.smithi089.stdout:1679: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:40:54.649 INFO:tasks.rados.rados.0.smithi089.stdout:1678: expect (ObjNum 326 snap 0 seq_num 326) 2024-04-22T18:40:54.661 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:54.662+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:54.670 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:54.670 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:54.670 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:54.670 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:54.679 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:54.762 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:54.764+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:54.782 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:54.782 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:54.782 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:54.782 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:54.782 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:54.790 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:54.863 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:54.865+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:54.875 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:54.876+0000 7f3e7be88640 -1 osd.0 120 handle_osd_map injecting map crc failure 2024-04-22T18:40:54.878 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.2 to 1 (10000) 2024-04-22T18:40:54.884 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:40:54.893 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-22T18:40:54.893 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:40:54.931 INFO:tasks.rados.rados.0.smithi089.stdout:1680: finishing write tid 1 to smithi08937929-45 2024-04-22T18:40:54.931 INFO:tasks.rados.rados.0.smithi089.stdout:1680: finishing write tid 2 to smithi08937929-45 2024-04-22T18:40:54.931 INFO:tasks.rados.rados.0.smithi089.stdout:1680: finishing write tid 3 to smithi08937929-45 2024-04-22T18:40:54.931 INFO:tasks.rados.rados.0.smithi089.stdout:1680: finishing write tid 4 to smithi08937929-45 2024-04-22T18:40:54.931 INFO:tasks.rados.rados.0.smithi089.stdout:1682: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-04-22T18:40:54.939 INFO:tasks.rados.rados.0.smithi089.stdout:1682: writing smithi08937929-33 from 598415 to 1395205 tid 1 2024-04-22T18:40:54.941 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:54.941 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:54.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:54.941 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:54.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.941 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:54.941 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:54.941 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 120.485178844, 2024-04-22T18:40:54.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:40:54.941 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:54.941 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.942 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:54.942 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:54.942 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.942 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:40:54.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.942 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:54.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:54.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:54.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:54.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:54.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:54.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:40:54.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:54.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:54.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:40:54.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:40:54.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:54.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:40:54.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:40:54.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:40:54.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:40:54.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:54.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:40:54.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:40:54.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:54.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:40:54.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:40:54.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:54.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:40:54.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:40:54.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:54.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:40:54.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:40:54.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:40:54.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:40:54.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:54.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:40:54.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:40:54.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:54.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:40:54.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:40:54.946 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.947 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:54.947 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.947 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:54.947 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:54.947 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 111.792121596, 2024-04-22T18:40:54.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:40:54.947 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:54.947 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.947 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:54.947 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:54.947 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.947 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:40:54.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:54.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:40:54.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:40:54.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:54.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:40:54.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:40:54.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:40:54.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:40:54.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:54.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:40:54.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:40:54.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:54.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:40:54.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:40:54.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:54.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:40:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:40:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:40:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:40:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:40:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:40:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:40:54.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:40:54.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:54.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:40:54.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:40:54.952 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.952 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:54.952 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.952 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:54.952 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:54.952 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 111.79210311999999, 2024-04-22T18:40:54.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:40:54.953 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:54.953 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:54.953 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:54.953 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:54.953 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.953 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:40:54.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.953 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:54.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:54.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:54.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:54.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:54.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:54.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:54.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:54.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:40:54.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:40:54.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:54.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:40:54.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:40:54.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:40:54.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:40:54.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:54.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:40:54.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:40:54.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:54.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:40:54.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:40:54.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:54.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:40:54.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:40:54.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:54.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:40:54.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:40:54.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:54.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:40:54.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:40:54.957 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.957 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:54.957 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.957 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:54.957 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:54.957 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 84.029054807999998, 2024-04-22T18:40:54.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:40:54.957 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:54.958 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.958 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:54.958 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:54.958 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.958 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:40:54.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.958 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:54.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:54.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:54.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:54.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:54.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:54.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:40:54.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:54.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:54.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:40:54.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:40:54.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:54.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:40:54.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:40:54.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:40:54.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:40:54.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:54.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:40:54.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:40:54.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:54.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:40:54.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:40:54.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:54.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:40:54.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:40:54.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:40:54.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:40:54.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:40:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:40:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:40:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:40:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:40:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:40:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:54.963 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.963 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:54.963 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:54.963 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.499895481999999, 2024-04-22T18:40:54.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:40:54.963 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:54.963 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:54.963 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:54.963 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:54.963 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.963 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:40:54.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.963 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:54.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:54.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:54.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:54.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:54.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:54.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:54.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:54.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:40:54.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:40:54.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:54.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:40:54.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:40:54.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:40:54.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:40:54.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:54.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:40:54.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:40:54.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:54.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:40:54.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:40:54.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:54.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:40:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:40:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:40:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:40:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:40:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:40:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.968 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:54.968 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:54.968 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.322188087000001, 2024-04-22T18:40:54.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:40:54.968 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:54.968 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.968 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:54.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:54.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:54.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:54.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:40:54.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:40:54.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:40:54.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:40:54.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:54.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:40:54.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:40:54.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:54.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:40:54.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:40:54.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:54.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:40:54.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:40:54.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:54.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:40:54.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:40:54.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:40:54.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:40:54.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:54.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:40:54.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:40:54.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:54.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:40:54.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:40:54.972 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.973 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:54.973 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.973 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:54.973 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:54.973 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.322119814999994, 2024-04-22T18:40:54.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:40:54.973 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:54.973 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.973 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:54.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:54.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:54.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:54.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:54.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:54.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:40:54.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:54.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:40:54.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:40:54.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:54.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:40:54.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:40:54.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:40:54.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:40:54.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:54.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:40:54.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:40:54.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:54.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:40:54.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:40:54.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:54.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:40:54.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:40:54.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:54.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:40:54.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:40:54.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:40:54.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:40:54.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:54.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:40:54.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:40:54.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:54.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:40:54.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:54.978 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.978 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:54.978 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.978 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:54.978 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:54.978 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.321951974000001, 2024-04-22T18:40:54.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:40:54.978 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:54.978 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.979 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:54.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:54.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:54.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:54.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:54.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:54.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:54.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:54.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:40:54.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:40:54.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:54.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:40:54.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:40:54.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:40:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:40:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:40:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:40:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:40:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:40:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:54.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:40:54.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:40:54.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:54.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:40:54.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:40:54.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:40:54.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:40:54.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:54.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:40:54.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:40:54.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:54.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:40:54.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:40:54.983 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.983 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:54.983 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.983 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:54.984 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:54.984 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.356626499000001, 2024-04-22T18:40:54.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:40:54.984 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:54.984 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.984 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:54.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:54.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:54.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:54.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:40:54.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:40:54.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:40:54.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:40:54.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:54.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:40:54.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:40:54.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:40:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:40:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:40:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:40:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:40:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:40:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:40:54.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:40:54.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:54.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:40:54.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:40:54.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:54.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:40:54.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:40:54.988 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.988 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:54.989 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.989 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:54.989 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:54.989 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.256474311999995, 2024-04-22T18:40:54.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:40:54.989 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:54.989 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:54.989 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:54.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:54.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:54.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:54.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:54.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:54.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:40:54.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:54.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:54.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:40:54.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:40:54.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:54.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:40:54.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:40:54.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:40:54.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:40:54.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:54.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:40:54.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:40:54.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:54.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:40:54.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:40:54.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:54.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:40:54.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:40:54.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:40:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:40:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:40:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:40:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:40:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:40:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:54.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:40:54.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:40:54.994 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.994 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:54.994 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.994 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:40:54.994 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:54.994 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 64.547609792000003, 2024-04-22T18:40:54.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:40:54.994 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:54.995 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:54.995 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:54.995 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:54.995 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:54.995 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:40:54.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.995 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:54.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:54.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:54.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:54.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:54.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:54.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:54.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:40:54.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:54.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:54.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:40:54.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:40:54.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:54.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:40:54.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:40:54.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:54.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:40:54.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:40:54.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:54.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:40:54.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:40:54.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:54.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:40:54.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:40:54.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:54.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:40:54.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:40:54.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:54.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:40:54.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:40:54.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:54.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:40:54.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:40:54.999 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.999 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:54.999 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:54.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:54.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:54.999 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:54.999 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:54.999 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 47.651571420000003, 2024-04-22T18:40:54.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:40:54.999 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:54.999 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:54.999 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.000 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.000 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.000 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:40:55.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.000 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:55.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:55.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:55.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:40:55.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:40:55.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:40:55.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:40:55.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:40:55.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:40:55.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:40:55.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:40:55.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:40:55.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:40:55.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:40:55.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:40:55.003 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.003 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.003 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.003 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3330 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-04-22T18:40:55.003 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:40:55.004 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.916493134, 2024-04-22T18:40:55.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43151673600000001, 2024-04-22T18:40:55.004 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.004 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:55.004 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.004 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.004 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.004 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3330 2024-04-22T18:40:55.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.004 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:40:55.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:40:55.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024041+0000", 2024-04-22T18:40:55.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:55.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024045+0000", 2024-04-22T18:40:55.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3950000000000001e-06 2024-04-22T18:40:55.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024051+0000", 2024-04-22T18:40:55.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0260000000000004e-06 2024-04-22T18:40:55.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024056+0000", 2024-04-22T18:40:55.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4489999999999998e-06 2024-04-22T18:40:55.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028383+0000", 2024-04-22T18:40:55.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004326557 2024-04-22T18:40:55.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028471+0000", 2024-04-22T18:40:55.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8387999999999995e-05 2024-04-22T18:40:55.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.455559+0000", 2024-04-22T18:40:55.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.42708770699999998 2024-04-22T18:40:55.007 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.007 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.007 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.008 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3377 4.6 4:6700eab5:::smithi08937929-32:head [read 1812992~307712] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-04-22T18:40:55.008 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:40:55.008 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.504167777999999, 2024-04-22T18:40:55.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.52001745799999999, 2024-04-22T18:40:55.008 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.008 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:55.008 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.008 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.008 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.008 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3377 2024-04-22T18:40:55.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.008 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:40:55.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:40:55.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436364+0000", 2024-04-22T18:40:55.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:55.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436376+0000", 2024-04-22T18:40:55.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2121e-05 2024-04-22T18:40:55.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436394+0000", 2024-04-22T18:40:55.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8094e-05 2024-04-22T18:40:55.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436405+0000", 2024-04-22T18:40:55.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0189e-05 2024-04-22T18:40:55.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436471+0000", 2024-04-22T18:40:55.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6817999999999996e-05 2024-04-22T18:40:55.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:55.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436481+0000", 2024-04-22T18:40:55.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2720000000000003e-06 2024-04-22T18:40:55.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956084+0000", 2024-04-22T18:40:55.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51960320800000004 2024-04-22T18:40:55.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956232+0000", 2024-04-22T18:40:55.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014778600000000001 2024-04-22T18:40:55.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956385+0000", 2024-04-22T18:40:55.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153394 2024-04-22T18:40:55.012 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.012 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.012 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.013 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:131 3.1 3:a0a19997:::ee8ca23a7050e771c77dff761e6ea5a11ed57796f4e8a17cbb5804a7fdc937b9:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:55.013 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:40:55.013 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.611298181999999, 2024-04-22T18:40:55.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51449564999999997, 2024-04-22T18:40:55.013 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.013 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:55.013 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:40:55.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:40:55.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329235+0000", 2024-04-22T18:40:55.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:55.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329244+0000", 2024-04-22T18:40:55.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6819999999999992e-06 2024-04-22T18:40:55.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329259+0000", 2024-04-22T18:40:55.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698000000000001e-05 2024-04-22T18:40:55.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329265+0000", 2024-04-22T18:40:55.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1510000000000001e-06 2024-04-22T18:40:55.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329311+0000", 2024-04-22T18:40:55.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6096e-05 2024-04-22T18:40:55.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:55.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.332269+0000", 2024-04-22T18:40:55.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029581910000000002 2024-04-22T18:40:55.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630603+0000", 2024-04-22T18:40:55.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29833418 2024-04-22T18:40:55.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630685+0000", 2024-04-22T18:40:55.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1217999999999994e-05 2024-04-22T18:40:55.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:55.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630813+0000", 2024-04-22T18:40:55.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128706 2024-04-22T18:40:55.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.631447+0000", 2024-04-22T18:40:55.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063427000000000004 2024-04-22T18:40:55.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:55.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843586+0000", 2024-04-22T18:40:55.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21213873 2024-04-22T18:40:55.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:55.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843623+0000", 2024-04-22T18:40:55.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6848999999999999e-05 2024-04-22T18:40:55.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843733+0000", 2024-04-22T18:40:55.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010982200000000001 2024-04-22T18:40:55.019 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.019 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.019 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.019 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3441 4.b 4:d3068ee3:::smithi08937929-29:head [write 558065~619869] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:55.019 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:40:55.019 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.022822121000001, 2024-04-22T18:40:55.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50272020399999995, 2024-04-22T18:40:55.019 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.019 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:55.019 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.019 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.019 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.019 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3441 2024-04-22T18:40:55.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.020 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:40:55.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:40:55.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917711+0000", 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917838+0000", 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012734899999999999 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917854+0000", 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6130999999999999e-05 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917863+0000", 2024-04-22T18:40:55.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5969999999999992e-06 2024-04-22T18:40:55.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992845+0000", 2024-04-22T18:40:55.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.074982193000000003 2024-04-22T18:40:55.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992919+0000", 2024-04-22T18:40:55.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:40:55.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:55.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.993019+0000", 2024-04-22T18:40:55.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9718999999999999e-05 2024-04-22T18:40:55.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095554+0000", 2024-04-22T18:40:55.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102535193 2024-04-22T18:40:55.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:55.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420183+0000", 2024-04-22T18:40:55.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32462853600000002 2024-04-22T18:40:55.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:55.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420401+0000", 2024-04-22T18:40:55.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021881400000000001 2024-04-22T18:40:55.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420433+0000", 2024-04-22T18:40:55.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079000000000002e-05 2024-04-22T18:40:55.024 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.024 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.024 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.024 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3442 4.b 4:d3068ee3:::smithi08937929-29:head [write 1579835~486765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:55.024 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:40:55.024 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.018135409999999, 2024-04-22T18:40:55.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49822799099999998, 2024-04-22T18:40:55.025 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.025 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:55.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.025 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3442 2024-04-22T18:40:55.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.025 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:40:55.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:40:55.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922399+0000", 2024-04-22T18:40:55.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:55.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922502+0000", 2024-04-22T18:40:55.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102784 2024-04-22T18:40:55.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922513+0000", 2024-04-22T18:40:55.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1051e-05 2024-04-22T18:40:55.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922518+0000", 2024-04-22T18:40:55.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8529999999999997e-06 2024-04-22T18:40:55.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093158+0000", 2024-04-22T18:40:55.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.17063958800000001 2024-04-22T18:40:55.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093230+0000", 2024-04-22T18:40:55.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1803000000000004e-05 2024-04-22T18:40:55.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:55.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093326+0000", 2024-04-22T18:40:55.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6454999999999997e-05 2024-04-22T18:40:55.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095557+0000", 2024-04-22T18:40:55.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022306399999999999 2024-04-22T18:40:55.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:55.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420484+0000", 2024-04-22T18:40:55.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32492717700000001 2024-04-22T18:40:55.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:55.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420606+0000", 2024-04-22T18:40:55.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012225800000000001 2024-04-22T18:40:55.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420628+0000", 2024-04-22T18:40:55.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1596000000000001e-05 2024-04-22T18:40:55.030 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.030 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.030 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.030 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3443 4.b 4:d3068ee3:::smithi08937929-29:head [write 2468371~489694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:55.030 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:40:55.030 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.013533363000001, 2024-04-22T18:40:55.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49372487199999998, 2024-04-22T18:40:55.030 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.030 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:55.031 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.031 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.031 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.031 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3443 2024-04-22T18:40:55.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.031 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:40:55.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:40:55.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927001+0000", 2024-04-22T18:40:55.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:55.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927117+0000", 2024-04-22T18:40:55.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011637 2024-04-22T18:40:55.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927128+0000", 2024-04-22T18:40:55.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0696000000000001e-05 2024-04-22T18:40:55.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927134+0000", 2024-04-22T18:40:55.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8640000000000001e-06 2024-04-22T18:40:55.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094288+0000", 2024-04-22T18:40:55.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16715412900000001 2024-04-22T18:40:55.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094359+0000", 2024-04-22T18:40:55.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1639999999999998e-05 2024-04-22T18:40:55.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:55.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094451+0000", 2024-04-22T18:40:55.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1252e-05 2024-04-22T18:40:55.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095560+0000", 2024-04-22T18:40:55.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011095530000000001 2024-04-22T18:40:55.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:55.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420661+0000", 2024-04-22T18:40:55.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32510105500000003 2024-04-22T18:40:55.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:55.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420710+0000", 2024-04-22T18:40:55.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9018999999999999e-05 2024-04-22T18:40:55.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420727+0000", 2024-04-22T18:40:55.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6722e-05 2024-04-22T18:40:55.035 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.036 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.036 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.036 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3444 4.b 4:d3068ee3:::smithi08937929-29:head [setxattr _header (56) in=7b,truncate 2958065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:55.036 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:40:55.036 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.013397646000001, 2024-04-22T18:40:55.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49366935299999998, 2024-04-22T18:40:55.036 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.036 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:55.036 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.036 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.036 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.036 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3444 2024-04-22T18:40:55.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927137+0000", 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927141+0000", 2024-04-22T18:40:55.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3270000000000001e-06 2024-04-22T18:40:55.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927144+0000", 2024-04-22T18:40:55.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9730000000000002e-06 2024-04-22T18:40:55.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927147+0000", 2024-04-22T18:40:55.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.959e-06 2024-04-22T18:40:55.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095118+0000", 2024-04-22T18:40:55.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16797115500000001 2024-04-22T18:40:55.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095169+0000", 2024-04-22T18:40:55.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1316000000000001e-05 2024-04-22T18:40:55.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:55.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095260+0000", 2024-04-22T18:40:55.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1184e-05 2024-04-22T18:40:55.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095999+0000", 2024-04-22T18:40:55.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073868200000000005 2024-04-22T18:40:55.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:55.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420757+0000", 2024-04-22T18:40:55.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32475783499999999 2024-04-22T18:40:55.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:55.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420781+0000", 2024-04-22T18:40:55.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4022000000000001e-05 2024-04-22T18:40:55.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420807+0000", 2024-04-22T18:40:55.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6366000000000001e-05 2024-04-22T18:40:55.041 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.041 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.041 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.041 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3445 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:55.041 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:55.041 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.013386552, 2024-04-22T18:40:55.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.493780049, 2024-04-22T18:40:55.041 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.042 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:55.042 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.042 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.042 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.042 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3445 2024-04-22T18:40:55.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.042 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:55.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:55.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:55.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927150+0000", 2024-04-22T18:40:55.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.542e-06 2024-04-22T18:40:55.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927153+0000", 2024-04-22T18:40:55.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0520000000000002e-06 2024-04-22T18:40:55.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027218+0000", 2024-04-22T18:40:55.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100064604 2024-04-22T18:40:55.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095501+0000", 2024-04-22T18:40:55.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.068283431000000006 2024-04-22T18:40:55.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:55.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095545+0000", 2024-04-22T18:40:55.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4295999999999997e-05 2024-04-22T18:40:55.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420833+0000", 2024-04-22T18:40:55.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32528718400000001 2024-04-22T18:40:55.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420877+0000", 2024-04-22T18:40:55.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4403999999999997e-05 2024-04-22T18:40:55.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420929+0000", 2024-04-22T18:40:55.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1848999999999998e-05 2024-04-22T18:40:55.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.046 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.046 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.046 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:55.047 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:54.951+0000 7f3cfee88640 -1 osd.6 120 handle_osd_map injecting map crc failure 2024-04-22T18:40:55.047 INFO:tasks.rados.rados.0.smithi089.stdout:1682: writing smithi08937929-33 from 1922170 to 2667954 tid 2 2024-04-22T18:40:55.048 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:55.050 INFO:tasks.rados.rados.0.smithi089.stdout:1682: writing smithi08937929-33 from 3141403 to 3556718 tid 3 2024-04-22T18:40:55.050 INFO:tasks.rados.rados.0.smithi089.stdout:1676: done (5 left) 2024-04-22T18:40:55.050 INFO:tasks.rados.rados.0.smithi089.stdout:1678: done (4 left) 2024-04-22T18:40:55.050 INFO:tasks.rados.rados.0.smithi089.stdout:1679: done (3 left) 2024-04-22T18:40:55.050 INFO:tasks.rados.rados.0.smithi089.stdout:1681: done (2 left) 2024-04-22T18:40:55.050 INFO:tasks.rados.rados.0.smithi089.stdout:1683: read oid 6 snap -1 2024-04-22T18:40:55.050 INFO:tasks.rados.rados.0.smithi089.stdout:1683: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:40:55.050 INFO:tasks.rados.rados.0.smithi089.stdout:1684: tier_promote oid 15 2024-04-22T18:40:55.050 INFO:tasks.rados.rados.0.smithi089.stdout:1685: read oid 5 snap -1 2024-04-22T18:40:55.050 INFO:tasks.rados.rados.0.smithi089.stdout:1685: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:40:55.050 INFO:tasks.rados.rados.0.smithi089.stdout:1686: read oid 34 snap -1 2024-04-22T18:40:55.050 INFO:tasks.rados.rados.0.smithi089.stdout:1686: expect (ObjNum 277 snap 0 seq_num 277) 2024-04-22T18:40:55.051 INFO:tasks.rados.rados.0.smithi089.stdout:1687: read oid 46 snap -1 2024-04-22T18:40:55.051 INFO:tasks.rados.rados.0.smithi089.stdout:1687: expect (ObjNum 343 snap 0 seq_num 343) 2024-04-22T18:40:55.051 INFO:tasks.rados.rados.0.smithi089.stdout:1688: write oid 7 current snap is 0 2024-04-22T18:40:55.051 INFO:tasks.rados.rados.0.smithi089.stdout:1688: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-04-22T18:40:55.051 INFO:tasks.rados.rados.0.smithi089.stdout:1688: writing smithi08937929-7 from 615222 to 1087261 tid 1 2024-04-22T18:40:55.051 INFO:tasks.rados.rados.0.smithi089.stdout:1688: writing smithi08937929-7 from 1580652 to 2070094 tid 2 2024-04-22T18:40:55.051 INFO:tasks.rados.rados.0.smithi089.stdout:1688: writing smithi08937929-7 from 2768826 to 3395413 tid 3 2024-04-22T18:40:55.051 INFO:tasks.rados.rados.0.smithi089.stdout:1689: read oid 16 snap -1 2024-04-22T18:40:55.051 INFO:tasks.rados.rados.0.smithi089.stdout:1689: expect (ObjNum 333 snap 0 seq_num 333) 2024-04-22T18:40:55.051 INFO:tasks.rados.rados.0.smithi089.stdout:1690: set_chunk oid 35 target oid 2024-04-22T18:40:55.051 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:55.048+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:55.148 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:55.149+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:55.193 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:55.193 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:55.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.194 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4036 4.b 4.7c11c49b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:55.194 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:55.142140+0000", 2024-04-22T18:40:55.194 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.052812442000000001, 2024-04-22T18:40:55.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.052829683000000002, 2024-04-22T18:40:55.194 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.194 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:55.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.194 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4036 2024-04-22T18:40:55.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.194 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.142140+0000", 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.142140+0000", 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.142139+0000", 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.142142+0000", 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2569999999999999e-06 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.142148+0000", 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3990000000000003e-06 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.142152+0000", 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8520000000000003e-06 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.196 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.197 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4037 4.b 4.7c11c49b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:55.197 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:55.142175+0000", 2024-04-22T18:40:55.197 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.052777515999999997, 2024-04-22T18:40:55.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.052830153999999997, 2024-04-22T18:40:55.197 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.197 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:55.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.197 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4037 2024-04-22T18:40:55.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.197 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.142175+0000", 2024-04-22T18:40:55.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.142175+0000", 2024-04-22T18:40:55.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.142174+0000", 2024-04-22T18:40:55.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.142179+0000", 2024-04-22T18:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2800000000000003e-06 2024-04-22T18:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.142185+0000", 2024-04-22T18:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4880000000000004e-06 2024-04-22T18:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.142191+0000", 2024-04-22T18:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4779999999999999e-06 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.2.13:5 3.0 3:2b6e3ad6:::a13c5d28375df9492f4c5792e5ba50a86d2d30b6761d3431a19bb7ac7af485f2:head [call cas.chunk_put_ref in=77b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e121)", 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:55.170181+0000", 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.024770973000000002, 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.024858631999999999, 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170181+0000", 2024-04-22T18:40:55.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170181+0000", 2024-04-22T18:40:55.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170179+0000", 2024-04-22T18:40:55.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:55.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170188+0000", 2024-04-22T18:40:55.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5900000000000008e-06 2024-04-22T18:40:55.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170200+0000", 2024-04-22T18:40:55.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1817999999999999e-05 2024-04-22T18:40:55.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170208+0000", 2024-04-22T18:40:55.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7320000000000004e-06 2024-04-22T18:40:55.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170268+0000", 2024-04-22T18:40:55.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9546000000000002e-05 2024-04-22T18:40:55.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170366+0000", 2024-04-22T18:40:55.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8307999999999995e-05 2024-04-22T18:40:55.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:55.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170527+0000", 2024-04-22T18:40:55.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016106700000000001 2024-04-22T18:40:55.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171178+0000", 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065053799999999998 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4012 4.7 4:e117a562:::smithi08937929-33:head [write 598415~796790 in=796790b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.941031+0000", 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.25392092999999999, 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.25404842500000002, 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.204 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4012 2024-04-22T18:40:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.941031+0000", 2024-04-22T18:40:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.941031+0000", 2024-04-22T18:40:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.941028+0000", 2024-04-22T18:40:55.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:55.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.941216+0000", 2024-04-22T18:40:55.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018819999999999999 2024-04-22T18:40:55.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.941241+0000", 2024-04-22T18:40:55.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4859999999999999e-05 2024-04-22T18:40:55.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041330+0000", 2024-04-22T18:40:55.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100088978 2024-04-22T18:40:55.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041390+0000", 2024-04-22T18:40:55.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0661000000000001e-05 2024-04-22T18:40:55.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041460+0000", 2024-04-22T18:40:55.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9227999999999995e-05 2024-04-22T18:40:55.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041556+0000", 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6317000000000006e-05 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.145437+0000", 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10388099000000001 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4013 4.7 4:e117a562:::smithi08937929-33:head [write 1922170~745784 in=745784b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:55.208 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:55.041350+0000", 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.15360195900000001, 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.15377210599999999, 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4013 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041350+0000", 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041350+0000", 2024-04-22T18:40:55.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041347+0000", 2024-04-22T18:40:55.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:55.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041686+0000", 2024-04-22T18:40:55.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00033850599999999997 2024-04-22T18:40:55.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041695+0000", 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4089999999999992e-06 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041701+0000", 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7320000000000003e-06 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.043062+0000", 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001361258 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.043107+0000", 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5231000000000001e-05 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.043198+0000", 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1038999999999994e-05 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.145441+0000", 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102242187 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4014 4.7 4:e117a562:::smithi08937929-33:head [write 3141403~415315 in=415315b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:55.041705+0000", 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.153247525, 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.15345713699999999, 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4014 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041705+0000", 2024-04-22T18:40:55.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041705+0000", 2024-04-22T18:40:55.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041704+0000", 2024-04-22T18:40:55.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041891+0000", 2024-04-22T18:40:55.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018729700000000001 2024-04-22T18:40:55.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041897+0000", 2024-04-22T18:40:55.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2480000000000004e-06 2024-04-22T18:40:55.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041902+0000", 2024-04-22T18:40:55.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8429999999999998e-06 2024-04-22T18:40:55.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.044105+0000", 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00220244 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.044152+0000", 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.044231+0000", 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8901999999999998e-05 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.145443+0000", 2024-04-22T18:40:55.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10121255 2024-04-22T18:40:55.217 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.217 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.217 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.217 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4015 4.7 4:e117a562:::smithi08937929-33:head [setxattr _header (56) in=63b,truncate 3798415] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:55.217 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:55.041905+0000", 2024-04-22T18:40:55.217 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.15304670500000001, 2024-04-22T18:40:55.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.15329688599999999, 2024-04-22T18:40:55.217 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.217 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:40:55.217 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.217 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.217 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.218 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4015 2024-04-22T18:40:55.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.218 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041905+0000", 2024-04-22T18:40:55.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041905+0000", 2024-04-22T18:40:55.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041905+0000", 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041908+0000", 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.897e-06 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041912+0000", 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6770000000000001e-06 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041914+0000", 2024-04-22T18:40:55.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5560000000000002e-06 2024-04-22T18:40:55.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.045055+0000", 2024-04-22T18:40:55.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031407539999999999 2024-04-22T18:40:55.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.045108+0000", 2024-04-22T18:40:55.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3557999999999998e-05 2024-04-22T18:40:55.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:55.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.045179+0000", 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0685999999999994e-05 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.145446+0000", 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10026700299999999 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4016 4.7 4:e117a562:::smithi08937929-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:55.041916+0000", 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.15303567800000001, 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.15332626899999999, 2024-04-22T18:40:55.221 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.222 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for rw locks", 2024-04-22T18:40:55.222 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.222 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.222 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.222 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4016 2024-04-22T18:40:55.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.222 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041916+0000", 2024-04-22T18:40:55.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041916+0000", 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041916+0000", 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041918+0000", 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.53e-06 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041921+0000", 2024-04-22T18:40:55.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1240000000000001e-06 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.041923+0000", 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.632e-06 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.145363+0000", 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103439762 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.145426+0000", 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2416999999999995e-05 2024-04-22T18:40:55.224 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.225 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.225 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.225 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.225 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:55.225 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 8 2024-04-22T18:40:55.225 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:55.226 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:55.232 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:40:55.232 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:40:55.244 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-04-22T18:40:54.984761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":269037525,"num_objects":209,"num_object_clones":0,"num_object_copies":418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":4062,"num_read_kb":3552835,"num_write":2391,"num_write_kb":573018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":195615072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":31,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2485,"ondisk_log_size":2485,"up":78,"acting":78,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":503524,"kb_used_data":332032,"kb_used_omap":7,"kb_used_meta":155512,"kb_avail":471355676,"statfs":{"total":483183820800,"available":482668212224,"internally_reserved":0,"allocated":340000768,"data_stored":345814897,"data_compressed":9518582,"data_compressed_allocated":10149888,"data_compressed_original":20877312,"omap_allocated":7962,"internal_metadata":159244518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13398660,"num_objects":-6,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":198,"num_read_kb":222577,"num_write":106,"num_write_kb":35682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":939979,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000805"},"pg_stats":[{"pgid":"4.19","version":"105'114","reported_seq":470,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.158197+0000","last_change":"2024-04-22T18:40:42.952084+0000","last_active":"2024-04-22T18:40:50.158197+0000","last_peered":"2024-04-22T18:40:50.158197+0000","last_clean":"2024-04-22T18:40:50.158197+0000","last_became_active":"2024-04-22T18:40:42.947705+0000","last_became_peered":"2024-04-22T18:40:42.947705+0000","last_unstale":"2024-04-22T18:40:50.158197+0000","last_undegraded":"2024-04-22T18:40:50.158197+0000","last_fullsized":"2024-04-22T18:40:50.158197+0000","mapping_epoch":110,"log_start":"84'88","ondisk_log_start":"84'88","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":26,"log_dups_size":106,"ondisk_log_size":26,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6680865,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":57,"num_read_kb":64438,"num_write":19,"num_write_kb":5735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4990114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"113'56","reported_seq":265,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.659800+0000","last_change":"2024-04-22T18:40:42.951030+0000","last_active":"2024-04-22T18:40:50.659800+0000","last_peered":"2024-04-22T18:40:50.659800+0000","last_clean":"2024-04-22T18:40:50.659800+0000","last_became_active":"2024-04-22T18:40:42.950080+0000","last_became_peered":"2024-04-22T18:40:42.950080+0000","last_unstale":"2024-04-22T18:40:50.659800+0000","last_undegraded":"2024-04-22T18:40:50.659800+0000","last_fullsized":"2024-04-22T18:40:50.659800+0000","mapping_epoch":110,"log_start":"79'52","ondisk_log_start":"79'52","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":61,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5025326,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":108,"num_read_kb":151814,"num_write":45,"num_write_kb":12231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2525724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"107'52","reported_seq":310,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.158489+0000","last_change":"2024-04-22T18:40:42.753148+0000","last_active":"2024-04-22T18:40:50.158489+0000","last_peered":"2024-04-22T18:40:50.158489+0000","last_clean":"2024-04-22T18:40:50.158489+0000","last_became_active":"2024-04-22T18:40:42.752884+0000","last_became_peered":"2024-04-22T18:40:42.752884+0000","last_unstale":"2024-04-22T18:40:50.158489+0000","last_undegraded":"2024-04-22T18:40:50.158489+0000","last_fullsized":"2024-04-22T18:40:50.158489+0000","mapping_epoch":110,"log_start":"83'49","ondisk_log_start":"83'49","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":5,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092497,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":66769,"num_write":16,"num_write_kb":3993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2254766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"115'90","reported_seq":376,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.844415+0000","last_change":"2024-04-22T18:40:47.049866+0000","last_active":"2024-04-22T18:40:50.844415+0000","last_peered":"2024-04-22T18:40:50.844415+0000","last_clean":"2024-04-22T18:40:50.844415+0000","last_became_active":"2024-04-22T18:40:47.049599+0000","last_became_peered":"2024-04-22T18:40:47.049599+0000","last_unstale":"2024-04-22T18:40:50.844415+0000","last_undegraded":"2024-04-22T18:40:50.844415+0000","last_fullsized":"2024-04-22T18:40:50.844415+0000","mapping_epoch":114,"log_start":"96'71","ondisk_log_start":"96'71","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":81,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.22461577699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5142992,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":57,"num_read_kb":73378,"num_write":21,"num_write_kb":6576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1847758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.15","version":"54'29","reported_seq":46,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.659865+0000","last_change":"2024-04-22T18:40:33.895403+0000","last_active":"2024-04-22T18:40:50.659865+0000","last_peered":"2024-04-22T18:40:50.659865+0000","last_clean":"2024-04-22T18:40:50.659865+0000","last_became_active":"2024-04-22T18:40:13.180144+0000","last_became_peered":"2024-04-22T18:40:13.180144+0000","last_unstale":"2024-04-22T18:40:50.659865+0000","last_undegraded":"2024-04-22T18:40:50.659865+0000","last_fullsized":"2024-04-22T18:40:50.659865+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'29","last_scrub_stamp":"2024-04-22T18:39:56.614637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:56.614637+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2915998,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":45460,"num_write":12,"num_write_kb":3102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"51'19","reported_seq":46,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:49.860218+0000","last_change":"2024-04-22T18:40:11.079224+0000","last_active":"2024-04-22T18:40:49.860218+0000","last_peered":"2024-04-22T18:40:49.860218+0000","last_clean":"2024-04-22T18:40:49.860218+0000","last_became_active":"2024-04-22T18:40:10.978840+0000","last_became_peered":"2024-04-22T18:40:10.978840+0000","last_unstale":"2024-04-22T18:40:49.860218+0000","last_undegraded":"2024-04-22T18:40:49.860218+0000","last_fullsized":"2024-04-22T18:40:49.860218+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1882368,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":22802,"num_write":10,"num_write_kb":3047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1349842,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"115'80","reported_seq":112,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-22T18:40:47.999205+0000","last_change":"2024-04-22T18:40:29.100526+0000","last_active":"2024-04-22T18:40:47.999205+0000","last_peered":"2024-04-22T18:40:47.999205+0000","last_clean":"2024-04-22T18:40:47.999205+0000","last_became_active":"2024-04-22T18:40:28.877828+0000","last_became_peered":"2024-04-22T18:40:28.877828+0000","last_unstale":"2024-04-22T18:40:47.999205+0000","last_undegraded":"2024-04-22T18:40:47.999205+0000","last_fullsized":"2024-04-22T18:40:47.999205+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'75","last_scrub_stamp":"2024-04-22T18:40:22.670296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.670296+0000","objects_scrubbed":2,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:44.049940+0000","scrub_duration":0.20951747300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218623,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":28169,"num_write":5,"num_write_kb":1772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4062110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"115'44","reported_seq":223,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.340660+0000","last_change":"2024-04-22T18:40:03.451290+0000","last_active":"2024-04-22T18:40:51.340660+0000","last_peered":"2024-04-22T18:40:51.340660+0000","last_clean":"2024-04-22T18:40:51.340660+0000","last_became_active":"2024-04-22T18:40:02.844371+0000","last_became_peered":"2024-04-22T18:40:02.844371+0000","last_unstale":"2024-04-22T18:40:51.340660+0000","last_undegraded":"2024-04-22T18:40:51.340660+0000","last_fullsized":"2024-04-22T18:40:51.340660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4008503,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":102,"num_read_kb":93785,"num_write":26,"num_write_kb":7134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":970119,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"115'135","reported_seq":414,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.158210+0000","last_change":"2024-04-22T18:40:26.844287+0000","last_active":"2024-04-22T18:40:50.158210+0000","last_peered":"2024-04-22T18:40:50.158210+0000","last_clean":"2024-04-22T18:40:50.158210+0000","last_became_active":"2024-04-22T18:40:26.742351+0000","last_became_peered":"2024-04-22T18:40:26.742351+0000","last_unstale":"2024-04-22T18:40:50.158210+0000","last_undegraded":"2024-04-22T18:40:50.158210+0000","last_fullsized":"2024-04-22T18:40:50.158210+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'113","last_scrub_stamp":"2024-04-22T18:40:20.291988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.291988+0000","objects_scrubbed":2,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:22.684852+0000","scrub_duration":0.10270805500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6823708,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":8729,"num_write":27,"num_write_kb":7833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3940523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"31'5","reported_seq":150,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:49.860073+0000","last_change":"2024-04-22T18:40:10.212897+0000","last_active":"2024-04-22T18:40:49.860073+0000","last_peered":"2024-04-22T18:40:49.860073+0000","last_clean":"2024-04-22T18:40:49.860073+0000","last_became_active":"2024-04-22T18:39:59.265189+0000","last_became_peered":"2024-04-22T18:39:59.265189+0000","last_unstale":"2024-04-22T18:40:49.860073+0000","last_undegraded":"2024-04-22T18:40:49.860073+0000","last_fullsized":"2024-04-22T18:40:49.860073+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":5,"last_scrub":"31'5","last_scrub_stamp":"2024-04-22T18:40:10.212815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.212815+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:17.620115+0000","scrub_duration":0.0015068670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.153662+0000","last_change":"2024-04-22T18:40:19.137046+0000","last_active":"2024-04-22T18:40:51.153662+0000","last_peered":"2024-04-22T18:40:51.153662+0000","last_clean":"2024-04-22T18:40:51.153662+0000","last_became_active":"2024-04-22T18:38:39.025741+0000","last_became_peered":"2024-04-22T18:38:39.025741+0000","last_unstale":"2024-04-22T18:40:51.153662+0000","last_undegraded":"2024-04-22T18:40:51.153662+0000","last_fullsized":"2024-04-22T18:40:51.153662+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:19.136933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:19.136933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:21.035233+0000","scrub_duration":0.0023924060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"117'161","reported_seq":550,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.355659+0000","last_change":"2024-04-22T18:40:36.329312+0000","last_active":"2024-04-22T18:40:51.355659+0000","last_peered":"2024-04-22T18:40:51.355659+0000","last_clean":"2024-04-22T18:40:51.355659+0000","last_became_active":"2024-04-22T18:40:34.914043+0000","last_became_peered":"2024-04-22T18:40:34.914043+0000","last_unstale":"2024-04-22T18:40:51.355659+0000","last_undegraded":"2024-04-22T18:40:51.355659+0000","last_fullsized":"2024-04-22T18:40:51.355659+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:06.180858+0000","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13395278,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":246,"num_read_kb":286454,"num_write":56,"num_write_kb":16276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":26002234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":110,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.158564+0000","last_change":"2024-04-22T18:38:39.110499+0000","last_active":"2024-04-22T18:40:50.158564+0000","last_peered":"2024-04-22T18:40:50.158564+0000","last_clean":"2024-04-22T18:40:50.158564+0000","last_became_active":"2024-04-22T18:38:39.010173+0000","last_became_peered":"2024-04-22T18:38:39.010173+0000","last_unstale":"2024-04-22T18:40:50.158564+0000","last_undegraded":"2024-04-22T18:40:50.158564+0000","last_fullsized":"2024-04-22T18:40:50.158564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"115'19","reported_seq":179,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.007211+0000","last_change":"2024-04-22T18:39:54.378188+0000","last_active":"2024-04-22T18:40:51.007211+0000","last_peered":"2024-04-22T18:40:51.007211+0000","last_clean":"2024-04-22T18:40:51.007211+0000","last_became_active":"2024-04-22T18:39:46.805569+0000","last_became_peered":"2024-04-22T18:39:46.805569+0000","last_unstale":"2024-04-22T18:40:51.007211+0000","last_undegraded":"2024-04-22T18:40:51.007211+0000","last_fullsized":"2024-04-22T18:40:51.007211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'13","last_scrub_stamp":"2024-04-22T18:39:54.377980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.377980+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:00.921987+0000","scrub_duration":0.30260881099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":85369,"num_write":6,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.659914+0000","last_change":"2024-04-22T18:40:35.007077+0000","last_active":"2024-04-22T18:40:50.659914+0000","last_peered":"2024-04-22T18:40:50.659914+0000","last_clean":"2024-04-22T18:40:50.659914+0000","last_became_active":"2024-04-22T18:40:35.006777+0000","last_became_peered":"2024-04-22T18:40:35.006777+0000","last_unstale":"2024-04-22T18:40:50.659914+0000","last_undegraded":"2024-04-22T18:40:50.659914+0000","last_fullsized":"2024-04-22T18:40:50.659914+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:09.538542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:09.538542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:18.970457+0000","scrub_duration":0.35201753200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"114'85","reported_seq":304,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.237142+0000","last_change":"2024-04-22T18:40:35.964284+0000","last_active":"2024-04-22T18:40:51.237142+0000","last_peered":"2024-04-22T18:40:51.237142+0000","last_clean":"2024-04-22T18:40:51.237142+0000","last_became_active":"2024-04-22T18:40:34.910015+0000","last_became_peered":"2024-04-22T18:40:34.910015+0000","last_unstale":"2024-04-22T18:40:51.237142+0000","last_undegraded":"2024-04-22T18:40:51.237142+0000","last_fullsized":"2024-04-22T18:40:51.237142+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'67","last_scrub_stamp":"2024-04-22T18:39:53.916646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.916646+0000","objects_scrubbed":2,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:06.623573+0000","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554796,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":64717,"num_write":17,"num_write_kb":4399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7812071,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.258398+0000","last_change":"2024-04-22T18:40:35.014535+0000","last_active":"2024-04-22T18:40:50.258398+0000","last_peered":"2024-04-22T18:40:50.258398+0000","last_clean":"2024-04-22T18:40:50.258398+0000","last_became_active":"2024-04-22T18:40:35.014154+0000","last_became_peered":"2024-04-22T18:40:35.014154+0000","last_unstale":"2024-04-22T18:40:50.258398+0000","last_undegraded":"2024-04-22T18:40:50.258398+0000","last_fullsized":"2024-04-22T18:40:50.258398+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"64'38","reported_seq":241,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.256438+0000","last_change":"2024-04-22T18:40:35.556480+0000","last_active":"2024-04-22T18:40:50.256438+0000","last_peered":"2024-04-22T18:40:50.256438+0000","last_clean":"2024-04-22T18:40:50.256438+0000","last_became_active":"2024-04-22T18:40:34.911555+0000","last_became_peered":"2024-04-22T18:40:34.911555+0000","last_unstale":"2024-04-22T18:40:50.256438+0000","last_undegraded":"2024-04-22T18:40:50.256438+0000","last_fullsized":"2024-04-22T18:40:50.256438+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2932821,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":104,"num_read_kb":148000,"num_write":20,"num_write_kb":5595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":8531127,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'293","reported_seq":791,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.153290+0000","last_change":"2024-04-22T18:40:47.165729+0000","last_active":"2024-04-22T18:40:51.153290+0000","last_peered":"2024-04-22T18:40:51.153290+0000","last_clean":"2024-04-22T18:40:51.153290+0000","last_became_active":"2024-04-22T18:40:35.239662+0000","last_became_peered":"2024-04-22T18:40:35.239662+0000","last_unstale":"2024-04-22T18:40:51.153290+0000","last_undegraded":"2024-04-22T18:40:51.153290+0000","last_fullsized":"2024-04-22T18:40:51.153290+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29489419,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":251,"num_read_kb":4944,"num_write":356,"num_write_kb":87998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":28207650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.256494+0000","last_change":"2024-04-22T18:40:33.899901+0000","last_active":"2024-04-22T18:40:50.256494+0000","last_peered":"2024-04-22T18:40:50.256494+0000","last_clean":"2024-04-22T18:40:50.256494+0000","last_became_active":"2024-04-22T18:40:02.945025+0000","last_became_peered":"2024-04-22T18:40:02.945025+0000","last_unstale":"2024-04-22T18:40:50.256494+0000","last_undegraded":"2024-04-22T18:40:50.256494+0000","last_fullsized":"2024-04-22T18:40:50.256494+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"115'32","reported_seq":228,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.258383+0000","last_change":"2024-04-22T18:40:21.426510+0000","last_active":"2024-04-22T18:40:50.258383+0000","last_peered":"2024-04-22T18:40:50.258383+0000","last_clean":"2024-04-22T18:40:50.258383+0000","last_became_active":"2024-04-22T18:39:51.943071+0000","last_became_peered":"2024-04-22T18:39:51.943071+0000","last_unstale":"2024-04-22T18:40:50.258383+0000","last_undegraded":"2024-04-22T18:40:50.258383+0000","last_fullsized":"2024-04-22T18:40:50.258383+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-04-22T18:40:21.426350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.426350+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.678685+0000","scrub_duration":0.30334366400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2605118,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":51,"num_read_kb":55069,"num_write":4,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"117'219","reported_seq":554,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.442512+0000","last_change":"2024-04-22T18:40:20.465005+0000","last_active":"2024-04-22T18:40:51.442512+0000","last_peered":"2024-04-22T18:40:51.442512+0000","last_clean":"2024-04-22T18:40:51.442512+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:40:51.442512+0000","last_undegraded":"2024-04-22T18:40:51.442512+0000","last_fullsized":"2024-04-22T18:40:51.442512+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'176","last_scrub_stamp":"2024-04-22T18:40:20.464799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.464799+0000","objects_scrubbed":31,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:45.914514+0000","scrub_duration":0.30629975799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27244592,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":187,"num_read_kb":2520,"num_write":277,"num_write_kb":62747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":75,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-22T18:40:46.846876+0000","last_change":"2024-04-22T18:40:10.403338+0000","last_active":"2024-04-22T18:40:46.846876+0000","last_peered":"2024-04-22T18:40:46.846876+0000","last_clean":"2024-04-22T18:40:46.846876+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:40:46.846876+0000","last_undegraded":"2024-04-22T18:40:46.846876+0000","last_fullsized":"2024-04-22T18:40:46.846876+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:10.403189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.403189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:34.232014+0000","scrub_duration":0.201433005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"113'56","reported_seq":319,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.344928+0000","last_change":"2024-04-22T18:40:42.750352+0000","last_active":"2024-04-22T18:40:51.344928+0000","last_peered":"2024-04-22T18:40:51.344928+0000","last_clean":"2024-04-22T18:40:51.344928+0000","last_became_active":"2024-04-22T18:40:42.750084+0000","last_became_peered":"2024-04-22T18:40:42.750084+0000","last_unstale":"2024-04-22T18:40:51.344928+0000","last_undegraded":"2024-04-22T18:40:51.344928+0000","last_fullsized":"2024-04-22T18:40:51.344928+0000","mapping_epoch":106,"log_start":"83'49","ondisk_log_start":"83'49","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":49,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3310987,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":67,"num_read_kb":74547,"num_write":20,"num_write_kb":5725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2254765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"115'261","reported_seq":729,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:49.960379+0000","last_change":"2024-04-22T18:40:47.255770+0000","last_active":"2024-04-22T18:40:49.960379+0000","last_peered":"2024-04-22T18:40:49.960379+0000","last_clean":"2024-04-22T18:40:49.960379+0000","last_became_active":"2024-04-22T18:40:35.038135+0000","last_became_peered":"2024-04-22T18:40:35.038135+0000","last_unstale":"2024-04-22T18:40:49.960379+0000","last_undegraded":"2024-04-22T18:40:49.960379+0000","last_fullsized":"2024-04-22T18:40:49.960379+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'209","last_scrub_stamp":"2024-04-22T18:40:18.683442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:18.683442+0000","objects_scrubbed":24,"log_size":261,"log_dups_size":0,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:21.496831+0000","scrub_duration":0.204344365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29514217,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":227,"num_read_kb":5187,"num_write":321,"num_write_kb":79183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":23971897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":119,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.659784+0000","last_change":"2024-04-22T18:40:22.521828+0000","last_active":"2024-04-22T18:40:50.659784+0000","last_peered":"2024-04-22T18:40:50.659784+0000","last_clean":"2024-04-22T18:40:50.659784+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:40:50.659784+0000","last_undegraded":"2024-04-22T18:40:50.659784+0000","last_fullsized":"2024-04-22T18:40:50.659784+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.521638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:43.407910+0000","scrub_duration":0.30190717299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"115'88","reported_seq":366,"reported_epoch":116,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:40:50.257232+0000","last_change":"2024-04-22T18:40:43.158709+0000","last_active":"2024-04-22T18:40:50.257232+0000","last_peered":"2024-04-22T18:40:50.257232+0000","last_clean":"2024-04-22T18:40:32.014508+0000","last_became_active":"2024-04-22T18:40:43.046938+0000","last_became_peered":"2024-04-22T18:40:43.046938+0000","last_unstale":"2024-04-22T18:40:50.257232+0000","last_undegraded":"2024-04-22T18:40:50.257232+0000","last_fullsized":"2024-04-22T18:40:50.257232+0000","mapping_epoch":104,"log_start":"96'71","ondisk_log_start":"96'71","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":2,"log_size":17,"log_dups_size":81,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.22461577699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1815172,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":99604,"num_write":17,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,0],"avail_no_missing":["1","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"115'250","reported_seq":552,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.153304+0000","last_change":"2024-04-22T18:40:21.871081+0000","last_active":"2024-04-22T18:40:51.153304+0000","last_peered":"2024-04-22T18:40:51.153304+0000","last_clean":"2024-04-22T18:40:51.153304+0000","last_became_active":"2024-04-22T18:40:10.784978+0000","last_became_peered":"2024-04-22T18:40:10.784978+0000","last_unstale":"2024-04-22T18:40:51.153304+0000","last_undegraded":"2024-04-22T18:40:51.153304+0000","last_fullsized":"2024-04-22T18:40:51.153304+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'212","last_scrub_stamp":"2024-04-22T18:40:21.870855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.870855+0000","objects_scrubbed":46,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:28.179796+0000","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33918065,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":230,"num_read_kb":6186,"num_write":320,"num_write_kb":68546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":13435025,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":130,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-22T18:40:46.947165+0000","last_change":"2024-04-22T18:40:07.420765+0000","last_active":"2024-04-22T18:40:46.947165+0000","last_peered":"2024-04-22T18:40:46.947165+0000","last_clean":"2024-04-22T18:40:46.947165+0000","last_became_active":"2024-04-22T18:40:02.741862+0000","last_became_peered":"2024-04-22T18:40:02.741862+0000","last_unstale":"2024-04-22T18:40:46.947165+0000","last_undegraded":"2024-04-22T18:40:46.947165+0000","last_fullsized":"2024-04-22T18:40:46.947165+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:07.420618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:07.420618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.642486+0000","scrub_duration":0.30164867299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":284,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.659869+0000","last_change":"2024-04-22T18:40:35.216423+0000","last_active":"2024-04-22T18:40:50.659869+0000","last_peered":"2024-04-22T18:40:50.659869+0000","last_clean":"2024-04-22T18:40:50.659869+0000","last_became_active":"2024-04-22T18:40:34.907666+0000","last_became_peered":"2024-04-22T18:40:34.907666+0000","last_unstale":"2024-04-22T18:40:50.659869+0000","last_undegraded":"2024-04-22T18:40:50.659869+0000","last_fullsized":"2024-04-22T18:40:50.659869+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:39:53.346787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.346787+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:15.525107+0000","scrub_duration":0.20273052499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"104'59","reported_seq":261,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.153456+0000","last_change":"2024-04-22T18:40:29.696372+0000","last_active":"2024-04-22T18:40:51.153456+0000","last_peered":"2024-04-22T18:40:51.153456+0000","last_clean":"2024-04-22T18:40:51.153456+0000","last_became_active":"2024-04-22T18:40:28.989193+0000","last_became_peered":"2024-04-22T18:40:28.989193+0000","last_unstale":"2024-04-22T18:40:51.153456+0000","last_undegraded":"2024-04-22T18:40:51.153456+0000","last_fullsized":"2024-04-22T18:40:51.153456+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'38","last_scrub_stamp":"2024-04-22T18:39:55.746067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:55.746067+0000","objects_scrubbed":1,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:20.212013+0000","scrub_duration":0.60877772299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7613571,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":83848,"num_write":22,"num_write_kb":6499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4764796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"79'52","reported_seq":254,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.659699+0000","last_change":"2024-04-22T18:40:42.849298+0000","last_active":"2024-04-22T18:40:50.659699+0000","last_peered":"2024-04-22T18:40:50.659699+0000","last_clean":"2024-04-22T18:40:50.659699+0000","last_became_active":"2024-04-22T18:40:42.848954+0000","last_became_peered":"2024-04-22T18:40:42.848954+0000","last_unstale":"2024-04-22T18:40:50.659699+0000","last_undegraded":"2024-04-22T18:40:50.659699+0000","last_fullsized":"2024-04-22T18:40:50.659699+0000","mapping_epoch":72,"log_start":"79'52","ondisk_log_start":"79'52","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":61,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1747928,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":89,"num_read_kb":120607,"num_write":39,"num_write_kb":10408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2525724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"116'121","reported_seq":479,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.942978+0000","last_change":"2024-04-22T18:40:43.053292+0000","last_active":"2024-04-22T18:40:50.942978+0000","last_peered":"2024-04-22T18:40:50.942978+0000","last_clean":"2024-04-22T18:40:50.942978+0000","last_became_active":"2024-04-22T18:40:43.052945+0000","last_became_peered":"2024-04-22T18:40:43.052945+0000","last_unstale":"2024-04-22T18:40:50.942978+0000","last_undegraded":"2024-04-22T18:40:50.942978+0000","last_fullsized":"2024-04-22T18:40:50.942978+0000","mapping_epoch":101,"log_start":"84'88","ondisk_log_start":"84'88","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":33,"log_dups_size":106,"ondisk_log_size":33,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4848869,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":61,"num_read_kb":70782,"num_write":27,"num_write_kb":8118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4990114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"115'105","reported_seq":341,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-22T18:40:49.544295+0000","last_change":"2024-04-22T18:40:22.140818+0000","last_active":"2024-04-22T18:40:49.544295+0000","last_peered":"2024-04-22T18:40:49.544295+0000","last_clean":"2024-04-22T18:40:49.544295+0000","last_became_active":"2024-04-22T18:38:48.277922+0000","last_became_peered":"2024-04-22T18:38:48.277922+0000","last_unstale":"2024-04-22T18:40:49.544295+0000","last_undegraded":"2024-04-22T18:40:49.544295+0000","last_fullsized":"2024-04-22T18:40:49.544295+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:39.617182+0000","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13378471,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":287,"num_read_kb":338955,"num_write":106,"num_write_kb":28032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"115'138","reported_seq":464,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.158585+0000","last_change":"2024-04-22T18:40:36.165101+0000","last_active":"2024-04-22T18:40:50.158585+0000","last_peered":"2024-04-22T18:40:50.158585+0000","last_clean":"2024-04-22T18:40:50.158585+0000","last_became_active":"2024-04-22T18:40:35.110751+0000","last_became_peered":"2024-04-22T18:40:35.110751+0000","last_unstale":"2024-04-22T18:40:50.158585+0000","last_undegraded":"2024-04-22T18:40:50.158585+0000","last_fullsized":"2024-04-22T18:40:50.158585+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'109","last_scrub_stamp":"2024-04-22T18:40:17.658363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:17.658363+0000","objects_scrubbed":5,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:37.958968+0000","scrub_duration":0.202176033,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15496186,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":467,"num_read_kb":515535,"num_write":146,"num_write_kb":42951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8880696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"114'118","reported_seq":402,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:49.860220+0000","last_change":"2024-04-22T18:40:29.506777+0000","last_active":"2024-04-22T18:40:49.860220+0000","last_peered":"2024-04-22T18:40:49.860220+0000","last_clean":"2024-04-22T18:40:49.860220+0000","last_became_active":"2024-04-22T18:40:28.975797+0000","last_became_peered":"2024-04-22T18:40:28.975797+0000","last_unstale":"2024-04-22T18:40:49.860220+0000","last_undegraded":"2024-04-22T18:40:49.860220+0000","last_fullsized":"2024-04-22T18:40:49.860220+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12487700,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":415,"num_read_kb":485666,"num_write":123,"num_write_kb":35069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7572471,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"115'60","reported_seq":291,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.261470+0000","last_change":"2024-04-22T18:40:34.755224+0000","last_active":"2024-04-22T18:40:50.261470+0000","last_peered":"2024-04-22T18:40:50.261470+0000","last_clean":"2024-04-22T18:40:50.261470+0000","last_became_active":"2024-04-22T18:40:28.778850+0000","last_became_peered":"2024-04-22T18:40:28.778850+0000","last_unstale":"2024-04-22T18:40:50.261470+0000","last_undegraded":"2024-04-22T18:40:50.261470+0000","last_fullsized":"2024-04-22T18:40:50.261470+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'46","last_scrub_stamp":"2024-04-22T18:40:34.755106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:34.755106+0000","objects_scrubbed":2,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:52.199053+0000","scrub_duration":0.40192567099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7075813,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":194114,"num_write":64,"num_write_kb":19036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":10972843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"113'35","reported_seq":211,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.153453+0000","last_change":"2024-04-22T18:40:29.090636+0000","last_active":"2024-04-22T18:40:51.153453+0000","last_peered":"2024-04-22T18:40:51.153453+0000","last_clean":"2024-04-22T18:40:51.153453+0000","last_became_active":"2024-04-22T18:40:28.778111+0000","last_became_peered":"2024-04-22T18:40:28.778111+0000","last_unstale":"2024-04-22T18:40:51.153453+0000","last_undegraded":"2024-04-22T18:40:51.153453+0000","last_fullsized":"2024-04-22T18:40:51.153453+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'23","last_scrub_stamp":"2024-04-22T18:40:20.587606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.587606+0000","objects_scrubbed":1,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.710668+0000","scrub_duration":0.10225366599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899323,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":153548,"num_write":35,"num_write_kb":11536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2364451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"110'74","reported_seq":272,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.136777+0000","last_change":"2024-04-22T18:38:48.173640+0000","last_active":"2024-04-22T18:40:51.136777+0000","last_peered":"2024-04-22T18:40:51.136777+0000","last_clean":"2024-04-22T18:40:51.136777+0000","last_became_active":"2024-04-22T18:38:48.173369+0000","last_became_peered":"2024-04-22T18:38:48.173369+0000","last_unstale":"2024-04-22T18:40:51.136777+0000","last_undegraded":"2024-04-22T18:40:51.136777+0000","last_fullsized":"2024-04-22T18:40:51.136777+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7327951,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":216,"num_read_kb":201739,"num_write":75,"num_write_kb":19712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":148280864,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":3051,"num_read_kb":3533898,"num_write":958,"num_write_kb":272128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":108612248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":31,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":227815424,"data_stored":227283369,"data_compressed":667075,"data_compressed_allocated":704512,"data_compressed_original":1241088,"omap_allocated":0,"internal_metadata":0},"log_size":1345,"ondisk_log_size":1345,"up":52,"acting":52,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":120166293,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":895,"num_read_kb":18837,"num_write":1274,"num_write_kb":298474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":86473896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218955776,"data_stored":231353426,"data_compressed":12437636,"data_compressed_allocated":13242368,"data_compressed_original":26378240,"omap_allocated":0,"internal_metadata":0},"log_size":1023,"ondisk_log_size":1023,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1586784,"data_compressed":90130,"data_compressed_allocated":122880,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82592,"kb_used_data":49836,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94289248,"statfs":{"total":96636764160,"available":96552189952,"internally_reserved":0,"allocated":51032064,"data_stored":51835378,"data_compressed":1442093,"data_compressed_allocated":1540096,"data_compressed_original":3215360,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":9.8949999999999996,"5min":2.3170000000000002,"15min":1.054},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":139.84299999999999,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":13.949,"5min":6.3959999999999999,"15min":5.1369999999999996},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":186.97900000000001,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.49199999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":4.1959999999999997,"15min":4.6420000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":27.027000000000001,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":14.961,"5min":13.43,"15min":13.175000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":183.16300000000001,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.38200000000000001}]},{"osd":2,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":3.3479999999999999,"15min":3.839},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64100000000000001,"5min":77.320999999999998,"15min":77.320999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47199999999999998,"15min":0.47299999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":3.4079999999999999,"15min":3.903},"min":{"1min":0.32000000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60799999999999998,"5min":77.361000000000004,"15min":77.361000000000004},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.30299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.65000000000000002,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.501}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":5.6710000000000003,"5min":4.7009999999999996,"15min":4.5389999999999997},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":69.097999999999999,"5min":85.509,"15min":85.509},"last":0.46000000000000002},{"interface":"front","average":{"1min":7.9509999999999996,"5min":19.887,"15min":21.876999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":181.97,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.46899999999999997}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.5379999999999998,"15min":2.8929999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67800000000000005,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.29599999999999999},{"interface":"front","average":{"1min":3.3170000000000002,"5min":6.3929999999999998,"15min":6.9050000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":69.063000000000002,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.43099999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":17.294,"5min":14.079000000000001,"15min":13.542999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":139.428,"5min":139.428,"15min":139.428},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":3.5779999999999998,"15min":3.948},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":22.878,"5min":77.405000000000001,"15min":77.405000000000001},"last":0.36199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112980,"kb_used_data":79968,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94258860,"statfs":{"total":96636764160,"available":96521072640,"internally_reserved":0,"allocated":81887232,"data_stored":83763829,"data_compressed":2315210,"data_compressed_allocated":2469888,"data_compressed_original":5435392,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:43 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":1.516,"15min":1.7010000000000001},"min":{"1min":0.24099999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.57199999999999995,"5min":32.097999999999999,"15min":32.097999999999999},"last":0.251},{"interface":"front","average":{"1min":5.9290000000000003,"5min":21.832000000000001,"15min":24.481999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":52.465000000000003,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:40:43 2024","interfaces":[{"interface":"back","average":{"1min":11.403,"5min":3.911,"15min":2.6629999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":199.047,"5min":199.047,"15min":199.047},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.6599999999999999,"5min":8.4049999999999994,"15min":9.5289999999999999},"min":{"1min":0.318,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":22.747,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:40:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.45000000000000001,"15min":0.45600000000000002},"min":{"1min":0.29999999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56599999999999995,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.34899999999999998},{"interface":"front","average":{"1min":5.944,"5min":16.231999999999999,"15min":17.946000000000002},"min":{"1min":0.32100000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":52.515999999999998,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.40300000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:40:43 2024","interfaces":[{"interface":"back","average":{"1min":9.0640000000000001,"5min":19.027999999999999,"15min":20.687999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":103.976,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":4.9039999999999999,"5min":1.339,"15min":0.745},"min":{"1min":0.33100000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":77.097999999999999,"5min":77.097999999999999,"15min":77.097999999999999},"last":0.309}]},{"osd":4,"last update":"Mon Apr 22 18:40:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":6.1600000000000001,"15min":7.1219999999999999},"min":{"1min":0.26400000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53000000000000003,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.372},{"interface":"front","average":{"1min":13.000999999999999,"5min":18.062000000000001,"15min":18.905999999999999},"min":{"1min":0.255,"5min":0.218,"15min":0.218},"max":{"1min":99.771000000000001,"5min":166.733,"15min":166.733},"last":0.42699999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:40:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39600000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.252,"15min":0.252},"max":{"1min":0.56000000000000005,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44700000000000001},"min":{"1min":0.33700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.70199999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.41599999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:40:43 2024","interfaces":[{"interface":"back","average":{"1min":9.9819999999999993,"5min":17.077000000000002,"15min":18.259},"min":{"1min":0.309,"5min":0.308,"15min":0.308},"max":{"1min":72.415000000000006,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46200000000000002,"15min":0.46400000000000002},"min":{"1min":0.312,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61899999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":13.077}]}]},{"osd":5,"up_from":0,"seq":55834574877,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":55834574877,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106596,"kb_used_data":70516,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94265244,"statfs":{"total":96636764160,"available":96527609856,"internally_reserved":0,"allocated":72208384,"data_stored":73645262,"data_compressed":2562537,"data_compressed_allocated":2719744,"data_compressed_original":5156864,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":9.9700000000000006,"15min":11.456},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":11.797000000000001,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.182},{"interface":"front","average":{"1min":0.26700000000000002,"5min":0.316,"15min":0.32400000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.43099999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.214}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":25.978000000000002,"5min":9.2309999999999999,"15min":6.4400000000000004},"min":{"1min":0.20999999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":197.02099999999999,"5min":197.02099999999999,"15min":197.02099999999999},"last":78.805000000000007},{"interface":"front","average":{"1min":1.623,"5min":0.59199999999999997,"15min":0.41999999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":16.388999999999999,"5min":16.388999999999999,"15min":16.388999999999999},"last":0.20999999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":6.0090000000000003,"5min":2.0059999999999998,"15min":1.339},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":87.635999999999996,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.51800000000000002,"15min":0.45600000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":11.859999999999999,"5min":11.859999999999999,"15min":11.859999999999999},"last":0.253}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":3.379,"15min":3.8650000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79100000000000004,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.28999999999999998},{"interface":"front","average":{"1min":6.1719999999999997,"5min":4.5439999999999996,"15min":4.2729999999999997},"min":{"1min":0.34799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":87.724000000000004,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.27600000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":6.1539999999999999,"5min":2.1760000000000002,"15min":1.5129999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":87.784000000000006,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.30299999999999999},{"interface":"front","average":{"1min":1.038,"5min":1.2130000000000001,"15min":1.242},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":11.911,"5min":13.965,"15min":13.965},"last":0.35699999999999998}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.52700000000000002,"15min":0.53700000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72999999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.32100000000000001},{"interface":"front","average":{"1min":9.9130000000000003,"5min":2.4079999999999999,"15min":1.157},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":96.140000000000001,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.38}]},{"osd":7,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":0.69599999999999995,"15min":0.621},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":11.887,"5min":11.887,"15min":11.887},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.55000000000000004,"15min":0.56200000000000006},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75600000000000001,"5min":0.995,"15min":0.995},"last":0.37}]}]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57952,"kb_used_data":29236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94313888,"statfs":{"total":96636764160,"available":96577421312,"internally_reserved":0,"allocated":29937664,"data_stored":30366509,"data_compressed":1083429,"data_compressed_allocated":1159168,"data_compressed_original":2371584,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":7.4240000000000004,"15min":7.8920000000000003},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":59.002000000000002,"5min":149.03,"15min":149.03},"last":0.246},{"interface":"front","average":{"1min":0.308,"5min":1.649,"15min":1.873},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.442,"5min":31.866,"15min":31.866},"last":0.28599999999999998}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":3.5579999999999998,"15min":1.8740000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":97.683999999999997,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.27100000000000002},{"interface":"front","average":{"1min":22.157,"5min":16.338999999999999,"15min":15.369},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":393.42399999999998,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.33500000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":18.393000000000001,"5min":3.9689999999999999,"15min":1.5649999999999999},"min":{"1min":0.252,"5min":0.245,"15min":0.245},"max":{"1min":147.721,"5min":147.721,"15min":147.721},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.36899999999999999,"15min":0.377},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.53200000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.307}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":3.4060000000000001,"5min":15.698,"15min":17.747},"min":{"1min":0.311,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":52.683,"5min":160.584,"15min":160.584},"last":0.5},{"interface":"front","average":{"1min":10.984,"5min":4.1550000000000002,"15min":3.0169999999999999},"min":{"1min":0.32200000000000001,"5min":0.315,"15min":0.315},"max":{"1min":97.887,"5min":97.887,"15min":97.887},"last":0.56999999999999995}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":2.5409999999999999,"15min":2.8809999999999998},"min":{"1min":0.33300000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.74299999999999999,"5min":48.631,"15min":48.631},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":2.8820000000000001,"15min":3.278},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.83499999999999996,"5min":56.18,"15min":56.18},"last":0.53100000000000003}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":10.375,"5min":2.5030000000000001,"15min":1.1910000000000001},"min":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":97.832999999999998,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.33200000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.49399999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.63800000000000001,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"last":0.55800000000000005},{"interface":"front","average":{"1min":10.396000000000001,"5min":2.5470000000000002,"15min":1.2390000000000001},"min":{"1min":0.38800000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":97.861999999999995,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.54000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":143404,"kb_used_data":102476,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94228436,"statfs":{"total":96636764160,"available":96489918464,"internally_reserved":0,"allocated":104935424,"data_stored":106203919,"data_compressed":2115313,"data_compressed_allocated":2260992,"data_compressed_original":4698112,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":3.9870000000000001,"5min":7.0839999999999996,"15min":7.5999999999999996},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":66.480999999999995,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.33400000000000002},{"interface":"front","average":{"1min":4.0430000000000001,"5min":7.0990000000000002,"15min":7.6079999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":66.498000000000005,"5min":88.156000000000006,"15min":88.156000000000006},"last":0.27700000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":6.9429999999999996,"15min":8.0470000000000006},"min":{"1min":0.20000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.626,"5min":183.04400000000001,"15min":183.04400000000001},"last":0.308},{"interface":"front","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.39100000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.33700000000000002,"15min":0.33400000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":11.664,"15min":13.548999999999999},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.878,"5min":231.864,"15min":231.864},"last":0.32100000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":7.29,"5min":11.845000000000001,"15min":12.603999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":109.399,"5min":192.42099999999999,"15min":192.42099999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.47899999999999998}]},{"osd":5,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":11.359999999999999,"15min":13.167},"min":{"1min":0.34399999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.052,"5min":138.089,"15min":138.089},"last":0.435},{"interface":"front","average":{"1min":0.48999999999999999,"5min":16.158999999999999,"15min":18.771000000000001},"min":{"1min":0.35099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69899999999999995,"5min":183.08199999999999,"15min":183.08199999999999},"last":0.44500000000000001}]},{"osd":6,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":4.8520000000000003,"5min":3.1920000000000002,"15min":2.915},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":78.805000000000007,"5min":78.805000000000007,"15min":78.805000000000007},"last":0.41999999999999998},{"interface":"front","average":{"1min":5.9400000000000004,"5min":14.413,"15min":15.824999999999999},"min":{"1min":0.36599999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":98.968000000000004,"5min":183.10900000000001,"15min":183.10900000000001},"last":33.441000000000003}]},{"osd":7,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.38500000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.503},{"interface":"front","average":{"1min":0.52100000000000002,"5min":2.5350000000000001,"15min":2.871},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.0189999999999999,"5min":51.097000000000001,"15min":51.097000000000001},"last":0.40300000000000002}]}]},{"osd":0,"up_from":0,"seq":55834574877,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33095680,"data_stored":34790173,"data_compressed":1758468,"data_compressed_allocated":1871872,"data_compressed_original":3694592,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35766272,"data_stored":37969348,"data_compressed":1987296,"data_compressed_allocated":2125824,"data_compressed_original":4435968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16535552,"data_stored":17706169,"data_compressed":1083429,"data_compressed_allocated":1159168,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40054784,"data_stored":42280193,"data_compressed":2453667,"data_compressed_allocated":2605056,"data_compressed_original":4960256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":34144256,"data_stored":36043524,"data_compressed":1815904,"data_compressed_allocated":1929216,"data_compressed_original":3956736,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":223399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":38842368,"data_stored":40993712,"data_compressed":2086377,"data_compressed_allocated":2220032,"data_compressed_original":4517888,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20287488,"data_stored":21346908,"data_compressed":1252495,"data_compressed_allocated":1331200,"data_compressed_original":2441216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2359296,"data_stored":2349834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":68227072,"data_stored":68082959,"data_compressed":146679,"data_compressed_allocated":155648,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12144640,"data_stored":12084933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30617600,"data_stored":30549118,"data_compressed":108870,"data_compressed_allocated":114688,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":10027008,"data_stored":9980228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33280000,"data_stored":33118894,"data_compressed":53657,"data_compressed_allocated":57344,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":41676800,"data_stored":41655503,"data_compressed":197839,"data_compressed_allocated":208896,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":29483008,"data_stored":29461900,"data_compressed":160030,"data_compressed_allocated":167936,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:40:55.246 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:40:55.269 INFO:tasks.rados.rados.0.smithi089.stdout:1690: set_chunk oid 35 offset: 301568 length: 2463232 target oid 730fd78d9ff31a03ae4406bd419bd22e1875cc3e8b8e073e7d5ed9c14772c194 offset: 0 snap -1 2024-04-22T18:40:55.269 INFO:tasks.rados.rados.0.smithi089.stdout:1691: write (excl) oid 1 current snap is 0 2024-04-22T18:40:55.269 INFO:tasks.rados.rados.0.smithi089.stdout:1688: finishing write tid 1 to smithi08937929-7 2024-04-22T18:40:55.269 INFO:tasks.rados.rados.0.smithi089.stdout:1680: finishing write tid 5 to smithi08937929-45 2024-04-22T18:40:55.269 INFO:tasks.rados.rados.0.smithi089.stdout:1680: oid 45 updating version 0 to 91 2024-04-22T18:40:55.270 INFO:tasks.rados.rados.0.smithi089.stdout:1680: oid 45 version 91 is already newer than 89 2024-04-22T18:40:55.270 INFO:tasks.rados.rados.0.smithi089.stdout:1680: oid 45 version 91 is already newer than 90 2024-04-22T18:40:55.270 INFO:tasks.rados.rados.0.smithi089.stdout:1680: oid 45 updating version 91 to 92 2024-04-22T18:40:55.270 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 92 (ObjNum 344 snap 0 seq_num 344) dirty exists 2024-04-22T18:40:55.270 INFO:tasks.rados.rados.0.smithi089.stdout:1680: left oid 45 (ObjNum 344 snap 0 seq_num 344) 2024-04-22T18:40:55.270 INFO:tasks.rados.rados.0.smithi089.stdout:1691: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-04-22T18:40:55.270 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:55.270+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:55.275 INFO:tasks.rados.rados.0.smithi089.stdout:1691: writing smithi08937929-1 from 632029 to 1179317 tid 1 2024-04-22T18:40:55.281 INFO:tasks.rados.rados.0.smithi089.stdout:1691: writing smithi08937929-1 from 1639134 to 2388587 tid 2 2024-04-22T18:40:55.286 INFO:tasks.rados.rados.0.smithi089.stdout:1691: writing smithi08937929-1 from 2796249 to 3234108 tid 3 2024-04-22T18:40:55.290 INFO:tasks.rados.rados.0.smithi089.stdout:1691: writing smithi08937929-1 from 3653524 to 3832029 tid 4 2024-04-22T18:40:55.290 INFO:tasks.rados.rados.0.smithi089.stdout:1680: done (10 left) 2024-04-22T18:40:55.290 INFO:tasks.rados.rados.0.smithi089.stdout:1692: read oid 27 snap -1 2024-04-22T18:40:55.290 INFO:tasks.rados.rados.0.smithi089.stdout:1692: expect (ObjNum 287 snap 0 seq_num 287) 2024-04-22T18:40:55.290 INFO:tasks.rados.rados.0.smithi089.stdout:1693: read oid 19 snap -1 2024-04-22T18:40:55.290 INFO:tasks.rados.rados.0.smithi089.stdout:1693: expect (ObjNum 331 snap 0 seq_num 331) 2024-04-22T18:40:55.290 INFO:tasks.rados.rados.0.smithi089.stdout:1694: tier_evict oid 45 2024-04-22T18:40:55.290 INFO:tasks.rados.rados.0.smithi089.stdout:1694: tier_evict oid 45 snap -1 2024-04-22T18:40:55.290 INFO:tasks.rados.rados.0.smithi089.stdout:1695: write oid 13 current snap is 0 2024-04-22T18:40:55.290 INFO:tasks.rados.rados.0.smithi089.stdout:1686: expect (ObjNum 277 snap 0 seq_num 277) 2024-04-22T18:40:55.344 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:55.344 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:55.344 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:55.344 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:55.344 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:55.350 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:55.351+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:55.354 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:55.451 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:55.452+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:55.493 INFO:tasks.rados.rados.0.smithi089.stdout:1688: finishing write tid 2 to smithi08937929-7 2024-04-22T18:40:55.493 INFO:tasks.rados.rados.0.smithi089.stdout:1689: expect (ObjNum 333 snap 0 seq_num 333) 2024-04-22T18:40:55.503 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:55.503 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:55.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:55.503 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:55.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:9 4.f 4:f2c48a0e:::smithi08937929-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.385941696, 2024-04-22T18:40:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49290714600000002, 2024-04-22T18:40:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 9 2024-04-22T18:40:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:55.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:55.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116773+0000", 2024-04-22T18:40:55.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:55.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117024+0000", 2024-04-22T18:40:55.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025191299999999999 2024-04-22T18:40:55.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117036+0000", 2024-04-22T18:40:55.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1595e-05 2024-04-22T18:40:55.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117046+0000", 2024-04-22T18:40:55.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0456e-05 2024-04-22T18:40:55.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414324+0000", 2024-04-22T18:40:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297277603 2024-04-22T18:40:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414404+0000", 2024-04-22T18:40:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9738000000000001e-05 2024-04-22T18:40:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414494+0000", 2024-04-22T18:40:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9733000000000003e-05 2024-04-22T18:40:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518194+0000", 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10370061 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609618+0000", 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091423360999999995 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609655+0000", 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7580000000000003e-05 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609681+0000", 2024-04-22T18:40:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5908e-05 2024-04-22T18:40:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:10 4.f 4:f2c48a0e:::smithi08937929-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.38008943600001, 2024-04-22T18:40:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48712681400000002, 2024-04-22T18:40:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:55.509 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.510 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.510 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.510 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 10 2024-04-22T18:40:55.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.510 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:55.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:55.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122625+0000", 2024-04-22T18:40:55.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:55.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122828+0000", 2024-04-22T18:40:55.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020359499999999999 2024-04-22T18:40:55.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122840+0000", 2024-04-22T18:40:55.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1239e-05 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122845+0000", 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2719999999999997e-06 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415105+0000", 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29225972099999997 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415169+0000", 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4170000000000004e-05 2024-04-22T18:40:55.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:55.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415257+0000", 2024-04-22T18:40:55.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8331999999999994e-05 2024-04-22T18:40:55.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518199+0000", 2024-04-22T18:40:55.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10294174 2024-04-22T18:40:55.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:55.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609707+0000", 2024-04-22T18:40:55.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091508408999999999 2024-04-22T18:40:55.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:55.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609729+0000", 2024-04-22T18:40:55.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1917e-05 2024-04-22T18:40:55.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609753+0000", 2024-04-22T18:40:55.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3776000000000001e-05 2024-04-22T18:40:55.514 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.514 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.514 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.515 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:11 4.f 4:f2c48a0e:::smithi08937929-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:55.515 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:55.515 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.37986606299999, 2024-04-22T18:40:55.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49452214500000002, 2024-04-22T18:40:55.515 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.515 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:55.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.515 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 11 2024-04-22T18:40:55.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.515 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122849+0000", 2024-04-22T18:40:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122853+0000", 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9029999999999997e-06 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122856+0000", 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.01e-06 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122859+0000", 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415755+0000", 2024-04-22T18:40:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29289606800000001 2024-04-22T18:40:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415808+0000", 2024-04-22T18:40:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3415999999999998e-05 2024-04-22T18:40:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415888+0000", 2024-04-22T18:40:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9061999999999996e-05 2024-04-22T18:40:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518203+0000", 2024-04-22T18:40:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102315533 2024-04-22T18:40:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617164+0000", 2024-04-22T18:40:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098960548999999995 2024-04-22T18:40:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617241+0000", 2024-04-22T18:40:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7292000000000005e-05 2024-04-22T18:40:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617372+0000", 2024-04-22T18:40:55.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013075399999999999 2024-04-22T18:40:55.520 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.520 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.520 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.520 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:12 4.f 4:f2c48a0e:::smithi08937929-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:55.520 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:55.520 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.37985419899999, 2024-04-22T18:40:55.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49473910999999998, 2024-04-22T18:40:55.520 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.520 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:55.520 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.521 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.521 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.521 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 12 2024-04-22T18:40:55.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.521 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:55.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:55.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:55.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122863+0000", 2024-04-22T18:40:55.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.863e-06 2024-04-22T18:40:55.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122866+0000", 2024-04-22T18:40:55.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:40:55.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122869+0000", 2024-04-22T18:40:55.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5950000000000001e-06 2024-04-22T18:40:55.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416008+0000", 2024-04-22T18:40:55.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29313975199999998 2024-04-22T18:40:55.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:55.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416048+0000", 2024-04-22T18:40:55.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9888000000000002e-05 2024-04-22T18:40:55.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617447+0000", 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20139881400000001 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617537+0000", 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9606999999999998e-05 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617600+0000", 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3884e-05 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.524 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:33 4.f 4:f0ed3d49:::smithi08937929-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.315913998, 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62359692099999997, 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 33 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:55.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:55.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186799+0000", 2024-04-22T18:40:55.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:55.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187103+0000", 2024-04-22T18:40:55.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030354900000000002 2024-04-22T18:40:55.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187117+0000", 2024-04-22T18:40:55.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4578e-05 2024-04-22T18:40:55.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187129+0000", 2024-04-22T18:40:55.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1421e-05 2024-04-22T18:40:55.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516134+0000", 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32900553900000001 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516215+0000", 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0537999999999994e-05 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516302+0000", 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6601000000000007e-05 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518207+0000", 2024-04-22T18:40:55.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019058370000000001 2024-04-22T18:40:55.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:55.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810302+0000", 2024-04-22T18:40:55.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29209509700000003 2024-04-22T18:40:55.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:55.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810348+0000", 2024-04-22T18:40:55.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5992999999999998e-05 2024-04-22T18:40:55.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810398+0000", 2024-04-22T18:40:55.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0046000000000002e-05 2024-04-22T18:40:55.530 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.530 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.530 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.530 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:34 4.f 4:f0ed3d49:::smithi08937929-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:55.530 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:55.530 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.31131198600001, 2024-04-22T18:40:55.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.61954655800000002, 2024-04-22T18:40:55.530 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.530 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:55.530 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.530 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.530 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.531 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34 2024-04-22T18:40:55.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.531 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:55.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:55.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191401+0000", 2024-04-22T18:40:55.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:55.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191503+0000", 2024-04-22T18:40:55.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010190400000000001 2024-04-22T18:40:55.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191517+0000", 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3267e-05 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191523+0000", 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517062+0000", 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32553894300000003 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517141+0000", 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9667999999999997e-05 2024-04-22T18:40:55.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:55.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517267+0000", 2024-04-22T18:40:55.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000125494 2024-04-22T18:40:55.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518213+0000", 2024-04-22T18:40:55.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000946275 2024-04-22T18:40:55.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:55.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810901+0000", 2024-04-22T18:40:55.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29268821699999997 2024-04-22T18:40:55.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:55.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810929+0000", 2024-04-22T18:40:55.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7645999999999999e-05 2024-04-22T18:40:55.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810950+0000", 2024-04-22T18:40:55.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1333999999999999e-05 2024-04-22T18:40:55.535 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.535 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.535 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.31118901799999, 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:55.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:55.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:40:55.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:40:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:40:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:40:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:40:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:40:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:40:55.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:40:55.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:40:55.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:40:55.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:55.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:40:55.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:40:55.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:40:55.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:40:55.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:55.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:40:55.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:40:55.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:55.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:40:55.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:40:55.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:40:55.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:40:55.543 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.543 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.311177943, 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:55.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:55.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:40:55.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:40:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:40:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:40:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:40:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:55.548 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:55.549 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 117.70980844899999, 2024-04-22T18:40:55.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:40:55.549 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.549 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:55.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.549 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:40:55.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.549 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:55.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:55.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.554 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:7 3.0 3:1554f608:::d614c1f2f74bb29087660bd6e888eafe46b1a7ed3cbdede85b36de771c7f0d66:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:55.554 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:55.554 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.330530922999998, 2024-04-22T18:40:55.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48116529899999999, 2024-04-22T18:40:55.554 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.554 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:55.554 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:55.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172184+0000", 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172349+0000", 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000165074 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172355+0000", 2024-04-22T18:40:55.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8900000000000004e-06 2024-04-22T18:40:55.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172359+0000", 2024-04-22T18:40:55.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.569e-06 2024-04-22T18:40:55.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271561+0000", 2024-04-22T18:40:55.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099202569000000004 2024-04-22T18:40:55.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271659+0000", 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8171000000000006e-05 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271775+0000", 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000116137 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.272327+0000", 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00055106999999999997 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653290+0000", 2024-04-22T18:40:55.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.38096334500000001 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653326+0000", 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6168000000000003e-05 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653350+0000", 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3921999999999999e-05 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.558 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3193 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:55.559 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:40:55.559 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.459969653999998, 2024-04-22T18:40:55.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69912102700000001, 2024-04-22T18:40:55.559 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.559 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:55.559 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.559 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.559 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.559 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3193 2024-04-22T18:40:55.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.559 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:40:55.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042744+0000", 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043013+0000", 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026906 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043028+0000", 2024-04-22T18:40:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5359999999999999e-05 2024-04-22T18:40:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043036+0000", 2024-04-22T18:40:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8420000000000005e-06 2024-04-22T18:40:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043081+0000", 2024-04-22T18:40:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5019999999999999e-05 2024-04-22T18:40:55.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043087+0000", 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5420000000000002e-06 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741856+0000", 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69876864500000002 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741867+0000", 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.084e-05 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.562 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.563 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.563 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3194 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:55.563 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:40:55.563 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.452719674000001, 2024-04-22T18:40:55.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69190137100000004, 2024-04-22T18:40:55.563 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.563 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:55.563 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.563 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.563 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.563 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3194 2024-04-22T18:40:55.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.564 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:40:55.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:40:55.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049991+0000", 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999957 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050284+0000", 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029296699999999999 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050304+0000", 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.969e-05 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050314+0000", 2024-04-22T18:40:55.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0268999999999999e-05 2024-04-22T18:40:55.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150531+0000", 2024-04-22T18:40:55.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100216868 2024-04-22T18:40:55.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:55.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150545+0000", 2024-04-22T18:40:55.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4063000000000001e-05 2024-04-22T18:40:55.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741890+0000", 2024-04-22T18:40:55.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59134479100000004 2024-04-22T18:40:55.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741897+0000", 2024-04-22T18:40:55.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1189999999999999e-06 2024-04-22T18:40:55.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.567 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.567 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3195 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:55.567 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:55.567 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.452387234, 2024-04-22T18:40:55.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69159474200000004, 2024-04-22T18:40:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3195 2024-04-22T18:40:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:55.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:55.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:55.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050333+0000", 2024-04-22T18:40:55.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0289999999999996e-06 2024-04-22T18:40:55.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050341+0000", 2024-04-22T18:40:55.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0050000000000004e-06 2024-04-22T18:40:55.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050346+0000", 2024-04-22T18:40:55.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8329999999999999e-06 2024-04-22T18:40:55.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150628+0000", 2024-04-22T18:40:55.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100281548 2024-04-22T18:40:55.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150672+0000", 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3875999999999998e-05 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741916+0000", 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59124448799999996 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741923+0000", 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7689999999999999e-06 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3196 4.11 4.9f8e4db1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.452357114999998, 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69158947400000004, 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3196 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050361+0000", 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2040000000000002e-06 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050367+0000", 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3119999999999999e-06 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050372+0000", 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0869999999999997e-06 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150688+0000", 2024-04-22T18:40:55.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100315381 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150691+0000", 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8899999999999999e-06 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741941+0000", 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59125044900000001 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741948+0000", 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.827e-06 2024-04-22T18:40:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3316 4.11 4:8f787b23:::smithi08937929-41:head [write 1969239~724762] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.539840+0000", 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.962875592, 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50791082700000001, 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3316 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.576 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539840+0000", 2024-04-22T18:40:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539840+0000", 2024-04-22T18:40:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539839+0000", 2024-04-22T18:40:55.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540217+0000", 2024-04-22T18:40:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037753899999999998 2024-04-22T18:40:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540225+0000", 2024-04-22T18:40:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4630000000000001e-06 2024-04-22T18:40:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540231+0000", 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0440000000000001e-06 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540273+0000", 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1480000000000003e-05 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540282+0000", 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.935e-06 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.844412+0000", 2024-04-22T18:40:55.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30413055500000002 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.844463+0000", 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0918000000000003e-05 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.844537+0000", 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3983e-05 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.948265+0000", 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103727471 2024-04-22T18:40:55.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.047658+0000", 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099393426000000007 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.047707+0000", 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8733000000000002e-05 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.047751+0000", 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3847000000000002e-05 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.581 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.582 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.582 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3317 4.11 4:8f787b23:::smithi08937929-41:head [setxattr _header (56) in=7b,truncate 2773188] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:55.582 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.540240+0000", 2024-04-22T18:40:55.582 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.962475734000002, 2024-04-22T18:40:55.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62115056400000002, 2024-04-22T18:40:55.582 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.582 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:55.582 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.582 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.582 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.582 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3317 2024-04-22T18:40:55.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.582 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540240+0000", 2024-04-22T18:40:55.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540240+0000", 2024-04-22T18:40:55.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540239+0000", 2024-04-22T18:40:55.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:55.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540243+0000", 2024-04-22T18:40:55.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4569999999999998e-06 2024-04-22T18:40:55.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540250+0000", 2024-04-22T18:40:55.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6440000000000003e-06 2024-04-22T18:40:55.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540256+0000", 2024-04-22T18:40:55.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8100000000000003e-06 2024-04-22T18:40:55.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540302+0000", 2024-04-22T18:40:55.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5859999999999998e-05 2024-04-22T18:40:55.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:55.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540309+0000", 2024-04-22T18:40:55.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3950000000000003e-06 2024-04-22T18:40:55.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.947528+0000", 2024-04-22T18:40:55.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40721847999999999 2024-04-22T18:40:55.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.947588+0000", 2024-04-22T18:40:55.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0364e-05 2024-04-22T18:40:55.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:55.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.947687+0000", 2024-04-22T18:40:55.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8733999999999994e-05 2024-04-22T18:40:55.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:55.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.948414+0000", 2024-04-22T18:40:55.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072766900000000002 2024-04-22T18:40:55.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:55.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161295+0000", 2024-04-22T18:40:55.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.212880818 2024-04-22T18:40:55.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:55.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161342+0000", 2024-04-22T18:40:55.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6894000000000002e-05 2024-04-22T18:40:55.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161390+0000", 2024-04-22T18:40:55.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8245999999999999e-05 2024-04-22T18:40:55.588 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.588 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.588 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.588 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3318 4.11 4:8f787b23:::smithi08937929-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:55.588 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:55.588 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.962445542000001, 2024-04-22T18:40:55.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62124934899999995, 2024-04-22T18:40:55.588 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.588 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3318 2024-04-22T18:40:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:55.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:55.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540276+0000", 2024-04-22T18:40:55.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.049e-06 2024-04-22T18:40:55.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540282+0000", 2024-04-22T18:40:55.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7050000000000002e-06 2024-04-22T18:40:55.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540288+0000", 2024-04-22T18:40:55.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2900000000000002e-06 2024-04-22T18:40:55.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540326+0000", 2024-04-22T18:40:55.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8818e-05 2024-04-22T18:40:55.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:55.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540330+0000", 2024-04-22T18:40:55.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6260000000000001e-06 2024-04-22T18:40:55.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.948024+0000", 2024-04-22T18:40:55.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40769397600000001 2024-04-22T18:40:55.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:55.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.948063+0000", 2024-04-22T18:40:55.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9127e-05 2024-04-22T18:40:55.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161420+0000", 2024-04-22T18:40:55.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.213356876 2024-04-22T18:40:55.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161463+0000", 2024-04-22T18:40:55.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3219000000000001e-05 2024-04-22T18:40:55.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161519+0000", 2024-04-22T18:40:55.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6155e-05 2024-04-22T18:40:55.593 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.593 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3337 4.11 4:8db271f9:::smithi08937929-19:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e96)", 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:27.149915+0000", 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.352800532, 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.42102647599999998, 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3337 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149915+0000", 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149915+0000", 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149915+0000", 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149918+0000", 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1030000000000002e-06 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149921+0000", 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0309999999999999e-06 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149924+0000", 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9510000000000002e-06 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149991+0000", 2024-04-22T18:40:55.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6773999999999995e-05 2024-04-22T18:40:55.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.150065+0000", 2024-04-22T18:40:55.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:40:55.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.570942+0000", 2024-04-22T18:40:55.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.420876947 2024-04-22T18:40:55.597 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.597 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.597 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.598 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3504 4.9 4:96737e19:::smithi08937929-4:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:55.598 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:40:55.598 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.091853939, 2024-04-22T18:40:55.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.951755291, 2024-04-22T18:40:55.598 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.598 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:55.598 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.598 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.598 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.598 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3504 2024-04-22T18:40:55.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.598 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410861+0000", 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410863+0000", 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.635e-06 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410866+0000", 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8499999999999998e-06 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410868+0000", 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.497e-06 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615137+0000", 2024-04-22T18:40:55.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20426903499999999 2024-04-22T18:40:55.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615218+0000", 2024-04-22T18:40:55.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0860999999999997e-05 2024-04-22T18:40:55.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.362617+0000", 2024-04-22T18:40:55.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.74739882700000004 2024-04-22T18:40:55.601 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.601 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.601 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3758 4.7 4:e117a562:::smithi08937929-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:40:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:45.699568+0000", 2024-04-22T18:40:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.8031473899999995, 2024-04-22T18:40:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50699397000000002, 2024-04-22T18:40:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3758 2024-04-22T18:40:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.699568+0000", 2024-04-22T18:40:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.699568+0000", 2024-04-22T18:40:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.699568+0000", 2024-04-22T18:40:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.699571+0000", 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9919999999999999e-06 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.699575+0000", 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8570000000000003e-06 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.699581+0000", 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6999999999999996e-06 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.799472+0000", 2024-04-22T18:40:55.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099890548999999995 2024-04-22T18:40:55.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:55.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.799517+0000", 2024-04-22T18:40:55.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5779000000000003e-05 2024-04-22T18:40:55.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:55.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.206434+0000", 2024-04-22T18:40:55.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40691630099999998 2024-04-22T18:40:55.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:55.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.206492+0000", 2024-04-22T18:40:55.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8350000000000002e-05 2024-04-22T18:40:55.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:55.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:55.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:55.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.206562+0000", 2024-04-22T18:40:55.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0094999999999999e-05 2024-04-22T18:40:55.606 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.606 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.606 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.606 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:55.606 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:55.606 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:55.608 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:55.609 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:40:55.610 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:40:55.611 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-04-22T18:40:54.984761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":269037525,"num_objects":209,"num_object_clones":0,"num_object_copies":418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":4062,"num_read_kb":3552835,"num_write":2391,"num_write_kb":573018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":195615072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":31,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2485,"ondisk_log_size":2485,"up":78,"acting":78,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":503524,"kb_used_data":332032,"kb_used_omap":7,"kb_used_meta":155512,"kb_avail":471355676,"statfs":{"total":483183820800,"available":482668212224,"internally_reserved":0,"allocated":340000768,"data_stored":345814897,"data_compressed":9518582,"data_compressed_allocated":10149888,"data_compressed_original":20877312,"omap_allocated":7962,"internal_metadata":159244518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13398660,"num_objects":-6,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":198,"num_read_kb":222577,"num_write":106,"num_write_kb":35682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":939979,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000805"},"pg_stats":[{"pgid":"4.19","version":"105'114","reported_seq":470,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.158197+0000","last_change":"2024-04-22T18:40:42.952084+0000","last_active":"2024-04-22T18:40:50.158197+0000","last_peered":"2024-04-22T18:40:50.158197+0000","last_clean":"2024-04-22T18:40:50.158197+0000","last_became_active":"2024-04-22T18:40:42.947705+0000","last_became_peered":"2024-04-22T18:40:42.947705+0000","last_unstale":"2024-04-22T18:40:50.158197+0000","last_undegraded":"2024-04-22T18:40:50.158197+0000","last_fullsized":"2024-04-22T18:40:50.158197+0000","mapping_epoch":110,"log_start":"84'88","ondisk_log_start":"84'88","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":26,"log_dups_size":106,"ondisk_log_size":26,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6680865,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":57,"num_read_kb":64438,"num_write":19,"num_write_kb":5735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4990114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"113'56","reported_seq":265,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.659800+0000","last_change":"2024-04-22T18:40:42.951030+0000","last_active":"2024-04-22T18:40:50.659800+0000","last_peered":"2024-04-22T18:40:50.659800+0000","last_clean":"2024-04-22T18:40:50.659800+0000","last_became_active":"2024-04-22T18:40:42.950080+0000","last_became_peered":"2024-04-22T18:40:42.950080+0000","last_unstale":"2024-04-22T18:40:50.659800+0000","last_undegraded":"2024-04-22T18:40:50.659800+0000","last_fullsized":"2024-04-22T18:40:50.659800+0000","mapping_epoch":110,"log_start":"79'52","ondisk_log_start":"79'52","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":61,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5025326,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":108,"num_read_kb":151814,"num_write":45,"num_write_kb":12231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2525724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"107'52","reported_seq":310,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.158489+0000","last_change":"2024-04-22T18:40:42.753148+0000","last_active":"2024-04-22T18:40:50.158489+0000","last_peered":"2024-04-22T18:40:50.158489+0000","last_clean":"2024-04-22T18:40:50.158489+0000","last_became_active":"2024-04-22T18:40:42.752884+0000","last_became_peered":"2024-04-22T18:40:42.752884+0000","last_unstale":"2024-04-22T18:40:50.158489+0000","last_undegraded":"2024-04-22T18:40:50.158489+0000","last_fullsized":"2024-04-22T18:40:50.158489+0000","mapping_epoch":110,"log_start":"83'49","ondisk_log_start":"83'49","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":5,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092497,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":66769,"num_write":16,"num_write_kb":3993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2254766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"115'90","reported_seq":376,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.844415+0000","last_change":"2024-04-22T18:40:47.049866+0000","last_active":"2024-04-22T18:40:50.844415+0000","last_peered":"2024-04-22T18:40:50.844415+0000","last_clean":"2024-04-22T18:40:50.844415+0000","last_became_active":"2024-04-22T18:40:47.049599+0000","last_became_peered":"2024-04-22T18:40:47.049599+0000","last_unstale":"2024-04-22T18:40:50.844415+0000","last_undegraded":"2024-04-22T18:40:50.844415+0000","last_fullsized":"2024-04-22T18:40:50.844415+0000","mapping_epoch":114,"log_start":"96'71","ondisk_log_start":"96'71","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":81,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.22461577699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5142992,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":57,"num_read_kb":73378,"num_write":21,"num_write_kb":6576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1847758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.15","version":"54'29","reported_seq":46,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.659865+0000","last_change":"2024-04-22T18:40:33.895403+0000","last_active":"2024-04-22T18:40:50.659865+0000","last_peered":"2024-04-22T18:40:50.659865+0000","last_clean":"2024-04-22T18:40:50.659865+0000","last_became_active":"2024-04-22T18:40:13.180144+0000","last_became_peered":"2024-04-22T18:40:13.180144+0000","last_unstale":"2024-04-22T18:40:50.659865+0000","last_undegraded":"2024-04-22T18:40:50.659865+0000","last_fullsized":"2024-04-22T18:40:50.659865+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'29","last_scrub_stamp":"2024-04-22T18:39:56.614637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:56.614637+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2915998,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":45460,"num_write":12,"num_write_kb":3102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"51'19","reported_seq":46,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:49.860218+0000","last_change":"2024-04-22T18:40:11.079224+0000","last_active":"2024-04-22T18:40:49.860218+0000","last_peered":"2024-04-22T18:40:49.860218+0000","last_clean":"2024-04-22T18:40:49.860218+0000","last_became_active":"2024-04-22T18:40:10.978840+0000","last_became_peered":"2024-04-22T18:40:10.978840+0000","last_unstale":"2024-04-22T18:40:49.860218+0000","last_undegraded":"2024-04-22T18:40:49.860218+0000","last_fullsized":"2024-04-22T18:40:49.860218+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1882368,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":22802,"num_write":10,"num_write_kb":3047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1349842,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"115'80","reported_seq":112,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-22T18:40:47.999205+0000","last_change":"2024-04-22T18:40:29.100526+0000","last_active":"2024-04-22T18:40:47.999205+0000","last_peered":"2024-04-22T18:40:47.999205+0000","last_clean":"2024-04-22T18:40:47.999205+0000","last_became_active":"2024-04-22T18:40:28.877828+0000","last_became_peered":"2024-04-22T18:40:28.877828+0000","last_unstale":"2024-04-22T18:40:47.999205+0000","last_undegraded":"2024-04-22T18:40:47.999205+0000","last_fullsized":"2024-04-22T18:40:47.999205+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'75","last_scrub_stamp":"2024-04-22T18:40:22.670296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.670296+0000","objects_scrubbed":2,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:44.049940+0000","scrub_duration":0.20951747300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218623,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":28169,"num_write":5,"num_write_kb":1772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4062110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"115'44","reported_seq":223,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.340660+0000","last_change":"2024-04-22T18:40:03.451290+0000","last_active":"2024-04-22T18:40:51.340660+0000","last_peered":"2024-04-22T18:40:51.340660+0000","last_clean":"2024-04-22T18:40:51.340660+0000","last_became_active":"2024-04-22T18:40:02.844371+0000","last_became_peered":"2024-04-22T18:40:02.844371+0000","last_unstale":"2024-04-22T18:40:51.340660+0000","last_undegraded":"2024-04-22T18:40:51.340660+0000","last_fullsized":"2024-04-22T18:40:51.340660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4008503,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":102,"num_read_kb":93785,"num_write":26,"num_write_kb":7134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":970119,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"115'135","reported_seq":414,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.158210+0000","last_change":"2024-04-22T18:40:26.844287+0000","last_active":"2024-04-22T18:40:50.158210+0000","last_peered":"2024-04-22T18:40:50.158210+0000","last_clean":"2024-04-22T18:40:50.158210+0000","last_became_active":"2024-04-22T18:40:26.742351+0000","last_became_peered":"2024-04-22T18:40:26.742351+0000","last_unstale":"2024-04-22T18:40:50.158210+0000","last_undegraded":"2024-04-22T18:40:50.158210+0000","last_fullsized":"2024-04-22T18:40:50.158210+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'113","last_scrub_stamp":"2024-04-22T18:40:20.291988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.291988+0000","objects_scrubbed":2,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:22.684852+0000","scrub_duration":0.10270805500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6823708,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":8729,"num_write":27,"num_write_kb":7833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3940523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"31'5","reported_seq":150,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:49.860073+0000","last_change":"2024-04-22T18:40:10.212897+0000","last_active":"2024-04-22T18:40:49.860073+0000","last_peered":"2024-04-22T18:40:49.860073+0000","last_clean":"2024-04-22T18:40:49.860073+0000","last_became_active":"2024-04-22T18:39:59.265189+0000","last_became_peered":"2024-04-22T18:39:59.265189+0000","last_unstale":"2024-04-22T18:40:49.860073+0000","last_undegraded":"2024-04-22T18:40:49.860073+0000","last_fullsized":"2024-04-22T18:40:49.860073+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":5,"last_scrub":"31'5","last_scrub_stamp":"2024-04-22T18:40:10.212815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.212815+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:17.620115+0000","scrub_duration":0.0015068670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.153662+0000","last_change":"2024-04-22T18:40:19.137046+0000","last_active":"2024-04-22T18:40:51.153662+0000","last_peered":"2024-04-22T18:40:51.153662+0000","last_clean":"2024-04-22T18:40:51.153662+0000","last_became_active":"2024-04-22T18:38:39.025741+0000","last_became_peered":"2024-04-22T18:38:39.025741+0000","last_unstale":"2024-04-22T18:40:51.153662+0000","last_undegraded":"2024-04-22T18:40:51.153662+0000","last_fullsized":"2024-04-22T18:40:51.153662+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:19.136933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:19.136933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:21.035233+0000","scrub_duration":0.0023924060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"117'161","reported_seq":550,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.355659+0000","last_change":"2024-04-22T18:40:36.329312+0000","last_active":"2024-04-22T18:40:51.355659+0000","last_peered":"2024-04-22T18:40:51.355659+0000","last_clean":"2024-04-22T18:40:51.355659+0000","last_became_active":"2024-04-22T18:40:34.914043+0000","last_became_peered":"2024-04-22T18:40:34.914043+0000","last_unstale":"2024-04-22T18:40:51.355659+0000","last_undegraded":"2024-04-22T18:40:51.355659+0000","last_fullsized":"2024-04-22T18:40:51.355659+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:06.180858+0000","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13395278,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":246,"num_read_kb":286454,"num_write":56,"num_write_kb":16276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":26002234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":110,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.158564+0000","last_change":"2024-04-22T18:38:39.110499+0000","last_active":"2024-04-22T18:40:50.158564+0000","last_peered":"2024-04-22T18:40:50.158564+0000","last_clean":"2024-04-22T18:40:50.158564+0000","last_became_active":"2024-04-22T18:38:39.010173+0000","last_became_peered":"2024-04-22T18:38:39.010173+0000","last_unstale":"2024-04-22T18:40:50.158564+0000","last_undegraded":"2024-04-22T18:40:50.158564+0000","last_fullsized":"2024-04-22T18:40:50.158564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"115'19","reported_seq":179,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.007211+0000","last_change":"2024-04-22T18:39:54.378188+0000","last_active":"2024-04-22T18:40:51.007211+0000","last_peered":"2024-04-22T18:40:51.007211+0000","last_clean":"2024-04-22T18:40:51.007211+0000","last_became_active":"2024-04-22T18:39:46.805569+0000","last_became_peered":"2024-04-22T18:39:46.805569+0000","last_unstale":"2024-04-22T18:40:51.007211+0000","last_undegraded":"2024-04-22T18:40:51.007211+0000","last_fullsized":"2024-04-22T18:40:51.007211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'13","last_scrub_stamp":"2024-04-22T18:39:54.377980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.377980+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:00.921987+0000","scrub_duration":0.30260881099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":85369,"num_write":6,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":127,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.659914+0000","last_change":"2024-04-22T18:40:35.007077+0000","last_active":"2024-04-22T18:40:50.659914+0000","last_peered":"2024-04-22T18:40:50.659914+0000","last_clean":"2024-04-22T18:40:50.659914+0000","last_became_active":"2024-04-22T18:40:35.006777+0000","last_became_peered":"2024-04-22T18:40:35.006777+0000","last_unstale":"2024-04-22T18:40:50.659914+0000","last_undegraded":"2024-04-22T18:40:50.659914+0000","last_fullsized":"2024-04-22T18:40:50.659914+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:09.538542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:09.538542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:18.970457+0000","scrub_duration":0.35201753200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"114'85","reported_seq":304,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.237142+0000","last_change":"2024-04-22T18:40:35.964284+0000","last_active":"2024-04-22T18:40:51.237142+0000","last_peered":"2024-04-22T18:40:51.237142+0000","last_clean":"2024-04-22T18:40:51.237142+0000","last_became_active":"2024-04-22T18:40:34.910015+0000","last_became_peered":"2024-04-22T18:40:34.910015+0000","last_unstale":"2024-04-22T18:40:51.237142+0000","last_undegraded":"2024-04-22T18:40:51.237142+0000","last_fullsized":"2024-04-22T18:40:51.237142+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'67","last_scrub_stamp":"2024-04-22T18:39:53.916646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.916646+0000","objects_scrubbed":2,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:06.623573+0000","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554796,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":64717,"num_write":17,"num_write_kb":4399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7812071,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.258398+0000","last_change":"2024-04-22T18:40:35.014535+0000","last_active":"2024-04-22T18:40:50.258398+0000","last_peered":"2024-04-22T18:40:50.258398+0000","last_clean":"2024-04-22T18:40:50.258398+0000","last_became_active":"2024-04-22T18:40:35.014154+0000","last_became_peered":"2024-04-22T18:40:35.014154+0000","last_unstale":"2024-04-22T18:40:50.258398+0000","last_undegraded":"2024-04-22T18:40:50.258398+0000","last_fullsized":"2024-04-22T18:40:50.258398+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"64'38","reported_seq":241,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.256438+0000","last_change":"2024-04-22T18:40:35.556480+0000","last_active":"2024-04-22T18:40:50.256438+0000","last_peered":"2024-04-22T18:40:50.256438+0000","last_clean":"2024-04-22T18:40:50.256438+0000","last_became_active":"2024-04-22T18:40:34.911555+0000","last_became_peered":"2024-04-22T18:40:34.911555+0000","last_unstale":"2024-04-22T18:40:50.256438+0000","last_undegraded":"2024-04-22T18:40:50.256438+0000","last_fullsized":"2024-04-22T18:40:50.256438+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2932821,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":104,"num_read_kb":148000,"num_write":20,"num_write_kb":5595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":8531127,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"115'293","reported_seq":791,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.153290+0000","last_change":"2024-04-22T18:40:47.165729+0000","last_active":"2024-04-22T18:40:51.153290+0000","last_peered":"2024-04-22T18:40:51.153290+0000","last_clean":"2024-04-22T18:40:51.153290+0000","last_became_active":"2024-04-22T18:40:35.239662+0000","last_became_peered":"2024-04-22T18:40:35.239662+0000","last_unstale":"2024-04-22T18:40:51.153290+0000","last_undegraded":"2024-04-22T18:40:51.153290+0000","last_fullsized":"2024-04-22T18:40:51.153290+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:38:46.054802+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29489419,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":251,"num_read_kb":4944,"num_write":356,"num_write_kb":87998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":28207650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.256494+0000","last_change":"2024-04-22T18:40:33.899901+0000","last_active":"2024-04-22T18:40:50.256494+0000","last_peered":"2024-04-22T18:40:50.256494+0000","last_clean":"2024-04-22T18:40:50.256494+0000","last_became_active":"2024-04-22T18:40:02.945025+0000","last_became_peered":"2024-04-22T18:40:02.945025+0000","last_unstale":"2024-04-22T18:40:50.256494+0000","last_undegraded":"2024-04-22T18:40:50.256494+0000","last_fullsized":"2024-04-22T18:40:50.256494+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"115'32","reported_seq":228,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.258383+0000","last_change":"2024-04-22T18:40:21.426510+0000","last_active":"2024-04-22T18:40:50.258383+0000","last_peered":"2024-04-22T18:40:50.258383+0000","last_clean":"2024-04-22T18:40:50.258383+0000","last_became_active":"2024-04-22T18:39:51.943071+0000","last_became_peered":"2024-04-22T18:39:51.943071+0000","last_unstale":"2024-04-22T18:40:50.258383+0000","last_undegraded":"2024-04-22T18:40:50.258383+0000","last_fullsized":"2024-04-22T18:40:50.258383+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-04-22T18:40:21.426350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.426350+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.678685+0000","scrub_duration":0.30334366400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2605118,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":51,"num_read_kb":55069,"num_write":4,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"117'219","reported_seq":554,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.442512+0000","last_change":"2024-04-22T18:40:20.465005+0000","last_active":"2024-04-22T18:40:51.442512+0000","last_peered":"2024-04-22T18:40:51.442512+0000","last_clean":"2024-04-22T18:40:51.442512+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:40:51.442512+0000","last_undegraded":"2024-04-22T18:40:51.442512+0000","last_fullsized":"2024-04-22T18:40:51.442512+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'176","last_scrub_stamp":"2024-04-22T18:40:20.464799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.464799+0000","objects_scrubbed":31,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:45.914514+0000","scrub_duration":0.30629975799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27244592,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":187,"num_read_kb":2520,"num_write":277,"num_write_kb":62747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":75,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-22T18:40:46.846876+0000","last_change":"2024-04-22T18:40:10.403338+0000","last_active":"2024-04-22T18:40:46.846876+0000","last_peered":"2024-04-22T18:40:46.846876+0000","last_clean":"2024-04-22T18:40:46.846876+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:40:46.846876+0000","last_undegraded":"2024-04-22T18:40:46.846876+0000","last_fullsized":"2024-04-22T18:40:46.846876+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:10.403189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.403189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:34.232014+0000","scrub_duration":0.201433005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"113'56","reported_seq":319,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.344928+0000","last_change":"2024-04-22T18:40:42.750352+0000","last_active":"2024-04-22T18:40:51.344928+0000","last_peered":"2024-04-22T18:40:51.344928+0000","last_clean":"2024-04-22T18:40:51.344928+0000","last_became_active":"2024-04-22T18:40:42.750084+0000","last_became_peered":"2024-04-22T18:40:42.750084+0000","last_unstale":"2024-04-22T18:40:51.344928+0000","last_undegraded":"2024-04-22T18:40:51.344928+0000","last_fullsized":"2024-04-22T18:40:51.344928+0000","mapping_epoch":106,"log_start":"83'49","ondisk_log_start":"83'49","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":49,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3310987,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":67,"num_read_kb":74547,"num_write":20,"num_write_kb":5725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2254765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"115'261","reported_seq":729,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:49.960379+0000","last_change":"2024-04-22T18:40:47.255770+0000","last_active":"2024-04-22T18:40:49.960379+0000","last_peered":"2024-04-22T18:40:49.960379+0000","last_clean":"2024-04-22T18:40:49.960379+0000","last_became_active":"2024-04-22T18:40:35.038135+0000","last_became_peered":"2024-04-22T18:40:35.038135+0000","last_unstale":"2024-04-22T18:40:49.960379+0000","last_undegraded":"2024-04-22T18:40:49.960379+0000","last_fullsized":"2024-04-22T18:40:49.960379+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'209","last_scrub_stamp":"2024-04-22T18:40:18.683442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:18.683442+0000","objects_scrubbed":24,"log_size":261,"log_dups_size":0,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:21.496831+0000","scrub_duration":0.204344365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29514217,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":227,"num_read_kb":5187,"num_write":321,"num_write_kb":79183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":23971897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":119,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.659784+0000","last_change":"2024-04-22T18:40:22.521828+0000","last_active":"2024-04-22T18:40:50.659784+0000","last_peered":"2024-04-22T18:40:50.659784+0000","last_clean":"2024-04-22T18:40:50.659784+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:40:50.659784+0000","last_undegraded":"2024-04-22T18:40:50.659784+0000","last_fullsized":"2024-04-22T18:40:50.659784+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.521638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:43.407910+0000","scrub_duration":0.30190717299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"115'88","reported_seq":366,"reported_epoch":116,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:40:50.257232+0000","last_change":"2024-04-22T18:40:43.158709+0000","last_active":"2024-04-22T18:40:50.257232+0000","last_peered":"2024-04-22T18:40:50.257232+0000","last_clean":"2024-04-22T18:40:32.014508+0000","last_became_active":"2024-04-22T18:40:43.046938+0000","last_became_peered":"2024-04-22T18:40:43.046938+0000","last_unstale":"2024-04-22T18:40:50.257232+0000","last_undegraded":"2024-04-22T18:40:50.257232+0000","last_fullsized":"2024-04-22T18:40:50.257232+0000","mapping_epoch":104,"log_start":"96'71","ondisk_log_start":"96'71","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":2,"log_size":17,"log_dups_size":81,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.22461577699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1815172,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":99604,"num_write":17,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,0],"avail_no_missing":["1","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"115'250","reported_seq":552,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.153304+0000","last_change":"2024-04-22T18:40:21.871081+0000","last_active":"2024-04-22T18:40:51.153304+0000","last_peered":"2024-04-22T18:40:51.153304+0000","last_clean":"2024-04-22T18:40:51.153304+0000","last_became_active":"2024-04-22T18:40:10.784978+0000","last_became_peered":"2024-04-22T18:40:10.784978+0000","last_unstale":"2024-04-22T18:40:51.153304+0000","last_undegraded":"2024-04-22T18:40:51.153304+0000","last_fullsized":"2024-04-22T18:40:51.153304+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'212","last_scrub_stamp":"2024-04-22T18:40:21.870855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.870855+0000","objects_scrubbed":46,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:28.179796+0000","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33918065,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":230,"num_read_kb":6186,"num_write":320,"num_write_kb":68546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":13435025,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":130,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-22T18:40:46.947165+0000","last_change":"2024-04-22T18:40:07.420765+0000","last_active":"2024-04-22T18:40:46.947165+0000","last_peered":"2024-04-22T18:40:46.947165+0000","last_clean":"2024-04-22T18:40:46.947165+0000","last_became_active":"2024-04-22T18:40:02.741862+0000","last_became_peered":"2024-04-22T18:40:02.741862+0000","last_unstale":"2024-04-22T18:40:46.947165+0000","last_undegraded":"2024-04-22T18:40:46.947165+0000","last_fullsized":"2024-04-22T18:40:46.947165+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:07.420618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:07.420618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.642486+0000","scrub_duration":0.30164867299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":284,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.659869+0000","last_change":"2024-04-22T18:40:35.216423+0000","last_active":"2024-04-22T18:40:50.659869+0000","last_peered":"2024-04-22T18:40:50.659869+0000","last_clean":"2024-04-22T18:40:50.659869+0000","last_became_active":"2024-04-22T18:40:34.907666+0000","last_became_peered":"2024-04-22T18:40:34.907666+0000","last_unstale":"2024-04-22T18:40:50.659869+0000","last_undegraded":"2024-04-22T18:40:50.659869+0000","last_fullsized":"2024-04-22T18:40:50.659869+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:39:53.346787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.346787+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:15.525107+0000","scrub_duration":0.20273052499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"104'59","reported_seq":261,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.153456+0000","last_change":"2024-04-22T18:40:29.696372+0000","last_active":"2024-04-22T18:40:51.153456+0000","last_peered":"2024-04-22T18:40:51.153456+0000","last_clean":"2024-04-22T18:40:51.153456+0000","last_became_active":"2024-04-22T18:40:28.989193+0000","last_became_peered":"2024-04-22T18:40:28.989193+0000","last_unstale":"2024-04-22T18:40:51.153456+0000","last_undegraded":"2024-04-22T18:40:51.153456+0000","last_fullsized":"2024-04-22T18:40:51.153456+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'38","last_scrub_stamp":"2024-04-22T18:39:55.746067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:55.746067+0000","objects_scrubbed":1,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:20.212013+0000","scrub_duration":0.60877772299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7613571,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":83848,"num_write":22,"num_write_kb":6499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4764796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"79'52","reported_seq":254,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.659699+0000","last_change":"2024-04-22T18:40:42.849298+0000","last_active":"2024-04-22T18:40:50.659699+0000","last_peered":"2024-04-22T18:40:50.659699+0000","last_clean":"2024-04-22T18:40:50.659699+0000","last_became_active":"2024-04-22T18:40:42.848954+0000","last_became_peered":"2024-04-22T18:40:42.848954+0000","last_unstale":"2024-04-22T18:40:50.659699+0000","last_undegraded":"2024-04-22T18:40:50.659699+0000","last_fullsized":"2024-04-22T18:40:50.659699+0000","mapping_epoch":72,"log_start":"79'52","ondisk_log_start":"79'52","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":61,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1747928,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":89,"num_read_kb":120607,"num_write":39,"num_write_kb":10408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2525724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"116'121","reported_seq":479,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.942978+0000","last_change":"2024-04-22T18:40:43.053292+0000","last_active":"2024-04-22T18:40:50.942978+0000","last_peered":"2024-04-22T18:40:50.942978+0000","last_clean":"2024-04-22T18:40:50.942978+0000","last_became_active":"2024-04-22T18:40:43.052945+0000","last_became_peered":"2024-04-22T18:40:43.052945+0000","last_unstale":"2024-04-22T18:40:50.942978+0000","last_undegraded":"2024-04-22T18:40:50.942978+0000","last_fullsized":"2024-04-22T18:40:50.942978+0000","mapping_epoch":101,"log_start":"84'88","ondisk_log_start":"84'88","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":33,"log_dups_size":106,"ondisk_log_size":33,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4848869,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":61,"num_read_kb":70782,"num_write":27,"num_write_kb":8118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4990114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"115'105","reported_seq":341,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-22T18:40:49.544295+0000","last_change":"2024-04-22T18:40:22.140818+0000","last_active":"2024-04-22T18:40:49.544295+0000","last_peered":"2024-04-22T18:40:49.544295+0000","last_clean":"2024-04-22T18:40:49.544295+0000","last_became_active":"2024-04-22T18:38:48.277922+0000","last_became_peered":"2024-04-22T18:38:48.277922+0000","last_unstale":"2024-04-22T18:40:49.544295+0000","last_undegraded":"2024-04-22T18:40:49.544295+0000","last_fullsized":"2024-04-22T18:40:49.544295+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:39.617182+0000","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13378471,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":287,"num_read_kb":338955,"num_write":106,"num_write_kb":28032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"115'138","reported_seq":464,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.158585+0000","last_change":"2024-04-22T18:40:36.165101+0000","last_active":"2024-04-22T18:40:50.158585+0000","last_peered":"2024-04-22T18:40:50.158585+0000","last_clean":"2024-04-22T18:40:50.158585+0000","last_became_active":"2024-04-22T18:40:35.110751+0000","last_became_peered":"2024-04-22T18:40:35.110751+0000","last_unstale":"2024-04-22T18:40:50.158585+0000","last_undegraded":"2024-04-22T18:40:50.158585+0000","last_fullsized":"2024-04-22T18:40:50.158585+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'109","last_scrub_stamp":"2024-04-22T18:40:17.658363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:17.658363+0000","objects_scrubbed":5,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:37.958968+0000","scrub_duration":0.202176033,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15496186,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":467,"num_read_kb":515535,"num_write":146,"num_write_kb":42951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8880696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"114'118","reported_seq":402,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:49.860220+0000","last_change":"2024-04-22T18:40:29.506777+0000","last_active":"2024-04-22T18:40:49.860220+0000","last_peered":"2024-04-22T18:40:49.860220+0000","last_clean":"2024-04-22T18:40:49.860220+0000","last_became_active":"2024-04-22T18:40:28.975797+0000","last_became_peered":"2024-04-22T18:40:28.975797+0000","last_unstale":"2024-04-22T18:40:49.860220+0000","last_undegraded":"2024-04-22T18:40:49.860220+0000","last_fullsized":"2024-04-22T18:40:49.860220+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12487700,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":415,"num_read_kb":485666,"num_write":123,"num_write_kb":35069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7572471,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"115'60","reported_seq":291,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:50.261470+0000","last_change":"2024-04-22T18:40:34.755224+0000","last_active":"2024-04-22T18:40:50.261470+0000","last_peered":"2024-04-22T18:40:50.261470+0000","last_clean":"2024-04-22T18:40:50.261470+0000","last_became_active":"2024-04-22T18:40:28.778850+0000","last_became_peered":"2024-04-22T18:40:28.778850+0000","last_unstale":"2024-04-22T18:40:50.261470+0000","last_undegraded":"2024-04-22T18:40:50.261470+0000","last_fullsized":"2024-04-22T18:40:50.261470+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'46","last_scrub_stamp":"2024-04-22T18:40:34.755106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:34.755106+0000","objects_scrubbed":2,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:52.199053+0000","scrub_duration":0.40192567099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7075813,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":194114,"num_write":64,"num_write_kb":19036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":10972843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"113'35","reported_seq":211,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.153453+0000","last_change":"2024-04-22T18:40:29.090636+0000","last_active":"2024-04-22T18:40:51.153453+0000","last_peered":"2024-04-22T18:40:51.153453+0000","last_clean":"2024-04-22T18:40:51.153453+0000","last_became_active":"2024-04-22T18:40:28.778111+0000","last_became_peered":"2024-04-22T18:40:28.778111+0000","last_unstale":"2024-04-22T18:40:51.153453+0000","last_undegraded":"2024-04-22T18:40:51.153453+0000","last_fullsized":"2024-04-22T18:40:51.153453+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'23","last_scrub_stamp":"2024-04-22T18:40:20.587606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.587606+0000","objects_scrubbed":1,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.710668+0000","scrub_duration":0.10225366599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899323,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":153548,"num_write":35,"num_write_kb":11536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2364451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"110'74","reported_seq":272,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-22T18:40:51.136777+0000","last_change":"2024-04-22T18:38:48.173640+0000","last_active":"2024-04-22T18:40:51.136777+0000","last_peered":"2024-04-22T18:40:51.136777+0000","last_clean":"2024-04-22T18:40:51.136777+0000","last_became_active":"2024-04-22T18:38:48.173369+0000","last_became_peered":"2024-04-22T18:38:48.173369+0000","last_unstale":"2024-04-22T18:40:51.136777+0000","last_undegraded":"2024-04-22T18:40:51.136777+0000","last_fullsized":"2024-04-22T18:40:51.136777+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7327951,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":216,"num_read_kb":201739,"num_write":75,"num_write_kb":19712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":148280864,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":3051,"num_read_kb":3533898,"num_write":958,"num_write_kb":272128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":108612248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":31,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":227815424,"data_stored":227283369,"data_compressed":667075,"data_compressed_allocated":704512,"data_compressed_original":1241088,"omap_allocated":0,"internal_metadata":0},"log_size":1345,"ondisk_log_size":1345,"up":52,"acting":52,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":120166293,"num_objects":160,"num_object_clones":0,"num_object_copies":320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":895,"num_read_kb":18837,"num_write":1274,"num_write_kb":298474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":86473896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218955776,"data_stored":231353426,"data_compressed":12437636,"data_compressed_allocated":13242368,"data_compressed_original":26378240,"omap_allocated":0,"internal_metadata":0},"log_size":1023,"ondisk_log_size":1023,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1586784,"data_compressed":90130,"data_compressed_allocated":122880,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82592,"kb_used_data":49836,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94289248,"statfs":{"total":96636764160,"available":96552189952,"internally_reserved":0,"allocated":51032064,"data_stored":51835378,"data_compressed":1442093,"data_compressed_allocated":1540096,"data_compressed_original":3215360,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":9.8949999999999996,"5min":2.3170000000000002,"15min":1.054},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":139.84299999999999,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":13.949,"5min":6.3959999999999999,"15min":5.1369999999999996},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":186.97900000000001,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.49199999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":4.1959999999999997,"15min":4.6420000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":27.027000000000001,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":14.961,"5min":13.43,"15min":13.175000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":183.16300000000001,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.38200000000000001}]},{"osd":2,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":3.3479999999999999,"15min":3.839},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64100000000000001,"5min":77.320999999999998,"15min":77.320999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47199999999999998,"15min":0.47299999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":3.4079999999999999,"15min":3.903},"min":{"1min":0.32000000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60799999999999998,"5min":77.361000000000004,"15min":77.361000000000004},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.30299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.65000000000000002,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.501}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":5.6710000000000003,"5min":4.7009999999999996,"15min":4.5389999999999997},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":69.097999999999999,"5min":85.509,"15min":85.509},"last":0.46000000000000002},{"interface":"front","average":{"1min":7.9509999999999996,"5min":19.887,"15min":21.876999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":181.97,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.46899999999999997}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.5379999999999998,"15min":2.8929999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67800000000000005,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.29599999999999999},{"interface":"front","average":{"1min":3.3170000000000002,"5min":6.3929999999999998,"15min":6.9050000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":69.063000000000002,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.43099999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":17.294,"5min":14.079000000000001,"15min":13.542999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":139.428,"5min":139.428,"15min":139.428},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":3.5779999999999998,"15min":3.948},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":22.878,"5min":77.405000000000001,"15min":77.405000000000001},"last":0.36199999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112980,"kb_used_data":79968,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94258860,"statfs":{"total":96636764160,"available":96521072640,"internally_reserved":0,"allocated":81887232,"data_stored":83763829,"data_compressed":2315210,"data_compressed_allocated":2469888,"data_compressed_original":5435392,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:43 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":1.516,"15min":1.7010000000000001},"min":{"1min":0.24099999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.57199999999999995,"5min":32.097999999999999,"15min":32.097999999999999},"last":0.251},{"interface":"front","average":{"1min":5.9290000000000003,"5min":21.832000000000001,"15min":24.481999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":52.465000000000003,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:40:43 2024","interfaces":[{"interface":"back","average":{"1min":11.403,"5min":3.911,"15min":2.6629999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":199.047,"5min":199.047,"15min":199.047},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.6599999999999999,"5min":8.4049999999999994,"15min":9.5289999999999999},"min":{"1min":0.318,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":22.747,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:40:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.45000000000000001,"15min":0.45600000000000002},"min":{"1min":0.29999999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56599999999999995,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.34899999999999998},{"interface":"front","average":{"1min":5.944,"5min":16.231999999999999,"15min":17.946000000000002},"min":{"1min":0.32100000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":52.515999999999998,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.40300000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:40:43 2024","interfaces":[{"interface":"back","average":{"1min":9.0640000000000001,"5min":19.027999999999999,"15min":20.687999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":103.976,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":4.9039999999999999,"5min":1.339,"15min":0.745},"min":{"1min":0.33100000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":77.097999999999999,"5min":77.097999999999999,"15min":77.097999999999999},"last":0.309}]},{"osd":4,"last update":"Mon Apr 22 18:40:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":6.1600000000000001,"15min":7.1219999999999999},"min":{"1min":0.26400000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53000000000000003,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.372},{"interface":"front","average":{"1min":13.000999999999999,"5min":18.062000000000001,"15min":18.905999999999999},"min":{"1min":0.255,"5min":0.218,"15min":0.218},"max":{"1min":99.771000000000001,"5min":166.733,"15min":166.733},"last":0.42699999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:40:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39600000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.252,"15min":0.252},"max":{"1min":0.56000000000000005,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.44700000000000001},"min":{"1min":0.33700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.70199999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.41599999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:40:43 2024","interfaces":[{"interface":"back","average":{"1min":9.9819999999999993,"5min":17.077000000000002,"15min":18.259},"min":{"1min":0.309,"5min":0.308,"15min":0.308},"max":{"1min":72.415000000000006,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46200000000000002,"15min":0.46400000000000002},"min":{"1min":0.312,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.61899999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":13.077}]}]},{"osd":5,"up_from":0,"seq":55834574877,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":55834574877,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106596,"kb_used_data":70516,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94265244,"statfs":{"total":96636764160,"available":96527609856,"internally_reserved":0,"allocated":72208384,"data_stored":73645262,"data_compressed":2562537,"data_compressed_allocated":2719744,"data_compressed_original":5156864,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":9.9700000000000006,"15min":11.456},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":11.797000000000001,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.182},{"interface":"front","average":{"1min":0.26700000000000002,"5min":0.316,"15min":0.32400000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.43099999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.214}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":25.978000000000002,"5min":9.2309999999999999,"15min":6.4400000000000004},"min":{"1min":0.20999999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":197.02099999999999,"5min":197.02099999999999,"15min":197.02099999999999},"last":78.805000000000007},{"interface":"front","average":{"1min":1.623,"5min":0.59199999999999997,"15min":0.41999999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":16.388999999999999,"5min":16.388999999999999,"15min":16.388999999999999},"last":0.20999999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":6.0090000000000003,"5min":2.0059999999999998,"15min":1.339},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":87.635999999999996,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.51800000000000002,"15min":0.45600000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":11.859999999999999,"5min":11.859999999999999,"15min":11.859999999999999},"last":0.253}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":3.379,"15min":3.8650000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79100000000000004,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.28999999999999998},{"interface":"front","average":{"1min":6.1719999999999997,"5min":4.5439999999999996,"15min":4.2729999999999997},"min":{"1min":0.34799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":87.724000000000004,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.27600000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":6.1539999999999999,"5min":2.1760000000000002,"15min":1.5129999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":87.784000000000006,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.30299999999999999},{"interface":"front","average":{"1min":1.038,"5min":1.2130000000000001,"15min":1.242},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":11.911,"5min":13.965,"15min":13.965},"last":0.35699999999999998}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.52700000000000002,"15min":0.53700000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72999999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.32100000000000001},{"interface":"front","average":{"1min":9.9130000000000003,"5min":2.4079999999999999,"15min":1.157},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":96.140000000000001,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.38}]},{"osd":7,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":0.69599999999999995,"15min":0.621},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":11.887,"5min":11.887,"15min":11.887},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.55000000000000004,"15min":0.56200000000000006},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75600000000000001,"5min":0.995,"15min":0.995},"last":0.37}]}]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57952,"kb_used_data":29236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94313888,"statfs":{"total":96636764160,"available":96577421312,"internally_reserved":0,"allocated":29937664,"data_stored":30366509,"data_compressed":1083429,"data_compressed_allocated":1159168,"data_compressed_original":2371584,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":7.4240000000000004,"15min":7.8920000000000003},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":59.002000000000002,"5min":149.03,"15min":149.03},"last":0.246},{"interface":"front","average":{"1min":0.308,"5min":1.649,"15min":1.873},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.442,"5min":31.866,"15min":31.866},"last":0.28599999999999998}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":3.5579999999999998,"15min":1.8740000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":97.683999999999997,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.27100000000000002},{"interface":"front","average":{"1min":22.157,"5min":16.338999999999999,"15min":15.369},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":393.42399999999998,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.33500000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":18.393000000000001,"5min":3.9689999999999999,"15min":1.5649999999999999},"min":{"1min":0.252,"5min":0.245,"15min":0.245},"max":{"1min":147.721,"5min":147.721,"15min":147.721},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.36899999999999999,"15min":0.377},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.53200000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.307}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":3.4060000000000001,"5min":15.698,"15min":17.747},"min":{"1min":0.311,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":52.683,"5min":160.584,"15min":160.584},"last":0.5},{"interface":"front","average":{"1min":10.984,"5min":4.1550000000000002,"15min":3.0169999999999999},"min":{"1min":0.32200000000000001,"5min":0.315,"15min":0.315},"max":{"1min":97.887,"5min":97.887,"15min":97.887},"last":0.56999999999999995}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":2.5409999999999999,"15min":2.8809999999999998},"min":{"1min":0.33300000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.74299999999999999,"5min":48.631,"15min":48.631},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":2.8820000000000001,"15min":3.278},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.83499999999999996,"5min":56.18,"15min":56.18},"last":0.53100000000000003}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":10.375,"5min":2.5030000000000001,"15min":1.1910000000000001},"min":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":97.832999999999998,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.33200000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.49399999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.63800000000000001,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"last":0.55800000000000005},{"interface":"front","average":{"1min":10.396000000000001,"5min":2.5470000000000002,"15min":1.2390000000000001},"min":{"1min":0.38800000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":97.861999999999995,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.54000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":143404,"kb_used_data":102476,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94228436,"statfs":{"total":96636764160,"available":96489918464,"internally_reserved":0,"allocated":104935424,"data_stored":106203919,"data_compressed":2115313,"data_compressed_allocated":2260992,"data_compressed_original":4698112,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":3.9870000000000001,"5min":7.0839999999999996,"15min":7.5999999999999996},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":66.480999999999995,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.33400000000000002},{"interface":"front","average":{"1min":4.0430000000000001,"5min":7.0990000000000002,"15min":7.6079999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":66.498000000000005,"5min":88.156000000000006,"15min":88.156000000000006},"last":0.27700000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":6.9429999999999996,"15min":8.0470000000000006},"min":{"1min":0.20000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.626,"5min":183.04400000000001,"15min":183.04400000000001},"last":0.308},{"interface":"front","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.39100000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.33700000000000002,"15min":0.33400000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":11.664,"15min":13.548999999999999},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.878,"5min":231.864,"15min":231.864},"last":0.32100000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":7.29,"5min":11.845000000000001,"15min":12.603999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":109.399,"5min":192.42099999999999,"15min":192.42099999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.47899999999999998}]},{"osd":5,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":11.359999999999999,"15min":13.167},"min":{"1min":0.34399999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.052,"5min":138.089,"15min":138.089},"last":0.435},{"interface":"front","average":{"1min":0.48999999999999999,"5min":16.158999999999999,"15min":18.771000000000001},"min":{"1min":0.35099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69899999999999995,"5min":183.08199999999999,"15min":183.08199999999999},"last":0.44500000000000001}]},{"osd":6,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":4.8520000000000003,"5min":3.1920000000000002,"15min":2.915},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":78.805000000000007,"5min":78.805000000000007,"15min":78.805000000000007},"last":0.41999999999999998},{"interface":"front","average":{"1min":5.9400000000000004,"5min":14.413,"15min":15.824999999999999},"min":{"1min":0.36599999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":98.968000000000004,"5min":183.10900000000001,"15min":183.10900000000001},"last":33.441000000000003}]},{"osd":7,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.38500000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.503},{"interface":"front","average":{"1min":0.52100000000000002,"5min":2.5350000000000001,"15min":2.871},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.0189999999999999,"5min":51.097000000000001,"15min":51.097000000000001},"last":0.40300000000000002}]}]},{"osd":0,"up_from":0,"seq":55834574877,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33095680,"data_stored":34790173,"data_compressed":1758468,"data_compressed_allocated":1871872,"data_compressed_original":3694592,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35766272,"data_stored":37969348,"data_compressed":1987296,"data_compressed_allocated":2125824,"data_compressed_original":4435968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16535552,"data_stored":17706169,"data_compressed":1083429,"data_compressed_allocated":1159168,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40054784,"data_stored":42280193,"data_compressed":2453667,"data_compressed_allocated":2605056,"data_compressed_original":4960256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":34144256,"data_stored":36043524,"data_compressed":1815904,"data_compressed_allocated":1929216,"data_compressed_original":3956736,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":223399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":38842368,"data_stored":40993712,"data_compressed":2086377,"data_compressed_allocated":2220032,"data_compressed_original":4517888,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20287488,"data_stored":21346908,"data_compressed":1252495,"data_compressed_allocated":1331200,"data_compressed_original":2441216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2359296,"data_stored":2349834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":68227072,"data_stored":68082959,"data_compressed":146679,"data_compressed_allocated":155648,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12144640,"data_stored":12084933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30617600,"data_stored":30549118,"data_compressed":108870,"data_compressed_allocated":114688,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":10027008,"data_stored":9980228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33280000,"data_stored":33118894,"data_compressed":53657,"data_compressed_allocated":57344,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":41676800,"data_stored":41655503,"data_compressed":197839,"data_compressed_allocated":208896,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":29483008,"data_stored":29461900,"data_compressed":160030,"data_compressed_allocated":167936,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:40:55.613 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-22T18:40:55.613 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:55.609+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:55.614 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:40:55.614 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [1, 3, 6, 7, 2] 2024-04-22T18:40:55.614 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 6 2024-04-22T18:40:55.687 INFO:tasks.rados.rados.0.smithi089.stdout:1682: finishing write tid 1 to smithi08937929-33 2024-04-22T18:40:55.687 INFO:tasks.rados.rados.0.smithi089.stdout:1682: finishing write tid 2 to smithi08937929-33 2024-04-22T18:40:55.687 INFO:tasks.rados.rados.0.smithi089.stdout:1683: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:40:55.708 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:55.710+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:55.755 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:55.755 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:55.755 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:55.755 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:55.765 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:55.809 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:55.811+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:55.834 INFO:tasks.rados.rados.0.smithi089.stdout:1688: finishing write tid 3 to smithi08937929-7 2024-04-22T18:40:55.834 INFO:tasks.rados.rados.0.smithi089.stdout:1688: finishing write tid 4 to smithi08937929-7 2024-04-22T18:40:55.834 INFO:tasks.rados.rados.0.smithi089.stdout:1691: finishing write tid 1 to smithi08937929-1 2024-04-22T18:40:55.834 INFO:tasks.rados.rados.0.smithi089.stdout:1695: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2024-04-22T18:40:55.842 INFO:tasks.rados.rados.0.smithi089.stdout:1695: writing smithi08937929-13 from 648836 to 1271373 tid 1 2024-04-22T18:40:55.844 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:40:55.849 INFO:tasks.rados.rados.0.smithi089.stdout:1695: writing smithi08937929-13 from 1981263 to 2474374 tid 2 2024-04-22T18:40:55.858 INFO:tasks.rados.rados.0.smithi089.stdout:1695: writing smithi08937929-13 from 3107319 to 3848836 tid 3 2024-04-22T18:40:55.859 INFO:tasks.rados.rados.0.smithi089.stdout:1683: done (13 left) 2024-04-22T18:40:55.859 INFO:tasks.rados.rados.0.smithi089.stdout:1686: done (12 left) 2024-04-22T18:40:55.859 INFO:tasks.rados.rados.0.smithi089.stdout:1689: done (11 left) 2024-04-22T18:40:55.859 INFO:tasks.rados.rados.0.smithi089.stdout:1696: set_chunk oid 41 target oid 2024-04-22T18:40:55.877 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:55.877 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:55.877 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:55.877 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:55.877 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:55.887 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:55.890 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:40:55.911 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:55.912+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:55.959 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-22T18:40:55.960+0000 7f9b4bbc8640 -1 mon.a@0(leader).osd e122 definitely_dead 0 2024-04-22T18:40:55.989 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:55.989 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:56.009 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:56.009 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:56.010 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:56.011 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:56.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:56.011 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:56.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.011 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:75 3.0 e34/19 3:18418e0f:::b7c35f165c374d5e075dde9ac2607e47ace3b0a1e2343ec6efb171380f725eea:head v 34'32, mlcod=31'29)", 2024-04-22T18:40:56.011 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:56.011 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 111.55194731, 2024-04-22T18:40:56.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104798078, 2024-04-22T18:40:56.011 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.011 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.011 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458939+0000", 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459092+0000", 2024-04-22T18:40:56.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153838 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459107+0000", 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4703e-05 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459115+0000", 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560093+0000", 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10097840499999999 2024-04-22T18:40:56.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560109+0000", 2024-04-22T18:40:56.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5631e-05 2024-04-22T18:40:56.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563706+0000", 2024-04-22T18:40:56.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035972059999999999 2024-04-22T18:40:56.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563738+0000", 2024-04-22T18:40:56.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2184000000000002e-05 2024-04-22T18:40:56.014 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.015 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.015 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.015 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:77 3.0 e34/19 3:28387c9f:::b6d7840f02d561da04e398fbf94d91b85e9066b426e48baf3b11dd4a4f6096c0:head v 34'33, mlcod=31'29)", 2024-04-22T18:40:56.015 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:56.015 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 111.55099722999999, 2024-04-22T18:40:56.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104186079, 2024-04-22T18:40:56.015 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.015 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.015 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:56.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:56.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459889+0000", 2024-04-22T18:40:56.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:56.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459963+0000", 2024-04-22T18:40:56.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3732999999999994e-05 2024-04-22T18:40:56.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459973+0000", 2024-04-22T18:40:56.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0285999999999999e-05 2024-04-22T18:40:56.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459987+0000", 2024-04-22T18:40:56.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3675999999999999e-05 2024-04-22T18:40:56.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560515+0000", 2024-04-22T18:40:56.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10052781299999999 2024-04-22T18:40:56.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560538+0000", 2024-04-22T18:40:56.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3861000000000001e-05 2024-04-22T18:40:56.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564044+0000", 2024-04-22T18:40:56.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035052009999999999 2024-04-22T18:40:56.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564076+0000", 2024-04-22T18:40:56.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2737000000000002e-05 2024-04-22T18:40:56.019 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.019 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.019 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.019 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:40:56.019 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:56.019 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 111.549715818, 2024-04-22T18:40:56.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:40:56.020 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.020 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.020 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:56.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:56.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:40:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:40:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:40:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:40:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:40:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:40:56.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:40:56.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:40:56.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:40:56.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:40:56.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:40:56.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:40:56.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:40:56.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:40:56.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:40:56.023 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.023 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.023 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.023 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:40:56.024 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:56.024 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.67304199599999, 2024-04-22T18:40:56.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:40:56.024 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.024 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.024 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:56.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:56.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:40:56.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:56.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:40:56.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:40:56.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:40:56.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:40:56.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:40:56.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:40:56.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:40:56.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:40:56.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:40:56.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:40:56.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:40:56.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:40:56.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:40:56.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:40:56.028 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.028 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.028 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.028 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:40:56.028 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:56.028 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.468238473, 2024-04-22T18:40:56.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:40:56.028 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.028 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.028 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:56.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:56.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:40:56.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:40:56.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:40:56.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:40:56.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:40:56.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:40:56.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:40:56.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:40:56.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:40:56.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:40:56.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:40:56.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:40:56.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:40:56.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:40:56.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:40:56.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:40:56.032 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.032 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.032 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.032 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:161 3.0 e76/63 3:3638311b:::f282598a146af3da1c05c8882972fba387afd249b39f6eeb6e6052a5de37fd2d:head v 76'157, mlcod=76'154)", 2024-04-22T18:40:56.032 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:56.032 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 48.619519042999997, 2024-04-22T18:40:56.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104845172, 2024-04-22T18:40:56.032 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:56.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:56.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391440+0000", 2024-04-22T18:40:56.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2045999999999994e-05 2024-04-22T18:40:56.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391448+0000", 2024-04-22T18:40:56.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8189999999999992e-06 2024-04-22T18:40:56.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391453+0000", 2024-04-22T18:40:56.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8379999999999998e-06 2024-04-22T18:40:56.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391839+0000", 2024-04-22T18:40:56.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000385325 2024-04-22T18:40:56.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391854+0000", 2024-04-22T18:40:56.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5006e-05 2024-04-22T18:40:56.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496191+0000", 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104336792 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496214+0000", 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3079999999999999e-05 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:165 3.0 e76/63 3:1e2f7d15:::93d9bab0d1f539405a95a28c02bcf2dcbdfa6f3e449e5e5e3f27f774df3c508a:head v 76'158, mlcod=76'154)", 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 48.619424860000002, 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104788752, 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.036 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.037 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:56.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:56.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391462+0000", 2024-04-22T18:40:56.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:56.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391522+0000", 2024-04-22T18:40:56.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0148000000000001e-05 2024-04-22T18:40:56.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391531+0000", 2024-04-22T18:40:56.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5709999999999998e-06 2024-04-22T18:40:56.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391536+0000", 2024-04-22T18:40:56.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6180000000000003e-06 2024-04-22T18:40:56.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392218+0000", 2024-04-22T18:40:56.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000681326 2024-04-22T18:40:56.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392237+0000", 2024-04-22T18:40:56.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9726999999999999e-05 2024-04-22T18:40:56.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496231+0000", 2024-04-22T18:40:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103993672 2024-04-22T18:40:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496251+0000", 2024-04-22T18:40:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0327999999999999e-05 2024-04-22T18:40:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:166 3.0 e76/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 76'159, mlcod=76'154)", 2024-04-22T18:40:56.041 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:56.041 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 48.619346962000002, 2024-04-22T18:40:56.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104724628, 2024-04-22T18:40:56.041 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.041 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.041 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:56.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:56.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391540+0000", 2024-04-22T18:40:56.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:56.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391627+0000", 2024-04-22T18:40:56.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7330000000000006e-05 2024-04-22T18:40:56.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391635+0000", 2024-04-22T18:40:56.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6709999999999999e-06 2024-04-22T18:40:56.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391640+0000", 2024-04-22T18:40:56.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3199999999999999e-06 2024-04-22T18:40:56.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492541+0000", 2024-04-22T18:40:56.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100900873 2024-04-22T18:40:56.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492560+0000", 2024-04-22T18:40:56.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8502999999999999e-05 2024-04-22T18:40:56.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496254+0000", 2024-04-22T18:40:56.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036945849999999998 2024-04-22T18:40:56.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496265+0000", 2024-04-22T18:40:56.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0869999999999999e-05 2024-04-22T18:40:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:167 3.0 e76/63 3:384c1ce2:::81dc8c58ebe106ae6fa9feadc06291c645d9079ad84b78214d4e6440fbc5def8:head v 76'160, mlcod=76'154)", 2024-04-22T18:40:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 48.619232087, 2024-04-22T18:40:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10487859400000001, 2024-04-22T18:40:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391707+0000", 2024-04-22T18:40:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2343999999999999e-05 2024-04-22T18:40:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391714+0000", 2024-04-22T18:40:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.596e-06 2024-04-22T18:40:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391719+0000", 2024-04-22T18:40:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9130000000000002e-06 2024-04-22T18:40:56.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493154+0000", 2024-04-22T18:40:56.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101435256 2024-04-22T18:40:56.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493177+0000", 2024-04-22T18:40:56.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2796e-05 2024-04-22T18:40:56.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496509+0000", 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033320139999999999 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496534+0000", 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5265999999999998e-05 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3450 4.d 4:bee1184f:::smithi08937929-7:head [sparse-read 0~2105792] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.083738960000002, 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103826771, 2024-04-22T18:40:56.049 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.050 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:56.050 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:56.050 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:56.050 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:56.050 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3450 2024-04-22T18:40:56.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.050 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:56.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:56.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927148+0000", 2024-04-22T18:40:56.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:56.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927151+0000", 2024-04-22T18:40:56.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7e-06 2024-04-22T18:40:56.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927156+0000", 2024-04-22T18:40:56.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6699999999999999e-06 2024-04-22T18:40:56.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927161+0000", 2024-04-22T18:40:56.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.933e-06 2024-04-22T18:40:56.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927255+0000", 2024-04-22T18:40:56.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4195000000000002e-05 2024-04-22T18:40:56.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:56.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927262+0000", 2024-04-22T18:40:56.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9480000000000002e-06 2024-04-22T18:40:56.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.030462+0000", 2024-04-22T18:40:56.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103199402 2024-04-22T18:40:56.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.030553+0000", 2024-04-22T18:40:56.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1080000000000002e-05 2024-04-22T18:40:56.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.030975+0000", 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000422426 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3451 4.d 4:bee1184f:::smithi08937929-7:head [read 0~3613505,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.083708145999999, 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20164568099999999, 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:56.054 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3451 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927183+0000", 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1110000000000001e-06 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927188+0000", 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1569999999999999e-06 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927192+0000", 2024-04-22T18:40:56.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4710000000000001e-06 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027382+0000", 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100189341 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027387+0000", 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8400000000000002e-06 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127638+0000", 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10025102499999999 2024-04-22T18:40:56.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127740+0000", 2024-04-22T18:40:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102749 2024-04-22T18:40:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.128825+0000", 2024-04-22T18:40:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001084593 2024-04-22T18:40:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.058 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:151 3.0 e114/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 114'213, mlcod=110'210)", 2024-04-22T18:40:56.059 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:40:56.059 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.493240063, 2024-04-22T18:40:56.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20074001299999999, 2024-04-22T18:40:56.059 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.059 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.059 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:40:56.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:40:56.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517640+0000", 2024-04-22T18:40:56.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999928 2024-04-22T18:40:56.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517652+0000", 2024-04-22T18:40:56.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1824000000000001e-05 2024-04-22T18:40:56.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517660+0000", 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1419999999999993e-06 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617729+0000", 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068723 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617770+0000", 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1391999999999999e-05 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617793+0000", 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2807999999999999e-05 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718361+0000", 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100567473 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718387+0000", 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6826e-05 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3862 4.d 4:bee1184f:::smithi08937929-7:head [write 3301257~278667] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.7483377930000001, 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301875958, 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3862 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:40:56.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262548+0000", 2024-04-22T18:40:56.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:56.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262622+0000", 2024-04-22T18:40:56.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3747999999999999e-05 2024-04-22T18:40:56.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262632+0000", 2024-04-22T18:40:56.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0767e-05 2024-04-22T18:40:56.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362705+0000", 2024-04-22T18:40:56.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073157 2024-04-22T18:40:56.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:40:56.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2846999999999997e-05 2024-04-22T18:40:56.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362853+0000", 2024-04-22T18:40:56.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4498000000000002e-05 2024-04-22T18:40:56.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:56.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362992+0000", 2024-04-22T18:40:56.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013879699999999999 2024-04-22T18:40:56.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:56.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.366534+0000", 2024-04-22T18:40:56.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035427969999999999 2024-04-22T18:40:56.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:56.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564353+0000", 2024-04-22T18:40:56.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.197818893 2024-04-22T18:40:56.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564385+0000", 2024-04-22T18:40:56.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1717000000000001e-05 2024-04-22T18:40:56.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564426+0000", 2024-04-22T18:40:56.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0654000000000001e-05 2024-04-22T18:40:56.068 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.068 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.068 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.068 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3863 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3579924] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:40:56.068 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:40:56.068 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.6481704009999998, 2024-04-22T18:40:56.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20206560900000001, 2024-04-22T18:40:56.068 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.068 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.068 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:56.068 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:56.068 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:56.068 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3863 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362715+0000", 2024-04-22T18:40:56.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:56.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362725+0000", 2024-04-22T18:40:56.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.006e-05 2024-04-22T18:40:56.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362732+0000", 2024-04-22T18:40:56.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6780000000000003e-06 2024-04-22T18:40:56.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362737+0000", 2024-04-22T18:40:56.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2089999999999996e-06 2024-04-22T18:40:56.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462875+0000", 2024-04-22T18:40:56.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013834100000001 2024-04-22T18:40:56.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462963+0000", 2024-04-22T18:40:56.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7792000000000002e-05 2024-04-22T18:40:56.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:56.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463127+0000", 2024-04-22T18:40:56.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000163688 2024-04-22T18:40:56.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:56.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463692+0000", 2024-04-22T18:40:56.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056466299999999997 2024-04-22T18:40:56.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:56.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564720+0000", 2024-04-22T18:40:56.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10102826199999999 2024-04-22T18:40:56.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564752+0000", 2024-04-22T18:40:56.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1936000000000001e-05 2024-04-22T18:40:56.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564783+0000", 2024-04-22T18:40:56.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0919e-05 2024-04-22T18:40:56.073 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.073 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.073 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.073 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3864 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:40:56.073 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:40:56.074 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.6481370169999998, 2024-04-22T18:40:56.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20218423599999999, 2024-04-22T18:40:56.074 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.074 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:56.074 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:56.074 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:56.074 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:56.074 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3864 2024-04-22T18:40:56.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.074 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:40:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:40:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:40:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362753+0000", 2024-04-22T18:40:56.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6989999999999999e-06 2024-04-22T18:40:56.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:40:56.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:40:56.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462871+0000", 2024-04-22T18:40:56.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100113641 2024-04-22T18:40:56.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463385+0000", 2024-04-22T18:40:56.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051307100000000003 2024-04-22T18:40:56.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:56.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463457+0000", 2024-04-22T18:40:56.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2778e-05 2024-04-22T18:40:56.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564814+0000", 2024-04-22T18:40:56.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10135696299999999 2024-04-22T18:40:56.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564868+0000", 2024-04-22T18:40:56.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3872000000000002e-05 2024-04-22T18:40:56.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564935+0000", 2024-04-22T18:40:56.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6545999999999996e-05 2024-04-22T18:40:56.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.078 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.078 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4030 4.d 4:bee1184f:::smithi08937929-7:head [write 615222~472039] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:56.078 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:40:56.079 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.0516881149999999, 2024-04-22T18:40:56.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109028217, 2024-04-22T18:40:56.079 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.079 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.079 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:56.079 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:56.079 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:56.079 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4030 2024-04-22T18:40:56.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.079 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:40:56.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:40:56.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959177+0000", 2024-04-22T18:40:56.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999776 2024-04-22T18:40:56.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959331+0000", 2024-04-22T18:40:56.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015334600000000001 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959358+0000", 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7220999999999999e-05 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959371+0000", 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3489000000000001e-05 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059523+0000", 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100152059 2024-04-22T18:40:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059588+0000", 2024-04-22T18:40:56.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4493999999999996e-05 2024-04-22T18:40:56.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:56.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059694+0000", 2024-04-22T18:40:56.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010584199999999999 2024-04-22T18:40:56.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:56.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068082+0000", 2024-04-22T18:40:56.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083880589999999998 2024-04-22T18:40:56.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068138+0000", 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6196999999999997e-05 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068185+0000", 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7318000000000003e-05 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068228+0000", 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2376999999999998e-05 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.083 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4031 4.d 4:bee1184f:::smithi08937929-7:head [write 1580652~489442] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.0463871229999999, 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20571598299999999, 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4031 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:40:56.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:40:56.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964480+0000", 2024-04-22T18:40:56.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999795 2024-04-22T18:40:56.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964628+0000", 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148109 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964651+0000", 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3215999999999999e-05 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964664+0000", 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2673e-05 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061135+0000", 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096470822999999997 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061228+0000", 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3459000000000001e-05 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061376+0000", 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148128 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:56.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068085+0000", 2024-04-22T18:40:56.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.006708632 2024-04-22T18:40:56.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:56.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170038+0000", 2024-04-22T18:40:56.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101952716 2024-04-22T18:40:56.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170173+0000", 2024-04-22T18:40:56.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013508699999999999 2024-04-22T18:40:56.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170216+0000", 2024-04-22T18:40:56.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3498999999999998e-05 2024-04-22T18:40:56.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.089 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.089 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4032 4.d 4:bee1184f:::smithi08937929-7:head [write 2768826~626587] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:56.089 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:40:56.089 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.039182016, 2024-04-22T18:40:56.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19941629799999999, 2024-04-22T18:40:56.089 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.089 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.089 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:56.089 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:56.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:56.090 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4032 2024-04-22T18:40:56.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.090 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:40:56.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:40:56.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971703+0000", 2024-04-22T18:40:56.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:56.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971996+0000", 2024-04-22T18:40:56.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029351300000000002 2024-04-22T18:40:56.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972012+0000", 2024-04-22T18:40:56.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5569e-05 2024-04-22T18:40:56.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972019+0000", 2024-04-22T18:40:56.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7950000000000008e-06 2024-04-22T18:40:56.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.063940+0000", 2024-04-22T18:40:56.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091920162 2024-04-22T18:40:56.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064021+0000", 2024-04-22T18:40:56.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1865000000000003e-05 2024-04-22T18:40:56.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:56.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064145+0000", 2024-04-22T18:40:56.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123893 2024-04-22T18:40:56.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:56.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068317+0000", 2024-04-22T18:40:56.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041720799999999999 2024-04-22T18:40:56.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:56.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171042+0000", 2024-04-22T18:40:56.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102724893 2024-04-22T18:40:56.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171085+0000", 2024-04-22T18:40:56.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2619e-05 2024-04-22T18:40:56.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171122+0000", 2024-04-22T18:40:56.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6838999999999997e-05 2024-04-22T18:40:56.094 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.095 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.095 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.095 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4033 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3815222] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:56.095 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:40:56.095 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.0388585340000001, 2024-04-22T18:40:56.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297586038, 2024-04-22T18:40:56.095 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.095 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:56.095 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:56.095 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:56.095 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:56.095 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4033 2024-04-22T18:40:56.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.096 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:40:56.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:40:56.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972028+0000", 2024-04-22T18:40:56.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:56.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972032+0000", 2024-04-22T18:40:56.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8330000000000004e-06 2024-04-22T18:40:56.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972035+0000", 2024-04-22T18:40:56.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:40:56.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972038+0000", 2024-04-22T18:40:56.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9579999999999999e-06 2024-04-22T18:40:56.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067511+0000", 2024-04-22T18:40:56.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.095472308000000006 2024-04-22T18:40:56.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067607+0000", 2024-04-22T18:40:56.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6546999999999996e-05 2024-04-22T18:40:56.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:56.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067776+0000", 2024-04-22T18:40:56.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000169202 2024-04-22T18:40:56.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:56.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068470+0000", 2024-04-22T18:40:56.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069403100000000001 2024-04-22T18:40:56.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:56.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269540+0000", 2024-04-22T18:40:56.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20106934000000001 2024-04-22T18:40:56.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269574+0000", 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4332000000000003e-05 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269615+0000", 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0988000000000001e-05 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4034 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:40:56.100 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.038846897, 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39788621000000002, 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4034 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:56.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:40:56.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:56.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:56.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972040+0000", 2024-04-22T18:40:56.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:56.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:56.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972042+0000", 2024-04-22T18:40:56.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.519e-06 2024-04-22T18:40:56.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972045+0000", 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972048+0000", 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.773e-06 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068266+0000", 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096218029999999996 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068308+0000", 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.21e-05 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369703+0000", 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301395578 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369802+0000", 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8788000000000004e-05 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:56.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369927+0000", 2024-04-22T18:40:56.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012478200000000001 2024-04-22T18:40:56.105 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.105 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.105 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.105 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:56.105 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:56.105 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:56.106 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:56.107 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:56.107 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:56.107 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:56.108 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:56.108 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:56.108 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:56.108 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:56.109 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:56.109 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:56.109 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:56.110 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:56.110 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:56.110 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:56.107+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:56.206 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:56.207+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:56.211 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:56.211 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:56.220 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:56.256 INFO:tasks.rados.rados.0.smithi089.stdout:1696: set_chunk oid 41 offset: 2033152 length: 1023488 target oid f546de5dea93dd500a2cd47bce70d61ac53a42ebe56ca4e3e28c58cc37943d5a offset: 0 snap -1 2024-04-22T18:40:56.256 INFO:tasks.rados.rados.0.smithi089.stdout:1697: read oid 3 snap -1 2024-04-22T18:40:56.256 INFO:tasks.rados.rados.0.smithi089.stdout:1697: expect (ObjNum 259 snap 0 seq_num 259) 2024-04-22T18:40:56.256 INFO:tasks.rados.rados.0.smithi089.stdout:1698: write oid 21 current snap is 0 2024-04-22T18:40:56.256 INFO:tasks.rados.rados.0.smithi089.stdout:1698: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2024-04-22T18:40:56.263 INFO:tasks.rados.rados.0.smithi089.stdout:1698: writing smithi08937929-21 from 665643 to 1363429 tid 1 2024-04-22T18:40:56.271 INFO:tasks.rados.rados.0.smithi089.stdout:1698: writing smithi08937929-21 from 2039745 to 2792867 tid 2 2024-04-22T18:40:56.277 INFO:tasks.rados.rados.0.smithi089.stdout:1698: writing smithi08937929-21 from 3534742 to 3865643 tid 3 2024-04-22T18:40:56.277 INFO:tasks.rados.rados.0.smithi089.stdout:1699: read oid 39 snap -1 2024-04-22T18:40:56.277 INFO:tasks.rados.rados.0.smithi089.stdout:1699: expect (ObjNum 340 snap 0 seq_num 340) 2024-04-22T18:40:56.277 INFO:tasks.rados.rados.0.smithi089.stdout:1700: read oid 29 snap -1 2024-04-22T18:40:56.278 INFO:tasks.rados.rados.0.smithi089.stdout:1700: expect (ObjNum 295 snap 0 seq_num 295) 2024-04-22T18:40:56.278 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:56.278 INFO:tasks.rados.rados.0.smithi089.stdout:1690: finishing set_chunk to oid 35 2024-04-22T18:40:56.278 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 115 (ObjNum 304 snap 0 seq_num 304) dirty exists 2024-04-22T18:40:56.278 INFO:tasks.rados.rados.0.smithi089.stdout:1685: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:40:56.307 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:56.308+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:56.357 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:56.357 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:56.367 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:56.408 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:56.409+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:56.509 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:56.511+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:56.511 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:56.512 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:56.516 INFO:tasks.rados.rados.0.smithi089.stdout:1687: expect (ObjNum 343 snap 0 seq_num 343) 2024-04-22T18:40:56.521 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:40:56.610 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:56.611+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:56.634 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:56.634 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:56.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:40:56.645 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:56.646+0000 7f8607f36640 -1 osd.3 122 handle_osd_map injecting map crc failure 2024-04-22T18:40:56.646 INFO:teuthology.orchestra.run.smithi089.stderr:marked out osd.6. 2024-04-22T18:40:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:1688: finishing write tid 5 to smithi08937929-7 2024-04-22T18:40:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:1688: oid 7 updating version 0 to 62 2024-04-22T18:40:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:1688: oid 7 updating version 62 to 63 2024-04-22T18:40:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:1688: oid 7 updating version 63 to 64 2024-04-22T18:40:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:1688: oid 7 updating version 64 to 65 2024-04-22T18:40:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 65 (ObjNum 346 snap 0 seq_num 346) dirty exists 2024-04-22T18:40:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:1688: left oid 7 (ObjNum 346 snap 0 seq_num 346) 2024-04-22T18:40:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:1691: finishing write tid 2 to smithi08937929-1 2024-04-22T18:40:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:1685: done (15 left) 2024-04-22T18:40:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:1687: done (14 left) 2024-04-22T18:40:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:1688: done (13 left) 2024-04-22T18:40:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:1690: done (12 left) 2024-04-22T18:40:56.678 INFO:tasks.rados.rados.0.smithi089.stdout:1701: delete oid 48 current snap is 0 2024-04-22T18:40:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:1682: finishing write tid 3 to smithi08937929-33 2024-04-22T18:40:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:1682: finishing write tid 4 to smithi08937929-33 2024-04-22T18:40:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:1682: finishing write tid 5 to smithi08937929-33 2024-04-22T18:40:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:1682: oid 33 updating version 0 to 58 2024-04-22T18:40:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:1682: oid 33 version 58 is already newer than 57 2024-04-22T18:40:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:1682: oid 33 updating version 58 to 59 2024-04-22T18:40:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:1682: oid 33 updating version 59 to 60 2024-04-22T18:40:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 60 (ObjNum 345 snap 0 seq_num 345) dirty exists 2024-04-22T18:40:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:1682: left oid 33 (ObjNum 345 snap 0 seq_num 345) 2024-04-22T18:40:56.679 INFO:tasks.rados.rados.0.smithi089.stdout:1692: expect (ObjNum 287 snap 0 seq_num 287) 2024-04-22T18:40:56.727 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:56.728+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:56.785 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:56.786 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:56.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:40:56.811 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:56.813+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:56.841 INFO:tasks.rados.rados.0.smithi089.stdout:1694: got (22) Invalid argument 2024-04-22T18:40:56.841 INFO:tasks.rados.rados.0.smithi089.stdout:1684: got (0) Success 2024-04-22T18:40:56.841 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 94 (ObjNum 301 snap 0 seq_num 301) dirty exists 2024-04-22T18:40:56.841 INFO:tasks.rados.rados.0.smithi089.stdout:1693: expect (ObjNum 331 snap 0 seq_num 331) 2024-04-22T18:40:56.912 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:56.913+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:56.937 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:56.937 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:56.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:40:57.013 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:57.015+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:57.050 INFO:tasks.rados.rados.0.smithi089.stdout:1691: finishing write tid 3 to smithi08937929-1 2024-04-22T18:40:57.050 INFO:tasks.rados.rados.0.smithi089.stdout:1691: finishing write tid 4 to smithi08937929-1 2024-04-22T18:40:57.050 INFO:tasks.rados.rados.0.smithi089.stdout:1691: finishing write tid 5 to smithi08937929-1 2024-04-22T18:40:57.050 INFO:tasks.rados.rados.0.smithi089.stdout:1691: finishing write tid 6 to smithi08937929-1 2024-04-22T18:40:57.050 INFO:tasks.rados.rados.0.smithi089.stdout:1691: oid 1 updating version 0 to 57 2024-04-22T18:40:57.050 INFO:tasks.rados.rados.0.smithi089.stdout:1691: oid 1 version 57 is already newer than 54 2024-04-22T18:40:57.050 INFO:tasks.rados.rados.0.smithi089.stdout:1691: oid 1 version 57 is already newer than 55 2024-04-22T18:40:57.050 INFO:tasks.rados.rados.0.smithi089.stdout:1691: oid 1 version 57 is already newer than 53 2024-04-22T18:40:57.050 INFO:tasks.rados.rados.0.smithi089.stdout:1691: oid 1 version 57 is already newer than 56 2024-04-22T18:40:57.050 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 57 (ObjNum 347 snap 0 seq_num 347) dirty exists 2024-04-22T18:40:57.051 INFO:tasks.rados.rados.0.smithi089.stdout:1691: left oid 1 (ObjNum 347 snap 0 seq_num 347) 2024-04-22T18:40:57.051 INFO:tasks.rados.rados.0.smithi089.stdout:1682: done (12 left) 2024-04-22T18:40:57.051 INFO:tasks.rados.rados.0.smithi089.stdout:1684: done (11 left) 2024-04-22T18:40:57.051 INFO:tasks.rados.rados.0.smithi089.stdout:1691: done (10 left) 2024-04-22T18:40:57.051 INFO:tasks.rados.rados.0.smithi089.stdout:1692: done (9 left) 2024-04-22T18:40:57.051 INFO:tasks.rados.rados.0.smithi089.stdout:1693: done (8 left) 2024-04-22T18:40:57.051 INFO:tasks.rados.rados.0.smithi089.stdout:1694: done (7 left) 2024-04-22T18:40:57.051 INFO:tasks.rados.rados.0.smithi089.stdout:1701: done (6 left) 2024-04-22T18:40:57.051 INFO:tasks.rados.rados.0.smithi089.stdout:1702: read oid 50 snap -1 2024-04-22T18:40:57.051 INFO:tasks.rados.rados.0.smithi089.stdout:1702: expect (ObjNum 189 snap 0 seq_num 189) 2024-04-22T18:40:57.051 INFO:tasks.rados.rados.0.smithi089.stdout:1703: read oid 38 snap -1 2024-04-22T18:40:57.051 INFO:tasks.rados.rados.0.smithi089.stdout:1703: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:40:57.051 INFO:tasks.rados.rados.0.smithi089.stdout:1704: write oid 48 current snap is 0 2024-04-22T18:40:57.051 INFO:tasks.rados.rados.0.smithi089.stdout:1695: finishing write tid 1 to smithi08937929-13 2024-04-22T18:40:57.051 INFO:tasks.rados.rados.0.smithi089.stdout:1695: finishing write tid 2 to smithi08937929-13 2024-04-22T18:40:57.052 INFO:tasks.rados.rados.0.smithi089.stdout:1695: finishing write tid 3 to smithi08937929-13 2024-04-22T18:40:57.052 INFO:tasks.rados.rados.0.smithi089.stdout:1704: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2024-04-22T18:40:57.056 INFO:tasks.rados.rados.0.smithi089.stdout:1704: writing smithi08937929-48 from 699257 to 1263894 tid 1 2024-04-22T18:40:57.063 INFO:tasks.rados.rados.0.smithi089.stdout:1704: writing smithi08937929-48 from 1873062 to 2513500 tid 2 2024-04-22T18:40:57.070 INFO:tasks.rados.rados.0.smithi089.stdout:1704: writing smithi08937929-48 from 3305941 to 3899257 tid 3 2024-04-22T18:40:57.070 INFO:tasks.rados.rados.0.smithi089.stdout:1705: set_chunk oid 9 target oid 2024-04-22T18:40:57.087 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:57.087 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:57.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:40:57.115 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:57.116+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:57.198 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:57.198 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:57.207 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:40:57.250 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:40:57.251+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:40:57.267 INFO:tasks.rados.rados.0.smithi089.stdout:1705: set_chunk oid 9 offset: 872448 length: 1147904 target oid 6088fd2701113dfbf040faf44dba499d13abfb2a915fcd775777d91f91676353 offset: 0 snap -1 2024-04-22T18:40:57.267 INFO:tasks.rados.rados.0.smithi089.stdout:1706: read oid 42 snap -1 2024-04-22T18:40:57.267 INFO:tasks.rados.rados.0.smithi089.stdout:1706: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:40:57.267 INFO:tasks.rados.rados.0.smithi089.stdout:1707: set_chunk oid 33 target oid 2024-04-22T18:40:57.309 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:57.309 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:57.316 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:57.317+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:57.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:40:57.417 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:57.418+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:57.459 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:57.459 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:57.468 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:40:57.518 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:57.520+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:57.609 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:57.609 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:57.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:40:57.619 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:40:57.620+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:40:57.647 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:57.648+0000 7f8605f32640 -1 osd.3 123 handle_osd_map injecting map crc failure 2024-04-22T18:40:57.720 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:57.721+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:57.721 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:57.721 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:57.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:40:57.822 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:57.823+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:57.873 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:57.873 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:57.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:40:57.912 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:40:57.955 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:57.957+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:57.984 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:57.984 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:57.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:40:58.023 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:58.024+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:58.095 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:58.095 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:58.104 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:40:58.120 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:40:58.124 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:58.125+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:58.212 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:40:58.212 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:40:58.222 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:40:58.224 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:58.226+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:40:58.270 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:40:58.271 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:40:58.276 INFO:tasks.rados.rados.0.smithi089.stdout:1707: set_chunk oid 33 offset: 1007104 length: 2410496 target oid 9981734a407db6626447d68240510009c8a94f4701c6c1e3baef8e9fb8fa502b offset: 0 snap -1 2024-04-22T18:40:58.276 INFO:tasks.rados.rados.0.smithi089.stdout:1708: read oid 36 snap -1 2024-04-22T18:40:58.276 INFO:tasks.rados.rados.0.smithi089.stdout:1708: expect (ObjNum 208 snap 0 seq_num 208) 2024-04-22T18:40:58.276 INFO:tasks.rados.rados.0.smithi089.stdout:1709: write oid 20 current snap is 0 2024-04-22T18:40:58.276 INFO:tasks.rados.rados.0.smithi089.stdout:1709: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-04-22T18:40:58.283 INFO:tasks.rados.rados.0.smithi089.stdout:1709: writing smithi08937929-20 from 716064 to 1355950 tid 1 2024-04-22T18:40:58.288 INFO:tasks.rados.rados.0.smithi089.stdout:1709: writing smithi08937929-20 from 1815191 to 2315640 tid 2 2024-04-22T18:40:58.294 INFO:tasks.rados.rados.0.smithi089.stdout:1709: writing smithi08937929-20 from 2817011 to 3343936 tid 3 2024-04-22T18:40:58.294 INFO:tasks.rados.rados.0.smithi089.stdout:1710: set_chunk oid 8 target oid 2024-04-22T18:40:58.325 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:58.327+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:58.329 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:58.329 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:58.330 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:58.338 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:40:58.382 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:40:58.382 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:40:58.398 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:40:58.398 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:40:58.414 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:40:58.415 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:40:58.426 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:58.427+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:58.434 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:40:58.434 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:40:58.448 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:58.448 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:58.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:58.448 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:58.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.448 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:58.448 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:58.448 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 123.99280517, 2024-04-22T18:40:58.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:40:58.449 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.449 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:58.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:58.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:58.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:58.449 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:40:58.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.449 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:58.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:40:58.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:40:58.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:58.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:40:58.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:40:58.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:40:58.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:40:58.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:40:58.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:40:58.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:40:58.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:58.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:40:58.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:40:58.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:58.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:40:58.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:40:58.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:58.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:40:58.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:40:58.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:40:58.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:40:58.453 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.453 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.454 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.454 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:58.454 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:58.454 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 115.29974792199999, 2024-04-22T18:40:58.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:40:58.454 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.454 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:58.454 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:58.454 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:58.454 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:58.454 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:40:58.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:40:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:40:58.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:58.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:40:58.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:40:58.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:40:58.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:40:58.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:40:58.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:40:58.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:40:58.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:40:58.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:40:58.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:40:58.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:58.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:40:58.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:40:58.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:58.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:40:58.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:40:58.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:58.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:40:58.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:40:58.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:58.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:40:58.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:40:58.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:40:58.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:40:58.459 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.459 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.459 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.459 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:40:58.460 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:58.460 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 115.299729446, 2024-04-22T18:40:58.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:40:58.460 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.460 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:58.460 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:58.460 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:58.460 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:58.460 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:40:58.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.460 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:58.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:58.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:40:58.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:58.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:40:58.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:40:58.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:40:58.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:40:58.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:40:58.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:40:58.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:40:58.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:40:58.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:58.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:40:58.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:40:58.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:40:58.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:40:58.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:40:58.464 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:58.465 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 87.536681134000005, 2024-04-22T18:40:58.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:40:58.465 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.465 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:58.465 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:58.465 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:58.465 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:58.465 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:40:58.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.465 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:58.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:40:58.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:40:58.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:58.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:40:58.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:40:58.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:40:58.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:40:58.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:40:58.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:40:58.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:40:58.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:40:58.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:40:58.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:40:58.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:58.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:40:58.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:40:58.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:58.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:40:58.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:40:58.470 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.470 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.470 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.470 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:58.470 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:58.470 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.007521808000007, 2024-04-22T18:40:58.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:40:58.470 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.470 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:58.470 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:58.470 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:58.470 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:58.471 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:40:58.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.471 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:58.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:58.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:40:58.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:58.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:40:58.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:40:58.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:40:58.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:40:58.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:40:58.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:40:58.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:40:58.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:40:58.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:58.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:40:58.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:40:58.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:40:58.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:40:58.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:40:58.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:40:58.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:40:58.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:40:58.475 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.475 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.475 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.475 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:58.475 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:58.475 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 81.829814412999994, 2024-04-22T18:40:58.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:40:58.475 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.475 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:58.475 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:58.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:40:58.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:40:58.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:58.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:40:58.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:40:58.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:40:58.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:40:58.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:40:58.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:40:58.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:40:58.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:40:58.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:40:58.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:40:58.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:58.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:40:58.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:40:58.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:58.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:40:58.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:40:58.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:58.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:40:58.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:40:58.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:58.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:40:58.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:40:58.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:40:58.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:40:58.480 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.480 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.480 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.480 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:58.481 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:58.481 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 81.829746141000001, 2024-04-22T18:40:58.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:40:58.481 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.481 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:58.481 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:58.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:40:58.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:40:58.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:58.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:40:58.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:40:58.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:40:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:40:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:40:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:40:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:40:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:40:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:40:58.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:40:58.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:58.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:40:58.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:40:58.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:58.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:40:58.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:40:58.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:58.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:40:58.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:40:58.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:58.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:40:58.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:40:58.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:40:58.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:40:58.485 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.486 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.486 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.486 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:40:58.486 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:58.486 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 81.829578299999994, 2024-04-22T18:40:58.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:40:58.486 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.486 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:58.486 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:58.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:58.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:40:58.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:58.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:40:58.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:40:58.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:40:58.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:40:58.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:40:58.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:40:58.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:40:58.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:40:58.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:40:58.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:40:58.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:58.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:40:58.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:40:58.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:58.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:40:58.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:40:58.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:58.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:40:58.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:40:58.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:58.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:40:58.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:40:58.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:40:58.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:40:58.491 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.491 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.491 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.491 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:58.491 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:58.491 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 77.864252824999994, 2024-04-22T18:40:58.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:40:58.492 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.492 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:58.492 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:58.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:40:58.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:40:58.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:40:58.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:40:58.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:40:58.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:40:58.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:40:58.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:40:58.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:58.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:40:58.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:40:58.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:58.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:40:58.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:40:58.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:58.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:40:58.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:40:58.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:58.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:40:58.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:40:58.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:40:58.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:40:58.496 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.496 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.497 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.497 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:40:58.497 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:58.497 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 77.764100638000002, 2024-04-22T18:40:58.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:40:58.497 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.497 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:58.497 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:58.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:40:58.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:40:58.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:58.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:40:58.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:40:58.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:40:58.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:40:58.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:40:58.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:40:58.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:40:58.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:40:58.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:40:58.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:40:58.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:40:58.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:40:58.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:40:58.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:58.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:40:58.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:40:58.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:58.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:40:58.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:40:58.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:58.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:40:58.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:40:58.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:40:58.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:40:58.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.502 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.502 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:40:58.502 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:58.502 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.055236117999996, 2024-04-22T18:40:58.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:40:58.503 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.503 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:58.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:58.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:58.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:58.503 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:40:58.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.503 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:58.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:40:58.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:40:58.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:58.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:40:58.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:40:58.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:40:58.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:40:58.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:40:58.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:40:58.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:40:58.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:40:58.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:58.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:40:58.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:40:58.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:40:58.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:40:58.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:40:58.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:40:58.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:40:58.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:40:58.507 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.507 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.507 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.507 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:40:58.508 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:58.508 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.159197745999997, 2024-04-22T18:40:58.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:40:58.508 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.508 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:58.508 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:58.508 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:58.508 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:58.508 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:40:58.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.508 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:58.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:58.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:40:58.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:58.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:40:58.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:40:58.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:40:58.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:40:58.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:40:58.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:40:58.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:40:58.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:40:58.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:40:58.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:40:58.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:40:58.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:40:58.512 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.512 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.512 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.512 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3330 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-04-22T18:40:58.512 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:40:58.512 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.42411946, 2024-04-22T18:40:58.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43151673600000001, 2024-04-22T18:40:58.512 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.512 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:58.512 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:58.512 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:58.513 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:58.513 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3330 2024-04-22T18:40:58.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.513 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:40:58.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:40:58.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024041+0000", 2024-04-22T18:40:58.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:58.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024045+0000", 2024-04-22T18:40:58.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3950000000000001e-06 2024-04-22T18:40:58.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024051+0000", 2024-04-22T18:40:58.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0260000000000004e-06 2024-04-22T18:40:58.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024056+0000", 2024-04-22T18:40:58.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4489999999999998e-06 2024-04-22T18:40:58.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028383+0000", 2024-04-22T18:40:58.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004326557 2024-04-22T18:40:58.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028471+0000", 2024-04-22T18:40:58.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8387999999999995e-05 2024-04-22T18:40:58.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.455559+0000", 2024-04-22T18:40:58.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.42708770699999998 2024-04-22T18:40:58.516 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.516 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.516 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.517 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3377 4.6 4:6700eab5:::smithi08937929-32:head [read 1812992~307712] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-04-22T18:40:58.517 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:40:58.517 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.011794104, 2024-04-22T18:40:58.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.52001745799999999, 2024-04-22T18:40:58.517 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.517 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:58.517 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:58.517 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:58.517 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:58.517 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3377 2024-04-22T18:40:58.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.517 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:40:58.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:40:58.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436364+0000", 2024-04-22T18:40:58.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:40:58.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436376+0000", 2024-04-22T18:40:58.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2121e-05 2024-04-22T18:40:58.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436394+0000", 2024-04-22T18:40:58.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8094e-05 2024-04-22T18:40:58.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436405+0000", 2024-04-22T18:40:58.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0189e-05 2024-04-22T18:40:58.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436471+0000", 2024-04-22T18:40:58.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6817999999999996e-05 2024-04-22T18:40:58.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:58.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436481+0000", 2024-04-22T18:40:58.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2720000000000003e-06 2024-04-22T18:40:58.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956084+0000", 2024-04-22T18:40:58.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51960320800000004 2024-04-22T18:40:58.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956232+0000", 2024-04-22T18:40:58.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014778600000000001 2024-04-22T18:40:58.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956385+0000", 2024-04-22T18:40:58.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153394 2024-04-22T18:40:58.521 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.521 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.521 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.522 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:131 3.1 3:a0a19997:::ee8ca23a7050e771c77dff761e6ea5a11ed57796f4e8a17cbb5804a7fdc937b9:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:40:58.522 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:40:58.522 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.118924507999999, 2024-04-22T18:40:58.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51449564999999997, 2024-04-22T18:40:58.522 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.522 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:58.522 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:40:58.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:40:58.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329235+0000", 2024-04-22T18:40:58.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:58.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329244+0000", 2024-04-22T18:40:58.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6819999999999992e-06 2024-04-22T18:40:58.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329259+0000", 2024-04-22T18:40:58.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698000000000001e-05 2024-04-22T18:40:58.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329265+0000", 2024-04-22T18:40:58.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1510000000000001e-06 2024-04-22T18:40:58.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329311+0000", 2024-04-22T18:40:58.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6096e-05 2024-04-22T18:40:58.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:58.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.332269+0000", 2024-04-22T18:40:58.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029581910000000002 2024-04-22T18:40:58.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630603+0000", 2024-04-22T18:40:58.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29833418 2024-04-22T18:40:58.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630685+0000", 2024-04-22T18:40:58.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1217999999999994e-05 2024-04-22T18:40:58.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:58.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630813+0000", 2024-04-22T18:40:58.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128706 2024-04-22T18:40:58.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:58.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.631447+0000", 2024-04-22T18:40:58.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063427000000000004 2024-04-22T18:40:58.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843586+0000", 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21213873 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843623+0000", 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6848999999999999e-05 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843733+0000", 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010982200000000001 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.527 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.528 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3441 4.b 4:d3068ee3:::smithi08937929-29:head [write 558065~619869] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:58.528 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:40:58.528 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.530448447000001, 2024-04-22T18:40:58.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50272020399999995, 2024-04-22T18:40:58.528 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.528 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:58.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:58.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:58.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:58.528 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3441 2024-04-22T18:40:58.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.528 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:40:58.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:40:58.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917711+0000", 2024-04-22T18:40:58.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:58.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917838+0000", 2024-04-22T18:40:58.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012734899999999999 2024-04-22T18:40:58.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917854+0000", 2024-04-22T18:40:58.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6130999999999999e-05 2024-04-22T18:40:58.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917863+0000", 2024-04-22T18:40:58.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5969999999999992e-06 2024-04-22T18:40:58.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992845+0000", 2024-04-22T18:40:58.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.074982193000000003 2024-04-22T18:40:58.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992919+0000", 2024-04-22T18:40:58.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:40:58.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:58.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.993019+0000", 2024-04-22T18:40:58.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9718999999999999e-05 2024-04-22T18:40:58.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:58.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095554+0000", 2024-04-22T18:40:58.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102535193 2024-04-22T18:40:58.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:58.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420183+0000", 2024-04-22T18:40:58.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32462853600000002 2024-04-22T18:40:58.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:58.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420401+0000", 2024-04-22T18:40:58.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021881400000000001 2024-04-22T18:40:58.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420433+0000", 2024-04-22T18:40:58.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079000000000002e-05 2024-04-22T18:40:58.533 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.533 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.533 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.533 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3442 4.b 4:d3068ee3:::smithi08937929-29:head [write 1579835~486765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:58.533 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:40:58.534 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.525761736, 2024-04-22T18:40:58.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49822799099999998, 2024-04-22T18:40:58.534 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.534 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:58.534 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:58.534 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:58.534 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:58.534 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3442 2024-04-22T18:40:58.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.534 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:40:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:40:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922399+0000", 2024-04-22T18:40:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922502+0000", 2024-04-22T18:40:58.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102784 2024-04-22T18:40:58.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922513+0000", 2024-04-22T18:40:58.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1051e-05 2024-04-22T18:40:58.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922518+0000", 2024-04-22T18:40:58.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8529999999999997e-06 2024-04-22T18:40:58.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093158+0000", 2024-04-22T18:40:58.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.17063958800000001 2024-04-22T18:40:58.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093230+0000", 2024-04-22T18:40:58.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1803000000000004e-05 2024-04-22T18:40:58.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:58.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093326+0000", 2024-04-22T18:40:58.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6454999999999997e-05 2024-04-22T18:40:58.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:58.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095557+0000", 2024-04-22T18:40:58.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022306399999999999 2024-04-22T18:40:58.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:58.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420484+0000", 2024-04-22T18:40:58.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32492717700000001 2024-04-22T18:40:58.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:58.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420606+0000", 2024-04-22T18:40:58.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012225800000000001 2024-04-22T18:40:58.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420628+0000", 2024-04-22T18:40:58.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1596000000000001e-05 2024-04-22T18:40:58.539 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.539 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.539 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.539 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3443 4.b 4:d3068ee3:::smithi08937929-29:head [write 2468371~489694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:58.539 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:40:58.539 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.521159689000001, 2024-04-22T18:40:58.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49372487199999998, 2024-04-22T18:40:58.539 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.539 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:58.540 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:58.540 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:58.540 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:58.540 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3443 2024-04-22T18:40:58.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.540 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:40:58.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:40:58.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927001+0000", 2024-04-22T18:40:58.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:58.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927117+0000", 2024-04-22T18:40:58.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011637 2024-04-22T18:40:58.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927128+0000", 2024-04-22T18:40:58.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0696000000000001e-05 2024-04-22T18:40:58.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927134+0000", 2024-04-22T18:40:58.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8640000000000001e-06 2024-04-22T18:40:58.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094288+0000", 2024-04-22T18:40:58.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16715412900000001 2024-04-22T18:40:58.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094359+0000", 2024-04-22T18:40:58.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1639999999999998e-05 2024-04-22T18:40:58.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:58.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094451+0000", 2024-04-22T18:40:58.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1252e-05 2024-04-22T18:40:58.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:58.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095560+0000", 2024-04-22T18:40:58.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011095530000000001 2024-04-22T18:40:58.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:58.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420661+0000", 2024-04-22T18:40:58.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32510105500000003 2024-04-22T18:40:58.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:58.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420710+0000", 2024-04-22T18:40:58.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9018999999999999e-05 2024-04-22T18:40:58.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420727+0000", 2024-04-22T18:40:58.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6722e-05 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3444 4.b 4:d3068ee3:::smithi08937929-29:head [setxattr _header (56) in=7b,truncate 2958065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.521023971999998, 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49366935299999998, 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3444 2024-04-22T18:40:58.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.546 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:40:58.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:40:58.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927137+0000", 2024-04-22T18:40:58.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:58.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927141+0000", 2024-04-22T18:40:58.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3270000000000001e-06 2024-04-22T18:40:58.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927144+0000", 2024-04-22T18:40:58.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9730000000000002e-06 2024-04-22T18:40:58.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927147+0000", 2024-04-22T18:40:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.959e-06 2024-04-22T18:40:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095118+0000", 2024-04-22T18:40:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16797115500000001 2024-04-22T18:40:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095169+0000", 2024-04-22T18:40:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1316000000000001e-05 2024-04-22T18:40:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:58.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095260+0000", 2024-04-22T18:40:58.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1184e-05 2024-04-22T18:40:58.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:58.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095999+0000", 2024-04-22T18:40:58.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073868200000000005 2024-04-22T18:40:58.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:58.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420757+0000", 2024-04-22T18:40:58.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32475783499999999 2024-04-22T18:40:58.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:58.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420781+0000", 2024-04-22T18:40:58.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4022000000000001e-05 2024-04-22T18:40:58.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420807+0000", 2024-04-22T18:40:58.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6366000000000001e-05 2024-04-22T18:40:58.550 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.550 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.550 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.550 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3445 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:58.551 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:58.551 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.521012878000001, 2024-04-22T18:40:58.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.493780049, 2024-04-22T18:40:58.551 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.551 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:58.551 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:58.551 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:58.551 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:58.551 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3445 2024-04-22T18:40:58.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.551 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:58.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:58.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:40:58.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:58.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927150+0000", 2024-04-22T18:40:58.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.542e-06 2024-04-22T18:40:58.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927153+0000", 2024-04-22T18:40:58.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0520000000000002e-06 2024-04-22T18:40:58.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027218+0000", 2024-04-22T18:40:58.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100064604 2024-04-22T18:40:58.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095501+0000", 2024-04-22T18:40:58.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.068283431000000006 2024-04-22T18:40:58.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:58.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095545+0000", 2024-04-22T18:40:58.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4295999999999997e-05 2024-04-22T18:40:58.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420833+0000", 2024-04-22T18:40:58.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32528718400000001 2024-04-22T18:40:58.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:58.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420877+0000", 2024-04-22T18:40:58.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4403999999999997e-05 2024-04-22T18:40:58.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:58.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420929+0000", 2024-04-22T18:40:58.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1848999999999998e-05 2024-04-22T18:40:58.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.555 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.555 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.555 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:58.556 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:40:58.556 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:40:58.557 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:40:58.558 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:40:58.558 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:40:58.558 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:40:58.559 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:40:58.559 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:58.558+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:58.581 INFO:tasks.rados.rados.0.smithi089.stdout:1710: set_chunk oid 8 offset: 1234944 length: 669696 target oid 306c652ca18b0f20236b7817c007088469b25e4989bd8a337ad52b8f778776ad offset: 0 snap -1 2024-04-22T18:40:58.581 INFO:tasks.rados.rados.0.smithi089.stdout:1697: expect (ObjNum 259 snap 0 seq_num 259) 2024-04-22T18:40:58.650 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:40:58.657 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:40:58.658+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:40:58.690 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4107 4.e 4:744777ef:::smithi08937929-8:head [assert-version v33,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e124)", 2024-04-22T18:40:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:58.583244+0000", 2024-04-22T18:40:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.10920270999999999, 2024-04-22T18:40:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10923066200000001, 2024-04-22T18:40:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for degraded object", 2024-04-22T18:40:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:58.691 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4107 2024-04-22T18:40:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.583244+0000", 2024-04-22T18:40:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.583244+0000", 2024-04-22T18:40:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.583242+0000", 2024-04-22T18:40:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.583251+0000", 2024-04-22T18:40:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8980000000000002e-06 2024-04-22T18:40:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.583262+0000", 2024-04-22T18:40:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0409e-05 2024-04-22T18:40:58.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.583268+0000", 2024-04-22T18:40:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5629999999999997e-06 2024-04-22T18:40:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.583615+0000", 2024-04-22T18:40:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00034738999999999999 2024-04-22T18:40:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:40:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.583678+0000", 2024-04-22T18:40:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2986999999999999e-05 2024-04-22T18:40:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPushReply(4.a 124/123 [PushReplyOp(4:537c89f8:::smithi08937929-47:head)])", 2024-04-22T18:40:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:58.594207+0000", 2024-04-22T18:40:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.098239736999999994, 2024-04-22T18:40:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098336178999999996, 2024-04-22T18:40:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.695 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "no flag points reached", 2024-04-22T18:40:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.594207+0000", 2024-04-22T18:40:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.594207+0000", 2024-04-22T18:40:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.594206+0000", 2024-04-22T18:40:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.594214+0000", 2024-04-22T18:40:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6660000000000008e-06 2024-04-22T18:40:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.594226+0000", 2024-04-22T18:40:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1935e-05 2024-04-22T18:40:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.697 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(4.c 124/123 [PushOp(4:3227431c:::smithi08937929-35:head, version: 121'119, data_included: [709328~613463,1989820~435113], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:3227431c:::smithi08937929-35:head@121'119, size: 3109328, copy_subset: [0~3109328], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2424933, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-22T18:40:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:58.683517+0000", 2024-04-22T18:40:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0089304469999999993, 2024-04-22T18:40:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0090876970000000005, 2024-04-22T18:40:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.683517+0000", 2024-04-22T18:40:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.683517+0000", 2024-04-22T18:40:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.683515+0000", 2024-04-22T18:40:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.684078+0000", 2024-04-22T18:40:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056379800000000001 2024-04-22T18:40:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.684094+0000", 2024-04-22T18:40:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5311e-05 2024-04-22T18:40:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.684108+0000", 2024-04-22T18:40:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3985000000000001e-05 2024-04-22T18:40:58.700 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.701 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:58.701 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.701 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:58.701 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:58.701 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 3 2024-04-22T18:40:58.701 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:58.702 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:40:58.754 INFO:tasks.rados.rados.0.smithi089.stdout:1698: finishing write tid 1 to smithi08937929-21 2024-04-22T18:40:58.754 INFO:tasks.rados.rados.0.smithi089.stdout:1698: finishing write tid 2 to smithi08937929-21 2024-04-22T18:40:58.754 INFO:tasks.rados.rados.0.smithi089.stdout:1711: read oid 24 snap -1 2024-04-22T18:40:58.754 INFO:tasks.rados.rados.0.smithi089.stdout:1711: expect deleted 2024-04-22T18:40:58.754 INFO:tasks.rados.rados.0.smithi089.stdout:1697: done (15 left) 2024-04-22T18:40:58.754 INFO:tasks.rados.rados.0.smithi089.stdout:1712: set_chunk oid 47 target oid 2024-04-22T18:40:58.757 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:58.758+0000 7f1e2dd22640 -1 osd.2 124 handle_osd_map injecting map crc failure 2024-04-22T18:40:58.758 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:58.759+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:58.847 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:58.847 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:58.847 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:58.847 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:58.847 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:58.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:40:58.859 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:58.859+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:58.960 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:58.961+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:59.002 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:59.002 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:59.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:59.003 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:59.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.003 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:9 4.f 4:f2c48a0e:::smithi08937929-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:59.003 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:59.003 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 124.885841015, 2024-04-22T18:40:59.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49290714600000002, 2024-04-22T18:40:59.003 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.003 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.003 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.003 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.003 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.003 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 9 2024-04-22T18:40:59.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.004 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:59.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116774+0000", 2024-04-22T18:40:59.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.116773+0000", 2024-04-22T18:40:59.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:59.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117024+0000", 2024-04-22T18:40:59.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025191299999999999 2024-04-22T18:40:59.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117036+0000", 2024-04-22T18:40:59.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1595e-05 2024-04-22T18:40:59.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.117046+0000", 2024-04-22T18:40:59.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0456e-05 2024-04-22T18:40:59.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414324+0000", 2024-04-22T18:40:59.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297277603 2024-04-22T18:40:59.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414404+0000", 2024-04-22T18:40:59.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9738000000000001e-05 2024-04-22T18:40:59.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:59.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.414494+0000", 2024-04-22T18:40:59.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9733000000000003e-05 2024-04-22T18:40:59.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518194+0000", 2024-04-22T18:40:59.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10370061 2024-04-22T18:40:59.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609618+0000", 2024-04-22T18:40:59.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091423360999999995 2024-04-22T18:40:59.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609655+0000", 2024-04-22T18:40:59.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7580000000000003e-05 2024-04-22T18:40:59.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609681+0000", 2024-04-22T18:40:59.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5908e-05 2024-04-22T18:40:59.008 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.008 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.008 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.008 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:10 4.f 4:f2c48a0e:::smithi08937929-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:59.009 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:59.009 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 124.879988755, 2024-04-22T18:40:59.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48712681400000002, 2024-04-22T18:40:59.009 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.009 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.009 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.009 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.009 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.009 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 10 2024-04-22T18:40:59.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.009 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:59.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122626+0000", 2024-04-22T18:40:59.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122625+0000", 2024-04-22T18:40:59.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:59.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122828+0000", 2024-04-22T18:40:59.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020359499999999999 2024-04-22T18:40:59.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122840+0000", 2024-04-22T18:40:59.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1239e-05 2024-04-22T18:40:59.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122845+0000", 2024-04-22T18:40:59.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2719999999999997e-06 2024-04-22T18:40:59.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415105+0000", 2024-04-22T18:40:59.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29225972099999997 2024-04-22T18:40:59.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415169+0000", 2024-04-22T18:40:59.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4170000000000004e-05 2024-04-22T18:40:59.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:59.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415257+0000", 2024-04-22T18:40:59.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8331999999999994e-05 2024-04-22T18:40:59.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518199+0000", 2024-04-22T18:40:59.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10294174 2024-04-22T18:40:59.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609707+0000", 2024-04-22T18:40:59.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091508408999999999 2024-04-22T18:40:59.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609729+0000", 2024-04-22T18:40:59.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1917e-05 2024-04-22T18:40:59.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609753+0000", 2024-04-22T18:40:59.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3776000000000001e-05 2024-04-22T18:40:59.014 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.014 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.014 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.014 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:11 4.f 4:f2c48a0e:::smithi08937929-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:59.014 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:59.014 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 124.879765382, 2024-04-22T18:40:59.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49452214500000002, 2024-04-22T18:40:59.014 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.014 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.014 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.014 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.015 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.015 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 11 2024-04-22T18:40:59.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.015 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:59.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122850+0000", 2024-04-22T18:40:59.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122849+0000", 2024-04-22T18:40:59.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122853+0000", 2024-04-22T18:40:59.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9029999999999997e-06 2024-04-22T18:40:59.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122856+0000", 2024-04-22T18:40:59.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.01e-06 2024-04-22T18:40:59.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122859+0000", 2024-04-22T18:40:59.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:59.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415755+0000", 2024-04-22T18:40:59.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29289606800000001 2024-04-22T18:40:59.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415808+0000", 2024-04-22T18:40:59.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3415999999999998e-05 2024-04-22T18:40:59.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:59.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.415888+0000", 2024-04-22T18:40:59.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9061999999999996e-05 2024-04-22T18:40:59.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518203+0000", 2024-04-22T18:40:59.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102315533 2024-04-22T18:40:59.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617164+0000", 2024-04-22T18:40:59.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098960548999999995 2024-04-22T18:40:59.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617241+0000", 2024-04-22T18:40:59.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7292000000000005e-05 2024-04-22T18:40:59.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617372+0000", 2024-04-22T18:40:59.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013075399999999999 2024-04-22T18:40:59.020 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.020 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.020 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.021 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:12 4.f 4:f2c48a0e:::smithi08937929-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e24)", 2024-04-22T18:40:59.021 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:59.021 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 124.879753518, 2024-04-22T18:40:59.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49473910999999998, 2024-04-22T18:40:59.022 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.022 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:59.022 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.022 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.022 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.022 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 12 2024-04-22T18:40:59.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.022 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:59.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:59.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122861+0000", 2024-04-22T18:40:59.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122863+0000", 2024-04-22T18:40:59.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.863e-06 2024-04-22T18:40:59.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122866+0000", 2024-04-22T18:40:59.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:40:59.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.122869+0000", 2024-04-22T18:40:59.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5950000000000001e-06 2024-04-22T18:40:59.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416008+0000", 2024-04-22T18:40:59.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29313975199999998 2024-04-22T18:40:59.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:59.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.416048+0000", 2024-04-22T18:40:59.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9888000000000002e-05 2024-04-22T18:40:59.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617447+0000", 2024-04-22T18:40:59.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20139881400000001 2024-04-22T18:40:59.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617537+0000", 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9606999999999998e-05 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.617600+0000", 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3884e-05 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:33 4.f 4:f0ed3d49:::smithi08937929-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 124.81581331700001, 2024-04-22T18:40:59.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62359692099999997, 2024-04-22T18:40:59.027 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.027 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.027 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.027 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.027 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.027 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 33 2024-04-22T18:40:59.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.027 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:59.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:40:59.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186799+0000", 2024-04-22T18:40:59.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:59.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187103+0000", 2024-04-22T18:40:59.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030354900000000002 2024-04-22T18:40:59.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187117+0000", 2024-04-22T18:40:59.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4578e-05 2024-04-22T18:40:59.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187129+0000", 2024-04-22T18:40:59.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1421e-05 2024-04-22T18:40:59.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516134+0000", 2024-04-22T18:40:59.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32900553900000001 2024-04-22T18:40:59.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516215+0000", 2024-04-22T18:40:59.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0537999999999994e-05 2024-04-22T18:40:59.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:59.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516302+0000", 2024-04-22T18:40:59.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6601000000000007e-05 2024-04-22T18:40:59.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518207+0000", 2024-04-22T18:40:59.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019058370000000001 2024-04-22T18:40:59.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810302+0000", 2024-04-22T18:40:59.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29209509700000003 2024-04-22T18:40:59.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810348+0000", 2024-04-22T18:40:59.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5992999999999998e-05 2024-04-22T18:40:59.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810398+0000", 2024-04-22T18:40:59.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0046000000000002e-05 2024-04-22T18:40:59.031 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.032 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.032 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.032 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:34 4.f 4:f0ed3d49:::smithi08937929-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:59.032 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:59.032 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 124.811211305, 2024-04-22T18:40:59.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.61954655800000002, 2024-04-22T18:40:59.032 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.032 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.032 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.032 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.032 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191404+0000", 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191401+0000", 2024-04-22T18:40:59.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:59.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191503+0000", 2024-04-22T18:40:59.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010190400000000001 2024-04-22T18:40:59.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191517+0000", 2024-04-22T18:40:59.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3267e-05 2024-04-22T18:40:59.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191523+0000", 2024-04-22T18:40:59.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-04-22T18:40:59.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517062+0000", 2024-04-22T18:40:59.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32553894300000003 2024-04-22T18:40:59.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517141+0000", 2024-04-22T18:40:59.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9667999999999997e-05 2024-04-22T18:40:59.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:59.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517267+0000", 2024-04-22T18:40:59.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000125494 2024-04-22T18:40:59.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518213+0000", 2024-04-22T18:40:59.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000946275 2024-04-22T18:40:59.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810901+0000", 2024-04-22T18:40:59.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29268821699999997 2024-04-22T18:40:59.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810929+0000", 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7645999999999999e-05 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810950+0000", 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1333999999999999e-05 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 124.811088337, 2024-04-22T18:40:59.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:40:59.038 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.038 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.038 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.038 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.038 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.038 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:40:59.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.038 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:59.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:40:59.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:40:59.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:40:59.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:40:59.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:40:59.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:40:59.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:40:59.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:40:59.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:40:59.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:40:59.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:40:59.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:40:59.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:59.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:40:59.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:40:59.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:40:59.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:40:59.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:40:59.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:40:59.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:40:59.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:40:59.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 124.811077262, 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.043 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:40:59.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:40:59.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:40:59.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:40:59.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:59.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:40:59.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:40:59.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:40:59.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.047 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:40:59.048 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:59.048 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.209707768, 2024-04-22T18:40:59.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:40:59.048 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.048 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.048 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.048 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.048 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.048 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:40:59.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.048 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:59.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:40:59.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:40:59.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:59.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:40:59.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:40:59.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:40:59.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:40:59.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:40:59.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:40:59.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:40:59.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:40:59.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:40:59.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:40:59.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:40:59.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:40:59.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:40:59.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:7 3.0 3:1554f608:::d614c1f2f74bb29087660bd6e888eafe46b1a7ed3cbdede85b36de771c7f0d66:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.830430241999998, 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.48116529899999999, 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:59.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172185+0000", 2024-04-22T18:40:59.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172184+0000", 2024-04-22T18:40:59.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172349+0000", 2024-04-22T18:40:59.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000165074 2024-04-22T18:40:59.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172355+0000", 2024-04-22T18:40:59.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8900000000000004e-06 2024-04-22T18:40:59.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172359+0000", 2024-04-22T18:40:59.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.569e-06 2024-04-22T18:40:59.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271561+0000", 2024-04-22T18:40:59.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099202569000000004 2024-04-22T18:40:59.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271659+0000", 2024-04-22T18:40:59.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8171000000000006e-05 2024-04-22T18:40:59.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-04-22T18:40:59.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.271775+0000", 2024-04-22T18:40:59.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000116137 2024-04-22T18:40:59.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.272327+0000", 2024-04-22T18:40:59.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00055106999999999997 2024-04-22T18:40:59.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653290+0000", 2024-04-22T18:40:59.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.38096334500000001 2024-04-22T18:40:59.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653326+0000", 2024-04-22T18:40:59.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6168000000000003e-05 2024-04-22T18:40:59.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.653350+0000", 2024-04-22T18:40:59.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3921999999999999e-05 2024-04-22T18:40:59.058 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.058 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.058 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.058 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3193 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:59.058 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:40:59.058 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.959868972999999, 2024-04-22T18:40:59.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69912102700000001, 2024-04-22T18:40:59.058 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.059 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:59.059 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.059 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.059 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.059 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3193 2024-04-22T18:40:59.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.059 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:40:59.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:40:59.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042744+0000", 2024-04-22T18:40:59.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:40:59.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043013+0000", 2024-04-22T18:40:59.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026906 2024-04-22T18:40:59.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043028+0000", 2024-04-22T18:40:59.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5359999999999999e-05 2024-04-22T18:40:59.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043036+0000", 2024-04-22T18:40:59.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8420000000000005e-06 2024-04-22T18:40:59.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043081+0000", 2024-04-22T18:40:59.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5019999999999999e-05 2024-04-22T18:40:59.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:59.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043087+0000", 2024-04-22T18:40:59.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5420000000000002e-06 2024-04-22T18:40:59.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741856+0000", 2024-04-22T18:40:59.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69876864500000002 2024-04-22T18:40:59.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741867+0000", 2024-04-22T18:40:59.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.084e-05 2024-04-22T18:40:59.063 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.063 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.063 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.063 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3194 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:59.063 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:40:59.063 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.952618993000002, 2024-04-22T18:40:59.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69190137100000004, 2024-04-22T18:40:59.063 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.063 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:59.063 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.063 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.064 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.064 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3194 2024-04-22T18:40:59.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.064 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:40:59.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:40:59.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049991+0000", 2024-04-22T18:40:59.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999957 2024-04-22T18:40:59.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050284+0000", 2024-04-22T18:40:59.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029296699999999999 2024-04-22T18:40:59.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050304+0000", 2024-04-22T18:40:59.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.969e-05 2024-04-22T18:40:59.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050314+0000", 2024-04-22T18:40:59.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0268999999999999e-05 2024-04-22T18:40:59.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150531+0000", 2024-04-22T18:40:59.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100216868 2024-04-22T18:40:59.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:59.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150545+0000", 2024-04-22T18:40:59.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4063000000000001e-05 2024-04-22T18:40:59.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741890+0000", 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59134479100000004 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741897+0000", 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1189999999999999e-06 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3195 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:59.067 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:59.068 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.952286553, 2024-04-22T18:40:59.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69159474200000004, 2024-04-22T18:40:59.068 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.068 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:59.068 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.068 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.068 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.068 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3195 2024-04-22T18:40:59.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.068 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:59.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:59.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:40:59.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:59.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050333+0000", 2024-04-22T18:40:59.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0289999999999996e-06 2024-04-22T18:40:59.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050341+0000", 2024-04-22T18:40:59.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0050000000000004e-06 2024-04-22T18:40:59.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050346+0000", 2024-04-22T18:40:59.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8329999999999999e-06 2024-04-22T18:40:59.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150628+0000", 2024-04-22T18:40:59.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100281548 2024-04-22T18:40:59.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:59.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150672+0000", 2024-04-22T18:40:59.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3875999999999998e-05 2024-04-22T18:40:59.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741916+0000", 2024-04-22T18:40:59.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59124448799999996 2024-04-22T18:40:59.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741923+0000", 2024-04-22T18:40:59.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7689999999999999e-06 2024-04-22T18:40:59.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.072 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.072 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3196 4.11 4.9f8e4db1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:40:59.072 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:59.072 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.952256433999999, 2024-04-22T18:40:59.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69158947400000004, 2024-04-22T18:40:59.072 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.073 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:40:59.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.073 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3196 2024-04-22T18:40:59.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.073 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:59.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:59.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:40:59.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050361+0000", 2024-04-22T18:40:59.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2040000000000002e-06 2024-04-22T18:40:59.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050367+0000", 2024-04-22T18:40:59.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3119999999999999e-06 2024-04-22T18:40:59.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050372+0000", 2024-04-22T18:40:59.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0869999999999997e-06 2024-04-22T18:40:59.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150688+0000", 2024-04-22T18:40:59.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100315381 2024-04-22T18:40:59.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:59.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150691+0000", 2024-04-22T18:40:59.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8899999999999999e-06 2024-04-22T18:40:59.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741941+0000", 2024-04-22T18:40:59.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59125044900000001 2024-04-22T18:40:59.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741948+0000", 2024-04-22T18:40:59.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.827e-06 2024-04-22T18:40:59.077 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.077 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.077 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.077 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3316 4.11 4:8f787b23:::smithi08937929-41:head [write 1969239~724762] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:59.077 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.539840+0000", 2024-04-22T18:40:59.077 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.462774910999997, 2024-04-22T18:40:59.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50791082700000001, 2024-04-22T18:40:59.077 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.077 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3316 2024-04-22T18:40:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539840+0000", 2024-04-22T18:40:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539840+0000", 2024-04-22T18:40:59.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.539839+0000", 2024-04-22T18:40:59.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540217+0000", 2024-04-22T18:40:59.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037753899999999998 2024-04-22T18:40:59.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540225+0000", 2024-04-22T18:40:59.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4630000000000001e-06 2024-04-22T18:40:59.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540231+0000", 2024-04-22T18:40:59.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0440000000000001e-06 2024-04-22T18:40:59.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540273+0000", 2024-04-22T18:40:59.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1480000000000003e-05 2024-04-22T18:40:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540282+0000", 2024-04-22T18:40:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.935e-06 2024-04-22T18:40:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.844412+0000", 2024-04-22T18:40:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30413055500000002 2024-04-22T18:40:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.844463+0000", 2024-04-22T18:40:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0918000000000003e-05 2024-04-22T18:40:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.844537+0000", 2024-04-22T18:40:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3983e-05 2024-04-22T18:40:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.948265+0000", 2024-04-22T18:40:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103727471 2024-04-22T18:40:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.047658+0000", 2024-04-22T18:40:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099393426000000007 2024-04-22T18:40:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.047707+0000", 2024-04-22T18:40:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8733000000000002e-05 2024-04-22T18:40:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.047751+0000", 2024-04-22T18:40:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3847000000000002e-05 2024-04-22T18:40:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.083 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3317 4.11 4:8f787b23:::smithi08937929-41:head [setxattr _header (56) in=7b,truncate 2773188] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.540240+0000", 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.462375053000002, 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62115056400000002, 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3317 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540240+0000", 2024-04-22T18:40:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540240+0000", 2024-04-22T18:40:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540239+0000", 2024-04-22T18:40:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:59.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540243+0000", 2024-04-22T18:40:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4569999999999998e-06 2024-04-22T18:40:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540250+0000", 2024-04-22T18:40:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6440000000000003e-06 2024-04-22T18:40:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540256+0000", 2024-04-22T18:40:59.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8100000000000003e-06 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540302+0000", 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5859999999999998e-05 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540309+0000", 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3950000000000003e-06 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.947528+0000", 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40721847999999999 2024-04-22T18:40:59.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.947588+0000", 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0364e-05 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.947687+0000", 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8733999999999994e-05 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.948414+0000", 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072766900000000002 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161295+0000", 2024-04-22T18:40:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.212880818 2024-04-22T18:40:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161342+0000", 2024-04-22T18:40:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6894000000000002e-05 2024-04-22T18:40:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161390+0000", 2024-04-22T18:40:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8245999999999999e-05 2024-04-22T18:40:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.089 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.090 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.090 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3318 4.11 4:8f787b23:::smithi08937929-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:40:59.090 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:59.090 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.462344860999998, 2024-04-22T18:40:59.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62124934899999995, 2024-04-22T18:40:59.090 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.090 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:59.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.090 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3318 2024-04-22T18:40:59.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.091 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:59.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:59.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:40:59.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540276+0000", 2024-04-22T18:40:59.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.049e-06 2024-04-22T18:40:59.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540282+0000", 2024-04-22T18:40:59.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7050000000000002e-06 2024-04-22T18:40:59.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540288+0000", 2024-04-22T18:40:59.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2900000000000002e-06 2024-04-22T18:40:59.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540326+0000", 2024-04-22T18:40:59.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8818e-05 2024-04-22T18:40:59.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:40:59.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540330+0000", 2024-04-22T18:40:59.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6260000000000001e-06 2024-04-22T18:40:59.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.948024+0000", 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40769397600000001 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.948063+0000", 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9127e-05 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161420+0000", 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.213356876 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161463+0000", 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3219000000000001e-05 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161519+0000", 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6155e-05 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3337 4.11 4:8db271f9:::smithi08937929-19:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e96)", 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:27.149915+0000", 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.852699851000001, 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.42102647599999998, 2024-04-22T18:40:59.095 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.096 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:59.096 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.096 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.096 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.096 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3337 2024-04-22T18:40:59.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.096 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149915+0000", 2024-04-22T18:40:59.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149915+0000", 2024-04-22T18:40:59.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149915+0000", 2024-04-22T18:40:59.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149918+0000", 2024-04-22T18:40:59.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1030000000000002e-06 2024-04-22T18:40:59.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149921+0000", 2024-04-22T18:40:59.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0309999999999999e-06 2024-04-22T18:40:59.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149924+0000", 2024-04-22T18:40:59.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9510000000000002e-06 2024-04-22T18:40:59.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.149991+0000", 2024-04-22T18:40:59.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6773999999999995e-05 2024-04-22T18:40:59.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.150065+0000", 2024-04-22T18:40:59.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:40:59.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.570942+0000", 2024-04-22T18:40:59.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.420876947 2024-04-22T18:40:59.099 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.100 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.100 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.100 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3504 4.9 4:96737e19:::smithi08937929-4:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:40:59.100 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:40:59.100 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.591753258000001, 2024-04-22T18:40:59.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.951755291, 2024-04-22T18:40:59.100 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.100 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:59.100 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.100 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.100 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.100 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3504 2024-04-22T18:40:59.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.101 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:40:59.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:40:59.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410861+0000", 2024-04-22T18:40:59.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410863+0000", 2024-04-22T18:40:59.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.635e-06 2024-04-22T18:40:59.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410866+0000", 2024-04-22T18:40:59.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8499999999999998e-06 2024-04-22T18:40:59.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410868+0000", 2024-04-22T18:40:59.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.497e-06 2024-04-22T18:40:59.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615137+0000", 2024-04-22T18:40:59.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20426903499999999 2024-04-22T18:40:59.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615218+0000", 2024-04-22T18:40:59.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0860999999999997e-05 2024-04-22T18:40:59.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.362617+0000", 2024-04-22T18:40:59.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.74739882700000004 2024-04-22T18:40:59.104 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.104 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.104 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.104 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3758 4.7 4:e117a562:::smithi08937929-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:40:59.104 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:45.699568+0000", 2024-04-22T18:40:59.104 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.303046709, 2024-04-22T18:40:59.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50699397000000002, 2024-04-22T18:40:59.104 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.104 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:59.104 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.104 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.105 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.105 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3758 2024-04-22T18:40:59.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.105 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.699568+0000", 2024-04-22T18:40:59.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.699568+0000", 2024-04-22T18:40:59.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.699568+0000", 2024-04-22T18:40:59.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.699571+0000", 2024-04-22T18:40:59.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9919999999999999e-06 2024-04-22T18:40:59.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.699575+0000", 2024-04-22T18:40:59.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8570000000000003e-06 2024-04-22T18:40:59.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.699581+0000", 2024-04-22T18:40:59.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6999999999999996e-06 2024-04-22T18:40:59.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.799472+0000", 2024-04-22T18:40:59.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099890548999999995 2024-04-22T18:40:59.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:59.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.799517+0000", 2024-04-22T18:40:59.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5779000000000003e-05 2024-04-22T18:40:59.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.206434+0000", 2024-04-22T18:40:59.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40691630099999998 2024-04-22T18:40:59.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.206492+0000", 2024-04-22T18:40:59.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8350000000000002e-05 2024-04-22T18:40:59.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.206562+0000", 2024-04-22T18:40:59.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0094999999999999e-05 2024-04-22T18:40:59.109 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.109 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.109 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.109 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.109 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.109 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:59.110 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:40:59.111 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:59.111+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:59.158 INFO:tasks.rados.rados.0.smithi089.stdout:1712: set_chunk oid 47 offset: 1432064 length: 1689600 target oid ea019d54f4933e610b6ffbf93bcaceac006859815d8ad1e7bb13ddaf8d28a20e offset: 0 snap -1 2024-04-22T18:40:59.158 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:59.158 INFO:tasks.rados.rados.0.smithi089.stdout:1695: finishing write tid 4 to smithi08937929-13 2024-04-22T18:40:59.158 INFO:tasks.rados.rados.0.smithi089.stdout:1695: finishing write tid 5 to smithi08937929-13 2024-04-22T18:40:59.158 INFO:tasks.rados.rados.0.smithi089.stdout:1695: oid 13 updating version 0 to 108 2024-04-22T18:40:59.159 INFO:tasks.rados.rados.0.smithi089.stdout:1695: oid 13 version 108 is already newer than 107 2024-04-22T18:40:59.159 INFO:tasks.rados.rados.0.smithi089.stdout:1695: oid 13 updating version 108 to 110 2024-04-22T18:40:59.159 INFO:tasks.rados.rados.0.smithi089.stdout:1695: oid 13 version 110 is already newer than 109 2024-04-22T18:40:59.159 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 13 v 110 (ObjNum 348 snap 0 seq_num 348) dirty exists 2024-04-22T18:40:59.159 INFO:tasks.rados.rados.0.smithi089.stdout:1695: left oid 13 (ObjNum 348 snap 0 seq_num 348) 2024-04-22T18:40:59.159 INFO:tasks.rados.rados.0.smithi089.stdout:1698: finishing write tid 3 to smithi08937929-21 2024-04-22T18:40:59.159 INFO:tasks.rados.rados.0.smithi089.stdout:1695: done (15 left) 2024-04-22T18:40:59.159 INFO:tasks.rados.rados.0.smithi089.stdout:1713: write (excl) oid 26 current snap is 0 2024-04-22T18:40:59.159 INFO:tasks.rados.rados.0.smithi089.stdout:1713: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2024-04-22T18:40:59.165 INFO:tasks.rados.rados.0.smithi089.stdout:1713: writing smithi08937929-26 from 732871 to 1448006 tid 1 2024-04-22T18:40:59.171 INFO:tasks.rados.rados.0.smithi089.stdout:1713: writing smithi08937929-26 from 1873673 to 2517780 tid 2 2024-04-22T18:40:59.179 INFO:tasks.rados.rados.0.smithi089.stdout:1713: writing smithi08937929-26 from 3244434 to 3932871 tid 3 2024-04-22T18:40:59.179 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:40:59.179 INFO:tasks.rados.rados.0.smithi089.stdout:1699: expect (ObjNum 340 snap 0 seq_num 340) 2024-04-22T18:40:59.211 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:40:59.212+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:40:59.256 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:59.256 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:59.256 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:40:59.256 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:59.266 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:40:59.350 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:59.352+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:59.413 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:59.414+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:59.414 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:59.414 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:59.414 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:59.414 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:59.414 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:59.423 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:40:59.472 INFO:tasks.rados.rados.0.smithi089.stdout:1700: expect (ObjNum 295 snap 0 seq_num 295) 2024-04-22T18:40:59.514 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:40:59.515+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:40:59.568 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:59.568 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:40:59.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:40:59.568 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:59.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.568 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:75 3.0 e34/19 3:18418e0f:::b7c35f165c374d5e075dde9ac2607e47ace3b0a1e2343ec6efb171380f725eea:head v 34'32, mlcod=31'29)", 2024-04-22T18:40:59.568 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:59.568 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 115.10952851099999, 2024-04-22T18:40:59.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104798078, 2024-04-22T18:40:59.569 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.569 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.569 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:59.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458940+0000", 2024-04-22T18:40:59.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.458939+0000", 2024-04-22T18:40:59.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:59.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459092+0000", 2024-04-22T18:40:59.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153838 2024-04-22T18:40:59.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459107+0000", 2024-04-22T18:40:59.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4703e-05 2024-04-22T18:40:59.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459115+0000", 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560093+0000", 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10097840499999999 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560109+0000", 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5631e-05 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563706+0000", 2024-04-22T18:40:59.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035972059999999999 2024-04-22T18:40:59.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.563738+0000", 2024-04-22T18:40:59.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2184000000000002e-05 2024-04-22T18:40:59.572 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.572 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.572 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.572 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:77 3.0 e34/19 3:28387c9f:::b6d7840f02d561da04e398fbf94d91b85e9066b426e48baf3b11dd4a4f6096c0:head v 34'33, mlcod=31'29)", 2024-04-22T18:40:59.572 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:59.572 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 115.108578431, 2024-04-22T18:40:59.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104186079, 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459890+0000", 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459889+0000", 2024-04-22T18:40:59.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:59.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459963+0000", 2024-04-22T18:40:59.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3732999999999994e-05 2024-04-22T18:40:59.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459973+0000", 2024-04-22T18:40:59.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0285999999999999e-05 2024-04-22T18:40:59.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.459987+0000", 2024-04-22T18:40:59.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3675999999999999e-05 2024-04-22T18:40:59.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560515+0000", 2024-04-22T18:40:59.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10052781299999999 2024-04-22T18:40:59.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.560538+0000", 2024-04-22T18:40:59.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3861000000000001e-05 2024-04-22T18:40:59.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564044+0000", 2024-04-22T18:40:59.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035052009999999999 2024-04-22T18:40:59.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.564076+0000", 2024-04-22T18:40:59.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2737000000000002e-05 2024-04-22T18:40:59.576 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.576 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.576 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.576 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:40:59.576 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:59.576 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 115.107297019, 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:40:59.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:59.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:40:59.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:40:59.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:40:59.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:40:59.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:40:59.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:40:59.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:40:59.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:40:59.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:40:59.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:40:59.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:40:59.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:40:59.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:40:59.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:40:59.580 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.580 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.580 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.580 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:40:59.580 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:59.581 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 111.230623197, 2024-04-22T18:40:59.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:40:59.581 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.581 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.581 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:59.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:40:59.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:40:59.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:40:59.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:40:59.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:40:59.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:40:59.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:40:59.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:40:59.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:40:59.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:40:59.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:40:59.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:40:59.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:40:59.584 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:59.585 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.025819673999997, 2024-04-22T18:40:59.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:40:59.585 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.585 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.585 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:59.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:40:59.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:40:59.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:40:59.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:40:59.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:40:59.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:40:59.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:40:59.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:40:59.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:40:59.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:40:59.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:40:59.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:40:59.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:40:59.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:40:59.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:40:59.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:40:59.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:40:59.588 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.588 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.588 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.589 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:161 3.0 e76/63 3:3638311b:::f282598a146af3da1c05c8882972fba387afd249b39f6eeb6e6052a5de37fd2d:head v 76'157, mlcod=76'154)", 2024-04-22T18:40:59.589 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:59.589 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.177100244000002, 2024-04-22T18:40:59.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104845172, 2024-04-22T18:40:59.589 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.589 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.589 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:59.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:59.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:40:59.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:59.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391440+0000", 2024-04-22T18:40:59.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2045999999999994e-05 2024-04-22T18:40:59.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391448+0000", 2024-04-22T18:40:59.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8189999999999992e-06 2024-04-22T18:40:59.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391453+0000", 2024-04-22T18:40:59.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8379999999999998e-06 2024-04-22T18:40:59.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391839+0000", 2024-04-22T18:40:59.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000385325 2024-04-22T18:40:59.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391854+0000", 2024-04-22T18:40:59.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5006e-05 2024-04-22T18:40:59.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496191+0000", 2024-04-22T18:40:59.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104336792 2024-04-22T18:40:59.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496214+0000", 2024-04-22T18:40:59.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3079999999999999e-05 2024-04-22T18:40:59.593 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.593 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.593 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.593 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:165 3.0 e76/63 3:1e2f7d15:::93d9bab0d1f539405a95a28c02bcf2dcbdfa6f3e449e5e5e3f27f774df3c508a:head v 76'158, mlcod=76'154)", 2024-04-22T18:40:59.593 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:59.593 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.177006061, 2024-04-22T18:40:59.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104788752, 2024-04-22T18:40:59.593 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.593 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.593 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:59.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391463+0000", 2024-04-22T18:40:59.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391462+0000", 2024-04-22T18:40:59.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391522+0000", 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0148000000000001e-05 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391531+0000", 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5709999999999998e-06 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391536+0000", 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6180000000000003e-06 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392218+0000", 2024-04-22T18:40:59.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000681326 2024-04-22T18:40:59.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.392237+0000", 2024-04-22T18:40:59.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9726999999999999e-05 2024-04-22T18:40:59.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496231+0000", 2024-04-22T18:40:59.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.103993672 2024-04-22T18:40:59.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496251+0000", 2024-04-22T18:40:59.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0327999999999999e-05 2024-04-22T18:40:59.597 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.597 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.597 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.597 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:166 3.0 e76/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 76'159, mlcod=76'154)", 2024-04-22T18:40:59.597 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:59.597 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.176928162999999, 2024-04-22T18:40:59.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104724628, 2024-04-22T18:40:59.597 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.597 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.597 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:59.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391541+0000", 2024-04-22T18:40:59.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391540+0000", 2024-04-22T18:40:59.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391627+0000", 2024-04-22T18:40:59.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7330000000000006e-05 2024-04-22T18:40:59.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391635+0000", 2024-04-22T18:40:59.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6709999999999999e-06 2024-04-22T18:40:59.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391640+0000", 2024-04-22T18:40:59.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3199999999999999e-06 2024-04-22T18:40:59.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492541+0000", 2024-04-22T18:40:59.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100900873 2024-04-22T18:40:59.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.492560+0000", 2024-04-22T18:40:59.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8502999999999999e-05 2024-04-22T18:40:59.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496254+0000", 2024-04-22T18:40:59.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036945849999999998 2024-04-22T18:40:59.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496265+0000", 2024-04-22T18:40:59.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0869999999999999e-05 2024-04-22T18:40:59.601 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.601 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.601 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.601 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:167 3.0 e76/63 3:384c1ce2:::81dc8c58ebe106ae6fa9feadc06291c645d9079ad84b78214d4e6440fbc5def8:head v 76'160, mlcod=76'154)", 2024-04-22T18:40:59.601 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:59.601 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.176813287999998, 2024-04-22T18:40:59.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10487859400000001, 2024-04-22T18:40:59.601 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.601 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.602 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:59.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:59.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:40:59.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391707+0000", 2024-04-22T18:40:59.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2343999999999999e-05 2024-04-22T18:40:59.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391714+0000", 2024-04-22T18:40:59.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.596e-06 2024-04-22T18:40:59.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391719+0000", 2024-04-22T18:40:59.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9130000000000002e-06 2024-04-22T18:40:59.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493154+0000", 2024-04-22T18:40:59.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101435256 2024-04-22T18:40:59.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493177+0000", 2024-04-22T18:40:59.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2796e-05 2024-04-22T18:40:59.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496509+0000", 2024-04-22T18:40:59.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033320139999999999 2024-04-22T18:40:59.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496534+0000", 2024-04-22T18:40:59.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5265999999999998e-05 2024-04-22T18:40:59.605 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.605 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.605 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.605 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3451 4.d 4:bee1184f:::smithi08937929-7:head [read 0~3613505,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:40:59.605 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:59.605 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.641289347000001, 2024-04-22T18:40:59.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20164568099999999, 2024-04-22T18:40:59.606 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.606 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:59.606 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.606 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.606 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.606 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3451 2024-04-22T18:40:59.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.606 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:59.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:59.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:40:59.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927183+0000", 2024-04-22T18:40:59.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1110000000000001e-06 2024-04-22T18:40:59.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927188+0000", 2024-04-22T18:40:59.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1569999999999999e-06 2024-04-22T18:40:59.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927192+0000", 2024-04-22T18:40:59.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4710000000000001e-06 2024-04-22T18:40:59.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027382+0000", 2024-04-22T18:40:59.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100189341 2024-04-22T18:40:59.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027387+0000", 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8400000000000002e-06 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127638+0000", 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10025102499999999 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127740+0000", 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102749 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.128825+0000", 2024-04-22T18:40:59.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001084593 2024-04-22T18:40:59.610 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.610 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.610 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.610 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:151 3.0 e114/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 114'213, mlcod=110'210)", 2024-04-22T18:40:59.610 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:40:59.610 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.050821264, 2024-04-22T18:40:59.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20074001299999999, 2024-04-22T18:40:59.610 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.611 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.611 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:40:59.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:40:59.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517640+0000", 2024-04-22T18:40:59.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999928 2024-04-22T18:40:59.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517652+0000", 2024-04-22T18:40:59.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1824000000000001e-05 2024-04-22T18:40:59.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517660+0000", 2024-04-22T18:40:59.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1419999999999993e-06 2024-04-22T18:40:59.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617729+0000", 2024-04-22T18:40:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068723 2024-04-22T18:40:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617770+0000", 2024-04-22T18:40:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1391999999999999e-05 2024-04-22T18:40:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617793+0000", 2024-04-22T18:40:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2807999999999999e-05 2024-04-22T18:40:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718361+0000", 2024-04-22T18:40:59.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100567473 2024-04-22T18:40:59.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718387+0000", 2024-04-22T18:40:59.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6826e-05 2024-04-22T18:40:59.614 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.614 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.614 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3862 4.d 4:bee1184f:::smithi08937929-7:head [write 3301257~278667] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:40:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:40:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.305918994000001, 2024-04-22T18:40:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301875958, 2024-04-22T18:40:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3862 2024-04-22T18:40:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:40:59.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:40:59.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262548+0000", 2024-04-22T18:40:59.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:59.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262622+0000", 2024-04-22T18:40:59.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3747999999999999e-05 2024-04-22T18:40:59.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262632+0000", 2024-04-22T18:40:59.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0767e-05 2024-04-22T18:40:59.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362705+0000", 2024-04-22T18:40:59.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073157 2024-04-22T18:40:59.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:40:59.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2846999999999997e-05 2024-04-22T18:40:59.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362853+0000", 2024-04-22T18:40:59.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4498000000000002e-05 2024-04-22T18:40:59.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:59.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362992+0000", 2024-04-22T18:40:59.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013879699999999999 2024-04-22T18:40:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.366534+0000", 2024-04-22T18:40:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035427969999999999 2024-04-22T18:40:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564353+0000", 2024-04-22T18:40:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.197818893 2024-04-22T18:40:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564385+0000", 2024-04-22T18:40:59.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1717000000000001e-05 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564426+0000", 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0654000000000001e-05 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3863 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3579924] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.205751601999999, 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20206560900000001, 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.620 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3863 2024-04-22T18:40:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:40:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:40:59.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362715+0000", 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362725+0000", 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.006e-05 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362732+0000", 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6780000000000003e-06 2024-04-22T18:40:59.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362737+0000", 2024-04-22T18:40:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2089999999999996e-06 2024-04-22T18:40:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462875+0000", 2024-04-22T18:40:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013834100000001 2024-04-22T18:40:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462963+0000", 2024-04-22T18:40:59.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7792000000000002e-05 2024-04-22T18:40:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463127+0000", 2024-04-22T18:40:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000163688 2024-04-22T18:40:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463692+0000", 2024-04-22T18:40:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056466299999999997 2024-04-22T18:40:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564720+0000", 2024-04-22T18:40:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10102826199999999 2024-04-22T18:40:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564752+0000", 2024-04-22T18:40:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1936000000000001e-05 2024-04-22T18:40:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564783+0000", 2024-04-22T18:40:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0919e-05 2024-04-22T18:40:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3864 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:40:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:40:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.205718217999999, 2024-04-22T18:40:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20218423599999999, 2024-04-22T18:40:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3864 2024-04-22T18:40:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:40:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:40:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:40:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362753+0000", 2024-04-22T18:40:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6989999999999999e-06 2024-04-22T18:40:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:40:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:40:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462871+0000", 2024-04-22T18:40:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100113641 2024-04-22T18:40:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463385+0000", 2024-04-22T18:40:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051307100000000003 2024-04-22T18:40:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463457+0000", 2024-04-22T18:40:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2778e-05 2024-04-22T18:40:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564814+0000", 2024-04-22T18:40:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10135696299999999 2024-04-22T18:40:59.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564868+0000", 2024-04-22T18:40:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3872000000000002e-05 2024-04-22T18:40:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564935+0000", 2024-04-22T18:40:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6545999999999996e-05 2024-04-22T18:40:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4030 4.d 4:bee1184f:::smithi08937929-7:head [write 615222~472039] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.6092693159999998, 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109028217, 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4030 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:40:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:40:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959177+0000", 2024-04-22T18:40:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999776 2024-04-22T18:40:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959331+0000", 2024-04-22T18:40:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015334600000000001 2024-04-22T18:40:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959358+0000", 2024-04-22T18:40:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7220999999999999e-05 2024-04-22T18:40:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959371+0000", 2024-04-22T18:40:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3489000000000001e-05 2024-04-22T18:40:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059523+0000", 2024-04-22T18:40:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100152059 2024-04-22T18:40:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059588+0000", 2024-04-22T18:40:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4493999999999996e-05 2024-04-22T18:40:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059694+0000", 2024-04-22T18:40:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010584199999999999 2024-04-22T18:40:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068082+0000", 2024-04-22T18:40:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083880589999999998 2024-04-22T18:40:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068138+0000", 2024-04-22T18:40:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6196999999999997e-05 2024-04-22T18:40:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068185+0000", 2024-04-22T18:40:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7318000000000003e-05 2024-04-22T18:40:59.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068228+0000", 2024-04-22T18:40:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2376999999999998e-05 2024-04-22T18:40:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4031 4.d 4:bee1184f:::smithi08937929-7:head [write 1580652~489442] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:40:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.6039683240000002, 2024-04-22T18:40:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20571598299999999, 2024-04-22T18:40:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4031 2024-04-22T18:40:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:40:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:40:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964480+0000", 2024-04-22T18:40:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999795 2024-04-22T18:40:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964628+0000", 2024-04-22T18:40:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148109 2024-04-22T18:40:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964651+0000", 2024-04-22T18:40:59.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3215999999999999e-05 2024-04-22T18:40:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964664+0000", 2024-04-22T18:40:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2673e-05 2024-04-22T18:40:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061135+0000", 2024-04-22T18:40:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096470822999999997 2024-04-22T18:40:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061228+0000", 2024-04-22T18:40:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3459000000000001e-05 2024-04-22T18:40:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061376+0000", 2024-04-22T18:40:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148128 2024-04-22T18:40:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068085+0000", 2024-04-22T18:40:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.006708632 2024-04-22T18:40:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170038+0000", 2024-04-22T18:40:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101952716 2024-04-22T18:40:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170173+0000", 2024-04-22T18:40:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013508699999999999 2024-04-22T18:40:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170216+0000", 2024-04-22T18:40:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3498999999999998e-05 2024-04-22T18:40:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.642 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4032 4.d 4:bee1184f:::smithi08937929-7:head [write 2768826~626587] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:40:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.5967632170000003, 2024-04-22T18:40:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19941629799999999, 2024-04-22T18:40:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4032 2024-04-22T18:40:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:40:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:40:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971703+0000", 2024-04-22T18:40:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:40:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971996+0000", 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029351300000000002 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972012+0000", 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5569e-05 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972019+0000", 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7950000000000008e-06 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.063940+0000", 2024-04-22T18:40:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091920162 2024-04-22T18:40:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064021+0000", 2024-04-22T18:40:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1865000000000003e-05 2024-04-22T18:40:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064145+0000", 2024-04-22T18:40:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123893 2024-04-22T18:40:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068317+0000", 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041720799999999999 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171042+0000", 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102724893 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171085+0000", 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2619e-05 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171122+0000", 2024-04-22T18:40:59.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6838999999999997e-05 2024-04-22T18:40:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4033 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3815222] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:40:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.5964397349999997, 2024-04-22T18:40:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297586038, 2024-04-22T18:40:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:40:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.648 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4033 2024-04-22T18:40:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:40:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:40:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972028+0000", 2024-04-22T18:40:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972032+0000", 2024-04-22T18:40:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8330000000000004e-06 2024-04-22T18:40:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972035+0000", 2024-04-22T18:40:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:40:59.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972038+0000", 2024-04-22T18:40:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9579999999999999e-06 2024-04-22T18:40:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067511+0000", 2024-04-22T18:40:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.095472308000000006 2024-04-22T18:40:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067607+0000", 2024-04-22T18:40:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6546999999999996e-05 2024-04-22T18:40:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:40:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067776+0000", 2024-04-22T18:40:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000169202 2024-04-22T18:40:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:40:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068470+0000", 2024-04-22T18:40:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069403100000000001 2024-04-22T18:40:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269540+0000", 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20106934000000001 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269574+0000", 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4332000000000003e-05 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269615+0000", 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0988000000000001e-05 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4034 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:40:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:40:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.5964280979999996, 2024-04-22T18:40:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39788621000000002, 2024-04-22T18:40:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:40:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:40:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:40:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4034 2024-04-22T18:40:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:40:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:40:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972040+0000", 2024-04-22T18:40:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:40:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972042+0000", 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.519e-06 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972045+0000", 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972048+0000", 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.773e-06 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068266+0000", 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096218029999999996 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068308+0000", 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.21e-05 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369703+0000", 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301395578 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369802+0000", 2024-04-22T18:40:59.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8788000000000004e-05 2024-04-22T18:40:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369927+0000", 2024-04-22T18:40:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012478200000000001 2024-04-22T18:40:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(4.1 124/123 [PushOp(4:83c955bf:::smithi08937929-28:head, version: 115'157, data_included: [1890685~548185,2699879~469889], data_size: 1018074, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:83c955bf:::smithi08937929-28:head@115'157, size: 3630345, copy_subset: [0~3630345], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3630345, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1890685, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-22T18:40:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:40:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.92985998999999997, 2024-04-22T18:40:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20293218800000001, 2024-04-22T18:40:59.658 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:40:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:40:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:40:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638608+0000", 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639519+0000", 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091196199999999999 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639537+0000", 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7455000000000001e-05 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.739639+0000", 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100101942 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840271+0000", 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10063235299999999 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840281+0000", 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6509999999999996e-06 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.841541+0000", 2024-04-22T18:40:59.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012600230000000001 2024-04-22T18:40:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.662 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.662 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:59.663 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:40:59.664 INFO:tasks.rados.rados.0.smithi089.stdout:1698: finishing write tid 4 to smithi08937929-21 2024-04-22T18:40:59.664 INFO:tasks.rados.rados.0.smithi089.stdout:1698: finishing write tid 5 to smithi08937929-21 2024-04-22T18:40:59.664 INFO:tasks.rados.rados.0.smithi089.stdout:1698: oid 21 updating version 0 to 63 2024-04-22T18:40:59.664 INFO:tasks.rados.rados.0.smithi089.stdout:1698: oid 21 version 63 is already newer than 61 2024-04-22T18:40:59.664 INFO:tasks.rados.rados.0.smithi089.stdout:1698: oid 21 updating version 63 to 64 2024-04-22T18:40:59.664 INFO:tasks.rados.rados.0.smithi089.stdout:1698: oid 21 version 64 is already newer than 62 2024-04-22T18:40:59.664 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 64 (ObjNum 349 snap 0 seq_num 349) dirty exists 2024-04-22T18:40:59.664 INFO:tasks.rados.rados.0.smithi089.stdout:1698: left oid 21 (ObjNum 349 snap 0 seq_num 349) 2024-04-22T18:40:59.664 INFO:tasks.rados.rados.0.smithi089.stdout:1698: done (15 left) 2024-04-22T18:40:59.665 INFO:tasks.rados.rados.0.smithi089.stdout:1699: done (14 left) 2024-04-22T18:40:59.665 INFO:tasks.rados.rados.0.smithi089.stdout:1700: done (13 left) 2024-04-22T18:40:59.665 INFO:tasks.rados.rados.0.smithi089.stdout:1714: set_chunk oid 10 target oid 2024-04-22T18:40:59.665 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:40:59.664+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:40:59.764 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:59.765+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:59.768 INFO:tasks.rados.rados.0.smithi089.stdout:1714: set_chunk oid 10 offset: 1055232 length: 1348608 target oid 48f05881d8397501bc6119ae239e14ac22e37488d789da81ca18cc8233c5cc0b offset: 0 snap -1 2024-04-22T18:40:59.768 INFO:tasks.rados.rados.0.smithi089.stdout:1715: set_chunk oid 25 target oid 2024-04-22T18:40:59.807 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:40:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGRecoveryDeleteReply(3.2 e126,124 3:7a6cba00:::smithi08937929-32 low_tier:head,40'50)", 2024-04-22T18:40:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:59.769859+0000", 2024-04-22T18:40:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.039192176000000002, 2024-04-22T18:40:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.039211467999999999, 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.769859+0000", 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.769859+0000", 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.769858+0000", 2024-04-22T18:40:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:40:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.769863+0000", 2024-04-22T18:40:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6009999999999997e-06 2024-04-22T18:40:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.769871+0000", 2024-04-22T18:40:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8960000000000003e-06 2024-04-22T18:40:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:40:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.769876+0000", 2024-04-22T18:40:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.215e-06 2024-04-22T18:40:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:169 3.2 3.44498a92 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e126)", 2024-04-22T18:40:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:59.770492+0000", 2024-04-22T18:40:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.038558733999999997, 2024-04-22T18:40:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.038608558000000001, 2024-04-22T18:40:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:40:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "no flag points reached", 2024-04-22T18:40:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:40:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.770492+0000", 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.770492+0000", 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.770491+0000", 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:40:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.770497+0000", 2024-04-22T18:40:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0970000000000002e-06 2024-04-22T18:40:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:40:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:40:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:40:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.770506+0000", 2024-04-22T18:40:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4610000000000008e-06 2024-04-22T18:40:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:40:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:40:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:40:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 2 2024-04-22T18:40:59.812 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:59.816 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:40:59.865 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:40:59.867+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:40:59.922 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:40:59.922 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:40:59.922 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:40:59.922 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:40:59.922 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:40:59.931 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:40:59.973 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:40:59.975+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:00.034 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:00.034 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:00.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:00.034 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:00.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.034 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:289 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:00.034 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:00.034 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 57.393113618999998, 2024-04-22T18:41:00.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9176801189999999, 2024-04-22T18:41:00.035 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.035 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.035 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:00.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:00.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640723+0000", 2024-04-22T18:41:00.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:00.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640732+0000", 2024-04-22T18:41:00.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5809999999999997e-06 2024-04-22T18:41:00.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640744+0000", 2024-04-22T18:41:00.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.261e-05 2024-04-22T18:41:00.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640753+0000", 2024-04-22T18:41:00.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7460000000000006e-06 2024-04-22T18:41:00.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642189+0000", 2024-04-22T18:41:00.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001435551 2024-04-22T18:41:00.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:00.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642200+0000", 2024-04-22T18:41:00.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1389999999999999e-05 2024-04-22T18:41:00.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.847015+0000", 2024-04-22T18:41:00.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048149160000001 2024-04-22T18:41:00.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:00.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.848217+0000", 2024-04-22T18:41:00.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012020659999999999 2024-04-22T18:41:00.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351823+0000", 2024-04-22T18:41:00.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50360565099999999 2024-04-22T18:41:00.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351907+0000", 2024-04-22T18:41:00.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4141999999999995e-05 2024-04-22T18:41:00.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:00.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352051+0000", 2024-04-22T18:41:00.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014386600000000001 2024-04-22T18:41:00.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453617+0000", 2024-04-22T18:41:00.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156649500000001 2024-04-22T18:41:00.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558297+0000", 2024-04-22T18:41:00.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10468000399999999 2024-04-22T18:41:00.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558331+0000", 2024-04-22T18:41:00.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3518e-05 2024-04-22T18:41:00.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558405+0000", 2024-04-22T18:41:00.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4221000000000006e-05 2024-04-22T18:41:00.041 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.041 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.041 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.041 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:1 3.2 3:6d90b903:::64b5942574fb36ca1b9a2be186c6b740edf58e989056a2f88b83cd280451a438:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:00.041 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:00.041 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.861812659000002, 2024-04-22T18:41:00.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.285534422, 2024-04-22T18:41:00.041 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.041 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.041 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:00.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:00.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172024+0000", 2024-04-22T18:41:00.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:00.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173091+0000", 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010677569999999999 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173103+0000", 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1241e-05 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173109+0000", 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4300000000000003e-06 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173168+0000", 2024-04-22T18:41:00.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9011000000000001e-05 2024-04-22T18:41:00.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:00.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173177+0000", 2024-04-22T18:41:00.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-04-22T18:41:00.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948501+0000", 2024-04-22T18:41:00.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77532387199999997 2024-04-22T18:41:00.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:00.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948912+0000", 2024-04-22T18:41:00.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041183000000000002 2024-04-22T18:41:00.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950438+0000", 2024-04-22T18:41:00.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015253199999999999 2024-04-22T18:41:00.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950488+0000", 2024-04-22T18:41:00.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0285999999999999e-05 2024-04-22T18:41:00.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:00.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950601+0000", 2024-04-22T18:41:00.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011244899999999999 2024-04-22T18:41:00.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954048+0000", 2024-04-22T18:41:00.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003447313 2024-04-22T18:41:00.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457373+0000", 2024-04-22T18:41:00.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50332465299999996 2024-04-22T18:41:00.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457430+0000", 2024-04-22T18:41:00.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7574999999999997e-05 2024-04-22T18:41:00.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457560+0000", 2024-04-22T18:41:00.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013023900000000001 2024-04-22T18:41:00.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.047 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.048 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:8 3.2 3:5f64590d:::7c47933f331d40689081b2bd8f9d7f7bdae1be1fdccb52d696047a3969e902b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:00.048 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:00.048 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.860720290000003, 2024-04-22T18:41:00.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3854200569999999, 2024-04-22T18:41:00.048 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.048 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.048 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:00.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:00.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173117+0000", 2024-04-22T18:41:00.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:00.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173393+0000", 2024-04-22T18:41:00.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000275278 2024-04-22T18:41:00.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173399+0000", 2024-04-22T18:41:00.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4590000000000003e-06 2024-04-22T18:41:00.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173406+0000", 2024-04-22T18:41:00.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2450000000000001e-06 2024-04-22T18:41:00.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173795+0000", 2024-04-22T18:41:00.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038912400000000002 2024-04-22T18:41:00.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:00.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173804+0000", 2024-04-22T18:41:00.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-04-22T18:41:00.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.949506+0000", 2024-04-22T18:41:00.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77570185700000005 2024-04-22T18:41:00.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:00.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950166+0000", 2024-04-22T18:41:00.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066022800000000001 2024-04-22T18:41:00.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352843+0000", 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40267661799999999 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352911+0000", 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8126000000000005e-05 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.353031+0000", 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119919 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453622+0000", 2024-04-22T18:41:00.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100590793 2024-04-22T18:41:00.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558446+0000", 2024-04-22T18:41:00.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104824073 2024-04-22T18:41:00.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558479+0000", 2024-04-22T18:41:00.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3362999999999999e-05 2024-04-22T18:41:00.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558538+0000", 2024-04-22T18:41:00.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9046999999999999e-05 2024-04-22T18:41:00.054 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.054 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.054 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.054 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:10 3.2 3:518c37f9:::851eca84dafbf2f780c4f07291110c453349bdc9c623c17bdcd26e27badb30ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:00.054 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:00.054 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.860426697000001, 2024-04-22T18:41:00.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3852527729999999, 2024-04-22T18:41:00.054 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.054 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.054 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:00.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:00.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173411+0000", 2024-04-22T18:41:00.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:00.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173749+0000", 2024-04-22T18:41:00.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000338029 2024-04-22T18:41:00.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173755+0000", 2024-04-22T18:41:00.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0530000000000003e-06 2024-04-22T18:41:00.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173759+0000", 2024-04-22T18:41:00.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8190000000000002e-06 2024-04-22T18:41:00.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173855+0000", 2024-04-22T18:41:00.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5927999999999994e-05 2024-04-22T18:41:00.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:00.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173862+0000", 2024-04-22T18:41:00.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4449999999999999e-06 2024-04-22T18:41:00.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950835+0000", 2024-04-22T18:41:00.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77697248299999999 2024-04-22T18:41:00.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:00.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951632+0000", 2024-04-22T18:41:00.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079752299999999996 2024-04-22T18:41:00.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355407+0000", 2024-04-22T18:41:00.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40377480500000001 2024-04-22T18:41:00.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355462+0000", 2024-04-22T18:41:00.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4577999999999998e-05 2024-04-22T18:41:00.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:00.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355571+0000", 2024-04-22T18:41:00.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108964 2024-04-22T18:41:00.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453625+0000", 2024-04-22T18:41:00.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098054397000000001 2024-04-22T18:41:00.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558574+0000", 2024-04-22T18:41:00.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104948571 2024-04-22T18:41:00.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558602+0000", 2024-04-22T18:41:00.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7966999999999998e-05 2024-04-22T18:41:00.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558665+0000", 2024-04-22T18:41:00.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3032000000000002e-05 2024-04-22T18:41:00.060 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.060 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.060 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.061 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:11 3.2 3:718aa8aa:::9603c4f8e269a436b2f0bf376c5374e77bdd7cf58fccae60e4e65ac4696806d4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:00.061 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:00.061 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.860075078999998, 2024-04-22T18:41:00.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4852604440000001, 2024-04-22T18:41:00.061 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.061 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.061 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:00.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:00.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:00.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173813+0000", 2024-04-22T18:41:00.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0655999999999997e-05 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173819+0000", 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.468e-06 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173822+0000", 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4879999999999999e-06 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173983+0000", 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160519 2024-04-22T18:41:00.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:00.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173991+0000", 2024-04-22T18:41:00.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6490000000000008e-06 2024-04-22T18:41:00.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951688+0000", 2024-04-22T18:41:00.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77769743800000002 2024-04-22T18:41:00.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:00.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952419+0000", 2024-04-22T18:41:00.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073067499999999997 2024-04-22T18:41:00.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355901+0000", 2024-04-22T18:41:00.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40348237599999998 2024-04-22T18:41:00.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355953+0000", 2024-04-22T18:41:00.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1477000000000002e-05 2024-04-22T18:41:00.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:00.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356060+0000", 2024-04-22T18:41:00.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107114 2024-04-22T18:41:00.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453629+0000", 2024-04-22T18:41:00.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097569127000000005 2024-04-22T18:41:00.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658807+0000", 2024-04-22T18:41:00.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20517829800000001 2024-04-22T18:41:00.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658876+0000", 2024-04-22T18:41:00.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8454000000000006e-05 2024-04-22T18:41:00.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659024+0000", 2024-04-22T18:41:00.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148338 2024-04-22T18:41:00.067 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.067 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.067 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.067 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:12 3.2 3:475e1cd9:::efaab2bd13543cdbb143e869e20eb0ec424f7f5202707859b241ed91d4674936:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:00.067 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:00.067 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.860012134999998, 2024-04-22T18:41:00.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4853405259999999, 2024-04-22T18:41:00.068 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.068 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.068 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:00.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:00.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173940+0000", 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011465500000000001 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173946+0000", 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5289999999999999e-06 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173950+0000", 2024-04-22T18:41:00.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8290000000000001e-06 2024-04-22T18:41:00.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174277+0000", 2024-04-22T18:41:00.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032761299999999998 2024-04-22T18:41:00.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:00.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174285+0000", 2024-04-22T18:41:00.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.52e-06 2024-04-22T18:41:00.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952455+0000", 2024-04-22T18:41:00.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77817000700000005 2024-04-22T18:41:00.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:00.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954035+0000", 2024-04-22T18:41:00.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001580078 2024-04-22T18:41:00.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356379+0000", 2024-04-22T18:41:00.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40234440399999999 2024-04-22T18:41:00.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356428+0000", 2024-04-22T18:41:00.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8832999999999998e-05 2024-04-22T18:41:00.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:00.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356532+0000", 2024-04-22T18:41:00.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103235 2024-04-22T18:41:00.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453632+0000", 2024-04-22T18:41:00.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097100346000000004 2024-04-22T18:41:00.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659069+0000", 2024-04-22T18:41:00.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20543763800000001 2024-04-22T18:41:00.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659105+0000", 2024-04-22T18:41:00.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5240000000000001e-05 2024-04-22T18:41:00.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659167+0000", 2024-04-22T18:41:00.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2207999999999998e-05 2024-04-22T18:41:00.074 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.074 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.074 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.074 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:16 3.2 3:47c76cc8:::c00d18458eb20c0c0aa9c612cf9793a68445ab0ea7b9e650ecd0072c60a96c0f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:00.074 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:00.074 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.859884176000001, 2024-04-22T18:41:00.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.485347744, 2024-04-22T18:41:00.074 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.074 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.074 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:00.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:00.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:00.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:00.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174234+0000", 2024-04-22T18:41:00.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028047599999999999 2024-04-22T18:41:00.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174240+0000", 2024-04-22T18:41:00.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2079999999999997e-06 2024-04-22T18:41:00.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174244+0000", 2024-04-22T18:41:00.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5810000000000001e-06 2024-04-22T18:41:00.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174670+0000", 2024-04-22T18:41:00.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042640000000000001 2024-04-22T18:41:00.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:00.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174675+0000", 2024-04-22T18:41:00.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8709999999999996e-06 2024-04-22T18:41:00.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054168+0000", 2024-04-22T18:41:00.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87949257300000006 2024-04-22T18:41:00.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:00.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054877+0000", 2024-04-22T18:41:00.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070884300000000002 2024-04-22T18:41:00.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356836+0000", 2024-04-22T18:41:00.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30195926000000001 2024-04-22T18:41:00.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356887+0000", 2024-04-22T18:41:00.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0767999999999999e-05 2024-04-22T18:41:00.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:00.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356991+0000", 2024-04-22T18:41:00.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104072 2024-04-22T18:41:00.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453636+0000", 2024-04-22T18:41:00.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096644968999999997 2024-04-22T18:41:00.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659207+0000", 2024-04-22T18:41:00.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20557154699999999 2024-04-22T18:41:00.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659240+0000", 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2665e-05 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659302+0000", 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2234000000000001e-05 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:17 3.2 3:6ef31daa:::1cf4ff496c4480e58510f607481d976b47d84868e2c54016273c38baf590f3b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:00.080 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.859589896000003, 2024-04-22T18:41:00.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4851703729999999, 2024-04-22T18:41:00.081 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.081 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.081 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:00.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:00.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174248+0000", 2024-04-22T18:41:00.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174626+0000", 2024-04-22T18:41:00.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037839699999999999 2024-04-22T18:41:00.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174633+0000", 2024-04-22T18:41:00.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0990000000000004e-06 2024-04-22T18:41:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174636+0000", 2024-04-22T18:41:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5540000000000002e-06 2024-04-22T18:41:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174790+0000", 2024-04-22T18:41:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153807 2024-04-22T18:41:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174793+0000", 2024-04-22T18:41:00.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4970000000000001e-06 2024-04-22T18:41:00.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054902+0000", 2024-04-22T18:41:00.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88010886300000002 2024-04-22T18:41:00.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:00.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055366+0000", 2024-04-22T18:41:00.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046336199999999999 2024-04-22T18:41:00.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.452973+0000", 2024-04-22T18:41:00.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39760783999999999 2024-04-22T18:41:00.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453046+0000", 2024-04-22T18:41:00.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2374999999999997e-05 2024-04-22T18:41:00.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:00.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453193+0000", 2024-04-22T18:41:00.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014722499999999999 2024-04-22T18:41:00.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554698+0000", 2024-04-22T18:41:00.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101505096 2024-04-22T18:41:00.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659337+0000", 2024-04-22T18:41:00.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104639236 2024-04-22T18:41:00.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659365+0000", 2024-04-22T18:41:00.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7594e-05 2024-04-22T18:41:00.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659419+0000", 2024-04-22T18:41:00.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4058999999999999e-05 2024-04-22T18:41:00.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.087 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.087 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:18 3.2 3:792a4ed0:::e1c814961b3d1531fb5ed475b3d496a0405c3a819b68a92ff9b2391fe912370f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:00.087 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:00.087 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.859193640000001, 2024-04-22T18:41:00.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.585036954, 2024-04-22T18:41:00.088 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.088 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.088 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:00.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:00.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174640+0000", 2024-04-22T18:41:00.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-04-22T18:41:00.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174758+0000", 2024-04-22T18:41:00.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011819099999999999 2024-04-22T18:41:00.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174764+0000", 2024-04-22T18:41:00.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6729999999999997e-06 2024-04-22T18:41:00.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174767+0000", 2024-04-22T18:41:00.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3830000000000001e-06 2024-04-22T18:41:00.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373669+0000", 2024-04-22T18:41:00.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198901989 2024-04-22T18:41:00.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:00.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373686+0000", 2024-04-22T18:41:00.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7286e-05 2024-04-22T18:41:00.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055408+0000", 2024-04-22T18:41:00.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68172133400000001 2024-04-22T18:41:00.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:00.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055938+0000", 2024-04-22T18:41:00.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053016599999999999 2024-04-22T18:41:00.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453668+0000", 2024-04-22T18:41:00.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39773030799999998 2024-04-22T18:41:00.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453717+0000", 2024-04-22T18:41:00.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8884000000000002e-05 2024-04-22T18:41:00.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:00.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453823+0000", 2024-04-22T18:41:00.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105791 2024-04-22T18:41:00.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554715+0000", 2024-04-22T18:41:00.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100891628 2024-04-22T18:41:00.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759439+0000", 2024-04-22T18:41:00.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20472459400000001 2024-04-22T18:41:00.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759525+0000", 2024-04-22T18:41:00.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5653000000000001e-05 2024-04-22T18:41:00.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759682+0000", 2024-04-22T18:41:00.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015711800000000001 2024-04-22T18:41:00.094 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.094 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.094 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.094 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:75 3.1 3:ab43a94d:::f38e6e97cb24e2b90da964fee5d2d6d8674386870fc552915e0c3fdbefd4b126:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:00.094 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:00.094 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.786317123, 2024-04-22T18:41:00.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60149928399999997, 2024-04-22T18:41:00.094 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.094 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.094 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:00.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:00.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247520+0000", 2024-04-22T18:41:00.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:00.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247527+0000", 2024-04-22T18:41:00.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1670000000000002e-06 2024-04-22T18:41:00.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247539+0000", 2024-04-22T18:41:00.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503e-05 2024-04-22T18:41:00.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247549+0000", 2024-04-22T18:41:00.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0000000000000001e-05 2024-04-22T18:41:00.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247796+0000", 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024712699999999998 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247955+0000", 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158534 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.343985+0000", 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096030344000000004 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:00.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.344035+0000", 2024-04-22T18:41:00.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9969999999999998e-05 2024-04-22T18:41:00.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.745806+0000", 2024-04-22T18:41:00.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.401771029 2024-04-22T18:41:00.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746040+0000", 2024-04-22T18:41:00.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023339300000000001 2024-04-22T18:41:00.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:00.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746198+0000", 2024-04-22T18:41:00.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015793299999999999 2024-04-22T18:41:00.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848737+0000", 2024-04-22T18:41:00.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10253898 2024-04-22T18:41:00.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848751+0000", 2024-04-22T18:41:00.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4694e-05 2024-04-22T18:41:00.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848825+0000", 2024-04-22T18:41:00.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3443999999999997e-05 2024-04-22T18:41:00.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849021+0000", 2024-04-22T18:41:00.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000196047 2024-04-22T18:41:00.100 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.100 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.100 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.101 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:76 3.1 3:b44829a0:::3788286a255d538503614e5f04049b0bb54cacaa317ab102b5dee8f5b640b296:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:00.101 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:00.101 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.786278057000001, 2024-04-22T18:41:00.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60168681599999996, 2024-04-22T18:41:00.101 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.101 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.101 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:00.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:00.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:00.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:00.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247564+0000", 2024-04-22T18:41:00.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1150000000000004e-06 2024-04-22T18:41:00.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247570+0000", 2024-04-22T18:41:00.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3940000000000001e-06 2024-04-22T18:41:00.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247576+0000", 2024-04-22T18:41:00.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3619999999999997e-06 2024-04-22T18:41:00.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.340845+0000", 2024-04-22T18:41:00.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.093269199999999997 2024-04-22T18:41:00.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:00.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.341043+0000", 2024-04-22T18:41:00.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019834400000000001 2024-04-22T18:41:00.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345106+0000", 2024-04-22T18:41:00.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040629780000000001 2024-04-22T18:41:00.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:00.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345160+0000", 2024-04-22T18:41:00.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4246999999999997e-05 2024-04-22T18:41:00.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747047+0000", 2024-04-22T18:41:00.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40188678300000003 2024-04-22T18:41:00.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747162+0000", 2024-04-22T18:41:00.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011483400000000001 2024-04-22T18:41:00.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:00.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747429+0000", 2024-04-22T18:41:00.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026700299999999998 2024-04-22T18:41:00.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849032+0000", 2024-04-22T18:41:00.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101603203 2024-04-22T18:41:00.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849095+0000", 2024-04-22T18:41:00.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2597999999999999e-05 2024-04-22T18:41:00.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849136+0000", 2024-04-22T18:41:00.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1251000000000002e-05 2024-04-22T18:41:00.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849247+0000", 2024-04-22T18:41:00.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011127 2024-04-22T18:41:00.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.107 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.107 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3524 4.c 4:314c2fec:::smithi08937929-22:head [write 692521~538214] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:41:00.107 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.128717+0000", 2024-04-22T18:41:00.107 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.905121961999999, 2024-04-22T18:41:00.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40275804199999998, 2024-04-22T18:41:00.107 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.107 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.108 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:00.108 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:00.108 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.108 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3524 2024-04-22T18:41:00.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.108 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128717+0000", 2024-04-22T18:41:00.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128717+0000", 2024-04-22T18:41:00.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128715+0000", 2024-04-22T18:41:00.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:00.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128877+0000", 2024-04-22T18:41:00.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016187999999999999 2024-04-22T18:41:00.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128886+0000", 2024-04-22T18:41:00.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8780000000000004e-06 2024-04-22T18:41:00.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.228979+0000", 2024-04-22T18:41:00.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092734 2024-04-22T18:41:00.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229043+0000", 2024-04-22T18:41:00.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4036999999999997e-05 2024-04-22T18:41:00.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229146+0000", 2024-04-22T18:41:00.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010301700000000001 2024-04-22T18:41:00.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:00.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229286+0000", 2024-04-22T18:41:00.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014017000000000001 2024-04-22T18:41:00.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.232704+0000", 2024-04-22T18:41:00.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0034182169999999999 2024-04-22T18:41:00.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.531413+0000", 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29870898200000001 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.531445+0000", 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1751999999999997e-05 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.531475+0000", 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9906000000000001e-05 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3525 4.c 4:314c2fec:::smithi08937929-22:head [write 1931338~785088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.229011+0000", 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.804827364000001, 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40428670300000003, 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3525 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229011+0000", 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229011+0000", 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229010+0000", 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229182+0000", 2024-04-22T18:41:00.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000171645 2024-04-22T18:41:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229188+0000", 2024-04-22T18:41:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.674e-06 2024-04-22T18:41:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229193+0000", 2024-04-22T18:41:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8890000000000001e-06 2024-04-22T18:41:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.230814+0000", 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00162068 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.230865+0000", 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1700000000000003e-05 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.230967+0000", 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101481 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.234144+0000", 2024-04-22T18:41:00.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031771849999999999 2024-04-22T18:41:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.633226+0000", 2024-04-22T18:41:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39908168799999999 2024-04-22T18:41:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.633257+0000", 2024-04-22T18:41:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1096999999999997e-05 2024-04-22T18:41:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.633298+0000", 2024-04-22T18:41:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0960000000000001e-05 2024-04-22T18:41:00.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.118 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.118 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3526 4.c 4:314c2fec:::smithi08937929-22:head [setxattr _header (56) in=7b,truncate 3092521] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:41:00.118 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:00.118 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.804641277000002, 2024-04-22T18:41:00.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60463626000000004, 2024-04-22T18:41:00.118 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.118 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.118 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:00.118 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:00.118 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.119 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3526 2024-04-22T18:41:00.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.119 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:00.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:00.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229200+0000", 2024-04-22T18:41:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2320000000000001e-06 2024-04-22T18:41:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229203+0000", 2024-04-22T18:41:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:41:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329292+0000", 2024-04-22T18:41:00.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100088891 2024-04-22T18:41:00.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329357+0000", 2024-04-22T18:41:00.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5115000000000003e-05 2024-04-22T18:41:00.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329453+0000", 2024-04-22T18:41:00.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5802000000000002e-05 2024-04-22T18:41:00.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:00.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329594+0000", 2024-04-22T18:41:00.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014157500000000001 2024-04-22T18:41:00.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.330172+0000", 2024-04-22T18:41:00.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057782799999999996 2024-04-22T18:41:00.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833712+0000", 2024-04-22T18:41:00.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50353975299999998 2024-04-22T18:41:00.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833774+0000", 2024-04-22T18:41:00.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1832e-05 2024-04-22T18:41:00.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833834+0000", 2024-04-22T18:41:00.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9809999999999998e-05 2024-04-22T18:41:00.123 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.123 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.123 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.124 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3527 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:41:00.124 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.329322+0000", 2024-04-22T18:41:00.124 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.704516205000001, 2024-04-22T18:41:00.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50479107700000003, 2024-04-22T18:41:00.124 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.124 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:00.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:00.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:00.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.124 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3527 2024-04-22T18:41:00.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.124 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329322+0000", 2024-04-22T18:41:00.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329322+0000", 2024-04-22T18:41:00.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329321+0000", 2024-04-22T18:41:00.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:00.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329337+0000", 2024-04-22T18:41:00.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5820000000000001e-05 2024-04-22T18:41:00.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329345+0000", 2024-04-22T18:41:00.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4610000000000008e-06 2024-04-22T18:41:00.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.429437+0000", 2024-04-22T18:41:00.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092323 2024-04-22T18:41:00.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.429503+0000", 2024-04-22T18:41:00.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5446999999999999e-05 2024-04-22T18:41:00.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:00.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.429626+0000", 2024-04-22T18:41:00.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012354600000000001 2024-04-22T18:41:00.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833904+0000", 2024-04-22T18:41:00.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.404277901 2024-04-22T18:41:00.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.834004+0000", 2024-04-22T18:41:00.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9444999999999994e-05 2024-04-22T18:41:00.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.834113+0000", 2024-04-22T18:41:00.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010976099999999999 2024-04-22T18:41:00.128 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.128 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.128 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.128 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3735 4.0 4:01baf5a9:::smithi08937929-20:head [write 527819~624406] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:00.129 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:45.017334+0000", 2024-04-22T18:41:00.129 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 15.0165042, 2024-04-22T18:41:00.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50288105500000002, 2024-04-22T18:41:00.129 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.129 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.129 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:00.129 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:00.129 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.129 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3735 2024-04-22T18:41:00.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.129 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.017334+0000", 2024-04-22T18:41:00.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.017334+0000", 2024-04-22T18:41:00.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.017330+0000", 2024-04-22T18:41:00.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999962 2024-04-22T18:41:00.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.017601+0000", 2024-04-22T18:41:00.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027010400000000001 2024-04-22T18:41:00.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.017619+0000", 2024-04-22T18:41:00.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7978999999999998e-05 2024-04-22T18:41:00.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117701+0000", 2024-04-22T18:41:00.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100082011 2024-04-22T18:41:00.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117758+0000", 2024-04-22T18:41:00.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7191000000000002e-05 2024-04-22T18:41:00.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117826+0000", 2024-04-22T18:41:00.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8697999999999999e-05 2024-04-22T18:41:00.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:00.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117916+0000", 2024-04-22T18:41:00.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9888000000000004e-05 2024-04-22T18:41:00.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.120124+0000", 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022077360000000001 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.520160+0000", 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40003630499999998 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.520192+0000", 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1334999999999997e-05 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.520215+0000", 2024-04-22T18:41:00.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3668000000000001e-05 2024-04-22T18:41:00.134 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.134 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.134 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.134 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3736 4.0 4:01baf5a9:::smithi08937929-20:head [write 1833733~410771] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:00.134 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:45.117729+0000", 2024-04-22T18:41:00.134 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.916109621, 2024-04-22T18:41:00.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40259752999999998, 2024-04-22T18:41:00.134 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.134 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.134 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:00.134 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:00.134 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.134 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3736 2024-04-22T18:41:00.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.135 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117729+0000", 2024-04-22T18:41:00.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117729+0000", 2024-04-22T18:41:00.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117726+0000", 2024-04-22T18:41:00.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:00.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117983+0000", 2024-04-22T18:41:00.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025640400000000001 2024-04-22T18:41:00.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117995+0000", 2024-04-22T18:41:00.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1922999999999999e-05 2024-04-22T18:41:00.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.118004+0000", 2024-04-22T18:41:00.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9099999999999994e-06 2024-04-22T18:41:00.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.119171+0000", 2024-04-22T18:41:00.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001167682 2024-04-22T18:41:00.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.119237+0000", 2024-04-22T18:41:00.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6197000000000003e-05 2024-04-22T18:41:00.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:00.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.119338+0000", 2024-04-22T18:41:00.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000100636 2024-04-22T18:41:00.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.120523+0000", 2024-04-22T18:41:00.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001185052 2024-04-22T18:41:00.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.520260+0000", 2024-04-22T18:41:00.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39973705799999998 2024-04-22T18:41:00.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.520291+0000", 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0508999999999999e-05 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.520326+0000", 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5720999999999999e-05 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3737 4.0 4:01baf5a9:::smithi08937929-20:head [write 2722618~601581] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:00.139 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:45.118019+0000", 2024-04-22T18:41:00.140 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.915819268, 2024-04-22T18:41:00.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40237807599999997, 2024-04-22T18:41:00.140 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.140 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.140 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:00.140 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:00.140 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.140 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3737 2024-04-22T18:41:00.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.140 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.118019+0000", 2024-04-22T18:41:00.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.118019+0000", 2024-04-22T18:41:00.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.118017+0000", 2024-04-22T18:41:00.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:00.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.118393+0000", 2024-04-22T18:41:00.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037648599999999999 2024-04-22T18:41:00.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.118404+0000", 2024-04-22T18:41:00.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0693000000000001e-05 2024-04-22T18:41:00.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218509+0000", 2024-04-22T18:41:00.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10010458799999999 2024-04-22T18:41:00.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218576+0000", 2024-04-22T18:41:00.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7297000000000003e-05 2024-04-22T18:41:00.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218700+0000", 2024-04-22T18:41:00.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012466500000000001 2024-04-22T18:41:00.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:00.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218862+0000", 2024-04-22T18:41:00.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000161989 2024-04-22T18:41:00.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.321729+0000", 2024-04-22T18:41:00.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10286685199999999 2024-04-22T18:41:00.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.520356+0000", 2024-04-22T18:41:00.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198626833 2024-04-22T18:41:00.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.520379+0000", 2024-04-22T18:41:00.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.27e-05 2024-04-22T18:41:00.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.520397+0000", 2024-04-22T18:41:00.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8525e-05 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3738 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 3327819] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:45.218527+0000", 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.815311684999999, 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50245176700000005, 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3738 2024-04-22T18:41:00.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.146 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218527+0000", 2024-04-22T18:41:00.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218527+0000", 2024-04-22T18:41:00.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218525+0000", 2024-04-22T18:41:00.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:00.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218537+0000", 2024-04-22T18:41:00.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2668e-05 2024-04-22T18:41:00.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218546+0000", 2024-04-22T18:41:00.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1710000000000002e-06 2024-04-22T18:41:00.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218555+0000", 2024-04-22T18:41:00.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2660000000000007e-06 2024-04-22T18:41:00.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.220809+0000", 2024-04-22T18:41:00.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022538649999999999 2024-04-22T18:41:00.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.220903+0000", 2024-04-22T18:41:00.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4072000000000003e-05 2024-04-22T18:41:00.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:00.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.221042+0000", 2024-04-22T18:41:00.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013949700000000001 2024-04-22T18:41:00.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:00.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.321737+0000", 2024-04-22T18:41:00.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100694216 2024-04-22T18:41:00.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.720867+0000", 2024-04-22T18:41:00.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39913031199999999 2024-04-22T18:41:00.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:00.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.720921+0000", 2024-04-22T18:41:00.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4166999999999998e-05 2024-04-22T18:41:00.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.720979+0000", 2024-04-22T18:41:00.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7593000000000002e-05 2024-04-22T18:41:00.150 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.150 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.150 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.150 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3739 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:00.150 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:45.218575+0000", 2024-04-22T18:41:00.150 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.815263205000001, 2024-04-22T18:41:00.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50254516100000002, 2024-04-22T18:41:00.150 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3739 2024-04-22T18:41:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218575+0000", 2024-04-22T18:41:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:00.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218575+0000", 2024-04-22T18:41:00.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:00.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:00.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218575+0000", 2024-04-22T18:41:00.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:00.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218578+0000", 2024-04-22T18:41:00.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0319999999999999e-06 2024-04-22T18:41:00.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:00.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218585+0000", 2024-04-22T18:41:00.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7499999999999997e-06 2024-04-22T18:41:00.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218588+0000", 2024-04-22T18:41:00.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5329999999999999e-06 2024-04-22T18:41:00.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.321558+0000", 2024-04-22T18:41:00.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102970176 2024-04-22T18:41:00.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:00.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.321708+0000", 2024-04-22T18:41:00.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014996000000000001 2024-04-22T18:41:00.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:00.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.721012+0000", 2024-04-22T18:41:00.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39930342499999999 2024-04-22T18:41:00.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:00.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.721058+0000", 2024-04-22T18:41:00.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6220000000000001e-05 2024-04-22T18:41:00.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:00.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:00.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:00.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.721121+0000", 2024-04-22T18:41:00.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2687000000000005e-05 2024-04-22T18:41:00.155 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.155 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.155 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.155 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:00.155 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:00.155 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:00.156 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:00.157 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:00.157+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:00.172 INFO:tasks.rados.rados.0.smithi089.stdout:1715: set_chunk oid 25 offset: 1043456 length: 1428480 target oid 3ba8a2bc05f6a392bc5111bbddc3fecdb9cfa0eb2ead19d210aac296cf4788f1 offset: 0 snap -1 2024-04-22T18:41:00.172 INFO:tasks.rados.rados.0.smithi089.stdout:1716: read oid 5 snap -1 2024-04-22T18:41:00.172 INFO:tasks.rados.rados.0.smithi089.stdout:1716: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:41:00.172 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:41:00.172 INFO:tasks.rados.rados.0.smithi089.stdout:1696: finishing set_chunk to oid 41 2024-04-22T18:41:00.172 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 127 (ObjNum 313 snap 0 seq_num 313) dirty exists 2024-04-22T18:41:00.172 INFO:tasks.rados.rados.0.smithi089.stdout:1704: finishing write tid 1 to smithi08937929-48 2024-04-22T18:41:00.172 INFO:tasks.rados.rados.0.smithi089.stdout:1705: finishing set_chunk to oid 9 2024-04-22T18:41:00.173 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 9 set_chunk returned error code -95 offset: 872448 length: 1147904 2024-04-22T18:41:00.173 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 0 (ObjNum 283 snap 0 seq_num 283) dirty exists 2024-04-22T18:41:00.173 INFO:tasks.rados.rados.0.smithi089.stdout:1706: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:41:00.257 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:00.258+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:00.264 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:00.264 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:00.264 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:00.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:00.357 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:00.359+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:00.394 INFO:tasks.rados.rados.0.smithi089.stdout:1704: finishing write tid 2 to smithi08937929-48 2024-04-22T18:41:00.394 INFO:tasks.rados.rados.0.smithi089.stdout:1704: finishing write tid 3 to smithi08937929-48 2024-04-22T18:41:00.394 INFO:tasks.rados.rados.0.smithi089.stdout:1704: finishing write tid 4 to smithi08937929-48 2024-04-22T18:41:00.394 INFO:tasks.rados.rados.0.smithi089.stdout:1704: finishing write tid 5 to smithi08937929-48 2024-04-22T18:41:00.394 INFO:tasks.rados.rados.0.smithi089.stdout:1704: oid 48 updating version 0 to 95 2024-04-22T18:41:00.394 INFO:tasks.rados.rados.0.smithi089.stdout:1704: oid 48 version 95 is already newer than 92 2024-04-22T18:41:00.394 INFO:tasks.rados.rados.0.smithi089.stdout:1704: oid 48 version 95 is already newer than 94 2024-04-22T18:41:00.394 INFO:tasks.rados.rados.0.smithi089.stdout:1704: oid 48 version 95 is already newer than 93 2024-04-22T18:41:00.394 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 95 (ObjNum 351 snap 0 seq_num 351) dirty exists 2024-04-22T18:41:00.394 INFO:tasks.rados.rados.0.smithi089.stdout:1704: left oid 48 (ObjNum 351 snap 0 seq_num 351) 2024-04-22T18:41:00.395 INFO:tasks.rados.rados.0.smithi089.stdout:1703: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:41:00.414 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:00.415 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:00.415 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:00.415 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:00.415 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:00.423 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:00.467 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:00.468+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:00.521 INFO:tasks.rados.rados.0.smithi089.stdout:1707: finishing set_chunk to oid 33 2024-04-22T18:41:00.521 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 60 (ObjNum 345 snap 0 seq_num 345) dirty exists 2024-04-22T18:41:00.521 INFO:tasks.rados.rados.0.smithi089.stdout:1702: expect (ObjNum 189 snap 0 seq_num 189) 2024-04-22T18:41:00.526 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:00.527 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:00.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:00.527 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:00.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.527 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:00.527 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:41:00.527 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.36435366000001, 2024-04-22T18:41:00.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:41:00.527 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.527 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.527 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.527 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.527 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.527 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:41:00.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.528 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:41:00.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:41:00.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:41:00.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:00.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:41:00.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:41:00.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:41:00.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:41:00.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:41:00.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:41:00.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:41:00.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:41:00.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:41:00.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:41:00.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:41:00.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:41:00.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:41:00.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:00.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:41:00.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:41:00.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:41:00.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.532 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:00.533 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:41:00.533 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.26279483899999, 2024-04-22T18:41:00.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:41:00.533 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.533 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.533 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.533 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.533 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.533 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:41:00.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.533 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:41:00.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:41:00.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:41:00.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:00.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:41:00.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:41:00.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:41:00.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:41:00.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:00.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:41:00.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:41:00.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:00.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:41:00.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:41:00.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:41:00.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:41:00.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:00.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:41:00.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:41:00.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:41:00.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:41:00.538 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.538 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.538 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.538 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:00.538 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:41:00.538 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.26276843700001, 2024-04-22T18:41:00.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:41:00.539 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.539 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:00.539 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.539 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.539 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.539 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:41:00.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.539 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:41:00.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:41:00.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:41:00.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:41:00.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:41:00.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:00.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:41:00.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:41:00.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:41:00.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:41:00.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:41:00.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:41:00.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:41:00.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:41:00.543 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.543 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.543 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.543 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:00.543 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:00.543 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 125.864434108, 2024-04-22T18:41:00.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:41:00.543 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.543 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.543 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.544 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.544 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.544 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:41:00.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.544 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:00.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:00.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:00.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:41:00.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:41:00.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:41:00.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:41:00.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:00.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:41:00.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:41:00.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:41:00.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:41:00.548 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.548 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.548 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.548 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:00.549 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:41:00.549 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 125.76359780999999, 2024-04-22T18:41:00.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:41:00.549 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.549 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.549 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.549 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.549 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.549 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:41:00.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.549 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:41:00.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:41:00.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:41:00.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:00.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:41:00.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:41:00.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:41:00.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:41:00.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:41:00.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:41:00.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:41:00.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:41:00.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:41:00.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:41:00.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:00.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:41:00.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:41:00.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:00.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:41:00.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:41:00.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:41:00.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:41:00.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:00.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:41:00.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:41:00.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:41:00.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:41:00.554 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.554 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.554 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.554 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:00.554 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:41:00.554 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 125.76344054, 2024-04-22T18:41:00.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:41:00.554 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.554 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:41:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:41:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:41:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:41:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:41:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:41:00.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:41:00.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:41:00.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:41:00.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:41:00.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:41:00.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:00.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:41:00.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:41:00.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:00.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:41:00.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:41:00.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:41:00.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:41:00.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:00.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:41:00.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:41:00.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:41:00.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:41:00.559 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.559 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.559 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 125.763425931, 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:00.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:41:00.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:41:00.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:41:00.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:41:00.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:41:00.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:41:00.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:41:00.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:41:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:41:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:41:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:41:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:41:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:41:00.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 125.763415546, 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.564 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.565 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.565 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:41:00.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.565 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:00.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:00.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:41:00.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:41:00.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:41:00.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:41:00.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:41:00.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:41:00.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:41:00.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:41:00.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:41:00.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:41:00.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:41:00.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:00.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:41:00.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:41:00.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:00.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:41:00.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:41:00.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:41:00.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:41:00.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:00.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:41:00.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:41:00.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:41:00.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:41:00.570 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.570 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.570 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.570 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:00.570 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:41:00.570 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 125.76248561600001, 2024-04-22T18:41:00.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:41:00.570 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.571 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.571 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.571 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.571 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.571 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:41:00.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.571 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:41:00.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:41:00.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:41:00.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:00.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:41:00.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:41:00.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:41:00.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:41:00.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:41:00.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:41:00.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:41:00.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:41:00.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:41:00.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:41:00.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:00.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:41:00.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:41:00.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:00.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:41:00.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:41:00.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:41:00.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:41:00.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:00.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:41:00.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:41:00.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:41:00.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:41:00.576 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.576 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.576 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.576 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:00.576 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:41:00.576 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 125.76202349099999, 2024-04-22T18:41:00.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:41:00.577 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.577 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.577 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.577 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.577 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.577 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:41:00.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.577 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:41:00.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:41:00.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:41:00.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:00.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:41:00.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:41:00.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:41:00.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:41:00.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:41:00.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:41:00.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:41:00.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:41:00.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:41:00.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:41:00.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:00.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:41:00.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:41:00.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:00.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:41:00.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:41:00.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:41:00.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:41:00.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:00.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:41:00.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:41:00.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:41:00.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:41:00.582 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.582 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.582 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.582 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:00.582 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:41:00.582 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 125.762010687, 2024-04-22T18:41:00.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:41:00.582 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.582 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:41:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:41:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:41:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:41:00.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:41:00.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:41:00.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:41:00.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:41:00.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:41:00.586 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.586 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.587 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.587 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:41:00.587 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:41:00.587 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 122.43423823099999, 2024-04-22T18:41:00.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:41:00.587 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.587 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.587 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.587 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.587 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.587 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:41:00.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.587 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:41:00.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:41:00.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:41:00.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:00.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:41:00.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:41:00.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:41:00.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:41:00.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:41:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:41:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:41:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:41:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:41:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:41:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:00.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:41:00.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:41:00.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:00.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:41:00.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:41:00.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:41:00.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:41:00.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:00.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:41:00.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:41:00.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:41:00.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:41:00.592 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.592 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.592 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.592 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:41:00.592 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:00.593 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 121.425665119, 2024-04-22T18:41:00.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:41:00.593 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.593 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:00.593 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.593 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.593 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.593 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:41:00.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.593 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:00.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:00.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:41:00.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:00.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:41:00.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:41:00.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:41:00.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:41:00.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:00.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:41:00.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:41:00.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:00.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:41:00.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:41:00.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:00.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:41:00.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:41:00.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:00.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:41:00.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:41:00.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:41:00.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:41:00.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.598 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.598 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:41:00.598 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:00.598 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.391017603999998, 2024-04-22T18:41:00.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:41:00.598 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.599 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:00.599 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.599 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.599 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.599 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:41:00.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.599 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:00.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:00.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:41:00.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:41:00.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:41:00.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:41:00.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:41:00.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:41:00.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:41:00.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:41:00.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:41:00.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:41:00.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:41:00.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:41:00.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:41:00.602 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.602 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1684 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:41:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:41:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 89.921533640999996, 2024-04-22T18:41:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987885199999998, 2024-04-22T18:41:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1684 2024-04-22T18:41:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.603 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:41:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:41:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605278+0000", 2024-04-22T18:41:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605280+0000", 2024-04-22T18:41:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1770000000000001e-06 2024-04-22T18:41:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605285+0000", 2024-04-22T18:41:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-04-22T18:41:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605289+0000", 2024-04-22T18:41:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.687e-06 2024-04-22T18:41:00.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705352+0000", 2024-04-22T18:41:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006322199999999 2024-04-22T18:41:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705423+0000", 2024-04-22T18:41:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1136999999999993e-05 2024-04-22T18:41:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105017+0000", 2024-04-22T18:41:00.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.399593268 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105081+0000", 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4703999999999996e-05 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105157+0000", 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6205999999999995e-05 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.607 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2191 4.1 4.fdaa93c1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-04-22T18:41:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:41:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 74.976941390999997, 2024-04-22T18:41:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51064384399999996, 2024-04-22T18:41:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2191 2024-04-22T18:41:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:41:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:41:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:41:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549872+0000", 2024-04-22T18:41:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.387e-06 2024-04-22T18:41:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549875+0000", 2024-04-22T18:41:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:41:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549880+0000", 2024-04-22T18:41:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2529999999999998e-06 2024-04-22T18:41:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060496+0000", 2024-04-22T18:41:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51061597999999997 2024-04-22T18:41:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060515+0000", 2024-04-22T18:41:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9130000000000001e-05 2024-04-22T18:41:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.611 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2198 4.d 4:bc165c76:::smithi08937929-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:41:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:41:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 74.975673689999994, 2024-04-22T18:41:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50502289899999997, 2024-04-22T18:41:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2198 2024-04-22T18:41:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:41:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:41:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551138+0000", 2024-04-22T18:41:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551143+0000", 2024-04-22T18:41:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8949999999999997e-06 2024-04-22T18:41:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551151+0000", 2024-04-22T18:41:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8729999999999996e-06 2024-04-22T18:41:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551158+0000", 2024-04-22T18:41:00.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3099999999999997e-06 2024-04-22T18:41:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551180+0000", 2024-04-22T18:41:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2197000000000001e-05 2024-04-22T18:41:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:41:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551185+0000", 2024-04-22T18:41:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7010000000000004e-06 2024-04-22T18:41:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657133+0000", 2024-04-22T18:41:00.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105948498 2024-04-22T18:41:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657183+0000", 2024-04-22T18:41:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0124000000000003e-05 2024-04-22T18:41:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.055924+0000", 2024-04-22T18:41:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39874079299999998 2024-04-22T18:41:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056018+0000", 2024-04-22T18:41:00.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4032999999999999e-05 2024-04-22T18:41:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056161+0000", 2024-04-22T18:41:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143385 2024-04-22T18:41:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2212 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:41:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:41:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 74.975604047999994, 2024-04-22T18:41:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50989936400000002, 2024-04-22T18:41:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2212 2024-04-22T18:41:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:41:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:41:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:41:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551211+0000", 2024-04-22T18:41:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6280000000000001e-06 2024-04-22T18:41:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551214+0000", 2024-04-22T18:41:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1209999999999998e-06 2024-04-22T18:41:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551217+0000", 2024-04-22T18:41:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8490000000000002e-06 2024-04-22T18:41:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061086+0000", 2024-04-22T18:41:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50986975099999998 2024-04-22T18:41:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061108+0000", 2024-04-22T18:41:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1319e-05 2024-04-22T18:41:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2213 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:41:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:41:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 74.975593630999995, 2024-04-22T18:41:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50998486099999996, 2024-04-22T18:41:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2213 2024-04-22T18:41:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:41:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:41:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551218+0000", 2024-04-22T18:41:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551220+0000", 2024-04-22T18:41:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.319e-06 2024-04-22T18:41:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551223+0000", 2024-04-22T18:41:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9660000000000001e-06 2024-04-22T18:41:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551226+0000", 2024-04-22T18:41:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9050000000000001e-06 2024-04-22T18:41:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061162+0000", 2024-04-22T18:41:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.509936838 2024-04-22T18:41:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061204+0000", 2024-04-22T18:41:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1094999999999999e-05 2024-04-22T18:41:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2214 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:41:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:41:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 74.975584642000001, 2024-04-22T18:41:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50997458100000004, 2024-04-22T18:41:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2214 2024-04-22T18:41:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:41:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:41:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551227+0000", 2024-04-22T18:41:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551229+0000", 2024-04-22T18:41:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8649999999999999e-06 2024-04-22T18:41:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551232+0000", 2024-04-22T18:41:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:41:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551235+0000", 2024-04-22T18:41:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:41:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061194+0000", 2024-04-22T18:41:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50995819099999995 2024-04-22T18:41:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061202+0000", 2024-04-22T18:41:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.541e-06 2024-04-22T18:41:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:00.630 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:00.630 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:41:00.631 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:00.632+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:00.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:00.667 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:41:00.731 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:00.732+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:00.745 INFO:tasks.rados.rados.0.smithi089.stdout:1709: finishing write tid 1 to smithi08937929-20 2024-04-22T18:41:00.745 INFO:tasks.rados.rados.0.smithi089.stdout:1709: finishing write tid 2 to smithi08937929-20 2024-04-22T18:41:00.745 INFO:tasks.rados.rados.0.smithi089.stdout:1709: finishing write tid 3 to smithi08937929-20 2024-04-22T18:41:00.745 INFO:tasks.rados.rados.0.smithi089.stdout:1709: finishing write tid 4 to smithi08937929-20 2024-04-22T18:41:00.745 INFO:tasks.rados.rados.0.smithi089.stdout:1696: done (15 left) 2024-04-22T18:41:00.745 INFO:tasks.rados.rados.0.smithi089.stdout:1702: done (14 left) 2024-04-22T18:41:00.745 INFO:tasks.rados.rados.0.smithi089.stdout:1703: done (13 left) 2024-04-22T18:41:00.746 INFO:tasks.rados.rados.0.smithi089.stdout:1704: done (12 left) 2024-04-22T18:41:00.746 INFO:tasks.rados.rados.0.smithi089.stdout:1705: done (11 left) 2024-04-22T18:41:00.746 INFO:tasks.rados.rados.0.smithi089.stdout:1706: done (10 left) 2024-04-22T18:41:00.746 INFO:tasks.rados.rados.0.smithi089.stdout:1707: done (9 left) 2024-04-22T18:41:00.746 INFO:tasks.rados.rados.0.smithi089.stdout:1717: read oid 43 snap -1 2024-04-22T18:41:00.746 INFO:tasks.rados.rados.0.smithi089.stdout:1717: expect (ObjNum 226 snap 0 seq_num 226) 2024-04-22T18:41:00.746 INFO:tasks.rados.rados.0.smithi089.stdout:1718: read oid 7 snap -1 2024-04-22T18:41:00.746 INFO:tasks.rados.rados.0.smithi089.stdout:1718: expect (ObjNum 346 snap 0 seq_num 346) 2024-04-22T18:41:00.746 INFO:tasks.rados.rados.0.smithi089.stdout:1719: set_chunk oid 17 target oid 2024-04-22T18:41:00.758 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:00.758 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:00.775 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:00.775 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:00.777 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:00.777 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:00.777 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:00.777 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:00.786 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:00.831 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:00.833+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:00.864 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:00.864 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:00.881 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:00.881 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:00.887 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:00.888 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:00.888 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:00.888 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:00.888 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:00.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:00.898 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:00.898 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:00.917 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:00.917 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:00.932 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:00.934+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:00.933 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:00.934 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:00.950 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:00.950 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:00.956 INFO:tasks.rados.rados.0.smithi089.stdout:1719: set_chunk oid 17 offset: 744448 length: 515584 target oid 7bf3cd45d61bac1e9393c615b0f73873204708717158f9d510de7b1d11ccab6a offset: 0 snap -1 2024-04-22T18:41:00.956 INFO:tasks.rados.rados.0.smithi089.stdout:1720: set_chunk oid 39 target oid 2024-04-22T18:41:00.999 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:00.999 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:01.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:01.033 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:01.035+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:01.110 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:01.111 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:01.114 INFO:tasks.rados.rados.0.smithi089.stdout:1720: set_chunk oid 39 offset: 1010688 length: 886272 target oid 8a0e05deebfbdbd6608e66b0485500bc332ef877ec40eb0edf416c9092444a3b offset: 0 snap -1 2024-04-22T18:41:01.114 INFO:tasks.rados.rados.0.smithi089.stdout:1721: write oid 40 current snap is 0 2024-04-22T18:41:01.114 INFO:tasks.rados.rados.0.smithi089.stdout:1709: finishing write tid 5 to smithi08937929-20 2024-04-22T18:41:01.114 INFO:tasks.rados.rados.0.smithi089.stdout:1709: oid 20 updating version 0 to 22 2024-04-22T18:41:01.114 INFO:tasks.rados.rados.0.smithi089.stdout:1709: oid 20 updating version 22 to 23 2024-04-22T18:41:01.114 INFO:tasks.rados.rados.0.smithi089.stdout:1709: oid 20 version 23 is already newer than 21 2024-04-22T18:41:01.114 INFO:tasks.rados.rados.0.smithi089.stdout:1709: oid 20 updating version 23 to 24 2024-04-22T18:41:01.114 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 24 (ObjNum 352 snap 0 seq_num 352) dirty exists 2024-04-22T18:41:01.115 INFO:tasks.rados.rados.0.smithi089.stdout:1709: left oid 20 (ObjNum 352 snap 0 seq_num 352) 2024-04-22T18:41:01.115 INFO:tasks.rados.rados.0.smithi089.stdout:1710: finishing set_chunk to oid 8 2024-04-22T18:41:01.115 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 33 (ObjNum 311 snap 0 seq_num 311) dirty exists 2024-04-22T18:41:01.115 INFO:tasks.rados.rados.0.smithi089.stdout:1712: finishing set_chunk to oid 47 2024-04-22T18:41:01.115 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 47 set_chunk returned error code -95 offset: 1432064 length: 1689600 2024-04-22T18:41:01.115 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 0 (ObjNum 323 snap 0 seq_num 323) dirty exists 2024-04-22T18:41:01.115 INFO:tasks.rados.rados.0.smithi089.stdout:1708: expect (ObjNum 208 snap 0 seq_num 208) 2024-04-22T18:41:01.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:01.163 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:01.164+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:01.221 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:01.221 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:01.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:01.236 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:01.237+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:01.305 INFO:tasks.rados.rados.0.smithi089.stdout:1713: finishing write tid 1 to smithi08937929-26 2024-04-22T18:41:01.305 INFO:tasks.rados.rados.0.smithi089.stdout:1713: finishing write tid 2 to smithi08937929-26 2024-04-22T18:41:01.305 INFO:tasks.rados.rados.0.smithi089.stdout:1713: finishing write tid 3 to smithi08937929-26 2024-04-22T18:41:01.305 INFO:tasks.rados.rados.0.smithi089.stdout:1713: finishing write tid 4 to smithi08937929-26 2024-04-22T18:41:01.305 INFO:tasks.rados.rados.0.smithi089.stdout:1713: finishing write tid 5 to smithi08937929-26 2024-04-22T18:41:01.305 INFO:tasks.rados.rados.0.smithi089.stdout:1713: oid 26 updating version 0 to 132 2024-04-22T18:41:01.305 INFO:tasks.rados.rados.0.smithi089.stdout:1713: oid 26 version 132 is already newer than 131 2024-04-22T18:41:01.305 INFO:tasks.rados.rados.0.smithi089.stdout:1713: oid 26 updating version 132 to 134 2024-04-22T18:41:01.305 INFO:tasks.rados.rados.0.smithi089.stdout:1713: oid 26 version 134 is already newer than 133 2024-04-22T18:41:01.305 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 134 (ObjNum 353 snap 0 seq_num 353) dirty exists 2024-04-22T18:41:01.305 INFO:tasks.rados.rados.0.smithi089.stdout:1713: left oid 26 (ObjNum 353 snap 0 seq_num 353) 2024-04-22T18:41:01.306 INFO:tasks.rados.rados.0.smithi089.stdout:1714: finishing set_chunk to oid 10 2024-04-22T18:41:01.306 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 44 (ObjNum 336 snap 0 seq_num 336) dirty exists 2024-04-22T18:41:01.306 INFO:tasks.rados.rados.0.smithi089.stdout:1716: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:41:01.337 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:01.338+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:01.346 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:01.346+0000 7fa766d7d640 -1 osd.5 126 handle_osd_map injecting map crc failure 2024-04-22T18:41:01.374 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:01.374 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:01.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:01.467 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:01.468+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:01.525 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:01.525 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:01.535 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:01.537 INFO:tasks.rados.rados.0.smithi089.stdout:1715: finishing set_chunk to oid 25 2024-04-22T18:41:01.537 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 80 (ObjNum 326 snap 0 seq_num 326) dirty exists 2024-04-22T18:41:01.537 INFO:tasks.rados.rados.0.smithi089.stdout:1717: expect (ObjNum 226 snap 0 seq_num 226) 2024-04-22T18:41:01.576 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:01.577+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:01.638 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:01.640+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:41:01.658 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:01.658+0000 7fa766d7d640 -1 osd.5 127 handle_osd_map injecting map crc failure 2024-04-22T18:41:01.662 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 7, 2] out_osds: [5, 4, 0, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:41:01.662 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:41:01.662 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-04-22T18:41:01.663 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-04-22T18:41:01.679 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:01.679 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:01.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:01.739 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:01.740+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:01.789 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:01.789 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:01.798 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:01.813 INFO:tasks.rados.rados.0.smithi089.stdout:1718: expect (ObjNum 346 snap 0 seq_num 346) 2024-04-22T18:41:01.840 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:01.842+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:01.901 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:01.901 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:01.911 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:01.952 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:01.953+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:01.976 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-22T18:41:01.977+0000 7f9b4bbc8640 -1 mon.a@0(leader).osd e128 definitely_dead 0 2024-04-22T18:41:02.032 INFO:tasks.rados.rados.0.smithi089.stdout:1720: finishing set_chunk to oid 39 2024-04-22T18:41:02.032 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 39 set_chunk returned error code -95 offset: 1010688 length: 886272 2024-04-22T18:41:02.032 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 0 (ObjNum 340 snap 0 seq_num 340) dirty exists 2024-04-22T18:41:02.032 INFO:tasks.rados.rados.0.smithi089.stdout:1719: finishing set_chunk to oid 17 2024-04-22T18:41:02.032 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 85 (ObjNum 321 snap 0 seq_num 321) dirty exists 2024-04-22T18:41:02.032 INFO:tasks.rados.rados.0.smithi089.stdout:1721: seq_num 354 ranges {749678=790384,2332155=504118,3271857=549469} 2024-04-22T18:41:02.040 INFO:tasks.rados.rados.0.smithi089.stdout:1721: writing smithi08937929-40 from 749678 to 1540062 tid 1 2024-04-22T18:41:02.042 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:02.043+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:02.047 INFO:tasks.rados.rados.0.smithi089.stdout:1721: writing smithi08937929-40 from 2332155 to 2836273 tid 2 2024-04-22T18:41:02.054 INFO:tasks.rados.rados.0.smithi089.stdout:1721: writing smithi08937929-40 from 3271857 to 3821326 tid 3 2024-04-22T18:41:02.054 INFO:tasks.rados.rados.0.smithi089.stdout:1708: done (13 left) 2024-04-22T18:41:02.054 INFO:tasks.rados.rados.0.smithi089.stdout:1709: done (12 left) 2024-04-22T18:41:02.054 INFO:tasks.rados.rados.0.smithi089.stdout:1710: done (11 left) 2024-04-22T18:41:02.054 INFO:tasks.rados.rados.0.smithi089.stdout:1711: done (10 left) 2024-04-22T18:41:02.054 INFO:tasks.rados.rados.0.smithi089.stdout:1712: done (9 left) 2024-04-22T18:41:02.054 INFO:tasks.rados.rados.0.smithi089.stdout:1713: done (8 left) 2024-04-22T18:41:02.054 INFO:tasks.rados.rados.0.smithi089.stdout:1714: done (7 left) 2024-04-22T18:41:02.055 INFO:tasks.rados.rados.0.smithi089.stdout:1715: done (6 left) 2024-04-22T18:41:02.055 INFO:tasks.rados.rados.0.smithi089.stdout:1716: done (5 left) 2024-04-22T18:41:02.055 INFO:tasks.rados.rados.0.smithi089.stdout:1717: done (4 left) 2024-04-22T18:41:02.055 INFO:tasks.rados.rados.0.smithi089.stdout:1718: done (3 left) 2024-04-22T18:41:02.055 INFO:tasks.rados.rados.0.smithi089.stdout:1719: done (2 left) 2024-04-22T18:41:02.055 INFO:tasks.rados.rados.0.smithi089.stdout:1720: done (1 left) 2024-04-22T18:41:02.055 INFO:tasks.rados.rados.0.smithi089.stdout:1722: read oid 37 snap -1 2024-04-22T18:41:02.055 INFO:tasks.rados.rados.0.smithi089.stdout:1722: expect (ObjNum 267 snap 0 seq_num 267) 2024-04-22T18:41:02.055 INFO:tasks.rados.rados.0.smithi089.stdout:1723: read oid 9 snap -1 2024-04-22T18:41:02.055 INFO:tasks.rados.rados.0.smithi089.stdout:1723: expect (ObjNum 283 snap 0 seq_num 283) 2024-04-22T18:41:02.055 INFO:tasks.rados.rados.0.smithi089.stdout:1724: read oid 42 snap -1 2024-04-22T18:41:02.055 INFO:tasks.rados.rados.0.smithi089.stdout:1724: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:41:02.055 INFO:tasks.rados.rados.0.smithi089.stdout:1725: tier_evict oid 41 2024-04-22T18:41:02.055 INFO:tasks.rados.rados.0.smithi089.stdout:1725: tier_evict oid 41 snap -1 2024-04-22T18:41:02.056 INFO:tasks.rados.rados.0.smithi089.stdout:1726: write (excl) oid 28 current snap is 0 2024-04-22T18:41:02.056 INFO:tasks.rados.rados.0.smithi089.stdout:1726: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2024-04-22T18:41:02.056 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:02.056 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:02.059 INFO:tasks.rados.rados.0.smithi089.stdout:1726: writing smithi08937929-28 from 766485 to 1232118 tid 1 2024-04-22T18:41:02.065 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:02.067 INFO:tasks.rados.rados.0.smithi089.stdout:1726: writing smithi08937929-28 from 1990637 to 2638413 tid 2 2024-04-22T18:41:02.074 INFO:tasks.rados.rados.0.smithi089.stdout:1726: writing smithi08937929-28 from 3299280 to 3966485 tid 3 2024-04-22T18:41:02.074 INFO:tasks.rados.rados.0.smithi089.stdout:1727: write (excl) oid 8 current snap is 0 2024-04-22T18:41:02.074 INFO:tasks.rados.rados.0.smithi089.stdout:1727: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2024-04-22T18:41:02.081 INFO:tasks.rados.rados.0.smithi089.stdout:1727: writing smithi08937929-8 from 783292 to 1324174 tid 1 2024-04-22T18:41:02.087 INFO:tasks.rados.rados.0.smithi089.stdout:1727: writing smithi08937929-8 from 1932766 to 2440553 tid 2 2024-04-22T18:41:02.095 INFO:tasks.rados.rados.0.smithi089.stdout:1727: writing smithi08937929-8 from 3210350 to 3782363 tid 3 2024-04-22T18:41:02.095 INFO:tasks.rados.rados.0.smithi089.stdout:1728: read oid 10 snap -1 2024-04-22T18:41:02.096 INFO:tasks.rados.rados.0.smithi089.stdout:1728: expect (ObjNum 336 snap 0 seq_num 336) 2024-04-22T18:41:02.096 INFO:tasks.rados.rados.0.smithi089.stdout:1729: read oid 18 snap -1 2024-04-22T18:41:02.096 INFO:tasks.rados.rados.0.smithi089.stdout:1729: expect (ObjNum 307 snap 0 seq_num 307) 2024-04-22T18:41:02.096 INFO:tasks.rados.rados.0.smithi089.stdout:1730: read oid 12 snap -1 2024-04-22T18:41:02.096 INFO:tasks.rados.rados.0.smithi089.stdout:1730: expect (ObjNum 263 snap 0 seq_num 263) 2024-04-22T18:41:02.096 INFO:tasks.rados.rados.0.smithi089.stdout:1731: delete oid 45 current snap is 0 2024-04-22T18:41:02.098 INFO:tasks.rados.rados.0.smithi089.stdout:1730: expect (ObjNum 263 snap 0 seq_num 263) 2024-04-22T18:41:02.143 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:02.144+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:02.176 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:02.176 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:02.186 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:02.244 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:02.245+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:02.246 INFO:tasks.rados.rados.0.smithi089.stdout:1721: finishing write tid 1 to smithi08937929-40 2024-04-22T18:41:02.246 INFO:tasks.rados.rados.0.smithi089.stdout:1721: finishing write tid 2 to smithi08937929-40 2024-04-22T18:41:02.246 INFO:tasks.rados.rados.0.smithi089.stdout:1726: finishing write tid 1 to smithi08937929-28 2024-04-22T18:41:02.246 INFO:tasks.rados.rados.0.smithi089.stdout:1729: expect (ObjNum 307 snap 0 seq_num 307) 2024-04-22T18:41:02.329 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:02.329 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:02.338 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:02.380 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:02.381+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:02.446 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:02.447+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:02.451 INFO:tasks.rados.rados.0.smithi089.stdout:1721: finishing write tid 3 to smithi08937929-40 2024-04-22T18:41:02.451 INFO:tasks.rados.rados.0.smithi089.stdout:1721: finishing write tid 4 to smithi08937929-40 2024-04-22T18:41:02.452 INFO:tasks.rados.rados.0.smithi089.stdout:1721: finishing write tid 5 to smithi08937929-40 2024-04-22T18:41:02.452 INFO:tasks.rados.rados.0.smithi089.stdout:1721: oid 40 updating version 0 to 64 2024-04-22T18:41:02.452 INFO:tasks.rados.rados.0.smithi089.stdout:1721: oid 40 version 64 is already newer than 63 2024-04-22T18:41:02.452 INFO:tasks.rados.rados.0.smithi089.stdout:1721: oid 40 version 64 is already newer than 62 2024-04-22T18:41:02.452 INFO:tasks.rados.rados.0.smithi089.stdout:1721: oid 40 updating version 64 to 65 2024-04-22T18:41:02.452 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 65 (ObjNum 354 snap 0 seq_num 354) dirty exists 2024-04-22T18:41:02.452 INFO:tasks.rados.rados.0.smithi089.stdout:1721: left oid 40 (ObjNum 354 snap 0 seq_num 354) 2024-04-22T18:41:02.452 INFO:tasks.rados.rados.0.smithi089.stdout:1726: finishing write tid 2 to smithi08937929-28 2024-04-22T18:41:02.452 INFO:tasks.rados.rados.0.smithi089.stdout:1726: finishing write tid 3 to smithi08937929-28 2024-04-22T18:41:02.452 INFO:tasks.rados.rados.0.smithi089.stdout:1726: finishing write tid 4 to smithi08937929-28 2024-04-22T18:41:02.452 INFO:tasks.rados.rados.0.smithi089.stdout:1726: finishing write tid 5 to smithi08937929-28 2024-04-22T18:41:02.452 INFO:tasks.rados.rados.0.smithi089.stdout:1726: oid 28 updating version 0 to 162 2024-04-22T18:41:02.452 INFO:tasks.rados.rados.0.smithi089.stdout:1726: oid 28 updating version 162 to 165 2024-04-22T18:41:02.452 INFO:tasks.rados.rados.0.smithi089.stdout:1726: oid 28 version 165 is already newer than 164 2024-04-22T18:41:02.452 INFO:tasks.rados.rados.0.smithi089.stdout:1726: oid 28 version 165 is already newer than 163 2024-04-22T18:41:02.453 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 165 (ObjNum 355 snap 0 seq_num 355) dirty exists 2024-04-22T18:41:02.453 INFO:tasks.rados.rados.0.smithi089.stdout:1726: left oid 28 (ObjNum 355 snap 0 seq_num 355) 2024-04-22T18:41:02.453 INFO:tasks.rados.rados.0.smithi089.stdout:1723: expect (ObjNum 283 snap 0 seq_num 283) 2024-04-22T18:41:02.483 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:02.483 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:02.493 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:02.576 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:02.577+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:02.618 INFO:tasks.rados.rados.0.smithi089.stdout:1722: expect (ObjNum 267 snap 0 seq_num 267) 2024-04-22T18:41:02.635 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:02.635 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:02.644 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:02.648 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:02.650+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:02.731 INFO:tasks.rados.rados.0.smithi089.stdout:1724: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:41:02.749 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:02.751+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:02.787 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:02.787 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:02.796 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:02.878 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:02.880+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:02.915 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:02.916+0000 7f3d00e8c640 -1 osd.6 128 handle_osd_map injecting map crc failure 2024-04-22T18:41:02.916 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:02.918+0000 7f3e7be88640 -1 osd.0 128 handle_osd_map injecting map crc failure 2024-04-22T18:41:02.919 INFO:teuthology.orchestra.run.smithi089.stderr:marked in osd.4. 2024-04-22T18:41:02.920 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:02.921+0000 7fec55bea640 -1 osd.7 128 handle_osd_map injecting map crc failure 2024-04-22T18:41:02.934 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-04-22T18:41:02.943 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:02.943 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:02.951 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:02.952+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:02.952 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:02.960 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:41:02.963 INFO:tasks.rados.rados.0.smithi089.stdout:1725: got (22) Invalid argument 2024-04-22T18:41:02.963 INFO:tasks.rados.rados.0.smithi089.stdout:1721: done (10 left) 2024-04-22T18:41:02.963 INFO:tasks.rados.rados.0.smithi089.stdout:1722: done (9 left) 2024-04-22T18:41:02.963 INFO:tasks.rados.rados.0.smithi089.stdout:1723: done (8 left) 2024-04-22T18:41:02.963 INFO:tasks.rados.rados.0.smithi089.stdout:1724: done (7 left) 2024-04-22T18:41:02.964 INFO:tasks.rados.rados.0.smithi089.stdout:1725: done (6 left) 2024-04-22T18:41:02.964 INFO:tasks.rados.rados.0.smithi089.stdout:1726: done (5 left) 2024-04-22T18:41:02.964 INFO:tasks.rados.rados.0.smithi089.stdout:1729: done (4 left) 2024-04-22T18:41:02.964 INFO:tasks.rados.rados.0.smithi089.stdout:1730: done (3 left) 2024-04-22T18:41:02.964 INFO:tasks.rados.rados.0.smithi089.stdout:1731: done (2 left) 2024-04-22T18:41:02.964 INFO:tasks.rados.rados.0.smithi089.stdout:1732: read oid 27 snap -1 2024-04-22T18:41:02.964 INFO:tasks.rados.rados.0.smithi089.stdout:1732: expect (ObjNum 287 snap 0 seq_num 287) 2024-04-22T18:41:02.964 INFO:tasks.rados.rados.0.smithi089.stdout:1733: read oid 49 snap -1 2024-04-22T18:41:02.964 INFO:tasks.rados.rados.0.smithi089.stdout:1733: expect (ObjNum 316 snap 0 seq_num 316) 2024-04-22T18:41:02.964 INFO:tasks.rados.rados.0.smithi089.stdout:1734: read oid 12 snap -1 2024-04-22T18:41:02.964 INFO:tasks.rados.rados.0.smithi089.stdout:1734: expect (ObjNum 263 snap 0 seq_num 263) 2024-04-22T18:41:02.964 INFO:tasks.rados.rados.0.smithi089.stdout:1735: read oid 32 snap -1 2024-04-22T18:41:02.964 INFO:tasks.rados.rados.0.smithi089.stdout:1735: expect deleted 2024-04-22T18:41:02.964 INFO:tasks.rados.rados.0.smithi089.stdout:1736: read oid 20 snap -1 2024-04-22T18:41:02.965 INFO:tasks.rados.rados.0.smithi089.stdout:1736: expect (ObjNum 352 snap 0 seq_num 352) 2024-04-22T18:41:02.965 INFO:tasks.rados.rados.0.smithi089.stdout:1737: read oid 35 snap -1 2024-04-22T18:41:02.965 INFO:tasks.rados.rados.0.smithi089.stdout:1737: expect (ObjNum 304 snap 0 seq_num 304) 2024-04-22T18:41:02.965 INFO:tasks.rados.rados.0.smithi089.stdout:1738: tier_flush oid 43 2024-04-22T18:41:02.965 INFO:tasks.rados.rados.0.smithi089.stdout:1738: tier_flush oid 43 snap -1 2024-04-22T18:41:02.965 INFO:tasks.rados.rados.0.smithi089.stdout:1739: read oid 29 snap -1 2024-04-22T18:41:02.965 INFO:tasks.rados.rados.0.smithi089.stdout:1739: expect (ObjNum 295 snap 0 seq_num 295) 2024-04-22T18:41:02.965 INFO:tasks.rados.rados.0.smithi089.stdout:1740: read oid 14 snap -1 2024-04-22T18:41:02.965 INFO:tasks.rados.rados.0.smithi089.stdout:1740: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:41:02.965 INFO:tasks.rados.rados.0.smithi089.stdout:1741: tier_promote oid 25 2024-04-22T18:41:02.965 INFO:tasks.rados.rados.0.smithi089.stdout:1742: write oid 41 current snap is 0 2024-04-22T18:41:02.965 INFO:tasks.rados.rados.0.smithi089.stdout:1742: seq_num 358 ranges {416906=407733,1366083=511456} 2024-04-22T18:41:02.966 INFO:tasks.rados.rados.0.smithi089.stdout:1742: writing smithi08937929-41 from 416906 to 824639 tid 1 2024-04-22T18:41:02.971 INFO:tasks.rados.rados.0.smithi089.stdout:1742: writing smithi08937929-41 from 1366083 to 1877539 tid 2 2024-04-22T18:41:02.972 INFO:tasks.rados.rados.0.smithi089.stdout:1743: set_chunk oid 28 target oid 2024-04-22T18:41:03.052 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:03.053+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:03.071 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:03.071 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:03.080 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:03.087 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:03.105 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:03.106 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:03.123 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:03.123 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:03.139 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:03.140 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:03.153 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:03.154+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:03.156 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:03.157 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:03.173 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:03.173 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:03.187 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:03.187 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:03.187 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:03.190 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:03.191 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:03.196 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:03.207 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:03.208 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:03.254 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:03.255+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:03.310 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:03.310 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:03.310 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:03.310 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:03.310 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:03.319 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:03.355 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:03.356+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:03.423 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:03.423 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:03.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:03.423 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:03.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.424 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:289 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:03.424 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:03.424 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.782287599, 2024-04-22T18:41:03.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9176801189999999, 2024-04-22T18:41:03.424 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.424 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.424 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:03.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:03.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640723+0000", 2024-04-22T18:41:03.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:03.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640732+0000", 2024-04-22T18:41:03.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5809999999999997e-06 2024-04-22T18:41:03.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640744+0000", 2024-04-22T18:41:03.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.261e-05 2024-04-22T18:41:03.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640753+0000", 2024-04-22T18:41:03.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7460000000000006e-06 2024-04-22T18:41:03.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642189+0000", 2024-04-22T18:41:03.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001435551 2024-04-22T18:41:03.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642200+0000", 2024-04-22T18:41:03.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1389999999999999e-05 2024-04-22T18:41:03.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.847015+0000", 2024-04-22T18:41:03.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048149160000001 2024-04-22T18:41:03.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:03.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.848217+0000", 2024-04-22T18:41:03.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012020659999999999 2024-04-22T18:41:03.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351823+0000", 2024-04-22T18:41:03.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50360565099999999 2024-04-22T18:41:03.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351907+0000", 2024-04-22T18:41:03.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4141999999999995e-05 2024-04-22T18:41:03.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:03.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352051+0000", 2024-04-22T18:41:03.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014386600000000001 2024-04-22T18:41:03.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453617+0000", 2024-04-22T18:41:03.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156649500000001 2024-04-22T18:41:03.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558297+0000", 2024-04-22T18:41:03.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10468000399999999 2024-04-22T18:41:03.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558331+0000", 2024-04-22T18:41:03.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3518e-05 2024-04-22T18:41:03.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558405+0000", 2024-04-22T18:41:03.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4221000000000006e-05 2024-04-22T18:41:03.430 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.430 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.430 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.430 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:1 3.2 3:6d90b903:::64b5942574fb36ca1b9a2be186c6b740edf58e989056a2f88b83cd280451a438:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:03.430 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:03.430 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.250986638999997, 2024-04-22T18:41:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.285534422, 2024-04-22T18:41:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172024+0000", 2024-04-22T18:41:03.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:03.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173091+0000", 2024-04-22T18:41:03.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010677569999999999 2024-04-22T18:41:03.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173103+0000", 2024-04-22T18:41:03.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1241e-05 2024-04-22T18:41:03.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173109+0000", 2024-04-22T18:41:03.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4300000000000003e-06 2024-04-22T18:41:03.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173168+0000", 2024-04-22T18:41:03.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9011000000000001e-05 2024-04-22T18:41:03.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173177+0000", 2024-04-22T18:41:03.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-04-22T18:41:03.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948501+0000", 2024-04-22T18:41:03.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77532387199999997 2024-04-22T18:41:03.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:03.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948912+0000", 2024-04-22T18:41:03.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041183000000000002 2024-04-22T18:41:03.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950438+0000", 2024-04-22T18:41:03.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015253199999999999 2024-04-22T18:41:03.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950488+0000", 2024-04-22T18:41:03.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0285999999999999e-05 2024-04-22T18:41:03.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:03.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950601+0000", 2024-04-22T18:41:03.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011244899999999999 2024-04-22T18:41:03.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954048+0000", 2024-04-22T18:41:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003447313 2024-04-22T18:41:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457373+0000", 2024-04-22T18:41:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50332465299999996 2024-04-22T18:41:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457430+0000", 2024-04-22T18:41:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7574999999999997e-05 2024-04-22T18:41:03.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457560+0000", 2024-04-22T18:41:03.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013023900000000001 2024-04-22T18:41:03.437 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.437 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.437 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.437 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:8 3.2 3:5f64590d:::7c47933f331d40689081b2bd8f9d7f7bdae1be1fdccb52d696047a3969e902b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:03.437 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:03.437 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.249894269999999, 2024-04-22T18:41:03.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3854200569999999, 2024-04-22T18:41:03.437 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.438 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.438 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:03.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:03.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173117+0000", 2024-04-22T18:41:03.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:03.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173393+0000", 2024-04-22T18:41:03.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000275278 2024-04-22T18:41:03.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173399+0000", 2024-04-22T18:41:03.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4590000000000003e-06 2024-04-22T18:41:03.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173406+0000", 2024-04-22T18:41:03.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2450000000000001e-06 2024-04-22T18:41:03.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173795+0000", 2024-04-22T18:41:03.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038912400000000002 2024-04-22T18:41:03.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173804+0000", 2024-04-22T18:41:03.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-04-22T18:41:03.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.949506+0000", 2024-04-22T18:41:03.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77570185700000005 2024-04-22T18:41:03.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:03.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950166+0000", 2024-04-22T18:41:03.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066022800000000001 2024-04-22T18:41:03.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352843+0000", 2024-04-22T18:41:03.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40267661799999999 2024-04-22T18:41:03.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352911+0000", 2024-04-22T18:41:03.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8126000000000005e-05 2024-04-22T18:41:03.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:03.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.353031+0000", 2024-04-22T18:41:03.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119919 2024-04-22T18:41:03.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453622+0000", 2024-04-22T18:41:03.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100590793 2024-04-22T18:41:03.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558446+0000", 2024-04-22T18:41:03.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104824073 2024-04-22T18:41:03.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558479+0000", 2024-04-22T18:41:03.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3362999999999999e-05 2024-04-22T18:41:03.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558538+0000", 2024-04-22T18:41:03.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9046999999999999e-05 2024-04-22T18:41:03.444 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.444 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.444 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.444 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:10 3.2 3:518c37f9:::851eca84dafbf2f780c4f07291110c453349bdc9c623c17bdcd26e27badb30ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:03.444 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:03.444 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.249600676999997, 2024-04-22T18:41:03.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3852527729999999, 2024-04-22T18:41:03.444 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.444 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.444 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:03.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:03.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173411+0000", 2024-04-22T18:41:03.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:03.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173749+0000", 2024-04-22T18:41:03.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000338029 2024-04-22T18:41:03.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173755+0000", 2024-04-22T18:41:03.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0530000000000003e-06 2024-04-22T18:41:03.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173759+0000", 2024-04-22T18:41:03.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8190000000000002e-06 2024-04-22T18:41:03.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173855+0000", 2024-04-22T18:41:03.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5927999999999994e-05 2024-04-22T18:41:03.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173862+0000", 2024-04-22T18:41:03.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4449999999999999e-06 2024-04-22T18:41:03.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950835+0000", 2024-04-22T18:41:03.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77697248299999999 2024-04-22T18:41:03.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:03.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951632+0000", 2024-04-22T18:41:03.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079752299999999996 2024-04-22T18:41:03.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355407+0000", 2024-04-22T18:41:03.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40377480500000001 2024-04-22T18:41:03.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355462+0000", 2024-04-22T18:41:03.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4577999999999998e-05 2024-04-22T18:41:03.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:03.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355571+0000", 2024-04-22T18:41:03.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108964 2024-04-22T18:41:03.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453625+0000", 2024-04-22T18:41:03.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098054397000000001 2024-04-22T18:41:03.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558574+0000", 2024-04-22T18:41:03.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104948571 2024-04-22T18:41:03.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558602+0000", 2024-04-22T18:41:03.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7966999999999998e-05 2024-04-22T18:41:03.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558665+0000", 2024-04-22T18:41:03.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3032000000000002e-05 2024-04-22T18:41:03.451 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.451 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.451 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.451 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:11 3.2 3:718aa8aa:::9603c4f8e269a436b2f0bf376c5374e77bdd7cf58fccae60e4e65ac4696806d4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:03.451 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:03.451 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.249249059, 2024-04-22T18:41:03.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4852604440000001, 2024-04-22T18:41:03.451 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.451 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.451 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:03.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:03.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:03.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173813+0000", 2024-04-22T18:41:03.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0655999999999997e-05 2024-04-22T18:41:03.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173819+0000", 2024-04-22T18:41:03.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.468e-06 2024-04-22T18:41:03.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173822+0000", 2024-04-22T18:41:03.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4879999999999999e-06 2024-04-22T18:41:03.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173983+0000", 2024-04-22T18:41:03.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160519 2024-04-22T18:41:03.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173991+0000", 2024-04-22T18:41:03.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6490000000000008e-06 2024-04-22T18:41:03.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951688+0000", 2024-04-22T18:41:03.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77769743800000002 2024-04-22T18:41:03.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:03.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952419+0000", 2024-04-22T18:41:03.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073067499999999997 2024-04-22T18:41:03.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355901+0000", 2024-04-22T18:41:03.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40348237599999998 2024-04-22T18:41:03.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355953+0000", 2024-04-22T18:41:03.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1477000000000002e-05 2024-04-22T18:41:03.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:03.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356060+0000", 2024-04-22T18:41:03.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107114 2024-04-22T18:41:03.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453629+0000", 2024-04-22T18:41:03.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097569127000000005 2024-04-22T18:41:03.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658807+0000", 2024-04-22T18:41:03.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20517829800000001 2024-04-22T18:41:03.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658876+0000", 2024-04-22T18:41:03.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8454000000000006e-05 2024-04-22T18:41:03.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659024+0000", 2024-04-22T18:41:03.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148338 2024-04-22T18:41:03.458 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.458 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.458 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.458 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:12 3.2 3:475e1cd9:::efaab2bd13543cdbb143e869e20eb0ec424f7f5202707859b241ed91d4674936:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:03.458 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:03.458 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.249186115000001, 2024-04-22T18:41:03.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4853405259999999, 2024-04-22T18:41:03.458 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.458 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.458 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:03.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:03.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:03.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173940+0000", 2024-04-22T18:41:03.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011465500000000001 2024-04-22T18:41:03.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173946+0000", 2024-04-22T18:41:03.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5289999999999999e-06 2024-04-22T18:41:03.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173950+0000", 2024-04-22T18:41:03.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8290000000000001e-06 2024-04-22T18:41:03.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174277+0000", 2024-04-22T18:41:03.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032761299999999998 2024-04-22T18:41:03.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174285+0000", 2024-04-22T18:41:03.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.52e-06 2024-04-22T18:41:03.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952455+0000", 2024-04-22T18:41:03.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77817000700000005 2024-04-22T18:41:03.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:03.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954035+0000", 2024-04-22T18:41:03.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001580078 2024-04-22T18:41:03.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356379+0000", 2024-04-22T18:41:03.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40234440399999999 2024-04-22T18:41:03.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356428+0000", 2024-04-22T18:41:03.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8832999999999998e-05 2024-04-22T18:41:03.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:03.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356532+0000", 2024-04-22T18:41:03.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103235 2024-04-22T18:41:03.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453632+0000", 2024-04-22T18:41:03.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097100346000000004 2024-04-22T18:41:03.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659069+0000", 2024-04-22T18:41:03.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20543763800000001 2024-04-22T18:41:03.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659105+0000", 2024-04-22T18:41:03.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5240000000000001e-05 2024-04-22T18:41:03.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659167+0000", 2024-04-22T18:41:03.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2207999999999998e-05 2024-04-22T18:41:03.464 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.464 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:16 3.2 3:47c76cc8:::c00d18458eb20c0c0aa9c612cf9793a68445ab0ea7b9e650ecd0072c60a96c0f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.249058155999997, 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.485347744, 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:03.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:03.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:03.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174234+0000", 2024-04-22T18:41:03.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028047599999999999 2024-04-22T18:41:03.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174240+0000", 2024-04-22T18:41:03.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2079999999999997e-06 2024-04-22T18:41:03.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174244+0000", 2024-04-22T18:41:03.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5810000000000001e-06 2024-04-22T18:41:03.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174670+0000", 2024-04-22T18:41:03.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042640000000000001 2024-04-22T18:41:03.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174675+0000", 2024-04-22T18:41:03.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8709999999999996e-06 2024-04-22T18:41:03.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054168+0000", 2024-04-22T18:41:03.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87949257300000006 2024-04-22T18:41:03.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:03.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054877+0000", 2024-04-22T18:41:03.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070884300000000002 2024-04-22T18:41:03.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356836+0000", 2024-04-22T18:41:03.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30195926000000001 2024-04-22T18:41:03.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356887+0000", 2024-04-22T18:41:03.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0767999999999999e-05 2024-04-22T18:41:03.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:03.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356991+0000", 2024-04-22T18:41:03.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104072 2024-04-22T18:41:03.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453636+0000", 2024-04-22T18:41:03.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096644968999999997 2024-04-22T18:41:03.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659207+0000", 2024-04-22T18:41:03.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20557154699999999 2024-04-22T18:41:03.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659240+0000", 2024-04-22T18:41:03.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2665e-05 2024-04-22T18:41:03.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659302+0000", 2024-04-22T18:41:03.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2234000000000001e-05 2024-04-22T18:41:03.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.471 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.471 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:17 3.2 3:6ef31daa:::1cf4ff496c4480e58510f607481d976b47d84868e2c54016273c38baf590f3b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:03.471 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:03.471 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.248763875999998, 2024-04-22T18:41:03.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4851703729999999, 2024-04-22T18:41:03.472 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.472 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.472 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:03.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:03.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174248+0000", 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174626+0000", 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037839699999999999 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174633+0000", 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0990000000000004e-06 2024-04-22T18:41:03.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174636+0000", 2024-04-22T18:41:03.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5540000000000002e-06 2024-04-22T18:41:03.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174790+0000", 2024-04-22T18:41:03.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153807 2024-04-22T18:41:03.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174793+0000", 2024-04-22T18:41:03.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4970000000000001e-06 2024-04-22T18:41:03.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054902+0000", 2024-04-22T18:41:03.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88010886300000002 2024-04-22T18:41:03.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:03.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055366+0000", 2024-04-22T18:41:03.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046336199999999999 2024-04-22T18:41:03.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.452973+0000", 2024-04-22T18:41:03.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39760783999999999 2024-04-22T18:41:03.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453046+0000", 2024-04-22T18:41:03.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2374999999999997e-05 2024-04-22T18:41:03.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:03.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453193+0000", 2024-04-22T18:41:03.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014722499999999999 2024-04-22T18:41:03.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554698+0000", 2024-04-22T18:41:03.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101505096 2024-04-22T18:41:03.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659337+0000", 2024-04-22T18:41:03.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104639236 2024-04-22T18:41:03.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659365+0000", 2024-04-22T18:41:03.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7594e-05 2024-04-22T18:41:03.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659419+0000", 2024-04-22T18:41:03.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4058999999999999e-05 2024-04-22T18:41:03.478 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.478 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.478 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.478 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:18 3.2 3:792a4ed0:::e1c814961b3d1531fb5ed475b3d496a0405c3a819b68a92ff9b2391fe912370f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:03.478 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:03.478 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.248367620000003, 2024-04-22T18:41:03.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.585036954, 2024-04-22T18:41:03.478 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.478 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.478 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:03.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:03.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174640+0000", 2024-04-22T18:41:03.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-04-22T18:41:03.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174758+0000", 2024-04-22T18:41:03.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011819099999999999 2024-04-22T18:41:03.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174764+0000", 2024-04-22T18:41:03.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6729999999999997e-06 2024-04-22T18:41:03.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174767+0000", 2024-04-22T18:41:03.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3830000000000001e-06 2024-04-22T18:41:03.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373669+0000", 2024-04-22T18:41:03.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198901989 2024-04-22T18:41:03.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373686+0000", 2024-04-22T18:41:03.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7286e-05 2024-04-22T18:41:03.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055408+0000", 2024-04-22T18:41:03.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68172133400000001 2024-04-22T18:41:03.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:03.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055938+0000", 2024-04-22T18:41:03.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053016599999999999 2024-04-22T18:41:03.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453668+0000", 2024-04-22T18:41:03.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39773030799999998 2024-04-22T18:41:03.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453717+0000", 2024-04-22T18:41:03.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8884000000000002e-05 2024-04-22T18:41:03.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:03.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453823+0000", 2024-04-22T18:41:03.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105791 2024-04-22T18:41:03.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554715+0000", 2024-04-22T18:41:03.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100891628 2024-04-22T18:41:03.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759439+0000", 2024-04-22T18:41:03.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20472459400000001 2024-04-22T18:41:03.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759525+0000", 2024-04-22T18:41:03.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5653000000000001e-05 2024-04-22T18:41:03.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759682+0000", 2024-04-22T18:41:03.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015711800000000001 2024-04-22T18:41:03.484 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.485 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.485 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.485 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:75 3.1 3:ab43a94d:::f38e6e97cb24e2b90da964fee5d2d6d8674386870fc552915e0c3fdbefd4b126:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:03.485 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:03.485 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.175491102999999, 2024-04-22T18:41:03.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60149928399999997, 2024-04-22T18:41:03.485 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.485 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.485 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247520+0000", 2024-04-22T18:41:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247527+0000", 2024-04-22T18:41:03.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1670000000000002e-06 2024-04-22T18:41:03.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247539+0000", 2024-04-22T18:41:03.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503e-05 2024-04-22T18:41:03.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247549+0000", 2024-04-22T18:41:03.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0000000000000001e-05 2024-04-22T18:41:03.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247796+0000", 2024-04-22T18:41:03.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024712699999999998 2024-04-22T18:41:03.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:03.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247955+0000", 2024-04-22T18:41:03.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158534 2024-04-22T18:41:03.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.343985+0000", 2024-04-22T18:41:03.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096030344000000004 2024-04-22T18:41:03.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:03.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.344035+0000", 2024-04-22T18:41:03.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9969999999999998e-05 2024-04-22T18:41:03.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.745806+0000", 2024-04-22T18:41:03.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.401771029 2024-04-22T18:41:03.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746040+0000", 2024-04-22T18:41:03.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023339300000000001 2024-04-22T18:41:03.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:03.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746198+0000", 2024-04-22T18:41:03.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015793299999999999 2024-04-22T18:41:03.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848737+0000", 2024-04-22T18:41:03.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10253898 2024-04-22T18:41:03.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848751+0000", 2024-04-22T18:41:03.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4694e-05 2024-04-22T18:41:03.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848825+0000", 2024-04-22T18:41:03.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3443999999999997e-05 2024-04-22T18:41:03.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849021+0000", 2024-04-22T18:41:03.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000196047 2024-04-22T18:41:03.491 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.491 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.491 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.491 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:76 3.1 3:b44829a0:::3788286a255d538503614e5f04049b0bb54cacaa317ab102b5dee8f5b640b296:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:03.492 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:03.492 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.175452036999999, 2024-04-22T18:41:03.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60168681599999996, 2024-04-22T18:41:03.492 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.492 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.492 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:03.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:03.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:03.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:03.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247564+0000", 2024-04-22T18:41:03.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1150000000000004e-06 2024-04-22T18:41:03.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247570+0000", 2024-04-22T18:41:03.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3940000000000001e-06 2024-04-22T18:41:03.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247576+0000", 2024-04-22T18:41:03.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3619999999999997e-06 2024-04-22T18:41:03.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.340845+0000", 2024-04-22T18:41:03.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.093269199999999997 2024-04-22T18:41:03.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:03.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.341043+0000", 2024-04-22T18:41:03.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019834400000000001 2024-04-22T18:41:03.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345106+0000", 2024-04-22T18:41:03.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040629780000000001 2024-04-22T18:41:03.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:03.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345160+0000", 2024-04-22T18:41:03.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4246999999999997e-05 2024-04-22T18:41:03.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747047+0000", 2024-04-22T18:41:03.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40188678300000003 2024-04-22T18:41:03.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747162+0000", 2024-04-22T18:41:03.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011483400000000001 2024-04-22T18:41:03.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:03.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747429+0000", 2024-04-22T18:41:03.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026700299999999998 2024-04-22T18:41:03.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849032+0000", 2024-04-22T18:41:03.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101603203 2024-04-22T18:41:03.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849095+0000", 2024-04-22T18:41:03.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2597999999999999e-05 2024-04-22T18:41:03.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849136+0000", 2024-04-22T18:41:03.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1251000000000002e-05 2024-04-22T18:41:03.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849247+0000", 2024-04-22T18:41:03.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011127 2024-04-22T18:41:03.498 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.498 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.498 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.498 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3524 4.c 4:314c2fec:::smithi08937929-22:head [write 692521~538214] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:41:03.498 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.128717+0000", 2024-04-22T18:41:03.498 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.294295942000002, 2024-04-22T18:41:03.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40275804199999998, 2024-04-22T18:41:03.498 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.498 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.498 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:03.499 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:03.499 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.499 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3524 2024-04-22T18:41:03.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.499 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128717+0000", 2024-04-22T18:41:03.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128717+0000", 2024-04-22T18:41:03.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128715+0000", 2024-04-22T18:41:03.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:03.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128877+0000", 2024-04-22T18:41:03.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016187999999999999 2024-04-22T18:41:03.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.128886+0000", 2024-04-22T18:41:03.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8780000000000004e-06 2024-04-22T18:41:03.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.228979+0000", 2024-04-22T18:41:03.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092734 2024-04-22T18:41:03.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229043+0000", 2024-04-22T18:41:03.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4036999999999997e-05 2024-04-22T18:41:03.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229146+0000", 2024-04-22T18:41:03.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010301700000000001 2024-04-22T18:41:03.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:03.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229286+0000", 2024-04-22T18:41:03.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014017000000000001 2024-04-22T18:41:03.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.232704+0000", 2024-04-22T18:41:03.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0034182169999999999 2024-04-22T18:41:03.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.531413+0000", 2024-04-22T18:41:03.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29870898200000001 2024-04-22T18:41:03.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.531445+0000", 2024-04-22T18:41:03.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1751999999999997e-05 2024-04-22T18:41:03.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.531475+0000", 2024-04-22T18:41:03.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9906000000000001e-05 2024-04-22T18:41:03.503 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.503 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.503 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.504 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3525 4.c 4:314c2fec:::smithi08937929-22:head [write 1931338~785088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:41:03.504 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.229011+0000", 2024-04-22T18:41:03.504 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.194001344, 2024-04-22T18:41:03.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40428670300000003, 2024-04-22T18:41:03.504 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.504 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.504 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:03.504 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:03.504 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.504 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3525 2024-04-22T18:41:03.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.504 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229011+0000", 2024-04-22T18:41:03.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229011+0000", 2024-04-22T18:41:03.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229010+0000", 2024-04-22T18:41:03.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:03.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229182+0000", 2024-04-22T18:41:03.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000171645 2024-04-22T18:41:03.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229188+0000", 2024-04-22T18:41:03.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.674e-06 2024-04-22T18:41:03.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229193+0000", 2024-04-22T18:41:03.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8890000000000001e-06 2024-04-22T18:41:03.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.230814+0000", 2024-04-22T18:41:03.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00162068 2024-04-22T18:41:03.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.230865+0000", 2024-04-22T18:41:03.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1700000000000003e-05 2024-04-22T18:41:03.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:03.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.230967+0000", 2024-04-22T18:41:03.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101481 2024-04-22T18:41:03.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.234144+0000", 2024-04-22T18:41:03.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031771849999999999 2024-04-22T18:41:03.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.633226+0000", 2024-04-22T18:41:03.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39908168799999999 2024-04-22T18:41:03.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.633257+0000", 2024-04-22T18:41:03.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1096999999999997e-05 2024-04-22T18:41:03.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.633298+0000", 2024-04-22T18:41:03.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0960000000000001e-05 2024-04-22T18:41:03.509 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.509 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.509 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.509 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3526 4.c 4:314c2fec:::smithi08937929-22:head [setxattr _header (56) in=7b,truncate 3092521] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:41:03.509 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:03.509 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.193815257000001, 2024-04-22T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60463626000000004, 2024-04-22T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3526 2024-04-22T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:03.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:03.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229200+0000", 2024-04-22T18:41:03.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2320000000000001e-06 2024-04-22T18:41:03.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229203+0000", 2024-04-22T18:41:03.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:41:03.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329292+0000", 2024-04-22T18:41:03.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100088891 2024-04-22T18:41:03.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329357+0000", 2024-04-22T18:41:03.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5115000000000003e-05 2024-04-22T18:41:03.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329453+0000", 2024-04-22T18:41:03.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5802000000000002e-05 2024-04-22T18:41:03.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:03.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329594+0000", 2024-04-22T18:41:03.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014157500000000001 2024-04-22T18:41:03.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.330172+0000", 2024-04-22T18:41:03.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057782799999999996 2024-04-22T18:41:03.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833712+0000", 2024-04-22T18:41:03.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50353975299999998 2024-04-22T18:41:03.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833774+0000", 2024-04-22T18:41:03.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1832e-05 2024-04-22T18:41:03.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833834+0000", 2024-04-22T18:41:03.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9809999999999998e-05 2024-04-22T18:41:03.516 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.516 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.516 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.516 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3527 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:41:03.516 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.329322+0000", 2024-04-22T18:41:03.516 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.093690185, 2024-04-22T18:41:03.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50479107700000003, 2024-04-22T18:41:03.516 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.516 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:03.517 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:03.517 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:03.517 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.517 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3527 2024-04-22T18:41:03.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.517 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329322+0000", 2024-04-22T18:41:03.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329322+0000", 2024-04-22T18:41:03.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329321+0000", 2024-04-22T18:41:03.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:03.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329337+0000", 2024-04-22T18:41:03.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5820000000000001e-05 2024-04-22T18:41:03.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329345+0000", 2024-04-22T18:41:03.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4610000000000008e-06 2024-04-22T18:41:03.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.429437+0000", 2024-04-22T18:41:03.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092323 2024-04-22T18:41:03.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.429503+0000", 2024-04-22T18:41:03.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5446999999999999e-05 2024-04-22T18:41:03.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:03.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.429626+0000", 2024-04-22T18:41:03.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012354600000000001 2024-04-22T18:41:03.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833904+0000", 2024-04-22T18:41:03.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.404277901 2024-04-22T18:41:03.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.834004+0000", 2024-04-22T18:41:03.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9444999999999994e-05 2024-04-22T18:41:03.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.834113+0000", 2024-04-22T18:41:03.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010976099999999999 2024-04-22T18:41:03.521 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.521 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.521 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.521 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3735 4.0 4:01baf5a9:::smithi08937929-20:head [write 527819~624406] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:03.521 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:45.017334+0000", 2024-04-22T18:41:03.521 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.405678179999999, 2024-04-22T18:41:03.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50288105500000002, 2024-04-22T18:41:03.521 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.521 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.521 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:03.522 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:03.522 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.522 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3735 2024-04-22T18:41:03.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.522 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.017334+0000", 2024-04-22T18:41:03.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.017334+0000", 2024-04-22T18:41:03.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.017330+0000", 2024-04-22T18:41:03.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999962 2024-04-22T18:41:03.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.017601+0000", 2024-04-22T18:41:03.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027010400000000001 2024-04-22T18:41:03.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.017619+0000", 2024-04-22T18:41:03.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7978999999999998e-05 2024-04-22T18:41:03.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117701+0000", 2024-04-22T18:41:03.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100082011 2024-04-22T18:41:03.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117758+0000", 2024-04-22T18:41:03.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7191000000000002e-05 2024-04-22T18:41:03.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117826+0000", 2024-04-22T18:41:03.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8697999999999999e-05 2024-04-22T18:41:03.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:03.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117916+0000", 2024-04-22T18:41:03.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9888000000000004e-05 2024-04-22T18:41:03.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.120124+0000", 2024-04-22T18:41:03.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022077360000000001 2024-04-22T18:41:03.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.520160+0000", 2024-04-22T18:41:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40003630499999998 2024-04-22T18:41:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.520192+0000", 2024-04-22T18:41:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1334999999999997e-05 2024-04-22T18:41:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.520215+0000", 2024-04-22T18:41:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3668000000000001e-05 2024-04-22T18:41:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3736 4.0 4:01baf5a9:::smithi08937929-20:head [write 1833733~410771] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:45.117729+0000", 2024-04-22T18:41:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.305283600999999, 2024-04-22T18:41:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40259752999999998, 2024-04-22T18:41:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3736 2024-04-22T18:41:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117729+0000", 2024-04-22T18:41:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117729+0000", 2024-04-22T18:41:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117726+0000", 2024-04-22T18:41:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117983+0000", 2024-04-22T18:41:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025640400000000001 2024-04-22T18:41:03.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.117995+0000", 2024-04-22T18:41:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1922999999999999e-05 2024-04-22T18:41:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.118004+0000", 2024-04-22T18:41:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9099999999999994e-06 2024-04-22T18:41:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.119171+0000", 2024-04-22T18:41:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001167682 2024-04-22T18:41:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.119237+0000", 2024-04-22T18:41:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6197000000000003e-05 2024-04-22T18:41:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.119338+0000", 2024-04-22T18:41:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000100636 2024-04-22T18:41:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.120523+0000", 2024-04-22T18:41:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001185052 2024-04-22T18:41:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.520260+0000", 2024-04-22T18:41:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39973705799999998 2024-04-22T18:41:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.520291+0000", 2024-04-22T18:41:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0508999999999999e-05 2024-04-22T18:41:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.520326+0000", 2024-04-22T18:41:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5720999999999999e-05 2024-04-22T18:41:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3738 4.0 4:01baf5a9:::smithi08937929-20:head [setxattr _header (56) in=7b,truncate 3327819] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:45.218527+0000", 2024-04-22T18:41:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.204485665, 2024-04-22T18:41:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50245176700000005, 2024-04-22T18:41:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3738 2024-04-22T18:41:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218527+0000", 2024-04-22T18:41:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218527+0000", 2024-04-22T18:41:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218525+0000", 2024-04-22T18:41:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218537+0000", 2024-04-22T18:41:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2668e-05 2024-04-22T18:41:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218546+0000", 2024-04-22T18:41:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1710000000000002e-06 2024-04-22T18:41:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218555+0000", 2024-04-22T18:41:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2660000000000007e-06 2024-04-22T18:41:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.220809+0000", 2024-04-22T18:41:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022538649999999999 2024-04-22T18:41:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.220903+0000", 2024-04-22T18:41:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4072000000000003e-05 2024-04-22T18:41:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.221042+0000", 2024-04-22T18:41:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013949700000000001 2024-04-22T18:41:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.321737+0000", 2024-04-22T18:41:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100694216 2024-04-22T18:41:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.720867+0000", 2024-04-22T18:41:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39913031199999999 2024-04-22T18:41:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.720921+0000", 2024-04-22T18:41:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4166999999999998e-05 2024-04-22T18:41:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.720979+0000", 2024-04-22T18:41:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7593000000000002e-05 2024-04-22T18:41:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3739 4.0 4:01baf5a9:::smithi08937929-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:45.218575+0000", 2024-04-22T18:41:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.204437185, 2024-04-22T18:41:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50254516100000002, 2024-04-22T18:41:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3739 2024-04-22T18:41:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218575+0000", 2024-04-22T18:41:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218575+0000", 2024-04-22T18:41:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218575+0000", 2024-04-22T18:41:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218578+0000", 2024-04-22T18:41:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0319999999999999e-06 2024-04-22T18:41:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218585+0000", 2024-04-22T18:41:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7499999999999997e-06 2024-04-22T18:41:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.218588+0000", 2024-04-22T18:41:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5329999999999999e-06 2024-04-22T18:41:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.321558+0000", 2024-04-22T18:41:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102970176 2024-04-22T18:41:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.321708+0000", 2024-04-22T18:41:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014996000000000001 2024-04-22T18:41:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.721012+0000", 2024-04-22T18:41:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39930342499999999 2024-04-22T18:41:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.721058+0000", 2024-04-22T18:41:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6220000000000001e-05 2024-04-22T18:41:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:45.721121+0000", 2024-04-22T18:41:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2687000000000005e-05 2024-04-22T18:41:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:169 3.2 3:49519222:::48f05881d8397501bc6119ae239e14ac22e37488d789da81ca18cc8233c5cc0b:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e126)", 2024-04-22T18:41:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:59.770492+0000", 2024-04-22T18:41:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6525200939999998, 2024-04-22T18:41:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40355745999999998, 2024-04-22T18:41:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.770492+0000", 2024-04-22T18:41:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.770492+0000", 2024-04-22T18:41:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.770491+0000", 2024-04-22T18:41:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.770497+0000", 2024-04-22T18:41:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0970000000000002e-06 2024-04-22T18:41:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.770506+0000", 2024-04-22T18:41:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4610000000000008e-06 2024-04-22T18:41:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.870600+0000", 2024-04-22T18:41:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009386000000001 2024-04-22T18:41:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.870688+0000", 2024-04-22T18:41:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8220000000000006e-05 2024-04-22T18:41:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.870813+0000", 2024-04-22T18:41:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012526799999999999 2024-04-22T18:41:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.871056+0000", 2024-04-22T18:41:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000242554 2024-04-22T18:41:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:59.871698+0000", 2024-04-22T18:41:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00064254500000000001 2024-04-22T18:41:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:00.173885+0000", 2024-04-22T18:41:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.302186764 2024-04-22T18:41:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:00.173938+0000", 2024-04-22T18:41:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3270000000000003e-05 2024-04-22T18:41:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:00.174050+0000", 2024-04-22T18:41:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000111765 2024-04-22T18:41:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:03.552 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:03.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:03.554 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:03.554+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:03.654 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:03.655+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:03.667 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:41:03.669 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:03.669 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:03.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.669 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4177 4.19 4.f75b2379 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:03.669 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:03.669 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.50429985799999999, 2024-04-22T18:41:03.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50432084200000005, 2024-04-22T18:41:03.669 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.669 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:03.669 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.669 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.669 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.669 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4177 2024-04-22T18:41:03.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.670 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:03.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:03.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164847+0000", 2024-04-22T18:41:03.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:03.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164854+0000", 2024-04-22T18:41:03.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2620000000000001e-06 2024-04-22T18:41:03.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164864+0000", 2024-04-22T18:41:03.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0400999999999999e-05 2024-04-22T18:41:03.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164876+0000", 2024-04-22T18:41:03.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1204e-05 2024-04-22T18:41:03.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164975+0000", 2024-04-22T18:41:03.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9562999999999995e-05 2024-04-22T18:41:03.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164980+0000", 2024-04-22T18:41:03.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7310000000000001e-06 2024-04-22T18:41:03.672 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.672 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.673 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.673 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4182 4.c 4.5345b7ac (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:03.673 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:03.673 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.50426853900000002, 2024-04-22T18:41:03.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50433200899999997, 2024-04-22T18:41:03.673 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.673 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:03.673 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.673 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.673 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.673 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4182 2024-04-22T18:41:03.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.673 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164881+0000", 2024-04-22T18:41:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.305e-06 2024-04-22T18:41:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164885+0000", 2024-04-22T18:41:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.873e-06 2024-04-22T18:41:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164890+0000", 2024-04-22T18:41:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6299999999999997e-06 2024-04-22T18:41:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225475+0000", 2024-04-22T18:41:03.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060585181000000002 2024-04-22T18:41:03.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225483+0000", 2024-04-22T18:41:03.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6790000000000005e-06 2024-04-22T18:41:03.676 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.676 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.676 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.676 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4183 4.c 4.5345b7ac (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:03.677 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:03.677 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.504255283, 2024-04-22T18:41:03.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50435403700000003, 2024-04-22T18:41:03.677 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.677 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:03.677 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.677 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.677 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.677 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4183 2024-04-22T18:41:03.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.677 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:03.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:03.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164892+0000", 2024-04-22T18:41:03.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164894+0000", 2024-04-22T18:41:03.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.705e-06 2024-04-22T18:41:03.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164897+0000", 2024-04-22T18:41:03.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2739999999999999e-06 2024-04-22T18:41:03.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164901+0000", 2024-04-22T18:41:03.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.636e-06 2024-04-22T18:41:03.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225508+0000", 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060607564000000003 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225515+0000", 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1829999999999999e-06 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4190 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.50423836099999997, 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50437122199999995, 2024-04-22T18:41:03.680 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4190 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:03.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:03.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164912+0000", 2024-04-22T18:41:03.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6249999999999999e-06 2024-04-22T18:41:03.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164918+0000", 2024-04-22T18:41:03.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6039999999999998e-06 2024-04-22T18:41:03.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164922+0000", 2024-04-22T18:41:03.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9319999999999998e-06 2024-04-22T18:41:03.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225536+0000", 2024-04-22T18:41:03.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060613725 2024-04-22T18:41:03.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225540+0000", 2024-04-22T18:41:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5030000000000002e-06 2024-04-22T18:41:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4191 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.50422294000000001, 2024-04-22T18:41:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50438933699999999, 2024-04-22T18:41:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.685 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4191 2024-04-22T18:41:03.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.685 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:03.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:03.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:03.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164927+0000", 2024-04-22T18:41:03.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0959999999999999e-06 2024-04-22T18:41:03.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164937+0000", 2024-04-22T18:41:03.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0730999999999999e-05 2024-04-22T18:41:03.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164944+0000", 2024-04-22T18:41:03.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5130000000000002e-06 2024-04-22T18:41:03.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225565+0000", 2024-04-22T18:41:03.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060621421000000002 2024-04-22T18:41:03.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225569+0000", 2024-04-22T18:41:03.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.264e-06 2024-04-22T18:41:03.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.688 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.688 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4192 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:03.688 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:03.688 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.50419833199999997, 2024-04-22T18:41:03.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50439789499999999, 2024-04-22T18:41:03.688 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.688 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:03.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4192 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:03.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164953+0000", 2024-04-22T18:41:03.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1099999999999996e-06 2024-04-22T18:41:03.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164965+0000", 2024-04-22T18:41:03.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2298e-05 2024-04-22T18:41:03.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164972+0000", 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5230000000000001e-06 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225581+0000", 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060609356000000003 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225584+0000", 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.106e-06 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4197 4.13 4.97bf6253 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.50417172200000004, 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50441956899999996, 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4197 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164975+0000", 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164979+0000", 2024-04-22T18:41:03.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8959999999999996e-06 2024-04-22T18:41:03.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164983+0000", 2024-04-22T18:41:03.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.053e-06 2024-04-22T18:41:03.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164987+0000", 2024-04-22T18:41:03.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:41:03.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.264783+0000", 2024-04-22T18:41:03.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099796088000000005 2024-04-22T18:41:03.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.264789+0000", 2024-04-22T18:41:03.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4989999999999999e-06 2024-04-22T18:41:03.695 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.695 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.695 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.695 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4198 4.13 4.97bf6253 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:03.695 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:03.695 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.50415835399999998, 2024-04-22T18:41:03.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50444084099999997, 2024-04-22T18:41:03.696 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.696 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:03.696 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.696 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.696 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.696 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4198 2024-04-22T18:41:03.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.696 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:03.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:03.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:03.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164991+0000", 2024-04-22T18:41:03.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:41:03.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164994+0000", 2024-04-22T18:41:03.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2459999999999999e-06 2024-04-22T18:41:03.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164997+0000", 2024-04-22T18:41:03.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1200000000000002e-06 2024-04-22T18:41:03.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.364948+0000", 2024-04-22T18:41:03.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19995059200000001 2024-04-22T18:41:03.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.364969+0000", 2024-04-22T18:41:03.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1495999999999999e-05 2024-04-22T18:41:03.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.699 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.699 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4199 4.13 4.97bf6253 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:03.699 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:03.699 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.50414733099999998, 2024-04-22T18:41:03.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50446152700000002, 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4199 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:03.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:03.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165002+0000", 2024-04-22T18:41:03.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:41:03.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165006+0000", 2024-04-22T18:41:03.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:41:03.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165009+0000", 2024-04-22T18:41:03.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4599999999999999e-06 2024-04-22T18:41:03.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365066+0000", 2024-04-22T18:41:03.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20005682999999999 2024-04-22T18:41:03.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365078+0000", 2024-04-22T18:41:03.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2235000000000001e-05 2024-04-22T18:41:03.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.703 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.703 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4200 4.13 4.74213c93 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:03.703 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:03.703 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.50413645100000004, 2024-04-22T18:41:03.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.504482298, 2024-04-22T18:41:03.703 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.703 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:03.703 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.703 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.704 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.704 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4200 2024-04-22T18:41:03.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.704 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:03.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:03.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165013+0000", 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.635e-06 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165018+0000", 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2000000000000002e-06 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165024+0000", 2024-04-22T18:41:03.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5910000000000003e-06 2024-04-22T18:41:03.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365110+0000", 2024-04-22T18:41:03.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20008601400000001 2024-04-22T18:41:03.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365114+0000", 2024-04-22T18:41:03.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.651e-06 2024-04-22T18:41:03.706 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.706 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4205 4.1 4.fdaa93c1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.50412030100000005, 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50451310999999999, 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4205 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:03.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:03.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165027+0000", 2024-04-22T18:41:03.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165029+0000", 2024-04-22T18:41:03.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3549999999999999e-06 2024-04-22T18:41:03.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165035+0000", 2024-04-22T18:41:03.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8549999999999998e-06 2024-04-22T18:41:03.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165040+0000", 2024-04-22T18:41:03.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9200000000000003e-06 2024-04-22T18:41:03.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165116+0000", 2024-04-22T18:41:03.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5834999999999995e-05 2024-04-22T18:41:03.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:03.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165123+0000", 2024-04-22T18:41:03.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2139999999999999e-06 2024-04-22T18:41:03.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.710 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.711 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:41:03.711 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 11 2024-04-22T18:41:03.711 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:03.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:03.755 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:03.757+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:03.814 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:03.814 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:03.815 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:03.815 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:03.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:03.866 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:03.867+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:03.922 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:03.921+0000 7f3d00e8c640 -1 osd.6 129 handle_osd_map injecting map crc failure 2024-04-22T18:41:03.930 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:03.930 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:03.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:03.930 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:03.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.930 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:25 4.d 4:bee1184f:::smithi08937929-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:03.930 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:41:03.930 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.76734912699999, 2024-04-22T18:41:03.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49882675799999998, 2024-04-22T18:41:03.930 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.930 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.931 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.931 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 25 2024-04-22T18:41:03.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.931 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:41:03.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162459+0000", 2024-04-22T18:41:03.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162456+0000", 2024-04-22T18:41:03.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:03.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162849+0000", 2024-04-22T18:41:03.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039379600000000002 2024-04-22T18:41:03.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.162865+0000", 2024-04-22T18:41:03.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6033e-05 2024-04-22T18:41:03.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.262946+0000", 2024-04-22T18:41:03.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100080551 2024-04-22T18:41:03.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363090+0000", 2024-04-22T18:41:03.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014448500000001 2024-04-22T18:41:03.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:03.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363188+0000", 2024-04-22T18:41:03.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7413999999999999e-05 2024-04-22T18:41:03.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 2", 2024-04-22T18:41:03.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363281+0000", 2024-04-22T18:41:03.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3017999999999996e-05 2024-04-22T18:41:03.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:03.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367025+0000", 2024-04-22T18:41:03.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0037439249999999999 2024-04-22T18:41:03.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661149+0000", 2024-04-22T18:41:03.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29412436800000002 2024-04-22T18:41:03.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:03.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661206+0000", 2024-04-22T18:41:03.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6304999999999997e-05 2024-04-22T18:41:03.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:03.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.661285+0000", 2024-04-22T18:41:03.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.9849000000000001e-05 2024-04-22T18:41:03.936 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.936 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.936 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.936 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:31 4.e 4:744777ef:::smithi08937929-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:03.936 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:41:03.936 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.66579030599999, 2024-04-22T18:41:03.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48466376500000002, 2024-04-22T18:41:03.936 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.936 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.936 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.936 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.936 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.936 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 31 2024-04-22T18:41:03.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.937 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:41:03.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264017+0000", 2024-04-22T18:41:03.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264016+0000", 2024-04-22T18:41:03.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:03.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264025+0000", 2024-04-22T18:41:03.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.901e-06 2024-04-22T18:41:03.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264033+0000", 2024-04-22T18:41:03.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4989999999999997e-06 2024-04-22T18:41:03.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.938 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264039+0000", 2024-04-22T18:41:03.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6200000000000004e-06 2024-04-22T18:41:03.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367500+0000", 2024-04-22T18:41:03.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103461803 2024-04-22T18:41:03.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:03.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367577+0000", 2024-04-22T18:41:03.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6366999999999996e-05 2024-04-22T18:41:03.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.367667+0000", 2024-04-22T18:41:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0569999999999995e-05 2024-04-22T18:41:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467963+0000", 2024-04-22T18:41:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100295473 2024-04-22T18:41:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748613+0000", 2024-04-22T18:41:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28064997400000002 2024-04-22T18:41:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:03.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748646+0000", 2024-04-22T18:41:03.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2954000000000003e-05 2024-04-22T18:41:03.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:03.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748681+0000", 2024-04-22T18:41:03.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5546000000000002e-05 2024-04-22T18:41:03.941 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.941 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.941 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.941 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:32 4.e 4:744777ef:::smithi08937929-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:03.942 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:41:03.942 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.66576390399999, 2024-04-22T18:41:03.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48479826999999998, 2024-04-22T18:41:03.942 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.942 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:03.942 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.942 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.942 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.942 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 32 2024-04-22T18:41:03.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.942 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:41:03.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264044+0000", 2024-04-22T18:41:03.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264043+0000", 2024-04-22T18:41:03.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.943 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.943 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264046+0000", 2024-04-22T18:41:03.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5440000000000001e-06 2024-04-22T18:41:03.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264051+0000", 2024-04-22T18:41:03.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7200000000000003e-06 2024-04-22T18:41:03.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264056+0000", 2024-04-22T18:41:03.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7949999999999998e-06 2024-04-22T18:41:03.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467888+0000", 2024-04-22T18:41:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20383206400000001 2024-04-22T18:41:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.467945+0000", 2024-04-22T18:41:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6443999999999997e-05 2024-04-22T18:41:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748723+0000", 2024-04-22T18:41:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28077779600000002 2024-04-22T18:41:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748777+0000", 2024-04-22T18:41:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4092000000000003e-05 2024-04-22T18:41:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748842+0000", 2024-04-22T18:41:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5421999999999998e-05 2024-04-22T18:41:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:03.946 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:03.947 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.26742957499999, 2024-04-22T18:41:03.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:41:03.947 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.947 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.947 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.947 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.947 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.947 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:41:03.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.947 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:03.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:03.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:03.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:41:03.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:41:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:41:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:41:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:41:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:41:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.949 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:41:03.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:41:03.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:03.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:41:03.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:41:03.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:03.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:41:03.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:41:03.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:03.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:41:03.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:41:03.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:41:03.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:41:03.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:03.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:41:03.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:41:03.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:03.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:41:03.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:41:03.952 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.952 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.952 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.952 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:03.952 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:41:03.952 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.166593277, 2024-04-22T18:41:03.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:41:03.952 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.952 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.952 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.952 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:41:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:41:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:41:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:41:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:41:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:41:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:41:03.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:41:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:41:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:41:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:41:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:41:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:41:03.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:41:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:41:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:41:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:41:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:41:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:41:03.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:41:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:41:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:41:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:41:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:41:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:41:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.16643600699999, 2024-04-22T18:41:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:41:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:41:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:41:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:41:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:41:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:41:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:41:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:41:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:41:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:41:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:41:03.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:41:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:41:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:41:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:41:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:41:03.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:41:03.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:03.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:41:03.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:41:03.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:41:03.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:41:03.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:03.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:41:03.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:41:03.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:03.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:41:03.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:41:03.963 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.963 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.963 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.963 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:03.963 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:03.963 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.16642139800001, 2024-04-22T18:41:03.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:41:03.964 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.964 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:03.964 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.964 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.964 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.964 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:41:03.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.964 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:03.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:03.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:03.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:41:03.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:41:03.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:41:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:41:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:41:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:41:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:41:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:41:03.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:41:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:41:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:41:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:41:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:41:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:41:03.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:03.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:41:03.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:41:03.968 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.968 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.968 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.968 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:03.968 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:03.968 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.16641101299999, 2024-04-22T18:41:03.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:41:03.968 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.969 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.969 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.969 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.969 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.969 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:41:03.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.969 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:03.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:03.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:41:03.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:41:03.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:41:03.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:41:03.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:41:03.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:41:03.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:41:03.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:41:03.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:41:03.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:03.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:41:03.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:41:03.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:03.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:41:03.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:41:03.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:03.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:41:03.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:41:03.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:41:03.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:41:03.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:03.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:41:03.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:41:03.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:03.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:41:03.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:41:03.973 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.974 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.974 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.974 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:94 4.1 4:821263ac:::smithi08937929-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:03.974 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:41:03.974 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.165481083, 2024-04-22T18:41:03.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55373634599999999, 2024-04-22T18:41:03.974 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.974 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.974 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.974 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.974 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.974 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 94 2024-04-22T18:41:03.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.975 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:41:03.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764327+0000", 2024-04-22T18:41:03.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764326+0000", 2024-04-22T18:41:03.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:03.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764739+0000", 2024-04-22T18:41:03.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00041332899999999999 2024-04-22T18:41:03.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764776+0000", 2024-04-22T18:41:03.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6869000000000002e-05 2024-04-22T18:41:03.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764784+0000", 2024-04-22T18:41:03.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7440000000000007e-06 2024-04-22T18:41:03.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869348+0000", 2024-04-22T18:41:03.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.104564212 2024-04-22T18:41:03.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:03.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869449+0000", 2024-04-22T18:41:03.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101667 2024-04-22T18:41:03.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:03.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.869604+0000", 2024-04-22T18:41:03.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015436599999999999 2024-04-22T18:41:03.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:03.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873223+0000", 2024-04-22T18:41:03.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.003619386 2024-04-22T18:41:03.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318017+0000", 2024-04-22T18:41:03.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44479398599999997 2024-04-22T18:41:03.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:03.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318042+0000", 2024-04-22T18:41:03.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4831e-05 2024-04-22T18:41:03.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:03.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318063+0000", 2024-04-22T18:41:03.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0996e-05 2024-04-22T18:41:03.979 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.979 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.979 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.979 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:95 4.1 4:821263ac:::smithi08937929-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:03.979 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:41:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.16501895799999, 2024-04-22T18:41:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55335714499999999, 2024-04-22T18:41:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 95 2024-04-22T18:41:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:41:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764789+0000", 2024-04-22T18:41:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764788+0000", 2024-04-22T18:41:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764792+0000", 2024-04-22T18:41:03.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5750000000000001e-06 2024-04-22T18:41:03.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764796+0000", 2024-04-22T18:41:03.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4719999999999997e-06 2024-04-22T18:41:03.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764799+0000", 2024-04-22T18:41:03.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1879999999999998e-06 2024-04-22T18:41:03.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870595+0000", 2024-04-22T18:41:03.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105795443 2024-04-22T18:41:03.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:03.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870648+0000", 2024-04-22T18:41:03.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3004000000000003e-05 2024-04-22T18:41:03.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:03.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870771+0000", 2024-04-22T18:41:03.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000123115 2024-04-22T18:41:03.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:03.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.873704+0000", 2024-04-22T18:41:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0029328050000000001 2024-04-22T18:41:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318087+0000", 2024-04-22T18:41:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.444383578 2024-04-22T18:41:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318113+0000", 2024-04-22T18:41:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6148000000000001e-05 2024-04-22T18:41:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318146+0000", 2024-04-22T18:41:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2564000000000002e-05 2024-04-22T18:41:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:96 4.1 4:821263ac:::smithi08937929-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:41:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.165006154, 2024-04-22T18:41:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.553472415, 2024-04-22T18:41:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.985 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 96 2024-04-22T18:41:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:41:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764802+0000", 2024-04-22T18:41:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764801+0000", 2024-04-22T18:41:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764803+0000", 2024-04-22T18:41:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.528e-06 2024-04-22T18:41:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764807+0000", 2024-04-22T18:41:03.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1659999999999996e-06 2024-04-22T18:41:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764813+0000", 2024-04-22T18:41:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8420000000000001e-06 2024-04-22T18:41:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870963+0000", 2024-04-22T18:41:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106150146 2024-04-22T18:41:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871040+0000", 2024-04-22T18:41:03.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6593000000000003e-05 2024-04-22T18:41:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318172+0000", 2024-04-22T18:41:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44713280500000002 2024-04-22T18:41:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318220+0000", 2024-04-22T18:41:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7602000000000003e-05 2024-04-22T18:41:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:03.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.318274+0000", 2024-04-22T18:41:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4054000000000001e-05 2024-04-22T18:41:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:326 4.9 4:9ec4daef:::smithi08937929-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:41:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:41:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 125.83723369800001, 2024-04-22T18:41:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50355474200000006, 2024-04-22T18:41:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.990 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 326 2024-04-22T18:41:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:41:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092574+0000", 2024-04-22T18:41:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092573+0000", 2024-04-22T18:41:03.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:03.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092580+0000", 2024-04-22T18:41:03.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.34e-06 2024-04-22T18:41:03.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092591+0000", 2024-04-22T18:41:03.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1365e-05 2024-04-22T18:41:03.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.092602+0000", 2024-04-22T18:41:03.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0550000000000001e-05 2024-04-22T18:41:03.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093173+0000", 2024-04-22T18:41:03.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057078599999999999 2024-04-22T18:41:03.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:03.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.093225+0000", 2024-04-22T18:41:03.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2595e-05 2024-04-22T18:41:03.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:03.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495173+0000", 2024-04-22T18:41:03.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40194817300000002 2024-04-22T18:41:03.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:03.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.495736+0000", 2024-04-22T18:41:03.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056260000000000001 2024-04-22T18:41:03.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:03.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596007+0000", 2024-04-22T18:41:03.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100270656 2024-04-22T18:41:03.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:03.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596068+0000", 2024-04-22T18:41:03.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1804999999999995e-05 2024-04-22T18:41:03.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:03.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.596129+0000", 2024-04-22T18:41:03.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0319000000000003e-05 2024-04-22T18:41:03.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.995 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:03.995 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:03.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.996 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:41:03.996 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:03.996 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 124.828660586, 2024-04-22T18:41:03.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:41:03.996 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:03.996 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:03.996 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:03.996 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:03.996 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:03.996 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:41:03.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.996 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:03.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:03.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:03.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:03.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:03.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:03.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:03.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:41:03.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:03.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:03.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:41:03.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:41:03.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:03.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:41:03.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:41:03.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:03.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:41:03.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:41:03.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:03.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:41:03.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:41:03.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:03.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:41:03.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:41:03.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:03.999 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:03.999 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:03.999 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:41:03.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:41:04.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:41:04.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:41:04.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:41:04.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:41:04.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:41:04.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:41:04.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:41:04.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:41:04.001 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.001 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.001 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.001 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:41:04.001 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:04.001 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.794013070999995, 2024-04-22T18:41:04.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:41:04.002 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.002 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:04.002 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.002 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.002 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.002 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:41:04.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.002 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:04.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:04.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:41:04.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:04.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:41:04.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:41:04.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:41:04.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:41:04.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:41:04.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:41:04.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:41:04.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:41:04.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:41:04.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:41:04.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:41:04.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:41:04.005 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.005 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.005 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.006 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1684 4.c 4:314c2fec:::smithi08937929-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:41:04.006 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:41:04.006 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 93.324529107999993, 2024-04-22T18:41:04.006 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987885199999998, 2024-04-22T18:41:04.006 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.006 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:04.006 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.006 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.006 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.006 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1684 2024-04-22T18:41:04.006 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.006 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.006 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.006 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:41:04.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605279+0000", 2024-04-22T18:41:04.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.007 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.007 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605278+0000", 2024-04-22T18:41:04.007 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:04.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.007 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605280+0000", 2024-04-22T18:41:04.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1770000000000001e-06 2024-04-22T18:41:04.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605285+0000", 2024-04-22T18:41:04.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8539999999999996e-06 2024-04-22T18:41:04.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.008 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.008 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.008 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.605289+0000", 2024-04-22T18:41:04.008 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.687e-06 2024-04-22T18:41:04.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705352+0000", 2024-04-22T18:41:04.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10006322199999999 2024-04-22T18:41:04.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:04.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:30.705423+0000", 2024-04-22T18:41:04.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1136999999999993e-05 2024-04-22T18:41:04.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.009 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.009 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105017+0000", 2024-04-22T18:41:04.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.399593268 2024-04-22T18:41:04.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105081+0000", 2024-04-22T18:41:04.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4703999999999996e-05 2024-04-22T18:41:04.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.010 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:31.105157+0000", 2024-04-22T18:41:04.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6205999999999995e-05 2024-04-22T18:41:04.010 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.010 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.010 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.011 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2191 4.1 4.fdaa93c1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-04-22T18:41:04.011 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:41:04.011 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 78.379936857999994, 2024-04-22T18:41:04.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51064384399999996, 2024-04-22T18:41:04.011 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.011 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:04.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.011 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2191 2024-04-22T18:41:04.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.011 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:41:04.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:41:04.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549871+0000", 2024-04-22T18:41:04.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:04.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549872+0000", 2024-04-22T18:41:04.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.387e-06 2024-04-22T18:41:04.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549875+0000", 2024-04-22T18:41:04.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:41:04.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.549880+0000", 2024-04-22T18:41:04.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2529999999999998e-06 2024-04-22T18:41:04.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060496+0000", 2024-04-22T18:41:04.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.51061597999999997 2024-04-22T18:41:04.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.060515+0000", 2024-04-22T18:41:04.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9130000000000001e-05 2024-04-22T18:41:04.014 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.014 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.014 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.014 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2198 4.d 4:bc165c76:::smithi08937929-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:41:04.015 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:41:04.015 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 78.378669157000004, 2024-04-22T18:41:04.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50502289899999997, 2024-04-22T18:41:04.015 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.015 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:04.015 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.015 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.015 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.015 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2198 2024-04-22T18:41:04.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.015 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:41:04.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551139+0000", 2024-04-22T18:41:04.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551138+0000", 2024-04-22T18:41:04.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:04.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551143+0000", 2024-04-22T18:41:04.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8949999999999997e-06 2024-04-22T18:41:04.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551151+0000", 2024-04-22T18:41:04.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8729999999999996e-06 2024-04-22T18:41:04.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551158+0000", 2024-04-22T18:41:04.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3099999999999997e-06 2024-04-22T18:41:04.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551180+0000", 2024-04-22T18:41:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2197000000000001e-05 2024-04-22T18:41:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:41:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551185+0000", 2024-04-22T18:41:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7010000000000004e-06 2024-04-22T18:41:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657133+0000", 2024-04-22T18:41:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105948498 2024-04-22T18:41:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:04.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.657183+0000", 2024-04-22T18:41:04.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0124000000000003e-05 2024-04-22T18:41:04.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.055924+0000", 2024-04-22T18:41:04.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39874079299999998 2024-04-22T18:41:04.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056018+0000", 2024-04-22T18:41:04.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4032999999999999e-05 2024-04-22T18:41:04.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.056161+0000", 2024-04-22T18:41:04.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000143385 2024-04-22T18:41:04.020 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.020 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.020 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.020 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2212 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:41:04.020 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:41:04.020 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 78.378599515000005, 2024-04-22T18:41:04.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50989936400000002, 2024-04-22T18:41:04.020 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.020 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:04.021 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.021 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.021 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.021 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2212 2024-04-22T18:41:04.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.021 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:41:04.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:41:04.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551208+0000", 2024-04-22T18:41:04.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:04.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551211+0000", 2024-04-22T18:41:04.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6280000000000001e-06 2024-04-22T18:41:04.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551214+0000", 2024-04-22T18:41:04.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1209999999999998e-06 2024-04-22T18:41:04.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551217+0000", 2024-04-22T18:41:04.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8490000000000002e-06 2024-04-22T18:41:04.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061086+0000", 2024-04-22T18:41:04.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50986975099999998 2024-04-22T18:41:04.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061108+0000", 2024-04-22T18:41:04.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1319e-05 2024-04-22T18:41:04.024 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.024 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.024 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.024 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2213 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:41:04.024 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:41:04.024 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 78.378589098000006, 2024-04-22T18:41:04.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50998486099999996, 2024-04-22T18:41:04.024 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.024 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:04.024 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.025 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.025 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.025 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2213 2024-04-22T18:41:04.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.025 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:41:04.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551219+0000", 2024-04-22T18:41:04.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551218+0000", 2024-04-22T18:41:04.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:04.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551220+0000", 2024-04-22T18:41:04.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.319e-06 2024-04-22T18:41:04.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551223+0000", 2024-04-22T18:41:04.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9660000000000001e-06 2024-04-22T18:41:04.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551226+0000", 2024-04-22T18:41:04.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9050000000000001e-06 2024-04-22T18:41:04.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061162+0000", 2024-04-22T18:41:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.509936838 2024-04-22T18:41:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061204+0000", 2024-04-22T18:41:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1094999999999999e-05 2024-04-22T18:41:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2214 4.9 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-04-22T18:41:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:41:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 78.378580108999998, 2024-04-22T18:41:04.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50997458100000004, 2024-04-22T18:41:04.029 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.029 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:04.029 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.029 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.029 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.029 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2214 2024-04-22T18:41:04.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.029 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:41:04.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551228+0000", 2024-04-22T18:41:04.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551227+0000", 2024-04-22T18:41:04.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:04.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551229+0000", 2024-04-22T18:41:04.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8649999999999999e-06 2024-04-22T18:41:04.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551232+0000", 2024-04-22T18:41:04.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:41:04.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:45.551235+0000", 2024-04-22T18:41:04.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:41:04.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061194+0000", 2024-04-22T18:41:04.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50995819099999995 2024-04-22T18:41:04.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:46.061202+0000", 2024-04-22T18:41:04.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.541e-06 2024-04-22T18:41:04.032 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.032 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.032 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.032 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.032 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.032 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:04.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:04.034 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:04.035+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:04.134 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:04.134+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:04.177 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:04.177 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:04.177 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:04.177 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:04.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:04.234 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:04.236+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:04.328 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:04.335 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:04.336+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:04.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:04.437 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:04.438+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:04.489 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:04.489 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:04.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:04.489 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:04.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.490 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:41:04.490 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:04.490 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.39558201099999, 2024-04-22T18:41:04.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:41:04.490 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.490 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.490 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.490 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.490 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.490 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:41:04.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.490 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:04.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:04.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:41:04.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:04.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:41:04.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:41:04.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:41:04.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:41:04.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:41:04.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:41:04.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:41:04.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:41:04.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:41:04.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:41:04.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:04.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:41:04.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:41:04.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:41:04.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:41:04.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:41:04.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:41:04.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:41:04.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:41:04.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:41:04.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:41:04.495 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.495 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.495 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.495 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:41:04.495 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:04.496 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.49608413599999, 2024-04-22T18:41:04.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:41:04.496 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.496 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.496 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:04.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:04.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:41:04.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:04.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:41:04.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:41:04.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:41:04.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:41:04.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:41:04.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:41:04.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:41:04.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:41:04.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:04.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:41:04.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:41:04.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:41:04.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:41:04.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:41:04.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:41:04.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:04.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:41:04.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:41:04.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:41:04.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:41:04.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:41:04.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:41:04.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:41:04.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:41:04.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:41:04.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:41:04.501 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.501 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.501 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.501 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:41:04.501 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:04.502 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 125.387784345, 2024-04-22T18:41:04.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:41:04.502 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.502 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.502 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.502 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.502 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.502 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:41:04.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.502 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:04.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:04.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:41:04.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:04.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.503 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:41:04.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:41:04.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:41:04.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:41:04.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:41:04.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:41:04.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:41:04.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:41:04.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:41:04.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:41:04.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:04.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:41:04.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:41:04.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:41:04.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:41:04.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:41:04.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:41:04.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:41:04.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:41:04.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:41:04.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:41:04.507 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.507 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.507 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.507 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:04.507 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:04.507 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 112.05624071699999, 2024-04-22T18:41:04.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:41:04.507 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.508 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.508 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:04.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:04.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:41:04.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:04.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:41:04.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:41:04.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:41:04.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:41:04.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:41:04.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:41:04.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:41:04.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:41:04.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:41:04.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:41:04.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:04.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:41:04.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:41:04.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:41:04.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:41:04.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:41:04.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:41:04.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:41:04.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:41:04.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:41:04.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:41:04.512 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.512 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.512 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.512 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:04.513 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:04.513 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.96515490900001, 2024-04-22T18:41:04.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:41:04.513 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.513 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.513 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.513 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.513 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.513 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:41:04.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.513 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:04.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:04.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:41:04.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:04.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:41:04.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:41:04.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:41:04.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:41:04.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:41:04.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:41:04.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:41:04.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:41:04.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:41:04.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:41:04.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:04.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:41:04.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:41:04.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:41:04.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:41:04.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:41:04.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:41:04.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:41:04.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:41:04.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:41:04.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:41:04.518 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.518 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.518 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.518 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:04.518 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:04.518 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.95935464900001, 2024-04-22T18:41:04.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:41:04.518 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:41:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.519 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:04.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:41:04.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:04.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:41:04.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:41:04.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:41:04.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:41:04.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:41:04.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:41:04.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:41:04.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:41:04.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:41:04.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:41:04.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:04.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:41:04.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:41:04.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:41:04.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:41:04.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:41:04.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:41:04.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:41:04.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:41:04.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:41:04.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:41:04.523 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.524 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.524 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.524 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:04.524 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:04.524 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.954614306, 2024-04-22T18:41:04.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:41:04.524 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.524 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.524 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.524 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.524 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.524 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:41:04.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.525 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:04.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:04.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:41:04.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:04.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:41:04.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:41:04.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:41:04.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:41:04.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:41:04.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:41:04.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:41:04.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:41:04.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:41:04.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:41:04.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:04.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:41:04.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:41:04.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:41:04.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:41:04.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:41:04.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:41:04.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:41:04.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:41:04.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:41:04.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:41:04.529 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.529 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.529 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.529 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:04.529 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:04.530 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.954141408, 2024-04-22T18:41:04.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:41:04.530 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.530 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.530 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.530 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.530 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.530 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:41:04.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.530 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:04.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:04.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:41:04.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:04.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:41:04.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:41:04.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:41:04.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:41:04.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:41:04.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:41:04.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:41:04.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:41:04.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:41:04.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:41:04.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:04.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:41:04.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:41:04.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:41:04.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:41:04.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:41:04.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:41:04.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:41:04.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:41:04.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 111.95412148699999, 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.537 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.538 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:41:04.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.538 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:04.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:04.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:04.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:04.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:41:04.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:41:04.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:41:04.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:41:04.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:41:04.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:41:04.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:41:04.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:41:04.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:04.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:41:04.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:41:04.542 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.542 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.542 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.542 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:04.542 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:04.542 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 106.64479772200001, 2024-04-22T18:41:04.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:41:04.542 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.542 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.542 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:04.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:04.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:41:04.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:04.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:41:04.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:41:04.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:41:04.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:41:04.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:41:04.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:41:04.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:41:04.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:41:04.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:41:04.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:41:04.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:04.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:41:04.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:41:04.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:41:04.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:41:04.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:41:04.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:41:04.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:41:04.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:41:04.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:41:04.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:41:04.547 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.547 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.547 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.547 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:04.547 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:04.547 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 106.644590186, 2024-04-22T18:41:04.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:41:04.548 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.548 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.548 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:04.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:04.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:41:04.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:04.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:41:04.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:41:04.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:41:04.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:41:04.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:41:04.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:41:04.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:41:04.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:41:04.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:41:04.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:41:04.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:04.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:41:04.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:41:04.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:41:04.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:41:04.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:41:04.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:41:04.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:41:04.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:41:04.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:41:04.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:41:04.552 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.553 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.553 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.553 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:04.553 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:04.553 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.778940453000004, 2024-04-22T18:41:04.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:41:04.553 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.553 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.553 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:04.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:41:04.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:41:04.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:41:04.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:41:04.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:41:04.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:41:04.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:41:04.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:41:04.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:41:04.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:41:04.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:41:04.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:41:04.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:41:04.557 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.557 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.558 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.558 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:04.558 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:04.558 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.565102068000002, 2024-04-22T18:41:04.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:41:04.558 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.558 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.558 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:04.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:41:04.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:41:04.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:41:04.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:41:04.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:41:04.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:41:04.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:41:04.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:41:04.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:41:04.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:04.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:41:04.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:41:04.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:41:04.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:41:04.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:41:04.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:41:04.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:41:04.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:41:04.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:41:04.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:41:04.563 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.563 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.563 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.563 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:04.564 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:04.564 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.565068658000001, 2024-04-22T18:41:04.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:41:04.564 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.564 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.564 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:04.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:41:04.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:41:04.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:41:04.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:41:04.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:41:04.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:41:04.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:41:04.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:41:04.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:04.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:41:04.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:41:04.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:41:04.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:41:04.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:41:04.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:41:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:41:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:41:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:41:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:41:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.569 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:04.569 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:04.569 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.506186004, 2024-04-22T18:41:04.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:41:04.569 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.569 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.569 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:04.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:04.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:41:04.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:04.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:41:04.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:41:04.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:41:04.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:41:04.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:04.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:41:04.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:41:04.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:41:04.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:41:04.573 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.574 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.574 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.574 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:04.574 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:04.574 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.506047387999999, 2024-04-22T18:41:04.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:41:04.574 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.574 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.574 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:04.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:04.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:04.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:04.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:41:04.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:41:04.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:41:04.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:41:04.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:41:04.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:41:04.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:41:04.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:41:04.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:41:04.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:41:04.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:04.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:41:04.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:41:04.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:41:04.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:41:04.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:41:04.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:41:04.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:41:04.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:41:04.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:41:04.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:41:04.579 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.579 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.579 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.580 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:04.580 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:04.580 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.506028384999993, 2024-04-22T18:41:04.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:41:04.580 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.580 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.580 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:04.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:04.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:04.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:04.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:41:04.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:41:04.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:41:04.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:41:04.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:41:04.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:41:04.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:41:04.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:41:04.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:41:04.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:41:04.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:04.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:41:04.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:41:04.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:41:04.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:41:04.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:41:04.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:41:04.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:41:04.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:41:04.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:41:04.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:41:04.585 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.585 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.585 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.585 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:04.586 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:04.586 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.885015851999995, 2024-04-22T18:41:04.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:41:04.586 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.586 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.586 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.586 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.586 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.586 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:41:04.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.586 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:04.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:04.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:41:04.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:04.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:41:04.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:41:04.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:41:04.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:41:04.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:41:04.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:41:04.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:41:04.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:41:04.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:41:04.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:41:04.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:04.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:41:04.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:41:04.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:41:04.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:41:04.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:41:04.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:41:04.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:41:04.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:41:04.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:41:04.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:41:04.591 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.591 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.591 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.591 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:04.592 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:04.592 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.884496260999995, 2024-04-22T18:41:04.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:41:04.592 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.592 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:04.592 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.592 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.592 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.592 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:41:04.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.592 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:04.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:04.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:41:04.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:04.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:41:04.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:41:04.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:41:04.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:41:04.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:41:04.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:41:04.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:41:04.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:41:04.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:41:04.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:41:04.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:04.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:41:04.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:41:04.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:04.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:41:04.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:41:04.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:04.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:41:04.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:41:04.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:04.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:41:04.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:41:04.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:41:04.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:41:04.597 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.597 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.597 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.597 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:04.597 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:04.597 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 83.784343378000003, 2024-04-22T18:41:04.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:41:04.598 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:04.598 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:04.598 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:04.598 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:04.598 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:04.598 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:41:04.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.598 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:04.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:04.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:04.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:04.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:04.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:04.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:04.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:41:04.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:04.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:04.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:41:04.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:41:04.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:04.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:41:04.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:41:04.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:04.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:41:04.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:41:04.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:41:04.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:41:04.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:04.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:41:04.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:41:04.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:04.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:41:04.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:41:04.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:04.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:41:04.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:41:04.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:04.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:04.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:04.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:41:04.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:41:04.602 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.602 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.602 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.602 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:04.602 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:04.602 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:04.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:04.604 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:04.605+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:04.705 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:04.706+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:04.746 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:04.747+0000 7fa766d7d640 -1 osd.5 129 handle_osd_map injecting map crc failure 2024-04-22T18:41:04.748 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:04.748 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:04.748 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:04.748 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:04.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:04.806 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:04.807+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:04.899 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:04.899 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:04.899 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:04.899 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:04.899 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:04.907 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:04.908+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:04.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:04.921 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:04.923+0000 7f8607f36640 -1 osd.3 130 handle_osd_map injecting map crc failure 2024-04-22T18:41:04.924 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:41:05.008 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:05.008+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:05.055 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.79584998000001, 2024-04-22T18:41:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:41:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:41:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:41:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:41:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:41:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:41:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:41:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:41:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:41:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:41:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:41:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:41:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:41:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:41:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.79106755000001, 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:41:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:41:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:41:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:41:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:41:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:41:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:41:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:41:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:41:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:41:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:41:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:41:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:41:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:41:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:41:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.790973347, 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:41:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:41:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:41:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:41:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:41:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:41:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:41:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:41:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:41:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:41:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:41:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:41:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:41:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:41:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:41:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:41:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.79095245900001, 2024-04-22T18:41:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:41:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:41:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:41:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:41:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:41:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:41:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:41:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:41:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:41:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:41:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:41:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:41:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:41:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:41:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:41:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:41:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:41:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:41:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:41:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.007944796, 2024-04-22T18:41:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:41:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:41:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:41:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:41:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:41:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:41:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:41:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:41:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:41:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:41:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:41:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:41:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:41:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:41:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:41:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:41:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:41:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:41:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:41:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:41:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:41:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:41:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:41:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.00011339, 2024-04-22T18:41:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:41:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:41:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:41:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:41:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:41:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:41:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:41:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:41:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:41:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:41:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:41:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:41:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:41:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:41:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:41:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:41:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:41:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:41:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:41:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:41:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:41:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:41:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:41:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.848240759999996, 2024-04-22T18:41:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:41:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:41:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:41:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:41:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:41:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:41:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:41:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:41:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:41:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:41:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:41:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:41:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:41:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:41:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:41:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:41:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:41:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:41:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:41:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:41:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:41:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:41:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:41:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.847835852000003, 2024-04-22T18:41:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:41:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:41:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:41:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:41:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:41:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:41:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:41:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:41:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:41:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:41:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:41:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:41:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:41:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:41:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:41:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:41:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:41:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:41:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3213 4.a 4:5d2aee44:::smithi08937929-13:head [assert-version v66,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 42.486935963999997, 2024-04-22T18:41:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69928701400000004, 2024-04-22T18:41:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3213 2024-04-22T18:41:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568559+0000", 2024-04-22T18:41:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568567+0000", 2024-04-22T18:41:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-04-22T18:41:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568579+0000", 2024-04-22T18:41:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-04-22T18:41:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.668711+0000", 2024-04-22T18:41:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100131866 2024-04-22T18:41:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.768945+0000", 2024-04-22T18:41:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100234252 2024-04-22T18:41:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769002+0000", 2024-04-22T18:41:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7092999999999997e-05 2024-04-22T18:41:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.166404+0000", 2024-04-22T18:41:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39740241999999998 2024-04-22T18:41:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.191686+0000", 2024-04-22T18:41:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025281739000000001 2024-04-22T18:41:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267758+0000", 2024-04-22T18:41:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076071606999999999 2024-04-22T18:41:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267812+0000", 2024-04-22T18:41:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4478999999999998e-05 2024-04-22T18:41:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267848+0000", 2024-04-22T18:41:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6062999999999997e-05 2024-04-22T18:41:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3319 4.a 4:5d2aee44:::smithi08937929-13:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:41:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 38.595188213, 2024-04-22T18:41:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476623401, 2024-04-22T18:41:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3319 2024-04-22T18:41:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460306+0000", 2024-04-22T18:41:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460318+0000", 2024-04-22T18:41:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2896e-05 2024-04-22T18:41:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460345+0000", 2024-04-22T18:41:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6650000000000001e-05 2024-04-22T18:41:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460360+0000", 2024-04-22T18:41:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4729000000000001e-05 2024-04-22T18:41:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460502+0000", 2024-04-22T18:41:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141729 2024-04-22T18:41:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460644+0000", 2024-04-22T18:41:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142755 2024-04-22T18:41:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.936932+0000", 2024-04-22T18:41:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476288078 2024-04-22T18:41:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3412 4.a 4:519c08f7:::smithi08937929-15:head [write 2529878~394573] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.913353651999998, 2024-04-22T18:41:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48918094899999998, 2024-04-22T18:41:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3412 2024-04-22T18:41:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142141+0000", 2024-04-22T18:41:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142496+0000", 2024-04-22T18:41:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035444699999999998 2024-04-22T18:41:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142510+0000", 2024-04-22T18:41:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3841e-05 2024-04-22T18:41:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142518+0000", 2024-04-22T18:41:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8089999999999993e-06 2024-04-22T18:41:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142560+0000", 2024-04-22T18:41:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2097e-05 2024-04-22T18:41:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142637+0000", 2024-04-22T18:41:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6965000000000006e-05 2024-04-22T18:41:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142715+0000", 2024-04-22T18:41:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8694000000000002e-05 2024-04-22T18:41:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144179+0000", 2024-04-22T18:41:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146342 2024-04-22T18:41:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631229+0000", 2024-04-22T18:41:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48705018500000002 2024-04-22T18:41:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631281+0000", 2024-04-22T18:41:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2216999999999998e-05 2024-04-22T18:41:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631324+0000", 2024-04-22T18:41:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3344999999999999e-05 2024-04-22T18:41:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3413 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2924451] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.912972512000003, 2024-04-22T18:41:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58904086899999997, 2024-04-22T18:41:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3413 2024-04-22T18:41:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142523+0000", 2024-04-22T18:41:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142529+0000", 2024-04-22T18:41:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6049999999999997e-06 2024-04-22T18:41:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142535+0000", 2024-04-22T18:41:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4740000000000002e-06 2024-04-22T18:41:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142541+0000", 2024-04-22T18:41:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1839999999999998e-06 2024-04-22T18:41:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143553+0000", 2024-04-22T18:41:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010126689999999999 2024-04-22T18:41:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143659+0000", 2024-04-22T18:41:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105393 2024-04-22T18:41:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143825+0000", 2024-04-22T18:41:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165809 2024-04-22T18:41:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144329+0000", 2024-04-22T18:41:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050483799999999997 2024-04-22T18:41:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731480+0000", 2024-04-22T18:41:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58715100600000003 2024-04-22T18:41:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731522+0000", 2024-04-22T18:41:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2098000000000002e-05 2024-04-22T18:41:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731566+0000", 2024-04-22T18:41:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3066999999999999e-05 2024-04-22T18:41:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3414 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.912943908000003, 2024-04-22T18:41:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58926020199999996, 2024-04-22T18:41:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3414 2024-04-22T18:41:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142552+0000", 2024-04-22T18:41:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142556+0000", 2024-04-22T18:41:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0289999999999999e-06 2024-04-22T18:41:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142574+0000", 2024-04-22T18:41:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7266999999999999e-05 2024-04-22T18:41:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142581+0000", 2024-04-22T18:41:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-04-22T18:41:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144121+0000", 2024-04-22T18:41:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015397480000000001 2024-04-22T18:41:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144170+0000", 2024-04-22T18:41:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8803e-05 2024-04-22T18:41:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731624+0000", 2024-04-22T18:41:05.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58745365800000005 2024-04-22T18:41:05.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731712+0000", 2024-04-22T18:41:05.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8016999999999994e-05 2024-04-22T18:41:05.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731813+0000", 2024-04-22T18:41:05.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101817 2024-04-22T18:41:05.127 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.127 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.127 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.127 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:77 3.3 3:dcc8aeec:::3590d5741efdda41a26b62a0982727439225b7b4b51097b313f30d8dc10bf6f6:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:05.127 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:05.127 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.807925586, 2024-04-22T18:41:05.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011812878, 2024-04-22T18:41:05.127 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.127 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:05.128 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:05.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:05.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247571+0000", 2024-04-22T18:41:05.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:05.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247574+0000", 2024-04-22T18:41:05.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:41:05.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247577+0000", 2024-04-22T18:41:05.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3069999999999998e-06 2024-04-22T18:41:05.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247580+0000", 2024-04-22T18:41:05.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8870000000000001e-06 2024-04-22T18:41:05.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.340928+0000", 2024-04-22T18:41:05.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093348678000000004 2024-04-22T18:41:05.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:05.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341122+0000", 2024-04-22T18:41:05.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000193227 2024-04-22T18:41:05.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618335+0000", 2024-04-22T18:41:05.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.277213775 2024-04-22T18:41:05.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:05.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618422+0000", 2024-04-22T18:41:05.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6537000000000002e-05 2024-04-22T18:41:05.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954387+0000", 2024-04-22T18:41:05.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33596509699999999 2024-04-22T18:41:05.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954548+0000", 2024-04-22T18:41:05.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016042199999999999 2024-04-22T18:41:05.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:05.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954858+0000", 2024-04-22T18:41:05.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000310396 2024-04-22T18:41:05.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:05.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157019+0000", 2024-04-22T18:41:05.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20216071299999999 2024-04-22T18:41:05.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:05.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259213+0000", 2024-04-22T18:41:05.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102194734 2024-04-22T18:41:05.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:05.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259262+0000", 2024-04-22T18:41:05.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9052999999999999e-05 2024-04-22T18:41:05.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259384+0000", 2024-04-22T18:41:05.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122052 2024-04-22T18:41:05.134 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.134 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.134 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.134 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:78 3.3 3:df7e90d2:::ab0657e24bd96704775de2a378746bc22309c5a7366a5eaf9b4e1f1f2bef0a4c:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:05.134 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:05.134 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.807910389, 2024-04-22T18:41:05.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011974739, 2024-04-22T18:41:05.134 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.134 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:05.134 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:05.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:05.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247586+0000", 2024-04-22T18:41:05.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:05.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247589+0000", 2024-04-22T18:41:05.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.181e-06 2024-04-22T18:41:05.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247592+0000", 2024-04-22T18:41:05.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0230000000000001e-06 2024-04-22T18:41:05.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247595+0000", 2024-04-22T18:41:05.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.428e-06 2024-04-22T18:41:05.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341160+0000", 2024-04-22T18:41:05.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093564571999999999 2024-04-22T18:41:05.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:05.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341264+0000", 2024-04-22T18:41:05.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010391999999999999 2024-04-22T18:41:05.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619269+0000", 2024-04-22T18:41:05.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27800575100000002 2024-04-22T18:41:05.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:05.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619346+0000", 2024-04-22T18:41:05.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6459999999999996e-05 2024-04-22T18:41:05.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.155957+0000", 2024-04-22T18:41:05.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53661109299999998 2024-04-22T18:41:05.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156049+0000", 2024-04-22T18:41:05.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2272e-05 2024-04-22T18:41:05.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:05.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156236+0000", 2024-04-22T18:41:05.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000186808 2024-04-22T18:41:05.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:05.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.158088+0000", 2024-04-22T18:41:05.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018522079999999999 2024-04-22T18:41:05.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:05.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259421+0000", 2024-04-22T18:41:05.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101332718 2024-04-22T18:41:05.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:05.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259460+0000", 2024-04-22T18:41:05.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8912000000000003e-05 2024-04-22T18:41:05.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259562+0000", 2024-04-22T18:41:05.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010175600000000001 2024-04-22T18:41:05.140 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.140 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:80 3.3 3:f90f47f2:::e5603f596d699ee3e27e255225eeb90a9f0c1b4a4d099d1c586b19019220aa3f:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.807871312, 2024-04-22T18:41:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0121096759999999, 2024-04-22T18:41:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:05.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247625+0000", 2024-04-22T18:41:05.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:05.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247628+0000", 2024-04-22T18:41:05.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6129999999999998e-06 2024-04-22T18:41:05.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247631+0000", 2024-04-22T18:41:05.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3869999999999999e-06 2024-04-22T18:41:05.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247634+0000", 2024-04-22T18:41:05.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9919999999999999e-06 2024-04-22T18:41:05.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341550+0000", 2024-04-22T18:41:05.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093915584999999996 2024-04-22T18:41:05.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:05.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341700+0000", 2024-04-22T18:41:05.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015000300000000001 2024-04-22T18:41:05.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620744+0000", 2024-04-22T18:41:05.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27904399200000002 2024-04-22T18:41:05.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:05.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620794+0000", 2024-04-22T18:41:05.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9579000000000001e-05 2024-04-22T18:41:05.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157597+0000", 2024-04-22T18:41:05.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53680343799999997 2024-04-22T18:41:05.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157678+0000", 2024-04-22T18:41:05.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0829999999999997e-05 2024-04-22T18:41:05.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:05.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157832+0000", 2024-04-22T18:41:05.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154184 2024-04-22T18:41:05.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:05.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259572+0000", 2024-04-22T18:41:05.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10174003600000001 2024-04-22T18:41:05.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:05.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259601+0000", 2024-04-22T18:41:05.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8841999999999999e-05 2024-04-22T18:41:05.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:05.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259638+0000", 2024-04-22T18:41:05.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7308999999999998e-05 2024-04-22T18:41:05.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259736+0000", 2024-04-22T18:41:05.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7283999999999999e-05 2024-04-22T18:41:05.147 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.147 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.147 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.147 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:135 3.3 3:f89134e5:::3e4e73f5d4a0122957865f0faeac2483204ee98d7d497ef3b54afe2a02b39a3d:head [read 0~802304 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:05.148 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:05.148 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.440056233, 2024-04-22T18:41:05.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.64370653499999997, 2024-04-22T18:41:05.148 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.148 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:05.148 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:05.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:05.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615438+0000", 2024-04-22T18:41:05.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:05.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615449+0000", 2024-04-22T18:41:05.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1219999999999999e-05 2024-04-22T18:41:05.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615460+0000", 2024-04-22T18:41:05.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0403e-05 2024-04-22T18:41:05.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.715588+0000", 2024-04-22T18:41:05.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100128881 2024-04-22T18:41:05.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722533+0000", 2024-04-22T18:41:05.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069444010000000002 2024-04-22T18:41:05.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:05.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722658+0000", 2024-04-22T18:41:05.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012560699999999999 2024-04-22T18:41:05.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826437+0000", 2024-04-22T18:41:05.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103778194 2024-04-22T18:41:05.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:05.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826532+0000", 2024-04-22T18:41:05.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5111000000000005e-05 2024-04-22T18:41:05.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258787+0000", 2024-04-22T18:41:05.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43225529099999999 2024-04-22T18:41:05.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258905+0000", 2024-04-22T18:41:05.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118348 2024-04-22T18:41:05.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259148+0000", 2024-04-22T18:41:05.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000242093 2024-04-22T18:41:05.153 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.153 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.153 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.154 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3890 4.1 4:83c955bf:::smithi08937929-28:head [write 2699879~469889] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:05.154 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:05.154 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.441381389, 2024-04-22T18:41:05.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49042297000000001, 2024-04-22T18:41:05.154 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.154 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:05.154 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.154 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.154 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.154 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3890 2024-04-22T18:41:05.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.154 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:05.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:05.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614115+0000", 2024-04-22T18:41:05.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:05.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614496+0000", 2024-04-22T18:41:05.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038181700000000001 2024-04-22T18:41:05.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614509+0000", 2024-04-22T18:41:05.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2381000000000001e-05 2024-04-22T18:41:05.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614517+0000", 2024-04-22T18:41:05.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0879999999999995e-06 2024-04-22T18:41:05.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615610+0000", 2024-04-22T18:41:05.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010933519999999999 2024-04-22T18:41:05.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615655+0000", 2024-04-22T18:41:05.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4966000000000003e-05 2024-04-22T18:41:05.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:05.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615769+0000", 2024-04-22T18:41:05.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113986 2024-04-22T18:41:05.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704882+0000", 2024-04-22T18:41:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089112825000000007 2024-04-22T18:41:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104488+0000", 2024-04-22T18:41:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39960572999999999 2024-04-22T18:41:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104517+0000", 2024-04-22T18:41:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9006e-05 2024-04-22T18:41:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104539+0000", 2024-04-22T18:41:05.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2073e-05 2024-04-22T18:41:05.159 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.159 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.159 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.159 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3891 4.1 4:83c955bf:::smithi08937929-28:head [setxattr _header (56) in=7b,truncate 3630345] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:05.159 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:05.159 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.440973158, 2024-04-22T18:41:05.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59018588699999996, 2024-04-22T18:41:05.159 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.159 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:05.159 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.160 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.160 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.160 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3891 2024-04-22T18:41:05.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.160 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:05.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:05.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614523+0000", 2024-04-22T18:41:05.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:05.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614528+0000", 2024-04-22T18:41:05.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0379999999999999e-06 2024-04-22T18:41:05.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614540+0000", 2024-04-22T18:41:05.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018000000000001e-05 2024-04-22T18:41:05.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614546+0000", 2024-04-22T18:41:05.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4990000000000002e-06 2024-04-22T18:41:05.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704438+0000", 2024-04-22T18:41:05.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089892044000000004 2024-04-22T18:41:05.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704504+0000", 2024-04-22T18:41:05.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6365000000000006e-05 2024-04-22T18:41:05.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:05.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704602+0000", 2024-04-22T18:41:05.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8048999999999996e-05 2024-04-22T18:41:05.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:05.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.806370+0000", 2024-04-22T18:41:05.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10176768 2024-04-22T18:41:05.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:05.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204629+0000", 2024-04-22T18:41:05.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39825917999999999 2024-04-22T18:41:05.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:05.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204673+0000", 2024-04-22T18:41:05.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3516999999999997e-05 2024-04-22T18:41:05.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204710+0000", 2024-04-22T18:41:05.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7391000000000002e-05 2024-04-22T18:41:05.164 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.164 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.164 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.165 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3892 4.1 4:83c955bf:::smithi08937929-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:05.165 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:05.165 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.440946577, 2024-04-22T18:41:05.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59031281800000002, 2024-04-22T18:41:05.165 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:05.165 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:05.165 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:05.165 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:05.165 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:05.165 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3892 2024-04-22T18:41:05.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.165 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:05.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:05.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:05.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:05.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:05.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:05.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:05.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614550+0000", 2024-04-22T18:41:05.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:05.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:05.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614553+0000", 2024-04-22T18:41:05.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0879999999999999e-06 2024-04-22T18:41:05.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:05.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614559+0000", 2024-04-22T18:41:05.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9259999999999997e-06 2024-04-22T18:41:05.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:05.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614565+0000", 2024-04-22T18:41:05.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7950000000000004e-06 2024-04-22T18:41:05.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704818+0000", 2024-04-22T18:41:05.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090252624000000004 2024-04-22T18:41:05.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:05.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704866+0000", 2024-04-22T18:41:05.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8667e-05 2024-04-22T18:41:05.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:05.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204742+0000", 2024-04-22T18:41:05.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987551899999999 2024-04-22T18:41:05.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:05.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204799+0000", 2024-04-22T18:41:05.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7299000000000001e-05 2024-04-22T18:41:05.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:05.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:05.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:05.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204863+0000", 2024-04-22T18:41:05.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4455000000000005e-05 2024-04-22T18:41:05.169 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.169 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.169 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.170 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:05.170 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:05.170 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:05.171 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:05.172 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:05.172+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:05.218 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:41:05.272 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:05.273+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:05.314 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:05.314 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:05.314 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:05.314 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:05.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:05.338 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:05.338 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:05.339 INFO:tasks.rados.rados.0.smithi089.stdout:1743: set_chunk oid 28 offset: 3065344 length: 859648 target oid 3b5a41efae8260015fb0562ac673c25a9d4becbab8172a50240e0a2be7e3fefe offset: 0 snap -1 2024-04-22T18:41:05.340 INFO:tasks.rados.rados.0.smithi089.stdout:1744: tier_evict oid 23 2024-04-22T18:41:05.340 INFO:tasks.rados.rados.0.smithi089.stdout:1744: tier_evict oid 23 snap -1 2024-04-22T18:41:05.340 INFO:tasks.rados.rados.0.smithi089.stdout:1728: expect (ObjNum 336 snap 0 seq_num 336) 2024-04-22T18:41:05.355 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:05.356 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:05.372 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:05.372 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:05.372 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:05.373+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:05.387 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:05.388 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:05.404 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:05.404 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:05.420 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:05.420 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:05.437 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:05.437 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:05.454 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:05.455 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:05.462 INFO:tasks.rados.rados.0.smithi089.stdout:1727: finishing write tid 1 to smithi08937929-8 2024-04-22T18:41:05.462 INFO:tasks.rados.rados.0.smithi089.stdout:1727: finishing write tid 2 to smithi08937929-8 2024-04-22T18:41:05.462 INFO:tasks.rados.rados.0.smithi089.stdout:1728: done (14 left) 2024-04-22T18:41:05.462 INFO:tasks.rados.rados.0.smithi089.stdout:1745: set_chunk oid 26 target oid 2024-04-22T18:41:05.467 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:05.468 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:05.473 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:05.474+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:05.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:05.574 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:05.575+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:05.621 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:05.621 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:05.630 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:05.659 INFO:tasks.rados.rados.0.smithi089.stdout:1745: set_chunk oid 26 offset: 873984 length: 2621952 target oid baf310ff854e4260036711bf8b1cb8b84ea79e4b0e0379b7575e20ee98fd0e62 offset: 0 snap -1 2024-04-22T18:41:05.659 INFO:tasks.rados.rados.0.smithi089.stdout:1746: delete oid 2 current snap is 0 2024-04-22T18:41:05.660 INFO:tasks.rados.rados.0.smithi089.stdout:1727: finishing write tid 3 to smithi08937929-8 2024-04-22T18:41:05.660 INFO:tasks.rados.rados.0.smithi089.stdout:1727: finishing write tid 4 to smithi08937929-8 2024-04-22T18:41:05.660 INFO:tasks.rados.rados.0.smithi089.stdout:1727: finishing write tid 5 to smithi08937929-8 2024-04-22T18:41:05.660 INFO:tasks.rados.rados.0.smithi089.stdout:1727: oid 8 updating version 0 to 43 2024-04-22T18:41:05.660 INFO:tasks.rados.rados.0.smithi089.stdout:1727: oid 8 version 43 is already newer than 41 2024-04-22T18:41:05.660 INFO:tasks.rados.rados.0.smithi089.stdout:1727: oid 8 updating version 43 to 44 2024-04-22T18:41:05.660 INFO:tasks.rados.rados.0.smithi089.stdout:1727: oid 8 version 44 is already newer than 42 2024-04-22T18:41:05.660 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 44 (ObjNum 356 snap 0 seq_num 356) dirty exists 2024-04-22T18:41:05.660 INFO:tasks.rados.rados.0.smithi089.stdout:1727: left oid 8 (ObjNum 356 snap 0 seq_num 356) 2024-04-22T18:41:05.660 INFO:tasks.rados.rados.0.smithi089.stdout:1738: got (0) Success 2024-04-22T18:41:05.660 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 51 (ObjNum 226 snap 0 seq_num 226) dirty exists 2024-04-22T18:41:05.660 INFO:tasks.rados.rados.0.smithi089.stdout:1736: expect (ObjNum 352 snap 0 seq_num 352) 2024-04-22T18:41:05.674 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:05.675+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:05.734 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:05.734 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:05.743 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:05.784 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:05.785+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:05.791 INFO:tasks.rados.rados.0.smithi089.stdout:1742: finishing write tid 1 to smithi08937929-41 2024-04-22T18:41:05.792 INFO:tasks.rados.rados.0.smithi089.stdout:1742: finishing write tid 2 to smithi08937929-41 2024-04-22T18:41:05.792 INFO:tasks.rados.rados.0.smithi089.stdout:1742: finishing write tid 3 to smithi08937929-41 2024-04-22T18:41:05.792 INFO:tasks.rados.rados.0.smithi089.stdout:1742: finishing write tid 4 to smithi08937929-41 2024-04-22T18:41:05.792 INFO:tasks.rados.rados.0.smithi089.stdout:1742: oid 41 updating version 0 to 138 2024-04-22T18:41:05.792 INFO:tasks.rados.rados.0.smithi089.stdout:1742: oid 41 updating version 138 to 139 2024-04-22T18:41:05.792 INFO:tasks.rados.rados.0.smithi089.stdout:1742: oid 41 version 139 is already newer than 137 2024-04-22T18:41:05.792 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 139 (ObjNum 358 snap 0 seq_num 358) dirty exists 2024-04-22T18:41:05.792 INFO:tasks.rados.rados.0.smithi089.stdout:1742: left oid 41 (ObjNum 358 snap 0 seq_num 358) 2024-04-22T18:41:05.792 INFO:tasks.rados.rados.0.smithi089.stdout:1727: done (15 left) 2024-04-22T18:41:05.792 INFO:tasks.rados.rados.0.smithi089.stdout:1735: done (14 left) 2024-04-22T18:41:05.792 INFO:tasks.rados.rados.0.smithi089.stdout:1736: done (13 left) 2024-04-22T18:41:05.792 INFO:tasks.rados.rados.0.smithi089.stdout:1738: done (12 left) 2024-04-22T18:41:05.793 INFO:tasks.rados.rados.0.smithi089.stdout:1742: done (11 left) 2024-04-22T18:41:05.793 INFO:tasks.rados.rados.0.smithi089.stdout:1746: done (10 left) 2024-04-22T18:41:05.793 INFO:tasks.rados.rados.0.smithi089.stdout:1747: tier_evict oid 3 2024-04-22T18:41:05.793 INFO:tasks.rados.rados.0.smithi089.stdout:1747: tier_evict oid 3 snap -1 2024-04-22T18:41:05.793 INFO:tasks.rados.rados.0.smithi089.stdout:1733: expect (ObjNum 316 snap 0 seq_num 316) 2024-04-22T18:41:05.876 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:05.878+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:05.885 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:05.885 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:05.894 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:05.977 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:05.978+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:06.015 INFO:tasks.rados.rados.0.smithi089.stdout:1740: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:41:06.039 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:06.039 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:06.048 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:06.078 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:06.079+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:06.146 INFO:tasks.rados.rados.0.smithi089.stdout:1739: expect (ObjNum 295 snap 0 seq_num 295) 2024-04-22T18:41:06.150 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:06.151 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:06.160 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:06.203 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:06.205+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:06.258 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:06.259+0000 7f3cfee88640 -1 osd.6 131 handle_osd_map injecting map crc failure 2024-04-22T18:41:06.262 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:06.262 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:06.272 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:06.309 INFO:tasks.rados.rados.0.smithi089.stdout:1734: expect (ObjNum 263 snap 0 seq_num 263) 2024-04-22T18:41:06.310 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:06.311+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:06.381 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:06.382+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:06.412 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:06.412 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:06.421 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:06.458 INFO:tasks.rados.rados.0.smithi089.stdout:1737: expect (ObjNum 304 snap 0 seq_num 304) 2024-04-22T18:41:06.482 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:06.483+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:06.523 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:06.524 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:06.533 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:06.563 INFO:tasks.rados.rados.0.smithi089.stdout:1744: got (22) Invalid argument 2024-04-22T18:41:06.563 INFO:tasks.rados.rados.0.smithi089.stdout:1741: got (0) Success 2024-04-22T18:41:06.564 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 80 (ObjNum 326 snap 0 seq_num 326) dirty exists 2024-04-22T18:41:06.564 INFO:tasks.rados.rados.0.smithi089.stdout:1732: expect (ObjNum 287 snap 0 seq_num 287) 2024-04-22T18:41:06.607 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:06.608+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:06.667 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:06.668 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:06.677 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:06.719 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:06.720+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:06.735 INFO:tasks.rados.rados.0.smithi089.stdout:1743: finishing set_chunk to oid 28 2024-04-22T18:41:06.736 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 165 (ObjNum 355 snap 0 seq_num 355) dirty exists 2024-04-22T18:41:06.736 INFO:tasks.rados.rados.0.smithi089.stdout:1732: done (10 left) 2024-04-22T18:41:06.736 INFO:tasks.rados.rados.0.smithi089.stdout:1733: done (9 left) 2024-04-22T18:41:06.736 INFO:tasks.rados.rados.0.smithi089.stdout:1734: done (8 left) 2024-04-22T18:41:06.736 INFO:tasks.rados.rados.0.smithi089.stdout:1737: done (7 left) 2024-04-22T18:41:06.736 INFO:tasks.rados.rados.0.smithi089.stdout:1739: done (6 left) 2024-04-22T18:41:06.736 INFO:tasks.rados.rados.0.smithi089.stdout:1740: done (5 left) 2024-04-22T18:41:06.736 INFO:tasks.rados.rados.0.smithi089.stdout:1741: done (4 left) 2024-04-22T18:41:06.736 INFO:tasks.rados.rados.0.smithi089.stdout:1743: done (3 left) 2024-04-22T18:41:06.736 INFO:tasks.rados.rados.0.smithi089.stdout:1744: done (2 left) 2024-04-22T18:41:06.736 INFO:tasks.rados.rados.0.smithi089.stdout:1748: read oid 18 snap -1 2024-04-22T18:41:06.736 INFO:tasks.rados.rados.0.smithi089.stdout:1748: expect (ObjNum 307 snap 0 seq_num 307) 2024-04-22T18:41:06.736 INFO:tasks.rados.rados.0.smithi089.stdout:1749: read oid 11 snap -1 2024-04-22T18:41:06.736 INFO:tasks.rados.rados.0.smithi089.stdout:1749: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:41:06.737 INFO:tasks.rados.rados.0.smithi089.stdout:1750: write (excl) oid 8 current snap is 0 2024-04-22T18:41:06.737 INFO:tasks.rados.rados.0.smithi089.stdout:1747: got (22) Invalid argument 2024-04-22T18:41:06.737 INFO:tasks.rados.rados.0.smithi089.stdout:1745: finishing set_chunk to oid 26 2024-04-22T18:41:06.737 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 134 (ObjNum 353 snap 0 seq_num 353) dirty exists 2024-04-22T18:41:06.737 INFO:tasks.rados.rados.0.smithi089.stdout:1750: seq_num 360 ranges {450520=558231,1483047=515125} 2024-04-22T18:41:06.741 INFO:tasks.rados.rados.0.smithi089.stdout:1750: writing smithi08937929-8 from 450520 to 1008751 tid 1 2024-04-22T18:41:06.746 INFO:tasks.rados.rados.0.smithi089.stdout:1750: writing smithi08937929-8 from 1483047 to 1998172 tid 2 2024-04-22T18:41:06.746 INFO:tasks.rados.rados.0.smithi089.stdout:1745: done (4 left) 2024-04-22T18:41:06.746 INFO:tasks.rados.rados.0.smithi089.stdout:1747: done (3 left) 2024-04-22T18:41:06.747 INFO:tasks.rados.rados.0.smithi089.stdout:1751: set_chunk oid 27 target oid 2024-04-22T18:41:06.785 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:06.786+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:06.821 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:06.821 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:06.830 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:06.915 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:06.916+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:06.940 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:41:06.950 INFO:tasks.rados.rados.0.smithi089.stdout:1751: set_chunk oid 27 offset: 757248 length: 116224 target oid c4ad57e0dba5f7f0a7ff5034cdca14474ca6e5d03370b2f1d6ec711be2acd74a offset: 0 snap -1 2024-04-22T18:41:06.950 INFO:tasks.rados.rados.0.smithi089.stdout:1752: write (excl) oid 4 current snap is 0 2024-04-22T18:41:06.950 INFO:tasks.rados.rados.0.smithi089.stdout:1752: seq_num 361 ranges {467327=633480,1825176=242151} 2024-04-22T18:41:06.955 INFO:tasks.rados.rados.0.smithi089.stdout:1752: writing smithi08937929-4 from 467327 to 1100807 tid 1 2024-04-22T18:41:06.961 INFO:tasks.rados.rados.0.smithi089.stdout:1752: writing smithi08937929-4 from 1825176 to 2067327 tid 2 2024-04-22T18:41:06.961 INFO:tasks.rados.rados.0.smithi089.stdout:1753: set_chunk oid 31 target oid 2024-04-22T18:41:06.961 INFO:tasks.rados.rados.0.smithi089.stdout:1753: set_chunk oid 31 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:41:06.961 INFO:tasks.rados.rados.0.smithi089.stdout:1754: write (excl) oid 12 current snap is 0 2024-04-22T18:41:06.961 INFO:tasks.rados.rados.0.smithi089.stdout:1754: seq_num 362 ranges {484134=708729,1883658=200476} 2024-04-22T18:41:06.969 INFO:tasks.rados.rados.0.smithi089.stdout:1754: writing smithi08937929-12 from 484134 to 1192863 tid 1 2024-04-22T18:41:06.975 INFO:tasks.rados.rados.0.smithi089.stdout:1754: writing smithi08937929-12 from 1883658 to 2084134 tid 2 2024-04-22T18:41:06.975 INFO:tasks.rados.rados.0.smithi089.stdout:1755: set_chunk oid 41 target oid 2024-04-22T18:41:06.982 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:06.982 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:06.986 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:06.987+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:06.992 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:07.087 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:07.089+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:07.141 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:07.141 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:07.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:07.169 INFO:tasks.rados.rados.0.smithi089.stdout:1755: set_chunk oid 41 offset: 547328 length: 647680 target oid a9a73adc037e707662cc496a50b77560a2a7b5749b463dfc3fb6ce530f1cf2dc offset: 0 snap -1 2024-04-22T18:41:07.169 INFO:tasks.rados.rados.0.smithi089.stdout:1756: delete oid 35 current snap is 0 2024-04-22T18:41:07.169 INFO:tasks.rados.rados.0.smithi089.stdout:1748: expect (ObjNum 307 snap 0 seq_num 307) 2024-04-22T18:41:07.190 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:07.191+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:07.253 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:07.253 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:07.262 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:07.302 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:07.304+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:07.370 INFO:tasks.rados.rados.0.smithi089.stdout:1749: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:41:07.376 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:07.376 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:07.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:07.428 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:07.429+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:07.466 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:41:07.487 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:07.487 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:07.496 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:07.509 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:07.511+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:07.578 INFO:tasks.rados.rados.0.smithi089.stdout:1750: finishing write tid 1 to smithi08937929-8 2024-04-22T18:41:07.578 INFO:tasks.rados.rados.0.smithi089.stdout:1750: finishing write tid 2 to smithi08937929-8 2024-04-22T18:41:07.578 INFO:tasks.rados.rados.0.smithi089.stdout:1750: finishing write tid 3 to smithi08937929-8 2024-04-22T18:41:07.578 INFO:tasks.rados.rados.0.smithi089.stdout:1750: finishing write tid 4 to smithi08937929-8 2024-04-22T18:41:07.578 INFO:tasks.rados.rados.0.smithi089.stdout:1750: oid 8 updating version 0 to 47 2024-04-22T18:41:07.578 INFO:tasks.rados.rados.0.smithi089.stdout:1750: oid 8 version 47 is already newer than 45 2024-04-22T18:41:07.578 INFO:tasks.rados.rados.0.smithi089.stdout:1750: oid 8 version 47 is already newer than 46 2024-04-22T18:41:07.578 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 47 (ObjNum 360 snap 0 seq_num 360) dirty exists 2024-04-22T18:41:07.578 INFO:tasks.rados.rados.0.smithi089.stdout:1750: left oid 8 (ObjNum 360 snap 0 seq_num 360) 2024-04-22T18:41:07.578 INFO:tasks.rados.rados.0.smithi089.stdout:1751: finishing set_chunk to oid 27 2024-04-22T18:41:07.579 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 757248 length: 116224 2024-04-22T18:41:07.579 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 0 (ObjNum 287 snap 0 seq_num 287) dirty exists 2024-04-22T18:41:07.579 INFO:tasks.rados.rados.0.smithi089.stdout:1752: finishing write tid 1 to smithi08937929-4 2024-04-22T18:41:07.579 INFO:tasks.rados.rados.0.smithi089.stdout:1748: done (8 left) 2024-04-22T18:41:07.579 INFO:tasks.rados.rados.0.smithi089.stdout:1749: done (7 left) 2024-04-22T18:41:07.579 INFO:tasks.rados.rados.0.smithi089.stdout:1750: done (6 left) 2024-04-22T18:41:07.579 INFO:tasks.rados.rados.0.smithi089.stdout:1751: done (5 left) 2024-04-22T18:41:07.579 INFO:tasks.rados.rados.0.smithi089.stdout:1756: done (4 left) 2024-04-22T18:41:07.579 INFO:tasks.rados.rados.0.smithi089.stdout:1757: read oid 35 snap -1 2024-04-22T18:41:07.579 INFO:tasks.rados.rados.0.smithi089.stdout:1757: expect deleted 2024-04-22T18:41:07.579 INFO:tasks.rados.rados.0.smithi089.stdout:1758: read oid 5 snap -1 2024-04-22T18:41:07.579 INFO:tasks.rados.rados.0.smithi089.stdout:1758: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:41:07.579 INFO:tasks.rados.rados.0.smithi089.stdout:1759: write (excl) oid 43 current snap is 0 2024-04-22T18:41:07.579 INFO:tasks.rados.rados.0.smithi089.stdout:1752: finishing write tid 2 to smithi08937929-4 2024-04-22T18:41:07.580 INFO:tasks.rados.rados.0.smithi089.stdout:1752: finishing write tid 3 to smithi08937929-4 2024-04-22T18:41:07.580 INFO:tasks.rados.rados.0.smithi089.stdout:1752: finishing write tid 4 to smithi08937929-4 2024-04-22T18:41:07.580 INFO:tasks.rados.rados.0.smithi089.stdout:1752: oid 4 updating version 0 to 136 2024-04-22T18:41:07.580 INFO:tasks.rados.rados.0.smithi089.stdout:1752: oid 4 updating version 136 to 138 2024-04-22T18:41:07.580 INFO:tasks.rados.rados.0.smithi089.stdout:1752: oid 4 version 138 is already newer than 137 2024-04-22T18:41:07.580 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 138 (ObjNum 361 snap 0 seq_num 361) dirty exists 2024-04-22T18:41:07.580 INFO:tasks.rados.rados.0.smithi089.stdout:1752: left oid 4 (ObjNum 361 snap 0 seq_num 361) 2024-04-22T18:41:07.580 INFO:tasks.rados.rados.0.smithi089.stdout:1753: finishing set_chunk to oid 31 2024-04-22T18:41:07.580 INFO:tasks.rados.rados.0.smithi089.stdout:1753: got expected ENOENT (src dne) 2024-04-22T18:41:07.580 INFO:tasks.rados.rados.0.smithi089.stdout:1754: finishing write tid 1 to smithi08937929-12 2024-04-22T18:41:07.580 INFO:tasks.rados.rados.0.smithi089.stdout:1754: finishing write tid 2 to smithi08937929-12 2024-04-22T18:41:07.580 INFO:tasks.rados.rados.0.smithi089.stdout:1754: finishing write tid 3 to smithi08937929-12 2024-04-22T18:41:07.580 INFO:tasks.rados.rados.0.smithi089.stdout:1754: finishing write tid 4 to smithi08937929-12 2024-04-22T18:41:07.581 INFO:tasks.rados.rados.0.smithi089.stdout:1754: oid 12 updating version 0 to 120 2024-04-22T18:41:07.581 INFO:tasks.rados.rados.0.smithi089.stdout:1754: oid 12 updating version 120 to 121 2024-04-22T18:41:07.581 INFO:tasks.rados.rados.0.smithi089.stdout:1754: oid 12 updating version 121 to 122 2024-04-22T18:41:07.581 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 12 v 122 (ObjNum 362 snap 0 seq_num 362) dirty exists 2024-04-22T18:41:07.581 INFO:tasks.rados.rados.0.smithi089.stdout:1754: left oid 12 (ObjNum 362 snap 0 seq_num 362) 2024-04-22T18:41:07.581 INFO:tasks.rados.rados.0.smithi089.stdout:1755: finishing set_chunk to oid 41 2024-04-22T18:41:07.581 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 139 (ObjNum 358 snap 0 seq_num 358) dirty exists 2024-04-22T18:41:07.581 INFO:tasks.rados.rados.0.smithi089.stdout:1759: seq_num 364 ranges {517748=459227,1600622=406110} 2024-04-22T18:41:07.583 INFO:tasks.rados.rados.0.smithi089.stdout:1759: writing smithi08937929-43 from 517748 to 976975 tid 1 2024-04-22T18:41:07.588 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:07.589 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:07.589 INFO:tasks.rados.rados.0.smithi089.stdout:1759: writing smithi08937929-43 from 1600622 to 2006732 tid 2 2024-04-22T18:41:07.589 INFO:tasks.rados.rados.0.smithi089.stdout:1752: done (6 left) 2024-04-22T18:41:07.589 INFO:tasks.rados.rados.0.smithi089.stdout:1753: done (5 left) 2024-04-22T18:41:07.589 INFO:tasks.rados.rados.0.smithi089.stdout:1754: done (4 left) 2024-04-22T18:41:07.590 INFO:tasks.rados.rados.0.smithi089.stdout:1755: done (3 left) 2024-04-22T18:41:07.590 INFO:tasks.rados.rados.0.smithi089.stdout:1760: write oid 8 current snap is 0 2024-04-22T18:41:07.590 INFO:tasks.rados.rados.0.smithi089.stdout:1760: seq_num 365 ranges {534555=650829,1659104=475451} 2024-04-22T18:41:07.593 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:07.594+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:07.597 INFO:tasks.rados.rados.0.smithi089.stdout:1760: writing smithi08937929-8 from 534555 to 1185384 tid 1 2024-04-22T18:41:07.603 INFO:tasks.rados.rados.0.smithi089.stdout:1760: writing smithi08937929-8 from 1659104 to 2134555 tid 2 2024-04-22T18:41:07.603 INFO:tasks.rados.rados.0.smithi089.stdout:1761: set_chunk oid 2 target oid 2024-04-22T18:41:07.603 INFO:tasks.rados.rados.0.smithi089.stdout:1761: set_chunk oid 2 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:41:07.603 INFO:tasks.rados.rados.0.smithi089.stdout:1762: tier_promote oid 39 2024-04-22T18:41:07.603 INFO:tasks.rados.rados.0.smithi089.stdout:1758: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:41:07.612 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:07.613 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:07.628 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:07.629 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:07.640 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:07.640 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:07.644 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:07.644 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:07.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:07.660 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:07.660 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:07.677 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:07.677 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:07.693 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:07.694 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:07.711 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:07.711 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:07.734 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:07.735+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:07.794 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:07.794 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:07.794 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:07.794 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:07.794 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:07.795+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:07.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:07.843 INFO:tasks.rados.rados.0.smithi089.stdout:1759: finishing write tid 1 to smithi08937929-43 2024-04-22T18:41:07.843 INFO:tasks.rados.rados.0.smithi089.stdout:1760: finishing write tid 1 to smithi08937929-8 2024-04-22T18:41:07.844 INFO:tasks.rados.rados.0.smithi089.stdout:1760: finishing write tid 2 to smithi08937929-8 2024-04-22T18:41:07.844 INFO:tasks.rados.rados.0.smithi089.stdout:1761: finishing set_chunk to oid 2 2024-04-22T18:41:07.844 INFO:tasks.rados.rados.0.smithi089.stdout:1761: got expected ENOENT (src dne) 2024-04-22T18:41:07.844 INFO:tasks.rados.rados.0.smithi089.stdout:1759: finishing write tid 2 to smithi08937929-43 2024-04-22T18:41:07.844 INFO:tasks.rados.rados.0.smithi089.stdout:1759: finishing write tid 3 to smithi08937929-43 2024-04-22T18:41:07.844 INFO:tasks.rados.rados.0.smithi089.stdout:1759: finishing write tid 4 to smithi08937929-43 2024-04-22T18:41:07.844 INFO:tasks.rados.rados.0.smithi089.stdout:1759: oid 43 updating version 0 to 67 2024-04-22T18:41:07.844 INFO:tasks.rados.rados.0.smithi089.stdout:1759: oid 43 updating version 67 to 68 2024-04-22T18:41:07.844 INFO:tasks.rados.rados.0.smithi089.stdout:1759: oid 43 version 68 is already newer than 66 2024-04-22T18:41:07.844 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 68 (ObjNum 364 snap 0 seq_num 364) dirty exists 2024-04-22T18:41:07.844 INFO:tasks.rados.rados.0.smithi089.stdout:1759: left oid 43 (ObjNum 364 snap 0 seq_num 364) 2024-04-22T18:41:07.845 INFO:tasks.rados.rados.0.smithi089.stdout:1760: finishing write tid 3 to smithi08937929-8 2024-04-22T18:41:07.845 INFO:tasks.rados.rados.0.smithi089.stdout:1757: done (5 left) 2024-04-22T18:41:07.845 INFO:tasks.rados.rados.0.smithi089.stdout:1758: done (4 left) 2024-04-22T18:41:07.845 INFO:tasks.rados.rados.0.smithi089.stdout:1759: done (3 left) 2024-04-22T18:41:07.845 INFO:tasks.rados.rados.0.smithi089.stdout:1761: done (2 left) 2024-04-22T18:41:07.845 INFO:tasks.rados.rados.0.smithi089.stdout:1763: read oid 23 snap -1 2024-04-22T18:41:07.845 INFO:tasks.rados.rados.0.smithi089.stdout:1763: expect (ObjNum 339 snap 0 seq_num 339) 2024-04-22T18:41:07.845 INFO:tasks.rados.rados.0.smithi089.stdout:1764: read oid 29 snap -1 2024-04-22T18:41:07.845 INFO:tasks.rados.rados.0.smithi089.stdout:1764: expect (ObjNum 295 snap 0 seq_num 295) 2024-04-22T18:41:07.845 INFO:tasks.rados.rados.0.smithi089.stdout:1765: read oid 47 snap -1 2024-04-22T18:41:07.845 INFO:tasks.rados.rados.0.smithi089.stdout:1765: expect (ObjNum 323 snap 0 seq_num 323) 2024-04-22T18:41:07.846 INFO:tasks.rados.rados.0.smithi089.stdout:1766: tier_evict oid 21 2024-04-22T18:41:07.846 INFO:tasks.rados.rados.0.smithi089.stdout:1766: tier_evict oid 21 snap -1 2024-04-22T18:41:07.846 INFO:tasks.rados.rados.0.smithi089.stdout:1767: set_chunk oid 40 target oid 2024-04-22T18:41:07.895 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:07.896+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:07.934 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 7, 2, 4] out_osds: [5, 0, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:41:07.934 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:41:07.935 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-04-22T18:41:07.935 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:41:07.937 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:41:07.948 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:07.948 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:07.948 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:07.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:07.995 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:07.997+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:08.080 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:08.081+0000 7fec55bea640 -1 osd.7 132 handle_osd_map injecting map crc failure 2024-04-22T18:41:08.097 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:08.098+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:08.103 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:08.103 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:08.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:08.103 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:08.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.103 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:41:08.103 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.009367508, 2024-04-22T18:41:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:41:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:41:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:08.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:41:08.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:08.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:41:08.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:41:08.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:41:08.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:41:08.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:41:08.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:41:08.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:41:08.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:41:08.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:41:08.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:41:08.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:08.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:41:08.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:41:08.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:41:08.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:41:08.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:41:08.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:41:08.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:41:08.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:41:08.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:41:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:41:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:41:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.10986963299999, 2024-04-22T18:41:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:41:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:08.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:08.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:41:08.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:08.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:41:08.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:41:08.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:41:08.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:41:08.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:41:08.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:41:08.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:41:08.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:41:08.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:08.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:41:08.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:41:08.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:41:08.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:41:08.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:41:08.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:41:08.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:08.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:41:08.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:41:08.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:41:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:41:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:41:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:41:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:41:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:41:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 129.00156984200001, 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:41:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:41:08.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:08.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:41:08.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:41:08.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:41:08.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:41:08.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:41:08.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:41:08.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:41:08.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:41:08.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:41:08.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:41:08.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:08.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:41:08.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:41:08.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:41:08.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:41:08.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:41:08.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:41:08.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:41:08.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:41:08.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:41:08.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:41:08.120 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 115.670026214, 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:41:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:41:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:41:08.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:41:08.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:41:08.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:41:08.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:41:08.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:41:08.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:41:08.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:41:08.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:41:08.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:08.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:41:08.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:41:08.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:41:08.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:41:08.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:41:08.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:41:08.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:41:08.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:41:08.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:41:08.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:41:08.126 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.126 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.126 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.126 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:08.126 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:08.126 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 115.578940406, 2024-04-22T18:41:08.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:41:08.126 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.126 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.126 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.127 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.127 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.127 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:41:08.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.127 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:08.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:08.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:41:08.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:08.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:41:08.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:41:08.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:41:08.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:41:08.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:41:08.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:41:08.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:41:08.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:41:08.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:41:08.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:41:08.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:08.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:41:08.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:41:08.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:41:08.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:41:08.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:41:08.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:41:08.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:41:08.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:41:08.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:41:08.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:41:08.132 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.132 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.132 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.132 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:08.132 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:08.132 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 115.573140146, 2024-04-22T18:41:08.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:41:08.132 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.132 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.133 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.133 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.133 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.133 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:41:08.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.133 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:08.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:08.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:41:08.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:08.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:41:08.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:41:08.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:41:08.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:41:08.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:41:08.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:41:08.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:41:08.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:41:08.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:41:08.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:41:08.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:41:08.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:41:08.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:41:08.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:41:08.138 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.138 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.138 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.138 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:08.138 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:08.138 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 115.56839980300001, 2024-04-22T18:41:08.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:41:08.138 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.138 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.139 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.139 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.139 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.139 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:41:08.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.139 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:08.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:08.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:41:08.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:08.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:41:08.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:41:08.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:41:08.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:41:08.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:41:08.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:41:08.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:41:08.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:41:08.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:41:08.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:41:08.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:08.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:41:08.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:41:08.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:41:08.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:41:08.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:41:08.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:41:08.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:41:08.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:41:08.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:41:08.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:41:08.145 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.145 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.145 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.145 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:08.145 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:08.145 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 115.56792690499999, 2024-04-22T18:41:08.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:41:08.145 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.145 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.145 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.145 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.146 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.146 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:41:08.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.146 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:08.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:08.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:41:08.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:08.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:41:08.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:41:08.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:41:08.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:41:08.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:41:08.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:41:08.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:41:08.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:41:08.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:41:08.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:41:08.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:08.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:41:08.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:41:08.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:41:08.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:41:08.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:41:08.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:41:08.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:41:08.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:41:08.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:41:08.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:41:08.150 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.150 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.150 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.151 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:08.151 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:08.151 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 115.567906984, 2024-04-22T18:41:08.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:41:08.151 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.151 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:08.151 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.151 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.151 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.151 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:41:08.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.151 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:41:08.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:41:08.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:41:08.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:41:08.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:41:08.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:41:08.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:41:08.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:41:08.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:08.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:41:08.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:41:08.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:41:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:41:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:41:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:41:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:41:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:41:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.155 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.156 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:08.156 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:08.156 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 110.258583219, 2024-04-22T18:41:08.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:41:08.156 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.156 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.156 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:08.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:08.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:41:08.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:08.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:41:08.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:41:08.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:41:08.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:41:08.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:41:08.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:41:08.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:41:08.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:41:08.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:41:08.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:41:08.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:41:08.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:41:08.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:41:08.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:41:08.160 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.160 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.161 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.161 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:08.161 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:08.161 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 110.258375683, 2024-04-22T18:41:08.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:41:08.161 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.161 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.161 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:08.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:08.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:41:08.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:08.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:41:08.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:41:08.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:41:08.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:41:08.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:41:08.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:41:08.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:41:08.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:41:08.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:41:08.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:41:08.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:08.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:41:08.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:41:08.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:41:08.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:41:08.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:41:08.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:41:08.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:41:08.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:41:08.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:41:08.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:41:08.166 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.166 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.166 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.166 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:08.166 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:08.166 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 91.392725949999999, 2024-04-22T18:41:08.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:41:08.166 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.166 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:08.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:41:08.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:41:08.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:41:08.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:41:08.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:41:08.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:41:08.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:41:08.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:41:08.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:41:08.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:41:08.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:41:08.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:41:08.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 91.178887564999997, 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:08.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:08.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:41:08.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:08.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:41:08.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:41:08.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:41:08.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:41:08.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:41:08.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:41:08.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:41:08.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:41:08.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:41:08.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:41:08.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:08.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:41:08.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:41:08.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:41:08.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:41:08.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:41:08.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:41:08.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:41:08.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:41:08.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:41:08.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:41:08.176 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.176 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.176 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.176 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:08.176 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:08.176 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 91.178854154999996, 2024-04-22T18:41:08.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:41:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:41:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:41:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:41:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:41:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:41:08.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:41:08.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:41:08.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:41:08.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:41:08.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:41:08.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:08.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:41:08.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:41:08.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:41:08.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:41:08.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:41:08.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:41:08.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:41:08.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:41:08.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:41:08.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:41:08.181 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.181 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.181 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.182 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:08.182 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:08.182 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 91.119971500999995, 2024-04-22T18:41:08.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:41:08.182 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.182 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.182 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:08.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:08.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:41:08.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:08.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:41:08.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:41:08.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:41:08.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:41:08.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:41:08.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:41:08.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:41:08.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:41:08.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:41:08.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:41:08.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:08.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:41:08.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:41:08.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:41:08.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:41:08.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:41:08.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:41:08.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:41:08.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:41:08.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:41:08.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:41:08.187 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.187 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.187 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.187 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:08.187 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:08.187 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 91.119832884999994, 2024-04-22T18:41:08.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:41:08.187 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.188 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.188 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:08.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:08.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:08.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:08.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:41:08.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:41:08.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:41:08.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:41:08.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:41:08.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:41:08.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:41:08.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:41:08.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:41:08.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:41:08.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:08.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:41:08.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:41:08.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:41:08.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:41:08.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:41:08.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:41:08.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:41:08.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:41:08.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:41:08.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:41:08.192 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.192 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.192 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.193 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:08.193 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:08.193 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 91.119813882000003, 2024-04-22T18:41:08.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:41:08.193 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.193 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.193 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:08.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:08.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:08.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:08.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:41:08.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:41:08.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:41:08.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:41:08.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:41:08.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:41:08.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:41:08.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:41:08.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:41:08.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:41:08.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:08.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:41:08.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:41:08.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:41:08.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:41:08.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:41:08.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:41:08.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:41:08.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:41:08.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:41:08.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:41:08.198 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.198 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.198 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.198 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:08.198 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:08.198 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.498801349000004, 2024-04-22T18:41:08.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:41:08.198 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.198 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.198 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.198 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.198 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.199 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:41:08.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.199 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:08.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:08.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:41:08.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:08.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:41:08.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:41:08.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:41:08.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:41:08.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:41:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:41:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:41:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:41:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:41:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:41:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:08.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:41:08.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:41:08.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:41:08.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:41:08.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:41:08.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:41:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:41:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:41:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:41:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:41:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.204 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:08.204 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:08.204 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.498281758000005, 2024-04-22T18:41:08.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:41:08.204 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.204 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.204 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.204 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.204 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.204 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:41:08.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.204 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:08.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:08.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:41:08.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:08.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:41:08.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:41:08.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:41:08.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:41:08.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:41:08.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:41:08.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:41:08.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:41:08.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:41:08.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:41:08.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:08.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:41:08.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:41:08.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:41:08.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:41:08.209 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.209 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.209 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.209 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:08.209 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:08.209 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.398128874999998, 2024-04-22T18:41:08.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:41:08.209 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.209 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:08.209 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.210 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.210 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.210 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:41:08.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.210 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:08.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:08.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:41:08.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:08.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:41:08.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:41:08.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:41:08.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:41:08.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:41:08.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:41:08.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:41:08.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:41:08.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:08.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:41:08.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:41:08.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:41:08.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:41:08.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:41:08.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:41:08.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:41:08.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:41:08.215 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.215 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.216 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.216 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.216 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.216 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:08.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:08.218 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:08.218+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:08.220 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:41:08.220 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:41:08.231 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-04-22T18:41:07.936348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285498160,"num_objects":213,"num_object_clones":0,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":128,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":4240,"num_read_kb":3750999,"num_write":2467,"num_write_kb":607978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":693,"num_bytes_recovered":293173607,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":32,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2606,"ondisk_log_size":2606,"up":78,"acting":78,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":512636,"kb_used_data":354276,"kb_used_omap":7,"kb_used_meta":155512,"kb_avail":471346564,"statfs":{"total":483183820800,"available":482658881536,"internally_reserved":0,"allocated":362778624,"data_stored":368805298,"data_compressed":10308520,"data_compressed_allocated":10997760,"data_compressed_original":22343680,"omap_allocated":7963,"internal_metadata":159244517},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4362303,"num_objects":-5,"num_object_clones":0,"num_object_copies":-10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":50,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":87,"num_read_kb":108017,"num_write":32,"num_write_kb":10821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":46523663,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.950185"},"pg_stats":[{"pgid":"4.19","version":"105'114","reported_seq":499,"reported_epoch":131,"state":"active+remapped+backfill_wait","last_fresh":"2024-04-22T18:41:05.540161+0000","last_change":"2024-04-22T18:41:05.438520+0000","last_active":"2024-04-22T18:41:05.540161+0000","last_peered":"2024-04-22T18:41:05.540161+0000","last_clean":"2024-04-22T18:41:01.651846+0000","last_became_active":"2024-04-22T18:41:05.337692+0000","last_became_peered":"2024-04-22T18:41:05.337692+0000","last_unstale":"2024-04-22T18:41:05.540161+0000","last_undegraded":"2024-04-22T18:41:05.540161+0000","last_fullsized":"2024-04-22T18:41:05.540161+0000","mapping_epoch":130,"log_start":"84'88","ondisk_log_start":"84'88","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":26,"log_dups_size":106,"ondisk_log_size":26,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6680865,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":93028,"num_write":19,"num_write_kb":5735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4990114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"122'64","reported_seq":332,"reported_epoch":131,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:41:05.348655+0000","last_change":"2024-04-22T18:41:05.348597+0000","last_active":"2024-04-22T18:41:05.348655+0000","last_peered":"2024-04-22T18:41:05.348655+0000","last_clean":"2024-04-22T18:41:01.854833+0000","last_became_active":"2024-04-22T18:41:05.127081+0000","last_became_peered":"2024-04-22T18:41:05.127081+0000","last_unstale":"2024-04-22T18:41:05.348655+0000","last_undegraded":"2024-04-22T18:41:05.348655+0000","last_fullsized":"2024-04-22T18:41:05.348655+0000","mapping_epoch":130,"log_start":"79'52","ondisk_log_start":"79'52","created":110,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":61,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5613571,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":119,"num_read_kb":168057,"num_write":55,"num_write_kb":15750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7945966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,3],"avail_no_missing":["7","3","6"],"object_location_counts":[{"shards":"3,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"107'52","reported_seq":339,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.927343+0000","last_change":"2024-04-22T18:41:00.078939+0000","last_active":"2024-04-22T18:41:04.927343+0000","last_peered":"2024-04-22T18:41:04.927343+0000","last_clean":"2024-04-22T18:41:04.927343+0000","last_became_active":"2024-04-22T18:41:00.078669+0000","last_became_peered":"2024-04-22T18:41:00.078669+0000","last_unstale":"2024-04-22T18:41:04.927343+0000","last_undegraded":"2024-04-22T18:41:04.927343+0000","last_fullsized":"2024-04-22T18:41:04.927343+0000","mapping_epoch":125,"log_start":"83'49","ondisk_log_start":"83'49","created":110,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":5,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092497,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":66769,"num_write":16,"num_write_kb":3993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2254766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"123'95","reported_seq":406,"reported_epoch":131,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:41:04.929141+0000","last_change":"2024-04-22T18:40:56.200294+0000","last_active":"2024-04-22T18:41:04.929141+0000","last_peered":"2024-04-22T18:41:04.929141+0000","last_clean":"2024-04-22T18:40:53.248441+0000","last_became_active":"2024-04-22T18:40:55.998994+0000","last_became_peered":"2024-04-22T18:40:55.998994+0000","last_unstale":"2024-04-22T18:41:04.929141+0000","last_undegraded":"2024-04-22T18:41:04.929141+0000","last_fullsized":"2024-04-22T18:41:04.929141+0000","mapping_epoch":121,"log_start":"96'71","ondisk_log_start":"96'71","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":2,"log_size":24,"log_dups_size":81,"ondisk_log_size":24,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.22461577699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5428711,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":73379,"num_write":27,"num_write_kb":8334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1847758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.15","version":"54'29","reported_seq":73,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:05.027012+0000","last_change":"2024-04-22T18:40:55.328642+0000","last_active":"2024-04-22T18:41:05.027012+0000","last_peered":"2024-04-22T18:41:05.027012+0000","last_clean":"2024-04-22T18:41:05.027012+0000","last_became_active":"2024-04-22T18:40:13.180144+0000","last_became_peered":"2024-04-22T18:40:13.180144+0000","last_unstale":"2024-04-22T18:41:05.027012+0000","last_undegraded":"2024-04-22T18:41:05.027012+0000","last_fullsized":"2024-04-22T18:41:05.027012+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'29","last_scrub_stamp":"2024-04-22T18:40:55.328529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.328529+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:10.967288+0000","scrub_duration":0.30217712000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"51'19","reported_seq":73,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.927327+0000","last_change":"2024-04-22T18:40:57.754351+0000","last_active":"2024-04-22T18:41:04.927327+0000","last_peered":"2024-04-22T18:41:04.927327+0000","last_clean":"2024-04-22T18:41:04.927327+0000","last_became_active":"2024-04-22T18:40:10.978840+0000","last_became_peered":"2024-04-22T18:40:10.978840+0000","last_unstale":"2024-04-22T18:41:04.927327+0000","last_undegraded":"2024-04-22T18:41:04.927327+0000","last_fullsized":"2024-04-22T18:41:04.927327+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"51'19","last_scrub_stamp":"2024-04-22T18:40:57.754261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:57.754261+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:17.927283+0000","scrub_duration":0.59184710600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"131'83","reported_seq":23,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-22T18:41:05.968940+0000","last_change":"2024-04-22T18:41:04.562945+0000","last_active":"2024-04-22T18:41:05.968940+0000","last_peered":"2024-04-22T18:41:05.968940+0000","last_clean":"2024-04-22T18:41:05.968940+0000","last_became_active":"2024-04-22T18:41:04.139159+0000","last_became_peered":"2024-04-22T18:41:04.139159+0000","last_unstale":"2024-04-22T18:41:05.968940+0000","last_undegraded":"2024-04-22T18:41:05.968940+0000","last_fullsized":"2024-04-22T18:41:05.968940+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'75","last_scrub_stamp":"2024-04-22T18:40:22.670296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.670296+0000","objects_scrubbed":2,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:49.126896+0000","scrub_duration":0.20951747300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218623,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":37607,"num_write":6,"num_write_kb":3167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12459808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"126'45","reported_seq":255,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.928919+0000","last_change":"2024-04-22T18:40:59.453830+0000","last_active":"2024-04-22T18:41:04.928919+0000","last_peered":"2024-04-22T18:41:04.928919+0000","last_clean":"2024-04-22T18:41:04.928919+0000","last_became_active":"2024-04-22T18:40:58.107873+0000","last_became_peered":"2024-04-22T18:40:58.107873+0000","last_unstale":"2024-04-22T18:41:04.928919+0000","last_undegraded":"2024-04-22T18:41:04.928919+0000","last_fullsized":"2024-04-22T18:41:04.928919+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4008503,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":111,"num_read_kb":105496,"num_write":26,"num_write_kb":7134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2641926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"129'139","reported_seq":438,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.927693+0000","last_change":"2024-04-22T18:40:26.844287+0000","last_active":"2024-04-22T18:41:04.927693+0000","last_peered":"2024-04-22T18:41:04.927693+0000","last_clean":"2024-04-22T18:41:04.927693+0000","last_became_active":"2024-04-22T18:40:26.742351+0000","last_became_peered":"2024-04-22T18:40:26.742351+0000","last_unstale":"2024-04-22T18:41:04.927693+0000","last_undegraded":"2024-04-22T18:41:04.927693+0000","last_fullsized":"2024-04-22T18:41:04.927693+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'113","last_scrub_stamp":"2024-04-22T18:40:20.291988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.291988+0000","objects_scrubbed":2,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:22.684852+0000","scrub_duration":0.10270805500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5580023,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":21946,"num_write":31,"num_write_kb":8732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3940523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"31'5","reported_seq":165,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.927325+0000","last_change":"2024-04-22T18:40:10.212897+0000","last_active":"2024-04-22T18:41:04.927325+0000","last_peered":"2024-04-22T18:41:04.927325+0000","last_clean":"2024-04-22T18:41:04.927325+0000","last_became_active":"2024-04-22T18:39:59.265189+0000","last_became_peered":"2024-04-22T18:39:59.265189+0000","last_unstale":"2024-04-22T18:41:04.927325+0000","last_undegraded":"2024-04-22T18:41:04.927325+0000","last_fullsized":"2024-04-22T18:41:04.927325+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":5,"last_scrub":"31'5","last_scrub_stamp":"2024-04-22T18:40:10.212815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.212815+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:17.620115+0000","scrub_duration":0.0015068670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.927033+0000","last_change":"2024-04-22T18:40:58.207745+0000","last_active":"2024-04-22T18:41:04.927033+0000","last_peered":"2024-04-22T18:41:04.927033+0000","last_clean":"2024-04-22T18:41:04.927033+0000","last_became_active":"2024-04-22T18:40:58.207451+0000","last_became_peered":"2024-04-22T18:40:58.207451+0000","last_unstale":"2024-04-22T18:41:04.927033+0000","last_undegraded":"2024-04-22T18:41:04.927033+0000","last_fullsized":"2024-04-22T18:41:04.927033+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:19.136933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:19.136933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:38.724946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"131'166","reported_seq":591,"reported_epoch":131,"state":"active+recovery_wait+remapped","last_fresh":"2024-04-22T18:41:05.645379+0000","last_change":"2024-04-22T18:41:05.231050+0000","last_active":"2024-04-22T18:41:05.645379+0000","last_peered":"2024-04-22T18:41:05.645379+0000","last_clean":"2024-04-22T18:41:02.169513+0000","last_became_active":"2024-04-22T18:41:05.229296+0000","last_became_peered":"2024-04-22T18:41:05.229296+0000","last_unstale":"2024-04-22T18:41:05.645379+0000","last_undegraded":"2024-04-22T18:41:05.645379+0000","last_fullsized":"2024-04-22T18:41:05.645379+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13731418,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":265,"num_read_kb":310309,"num_write":61,"num_write_kb":18016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":34184641,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":141,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.930270+0000","last_change":"2024-04-22T18:41:04.037422+0000","last_active":"2024-04-22T18:41:04.930270+0000","last_peered":"2024-04-22T18:41:04.930270+0000","last_clean":"2024-04-22T18:41:04.930270+0000","last_became_active":"2024-04-22T18:41:04.037248+0000","last_became_peered":"2024-04-22T18:41:04.037248+0000","last_unstale":"2024-04-22T18:41:04.930270+0000","last_undegraded":"2024-04-22T18:41:04.930270+0000","last_fullsized":"2024-04-22T18:41:04.930270+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"124'24","reported_seq":203,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.927410+0000","last_change":"2024-04-22T18:39:54.378188+0000","last_active":"2024-04-22T18:41:04.927410+0000","last_peered":"2024-04-22T18:41:04.927410+0000","last_clean":"2024-04-22T18:41:04.927410+0000","last_became_active":"2024-04-22T18:39:46.805569+0000","last_became_peered":"2024-04-22T18:39:46.805569+0000","last_unstale":"2024-04-22T18:41:04.927410+0000","last_undegraded":"2024-04-22T18:41:04.927410+0000","last_fullsized":"2024-04-22T18:41:04.927410+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'13","last_scrub_stamp":"2024-04-22T18:39:54.377980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.377980+0000","objects_scrubbed":1,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30260881099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3916064,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":64,"num_read_kb":108324,"num_write":11,"num_write_kb":3229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":150,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:05.027091+0000","last_change":"2024-04-22T18:41:04.428281+0000","last_active":"2024-04-22T18:41:05.027091+0000","last_peered":"2024-04-22T18:41:05.027091+0000","last_clean":"2024-04-22T18:41:05.027091+0000","last_became_active":"2024-04-22T18:41:04.427985+0000","last_became_peered":"2024-04-22T18:41:04.427985+0000","last_unstale":"2024-04-22T18:41:05.027091+0000","last_undegraded":"2024-04-22T18:41:05.027091+0000","last_fullsized":"2024-04-22T18:41:05.027091+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:09.538542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:09.538542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:36.605250+0000","scrub_duration":0.35201753200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"127'86","reported_seq":327,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.926823+0000","last_change":"2024-04-22T18:40:35.964284+0000","last_active":"2024-04-22T18:41:04.926823+0000","last_peered":"2024-04-22T18:41:04.926823+0000","last_clean":"2024-04-22T18:41:04.926823+0000","last_became_active":"2024-04-22T18:40:34.910015+0000","last_became_peered":"2024-04-22T18:40:34.910015+0000","last_unstale":"2024-04-22T18:41:04.926823+0000","last_undegraded":"2024-04-22T18:41:04.926823+0000","last_fullsized":"2024-04-22T18:41:04.926823+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'67","last_scrub_stamp":"2024-04-22T18:39:53.916646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.916646+0000","objects_scrubbed":2,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:06.623573+0000","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554796,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":87189,"num_write":17,"num_write_kb":4399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7812071,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":133,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.930677+0000","last_change":"2024-04-22T18:40:35.014535+0000","last_active":"2024-04-22T18:41:04.930677+0000","last_peered":"2024-04-22T18:41:04.930677+0000","last_clean":"2024-04-22T18:41:04.930677+0000","last_became_active":"2024-04-22T18:40:35.014154+0000","last_became_peered":"2024-04-22T18:40:35.014154+0000","last_unstale":"2024-04-22T18:41:04.930677+0000","last_undegraded":"2024-04-22T18:41:04.930677+0000","last_fullsized":"2024-04-22T18:41:04.930677+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"64'38","reported_seq":270,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.929451+0000","last_change":"2024-04-22T18:41:04.141946+0000","last_active":"2024-04-22T18:41:04.929451+0000","last_peered":"2024-04-22T18:41:04.929451+0000","last_clean":"2024-04-22T18:41:04.929451+0000","last_became_active":"2024-04-22T18:41:03.930474+0000","last_became_peered":"2024-04-22T18:41:03.930474+0000","last_unstale":"2024-04-22T18:41:04.929451+0000","last_undegraded":"2024-04-22T18:41:04.929451+0000","last_fullsized":"2024-04-22T18:41:04.929451+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2932821,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":105,"num_read_kb":149879,"num_write":20,"num_write_kb":5595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":11374836,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"131'305","reported_seq":935,"reported_epoch":132,"state":"active+recovering+remapped","last_fresh":"2024-04-22T18:41:06.570216+0000","last_change":"2024-04-22T18:41:05.443806+0000","last_active":"2024-04-22T18:41:06.570216+0000","last_peered":"2024-04-22T18:41:06.570216+0000","last_clean":"2024-04-22T18:40:48.123953+0000","last_became_active":"2024-04-22T18:41:05.133127+0000","last_became_peered":"2024-04-22T18:41:05.133127+0000","last_unstale":"2024-04-22T18:41:06.570216+0000","last_undegraded":"2024-04-22T18:41:06.570216+0000","last_fullsized":"2024-04-22T18:41:06.570216+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'293","last_scrub_stamp":"2024-04-22T18:40:53.582354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:53.582354+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:17.561029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33593320,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":257,"num_read_kb":4163,"num_write":368,"num_write_kb":93216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":38218668,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[1,4],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,3","objects":3},{"shards":"1,3,6","objects":2},{"shards":"1,4","objects":7},{"shards":"1,6","objects":28}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":94,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.929526+0000","last_change":"2024-04-22T18:40:57.953864+0000","last_active":"2024-04-22T18:41:04.929526+0000","last_peered":"2024-04-22T18:41:04.929526+0000","last_clean":"2024-04-22T18:41:04.929526+0000","last_became_active":"2024-04-22T18:40:57.853059+0000","last_became_peered":"2024-04-22T18:40:57.853059+0000","last_unstale":"2024-04-22T18:41:04.929526+0000","last_undegraded":"2024-04-22T18:41:04.929526+0000","last_fullsized":"2024-04-22T18:41:04.929526+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:55.558446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.558446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:05.813196+0000","scrub_duration":0.56126016300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"115'32","reported_seq":252,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.930591+0000","last_change":"2024-04-22T18:40:21.426510+0000","last_active":"2024-04-22T18:41:04.930591+0000","last_peered":"2024-04-22T18:41:04.930591+0000","last_clean":"2024-04-22T18:41:04.930591+0000","last_became_active":"2024-04-22T18:39:51.943071+0000","last_became_peered":"2024-04-22T18:39:51.943071+0000","last_unstale":"2024-04-22T18:41:04.930591+0000","last_undegraded":"2024-04-22T18:41:04.930591+0000","last_fullsized":"2024-04-22T18:41:04.930591+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-04-22T18:40:21.426350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.426350+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.678685+0000","scrub_duration":0.30334366400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2605118,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":78568,"num_write":4,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"129'223","reported_seq":578,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:05.027711+0000","last_change":"2024-04-22T18:40:20.465005+0000","last_active":"2024-04-22T18:41:05.027711+0000","last_peered":"2024-04-22T18:41:05.027711+0000","last_clean":"2024-04-22T18:41:05.027711+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:41:05.027711+0000","last_undegraded":"2024-04-22T18:41:05.027711+0000","last_fullsized":"2024-04-22T18:41:05.027711+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'176","last_scrub_stamp":"2024-04-22T18:40:20.464799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.464799+0000","objects_scrubbed":31,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:45.914514+0000","scrub_duration":0.30629975799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26351664,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":195,"num_read_kb":3220,"num_write":281,"num_write_kb":63747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":90,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.128796+0000","last_change":"2024-04-22T18:40:10.403338+0000","last_active":"2024-04-22T18:41:04.128796+0000","last_peered":"2024-04-22T18:41:04.128796+0000","last_clean":"2024-04-22T18:41:04.128796+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:41:04.128796+0000","last_undegraded":"2024-04-22T18:41:04.128796+0000","last_fullsized":"2024-04-22T18:41:04.128796+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:10.403189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.403189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:34.232014+0000","scrub_duration":0.201433005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"124'61","reported_seq":345,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:05.029309+0000","last_change":"2024-04-22T18:40:42.750352+0000","last_active":"2024-04-22T18:41:05.029309+0000","last_peered":"2024-04-22T18:41:05.029309+0000","last_clean":"2024-04-22T18:41:05.029309+0000","last_became_active":"2024-04-22T18:40:42.750084+0000","last_became_peered":"2024-04-22T18:40:42.750084+0000","last_unstale":"2024-04-22T18:41:05.029309+0000","last_undegraded":"2024-04-22T18:41:05.029309+0000","last_fullsized":"2024-04-22T18:41:05.029309+0000","mapping_epoch":106,"log_start":"83'49","ondisk_log_start":"83'49","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":49,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3747969,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":92629,"num_write":25,"num_write_kb":7639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2254765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"131'268","reported_seq":787,"reported_epoch":132,"state":"active+recovering+remapped","last_fresh":"2024-04-22T18:41:06.476067+0000","last_change":"2024-04-22T18:41:05.437289+0000","last_active":"2024-04-22T18:41:06.476067+0000","last_peered":"2024-04-22T18:41:06.476067+0000","last_clean":"2024-04-22T18:41:01.661225+0000","last_became_active":"2024-04-22T18:41:05.030149+0000","last_became_peered":"2024-04-22T18:41:05.030149+0000","last_unstale":"2024-04-22T18:41:06.476067+0000","last_undegraded":"2024-04-22T18:41:06.476067+0000","last_fullsized":"2024-04-22T18:41:06.476067+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'209","last_scrub_stamp":"2024-04-22T18:40:18.683442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:18.683442+0000","objects_scrubbed":24,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:45.661944+0000","scrub_duration":0.204344365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32145897,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":30,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":236,"num_read_kb":6472,"num_write":328,"num_write_kb":84515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":29254713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[3,4],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,4","objects":6},{"shards":"3,7","objects":30}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":134,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.927293+0000","last_change":"2024-04-22T18:40:22.521828+0000","last_active":"2024-04-22T18:41:04.927293+0000","last_peered":"2024-04-22T18:41:04.927293+0000","last_clean":"2024-04-22T18:41:04.927293+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:41:04.927293+0000","last_undegraded":"2024-04-22T18:41:04.927293+0000","last_fullsized":"2024-04-22T18:41:04.927293+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.521638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:43.407910+0000","scrub_duration":0.30190717299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"128'93","reported_seq":387,"reported_epoch":131,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:41:05.029299+0000","last_change":"2024-04-22T18:40:43.158709+0000","last_active":"2024-04-22T18:41:05.029299+0000","last_peered":"2024-04-22T18:41:05.029299+0000","last_clean":"2024-04-22T18:40:32.014508+0000","last_became_active":"2024-04-22T18:40:43.046938+0000","last_became_peered":"2024-04-22T18:40:43.046938+0000","last_unstale":"2024-04-22T18:41:05.029299+0000","last_undegraded":"2024-04-22T18:41:05.029299+0000","last_fullsized":"2024-04-22T18:41:05.029299+0000","mapping_epoch":104,"log_start":"96'71","ondisk_log_start":"96'71","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":81,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.22461577699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":99605,"num_write":23,"num_write_kb":6431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,0],"avail_no_missing":["1","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"128'260","reported_seq":718,"reported_epoch":132,"state":"active+recovering+remapped","last_fresh":"2024-04-22T18:41:06.379195+0000","last_change":"2024-04-22T18:41:05.143862+0000","last_active":"2024-04-22T18:41:06.379195+0000","last_peered":"2024-04-22T18:41:06.379195+0000","last_clean":"2024-04-22T18:40:55.265085+0000","last_became_active":"2024-04-22T18:41:04.934615+0000","last_became_peered":"2024-04-22T18:41:04.934615+0000","last_unstale":"2024-04-22T18:41:06.379195+0000","last_undegraded":"2024-04-22T18:41:06.379195+0000","last_fullsized":"2024-04-22T18:41:06.379195+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'212","last_scrub_stamp":"2024-04-22T18:40:21.870855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.870855+0000","objects_scrubbed":46,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:31.749601+0000","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40325745,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":44,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":242,"num_read_kb":8320,"num_write":330,"num_write_kb":76410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":27663656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[3,4],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"1,3","objects":1},{"shards":"1,3,6","objects":5},{"shards":"3,4","objects":9},{"shards":"3,6","objects":38}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":145,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.128747+0000","last_change":"2024-04-22T18:40:07.420765+0000","last_active":"2024-04-22T18:41:04.128747+0000","last_peered":"2024-04-22T18:41:04.128747+0000","last_clean":"2024-04-22T18:41:04.128747+0000","last_became_active":"2024-04-22T18:40:02.741862+0000","last_became_peered":"2024-04-22T18:40:02.741862+0000","last_unstale":"2024-04-22T18:41:04.128747+0000","last_undegraded":"2024-04-22T18:41:04.128747+0000","last_fullsized":"2024-04-22T18:41:04.128747+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:07.420618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:07.420618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.642486+0000","scrub_duration":0.30164867299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":313,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.927342+0000","last_change":"2024-04-22T18:40:58.257511+0000","last_active":"2024-04-22T18:41:04.927342+0000","last_peered":"2024-04-22T18:41:04.927342+0000","last_clean":"2024-04-22T18:41:04.927342+0000","last_became_active":"2024-04-22T18:40:57.952303+0000","last_became_peered":"2024-04-22T18:40:57.952303+0000","last_unstale":"2024-04-22T18:41:04.927342+0000","last_undegraded":"2024-04-22T18:41:04.927342+0000","last_fullsized":"2024-04-22T18:41:04.927342+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:39:53.346787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.346787+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:22.397484+0000","scrub_duration":0.20273052499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"128'65","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-22T18:41:05.956701+0000","last_change":"2024-04-22T18:41:04.681901+0000","last_active":"2024-04-22T18:41:05.956701+0000","last_peered":"2024-04-22T18:41:05.956701+0000","last_clean":"2024-04-22T18:41:05.956701+0000","last_became_active":"2024-04-22T18:41:04.227104+0000","last_became_peered":"2024-04-22T18:41:04.227104+0000","last_unstale":"2024-04-22T18:41:05.956701+0000","last_undegraded":"2024-04-22T18:41:05.956701+0000","last_fullsized":"2024-04-22T18:41:05.956701+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'38","last_scrub_stamp":"2024-04-22T18:39:55.746067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:55.746067+0000","objects_scrubbed":1,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:16.501045+0000","scrub_duration":0.60877772299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7748060,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":62,"num_read_kb":101952,"num_write":47,"num_write_kb":12021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15782313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"121'57","reported_seq":309,"reported_epoch":131,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:41:04.927492+0000","last_change":"2024-04-22T18:41:04.027077+0000","last_active":"2024-04-22T18:41:04.927492+0000","last_peered":"2024-04-22T18:41:04.927492+0000","last_clean":"2024-04-22T18:40:56.376630+0000","last_became_active":"2024-04-22T18:41:04.026361+0000","last_became_peered":"2024-04-22T18:41:04.026361+0000","last_unstale":"2024-04-22T18:41:04.927492+0000","last_undegraded":"2024-04-22T18:41:04.927492+0000","last_fullsized":"2024-04-22T18:41:04.927492+0000","mapping_epoch":129,"log_start":"79'52","ondisk_log_start":"79'52","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'57","last_scrub_stamp":"2024-04-22T18:40:56.376497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:56.376497+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":61,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:18.705090+0000","scrub_duration":0.30191299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3832029,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"131'135","reported_seq":523,"reported_epoch":131,"state":"active+remapped+backfill_wait","last_fresh":"2024-04-22T18:41:05.866842+0000","last_change":"2024-04-22T18:41:05.331922+0000","last_active":"2024-04-22T18:41:05.866842+0000","last_peered":"2024-04-22T18:41:05.866842+0000","last_clean":"2024-04-22T18:41:01.651924+0000","last_became_active":"2024-04-22T18:41:05.331724+0000","last_became_peered":"2024-04-22T18:41:05.331724+0000","last_unstale":"2024-04-22T18:41:05.866842+0000","last_undegraded":"2024-04-22T18:41:05.866842+0000","last_fullsized":"2024-04-22T18:41:05.866842+0000","mapping_epoch":130,"log_start":"84'88","ondisk_log_start":"84'88","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":47,"log_dups_size":106,"ondisk_log_size":47,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8966617,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":92237,"num_write":43,"num_write_kb":13219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4990114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"122'110","reported_seq":380,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:04.929374+0000","last_change":"2024-04-22T18:40:58.703325+0000","last_active":"2024-04-22T18:41:04.929374+0000","last_peered":"2024-04-22T18:41:04.929374+0000","last_clean":"2024-04-22T18:41:04.929374+0000","last_became_active":"2024-04-22T18:40:58.054356+0000","last_became_peered":"2024-04-22T18:40:58.054356+0000","last_unstale":"2024-04-22T18:41:04.929374+0000","last_undegraded":"2024-04-22T18:41:04.929374+0000","last_fullsized":"2024-04-22T18:41:04.929374+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:26.806356+0000","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13680997,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":295,"num_read_kb":347839,"num_write":111,"num_write_kb":29847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9109192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"115'138","reported_seq":517,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-04-22T18:41:05.443061+0000","last_change":"2024-04-22T18:41:05.342209+0000","last_active":"2024-04-22T18:41:05.443061+0000","last_peered":"2024-04-22T18:41:05.443061+0000","last_clean":"2024-04-22T18:41:05.443061+0000","last_became_active":"2024-04-22T18:41:04.137033+0000","last_became_peered":"2024-04-22T18:41:04.137033+0000","last_unstale":"2024-04-22T18:41:05.443061+0000","last_undegraded":"2024-04-22T18:41:05.443061+0000","last_fullsized":"2024-04-22T18:41:05.443061+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'109","last_scrub_stamp":"2024-04-22T18:40:17.658363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:17.658363+0000","objects_scrubbed":5,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:18.784031+0000","scrub_duration":0.202176033,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15496186,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":509,"num_read_kb":574500,"num_write":146,"num_write_kb":42951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":19423447,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"121'119","reported_seq":441,"reported_epoch":132,"state":"active+recovery_wait+remapped","last_fresh":"2024-04-22T18:41:06.033178+0000","last_change":"2024-04-22T18:41:05.134017+0000","last_active":"2024-04-22T18:41:06.033178+0000","last_peered":"2024-04-22T18:41:06.033178+0000","last_clean":"2024-04-22T18:40:55.895030+0000","last_became_active":"2024-04-22T18:41:05.129696+0000","last_became_peered":"2024-04-22T18:41:05.129696+0000","last_unstale":"2024-04-22T18:41:06.033178+0000","last_undegraded":"2024-04-22T18:41:06.033178+0000","last_fullsized":"2024-04-22T18:41:06.033178+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'119","last_scrub_stamp":"2024-04-22T18:40:55.573199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.573199+0000","objects_scrubbed":4,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:25.570996+0000","scrub_duration":0.32976813900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12487700,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":435,"num_read_kb":515582,"num_write":123,"num_write_kb":35069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7572471,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"120'65","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-22T18:41:05.956704+0000","last_change":"2024-04-22T18:41:04.456562+0000","last_active":"2024-04-22T18:41:05.956704+0000","last_peered":"2024-04-22T18:41:05.956704+0000","last_clean":"2024-04-22T18:41:05.956704+0000","last_became_active":"2024-04-22T18:41:03.929675+0000","last_became_peered":"2024-04-22T18:41:03.929675+0000","last_unstale":"2024-04-22T18:41:05.956704+0000","last_undegraded":"2024-04-22T18:41:05.956704+0000","last_fullsized":"2024-04-22T18:41:05.956704+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'46","last_scrub_stamp":"2024-04-22T18:40:34.755106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:34.755106+0000","objects_scrubbed":2,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:44.124447+0000","scrub_duration":0.40192567099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7311111,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":193,"num_read_kb":201357,"num_write":69,"num_write_kb":20587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":16385253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"128'44","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-22T18:41:05.956722+0000","last_change":"2024-04-22T18:41:04.469353+0000","last_active":"2024-04-22T18:41:05.956722+0000","last_peered":"2024-04-22T18:41:05.956722+0000","last_clean":"2024-04-22T18:41:05.956722+0000","last_became_active":"2024-04-22T18:41:04.036865+0000","last_became_peered":"2024-04-22T18:41:04.036865+0000","last_unstale":"2024-04-22T18:41:05.956722+0000","last_undegraded":"2024-04-22T18:41:05.956722+0000","last_fullsized":"2024-04-22T18:41:05.956722+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'23","last_scrub_stamp":"2024-04-22T18:40:20.587606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.587606+0000","objects_scrubbed":1,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.986752+0000","scrub_duration":0.10225366599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7748093,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":143,"num_read_kb":154202,"num_write":45,"num_write_kb":15026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":11149426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"131'75","reported_seq":302,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-04-22T18:41:06.033279+0000","last_change":"2024-04-22T18:38:48.173640+0000","last_active":"2024-04-22T18:41:06.033279+0000","last_peered":"2024-04-22T18:41:06.033279+0000","last_clean":"2024-04-22T18:41:06.033279+0000","last_became_active":"2024-04-22T18:38:48.173369+0000","last_became_peered":"2024-04-22T18:38:48.173369+0000","last_unstale":"2024-04-22T18:41:06.033279+0000","last_undegraded":"2024-04-22T18:41:06.033279+0000","last_fullsized":"2024-04-22T18:41:06.033279+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5109394,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":254,"num_read_kb":248270,"num_write":76,"num_write_kb":19712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":152491166,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":21,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":3194,"num_read_kb":3728724,"num_write":1001,"num_write_kb":287674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":176119390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":32,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":233287680,"data_stored":233000998,"data_compressed":889609,"data_compressed_allocated":946176,"data_compressed_original":1757184,"omap_allocated":0,"internal_metadata":0},"log_size":1433,"ondisk_log_size":1433,"up":52,"acting":52,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":132416626,"num_objects":164,"num_object_clones":0,"num_object_copies":328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":107,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":930,"num_read_kb":22175,"num_write":1307,"num_write_kb":317888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":600,"num_bytes_recovered":115996361,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":233680896,"data_stored":246913321,"data_compressed":13454868,"data_compressed_allocated":14327808,"data_compressed_original":28217344,"omap_allocated":0,"internal_metadata":0},"log_size":1056,"ondisk_log_size":1056,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2115712,"data_compressed":121124,"data_compressed_allocated":163840,"data_compressed_original":2072576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96704,"kb_used_data":65584,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94275136,"statfs":{"total":96636764160,"available":96537739264,"internally_reserved":0,"allocated":67158016,"data_stored":68023256,"data_compressed":1568203,"data_compressed_allocated":1679360,"data_compressed_original":3604480,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":9.8949999999999996,"5min":2.3170000000000002,"15min":1.054},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":139.84299999999999,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":13.949,"5min":6.3959999999999999,"15min":5.1369999999999996},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":186.97900000000001,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.48899999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":4.1959999999999997,"15min":4.6420000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":27.027000000000001,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.29299999999999998},{"interface":"front","average":{"1min":14.961,"5min":13.43,"15min":13.175000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":183.16300000000001,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.31}]},{"osd":2,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":3.3479999999999999,"15min":3.839},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64100000000000001,"5min":77.320999999999998,"15min":77.320999999999998},"last":0.37},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47199999999999998,"15min":0.47299999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.46200000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":3.4079999999999999,"15min":3.903},"min":{"1min":0.32000000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60799999999999998,"5min":77.361000000000004,"15min":77.361000000000004},"last":0.504},{"interface":"front","average":{"1min":0.497,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.30299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.65000000000000002,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.54500000000000004}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":5.6710000000000003,"5min":4.7009999999999996,"15min":4.5389999999999997},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":69.097999999999999,"5min":85.509,"15min":85.509},"last":0.27000000000000002},{"interface":"front","average":{"1min":7.9509999999999996,"5min":19.887,"15min":21.876999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":181.97,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.32500000000000001}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.5379999999999998,"15min":2.8929999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67800000000000005,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":3.3170000000000002,"5min":6.3929999999999998,"15min":6.9050000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":69.063000000000002,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.42199999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":17.294,"5min":14.079000000000001,"15min":13.542999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":139.428,"5min":139.428,"15min":139.428},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.3580000000000001,"5min":3.5779999999999998,"15min":3.948},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":22.878,"5min":77.405000000000001,"15min":77.405000000000001},"last":0.441}]}]},{"osd":6,"up_from":0,"seq":55834574880,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574880,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574880,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80196,"kb_used_data":47696,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94291644,"statfs":{"total":96636764160,"available":96554643456,"internally_reserved":0,"allocated":48840704,"data_stored":49153366,"data_compressed":1224834,"data_compressed_allocated":1294336,"data_compressed_original":2539520,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":13.022,"5min":15.922000000000001,"15min":16.405999999999999},"min":{"1min":0.27600000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":252.96199999999999,"5min":252.96199999999999,"15min":252.96199999999999},"last":0.502},{"interface":"front","average":{"1min":20.289000000000001,"5min":10.912000000000001,"15min":9.3490000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":187.148,"5min":187.148,"15min":187.148},"last":0.433}]},{"osd":1,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":9.9629999999999992,"5min":2.327,"15min":1.0549999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":96.486999999999995,"5min":96.486999999999995,"15min":96.486999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":19.181999999999999,"5min":8.1449999999999996,"15min":6.3049999999999997},"min":{"1min":0.30199999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":264.68299999999999,"5min":264.68299999999999,"15min":264.68299999999999},"last":0.40699999999999997}]},{"osd":2,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":18.23,"5min":10.502000000000001,"15min":9.2140000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":186.94300000000001,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.29299999999999998},{"interface":"front","average":{"1min":9.7699999999999996,"5min":2.3100000000000001,"15min":1.0669999999999999},"min":{"1min":0.33100000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":96.471999999999994,"5min":96.471999999999994,"15min":96.471999999999994},"last":0.42399999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":21.619,"5min":7.5410000000000004,"15min":5.1950000000000003},"min":{"1min":0.32200000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":252.91900000000001,"5min":252.91900000000001,"15min":252.91900000000001},"last":13.343999999999999},{"interface":"front","average":{"1min":22.878,"5min":21.350000000000001,"15min":21.094999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":187.11099999999999,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.371}]},{"osd":5,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":9.75,"5min":2.2559999999999998,"15min":1.0069999999999999},"min":{"1min":0.23100000000000001,"5min":0.218,"15min":0.218},"max":{"1min":96.408000000000001,"5min":96.408000000000001,"15min":96.408000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":3.3159999999999998,"15min":3.7970000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.76500000000000001,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.46899999999999997}]},{"osd":6,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":28.026,"5min":16.132999999999999,"15min":14.151},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":197.72999999999999,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":9.7560000000000002,"5min":2.3039999999999998,"15min":1.0620000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":96.438000000000002,"5min":96.438000000000002,"15min":96.438000000000002},"last":0.38100000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":21.757000000000001,"5min":11.199,"15min":9.4390000000000001},"min":{"1min":0.30099999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":186.79300000000001,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":9.7880000000000003,"5min":2.343,"15min":1.1020000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":96.456999999999994,"5min":96.456999999999994,"15min":96.456999999999994},"last":0.32100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115876,"kb_used_data":84752,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94255964,"statfs":{"total":96636764160,"available":96518107136,"internally_reserved":0,"allocated":86786048,"data_stored":88898496,"data_compressed":3425740,"data_compressed_allocated":3645440,"data_compressed_original":6860800,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":9.9700000000000006,"15min":11.456},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":11.797000000000001,"5min":98.957999999999998,"15min":98.957999999999998},"last":44.848999999999997},{"interface":"front","average":{"1min":0.26700000000000002,"5min":0.316,"15min":0.32400000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.43099999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.26300000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":25.978000000000002,"5min":9.2309999999999999,"15min":6.4400000000000004},"min":{"1min":0.20999999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":197.02099999999999,"5min":197.02099999999999,"15min":197.02099999999999},"last":0.23200000000000001},{"interface":"front","average":{"1min":1.623,"5min":0.59199999999999997,"15min":0.41999999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":16.388999999999999,"5min":16.388999999999999,"15min":16.388999999999999},"last":44.810000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":6.0090000000000003,"5min":2.0059999999999998,"15min":1.339},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":87.635999999999996,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.51800000000000002,"15min":0.45600000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":11.859999999999999,"5min":11.859999999999999,"15min":11.859999999999999},"last":44.749000000000002}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":3.379,"15min":3.8650000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79100000000000004,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":6.1719999999999997,"5min":4.5439999999999996,"15min":4.2729999999999997},"min":{"1min":0.34799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":87.724000000000004,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.443}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":6.1539999999999999,"5min":2.1760000000000002,"15min":1.5129999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":87.784000000000006,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.378},{"interface":"front","average":{"1min":1.038,"5min":1.2130000000000001,"15min":1.242},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":11.911,"5min":13.965,"15min":13.965},"last":44.938000000000002}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.52700000000000002,"15min":0.53700000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72999999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.36199999999999999},{"interface":"front","average":{"1min":9.9130000000000003,"5min":2.4079999999999999,"15min":1.157},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":96.140000000000001,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.41799999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":0.69599999999999995,"15min":0.621},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":11.887,"5min":11.887,"15min":11.887},"last":44.908000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.55000000000000004,"15min":0.56200000000000006},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75600000000000001,"5min":0.995,"15min":0.995},"last":0.42999999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64164,"kb_used_data":37140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94307676,"statfs":{"total":96636764160,"available":96571060224,"internally_reserved":0,"allocated":38031360,"data_stored":38317267,"data_compressed":1083429,"data_compressed_allocated":1159168,"data_compressed_original":2371584,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":7.4240000000000004,"15min":7.8920000000000003},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":59.002000000000002,"5min":149.03,"15min":149.03},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.308,"5min":1.649,"15min":1.873},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.442,"5min":31.866,"15min":31.866},"last":0.33700000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":3.5579999999999998,"15min":1.8740000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":97.683999999999997,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.26300000000000001},{"interface":"front","average":{"1min":22.157,"5min":16.338999999999999,"15min":15.369},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":393.42399999999998,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.30599999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":18.393000000000001,"5min":3.9689999999999999,"15min":1.5649999999999999},"min":{"1min":0.252,"5min":0.245,"15min":0.245},"max":{"1min":147.721,"5min":147.721,"15min":147.721},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.317,"5min":0.36899999999999999,"15min":0.377},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.53200000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.23699999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":3.4060000000000001,"5min":15.698,"15min":17.747},"min":{"1min":0.311,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":52.683,"5min":160.584,"15min":160.584},"last":0.48299999999999998},{"interface":"front","average":{"1min":10.984,"5min":4.1550000000000002,"15min":3.0169999999999999},"min":{"1min":0.32200000000000001,"5min":0.315,"15min":0.315},"max":{"1min":97.887,"5min":97.887,"15min":97.887},"last":0.53200000000000003}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":2.5409999999999999,"15min":2.8809999999999998},"min":{"1min":0.33300000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.74299999999999999,"5min":48.631,"15min":48.631},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":2.8820000000000001,"15min":3.278},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.83499999999999996,"5min":56.18,"15min":56.18},"last":0.498}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":10.375,"5min":2.5030000000000001,"15min":1.1910000000000001},"min":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":97.832999999999998,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.33200000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.61199999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.63800000000000001,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":10.396000000000001,"5min":2.5470000000000002,"15min":1.2390000000000001},"min":{"1min":0.38800000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":97.861999999999995,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.57199999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574880,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155696,"kb_used_data":119104,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94216144,"statfs":{"total":96636764160,"available":96477331456,"internally_reserved":0,"allocated":121962496,"data_stored":124412913,"data_compressed":3006314,"data_compressed_allocated":3219456,"data_compressed_original":6967296,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":3.9870000000000001,"5min":7.0839999999999996,"15min":7.5999999999999996},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":66.480999999999995,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.27100000000000002},{"interface":"front","average":{"1min":4.0430000000000001,"5min":7.0990000000000002,"15min":7.6079999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":66.498000000000005,"5min":88.156000000000006,"15min":88.156000000000006},"last":0.307}]},{"osd":2,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":6.9429999999999996,"15min":8.0470000000000006},"min":{"1min":0.20000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.626,"5min":183.04400000000001,"15min":183.04400000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.32400000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.33700000000000002,"15min":0.33400000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.376},{"interface":"front","average":{"1min":0.35399999999999998,"5min":11.664,"15min":13.548999999999999},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.878,"5min":231.864,"15min":231.864},"last":0.35699999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":7.29,"5min":11.845000000000001,"15min":12.603999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":109.399,"5min":192.42099999999999,"15min":192.42099999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.44900000000000001}]},{"osd":5,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":11.359999999999999,"15min":13.167},"min":{"1min":0.34399999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.052,"5min":138.089,"15min":138.089},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":16.158999999999999,"15min":18.771000000000001},"min":{"1min":0.35099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69899999999999995,"5min":183.08199999999999,"15min":183.08199999999999},"last":0.435}]},{"osd":6,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":4.8520000000000003,"5min":3.1920000000000002,"15min":2.915},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":78.805000000000007,"5min":78.805000000000007,"15min":78.805000000000007},"last":0.38900000000000001},{"interface":"front","average":{"1min":5.9400000000000004,"5min":14.413,"15min":15.824999999999999},"min":{"1min":0.36599999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":98.968000000000004,"5min":183.10900000000001,"15min":183.10900000000001},"last":0.46400000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.38500000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":2.5350000000000001,"15min":2.871},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.0189999999999999,"5min":51.097000000000001,"15min":51.097000000000001},"last":0.41299999999999998}]}]},{"osd":0,"up_from":0,"seq":55834574880,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33095680,"data_stored":34790173,"data_compressed":1758468,"data_compressed_allocated":1871872,"data_compressed_original":3694592,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49758208,"data_stored":52773688,"data_compressed":2788764,"data_compressed_allocated":2981888,"data_compressed_original":6115328,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16535552,"data_stored":17706169,"data_compressed":1083429,"data_compressed_allocated":1159168,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":56782848,"data_stored":59689832,"data_compressed":3205842,"data_compressed_allocated":3411968,"data_compressed_original":6467584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":14774272,"data_stored":15892992,"data_compressed":1080104,"data_compressed_allocated":1142784,"data_compressed_original":2281472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":223399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":42012672,"data_stored":44215728,"data_compressed":2229009,"data_compressed_allocated":2367488,"data_compressed_original":4714496,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70557696,"data_stored":70354704,"data_compressed":186556,"data_compressed_allocated":196608,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20058112,"data_stored":19952057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28516352,"data_stored":28539882,"data_compressed":219898,"data_compressed_allocated":233472,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33800192,"data_stored":33910615,"data_compressed":200115,"data_compressed_allocated":212992,"data_compressed_original":454656,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33280000,"data_stored":33118894,"data_compressed":53657,"data_compressed_allocated":57344,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1929216,"data_stored":1913105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45146112,"data_stored":45211741,"data_compressed":229383,"data_compressed_allocated":245760,"data_compressed_original":520192,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:41:08.234 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-04-22T18:41:08.234 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 36 2024-04-22T18:41:08.318 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:08.319+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:08.359 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:08.359 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:08.359 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:08.359 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:08.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:08.419 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:08.420+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:08.469 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:08.469 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:08.469 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:08.469 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:08.469 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:08.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:08.520 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:08.521+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:08.580 INFO:tasks.rados.rados.0.smithi089.stdout:1767: set_chunk oid 40 offset: 315904 length: 2635776 target oid cda848272f4ef42ebafc2089aaddeacaf12eabd82aef5e42e32ae93a8a5da0b6 offset: 0 snap -1 2024-04-22T18:41:08.580 INFO:tasks.rados.rados.0.smithi089.stdout:1768: set_chunk oid 7 target oid 2024-04-22T18:41:08.582 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:08.583 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:08.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:08.583 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:08.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.583 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:08.583 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:08.583 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.32273955799999, 2024-04-22T18:41:08.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:41:08.583 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.583 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.583 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.583 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.583 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.584 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:41:08.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.584 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:08.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:08.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:41:08.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:08.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:41:08.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:41:08.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:41:08.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:41:08.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:41:08.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:41:08.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:41:08.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:41:08.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:41:08.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:41:08.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:08.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:41:08.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:41:08.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:41:08.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:41:08.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:41:08.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:41:08.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:41:08.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:41:08.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:41:08.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:41:08.588 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.588 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.589 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.589 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:08.589 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:08.589 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.31795712799999, 2024-04-22T18:41:08.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:41:08.589 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.589 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.589 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.589 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.589 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.589 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:41:08.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.590 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:08.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:08.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:41:08.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:08.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:41:08.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:41:08.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:41:08.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:41:08.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:41:08.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:41:08.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:41:08.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:41:08.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:41:08.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:41:08.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:08.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:41:08.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:41:08.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:41:08.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:41:08.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:41:08.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:41:08.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:41:08.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:41:08.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:41:08.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:41:08.594 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.594 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.594 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.31786292499999, 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:08.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:08.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:41:08.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:08.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:41:08.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:41:08.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:41:08.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:41:08.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:41:08.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:41:08.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:41:08.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:41:08.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:41:08.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:41:08.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:08.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:41:08.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:41:08.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:41:08.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:41:08.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:41:08.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:41:08.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:41:08.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:41:08.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:41:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:41:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.31784203699999, 2024-04-22T18:41:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:41:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:08.600 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.601 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.601 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.601 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:41:08.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.601 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:08.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:08.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:41:08.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:08.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:41:08.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:41:08.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:41:08.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:41:08.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:41:08.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:41:08.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:41:08.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:41:08.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:08.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:41:08.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:41:08.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:41:08.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:41:08.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:41:08.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:41:08.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:41:08.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:41:08.605 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.605 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.605 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.605 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:08.605 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:08.605 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 133.53483437400001, 2024-04-22T18:41:08.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:41:08.605 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.605 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.605 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.605 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.605 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.606 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:41:08.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.606 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:08.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:08.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:41:08.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:08.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:41:08.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:41:08.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:41:08.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:41:08.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:41:08.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:41:08.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:41:08.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:41:08.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:41:08.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:41:08.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:08.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:41:08.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:41:08.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:41:08.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:41:08.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:41:08.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:41:08.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:41:08.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:41:08.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:41:08.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:41:08.610 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.610 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.610 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.611 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:08.611 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:08.611 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 133.52700296800001, 2024-04-22T18:41:08.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:41:08.611 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.611 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.611 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.611 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.611 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.611 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:41:08.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.612 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:08.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:08.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:41:08.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:08.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:41:08.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:41:08.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:41:08.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:41:08.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:41:08.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:41:08.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:41:08.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:41:08.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:41:08.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:41:08.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:08.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:41:08.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:41:08.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:41:08.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:41:08.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:41:08.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:41:08.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:41:08.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:41:08.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:41:08.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:41:08.616 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.616 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.616 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.375130338000005, 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:41:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:41:08.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:41:08.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:41:08.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:41:08.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:41:08.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:41:08.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:41:08.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:41:08.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:41:08.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:41:08.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:08.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:41:08.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:41:08.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:41:08.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:41:08.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:41:08.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:41:08.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:41:08.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:41:08.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:41:08.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:41:08.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.624 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.624 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:08.624 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:08.624 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 73.374725429999998, 2024-04-22T18:41:08.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:41:08.625 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.625 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:08.625 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.625 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.625 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.625 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:41:08.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.625 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:08.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:08.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:08.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:08.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:41:08.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:41:08.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:41:08.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:41:08.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:41:08.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:41:08.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:41:08.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:41:08.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:08.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:41:08.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:41:08.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:41:08.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:41:08.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:41:08.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:41:08.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:41:08.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:41:08.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.629 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.629 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3213 4.a 4:5d2aee44:::smithi08937929-13:head [assert-version v66,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:08.630 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:08.630 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 46.013825541999999, 2024-04-22T18:41:08.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69928701400000004, 2024-04-22T18:41:08.630 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.630 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.630 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3213 2024-04-22T18:41:08.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.630 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:08.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:08.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568559+0000", 2024-04-22T18:41:08.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:08.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568567+0000", 2024-04-22T18:41:08.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-04-22T18:41:08.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568579+0000", 2024-04-22T18:41:08.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-04-22T18:41:08.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.668711+0000", 2024-04-22T18:41:08.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100131866 2024-04-22T18:41:08.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.768945+0000", 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100234252 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769002+0000", 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7092999999999997e-05 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.166404+0000", 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39740241999999998 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.191686+0000", 2024-04-22T18:41:08.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025281739000000001 2024-04-22T18:41:08.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267758+0000", 2024-04-22T18:41:08.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076071606999999999 2024-04-22T18:41:08.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267812+0000", 2024-04-22T18:41:08.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4478999999999998e-05 2024-04-22T18:41:08.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267848+0000", 2024-04-22T18:41:08.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6062999999999997e-05 2024-04-22T18:41:08.635 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.635 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.635 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.635 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3319 4.a 4:5d2aee44:::smithi08937929-13:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:41:08.635 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:08.635 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 42.122077791000002, 2024-04-22T18:41:08.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476623401, 2024-04-22T18:41:08.635 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.635 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:08.635 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.636 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.636 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.636 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3319 2024-04-22T18:41:08.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.636 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:08.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:08.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460306+0000", 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460318+0000", 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2896e-05 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460345+0000", 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6650000000000001e-05 2024-04-22T18:41:08.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460360+0000", 2024-04-22T18:41:08.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4729000000000001e-05 2024-04-22T18:41:08.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460502+0000", 2024-04-22T18:41:08.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141729 2024-04-22T18:41:08.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460644+0000", 2024-04-22T18:41:08.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142755 2024-04-22T18:41:08.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.936932+0000", 2024-04-22T18:41:08.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476288078 2024-04-22T18:41:08.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.639 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.639 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3412 4.a 4:519c08f7:::smithi08937929-15:head [write 2529878~394573] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:08.639 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:08.639 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.44024323, 2024-04-22T18:41:08.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48918094899999998, 2024-04-22T18:41:08.640 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.640 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.640 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.640 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.640 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.640 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3412 2024-04-22T18:41:08.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.640 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:08.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:08.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142141+0000", 2024-04-22T18:41:08.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:08.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142496+0000", 2024-04-22T18:41:08.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035444699999999998 2024-04-22T18:41:08.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142510+0000", 2024-04-22T18:41:08.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3841e-05 2024-04-22T18:41:08.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142518+0000", 2024-04-22T18:41:08.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8089999999999993e-06 2024-04-22T18:41:08.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142560+0000", 2024-04-22T18:41:08.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2097e-05 2024-04-22T18:41:08.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142637+0000", 2024-04-22T18:41:08.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6965000000000006e-05 2024-04-22T18:41:08.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:08.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142715+0000", 2024-04-22T18:41:08.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8694000000000002e-05 2024-04-22T18:41:08.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144179+0000", 2024-04-22T18:41:08.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146342 2024-04-22T18:41:08.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631229+0000", 2024-04-22T18:41:08.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48705018500000002 2024-04-22T18:41:08.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631281+0000", 2024-04-22T18:41:08.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2216999999999998e-05 2024-04-22T18:41:08.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631324+0000", 2024-04-22T18:41:08.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3344999999999999e-05 2024-04-22T18:41:08.645 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.645 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.645 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.645 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3413 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2924451] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:08.645 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:08.645 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.439862089999998, 2024-04-22T18:41:08.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58904086899999997, 2024-04-22T18:41:08.645 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.645 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.645 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.645 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.645 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.646 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3413 2024-04-22T18:41:08.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.646 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:08.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:08.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142523+0000", 2024-04-22T18:41:08.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:08.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142529+0000", 2024-04-22T18:41:08.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6049999999999997e-06 2024-04-22T18:41:08.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142535+0000", 2024-04-22T18:41:08.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4740000000000002e-06 2024-04-22T18:41:08.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142541+0000", 2024-04-22T18:41:08.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1839999999999998e-06 2024-04-22T18:41:08.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143553+0000", 2024-04-22T18:41:08.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010126689999999999 2024-04-22T18:41:08.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143659+0000", 2024-04-22T18:41:08.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105393 2024-04-22T18:41:08.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:08.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143825+0000", 2024-04-22T18:41:08.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165809 2024-04-22T18:41:08.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144329+0000", 2024-04-22T18:41:08.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050483799999999997 2024-04-22T18:41:08.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731480+0000", 2024-04-22T18:41:08.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58715100600000003 2024-04-22T18:41:08.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731522+0000", 2024-04-22T18:41:08.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2098000000000002e-05 2024-04-22T18:41:08.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731566+0000", 2024-04-22T18:41:08.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3066999999999999e-05 2024-04-22T18:41:08.650 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.650 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.650 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.651 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3414 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:08.651 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:08.651 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.439833485999998, 2024-04-22T18:41:08.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58926020199999996, 2024-04-22T18:41:08.651 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.651 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:08.651 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.651 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.651 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.651 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3414 2024-04-22T18:41:08.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.651 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:08.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:08.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142552+0000", 2024-04-22T18:41:08.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:08.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142556+0000", 2024-04-22T18:41:08.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0289999999999999e-06 2024-04-22T18:41:08.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142574+0000", 2024-04-22T18:41:08.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7266999999999999e-05 2024-04-22T18:41:08.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142581+0000", 2024-04-22T18:41:08.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-04-22T18:41:08.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144121+0000", 2024-04-22T18:41:08.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015397480000000001 2024-04-22T18:41:08.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:08.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144170+0000", 2024-04-22T18:41:08.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8803e-05 2024-04-22T18:41:08.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731624+0000", 2024-04-22T18:41:08.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58745365800000005 2024-04-22T18:41:08.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731712+0000", 2024-04-22T18:41:08.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8016999999999994e-05 2024-04-22T18:41:08.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731813+0000", 2024-04-22T18:41:08.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101817 2024-04-22T18:41:08.655 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.655 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.655 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.656 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:77 3.3 3:dcc8aeec:::3590d5741efdda41a26b62a0982727439225b7b4b51097b313f30d8dc10bf6f6:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:08.656 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:08.656 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.334815163999998, 2024-04-22T18:41:08.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011812878, 2024-04-22T18:41:08.656 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.656 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.656 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:08.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:08.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247571+0000", 2024-04-22T18:41:08.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:08.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247574+0000", 2024-04-22T18:41:08.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:41:08.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247577+0000", 2024-04-22T18:41:08.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3069999999999998e-06 2024-04-22T18:41:08.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247580+0000", 2024-04-22T18:41:08.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8870000000000001e-06 2024-04-22T18:41:08.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.340928+0000", 2024-04-22T18:41:08.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093348678000000004 2024-04-22T18:41:08.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:08.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341122+0000", 2024-04-22T18:41:08.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000193227 2024-04-22T18:41:08.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618335+0000", 2024-04-22T18:41:08.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.277213775 2024-04-22T18:41:08.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:08.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618422+0000", 2024-04-22T18:41:08.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6537000000000002e-05 2024-04-22T18:41:08.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954387+0000", 2024-04-22T18:41:08.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33596509699999999 2024-04-22T18:41:08.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954548+0000", 2024-04-22T18:41:08.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016042199999999999 2024-04-22T18:41:08.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:08.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954858+0000", 2024-04-22T18:41:08.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000310396 2024-04-22T18:41:08.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157019+0000", 2024-04-22T18:41:08.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20216071299999999 2024-04-22T18:41:08.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259213+0000", 2024-04-22T18:41:08.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102194734 2024-04-22T18:41:08.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259262+0000", 2024-04-22T18:41:08.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9052999999999999e-05 2024-04-22T18:41:08.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259384+0000", 2024-04-22T18:41:08.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122052 2024-04-22T18:41:08.662 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.662 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.662 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.662 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:78 3.3 3:df7e90d2:::ab0657e24bd96704775de2a378746bc22309c5a7366a5eaf9b4e1f1f2bef0a4c:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.334799967000002, 2024-04-22T18:41:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011974739, 2024-04-22T18:41:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:08.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247586+0000", 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247589+0000", 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.181e-06 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247592+0000", 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0230000000000001e-06 2024-04-22T18:41:08.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247595+0000", 2024-04-22T18:41:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.428e-06 2024-04-22T18:41:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341160+0000", 2024-04-22T18:41:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093564571999999999 2024-04-22T18:41:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341264+0000", 2024-04-22T18:41:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010391999999999999 2024-04-22T18:41:08.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619269+0000", 2024-04-22T18:41:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27800575100000002 2024-04-22T18:41:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619346+0000", 2024-04-22T18:41:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6459999999999996e-05 2024-04-22T18:41:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.155957+0000", 2024-04-22T18:41:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53661109299999998 2024-04-22T18:41:08.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156049+0000", 2024-04-22T18:41:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2272e-05 2024-04-22T18:41:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156236+0000", 2024-04-22T18:41:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000186808 2024-04-22T18:41:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.158088+0000", 2024-04-22T18:41:08.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018522079999999999 2024-04-22T18:41:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259421+0000", 2024-04-22T18:41:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101332718 2024-04-22T18:41:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259460+0000", 2024-04-22T18:41:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8912000000000003e-05 2024-04-22T18:41:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259562+0000", 2024-04-22T18:41:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010175600000000001 2024-04-22T18:41:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:80 3.3 3:f90f47f2:::e5603f596d699ee3e27e255225eeb90a9f0c1b4a4d099d1c586b19019220aa3f:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.334760889999998, 2024-04-22T18:41:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0121096759999999, 2024-04-22T18:41:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247625+0000", 2024-04-22T18:41:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247628+0000", 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6129999999999998e-06 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247631+0000", 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3869999999999999e-06 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247634+0000", 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9919999999999999e-06 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341550+0000", 2024-04-22T18:41:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093915584999999996 2024-04-22T18:41:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341700+0000", 2024-04-22T18:41:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015000300000000001 2024-04-22T18:41:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620744+0000", 2024-04-22T18:41:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27904399200000002 2024-04-22T18:41:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620794+0000", 2024-04-22T18:41:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9579000000000001e-05 2024-04-22T18:41:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157597+0000", 2024-04-22T18:41:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53680343799999997 2024-04-22T18:41:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157678+0000", 2024-04-22T18:41:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0829999999999997e-05 2024-04-22T18:41:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157832+0000", 2024-04-22T18:41:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154184 2024-04-22T18:41:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259572+0000", 2024-04-22T18:41:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10174003600000001 2024-04-22T18:41:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259601+0000", 2024-04-22T18:41:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8841999999999999e-05 2024-04-22T18:41:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259638+0000", 2024-04-22T18:41:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7308999999999998e-05 2024-04-22T18:41:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259736+0000", 2024-04-22T18:41:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7283999999999999e-05 2024-04-22T18:41:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:135 3.3 3:f89134e5:::3e4e73f5d4a0122957865f0faeac2483204ee98d7d497ef3b54afe2a02b39a3d:head [read 0~802304 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:08.675 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 32.966945811000002, 2024-04-22T18:41:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.64370653499999997, 2024-04-22T18:41:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615438+0000", 2024-04-22T18:41:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615449+0000", 2024-04-22T18:41:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1219999999999999e-05 2024-04-22T18:41:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615460+0000", 2024-04-22T18:41:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0403e-05 2024-04-22T18:41:08.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.715588+0000", 2024-04-22T18:41:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100128881 2024-04-22T18:41:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722533+0000", 2024-04-22T18:41:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069444010000000002 2024-04-22T18:41:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:08.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722658+0000", 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012560699999999999 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826437+0000", 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103778194 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826532+0000", 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5111000000000005e-05 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258787+0000", 2024-04-22T18:41:08.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43225529099999999 2024-04-22T18:41:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258905+0000", 2024-04-22T18:41:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118348 2024-04-22T18:41:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259148+0000", 2024-04-22T18:41:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000242093 2024-04-22T18:41:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3890 4.1 4:83c955bf:::smithi08937929-28:head [write 2699879~469889] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.968270966999999, 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49042297000000001, 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3890 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:08.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614115+0000", 2024-04-22T18:41:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614496+0000", 2024-04-22T18:41:08.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038181700000000001 2024-04-22T18:41:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614509+0000", 2024-04-22T18:41:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2381000000000001e-05 2024-04-22T18:41:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614517+0000", 2024-04-22T18:41:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0879999999999995e-06 2024-04-22T18:41:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615610+0000", 2024-04-22T18:41:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010933519999999999 2024-04-22T18:41:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615655+0000", 2024-04-22T18:41:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4966000000000003e-05 2024-04-22T18:41:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615769+0000", 2024-04-22T18:41:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113986 2024-04-22T18:41:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704882+0000", 2024-04-22T18:41:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089112825000000007 2024-04-22T18:41:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104488+0000", 2024-04-22T18:41:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39960572999999999 2024-04-22T18:41:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104517+0000", 2024-04-22T18:41:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9006e-05 2024-04-22T18:41:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104539+0000", 2024-04-22T18:41:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2073e-05 2024-04-22T18:41:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3891 4.1 4:83c955bf:::smithi08937929-28:head [setxattr _header (56) in=7b,truncate 3630345] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.967862736000001, 2024-04-22T18:41:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59018588699999996, 2024-04-22T18:41:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:08.686 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3891 2024-04-22T18:41:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:08.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614523+0000", 2024-04-22T18:41:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614528+0000", 2024-04-22T18:41:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0379999999999999e-06 2024-04-22T18:41:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614540+0000", 2024-04-22T18:41:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018000000000001e-05 2024-04-22T18:41:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614546+0000", 2024-04-22T18:41:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4990000000000002e-06 2024-04-22T18:41:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704438+0000", 2024-04-22T18:41:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089892044000000004 2024-04-22T18:41:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704504+0000", 2024-04-22T18:41:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6365000000000006e-05 2024-04-22T18:41:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704602+0000", 2024-04-22T18:41:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8048999999999996e-05 2024-04-22T18:41:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.806370+0000", 2024-04-22T18:41:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10176768 2024-04-22T18:41:08.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204629+0000", 2024-04-22T18:41:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39825917999999999 2024-04-22T18:41:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204673+0000", 2024-04-22T18:41:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3516999999999997e-05 2024-04-22T18:41:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204710+0000", 2024-04-22T18:41:08.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7391000000000002e-05 2024-04-22T18:41:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3892 4.1 4:83c955bf:::smithi08937929-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.967836155000001, 2024-04-22T18:41:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59031281800000002, 2024-04-22T18:41:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:08.692 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3892 2024-04-22T18:41:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614550+0000", 2024-04-22T18:41:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614553+0000", 2024-04-22T18:41:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0879999999999999e-06 2024-04-22T18:41:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614559+0000", 2024-04-22T18:41:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9259999999999997e-06 2024-04-22T18:41:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614565+0000", 2024-04-22T18:41:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7950000000000004e-06 2024-04-22T18:41:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704818+0000", 2024-04-22T18:41:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090252624000000004 2024-04-22T18:41:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704866+0000", 2024-04-22T18:41:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8667e-05 2024-04-22T18:41:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204742+0000", 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987551899999999 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204799+0000", 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7299000000000001e-05 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204863+0000", 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4455000000000005e-05 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.696 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:08.697 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:08.697 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:08.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:08.698 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:08.699+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:08.784 INFO:tasks.rados.rados.0.smithi089.stdout:1768: set_chunk oid 7 offset: 684544 length: 2206720 target oid e4d8f1c60e11099cd23d36ecf95438dc35a3631015ca9f03c7a84c5ae2008074 offset: 0 snap -1 2024-04-22T18:41:08.785 INFO:tasks.rados.rados.0.smithi089.stdout:1769: tier_flush oid 43 2024-04-22T18:41:08.785 INFO:tasks.rados.rados.0.smithi089.stdout:1769: tier_flush oid 43 snap -1 2024-04-22T18:41:08.785 INFO:tasks.rados.rados.0.smithi089.stdout:1760: finishing write tid 4 to smithi08937929-8 2024-04-22T18:41:08.785 INFO:tasks.rados.rados.0.smithi089.stdout:1760: oid 8 updating version 0 to 49 2024-04-22T18:41:08.785 INFO:tasks.rados.rados.0.smithi089.stdout:1760: oid 8 version 49 is already newer than 48 2024-04-22T18:41:08.785 INFO:tasks.rados.rados.0.smithi089.stdout:1760: oid 8 updating version 49 to 50 2024-04-22T18:41:08.785 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 8 v 50 (ObjNum 365 snap 0 seq_num 365) dirty exists 2024-04-22T18:41:08.785 INFO:tasks.rados.rados.0.smithi089.stdout:1760: left oid 8 (ObjNum 365 snap 0 seq_num 365) 2024-04-22T18:41:08.785 INFO:tasks.rados.rados.0.smithi089.stdout:1760: done (8 left) 2024-04-22T18:41:08.785 INFO:tasks.rados.rados.0.smithi089.stdout:1770: read oid 24 snap -1 2024-04-22T18:41:08.785 INFO:tasks.rados.rados.0.smithi089.stdout:1770: expect deleted 2024-04-22T18:41:08.785 INFO:tasks.rados.rados.0.smithi089.stdout:1771: read oid 50 snap -1 2024-04-22T18:41:08.785 INFO:tasks.rados.rados.0.smithi089.stdout:1771: expect (ObjNum 189 snap 0 seq_num 189) 2024-04-22T18:41:08.786 INFO:tasks.rados.rados.0.smithi089.stdout:1772: tier_evict oid 1 2024-04-22T18:41:08.786 INFO:tasks.rados.rados.0.smithi089.stdout:1772: tier_evict oid 1 snap -1 2024-04-22T18:41:08.786 INFO:tasks.rados.rados.0.smithi089.stdout:1766: got (22) Invalid argument 2024-04-22T18:41:08.786 INFO:tasks.rados.rados.0.smithi089.stdout:1763: expect (ObjNum 339 snap 0 seq_num 339) 2024-04-22T18:41:08.801 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:08.800+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:08.842 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:08.842 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:08.842 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:08.842 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:08.850 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:08.899 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:08.901+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:08.937 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:08.938+0000 7fec57bee640 -1 osd.7 133 handle_osd_map injecting map crc failure 2024-04-22T18:41:08.939 INFO:teuthology.orchestra.run.smithi089.stderr:set pool 4 pg_num to 36 2024-04-22T18:41:08.993 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:08.993 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:08.993 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:08.993 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:08.993 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:09.001 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:09.002+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:09.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:09.005 INFO:tasks.rados.rados.0.smithi089.stdout:1764: expect (ObjNum 295 snap 0 seq_num 295) 2024-04-22T18:41:09.101 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:09.102+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:09.106 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:09.107 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:09.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:09.107 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:09.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.107 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:41:09.107 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:41:09.107 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 149.36502862200001, 2024-04-22T18:41:09.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40512221599999998, 2024-04-22T18:41:09.107 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.107 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.107 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.107 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:09.108 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:09.108 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 36 2024-04-22T18:41:09.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.108 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:41:09.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741984+0000", 2024-04-22T18:41:09.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.741983+0000", 2024-04-22T18:41:09.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:09.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742005+0000", 2024-04-22T18:41:09.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1656999999999999e-05 2024-04-22T18:41:09.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742012+0000", 2024-04-22T18:41:09.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.362e-06 2024-04-22T18:41:09.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742019+0000", 2024-04-22T18:41:09.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.81e-06 2024-04-22T18:41:09.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845014+0000", 2024-04-22T18:41:09.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102994556 2024-04-22T18:41:09.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845097+0000", 2024-04-22T18:41:09.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3176999999999993e-05 2024-04-22T18:41:09.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:09.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.845225+0000", 2024-04-22T18:41:09.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000127699 2024-04-22T18:41:09.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:09.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944769+0000", 2024-04-22T18:41:09.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099544787999999995 2024-04-22T18:41:09.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.146984+0000", 2024-04-22T18:41:09.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221493700000001 2024-04-22T18:41:09.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:09.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147048+0000", 2024-04-22T18:41:09.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3373999999999993e-05 2024-04-22T18:41:09.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147106+0000", 2024-04-22T18:41:09.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8724000000000002e-05 2024-04-22T18:41:09.112 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.112 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.112 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 149.364988931, 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.405186033, 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 37 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:41:09.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742024+0000", 2024-04-22T18:41:09.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742023+0000", 2024-04-22T18:41:09.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:09.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742027+0000", 2024-04-22T18:41:09.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.642e-06 2024-04-22T18:41:09.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.742032+0000", 2024-04-22T18:41:09.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4809999999999997e-06 2024-04-22T18:41:09.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.842182+0000", 2024-04-22T18:41:09.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100149974 2024-04-22T18:41:09.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944137+0000", 2024-04-22T18:41:09.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101955086 2024-04-22T18:41:09.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944280+0000", 2024-04-22T18:41:09.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142108 2024-04-22T18:41:09.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:09.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944450+0000", 2024-04-22T18:41:09.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00017076099999999999 2024-04-22T18:41:09.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:09.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:39.944999+0000", 2024-04-22T18:41:09.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00054922399999999996 2024-04-22T18:41:09.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147152+0000", 2024-04-22T18:41:09.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.202152999 2024-04-22T18:41:09.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:09.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147185+0000", 2024-04-22T18:41:09.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2994000000000003e-05 2024-04-22T18:41:09.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:40.147210+0000", 2024-04-22T18:41:09.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4479999999999999e-05 2024-04-22T18:41:09.118 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.118 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.118 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.118 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-22T18:41:09.118 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:41:09.118 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 147.12079209500001, 2024-04-22T18:41:09.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.402512329, 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 69 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986221+0000", 2024-04-22T18:41:09.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986220+0000", 2024-04-22T18:41:09.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986268+0000", 2024-04-22T18:41:09.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7512000000000002e-05 2024-04-22T18:41:09.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:41.986275+0000", 2024-04-22T18:41:09.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3359999999999997e-06 2024-04-22T18:41:09.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086403+0000", 2024-04-22T18:41:09.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100127745 2024-04-22T18:41:09.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086519+0000", 2024-04-22T18:41:09.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115982 2024-04-22T18:41:09.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086674+0000", 2024-04-22T18:41:09.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000155415 2024-04-22T18:41:09.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:09.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.086863+0000", 2024-04-22T18:41:09.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00018835 2024-04-22T18:41:09.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:09.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.087755+0000", 2024-04-22T18:41:09.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00089285099999999995 2024-04-22T18:41:09.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388663+0000", 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30090760799999999 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388697+0000", 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4174000000000002e-05 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:42.388733+0000", 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5955999999999999e-05 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.123 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.124 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.124 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:09.124 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:41:09.124 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.25280226699999, 2024-04-22T18:41:09.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40760861900000001, 2024-04-22T18:41:09.124 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.124 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.124 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.124 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:09.124 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:09.124 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 111 2024-04-22T18:41:09.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.125 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:41:09.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854211+0000", 2024-04-22T18:41:09.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854210+0000", 2024-04-22T18:41:09.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854213+0000", 2024-04-22T18:41:09.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:41:09.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854216+0000", 2024-04-22T18:41:09.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.208e-06 2024-04-22T18:41:09.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854219+0000", 2024-04-22T18:41:09.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7920000000000001e-06 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058127+0000", 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.203908122 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058226+0000", 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8888999999999995e-05 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.058378+0000", 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000151924 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:09.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159554+0000", 2024-04-22T18:41:09.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10117661999999999 2024-04-22T18:41:09.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261764+0000", 2024-04-22T18:41:09.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102209393 2024-04-22T18:41:09.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:09.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261796+0000", 2024-04-22T18:41:09.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2211e-05 2024-04-22T18:41:09.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261819+0000", 2024-04-22T18:41:09.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3269e-05 2024-04-22T18:41:09.129 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.129 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.129 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.129 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:09.129 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:09.129 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.252786609, 2024-04-22T18:41:09.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:41:09.129 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.129 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.130 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.130 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:09.130 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:09.130 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:41:09.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.130 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:09.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:09.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:09.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:41:09.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:41:09.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:41:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:41:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:41:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:41:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:41:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:41:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:41:09.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:41:09.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:09.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:41:09.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:41:09.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:09.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:41:09.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:41:09.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:41:09.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:41:09.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:09.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:41:09.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:41:09.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:41:09.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:41:09.135 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.135 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.135 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.135 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:09.135 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:09.135 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.25273544699999, 2024-04-22T18:41:09.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:41:09.135 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.135 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.135 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.135 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:09.135 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:09.136 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:41:09.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.136 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:09.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:09.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:09.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:41:09.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:41:09.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:41:09.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:41:09.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:41:09.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:41:09.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:41:09.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:41:09.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:41:09.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:41:09.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:09.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:41:09.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:41:09.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:09.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:41:09.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:41:09.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:41:09.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:41:09.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:09.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:41:09.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:41:09.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:41:09.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:41:09.140 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.141 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.141 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.141 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:09.141 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:09.141 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.25272022300001, 2024-04-22T18:41:09.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:41:09.141 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.141 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.141 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.141 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:09.141 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:09.142 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:41:09.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.142 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:09.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:09.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:41:09.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:41:09.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:41:09.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:41:09.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:09.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:41:09.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:41:09.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:41:09.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:41:09.146 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.146 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.146 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.146 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:09.146 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:09.146 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.25266064600001, 2024-04-22T18:41:09.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:41:09.147 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.147 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.147 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.147 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:09.147 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:09.147 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:41:09.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.147 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:09.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:41:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:41:09.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:41:09.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:41:09.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:41:09.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:41:09.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:41:09.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:41:09.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:41:09.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:41:09.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:09.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:41:09.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:41:09.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:09.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:41:09.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:41:09.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:41:09.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:41:09.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:09.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:41:09.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:41:09.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:41:09.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:41:09.152 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.152 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.152 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.152 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:09.152 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:09.152 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.25257833000001, 2024-04-22T18:41:09.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:41:09.153 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.153 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.153 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.153 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:09.153 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:09.153 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:41:09.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.153 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:09.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:09.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:41:09.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:41:09.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:41:09.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:41:09.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:41:09.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:09.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:41:09.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:41:09.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:09.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:41:09.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:41:09.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.157 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.158 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.158 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:09.158 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:09.158 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.252568174, 2024-04-22T18:41:09.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:41:09.158 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.158 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.158 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.158 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:09.158 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:09.158 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:41:09.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.159 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:09.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:09.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:41:09.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:41:09.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:41:09.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:41:09.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:41:09.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:41:09.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:41:09.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:41:09.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:41:09.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:41:09.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:41:09.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:09.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:41:09.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:41:09.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:09.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:41:09.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:41:09.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:41:09.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:41:09.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:41:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:41:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:41:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:41:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2835 4.18 4:187adac1:::smithi08937929-36:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:41:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:41:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.490643783000003, 2024-04-22T18:41:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.403020987, 2024-04-22T18:41:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2835 2024-04-22T18:41:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:41:09.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616369+0000", 2024-04-22T18:41:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616367+0000", 2024-04-22T18:41:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616377+0000", 2024-04-22T18:41:09.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0618999999999999e-05 2024-04-22T18:41:09.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616410+0000", 2024-04-22T18:41:09.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3021000000000001e-05 2024-04-22T18:41:09.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.616437+0000", 2024-04-22T18:41:09.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7158e-05 2024-04-22T18:41:09.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.716588+0000", 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100150674 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:07.716679+0000", 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1231999999999997e-05 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:08.019390+0000", 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30271076299999999 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3163 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.269555736000001, 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1009801299999999, 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3163 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:09.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:09.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837455+0000", 2024-04-22T18:41:09.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:09.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837464+0000", 2024-04-22T18:41:09.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5319999999999995e-06 2024-04-22T18:41:09.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837476+0000", 2024-04-22T18:41:09.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1759e-05 2024-04-22T18:41:09.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837488+0000", 2024-04-22T18:41:09.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1666e-05 2024-04-22T18:41:09.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837541+0000", 2024-04-22T18:41:09.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2939999999999998e-05 2024-04-22T18:41:09.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:09.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837547+0000", 2024-04-22T18:41:09.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6409999999999996e-06 2024-04-22T18:41:09.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937423+0000", 2024-04-22T18:41:09.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.099875545 2024-04-22T18:41:09.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937567+0000", 2024-04-22T18:41:09.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014363099999999999 2024-04-22T18:41:09.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938437+0000", 2024-04-22T18:41:09.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087060399999999997 2024-04-22T18:41:09.172 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.172 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.172 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.172 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3164 4.8 4:187adac1:::smithi08937929-36:head [sparse-read 0~2643888] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:09.172 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:09.173 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.269516367000001, 2024-04-22T18:41:09.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1013670310000001, 2024-04-22T18:41:09.173 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.173 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:09.173 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.173 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:09.173 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.173 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3164 2024-04-22T18:41:09.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.173 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:09.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:09.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:09.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837499+0000", 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.994e-06 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837503+0000", 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7249999999999999e-06 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837506+0000", 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5980000000000001e-06 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937641+0000", 2024-04-22T18:41:09.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10013467700000001 2024-04-22T18:41:09.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:09.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937650+0000", 2024-04-22T18:41:09.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3489999999999998e-06 2024-04-22T18:41:09.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938468+0000", 2024-04-22T18:41:09.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0008176710000001 2024-04-22T18:41:09.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938582+0000", 2024-04-22T18:41:09.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114287 2024-04-22T18:41:09.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938864+0000", 2024-04-22T18:41:09.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028120899999999998 2024-04-22T18:41:09.177 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.177 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.177 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.177 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3165 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:09.177 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:09.177 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.269497598999997, 2024-04-22T18:41:09.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2022754499999999, 2024-04-22T18:41:09.178 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.178 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:09.178 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.178 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:09.178 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.178 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3165 2024-04-22T18:41:09.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.178 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:09.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:09.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:09.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837517+0000", 2024-04-22T18:41:09.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3369999999999998e-06 2024-04-22T18:41:09.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837523+0000", 2024-04-22T18:41:09.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5450000000000003e-06 2024-04-22T18:41:09.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837526+0000", 2024-04-22T18:41:09.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.354e-06 2024-04-22T18:41:09.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937682+0000", 2024-04-22T18:41:09.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100155917 2024-04-22T18:41:09.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:09.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937688+0000", 2024-04-22T18:41:09.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:41:09.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.038953+0000", 2024-04-22T18:41:09.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012648110000001 2024-04-22T18:41:09.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039034+0000", 2024-04-22T18:41:09.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0969000000000004e-05 2024-04-22T18:41:09.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039791+0000", 2024-04-22T18:41:09.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075672900000000004 2024-04-22T18:41:09.182 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.182 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.182 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.182 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3183 4.8 4:1bc4668a:::smithi08937929-21:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:09.182 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:09.182 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.071094401000003, 2024-04-22T18:41:09.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0040478260000001, 2024-04-22T18:41:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3183 2024-04-22T18:41:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035913+0000", 2024-04-22T18:41:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999943 2024-04-22T18:41:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035935+0000", 2024-04-22T18:41:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2232999999999999e-05 2024-04-22T18:41:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035969+0000", 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3969e-05 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035987+0000", 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8397e-05 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2916000000000003e-05 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:09.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:41:09.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0103e-05 2024-04-22T18:41:09.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039841+0000", 2024-04-22T18:41:09.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0037508959999999 2024-04-22T18:41:09.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039966+0000", 2024-04-22T18:41:09.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012514500000000001 2024-04-22T18:41:09.187 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.187 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.187 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.187 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3184 4.8 4:12d9f826:::smithi08937929-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:09.187 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:09.187 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.071006441999998, 2024-04-22T18:41:09.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.00402868, 2024-04-22T18:41:09.187 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.187 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:09.187 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.187 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:09.187 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.188 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3184 2024-04-22T18:41:09.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.188 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:09.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:09.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036005+0000", 2024-04-22T18:41:09.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:09.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036026+0000", 2024-04-22T18:41:09.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1092999999999999e-05 2024-04-22T18:41:09.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036030+0000", 2024-04-22T18:41:09.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.467e-06 2024-04-22T18:41:09.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036033+0000", 2024-04-22T18:41:09.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9780000000000001e-06 2024-04-22T18:41:09.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036120+0000", 2024-04-22T18:41:09.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7495000000000002e-05 2024-04-22T18:41:09.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036125+0000", 2024-04-22T18:41:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8720000000000001e-06 2024-04-22T18:41:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039982+0000", 2024-04-22T18:41:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0038571919999999 2024-04-22T18:41:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040035+0000", 2024-04-22T18:41:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2992999999999999e-05 2024-04-22T18:41:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.191 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3185 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.07095253, 2024-04-22T18:41:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0041404039999999, 2024-04-22T18:41:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.192 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3185 2024-04-22T18:41:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036062+0000", 2024-04-22T18:41:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9099999999999999e-06 2024-04-22T18:41:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036066+0000", 2024-04-22T18:41:09.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1699999999999999e-06 2024-04-22T18:41:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036077+0000", 2024-04-22T18:41:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0637e-05 2024-04-22T18:41:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036147+0000", 2024-04-22T18:41:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0025999999999997e-05 2024-04-22T18:41:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036151+0000", 2024-04-22T18:41:09.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:41:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040068+0000", 2024-04-22T18:41:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.003916604 2024-04-22T18:41:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040201+0000", 2024-04-22T18:41:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013275400000000001 2024-04-22T18:41:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.196 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3186 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.070932974000002, 2024-04-22T18:41:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2044167510000001, 2024-04-22T18:41:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3186 2024-04-22T18:41:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036079+0000", 2024-04-22T18:41:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036081+0000", 2024-04-22T18:41:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6449999999999999e-06 2024-04-22T18:41:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036084+0000", 2024-04-22T18:41:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9399999999999998e-06 2024-04-22T18:41:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:41:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.879e-06 2024-04-22T18:41:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036168+0000", 2024-04-22T18:41:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7657999999999994e-05 2024-04-22T18:41:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036171+0000", 2024-04-22T18:41:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7639999999999999e-06 2024-04-22T18:41:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240345+0000", 2024-04-22T18:41:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2041734260000001 2024-04-22T18:41:09.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240497+0000", 2024-04-22T18:41:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015182800000000001 2024-04-22T18:41:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3723 4.18 4:1bc4668a:::smithi08937929-21:head [setxattr _header (56) in=7b,truncate 3277398] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:45.060393+0000", 2024-04-22T18:41:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.046620144999999, 2024-04-22T18:41:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40389026300000003, 2024-04-22T18:41:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.201 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3723 2024-04-22T18:41:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060393+0000", 2024-04-22T18:41:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060393+0000", 2024-04-22T18:41:09.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060392+0000", 2024-04-22T18:41:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060396+0000", 2024-04-22T18:41:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7100000000000001e-06 2024-04-22T18:41:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060399+0000", 2024-04-22T18:41:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0400000000000001e-06 2024-04-22T18:41:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060402+0000", 2024-04-22T18:41:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6350000000000002e-06 2024-04-22T18:41:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165420+0000", 2024-04-22T18:41:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10501865000000001 2024-04-22T18:41:09.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165496+0000", 2024-04-22T18:41:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5768999999999999e-05 2024-04-22T18:41:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165616+0000", 2024-04-22T18:41:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120024 2024-04-22T18:41:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165906+0000", 2024-04-22T18:41:09.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028992200000000002 2024-04-22T18:41:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.464190+0000", 2024-04-22T18:41:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29828380599999998 2024-04-22T18:41:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.464229+0000", 2024-04-22T18:41:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9495e-05 2024-04-22T18:41:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.464283+0000", 2024-04-22T18:41:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3701e-05 2024-04-22T18:41:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3724 4.18 4:1bc4668a:::smithi08937929-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.046608597999999, 2024-04-22T18:41:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50413114699999995, 2024-04-22T18:41:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:09.207 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3724 2024-04-22T18:41:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:09.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060405+0000", 2024-04-22T18:41:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.395e-06 2024-04-22T18:41:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060408+0000", 2024-04-22T18:41:09.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:41:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060411+0000", 2024-04-22T18:41:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6060000000000001e-06 2024-04-22T18:41:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165799+0000", 2024-04-22T18:41:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105388369 2024-04-22T18:41:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:09.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165843+0000", 2024-04-22T18:41:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3891999999999999e-05 2024-04-22T18:41:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564408+0000", 2024-04-22T18:41:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39856466000000002 2024-04-22T18:41:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564461+0000", 2024-04-22T18:41:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3529999999999997e-05 2024-04-22T18:41:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:09.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564535+0000", 2024-04-22T18:41:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4147999999999995e-05 2024-04-22T18:41:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:09.212 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:09.212 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:09.213 INFO:tasks.rados.rados.0.smithi089.stdout:1765: expect (ObjNum 323 snap 0 seq_num 323) 2024-04-22T18:41:09.213 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:09.214 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:09.214+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:09.314 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:09.315+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:09.316 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:09.317 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:09.317 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:09.317 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:09.326 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:09.399 INFO:tasks.rados.rados.0.smithi089.stdout:1762: got (0) Success 2024-04-22T18:41:09.399 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 125 (ObjNum 340 snap 0 seq_num 340) dirty exists 2024-04-22T18:41:09.399 INFO:tasks.rados.rados.0.smithi089.stdout:1772: got (22) Invalid argument 2024-04-22T18:41:09.399 INFO:tasks.rados.rados.0.smithi089.stdout:1771: expect (ObjNum 189 snap 0 seq_num 189) 2024-04-22T18:41:09.414 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:09.415+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:09.427 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:09.427 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:09.427 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:09.428 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:09.428 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:09.438 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:09.515 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:09.517+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:09.543 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:09.543 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:09.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:09.543 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:09.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.544 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:09.544 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:09.544 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 135.08763203199999, 2024-04-22T18:41:09.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:41:09.544 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.544 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.544 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:09.544 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:09.544 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.544 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:41:09.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.544 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:09.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:09.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:41:09.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:09.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:41:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:41:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:41:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:41:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:41:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:41:09.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:41:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:41:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:41:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:41:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:41:09.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:41:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:41:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:41:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:41:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:41:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:41:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:41:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:41:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:41:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:09.549 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 126.394574784, 2024-04-22T18:41:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:41:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:41:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:09.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:41:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:41:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:41:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:41:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:41:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:41:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:41:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:41:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:41:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:41:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:41:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:41:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:41:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:41:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:41:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 126.39455630800001, 2024-04-22T18:41:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:41:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:09.555 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:41:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:41:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:41:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:41:09.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:41:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:41:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:41:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:41:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:41:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:41:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:41:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:41:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:41:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:41:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:41:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:41:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:41:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:41:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 98.631507995999996, 2024-04-22T18:41:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:41:09.560 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:41:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:41:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:41:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:41:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:41:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:41:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:41:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:41:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:41:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:41:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:41:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:41:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:41:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:41:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:41:09.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:41:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:41:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:41:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:41:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:41:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:41:09.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:41:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 93.102348669999998, 2024-04-22T18:41:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:41:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:09.566 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:41:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:41:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:41:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:41:09.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:41:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:41:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:41:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:41:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:41:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:09.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:41:09.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:41:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:41:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.924641274999999, 2024-04-22T18:41:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:41:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:41:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:41:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:41:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:41:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:41:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:41:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:41:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:41:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:41:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:41:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:41:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:41:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:41:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:41:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:41:09.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:41:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:41:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:41:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.924573003000006, 2024-04-22T18:41:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:41:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.576 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:41:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:41:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:41:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:41:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:41:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:41:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:41:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:41:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:41:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:41:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:41:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:41:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:41:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:41:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:41:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:41:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:41:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:41:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:41:09.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:41:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:41:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:41:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:41:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.581 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.924405161999999, 2024-04-22T18:41:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:41:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:41:09.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:41:09.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:41:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:41:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:41:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:41:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:09.586 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 88.959079686999999, 2024-04-22T18:41:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:41:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:41:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:41:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:41:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:41:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:41:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:41:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:41:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:41:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:41:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:41:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:41:09.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:41:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:41:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:41:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:41:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:41:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:41:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 88.858927499999993, 2024-04-22T18:41:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:41:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:41:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:41:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:41:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:41:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:41:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:41:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:41:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:41:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:41:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:41:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:41:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:41:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:41:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:41:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:41:09.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:41:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:41:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:41:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:41:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:41:09.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:41:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:41:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:41:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:41:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:09.597 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 79.150062980000001, 2024-04-22T18:41:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:41:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:41:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:41:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:41:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:41:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:41:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:41:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:41:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:41:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:41:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:41:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:41:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:41:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:41:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:41:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:41:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:41:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:41:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:41:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:41:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 62.254024608000002, 2024-04-22T18:41:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:41:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:41:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:41:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:41:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:41:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:41:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:41:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:41:09.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:41:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:41:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:41:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:41:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:41:09.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:41:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3330 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-04-22T18:41:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:41:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.518946321999998, 2024-04-22T18:41:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43151673600000001, 2024-04-22T18:41:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.607 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3330 2024-04-22T18:41:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:41:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:41:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024041+0000", 2024-04-22T18:41:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024045+0000", 2024-04-22T18:41:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3950000000000001e-06 2024-04-22T18:41:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024051+0000", 2024-04-22T18:41:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0260000000000004e-06 2024-04-22T18:41:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024056+0000", 2024-04-22T18:41:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4489999999999998e-06 2024-04-22T18:41:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028383+0000", 2024-04-22T18:41:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004326557 2024-04-22T18:41:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028471+0000", 2024-04-22T18:41:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8387999999999995e-05 2024-04-22T18:41:09.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.455559+0000", 2024-04-22T18:41:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.42708770699999998 2024-04-22T18:41:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3377 4.6 4:6700eab5:::smithi08937929-32:head [read 1812992~307712] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-04-22T18:41:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.106620966000001, 2024-04-22T18:41:09.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.52001745799999999, 2024-04-22T18:41:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3377 2024-04-22T18:41:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436364+0000", 2024-04-22T18:41:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436376+0000", 2024-04-22T18:41:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2121e-05 2024-04-22T18:41:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436394+0000", 2024-04-22T18:41:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8094e-05 2024-04-22T18:41:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436405+0000", 2024-04-22T18:41:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0189e-05 2024-04-22T18:41:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436471+0000", 2024-04-22T18:41:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6817999999999996e-05 2024-04-22T18:41:09.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436481+0000", 2024-04-22T18:41:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2720000000000003e-06 2024-04-22T18:41:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956084+0000", 2024-04-22T18:41:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51960320800000004 2024-04-22T18:41:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956232+0000", 2024-04-22T18:41:09.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014778600000000001 2024-04-22T18:41:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956385+0000", 2024-04-22T18:41:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153394 2024-04-22T18:41:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:131 3.1 3:a0a19997:::ee8ca23a7050e771c77dff761e6ea5a11ed57796f4e8a17cbb5804a7fdc937b9:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.213751369999997, 2024-04-22T18:41:09.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51449564999999997, 2024-04-22T18:41:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329235+0000", 2024-04-22T18:41:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329244+0000", 2024-04-22T18:41:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6819999999999992e-06 2024-04-22T18:41:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329259+0000", 2024-04-22T18:41:09.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698000000000001e-05 2024-04-22T18:41:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329265+0000", 2024-04-22T18:41:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1510000000000001e-06 2024-04-22T18:41:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329311+0000", 2024-04-22T18:41:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6096e-05 2024-04-22T18:41:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.332269+0000", 2024-04-22T18:41:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029581910000000002 2024-04-22T18:41:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630603+0000", 2024-04-22T18:41:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29833418 2024-04-22T18:41:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630685+0000", 2024-04-22T18:41:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1217999999999994e-05 2024-04-22T18:41:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630813+0000", 2024-04-22T18:41:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128706 2024-04-22T18:41:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.631447+0000", 2024-04-22T18:41:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063427000000000004 2024-04-22T18:41:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843586+0000", 2024-04-22T18:41:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21213873 2024-04-22T18:41:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843623+0000", 2024-04-22T18:41:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6848999999999999e-05 2024-04-22T18:41:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843733+0000", 2024-04-22T18:41:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010982200000000001 2024-04-22T18:41:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3441 4.b 4:d3068ee3:::smithi08937929-29:head [write 558065~619869] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.625275309000003, 2024-04-22T18:41:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50272020399999995, 2024-04-22T18:41:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3441 2024-04-22T18:41:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917711+0000", 2024-04-22T18:41:09.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917838+0000", 2024-04-22T18:41:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012734899999999999 2024-04-22T18:41:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917854+0000", 2024-04-22T18:41:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6130999999999999e-05 2024-04-22T18:41:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917863+0000", 2024-04-22T18:41:09.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5969999999999992e-06 2024-04-22T18:41:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992845+0000", 2024-04-22T18:41:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.074982193000000003 2024-04-22T18:41:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992919+0000", 2024-04-22T18:41:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:41:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:09.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.993019+0000", 2024-04-22T18:41:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9718999999999999e-05 2024-04-22T18:41:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095554+0000", 2024-04-22T18:41:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102535193 2024-04-22T18:41:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420183+0000", 2024-04-22T18:41:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32462853600000002 2024-04-22T18:41:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420401+0000", 2024-04-22T18:41:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021881400000000001 2024-04-22T18:41:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420433+0000", 2024-04-22T18:41:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079000000000002e-05 2024-04-22T18:41:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3442 4.b 4:d3068ee3:::smithi08937929-29:head [write 1579835~486765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:09.628 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.620588597999998, 2024-04-22T18:41:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49822799099999998, 2024-04-22T18:41:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3442 2024-04-22T18:41:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:09.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922399+0000", 2024-04-22T18:41:09.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:09.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922502+0000", 2024-04-22T18:41:09.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102784 2024-04-22T18:41:09.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922513+0000", 2024-04-22T18:41:09.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1051e-05 2024-04-22T18:41:09.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922518+0000", 2024-04-22T18:41:09.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8529999999999997e-06 2024-04-22T18:41:09.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093158+0000", 2024-04-22T18:41:09.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.17063958800000001 2024-04-22T18:41:09.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093230+0000", 2024-04-22T18:41:09.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1803000000000004e-05 2024-04-22T18:41:09.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:09.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093326+0000", 2024-04-22T18:41:09.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6454999999999997e-05 2024-04-22T18:41:09.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:09.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095557+0000", 2024-04-22T18:41:09.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022306399999999999 2024-04-22T18:41:09.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420484+0000", 2024-04-22T18:41:09.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32492717700000001 2024-04-22T18:41:09.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:09.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420606+0000", 2024-04-22T18:41:09.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012225800000000001 2024-04-22T18:41:09.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420628+0000", 2024-04-22T18:41:09.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1596000000000001e-05 2024-04-22T18:41:09.634 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.634 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.634 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.634 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3443 4.b 4:d3068ee3:::smithi08937929-29:head [write 2468371~489694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:09.634 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:09.634 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.615986550999999, 2024-04-22T18:41:09.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49372487199999998, 2024-04-22T18:41:09.634 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.634 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.634 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:09.634 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:09.634 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.634 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3443 2024-04-22T18:41:09.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.635 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:09.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:09.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927001+0000", 2024-04-22T18:41:09.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:09.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927117+0000", 2024-04-22T18:41:09.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011637 2024-04-22T18:41:09.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927128+0000", 2024-04-22T18:41:09.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0696000000000001e-05 2024-04-22T18:41:09.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927134+0000", 2024-04-22T18:41:09.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8640000000000001e-06 2024-04-22T18:41:09.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094288+0000", 2024-04-22T18:41:09.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16715412900000001 2024-04-22T18:41:09.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094359+0000", 2024-04-22T18:41:09.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1639999999999998e-05 2024-04-22T18:41:09.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:09.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094451+0000", 2024-04-22T18:41:09.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1252e-05 2024-04-22T18:41:09.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:09.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095560+0000", 2024-04-22T18:41:09.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011095530000000001 2024-04-22T18:41:09.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420661+0000", 2024-04-22T18:41:09.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32510105500000003 2024-04-22T18:41:09.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:09.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420710+0000", 2024-04-22T18:41:09.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9018999999999999e-05 2024-04-22T18:41:09.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420727+0000", 2024-04-22T18:41:09.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6722e-05 2024-04-22T18:41:09.639 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.639 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.639 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.641 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3444 4.b 4:d3068ee3:::smithi08937929-29:head [setxattr _header (56) in=7b,truncate 2958065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:09.641 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:09.641 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.615850834, 2024-04-22T18:41:09.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49366935299999998, 2024-04-22T18:41:09.641 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.641 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:09.641 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:09.641 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:09.641 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.641 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3444 2024-04-22T18:41:09.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.642 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:09.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:09.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927137+0000", 2024-04-22T18:41:09.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927141+0000", 2024-04-22T18:41:09.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3270000000000001e-06 2024-04-22T18:41:09.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927144+0000", 2024-04-22T18:41:09.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9730000000000002e-06 2024-04-22T18:41:09.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927147+0000", 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.959e-06 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095118+0000", 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16797115500000001 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095169+0000", 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1316000000000001e-05 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:09.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095260+0000", 2024-04-22T18:41:09.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1184e-05 2024-04-22T18:41:09.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:09.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095999+0000", 2024-04-22T18:41:09.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073868200000000005 2024-04-22T18:41:09.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:09.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420757+0000", 2024-04-22T18:41:09.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32475783499999999 2024-04-22T18:41:09.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420781+0000", 2024-04-22T18:41:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4022000000000001e-05 2024-04-22T18:41:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420807+0000", 2024-04-22T18:41:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6366000000000001e-05 2024-04-22T18:41:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3445 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.615839739999998, 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.493780049, 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3445 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927150+0000", 2024-04-22T18:41:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.542e-06 2024-04-22T18:41:09.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927153+0000", 2024-04-22T18:41:09.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0520000000000002e-06 2024-04-22T18:41:09.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027218+0000", 2024-04-22T18:41:09.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100064604 2024-04-22T18:41:09.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095501+0000", 2024-04-22T18:41:09.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.068283431000000006 2024-04-22T18:41:09.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:09.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095545+0000", 2024-04-22T18:41:09.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4295999999999997e-05 2024-04-22T18:41:09.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420833+0000", 2024-04-22T18:41:09.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32528718400000001 2024-04-22T18:41:09.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420877+0000", 2024-04-22T18:41:09.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4403999999999997e-05 2024-04-22T18:41:09.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:09.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420929+0000", 2024-04-22T18:41:09.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1848999999999998e-05 2024-04-22T18:41:09.651 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.651 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.651 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.651 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.651 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.651 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:09.653 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:09.654 INFO:tasks.rados.rados.0.smithi089.stdout:1768: finishing set_chunk to oid 7 2024-04-22T18:41:09.654 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 65 (ObjNum 346 snap 0 seq_num 346) dirty exists 2024-04-22T18:41:09.654 INFO:tasks.rados.rados.0.smithi089.stdout:1767: finishing set_chunk to oid 40 2024-04-22T18:41:09.654 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 65 (ObjNum 354 snap 0 seq_num 354) dirty exists 2024-04-22T18:41:09.654 INFO:tasks.rados.rados.0.smithi089.stdout:1769: got (0) Success 2024-04-22T18:41:09.654 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 0 (ObjNum 364 snap 0 seq_num 364) dirty exists 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1762: done (10 left) 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1763: done (9 left) 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1764: done (8 left) 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1765: done (7 left) 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1766: done (6 left) 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1767: done (5 left) 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1768: done (4 left) 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1769: done (3 left) 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1770: done (2 left) 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1771: done (1 left) 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1772: done (0 left) 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1773: tier_promote oid 15 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1774: write (excl) oid 17 current snap is 0 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1774: seq_num 366 ranges {551362=726078,1717586=409779} 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1774: writing smithi08937929-17 from 551362 to 1277440 tid 1 2024-04-22T18:41:09.655 INFO:tasks.rados.rados.0.smithi089.stdout:1774: writing smithi08937929-17 from 1717586 to 2127365 tid 2 2024-04-22T18:41:09.656 INFO:tasks.rados.rados.0.smithi089.stdout:1775: read oid 13 snap -1 2024-04-22T18:41:09.656 INFO:tasks.rados.rados.0.smithi089.stdout:1775: expect (ObjNum 348 snap 0 seq_num 348) 2024-04-22T18:41:09.656 INFO:tasks.rados.rados.0.smithi089.stdout:1776: set_chunk oid 18 target oid 2024-04-22T18:41:09.656 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:09.654+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:09.722 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:41:09.779 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:09.781+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:09.799 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:09.799 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:09.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.799 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGRecoveryDeleteReply(3.3 e134,130 3:eae3b13c:::b4288df75b8148ed492787eb76c8ece0c94303d4960b7832674dbea657b7d216:head,65'211)", 2024-04-22T18:41:09.799 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:09.733891+0000", 2024-04-22T18:41:09.799 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.066908861, 2024-04-22T18:41:09.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.066929330999999995, 2024-04-22T18:41:09.799 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.799 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:41:09.799 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.733891+0000", 2024-04-22T18:41:09.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.733891+0000", 2024-04-22T18:41:09.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.733891+0000", 2024-04-22T18:41:09.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:09.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.733893+0000", 2024-04-22T18:41:09.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4559999999999999e-06 2024-04-22T18:41:09.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.733899+0000", 2024-04-22T18:41:09.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9120000000000003e-06 2024-04-22T18:41:09.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.733903+0000", 2024-04-22T18:41:09.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8029999999999998e-06 2024-04-22T18:41:09.801 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.801 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.802 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.802 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4280 4.a 4:519c08f7:::smithi08937929-15:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-04-22T18:41:09.802 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:09.625500+0000", 2024-04-22T18:41:09.802 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.175299857, 2024-04-22T18:41:09.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.17535614099999999, 2024-04-22T18:41:09.802 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:09.802 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:09.802 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:09.802 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:09.802 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:09.802 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4280 2024-04-22T18:41:09.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.803 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:09.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:09.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.625500+0000", 2024-04-22T18:41:09.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:09.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.625500+0000", 2024-04-22T18:41:09.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:09.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:09.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.625498+0000", 2024-04-22T18:41:09.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:09.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:09.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.625505+0000", 2024-04-22T18:41:09.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-04-22T18:41:09.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:09.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.625517+0000", 2024-04-22T18:41:09.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1817999999999999e-05 2024-04-22T18:41:09.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:09.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.625523+0000", 2024-04-22T18:41:09.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0650000000000004e-06 2024-04-22T18:41:09.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:09.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.725700+0000", 2024-04-22T18:41:09.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10017693 2024-04-22T18:41:09.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:09.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:09.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:09.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.725766+0000", 2024-04-22T18:41:09.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5895999999999994e-05 2024-04-22T18:41:09.806 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.806 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:09.806 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.806 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:09.806 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:41:09.806 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 2 2024-04-22T18:41:09.806 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:09.809 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:09.858 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:09.858 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:09.859 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:09.860+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:09.875 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:09.876 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:09.951 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:41:09.952 INFO:tasks.rados.rados.0.smithi089.stdout:1776: set_chunk oid 18 offset: 197120 length: 2130944 target oid 9341e24c4a92f7ec719282d2faa5711f9644dee45a94ba5b4c4c1071ddc0e536 offset: 0 snap -1 2024-04-22T18:41:09.952 INFO:tasks.rados.rados.0.smithi089.stdout:1777: set_chunk oid 41 target oid 2024-04-22T18:41:09.955 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:09.955+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:09.956 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:09.956 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:09.966 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:09.967 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:09.968 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:09.984 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:09.984 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:10.056 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:10.057+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:10.066 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:10.066 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:10.075 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:10.076 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:10.085 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:10.140 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:10.140 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:10.157 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:10.157+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:10.158 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:10.158 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:10.175 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:10.175 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:10.228 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:10.228 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:10.237 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:10.258 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:10.259+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:10.338 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:10.338 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:10.347 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:10.356 INFO:tasks.rados.rados.0.smithi089.stdout:1777: set_chunk oid 41 offset: 372736 length: 181760 target oid bceb889cef88762e5841b42cac4bc063706e6755c73a7efad92c922e8e034607 offset: 0 snap -1 2024-04-22T18:41:10.357 INFO:tasks.rados.rados.0.smithi089.stdout:1778: read oid 3 snap -1 2024-04-22T18:41:10.357 INFO:tasks.rados.rados.0.smithi089.stdout:1778: expect (ObjNum 259 snap 0 seq_num 259) 2024-04-22T18:41:10.357 INFO:tasks.rados.rados.0.smithi089.stdout:1779: delete oid 24 current snap is 0 2024-04-22T18:41:10.357 INFO:tasks.rados.rados.0.smithi089.stdout:1775: expect (ObjNum 348 snap 0 seq_num 348) 2024-04-22T18:41:10.359 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:10.360+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:10.449 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:10.449 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:10.459 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:10.461 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:10.461+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:10.523 INFO:tasks.rados.rados.0.smithi089.stdout:1773: got (0) Success 2024-04-22T18:41:10.523 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 94 (ObjNum 301 snap 0 seq_num 301) dirty exists 2024-04-22T18:41:10.523 INFO:tasks.rados.rados.0.smithi089.stdout:1774: finishing write tid 1 to smithi08937929-17 2024-04-22T18:41:10.523 INFO:tasks.rados.rados.0.smithi089.stdout:1774: finishing write tid 2 to smithi08937929-17 2024-04-22T18:41:10.523 INFO:tasks.rados.rados.0.smithi089.stdout:1774: finishing write tid 3 to smithi08937929-17 2024-04-22T18:41:10.523 INFO:tasks.rados.rados.0.smithi089.stdout:1774: finishing write tid 4 to smithi08937929-17 2024-04-22T18:41:10.523 INFO:tasks.rados.rados.0.smithi089.stdout:1774: oid 17 updating version 0 to 87 2024-04-22T18:41:10.523 INFO:tasks.rados.rados.0.smithi089.stdout:1774: oid 17 updating version 87 to 88 2024-04-22T18:41:10.523 INFO:tasks.rados.rados.0.smithi089.stdout:1774: oid 17 updating version 88 to 89 2024-04-22T18:41:10.523 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 17 v 89 (ObjNum 366 snap 0 seq_num 366) dirty exists 2024-04-22T18:41:10.523 INFO:tasks.rados.rados.0.smithi089.stdout:1774: left oid 17 (ObjNum 366 snap 0 seq_num 366) 2024-04-22T18:41:10.523 INFO:tasks.rados.rados.0.smithi089.stdout:1776: finishing set_chunk to oid 18 2024-04-22T18:41:10.523 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 18 set_chunk returned error code -95 offset: 197120 length: 2130944 2024-04-22T18:41:10.524 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 0 (ObjNum 307 snap 0 seq_num 307) dirty exists 2024-04-22T18:41:10.524 INFO:tasks.rados.rados.0.smithi089.stdout:1777: finishing set_chunk to oid 41 2024-04-22T18:41:10.524 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 41 set_chunk returned error code -95 offset: 372736 length: 181760 2024-04-22T18:41:10.524 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 41 v 0 (ObjNum 358 snap 0 seq_num 358) dirty exists 2024-04-22T18:41:10.561 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:10.562+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:10.601 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:10.601 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:10.610 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:10.693 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:10.695+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:10.751 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:10.751 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:10.760 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:10.801 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:10.802+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:10.863 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:10.864+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:10.901 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:10.901 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:10.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:41:10.964 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:10.965+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:11.016 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:11.016 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:11.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:11.068 INFO:tasks.rados.rados.0.smithi089.stdout:1778: expect (ObjNum 259 snap 0 seq_num 259) 2024-04-22T18:41:11.069 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:11.070+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:11.128 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:11.129 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:11.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:11.178 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:11.179+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:11.267 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:11.268+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:11.279 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:11.280 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:11.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:11.367 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:11.368+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:11.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:11.481 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:11.482+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:11.538 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:11.538 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:11.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:11.569 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:11.571+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:11.669 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:11.671+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:11.691 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:11.691 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:11.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:11.770 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:11.772+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:11.802 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:11.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:11.871 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:11.872+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:11.913 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:11.913 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:11.921 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:11.972 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:11.972+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:12.065 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:12.065 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:12.069 INFO:tasks.rados.rados.0.smithi089.stdout:1773: done (6 left) 2024-04-22T18:41:12.069 INFO:tasks.rados.rados.0.smithi089.stdout:1774: done (5 left) 2024-04-22T18:41:12.069 INFO:tasks.rados.rados.0.smithi089.stdout:1775: done (4 left) 2024-04-22T18:41:12.069 INFO:tasks.rados.rados.0.smithi089.stdout:1776: done (3 left) 2024-04-22T18:41:12.069 INFO:tasks.rados.rados.0.smithi089.stdout:1777: done (2 left) 2024-04-22T18:41:12.069 INFO:tasks.rados.rados.0.smithi089.stdout:1778: done (1 left) 2024-04-22T18:41:12.069 INFO:tasks.rados.rados.0.smithi089.stdout:1779: done (0 left) 2024-04-22T18:41:12.069 INFO:tasks.rados.rados.0.smithi089.stdout:1780: write oid 37 current snap is 0 2024-04-22T18:41:12.069 INFO:tasks.rados.rados.0.smithi089.stdout:1780: seq_num 368 ranges {584976=476576,1834550=350426} 2024-04-22T18:41:12.072 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:12.073+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:12.074 INFO:tasks.rados.rados.0.smithi089.stdout:1780: writing smithi08937929-37 from 584976 to 1061552 tid 1 2024-04-22T18:41:12.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:12.081 INFO:tasks.rados.rados.0.smithi089.stdout:1780: writing smithi08937929-37 from 1834550 to 2184976 tid 2 2024-04-22T18:41:12.081 INFO:tasks.rados.rados.0.smithi089.stdout:1781: read oid 9 snap -1 2024-04-22T18:41:12.081 INFO:tasks.rados.rados.0.smithi089.stdout:1781: expect (ObjNum 283 snap 0 seq_num 283) 2024-04-22T18:41:12.081 INFO:tasks.rados.rados.0.smithi089.stdout:1782: write oid 50 current snap is 0 2024-04-22T18:41:12.081 INFO:tasks.rados.rados.0.smithi089.stdout:1782: seq_num 369 ranges {601783=551825,1776679=425104} 2024-04-22T18:41:12.087 INFO:tasks.rados.rados.0.smithi089.stdout:1782: writing smithi08937929-50 from 601783 to 1153608 tid 1 2024-04-22T18:41:12.092 INFO:tasks.rados.rados.0.smithi089.stdout:1782: writing smithi08937929-50 from 1776679 to 2201783 tid 2 2024-04-22T18:41:12.093 INFO:tasks.rados.rados.0.smithi089.stdout:1783: set_chunk oid 23 target oid 2024-04-22T18:41:12.173 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:12.174+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:12.180 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:12.180 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:12.187 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:41:12.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:12.274 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:12.275+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:12.307 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:12.307 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:12.323 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:12.324 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:12.334 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:12.339 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:12.340 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:12.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:12.355 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:12.355 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:12.384 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:12.385+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:12.472 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:12.472 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:12.475 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:12.476+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:12.489 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:12.489 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:12.489 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:12.489 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:12.499 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:12.505 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:12.506 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:12.522 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:12.523 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:12.576 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:12.577+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:12.603 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:12.603 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:12.604 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:12.604 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:12.612 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:12.677 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:12.677+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:12.714 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:12.714 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:12.714 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:12.715 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:12.715 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:12.723 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:12.778 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:12.779+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:12.798 INFO:tasks.rados.rados.0.smithi089.stdout:1783: set_chunk oid 23 offset: 1843200 length: 1201152 target oid a4ca16af619d516621f40f5ca35c6f0f9b8052449dac48568a5f1c009853b706 offset: 0 snap -1 2024-04-22T18:41:12.798 INFO:tasks.rados.rados.0.smithi089.stdout:1784: read oid 15 snap -1 2024-04-22T18:41:12.798 INFO:tasks.rados.rados.0.smithi089.stdout:1784: expect (ObjNum 301 snap 0 seq_num 301) 2024-04-22T18:41:12.799 INFO:tasks.rados.rados.0.smithi089.stdout:1785: write (excl) oid 43 current snap is 0 2024-04-22T18:41:12.799 INFO:tasks.rados.rados.0.smithi089.stdout:1785: seq_num 370 ranges {618590=627074,1835161=383429} 2024-04-22T18:41:12.805 INFO:tasks.rados.rados.0.smithi089.stdout:1785: writing smithi08937929-43 from 618590 to 1245664 tid 1 2024-04-22T18:41:12.810 INFO:tasks.rados.rados.0.smithi089.stdout:1785: writing smithi08937929-43 from 1835161 to 2218590 tid 2 2024-04-22T18:41:12.810 INFO:tasks.rados.rados.0.smithi089.stdout:1786: read oid 22 snap -1 2024-04-22T18:41:12.810 INFO:tasks.rados.rados.0.smithi089.stdout:1786: expect (ObjNum 303 snap 0 seq_num 303) 2024-04-22T18:41:12.811 INFO:tasks.rados.rados.0.smithi089.stdout:1787: set_chunk oid 18 target oid 2024-04-22T18:41:12.869 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 138.41404774599999, 2024-04-22T18:41:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:41:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:41:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:41:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:41:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:41:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:41:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:41:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:41:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:41:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:41:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:41:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:41:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:41:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:41:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:41:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:41:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:41:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:41:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:41:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:41:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:41:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:41:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 129.72099049799999, 2024-04-22T18:41:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:41:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:41:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:41:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:41:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:41:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:41:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:41:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:41:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:41:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:41:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:41:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:41:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:41:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:41:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:41:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:41:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:41:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:41:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:41:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:41:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:41:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:41:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 129.72097202200001, 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:41:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:41:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:41:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:41:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:41:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:41:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:41:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:41:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:41:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:41:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:41:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:41:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 101.95792371, 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:41:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:41:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:41:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:41:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:41:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:41:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:41:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:41:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:41:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:41:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:41:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:41:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:41:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:41:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:41:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:41:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:41:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:41:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:41:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:41:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.428764384000004, 2024-04-22T18:41:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:41:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:41:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:41:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:41:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:41:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:41:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:41:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:41:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:41:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:41:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:41:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:41:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.251056989000006, 2024-04-22T18:41:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:41:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:41:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:41:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:41:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:41:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:41:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:41:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:41:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.250988716999998, 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:41:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:41:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.250820876000006, 2024-04-22T18:41:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:41:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:41:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:41:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:41:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:41:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:41:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:41:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:41:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:41:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.285495401000006, 2024-04-22T18:41:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:41:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:41:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:41:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:41:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:41:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:41:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:41:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:41:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:41:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:41:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:41:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:41:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:41:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:41:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.185343214, 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:41:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:41:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:41:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:41:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:41:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:41:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:41:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:41:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:41:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:41:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:41:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:41:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:41:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:41:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:41:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:41:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:41:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:41:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:41:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:41:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:41:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.476478693999994, 2024-04-22T18:41:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:41:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:41:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:41:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:41:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:41:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:41:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:41:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:41:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:41:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:41:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:41:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:41:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:41:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:41:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:41:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:41:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:41:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:41:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:41:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.580440322000001, 2024-04-22T18:41:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:41:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:41:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:41:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:41:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:41:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:41:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:41:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:41:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:41:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:41:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:41:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:41:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:41:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:41:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3330 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-04-22T18:41:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:41:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.845362035999997, 2024-04-22T18:41:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43151673600000001, 2024-04-22T18:41:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3330 2024-04-22T18:41:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:41:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:41:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024041+0000", 2024-04-22T18:41:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024045+0000", 2024-04-22T18:41:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3950000000000001e-06 2024-04-22T18:41:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024051+0000", 2024-04-22T18:41:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0260000000000004e-06 2024-04-22T18:41:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024056+0000", 2024-04-22T18:41:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4489999999999998e-06 2024-04-22T18:41:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028383+0000", 2024-04-22T18:41:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004326557 2024-04-22T18:41:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028471+0000", 2024-04-22T18:41:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8387999999999995e-05 2024-04-22T18:41:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.455559+0000", 2024-04-22T18:41:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.42708770699999998 2024-04-22T18:41:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3377 4.6 4:6700eab5:::smithi08937929-32:head [read 1812992~307712] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-04-22T18:41:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.433036680000001, 2024-04-22T18:41:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.52001745799999999, 2024-04-22T18:41:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3377 2024-04-22T18:41:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436364+0000", 2024-04-22T18:41:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436376+0000", 2024-04-22T18:41:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2121e-05 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436394+0000", 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8094e-05 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436405+0000", 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0189e-05 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436471+0000", 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6817999999999996e-05 2024-04-22T18:41:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436481+0000", 2024-04-22T18:41:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2720000000000003e-06 2024-04-22T18:41:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956084+0000", 2024-04-22T18:41:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51960320800000004 2024-04-22T18:41:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956232+0000", 2024-04-22T18:41:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014778600000000001 2024-04-22T18:41:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956385+0000", 2024-04-22T18:41:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153394 2024-04-22T18:41:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:131 3.1 3:a0a19997:::ee8ca23a7050e771c77dff761e6ea5a11ed57796f4e8a17cbb5804a7fdc937b9:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.540167083999997, 2024-04-22T18:41:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51449564999999997, 2024-04-22T18:41:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329235+0000", 2024-04-22T18:41:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329244+0000", 2024-04-22T18:41:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6819999999999992e-06 2024-04-22T18:41:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329259+0000", 2024-04-22T18:41:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698000000000001e-05 2024-04-22T18:41:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329265+0000", 2024-04-22T18:41:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1510000000000001e-06 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329311+0000", 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6096e-05 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.332269+0000", 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029581910000000002 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630603+0000", 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29833418 2024-04-22T18:41:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630685+0000", 2024-04-22T18:41:12.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1217999999999994e-05 2024-04-22T18:41:12.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:12.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630813+0000", 2024-04-22T18:41:12.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128706 2024-04-22T18:41:12.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:12.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.631447+0000", 2024-04-22T18:41:12.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063427000000000004 2024-04-22T18:41:12.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843586+0000", 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21213873 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843623+0000", 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6848999999999999e-05 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843733+0000", 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010982200000000001 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.944 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3441 4.b 4:d3068ee3:::smithi08937929-29:head [write 558065~619869] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:12.944 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:12.944 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.951691023000002, 2024-04-22T18:41:12.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50272020399999995, 2024-04-22T18:41:12.944 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.944 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:12.944 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:12.944 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:12.944 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:12.944 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3441 2024-04-22T18:41:12.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.944 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917711+0000", 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917838+0000", 2024-04-22T18:41:12.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012734899999999999 2024-04-22T18:41:12.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917854+0000", 2024-04-22T18:41:12.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6130999999999999e-05 2024-04-22T18:41:12.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917863+0000", 2024-04-22T18:41:12.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5969999999999992e-06 2024-04-22T18:41:12.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992845+0000", 2024-04-22T18:41:12.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.074982193000000003 2024-04-22T18:41:12.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992919+0000", 2024-04-22T18:41:12.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:41:12.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:12.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.993019+0000", 2024-04-22T18:41:12.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9718999999999999e-05 2024-04-22T18:41:12.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:12.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095554+0000", 2024-04-22T18:41:12.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102535193 2024-04-22T18:41:12.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:12.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420183+0000", 2024-04-22T18:41:12.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32462853600000002 2024-04-22T18:41:12.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:12.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420401+0000", 2024-04-22T18:41:12.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021881400000000001 2024-04-22T18:41:12.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420433+0000", 2024-04-22T18:41:12.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079000000000002e-05 2024-04-22T18:41:12.949 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.949 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.949 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.949 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3442 4.b 4:d3068ee3:::smithi08937929-29:head [write 1579835~486765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:12.949 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:12.949 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.947004311999997, 2024-04-22T18:41:12.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49822799099999998, 2024-04-22T18:41:12.949 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.949 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:12.950 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:12.950 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:12.950 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:12.950 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3442 2024-04-22T18:41:12.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.950 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:12.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:12.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922399+0000", 2024-04-22T18:41:12.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:12.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922502+0000", 2024-04-22T18:41:12.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102784 2024-04-22T18:41:12.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922513+0000", 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1051e-05 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922518+0000", 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8529999999999997e-06 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093158+0000", 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.17063958800000001 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093230+0000", 2024-04-22T18:41:12.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1803000000000004e-05 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093326+0000", 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6454999999999997e-05 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095557+0000", 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022306399999999999 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420484+0000", 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32492717700000001 2024-04-22T18:41:12.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:12.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420606+0000", 2024-04-22T18:41:12.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012225800000000001 2024-04-22T18:41:12.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420628+0000", 2024-04-22T18:41:12.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1596000000000001e-05 2024-04-22T18:41:12.954 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.954 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.954 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3443 4.b 4:d3068ee3:::smithi08937929-29:head [write 2468371~489694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.942402264999998, 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49372487199999998, 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3443 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:12.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:12.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927001+0000", 2024-04-22T18:41:12.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:12.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927117+0000", 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011637 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927128+0000", 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0696000000000001e-05 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927134+0000", 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8640000000000001e-06 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094288+0000", 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16715412900000001 2024-04-22T18:41:12.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094359+0000", 2024-04-22T18:41:12.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1639999999999998e-05 2024-04-22T18:41:12.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:12.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094451+0000", 2024-04-22T18:41:12.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1252e-05 2024-04-22T18:41:12.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:12.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095560+0000", 2024-04-22T18:41:12.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011095530000000001 2024-04-22T18:41:12.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:12.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420661+0000", 2024-04-22T18:41:12.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32510105500000003 2024-04-22T18:41:12.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:12.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420710+0000", 2024-04-22T18:41:12.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9018999999999999e-05 2024-04-22T18:41:12.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420727+0000", 2024-04-22T18:41:12.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6722e-05 2024-04-22T18:41:12.959 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.960 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.960 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.960 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3444 4.b 4:d3068ee3:::smithi08937929-29:head [setxattr _header (56) in=7b,truncate 2958065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:12.960 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:12.960 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.942266547999999, 2024-04-22T18:41:12.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49366935299999998, 2024-04-22T18:41:12.960 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.960 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:12.960 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:12.960 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:12.960 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:12.960 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3444 2024-04-22T18:41:12.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.961 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:12.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:12.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927137+0000", 2024-04-22T18:41:12.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:12.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927141+0000", 2024-04-22T18:41:12.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3270000000000001e-06 2024-04-22T18:41:12.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927144+0000", 2024-04-22T18:41:12.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9730000000000002e-06 2024-04-22T18:41:12.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927147+0000", 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.959e-06 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095118+0000", 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16797115500000001 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095169+0000", 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1316000000000001e-05 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:12.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095260+0000", 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1184e-05 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095999+0000", 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073868200000000005 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420757+0000", 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32475783499999999 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420781+0000", 2024-04-22T18:41:12.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4022000000000001e-05 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420807+0000", 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6366000000000001e-05 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3445 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.942255453999998, 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.493780049, 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:12.965 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:12.966 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:12.966 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:12.966 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:12.966 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3445 2024-04-22T18:41:12.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.966 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:12.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:12.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:12.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:12.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:12.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:12.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:12.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:12.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:12.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:12.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927150+0000", 2024-04-22T18:41:12.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.542e-06 2024-04-22T18:41:12.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:12.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927153+0000", 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0520000000000002e-06 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027218+0000", 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100064604 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095501+0000", 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.068283431000000006 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095545+0000", 2024-04-22T18:41:12.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4295999999999997e-05 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420833+0000", 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32528718400000001 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420877+0000", 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4403999999999997e-05 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420929+0000", 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1848999999999998e-05 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.969 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.970 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.970 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:12.970 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:12.970 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:12.970 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:41:12.971 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:12.972+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:12.972 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:13.072 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:13.072+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:13.101 INFO:tasks.rados.rados.0.smithi089.stdout:1787: set_chunk oid 18 offset: 766464 length: 330240 target oid e7e5c0ec2ff8b6ccf5de829756c2cb752b68ba77fd6e194d5970c12f9d6883c1 offset: 0 snap -1 2024-04-22T18:41:13.101 INFO:tasks.rados.rados.0.smithi089.stdout:1788: read oid 31 snap -1 2024-04-22T18:41:13.101 INFO:tasks.rados.rados.0.smithi089.stdout:1788: expect deleted 2024-04-22T18:41:13.101 INFO:tasks.rados.rados.0.smithi089.stdout:1789: read oid 20 snap -1 2024-04-22T18:41:13.101 INFO:tasks.rados.rados.0.smithi089.stdout:1789: expect (ObjNum 352 snap 0 seq_num 352) 2024-04-22T18:41:13.101 INFO:tasks.rados.rados.0.smithi089.stdout:1780: finishing write tid 1 to smithi08937929-37 2024-04-22T18:41:13.101 INFO:tasks.rados.rados.0.smithi089.stdout:1780: finishing write tid 2 to smithi08937929-37 2024-04-22T18:41:13.101 INFO:tasks.rados.rados.0.smithi089.stdout:1780: finishing write tid 3 to smithi08937929-37 2024-04-22T18:41:13.101 INFO:tasks.rados.rados.0.smithi089.stdout:1781: expect (ObjNum 283 snap 0 seq_num 283) 2024-04-22T18:41:13.117 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPushReply(3.3 137/130 [PushReplyOp(3:ddfd2d2d:::88f5dcc6382512d967c31e9ff351c76732da7aebba828b2f554e518334424da5:head)])", 2024-04-22T18:41:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:13.082750+0000", 2024-04-22T18:41:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.036019455999999998, 2024-04-22T18:41:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.036039724000000002, 2024-04-22T18:41:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:41:13.117 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:13.082750+0000", 2024-04-22T18:41:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:13.082750+0000", 2024-04-22T18:41:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:13.082749+0000", 2024-04-22T18:41:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:13.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:13.082756+0000", 2024-04-22T18:41:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.861e-06 2024-04-22T18:41:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:13.082769+0000", 2024-04-22T18:41:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3046e-05 2024-04-22T18:41:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:13.082778+0000", 2024-04-22T18:41:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8410000000000006e-06 2024-04-22T18:41:13.119 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4327 4.7 4.b1df97e7 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e137)", 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:13.102957+0000", 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.015812479000000001, 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.015866195999999999, 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4327 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.120 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:13.102957+0000", 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:13.102957+0000", 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:13.102956+0000", 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:13.102961+0000", 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2690000000000001e-06 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:13.102967+0000", 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.511e-06 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:13.102973+0000", 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2670000000000003e-06 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.122 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:41:13.123 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 2 2024-04-22T18:41:13.123 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:13.128 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:13.212 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:13.212+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:13.270 INFO:tasks.rados.rados.0.smithi089.stdout:1780: finishing write tid 4 to smithi08937929-37 2024-04-22T18:41:13.270 INFO:tasks.rados.rados.0.smithi089.stdout:1780: oid 37 updating version 0 to 140 2024-04-22T18:41:13.270 INFO:tasks.rados.rados.0.smithi089.stdout:1780: oid 37 updating version 140 to 141 2024-04-22T18:41:13.270 INFO:tasks.rados.rados.0.smithi089.stdout:1780: oid 37 version 141 is already newer than 139 2024-04-22T18:41:13.270 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 141 (ObjNum 368 snap 0 seq_num 368) dirty exists 2024-04-22T18:41:13.271 INFO:tasks.rados.rados.0.smithi089.stdout:1780: left oid 37 (ObjNum 368 snap 0 seq_num 368) 2024-04-22T18:41:13.271 INFO:tasks.rados.rados.0.smithi089.stdout:1782: finishing write tid 1 to smithi08937929-50 2024-04-22T18:41:13.271 INFO:tasks.rados.rados.0.smithi089.stdout:1782: finishing write tid 2 to smithi08937929-50 2024-04-22T18:41:13.271 INFO:tasks.rados.rados.0.smithi089.stdout:1782: finishing write tid 3 to smithi08937929-50 2024-04-22T18:41:13.271 INFO:tasks.rados.rados.0.smithi089.stdout:1782: finishing write tid 4 to smithi08937929-50 2024-04-22T18:41:13.271 INFO:tasks.rados.rados.0.smithi089.stdout:1782: oid 50 updating version 0 to 169 2024-04-22T18:41:13.271 INFO:tasks.rados.rados.0.smithi089.stdout:1782: oid 50 updating version 169 to 170 2024-04-22T18:41:13.271 INFO:tasks.rados.rados.0.smithi089.stdout:1782: oid 50 version 170 is already newer than 168 2024-04-22T18:41:13.271 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 170 (ObjNum 369 snap 0 seq_num 369) dirty exists 2024-04-22T18:41:13.271 INFO:tasks.rados.rados.0.smithi089.stdout:1782: left oid 50 (ObjNum 369 snap 0 seq_num 369) 2024-04-22T18:41:13.271 INFO:tasks.rados.rados.0.smithi089.stdout:1786: expect (ObjNum 303 snap 0 seq_num 303) 2024-04-22T18:41:13.272 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:13.272 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:13.272 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:13.272 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:13.272 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:13.274 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:13.274+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:13.280 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:13.375 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:13.375+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:13.426 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:33 4.f 4:f0ed3d49:::smithi08937929-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:41:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 139.23937885999999, 2024-04-22T18:41:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62359692099999997, 2024-04-22T18:41:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 33 2024-04-22T18:41:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:41:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186802+0000", 2024-04-22T18:41:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.186799+0000", 2024-04-22T18:41:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187103+0000", 2024-04-22T18:41:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030354900000000002 2024-04-22T18:41:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187117+0000", 2024-04-22T18:41:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4578e-05 2024-04-22T18:41:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.187129+0000", 2024-04-22T18:41:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1421e-05 2024-04-22T18:41:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516134+0000", 2024-04-22T18:41:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32900553900000001 2024-04-22T18:41:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516215+0000", 2024-04-22T18:41:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0537999999999994e-05 2024-04-22T18:41:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.516302+0000", 2024-04-22T18:41:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6601000000000007e-05 2024-04-22T18:41:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518207+0000", 2024-04-22T18:41:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019058370000000001 2024-04-22T18:41:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810302+0000", 2024-04-22T18:41:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29209509700000003 2024-04-22T18:41:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810348+0000", 2024-04-22T18:41:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5992999999999998e-05 2024-04-22T18:41:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.810398+0000", 2024-04-22T18:41:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0046000000000002e-05 2024-04-22T18:41:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 139.23465388, 2024-04-22T18:41:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:41:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:41:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:41:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:41:13.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:41:13.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:41:13.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:41:13.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:41:13.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:41:13.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:41:13.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:41:13.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:13.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:41:13.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:41:13.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:13.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:41:13.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:41:13.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:13.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.437 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:13.438 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:13.438 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 139.23464280499999, 2024-04-22T18:41:13.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:41:13.438 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.438 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.438 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.438 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.438 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.438 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:41:13.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.438 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:13.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:41:13.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:41:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:41:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:41:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:41:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:41:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:41:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:41:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:41:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:41:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:41:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 135.63327331100001, 2024-04-22T18:41:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:41:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.442 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:41:13.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:41:13.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:41:13.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:41:13.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:41:13.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:41:13.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:41:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:41:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:41:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:41:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:41:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:41:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:41:13.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:41:13.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:41:13.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:41:13.447 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.447 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.447 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.447 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3193 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:13.447 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.383434516000001, 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69912102700000001, 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3193 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042744+0000", 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043013+0000", 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026906 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043028+0000", 2024-04-22T18:41:13.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5359999999999999e-05 2024-04-22T18:41:13.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043036+0000", 2024-04-22T18:41:13.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8420000000000005e-06 2024-04-22T18:41:13.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043081+0000", 2024-04-22T18:41:13.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5019999999999999e-05 2024-04-22T18:41:13.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:13.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043087+0000", 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5420000000000002e-06 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741856+0000", 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69876864500000002 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741867+0000", 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.084e-05 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3194 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.376184535999997, 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69190137100000004, 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3194 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:41:13.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049991+0000", 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999957 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050284+0000", 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029296699999999999 2024-04-22T18:41:13.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050304+0000", 2024-04-22T18:41:13.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.969e-05 2024-04-22T18:41:13.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050314+0000", 2024-04-22T18:41:13.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0268999999999999e-05 2024-04-22T18:41:13.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150531+0000", 2024-04-22T18:41:13.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100216868 2024-04-22T18:41:13.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150545+0000", 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4063000000000001e-05 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741890+0000", 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59134479100000004 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741897+0000", 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1189999999999999e-06 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.455 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3195 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.375852096000003, 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69159474200000004, 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3195 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:41:13.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:41:13.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:41:13.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:13.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050333+0000", 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0289999999999996e-06 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050341+0000", 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0050000000000004e-06 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050346+0000", 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8329999999999999e-06 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150628+0000", 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100281548 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150672+0000", 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3875999999999998e-05 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741916+0000", 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59124448799999996 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741923+0000", 2024-04-22T18:41:13.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7689999999999999e-06 2024-04-22T18:41:13.460 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.460 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.460 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.460 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3196 4.11 4.9f8e4db1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:13.460 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:41:13.460 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.375821977000001, 2024-04-22T18:41:13.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69158947400000004, 2024-04-22T18:41:13.460 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.460 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:13.460 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.460 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.460 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3196 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050361+0000", 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2040000000000002e-06 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050367+0000", 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3119999999999999e-06 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050372+0000", 2024-04-22T18:41:13.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0869999999999997e-06 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150688+0000", 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100315381 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150691+0000", 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8899999999999999e-06 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741941+0000", 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59125044900000001 2024-04-22T18:41:13.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741948+0000", 2024-04-22T18:41:13.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.827e-06 2024-04-22T18:41:13.464 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.464 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.464 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.464 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3317 4.11 4:8f787b23:::smithi08937929-41:head [setxattr _header (56) in=7b,truncate 2773188] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:41:13.464 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.540240+0000", 2024-04-22T18:41:13.464 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.885940595999998, 2024-04-22T18:41:13.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62115056400000002, 2024-04-22T18:41:13.464 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.464 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3317 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540240+0000", 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540240+0000", 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540239+0000", 2024-04-22T18:41:13.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:13.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540243+0000", 2024-04-22T18:41:13.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4569999999999998e-06 2024-04-22T18:41:13.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540250+0000", 2024-04-22T18:41:13.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6440000000000003e-06 2024-04-22T18:41:13.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540256+0000", 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8100000000000003e-06 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540302+0000", 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5859999999999998e-05 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540309+0000", 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3950000000000003e-06 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.947528+0000", 2024-04-22T18:41:13.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40721847999999999 2024-04-22T18:41:13.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.947588+0000", 2024-04-22T18:41:13.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0364e-05 2024-04-22T18:41:13.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:13.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.947687+0000", 2024-04-22T18:41:13.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8733999999999994e-05 2024-04-22T18:41:13.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:13.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.948414+0000", 2024-04-22T18:41:13.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072766900000000002 2024-04-22T18:41:13.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:13.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161295+0000", 2024-04-22T18:41:13.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.212880818 2024-04-22T18:41:13.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161342+0000", 2024-04-22T18:41:13.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6894000000000002e-05 2024-04-22T18:41:13.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161390+0000", 2024-04-22T18:41:13.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8245999999999999e-05 2024-04-22T18:41:13.470 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.470 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.470 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.470 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3318 4.11 4:8f787b23:::smithi08937929-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:41:13.470 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:41:13.470 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.885910404000001, 2024-04-22T18:41:13.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.62124934899999995, 2024-04-22T18:41:13.470 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.470 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3318 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540270+0000", 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540276+0000", 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.049e-06 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540282+0000", 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7050000000000002e-06 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540288+0000", 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2900000000000002e-06 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540326+0000", 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8818e-05 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.540330+0000", 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6260000000000001e-06 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.948024+0000", 2024-04-22T18:41:13.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40769397600000001 2024-04-22T18:41:13.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:13.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:26.948063+0000", 2024-04-22T18:41:13.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9127e-05 2024-04-22T18:41:13.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161420+0000", 2024-04-22T18:41:13.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.213356876 2024-04-22T18:41:13.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161463+0000", 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3219000000000001e-05 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.161519+0000", 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6155e-05 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3504 4.9 4:96737e19:::smithi08937929-4:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.015318800999999, 2024-04-22T18:41:13.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.951755291, 2024-04-22T18:41:13.476 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.476 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.476 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.476 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.476 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.476 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3504 2024-04-22T18:41:13.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.476 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:13.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410861+0000", 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410863+0000", 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.635e-06 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410866+0000", 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8499999999999998e-06 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410868+0000", 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.497e-06 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615137+0000", 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20426903499999999 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615218+0000", 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0860999999999997e-05 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.362617+0000", 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.74739882700000004 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4194 4.b 4:d3068ee3:::smithi08937929-29:head [sparse-read 0~1980000,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.262059730000001, 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.294164774, 2024-04-22T18:41:13.479 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4194 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:13.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164120+0000", 2024-04-22T18:41:13.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164125+0000", 2024-04-22T18:41:13.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2899999999999996e-06 2024-04-22T18:41:13.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164128+0000", 2024-04-22T18:41:13.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8700000000000002e-06 2024-04-22T18:41:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164132+0000", 2024-04-22T18:41:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.151e-06 2024-04-22T18:41:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164188+0000", 2024-04-22T18:41:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6912e-05 2024-04-22T18:41:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:13.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164201+0000", 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2466000000000001e-05 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.242879+0000", 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0786779710000001 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243364+0000", 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000485081 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456588+0000", 2024-04-22T18:41:13.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21322370900000001 2024-04-22T18:41:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456674+0000", 2024-04-22T18:41:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6079e-05 2024-04-22T18:41:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.458285+0000", 2024-04-22T18:41:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001611716 2024-04-22T18:41:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4195 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.262042107999999, 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.395681808, 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4195 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:13.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164141+0000", 2024-04-22T18:41:13.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7160000000000001e-06 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164147+0000", 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0509999999999996e-06 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264244+0000", 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100097063 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264319+0000", 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5729000000000006e-05 2024-04-22T18:41:13.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264331+0000", 2024-04-22T18:41:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048999999999999e-05 2024-04-22T18:41:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243392+0000", 2024-04-22T18:41:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97906041099999996 2024-04-22T18:41:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:13.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243451+0000", 2024-04-22T18:41:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8717e-05 2024-04-22T18:41:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558398+0000", 2024-04-22T18:41:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31494718500000002 2024-04-22T18:41:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558490+0000", 2024-04-22T18:41:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2529999999999997e-05 2024-04-22T18:41:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559820+0000", 2024-04-22T18:41:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013299060000000001 2024-04-22T18:41:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4196 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.161903361, 2024-04-22T18:41:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2963829229999999, 2024-04-22T18:41:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4196 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264275+0000", 2024-04-22T18:41:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264290+0000", 2024-04-22T18:41:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5373e-05 2024-04-22T18:41:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264300+0000", 2024-04-22T18:41:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.747e-06 2024-04-22T18:41:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264314+0000", 2024-04-22T18:41:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.396e-05 2024-04-22T18:41:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364450+0000", 2024-04-22T18:41:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013662500000001 2024-04-22T18:41:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364459+0000", 2024-04-22T18:41:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0710000000000001e-06 2024-04-22T18:41:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245690+0000", 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88123066400000005 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245766+0000", 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5938000000000004e-05 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559874+0000", 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31410827499999999 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559930+0000", 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5367e-05 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.560660+0000", 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073043800000000003 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4182 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4982205159999999, 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2059652709999999, 2024-04-22T18:41:13.495 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4182 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927959+0000", 2024-04-22T18:41:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927962+0000", 2024-04-22T18:41:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4269999999999998e-06 2024-04-22T18:41:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927968+0000", 2024-04-22T18:41:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:41:13.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028037+0000", 2024-04-22T18:41:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068544 2024-04-22T18:41:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028078+0000", 2024-04-22T18:41:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1523999999999997e-05 2024-04-22T18:41:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028087+0000", 2024-04-22T18:41:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3140000000000004e-06 2024-04-22T18:41:13.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.129951+0000", 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1018649229999999 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.130067+0000", 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115222 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.133925+0000", 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00385849 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.499 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4183 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.3981300839999999, 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1063017610000001, 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4183 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028049+0000", 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028055+0000", 2024-04-22T18:41:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-04-22T18:41:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028064+0000", 2024-04-22T18:41:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2830000000000001e-06 2024-04-22T18:41:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028071+0000", 2024-04-22T18:41:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2740000000000002e-06 2024-04-22T18:41:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028126+0000", 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4771999999999997e-05 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028134+0000", 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5599999999999994e-06 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134109+0000", 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.105974864 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134169+0000", 2024-04-22T18:41:13.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0167999999999997e-05 2024-04-22T18:41:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134352+0000", 2024-04-22T18:41:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018293699999999999 2024-04-22T18:41:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4190 4.c 4:3227431c:::smithi08937929-35:head [sparse-read 0~1258562,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.3981045810000001, 2024-04-22T18:41:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1098740030000001, 2024-04-22T18:41:13.504 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4190 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:13.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028075+0000", 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028080+0000", 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.561e-06 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028085+0000", 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0150000000000003e-06 2024-04-22T18:41:13.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028090+0000", 2024-04-22T18:41:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.169e-06 2024-04-22T18:41:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028175+0000", 2024-04-22T18:41:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4814999999999993e-05 2024-04-22T18:41:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028181+0000", 2024-04-22T18:41:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5490000000000003e-06 2024-04-22T18:41:13.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134400+0000", 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1062186869999999 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134552+0000", 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015210500000000001 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137950+0000", 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033978210000000001 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.508 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4191 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.3980218329999996, 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.110917953, 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4191 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028157+0000", 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028164+0000", 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.511e-06 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028172+0000", 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0339999999999997e-06 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028179+0000", 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8630000000000002e-06 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028202+0000", 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3635999999999999e-05 2024-04-22T18:41:13.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028209+0000", 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4629999999999998e-06 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137970+0000", 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1097613479999999 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.138034+0000", 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3551000000000002e-05 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.139077+0000", 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001042715 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4192 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.3979965330000006, 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.212555542, 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4192 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028183+0000", 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028187+0000", 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6119999999999999e-06 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028192+0000", 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3489999999999999e-06 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028197+0000", 2024-04-22T18:41:13.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7489999999999998e-06 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028228+0000", 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1328999999999998e-05 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028234+0000", 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1129999999999997e-06 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239245+0000", 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.211011032 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239384+0000", 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013902000000000001 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.240739+0000", 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013550389999999999 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4255 4.9 4:905c3da2:::smithi08937929-39:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.705522+0000", 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.7206584249999999, 2024-04-22T18:41:13.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68002623299999998, 2024-04-22T18:41:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4255 2024-04-22T18:41:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705522+0000", 2024-04-22T18:41:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705522+0000", 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705520+0000", 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705527+0000", 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2740000000000004e-06 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705533+0000", 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9819999999999996e-06 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705537+0000", 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2869999999999998e-06 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705582+0000", 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5247000000000002e-05 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705734+0000", 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000152029 2024-04-22T18:41:13.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.385548+0000", 2024-04-22T18:41:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.67981406899999997 2024-04-22T18:41:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.521 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.521 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:13.522 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:13.523 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:13.524+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:13.525 INFO:tasks.rados.rados.0.smithi089.stdout:1784: expect (ObjNum 301 snap 0 seq_num 301) 2024-04-22T18:41:13.623 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:13.623+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:13.628 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:13.628 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:13.636 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:13.672 INFO:tasks.rados.rados.0.smithi089.stdout:1783: finishing set_chunk to oid 23 2024-04-22T18:41:13.673 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 161 (ObjNum 339 snap 0 seq_num 339) dirty exists 2024-04-22T18:41:13.673 INFO:tasks.rados.rados.0.smithi089.stdout:1790: read oid 5 snap -1 2024-04-22T18:41:13.673 INFO:tasks.rados.rados.0.smithi089.stdout:1790: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:41:13.673 INFO:tasks.rados.rados.0.smithi089.stdout:1780: done (10 left) 2024-04-22T18:41:13.673 INFO:tasks.rados.rados.0.smithi089.stdout:1781: done (9 left) 2024-04-22T18:41:13.673 INFO:tasks.rados.rados.0.smithi089.stdout:1782: done (8 left) 2024-04-22T18:41:13.673 INFO:tasks.rados.rados.0.smithi089.stdout:1783: done (7 left) 2024-04-22T18:41:13.673 INFO:tasks.rados.rados.0.smithi089.stdout:1784: done (6 left) 2024-04-22T18:41:13.673 INFO:tasks.rados.rados.0.smithi089.stdout:1786: done (5 left) 2024-04-22T18:41:13.673 INFO:tasks.rados.rados.0.smithi089.stdout:1791: write oid 32 current snap is 0 2024-04-22T18:41:13.673 INFO:tasks.rados.rados.0.smithi089.stdout:1789: expect (ObjNum 352 snap 0 seq_num 352) 2024-04-22T18:41:13.724 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:13.726+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:13.776 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:13.776 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:13.777 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:13.777 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:13.777 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:13.785 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:13.806 INFO:tasks.rados.rados.0.smithi089.stdout:1785: finishing write tid 1 to smithi08937929-43 2024-04-22T18:41:13.806 INFO:tasks.rados.rados.0.smithi089.stdout:1785: finishing write tid 2 to smithi08937929-43 2024-04-22T18:41:13.806 INFO:tasks.rados.rados.0.smithi089.stdout:1785: finishing write tid 3 to smithi08937929-43 2024-04-22T18:41:13.806 INFO:tasks.rados.rados.0.smithi089.stdout:1785: finishing write tid 4 to smithi08937929-43 2024-04-22T18:41:13.806 INFO:tasks.rados.rados.0.smithi089.stdout:1785: oid 43 updating version 0 to 71 2024-04-22T18:41:13.806 INFO:tasks.rados.rados.0.smithi089.stdout:1785: oid 43 updating version 71 to 72 2024-04-22T18:41:13.806 INFO:tasks.rados.rados.0.smithi089.stdout:1785: oid 43 updating version 72 to 73 2024-04-22T18:41:13.806 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 43 v 73 (ObjNum 370 snap 0 seq_num 370) dirty exists 2024-04-22T18:41:13.807 INFO:tasks.rados.rados.0.smithi089.stdout:1785: left oid 43 (ObjNum 370 snap 0 seq_num 370) 2024-04-22T18:41:13.807 INFO:tasks.rados.rados.0.smithi089.stdout:1787: finishing set_chunk to oid 18 2024-04-22T18:41:13.807 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 81 (ObjNum 307 snap 0 seq_num 307) dirty exists 2024-04-22T18:41:13.807 INFO:tasks.rados.rados.0.smithi089.stdout:1791: seq_num 371 ranges {635397=702323,1893643=341754} 2024-04-22T18:41:13.813 INFO:tasks.rados.rados.0.smithi089.stdout:1791: writing smithi08937929-32 from 635397 to 1337720 tid 1 2024-04-22T18:41:13.818 INFO:tasks.rados.rados.0.smithi089.stdout:1791: writing smithi08937929-32 from 1893643 to 2235397 tid 2 2024-04-22T18:41:13.819 INFO:tasks.rados.rados.0.smithi089.stdout:1785: done (5 left) 2024-04-22T18:41:13.819 INFO:tasks.rados.rados.0.smithi089.stdout:1787: done (4 left) 2024-04-22T18:41:13.819 INFO:tasks.rados.rados.0.smithi089.stdout:1788: done (3 left) 2024-04-22T18:41:13.819 INFO:tasks.rados.rados.0.smithi089.stdout:1789: done (2 left) 2024-04-22T18:41:13.819 INFO:tasks.rados.rados.0.smithi089.stdout:1792: write (excl) oid 16 current snap is 0 2024-04-22T18:41:13.819 INFO:tasks.rados.rados.0.smithi089.stdout:1792: seq_num 372 ranges {652204=777572,1952125=300079} 2024-04-22T18:41:13.825 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:13.826+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:13.826 INFO:tasks.rados.rados.0.smithi089.stdout:1792: writing smithi08937929-16 from 652204 to 1429776 tid 1 2024-04-22T18:41:13.831 INFO:tasks.rados.rados.0.smithi089.stdout:1792: writing smithi08937929-16 from 1952125 to 2252204 tid 2 2024-04-22T18:41:13.831 INFO:tasks.rados.rados.0.smithi089.stdout:1793: tier_promote oid 40 2024-04-22T18:41:13.831 INFO:tasks.rados.rados.0.smithi089.stdout:1790: expect (ObjNum 291 snap 0 seq_num 291) 2024-04-22T18:41:13.891 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 129.430612841, 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:41:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:41:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:41:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:41:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:41:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:41:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:41:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:41:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:41:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:41:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:41:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:41:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:41:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:41:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:41:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:41:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 125.553939019, 2024-04-22T18:41:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:41:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:41:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:41:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:41:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:41:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:41:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:41:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:41:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:41:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:41:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:41:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:41:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:41:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:41:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:41:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:41:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:41:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.349135496000002, 2024-04-22T18:41:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:41:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:41:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:41:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:41:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:41:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:41:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:41:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:41:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:41:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:41:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:41:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:41:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:41:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:41:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:41:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:41:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:41:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:161 3.0 e76/63 3:3638311b:::f282598a146af3da1c05c8882972fba387afd249b39f6eeb6e6052a5de37fd2d:head v 76'157, mlcod=76'154)", 2024-04-22T18:41:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:41:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 66.500416066, 2024-04-22T18:41:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104845172, 2024-04-22T18:41:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:41:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:41:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:41:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391440+0000", 2024-04-22T18:41:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2045999999999994e-05 2024-04-22T18:41:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391448+0000", 2024-04-22T18:41:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8189999999999992e-06 2024-04-22T18:41:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391453+0000", 2024-04-22T18:41:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8379999999999998e-06 2024-04-22T18:41:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391839+0000", 2024-04-22T18:41:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000385325 2024-04-22T18:41:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391854+0000", 2024-04-22T18:41:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5006e-05 2024-04-22T18:41:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496191+0000", 2024-04-22T18:41:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104336792 2024-04-22T18:41:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496214+0000", 2024-04-22T18:41:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3079999999999999e-05 2024-04-22T18:41:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:167 3.0 e76/63 3:384c1ce2:::81dc8c58ebe106ae6fa9feadc06291c645d9079ad84b78214d4e6440fbc5def8:head v 76'160, mlcod=76'154)", 2024-04-22T18:41:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:41:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 66.500129110000003, 2024-04-22T18:41:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10487859400000001, 2024-04-22T18:41:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:41:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:41:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:41:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391707+0000", 2024-04-22T18:41:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2343999999999999e-05 2024-04-22T18:41:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391714+0000", 2024-04-22T18:41:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.596e-06 2024-04-22T18:41:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391719+0000", 2024-04-22T18:41:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9130000000000002e-06 2024-04-22T18:41:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493154+0000", 2024-04-22T18:41:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101435256 2024-04-22T18:41:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493177+0000", 2024-04-22T18:41:13.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2796e-05 2024-04-22T18:41:13.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496509+0000", 2024-04-22T18:41:13.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033320139999999999 2024-04-22T18:41:13.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496534+0000", 2024-04-22T18:41:13.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5265999999999998e-05 2024-04-22T18:41:13.912 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.912 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.912 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.913 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3451 4.d 4:bee1184f:::smithi08937929-7:head [read 0~3613505,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:13.913 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:13.913 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.964605169000002, 2024-04-22T18:41:13.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20164568099999999, 2024-04-22T18:41:13.913 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.913 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.913 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.913 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.913 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.913 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3451 2024-04-22T18:41:13.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.913 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:13.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:13.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:13.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927183+0000", 2024-04-22T18:41:13.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1110000000000001e-06 2024-04-22T18:41:13.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927188+0000", 2024-04-22T18:41:13.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1569999999999999e-06 2024-04-22T18:41:13.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927192+0000", 2024-04-22T18:41:13.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4710000000000001e-06 2024-04-22T18:41:13.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027382+0000", 2024-04-22T18:41:13.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100189341 2024-04-22T18:41:13.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:41:13.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027387+0000", 2024-04-22T18:41:13.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8400000000000002e-06 2024-04-22T18:41:13.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127638+0000", 2024-04-22T18:41:13.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10025102499999999 2024-04-22T18:41:13.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127740+0000", 2024-04-22T18:41:13.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102749 2024-04-22T18:41:13.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.128825+0000", 2024-04-22T18:41:13.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001084593 2024-04-22T18:41:13.917 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.917 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.917 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.917 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:151 3.0 e114/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 114'213, mlcod=110'210)", 2024-04-22T18:41:13.917 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:13.918 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.374137086000001, 2024-04-22T18:41:13.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20074001299999999, 2024-04-22T18:41:13.918 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.918 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.918 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:13.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:13.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517640+0000", 2024-04-22T18:41:13.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999928 2024-04-22T18:41:13.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517652+0000", 2024-04-22T18:41:13.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1824000000000001e-05 2024-04-22T18:41:13.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517660+0000", 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1419999999999993e-06 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617729+0000", 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068723 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617770+0000", 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1391999999999999e-05 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617793+0000", 2024-04-22T18:41:13.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2807999999999999e-05 2024-04-22T18:41:13.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718361+0000", 2024-04-22T18:41:13.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100567473 2024-04-22T18:41:13.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718387+0000", 2024-04-22T18:41:13.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6826e-05 2024-04-22T18:41:13.921 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.921 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.921 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.922 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3862 4.d 4:bee1184f:::smithi08937929-7:head [write 3301257~278667] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:13.922 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:13.922 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.629234816, 2024-04-22T18:41:13.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301875958, 2024-04-22T18:41:13.922 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.922 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.922 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.922 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.922 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.922 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3862 2024-04-22T18:41:13.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.922 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:13.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:13.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262548+0000", 2024-04-22T18:41:13.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:13.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262622+0000", 2024-04-22T18:41:13.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3747999999999999e-05 2024-04-22T18:41:13.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262632+0000", 2024-04-22T18:41:13.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0767e-05 2024-04-22T18:41:13.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362705+0000", 2024-04-22T18:41:13.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073157 2024-04-22T18:41:13.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:41:13.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2846999999999997e-05 2024-04-22T18:41:13.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362853+0000", 2024-04-22T18:41:13.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4498000000000002e-05 2024-04-22T18:41:13.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:13.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362992+0000", 2024-04-22T18:41:13.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013879699999999999 2024-04-22T18:41:13.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:13.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.366534+0000", 2024-04-22T18:41:13.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035427969999999999 2024-04-22T18:41:13.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:13.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564353+0000", 2024-04-22T18:41:13.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.197818893 2024-04-22T18:41:13.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564385+0000", 2024-04-22T18:41:13.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1717000000000001e-05 2024-04-22T18:41:13.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564426+0000", 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0654000000000001e-05 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3863 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3579924] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.529067424000001, 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20206560900000001, 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.927 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.928 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.928 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.928 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3863 2024-04-22T18:41:13.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.928 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:13.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:13.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362715+0000", 2024-04-22T18:41:13.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:13.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362725+0000", 2024-04-22T18:41:13.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.006e-05 2024-04-22T18:41:13.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362732+0000", 2024-04-22T18:41:13.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6780000000000003e-06 2024-04-22T18:41:13.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362737+0000", 2024-04-22T18:41:13.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2089999999999996e-06 2024-04-22T18:41:13.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462875+0000", 2024-04-22T18:41:13.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013834100000001 2024-04-22T18:41:13.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462963+0000", 2024-04-22T18:41:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7792000000000002e-05 2024-04-22T18:41:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463127+0000", 2024-04-22T18:41:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000163688 2024-04-22T18:41:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463692+0000", 2024-04-22T18:41:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056466299999999997 2024-04-22T18:41:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564720+0000", 2024-04-22T18:41:13.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10102826199999999 2024-04-22T18:41:13.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564752+0000", 2024-04-22T18:41:13.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1936000000000001e-05 2024-04-22T18:41:13.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564783+0000", 2024-04-22T18:41:13.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0919e-05 2024-04-22T18:41:13.933 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.933 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.933 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.934 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3864 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:13.934 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:13.934 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.529034039999999, 2024-04-22T18:41:13.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20218423599999999, 2024-04-22T18:41:13.934 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.934 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.934 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.934 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.934 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.934 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3864 2024-04-22T18:41:13.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.934 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362753+0000", 2024-04-22T18:41:13.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6989999999999999e-06 2024-04-22T18:41:13.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:41:13.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:41:13.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462871+0000", 2024-04-22T18:41:13.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100113641 2024-04-22T18:41:13.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463385+0000", 2024-04-22T18:41:13.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051307100000000003 2024-04-22T18:41:13.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:13.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463457+0000", 2024-04-22T18:41:13.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2778e-05 2024-04-22T18:41:13.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564814+0000", 2024-04-22T18:41:13.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10135696299999999 2024-04-22T18:41:13.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564868+0000", 2024-04-22T18:41:13.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3872000000000002e-05 2024-04-22T18:41:13.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564935+0000", 2024-04-22T18:41:13.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6545999999999996e-05 2024-04-22T18:41:13.938 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.938 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.938 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.938 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4030 4.d 4:bee1184f:::smithi08937929-7:head [write 615222~472039] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:13.939 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:13.939 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.932585138, 2024-04-22T18:41:13.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109028217, 2024-04-22T18:41:13.939 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.939 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.939 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.939 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.939 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.939 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4030 2024-04-22T18:41:13.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.939 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:13.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:13.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959177+0000", 2024-04-22T18:41:13.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999776 2024-04-22T18:41:13.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959331+0000", 2024-04-22T18:41:13.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015334600000000001 2024-04-22T18:41:13.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959358+0000", 2024-04-22T18:41:13.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7220999999999999e-05 2024-04-22T18:41:13.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959371+0000", 2024-04-22T18:41:13.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3489000000000001e-05 2024-04-22T18:41:13.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059523+0000", 2024-04-22T18:41:13.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100152059 2024-04-22T18:41:13.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059588+0000", 2024-04-22T18:41:13.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4493999999999996e-05 2024-04-22T18:41:13.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:13.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059694+0000", 2024-04-22T18:41:13.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010584199999999999 2024-04-22T18:41:13.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:13.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068082+0000", 2024-04-22T18:41:13.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083880589999999998 2024-04-22T18:41:13.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:13.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068138+0000", 2024-04-22T18:41:13.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6196999999999997e-05 2024-04-22T18:41:13.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068185+0000", 2024-04-22T18:41:13.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7318000000000003e-05 2024-04-22T18:41:13.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068228+0000", 2024-04-22T18:41:13.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2376999999999998e-05 2024-04-22T18:41:13.944 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.944 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.944 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.944 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4031 4.d 4:bee1184f:::smithi08937929-7:head [write 1580652~489442] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:13.944 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:13.944 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.927284146000002, 2024-04-22T18:41:13.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20571598299999999, 2024-04-22T18:41:13.944 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.944 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.944 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.944 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.945 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.945 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4031 2024-04-22T18:41:13.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.945 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:13.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:13.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964480+0000", 2024-04-22T18:41:13.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999795 2024-04-22T18:41:13.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964628+0000", 2024-04-22T18:41:13.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148109 2024-04-22T18:41:13.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964651+0000", 2024-04-22T18:41:13.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3215999999999999e-05 2024-04-22T18:41:13.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964664+0000", 2024-04-22T18:41:13.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2673e-05 2024-04-22T18:41:13.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061135+0000", 2024-04-22T18:41:13.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096470822999999997 2024-04-22T18:41:13.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061228+0000", 2024-04-22T18:41:13.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3459000000000001e-05 2024-04-22T18:41:13.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:13.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061376+0000", 2024-04-22T18:41:13.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148128 2024-04-22T18:41:13.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:13.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068085+0000", 2024-04-22T18:41:13.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.006708632 2024-04-22T18:41:13.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:13.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170038+0000", 2024-04-22T18:41:13.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101952716 2024-04-22T18:41:13.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170173+0000", 2024-04-22T18:41:13.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013508699999999999 2024-04-22T18:41:13.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170216+0000", 2024-04-22T18:41:13.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3498999999999998e-05 2024-04-22T18:41:13.949 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.949 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.950 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.950 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4032 4.d 4:bee1184f:::smithi08937929-7:head [write 2768826~626587] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:13.950 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:13.950 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.920079039000001, 2024-04-22T18:41:13.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19941629799999999, 2024-04-22T18:41:13.950 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.950 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.950 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.950 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.950 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.950 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4032 2024-04-22T18:41:13.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.951 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:13.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:13.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971703+0000", 2024-04-22T18:41:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971996+0000", 2024-04-22T18:41:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029351300000000002 2024-04-22T18:41:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972012+0000", 2024-04-22T18:41:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5569e-05 2024-04-22T18:41:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972019+0000", 2024-04-22T18:41:13.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7950000000000008e-06 2024-04-22T18:41:13.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.063940+0000", 2024-04-22T18:41:13.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091920162 2024-04-22T18:41:13.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064021+0000", 2024-04-22T18:41:13.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1865000000000003e-05 2024-04-22T18:41:13.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:13.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064145+0000", 2024-04-22T18:41:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123893 2024-04-22T18:41:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068317+0000", 2024-04-22T18:41:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041720799999999999 2024-04-22T18:41:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171042+0000", 2024-04-22T18:41:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102724893 2024-04-22T18:41:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171085+0000", 2024-04-22T18:41:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2619e-05 2024-04-22T18:41:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171122+0000", 2024-04-22T18:41:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6838999999999997e-05 2024-04-22T18:41:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4033 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3815222] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:13.955 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.919755556999998, 2024-04-22T18:41:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297586038, 2024-04-22T18:41:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4033 2024-04-22T18:41:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:13.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972028+0000", 2024-04-22T18:41:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972032+0000", 2024-04-22T18:41:13.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8330000000000004e-06 2024-04-22T18:41:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972035+0000", 2024-04-22T18:41:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:41:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972038+0000", 2024-04-22T18:41:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9579999999999999e-06 2024-04-22T18:41:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067511+0000", 2024-04-22T18:41:13.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.095472308000000006 2024-04-22T18:41:13.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067607+0000", 2024-04-22T18:41:13.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6546999999999996e-05 2024-04-22T18:41:13.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:13.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067776+0000", 2024-04-22T18:41:13.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000169202 2024-04-22T18:41:13.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:13.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068470+0000", 2024-04-22T18:41:13.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069403100000000001 2024-04-22T18:41:13.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:13.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269540+0000", 2024-04-22T18:41:13.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20106934000000001 2024-04-22T18:41:13.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269574+0000", 2024-04-22T18:41:13.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4332000000000003e-05 2024-04-22T18:41:13.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269615+0000", 2024-04-22T18:41:13.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0988000000000001e-05 2024-04-22T18:41:13.961 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.961 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.961 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.961 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4034 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:13.961 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:13.961 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.919743919999998, 2024-04-22T18:41:13.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39788621000000002, 2024-04-22T18:41:13.961 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.961 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.962 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.962 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.962 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.962 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4034 2024-04-22T18:41:13.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.962 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:13.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:13.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972040+0000", 2024-04-22T18:41:13.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972042+0000", 2024-04-22T18:41:13.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.519e-06 2024-04-22T18:41:13.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972045+0000", 2024-04-22T18:41:13.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:41:13.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972048+0000", 2024-04-22T18:41:13.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.773e-06 2024-04-22T18:41:13.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068266+0000", 2024-04-22T18:41:13.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096218029999999996 2024-04-22T18:41:13.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:13.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068308+0000", 2024-04-22T18:41:13.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.21e-05 2024-04-22T18:41:13.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369703+0000", 2024-04-22T18:41:13.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301395578 2024-04-22T18:41:13.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369802+0000", 2024-04-22T18:41:13.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8788000000000004e-05 2024-04-22T18:41:13.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369927+0000", 2024-04-22T18:41:13.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012478200000000001 2024-04-22T18:41:13.966 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.966 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.966 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.966 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(4.1 124/123 [PushOp(4:83c955bf:::smithi08937929-28:head, version: 115'157, data_included: [1890685~548185,2699879~469889], data_size: 1018074, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:83c955bf:::smithi08937929-28:head@115'157, size: 3630345, copy_subset: [0~3630345], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3630345, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1890685, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-22T18:41:13.966 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:13.966 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 15.253175812, 2024-04-22T18:41:13.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20293218800000001, 2024-04-22T18:41:13.966 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.966 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.966 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:13.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:13.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638608+0000", 2024-04-22T18:41:13.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:13.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639519+0000", 2024-04-22T18:41:13.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091196199999999999 2024-04-22T18:41:13.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639537+0000", 2024-04-22T18:41:13.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7455000000000001e-05 2024-04-22T18:41:13.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.739639+0000", 2024-04-22T18:41:13.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100101942 2024-04-22T18:41:13.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840271+0000", 2024-04-22T18:41:13.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10063235299999999 2024-04-22T18:41:13.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840281+0000", 2024-04-22T18:41:13.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6509999999999996e-06 2024-04-22T18:41:13.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.841541+0000", 2024-04-22T18:41:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012600230000000001 2024-04-22T18:41:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4205 4.1 4:83c955bf:::smithi08937929-28:head [read 3065344~859648] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.9642539709999998, 2024-04-22T18:41:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.303433565, 2024-04-22T18:41:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4205 2024-04-22T18:41:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927528+0000", 2024-04-22T18:41:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927535+0000", 2024-04-22T18:41:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1049999999999997e-06 2024-04-22T18:41:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927550+0000", 2024-04-22T18:41:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4365000000000001e-05 2024-04-22T18:41:13.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927560+0000", 2024-04-22T18:41:13.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0360000000000001e-05 2024-04-22T18:41:13.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028542+0000", 2024-04-22T18:41:13.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100981837 2024-04-22T18:41:13.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:13.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028551+0000", 2024-04-22T18:41:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6260000000000001e-06 2024-04-22T18:41:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229700+0000", 2024-04-22T18:41:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.201149453 2024-04-22T18:41:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229861+0000", 2024-04-22T18:41:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160605 2024-04-22T18:41:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.230964+0000", 2024-04-22T18:41:13.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00110351 2024-04-22T18:41:13.975 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.975 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.975 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.975 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4258 4.1 4:81200c09:::smithi08937929-23:head [read 0~3697573,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:13.975 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.875931+0000", 2024-04-22T18:41:13.975 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.0158536839999996, 2024-04-22T18:41:13.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105556871, 2024-04-22T18:41:13.975 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.975 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:13.975 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:13.976 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:13.976 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:13.976 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4258 2024-04-22T18:41:13.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.976 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.875931+0000", 2024-04-22T18:41:13.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.875931+0000", 2024-04-22T18:41:13.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.875930+0000", 2024-04-22T18:41:13.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:13.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.875936+0000", 2024-04-22T18:41:13.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0290000000000002e-06 2024-04-22T18:41:13.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.875944+0000", 2024-04-22T18:41:13.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8609999999999998e-06 2024-04-22T18:41:13.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.875950+0000", 2024-04-22T18:41:13.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3670000000000002e-06 2024-04-22T18:41:13.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.980155+0000", 2024-04-22T18:41:13.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104204899 2024-04-22T18:41:13.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.980272+0000", 2024-04-22T18:41:13.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011677500000000001 2024-04-22T18:41:13.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.981488+0000", 2024-04-22T18:41:13.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001215566 2024-04-22T18:41:13.979 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.979 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.979 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.979 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:359 3.0 e133/63 3:351d2474:::ba270d71d9c7ccc8d5c74eded70a7073f0e08c1ce33f7199f7c9db1158888734:head v 133'233, mlcod=132'228)", 2024-04-22T18:41:13.979 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:13.979 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.9894595439999998, 2024-04-22T18:41:13.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20122315599999999, 2024-04-22T18:41:13.980 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.980 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.980 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:13.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:13.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902324+0000", 2024-04-22T18:41:13.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:13.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902347+0000", 2024-04-22T18:41:13.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3577000000000001e-05 2024-04-22T18:41:13.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902356+0000", 2024-04-22T18:41:13.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8179999999999993e-06 2024-04-22T18:41:13.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.002447+0000", 2024-04-22T18:41:13.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100090255 2024-04-22T18:41:13.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102568+0000", 2024-04-22T18:41:13.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100121611 2024-04-22T18:41:13.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102592+0000", 2024-04-22T18:41:13.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3515000000000001e-05 2024-04-22T18:41:13.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103514+0000", 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000922174 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103548+0000", 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4316999999999997e-05 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:4289 3.0 e134/63 3:2a3d8f37:::9341e24c4a92f7ec719282d2faa5711f9644dee45a94ba5b4c4c1071ddc0e536:head v 134'235, mlcod=133'233)", 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:13.983 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.0448900319999996, 2024-04-22T18:41:13.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10669205499999999, 2024-04-22T18:41:13.984 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:13.984 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:13.984 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:13.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:13.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:13.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:13.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:13.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:13.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:13.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846891+0000", 2024-04-22T18:41:13.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:13.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:13.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847884+0000", 2024-04-22T18:41:13.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00099224800000000004 2024-04-22T18:41:13.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:13.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847900+0000", 2024-04-22T18:41:13.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6154000000000002e-05 2024-04-22T18:41:13.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:13.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.947999+0000", 2024-04-22T18:41:13.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009889199999999 2024-04-22T18:41:13.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:13.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948046+0000", 2024-04-22T18:41:13.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7617000000000002e-05 2024-04-22T18:41:13.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:13.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948072+0000", 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5559999999999999e-05 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953539+0000", 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054668599999999996 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953587+0000", 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7781999999999998e-05 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:13.987 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:13.988 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:13.988 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 7, 2, 4] out_osds: [5, 0, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:41:13.988 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:41:13.988 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-04-22T18:41:13.988 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-04-22T18:41:13.989 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:41:13.990 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:13.990+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:13.990 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:13.991 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:41:14.072 INFO:tasks.rados.rados.0.smithi089.stdout:1791: finishing write tid 1 to smithi08937929-32 2024-04-22T18:41:14.072 INFO:tasks.rados.rados.0.smithi089.stdout:1792: finishing write tid 1 to smithi08937929-16 2024-04-22T18:41:14.072 INFO:tasks.rados.rados.0.smithi089.stdout:1792: finishing write tid 2 to smithi08937929-16 2024-04-22T18:41:14.072 INFO:tasks.rados.rados.0.smithi089.stdout:1790: done (3 left) 2024-04-22T18:41:14.072 INFO:tasks.rados.rados.0.smithi089.stdout:1794: read oid 42 snap -1 2024-04-22T18:41:14.072 INFO:tasks.rados.rados.0.smithi089.stdout:1794: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:41:14.072 INFO:tasks.rados.rados.0.smithi089.stdout:1795: set_chunk oid 10 target oid 2024-04-22T18:41:14.112 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:14.113+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:14.139 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:14.139 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:14.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.139 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPushReply(3.1 138/130 [PushReplyOp(3:a0fb5fab:::9e0118a6710d4e50d3f23214dfcbdca1fc365927055527e8d4871d8b493dda17:head)])", 2024-04-22T18:41:14.139 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:14.130219+0000", 2024-04-22T18:41:14.139 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.010751649, 2024-04-22T18:41:14.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.010771819, 2024-04-22T18:41:14.140 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.140 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:41:14.140 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:14.130219+0000", 2024-04-22T18:41:14.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:14.130219+0000", 2024-04-22T18:41:14.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:14.130219+0000", 2024-04-22T18:41:14.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:14.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:14.130225+0000", 2024-04-22T18:41:14.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0809999999999999e-06 2024-04-22T18:41:14.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:14.130234+0000", 2024-04-22T18:41:14.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6290000000000005e-06 2024-04-22T18:41:14.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:14.130244+0000", 2024-04-22T18:41:14.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6560000000000004e-06 2024-04-22T18:41:14.142 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.142 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.142 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.142 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.142 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:41:14.142 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 1 2024-04-22T18:41:14.142 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:14.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:14.154 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:14.155+0000 7f3cfee88640 -1 osd.6 137 handle_osd_map injecting map crc failure 2024-04-22T18:41:14.193 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:14.193+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:14.291 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:14.291+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:14.295 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:14.296 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:14.296 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:14.296 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:14.296 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:14.305 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:14.333 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:41:14.334 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:41:14.345 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-04-22T18:41:13.965718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":279283263,"num_objects":206,"num_object_clones":0,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":76,"num_objects_unfound":0,"num_objects_dirty":202,"num_whiteouts":0,"num_read":4364,"num_read_kb":3864401,"num_write":2546,"num_write_kb":623232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":854,"num_bytes_recovered":315567110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":32,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3555,"ondisk_log_size":3555,"up":98,"acting":98,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":91,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":529828,"kb_used_data":370032,"kb_used_omap":7,"kb_used_meta":159608,"kb_avail":471329372,"statfs":{"total":483183820800,"available":482641276928,"internally_reserved":0,"allocated":378912768,"data_stored":386126398,"data_compressed":11406784,"data_compressed_allocated":12193792,"data_compressed_original":24907776,"omap_allocated":7963,"internal_metadata":163438821},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6214897,"num_objects":-7,"num_object_clones":0,"num_object_copies":-14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-52,"num_objects_unfound":0,"num_objects_dirty":-7,"num_whiteouts":0,"num_read":124,"num_read_kb":113402,"num_write":79,"num_write_kb":15254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":22393503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.978087"},"pg_stats":[{"pgid":"4.23","version":"134'89","reported_seq":351,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.257392+0000","last_change":"2024-04-22T18:41:11.257031+0000","last_active":"2024-04-22T18:41:11.257392+0000","last_peered":"2024-04-22T18:41:11.257392+0000","last_clean":"2024-04-22T18:41:11.257392+0000","last_became_active":"2024-04-22T18:41:11.256832+0000","last_became_peered":"2024-04-22T18:41:11.256832+0000","last_unstale":"2024-04-22T18:41:11.257392+0000","last_undegraded":"2024-04-22T18:41:11.257392+0000","last_fullsized":"2024-04-22T18:41:11.257392+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'67","last_scrub_stamp":"2024-04-22T18:39:53.916646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.916646+0000","objects_scrubbed":2,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655556,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":50731,"num_write":11,"num_write_kb":2755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3906036,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.22","version":"64'38","reported_seq":285,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.158355+0000","last_change":"2024-04-22T18:41:11.156704+0000","last_active":"2024-04-22T18:41:11.158355+0000","last_peered":"2024-04-22T18:41:11.158355+0000","last_clean":"2024-04-22T18:41:11.158355+0000","last_became_active":"2024-04-22T18:41:11.069578+0000","last_became_peered":"2024-04-22T18:41:11.069578+0000","last_unstale":"2024-04-22T18:41:11.158355+0000","last_undegraded":"2024-04-22T18:41:11.158355+0000","last_fullsized":"2024-04-22T18:41:11.158355+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1466411,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":57,"num_read_kb":82765,"num_write":10,"num_write_kb":2798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5687418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"131'166","reported_seq":600,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.255749+0000","last_change":"2024-04-22T18:41:11.255428+0000","last_active":"2024-04-22T18:41:11.255749+0000","last_peered":"2024-04-22T18:41:11.255749+0000","last_clean":"2024-04-22T18:41:11.255749+0000","last_became_active":"2024-04-22T18:41:11.255094+0000","last_became_peered":"2024-04-22T18:41:11.255094+0000","last_unstale":"2024-04-22T18:41:11.255749+0000","last_undegraded":"2024-04-22T18:41:11.255749+0000","last_fullsized":"2024-04-22T18:41:11.255749+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":6,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6865709,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":175134,"num_write":31,"num_write_kb":9008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":17092321,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"124'24","reported_seq":215,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.057953+0000","last_change":"2024-04-22T18:41:11.057465+0000","last_active":"2024-04-22T18:41:11.057953+0000","last_peered":"2024-04-22T18:41:11.057953+0000","last_clean":"2024-04-22T18:41:11.057953+0000","last_became_active":"2024-04-22T18:41:11.057202+0000","last_became_peered":"2024-04-22T18:41:11.057202+0000","last_unstale":"2024-04-22T18:41:11.057953+0000","last_undegraded":"2024-04-22T18:41:11.057953+0000","last_fullsized":"2024-04-22T18:41:11.057953+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'13","last_scrub_stamp":"2024-04-22T18:39:54.377980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.377980+0000","objects_scrubbed":1,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30260881099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1958032,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":54162,"num_write":6,"num_write_kb":1615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"131'75","reported_seq":315,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.172201+0000","last_change":"2024-04-22T18:41:11.171942+0000","last_active":"2024-04-22T18:41:11.172201+0000","last_peered":"2024-04-22T18:41:11.172201+0000","last_clean":"2024-04-22T18:41:11.172201+0000","last_became_active":"2024-04-22T18:41:11.171671+0000","last_became_peered":"2024-04-22T18:41:11.171671+0000","last_unstale":"2024-04-22T18:41:11.172201+0000","last_undegraded":"2024-04-22T18:41:11.172201+0000","last_fullsized":"2024-04-22T18:41:11.172201+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2554697,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":127,"num_read_kb":124135,"num_write":38,"num_write_kb":9856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"132'50","reported_seq":39,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.254529+0000","last_change":"2024-04-22T18:41:11.253830+0000","last_active":"2024-04-22T18:41:11.254529+0000","last_peered":"2024-04-22T18:41:11.254529+0000","last_clean":"2024-04-22T18:41:11.254529+0000","last_became_active":"2024-04-22T18:41:11.253471+0000","last_became_peered":"2024-04-22T18:41:11.253471+0000","last_unstale":"2024-04-22T18:41:11.254529+0000","last_undegraded":"2024-04-22T18:41:11.254529+0000","last_fullsized":"2024-04-22T18:41:11.254529+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'23","last_scrub_stamp":"2024-04-22T18:40:20.587606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.587606+0000","objects_scrubbed":1,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10225366599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949678,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":73,"num_read_kb":77102,"num_write":27,"num_write_kb":8589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5574713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"133'66","reported_seq":33,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.355744+0000","last_change":"2024-04-22T18:41:11.355276+0000","last_active":"2024-04-22T18:41:11.355744+0000","last_peered":"2024-04-22T18:41:11.355744+0000","last_clean":"2024-04-22T18:41:11.355744+0000","last_became_active":"2024-04-22T18:41:11.354958+0000","last_became_peered":"2024-04-22T18:41:11.354958+0000","last_unstale":"2024-04-22T18:41:11.355744+0000","last_undegraded":"2024-04-22T18:41:11.355744+0000","last_fullsized":"2024-04-22T18:41:11.355744+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"96'46","last_scrub_stamp":"2024-04-22T18:40:34.755106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:34.755106+0000","objects_scrubbed":2,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40192567099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3655556,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":97,"num_read_kb":101756,"num_write":35,"num_write_kb":10294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8192627,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"132'123","reported_seq":456,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.469917+0000","last_change":"2024-04-22T18:41:11.469596+0000","last_active":"2024-04-22T18:41:11.469917+0000","last_peered":"2024-04-22T18:41:11.469917+0000","last_clean":"2024-04-22T18:41:11.469917+0000","last_became_active":"2024-04-22T18:41:11.469282+0000","last_became_peered":"2024-04-22T18:41:11.469282+0000","last_unstale":"2024-04-22T18:41:11.469917+0000","last_undegraded":"2024-04-22T18:41:11.469917+0000","last_fullsized":"2024-04-22T18:41:11.469917+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"121'119","last_scrub_stamp":"2024-04-22T18:40:55.573199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.573199+0000","objects_scrubbed":4,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32976813900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4521133,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":219,"num_read_kb":257792,"num_write":64,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3786236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"115'138","reported_seq":533,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.554963+0000","last_change":"2024-04-22T18:41:11.457927+0000","last_active":"2024-04-22T18:41:11.554963+0000","last_peered":"2024-04-22T18:41:11.554963+0000","last_clean":"2024-04-22T18:41:11.554963+0000","last_became_active":"2024-04-22T18:41:11.457457+0000","last_became_peered":"2024-04-22T18:41:11.457457+0000","last_unstale":"2024-04-22T18:41:11.554963+0000","last_undegraded":"2024-04-22T18:41:11.554963+0000","last_fullsized":"2024-04-22T18:41:11.554963+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"85'109","last_scrub_stamp":"2024-04-22T18:40:17.658363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:17.658363+0000","objects_scrubbed":5,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202176033,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7748093,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":260,"num_read_kb":295919,"num_write":73,"num_write_kb":21476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9711724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"134'111","reported_seq":400,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:10.956672+0000","last_change":"2024-04-22T18:41:10.956373+0000","last_active":"2024-04-22T18:41:10.956672+0000","last_peered":"2024-04-22T18:41:10.956672+0000","last_clean":"2024-04-22T18:41:10.956672+0000","last_became_active":"2024-04-22T18:41:10.956174+0000","last_became_peered":"2024-04-22T18:41:10.956174+0000","last_unstale":"2024-04-22T18:41:10.956672+0000","last_undegraded":"2024-04-22T18:41:10.956672+0000","last_fullsized":"2024-04-22T18:41:10.956672+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840499,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":181703,"num_write":56,"num_write_kb":14924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4554596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"105'114","reported_seq":505,"reported_epoch":135,"state":"active+remapped+backfill_wait","last_fresh":"2024-04-22T18:41:10.047877+0000","last_change":"2024-04-22T18:41:05.438520+0000","last_active":"2024-04-22T18:41:10.047877+0000","last_peered":"2024-04-22T18:41:10.047877+0000","last_clean":"2024-04-22T18:41:01.651846+0000","last_became_active":"2024-04-22T18:41:05.337692+0000","last_became_peered":"2024-04-22T18:41:05.337692+0000","last_unstale":"2024-04-22T18:41:10.047877+0000","last_undegraded":"2024-04-22T18:41:10.047877+0000","last_fullsized":"2024-04-22T18:41:10.047877+0000","mapping_epoch":130,"log_start":"84'88","ondisk_log_start":"84'88","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":26,"log_dups_size":106,"ondisk_log_size":26,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6680865,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":93142,"num_write":19,"num_write_kb":5735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4990114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"122'64","reported_seq":336,"reported_epoch":135,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:41:10.125843+0000","last_change":"2024-04-22T18:41:05.348597+0000","last_active":"2024-04-22T18:41:10.125843+0000","last_peered":"2024-04-22T18:41:10.125843+0000","last_clean":"2024-04-22T18:41:01.854833+0000","last_became_active":"2024-04-22T18:41:05.127081+0000","last_became_peered":"2024-04-22T18:41:05.127081+0000","last_unstale":"2024-04-22T18:41:10.125843+0000","last_undegraded":"2024-04-22T18:41:10.125843+0000","last_fullsized":"2024-04-22T18:41:10.125843+0000","mapping_epoch":130,"log_start":"79'52","ondisk_log_start":"79'52","created":110,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":61,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5613571,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":119,"num_read_kb":168057,"num_write":55,"num_write_kb":15750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7945966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,3],"avail_no_missing":["7","3","6"],"object_location_counts":[{"shards":"3,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"107'52","reported_seq":343,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:09.954192+0000","last_change":"2024-04-22T18:41:00.078939+0000","last_active":"2024-04-22T18:41:09.954192+0000","last_peered":"2024-04-22T18:41:09.954192+0000","last_clean":"2024-04-22T18:41:09.954192+0000","last_became_active":"2024-04-22T18:41:00.078669+0000","last_became_peered":"2024-04-22T18:41:00.078669+0000","last_unstale":"2024-04-22T18:41:09.954192+0000","last_undegraded":"2024-04-22T18:41:09.954192+0000","last_fullsized":"2024-04-22T18:41:09.954192+0000","mapping_epoch":125,"log_start":"83'49","ondisk_log_start":"83'49","created":110,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":5,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092497,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":66769,"num_write":16,"num_write_kb":3993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2254766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"123'95","reported_seq":410,"reported_epoch":135,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:41:10.148256+0000","last_change":"2024-04-22T18:40:56.200294+0000","last_active":"2024-04-22T18:41:10.148256+0000","last_peered":"2024-04-22T18:41:10.148256+0000","last_clean":"2024-04-22T18:40:53.248441+0000","last_became_active":"2024-04-22T18:40:55.998994+0000","last_became_peered":"2024-04-22T18:40:55.998994+0000","last_unstale":"2024-04-22T18:41:10.148256+0000","last_undegraded":"2024-04-22T18:41:10.148256+0000","last_fullsized":"2024-04-22T18:41:10.148256+0000","mapping_epoch":121,"log_start":"96'71","ondisk_log_start":"96'71","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":2,"log_size":24,"log_dups_size":81,"ondisk_log_size":24,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.22461577699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5428711,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":73379,"num_write":27,"num_write_kb":8334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1847758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.15","version":"54'29","reported_seq":77,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:10.145818+0000","last_change":"2024-04-22T18:40:55.328642+0000","last_active":"2024-04-22T18:41:10.145818+0000","last_peered":"2024-04-22T18:41:10.145818+0000","last_clean":"2024-04-22T18:41:10.145818+0000","last_became_active":"2024-04-22T18:40:13.180144+0000","last_became_peered":"2024-04-22T18:40:13.180144+0000","last_unstale":"2024-04-22T18:41:10.145818+0000","last_undegraded":"2024-04-22T18:41:10.145818+0000","last_fullsized":"2024-04-22T18:41:10.145818+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'29","last_scrub_stamp":"2024-04-22T18:40:55.328529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.328529+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:10.967288+0000","scrub_duration":0.30217712000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"51'19","reported_seq":77,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:09.954269+0000","last_change":"2024-04-22T18:40:57.754351+0000","last_active":"2024-04-22T18:41:09.954269+0000","last_peered":"2024-04-22T18:41:09.954269+0000","last_clean":"2024-04-22T18:41:09.954269+0000","last_became_active":"2024-04-22T18:40:10.978840+0000","last_became_peered":"2024-04-22T18:40:10.978840+0000","last_unstale":"2024-04-22T18:41:09.954269+0000","last_undegraded":"2024-04-22T18:41:09.954269+0000","last_fullsized":"2024-04-22T18:41:09.954269+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"51'19","last_scrub_stamp":"2024-04-22T18:40:57.754261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:57.754261+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:17.927283+0000","scrub_duration":0.59184710600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"131'83","reported_seq":27,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:10.046025+0000","last_change":"2024-04-22T18:41:04.562945+0000","last_active":"2024-04-22T18:41:10.046025+0000","last_peered":"2024-04-22T18:41:10.046025+0000","last_clean":"2024-04-22T18:41:10.046025+0000","last_became_active":"2024-04-22T18:41:04.139159+0000","last_became_peered":"2024-04-22T18:41:04.139159+0000","last_unstale":"2024-04-22T18:41:10.046025+0000","last_undegraded":"2024-04-22T18:41:10.046025+0000","last_fullsized":"2024-04-22T18:41:10.046025+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'75","last_scrub_stamp":"2024-04-22T18:40:22.670296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.670296+0000","objects_scrubbed":2,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:49.126896+0000","scrub_duration":0.20951747300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218623,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":37607,"num_write":6,"num_write_kb":3167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12459808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"126'45","reported_seq":259,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:09.947723+0000","last_change":"2024-04-22T18:40:59.453830+0000","last_active":"2024-04-22T18:41:09.947723+0000","last_peered":"2024-04-22T18:41:09.947723+0000","last_clean":"2024-04-22T18:41:09.947723+0000","last_became_active":"2024-04-22T18:40:58.107873+0000","last_became_peered":"2024-04-22T18:40:58.107873+0000","last_unstale":"2024-04-22T18:41:09.947723+0000","last_undegraded":"2024-04-22T18:41:09.947723+0000","last_fullsized":"2024-04-22T18:41:09.947723+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4008503,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":111,"num_read_kb":105496,"num_write":26,"num_write_kb":7134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2641926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"132'140","reported_seq":445,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:10.155427+0000","last_change":"2024-04-22T18:40:26.844287+0000","last_active":"2024-04-22T18:41:10.155427+0000","last_peered":"2024-04-22T18:41:10.155427+0000","last_clean":"2024-04-22T18:41:10.155427+0000","last_became_active":"2024-04-22T18:40:26.742351+0000","last_became_peered":"2024-04-22T18:40:26.742351+0000","last_unstale":"2024-04-22T18:41:10.155427+0000","last_undegraded":"2024-04-22T18:41:10.155427+0000","last_fullsized":"2024-04-22T18:41:10.155427+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'113","last_scrub_stamp":"2024-04-22T18:40:20.291988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.291988+0000","objects_scrubbed":2,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:22.684852+0000","scrub_duration":0.10270805500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5580023,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":22757,"num_write":31,"num_write_kb":8732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3940523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"31'5","reported_seq":169,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:10.054579+0000","last_change":"2024-04-22T18:40:10.212897+0000","last_active":"2024-04-22T18:41:10.054579+0000","last_peered":"2024-04-22T18:41:10.054579+0000","last_clean":"2024-04-22T18:41:10.054579+0000","last_became_active":"2024-04-22T18:39:59.265189+0000","last_became_peered":"2024-04-22T18:39:59.265189+0000","last_unstale":"2024-04-22T18:41:10.054579+0000","last_undegraded":"2024-04-22T18:41:10.054579+0000","last_fullsized":"2024-04-22T18:41:10.054579+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":5,"last_scrub":"31'5","last_scrub_stamp":"2024-04-22T18:40:10.212815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.212815+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:17.620115+0000","scrub_duration":0.0015068670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:10.045687+0000","last_change":"2024-04-22T18:40:58.207745+0000","last_active":"2024-04-22T18:41:10.045687+0000","last_peered":"2024-04-22T18:41:10.045687+0000","last_clean":"2024-04-22T18:41:10.045687+0000","last_became_active":"2024-04-22T18:40:58.207451+0000","last_became_peered":"2024-04-22T18:40:58.207451+0000","last_unstale":"2024-04-22T18:41:10.045687+0000","last_undegraded":"2024-04-22T18:41:10.045687+0000","last_fullsized":"2024-04-22T18:41:10.045687+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:19.136933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:19.136933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:38.724946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"131'166","reported_seq":593,"reported_epoch":136,"state":"peering","last_fresh":"2024-04-22T18:41:11.254426+0000","last_change":"2024-04-22T18:41:11.254426+0000","last_active":"2024-04-22T18:41:09.957882+0000","last_peered":"2024-04-22T18:41:09.595896+0000","last_clean":"2024-04-22T18:41:02.169513+0000","last_became_active":"2024-04-22T18:41:05.229296+0000","last_became_peered":"2024-04-22T18:41:05.229296+0000","last_unstale":"2024-04-22T18:41:11.254426+0000","last_undegraded":"2024-04-22T18:41:11.254426+0000","last_fullsized":"2024-04-22T18:41:11.254426+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6865709,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":145,"num_read_kb":175133,"num_write":30,"num_write_kb":9008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":17092320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":145,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:10.057491+0000","last_change":"2024-04-22T18:41:04.037422+0000","last_active":"2024-04-22T18:41:10.057491+0000","last_peered":"2024-04-22T18:41:10.057491+0000","last_clean":"2024-04-22T18:41:10.057491+0000","last_became_active":"2024-04-22T18:41:04.037248+0000","last_became_peered":"2024-04-22T18:41:04.037248+0000","last_unstale":"2024-04-22T18:41:10.057491+0000","last_undegraded":"2024-04-22T18:41:10.057491+0000","last_fullsized":"2024-04-22T18:41:10.057491+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"124'24","reported_seq":216,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.057678+0000","last_change":"2024-04-22T18:41:11.057678+0000","last_active":"2024-04-22T18:41:11.057678+0000","last_peered":"2024-04-22T18:41:11.057678+0000","last_clean":"2024-04-22T18:41:11.057678+0000","last_became_active":"2024-04-22T18:41:11.057432+0000","last_became_peered":"2024-04-22T18:41:11.057432+0000","last_unstale":"2024-04-22T18:41:11.057678+0000","last_undegraded":"2024-04-22T18:41:11.057678+0000","last_fullsized":"2024-04-22T18:41:11.057678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'13","last_scrub_stamp":"2024-04-22T18:39:54.377980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.377980+0000","objects_scrubbed":1,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30260881099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1958032,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":54162,"num_write":5,"num_write_kb":1614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":154,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:10.045665+0000","last_change":"2024-04-22T18:41:04.428281+0000","last_active":"2024-04-22T18:41:10.045665+0000","last_peered":"2024-04-22T18:41:10.045665+0000","last_clean":"2024-04-22T18:41:10.045665+0000","last_became_active":"2024-04-22T18:41:04.427985+0000","last_became_peered":"2024-04-22T18:41:04.427985+0000","last_unstale":"2024-04-22T18:41:10.045665+0000","last_undegraded":"2024-04-22T18:41:10.045665+0000","last_fullsized":"2024-04-22T18:41:10.045665+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:09.538542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:09.538542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:36.605250+0000","scrub_duration":0.35201753200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"134'89","reported_seq":351,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.257020+0000","last_change":"2024-04-22T18:41:11.257020+0000","last_active":"2024-04-22T18:41:11.257020+0000","last_peered":"2024-04-22T18:41:11.257020+0000","last_clean":"2024-04-22T18:41:11.257020+0000","last_became_active":"2024-04-22T18:41:11.256784+0000","last_became_peered":"2024-04-22T18:41:11.256784+0000","last_unstale":"2024-04-22T18:41:11.257020+0000","last_undegraded":"2024-04-22T18:41:11.257020+0000","last_fullsized":"2024-04-22T18:41:11.257020+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'67","last_scrub_stamp":"2024-04-22T18:39:53.916646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.916646+0000","objects_scrubbed":2,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655555,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":50730,"num_write":10,"num_write_kb":2755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3906035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":137,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:09.949754+0000","last_change":"2024-04-22T18:40:35.014535+0000","last_active":"2024-04-22T18:41:09.949754+0000","last_peered":"2024-04-22T18:41:09.949754+0000","last_clean":"2024-04-22T18:41:09.949754+0000","last_became_active":"2024-04-22T18:40:35.014154+0000","last_became_peered":"2024-04-22T18:40:35.014154+0000","last_unstale":"2024-04-22T18:41:09.949754+0000","last_undegraded":"2024-04-22T18:41:09.949754+0000","last_fullsized":"2024-04-22T18:41:09.949754+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"64'38","reported_seq":286,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.069789+0000","last_change":"2024-04-22T18:41:11.069789+0000","last_active":"2024-04-22T18:41:11.069789+0000","last_peered":"2024-04-22T18:41:11.069789+0000","last_clean":"2024-04-22T18:41:11.069789+0000","last_became_active":"2024-04-22T18:41:11.069229+0000","last_became_peered":"2024-04-22T18:41:11.069229+0000","last_unstale":"2024-04-22T18:41:11.069789+0000","last_undegraded":"2024-04-22T18:41:11.069789+0000","last_fullsized":"2024-04-22T18:41:11.069789+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1466410,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":57,"num_read_kb":82764,"num_write":10,"num_write_kb":2797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5687418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"134'312","reported_seq":1074,"reported_epoch":136,"state":"active+recovering+remapped","last_fresh":"2024-04-22T18:41:11.666276+0000","last_change":"2024-04-22T18:41:05.443806+0000","last_active":"2024-04-22T18:41:11.666276+0000","last_peered":"2024-04-22T18:41:11.666276+0000","last_clean":"2024-04-22T18:40:48.123953+0000","last_became_active":"2024-04-22T18:41:05.133127+0000","last_became_peered":"2024-04-22T18:41:05.133127+0000","last_unstale":"2024-04-22T18:41:11.666276+0000","last_undegraded":"2024-04-22T18:41:11.666276+0000","last_fullsized":"2024-04-22T18:41:11.666276+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'293","last_scrub_stamp":"2024-04-22T18:40:53.582354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:53.582354+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:17.561029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31522221,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":264,"num_read_kb":4169,"num_write":376,"num_write_kb":93482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":49701665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[1,4],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,3","objects":2},{"shards":"1,4","objects":20},{"shards":"1,6","objects":13}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":98,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:09.949147+0000","last_change":"2024-04-22T18:40:57.953864+0000","last_active":"2024-04-22T18:41:09.949147+0000","last_peered":"2024-04-22T18:41:09.949147+0000","last_clean":"2024-04-22T18:41:09.949147+0000","last_became_active":"2024-04-22T18:40:57.853059+0000","last_became_peered":"2024-04-22T18:40:57.853059+0000","last_unstale":"2024-04-22T18:41:09.949147+0000","last_undegraded":"2024-04-22T18:41:09.949147+0000","last_fullsized":"2024-04-22T18:41:09.949147+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:55.558446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.558446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:05.813196+0000","scrub_duration":0.56126016300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"115'32","reported_seq":256,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:10.049776+0000","last_change":"2024-04-22T18:40:21.426510+0000","last_active":"2024-04-22T18:41:10.049776+0000","last_peered":"2024-04-22T18:41:10.049776+0000","last_clean":"2024-04-22T18:41:10.049776+0000","last_became_active":"2024-04-22T18:39:51.943071+0000","last_became_peered":"2024-04-22T18:39:51.943071+0000","last_unstale":"2024-04-22T18:41:10.049776+0000","last_undegraded":"2024-04-22T18:41:10.049776+0000","last_fullsized":"2024-04-22T18:41:10.049776+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-04-22T18:40:21.426350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.426350+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.678685+0000","scrub_duration":0.30334366400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2605118,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":78568,"num_write":4,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"134'235","reported_seq":594,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:09.953811+0000","last_change":"2024-04-22T18:40:20.465005+0000","last_active":"2024-04-22T18:41:09.953811+0000","last_peered":"2024-04-22T18:41:09.953811+0000","last_clean":"2024-04-22T18:41:09.953811+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:41:09.953811+0000","last_undegraded":"2024-04-22T18:41:09.953811+0000","last_fullsized":"2024-04-22T18:41:09.953811+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'176","last_scrub_stamp":"2024-04-22T18:40:20.464799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.464799+0000","objects_scrubbed":31,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:45.914514+0000","scrub_duration":0.30629975799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28937789,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":205,"num_read_kb":3225,"num_write":297,"num_write_kb":66969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":95,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:09.957472+0000","last_change":"2024-04-22T18:40:10.403338+0000","last_active":"2024-04-22T18:41:09.957472+0000","last_peered":"2024-04-22T18:41:09.957472+0000","last_clean":"2024-04-22T18:41:09.957472+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:41:09.957472+0000","last_undegraded":"2024-04-22T18:41:09.957472+0000","last_fullsized":"2024-04-22T18:41:09.957472+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:10.403189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.403189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:34.232014+0000","scrub_duration":0.201433005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"124'61","reported_seq":349,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:09.950207+0000","last_change":"2024-04-22T18:40:42.750352+0000","last_active":"2024-04-22T18:41:09.950207+0000","last_peered":"2024-04-22T18:41:09.950207+0000","last_clean":"2024-04-22T18:41:09.950207+0000","last_became_active":"2024-04-22T18:40:42.750084+0000","last_became_peered":"2024-04-22T18:40:42.750084+0000","last_unstale":"2024-04-22T18:41:09.950207+0000","last_undegraded":"2024-04-22T18:41:09.950207+0000","last_fullsized":"2024-04-22T18:41:09.950207+0000","mapping_epoch":106,"log_start":"83'49","ondisk_log_start":"83'49","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":49,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3747969,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":92629,"num_write":25,"num_write_kb":7639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2254765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"135'282","reported_seq":899,"reported_epoch":136,"state":"active+recovering+remapped","last_fresh":"2024-04-22T18:41:11.662488+0000","last_change":"2024-04-22T18:41:05.437289+0000","last_active":"2024-04-22T18:41:11.662488+0000","last_peered":"2024-04-22T18:41:11.662488+0000","last_clean":"2024-04-22T18:41:01.661225+0000","last_became_active":"2024-04-22T18:41:05.030149+0000","last_became_peered":"2024-04-22T18:41:05.030149+0000","last_unstale":"2024-04-22T18:41:11.662488+0000","last_undegraded":"2024-04-22T18:41:11.662488+0000","last_fullsized":"2024-04-22T18:41:11.662488+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'209","last_scrub_stamp":"2024-04-22T18:40:18.683442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:18.683442+0000","objects_scrubbed":24,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:45.661944+0000","scrub_duration":0.204344365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34525642,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":247,"num_read_kb":6478,"num_write":346,"num_write_kb":87619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":32687971,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[3,4],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,4","objects":20},{"shards":"3,7","objects":19}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":138,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:10.125950+0000","last_change":"2024-04-22T18:40:22.521828+0000","last_active":"2024-04-22T18:41:10.125950+0000","last_peered":"2024-04-22T18:41:10.125950+0000","last_clean":"2024-04-22T18:41:10.125950+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:41:10.125950+0000","last_undegraded":"2024-04-22T18:41:10.125950+0000","last_fullsized":"2024-04-22T18:41:10.125950+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.521638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:43.407910+0000","scrub_duration":0.30190717299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"128'93","reported_seq":391,"reported_epoch":135,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:41:10.057377+0000","last_change":"2024-04-22T18:40:43.158709+0000","last_active":"2024-04-22T18:41:10.057377+0000","last_peered":"2024-04-22T18:41:10.057377+0000","last_clean":"2024-04-22T18:40:32.014508+0000","last_became_active":"2024-04-22T18:40:43.046938+0000","last_became_peered":"2024-04-22T18:40:43.046938+0000","last_unstale":"2024-04-22T18:41:10.057377+0000","last_undegraded":"2024-04-22T18:41:10.057377+0000","last_fullsized":"2024-04-22T18:41:10.057377+0000","mapping_epoch":104,"log_start":"96'71","ondisk_log_start":"96'71","created":20,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":81,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.22461577699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":99605,"num_write":23,"num_write_kb":6431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,0],"avail_no_missing":["1","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"133'270","reported_seq":829,"reported_epoch":136,"state":"active+recovering+remapped","last_fresh":"2024-04-22T18:41:11.576369+0000","last_change":"2024-04-22T18:41:05.143862+0000","last_active":"2024-04-22T18:41:11.576369+0000","last_peered":"2024-04-22T18:41:11.576369+0000","last_clean":"2024-04-22T18:40:55.265085+0000","last_became_active":"2024-04-22T18:41:04.934615+0000","last_became_peered":"2024-04-22T18:41:04.934615+0000","last_unstale":"2024-04-22T18:41:11.576369+0000","last_undegraded":"2024-04-22T18:41:11.576369+0000","last_fullsized":"2024-04-22T18:41:11.576369+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'212","last_scrub_stamp":"2024-04-22T18:40:21.870855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.870855+0000","objects_scrubbed":46,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:31.749601+0000","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39888324,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":30,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":252,"num_read_kb":8424,"num_write":341,"num_write_kb":79122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":35140904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[3,4],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"1,3","objects":1},{"shards":"3,4","objects":18},{"shards":"3,6","objects":29}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":150,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:09.957276+0000","last_change":"2024-04-22T18:40:07.420765+0000","last_active":"2024-04-22T18:41:09.957276+0000","last_peered":"2024-04-22T18:41:09.957276+0000","last_clean":"2024-04-22T18:41:09.957276+0000","last_became_active":"2024-04-22T18:40:02.741862+0000","last_became_peered":"2024-04-22T18:40:02.741862+0000","last_unstale":"2024-04-22T18:41:09.957276+0000","last_undegraded":"2024-04-22T18:41:09.957276+0000","last_fullsized":"2024-04-22T18:41:09.957276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:07.420618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:07.420618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.642486+0000","scrub_duration":0.30164867299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":317,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:10.125983+0000","last_change":"2024-04-22T18:40:58.257511+0000","last_active":"2024-04-22T18:41:10.125983+0000","last_peered":"2024-04-22T18:41:10.125983+0000","last_clean":"2024-04-22T18:41:10.125983+0000","last_became_active":"2024-04-22T18:40:57.952303+0000","last_became_peered":"2024-04-22T18:40:57.952303+0000","last_unstale":"2024-04-22T18:41:10.125983+0000","last_undegraded":"2024-04-22T18:41:10.125983+0000","last_fullsized":"2024-04-22T18:41:10.125983+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:39:53.346787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.346787+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:22.397484+0000","scrub_duration":0.20273052499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"134'70","reported_seq":29,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-04-22T18:41:10.046302+0000","last_change":"2024-04-22T18:41:04.681901+0000","last_active":"2024-04-22T18:41:10.046302+0000","last_peered":"2024-04-22T18:41:10.046302+0000","last_clean":"2024-04-22T18:41:10.046302+0000","last_became_active":"2024-04-22T18:41:04.227104+0000","last_became_peered":"2024-04-22T18:41:04.227104+0000","last_unstale":"2024-04-22T18:41:10.046302+0000","last_undegraded":"2024-04-22T18:41:10.046302+0000","last_fullsized":"2024-04-22T18:41:10.046302+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'38","last_scrub_stamp":"2024-04-22T18:39:55.746067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:55.746067+0000","objects_scrubbed":1,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:16.501045+0000","scrub_duration":0.60877772299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067426,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":65,"num_read_kb":104527,"num_write":51,"num_write_kb":12867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15782313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"121'57","reported_seq":313,"reported_epoch":135,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:41:10.046391+0000","last_change":"2024-04-22T18:41:04.027077+0000","last_active":"2024-04-22T18:41:10.046391+0000","last_peered":"2024-04-22T18:41:10.046391+0000","last_clean":"2024-04-22T18:40:56.376630+0000","last_became_active":"2024-04-22T18:41:04.026361+0000","last_became_peered":"2024-04-22T18:41:04.026361+0000","last_unstale":"2024-04-22T18:41:10.046391+0000","last_undegraded":"2024-04-22T18:41:10.046391+0000","last_fullsized":"2024-04-22T18:41:10.046391+0000","mapping_epoch":129,"log_start":"79'52","ondisk_log_start":"79'52","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'57","last_scrub_stamp":"2024-04-22T18:40:56.376497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:56.376497+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":61,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:18.705090+0000","scrub_duration":0.30191299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3832029,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"133'140","reported_seq":533,"reported_epoch":135,"state":"active+remapped+backfill_wait","last_fresh":"2024-04-22T18:41:09.947862+0000","last_change":"2024-04-22T18:41:05.331922+0000","last_active":"2024-04-22T18:41:09.947862+0000","last_peered":"2024-04-22T18:41:09.947862+0000","last_clean":"2024-04-22T18:41:01.651924+0000","last_became_active":"2024-04-22T18:41:05.331724+0000","last_became_peered":"2024-04-22T18:41:05.331724+0000","last_unstale":"2024-04-22T18:41:09.947862+0000","last_undegraded":"2024-04-22T18:41:09.947862+0000","last_fullsized":"2024-04-22T18:41:09.947862+0000","mapping_epoch":130,"log_start":"84'88","ondisk_log_start":"84'88","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":52,"log_dups_size":106,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8512861,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":92238,"num_write":48,"num_write_kb":14172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4990114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"134'111","reported_seq":400,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.054130+0000","last_change":"2024-04-22T18:41:11.054130+0000","last_active":"2024-04-22T18:41:11.054130+0000","last_peered":"2024-04-22T18:41:11.054130+0000","last_clean":"2024-04-22T18:41:11.054130+0000","last_became_active":"2024-04-22T18:41:11.053760+0000","last_became_peered":"2024-04-22T18:41:11.053760+0000","last_unstale":"2024-04-22T18:41:11.054130+0000","last_undegraded":"2024-04-22T18:41:11.054130+0000","last_fullsized":"2024-04-22T18:41:11.054130+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840498,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":181703,"num_write":55,"num_write_kb":14923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4554596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"115'138","reported_seq":533,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.354483+0000","last_change":"2024-04-22T18:41:11.354483+0000","last_active":"2024-04-22T18:41:11.354483+0000","last_peered":"2024-04-22T18:41:11.354483+0000","last_clean":"2024-04-22T18:41:11.354483+0000","last_became_active":"2024-04-22T18:41:11.254149+0000","last_became_peered":"2024-04-22T18:41:11.254149+0000","last_unstale":"2024-04-22T18:41:11.354483+0000","last_undegraded":"2024-04-22T18:41:11.354483+0000","last_fullsized":"2024-04-22T18:41:11.354483+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'109","last_scrub_stamp":"2024-04-22T18:40:17.658363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:17.658363+0000","objects_scrubbed":5,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202176033,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7748093,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":259,"num_read_kb":295919,"num_write":73,"num_write_kb":21475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9711723,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"132'123","reported_seq":449,"reported_epoch":136,"state":"peering","last_fresh":"2024-04-22T18:41:11.155517+0000","last_change":"2024-04-22T18:41:11.155517+0000","last_active":"2024-04-22T18:41:09.948170+0000","last_peered":"2024-04-22T18:41:09.304990+0000","last_clean":"2024-04-22T18:40:55.895030+0000","last_became_active":"2024-04-22T18:41:05.129696+0000","last_became_peered":"2024-04-22T18:41:05.129696+0000","last_unstale":"2024-04-22T18:41:11.155517+0000","last_undegraded":"2024-04-22T18:41:11.155517+0000","last_fullsized":"2024-04-22T18:41:11.155517+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'119","last_scrub_stamp":"2024-04-22T18:40:55.573199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.573199+0000","objects_scrubbed":4,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32976813900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4521132,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":219,"num_read_kb":257791,"num_write":64,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3786235,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"133'66","reported_seq":32,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.254661+0000","last_change":"2024-04-22T18:41:11.254661+0000","last_active":"2024-04-22T18:41:11.254661+0000","last_peered":"2024-04-22T18:41:11.254661+0000","last_clean":"2024-04-22T18:41:11.254661+0000","last_became_active":"2024-04-22T18:41:11.154301+0000","last_became_peered":"2024-04-22T18:41:11.154301+0000","last_unstale":"2024-04-22T18:41:11.254661+0000","last_undegraded":"2024-04-22T18:41:11.254661+0000","last_fullsized":"2024-04-22T18:41:11.254661+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'46","last_scrub_stamp":"2024-04-22T18:40:34.755106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:34.755106+0000","objects_scrubbed":2,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40192567099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3655555,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":97,"num_read_kb":101756,"num_write":34,"num_write_kb":10293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8192626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"132'50","reported_seq":39,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.353806+0000","last_change":"2024-04-22T18:41:11.353806+0000","last_active":"2024-04-22T18:41:11.353806+0000","last_peered":"2024-04-22T18:41:11.353806+0000","last_clean":"2024-04-22T18:41:11.353806+0000","last_became_active":"2024-04-22T18:41:11.253409+0000","last_became_peered":"2024-04-22T18:41:11.253409+0000","last_unstale":"2024-04-22T18:41:11.353806+0000","last_undegraded":"2024-04-22T18:41:11.353806+0000","last_fullsized":"2024-04-22T18:41:11.353806+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'23","last_scrub_stamp":"2024-04-22T18:40:20.587606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.587606+0000","objects_scrubbed":1,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10225366599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949678,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":73,"num_read_kb":77102,"num_write":26,"num_write_kb":8588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5574713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"131'75","reported_seq":318,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-22T18:41:11.070420+0000","last_change":"2024-04-22T18:41:11.069650+0000","last_active":"2024-04-22T18:41:11.070420+0000","last_peered":"2024-04-22T18:41:11.070420+0000","last_clean":"2024-04-22T18:41:11.070420+0000","last_became_active":"2024-04-22T18:41:11.068781+0000","last_became_peered":"2024-04-22T18:41:11.068781+0000","last_unstale":"2024-04-22T18:41:11.070420+0000","last_undegraded":"2024-04-22T18:41:11.070420+0000","last_fullsized":"2024-04-22T18:41:11.070420+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2554697,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":134,"num_read_kb":128972,"num_write":38,"num_write_kb":9856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":36,"stat_sum":{"num_bytes":143818919,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":3280,"num_read_kb":3842005,"num_write":1027,"num_write_kb":293624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":176119390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":32,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":222531584,"data_stored":222187900,"data_compressed":843125,"data_compressed_allocated":897024,"data_compressed_original":1634304,"omap_allocated":0,"internal_metadata":0},"log_size":2339,"ondisk_log_size":2339,"up":72,"acting":72,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":134873976,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":64,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":968,"num_read_kb":22296,"num_write":1360,"num_write_kb":327192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":759,"num_bytes_recovered":138389864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":260345856,"data_stored":274862866,"data_compressed":14550567,"data_compressed_allocated":15519744,"data_compressed_original":30773248,"omap_allocated":0,"internal_metadata":0},"log_size":1099,"ondisk_log_size":1099,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2115712,"data_compressed":121124,"data_compressed_allocated":163840,"data_compressed_original":2072576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94072,"kb_used_data":62956,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94277768,"statfs":{"total":96636764160,"available":96540434432,"internally_reserved":0,"allocated":64466944,"data_stored":65274475,"data_compressed":1544961,"data_compressed_allocated":1654784,"data_compressed_original":3543040,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":9.8949999999999996,"5min":2.3170000000000002,"15min":1.054},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":139.84299999999999,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":13.949,"5min":6.3959999999999999,"15min":5.1369999999999996},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":186.97900000000001,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.41799999999999998}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":4.1959999999999997,"15min":4.6420000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":27.027000000000001,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":14.961,"5min":13.43,"15min":13.175000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":183.16300000000001,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.49299999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":3.3479999999999999,"15min":3.839},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64100000000000001,"5min":77.320999999999998,"15min":77.320999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47199999999999998,"15min":0.47299999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.437}]},{"osd":3,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":3.4079999999999999,"15min":3.903},"min":{"1min":0.32000000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60799999999999998,"5min":77.361000000000004,"15min":77.361000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.30299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.65000000000000002,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.61399999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":5.6710000000000003,"5min":4.7009999999999996,"15min":4.5389999999999997},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":69.097999999999999,"5min":85.509,"15min":85.509},"last":0.53100000000000003},{"interface":"front","average":{"1min":7.9509999999999996,"5min":19.887,"15min":21.876999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":181.97,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.39800000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.5379999999999998,"15min":2.8929999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67800000000000005,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.32500000000000001},{"interface":"front","average":{"1min":3.3170000000000002,"5min":6.3929999999999998,"15min":6.9050000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":69.063000000000002,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.59299999999999997}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":17.294,"5min":14.079000000000001,"15min":13.542999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":139.428,"5min":139.428,"15min":139.428},"last":0.624},{"interface":"front","average":{"1min":1.3580000000000001,"5min":3.5779999999999998,"15min":3.948},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":22.878,"5min":77.405000000000001,"15min":77.405000000000001},"last":0.54400000000000004}]}]},{"osd":6,"up_from":0,"seq":55834574881,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574881,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101772,"kb_used_data":66552,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94270068,"statfs":{"total":96636764160,"available":96532549632,"internally_reserved":0,"allocated":68149248,"data_stored":69565560,"data_compressed":2230485,"data_compressed_allocated":2383872,"data_compressed_original":4825088,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":13.022,"5min":15.922000000000001,"15min":16.405999999999999},"min":{"1min":0.27600000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":252.96199999999999,"5min":252.96199999999999,"15min":252.96199999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":20.289000000000001,"5min":10.912000000000001,"15min":9.3490000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":187.148,"5min":187.148,"15min":187.148},"last":0.39600000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":9.9629999999999992,"5min":2.327,"15min":1.0549999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":96.486999999999995,"5min":96.486999999999995,"15min":96.486999999999995},"last":0.34999999999999998},{"interface":"front","average":{"1min":19.181999999999999,"5min":8.1449999999999996,"15min":6.3049999999999997},"min":{"1min":0.30199999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":264.68299999999999,"5min":264.68299999999999,"15min":264.68299999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":18.23,"5min":10.502000000000001,"15min":9.2140000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":186.94300000000001,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":9.7699999999999996,"5min":2.3100000000000001,"15min":1.0669999999999999},"min":{"1min":0.33100000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":96.471999999999994,"5min":96.471999999999994,"15min":96.471999999999994},"last":0.41999999999999998}]},{"osd":3,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":21.619,"5min":7.5410000000000004,"15min":5.1950000000000003},"min":{"1min":0.32200000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":252.91900000000001,"5min":252.91900000000001,"15min":252.91900000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":22.878,"5min":21.350000000000001,"15min":21.094999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":187.11099999999999,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.41399999999999998}]},{"osd":5,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":9.75,"5min":2.2559999999999998,"15min":1.0069999999999999},"min":{"1min":0.23100000000000001,"5min":0.218,"15min":0.218},"max":{"1min":96.408000000000001,"5min":96.408000000000001,"15min":96.408000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.43099999999999999,"5min":3.3159999999999998,"15min":3.7970000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.76500000000000001,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.372}]},{"osd":6,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":28.026,"5min":16.132999999999999,"15min":14.151},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":197.72999999999999,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":9.7560000000000002,"5min":2.3039999999999998,"15min":1.0620000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":96.438000000000002,"5min":96.438000000000002,"15min":96.438000000000002},"last":0.38500000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":21.757000000000001,"5min":11.199,"15min":9.4390000000000001},"min":{"1min":0.30099999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":186.79300000000001,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":9.7880000000000003,"5min":2.343,"15min":1.1020000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":96.456999999999994,"5min":96.456999999999994,"15min":96.456999999999994},"last":0.31}]}]},{"osd":3,"up_from":13,"seq":55834574881,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116904,"kb_used_data":85784,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94254936,"statfs":{"total":96636764160,"available":96517054464,"internally_reserved":0,"allocated":87842816,"data_stored":90004091,"data_compressed":3484904,"data_compressed_allocated":3710976,"data_compressed_original":6991872,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":9.9700000000000006,"15min":11.456},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":11.797000000000001,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.26700000000000002,"5min":0.316,"15min":0.32400000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.43099999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.30199999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":25.978000000000002,"5min":9.2309999999999999,"15min":6.4400000000000004},"min":{"1min":0.20999999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":197.02099999999999,"5min":197.02099999999999,"15min":197.02099999999999},"last":31.143000000000001},{"interface":"front","average":{"1min":1.623,"5min":0.59199999999999997,"15min":0.41999999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":16.388999999999999,"5min":16.388999999999999,"15min":16.388999999999999},"last":0.29199999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":6.0090000000000003,"5min":2.0059999999999998,"15min":1.339},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":87.635999999999996,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.51800000000000002,"15min":0.45600000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":11.859999999999999,"5min":11.859999999999999,"15min":11.859999999999999},"last":0.32300000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":3.379,"15min":3.8650000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79100000000000004,"5min":71.998000000000005,"15min":71.998000000000005},"last":26.141999999999999},{"interface":"front","average":{"1min":6.1719999999999997,"5min":4.5439999999999996,"15min":4.2729999999999997},"min":{"1min":0.34799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":87.724000000000004,"5min":87.724000000000004,"15min":87.724000000000004},"last":26.119}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":6.1539999999999999,"5min":2.1760000000000002,"15min":1.5129999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":87.784000000000006,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.038,"5min":1.2130000000000001,"15min":1.242},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":11.911,"5min":13.965,"15min":13.965},"last":0.434}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.52700000000000002,"15min":0.53700000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72999999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.38},{"interface":"front","average":{"1min":9.9130000000000003,"5min":2.4079999999999999,"15min":1.157},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":96.140000000000001,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.41699999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":0.69599999999999995,"15min":0.621},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":11.887,"5min":11.887,"15min":11.887},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.55000000000000004,"15min":0.56200000000000006},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75600000000000001,"5min":0.995,"15min":0.995},"last":0.504}]}]},{"osd":2,"up_from":13,"seq":55834574881,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66200,"kb_used_data":39172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94305640,"statfs":{"total":96636764160,"available":96568975360,"internally_reserved":0,"allocated":40112128,"data_stored":40473541,"data_compressed":1159502,"data_compressed_allocated":1241088,"data_compressed_original":2568192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":7.4240000000000004,"15min":7.8920000000000003},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":59.002000000000002,"5min":149.03,"15min":149.03},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.308,"5min":1.649,"15min":1.873},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.442,"5min":31.866,"15min":31.866},"last":0.62}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":3.5579999999999998,"15min":1.8740000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":97.683999999999997,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":22.157,"5min":16.338999999999999,"15min":15.369},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":393.42399999999998,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.70299999999999996}]},{"osd":3,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":18.393000000000001,"5min":3.9689999999999999,"15min":1.5649999999999999},"min":{"1min":0.252,"5min":0.245,"15min":0.245},"max":{"1min":147.721,"5min":147.721,"15min":147.721},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.317,"5min":0.36899999999999999,"15min":0.377},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.53200000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.77000000000000002}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":3.4060000000000001,"5min":15.698,"15min":17.747},"min":{"1min":0.311,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":52.683,"5min":160.584,"15min":160.584},"last":1.008},{"interface":"front","average":{"1min":10.984,"5min":4.1550000000000002,"15min":3.0169999999999999},"min":{"1min":0.32200000000000001,"5min":0.315,"15min":0.315},"max":{"1min":97.887,"5min":97.887,"15min":97.887},"last":0.96499999999999997}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":2.5409999999999999,"15min":2.8809999999999998},"min":{"1min":0.33300000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.74299999999999999,"5min":48.631,"15min":48.631},"last":0.879},{"interface":"front","average":{"1min":0.50800000000000001,"5min":2.8820000000000001,"15min":3.278},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.83499999999999996,"5min":56.18,"15min":56.18},"last":0.91800000000000004}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":10.375,"5min":2.5030000000000001,"15min":1.1910000000000001},"min":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":97.832999999999998,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.33200000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.95299999999999996}]},{"osd":7,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.63800000000000001,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"last":0.97199999999999998},{"interface":"front","average":{"1min":10.396000000000001,"5min":2.5470000000000002,"15min":1.2390000000000001},"min":{"1min":0.38800000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":97.861999999999995,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.88900000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":150880,"kb_used_data":115568,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94220960,"statfs":{"total":96636764160,"available":96482263040,"internally_reserved":0,"allocated":118341632,"data_stored":120808731,"data_compressed":2986932,"data_compressed_allocated":3203072,"data_compressed_original":6979584,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":3.9870000000000001,"5min":7.0839999999999996,"15min":7.5999999999999996},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":66.480999999999995,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.34999999999999998},{"interface":"front","average":{"1min":4.0430000000000001,"5min":7.0990000000000002,"15min":7.6079999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":66.498000000000005,"5min":88.156000000000006,"15min":88.156000000000006},"last":0.24399999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":6.9429999999999996,"15min":8.0470000000000006},"min":{"1min":0.20000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.626,"5min":183.04400000000001,"15min":183.04400000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.32300000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.33700000000000002,"15min":0.33400000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":11.664,"15min":13.548999999999999},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.878,"5min":231.864,"15min":231.864},"last":0.27800000000000002}]},{"osd":4,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":7.29,"5min":11.845000000000001,"15min":12.603999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":109.399,"5min":192.42099999999999,"15min":192.42099999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":92.442999999999998}]},{"osd":5,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":11.359999999999999,"15min":13.167},"min":{"1min":0.34399999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.052,"5min":138.089,"15min":138.089},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.48999999999999999,"5min":16.158999999999999,"15min":18.771000000000001},"min":{"1min":0.35099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69899999999999995,"5min":183.08199999999999,"15min":183.08199999999999},"last":0.67600000000000005}]},{"osd":6,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":4.8520000000000003,"5min":3.1920000000000002,"15min":2.915},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":78.805000000000007,"5min":78.805000000000007,"15min":78.805000000000007},"last":0.57799999999999996},{"interface":"front","average":{"1min":5.9400000000000004,"5min":14.413,"15min":15.824999999999999},"min":{"1min":0.36599999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":98.968000000000004,"5min":183.10900000000001,"15min":183.10900000000001},"last":0.58799999999999997}]},{"osd":7,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.38500000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.503},{"interface":"front","average":{"1min":0.52100000000000002,"5min":2.5350000000000001,"15min":2.871},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.0189999999999999,"5min":51.097000000000001,"15min":51.097000000000001},"last":0.61899999999999999}]}]},{"osd":0,"up_from":0,"seq":55834574881,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33095680,"data_stored":34790173,"data_compressed":1758468,"data_compressed_allocated":1871872,"data_compressed_original":3694592,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49922048,"data_stored":52964618,"data_compressed":2769382,"data_compressed_allocated":2965504,"data_compressed_original":6127616,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18558976,"data_stored":19833542,"data_compressed":1159502,"data_compressed_allocated":1241088,"data_compressed_original":2568192,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":59305984,"data_stored":62276923,"data_compressed":3265006,"data_compressed_allocated":3477504,"data_compressed_original":6598656,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36728832,"data_stored":38937143,"data_compressed":2059948,"data_compressed_allocated":2203648,"data_compressed_original":4497408,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":223399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":42012672,"data_stored":44215728,"data_compressed":2229009,"data_compressed_allocated":2367488,"data_compressed_original":4714496,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66818048,"data_stored":66627399,"data_compressed":186556,"data_compressed_allocated":196608,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20058112,"data_stored":19952057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27000832,"data_stored":27034930,"data_compressed":219898,"data_compressed_allocated":233472,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":31031296,"data_stored":31101090,"data_compressed":176873,"data_compressed_allocated":188416,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33280000,"data_stored":33118894,"data_compressed":53657,"data_compressed_allocated":57344,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1929216,"data_stored":1913105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":42414080,"data_stored":42440425,"data_compressed":206141,"data_compressed_allocated":221184,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:41:14.348 INFO:tasks.ceph.ceph_manager.ceph:decrease pool size by 10 2024-04-22T18:41:14.348 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-04-22T18:41:14.392 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:14.394+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:14.439 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:14.440 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:14.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:14.440 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:14.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.440 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:289 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:14.440 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:14.440 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.798478317999994, 2024-04-22T18:41:14.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9176801189999999, 2024-04-22T18:41:14.440 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.440 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.440 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:14.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:14.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640723+0000", 2024-04-22T18:41:14.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:14.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640732+0000", 2024-04-22T18:41:14.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5809999999999997e-06 2024-04-22T18:41:14.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640744+0000", 2024-04-22T18:41:14.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.261e-05 2024-04-22T18:41:14.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640753+0000", 2024-04-22T18:41:14.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7460000000000006e-06 2024-04-22T18:41:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642189+0000", 2024-04-22T18:41:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001435551 2024-04-22T18:41:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642200+0000", 2024-04-22T18:41:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1389999999999999e-05 2024-04-22T18:41:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.847015+0000", 2024-04-22T18:41:14.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048149160000001 2024-04-22T18:41:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.848217+0000", 2024-04-22T18:41:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012020659999999999 2024-04-22T18:41:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351823+0000", 2024-04-22T18:41:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50360565099999999 2024-04-22T18:41:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351907+0000", 2024-04-22T18:41:14.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4141999999999995e-05 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352051+0000", 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014386600000000001 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453617+0000", 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156649500000001 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558297+0000", 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10468000399999999 2024-04-22T18:41:14.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558331+0000", 2024-04-22T18:41:14.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3518e-05 2024-04-22T18:41:14.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558405+0000", 2024-04-22T18:41:14.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4221000000000006e-05 2024-04-22T18:41:14.446 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.446 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.447 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.447 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:1 3.2 3:6d90b903:::64b5942574fb36ca1b9a2be186c6b740edf58e989056a2f88b83cd280451a438:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:14.447 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:14.447 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.267177357999998, 2024-04-22T18:41:14.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.285534422, 2024-04-22T18:41:14.447 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.447 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.447 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:14.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:14.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172024+0000", 2024-04-22T18:41:14.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:14.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173091+0000", 2024-04-22T18:41:14.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010677569999999999 2024-04-22T18:41:14.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173103+0000", 2024-04-22T18:41:14.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1241e-05 2024-04-22T18:41:14.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173109+0000", 2024-04-22T18:41:14.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4300000000000003e-06 2024-04-22T18:41:14.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173168+0000", 2024-04-22T18:41:14.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9011000000000001e-05 2024-04-22T18:41:14.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:14.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173177+0000", 2024-04-22T18:41:14.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-04-22T18:41:14.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948501+0000", 2024-04-22T18:41:14.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77532387199999997 2024-04-22T18:41:14.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948912+0000", 2024-04-22T18:41:14.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041183000000000002 2024-04-22T18:41:14.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950438+0000", 2024-04-22T18:41:14.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015253199999999999 2024-04-22T18:41:14.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950488+0000", 2024-04-22T18:41:14.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0285999999999999e-05 2024-04-22T18:41:14.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:14.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950601+0000", 2024-04-22T18:41:14.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011244899999999999 2024-04-22T18:41:14.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954048+0000", 2024-04-22T18:41:14.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003447313 2024-04-22T18:41:14.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457373+0000", 2024-04-22T18:41:14.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50332465299999996 2024-04-22T18:41:14.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457430+0000", 2024-04-22T18:41:14.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7574999999999997e-05 2024-04-22T18:41:14.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457560+0000", 2024-04-22T18:41:14.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013023900000000001 2024-04-22T18:41:14.453 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.453 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.453 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.453 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:8 3.2 3:5f64590d:::7c47933f331d40689081b2bd8f9d7f7bdae1be1fdccb52d696047a3969e902b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:14.454 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:14.454 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.266084989000007, 2024-04-22T18:41:14.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3854200569999999, 2024-04-22T18:41:14.454 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.454 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.454 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:14.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:14.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173117+0000", 2024-04-22T18:41:14.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:14.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173393+0000", 2024-04-22T18:41:14.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000275278 2024-04-22T18:41:14.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173399+0000", 2024-04-22T18:41:14.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4590000000000003e-06 2024-04-22T18:41:14.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173406+0000", 2024-04-22T18:41:14.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2450000000000001e-06 2024-04-22T18:41:14.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173795+0000", 2024-04-22T18:41:14.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038912400000000002 2024-04-22T18:41:14.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:14.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173804+0000", 2024-04-22T18:41:14.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-04-22T18:41:14.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.949506+0000", 2024-04-22T18:41:14.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77570185700000005 2024-04-22T18:41:14.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950166+0000", 2024-04-22T18:41:14.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066022800000000001 2024-04-22T18:41:14.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352843+0000", 2024-04-22T18:41:14.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40267661799999999 2024-04-22T18:41:14.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352911+0000", 2024-04-22T18:41:14.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8126000000000005e-05 2024-04-22T18:41:14.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:14.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.353031+0000", 2024-04-22T18:41:14.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119919 2024-04-22T18:41:14.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453622+0000", 2024-04-22T18:41:14.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100590793 2024-04-22T18:41:14.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558446+0000", 2024-04-22T18:41:14.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104824073 2024-04-22T18:41:14.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558479+0000", 2024-04-22T18:41:14.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3362999999999999e-05 2024-04-22T18:41:14.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558538+0000", 2024-04-22T18:41:14.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9046999999999999e-05 2024-04-22T18:41:14.460 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.460 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.460 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.460 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:10 3.2 3:518c37f9:::851eca84dafbf2f780c4f07291110c453349bdc9c623c17bdcd26e27badb30ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:14.460 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:14.460 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.265791395999997, 2024-04-22T18:41:14.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3852527729999999, 2024-04-22T18:41:14.461 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.461 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.461 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:14.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:14.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173411+0000", 2024-04-22T18:41:14.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:14.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173749+0000", 2024-04-22T18:41:14.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000338029 2024-04-22T18:41:14.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173755+0000", 2024-04-22T18:41:14.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0530000000000003e-06 2024-04-22T18:41:14.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173759+0000", 2024-04-22T18:41:14.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8190000000000002e-06 2024-04-22T18:41:14.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173855+0000", 2024-04-22T18:41:14.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5927999999999994e-05 2024-04-22T18:41:14.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:14.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173862+0000", 2024-04-22T18:41:14.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4449999999999999e-06 2024-04-22T18:41:14.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950835+0000", 2024-04-22T18:41:14.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77697248299999999 2024-04-22T18:41:14.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951632+0000", 2024-04-22T18:41:14.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079752299999999996 2024-04-22T18:41:14.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355407+0000", 2024-04-22T18:41:14.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40377480500000001 2024-04-22T18:41:14.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355462+0000", 2024-04-22T18:41:14.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4577999999999998e-05 2024-04-22T18:41:14.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:14.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355571+0000", 2024-04-22T18:41:14.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108964 2024-04-22T18:41:14.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453625+0000", 2024-04-22T18:41:14.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098054397000000001 2024-04-22T18:41:14.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558574+0000", 2024-04-22T18:41:14.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104948571 2024-04-22T18:41:14.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558602+0000", 2024-04-22T18:41:14.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7966999999999998e-05 2024-04-22T18:41:14.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558665+0000", 2024-04-22T18:41:14.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3032000000000002e-05 2024-04-22T18:41:14.467 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.467 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.467 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.467 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:11 3.2 3:718aa8aa:::9603c4f8e269a436b2f0bf376c5374e77bdd7cf58fccae60e4e65ac4696806d4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:14.467 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:14.467 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.265439778000001, 2024-04-22T18:41:14.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4852604440000001, 2024-04-22T18:41:14.468 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.468 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.468 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:14.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:14.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:14.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:14.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173813+0000", 2024-04-22T18:41:14.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0655999999999997e-05 2024-04-22T18:41:14.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173819+0000", 2024-04-22T18:41:14.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.468e-06 2024-04-22T18:41:14.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173822+0000", 2024-04-22T18:41:14.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4879999999999999e-06 2024-04-22T18:41:14.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173983+0000", 2024-04-22T18:41:14.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160519 2024-04-22T18:41:14.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:14.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173991+0000", 2024-04-22T18:41:14.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6490000000000008e-06 2024-04-22T18:41:14.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951688+0000", 2024-04-22T18:41:14.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77769743800000002 2024-04-22T18:41:14.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952419+0000", 2024-04-22T18:41:14.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073067499999999997 2024-04-22T18:41:14.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355901+0000", 2024-04-22T18:41:14.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40348237599999998 2024-04-22T18:41:14.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355953+0000", 2024-04-22T18:41:14.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1477000000000002e-05 2024-04-22T18:41:14.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:14.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356060+0000", 2024-04-22T18:41:14.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107114 2024-04-22T18:41:14.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453629+0000", 2024-04-22T18:41:14.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097569127000000005 2024-04-22T18:41:14.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658807+0000", 2024-04-22T18:41:14.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20517829800000001 2024-04-22T18:41:14.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658876+0000", 2024-04-22T18:41:14.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8454000000000006e-05 2024-04-22T18:41:14.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659024+0000", 2024-04-22T18:41:14.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148338 2024-04-22T18:41:14.474 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.474 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.474 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.474 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:12 3.2 3:475e1cd9:::efaab2bd13543cdbb143e869e20eb0ec424f7f5202707859b241ed91d4674936:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:14.474 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:14.474 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.265376833999994, 2024-04-22T18:41:14.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4853405259999999, 2024-04-22T18:41:14.474 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.474 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.475 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:14.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:14.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:14.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173940+0000", 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011465500000000001 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173946+0000", 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5289999999999999e-06 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173950+0000", 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8290000000000001e-06 2024-04-22T18:41:14.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174277+0000", 2024-04-22T18:41:14.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032761299999999998 2024-04-22T18:41:14.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:14.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174285+0000", 2024-04-22T18:41:14.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.52e-06 2024-04-22T18:41:14.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952455+0000", 2024-04-22T18:41:14.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77817000700000005 2024-04-22T18:41:14.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954035+0000", 2024-04-22T18:41:14.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001580078 2024-04-22T18:41:14.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356379+0000", 2024-04-22T18:41:14.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40234440399999999 2024-04-22T18:41:14.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356428+0000", 2024-04-22T18:41:14.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8832999999999998e-05 2024-04-22T18:41:14.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:14.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356532+0000", 2024-04-22T18:41:14.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103235 2024-04-22T18:41:14.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453632+0000", 2024-04-22T18:41:14.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097100346000000004 2024-04-22T18:41:14.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659069+0000", 2024-04-22T18:41:14.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20543763800000001 2024-04-22T18:41:14.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659105+0000", 2024-04-22T18:41:14.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5240000000000001e-05 2024-04-22T18:41:14.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659167+0000", 2024-04-22T18:41:14.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2207999999999998e-05 2024-04-22T18:41:14.480 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.480 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.481 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.481 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:16 3.2 3:47c76cc8:::c00d18458eb20c0c0aa9c612cf9793a68445ab0ea7b9e650ecd0072c60a96c0f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:14.481 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:14.481 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.265248874999997, 2024-04-22T18:41:14.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.485347744, 2024-04-22T18:41:14.481 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.481 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.481 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:14.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:14.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:14.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:14.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174234+0000", 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028047599999999999 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174240+0000", 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2079999999999997e-06 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174244+0000", 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5810000000000001e-06 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174670+0000", 2024-04-22T18:41:14.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042640000000000001 2024-04-22T18:41:14.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:14.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174675+0000", 2024-04-22T18:41:14.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8709999999999996e-06 2024-04-22T18:41:14.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054168+0000", 2024-04-22T18:41:14.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87949257300000006 2024-04-22T18:41:14.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054877+0000", 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070884300000000002 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356836+0000", 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30195926000000001 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356887+0000", 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0767999999999999e-05 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356991+0000", 2024-04-22T18:41:14.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104072 2024-04-22T18:41:14.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453636+0000", 2024-04-22T18:41:14.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096644968999999997 2024-04-22T18:41:14.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659207+0000", 2024-04-22T18:41:14.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20557154699999999 2024-04-22T18:41:14.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659240+0000", 2024-04-22T18:41:14.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2665e-05 2024-04-22T18:41:14.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659302+0000", 2024-04-22T18:41:14.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2234000000000001e-05 2024-04-22T18:41:14.487 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.487 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.487 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.487 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:17 3.2 3:6ef31daa:::1cf4ff496c4480e58510f607481d976b47d84868e2c54016273c38baf590f3b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:14.487 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:14.488 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.264954595000006, 2024-04-22T18:41:14.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4851703729999999, 2024-04-22T18:41:14.488 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.488 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.488 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:14.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:14.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174248+0000", 2024-04-22T18:41:14.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:14.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174626+0000", 2024-04-22T18:41:14.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037839699999999999 2024-04-22T18:41:14.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174633+0000", 2024-04-22T18:41:14.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0990000000000004e-06 2024-04-22T18:41:14.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174636+0000", 2024-04-22T18:41:14.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5540000000000002e-06 2024-04-22T18:41:14.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174790+0000", 2024-04-22T18:41:14.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153807 2024-04-22T18:41:14.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:14.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174793+0000", 2024-04-22T18:41:14.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4970000000000001e-06 2024-04-22T18:41:14.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054902+0000", 2024-04-22T18:41:14.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88010886300000002 2024-04-22T18:41:14.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055366+0000", 2024-04-22T18:41:14.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046336199999999999 2024-04-22T18:41:14.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.452973+0000", 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39760783999999999 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453046+0000", 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2374999999999997e-05 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453193+0000", 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014722499999999999 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554698+0000", 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101505096 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659337+0000", 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104639236 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659365+0000", 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7594e-05 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659419+0000", 2024-04-22T18:41:14.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4058999999999999e-05 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:18 3.2 3:792a4ed0:::e1c814961b3d1531fb5ed475b3d496a0405c3a819b68a92ff9b2391fe912370f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.264558339000004, 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.585036954, 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:14.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174640+0000", 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174758+0000", 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011819099999999999 2024-04-22T18:41:14.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174764+0000", 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6729999999999997e-06 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174767+0000", 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3830000000000001e-06 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373669+0000", 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198901989 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373686+0000", 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7286e-05 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055408+0000", 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68172133400000001 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055938+0000", 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053016599999999999 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453668+0000", 2024-04-22T18:41:14.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39773030799999998 2024-04-22T18:41:14.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453717+0000", 2024-04-22T18:41:14.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8884000000000002e-05 2024-04-22T18:41:14.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:14.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453823+0000", 2024-04-22T18:41:14.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105791 2024-04-22T18:41:14.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554715+0000", 2024-04-22T18:41:14.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100891628 2024-04-22T18:41:14.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759439+0000", 2024-04-22T18:41:14.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20472459400000001 2024-04-22T18:41:14.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759525+0000", 2024-04-22T18:41:14.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5653000000000001e-05 2024-04-22T18:41:14.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759682+0000", 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015711800000000001 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:75 3.1 3:ab43a94d:::f38e6e97cb24e2b90da964fee5d2d6d8674386870fc552915e0c3fdbefd4b126:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.191681822, 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60149928399999997, 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:14.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:14.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247520+0000", 2024-04-22T18:41:14.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:14.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247527+0000", 2024-04-22T18:41:14.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1670000000000002e-06 2024-04-22T18:41:14.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247539+0000", 2024-04-22T18:41:14.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503e-05 2024-04-22T18:41:14.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247549+0000", 2024-04-22T18:41:14.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0000000000000001e-05 2024-04-22T18:41:14.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247796+0000", 2024-04-22T18:41:14.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024712699999999998 2024-04-22T18:41:14.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:14.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247955+0000", 2024-04-22T18:41:14.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158534 2024-04-22T18:41:14.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.343985+0000", 2024-04-22T18:41:14.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096030344000000004 2024-04-22T18:41:14.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.344035+0000", 2024-04-22T18:41:14.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9969999999999998e-05 2024-04-22T18:41:14.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.745806+0000", 2024-04-22T18:41:14.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.401771029 2024-04-22T18:41:14.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746040+0000", 2024-04-22T18:41:14.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023339300000000001 2024-04-22T18:41:14.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:14.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746198+0000", 2024-04-22T18:41:14.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015793299999999999 2024-04-22T18:41:14.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848737+0000", 2024-04-22T18:41:14.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10253898 2024-04-22T18:41:14.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848751+0000", 2024-04-22T18:41:14.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4694e-05 2024-04-22T18:41:14.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848825+0000", 2024-04-22T18:41:14.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3443999999999997e-05 2024-04-22T18:41:14.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849021+0000", 2024-04-22T18:41:14.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000196047 2024-04-22T18:41:14.506 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.506 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.506 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.507 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:76 3.1 3:b44829a0:::3788286a255d538503614e5f04049b0bb54cacaa317ab102b5dee8f5b640b296:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:14.507 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:14.507 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.191642756, 2024-04-22T18:41:14.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60168681599999996, 2024-04-22T18:41:14.507 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.507 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.507 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:14.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:14.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:14.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:14.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247564+0000", 2024-04-22T18:41:14.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1150000000000004e-06 2024-04-22T18:41:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247570+0000", 2024-04-22T18:41:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3940000000000001e-06 2024-04-22T18:41:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247576+0000", 2024-04-22T18:41:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3619999999999997e-06 2024-04-22T18:41:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.340845+0000", 2024-04-22T18:41:14.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.093269199999999997 2024-04-22T18:41:14.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:14.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.341043+0000", 2024-04-22T18:41:14.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019834400000000001 2024-04-22T18:41:14.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345106+0000", 2024-04-22T18:41:14.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040629780000000001 2024-04-22T18:41:14.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345160+0000", 2024-04-22T18:41:14.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4246999999999997e-05 2024-04-22T18:41:14.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747047+0000", 2024-04-22T18:41:14.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40188678300000003 2024-04-22T18:41:14.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747162+0000", 2024-04-22T18:41:14.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011483400000000001 2024-04-22T18:41:14.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:14.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747429+0000", 2024-04-22T18:41:14.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026700299999999998 2024-04-22T18:41:14.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849032+0000", 2024-04-22T18:41:14.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101603203 2024-04-22T18:41:14.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849095+0000", 2024-04-22T18:41:14.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2597999999999999e-05 2024-04-22T18:41:14.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849136+0000", 2024-04-22T18:41:14.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1251000000000002e-05 2024-04-22T18:41:14.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849247+0000", 2024-04-22T18:41:14.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011127 2024-04-22T18:41:14.514 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.514 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.514 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.514 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3526 4.c 4:314c2fec:::smithi08937929-22:head [setxattr _header (56) in=7b,truncate 3092521] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:41:14.514 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:14.514 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.210005975999998, 2024-04-22T18:41:14.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60463626000000004, 2024-04-22T18:41:14.514 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.514 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.514 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3526 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229200+0000", 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2320000000000001e-06 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229203+0000", 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329292+0000", 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100088891 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329357+0000", 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5115000000000003e-05 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329453+0000", 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5802000000000002e-05 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:14.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329594+0000", 2024-04-22T18:41:14.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014157500000000001 2024-04-22T18:41:14.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.330172+0000", 2024-04-22T18:41:14.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057782799999999996 2024-04-22T18:41:14.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833712+0000", 2024-04-22T18:41:14.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50353975299999998 2024-04-22T18:41:14.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833774+0000", 2024-04-22T18:41:14.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1832e-05 2024-04-22T18:41:14.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833834+0000", 2024-04-22T18:41:14.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9809999999999998e-05 2024-04-22T18:41:14.519 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.519 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.519 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.519 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:339 3.2 3:4159b8d1:::bb1f8e997816dd9d2f29d36c4c1859fcaffbf8763b5c8d63c7a818d8d84bac37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:14.520 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:14.520 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.6637522169999999, 2024-04-22T18:41:14.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0092773479999999, 2024-04-22T18:41:14.520 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.520 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.520 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:14.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:14.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775450+0000", 2024-04-22T18:41:14.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:14.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775457+0000", 2024-04-22T18:41:14.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2949999999999996e-06 2024-04-22T18:41:14.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775467+0000", 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0180000000000001e-05 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775475+0000", 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.678e-06 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.877787+0000", 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102312559 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.880686+0000", 2024-04-22T18:41:14.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002898748 2024-04-22T18:41:14.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.282867+0000", 2024-04-22T18:41:14.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.402180961 2024-04-22T18:41:14.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.282945+0000", 2024-04-22T18:41:14.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7775999999999995e-05 2024-04-22T18:41:14.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283065+0000", 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0001199 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.285514+0000", 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0024491729999999998 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784375+0000", 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49886154100000002 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784475+0000", 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9074999999999996e-05 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784728+0000", 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025386799999999998 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:340 3.2 3:4bbc2d06:::e1c179ab87fb8e08c4a1dfc0c7cc725138181c5734d6df18f3f2545b93731e97:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.6637197979999998, 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009550757, 2024-04-22T18:41:14.525 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.526 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.526 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:14.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:14.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:14.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:14.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775487+0000", 2024-04-22T18:41:14.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6839999999999998e-06 2024-04-22T18:41:14.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775492+0000", 2024-04-22T18:41:14.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6899999999999997e-06 2024-04-22T18:41:14.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775497+0000", 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8590000000000004e-06 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.880737+0000", 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105240267 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881308+0000", 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057098399999999997 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283810+0000", 2024-04-22T18:41:14.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40250177500000001 2024-04-22T18:41:14.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283883+0000", 2024-04-22T18:41:14.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2430999999999998e-05 2024-04-22T18:41:14.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:14.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284019+0000", 2024-04-22T18:41:14.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013652299999999999 2024-04-22T18:41:14.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.287390+0000", 2024-04-22T18:41:14.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033710089999999999 2024-04-22T18:41:14.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784829+0000", 2024-04-22T18:41:14.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49743879400000002 2024-04-22T18:41:14.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784893+0000", 2024-04-22T18:41:14.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4184999999999996e-05 2024-04-22T18:41:14.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785034+0000", 2024-04-22T18:41:14.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014114600000000001 2024-04-22T18:41:14.531 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.531 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.531 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.531 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:341 3.2 3:5b8bea68:::ab73be59042e3e18c4fe1fdbc2bbcc205779424ddf3bb423a595330be9026399:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:14.532 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:14.532 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.663700446, 2024-04-22T18:41:14.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009670804, 2024-04-22T18:41:14.532 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.532 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.532 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:14.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:14.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775502+0000", 2024-04-22T18:41:14.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:14.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775505+0000", 2024-04-22T18:41:14.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0809999999999998e-06 2024-04-22T18:41:14.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775511+0000", 2024-04-22T18:41:14.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2039999999999996e-06 2024-04-22T18:41:14.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775515+0000", 2024-04-22T18:41:14.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8790000000000002e-06 2024-04-22T18:41:14.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881376+0000", 2024-04-22T18:41:14.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105861 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881839+0000", 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046217200000000001 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284569+0000", 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40273058900000003 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284730+0000", 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077800000000001 2024-04-22T18:41:14.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284923+0000", 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000193253 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.287593+0000", 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026699359999999999 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785074+0000", 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49748046000000001 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785103+0000", 2024-04-22T18:41:14.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8935e-05 2024-04-22T18:41:14.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785174+0000", 2024-04-22T18:41:14.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1001999999999995e-05 2024-04-22T18:41:14.537 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.537 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.537 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.537 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:342 3.2 3:634dda9f:::10622dc7de8047e7908c0b5cca7a43d84cdf1b7629e6f7e159b8207feba82b01:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:14.538 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:14.538 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.6636822340000004, 2024-04-22T18:41:14.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.022499579, 2024-04-22T18:41:14.538 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.538 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.538 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:14.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775524+0000", 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9629999999999998e-06 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775529+0000", 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0320000000000003e-06 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775533+0000", 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7380000000000002e-06 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881884+0000", 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106350294 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.882492+0000", 2024-04-22T18:41:14.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060872699999999999 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381559+0000", 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49906679199999998 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381677+0000", 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011743699999999999 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381899+0000", 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022272599999999999 2024-04-22T18:41:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.383440+0000", 2024-04-22T18:41:14.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015407719999999999 2024-04-22T18:41:14.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.797827+0000", 2024-04-22T18:41:14.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.414386741 2024-04-22T18:41:14.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.797887+0000", 2024-04-22T18:41:14.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9889000000000002e-05 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.798021+0000", 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000133948 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:343 3.2 3:7db7298e:::00597d8527769aa920807495342cf3ba06efb8f507feae865dd89cd8ad10bda0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.6636632789999997, 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.110995373, 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.543 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775542+0000", 2024-04-22T18:41:14.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9079999999999999e-06 2024-04-22T18:41:14.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775547+0000", 2024-04-22T18:41:14.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.994e-06 2024-04-22T18:41:14.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775553+0000", 2024-04-22T18:41:14.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2299999999999999e-06 2024-04-22T18:41:14.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.882519+0000", 2024-04-22T18:41:14.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106966453 2024-04-22T18:41:14.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883103+0000", 2024-04-22T18:41:14.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058419400000000001 2024-04-22T18:41:14.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.581957+0000", 2024-04-22T18:41:14.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69885328599999996 2024-04-22T18:41:14.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582023+0000", 2024-04-22T18:41:14.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6210999999999993e-05 2024-04-22T18:41:14.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:14.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582146+0000", 2024-04-22T18:41:14.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123669 2024-04-22T18:41:14.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.682293+0000", 2024-04-22T18:41:14.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10014662100000001 2024-04-22T18:41:14.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886410+0000", 2024-04-22T18:41:14.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20411678899999999 2024-04-22T18:41:14.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886445+0000", 2024-04-22T18:41:14.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5462e-05 2024-04-22T18:41:14.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886535+0000", 2024-04-22T18:41:14.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9995999999999997e-05 2024-04-22T18:41:14.549 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.549 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.549 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.549 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:344 3.1 3:9da67ae4:::04c6b48a110b3f74550f78f0899df697160a1eeab7f41463f6ac18091ca67078:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:14.549 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:14.549 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.6636279720000005, 2024-04-22T18:41:14.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60604869800000005, 2024-04-22T18:41:14.550 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.550 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.550 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:14.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:14.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:14.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:14.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775578+0000", 2024-04-22T18:41:14.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3440000000000001e-06 2024-04-22T18:41:14.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775584+0000", 2024-04-22T18:41:14.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6969999999999998e-06 2024-04-22T18:41:14.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775589+0000", 2024-04-22T18:41:14.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0540000000000002e-06 2024-04-22T18:41:14.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775729+0000", 2024-04-22T18:41:14.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014061299999999999 2024-04-22T18:41:14.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776045+0000", 2024-04-22T18:41:14.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000315168 2024-04-22T18:41:14.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080584+0000", 2024-04-22T18:41:14.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30453977700000001 2024-04-22T18:41:14.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080635+0000", 2024-04-22T18:41:14.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1007999999999997e-05 2024-04-22T18:41:14.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:14.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080741+0000", 2024-04-22T18:41:14.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105646 2024-04-22T18:41:14.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.180821+0000", 2024-04-22T18:41:14.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10007980900000001 2024-04-22T18:41:14.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381437+0000", 2024-04-22T18:41:14.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20061589399999999 2024-04-22T18:41:14.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381491+0000", 2024-04-22T18:41:14.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4531999999999999e-05 2024-04-22T18:41:14.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381624+0000", 2024-04-22T18:41:14.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000132745 2024-04-22T18:41:14.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.555 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.556 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:345 3.1 3:bd847077:::0e274616605e19dd653fcf05d310933ab02835574414ca18340b3fc325d2da4f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:14.556 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:14.556 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.6636036120000002, 2024-04-22T18:41:14.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.90876140699999997, 2024-04-22T18:41:14.556 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.556 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.556 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:14.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:14.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775599+0000", 2024-04-22T18:41:14.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:14.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775602+0000", 2024-04-22T18:41:14.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.976e-06 2024-04-22T18:41:14.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775607+0000", 2024-04-22T18:41:14.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2229999999999998e-06 2024-04-22T18:41:14.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775612+0000", 2024-04-22T18:41:14.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6539999999999998e-06 2024-04-22T18:41:14.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776072+0000", 2024-04-22T18:41:14.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046006299999999998 2024-04-22T18:41:14.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:14.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776256+0000", 2024-04-22T18:41:14.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000183735 2024-04-22T18:41:14.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081157+0000", 2024-04-22T18:41:14.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30490169099999997 2024-04-22T18:41:14.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081203+0000", 2024-04-22T18:41:14.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5677999999999999e-05 2024-04-22T18:41:14.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:14.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081301+0000", 2024-04-22T18:41:14.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7893999999999995e-05 2024-04-22T18:41:14.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.180827+0000", 2024-04-22T18:41:14.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099525725999999995 2024-04-22T18:41:14.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684197+0000", 2024-04-22T18:41:14.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50336990699999995 2024-04-22T18:41:14.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684247+0000", 2024-04-22T18:41:14.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0195000000000003e-05 2024-04-22T18:41:14.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684361+0000", 2024-04-22T18:41:14.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114189 2024-04-22T18:41:14.561 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.561 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.561 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.562 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4267 3.2 3:584fd9a2:::cda848272f4ef42ebafc2089aaddeacaf12eabd82aef5e42e32ae93a8a5da0b6:head [write 0~2635776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:14.562 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:14.562 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.5832537559999995, 2024-04-22T18:41:14.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.72658671900000005, 2024-04-22T18:41:14.562 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:14.562 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:14.562 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:14.562 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:14.562 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:14.562 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4267 2024-04-22T18:41:14.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.562 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:14.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:14.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:14.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:14.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:14.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:14.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:14.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855946+0000", 2024-04-22T18:41:14.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:14.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:14.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.856984+0000", 2024-04-22T18:41:14.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001037764 2024-04-22T18:41:14.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:14.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.857003+0000", 2024-04-22T18:41:14.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8876e-05 2024-04-22T18:41:14.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:14.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.857016+0000", 2024-04-22T18:41:14.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3337e-05 2024-04-22T18:41:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883170+0000", 2024-04-22T18:41:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026154047 2024-04-22T18:41:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883262+0000", 2024-04-22T18:41:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1996999999999993e-05 2024-04-22T18:41:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:14.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883360+0000", 2024-04-22T18:41:14.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7429000000000005e-05 2024-04-22T18:41:14.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:14.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.887277+0000", 2024-04-22T18:41:14.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039169449999999998 2024-04-22T18:41:14.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:14.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582423+0000", 2024-04-22T18:41:14.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69514625200000002 2024-04-22T18:41:14.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:14.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582458+0000", 2024-04-22T18:41:14.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5293999999999998e-05 2024-04-22T18:41:14.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:14.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:14.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:14.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582536+0000", 2024-04-22T18:41:14.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7855000000000006e-05 2024-04-22T18:41:14.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.567 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:14.567 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:14.567 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:14.568 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:41:14.569 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:14.570+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:14.570 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:14.570 INFO:tasks.rados.rados.0.smithi089.stdout:1795: set_chunk oid 10 offset: 1276928 length: 1517056 target oid ce8f4919bd96542169a78742da39aabd7c54c9b1855d85be270b88232813da9e offset: 0 snap -1 2024-04-22T18:41:14.570 INFO:tasks.rados.rados.0.smithi089.stdout:1796: set_chunk oid 33 target oid 2024-04-22T18:41:14.670 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:14.671+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:14.677 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:14.677 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:14.677 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:14.677 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:14.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:14.726 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:14.727 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:14.753 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:14.753 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:14.768 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:14.769 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:14.771 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:14.772+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:14.784 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:14.784 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:14.828 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:14.829 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:14.829 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:14.829 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:14.829 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:14.836 INFO:tasks.rados.rados.0.smithi089.stdout:1796: set_chunk oid 33 offset: 1400832 length: 1824768 target oid 76557f7b04b9068f582fad0ed7168181a199ce00b3576abfae4dbc9eae3d3b8e offset: 0 snap -1 2024-04-22T18:41:14.836 INFO:tasks.rados.rados.0.smithi089.stdout:1797: read oid 17 snap -1 2024-04-22T18:41:14.836 INFO:tasks.rados.rados.0.smithi089.stdout:1797: expect (ObjNum 366 snap 0 seq_num 366) 2024-04-22T18:41:14.836 INFO:tasks.rados.rados.0.smithi089.stdout:1798: tier_evict oid 1 2024-04-22T18:41:14.837 INFO:tasks.rados.rados.0.smithi089.stdout:1798: tier_evict oid 1 snap -1 2024-04-22T18:41:14.837 INFO:tasks.rados.rados.0.smithi089.stdout:1792: finishing write tid 3 to smithi08937929-16 2024-04-22T18:41:14.837 INFO:tasks.rados.rados.0.smithi089.stdout:1792: finishing write tid 4 to smithi08937929-16 2024-04-22T18:41:14.837 INFO:tasks.rados.rados.0.smithi089.stdout:1792: oid 16 updating version 0 to 143 2024-04-22T18:41:14.837 INFO:tasks.rados.rados.0.smithi089.stdout:1792: oid 16 updating version 143 to 144 2024-04-22T18:41:14.837 INFO:tasks.rados.rados.0.smithi089.stdout:1792: oid 16 version 144 is already newer than 142 2024-04-22T18:41:14.837 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 16 v 144 (ObjNum 372 snap 0 seq_num 372) dirty exists 2024-04-22T18:41:14.837 INFO:tasks.rados.rados.0.smithi089.stdout:1792: left oid 16 (ObjNum 372 snap 0 seq_num 372) 2024-04-22T18:41:14.837 INFO:tasks.rados.rados.0.smithi089.stdout:1793: got (0) Success 2024-04-22T18:41:14.837 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 40 v 65 (ObjNum 354 snap 0 seq_num 354) dirty exists 2024-04-22T18:41:14.837 INFO:tasks.rados.rados.0.smithi089.stdout:1794: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:41:14.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:14.860 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:14.860 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:14.876 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:14.876 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:14.880 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:14.881+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:14.892 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:14.892 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:14.908 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:14.908 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:14.968 INFO:teuthology.orchestra.run.smithi089.stderr:set pool 4 pg_num to 26 2024-04-22T18:41:14.973 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:14.973+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:14.982 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:14.982 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:14.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:15.064 INFO:tasks.rados.rados.0.smithi089.stdout:1795: finishing set_chunk to oid 10 2024-04-22T18:41:15.064 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 10 set_chunk returned error code -95 offset: 1276928 length: 1517056 2024-04-22T18:41:15.064 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 0 (ObjNum 336 snap 0 seq_num 336) dirty exists 2024-04-22T18:41:15.064 INFO:tasks.rados.rados.0.smithi089.stdout:1796: finishing set_chunk to oid 33 2024-04-22T18:41:15.064 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 33 set_chunk returned error code -95 offset: 1400832 length: 1824768 2024-04-22T18:41:15.064 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 0 (ObjNum 345 snap 0 seq_num 345) dirty exists 2024-04-22T18:41:15.064 INFO:tasks.rados.rados.0.smithi089.stdout:1798: got (22) Invalid argument 2024-04-22T18:41:15.064 INFO:tasks.rados.rados.0.smithi089.stdout:1797: expect (ObjNum 366 snap 0 seq_num 366) 2024-04-22T18:41:15.073 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:15.074+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:15.093 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:15.093 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:15.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:15.185 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:15.186+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:15.213 INFO:tasks.rados.rados.0.smithi089.stdout:1792: done (7 left) 2024-04-22T18:41:15.213 INFO:tasks.rados.rados.0.smithi089.stdout:1793: done (6 left) 2024-04-22T18:41:15.213 INFO:tasks.rados.rados.0.smithi089.stdout:1794: done (5 left) 2024-04-22T18:41:15.213 INFO:tasks.rados.rados.0.smithi089.stdout:1795: done (4 left) 2024-04-22T18:41:15.213 INFO:tasks.rados.rados.0.smithi089.stdout:1796: done (3 left) 2024-04-22T18:41:15.213 INFO:tasks.rados.rados.0.smithi089.stdout:1797: done (2 left) 2024-04-22T18:41:15.214 INFO:tasks.rados.rados.0.smithi089.stdout:1798: done (1 left) 2024-04-22T18:41:15.214 INFO:tasks.rados.rados.0.smithi089.stdout:1799: delete oid 15 current snap is 0 2024-04-22T18:41:15.243 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:15.243 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:15.252 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:15.269 INFO:tasks.rados.rados.0.smithi089.stdout:1791: finishing write tid 2 to smithi08937929-32 2024-04-22T18:41:15.269 INFO:tasks.rados.rados.0.smithi089.stdout:1791: finishing write tid 3 to smithi08937929-32 2024-04-22T18:41:15.275 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:15.276+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:15.314 INFO:tasks.rados.rados.0.smithi089.stdout:1799: done (1 left) 2024-04-22T18:41:15.315 INFO:tasks.rados.rados.0.smithi089.stdout:1800: write (excl) oid 50 current snap is 0 2024-04-22T18:41:15.315 INFO:tasks.rados.rados.0.smithi089.stdout:1800: seq_num 374 ranges {685818=644423,2069089=216729} 2024-04-22T18:41:15.321 INFO:tasks.rados.rados.0.smithi089.stdout:1800: writing smithi08937929-50 from 685818 to 1330241 tid 1 2024-04-22T18:41:15.326 INFO:tasks.rados.rados.0.smithi089.stdout:1800: writing smithi08937929-50 from 2069089 to 2285818 tid 2 2024-04-22T18:41:15.327 INFO:tasks.rados.rados.0.smithi089.stdout:1801: write (excl) oid 5 current snap is 0 2024-04-22T18:41:15.327 INFO:tasks.rados.rados.0.smithi089.stdout:1801: seq_num 375 ranges {702625=719672,2127571=175054} 2024-04-22T18:41:15.334 INFO:tasks.rados.rados.0.smithi089.stdout:1801: writing smithi08937929-5 from 702625 to 1422297 tid 1 2024-04-22T18:41:15.338 INFO:tasks.rados.rados.0.smithi089.stdout:1801: writing smithi08937929-5 from 2127571 to 2302625 tid 2 2024-04-22T18:41:15.338 INFO:tasks.rados.rados.0.smithi089.stdout:1802: write oid 37 current snap is 0 2024-04-22T18:41:15.338 INFO:tasks.rados.rados.0.smithi089.stdout:1802: seq_num 376 ranges {719432=794921,2186053=133379} 2024-04-22T18:41:15.346 INFO:tasks.rados.rados.0.smithi089.stdout:1802: writing smithi08937929-37 from 719432 to 1514353 tid 1 2024-04-22T18:41:15.351 INFO:tasks.rados.rados.0.smithi089.stdout:1802: writing smithi08937929-37 from 2186053 to 2319432 tid 2 2024-04-22T18:41:15.351 INFO:tasks.rados.rados.0.smithi089.stdout:1803: write oid 15 current snap is 0 2024-04-22T18:41:15.351 INFO:tasks.rados.rados.0.smithi089.stdout:1803: seq_num 377 ranges {736239=470170,1728182=571782} 2024-04-22T18:41:15.357 INFO:tasks.rados.rados.0.smithi089.stdout:1803: writing smithi08937929-15 from 736239 to 1206409 tid 1 2024-04-22T18:41:15.363 INFO:tasks.rados.rados.0.smithi089.stdout:1803: writing smithi08937929-15 from 1728182 to 2299964 tid 2 2024-04-22T18:41:15.363 INFO:tasks.rados.rados.0.smithi089.stdout:1804: set_chunk oid 27 target oid 2024-04-22T18:41:15.376 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:15.377+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:15.396 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:15.396 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:15.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:15.477 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:15.478+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:15.547 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:15.548 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:15.557 INFO:tasks.rados.rados.0.smithi089.stdout:1804: set_chunk oid 27 offset: 119296 length: 410112 target oid f12f4d65306f82c339ed0ee26ffbd661f6b3146cc20b41e6c9b1a00878cfd709 offset: 0 snap -1 2024-04-22T18:41:15.557 INFO:tasks.rados.rados.0.smithi089.stdout:1791: finishing write tid 4 to smithi08937929-32 2024-04-22T18:41:15.557 INFO:tasks.rados.rados.0.smithi089.stdout:1791: oid 32 updating version 0 to 95 2024-04-22T18:41:15.557 INFO:tasks.rados.rados.0.smithi089.stdout:1791: oid 32 version 95 is already newer than 94 2024-04-22T18:41:15.557 INFO:tasks.rados.rados.0.smithi089.stdout:1791: oid 32 updating version 95 to 96 2024-04-22T18:41:15.557 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 96 (ObjNum 371 snap 0 seq_num 371) dirty exists 2024-04-22T18:41:15.557 INFO:tasks.rados.rados.0.smithi089.stdout:1791: left oid 32 (ObjNum 371 snap 0 seq_num 371) 2024-04-22T18:41:15.557 INFO:tasks.rados.rados.0.smithi089.stdout:1805: read oid 2 snap -1 2024-04-22T18:41:15.557 INFO:tasks.rados.rados.0.smithi089.stdout:1805: expect deleted 2024-04-22T18:41:15.557 INFO:tasks.rados.rados.0.smithi089.stdout:1791: done (6 left) 2024-04-22T18:41:15.557 INFO:tasks.rados.rados.0.smithi089.stdout:1806: tier_evict oid 39 2024-04-22T18:41:15.557 INFO:tasks.rados.rados.0.smithi089.stdout:1806: tier_evict oid 39 snap -1 2024-04-22T18:41:15.557 INFO:tasks.rados.rados.0.smithi089.stdout:1800: finishing write tid 1 to smithi08937929-50 2024-04-22T18:41:15.558 INFO:tasks.rados.rados.0.smithi089.stdout:1800: finishing write tid 2 to smithi08937929-50 2024-04-22T18:41:15.558 INFO:tasks.rados.rados.0.smithi089.stdout:1800: finishing write tid 3 to smithi08937929-50 2024-04-22T18:41:15.558 INFO:tasks.rados.rados.0.smithi089.stdout:1800: finishing write tid 4 to smithi08937929-50 2024-04-22T18:41:15.558 INFO:tasks.rados.rados.0.smithi089.stdout:1800: oid 50 updating version 0 to 172 2024-04-22T18:41:15.558 INFO:tasks.rados.rados.0.smithi089.stdout:1800: oid 50 updating version 172 to 173 2024-04-22T18:41:15.558 INFO:tasks.rados.rados.0.smithi089.stdout:1800: oid 50 updating version 173 to 174 2024-04-22T18:41:15.558 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 174 (ObjNum 374 snap 0 seq_num 374) dirty exists 2024-04-22T18:41:15.558 INFO:tasks.rados.rados.0.smithi089.stdout:1800: left oid 50 (ObjNum 374 snap 0 seq_num 374) 2024-04-22T18:41:15.558 INFO:tasks.rados.rados.0.smithi089.stdout:1800: done (6 left) 2024-04-22T18:41:15.558 INFO:tasks.rados.rados.0.smithi089.stdout:1807: set_chunk oid 32 target oid 2024-04-22T18:41:15.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:15.577 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:15.578+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:15.663 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:15.663 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:15.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:15.678 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:15.679+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:15.775 INFO:tasks.rados.rados.0.smithi089.stdout:1807: set_chunk oid 32 offset: 312832 length: 1740288 target oid 74d959b5b495b676ec6d6ff3654e7416b4f5a0fc7763cad1d3c337fa80646641 offset: 0 snap -1 2024-04-22T18:41:15.775 INFO:tasks.rados.rados.0.smithi089.stdout:1808: set_chunk oid 44 target oid 2024-04-22T18:41:15.777 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:15.777 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:15.779 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:15.780+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:15.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:15.788 INFO:tasks.rados.rados.0.smithi089.stdout:1808: set_chunk oid 44 offset: 333312 length: 2080768 target oid 24c89265a3e3f2ada3c4c7f41c8ddcfb9ceb223f53bf4cabe4a1b0e31b1dd1d3 offset: 0 snap -1 2024-04-22T18:41:15.788 INFO:tasks.rados.rados.0.smithi089.stdout:1809: set_chunk oid 28 target oid 2024-04-22T18:41:15.880 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:15.881+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:15.929 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:15.930 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:15.938 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:15.976 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:15.976+0000 7f3093031640 -1 osd.1 139 handle_osd_map injecting map crc failure 2024-04-22T18:41:15.981 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:15.981+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:15.992 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:41:15.995 INFO:tasks.rados.rados.0.smithi089.stdout:1809: set_chunk oid 28 offset: 1609728 length: 1055744 target oid abc40dada9050c3185320a2227cac3e05d2253f144d6d1e31799d010cd3e441e offset: 0 snap -1 2024-04-22T18:41:15.995 INFO:tasks.rados.rados.0.smithi089.stdout:1810: read oid 42 snap -1 2024-04-22T18:41:15.995 INFO:tasks.rados.rados.0.smithi089.stdout:1810: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:41:15.995 INFO:tasks.rados.rados.0.smithi089.stdout:1811: write oid 30 current snap is 0 2024-04-22T18:41:15.995 INFO:tasks.rados.rados.0.smithi089.stdout:1801: finishing write tid 1 to smithi08937929-5 2024-04-22T18:41:15.995 INFO:tasks.rados.rados.0.smithi089.stdout:1811: seq_num 378 ranges {753046=545419,1786664=431793} 2024-04-22T18:41:16.001 INFO:tasks.rados.rados.0.smithi089.stdout:1811: writing smithi08937929-30 from 753046 to 1298465 tid 1 2024-04-22T18:41:16.008 INFO:tasks.rados.rados.0.smithi089.stdout:1811: writing smithi08937929-30 from 1786664 to 2218457 tid 2 2024-04-22T18:41:16.008 INFO:tasks.rados.rados.0.smithi089.stdout:1812: tier_evict oid 40 2024-04-22T18:41:16.008 INFO:tasks.rados.rados.0.smithi089.stdout:1812: tier_evict oid 40 snap -1 2024-04-22T18:41:16.008 INFO:tasks.rados.rados.0.smithi089.stdout:1801: finishing write tid 2 to smithi08937929-5 2024-04-22T18:41:16.008 INFO:tasks.rados.rados.0.smithi089.stdout:1801: finishing write tid 3 to smithi08937929-5 2024-04-22T18:41:16.009 INFO:tasks.rados.rados.0.smithi089.stdout:1813: delete oid 12 current snap is 0 2024-04-22T18:41:16.009 INFO:tasks.rados.rados.0.smithi089.stdout:1801: finishing write tid 4 to smithi08937929-5 2024-04-22T18:41:16.009 INFO:tasks.rados.rados.0.smithi089.stdout:1801: oid 5 updating version 0 to 177 2024-04-22T18:41:16.009 INFO:tasks.rados.rados.0.smithi089.stdout:1801: oid 5 version 177 is already newer than 175 2024-04-22T18:41:16.009 INFO:tasks.rados.rados.0.smithi089.stdout:1801: oid 5 version 177 is already newer than 176 2024-04-22T18:41:16.009 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 177 (ObjNum 375 snap 0 seq_num 375) dirty exists 2024-04-22T18:41:16.009 INFO:tasks.rados.rados.0.smithi089.stdout:1801: left oid 5 (ObjNum 375 snap 0 seq_num 375) 2024-04-22T18:41:16.009 INFO:tasks.rados.rados.0.smithi089.stdout:1802: finishing write tid 1 to smithi08937929-37 2024-04-22T18:41:16.009 INFO:tasks.rados.rados.0.smithi089.stdout:1802: finishing write tid 2 to smithi08937929-37 2024-04-22T18:41:16.009 INFO:tasks.rados.rados.0.smithi089.stdout:1802: finishing write tid 3 to smithi08937929-37 2024-04-22T18:41:16.009 INFO:tasks.rados.rados.0.smithi089.stdout:1802: finishing write tid 4 to smithi08937929-37 2024-04-22T18:41:16.009 INFO:tasks.rados.rados.0.smithi089.stdout:1802: oid 37 updating version 0 to 145 2024-04-22T18:41:16.009 INFO:tasks.rados.rados.0.smithi089.stdout:1802: oid 37 updating version 145 to 146 2024-04-22T18:41:16.009 INFO:tasks.rados.rados.0.smithi089.stdout:1802: oid 37 updating version 146 to 147 2024-04-22T18:41:16.009 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 147 (ObjNum 376 snap 0 seq_num 376) dirty exists 2024-04-22T18:41:16.010 INFO:tasks.rados.rados.0.smithi089.stdout:1802: left oid 37 (ObjNum 376 snap 0 seq_num 376) 2024-04-22T18:41:16.010 INFO:tasks.rados.rados.0.smithi089.stdout:1803: finishing write tid 1 to smithi08937929-15 2024-04-22T18:41:16.010 INFO:tasks.rados.rados.0.smithi089.stdout:1803: finishing write tid 2 to smithi08937929-15 2024-04-22T18:41:16.010 INFO:tasks.rados.rados.0.smithi089.stdout:1804: finishing set_chunk to oid 27 2024-04-22T18:41:16.010 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 119296 length: 410112 2024-04-22T18:41:16.010 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 0 (ObjNum 287 snap 0 seq_num 287) dirty exists 2024-04-22T18:41:16.010 INFO:tasks.rados.rados.0.smithi089.stdout:1806: got (22) Invalid argument 2024-04-22T18:41:16.010 INFO:tasks.rados.rados.0.smithi089.stdout:1803: finishing write tid 3 to smithi08937929-15 2024-04-22T18:41:16.010 INFO:tasks.rados.rados.0.smithi089.stdout:1803: finishing write tid 4 to smithi08937929-15 2024-04-22T18:41:16.010 INFO:tasks.rados.rados.0.smithi089.stdout:1803: oid 15 updating version 0 to 113 2024-04-22T18:41:16.010 INFO:tasks.rados.rados.0.smithi089.stdout:1803: oid 15 updating version 113 to 114 2024-04-22T18:41:16.010 INFO:tasks.rados.rados.0.smithi089.stdout:1803: oid 15 updating version 114 to 115 2024-04-22T18:41:16.010 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 115 (ObjNum 377 snap 0 seq_num 377) dirty exists 2024-04-22T18:41:16.010 INFO:tasks.rados.rados.0.smithi089.stdout:1803: left oid 15 (ObjNum 377 snap 0 seq_num 377) 2024-04-22T18:41:16.011 INFO:tasks.rados.rados.0.smithi089.stdout:1807: finishing set_chunk to oid 32 2024-04-22T18:41:16.011 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 96 (ObjNum 371 snap 0 seq_num 371) dirty exists 2024-04-22T18:41:16.011 INFO:tasks.rados.rados.0.smithi089.stdout:1808: finishing set_chunk to oid 44 2024-04-22T18:41:16.011 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 44 set_chunk returned error code -95 offset: 333312 length: 2080768 2024-04-22T18:41:16.011 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 0 (ObjNum 187 snap 0 seq_num 187) dirty exists 2024-04-22T18:41:16.080 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:16.080 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:16.082 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:16.082+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:16.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:16.095 INFO:tasks.rados.rados.0.smithi089.stdout:1810: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:41:16.182 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:16.183+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:16.192 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:16.192 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:16.202 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:16.283 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:16.283+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:16.296 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:16.296+0000 7f1e2dd22640 -1 osd.2 139 handle_osd_map injecting map crc failure 2024-04-22T18:41:16.307 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:16.307 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:16.316 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:16.324 INFO:tasks.rados.rados.0.smithi089.stdout:1812: got (22) Invalid argument 2024-04-22T18:41:16.450 INFO:tasks.rados.rados.0.smithi089.stdout:1801: done (12 left) 2024-04-22T18:41:16.451 INFO:tasks.rados.rados.0.smithi089.stdout:1802: done (11 left) 2024-04-22T18:41:16.451 INFO:tasks.rados.rados.0.smithi089.stdout:1803: done (10 left) 2024-04-22T18:41:16.451 INFO:tasks.rados.rados.0.smithi089.stdout:1804: done (9 left) 2024-04-22T18:41:16.451 INFO:tasks.rados.rados.0.smithi089.stdout:1805: done (8 left) 2024-04-22T18:41:16.451 INFO:tasks.rados.rados.0.smithi089.stdout:1806: done (7 left) 2024-04-22T18:41:16.451 INFO:tasks.rados.rados.0.smithi089.stdout:1807: done (6 left) 2024-04-22T18:41:16.451 INFO:tasks.rados.rados.0.smithi089.stdout:1808: done (5 left) 2024-04-22T18:41:16.451 INFO:tasks.rados.rados.0.smithi089.stdout:1810: done (4 left) 2024-04-22T18:41:16.451 INFO:tasks.rados.rados.0.smithi089.stdout:1812: done (3 left) 2024-04-22T18:41:16.451 INFO:tasks.rados.rados.0.smithi089.stdout:1813: done (2 left) 2024-04-22T18:41:16.451 INFO:tasks.rados.rados.0.smithi089.stdout:1814: read oid 34 snap -1 2024-04-22T18:41:16.451 INFO:tasks.rados.rados.0.smithi089.stdout:1814: expect (ObjNum 277 snap 0 seq_num 277) 2024-04-22T18:41:16.451 INFO:tasks.rados.rados.0.smithi089.stdout:1815: read oid 40 snap -1 2024-04-22T18:41:16.451 INFO:tasks.rados.rados.0.smithi089.stdout:1815: expect (ObjNum 354 snap 0 seq_num 354) 2024-04-22T18:41:16.452 INFO:tasks.rados.rados.0.smithi089.stdout:1816: tier_flush oid 21 2024-04-22T18:41:16.452 INFO:tasks.rados.rados.0.smithi089.stdout:1816: tier_flush oid 21 snap -1 2024-04-22T18:41:16.452 INFO:tasks.rados.rados.0.smithi089.stdout:1817: delete oid 36 current snap is 0 2024-04-22T18:41:16.453 INFO:tasks.rados.rados.0.smithi089.stdout:1815: expect (ObjNum 354 snap 0 seq_num 354) 2024-04-22T18:41:16.454 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:16.454 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:16.454 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:16.455 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:16.453+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:16.519 INFO:tasks.rados.rados.0.smithi089.stdout:1811: finishing write tid 1 to smithi08937929-30 2024-04-22T18:41:16.519 INFO:tasks.rados.rados.0.smithi089.stdout:1811: finishing write tid 2 to smithi08937929-30 2024-04-22T18:41:16.540 INFO:tasks.rados.rados.0.smithi089.stdout:1815: done (5 left) 2024-04-22T18:41:16.540 INFO:tasks.rados.rados.0.smithi089.stdout:1817: done (4 left) 2024-04-22T18:41:16.540 INFO:tasks.rados.rados.0.smithi089.stdout:1818: read oid 29 snap -1 2024-04-22T18:41:16.540 INFO:tasks.rados.rados.0.smithi089.stdout:1818: expect (ObjNum 295 snap 0 seq_num 295) 2024-04-22T18:41:16.540 INFO:tasks.rados.rados.0.smithi089.stdout:1819: tier_promote oid 32 2024-04-22T18:41:16.540 INFO:tasks.rados.rados.0.smithi089.stdout:1820: read oid 11 snap -1 2024-04-22T18:41:16.541 INFO:tasks.rados.rados.0.smithi089.stdout:1820: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:41:16.541 INFO:tasks.rados.rados.0.smithi089.stdout:1821: read oid 17 snap -1 2024-04-22T18:41:16.541 INFO:tasks.rados.rados.0.smithi089.stdout:1821: expect (ObjNum 366 snap 0 seq_num 366) 2024-04-22T18:41:16.541 INFO:tasks.rados.rados.0.smithi089.stdout:1822: delete oid 10 current snap is 0 2024-04-22T18:41:16.554 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:16.555+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:16.596 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:16.596 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:16.600 INFO:tasks.rados.rados.0.smithi089.stdout:1811: finishing write tid 3 to smithi08937929-30 2024-04-22T18:41:16.600 INFO:tasks.rados.rados.0.smithi089.stdout:1811: finishing write tid 4 to smithi08937929-30 2024-04-22T18:41:16.600 INFO:tasks.rados.rados.0.smithi089.stdout:1811: oid 30 updating version 0 to 67 2024-04-22T18:41:16.600 INFO:tasks.rados.rados.0.smithi089.stdout:1811: oid 30 updating version 67 to 68 2024-04-22T18:41:16.600 INFO:tasks.rados.rados.0.smithi089.stdout:1811: oid 30 updating version 68 to 69 2024-04-22T18:41:16.600 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 69 (ObjNum 378 snap 0 seq_num 378) dirty exists 2024-04-22T18:41:16.600 INFO:tasks.rados.rados.0.smithi089.stdout:1811: left oid 30 (ObjNum 378 snap 0 seq_num 378) 2024-04-22T18:41:16.600 INFO:tasks.rados.rados.0.smithi089.stdout:1809: finishing set_chunk to oid 28 2024-04-22T18:41:16.600 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 165 (ObjNum 355 snap 0 seq_num 355) dirty exists 2024-04-22T18:41:16.605 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:16.644 INFO:tasks.rados.rados.0.smithi089.stdout:1814: expect (ObjNum 277 snap 0 seq_num 277) 2024-04-22T18:41:16.653 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:41:16.654 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:16.655+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:16.753 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:16.753 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:16.755 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:16.755+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:16.762 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:16.842 INFO:tasks.rados.rados.0.smithi089.stdout:1818: expect (ObjNum 295 snap 0 seq_num 295) 2024-04-22T18:41:16.856 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:16.857+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:16.907 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:16.907 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:16.917 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:16.919 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:41:16.958 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:16.958+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:17.009 INFO:tasks.rados.rados.0.smithi089.stdout:1820: expect (ObjNum 196 snap 0 seq_num 196) 2024-04-22T18:41:17.057 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:17.058+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:17.064 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:17.064 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:17.073 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:17.079 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:17.079 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:17.097 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:17.097 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:17.116 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:17.116 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:17.158 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:17.160+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:17.183 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:17.183 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:17.184 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:17.192 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:17.219 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:17.219 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:17.221 INFO:tasks.rados.rados.0.smithi089.stdout:1819: got (0) Success 2024-04-22T18:41:17.221 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 32 v 96 (ObjNum 371 snap 0 seq_num 371) dirty exists 2024-04-22T18:41:17.222 INFO:tasks.rados.rados.0.smithi089.stdout:1816: got (0) Success 2024-04-22T18:41:17.222 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 0 (ObjNum 349 snap 0 seq_num 349) dirty exists 2024-04-22T18:41:17.222 INFO:tasks.rados.rados.0.smithi089.stdout:1809: done (8 left) 2024-04-22T18:41:17.222 INFO:tasks.rados.rados.0.smithi089.stdout:1811: done (7 left) 2024-04-22T18:41:17.222 INFO:tasks.rados.rados.0.smithi089.stdout:1814: done (6 left) 2024-04-22T18:41:17.222 INFO:tasks.rados.rados.0.smithi089.stdout:1816: done (5 left) 2024-04-22T18:41:17.222 INFO:tasks.rados.rados.0.smithi089.stdout:1818: done (4 left) 2024-04-22T18:41:17.222 INFO:tasks.rados.rados.0.smithi089.stdout:1819: done (3 left) 2024-04-22T18:41:17.222 INFO:tasks.rados.rados.0.smithi089.stdout:1820: done (2 left) 2024-04-22T18:41:17.222 INFO:tasks.rados.rados.0.smithi089.stdout:1822: done (1 left) 2024-04-22T18:41:17.222 INFO:tasks.rados.rados.0.smithi089.stdout:1823: read oid 7 snap -1 2024-04-22T18:41:17.223 INFO:tasks.rados.rados.0.smithi089.stdout:1823: expect (ObjNum 346 snap 0 seq_num 346) 2024-04-22T18:41:17.223 INFO:tasks.rados.rados.0.smithi089.stdout:1824: read oid 12 snap -1 2024-04-22T18:41:17.223 INFO:tasks.rados.rados.0.smithi089.stdout:1824: expect deleted 2024-04-22T18:41:17.223 INFO:tasks.rados.rados.0.smithi089.stdout:1825: tier_promote oid 36 2024-04-22T18:41:17.223 INFO:tasks.rados.rados.0.smithi089.stdout:1826: read oid 8 snap -1 2024-04-22T18:41:17.223 INFO:tasks.rados.rados.0.smithi089.stdout:1826: expect (ObjNum 365 snap 0 seq_num 365) 2024-04-22T18:41:17.223 INFO:tasks.rados.rados.0.smithi089.stdout:1827: read oid 37 snap -1 2024-04-22T18:41:17.223 INFO:tasks.rados.rados.0.smithi089.stdout:1827: expect (ObjNum 376 snap 0 seq_num 376) 2024-04-22T18:41:17.223 INFO:tasks.rados.rados.0.smithi089.stdout:1828: read oid 31 snap -1 2024-04-22T18:41:17.223 INFO:tasks.rados.rados.0.smithi089.stdout:1828: expect deleted 2024-04-22T18:41:17.224 INFO:tasks.rados.rados.0.smithi089.stdout:1829: delete oid 45 current snap is 0 2024-04-22T18:41:17.236 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:17.236 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:17.253 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:17.253 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:17.259 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:17.259+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:17.270 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:17.270 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:17.287 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:17.287 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:17.322 INFO:tasks.rados.rados.0.smithi089.stdout:1827: expect (ObjNum 376 snap 0 seq_num 376) 2024-04-22T18:41:17.325 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:17.325 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:17.325 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:17.325 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:17.325 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:17.335 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:17.360 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:17.361+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:17.456 INFO:tasks.rados.rados.0.smithi089.stdout:1823: expect (ObjNum 346 snap 0 seq_num 346) 2024-04-22T18:41:17.461 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:17.463+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:17.479 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:41:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 133.01720381199999, 2024-04-22T18:41:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:41:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:41:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:17.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:41:17.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:41:17.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:41:17.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:41:17.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:41:17.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:41:17.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:41:17.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:41:17.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:41:17.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:41:17.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:17.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:41:17.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:41:17.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:41:17.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:41:17.483 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.483 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.483 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.483 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:41:17.483 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:17.484 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 129.14052999, 2024-04-22T18:41:17.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:41:17.484 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.484 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:17.484 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:17.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:17.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:41:17.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:17.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:41:17.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:41:17.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:41:17.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:41:17.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:41:17.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:41:17.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:41:17.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:41:17.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:41:17.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:41:17.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:17.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:41:17.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:41:17.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:41:17.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:41:17.487 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.487 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.487 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.488 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:41:17.488 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:17.488 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.935726466999995, 2024-04-22T18:41:17.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:41:17.488 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.488 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:17.488 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:17.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:17.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:41:17.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:41:17.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:41:17.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:41:17.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:41:17.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:41:17.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:41:17.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:41:17.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:41:17.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:41:17.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:41:17.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:41:17.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:17.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:41:17.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:41:17.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:41:17.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:41:17.492 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.492 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.492 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.492 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:161 3.0 e76/63 3:3638311b:::f282598a146af3da1c05c8882972fba387afd249b39f6eeb6e6052a5de37fd2d:head v 76'157, mlcod=76'154)", 2024-04-22T18:41:17.492 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:41:17.492 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 70.087007037000006, 2024-04-22T18:41:17.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104845172, 2024-04-22T18:41:17.492 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.492 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:17.492 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:41:17.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:41:17.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391368+0000", 2024-04-22T18:41:17.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:17.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391440+0000", 2024-04-22T18:41:17.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2045999999999994e-05 2024-04-22T18:41:17.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391448+0000", 2024-04-22T18:41:17.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8189999999999992e-06 2024-04-22T18:41:17.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391453+0000", 2024-04-22T18:41:17.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8379999999999998e-06 2024-04-22T18:41:17.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391839+0000", 2024-04-22T18:41:17.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000385325 2024-04-22T18:41:17.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391854+0000", 2024-04-22T18:41:17.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5006e-05 2024-04-22T18:41:17.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496191+0000", 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104336792 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496214+0000", 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3079999999999999e-05 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.0.13:167 3.0 e76/63 3:384c1ce2:::81dc8c58ebe106ae6fa9feadc06291c645d9079ad84b78214d4e6440fbc5def8:head v 76'160, mlcod=76'154)", 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 70.086720080999996, 2024-04-22T18:41:17.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10487859400000001, 2024-04-22T18:41:17.497 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.497 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:17.497 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:41:17.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:41:17.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391655+0000", 2024-04-22T18:41:17.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:17.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391707+0000", 2024-04-22T18:41:17.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2343999999999999e-05 2024-04-22T18:41:17.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391714+0000", 2024-04-22T18:41:17.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.596e-06 2024-04-22T18:41:17.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.391719+0000", 2024-04-22T18:41:17.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9130000000000002e-06 2024-04-22T18:41:17.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493154+0000", 2024-04-22T18:41:17.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101435256 2024-04-22T18:41:17.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.493177+0000", 2024-04-22T18:41:17.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2796e-05 2024-04-22T18:41:17.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:17.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496509+0000", 2024-04-22T18:41:17.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033320139999999999 2024-04-22T18:41:17.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.496534+0000", 2024-04-22T18:41:17.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5265999999999998e-05 2024-04-22T18:41:17.500 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.500 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.500 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.501 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3451 4.d 4:bee1184f:::smithi08937929-7:head [read 0~3613505,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:17.501 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:17.501 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.551196140000002, 2024-04-22T18:41:17.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20164568099999999, 2024-04-22T18:41:17.501 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.501 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:17.501 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:17.501 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:17.501 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:17.501 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3451 2024-04-22T18:41:17.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.501 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:17.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:17.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:17.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:17.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927183+0000", 2024-04-22T18:41:17.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1110000000000001e-06 2024-04-22T18:41:17.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927188+0000", 2024-04-22T18:41:17.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1569999999999999e-06 2024-04-22T18:41:17.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927192+0000", 2024-04-22T18:41:17.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4710000000000001e-06 2024-04-22T18:41:17.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027382+0000", 2024-04-22T18:41:17.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100189341 2024-04-22T18:41:17.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:41:17.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027387+0000", 2024-04-22T18:41:17.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8400000000000002e-06 2024-04-22T18:41:17.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127638+0000", 2024-04-22T18:41:17.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10025102499999999 2024-04-22T18:41:17.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127740+0000", 2024-04-22T18:41:17.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102749 2024-04-22T18:41:17.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.128825+0000", 2024-04-22T18:41:17.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001084593 2024-04-22T18:41:17.505 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.505 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.505 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.505 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:151 3.0 e114/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 114'213, mlcod=110'210)", 2024-04-22T18:41:17.505 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:17.506 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.960728057000001, 2024-04-22T18:41:17.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20074001299999999, 2024-04-22T18:41:17.506 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.506 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:17.506 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:17.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:17.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517640+0000", 2024-04-22T18:41:17.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999928 2024-04-22T18:41:17.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517652+0000", 2024-04-22T18:41:17.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1824000000000001e-05 2024-04-22T18:41:17.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517660+0000", 2024-04-22T18:41:17.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1419999999999993e-06 2024-04-22T18:41:17.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617729+0000", 2024-04-22T18:41:17.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068723 2024-04-22T18:41:17.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617770+0000", 2024-04-22T18:41:17.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1391999999999999e-05 2024-04-22T18:41:17.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617793+0000", 2024-04-22T18:41:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2807999999999999e-05 2024-04-22T18:41:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718361+0000", 2024-04-22T18:41:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100567473 2024-04-22T18:41:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718387+0000", 2024-04-22T18:41:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6826e-05 2024-04-22T18:41:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.510 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3862 4.d 4:bee1184f:::smithi08937929-7:head [write 3301257~278667] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:17.510 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:17.510 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.215825787, 2024-04-22T18:41:17.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301875958, 2024-04-22T18:41:17.510 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.510 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:17.510 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:17.510 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:17.510 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:17.510 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3862 2024-04-22T18:41:17.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.510 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:17.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:17.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262548+0000", 2024-04-22T18:41:17.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:17.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262622+0000", 2024-04-22T18:41:17.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3747999999999999e-05 2024-04-22T18:41:17.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262632+0000", 2024-04-22T18:41:17.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0767e-05 2024-04-22T18:41:17.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362705+0000", 2024-04-22T18:41:17.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073157 2024-04-22T18:41:17.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:41:17.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2846999999999997e-05 2024-04-22T18:41:17.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362853+0000", 2024-04-22T18:41:17.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4498000000000002e-05 2024-04-22T18:41:17.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:17.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362992+0000", 2024-04-22T18:41:17.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013879699999999999 2024-04-22T18:41:17.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:17.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.366534+0000", 2024-04-22T18:41:17.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035427969999999999 2024-04-22T18:41:17.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:17.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564353+0000", 2024-04-22T18:41:17.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.197818893 2024-04-22T18:41:17.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:17.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564385+0000", 2024-04-22T18:41:17.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1717000000000001e-05 2024-04-22T18:41:17.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564426+0000", 2024-04-22T18:41:17.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0654000000000001e-05 2024-04-22T18:41:17.515 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.515 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.515 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.516 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3863 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3579924] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:17.516 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:17.516 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.115658395000001, 2024-04-22T18:41:17.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20206560900000001, 2024-04-22T18:41:17.516 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.516 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:17.516 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:17.516 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:17.516 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:17.516 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3863 2024-04-22T18:41:17.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.516 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:17.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:17.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362715+0000", 2024-04-22T18:41:17.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:17.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362725+0000", 2024-04-22T18:41:17.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.006e-05 2024-04-22T18:41:17.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362732+0000", 2024-04-22T18:41:17.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6780000000000003e-06 2024-04-22T18:41:17.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362737+0000", 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2089999999999996e-06 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462875+0000", 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013834100000001 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462963+0000", 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7792000000000002e-05 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463127+0000", 2024-04-22T18:41:17.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000163688 2024-04-22T18:41:17.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:17.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463692+0000", 2024-04-22T18:41:17.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056466299999999997 2024-04-22T18:41:17.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:17.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564720+0000", 2024-04-22T18:41:17.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10102826199999999 2024-04-22T18:41:17.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:17.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564752+0000", 2024-04-22T18:41:17.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1936000000000001e-05 2024-04-22T18:41:17.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564783+0000", 2024-04-22T18:41:17.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0919e-05 2024-04-22T18:41:17.521 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.521 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.521 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.521 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3864 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:17.521 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:17.521 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.115625010999999, 2024-04-22T18:41:17.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20218423599999999, 2024-04-22T18:41:17.522 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.522 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:17.522 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:17.522 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:17.522 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:17.522 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3864 2024-04-22T18:41:17.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.522 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:17.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:17.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:17.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:17.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362753+0000", 2024-04-22T18:41:17.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6989999999999999e-06 2024-04-22T18:41:17.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:41:17.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:41:17.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462871+0000", 2024-04-22T18:41:17.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100113641 2024-04-22T18:41:17.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463385+0000", 2024-04-22T18:41:17.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051307100000000003 2024-04-22T18:41:17.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463457+0000", 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2778e-05 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564814+0000", 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10135696299999999 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564868+0000", 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3872000000000002e-05 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564935+0000", 2024-04-22T18:41:17.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6545999999999996e-05 2024-04-22T18:41:17.526 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.526 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.526 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.526 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4030 4.d 4:bee1184f:::smithi08937929-7:head [write 615222~472039] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:17.526 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:17.526 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.519176109, 2024-04-22T18:41:17.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109028217, 2024-04-22T18:41:17.526 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.527 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:17.527 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:17.527 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:17.527 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:17.527 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4030 2024-04-22T18:41:17.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.527 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:17.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:17.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959177+0000", 2024-04-22T18:41:17.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999776 2024-04-22T18:41:17.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959331+0000", 2024-04-22T18:41:17.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015334600000000001 2024-04-22T18:41:17.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959358+0000", 2024-04-22T18:41:17.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7220999999999999e-05 2024-04-22T18:41:17.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959371+0000", 2024-04-22T18:41:17.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3489000000000001e-05 2024-04-22T18:41:17.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059523+0000", 2024-04-22T18:41:17.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100152059 2024-04-22T18:41:17.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059588+0000", 2024-04-22T18:41:17.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4493999999999996e-05 2024-04-22T18:41:17.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:17.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059694+0000", 2024-04-22T18:41:17.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010584199999999999 2024-04-22T18:41:17.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:17.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068082+0000", 2024-04-22T18:41:17.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083880589999999998 2024-04-22T18:41:17.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:17.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068138+0000", 2024-04-22T18:41:17.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6196999999999997e-05 2024-04-22T18:41:17.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:17.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068185+0000", 2024-04-22T18:41:17.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7318000000000003e-05 2024-04-22T18:41:17.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068228+0000", 2024-04-22T18:41:17.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2376999999999998e-05 2024-04-22T18:41:17.531 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.531 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.531 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4031 4.d 4:bee1184f:::smithi08937929-7:head [write 1580652~489442] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.513875117000001, 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20571598299999999, 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4031 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:17.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:17.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964480+0000", 2024-04-22T18:41:17.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999795 2024-04-22T18:41:17.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964628+0000", 2024-04-22T18:41:17.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148109 2024-04-22T18:41:17.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964651+0000", 2024-04-22T18:41:17.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3215999999999999e-05 2024-04-22T18:41:17.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964664+0000", 2024-04-22T18:41:17.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2673e-05 2024-04-22T18:41:17.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061135+0000", 2024-04-22T18:41:17.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096470822999999997 2024-04-22T18:41:17.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061228+0000", 2024-04-22T18:41:17.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3459000000000001e-05 2024-04-22T18:41:17.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:17.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061376+0000", 2024-04-22T18:41:17.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148128 2024-04-22T18:41:17.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:17.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068085+0000", 2024-04-22T18:41:17.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.006708632 2024-04-22T18:41:17.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:17.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170038+0000", 2024-04-22T18:41:17.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101952716 2024-04-22T18:41:17.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:17.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170173+0000", 2024-04-22T18:41:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013508699999999999 2024-04-22T18:41:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170216+0000", 2024-04-22T18:41:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3498999999999998e-05 2024-04-22T18:41:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4032 4.d 4:bee1184f:::smithi08937929-7:head [write 2768826~626587] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.506670010000001, 2024-04-22T18:41:17.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19941629799999999, 2024-04-22T18:41:17.538 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.538 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:17.538 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:17.538 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:17.538 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:17.538 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4032 2024-04-22T18:41:17.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.538 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:17.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:17.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971703+0000", 2024-04-22T18:41:17.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:17.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971996+0000", 2024-04-22T18:41:17.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029351300000000002 2024-04-22T18:41:17.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972012+0000", 2024-04-22T18:41:17.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5569e-05 2024-04-22T18:41:17.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972019+0000", 2024-04-22T18:41:17.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7950000000000008e-06 2024-04-22T18:41:17.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.063940+0000", 2024-04-22T18:41:17.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091920162 2024-04-22T18:41:17.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064021+0000", 2024-04-22T18:41:17.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1865000000000003e-05 2024-04-22T18:41:17.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:17.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064145+0000", 2024-04-22T18:41:17.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123893 2024-04-22T18:41:17.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:17.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068317+0000", 2024-04-22T18:41:17.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041720799999999999 2024-04-22T18:41:17.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171042+0000", 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102724893 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171085+0000", 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2619e-05 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171122+0000", 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6838999999999997e-05 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.543 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4033 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3815222] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:17.543 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:17.543 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.506346528000002, 2024-04-22T18:41:17.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297586038, 2024-04-22T18:41:17.543 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.543 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:17.543 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:17.543 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:17.543 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:17.543 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4033 2024-04-22T18:41:17.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.543 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:17.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:17.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972028+0000", 2024-04-22T18:41:17.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:17.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972032+0000", 2024-04-22T18:41:17.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8330000000000004e-06 2024-04-22T18:41:17.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972035+0000", 2024-04-22T18:41:17.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:41:17.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972038+0000", 2024-04-22T18:41:17.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9579999999999999e-06 2024-04-22T18:41:17.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067511+0000", 2024-04-22T18:41:17.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.095472308000000006 2024-04-22T18:41:17.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067607+0000", 2024-04-22T18:41:17.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6546999999999996e-05 2024-04-22T18:41:17.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:17.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067776+0000", 2024-04-22T18:41:17.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000169202 2024-04-22T18:41:17.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:17.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068470+0000", 2024-04-22T18:41:17.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069403100000000001 2024-04-22T18:41:17.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:17.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269540+0000", 2024-04-22T18:41:17.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20106934000000001 2024-04-22T18:41:17.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:17.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269574+0000", 2024-04-22T18:41:17.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4332000000000003e-05 2024-04-22T18:41:17.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269615+0000", 2024-04-22T18:41:17.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0988000000000001e-05 2024-04-22T18:41:17.548 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.548 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.548 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.548 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4034 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:17.548 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:17.548 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.506334891000002, 2024-04-22T18:41:17.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39788621000000002, 2024-04-22T18:41:17.549 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.549 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:17.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:17.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:17.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:17.549 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4034 2024-04-22T18:41:17.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.549 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:17.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:17.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972040+0000", 2024-04-22T18:41:17.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:17.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972042+0000", 2024-04-22T18:41:17.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.519e-06 2024-04-22T18:41:17.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972045+0000", 2024-04-22T18:41:17.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:41:17.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972048+0000", 2024-04-22T18:41:17.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.773e-06 2024-04-22T18:41:17.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068266+0000", 2024-04-22T18:41:17.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096218029999999996 2024-04-22T18:41:17.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:17.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068308+0000", 2024-04-22T18:41:17.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.21e-05 2024-04-22T18:41:17.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369703+0000", 2024-04-22T18:41:17.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301395578 2024-04-22T18:41:17.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369802+0000", 2024-04-22T18:41:17.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8788000000000004e-05 2024-04-22T18:41:17.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369927+0000", 2024-04-22T18:41:17.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012478200000000001 2024-04-22T18:41:17.553 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.553 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.553 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.553 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(4.1 124/123 [PushOp(4:83c955bf:::smithi08937929-28:head, version: 115'157, data_included: [1890685~548185,2699879~469889], data_size: 1018074, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:83c955bf:::smithi08937929-28:head@115'157, size: 3630345, copy_subset: [0~3630345], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3630345, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1890685, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-22T18:41:17.553 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:17.553 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.839766783000002, 2024-04-22T18:41:17.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20293218800000001, 2024-04-22T18:41:17.553 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.554 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:17.554 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:17.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:17.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638608+0000", 2024-04-22T18:41:17.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:17.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639519+0000", 2024-04-22T18:41:17.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091196199999999999 2024-04-22T18:41:17.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639537+0000", 2024-04-22T18:41:17.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7455000000000001e-05 2024-04-22T18:41:17.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.739639+0000", 2024-04-22T18:41:17.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100101942 2024-04-22T18:41:17.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840271+0000", 2024-04-22T18:41:17.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10063235299999999 2024-04-22T18:41:17.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840281+0000", 2024-04-22T18:41:17.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6509999999999996e-06 2024-04-22T18:41:17.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.841541+0000", 2024-04-22T18:41:17.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012600230000000001 2024-04-22T18:41:17.557 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.557 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.557 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.557 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4205 4.1 4:83c955bf:::smithi08937929-28:head [read 3065344~859648] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:17.557 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:17.557 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.550844941999999, 2024-04-22T18:41:17.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.303433565, 2024-04-22T18:41:17.557 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.557 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:17.557 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:17.557 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:17.558 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:17.558 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4205 2024-04-22T18:41:17.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.558 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:17.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:17.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927528+0000", 2024-04-22T18:41:17.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:17.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927535+0000", 2024-04-22T18:41:17.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1049999999999997e-06 2024-04-22T18:41:17.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927550+0000", 2024-04-22T18:41:17.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4365000000000001e-05 2024-04-22T18:41:17.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927560+0000", 2024-04-22T18:41:17.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0360000000000001e-05 2024-04-22T18:41:17.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028542+0000", 2024-04-22T18:41:17.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100981837 2024-04-22T18:41:17.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:17.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028551+0000", 2024-04-22T18:41:17.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6260000000000001e-06 2024-04-22T18:41:17.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229700+0000", 2024-04-22T18:41:17.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.201149453 2024-04-22T18:41:17.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229861+0000", 2024-04-22T18:41:17.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160605 2024-04-22T18:41:17.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.230964+0000", 2024-04-22T18:41:17.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00110351 2024-04-22T18:41:17.562 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.562 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.562 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.562 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4258 4.1 4:81200c09:::smithi08937929-23:head [read 0~3697573,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:17.562 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.875931+0000", 2024-04-22T18:41:17.562 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.6024446549999993, 2024-04-22T18:41:17.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105556871, 2024-04-22T18:41:17.562 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.562 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:17.562 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:17.562 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:17.562 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:17.563 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4258 2024-04-22T18:41:17.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.563 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.875931+0000", 2024-04-22T18:41:17.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.875931+0000", 2024-04-22T18:41:17.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.875930+0000", 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.875936+0000", 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0290000000000002e-06 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.875944+0000", 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8609999999999998e-06 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.875950+0000", 2024-04-22T18:41:17.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3670000000000002e-06 2024-04-22T18:41:17.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.980155+0000", 2024-04-22T18:41:17.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104204899 2024-04-22T18:41:17.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.980272+0000", 2024-04-22T18:41:17.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011677500000000001 2024-04-22T18:41:17.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.981488+0000", 2024-04-22T18:41:17.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001215566 2024-04-22T18:41:17.566 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.566 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.566 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.566 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:359 3.0 e133/63 3:351d2474:::ba270d71d9c7ccc8d5c74eded70a7073f0e08c1ce33f7199f7c9db1158888734:head v 133'233, mlcod=132'228)", 2024-04-22T18:41:17.566 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:17.566 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5760505150000004, 2024-04-22T18:41:17.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20122315599999999, 2024-04-22T18:41:17.566 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.566 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:17.567 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:17.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:17.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902324+0000", 2024-04-22T18:41:17.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902347+0000", 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3577000000000001e-05 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902356+0000", 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8179999999999993e-06 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.002447+0000", 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100090255 2024-04-22T18:41:17.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102568+0000", 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100121611 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102592+0000", 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3515000000000001e-05 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103514+0000", 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000922174 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103548+0000", 2024-04-22T18:41:17.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4316999999999997e-05 2024-04-22T18:41:17.570 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.570 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.570 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.570 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:4289 3.0 e134/63 3:2a3d8f37:::9341e24c4a92f7ec719282d2faa5711f9644dee45a94ba5b4c4c1071ddc0e536:head v 134'235, mlcod=133'233)", 2024-04-22T18:41:17.570 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:17.570 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.6314810030000002, 2024-04-22T18:41:17.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10669205499999999, 2024-04-22T18:41:17.570 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:17.570 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846891+0000", 2024-04-22T18:41:17.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:17.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:17.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847884+0000", 2024-04-22T18:41:17.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00099224800000000004 2024-04-22T18:41:17.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:17.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847900+0000", 2024-04-22T18:41:17.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6154000000000002e-05 2024-04-22T18:41:17.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:17.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.947999+0000", 2024-04-22T18:41:17.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009889199999999 2024-04-22T18:41:17.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:17.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948046+0000", 2024-04-22T18:41:17.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7617000000000002e-05 2024-04-22T18:41:17.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:17.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948072+0000", 2024-04-22T18:41:17.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5559999999999999e-05 2024-04-22T18:41:17.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:17.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953539+0000", 2024-04-22T18:41:17.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054668599999999996 2024-04-22T18:41:17.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:17.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:17.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:17.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953587+0000", 2024-04-22T18:41:17.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7781999999999998e-05 2024-04-22T18:41:17.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.574 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:17.574 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:17.574 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:17.576 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:17.576 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:17.575+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:17.650 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:17.651+0000 7f7f44bb3640 -1 osd.4 141 handle_osd_map injecting map crc failure 2024-04-22T18:41:17.668 INFO:tasks.rados.rados.0.smithi089.stdout:1825: got (2) No such file or directory 2024-04-22T18:41:17.668 INFO:tasks.rados.rados.0.smithi089.stdout:1825: got expected ENOENT (src dne) 2024-04-22T18:41:17.668 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 36 v 65 (ObjNum 129 snap 0 seq_num 2409330096) clean dne 2024-04-22T18:41:17.668 INFO:tasks.rados.rados.0.smithi089.stdout:1826: expect (ObjNum 365 snap 0 seq_num 365) 2024-04-22T18:41:17.676 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:17.676+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:17.720 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:17.720 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:17.720 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:17.720 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:17.729 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:17.804 INFO:tasks.rados.rados.0.smithi089.stdout:1823: done (7 left) 2024-04-22T18:41:17.804 INFO:tasks.rados.rados.0.smithi089.stdout:1824: done (6 left) 2024-04-22T18:41:17.804 INFO:tasks.rados.rados.0.smithi089.stdout:1825: done (5 left) 2024-04-22T18:41:17.804 INFO:tasks.rados.rados.0.smithi089.stdout:1826: done (4 left) 2024-04-22T18:41:17.804 INFO:tasks.rados.rados.0.smithi089.stdout:1827: done (3 left) 2024-04-22T18:41:17.804 INFO:tasks.rados.rados.0.smithi089.stdout:1828: done (2 left) 2024-04-22T18:41:17.804 INFO:tasks.rados.rados.0.smithi089.stdout:1829: done (1 left) 2024-04-22T18:41:17.804 INFO:tasks.rados.rados.0.smithi089.stdout:1830: set_chunk oid 7 target oid 2024-04-22T18:41:17.805 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:17.804+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:17.871 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:17.871 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:17.871 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:17.872 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:17.872 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:17.878 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:17.878+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:17.880 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:17.913 INFO:tasks.rados.rados.0.smithi089.stdout:1830: set_chunk oid 7 offset: 2080256 length: 1025536 target oid e613aed6b694cb6f7d04e499edd9fc8dc5435f2ffc56ae051dbc10eda7616c7e offset: 0 snap -1 2024-04-22T18:41:17.914 INFO:tasks.rados.rados.0.smithi089.stdout:1831: set_chunk oid 34 target oid 2024-04-22T18:41:17.979 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:17.980+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:18.020 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:289 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 75.379490782999994, 2024-04-22T18:41:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9176801189999999, 2024-04-22T18:41:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640723+0000", 2024-04-22T18:41:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640732+0000", 2024-04-22T18:41:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5809999999999997e-06 2024-04-22T18:41:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640744+0000", 2024-04-22T18:41:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.261e-05 2024-04-22T18:41:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640753+0000", 2024-04-22T18:41:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7460000000000006e-06 2024-04-22T18:41:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642189+0000", 2024-04-22T18:41:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001435551 2024-04-22T18:41:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642200+0000", 2024-04-22T18:41:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1389999999999999e-05 2024-04-22T18:41:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.847015+0000", 2024-04-22T18:41:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048149160000001 2024-04-22T18:41:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.848217+0000", 2024-04-22T18:41:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012020659999999999 2024-04-22T18:41:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351823+0000", 2024-04-22T18:41:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50360565099999999 2024-04-22T18:41:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351907+0000", 2024-04-22T18:41:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4141999999999995e-05 2024-04-22T18:41:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352051+0000", 2024-04-22T18:41:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014386600000000001 2024-04-22T18:41:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453617+0000", 2024-04-22T18:41:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156649500000001 2024-04-22T18:41:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558297+0000", 2024-04-22T18:41:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10468000399999999 2024-04-22T18:41:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558331+0000", 2024-04-22T18:41:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3518e-05 2024-04-22T18:41:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558405+0000", 2024-04-22T18:41:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4221000000000006e-05 2024-04-22T18:41:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:1 3.2 3:6d90b903:::64b5942574fb36ca1b9a2be186c6b740edf58e989056a2f88b83cd280451a438:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.848189822999998, 2024-04-22T18:41:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.285534422, 2024-04-22T18:41:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172024+0000", 2024-04-22T18:41:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173091+0000", 2024-04-22T18:41:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010677569999999999 2024-04-22T18:41:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173103+0000", 2024-04-22T18:41:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1241e-05 2024-04-22T18:41:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173109+0000", 2024-04-22T18:41:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4300000000000003e-06 2024-04-22T18:41:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173168+0000", 2024-04-22T18:41:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9011000000000001e-05 2024-04-22T18:41:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173177+0000", 2024-04-22T18:41:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-04-22T18:41:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948501+0000", 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77532387199999997 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948912+0000", 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041183000000000002 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950438+0000", 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015253199999999999 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950488+0000", 2024-04-22T18:41:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0285999999999999e-05 2024-04-22T18:41:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950601+0000", 2024-04-22T18:41:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011244899999999999 2024-04-22T18:41:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954048+0000", 2024-04-22T18:41:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003447313 2024-04-22T18:41:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457373+0000", 2024-04-22T18:41:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50332465299999996 2024-04-22T18:41:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457430+0000", 2024-04-22T18:41:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7574999999999997e-05 2024-04-22T18:41:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457560+0000", 2024-04-22T18:41:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013023900000000001 2024-04-22T18:41:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:8 3.2 3:5f64590d:::7c47933f331d40689081b2bd8f9d7f7bdae1be1fdccb52d696047a3969e902b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.847097453999993, 2024-04-22T18:41:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3854200569999999, 2024-04-22T18:41:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173117+0000", 2024-04-22T18:41:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173393+0000", 2024-04-22T18:41:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000275278 2024-04-22T18:41:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173399+0000", 2024-04-22T18:41:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4590000000000003e-06 2024-04-22T18:41:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173406+0000", 2024-04-22T18:41:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2450000000000001e-06 2024-04-22T18:41:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173795+0000", 2024-04-22T18:41:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038912400000000002 2024-04-22T18:41:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173804+0000", 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.949506+0000", 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77570185700000005 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950166+0000", 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066022800000000001 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352843+0000", 2024-04-22T18:41:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40267661799999999 2024-04-22T18:41:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352911+0000", 2024-04-22T18:41:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8126000000000005e-05 2024-04-22T18:41:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.353031+0000", 2024-04-22T18:41:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119919 2024-04-22T18:41:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453622+0000", 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100590793 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558446+0000", 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104824073 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558479+0000", 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3362999999999999e-05 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558538+0000", 2024-04-22T18:41:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9046999999999999e-05 2024-04-22T18:41:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:10 3.2 3:518c37f9:::851eca84dafbf2f780c4f07291110c453349bdc9c623c17bdcd26e27badb30ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.846803860999998, 2024-04-22T18:41:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3852527729999999, 2024-04-22T18:41:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173411+0000", 2024-04-22T18:41:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173749+0000", 2024-04-22T18:41:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000338029 2024-04-22T18:41:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173755+0000", 2024-04-22T18:41:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0530000000000003e-06 2024-04-22T18:41:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173759+0000", 2024-04-22T18:41:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8190000000000002e-06 2024-04-22T18:41:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173855+0000", 2024-04-22T18:41:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5927999999999994e-05 2024-04-22T18:41:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173862+0000", 2024-04-22T18:41:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4449999999999999e-06 2024-04-22T18:41:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950835+0000", 2024-04-22T18:41:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77697248299999999 2024-04-22T18:41:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951632+0000", 2024-04-22T18:41:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079752299999999996 2024-04-22T18:41:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355407+0000", 2024-04-22T18:41:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40377480500000001 2024-04-22T18:41:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355462+0000", 2024-04-22T18:41:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4577999999999998e-05 2024-04-22T18:41:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355571+0000", 2024-04-22T18:41:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108964 2024-04-22T18:41:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453625+0000", 2024-04-22T18:41:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098054397000000001 2024-04-22T18:41:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558574+0000", 2024-04-22T18:41:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104948571 2024-04-22T18:41:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558602+0000", 2024-04-22T18:41:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7966999999999998e-05 2024-04-22T18:41:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558665+0000", 2024-04-22T18:41:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3032000000000002e-05 2024-04-22T18:41:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:11 3.2 3:718aa8aa:::9603c4f8e269a436b2f0bf376c5374e77bdd7cf58fccae60e4e65ac4696806d4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.846452243000002, 2024-04-22T18:41:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4852604440000001, 2024-04-22T18:41:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173813+0000", 2024-04-22T18:41:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0655999999999997e-05 2024-04-22T18:41:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173819+0000", 2024-04-22T18:41:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.468e-06 2024-04-22T18:41:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173822+0000", 2024-04-22T18:41:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4879999999999999e-06 2024-04-22T18:41:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173983+0000", 2024-04-22T18:41:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160519 2024-04-22T18:41:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173991+0000", 2024-04-22T18:41:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6490000000000008e-06 2024-04-22T18:41:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951688+0000", 2024-04-22T18:41:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77769743800000002 2024-04-22T18:41:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952419+0000", 2024-04-22T18:41:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073067499999999997 2024-04-22T18:41:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355901+0000", 2024-04-22T18:41:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40348237599999998 2024-04-22T18:41:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355953+0000", 2024-04-22T18:41:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1477000000000002e-05 2024-04-22T18:41:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356060+0000", 2024-04-22T18:41:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107114 2024-04-22T18:41:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453629+0000", 2024-04-22T18:41:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097569127000000005 2024-04-22T18:41:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658807+0000", 2024-04-22T18:41:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20517829800000001 2024-04-22T18:41:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658876+0000", 2024-04-22T18:41:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8454000000000006e-05 2024-04-22T18:41:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659024+0000", 2024-04-22T18:41:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148338 2024-04-22T18:41:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:12 3.2 3:475e1cd9:::efaab2bd13543cdbb143e869e20eb0ec424f7f5202707859b241ed91d4674936:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.846389298999995, 2024-04-22T18:41:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4853405259999999, 2024-04-22T18:41:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.053 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173940+0000", 2024-04-22T18:41:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011465500000000001 2024-04-22T18:41:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173946+0000", 2024-04-22T18:41:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5289999999999999e-06 2024-04-22T18:41:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173950+0000", 2024-04-22T18:41:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8290000000000001e-06 2024-04-22T18:41:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174277+0000", 2024-04-22T18:41:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032761299999999998 2024-04-22T18:41:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174285+0000", 2024-04-22T18:41:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.52e-06 2024-04-22T18:41:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952455+0000", 2024-04-22T18:41:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77817000700000005 2024-04-22T18:41:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954035+0000", 2024-04-22T18:41:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001580078 2024-04-22T18:41:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356379+0000", 2024-04-22T18:41:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40234440399999999 2024-04-22T18:41:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356428+0000", 2024-04-22T18:41:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8832999999999998e-05 2024-04-22T18:41:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356532+0000", 2024-04-22T18:41:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103235 2024-04-22T18:41:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453632+0000", 2024-04-22T18:41:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097100346000000004 2024-04-22T18:41:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659069+0000", 2024-04-22T18:41:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20543763800000001 2024-04-22T18:41:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659105+0000", 2024-04-22T18:41:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5240000000000001e-05 2024-04-22T18:41:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659167+0000", 2024-04-22T18:41:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2207999999999998e-05 2024-04-22T18:41:18.059 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:16 3.2 3:47c76cc8:::c00d18458eb20c0c0aa9c612cf9793a68445ab0ea7b9e650ecd0072c60a96c0f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.846261339999998, 2024-04-22T18:41:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.485347744, 2024-04-22T18:41:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:18.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:18.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174234+0000", 2024-04-22T18:41:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028047599999999999 2024-04-22T18:41:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174240+0000", 2024-04-22T18:41:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2079999999999997e-06 2024-04-22T18:41:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174244+0000", 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5810000000000001e-06 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174670+0000", 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042640000000000001 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174675+0000", 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8709999999999996e-06 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054168+0000", 2024-04-22T18:41:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87949257300000006 2024-04-22T18:41:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054877+0000", 2024-04-22T18:41:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070884300000000002 2024-04-22T18:41:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356836+0000", 2024-04-22T18:41:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30195926000000001 2024-04-22T18:41:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356887+0000", 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0767999999999999e-05 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356991+0000", 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104072 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453636+0000", 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096644968999999997 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659207+0000", 2024-04-22T18:41:18.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20557154699999999 2024-04-22T18:41:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659240+0000", 2024-04-22T18:41:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2665e-05 2024-04-22T18:41:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659302+0000", 2024-04-22T18:41:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2234000000000001e-05 2024-04-22T18:41:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:17 3.2 3:6ef31daa:::1cf4ff496c4480e58510f607481d976b47d84868e2c54016273c38baf590f3b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.845967060000007, 2024-04-22T18:41:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4851703729999999, 2024-04-22T18:41:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174248+0000", 2024-04-22T18:41:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174626+0000", 2024-04-22T18:41:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037839699999999999 2024-04-22T18:41:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174633+0000", 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0990000000000004e-06 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174636+0000", 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5540000000000002e-06 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174790+0000", 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153807 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174793+0000", 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4970000000000001e-06 2024-04-22T18:41:18.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054902+0000", 2024-04-22T18:41:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88010886300000002 2024-04-22T18:41:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055366+0000", 2024-04-22T18:41:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046336199999999999 2024-04-22T18:41:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.452973+0000", 2024-04-22T18:41:18.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39760783999999999 2024-04-22T18:41:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453046+0000", 2024-04-22T18:41:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2374999999999997e-05 2024-04-22T18:41:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453193+0000", 2024-04-22T18:41:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014722499999999999 2024-04-22T18:41:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554698+0000", 2024-04-22T18:41:18.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101505096 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659337+0000", 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104639236 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659365+0000", 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7594e-05 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659419+0000", 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4058999999999999e-05 2024-04-22T18:41:18.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:18 3.2 3:792a4ed0:::e1c814961b3d1531fb5ed475b3d496a0405c3a819b68a92ff9b2391fe912370f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.845570804000005, 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.585036954, 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174640+0000", 2024-04-22T18:41:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-04-22T18:41:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174758+0000", 2024-04-22T18:41:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011819099999999999 2024-04-22T18:41:18.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174764+0000", 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6729999999999997e-06 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174767+0000", 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3830000000000001e-06 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373669+0000", 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198901989 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373686+0000", 2024-04-22T18:41:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7286e-05 2024-04-22T18:41:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055408+0000", 2024-04-22T18:41:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68172133400000001 2024-04-22T18:41:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055938+0000", 2024-04-22T18:41:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053016599999999999 2024-04-22T18:41:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453668+0000", 2024-04-22T18:41:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39773030799999998 2024-04-22T18:41:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453717+0000", 2024-04-22T18:41:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8884000000000002e-05 2024-04-22T18:41:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453823+0000", 2024-04-22T18:41:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105791 2024-04-22T18:41:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554715+0000", 2024-04-22T18:41:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100891628 2024-04-22T18:41:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759439+0000", 2024-04-22T18:41:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20472459400000001 2024-04-22T18:41:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759525+0000", 2024-04-22T18:41:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5653000000000001e-05 2024-04-22T18:41:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759682+0000", 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015711800000000001 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:75 3.1 3:ab43a94d:::f38e6e97cb24e2b90da964fee5d2d6d8674386870fc552915e0c3fdbefd4b126:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.772694287, 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60149928399999997, 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247520+0000", 2024-04-22T18:41:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247527+0000", 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1670000000000002e-06 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247539+0000", 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503e-05 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247549+0000", 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0000000000000001e-05 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247796+0000", 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024712699999999998 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247955+0000", 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158534 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.343985+0000", 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096030344000000004 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.344035+0000", 2024-04-22T18:41:18.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9969999999999998e-05 2024-04-22T18:41:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.745806+0000", 2024-04-22T18:41:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.401771029 2024-04-22T18:41:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746040+0000", 2024-04-22T18:41:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023339300000000001 2024-04-22T18:41:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746198+0000", 2024-04-22T18:41:18.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015793299999999999 2024-04-22T18:41:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848737+0000", 2024-04-22T18:41:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10253898 2024-04-22T18:41:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848751+0000", 2024-04-22T18:41:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4694e-05 2024-04-22T18:41:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848825+0000", 2024-04-22T18:41:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3443999999999997e-05 2024-04-22T18:41:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849021+0000", 2024-04-22T18:41:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000196047 2024-04-22T18:41:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:76 3.1 3:b44829a0:::3788286a255d538503614e5f04049b0bb54cacaa317ab102b5dee8f5b640b296:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.772655221000001, 2024-04-22T18:41:18.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60168681599999996, 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247564+0000", 2024-04-22T18:41:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1150000000000004e-06 2024-04-22T18:41:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247570+0000", 2024-04-22T18:41:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3940000000000001e-06 2024-04-22T18:41:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247576+0000", 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3619999999999997e-06 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.340845+0000", 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.093269199999999997 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.341043+0000", 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019834400000000001 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345106+0000", 2024-04-22T18:41:18.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040629780000000001 2024-04-22T18:41:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345160+0000", 2024-04-22T18:41:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4246999999999997e-05 2024-04-22T18:41:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747047+0000", 2024-04-22T18:41:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40188678300000003 2024-04-22T18:41:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747162+0000", 2024-04-22T18:41:18.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011483400000000001 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747429+0000", 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026700299999999998 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849032+0000", 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101603203 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849095+0000", 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2597999999999999e-05 2024-04-22T18:41:18.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849136+0000", 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1251000000000002e-05 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849247+0000", 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011127 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3526 4.c 4:314c2fec:::smithi08937929-22:head [setxattr _header (56) in=7b,truncate 3092521] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:41:18.091 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.791018440999999, 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60463626000000004, 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3526 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229200+0000", 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2320000000000001e-06 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229203+0000", 2024-04-22T18:41:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:41:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329292+0000", 2024-04-22T18:41:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100088891 2024-04-22T18:41:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329357+0000", 2024-04-22T18:41:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5115000000000003e-05 2024-04-22T18:41:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329453+0000", 2024-04-22T18:41:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5802000000000002e-05 2024-04-22T18:41:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329594+0000", 2024-04-22T18:41:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014157500000000001 2024-04-22T18:41:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.330172+0000", 2024-04-22T18:41:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057782799999999996 2024-04-22T18:41:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833712+0000", 2024-04-22T18:41:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50353975299999998 2024-04-22T18:41:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833774+0000", 2024-04-22T18:41:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1832e-05 2024-04-22T18:41:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833834+0000", 2024-04-22T18:41:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9809999999999998e-05 2024-04-22T18:41:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.096 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:339 3.2 3:4159b8d1:::bb1f8e997816dd9d2f29d36c4c1859fcaffbf8763b5c8d63c7a818d8d84bac37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.244764682, 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0092773479999999, 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775450+0000", 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775457+0000", 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2949999999999996e-06 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775467+0000", 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0180000000000001e-05 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775475+0000", 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.678e-06 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.877787+0000", 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102312559 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.880686+0000", 2024-04-22T18:41:18.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002898748 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.282867+0000", 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.402180961 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.282945+0000", 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7775999999999995e-05 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283065+0000", 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0001199 2024-04-22T18:41:18.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.285514+0000", 2024-04-22T18:41:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0024491729999999998 2024-04-22T18:41:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784375+0000", 2024-04-22T18:41:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49886154100000002 2024-04-22T18:41:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784475+0000", 2024-04-22T18:41:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9074999999999996e-05 2024-04-22T18:41:18.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784728+0000", 2024-04-22T18:41:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025386799999999998 2024-04-22T18:41:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:340 3.2 3:4bbc2d06:::e1c179ab87fb8e08c4a1dfc0c7cc725138181c5734d6df18f3f2545b93731e97:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.244732263, 2024-04-22T18:41:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009550757, 2024-04-22T18:41:18.102 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775487+0000", 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6839999999999998e-06 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775492+0000", 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6899999999999997e-06 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775497+0000", 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8590000000000004e-06 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.880737+0000", 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105240267 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881308+0000", 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057098399999999997 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283810+0000", 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40250177500000001 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283883+0000", 2024-04-22T18:41:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2430999999999998e-05 2024-04-22T18:41:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284019+0000", 2024-04-22T18:41:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013652299999999999 2024-04-22T18:41:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.287390+0000", 2024-04-22T18:41:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033710089999999999 2024-04-22T18:41:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784829+0000", 2024-04-22T18:41:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49743879400000002 2024-04-22T18:41:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784893+0000", 2024-04-22T18:41:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4184999999999996e-05 2024-04-22T18:41:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785034+0000", 2024-04-22T18:41:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014114600000000001 2024-04-22T18:41:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:341 3.2 3:5b8bea68:::ab73be59042e3e18c4fe1fdbc2bbcc205779424ddf3bb423a595330be9026399:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.244712911000001, 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009670804, 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775502+0000", 2024-04-22T18:41:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775505+0000", 2024-04-22T18:41:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0809999999999998e-06 2024-04-22T18:41:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775511+0000", 2024-04-22T18:41:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2039999999999996e-06 2024-04-22T18:41:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775515+0000", 2024-04-22T18:41:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8790000000000002e-06 2024-04-22T18:41:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881376+0000", 2024-04-22T18:41:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105861 2024-04-22T18:41:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881839+0000", 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046217200000000001 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284569+0000", 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40273058900000003 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284730+0000", 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077800000000001 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:18.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284923+0000", 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000193253 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.287593+0000", 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026699359999999999 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785074+0000", 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49748046000000001 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785103+0000", 2024-04-22T18:41:18.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8935e-05 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785174+0000", 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1001999999999995e-05 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:342 3.2 3:634dda9f:::10622dc7de8047e7908c0b5cca7a43d84cdf1b7629e6f7e159b8207feba82b01:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.244694699, 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.022499579, 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.113 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:18.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775524+0000", 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9629999999999998e-06 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775529+0000", 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0320000000000003e-06 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775533+0000", 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7380000000000002e-06 2024-04-22T18:41:18.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881884+0000", 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106350294 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.882492+0000", 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060872699999999999 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381559+0000", 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49906679199999998 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381677+0000", 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011743699999999999 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381899+0000", 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022272599999999999 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.383440+0000", 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015407719999999999 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.797827+0000", 2024-04-22T18:41:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.414386741 2024-04-22T18:41:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.797887+0000", 2024-04-22T18:41:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9889000000000002e-05 2024-04-22T18:41:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.798021+0000", 2024-04-22T18:41:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000133948 2024-04-22T18:41:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.118 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:343 3.2 3:7db7298e:::00597d8527769aa920807495342cf3ba06efb8f507feae865dd89cd8ad10bda0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.244675744, 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.110995373, 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775542+0000", 2024-04-22T18:41:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9079999999999999e-06 2024-04-22T18:41:18.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775547+0000", 2024-04-22T18:41:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.994e-06 2024-04-22T18:41:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775553+0000", 2024-04-22T18:41:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2299999999999999e-06 2024-04-22T18:41:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.882519+0000", 2024-04-22T18:41:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106966453 2024-04-22T18:41:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883103+0000", 2024-04-22T18:41:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058419400000000001 2024-04-22T18:41:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.581957+0000", 2024-04-22T18:41:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69885328599999996 2024-04-22T18:41:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582023+0000", 2024-04-22T18:41:18.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6210999999999993e-05 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582146+0000", 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123669 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.682293+0000", 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10014662100000001 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886410+0000", 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20411678899999999 2024-04-22T18:41:18.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886445+0000", 2024-04-22T18:41:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5462e-05 2024-04-22T18:41:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886535+0000", 2024-04-22T18:41:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9995999999999997e-05 2024-04-22T18:41:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.124 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:344 3.1 3:9da67ae4:::04c6b48a110b3f74550f78f0899df697160a1eeab7f41463f6ac18091ca67078:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.244640436999999, 2024-04-22T18:41:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60604869800000005, 2024-04-22T18:41:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:18.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775578+0000", 2024-04-22T18:41:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3440000000000001e-06 2024-04-22T18:41:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775584+0000", 2024-04-22T18:41:18.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6969999999999998e-06 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775589+0000", 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0540000000000002e-06 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775729+0000", 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014061299999999999 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776045+0000", 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000315168 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080584+0000", 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30453977700000001 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080635+0000", 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1007999999999997e-05 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080741+0000", 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105646 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.180821+0000", 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10007980900000001 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381437+0000", 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20061589399999999 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381491+0000", 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4531999999999999e-05 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381624+0000", 2024-04-22T18:41:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000132745 2024-04-22T18:41:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:345 3.1 3:bd847077:::0e274616605e19dd653fcf05d310933ab02835574414ca18340b3fc325d2da4f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.244616077, 2024-04-22T18:41:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.90876140699999997, 2024-04-22T18:41:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775599+0000", 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775602+0000", 2024-04-22T18:41:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.976e-06 2024-04-22T18:41:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775607+0000", 2024-04-22T18:41:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2229999999999998e-06 2024-04-22T18:41:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775612+0000", 2024-04-22T18:41:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6539999999999998e-06 2024-04-22T18:41:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776072+0000", 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046006299999999998 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776256+0000", 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000183735 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081157+0000", 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30490169099999997 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081203+0000", 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5677999999999999e-05 2024-04-22T18:41:18.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081301+0000", 2024-04-22T18:41:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7893999999999995e-05 2024-04-22T18:41:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.180827+0000", 2024-04-22T18:41:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099525725999999995 2024-04-22T18:41:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684197+0000", 2024-04-22T18:41:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50336990699999995 2024-04-22T18:41:18.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684247+0000", 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0195000000000003e-05 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684361+0000", 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114189 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4267 3.2 3:584fd9a2:::cda848272f4ef42ebafc2089aaddeacaf12eabd82aef5e42e32ae93a8a5da0b6:head [write 0~2635776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:18.135 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.164266221, 2024-04-22T18:41:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.72658671900000005, 2024-04-22T18:41:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4267 2024-04-22T18:41:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855946+0000", 2024-04-22T18:41:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.856984+0000", 2024-04-22T18:41:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001037764 2024-04-22T18:41:18.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.857003+0000", 2024-04-22T18:41:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8876e-05 2024-04-22T18:41:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.857016+0000", 2024-04-22T18:41:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3337e-05 2024-04-22T18:41:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883170+0000", 2024-04-22T18:41:18.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026154047 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883262+0000", 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1996999999999993e-05 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883360+0000", 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7429000000000005e-05 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.887277+0000", 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039169449999999998 2024-04-22T18:41:18.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582423+0000", 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69514625200000002 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582458+0000", 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5293999999999998e-05 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582536+0000", 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7855000000000006e-05 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.140 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.141 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.141 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:18.141 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:18.141 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:18.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:18.142 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:18.143+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:18.243 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:18.244+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.256 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGRecoveryDelete(3.2 e142,130 3:4330d3a2:::40b3874f17e3b043a354e1ae1e875047fa47972bfcea195b70fc992a4f14dd53:head,128'260)", 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:18.062055+0000", 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.19620674199999999, 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19622920199999999, 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.062055+0000", 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.062055+0000", 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.062054+0000", 2024-04-22T18:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.062063+0000", 2024-04-22T18:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.062107+0000", 2024-04-22T18:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3392000000000001e-05 2024-04-22T18:41:18.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.162186+0000", 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007961 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4437 4.19 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.34259218899999999, 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34264914400000002, 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:18.259 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4437 2024-04-22T18:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:41:18.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915669+0000", 2024-04-22T18:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915673+0000", 2024-04-22T18:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6770000000000001e-06 2024-04-22T18:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915679+0000", 2024-04-22T18:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.976e-06 2024-04-22T18:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915684+0000", 2024-04-22T18:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3430000000000003e-06 2024-04-22T18:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915725+0000", 2024-04-22T18:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0751999999999999e-05 2024-04-22T18:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915732+0000", 2024-04-22T18:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6279999999999999e-06 2024-04-22T18:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 2 2024-04-22T18:41:18.263 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:18.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:18.344 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:18.345+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:18.410 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:18.410 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:18.410 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:18.410 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:18.410 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:18.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:18.462 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:18.463+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:18.522 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:18.522 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 143.85975057100001, 2024-04-22T18:41:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:41:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:41:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:41:18.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:41:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:41:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:41:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:41:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:41:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:41:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:41:18.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:18.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:41:18.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:41:18.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:18.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:41:18.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:41:18.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:18.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:41:18.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:41:18.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:41:18.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:41:18.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:18.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:41:18.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:41:18.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:41:18.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:41:18.528 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.528 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.528 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.528 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:18.528 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:41:18.528 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 143.75891427299999, 2024-04-22T18:41:18.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:41:18.528 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.528 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.528 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.528 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.528 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.529 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:41:18.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.529 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:41:18.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:41:18.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:41:18.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:18.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:41:18.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:41:18.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:41:18.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:41:18.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:41:18.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:41:18.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:41:18.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:41:18.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:18.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:41:18.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:41:18.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:18.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:41:18.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:41:18.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:41:18.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:41:18.533 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.533 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.533 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.533 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:91 4.1 4:81200c09:::smithi08937929-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:18.534 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:41:18.534 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 143.758757003, 2024-04-22T18:41:18.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55437317699999999, 2024-04-22T18:41:18.534 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.534 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.534 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.534 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.534 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.534 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 91 2024-04-22T18:41:18.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.534 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:41:18.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763372+0000", 2024-04-22T18:41:18.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763371+0000", 2024-04-22T18:41:18.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763375+0000", 2024-04-22T18:41:18.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1999999999999999e-06 2024-04-22T18:41:18.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763378+0000", 2024-04-22T18:41:18.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.326e-06 2024-04-22T18:41:18.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763381+0000", 2024-04-22T18:41:18.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0290000000000001e-06 2024-04-22T18:41:18.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767649+0000", 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042676770000000001 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767729+0000", 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0439999999999996e-05 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.767863+0000", 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000134048 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:18.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871058+0000", 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103195333 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317677+0000", 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44661856999999999 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317707+0000", 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9641e-05 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317745+0000", 2024-04-22T18:41:18.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8339e-05 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 143.758742394, 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:41:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.540 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:18.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:18.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:18.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:41:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:41:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:41:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:41:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:41:18.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:41:18.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:18.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:41:18.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:41:18.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:41:18.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:41:18.543 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.543 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.543 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.543 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 143.758732009, 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:41:18.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:41:18.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:41:18.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:41:18.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:41:18.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.548 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 139.420981582, 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:18.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:18.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:41:18.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:18.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:41:18.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:41:18.554 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.554 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.554 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.554 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:41:18.554 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:18.554 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 112.38633406700001, 2024-04-22T18:41:18.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:41:18.554 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.554 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:18.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.555 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:41:18.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.555 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:18.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:18.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:41:18.557 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.558 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.558 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.558 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4177 4.19 4.f75b2379 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:18.558 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:18.558 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.357280769999999, 2024-04-22T18:41:18.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76236081700000002, 2024-04-22T18:41:18.558 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.558 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:18.558 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.558 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.558 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.558 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4177 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164847+0000", 2024-04-22T18:41:18.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164854+0000", 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2620000000000001e-06 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164864+0000", 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0400999999999999e-05 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164876+0000", 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1204e-05 2024-04-22T18:41:18.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164975+0000", 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9562999999999995e-05 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164980+0000", 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7310000000000001e-06 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927200+0000", 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.762219973 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927209+0000", 2024-04-22T18:41:18.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9050000000000003e-06 2024-04-22T18:41:18.562 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.562 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.562 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.562 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4182 4.c 4.5345b7ac (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:18.562 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:18.562 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.357249450999999, 2024-04-22T18:41:18.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76185930999999996, 2024-04-22T18:41:18.562 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.562 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:18.563 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.563 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.563 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.563 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4182 2024-04-22T18:41:18.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.563 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:18.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164881+0000", 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.305e-06 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164885+0000", 2024-04-22T18:41:18.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.873e-06 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164890+0000", 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6299999999999997e-06 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225475+0000", 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060585181000000002 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225483+0000", 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6790000000000005e-06 2024-04-22T18:41:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926728+0000", 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124542099999998 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926739+0000", 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0669000000000001e-05 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4183 4.c 4.5345b7ac (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:18.566 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.357236195, 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76187489500000005, 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4183 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164892+0000", 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164894+0000", 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.705e-06 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164897+0000", 2024-04-22T18:41:18.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2739999999999999e-06 2024-04-22T18:41:18.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164901+0000", 2024-04-22T18:41:18.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.636e-06 2024-04-22T18:41:18.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225508+0000", 2024-04-22T18:41:18.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060607564000000003 2024-04-22T18:41:18.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225515+0000", 2024-04-22T18:41:18.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1829999999999999e-06 2024-04-22T18:41:18.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926759+0000", 2024-04-22T18:41:18.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.701244859 2024-04-22T18:41:18.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926767+0000", 2024-04-22T18:41:18.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0180000000000001e-06 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4190 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.357219273, 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.761884546, 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4190 2024-04-22T18:41:18.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164912+0000", 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6249999999999999e-06 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164918+0000", 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6039999999999998e-06 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164922+0000", 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9319999999999998e-06 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225536+0000", 2024-04-22T18:41:18.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060613725 2024-04-22T18:41:18.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225540+0000", 2024-04-22T18:41:18.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5030000000000002e-06 2024-04-22T18:41:18.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926786+0000", 2024-04-22T18:41:18.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124673999999998 2024-04-22T18:41:18.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926794+0000", 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7689999999999996e-06 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4191 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.357203852, 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76190695500000005, 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.575 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4191 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:18.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164927+0000", 2024-04-22T18:41:18.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0959999999999999e-06 2024-04-22T18:41:18.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164937+0000", 2024-04-22T18:41:18.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0730999999999999e-05 2024-04-22T18:41:18.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164944+0000", 2024-04-22T18:41:18.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5130000000000002e-06 2024-04-22T18:41:18.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225565+0000", 2024-04-22T18:41:18.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060621421000000002 2024-04-22T18:41:18.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225569+0000", 2024-04-22T18:41:18.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.264e-06 2024-04-22T18:41:18.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926813+0000", 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124469 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926832+0000", 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8580000000000002e-05 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4192 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.357179243999999, 2024-04-22T18:41:18.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76191075399999997, 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4192 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:18.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164953+0000", 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1099999999999996e-06 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164965+0000", 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2298e-05 2024-04-22T18:41:18.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164972+0000", 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5230000000000001e-06 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225581+0000", 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060609356000000003 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225584+0000", 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.106e-06 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926852+0000", 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70126794999999997 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926860+0000", 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0919999999999998e-06 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4197 4.13 4:ca46fde9:::smithi08937929-14:head [sparse-read 0~1375265,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:18.583 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.357152634, 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.28429536, 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4197 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164975+0000", 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164979+0000", 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8959999999999996e-06 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164983+0000", 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.053e-06 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164987+0000", 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.264783+0000", 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099796088000000005 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.264789+0000", 2024-04-22T18:41:18.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4989999999999999e-06 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139375+0000", 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.87458606000000005 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139478+0000", 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010219 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.446961+0000", 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30748385900000003 2024-04-22T18:41:18.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.447101+0000", 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001393 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.449271+0000", 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021707950000000001 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4198 4.13 4:ca46fde9:::smithi08937929-14:head [sparse-read 0~1375265] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:18.588 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.357139266000001, 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2868951370000001, 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4198 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:18.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:18.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164991+0000", 2024-04-22T18:41:18.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:41:18.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164994+0000", 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2459999999999999e-06 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164997+0000", 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1200000000000002e-06 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.364948+0000", 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19995059200000001 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.364969+0000", 2024-04-22T18:41:18.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1495999999999999e-05 2024-04-22T18:41:18.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139617+0000", 2024-04-22T18:41:18.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77464723300000005 2024-04-22T18:41:18.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:18.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139643+0000", 2024-04-22T18:41:18.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6228e-05 2024-04-22T18:41:18.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451393+0000", 2024-04-22T18:41:18.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31175035200000001 2024-04-22T18:41:18.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:18.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451515+0000", 2024-04-22T18:41:18.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001214 2024-04-22T18:41:18.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451885+0000", 2024-04-22T18:41:18.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036992999999999998 2024-04-22T18:41:18.593 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.593 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.593 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.594 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4199 4.13 4:ca46fde9:::smithi08937929-14:head [read 0~2739541,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:18.594 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:18.594 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.357128243, 2024-04-22T18:41:18.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.291247504, 2024-04-22T18:41:18.594 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.594 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:18.594 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.594 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.594 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.594 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4199 2024-04-22T18:41:18.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.594 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165002+0000", 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165006+0000", 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165009+0000", 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4599999999999999e-06 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365066+0000", 2024-04-22T18:41:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20005682999999999 2024-04-22T18:41:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365078+0000", 2024-04-22T18:41:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2235000000000001e-05 2024-04-22T18:41:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139671+0000", 2024-04-22T18:41:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77459301700000005 2024-04-22T18:41:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:18.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139702+0000", 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0673999999999998e-05 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451930+0000", 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31222817600000002 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.452092+0000", 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001616 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.456248+0000", 2024-04-22T18:41:18.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041563700000000004 2024-04-22T18:41:18.599 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.599 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.599 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.599 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4200 4.13 4:c93c842e:::smithi08937929-25:head [tier-promote] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:18.599 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:18.599 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.357117363, 2024-04-22T18:41:18.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.177728814, 2024-04-22T18:41:18.599 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.599 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:18.599 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.599 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.599 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.599 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4200 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:18.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165013+0000", 2024-04-22T18:41:18.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.635e-06 2024-04-22T18:41:18.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165018+0000", 2024-04-22T18:41:18.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2000000000000002e-06 2024-04-22T18:41:18.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165024+0000", 2024-04-22T18:41:18.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5910000000000003e-06 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365110+0000", 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20008601400000001 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365114+0000", 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.651e-06 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139723+0000", 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77460836899999996 2024-04-22T18:41:18.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139787+0000", 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4707000000000002e-05 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.459491+0000", 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31970372000000002 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.459567+0000", 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6199999999999995e-05 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.562522+0000", 2024-04-22T18:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102954925 2024-04-22T18:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.562585+0000", 2024-04-22T18:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3287000000000006e-05 2024-04-22T18:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:05.342740+0000", 2024-04-22T18:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.78015481499999995 2024-04-22T18:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.604 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4205 4.1 4.fdaa93c1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.357101213, 2024-04-22T18:41:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76220996100000005, 2024-04-22T18:41:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4205 2024-04-22T18:41:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:18.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:18.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165027+0000", 2024-04-22T18:41:18.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165029+0000", 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3549999999999999e-06 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165035+0000", 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8549999999999998e-06 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165040+0000", 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9200000000000003e-06 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165116+0000", 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5834999999999995e-05 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165123+0000", 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2139999999999999e-06 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927230+0000", 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76210651100000004 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927237+0000", 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7409999999999992e-06 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4268 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e133)", 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.9393938009999996, 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70931800300000003, 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.609 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4268 2024-04-22T18:41:18.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.610 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:41:18.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:41:18.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582733+0000", 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582741+0000", 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3329999999999999e-06 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582751+0000", 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0628999999999999e-05 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582759+0000", 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4120000000000004e-06 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582795+0000", 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6157e-05 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582862+0000", 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7261999999999994e-05 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.987268+0000", 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40440629700000003 2024-04-22T18:41:18.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:18.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.987771+0000", 2024-04-22T18:41:18.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050267400000000005 2024-04-22T18:41:18.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:18.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.291974+0000", 2024-04-22T18:41:18.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30420264499999999 2024-04-22T18:41:18.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:18.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.292008+0000", 2024-04-22T18:41:18.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4011999999999999e-05 2024-04-22T18:41:18.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.292053+0000", 2024-04-22T18:41:18.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5305000000000001e-05 2024-04-22T18:41:18.614 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.614 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.614 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.614 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4272 4.5 4:a6d237d2:::smithi08937929-43:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e133)", 2024-04-22T18:41:18.614 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:18.614 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.736065601, 2024-04-22T18:41:18.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60731304699999999, 2024-04-22T18:41:18.614 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:18.615 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:18.615 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:18.615 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:18.615 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:18.615 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4272 2024-04-22T18:41:18.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.615 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:18.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:18.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:18.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:18.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:18.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:18.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:18.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:18.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:18.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:18.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786068+0000", 2024-04-22T18:41:18.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9910000000000002e-06 2024-04-22T18:41:18.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:18.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786079+0000", 2024-04-22T18:41:18.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1287000000000001e-05 2024-04-22T18:41:18.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:18.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786083+0000", 2024-04-22T18:41:18.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0949999999999998e-06 2024-04-22T18:41:18.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:18.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786121+0000", 2024-04-22T18:41:18.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8096999999999998e-05 2024-04-22T18:41:18.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:18.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786219+0000", 2024-04-22T18:41:18.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7429000000000005e-05 2024-04-22T18:41:18.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:18.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:18.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:18.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.393376+0000", 2024-04-22T18:41:18.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60715760600000002 2024-04-22T18:41:18.618 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.618 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.618 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.618 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:18.618 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:18.618 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:18.620 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:18.620 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:18.620+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:18.656 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:18.657+0000 7fec55bea640 -1 osd.7 142 handle_osd_map injecting map crc failure 2024-04-22T18:41:18.669 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:41:18.720 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:18.721+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:18.758 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:18.758 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:18.758 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:18.768 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:18.820 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:18.821+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:18.911 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:18.911 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:18.912 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:18.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:18.921 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:18.922+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:19.022 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:19.023+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:19.024 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:19.024 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:19.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:19.024 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:19.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.024 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:41:19.024 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:19.024 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.93024681099999, 2024-04-22T18:41:19.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:41:19.025 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.025 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.025 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.025 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.025 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.025 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:41:19.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.025 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:19.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:19.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:41:19.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:19.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:41:19.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:41:19.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:41:19.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:41:19.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:41:19.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:41:19.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:41:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:41:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:41:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:41:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:41:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:41:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:41:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:41:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:41:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:41:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:41:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:41:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:41:19.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:41:19.030 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.030 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.030 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.030 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:41:19.030 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:19.030 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 141.03074893600001, 2024-04-22T18:41:19.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:41:19.030 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.030 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.030 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:19.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:19.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:41:19.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:19.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:41:19.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:41:19.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:41:19.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:41:19.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:41:19.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:41:19.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:41:19.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:41:19.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:19.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:41:19.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:41:19.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:41:19.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:41:19.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:41:19.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:41:19.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:19.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:41:19.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:41:19.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:41:19.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:41:19.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:41:19.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:41:19.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:41:19.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:41:19.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:41:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:41:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:41:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 139.922449145, 2024-04-22T18:41:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:41:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:41:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:41:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:41:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:41:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:41:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:41:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:41:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:41:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:41:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:41:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:41:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:41:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:19.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:41:19.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:41:19.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:41:19.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:41:19.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:41:19.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:41:19.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:41:19.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:41:19.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:41:19.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:41:19.041 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.041 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.041 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.042 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:19.042 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:19.042 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.590905517, 2024-04-22T18:41:19.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:41:19.042 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.042 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.042 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:19.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:41:19.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:41:19.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:41:19.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:41:19.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:41:19.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:41:19.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:41:19.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:41:19.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:41:19.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:41:19.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:41:19.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:41:19.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:41:19.046 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.046 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.047 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.047 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:19.047 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:19.047 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.49981970899999, 2024-04-22T18:41:19.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:41:19.047 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.047 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.047 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.047 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.047 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.047 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:41:19.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.047 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:19.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:19.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:41:19.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:19.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:41:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:41:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:41:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:41:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:41:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:41:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:41:19.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:41:19.052 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.052 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.052 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.052 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:19.052 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:19.052 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.49401944900001, 2024-04-22T18:41:19.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:41:19.052 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.052 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.052 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.052 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.053 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:41:19.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.053 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:19.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:19.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:41:19.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:41:19.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:41:19.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:41:19.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:41:19.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:41:19.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:19.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:41:19.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:41:19.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:41:19.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:41:19.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:41:19.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:41:19.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:19.057 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.489279106, 2024-04-22T18:41:19.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:41:19.058 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.058 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.058 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.058 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.058 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.058 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:41:19.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.058 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:19.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:19.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:41:19.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:19.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:41:19.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:41:19.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:41:19.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:41:19.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:41:19.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:41:19.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:41:19.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:41:19.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:41:19.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:41:19.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:19.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:41:19.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:41:19.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:41:19.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:41:19.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:41:19.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:41:19.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:41:19.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:41:19.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:41:19.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:41:19.063 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.063 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.063 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.063 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:19.063 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:19.063 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.488806208, 2024-04-22T18:41:19.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:41:19.063 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.063 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.063 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.064 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.064 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.064 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:41:19.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.064 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:19.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:19.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:41:19.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:41:19.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:41:19.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:41:19.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:41:19.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:41:19.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:41:19.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:19.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:41:19.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:41:19.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:41:19.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:41:19.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:41:19.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:41:19.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:41:19.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:41:19.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:41:19.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:41:19.068 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.068 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.068 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.069 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:19.069 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:19.069 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 126.488786287, 2024-04-22T18:41:19.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:41:19.069 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.069 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:19.069 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.069 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.069 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.069 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:41:19.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.070 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:19.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:19.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:19.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:19.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:41:19.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:41:19.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:41:19.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:41:19.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:41:19.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:41:19.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:41:19.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:41:19.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:19.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:41:19.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:41:19.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:41:19.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:41:19.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:41:19.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:41:19.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:41:19.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:41:19.074 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.074 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.074 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.074 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:19.074 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:19.074 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 121.17946252199999, 2024-04-22T18:41:19.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:41:19.074 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.074 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.075 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:19.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:19.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:41:19.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:19.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:41:19.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:41:19.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:41:19.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:41:19.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:41:19.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:41:19.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:41:19.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:41:19.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:41:19.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:41:19.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:19.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:41:19.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:41:19.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:41:19.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:41:19.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:41:19.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:41:19.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:41:19.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:41:19.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:41:19.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:41:19.079 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.079 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.080 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.080 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:19.080 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:19.080 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 121.179254986, 2024-04-22T18:41:19.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:41:19.080 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.080 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.080 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:19.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:19.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:41:19.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:19.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:41:19.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:41:19.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:41:19.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:41:19.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:41:19.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:41:19.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:41:19.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:41:19.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:41:19.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:41:19.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:19.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:41:19.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:41:19.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:41:19.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:41:19.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:41:19.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:41:19.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:41:19.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:41:19.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:41:19.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:41:19.085 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.085 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.085 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.085 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:19.085 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:19.086 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 102.31360525300001, 2024-04-22T18:41:19.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:41:19.086 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.086 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.086 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:19.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:41:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:41:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:41:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:41:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:41:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:41:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:41:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:41:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:41:19.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:41:19.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:41:19.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:19.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:41:19.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:41:19.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:41:19.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:41:19.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:41:19.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:41:19.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:41:19.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:41:19.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:41:19.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:41:19.091 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.091 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.091 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.091 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:19.091 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:19.091 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 102.099766868, 2024-04-22T18:41:19.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:41:19.091 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.092 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.092 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:19.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:19.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:41:19.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:19.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:41:19.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:41:19.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:41:19.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:41:19.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:41:19.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:41:19.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:41:19.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:41:19.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:41:19.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:41:19.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:19.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:41:19.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:41:19.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:41:19.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:41:19.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:41:19.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:41:19.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:41:19.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:41:19.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:41:19.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:41:19.097 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.097 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.097 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.097 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:19.097 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:19.097 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 102.099733458, 2024-04-22T18:41:19.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:41:19.097 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.097 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.097 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:19.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:19.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:19.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:19.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:41:19.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:41:19.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:41:19.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:41:19.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:41:19.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:41:19.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:41:19.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:41:19.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:41:19.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:41:19.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:19.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:41:19.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:41:19.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:41:19.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:41:19.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:41:19.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:41:19.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:41:19.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:41:19.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:41:19.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:41:19.102 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.102 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.102 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.102 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:19.103 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:19.103 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 102.040850804, 2024-04-22T18:41:19.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:41:19.103 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.103 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.103 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:19.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:19.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:41:19.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:19.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:41:19.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:41:19.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:41:19.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:41:19.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:41:19.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:41:19.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:41:19.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:41:19.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:41:19.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:41:19.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:19.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:41:19.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:41:19.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:41:19.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:41:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:41:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:41:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:41:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:41:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:41:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:41:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 102.040712188, 2024-04-22T18:41:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:41:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:19.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:19.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:19.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:19.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:41:19.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:41:19.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:41:19.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:41:19.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:41:19.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:41:19.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:41:19.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:41:19.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:41:19.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:41:19.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:19.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:41:19.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:41:19.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:41:19.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:41:19.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:41:19.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:41:19.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:41:19.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 102.04069318499999, 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.113 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.114 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:19.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:19.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:19.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:19.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:41:19.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:41:19.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:41:19.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:41:19.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:41:19.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:41:19.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:41:19.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:41:19.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:41:19.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:41:19.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:19.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:41:19.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:41:19.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:41:19.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:41:19.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:41:19.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:41:19.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:41:19.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:41:19.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:41:19.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:41:19.118 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.118 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.118 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.119 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:19.119 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:19.119 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.419680651999997, 2024-04-22T18:41:19.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:41:19.119 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.119 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.119 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.119 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.119 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.119 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:41:19.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.119 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:19.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:19.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:41:19.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:19.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:41:19.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:41:19.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:41:19.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:41:19.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:41:19.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:41:19.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:41:19.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:41:19.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:41:19.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:41:19.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:19.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:41:19.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:41:19.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:41:19.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:41:19.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:41:19.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:41:19.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:41:19.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:41:19.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:41:19.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:41:19.124 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.124 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.124 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.124 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:19.124 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:19.124 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.419161060999997, 2024-04-22T18:41:19.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:41:19.125 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.125 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.125 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.125 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.125 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.125 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:41:19.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.125 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:19.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:19.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:41:19.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:19.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:41:19.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:41:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:41:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:41:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:41:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:41:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:41:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:41:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:41:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:41:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:41:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:41:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:41:19.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:41:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 98.319008178000004, 2024-04-22T18:41:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:41:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:41:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:41:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:41:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:41:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:41:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:41:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:41:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:41:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:41:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:41:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:41:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:41:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:41:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:41:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:41:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:41:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:41:19.134 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.135 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.135 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:19.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:19.137 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:19.137+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:19.138 INFO:tasks.rados.rados.0.smithi089.stdout:1831: set_chunk oid 34 offset: 304128 length: 651776 target oid ecb114dd8128c2ee55f0012396e3c703df81ed141276b0e8c72e2d08a6396004 offset: 0 snap -1 2024-04-22T18:41:19.139 INFO:tasks.rados.rados.0.smithi089.stdout:1832: read oid 30 snap -1 2024-04-22T18:41:19.139 INFO:tasks.rados.rados.0.smithi089.stdout:1832: expect (ObjNum 378 snap 0 seq_num 378) 2024-04-22T18:41:19.139 INFO:tasks.rados.rados.0.smithi089.stdout:1833: read oid 37 snap -1 2024-04-22T18:41:19.139 INFO:tasks.rados.rados.0.smithi089.stdout:1833: expect (ObjNum 376 snap 0 seq_num 376) 2024-04-22T18:41:19.139 INFO:tasks.rados.rados.0.smithi089.stdout:1834: read oid 49 snap -1 2024-04-22T18:41:19.139 INFO:tasks.rados.rados.0.smithi089.stdout:1834: expect (ObjNum 316 snap 0 seq_num 316) 2024-04-22T18:41:19.139 INFO:tasks.rados.rados.0.smithi089.stdout:1835: write (excl) oid 39 current snap is 0 2024-04-22T18:41:19.139 INFO:tasks.rados.rados.0.smithi089.stdout:1830: finishing set_chunk to oid 7 2024-04-22T18:41:19.139 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 7 set_chunk returned error code -95 offset: 2080256 length: 1025536 2024-04-22T18:41:19.139 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 0 (ObjNum 346 snap 0 seq_num 346) dirty exists 2024-04-22T18:41:19.139 INFO:tasks.rados.rados.0.smithi089.stdout:1835: seq_num 383 ranges {437081=638017,1679074=582789} 2024-04-22T18:41:19.139 INFO:tasks.rados.rados.0.smithi089.stdout:1835: writing smithi08937929-39 from 437081 to 1075098 tid 1 2024-04-22T18:41:19.139 INFO:tasks.rados.rados.0.smithi089.stdout:1835: writing smithi08937929-39 from 1679074 to 2261863 tid 2 2024-04-22T18:41:19.139 INFO:tasks.rados.rados.0.smithi089.stdout:1830: done (6 left) 2024-04-22T18:41:19.140 INFO:tasks.rados.rados.0.smithi089.stdout:1836: read oid 40 snap -1 2024-04-22T18:41:19.140 INFO:tasks.rados.rados.0.smithi089.stdout:1836: expect (ObjNum 354 snap 0 seq_num 354) 2024-04-22T18:41:19.140 INFO:tasks.rados.rados.0.smithi089.stdout:1837: read oid 5 snap -1 2024-04-22T18:41:19.140 INFO:tasks.rados.rados.0.smithi089.stdout:1837: expect (ObjNum 375 snap 0 seq_num 375) 2024-04-22T18:41:19.140 INFO:tasks.rados.rados.0.smithi089.stdout:1838: read oid 27 snap -1 2024-04-22T18:41:19.140 INFO:tasks.rados.rados.0.smithi089.stdout:1838: expect (ObjNum 287 snap 0 seq_num 287) 2024-04-22T18:41:19.140 INFO:tasks.rados.rados.0.smithi089.stdout:1839: set_chunk oid 23 target oid 2024-04-22T18:41:19.237 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:19.238+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:19.282 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:19.282 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:19.282 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:19.282 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:19.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:19.299 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:41:19.308 INFO:tasks.rados.rados.0.smithi089.stdout:1839: set_chunk oid 23 offset: 93184 length: 2834432 target oid 16bd782ee79c1c013e57d0aa1e4fa24bf46e2816c7fcdfd8f8d876a49cd9f826 offset: 0 snap -1 2024-04-22T18:41:19.308 INFO:tasks.rados.rados.0.smithi089.stdout:1840: read oid 48 snap -1 2024-04-22T18:41:19.308 INFO:tasks.rados.rados.0.smithi089.stdout:1840: expect (ObjNum 351 snap 0 seq_num 351) 2024-04-22T18:41:19.308 INFO:tasks.rados.rados.0.smithi089.stdout:1841: write oid 24 current snap is 0 2024-04-22T18:41:19.308 INFO:tasks.rados.rados.0.smithi089.stdout:1841: seq_num 384 ranges {453888=713266,1737556=442800} 2024-04-22T18:41:19.314 INFO:tasks.rados.rados.0.smithi089.stdout:1841: writing smithi08937929-24 from 453888 to 1167154 tid 1 2024-04-22T18:41:19.320 INFO:tasks.rados.rados.0.smithi089.stdout:1841: writing smithi08937929-24 from 1737556 to 2180356 tid 2 2024-04-22T18:41:19.320 INFO:tasks.rados.rados.0.smithi089.stdout:1842: write (excl) oid 2 current snap is 0 2024-04-22T18:41:19.320 INFO:tasks.rados.rados.0.smithi089.stdout:1821: expect (ObjNum 366 snap 0 seq_num 366) 2024-04-22T18:41:19.337 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:19.337+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:19.418 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:19.418 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:19.432 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:19.433 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:19.433 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:19.433 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:19.433 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:19.435 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:19.435 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:19.438 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:19.439+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:19.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:19.451 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:19.451 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:19.467 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:19.467 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:19.474 INFO:tasks.rados.rados.0.smithi089.stdout:1831: finishing set_chunk to oid 34 2024-04-22T18:41:19.474 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 34 set_chunk returned error code -95 offset: 304128 length: 651776 2024-04-22T18:41:19.474 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 0 (ObjNum 277 snap 0 seq_num 277) dirty exists 2024-04-22T18:41:19.475 INFO:tasks.rados.rados.0.smithi089.stdout:1832: expect (ObjNum 378 snap 0 seq_num 378) 2024-04-22T18:41:19.540 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:19.540+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:19.544 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:19.544 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:19.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:19.544 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:19.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.544 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:19.544 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:19.544 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 145.28457711799999, 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:19.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:41:19.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:19.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:41:19.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:41:19.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:41:19.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:41:19.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:41:19.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:41:19.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:41:19.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:41:19.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:41:19.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:41:19.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:19.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:41:19.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:41:19.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:41:19.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:41:19.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:41:19.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:41:19.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:41:19.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:41:19.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:41:19.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:41:19.549 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.550 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.550 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.550 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:19.550 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:19.550 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 145.27979468800001, 2024-04-22T18:41:19.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:41:19.550 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.550 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.550 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.550 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.550 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.550 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:41:19.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.551 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:19.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:19.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:41:19.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:19.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:41:19.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:41:19.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:41:19.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:41:19.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:41:19.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:41:19.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:41:19.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:41:19.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:41:19.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:41:19.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:19.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:41:19.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:41:19.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:41:19.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:41:19.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:41:19.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:41:19.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:41:19.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:41:19.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:41:19.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:41:19.555 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.555 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.555 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.555 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:19.555 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:19.556 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 145.27970048500001, 2024-04-22T18:41:19.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:41:19.556 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.556 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.556 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.556 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.556 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.556 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:41:19.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.556 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:19.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:19.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:41:19.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:19.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:41:19.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:41:19.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:41:19.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:41:19.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:41:19.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:41:19.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:41:19.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:41:19.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:41:19.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:41:19.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:19.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:41:19.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:41:19.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:41:19.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:41:19.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:41:19.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:41:19.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:41:19.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:41:19.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:41:19.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:41:19.561 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.561 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.561 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.561 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:19.561 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:19.561 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 145.27967959700001, 2024-04-22T18:41:19.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:41:19.561 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:19.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:41:19.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:41:19.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:41:19.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:41:19.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:41:19.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:19.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:41:19.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:41:19.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:41:19.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:41:19.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:41:19.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:41:19.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:41:19.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:41:19.566 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.566 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.566 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.566 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:19.566 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:19.566 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.49667193400001, 2024-04-22T18:41:19.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:41:19.566 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.566 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.566 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.566 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.566 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.566 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:41:19.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:19.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:41:19.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:41:19.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:41:19.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:41:19.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:41:19.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:41:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:41:19.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:41:19.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:41:19.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:41:19.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:41:19.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:41:19.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:41:19.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:41:19.571 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.571 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.571 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.571 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:19.571 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:19.571 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.488840528, 2024-04-22T18:41:19.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:41:19.572 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.572 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.572 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.572 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.572 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.572 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:41:19.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.572 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:19.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:19.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:41:19.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:19.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.573 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.573 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:41:19.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:41:19.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:41:19.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:41:19.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:41:19.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:41:19.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:41:19.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:41:19.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:41:19.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:41:19.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:19.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:41:19.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:41:19.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:41:19.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:41:19.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:41:19.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:41:19.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:41:19.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:41:19.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:41:19.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:41:19.576 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.577 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.577 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.577 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:19.577 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:19.577 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.336967897999997, 2024-04-22T18:41:19.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:41:19.577 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.577 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.577 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.577 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.577 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.577 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:41:19.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.578 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:19.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:19.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.578 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.578 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:41:19.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:41:19.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:41:19.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:41:19.582 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.582 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.582 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.582 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:19.582 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:19.582 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.336562990000004, 2024-04-22T18:41:19.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:41:19.582 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.582 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:19.582 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.583 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.583 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.583 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:41:19.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.583 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:19.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.584 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:41:19.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:41:19.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:41:19.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:41:19.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.585 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:41:19.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:41:19.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:19.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:41:19.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:41:19.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:41:19.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:41:19.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:41:19.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:41:19.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:41:19.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:41:19.587 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.587 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.587 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.587 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3213 4.a 4:5d2aee44:::smithi08937929-13:head [assert-version v66,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:19.587 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:19.587 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.975663101999999, 2024-04-22T18:41:19.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69928701400000004, 2024-04-22T18:41:19.587 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.587 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.587 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3213 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568559+0000", 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568567+0000", 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568579+0000", 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.668711+0000", 2024-04-22T18:41:19.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100131866 2024-04-22T18:41:19.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.768945+0000", 2024-04-22T18:41:19.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100234252 2024-04-22T18:41:19.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769002+0000", 2024-04-22T18:41:19.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7092999999999997e-05 2024-04-22T18:41:19.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:19.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.166404+0000", 2024-04-22T18:41:19.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39740241999999998 2024-04-22T18:41:19.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.191686+0000", 2024-04-22T18:41:19.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025281739000000001 2024-04-22T18:41:19.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267758+0000", 2024-04-22T18:41:19.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076071606999999999 2024-04-22T18:41:19.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267812+0000", 2024-04-22T18:41:19.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4478999999999998e-05 2024-04-22T18:41:19.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267848+0000", 2024-04-22T18:41:19.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6062999999999997e-05 2024-04-22T18:41:19.592 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.592 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.592 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.593 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3319 4.a 4:5d2aee44:::smithi08937929-13:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:41:19.593 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:19.593 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.083915351000002, 2024-04-22T18:41:19.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476623401, 2024-04-22T18:41:19.593 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.593 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:19.593 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.593 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.593 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.593 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3319 2024-04-22T18:41:19.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.593 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:19.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:19.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460306+0000", 2024-04-22T18:41:19.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:19.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460318+0000", 2024-04-22T18:41:19.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2896e-05 2024-04-22T18:41:19.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460345+0000", 2024-04-22T18:41:19.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6650000000000001e-05 2024-04-22T18:41:19.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460360+0000", 2024-04-22T18:41:19.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4729000000000001e-05 2024-04-22T18:41:19.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460502+0000", 2024-04-22T18:41:19.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141729 2024-04-22T18:41:19.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460644+0000", 2024-04-22T18:41:19.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142755 2024-04-22T18:41:19.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.936932+0000", 2024-04-22T18:41:19.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476288078 2024-04-22T18:41:19.597 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.597 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.597 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.597 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3412 4.a 4:519c08f7:::smithi08937929-15:head [write 2529878~394573] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:19.597 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:19.597 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 48.402080789999999, 2024-04-22T18:41:19.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48918094899999998, 2024-04-22T18:41:19.597 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.597 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.597 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.597 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.598 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.598 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3412 2024-04-22T18:41:19.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.598 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:19.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:19.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142141+0000", 2024-04-22T18:41:19.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:19.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142496+0000", 2024-04-22T18:41:19.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035444699999999998 2024-04-22T18:41:19.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142510+0000", 2024-04-22T18:41:19.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3841e-05 2024-04-22T18:41:19.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142518+0000", 2024-04-22T18:41:19.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8089999999999993e-06 2024-04-22T18:41:19.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142560+0000", 2024-04-22T18:41:19.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2097e-05 2024-04-22T18:41:19.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142637+0000", 2024-04-22T18:41:19.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6965000000000006e-05 2024-04-22T18:41:19.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:19.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142715+0000", 2024-04-22T18:41:19.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8694000000000002e-05 2024-04-22T18:41:19.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144179+0000", 2024-04-22T18:41:19.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146342 2024-04-22T18:41:19.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631229+0000", 2024-04-22T18:41:19.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48705018500000002 2024-04-22T18:41:19.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631281+0000", 2024-04-22T18:41:19.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2216999999999998e-05 2024-04-22T18:41:19.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631324+0000", 2024-04-22T18:41:19.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3344999999999999e-05 2024-04-22T18:41:19.602 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.602 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.602 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3413 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2924451] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 48.401699649999998, 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58904086899999997, 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3413 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:19.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:19.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142523+0000", 2024-04-22T18:41:19.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:19.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142529+0000", 2024-04-22T18:41:19.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6049999999999997e-06 2024-04-22T18:41:19.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142535+0000", 2024-04-22T18:41:19.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4740000000000002e-06 2024-04-22T18:41:19.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142541+0000", 2024-04-22T18:41:19.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1839999999999998e-06 2024-04-22T18:41:19.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143553+0000", 2024-04-22T18:41:19.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010126689999999999 2024-04-22T18:41:19.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143659+0000", 2024-04-22T18:41:19.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105393 2024-04-22T18:41:19.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:19.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143825+0000", 2024-04-22T18:41:19.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165809 2024-04-22T18:41:19.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144329+0000", 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050483799999999997 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731480+0000", 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58715100600000003 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731522+0000", 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2098000000000002e-05 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731566+0000", 2024-04-22T18:41:19.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3066999999999999e-05 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3414 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 48.401671045999997, 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58926020199999996, 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3414 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.608 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142552+0000", 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142556+0000", 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0289999999999999e-06 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142574+0000", 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7266999999999999e-05 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142581+0000", 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144121+0000", 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015397480000000001 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144170+0000", 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8803e-05 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731624+0000", 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58745365800000005 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731712+0000", 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8016999999999994e-05 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731813+0000", 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101817 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:77 3.3 3:dcc8aeec:::3590d5741efdda41a26b62a0982727439225b7b4b51097b313f30d8dc10bf6f6:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 44.296652723999998, 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011812878, 2024-04-22T18:41:19.612 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247571+0000", 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:19.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247574+0000", 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247577+0000", 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3069999999999998e-06 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247580+0000", 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8870000000000001e-06 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.340928+0000", 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093348678000000004 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341122+0000", 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000193227 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618335+0000", 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.277213775 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:19.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618422+0000", 2024-04-22T18:41:19.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6537000000000002e-05 2024-04-22T18:41:19.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954387+0000", 2024-04-22T18:41:19.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33596509699999999 2024-04-22T18:41:19.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954548+0000", 2024-04-22T18:41:19.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016042199999999999 2024-04-22T18:41:19.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:19.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954858+0000", 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000310396 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157019+0000", 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20216071299999999 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259213+0000", 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102194734 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259262+0000", 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9052999999999999e-05 2024-04-22T18:41:19.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259384+0000", 2024-04-22T18:41:19.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122052 2024-04-22T18:41:19.618 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.618 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.618 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.618 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:78 3.3 3:df7e90d2:::ab0657e24bd96704775de2a378746bc22309c5a7366a5eaf9b4e1f1f2bef0a4c:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:19.618 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:19.618 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 44.296637527000001, 2024-04-22T18:41:19.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011974739, 2024-04-22T18:41:19.618 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.618 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247586+0000", 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:19.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247589+0000", 2024-04-22T18:41:19.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.181e-06 2024-04-22T18:41:19.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247592+0000", 2024-04-22T18:41:19.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0230000000000001e-06 2024-04-22T18:41:19.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247595+0000", 2024-04-22T18:41:19.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.428e-06 2024-04-22T18:41:19.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341160+0000", 2024-04-22T18:41:19.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093564571999999999 2024-04-22T18:41:19.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:19.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341264+0000", 2024-04-22T18:41:19.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010391999999999999 2024-04-22T18:41:19.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619269+0000", 2024-04-22T18:41:19.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27800575100000002 2024-04-22T18:41:19.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:19.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619346+0000", 2024-04-22T18:41:19.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6459999999999996e-05 2024-04-22T18:41:19.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.155957+0000", 2024-04-22T18:41:19.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53661109299999998 2024-04-22T18:41:19.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156049+0000", 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2272e-05 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156236+0000", 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000186808 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.158088+0000", 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018522079999999999 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259421+0000", 2024-04-22T18:41:19.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101332718 2024-04-22T18:41:19.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259460+0000", 2024-04-22T18:41:19.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8912000000000003e-05 2024-04-22T18:41:19.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259562+0000", 2024-04-22T18:41:19.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010175600000000001 2024-04-22T18:41:19.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.624 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.625 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:80 3.3 3:f90f47f2:::e5603f596d699ee3e27e255225eeb90a9f0c1b4a4d099d1c586b19019220aa3f:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:19.625 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:19.625 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 44.296598449999998, 2024-04-22T18:41:19.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0121096759999999, 2024-04-22T18:41:19.625 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.625 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.625 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:19.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:19.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247625+0000", 2024-04-22T18:41:19.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:19.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247628+0000", 2024-04-22T18:41:19.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6129999999999998e-06 2024-04-22T18:41:19.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247631+0000", 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3869999999999999e-06 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247634+0000", 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9919999999999999e-06 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341550+0000", 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093915584999999996 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341700+0000", 2024-04-22T18:41:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015000300000000001 2024-04-22T18:41:19.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620744+0000", 2024-04-22T18:41:19.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27904399200000002 2024-04-22T18:41:19.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:19.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620794+0000", 2024-04-22T18:41:19.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9579000000000001e-05 2024-04-22T18:41:19.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157597+0000", 2024-04-22T18:41:19.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53680343799999997 2024-04-22T18:41:19.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157678+0000", 2024-04-22T18:41:19.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0829999999999997e-05 2024-04-22T18:41:19.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:19.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157832+0000", 2024-04-22T18:41:19.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154184 2024-04-22T18:41:19.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259572+0000", 2024-04-22T18:41:19.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10174003600000001 2024-04-22T18:41:19.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259601+0000", 2024-04-22T18:41:19.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8841999999999999e-05 2024-04-22T18:41:19.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259638+0000", 2024-04-22T18:41:19.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7308999999999998e-05 2024-04-22T18:41:19.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259736+0000", 2024-04-22T18:41:19.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7283999999999999e-05 2024-04-22T18:41:19.631 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.631 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.631 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.631 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:135 3.3 3:f89134e5:::3e4e73f5d4a0122957865f0faeac2483204ee98d7d497ef3b54afe2a02b39a3d:head [read 0~802304 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:19.631 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:19.631 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 43.928783371000002, 2024-04-22T18:41:19.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.64370653499999997, 2024-04-22T18:41:19.631 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.631 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:19.631 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615438+0000", 2024-04-22T18:41:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615449+0000", 2024-04-22T18:41:19.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1219999999999999e-05 2024-04-22T18:41:19.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615460+0000", 2024-04-22T18:41:19.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0403e-05 2024-04-22T18:41:19.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.715588+0000", 2024-04-22T18:41:19.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100128881 2024-04-22T18:41:19.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722533+0000", 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069444010000000002 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722658+0000", 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012560699999999999 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826437+0000", 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103778194 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826532+0000", 2024-04-22T18:41:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5111000000000005e-05 2024-04-22T18:41:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258787+0000", 2024-04-22T18:41:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43225529099999999 2024-04-22T18:41:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258905+0000", 2024-04-22T18:41:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118348 2024-04-22T18:41:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259148+0000", 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000242093 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3890 4.1 4:83c955bf:::smithi08937929-28:head [write 2699879~469889] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.930108527000002, 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49042297000000001, 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.636 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.637 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.637 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3890 2024-04-22T18:41:19.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.637 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:19.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:19.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614115+0000", 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614496+0000", 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038181700000000001 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614509+0000", 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2381000000000001e-05 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614517+0000", 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0879999999999995e-06 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615610+0000", 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010933519999999999 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615655+0000", 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4966000000000003e-05 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:19.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615769+0000", 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113986 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704882+0000", 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089112825000000007 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104488+0000", 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39960572999999999 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104517+0000", 2024-04-22T18:41:19.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9006e-05 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104539+0000", 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2073e-05 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3891 4.1 4:83c955bf:::smithi08937929-28:head [setxattr _header (56) in=7b,truncate 3630345] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.929700296, 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59018588699999996, 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:19.641 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.642 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.642 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.642 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3891 2024-04-22T18:41:19.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.642 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:19.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:19.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614523+0000", 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614528+0000", 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0379999999999999e-06 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614540+0000", 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018000000000001e-05 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614546+0000", 2024-04-22T18:41:19.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4990000000000002e-06 2024-04-22T18:41:19.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704438+0000", 2024-04-22T18:41:19.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089892044000000004 2024-04-22T18:41:19.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704504+0000", 2024-04-22T18:41:19.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6365000000000006e-05 2024-04-22T18:41:19.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:19.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704602+0000", 2024-04-22T18:41:19.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8048999999999996e-05 2024-04-22T18:41:19.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:19.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.806370+0000", 2024-04-22T18:41:19.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10176768 2024-04-22T18:41:19.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:19.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204629+0000", 2024-04-22T18:41:19.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39825917999999999 2024-04-22T18:41:19.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:19.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204673+0000", 2024-04-22T18:41:19.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3516999999999997e-05 2024-04-22T18:41:19.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204710+0000", 2024-04-22T18:41:19.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7391000000000002e-05 2024-04-22T18:41:19.646 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.646 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.646 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.647 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3892 4.1 4:83c955bf:::smithi08937929-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:19.647 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:19.647 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 29.929673715, 2024-04-22T18:41:19.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59031281800000002, 2024-04-22T18:41:19.647 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:19.647 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:19.647 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:19.647 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:19.647 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:19.647 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3892 2024-04-22T18:41:19.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.647 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:19.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:19.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614550+0000", 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614553+0000", 2024-04-22T18:41:19.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0879999999999999e-06 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614559+0000", 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9259999999999997e-06 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614565+0000", 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7950000000000004e-06 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704818+0000", 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090252624000000004 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:19.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704866+0000", 2024-04-22T18:41:19.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8667e-05 2024-04-22T18:41:19.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:19.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204742+0000", 2024-04-22T18:41:19.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987551899999999 2024-04-22T18:41:19.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:19.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204799+0000", 2024-04-22T18:41:19.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7299000000000001e-05 2024-04-22T18:41:19.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:19.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:19.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:19.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204863+0000", 2024-04-22T18:41:19.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4455000000000005e-05 2024-04-22T18:41:19.651 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.651 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.651 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.651 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:19.651 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:19.651 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:19.652 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:19.652 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:19.653 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:19.653 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:19.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:19.654 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:19.653+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:19.654 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:19.654 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:19.655 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:19.655 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:19.658 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:41:19.678 INFO:tasks.rados.rados.0.smithi089.stdout:1835: finishing write tid 1 to smithi08937929-39 2024-04-22T18:41:19.678 INFO:tasks.rados.rados.0.smithi089.stdout:1833: expect (ObjNum 376 snap 0 seq_num 376) 2024-04-22T18:41:19.753 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:19.754+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:19.763 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:19.763+0000 7fec55bea640 -1 osd.7 143 handle_osd_map injecting map crc failure 2024-04-22T18:41:19.796 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:19.796 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:19.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:19.807 INFO:tasks.rados.rados.0.smithi089.stdout:1834: expect (ObjNum 316 snap 0 seq_num 316) 2024-04-22T18:41:19.889 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:19.890+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:19.946 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:19.946 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:19.955 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:19.955 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:19.955+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:19.981 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 7, 2, 4] out_osds: [5, 0, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:41:19.982 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:41:19.982 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-04-22T18:41:19.982 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-22T18:41:19.982 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-22T18:41:19.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2024-04-22T18:41:20.023 INFO:tasks.rados.rados.0.smithi089.stdout:1835: finishing write tid 2 to smithi08937929-39 2024-04-22T18:41:20.023 INFO:tasks.rados.rados.0.smithi089.stdout:1835: finishing write tid 3 to smithi08937929-39 2024-04-22T18:41:20.023 INFO:tasks.rados.rados.0.smithi089.stdout:1835: finishing write tid 4 to smithi08937929-39 2024-04-22T18:41:20.023 INFO:tasks.rados.rados.0.smithi089.stdout:1835: oid 39 updating version 0 to 143 2024-04-22T18:41:20.023 INFO:tasks.rados.rados.0.smithi089.stdout:1835: oid 39 version 143 is already newer than 142 2024-04-22T18:41:20.024 INFO:tasks.rados.rados.0.smithi089.stdout:1835: oid 39 version 143 is already newer than 141 2024-04-22T18:41:20.024 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 143 (ObjNum 383 snap 0 seq_num 383) dirty exists 2024-04-22T18:41:20.024 INFO:tasks.rados.rados.0.smithi089.stdout:1835: left oid 39 (ObjNum 383 snap 0 seq_num 383) 2024-04-22T18:41:20.024 INFO:tasks.rados.rados.0.smithi089.stdout:1836: expect (ObjNum 354 snap 0 seq_num 354) 2024-04-22T18:41:20.055 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:20.056+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:20.098 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:20.098 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "" 2024-04-22T18:41:20.098 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:20.099 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:20.099 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:20.108 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:20.156 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:20.156+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:20.201 INFO:tasks.rados.rados.0.smithi089.stdout:1837: expect (ObjNum 375 snap 0 seq_num 375) 2024-04-22T18:41:20.211 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:20.211 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:20.220 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:20.262 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:20.262+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:20.358 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:20.358+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:20.363 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:20.363 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:20.370 INFO:tasks.rados.rados.0.smithi089.stdout:1838: expect (ObjNum 287 snap 0 seq_num 287) 2024-04-22T18:41:20.372 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:20.459 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:20.459+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:20.515 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:20.515 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:20.524 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:20.541 INFO:tasks.rados.rados.0.smithi089.stdout:1839: finishing set_chunk to oid 23 2024-04-22T18:41:20.541 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 23 set_chunk returned error code -95 offset: 93184 length: 2834432 2024-04-22T18:41:20.541 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 0 (ObjNum 339 snap 0 seq_num 339) dirty exists 2024-04-22T18:41:20.541 INFO:tasks.rados.rados.0.smithi089.stdout:1841: finishing write tid 1 to smithi08937929-24 2024-04-22T18:41:20.541 INFO:tasks.rados.rados.0.smithi089.stdout:1841: finishing write tid 2 to smithi08937929-24 2024-04-22T18:41:20.541 INFO:tasks.rados.rados.0.smithi089.stdout:1842: seq_num 385 ranges {470695=788515,1679685=586458} 2024-04-22T18:41:20.547 INFO:tasks.rados.rados.0.smithi089.stdout:1842: writing smithi08937929-2 from 470695 to 1259210 tid 1 2024-04-22T18:41:20.553 INFO:tasks.rados.rados.0.smithi089.stdout:1842: writing smithi08937929-2 from 1679685 to 2266143 tid 2 2024-04-22T18:41:20.553 INFO:tasks.rados.rados.0.smithi089.stdout:1821: done (12 left) 2024-04-22T18:41:20.553 INFO:tasks.rados.rados.0.smithi089.stdout:1831: done (11 left) 2024-04-22T18:41:20.553 INFO:tasks.rados.rados.0.smithi089.stdout:1832: done (10 left) 2024-04-22T18:41:20.553 INFO:tasks.rados.rados.0.smithi089.stdout:1833: done (9 left) 2024-04-22T18:41:20.553 INFO:tasks.rados.rados.0.smithi089.stdout:1834: done (8 left) 2024-04-22T18:41:20.554 INFO:tasks.rados.rados.0.smithi089.stdout:1835: done (7 left) 2024-04-22T18:41:20.554 INFO:tasks.rados.rados.0.smithi089.stdout:1836: done (6 left) 2024-04-22T18:41:20.554 INFO:tasks.rados.rados.0.smithi089.stdout:1837: done (5 left) 2024-04-22T18:41:20.554 INFO:tasks.rados.rados.0.smithi089.stdout:1838: done (4 left) 2024-04-22T18:41:20.554 INFO:tasks.rados.rados.0.smithi089.stdout:1839: done (3 left) 2024-04-22T18:41:20.554 INFO:tasks.rados.rados.0.smithi089.stdout:1843: read oid 46 snap -1 2024-04-22T18:41:20.554 INFO:tasks.rados.rados.0.smithi089.stdout:1843: expect (ObjNum 343 snap 0 seq_num 343) 2024-04-22T18:41:20.554 INFO:tasks.rados.rados.0.smithi089.stdout:1844: set_chunk oid 28 target oid 2024-04-22T18:41:20.560 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:20.561+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:20.571 INFO:tasks.rados.rados.0.smithi089.stdout:1844: set_chunk oid 28 offset: 1522688 length: 1690624 target oid 709e5cc5ca9f3cc840701c3edfbee99b3b0c0c8648db1c89fe6bfc1d382e2910 offset: 0 snap -1 2024-04-22T18:41:20.571 INFO:tasks.rados.rados.0.smithi089.stdout:1845: tier_promote oid 26 2024-04-22T18:41:20.571 INFO:tasks.rados.rados.0.smithi089.stdout:1840: expect (ObjNum 351 snap 0 seq_num 351) 2024-04-22T18:41:20.646 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:20.646 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:20.657 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:20.661 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:20.662+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:20.703 INFO:tasks.rados.rados.0.smithi089.stdout:1841: finishing write tid 3 to smithi08937929-24 2024-04-22T18:41:20.703 INFO:tasks.rados.rados.0.smithi089.stdout:1841: finishing write tid 4 to smithi08937929-24 2024-04-22T18:41:20.703 INFO:tasks.rados.rados.0.smithi089.stdout:1841: oid 24 updating version 0 to 180 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:1841: oid 24 updating version 180 to 181 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:1841: oid 24 version 181 is already newer than 179 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 181 (ObjNum 384 snap 0 seq_num 384) dirty exists 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:1841: left oid 24 (ObjNum 384 snap 0 seq_num 384) 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:1840: done (5 left) 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:1841: done (4 left) 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:1846: read oid 18 snap -1 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:1846: expect (ObjNum 307 snap 0 seq_num 307) 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:1847: read oid 35 snap -1 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:1847: expect deleted 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:1848: write (excl) oid 19 current snap is 0 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:1844: finishing set_chunk to oid 28 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 28 set_chunk returned error code -95 offset: 1522688 length: 1690624 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 28 v 0 (ObjNum 355 snap 0 seq_num 355) dirty exists 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:1842: finishing write tid 1 to smithi08937929-2 2024-04-22T18:41:20.704 INFO:tasks.rados.rados.0.smithi089.stdout:1842: finishing write tid 2 to smithi08937929-2 2024-04-22T18:41:20.705 INFO:tasks.rados.rados.0.smithi089.stdout:1842: finishing write tid 3 to smithi08937929-2 2024-04-22T18:41:20.705 INFO:tasks.rados.rados.0.smithi089.stdout:1842: finishing write tid 4 to smithi08937929-2 2024-04-22T18:41:20.705 INFO:tasks.rados.rados.0.smithi089.stdout:1842: oid 2 updating version 0 to 78 2024-04-22T18:41:20.705 INFO:tasks.rados.rados.0.smithi089.stdout:1842: oid 2 version 78 is already newer than 76 2024-04-22T18:41:20.705 INFO:tasks.rados.rados.0.smithi089.stdout:1842: oid 2 version 78 is already newer than 77 2024-04-22T18:41:20.705 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 78 (ObjNum 385 snap 0 seq_num 385) dirty exists 2024-04-22T18:41:20.705 INFO:tasks.rados.rados.0.smithi089.stdout:1842: left oid 2 (ObjNum 385 snap 0 seq_num 385) 2024-04-22T18:41:20.705 INFO:tasks.rados.rados.0.smithi089.stdout:1843: expect (ObjNum 343 snap 0 seq_num 343) 2024-04-22T18:41:20.762 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:20.762+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:20.800 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:20.801 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:20.810 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:20.863 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:20.864+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:20.867 INFO:tasks.rados.rados.0.smithi089.stdout:1845: got (0) Success 2024-04-22T18:41:20.867 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 134 (ObjNum 353 snap 0 seq_num 353) dirty exists 2024-04-22T18:41:20.867 INFO:tasks.rados.rados.0.smithi089.stdout:1846: expect (ObjNum 307 snap 0 seq_num 307) 2024-04-22T18:41:20.952 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:20.952 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:20.961 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:20.965 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:20.966+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:21.065 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:21.067+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:21.072 INFO:tasks.rados.rados.0.smithi089.stdout:1848: seq_num 386 ranges {487502=463764,1738167=446469} 2024-04-22T18:41:21.077 INFO:tasks.rados.rados.0.smithi089.stdout:1848: writing smithi08937929-19 from 487502 to 951266 tid 1 2024-04-22T18:41:21.085 INFO:tasks.rados.rados.0.smithi089.stdout:1848: writing smithi08937929-19 from 1738167 to 2184636 tid 2 2024-04-22T18:41:21.085 INFO:tasks.rados.rados.0.smithi089.stdout:1842: done (6 left) 2024-04-22T18:41:21.085 INFO:tasks.rados.rados.0.smithi089.stdout:1843: done (5 left) 2024-04-22T18:41:21.085 INFO:tasks.rados.rados.0.smithi089.stdout:1844: done (4 left) 2024-04-22T18:41:21.085 INFO:tasks.rados.rados.0.smithi089.stdout:1845: done (3 left) 2024-04-22T18:41:21.085 INFO:tasks.rados.rados.0.smithi089.stdout:1846: done (2 left) 2024-04-22T18:41:21.085 INFO:tasks.rados.rados.0.smithi089.stdout:1847: done (1 left) 2024-04-22T18:41:21.085 INFO:tasks.rados.rados.0.smithi089.stdout:1849: read oid 14 snap -1 2024-04-22T18:41:21.085 INFO:tasks.rados.rados.0.smithi089.stdout:1849: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:41:21.085 INFO:tasks.rados.rados.0.smithi089.stdout:1850: read oid 5 snap -1 2024-04-22T18:41:21.085 INFO:tasks.rados.rados.0.smithi089.stdout:1850: expect (ObjNum 375 snap 0 seq_num 375) 2024-04-22T18:41:21.085 INFO:tasks.rados.rados.0.smithi089.stdout:1851: read oid 39 snap -1 2024-04-22T18:41:21.085 INFO:tasks.rados.rados.0.smithi089.stdout:1851: expect (ObjNum 383 snap 0 seq_num 383) 2024-04-22T18:41:21.086 INFO:tasks.rados.rados.0.smithi089.stdout:1852: tier_evict oid 13 2024-04-22T18:41:21.086 INFO:tasks.rados.rados.0.smithi089.stdout:1852: tier_evict oid 13 snap -1 2024-04-22T18:41:21.086 INFO:tasks.rados.rados.0.smithi089.stdout:1853: write oid 20 current snap is 0 2024-04-22T18:41:21.086 INFO:tasks.rados.rados.0.smithi089.stdout:1853: seq_num 387 ranges {504309=539013,1796649=590127} 2024-04-22T18:41:21.089 INFO:tasks.rados.rados.0.smithi089.stdout:1853: writing smithi08937929-20 from 504309 to 1043322 tid 1 2024-04-22T18:41:21.097 INFO:tasks.rados.rados.0.smithi089.stdout:1853: writing smithi08937929-20 from 1796649 to 2386776 tid 2 2024-04-22T18:41:21.097 INFO:tasks.rados.rados.0.smithi089.stdout:1854: tier_evict oid 31 2024-04-22T18:41:21.097 INFO:tasks.rados.rados.0.smithi089.stdout:1854: tier_evict oid 31 snap -1 2024-04-22T18:41:21.097 INFO:tasks.rados.rados.0.smithi089.stdout:1853: finishing write tid 1 to smithi08937929-20 2024-04-22T18:41:21.098 INFO:tasks.rados.rados.0.smithi089.stdout:1855: delete oid 8 current snap is 0 2024-04-22T18:41:21.098 INFO:tasks.rados.rados.0.smithi089.stdout:1854: got (2) No such file or directory 2024-04-22T18:41:21.098 INFO:tasks.rados.rados.0.smithi089.stdout:1854: got expected ENOENT (src dne) 2024-04-22T18:41:21.102 INFO:tasks.rados.rados.0.smithi089.stdout:1853: finishing write tid 2 to smithi08937929-20 2024-04-22T18:41:21.102 INFO:tasks.rados.rados.0.smithi089.stdout:1853: finishing write tid 3 to smithi08937929-20 2024-04-22T18:41:21.102 INFO:tasks.rados.rados.0.smithi089.stdout:1853: finishing write tid 4 to smithi08937929-20 2024-04-22T18:41:21.102 INFO:tasks.rados.rados.0.smithi089.stdout:1853: oid 20 updating version 0 to 26 2024-04-22T18:41:21.102 INFO:tasks.rados.rados.0.smithi089.stdout:1853: oid 20 updating version 26 to 27 2024-04-22T18:41:21.103 INFO:tasks.rados.rados.0.smithi089.stdout:1853: oid 20 version 27 is already newer than 25 2024-04-22T18:41:21.103 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 27 (ObjNum 387 snap 0 seq_num 387) dirty exists 2024-04-22T18:41:21.103 INFO:tasks.rados.rados.0.smithi089.stdout:1853: left oid 20 (ObjNum 387 snap 0 seq_num 387) 2024-04-22T18:41:21.107 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:21.107 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:21.116 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:21.166 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:21.167+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:21.181 INFO:tasks.rados.rados.0.smithi089.stdout:1848: finishing write tid 1 to smithi08937929-19 2024-04-22T18:41:21.183 INFO:tasks.rados.rados.0.smithi089.stdout:1848: finishing write tid 2 to smithi08937929-19 2024-04-22T18:41:21.183 INFO:tasks.rados.rados.0.smithi089.stdout:1848: finishing write tid 3 to smithi08937929-19 2024-04-22T18:41:21.184 INFO:tasks.rados.rados.0.smithi089.stdout:1848: finishing write tid 4 to smithi08937929-19 2024-04-22T18:41:21.184 INFO:tasks.rados.rados.0.smithi089.stdout:1848: oid 19 updating version 0 to 142 2024-04-22T18:41:21.184 INFO:tasks.rados.rados.0.smithi089.stdout:1848: oid 19 version 142 is already newer than 141 2024-04-22T18:41:21.184 INFO:tasks.rados.rados.0.smithi089.stdout:1848: oid 19 updating version 142 to 143 2024-04-22T18:41:21.184 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 143 (ObjNum 386 snap 0 seq_num 386) dirty exists 2024-04-22T18:41:21.184 INFO:tasks.rados.rados.0.smithi089.stdout:1848: left oid 19 (ObjNum 386 snap 0 seq_num 386) 2024-04-22T18:41:21.184 INFO:tasks.rados.rados.0.smithi089.stdout:1852: got (22) Invalid argument 2024-04-22T18:41:21.192 INFO:tasks.rados.rados.0.smithi089.stdout:1849: expect (ObjNum 163 snap 0 seq_num 163) 2024-04-22T18:41:21.220 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:21.220 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:21.229 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:21.267 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:21.268+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:21.296 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:21.296+0000 7f1e2dd22640 -1 osd.2 144 handle_osd_map injecting map crc failure 2024-04-22T18:41:21.324 INFO:tasks.rados.rados.0.smithi089.stdout:1851: expect (ObjNum 383 snap 0 seq_num 383) 2024-04-22T18:41:21.368 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:21.369+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:21.373 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:21.373 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:21.382 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:21.469 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:21.470+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:21.523 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:21.523 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:21.532 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:21.532 INFO:tasks.rados.rados.0.smithi089.stdout:1850: expect (ObjNum 375 snap 0 seq_num 375) 2024-04-22T18:41:21.570 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:21.572+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:21.632 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:21.633 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:21.642 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:21.656 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:41:21.672 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:41:21.686 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:21.686+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:21.703 INFO:tasks.rados.rados.0.smithi089.stdout:1848: done (7 left) 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1849: done (6 left) 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1850: done (5 left) 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1851: done (4 left) 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1852: done (3 left) 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1853: done (2 left) 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1854: done (1 left) 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1855: done (0 left) 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1856: tier_flush oid 24 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1856: tier_flush oid 24 snap -1 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1857: read oid 25 snap -1 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1857: expect (ObjNum 326 snap 0 seq_num 326) 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1858: read oid 16 snap -1 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1858: expect (ObjNum 372 snap 0 seq_num 372) 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1859: tier_evict oid 12 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1859: tier_evict oid 12 snap -1 2024-04-22T18:41:21.704 INFO:tasks.rados.rados.0.smithi089.stdout:1860: set_chunk oid 3 target oid 2024-04-22T18:41:21.723 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:21.724+0000 7f3cfee88640 -1 osd.6 145 handle_osd_map injecting map crc failure 2024-04-22T18:41:21.749 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:21.749 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:21.760 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:21.801 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:21.802+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:21.815 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:21.815 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:21.832 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:21.832 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:21.856 INFO:tasks.rados.rados.0.smithi089.stdout:1860: set_chunk oid 3 offset: 333312 length: 935424 target oid d9c5d102ccc69e163ac5c2e6dc3d99e8afc418470ce44de4c86cac3f7de2507f offset: 0 snap -1 2024-04-22T18:41:21.856 INFO:tasks.rados.rados.0.smithi089.stdout:1861: read oid 42 snap -1 2024-04-22T18:41:21.856 INFO:tasks.rados.rados.0.smithi089.stdout:1861: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:41:21.856 INFO:tasks.rados.rados.0.smithi089.stdout:1859: got (22) Invalid argument 2024-04-22T18:41:21.856 INFO:tasks.rados.rados.0.smithi089.stdout:1857: expect (ObjNum 326 snap 0 seq_num 326) 2024-04-22T18:41:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:21.872 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:21.873+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:21.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:21.883 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:21.898 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:21.899 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:21.915 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:21.915 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:21.932 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:21.932 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:21.949 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:21.949 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:21.965 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:21.966 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:21.973 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:21.973+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:22.021 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:22.021 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:22.021 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:22.021 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:22.021 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:22.029 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:22.073 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:22.074+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 147.46994620199999, 2024-04-22T18:41:22.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:41:22.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:41:22.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:90 4.1 4:81200c09:::smithi08937929-23:head [write 2021192~348562] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 147.369109904, 2024-04-22T18:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55441794600000005, 2024-04-22T18:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.138 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.139 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.139 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.139 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 90 2024-04-22T18:41:22.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.139 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:41:22.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763214+0000", 2024-04-22T18:41:22.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763213+0000", 2024-04-22T18:41:22.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:22.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763354+0000", 2024-04-22T18:41:22.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014095500000000001 2024-04-22T18:41:22.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763362+0000", 2024-04-22T18:41:22.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7659999999999998e-06 2024-04-22T18:41:22.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763367+0000", 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1730000000000003e-06 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764390+0000", 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001023035 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764459+0000", 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8251999999999996e-05 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764579+0000", 2024-04-22T18:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000120184 2024-04-22T18:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871053+0000", 2024-04-22T18:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.106473849 2024-04-22T18:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317580+0000", 2024-04-22T18:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44652768799999998 2024-04-22T18:41:22.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317608+0000", 2024-04-22T18:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7752000000000001e-05 2024-04-22T18:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317632+0000", 2024-04-22T18:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.4437e-05 2024-04-22T18:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.143 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 147.36893802500001, 2024-04-22T18:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:22.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:22.148 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 147.36892764000001, 2024-04-22T18:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:41:22.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:41:22.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:41:22.153 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 143.03117721300001, 2024-04-22T18:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.154 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:41:22.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 115.996529698, 2024-04-22T18:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:41:22.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:41:22.163 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4177 4.19 4.f75b2379 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.967476400999999, 2024-04-22T18:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76236081700000002, 2024-04-22T18:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.164 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4177 2024-04-22T18:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164847+0000", 2024-04-22T18:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164854+0000", 2024-04-22T18:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2620000000000001e-06 2024-04-22T18:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164864+0000", 2024-04-22T18:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0400999999999999e-05 2024-04-22T18:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164876+0000", 2024-04-22T18:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1204e-05 2024-04-22T18:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164975+0000", 2024-04-22T18:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9562999999999995e-05 2024-04-22T18:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164980+0000", 2024-04-22T18:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7310000000000001e-06 2024-04-22T18:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927200+0000", 2024-04-22T18:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.762219973 2024-04-22T18:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927209+0000", 2024-04-22T18:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9050000000000003e-06 2024-04-22T18:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4182 4.c 4.5345b7ac (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.967445082000001, 2024-04-22T18:41:22.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76185930999999996, 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4182 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164881+0000", 2024-04-22T18:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.305e-06 2024-04-22T18:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164885+0000", 2024-04-22T18:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.873e-06 2024-04-22T18:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164890+0000", 2024-04-22T18:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6299999999999997e-06 2024-04-22T18:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225475+0000", 2024-04-22T18:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060585181000000002 2024-04-22T18:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225483+0000", 2024-04-22T18:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6790000000000005e-06 2024-04-22T18:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926728+0000", 2024-04-22T18:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124542099999998 2024-04-22T18:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926739+0000", 2024-04-22T18:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0669000000000001e-05 2024-04-22T18:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.172 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4183 4.c 4.5345b7ac (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.967431825999999, 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76187489500000005, 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4183 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164892+0000", 2024-04-22T18:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164894+0000", 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.705e-06 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164897+0000", 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2739999999999999e-06 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164901+0000", 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.636e-06 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225508+0000", 2024-04-22T18:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060607564000000003 2024-04-22T18:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225515+0000", 2024-04-22T18:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1829999999999999e-06 2024-04-22T18:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926759+0000", 2024-04-22T18:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.701244859 2024-04-22T18:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926767+0000", 2024-04-22T18:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0180000000000001e-06 2024-04-22T18:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4190 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.967414904000002, 2024-04-22T18:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.761884546, 2024-04-22T18:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:22.177 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4190 2024-04-22T18:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164912+0000", 2024-04-22T18:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6249999999999999e-06 2024-04-22T18:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164918+0000", 2024-04-22T18:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6039999999999998e-06 2024-04-22T18:41:22.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164922+0000", 2024-04-22T18:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9319999999999998e-06 2024-04-22T18:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225536+0000", 2024-04-22T18:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060613725 2024-04-22T18:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225540+0000", 2024-04-22T18:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5030000000000002e-06 2024-04-22T18:41:22.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926786+0000", 2024-04-22T18:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124673999999998 2024-04-22T18:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926794+0000", 2024-04-22T18:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7689999999999996e-06 2024-04-22T18:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.181 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4191 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.967399483000001, 2024-04-22T18:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76190695500000005, 2024-04-22T18:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4191 2024-04-22T18:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164927+0000", 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0959999999999999e-06 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164937+0000", 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0730999999999999e-05 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164944+0000", 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5130000000000002e-06 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225565+0000", 2024-04-22T18:41:22.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060621421000000002 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225569+0000", 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.264e-06 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926813+0000", 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124469 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926832+0000", 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8580000000000002e-05 2024-04-22T18:41:22.185 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4192 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.967374875000001, 2024-04-22T18:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76191075399999997, 2024-04-22T18:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4192 2024-04-22T18:41:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:22.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164953+0000", 2024-04-22T18:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1099999999999996e-06 2024-04-22T18:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164965+0000", 2024-04-22T18:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2298e-05 2024-04-22T18:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164972+0000", 2024-04-22T18:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5230000000000001e-06 2024-04-22T18:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225581+0000", 2024-04-22T18:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060609356000000003 2024-04-22T18:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225584+0000", 2024-04-22T18:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.106e-06 2024-04-22T18:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926852+0000", 2024-04-22T18:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70126794999999997 2024-04-22T18:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926860+0000", 2024-04-22T18:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0919999999999998e-06 2024-04-22T18:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4197 4.13 4:ca46fde9:::smithi08937929-14:head [sparse-read 0~1375265,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:22.190 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.967348264999998, 2024-04-22T18:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.28429536, 2024-04-22T18:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4197 2024-04-22T18:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164975+0000", 2024-04-22T18:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164979+0000", 2024-04-22T18:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8959999999999996e-06 2024-04-22T18:41:22.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164983+0000", 2024-04-22T18:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.053e-06 2024-04-22T18:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164987+0000", 2024-04-22T18:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.264783+0000", 2024-04-22T18:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099796088000000005 2024-04-22T18:41:22.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.264789+0000", 2024-04-22T18:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4989999999999999e-06 2024-04-22T18:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139375+0000", 2024-04-22T18:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.87458606000000005 2024-04-22T18:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139478+0000", 2024-04-22T18:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010219 2024-04-22T18:41:22.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.446961+0000", 2024-04-22T18:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30748385900000003 2024-04-22T18:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.447101+0000", 2024-04-22T18:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001393 2024-04-22T18:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.449271+0000", 2024-04-22T18:41:22.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021707950000000001 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4198 4.13 4:ca46fde9:::smithi08937929-14:head [sparse-read 0~1375265] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.967334897000001, 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2868951370000001, 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4198 2024-04-22T18:41:22.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164991+0000", 2024-04-22T18:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164994+0000", 2024-04-22T18:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2459999999999999e-06 2024-04-22T18:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164997+0000", 2024-04-22T18:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1200000000000002e-06 2024-04-22T18:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.364948+0000", 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19995059200000001 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.364969+0000", 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1495999999999999e-05 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139617+0000", 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77464723300000005 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139643+0000", 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6228e-05 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451393+0000", 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31175035200000001 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451515+0000", 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001214 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451885+0000", 2024-04-22T18:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036992999999999998 2024-04-22T18:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4199 4.13 4:ca46fde9:::smithi08937929-14:head [read 0~2739541,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.967323874000002, 2024-04-22T18:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.291247504, 2024-04-22T18:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.201 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4199 2024-04-22T18:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165002+0000", 2024-04-22T18:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165006+0000", 2024-04-22T18:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165009+0000", 2024-04-22T18:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4599999999999999e-06 2024-04-22T18:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365066+0000", 2024-04-22T18:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20005682999999999 2024-04-22T18:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365078+0000", 2024-04-22T18:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2235000000000001e-05 2024-04-22T18:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139671+0000", 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77459301700000005 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139702+0000", 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0673999999999998e-05 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451930+0000", 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31222817600000002 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.452092+0000", 2024-04-22T18:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001616 2024-04-22T18:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.456248+0000", 2024-04-22T18:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041563700000000004 2024-04-22T18:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4200 4.13 4:c93c842e:::smithi08937929-25:head [tier-promote] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:22.206 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.967312994, 2024-04-22T18:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.177728814, 2024-04-22T18:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4200 2024-04-22T18:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165013+0000", 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.635e-06 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165018+0000", 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2000000000000002e-06 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165024+0000", 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5910000000000003e-06 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365110+0000", 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20008601400000001 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:22.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365114+0000", 2024-04-22T18:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.651e-06 2024-04-22T18:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139723+0000", 2024-04-22T18:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77460836899999996 2024-04-22T18:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139787+0000", 2024-04-22T18:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4707000000000002e-05 2024-04-22T18:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.459491+0000", 2024-04-22T18:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31970372000000002 2024-04-22T18:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.459567+0000", 2024-04-22T18:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6199999999999995e-05 2024-04-22T18:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.562522+0000", 2024-04-22T18:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102954925 2024-04-22T18:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.562585+0000", 2024-04-22T18:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3287000000000006e-05 2024-04-22T18:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:05.342740+0000", 2024-04-22T18:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.78015481499999995 2024-04-22T18:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4205 4.1 4.fdaa93c1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.967296844, 2024-04-22T18:41:22.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76220996100000005, 2024-04-22T18:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4205 2024-04-22T18:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165027+0000", 2024-04-22T18:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165029+0000", 2024-04-22T18:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3549999999999999e-06 2024-04-22T18:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165035+0000", 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8549999999999998e-06 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165040+0000", 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9200000000000003e-06 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165116+0000", 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5834999999999995e-05 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165123+0000", 2024-04-22T18:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2139999999999999e-06 2024-04-22T18:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927230+0000", 2024-04-22T18:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76210651100000004 2024-04-22T18:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927237+0000", 2024-04-22T18:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7409999999999992e-06 2024-04-22T18:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.216 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4268 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e133)", 2024-04-22T18:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.549589431999999, 2024-04-22T18:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70931800300000003, 2024-04-22T18:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4268 2024-04-22T18:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.217 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582733+0000", 2024-04-22T18:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:22.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582741+0000", 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3329999999999999e-06 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582751+0000", 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0628999999999999e-05 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582759+0000", 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4120000000000004e-06 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582795+0000", 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6157e-05 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582862+0000", 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7261999999999994e-05 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.987268+0000", 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40440629700000003 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.987771+0000", 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050267400000000005 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.291974+0000", 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30420264499999999 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.292008+0000", 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4011999999999999e-05 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.292053+0000", 2024-04-22T18:41:22.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5305000000000001e-05 2024-04-22T18:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4272 4.5 4:a6d237d2:::smithi08937929-43:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e133)", 2024-04-22T18:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 13.346261232, 2024-04-22T18:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60731304699999999, 2024-04-22T18:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.222 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4272 2024-04-22T18:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786068+0000", 2024-04-22T18:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9910000000000002e-06 2024-04-22T18:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786079+0000", 2024-04-22T18:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1287000000000001e-05 2024-04-22T18:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786083+0000", 2024-04-22T18:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0949999999999998e-06 2024-04-22T18:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786121+0000", 2024-04-22T18:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8096999999999998e-05 2024-04-22T18:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786219+0000", 2024-04-22T18:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7429000000000005e-05 2024-04-22T18:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.393376+0000", 2024-04-22T18:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60715760600000002 2024-04-22T18:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4437 4.19 4:9ea6f51f:::smithi08937929-34:head [read 304128~651776] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 4.2166544579999998, 2024-04-22T18:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.042388732, 2024-04-22T18:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.226 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4437 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915669+0000", 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915673+0000", 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6770000000000001e-06 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915679+0000", 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.976e-06 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915684+0000", 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3430000000000003e-06 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915725+0000", 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0751999999999999e-05 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915732+0000", 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6279999999999999e-06 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.957228+0000", 2024-04-22T18:41:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0414961220000001 2024-04-22T18:41:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.957325+0000", 2024-04-22T18:41:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7422000000000003e-05 2024-04-22T18:41:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.958059+0000", 2024-04-22T18:41:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073350499999999996 2024-04-22T18:41:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.231 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.231 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:22.231 INFO:tasks.rados.rados.0.smithi089.stdout:1860: finishing set_chunk to oid 3 2024-04-22T18:41:22.231 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 3 set_chunk returned error code -95 offset: 333312 length: 935424 2024-04-22T18:41:22.231 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 0 (ObjNum 259 snap 0 seq_num 259) dirty exists 2024-04-22T18:41:22.232 INFO:tasks.rados.rados.0.smithi089.stdout:1858: expect (ObjNum 372 snap 0 seq_num 372) 2024-04-22T18:41:22.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:22.233 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:22.233+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:22.277 INFO:tasks.rados.rados.0.smithi089.stdout:1856: got (0) Success 2024-04-22T18:41:22.277 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 0 (ObjNum 384 snap 0 seq_num 384) dirty exists 2024-04-22T18:41:22.277 INFO:tasks.rados.rados.0.smithi089.stdout:1856: done (5 left) 2024-04-22T18:41:22.277 INFO:tasks.rados.rados.0.smithi089.stdout:1857: done (4 left) 2024-04-22T18:41:22.277 INFO:tasks.rados.rados.0.smithi089.stdout:1858: done (3 left) 2024-04-22T18:41:22.277 INFO:tasks.rados.rados.0.smithi089.stdout:1859: done (2 left) 2024-04-22T18:41:22.277 INFO:tasks.rados.rados.0.smithi089.stdout:1860: done (1 left) 2024-04-22T18:41:22.277 INFO:tasks.rados.rados.0.smithi089.stdout:1862: read oid 43 snap -1 2024-04-22T18:41:22.277 INFO:tasks.rados.rados.0.smithi089.stdout:1862: expect (ObjNum 370 snap 0 seq_num 370) 2024-04-22T18:41:22.277 INFO:tasks.rados.rados.0.smithi089.stdout:1863: delete oid 13 current snap is 0 2024-04-22T18:41:22.279 INFO:tasks.rados.rados.0.smithi089.stdout:1861: expect (ObjNum 274 snap 0 seq_num 274) 2024-04-22T18:41:22.333 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:22.333+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:22.374 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:22.374 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:22.374 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:22.374 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:22.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:22.434 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:22.434+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:22.484 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:22.484 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:22.484 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:22.484 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:22.484 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:22.492 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:22.497 INFO:tasks.rados.rados.0.smithi089.stdout:1862: expect (ObjNum 370 snap 0 seq_num 370) 2024-04-22T18:41:22.534 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:22.534+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:22.595 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:41:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:22.595 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.501328868, 2024-04-22T18:41:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:41:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:41:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:41:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:41:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:41:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:41:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:41:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:41:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:41:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:41:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:41:22.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:41:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:41:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:41:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:41:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:41:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:41:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.600 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:41:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.60183099299999, 2024-04-22T18:41:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:41:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:22.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:41:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:41:22.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:41:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:41:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:41:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:41:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:41:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:41:22.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:41:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:41:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:41:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:41:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:41:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:41:22.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:41:22.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:41:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:41:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:41:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:41:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:41:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 143.49353120200001, 2024-04-22T18:41:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:41:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:41:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:41:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:41:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:41:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:41:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:41:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:41:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:41:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:41:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:41:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:41:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:41:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:41:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:41:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:41:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:41:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:41:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:41:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:41:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:41:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:41:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:41:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.16198757399999, 2024-04-22T18:41:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:41:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:41:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:41:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:41:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:41:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:41:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:41:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:41:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:41:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:41:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:41:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:41:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:41:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:41:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:41:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:41:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:41:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:41:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:41:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:41:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:41:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:41:22.617 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.618 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.07090176599999, 2024-04-22T18:41:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:41:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:41:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:22.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:41:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:41:22.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:41:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:41:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:41:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:41:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:41:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:41:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:41:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:41:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:41:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:41:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:41:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:41:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:41:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:41:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:41:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:41:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:41:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:41:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:41:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.06510150599999, 2024-04-22T18:41:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:41:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.624 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:41:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:41:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:41:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:41:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:41:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:41:22.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:41:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:41:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:41:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:41:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:41:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:41:22.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:41:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:41:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:41:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:41:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:41:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:41:22.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:41:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:41:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:41:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:41:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.629 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.06036116300001, 2024-04-22T18:41:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:41:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:41:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:22.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:41:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:41:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:41:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:41:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:41:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:41:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:41:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:41:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:41:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:41:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:41:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:41:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:41:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:41:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:41:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:41:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:41:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:41:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:41:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:41:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:41:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.05988826500001, 2024-04-22T18:41:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:41:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.635 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:41:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:22.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:41:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:41:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:41:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:41:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:41:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:41:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:41:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:41:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:41:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:41:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:41:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:41:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:41:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:41:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:41:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.640 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.05986834399999, 2024-04-22T18:41:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:41:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:41:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:22.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:41:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:41:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:41:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:41:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:41:22.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:41:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:41:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:41:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:41:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:41:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:41:22.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:41:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:41:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:41:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:41:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:41:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 124.75054457900001, 2024-04-22T18:41:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:41:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:41:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:41:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:41:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:41:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:41:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:41:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:41:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:41:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:41:22.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:41:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:41:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:41:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:41:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:41:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:41:22.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:41:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:41:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:41:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:41:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:41:22.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 124.750337043, 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:22.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:41:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:41:22.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:41:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:41:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:41:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:41:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:41:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:41:22.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:41:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:41:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:41:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:41:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:41:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:41:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:41:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:41:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:41:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:41:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:41:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:41:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:41:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 105.88468731, 2024-04-22T18:41:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:41:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.656 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:41:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:41:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:41:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:41:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:41:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:41:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:41:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:41:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:41:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:41:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:41:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:41:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:41:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:41:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:41:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:41:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:41:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:41:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:41:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:41:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 105.670848925, 2024-04-22T18:41:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:41:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.661 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:41:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:22.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:41:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:41:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:41:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:41:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:41:22.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:41:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:41:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:41:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:41:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:41:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:41:22.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:41:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:41:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:41:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:41:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:41:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:41:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:41:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:41:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:41:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:22.666 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 105.670815515, 2024-04-22T18:41:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:41:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:41:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:41:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:41:22.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:41:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:41:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:41:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:41:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:41:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:41:22.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:41:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:41:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:41:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:41:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:41:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:41:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:41:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:41:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:41:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:41:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:41:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 105.611932861, 2024-04-22T18:41:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:41:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:41:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:41:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:41:22.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:41:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:41:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:41:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:41:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:41:22.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:41:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:41:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:41:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:41:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:41:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:41:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:41:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:41:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:41:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:41:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:41:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:41:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:41:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 105.611794245, 2024-04-22T18:41:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:41:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:41:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:41:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:41:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:41:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:41:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:41:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:41:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:41:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:41:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:41:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:41:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:41:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:41:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:41:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:41:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:41:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:41:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:41:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:41:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:41:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 105.61177524199999, 2024-04-22T18:41:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:41:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.682 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:22.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:41:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:41:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:41:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:41:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:41:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:41:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:41:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:41:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:41:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:41:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:22.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:41:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:41:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:41:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:41:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:41:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:41:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:41:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:41:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:41:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:41:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:22.687 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.99076270899999, 2024-04-22T18:41:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:41:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:41:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:22.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:41:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:41:22.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:41:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:41:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:41:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:41:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:41:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:41:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:41:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:41:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:41:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:41:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:41:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:41:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:41:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:41:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:41:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:41:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:41:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.990243118, 2024-04-22T18:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:41:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:41:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:41:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:41:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:41:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:41:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:41:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:41:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:41:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:41:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:41:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:41:22.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:41:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:41:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:41:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:41:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:41:22.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:41:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:41:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:41:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:41:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:41:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 101.890090235, 2024-04-22T18:41:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:41:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:41:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:22.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:41:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:22.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:41:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:41:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:41:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:41:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:41:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:41:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:41:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:41:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:41:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:41:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:41:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:41:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:41:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:41:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:41:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:41:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:22.703 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:22.703 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:22.705 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:41:22.706 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:22.706 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:22.705+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:22.707 INFO:tasks.rados.rados.0.smithi089.stdout:1861: done (2 left) 2024-04-22T18:41:22.707 INFO:tasks.rados.rados.0.smithi089.stdout:1862: done (1 left) 2024-04-22T18:41:22.708 INFO:tasks.rados.rados.0.smithi089.stdout:1863: done (0 left) 2024-04-22T18:41:22.708 INFO:tasks.rados.rados.0.smithi089.stdout:1864: tier_promote oid 15 2024-04-22T18:41:22.708 INFO:tasks.rados.rados.0.smithi089.stdout:1865: write (excl) oid 23 current snap is 0 2024-04-22T18:41:22.708 INFO:tasks.rados.rados.0.smithi089.stdout:1865: seq_num 390 ranges {554730=481113,1572095=737454} 2024-04-22T18:41:22.708 INFO:tasks.rados.rados.0.smithi089.stdout:1865: writing smithi08937929-23 from 554730 to 1035843 tid 1 2024-04-22T18:41:22.708 INFO:tasks.rados.rados.0.smithi089.stdout:1865: writing smithi08937929-23 from 1572095 to 2309549 tid 2 2024-04-22T18:41:22.708 INFO:tasks.rados.rados.0.smithi089.stdout:1866: read oid 24 snap -1 2024-04-22T18:41:22.708 INFO:tasks.rados.rados.0.smithi089.stdout:1866: expect (ObjNum 384 snap 0 seq_num 384) 2024-04-22T18:41:22.708 INFO:tasks.rados.rados.0.smithi089.stdout:1867: write oid 19 current snap is 0 2024-04-22T18:41:22.708 INFO:tasks.rados.rados.0.smithi089.stdout:1867: seq_num 391 ranges {571537=556362,1630577=597465} 2024-04-22T18:41:22.708 INFO:tasks.rados.rados.0.smithi089.stdout:1867: writing smithi08937929-19 from 571537 to 1127899 tid 1 2024-04-22T18:41:22.708 INFO:tasks.rados.rados.0.smithi089.stdout:1867: writing smithi08937929-19 from 1630577 to 2228042 tid 2 2024-04-22T18:41:22.708 INFO:tasks.rados.rados.0.smithi089.stdout:1868: write oid 20 current snap is 0 2024-04-22T18:41:22.708 INFO:tasks.rados.rados.0.smithi089.stdout:1868: seq_num 392 ranges {588344=631611,1689059=741123} 2024-04-22T18:41:22.713 INFO:tasks.rados.rados.0.smithi089.stdout:1868: writing smithi08937929-20 from 588344 to 1219955 tid 1 2024-04-22T18:41:22.720 INFO:tasks.rados.rados.0.smithi089.stdout:1868: writing smithi08937929-20 from 1689059 to 2430182 tid 2 2024-04-22T18:41:22.720 INFO:tasks.rados.rados.0.smithi089.stdout:1869: read oid 18 snap -1 2024-04-22T18:41:22.720 INFO:tasks.rados.rados.0.smithi089.stdout:1869: expect (ObjNum 307 snap 0 seq_num 307) 2024-04-22T18:41:22.720 INFO:tasks.rados.rados.0.smithi089.stdout:1870: read oid 50 snap -1 2024-04-22T18:41:22.720 INFO:tasks.rados.rados.0.smithi089.stdout:1870: expect (ObjNum 374 snap 0 seq_num 374) 2024-04-22T18:41:22.720 INFO:tasks.rados.rados.0.smithi089.stdout:1871: read oid 36 snap -1 2024-04-22T18:41:22.721 INFO:tasks.rados.rados.0.smithi089.stdout:1871: expect deleted 2024-04-22T18:41:22.721 INFO:tasks.rados.rados.0.smithi089.stdout:1872: write (excl) oid 21 current snap is 0 2024-04-22T18:41:22.721 INFO:tasks.rados.rados.0.smithi089.stdout:1872: seq_num 393 ranges {605151=706860,2031188=573963} 2024-04-22T18:41:22.727 INFO:tasks.rados.rados.0.smithi089.stdout:1872: writing smithi08937929-21 from 605151 to 1312011 tid 1 2024-04-22T18:41:22.733 INFO:tasks.rados.rados.0.smithi089.stdout:1872: writing smithi08937929-21 from 2031188 to 2605151 tid 2 2024-04-22T18:41:22.734 INFO:tasks.rados.rados.0.smithi089.stdout:1873: delete oid 16 current snap is 0 2024-04-22T18:41:22.748 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:22.749+0000 7fa766d7d640 -1 osd.5 146 handle_osd_map injecting map crc failure 2024-04-22T18:41:22.805 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:22.806+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:22.822 INFO:tasks.rados.rados.0.smithi089.stdout:1868: finishing write tid 1 to smithi08937929-20 2024-04-22T18:41:22.835 INFO:tasks.rados.rados.0.smithi089.stdout:1873: done (9 left) 2024-04-22T18:41:22.836 INFO:tasks.rados.rados.0.smithi089.stdout:1874: write (excl) oid 37 current snap is 0 2024-04-22T18:41:22.836 INFO:tasks.rados.rados.0.smithi089.stdout:1874: seq_num 395 ranges {638765=457358,1748152=604803} 2024-04-22T18:41:22.841 INFO:tasks.rados.rados.0.smithi089.stdout:1874: writing smithi08937929-37 from 638765 to 1096123 tid 1 2024-04-22T18:41:22.848 INFO:tasks.rados.rados.0.smithi089.stdout:1874: writing smithi08937929-37 from 1748152 to 2352955 tid 2 2024-04-22T18:41:22.848 INFO:tasks.rados.rados.0.smithi089.stdout:1875: read oid 22 snap -1 2024-04-22T18:41:22.848 INFO:tasks.rados.rados.0.smithi089.stdout:1875: expect (ObjNum 303 snap 0 seq_num 303) 2024-04-22T18:41:22.848 INFO:tasks.rados.rados.0.smithi089.stdout:1876: read oid 16 snap -1 2024-04-22T18:41:22.848 INFO:tasks.rados.rados.0.smithi089.stdout:1876: expect deleted 2024-04-22T18:41:22.848 INFO:tasks.rados.rados.0.smithi089.stdout:1877: set_chunk oid 48 target oid 2024-04-22T18:41:22.852 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:22.852 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:22.852 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:22.852 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:22.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:22.906 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:22.906+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:23.006 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:23.007+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:23.010 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:23.010 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:23.010 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:23.010 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:23.010 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:23.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:23.107 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:23.108+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:23.123 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:23.123 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:23.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:23.123 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:23.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.123 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:23.123 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:23.123 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.863290476, 2024-04-22T18:41:23.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:41:23.123 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.123 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.123 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:41:23.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:23.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:41:23.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:41:23.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:41:23.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:41:23.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:41:23.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:41:23.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:41:23.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:41:23.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:41:23.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:41:23.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:23.128 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.858508046, 2024-04-22T18:41:23.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:41:23.129 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.129 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.129 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.129 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.129 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.129 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:41:23.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.129 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:23.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:23.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.130 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:41:23.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:23.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.130 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:41:23.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:41:23.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:41:23.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:41:23.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:41:23.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:41:23.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:41:23.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:41:23.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:41:23.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:41:23.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:23.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:41:23.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:41:23.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:41:23.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:41:23.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:41:23.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:41:23.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:41:23.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:41:23.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:41:23.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:41:23.134 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.134 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.134 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.134 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:23.134 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:23.134 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.85841384299999, 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:23.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:41:23.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:23.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:41:23.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:41:23.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:41:23.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:41:23.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:41:23.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:41:23.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.137 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:41:23.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:41:23.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.137 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:41:23.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:41:23.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:23.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:41:23.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:41:23.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:41:23.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:41:23.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:41:23.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:41:23.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:41:23.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:41:23.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:41:23.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:41:23.139 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.139 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.139 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.858392955, 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:41:23.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:41:23.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:41:23.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:41:23.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:41:23.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:41:23.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:41:23.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:41:23.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:23.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:41:23.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:41:23.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:41:23.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:41:23.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.143 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.143 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:41:23.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:41:23.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:41:23.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:41:23.144 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.144 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.144 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.144 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:23.144 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:23.144 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.07538529199999, 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:23.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:41:23.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:23.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:41:23.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:41:23.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:41:23.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:41:23.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:41:23.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:41:23.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:41:23.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:41:23.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:41:23.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:41:23.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:23.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:41:23.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:41:23.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:41:23.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:41:23.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:41:23.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:41:23.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:41:23.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:41:23.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:41:23.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:41:23.150 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.150 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.150 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.150 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:23.150 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:23.150 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.06755388600001, 2024-04-22T18:41:23.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:41:23.150 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.150 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.150 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.150 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.150 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.151 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:41:23.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.151 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:23.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:23.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:41:23.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:23.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:41:23.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:41:23.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:41:23.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:41:23.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:41:23.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:41:23.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:41:23.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:41:23.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:41:23.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:41:23.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:23.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:41:23.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:41:23.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:41:23.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:41:23.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:41:23.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:41:23.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:23.155 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:23.156 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.915681255999999, 2024-04-22T18:41:23.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:41:23.156 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.156 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.156 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.156 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.156 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.156 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:41:23.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.156 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:23.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:23.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:41:23.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:23.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:41:23.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:41:23.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:41:23.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:41:23.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:41:23.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:41:23.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:41:23.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:41:23.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:41:23.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:41:23.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.159 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:23.159 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:41:23.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:41:23.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:41:23.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:41:23.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:41:23.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:41:23.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:41:23.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:41:23.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:41:23.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:41:23.161 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.161 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.161 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.161 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:23.161 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:23.161 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 87.915276348000006, 2024-04-22T18:41:23.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:41:23.162 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.162 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.162 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.162 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.162 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.162 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:41:23.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.162 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:23.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:41:23.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:41:23.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:41:23.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:41:23.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:41:23.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:41:23.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:41:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:41:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:41:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:41:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:41:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:41:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:41:23.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:41:23.166 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.166 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.166 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.166 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3213 4.a 4:5d2aee44:::smithi08937929-13:head [assert-version v66,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:23.166 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:23.166 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.55437646, 2024-04-22T18:41:23.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69928701400000004, 2024-04-22T18:41:23.166 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.167 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.167 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.167 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.167 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.167 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3213 2024-04-22T18:41:23.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.167 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:23.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:23.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568559+0000", 2024-04-22T18:41:23.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:23.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568567+0000", 2024-04-22T18:41:23.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-04-22T18:41:23.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568579+0000", 2024-04-22T18:41:23.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-04-22T18:41:23.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.668711+0000", 2024-04-22T18:41:23.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100131866 2024-04-22T18:41:23.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.768945+0000", 2024-04-22T18:41:23.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100234252 2024-04-22T18:41:23.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769002+0000", 2024-04-22T18:41:23.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7092999999999997e-05 2024-04-22T18:41:23.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:23.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.166404+0000", 2024-04-22T18:41:23.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39740241999999998 2024-04-22T18:41:23.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.191686+0000", 2024-04-22T18:41:23.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025281739000000001 2024-04-22T18:41:23.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267758+0000", 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076071606999999999 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267812+0000", 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4478999999999998e-05 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267848+0000", 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6062999999999997e-05 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.171 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.172 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.172 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3319 4.a 4:5d2aee44:::smithi08937929-13:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:41:23.172 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:23.172 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.662628709000003, 2024-04-22T18:41:23.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476623401, 2024-04-22T18:41:23.172 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.172 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.172 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.172 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.172 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.172 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3319 2024-04-22T18:41:23.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.173 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:23.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:23.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460306+0000", 2024-04-22T18:41:23.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:23.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460318+0000", 2024-04-22T18:41:23.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2896e-05 2024-04-22T18:41:23.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460345+0000", 2024-04-22T18:41:23.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6650000000000001e-05 2024-04-22T18:41:23.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460360+0000", 2024-04-22T18:41:23.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4729000000000001e-05 2024-04-22T18:41:23.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460502+0000", 2024-04-22T18:41:23.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141729 2024-04-22T18:41:23.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460644+0000", 2024-04-22T18:41:23.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142755 2024-04-22T18:41:23.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.936932+0000", 2024-04-22T18:41:23.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476288078 2024-04-22T18:41:23.176 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.176 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.176 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.176 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3412 4.a 4:519c08f7:::smithi08937929-15:head [write 2529878~394573] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:23.176 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:23.176 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.980794148000001, 2024-04-22T18:41:23.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48918094899999998, 2024-04-22T18:41:23.176 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.176 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.176 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3412 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142141+0000", 2024-04-22T18:41:23.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:23.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142496+0000", 2024-04-22T18:41:23.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035444699999999998 2024-04-22T18:41:23.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142510+0000", 2024-04-22T18:41:23.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3841e-05 2024-04-22T18:41:23.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142518+0000", 2024-04-22T18:41:23.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8089999999999993e-06 2024-04-22T18:41:23.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142560+0000", 2024-04-22T18:41:23.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2097e-05 2024-04-22T18:41:23.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142637+0000", 2024-04-22T18:41:23.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6965000000000006e-05 2024-04-22T18:41:23.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142715+0000", 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8694000000000002e-05 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144179+0000", 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146342 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631229+0000", 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48705018500000002 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631281+0000", 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2216999999999998e-05 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631324+0000", 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3344999999999999e-05 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3413 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2924451] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.980413007999999, 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58904086899999997, 2024-04-22T18:41:23.181 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.182 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.182 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.182 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.182 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.182 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3413 2024-04-22T18:41:23.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.182 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:23.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142523+0000", 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142529+0000", 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6049999999999997e-06 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142535+0000", 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4740000000000002e-06 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142541+0000", 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1839999999999998e-06 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143553+0000", 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010126689999999999 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143659+0000", 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105393 2024-04-22T18:41:23.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143825+0000", 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165809 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144329+0000", 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050483799999999997 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731480+0000", 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58715100600000003 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731522+0000", 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2098000000000002e-05 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731566+0000", 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3066999999999999e-05 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3414 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.980384403999999, 2024-04-22T18:41:23.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58926020199999996, 2024-04-22T18:41:23.187 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.187 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.187 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.187 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.187 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.187 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3414 2024-04-22T18:41:23.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.187 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:23.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:23.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142552+0000", 2024-04-22T18:41:23.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:23.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142556+0000", 2024-04-22T18:41:23.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0289999999999999e-06 2024-04-22T18:41:23.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142574+0000", 2024-04-22T18:41:23.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7266999999999999e-05 2024-04-22T18:41:23.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142581+0000", 2024-04-22T18:41:23.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-04-22T18:41:23.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144121+0000", 2024-04-22T18:41:23.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015397480000000001 2024-04-22T18:41:23.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:23.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144170+0000", 2024-04-22T18:41:23.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8803e-05 2024-04-22T18:41:23.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731624+0000", 2024-04-22T18:41:23.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58745365800000005 2024-04-22T18:41:23.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731712+0000", 2024-04-22T18:41:23.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8016999999999994e-05 2024-04-22T18:41:23.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731813+0000", 2024-04-22T18:41:23.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101817 2024-04-22T18:41:23.191 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.191 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.191 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.191 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:77 3.3 3:dcc8aeec:::3590d5741efdda41a26b62a0982727439225b7b4b51097b313f30d8dc10bf6f6:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:23.191 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:23.192 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.875366081999999, 2024-04-22T18:41:23.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011812878, 2024-04-22T18:41:23.192 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.192 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.192 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:23.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:23.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247571+0000", 2024-04-22T18:41:23.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247574+0000", 2024-04-22T18:41:23.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:41:23.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247577+0000", 2024-04-22T18:41:23.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3069999999999998e-06 2024-04-22T18:41:23.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247580+0000", 2024-04-22T18:41:23.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8870000000000001e-06 2024-04-22T18:41:23.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.340928+0000", 2024-04-22T18:41:23.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093348678000000004 2024-04-22T18:41:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341122+0000", 2024-04-22T18:41:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000193227 2024-04-22T18:41:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618335+0000", 2024-04-22T18:41:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.277213775 2024-04-22T18:41:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618422+0000", 2024-04-22T18:41:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6537000000000002e-05 2024-04-22T18:41:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954387+0000", 2024-04-22T18:41:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33596509699999999 2024-04-22T18:41:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954548+0000", 2024-04-22T18:41:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016042199999999999 2024-04-22T18:41:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954858+0000", 2024-04-22T18:41:23.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000310396 2024-04-22T18:41:23.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157019+0000", 2024-04-22T18:41:23.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20216071299999999 2024-04-22T18:41:23.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259213+0000", 2024-04-22T18:41:23.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102194734 2024-04-22T18:41:23.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259262+0000", 2024-04-22T18:41:23.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9052999999999999e-05 2024-04-22T18:41:23.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259384+0000", 2024-04-22T18:41:23.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122052 2024-04-22T18:41:23.198 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.198 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.198 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.199 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:78 3.3 3:df7e90d2:::ab0657e24bd96704775de2a378746bc22309c5a7366a5eaf9b4e1f1f2bef0a4c:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:23.199 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:23.199 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.875350885000003, 2024-04-22T18:41:23.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011974739, 2024-04-22T18:41:23.199 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.199 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.199 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:23.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:23.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247586+0000", 2024-04-22T18:41:23.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247589+0000", 2024-04-22T18:41:23.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.181e-06 2024-04-22T18:41:23.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247592+0000", 2024-04-22T18:41:23.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0230000000000001e-06 2024-04-22T18:41:23.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247595+0000", 2024-04-22T18:41:23.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.428e-06 2024-04-22T18:41:23.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341160+0000", 2024-04-22T18:41:23.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093564571999999999 2024-04-22T18:41:23.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:23.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341264+0000", 2024-04-22T18:41:23.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010391999999999999 2024-04-22T18:41:23.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619269+0000", 2024-04-22T18:41:23.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27800575100000002 2024-04-22T18:41:23.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:23.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619346+0000", 2024-04-22T18:41:23.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6459999999999996e-05 2024-04-22T18:41:23.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.155957+0000", 2024-04-22T18:41:23.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53661109299999998 2024-04-22T18:41:23.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156049+0000", 2024-04-22T18:41:23.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2272e-05 2024-04-22T18:41:23.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:23.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156236+0000", 2024-04-22T18:41:23.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000186808 2024-04-22T18:41:23.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.158088+0000", 2024-04-22T18:41:23.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018522079999999999 2024-04-22T18:41:23.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259421+0000", 2024-04-22T18:41:23.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101332718 2024-04-22T18:41:23.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259460+0000", 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8912000000000003e-05 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259562+0000", 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010175600000000001 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:80 3.3 3:f90f47f2:::e5603f596d699ee3e27e255225eeb90a9f0c1b4a4d099d1c586b19019220aa3f:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.875311807999999, 2024-04-22T18:41:23.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0121096759999999, 2024-04-22T18:41:23.206 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.206 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.206 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:23.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:23.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247625+0000", 2024-04-22T18:41:23.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247628+0000", 2024-04-22T18:41:23.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6129999999999998e-06 2024-04-22T18:41:23.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247631+0000", 2024-04-22T18:41:23.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3869999999999999e-06 2024-04-22T18:41:23.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247634+0000", 2024-04-22T18:41:23.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9919999999999999e-06 2024-04-22T18:41:23.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341550+0000", 2024-04-22T18:41:23.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093915584999999996 2024-04-22T18:41:23.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:23.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341700+0000", 2024-04-22T18:41:23.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015000300000000001 2024-04-22T18:41:23.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620744+0000", 2024-04-22T18:41:23.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27904399200000002 2024-04-22T18:41:23.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:23.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620794+0000", 2024-04-22T18:41:23.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9579000000000001e-05 2024-04-22T18:41:23.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157597+0000", 2024-04-22T18:41:23.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53680343799999997 2024-04-22T18:41:23.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157678+0000", 2024-04-22T18:41:23.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0829999999999997e-05 2024-04-22T18:41:23.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:23.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157832+0000", 2024-04-22T18:41:23.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154184 2024-04-22T18:41:23.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259572+0000", 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10174003600000001 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259601+0000", 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8841999999999999e-05 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259638+0000", 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7308999999999998e-05 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259736+0000", 2024-04-22T18:41:23.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7283999999999999e-05 2024-04-22T18:41:23.212 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.212 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.212 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.212 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:135 3.3 3:f89134e5:::3e4e73f5d4a0122957865f0faeac2483204ee98d7d497ef3b54afe2a02b39a3d:head [read 0~802304 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:23.212 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:23.212 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.507496729000003, 2024-04-22T18:41:23.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.64370653499999997, 2024-04-22T18:41:23.212 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.212 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615438+0000", 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615449+0000", 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1219999999999999e-05 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615460+0000", 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0403e-05 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.715588+0000", 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100128881 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722533+0000", 2024-04-22T18:41:23.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069444010000000002 2024-04-22T18:41:23.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:23.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722658+0000", 2024-04-22T18:41:23.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012560699999999999 2024-04-22T18:41:23.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826437+0000", 2024-04-22T18:41:23.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103778194 2024-04-22T18:41:23.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:23.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826532+0000", 2024-04-22T18:41:23.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5111000000000005e-05 2024-04-22T18:41:23.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258787+0000", 2024-04-22T18:41:23.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43225529099999999 2024-04-22T18:41:23.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258905+0000", 2024-04-22T18:41:23.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118348 2024-04-22T18:41:23.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259148+0000", 2024-04-22T18:41:23.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000242093 2024-04-22T18:41:23.217 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.217 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.217 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.217 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3890 4.1 4:83c955bf:::smithi08937929-28:head [write 2699879~469889] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:23.217 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:23.217 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.508821885000003, 2024-04-22T18:41:23.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49042297000000001, 2024-04-22T18:41:23.217 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.217 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.218 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.218 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.218 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.218 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3890 2024-04-22T18:41:23.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.218 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:23.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:23.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614115+0000", 2024-04-22T18:41:23.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:23.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614496+0000", 2024-04-22T18:41:23.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038181700000000001 2024-04-22T18:41:23.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614509+0000", 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2381000000000001e-05 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614517+0000", 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0879999999999995e-06 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615610+0000", 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010933519999999999 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615655+0000", 2024-04-22T18:41:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4966000000000003e-05 2024-04-22T18:41:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615769+0000", 2024-04-22T18:41:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113986 2024-04-22T18:41:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704882+0000", 2024-04-22T18:41:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089112825000000007 2024-04-22T18:41:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104488+0000", 2024-04-22T18:41:23.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39960572999999999 2024-04-22T18:41:23.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104517+0000", 2024-04-22T18:41:23.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9006e-05 2024-04-22T18:41:23.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104539+0000", 2024-04-22T18:41:23.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2073e-05 2024-04-22T18:41:23.222 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.222 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.222 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.223 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3891 4.1 4:83c955bf:::smithi08937929-28:head [setxattr _header (56) in=7b,truncate 3630345] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:23.223 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:23.223 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.508413654000002, 2024-04-22T18:41:23.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59018588699999996, 2024-04-22T18:41:23.223 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.223 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.223 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.223 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.223 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.223 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3891 2024-04-22T18:41:23.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.223 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:23.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:23.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614523+0000", 2024-04-22T18:41:23.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:23.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614528+0000", 2024-04-22T18:41:23.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0379999999999999e-06 2024-04-22T18:41:23.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614540+0000", 2024-04-22T18:41:23.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018000000000001e-05 2024-04-22T18:41:23.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614546+0000", 2024-04-22T18:41:23.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4990000000000002e-06 2024-04-22T18:41:23.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704438+0000", 2024-04-22T18:41:23.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089892044000000004 2024-04-22T18:41:23.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704504+0000", 2024-04-22T18:41:23.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6365000000000006e-05 2024-04-22T18:41:23.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:23.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704602+0000", 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8048999999999996e-05 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.806370+0000", 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10176768 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204629+0000", 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39825917999999999 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204673+0000", 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3516999999999997e-05 2024-04-22T18:41:23.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204710+0000", 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7391000000000002e-05 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3892 4.1 4:83c955bf:::smithi08937929-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.508387073000002, 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59031281800000002, 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.228 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3892 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614550+0000", 2024-04-22T18:41:23.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614553+0000", 2024-04-22T18:41:23.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0879999999999999e-06 2024-04-22T18:41:23.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614559+0000", 2024-04-22T18:41:23.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9259999999999997e-06 2024-04-22T18:41:23.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614565+0000", 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7950000000000004e-06 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704818+0000", 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090252624000000004 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704866+0000", 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8667e-05 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204742+0000", 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987551899999999 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204799+0000", 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7299000000000001e-05 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204863+0000", 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4455000000000005e-05 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.232 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.233 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.233 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:23.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:23.235 INFO:tasks.rados.rados.0.smithi089.stdout:1877: set_chunk oid 48 offset: 678912 length: 2576384 target oid f90dcf7aa55b4d0a91218032bc711b0052726133a5bb293a05ee684c38fcb8b4 offset: 0 snap -1 2024-04-22T18:41:23.235 INFO:tasks.rados.rados.0.smithi089.stdout:1878: write (excl) oid 46 current snap is 0 2024-04-22T18:41:23.236 INFO:tasks.rados.rados.0.smithi089.stdout:1878: seq_num 396 ranges {655572=648960,1922987=732585} 2024-04-22T18:41:23.236 INFO:tasks.rados.rados.0.smithi089.stdout:1878: writing smithi08937929-46 from 655572 to 1304532 tid 1 2024-04-22T18:41:23.236 INFO:tasks.rados.rados.0.smithi089.stdout:1878: writing smithi08937929-46 from 1922987 to 2655572 tid 2 2024-04-22T18:41:23.236 INFO:tasks.rados.rados.0.smithi089.stdout:1879: write (excl) oid 11 current snap is 0 2024-04-22T18:41:23.236 INFO:tasks.rados.rados.0.smithi089.stdout:1879: seq_num 397 ranges {672379=724209,1865116=608472} 2024-04-22T18:41:23.236 INFO:tasks.rados.rados.0.smithi089.stdout:1879: writing smithi08937929-11 from 672379 to 1396588 tid 1 2024-04-22T18:41:23.236 INFO:tasks.rados.rados.0.smithi089.stdout:1879: writing smithi08937929-11 from 1865116 to 2473588 tid 2 2024-04-22T18:41:23.236 INFO:tasks.rados.rados.0.smithi089.stdout:1880: write oid 7 current snap is 0 2024-04-22T18:41:23.236 INFO:tasks.rados.rados.0.smithi089.stdout:1874: finishing write tid 1 to smithi08937929-37 2024-04-22T18:41:23.236 INFO:tasks.rados.rados.0.smithi089.stdout:1866: expect (ObjNum 384 snap 0 seq_num 384) 2024-04-22T18:41:23.237 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:23.234+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:23.237 INFO:tasks.rados.rados.0.smithi089.stdout:1869: expect (ObjNum 307 snap 0 seq_num 307) 2024-04-22T18:41:23.335 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:23.335+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:23.338 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:23.338 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:23.338 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:23.338 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:23.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:23.432 INFO:tasks.rados.rados.0.smithi089.stdout:1868: finishing write tid 2 to smithi08937929-20 2024-04-22T18:41:23.432 INFO:tasks.rados.rados.0.smithi089.stdout:1867: finishing write tid 1 to smithi08937929-19 2024-04-22T18:41:23.432 INFO:tasks.rados.rados.0.smithi089.stdout:1874: finishing write tid 2 to smithi08937929-37 2024-04-22T18:41:23.432 INFO:tasks.rados.rados.0.smithi089.stdout:1880: seq_num 398 ranges {689186=799458,1923598=752130} 2024-04-22T18:41:23.436 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:23.437+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:23.440 INFO:tasks.rados.rados.0.smithi089.stdout:1880: writing smithi08937929-7 from 689186 to 1488644 tid 1 2024-04-22T18:41:23.447 INFO:tasks.rados.rados.0.smithi089.stdout:1880: writing smithi08937929-7 from 1923598 to 2675728 tid 2 2024-04-22T18:41:23.447 INFO:tasks.rados.rados.0.smithi089.stdout:1866: done (15 left) 2024-04-22T18:41:23.447 INFO:tasks.rados.rados.0.smithi089.stdout:1869: done (14 left) 2024-04-22T18:41:23.447 INFO:tasks.rados.rados.0.smithi089.stdout:1871: done (13 left) 2024-04-22T18:41:23.447 INFO:tasks.rados.rados.0.smithi089.stdout:1876: done (12 left) 2024-04-22T18:41:23.447 INFO:tasks.rados.rados.0.smithi089.stdout:1881: tier_flush oid 49 2024-04-22T18:41:23.447 INFO:tasks.rados.rados.0.smithi089.stdout:1881: tier_flush oid 49 snap -1 2024-04-22T18:41:23.447 INFO:tasks.rados.rados.0.smithi089.stdout:1867: finishing write tid 2 to smithi08937929-19 2024-04-22T18:41:23.447 INFO:tasks.rados.rados.0.smithi089.stdout:1867: finishing write tid 3 to smithi08937929-19 2024-04-22T18:41:23.447 INFO:tasks.rados.rados.0.smithi089.stdout:1867: finishing write tid 4 to smithi08937929-19 2024-04-22T18:41:23.447 INFO:tasks.rados.rados.0.smithi089.stdout:1867: oid 19 updating version 0 to 144 2024-04-22T18:41:23.448 INFO:tasks.rados.rados.0.smithi089.stdout:1867: oid 19 updating version 144 to 146 2024-04-22T18:41:23.448 INFO:tasks.rados.rados.0.smithi089.stdout:1867: oid 19 version 146 is already newer than 145 2024-04-22T18:41:23.448 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 146 (ObjNum 391 snap 0 seq_num 391) dirty exists 2024-04-22T18:41:23.448 INFO:tasks.rados.rados.0.smithi089.stdout:1867: left oid 19 (ObjNum 391 snap 0 seq_num 391) 2024-04-22T18:41:23.448 INFO:tasks.rados.rados.0.smithi089.stdout:1875: expect (ObjNum 303 snap 0 seq_num 303) 2024-04-22T18:41:23.450 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:23.450 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:23.450 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:23.451 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:23.451 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:23.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:23.544 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:23.545+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:23.605 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:23.606 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:23.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:23.606 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:23.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.606 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:23.606 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:23.606 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.75162567199999, 2024-04-22T18:41:23.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:41:23.606 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.606 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.606 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.606 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:23.607 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:23.607 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:41:23.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.607 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:23.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:23.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:23.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:41:23.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:41:23.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:41:23.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:41:23.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:41:23.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:41:23.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:41:23.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:41:23.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:41:23.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:41:23.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:23.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:41:23.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:41:23.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:41:23.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:41:23.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:41:23.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:41:23.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:41:23.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:41:23.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:41:23.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:41:23.611 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.611 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.611 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.612 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:23.612 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:23.612 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.75157451000001, 2024-04-22T18:41:23.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:41:23.612 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.612 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.612 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.612 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:23.612 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:23.612 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:41:23.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.612 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:23.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:23.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:23.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:41:23.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:41:23.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:41:23.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:41:23.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:41:23.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:41:23.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:41:23.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:41:23.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:41:23.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:41:23.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:23.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:41:23.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:41:23.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:41:23.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:41:23.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:41:23.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:41:23.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:41:23.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:41:23.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:41:23.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:41:23.617 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.617 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.617 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.617 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:23.617 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:23.617 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.751559286, 2024-04-22T18:41:23.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:41:23.617 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.617 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.618 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.618 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:23.618 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:23.618 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:41:23.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.618 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:23.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:23.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:41:23.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:41:23.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:41:23.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:41:23.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:41:23.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:41:23.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:41:23.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:41:23.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:41:23.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:41:23.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:41:23.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:23.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:41:23.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:41:23.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:41:23.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:41:23.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:41:23.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:41:23.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:41:23.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:41:23.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:41:23.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:41:23.622 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.622 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.622 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.623 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:23.623 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:23.623 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.751499709, 2024-04-22T18:41:23.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:41:23.623 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.623 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.623 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.623 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:23.623 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:23.623 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:41:23.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.623 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:23.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:23.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:23.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:41:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:41:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:41:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:41:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:41:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:41:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:41:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:41:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:41:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:41:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:41:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:41:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:41:23.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:41:23.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:41:23.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:41:23.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:41:23.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:41:23.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.751417393, 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.628 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:23.629 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:23.629 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:41:23.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.629 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:23.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:23.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:41:23.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:41:23.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:41:23.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:41:23.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:41:23.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:41:23.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:41:23.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:41:23.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:41:23.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:41:23.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:41:23.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:23.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:41:23.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:41:23.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:41:23.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:41:23.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:41:23.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:41:23.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:41:23.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:41:23.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:41:23.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:41:23.633 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.633 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.633 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.634 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:23.634 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:23.634 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.751407237, 2024-04-22T18:41:23.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:41:23.634 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.634 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:23.634 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.634 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:23.634 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:23.634 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:41:23.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.634 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:23.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:23.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:41:23.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:41:23.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:41:23.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:41:23.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:41:23.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:41:23.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:41:23.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:41:23.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:41:23.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:41:23.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:41:23.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:23.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:41:23.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:41:23.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:23.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:41:23.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:41:23.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:23.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:41:23.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:41:23.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:23.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:41:23.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:41:23.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.639 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:41:23.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:41:23.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.639 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.639 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3163 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:23.640 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:23.640 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.768394798999999, 2024-04-22T18:41:23.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1009801299999999, 2024-04-22T18:41:23.640 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.640 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.640 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.640 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.640 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.640 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3163 2024-04-22T18:41:23.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.640 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:23.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:23.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837455+0000", 2024-04-22T18:41:23.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:23.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837464+0000", 2024-04-22T18:41:23.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5319999999999995e-06 2024-04-22T18:41:23.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837476+0000", 2024-04-22T18:41:23.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1759e-05 2024-04-22T18:41:23.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837488+0000", 2024-04-22T18:41:23.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1666e-05 2024-04-22T18:41:23.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837541+0000", 2024-04-22T18:41:23.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2939999999999998e-05 2024-04-22T18:41:23.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:23.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837547+0000", 2024-04-22T18:41:23.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6409999999999996e-06 2024-04-22T18:41:23.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937423+0000", 2024-04-22T18:41:23.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.099875545 2024-04-22T18:41:23.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937567+0000", 2024-04-22T18:41:23.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014363099999999999 2024-04-22T18:41:23.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938437+0000", 2024-04-22T18:41:23.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087060399999999997 2024-04-22T18:41:23.644 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.644 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.645 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.645 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3164 4.8 4:187adac1:::smithi08937929-36:head [sparse-read 0~2643888] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:23.645 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:23.645 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.76835543, 2024-04-22T18:41:23.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1013670310000001, 2024-04-22T18:41:23.645 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.645 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.645 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.645 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.645 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.646 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3164 2024-04-22T18:41:23.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.646 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:23.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:23.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:23.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837499+0000", 2024-04-22T18:41:23.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.994e-06 2024-04-22T18:41:23.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837503+0000", 2024-04-22T18:41:23.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7249999999999999e-06 2024-04-22T18:41:23.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837506+0000", 2024-04-22T18:41:23.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5980000000000001e-06 2024-04-22T18:41:23.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937641+0000", 2024-04-22T18:41:23.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10013467700000001 2024-04-22T18:41:23.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:23.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937650+0000", 2024-04-22T18:41:23.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3489999999999998e-06 2024-04-22T18:41:23.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938468+0000", 2024-04-22T18:41:23.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0008176710000001 2024-04-22T18:41:23.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938582+0000", 2024-04-22T18:41:23.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114287 2024-04-22T18:41:23.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938864+0000", 2024-04-22T18:41:23.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028120899999999998 2024-04-22T18:41:23.650 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.650 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.650 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.650 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3165 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:23.650 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:23.650 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.768336662000003, 2024-04-22T18:41:23.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2022754499999999, 2024-04-22T18:41:23.650 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.650 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.651 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.651 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.651 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.651 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3165 2024-04-22T18:41:23.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.651 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:23.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:23.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:23.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837517+0000", 2024-04-22T18:41:23.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3369999999999998e-06 2024-04-22T18:41:23.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837523+0000", 2024-04-22T18:41:23.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5450000000000003e-06 2024-04-22T18:41:23.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837526+0000", 2024-04-22T18:41:23.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.354e-06 2024-04-22T18:41:23.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937682+0000", 2024-04-22T18:41:23.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100155917 2024-04-22T18:41:23.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:23.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937688+0000", 2024-04-22T18:41:23.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:41:23.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.038953+0000", 2024-04-22T18:41:23.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012648110000001 2024-04-22T18:41:23.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.654 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.654 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039034+0000", 2024-04-22T18:41:23.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0969000000000004e-05 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039791+0000", 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075672900000000004 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3183 4.8 4:1bc4668a:::smithi08937929-21:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.569933464000002, 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0040478260000001, 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.655 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:23.656 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.656 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.656 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.656 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3183 2024-04-22T18:41:23.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.656 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:23.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035913+0000", 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999943 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035935+0000", 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2232999999999999e-05 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035969+0000", 2024-04-22T18:41:23.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3969e-05 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035987+0000", 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8397e-05 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2916000000000003e-05 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0103e-05 2024-04-22T18:41:23.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039841+0000", 2024-04-22T18:41:23.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0037508959999999 2024-04-22T18:41:23.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039966+0000", 2024-04-22T18:41:23.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012514500000000001 2024-04-22T18:41:23.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.659 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.659 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3184 4.8 4:12d9f826:::smithi08937929-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:23.660 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:23.660 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.569845505000004, 2024-04-22T18:41:23.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.00402868, 2024-04-22T18:41:23.660 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.660 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:23.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.660 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3184 2024-04-22T18:41:23.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.660 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:23.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:23.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036005+0000", 2024-04-22T18:41:23.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:23.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036026+0000", 2024-04-22T18:41:23.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1092999999999999e-05 2024-04-22T18:41:23.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036030+0000", 2024-04-22T18:41:23.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.467e-06 2024-04-22T18:41:23.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036033+0000", 2024-04-22T18:41:23.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9780000000000001e-06 2024-04-22T18:41:23.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036120+0000", 2024-04-22T18:41:23.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7495000000000002e-05 2024-04-22T18:41:23.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:23.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036125+0000", 2024-04-22T18:41:23.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8720000000000001e-06 2024-04-22T18:41:23.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039982+0000", 2024-04-22T18:41:23.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0038571919999999 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040035+0000", 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2992999999999999e-05 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3185 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.569791592999998, 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0041404039999999, 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.664 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:23.665 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.665 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.665 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.665 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3185 2024-04-22T18:41:23.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.665 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:23.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:23.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:23.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036062+0000", 2024-04-22T18:41:23.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9099999999999999e-06 2024-04-22T18:41:23.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036066+0000", 2024-04-22T18:41:23.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1699999999999999e-06 2024-04-22T18:41:23.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036077+0000", 2024-04-22T18:41:23.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0637e-05 2024-04-22T18:41:23.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036147+0000", 2024-04-22T18:41:23.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0025999999999997e-05 2024-04-22T18:41:23.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:23.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036151+0000", 2024-04-22T18:41:23.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:41:23.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040068+0000", 2024-04-22T18:41:23.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.003916604 2024-04-22T18:41:23.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040201+0000", 2024-04-22T18:41:23.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013275400000000001 2024-04-22T18:41:23.669 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.669 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.669 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.669 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3186 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:23.669 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:23.669 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.569772037, 2024-04-22T18:41:23.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2044167510000001, 2024-04-22T18:41:23.669 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.669 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:23.669 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.669 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.670 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.670 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3186 2024-04-22T18:41:23.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.670 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:23.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:23.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036079+0000", 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036081+0000", 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6449999999999999e-06 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036084+0000", 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9399999999999998e-06 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:41:23.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.879e-06 2024-04-22T18:41:23.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036168+0000", 2024-04-22T18:41:23.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7657999999999994e-05 2024-04-22T18:41:23.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:23.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036171+0000", 2024-04-22T18:41:23.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7639999999999999e-06 2024-04-22T18:41:23.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240345+0000", 2024-04-22T18:41:23.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2041734260000001 2024-04-22T18:41:23.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240497+0000", 2024-04-22T18:41:23.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015182800000000001 2024-04-22T18:41:23.673 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.673 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.673 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3724 4.18 4:1bc4668a:::smithi08937929-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 38.545447660999997, 2024-04-22T18:41:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50413114699999995, 2024-04-22T18:41:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3724 2024-04-22T18:41:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060405+0000", 2024-04-22T18:41:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.395e-06 2024-04-22T18:41:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060408+0000", 2024-04-22T18:41:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:41:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060411+0000", 2024-04-22T18:41:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6060000000000001e-06 2024-04-22T18:41:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165799+0000", 2024-04-22T18:41:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105388369 2024-04-22T18:41:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165843+0000", 2024-04-22T18:41:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3891999999999999e-05 2024-04-22T18:41:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564408+0000", 2024-04-22T18:41:23.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39856466000000002 2024-04-22T18:41:23.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564461+0000", 2024-04-22T18:41:23.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3529999999999997e-05 2024-04-22T18:41:23.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564535+0000", 2024-04-22T18:41:23.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4147999999999995e-05 2024-04-22T18:41:23.678 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.679 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.679 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.679 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4413 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:23.679 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:23.679 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.9484676170000004, 2024-04-22T18:41:23.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2025380569999999, 2024-04-22T18:41:23.679 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.679 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.679 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.679 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.679 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.679 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4413 2024-04-22T18:41:23.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.680 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:23.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:23.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657377+0000", 2024-04-22T18:41:23.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999933 2024-04-22T18:41:23.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657393+0000", 2024-04-22T18:41:23.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5296999999999999e-05 2024-04-22T18:41:23.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657411+0000", 2024-04-22T18:41:23.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8105e-05 2024-04-22T18:41:23.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657424+0000", 2024-04-22T18:41:23.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3030000000000001e-05 2024-04-22T18:41:23.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658313+0000", 2024-04-22T18:41:23.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00088943099999999999 2024-04-22T18:41:23.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:23.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658324+0000", 2024-04-22T18:41:23.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0496000000000001e-05 2024-04-22T18:41:23.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859560+0000", 2024-04-22T18:41:23.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2012358350000001 2024-04-22T18:41:23.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859648+0000", 2024-04-22T18:41:23.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8744000000000003e-05 2024-04-22T18:41:23.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859922+0000", 2024-04-22T18:41:23.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027396499999999998 2024-04-22T18:41:23.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.683 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.684 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4414 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:23.684 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:23.684 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.9484132929999998, 2024-04-22T18:41:23.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2030255030000001, 2024-04-22T18:41:23.684 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.684 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.684 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.684 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.684 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.684 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4414 2024-04-22T18:41:23.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.685 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:23.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:23.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657437+0000", 2024-04-22T18:41:23.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:23.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657443+0000", 2024-04-22T18:41:23.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5899999999999998e-06 2024-04-22T18:41:23.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657450+0000", 2024-04-22T18:41:23.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4649999999999997e-06 2024-04-22T18:41:23.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657457+0000", 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6000000000000003e-06 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658364+0000", 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090735999999999996 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658373+0000", 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3250000000000008e-06 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860258+0000", 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018852769999999 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860308+0000", 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9938000000000002e-05 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860464+0000", 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015622800000000001 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.689 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4415 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:23.689 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:23.689 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.9483889689999998, 2024-04-22T18:41:23.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2038250989999999, 2024-04-22T18:41:23.689 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.689 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.689 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.689 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.689 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.689 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4415 2024-04-22T18:41:23.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.689 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657462+0000", 2024-04-22T18:41:23.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:23.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657467+0000", 2024-04-22T18:41:23.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5399999999999997e-06 2024-04-22T18:41:23.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657473+0000", 2024-04-22T18:41:23.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3949999999999998e-06 2024-04-22T18:41:23.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657480+0000", 2024-04-22T18:41:23.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2489999999999998e-06 2024-04-22T18:41:23.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658394+0000", 2024-04-22T18:41:23.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00091403299999999997 2024-04-22T18:41:23.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:23.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658398+0000", 2024-04-22T18:41:23.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7979999999999996e-06 2024-04-22T18:41:23.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860495+0000", 2024-04-22T18:41:23.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.202097081 2024-04-22T18:41:23.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860543+0000", 2024-04-22T18:41:23.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7907000000000001e-05 2024-04-22T18:41:23.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861288+0000", 2024-04-22T18:41:23.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074466500000000004 2024-04-22T18:41:23.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.693 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.693 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4413 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.9456724640000003, 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.201491952, 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4413 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:23.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:23.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660178+0000", 2024-04-22T18:41:23.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:23.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660185+0000", 2024-04-22T18:41:23.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4379999999999998e-06 2024-04-22T18:41:23.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660201+0000", 2024-04-22T18:41:23.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5252999999999999e-05 2024-04-22T18:41:23.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660210+0000", 2024-04-22T18:41:23.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1069999999999994e-06 2024-04-22T18:41:23.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660270+0000", 2024-04-22T18:41:23.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739000000000002e-05 2024-04-22T18:41:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660281+0000", 2024-04-22T18:41:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0664e-05 2024-04-22T18:41:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861304+0000", 2024-04-22T18:41:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2010225059999999 2024-04-22T18:41:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861342+0000", 2024-04-22T18:41:23.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8736000000000003e-05 2024-04-22T18:41:23.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861671+0000", 2024-04-22T18:41:23.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032915899999999998 2024-04-22T18:41:23.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.698 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.698 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.698 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4414 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:23.699 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:23.699 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.945628954, 2024-04-22T18:41:23.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2014303930000001, 2024-04-22T18:41:23.699 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.699 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.699 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.699 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4414 2024-04-22T18:41:23.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.699 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:23.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:23.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660222+0000", 2024-04-22T18:41:23.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:23.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660227+0000", 2024-04-22T18:41:23.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6789999999999996e-06 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660240+0000", 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3485e-05 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660245+0000", 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8269999999999997e-06 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660307+0000", 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2392999999999997e-05 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660316+0000", 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7790000000000001e-06 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861493+0000", 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.201177341 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861529+0000", 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5978e-05 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861653+0000", 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012391799999999999 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4415 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 5.9455837300000001, 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2019634779999999, 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:23.703 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4415 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:23.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660267+0000", 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660273+0000", 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5910000000000003e-06 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660280+0000", 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4000000000000003e-06 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660288+0000", 2024-04-22T18:41:23.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1769999999999998e-06 2024-04-22T18:41:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.760401+0000", 2024-04-22T18:41:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100112864 2024-04-22T18:41:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.760408+0000", 2024-04-22T18:41:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0299999999999996e-06 2024-04-22T18:41:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861682+0000", 2024-04-22T18:41:23.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012742520000001 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861716+0000", 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3674999999999999e-05 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.862232+0000", 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051554300000000003 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:23.707 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:23.710 INFO:tasks.rados.rados.0.smithi089.stdout:1874: finishing write tid 3 to smithi08937929-37 2024-04-22T18:41:23.710 INFO:tasks.rados.rados.0.smithi089.stdout:1874: finishing write tid 4 to smithi08937929-37 2024-04-22T18:41:23.710 INFO:tasks.rados.rados.0.smithi089.stdout:1874: oid 37 updating version 0 to 151 2024-04-22T18:41:23.710 INFO:tasks.rados.rados.0.smithi089.stdout:1874: oid 37 version 151 is already newer than 150 2024-04-22T18:41:23.710 INFO:tasks.rados.rados.0.smithi089.stdout:1874: oid 37 version 151 is already newer than 149 2024-04-22T18:41:23.710 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 151 (ObjNum 395 snap 0 seq_num 395) dirty exists 2024-04-22T18:41:23.710 INFO:tasks.rados.rados.0.smithi089.stdout:1874: left oid 37 (ObjNum 395 snap 0 seq_num 395) 2024-04-22T18:41:23.710 INFO:tasks.rados.rados.0.smithi089.stdout:1864: got (0) Success 2024-04-22T18:41:23.710 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 115 (ObjNum 377 snap 0 seq_num 377) dirty exists 2024-04-22T18:41:23.710 INFO:tasks.rados.rados.0.smithi089.stdout:1868: finishing write tid 3 to smithi08937929-20 2024-04-22T18:41:23.710 INFO:tasks.rados.rados.0.smithi089.stdout:1864: done (12 left) 2024-04-22T18:41:23.710 INFO:tasks.rados.rados.0.smithi089.stdout:1867: done (11 left) 2024-04-22T18:41:23.711 INFO:tasks.rados.rados.0.smithi089.stdout:1874: done (10 left) 2024-04-22T18:41:23.711 INFO:tasks.rados.rados.0.smithi089.stdout:1875: done (9 left) 2024-04-22T18:41:23.711 INFO:tasks.rados.rados.0.smithi089.stdout:1882: read oid 29 snap -1 2024-04-22T18:41:23.711 INFO:tasks.rados.rados.0.smithi089.stdout:1882: expect (ObjNum 295 snap 0 seq_num 295) 2024-04-22T18:41:23.711 INFO:tasks.rados.rados.0.smithi089.stdout:1883: read oid 13 snap -1 2024-04-22T18:41:23.711 INFO:tasks.rados.rados.0.smithi089.stdout:1883: expect deleted 2024-04-22T18:41:23.711 INFO:tasks.rados.rados.0.smithi089.stdout:1884: write oid 33 current snap is 0 2024-04-22T18:41:23.711 INFO:tasks.rados.rados.0.smithi089.stdout:1868: finishing write tid 4 to smithi08937929-20 2024-04-22T18:41:23.711 INFO:tasks.rados.rados.0.smithi089.stdout:1868: oid 20 updating version 0 to 30 2024-04-22T18:41:23.711 INFO:tasks.rados.rados.0.smithi089.stdout:1868: oid 20 version 30 is already newer than 28 2024-04-22T18:41:23.711 INFO:tasks.rados.rados.0.smithi089.stdout:1868: oid 20 version 30 is already newer than 29 2024-04-22T18:41:23.711 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 30 (ObjNum 392 snap 0 seq_num 392) dirty exists 2024-04-22T18:41:23.711 INFO:tasks.rados.rados.0.smithi089.stdout:1868: left oid 20 (ObjNum 392 snap 0 seq_num 392) 2024-04-22T18:41:23.712 INFO:tasks.rados.rados.0.smithi089.stdout:1872: finishing write tid 1 to smithi08937929-21 2024-04-22T18:41:23.712 INFO:tasks.rados.rados.0.smithi089.stdout:1872: finishing write tid 2 to smithi08937929-21 2024-04-22T18:41:23.712 INFO:tasks.rados.rados.0.smithi089.stdout:1865: finishing write tid 1 to smithi08937929-23 2024-04-22T18:41:23.712 INFO:tasks.rados.rados.0.smithi089.stdout:1865: finishing write tid 2 to smithi08937929-23 2024-04-22T18:41:23.712 INFO:tasks.rados.rados.0.smithi089.stdout:1865: finishing write tid 3 to smithi08937929-23 2024-04-22T18:41:23.712 INFO:tasks.rados.rados.0.smithi089.stdout:1865: finishing write tid 4 to smithi08937929-23 2024-04-22T18:41:23.712 INFO:tasks.rados.rados.0.smithi089.stdout:1865: oid 23 updating version 0 to 185 2024-04-22T18:41:23.712 INFO:tasks.rados.rados.0.smithi089.stdout:1865: oid 23 version 185 is already newer than 184 2024-04-22T18:41:23.712 INFO:tasks.rados.rados.0.smithi089.stdout:1865: oid 23 version 185 is already newer than 183 2024-04-22T18:41:23.712 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 185 (ObjNum 390 snap 0 seq_num 390) dirty exists 2024-04-22T18:41:23.712 INFO:tasks.rados.rados.0.smithi089.stdout:1865: left oid 23 (ObjNum 390 snap 0 seq_num 390) 2024-04-22T18:41:23.712 INFO:tasks.rados.rados.0.smithi089.stdout:1870: expect (ObjNum 374 snap 0 seq_num 374) 2024-04-22T18:41:23.713 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:23.713 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:23.709+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:23.809 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:23.809+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:23.817 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:23.817 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:23.817 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:23.817 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:23.826 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:23.884 INFO:tasks.rados.rados.0.smithi089.stdout:1872: finishing write tid 3 to smithi08937929-21 2024-04-22T18:41:23.884 INFO:tasks.rados.rados.0.smithi089.stdout:1872: finishing write tid 4 to smithi08937929-21 2024-04-22T18:41:23.884 INFO:tasks.rados.rados.0.smithi089.stdout:1872: oid 21 updating version 0 to 68 2024-04-22T18:41:23.884 INFO:tasks.rados.rados.0.smithi089.stdout:1872: oid 21 updating version 68 to 69 2024-04-22T18:41:23.884 INFO:tasks.rados.rados.0.smithi089.stdout:1872: oid 21 updating version 69 to 70 2024-04-22T18:41:23.885 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 21 v 70 (ObjNum 393 snap 0 seq_num 393) dirty exists 2024-04-22T18:41:23.885 INFO:tasks.rados.rados.0.smithi089.stdout:1872: left oid 21 (ObjNum 393 snap 0 seq_num 393) 2024-04-22T18:41:23.885 INFO:tasks.rados.rados.0.smithi089.stdout:1878: finishing write tid 1 to smithi08937929-46 2024-04-22T18:41:23.885 INFO:tasks.rados.rados.0.smithi089.stdout:1884: seq_num 399 ranges {705993=474707,1582080=612141} 2024-04-22T18:41:23.890 INFO:tasks.rados.rados.0.smithi089.stdout:1884: writing smithi08937929-33 from 705993 to 1180700 tid 1 2024-04-22T18:41:23.897 INFO:tasks.rados.rados.0.smithi089.stdout:1884: writing smithi08937929-33 from 1582080 to 2194221 tid 2 2024-04-22T18:41:23.897 INFO:tasks.rados.rados.0.smithi089.stdout:1865: done (11 left) 2024-04-22T18:41:23.897 INFO:tasks.rados.rados.0.smithi089.stdout:1868: done (10 left) 2024-04-22T18:41:23.897 INFO:tasks.rados.rados.0.smithi089.stdout:1870: done (9 left) 2024-04-22T18:41:23.897 INFO:tasks.rados.rados.0.smithi089.stdout:1872: done (8 left) 2024-04-22T18:41:23.898 INFO:tasks.rados.rados.0.smithi089.stdout:1885: read oid 35 snap -1 2024-04-22T18:41:23.898 INFO:tasks.rados.rados.0.smithi089.stdout:1885: expect deleted 2024-04-22T18:41:23.898 INFO:tasks.rados.rados.0.smithi089.stdout:1886: delete oid 45 current snap is 0 2024-04-22T18:41:23.898 INFO:tasks.rados.rados.0.smithi089.stdout:1878: finishing write tid 2 to smithi08937929-46 2024-04-22T18:41:23.898 INFO:tasks.rados.rados.0.smithi089.stdout:1878: finishing write tid 3 to smithi08937929-46 2024-04-22T18:41:23.898 INFO:tasks.rados.rados.0.smithi089.stdout:1878: finishing write tid 4 to smithi08937929-46 2024-04-22T18:41:23.898 INFO:tasks.rados.rados.0.smithi089.stdout:1878: oid 46 updating version 0 to 53 2024-04-22T18:41:23.898 INFO:tasks.rados.rados.0.smithi089.stdout:1878: oid 46 version 53 is already newer than 52 2024-04-22T18:41:23.898 INFO:tasks.rados.rados.0.smithi089.stdout:1878: oid 46 version 53 is already newer than 51 2024-04-22T18:41:23.898 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 46 v 53 (ObjNum 396 snap 0 seq_num 396) dirty exists 2024-04-22T18:41:23.898 INFO:tasks.rados.rados.0.smithi089.stdout:1878: left oid 46 (ObjNum 396 snap 0 seq_num 396) 2024-04-22T18:41:23.899 INFO:tasks.rados.rados.0.smithi089.stdout:1880: finishing write tid 1 to smithi08937929-7 2024-04-22T18:41:23.899 INFO:tasks.rados.rados.0.smithi089.stdout:1880: finishing write tid 2 to smithi08937929-7 2024-04-22T18:41:23.899 INFO:tasks.rados.rados.0.smithi089.stdout:1879: finishing write tid 1 to smithi08937929-11 2024-04-22T18:41:23.899 INFO:tasks.rados.rados.0.smithi089.stdout:1880: finishing write tid 3 to smithi08937929-7 2024-04-22T18:41:23.899 INFO:tasks.rados.rados.0.smithi089.stdout:1880: finishing write tid 4 to smithi08937929-7 2024-04-22T18:41:23.899 INFO:tasks.rados.rados.0.smithi089.stdout:1880: oid 7 updating version 0 to 71 2024-04-22T18:41:23.899 INFO:tasks.rados.rados.0.smithi089.stdout:1880: oid 7 version 71 is already newer than 70 2024-04-22T18:41:23.899 INFO:tasks.rados.rados.0.smithi089.stdout:1880: oid 7 updating version 71 to 72 2024-04-22T18:41:23.899 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 72 (ObjNum 398 snap 0 seq_num 398) dirty exists 2024-04-22T18:41:23.899 INFO:tasks.rados.rados.0.smithi089.stdout:1880: left oid 7 (ObjNum 398 snap 0 seq_num 398) 2024-04-22T18:41:23.899 INFO:tasks.rados.rados.0.smithi089.stdout:1879: finishing write tid 2 to smithi08937929-11 2024-04-22T18:41:23.899 INFO:tasks.rados.rados.0.smithi089.stdout:1877: finishing set_chunk to oid 48 2024-04-22T18:41:23.900 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 95 (ObjNum 351 snap 0 seq_num 351) dirty exists 2024-04-22T18:41:23.900 INFO:tasks.rados.rados.0.smithi089.stdout:1879: finishing write tid 3 to smithi08937929-11 2024-04-22T18:41:23.900 INFO:tasks.rados.rados.0.smithi089.stdout:1879: finishing write tid 4 to smithi08937929-11 2024-04-22T18:41:23.900 INFO:tasks.rados.rados.0.smithi089.stdout:1879: oid 11 updating version 0 to 40 2024-04-22T18:41:23.900 INFO:tasks.rados.rados.0.smithi089.stdout:1879: oid 11 version 40 is already newer than 39 2024-04-22T18:41:23.901 INFO:tasks.rados.rados.0.smithi089.stdout:1879: oid 11 updating version 40 to 41 2024-04-22T18:41:23.901 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 11 v 41 (ObjNum 397 snap 0 seq_num 397) dirty exists 2024-04-22T18:41:23.901 INFO:tasks.rados.rados.0.smithi089.stdout:1879: left oid 11 (ObjNum 397 snap 0 seq_num 397) 2024-04-22T18:41:23.901 INFO:tasks.rados.rados.0.smithi089.stdout:1882: expect (ObjNum 295 snap 0 seq_num 295) 2024-04-22T18:41:23.910 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:23.910+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:23.968 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:23.968 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:23.968 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:23.968 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:23.968 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:23.977 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:23.977 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:41:24.011 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:24.012+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:24.059 INFO:tasks.rados.rados.0.smithi089.stdout:1884: finishing write tid 1 to smithi08937929-33 2024-04-22T18:41:24.059 INFO:tasks.rados.rados.0.smithi089.stdout:1877: done (9 left) 2024-04-22T18:41:24.059 INFO:tasks.rados.rados.0.smithi089.stdout:1878: done (8 left) 2024-04-22T18:41:24.059 INFO:tasks.rados.rados.0.smithi089.stdout:1879: done (7 left) 2024-04-22T18:41:24.059 INFO:tasks.rados.rados.0.smithi089.stdout:1880: done (6 left) 2024-04-22T18:41:24.059 INFO:tasks.rados.rados.0.smithi089.stdout:1882: done (5 left) 2024-04-22T18:41:24.059 INFO:tasks.rados.rados.0.smithi089.stdout:1883: done (4 left) 2024-04-22T18:41:24.059 INFO:tasks.rados.rados.0.smithi089.stdout:1885: done (3 left) 2024-04-22T18:41:24.060 INFO:tasks.rados.rados.0.smithi089.stdout:1886: done (2 left) 2024-04-22T18:41:24.060 INFO:tasks.rados.rados.0.smithi089.stdout:1887: read oid 5 snap -1 2024-04-22T18:41:24.060 INFO:tasks.rados.rados.0.smithi089.stdout:1887: expect (ObjNum 375 snap 0 seq_num 375) 2024-04-22T18:41:24.060 INFO:tasks.rados.rados.0.smithi089.stdout:1888: read oid 15 snap -1 2024-04-22T18:41:24.060 INFO:tasks.rados.rados.0.smithi089.stdout:1888: expect (ObjNum 377 snap 0 seq_num 377) 2024-04-22T18:41:24.060 INFO:tasks.rados.rados.0.smithi089.stdout:1889: tier_promote oid 25 2024-04-22T18:41:24.060 INFO:tasks.rados.rados.0.smithi089.stdout:1881: got (0) Success 2024-04-22T18:41:24.060 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 0 (ObjNum 316 snap 0 seq_num 316) dirty exists 2024-04-22T18:41:24.060 INFO:tasks.rados.rados.0.smithi089.stdout:1884: finishing write tid 2 to smithi08937929-33 2024-04-22T18:41:24.060 INFO:tasks.rados.rados.0.smithi089.stdout:1884: finishing write tid 3 to smithi08937929-33 2024-04-22T18:41:24.060 INFO:tasks.rados.rados.0.smithi089.stdout:1884: finishing write tid 4 to smithi08937929-33 2024-04-22T18:41:24.060 INFO:tasks.rados.rados.0.smithi089.stdout:1884: oid 33 updating version 0 to 63 2024-04-22T18:41:24.060 INFO:tasks.rados.rados.0.smithi089.stdout:1884: oid 33 updating version 63 to 64 2024-04-22T18:41:24.060 INFO:tasks.rados.rados.0.smithi089.stdout:1884: oid 33 version 64 is already newer than 62 2024-04-22T18:41:24.060 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 64 (ObjNum 399 snap 0 seq_num 399) dirty exists 2024-04-22T18:41:24.061 INFO:tasks.rados.rados.0.smithi089.stdout:1884: left oid 33 (ObjNum 399 snap 0 seq_num 399) 2024-04-22T18:41:24.061 INFO:tasks.rados.rados.0.smithi089.stdout:1881: done (4 left) 2024-04-22T18:41:24.061 INFO:tasks.rados.rados.0.smithi089.stdout:1884: done (3 left) 2024-04-22T18:41:24.061 INFO:tasks.rados.rados.0.smithi089.stdout:1890: write (excl) oid 49 current snap is 0 2024-04-22T18:41:24.061 INFO:tasks.rados.rados.0.smithi089.stdout:1890: seq_num 401 ranges {739607=625205,1982691=499457} 2024-04-22T18:41:24.067 INFO:tasks.rados.rados.0.smithi089.stdout:1890: writing smithi08937929-49 from 739607 to 1364812 tid 1 2024-04-22T18:41:24.074 INFO:tasks.rados.rados.0.smithi089.stdout:1890: writing smithi08937929-49 from 1982691 to 2482148 tid 2 2024-04-22T18:41:24.074 INFO:tasks.rados.rados.0.smithi089.stdout:1891: delete oid 36 current snap is 0 2024-04-22T18:41:24.111 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:24.112+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:24.129 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:24.129 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:24.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:24.129 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:24.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.130 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:24.130 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:24.130 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 149.67419964800001, 2024-04-22T18:41:24.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:41:24.130 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.130 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:24.130 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:24.130 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:24.130 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:24.130 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:41:24.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.130 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:24.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:24.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:41:24.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:24.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:41:24.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:41:24.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:41:24.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:41:24.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:41:24.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:41:24.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:41:24.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:41:24.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:41:24.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:41:24.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:24.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:41:24.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:41:24.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:24.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:41:24.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:41:24.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:24.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:41:24.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:41:24.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:24.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:41:24.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:41:24.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:41:24.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:41:24.135 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.135 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.135 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.136 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:24.136 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:24.136 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 140.98114240000001, 2024-04-22T18:41:24.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:41:24.136 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.136 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:24.136 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:24.136 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:24.136 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:24.136 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:41:24.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.136 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:24.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:24.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:41:24.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:24.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:41:24.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:41:24.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:41:24.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:41:24.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:41:24.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:41:24.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:41:24.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:41:24.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:41:24.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:41:24.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:24.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:41:24.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:41:24.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:24.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:41:24.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:41:24.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:24.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:41:24.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:41:24.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:24.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:41:24.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:41:24.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:41:24.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:41:24.141 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.141 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.141 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.141 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:24.142 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:24.142 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 140.981123924, 2024-04-22T18:41:24.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:41:24.142 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.142 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:24.142 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:24.142 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:24.142 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:24.142 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:41:24.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.142 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:24.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:24.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:24.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:24.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:41:24.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:41:24.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:41:24.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:41:24.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:41:24.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:41:24.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:41:24.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:41:24.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:24.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:41:24.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:41:24.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:41:24.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:41:24.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:41:24.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:41:24.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:41:24.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:41:24.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.146 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.146 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:41:24.147 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:24.147 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 113.21807561200001, 2024-04-22T18:41:24.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:41:24.147 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.147 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:24.147 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:24.147 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:24.147 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:24.147 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:41:24.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.147 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:41:24.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:41:24.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:41:24.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:41:24.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:41:24.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:41:24.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:41:24.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:41:24.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:41:24.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:41:24.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:24.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:41:24.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:41:24.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:24.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:41:24.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:41:24.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:24.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:41:24.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:41:24.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:24.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:41:24.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:41:24.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:41:24.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:41:24.152 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.152 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.152 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.152 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:24.152 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:24.152 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.68891628599999, 2024-04-22T18:41:24.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:41:24.152 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.152 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:24.153 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:24.153 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:24.153 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:24.153 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:41:24.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.153 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:24.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:24.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:24.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:24.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:41:24.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:41:24.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:41:24.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:41:24.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:41:24.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:41:24.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:41:24.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:41:24.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:24.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:41:24.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:41:24.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:41:24.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:41:24.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:41:24.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:41:24.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:41:24.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:41:24.157 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.157 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.157 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.157 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:24.157 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:41:24.157 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.511208891, 2024-04-22T18:41:24.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:41:24.158 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.158 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:24.158 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:41:24.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:41:24.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:41:24.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:24.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:41:24.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:41:24.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:41:24.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:41:24.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:24.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:41:24.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:41:24.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:24.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:41:24.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:41:24.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:24.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:41:24.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:41:24.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:24.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:41:24.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:41:24.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:41:24.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:41:24.162 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.162 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.162 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.163 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:24.163 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:41:24.163 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.511140619, 2024-04-22T18:41:24.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:41:24.163 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.163 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:24.163 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:41:24.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:41:24.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:41:24.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:24.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:41:24.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:41:24.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:41:24.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:41:24.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:41:24.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:41:24.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:41:24.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:41:24.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:41:24.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:41:24.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:24.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:41:24.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:41:24.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:24.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:41:24.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:41:24.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:24.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:41:24.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:41:24.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:24.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:41:24.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:41:24.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.510972778, 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:24.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:24.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:24.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:24.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:41:24.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:41:24.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:41:24.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:41:24.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:41:24.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:41:24.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:41:24.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:41:24.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:41:24.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:41:24.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:24.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:41:24.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:41:24.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:24.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:41:24.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:41:24.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:24.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:41:24.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:41:24.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:24.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:41:24.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:41:24.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:41:24.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:41:24.173 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.173 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.173 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.173 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:24.173 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:24.174 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 103.545647303, 2024-04-22T18:41:24.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:41:24.174 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.174 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:24.174 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:24.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:24.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:41:24.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:24.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:41:24.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:41:24.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:41:24.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:41:24.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:41:24.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:41:24.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:41:24.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:41:24.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:41:24.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:41:24.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:24.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:41:24.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:41:24.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:24.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:41:24.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:41:24.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:24.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:41:24.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:41:24.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:24.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:41:24.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:41:24.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:41:24.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:41:24.179 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.179 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.179 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.179 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:24.179 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:41:24.179 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 103.445495116, 2024-04-22T18:41:24.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:41:24.179 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.179 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:24.180 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:41:24.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:41:24.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:41:24.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:24.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:41:24.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:41:24.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:41:24.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:41:24.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:24.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:41:24.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:41:24.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:24.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:41:24.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:41:24.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:24.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:41:24.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:41:24.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:24.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:41:24.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:41:24.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:41:24.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:41:24.184 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.184 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.184 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.184 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:41:24.184 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:24.185 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 93.736630595999998, 2024-04-22T18:41:24.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:41:24.185 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.185 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:24.185 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:24.185 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:24.185 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:24.185 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:41:24.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.185 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:24.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:24.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:41:24.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:24.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:41:24.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:41:24.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:41:24.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:41:24.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:41:24.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:41:24.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:41:24.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:41:24.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:41:24.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:41:24.189 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.189 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.189 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.189 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:41:24.189 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:24.189 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.840592224000005, 2024-04-22T18:41:24.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:41:24.190 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.190 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:24.190 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:24.190 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:24.190 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:24.190 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:41:24.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.190 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:24.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:24.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:24.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:24.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:41:24.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:41:24.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:41:24.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:41:24.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:41:24.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:41:24.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:41:24.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:41:24.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:41:24.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:41:24.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:41:24.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:41:24.193 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.193 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.193 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.194 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3330 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-04-22T18:41:24.194 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:41:24.194 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 57.105513938000001, 2024-04-22T18:41:24.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43151673600000001, 2024-04-22T18:41:24.194 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.194 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:24.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:24.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:24.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:24.194 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3330 2024-04-22T18:41:24.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.194 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:41:24.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:41:24.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024041+0000", 2024-04-22T18:41:24.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:24.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024045+0000", 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3950000000000001e-06 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024051+0000", 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0260000000000004e-06 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024056+0000", 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4489999999999998e-06 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028383+0000", 2024-04-22T18:41:24.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004326557 2024-04-22T18:41:24.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028471+0000", 2024-04-22T18:41:24.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8387999999999995e-05 2024-04-22T18:41:24.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.455559+0000", 2024-04-22T18:41:24.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.42708770699999998 2024-04-22T18:41:24.197 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.197 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.198 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.198 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3377 4.6 4:6700eab5:::smithi08937929-32:head [read 1812992~307712] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-04-22T18:41:24.198 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:24.198 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 53.693188581999998, 2024-04-22T18:41:24.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.52001745799999999, 2024-04-22T18:41:24.198 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.198 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:24.198 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:24.198 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:24.198 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:24.198 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3377 2024-04-22T18:41:24.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.199 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:24.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:24.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436364+0000", 2024-04-22T18:41:24.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:24.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436376+0000", 2024-04-22T18:41:24.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2121e-05 2024-04-22T18:41:24.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436394+0000", 2024-04-22T18:41:24.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8094e-05 2024-04-22T18:41:24.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436405+0000", 2024-04-22T18:41:24.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0189e-05 2024-04-22T18:41:24.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436471+0000", 2024-04-22T18:41:24.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6817999999999996e-05 2024-04-22T18:41:24.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:24.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436481+0000", 2024-04-22T18:41:24.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2720000000000003e-06 2024-04-22T18:41:24.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956084+0000", 2024-04-22T18:41:24.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51960320800000004 2024-04-22T18:41:24.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956232+0000", 2024-04-22T18:41:24.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014778600000000001 2024-04-22T18:41:24.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956385+0000", 2024-04-22T18:41:24.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153394 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:131 3.1 3:a0a19997:::ee8ca23a7050e771c77dff761e6ea5a11ed57796f4e8a17cbb5804a7fdc937b9:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.800318986000001, 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51449564999999997, 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:24.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329235+0000", 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329244+0000", 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6819999999999992e-06 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329259+0000", 2024-04-22T18:41:24.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698000000000001e-05 2024-04-22T18:41:24.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329265+0000", 2024-04-22T18:41:24.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1510000000000001e-06 2024-04-22T18:41:24.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329311+0000", 2024-04-22T18:41:24.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6096e-05 2024-04-22T18:41:24.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:24.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.332269+0000", 2024-04-22T18:41:24.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029581910000000002 2024-04-22T18:41:24.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630603+0000", 2024-04-22T18:41:24.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29833418 2024-04-22T18:41:24.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630685+0000", 2024-04-22T18:41:24.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1217999999999994e-05 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630813+0000", 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128706 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.631447+0000", 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063427000000000004 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843586+0000", 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21213873 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:24.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843623+0000", 2024-04-22T18:41:24.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6848999999999999e-05 2024-04-22T18:41:24.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843733+0000", 2024-04-22T18:41:24.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010982200000000001 2024-04-22T18:41:24.208 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.208 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.208 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3441 4.b 4:d3068ee3:::smithi08937929-29:head [write 558065~619869] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.211842924999999, 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50272020399999995, 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3441 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:24.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:24.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917711+0000", 2024-04-22T18:41:24.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:24.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917838+0000", 2024-04-22T18:41:24.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012734899999999999 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917854+0000", 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6130999999999999e-05 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917863+0000", 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5969999999999992e-06 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992845+0000", 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.074982193000000003 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992919+0000", 2024-04-22T18:41:24.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:41:24.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:24.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.993019+0000", 2024-04-22T18:41:24.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9718999999999999e-05 2024-04-22T18:41:24.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:24.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095554+0000", 2024-04-22T18:41:24.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102535193 2024-04-22T18:41:24.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:24.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420183+0000", 2024-04-22T18:41:24.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32462853600000002 2024-04-22T18:41:24.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:24.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420401+0000", 2024-04-22T18:41:24.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021881400000000001 2024-04-22T18:41:24.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420433+0000", 2024-04-22T18:41:24.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079000000000002e-05 2024-04-22T18:41:24.214 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.214 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.214 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.214 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3442 4.b 4:d3068ee3:::smithi08937929-29:head [write 1579835~486765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:24.214 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:24.214 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.207156214000001, 2024-04-22T18:41:24.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49822799099999998, 2024-04-22T18:41:24.214 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.214 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3442 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922399+0000", 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922502+0000", 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102784 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922513+0000", 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1051e-05 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922518+0000", 2024-04-22T18:41:24.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8529999999999997e-06 2024-04-22T18:41:24.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093158+0000", 2024-04-22T18:41:24.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.17063958800000001 2024-04-22T18:41:24.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093230+0000", 2024-04-22T18:41:24.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1803000000000004e-05 2024-04-22T18:41:24.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:24.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093326+0000", 2024-04-22T18:41:24.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6454999999999997e-05 2024-04-22T18:41:24.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:24.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095557+0000", 2024-04-22T18:41:24.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022306399999999999 2024-04-22T18:41:24.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:24.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420484+0000", 2024-04-22T18:41:24.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32492717700000001 2024-04-22T18:41:24.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:24.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420606+0000", 2024-04-22T18:41:24.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012225800000000001 2024-04-22T18:41:24.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420628+0000", 2024-04-22T18:41:24.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1596000000000001e-05 2024-04-22T18:41:24.220 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.220 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.220 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.220 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3443 4.b 4:d3068ee3:::smithi08937929-29:head [write 2468371~489694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:24.220 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:24.221 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.202554167000002, 2024-04-22T18:41:24.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49372487199999998, 2024-04-22T18:41:24.221 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.221 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:24.221 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:24.221 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:24.221 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:24.221 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3443 2024-04-22T18:41:24.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.221 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:24.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:24.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927001+0000", 2024-04-22T18:41:24.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:24.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927117+0000", 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011637 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927128+0000", 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0696000000000001e-05 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927134+0000", 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8640000000000001e-06 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094288+0000", 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16715412900000001 2024-04-22T18:41:24.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094359+0000", 2024-04-22T18:41:24.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1639999999999998e-05 2024-04-22T18:41:24.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:24.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094451+0000", 2024-04-22T18:41:24.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1252e-05 2024-04-22T18:41:24.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:24.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095560+0000", 2024-04-22T18:41:24.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011095530000000001 2024-04-22T18:41:24.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:24.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420661+0000", 2024-04-22T18:41:24.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32510105500000003 2024-04-22T18:41:24.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:24.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420710+0000", 2024-04-22T18:41:24.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9018999999999999e-05 2024-04-22T18:41:24.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420727+0000", 2024-04-22T18:41:24.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6722e-05 2024-04-22T18:41:24.226 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.226 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.226 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.226 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3444 4.b 4:d3068ee3:::smithi08937929-29:head [setxattr _header (56) in=7b,truncate 2958065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:24.226 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:24.226 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.202418450000003, 2024-04-22T18:41:24.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49366935299999998, 2024-04-22T18:41:24.226 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.226 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:24.227 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:24.227 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:24.227 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:24.227 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3444 2024-04-22T18:41:24.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.227 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:24.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:24.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927137+0000", 2024-04-22T18:41:24.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:24.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927141+0000", 2024-04-22T18:41:24.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3270000000000001e-06 2024-04-22T18:41:24.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927144+0000", 2024-04-22T18:41:24.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9730000000000002e-06 2024-04-22T18:41:24.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927147+0000", 2024-04-22T18:41:24.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.959e-06 2024-04-22T18:41:24.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095118+0000", 2024-04-22T18:41:24.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16797115500000001 2024-04-22T18:41:24.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095169+0000", 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1316000000000001e-05 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095260+0000", 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1184e-05 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095999+0000", 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073868200000000005 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:24.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420757+0000", 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32475783499999999 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420781+0000", 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4022000000000001e-05 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420807+0000", 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6366000000000001e-05 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.232 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3445 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:24.232 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:24.232 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.202407356000002, 2024-04-22T18:41:24.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.493780049, 2024-04-22T18:41:24.232 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:24.232 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:24.232 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:24.232 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:24.232 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:24.232 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3445 2024-04-22T18:41:24.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.232 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:24.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:24.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:24.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:24.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:24.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:24.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:24.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:24.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:24.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:24.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927150+0000", 2024-04-22T18:41:24.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.542e-06 2024-04-22T18:41:24.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:24.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927153+0000", 2024-04-22T18:41:24.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0520000000000002e-06 2024-04-22T18:41:24.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:24.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027218+0000", 2024-04-22T18:41:24.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100064604 2024-04-22T18:41:24.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095501+0000", 2024-04-22T18:41:24.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.068283431000000006 2024-04-22T18:41:24.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:24.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095545+0000", 2024-04-22T18:41:24.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4295999999999997e-05 2024-04-22T18:41:24.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:24.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420833+0000", 2024-04-22T18:41:24.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32528718400000001 2024-04-22T18:41:24.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:24.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420877+0000", 2024-04-22T18:41:24.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4403999999999997e-05 2024-04-22T18:41:24.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:24.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:24.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:24.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420929+0000", 2024-04-22T18:41:24.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1848999999999998e-05 2024-04-22T18:41:24.236 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.236 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:24.237 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:24.237 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:24.237 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:24.238 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:24.239 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:24.239 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:24.240 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:24.240 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:24.240 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:24.240 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:24.241 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:24.241 INFO:tasks.rados.rados.0.smithi089.stdout:1890: finishing write tid 1 to smithi08937929-49 2024-04-22T18:41:24.241 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:24.239+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:24.261 INFO:tasks.rados.rados.0.smithi089.stdout:1888: expect (ObjNum 377 snap 0 seq_num 377) 2024-04-22T18:41:24.304 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:24.305 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:24.321 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:24.321 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:24.338 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:24.338 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:24.338 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:24.338+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:24.349 INFO:tasks.rados.rados.0.smithi089.stdout:1890: finishing write tid 2 to smithi08937929-49 2024-04-22T18:41:24.349 INFO:tasks.rados.rados.0.smithi089.stdout:1890: finishing write tid 3 to smithi08937929-49 2024-04-22T18:41:24.349 INFO:tasks.rados.rados.0.smithi089.stdout:1890: finishing write tid 4 to smithi08937929-49 2024-04-22T18:41:24.349 INFO:tasks.rados.rados.0.smithi089.stdout:1890: oid 49 updating version 0 to 141 2024-04-22T18:41:24.350 INFO:tasks.rados.rados.0.smithi089.stdout:1890: oid 49 updating version 141 to 142 2024-04-22T18:41:24.350 INFO:tasks.rados.rados.0.smithi089.stdout:1890: oid 49 version 142 is already newer than 140 2024-04-22T18:41:24.350 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 142 (ObjNum 401 snap 0 seq_num 401) dirty exists 2024-04-22T18:41:24.350 INFO:tasks.rados.rados.0.smithi089.stdout:1890: left oid 49 (ObjNum 401 snap 0 seq_num 401) 2024-04-22T18:41:24.350 INFO:tasks.rados.rados.0.smithi089.stdout:1888: done (4 left) 2024-04-22T18:41:24.350 INFO:tasks.rados.rados.0.smithi089.stdout:1890: done (3 left) 2024-04-22T18:41:24.350 INFO:tasks.rados.rados.0.smithi089.stdout:1891: done (2 left) 2024-04-22T18:41:24.350 INFO:tasks.rados.rados.0.smithi089.stdout:1892: write (excl) oid 9 current snap is 0 2024-04-22T18:41:24.350 INFO:tasks.rados.rados.0.smithi089.stdout:1892: seq_num 403 ranges {773221=492056,1816008=503126} 2024-04-22T18:41:24.356 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:24.356 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:24.356 INFO:tasks.rados.rados.0.smithi089.stdout:1892: writing smithi08937929-9 from 773221 to 1265277 tid 1 2024-04-22T18:41:24.362 INFO:tasks.rados.rados.0.smithi089.stdout:1892: writing smithi08937929-9 from 1816008 to 2319134 tid 2 2024-04-22T18:41:24.362 INFO:tasks.rados.rados.0.smithi089.stdout:1893: read oid 1 snap -1 2024-04-22T18:41:24.362 INFO:tasks.rados.rados.0.smithi089.stdout:1893: expect (ObjNum 347 snap 0 seq_num 347) 2024-04-22T18:41:24.363 INFO:tasks.rados.rados.0.smithi089.stdout:1894: read oid 36 snap -1 2024-04-22T18:41:24.363 INFO:tasks.rados.rados.0.smithi089.stdout:1894: expect deleted 2024-04-22T18:41:24.363 INFO:tasks.rados.rados.0.smithi089.stdout:1895: set_chunk oid 44 target oid 2024-04-22T18:41:24.366 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:24.366 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:24.376 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:24.460 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:24.460+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:24.519 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:24.519 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:24.528 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:24.540 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:24.542+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:24.641 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:24.641+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:24.670 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:24.670 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:24.679 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:24.707 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:41:24.742 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:24.743+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:24.781 INFO:tasks.rados.rados.0.smithi089.stdout:1895: set_chunk oid 44 offset: 22528 length: 2908672 target oid dd283fd23988276ecb7174f2321a2765d30ee1d04f2f24c488ad6d59a5a33b4a offset: 0 snap -1 2024-04-22T18:41:24.781 INFO:tasks.rados.rados.0.smithi089.stdout:1896: read oid 43 snap -1 2024-04-22T18:41:24.781 INFO:tasks.rados.rados.0.smithi089.stdout:1896: expect (ObjNum 370 snap 0 seq_num 370) 2024-04-22T18:41:24.781 INFO:tasks.rados.rados.0.smithi089.stdout:1897: set_chunk oid 33 target oid 2024-04-22T18:41:24.827 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:24.827 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:24.837 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:24.843 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:24.845+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:24.944 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:24.946+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:24.980 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:24.980 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:24.986 INFO:tasks.rados.rados.0.smithi089.stdout:1897: set_chunk oid 33 offset: 512 length: 502272 target oid df3ba0d7d99746cc2af03952350e7386d61e324816f77465a32aa4681f212f24 offset: 0 snap -1 2024-04-22T18:41:24.986 INFO:tasks.rados.rados.0.smithi089.stdout:1898: read oid 30 snap -1 2024-04-22T18:41:24.986 INFO:tasks.rados.rados.0.smithi089.stdout:1898: expect (ObjNum 378 snap 0 seq_num 378) 2024-04-22T18:41:24.986 INFO:tasks.rados.rados.0.smithi089.stdout:1899: delete oid 45 current snap is 0 2024-04-22T18:41:24.986 INFO:tasks.rados.rados.0.smithi089.stdout:1892: finishing write tid 1 to smithi08937929-9 2024-04-22T18:41:24.986 INFO:tasks.rados.rados.0.smithi089.stdout:1892: finishing write tid 2 to smithi08937929-9 2024-04-22T18:41:24.986 INFO:tasks.rados.rados.0.smithi089.stdout:1893: expect (ObjNum 347 snap 0 seq_num 347) 2024-04-22T18:41:24.990 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:25.045 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:25.045+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:25.095 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:25.095 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:25.103 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:25.108 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 7, 2, 4] out_osds: [5, 0, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:41:25.108 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:41:25.108 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-04-22T18:41:25.108 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-04-22T18:41:25.113 INFO:tasks.rados.rados.0.smithi089.stdout:1892: finishing write tid 3 to smithi08937929-9 2024-04-22T18:41:25.113 INFO:tasks.rados.rados.0.smithi089.stdout:1892: finishing write tid 4 to smithi08937929-9 2024-04-22T18:41:25.113 INFO:tasks.rados.rados.0.smithi089.stdout:1892: oid 9 updating version 0 to 77 2024-04-22T18:41:25.113 INFO:tasks.rados.rados.0.smithi089.stdout:1892: oid 9 updating version 77 to 78 2024-04-22T18:41:25.114 INFO:tasks.rados.rados.0.smithi089.stdout:1892: oid 9 version 78 is already newer than 76 2024-04-22T18:41:25.114 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 78 (ObjNum 403 snap 0 seq_num 403) dirty exists 2024-04-22T18:41:25.114 INFO:tasks.rados.rados.0.smithi089.stdout:1892: left oid 9 (ObjNum 403 snap 0 seq_num 403) 2024-04-22T18:41:25.114 INFO:tasks.rados.rados.0.smithi089.stdout:1889: got (0) Success 2024-04-22T18:41:25.114 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 80 (ObjNum 326 snap 0 seq_num 326) dirty exists 2024-04-22T18:41:25.114 INFO:tasks.rados.rados.0.smithi089.stdout:1895: finishing set_chunk to oid 44 2024-04-22T18:41:25.114 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 44 set_chunk returned error code -95 offset: 22528 length: 2908672 2024-04-22T18:41:25.114 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 44 v 0 (ObjNum 187 snap 0 seq_num 187) dirty exists 2024-04-22T18:41:25.114 INFO:tasks.rados.rados.0.smithi089.stdout:1896: expect (ObjNum 370 snap 0 seq_num 370) 2024-04-22T18:41:25.146 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:25.148+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:25.219 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:25.219 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:25.228 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:25.272 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:25.272+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:25.309 INFO:tasks.rados.rados.0.smithi089.stdout:1897: finishing set_chunk to oid 33 2024-04-22T18:41:25.309 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 33 set_chunk returned error code -95 offset: 512 length: 502272 2024-04-22T18:41:25.309 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 0 (ObjNum 399 snap 0 seq_num 399) dirty exists 2024-04-22T18:41:25.309 INFO:tasks.rados.rados.0.smithi089.stdout:1898: expect (ObjNum 378 snap 0 seq_num 378) 2024-04-22T18:41:25.330 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:25.330 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:25.339 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:25.348 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:25.349+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:25.419 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-22T18:41:25.419+0000 7f9b4bbc8640 -1 mon.a@0(leader).osd e149 definitely_dead 0 2024-04-22T18:41:25.448 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:25.450+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:25.481 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:25.482 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:25.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:25.512 INFO:tasks.rados.rados.0.smithi089.stdout:1889: done (9 left) 2024-04-22T18:41:25.513 INFO:tasks.rados.rados.0.smithi089.stdout:1892: done (8 left) 2024-04-22T18:41:25.513 INFO:tasks.rados.rados.0.smithi089.stdout:1893: done (7 left) 2024-04-22T18:41:25.513 INFO:tasks.rados.rados.0.smithi089.stdout:1894: done (6 left) 2024-04-22T18:41:25.513 INFO:tasks.rados.rados.0.smithi089.stdout:1895: done (5 left) 2024-04-22T18:41:25.513 INFO:tasks.rados.rados.0.smithi089.stdout:1896: done (4 left) 2024-04-22T18:41:25.513 INFO:tasks.rados.rados.0.smithi089.stdout:1897: done (3 left) 2024-04-22T18:41:25.513 INFO:tasks.rados.rados.0.smithi089.stdout:1898: done (2 left) 2024-04-22T18:41:25.513 INFO:tasks.rados.rados.0.smithi089.stdout:1899: done (1 left) 2024-04-22T18:41:25.513 INFO:tasks.rados.rados.0.smithi089.stdout:1900: write oid 50 current snap is 0 2024-04-22T18:41:25.513 INFO:tasks.rados.rados.0.smithi089.stdout:1900: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2024-04-22T18:41:25.518 INFO:tasks.rados.rados.0.smithi089.stdout:1900: writing smithi08937929-50 from 406835 to 1049389 tid 1 2024-04-22T18:41:25.523 INFO:tasks.rados.rados.0.smithi089.stdout:1900: writing smithi08937929-50 from 1532972 to 2039767 tid 2 2024-04-22T18:41:25.528 INFO:tasks.rados.rados.0.smithi089.stdout:1900: writing smithi08937929-50 from 2772312 to 2806835 tid 3 2024-04-22T18:41:25.528 INFO:tasks.rados.rados.0.smithi089.stdout:1901: read oid 6 snap -1 2024-04-22T18:41:25.528 INFO:tasks.rados.rados.0.smithi089.stdout:1901: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:41:25.528 INFO:tasks.rados.rados.0.smithi089.stdout:1902: read oid 27 snap -1 2024-04-22T18:41:25.528 INFO:tasks.rados.rados.0.smithi089.stdout:1902: expect (ObjNum 287 snap 0 seq_num 287) 2024-04-22T18:41:25.528 INFO:tasks.rados.rados.0.smithi089.stdout:1903: read oid 26 snap -1 2024-04-22T18:41:25.528 INFO:tasks.rados.rados.0.smithi089.stdout:1903: expect (ObjNum 353 snap 0 seq_num 353) 2024-04-22T18:41:25.528 INFO:tasks.rados.rados.0.smithi089.stdout:1904: read oid 39 snap -1 2024-04-22T18:41:25.528 INFO:tasks.rados.rados.0.smithi089.stdout:1904: expect (ObjNum 383 snap 0 seq_num 383) 2024-04-22T18:41:25.528 INFO:tasks.rados.rados.0.smithi089.stdout:1905: write (excl) oid 19 current snap is 0 2024-04-22T18:41:25.529 INFO:tasks.rados.rados.0.smithi089.stdout:1905: seq_num 406 ranges {423642=717803,1875101=766806} 2024-04-22T18:41:25.533 INFO:tasks.rados.rados.0.smithi089.stdout:1905: writing smithi08937929-19 from 423642 to 1141445 tid 1 2024-04-22T18:41:25.542 INFO:tasks.rados.rados.0.smithi089.stdout:1905: writing smithi08937929-19 from 1875101 to 2641907 tid 2 2024-04-22T18:41:25.542 INFO:tasks.rados.rados.0.smithi089.stdout:1906: set_chunk oid 12 target oid 2024-04-22T18:41:25.543 INFO:tasks.rados.rados.0.smithi089.stdout:1906: set_chunk oid 12 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:41:25.543 INFO:tasks.rados.rados.0.smithi089.stdout:1907: read oid 38 snap -1 2024-04-22T18:41:25.543 INFO:tasks.rados.rados.0.smithi089.stdout:1907: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:41:25.543 INFO:tasks.rados.rados.0.smithi089.stdout:1908: write oid 29 current snap is 0 2024-04-22T18:41:25.543 INFO:tasks.rados.rados.0.smithi089.stdout:1908: seq_num 407 ranges {440449=793052,1933583=510464} 2024-04-22T18:41:25.549 INFO:tasks.rados.rados.0.smithi089.stdout:1908: writing smithi08937929-29 from 440449 to 1233501 tid 1 2024-04-22T18:41:25.549 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:25.550+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:25.555 INFO:tasks.rados.rados.0.smithi089.stdout:1908: writing smithi08937929-29 from 1933583 to 2444047 tid 2 2024-04-22T18:41:25.556 INFO:tasks.rados.rados.0.smithi089.stdout:1909: set_chunk oid 48 target oid 2024-04-22T18:41:25.607 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:25.607 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:25.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:25.650 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:25.650+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:25.662 INFO:tasks.rados.rados.0.smithi089.stdout:1909: set_chunk oid 48 offset: 1822720 length: 673792 target oid 84fb5139a05cffe9595db254fc8833eb054e94508b5094a75deb91e35395889d offset: 0 snap -1 2024-04-22T18:41:25.662 INFO:tasks.rados.rados.0.smithi089.stdout:1910: read oid 22 snap -1 2024-04-22T18:41:25.662 INFO:tasks.rados.rados.0.smithi089.stdout:1910: expect (ObjNum 303 snap 0 seq_num 303) 2024-04-22T18:41:25.662 INFO:tasks.rados.rados.0.smithi089.stdout:1911: write oid 3 current snap is 0 2024-04-22T18:41:25.662 INFO:tasks.rados.rados.0.smithi089.stdout:1911: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2024-04-22T18:41:25.667 INFO:tasks.rados.rados.0.smithi089.stdout:1911: writing smithi08937929-3 from 457256 to 925557 tid 1 2024-04-22T18:41:25.674 INFO:tasks.rados.rados.0.smithi089.stdout:1911: writing smithi08937929-3 from 1592065 to 2362540 tid 2 2024-04-22T18:41:25.677 INFO:tasks.rados.rados.0.smithi089.stdout:1911: writing smithi08937929-3 from 2854581 to 2857256 tid 3 2024-04-22T18:41:25.677 INFO:tasks.rados.rados.0.smithi089.stdout:1912: tier_promote oid 7 2024-04-22T18:41:25.677 INFO:tasks.rados.rados.0.smithi089.stdout:1901: expect (ObjNum 199 snap 0 seq_num 199) 2024-04-22T18:41:25.751 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:25.751+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:25.759 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:25.759 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:25.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:25.819 INFO:tasks.rados.rados.0.smithi089.stdout:1903: expect (ObjNum 353 snap 0 seq_num 353) 2024-04-22T18:41:25.852 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:25.852+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:25.870 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:25.870 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:25.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:25.952 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:25.952+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:25.983 INFO:tasks.rados.rados.0.smithi089.stdout:1906: finishing set_chunk to oid 12 2024-04-22T18:41:25.983 INFO:tasks.rados.rados.0.smithi089.stdout:1906: got expected ENOENT (src dne) 2024-04-22T18:41:25.984 INFO:tasks.rados.rados.0.smithi089.stdout:1901: done (13 left) 2024-04-22T18:41:25.984 INFO:tasks.rados.rados.0.smithi089.stdout:1903: done (12 left) 2024-04-22T18:41:25.984 INFO:tasks.rados.rados.0.smithi089.stdout:1906: done (11 left) 2024-04-22T18:41:25.984 INFO:tasks.rados.rados.0.smithi089.stdout:1913: read oid 43 snap -1 2024-04-22T18:41:25.984 INFO:tasks.rados.rados.0.smithi089.stdout:1913: expect (ObjNum 370 snap 0 seq_num 370) 2024-04-22T18:41:25.984 INFO:tasks.rados.rados.0.smithi089.stdout:1914: read oid 33 snap -1 2024-04-22T18:41:25.984 INFO:tasks.rados.rados.0.smithi089.stdout:1914: expect (ObjNum 399 snap 0 seq_num 399) 2024-04-22T18:41:25.984 INFO:tasks.rados.rados.0.smithi089.stdout:1915: read oid 20 snap -1 2024-04-22T18:41:25.984 INFO:tasks.rados.rados.0.smithi089.stdout:1915: expect (ObjNum 392 snap 0 seq_num 392) 2024-04-22T18:41:25.984 INFO:tasks.rados.rados.0.smithi089.stdout:1916: read oid 1 snap -1 2024-04-22T18:41:25.984 INFO:tasks.rados.rados.0.smithi089.stdout:1916: expect (ObjNum 347 snap 0 seq_num 347) 2024-04-22T18:41:25.984 INFO:tasks.rados.rados.0.smithi089.stdout:1917: delete oid 11 current snap is 0 2024-04-22T18:41:25.984 INFO:tasks.rados.rados.0.smithi089.stdout:1908: finishing write tid 1 to smithi08937929-29 2024-04-22T18:41:25.985 INFO:tasks.rados.rados.0.smithi089.stdout:1902: expect (ObjNum 287 snap 0 seq_num 287) 2024-04-22T18:41:26.006 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:41:26.008 INFO:teuthology.orchestra.run.smithi089.stderr:marked in osd.0. 2024-04-22T18:41:26.011 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:26.011+0000 7f1e2dd22640 -1 osd.2 149 handle_osd_map injecting map crc failure 2024-04-22T18:41:26.026 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-04-22T18:41:26.026 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:26.026 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:26.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:26.078 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:26.080+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:26.138 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:26.138 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:26.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:26.153 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:26.154+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:26.155 INFO:tasks.rados.rados.0.smithi089.stdout:1904: expect (ObjNum 383 snap 0 seq_num 383) 2024-04-22T18:41:26.254 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:26.255+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:26.288 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:26.288 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:26.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:26.355 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:26.355+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:26.356 INFO:tasks.rados.rados.0.smithi089.stdout:1907: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:41:26.368 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:41:26.438 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:26.438 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:26.446 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:26.456 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:26.456+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:26.485 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:26.486 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:26.487 INFO:tasks.rados.rados.0.smithi089.stdout:1905: finishing write tid 1 to smithi08937929-19 2024-04-22T18:41:26.487 INFO:tasks.rados.rados.0.smithi089.stdout:1905: finishing write tid 2 to smithi08937929-19 2024-04-22T18:41:26.487 INFO:tasks.rados.rados.0.smithi089.stdout:1902: done (15 left) 2024-04-22T18:41:26.487 INFO:tasks.rados.rados.0.smithi089.stdout:1904: done (14 left) 2024-04-22T18:41:26.487 INFO:tasks.rados.rados.0.smithi089.stdout:1907: done (13 left) 2024-04-22T18:41:26.487 INFO:tasks.rados.rados.0.smithi089.stdout:1917: done (12 left) 2024-04-22T18:41:26.487 INFO:tasks.rados.rados.0.smithi089.stdout:1918: read oid 11 snap -1 2024-04-22T18:41:26.488 INFO:tasks.rados.rados.0.smithi089.stdout:1918: expect deleted 2024-04-22T18:41:26.488 INFO:tasks.rados.rados.0.smithi089.stdout:1919: set_chunk oid 2 target oid 2024-04-22T18:41:26.502 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:26.502 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:26.518 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:26.518 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:26.533 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:26.534 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:26.547 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:26.547 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:26.550 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:26.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:26.557 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:26.557+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:26.565 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:26.566 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:26.582 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:26.582 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:26.599 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:26.600 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:26.657 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:26.658+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:26.704 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:26.704 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 152.44449707199999, 2024-04-22T18:41:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:41:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:41:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:41:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:41:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:41:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:41:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:41:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:41:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:41:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:41:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:41:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:41:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:41:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:41:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:41:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:26.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:41:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:41:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:41:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:41:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:41:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:41:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:41:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:41:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 152.43971464200001, 2024-04-22T18:41:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:41:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.710 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:41:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:41:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:41:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:41:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:41:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:41:26.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:41:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:41:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:41:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:41:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:41:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:41:26.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:41:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:41:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:41:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:41:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:41:26.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:41:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:41:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:41:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:41:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:41:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 152.43962043900001, 2024-04-22T18:41:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:41:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:41:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:41:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:41:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:41:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:41:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:41:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:41:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:41:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:41:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:41:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:41:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:41:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:41:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:41:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:41:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:41:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:41:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:41:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:41:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:41:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 152.43959955099999, 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.721 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:41:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:41:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:41:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:41:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:41:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:41:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:41:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.725 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 151.65659188800001, 2024-04-22T18:41:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:41:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:41:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.726 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:41:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:26.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:41:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:41:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:41:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:41:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:41:26.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:41:26.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:41:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:41:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:41:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:41:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:41:26.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:41:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:41:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:41:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 151.648760482, 2024-04-22T18:41:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:41:26.731 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:41:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:41:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:41:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:41:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:41:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:41:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:41:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:41:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:41:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:41:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:41:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:41:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:41:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:41:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:41:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:41:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:41:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:41:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:41:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:41:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:41:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:41:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.736 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 91.496887852, 2024-04-22T18:41:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:41:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:41:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:41:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:26.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:41:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:41:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:41:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:41:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:41:26.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:41:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:41:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:41:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:41:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:41:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:26.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:41:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:41:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:41:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:41:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:41:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:41:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:26.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:41:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:41:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:41:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:41:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:26.742 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 91.496482943999993, 2024-04-22T18:41:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:41:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:41:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:41:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:41:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:41:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:41:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:41:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:41:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:41:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:41:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:41:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:41:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:41:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:41:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:41:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:41:26.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:41:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:41:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3213 4.a 4:5d2aee44:::smithi08937929-13:head [assert-version v66,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 64.135583056000002, 2024-04-22T18:41:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69928701400000004, 2024-04-22T18:41:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.747 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3213 2024-04-22T18:41:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568559+0000", 2024-04-22T18:41:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568567+0000", 2024-04-22T18:41:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-04-22T18:41:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568579+0000", 2024-04-22T18:41:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-04-22T18:41:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.668711+0000", 2024-04-22T18:41:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100131866 2024-04-22T18:41:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.768945+0000", 2024-04-22T18:41:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100234252 2024-04-22T18:41:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769002+0000", 2024-04-22T18:41:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7092999999999997e-05 2024-04-22T18:41:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.166404+0000", 2024-04-22T18:41:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39740241999999998 2024-04-22T18:41:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.191686+0000", 2024-04-22T18:41:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025281739000000001 2024-04-22T18:41:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267758+0000", 2024-04-22T18:41:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076071606999999999 2024-04-22T18:41:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267812+0000", 2024-04-22T18:41:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4478999999999998e-05 2024-04-22T18:41:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267848+0000", 2024-04-22T18:41:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6062999999999997e-05 2024-04-22T18:41:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.752 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3319 4.a 4:5d2aee44:::smithi08937929-13:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:41:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 60.243835304999998, 2024-04-22T18:41:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476623401, 2024-04-22T18:41:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3319 2024-04-22T18:41:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.753 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460306+0000", 2024-04-22T18:41:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460318+0000", 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2896e-05 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460345+0000", 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6650000000000001e-05 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460360+0000", 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4729000000000001e-05 2024-04-22T18:41:26.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460502+0000", 2024-04-22T18:41:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141729 2024-04-22T18:41:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460644+0000", 2024-04-22T18:41:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142755 2024-04-22T18:41:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.936932+0000", 2024-04-22T18:41:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476288078 2024-04-22T18:41:26.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3412 4.a 4:519c08f7:::smithi08937929-15:head [write 2529878~394573] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.562000744000002, 2024-04-22T18:41:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48918094899999998, 2024-04-22T18:41:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3412 2024-04-22T18:41:26.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142141+0000", 2024-04-22T18:41:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142496+0000", 2024-04-22T18:41:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035444699999999998 2024-04-22T18:41:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142510+0000", 2024-04-22T18:41:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3841e-05 2024-04-22T18:41:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142518+0000", 2024-04-22T18:41:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8089999999999993e-06 2024-04-22T18:41:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142560+0000", 2024-04-22T18:41:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2097e-05 2024-04-22T18:41:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142637+0000", 2024-04-22T18:41:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6965000000000006e-05 2024-04-22T18:41:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:26.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142715+0000", 2024-04-22T18:41:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8694000000000002e-05 2024-04-22T18:41:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144179+0000", 2024-04-22T18:41:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146342 2024-04-22T18:41:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631229+0000", 2024-04-22T18:41:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48705018500000002 2024-04-22T18:41:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:26.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631281+0000", 2024-04-22T18:41:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2216999999999998e-05 2024-04-22T18:41:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631324+0000", 2024-04-22T18:41:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3344999999999999e-05 2024-04-22T18:41:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3413 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2924451] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.561619604000001, 2024-04-22T18:41:26.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58904086899999997, 2024-04-22T18:41:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3413 2024-04-22T18:41:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142523+0000", 2024-04-22T18:41:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142529+0000", 2024-04-22T18:41:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6049999999999997e-06 2024-04-22T18:41:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142535+0000", 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4740000000000002e-06 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142541+0000", 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1839999999999998e-06 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143553+0000", 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010126689999999999 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143659+0000", 2024-04-22T18:41:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105393 2024-04-22T18:41:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143825+0000", 2024-04-22T18:41:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165809 2024-04-22T18:41:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144329+0000", 2024-04-22T18:41:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050483799999999997 2024-04-22T18:41:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731480+0000", 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58715100600000003 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731522+0000", 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2098000000000002e-05 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731566+0000", 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3066999999999999e-05 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3414 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 55.561591, 2024-04-22T18:41:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58926020199999996, 2024-04-22T18:41:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3414 2024-04-22T18:41:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142552+0000", 2024-04-22T18:41:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142556+0000", 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0289999999999999e-06 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142574+0000", 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7266999999999999e-05 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142581+0000", 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144121+0000", 2024-04-22T18:41:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015397480000000001 2024-04-22T18:41:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144170+0000", 2024-04-22T18:41:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8803e-05 2024-04-22T18:41:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731624+0000", 2024-04-22T18:41:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58745365800000005 2024-04-22T18:41:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731712+0000", 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8016999999999994e-05 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731813+0000", 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101817 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:77 3.3 3:dcc8aeec:::3590d5741efdda41a26b62a0982727439225b7b4b51097b313f30d8dc10bf6f6:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.456572678000001, 2024-04-22T18:41:26.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011812878, 2024-04-22T18:41:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247571+0000", 2024-04-22T18:41:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247574+0000", 2024-04-22T18:41:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:41:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247577+0000", 2024-04-22T18:41:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3069999999999998e-06 2024-04-22T18:41:26.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247580+0000", 2024-04-22T18:41:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8870000000000001e-06 2024-04-22T18:41:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.340928+0000", 2024-04-22T18:41:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093348678000000004 2024-04-22T18:41:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341122+0000", 2024-04-22T18:41:26.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000193227 2024-04-22T18:41:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618335+0000", 2024-04-22T18:41:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.277213775 2024-04-22T18:41:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618422+0000", 2024-04-22T18:41:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6537000000000002e-05 2024-04-22T18:41:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954387+0000", 2024-04-22T18:41:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33596509699999999 2024-04-22T18:41:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954548+0000", 2024-04-22T18:41:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016042199999999999 2024-04-22T18:41:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954858+0000", 2024-04-22T18:41:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000310396 2024-04-22T18:41:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157019+0000", 2024-04-22T18:41:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20216071299999999 2024-04-22T18:41:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259213+0000", 2024-04-22T18:41:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102194734 2024-04-22T18:41:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259262+0000", 2024-04-22T18:41:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9052999999999999e-05 2024-04-22T18:41:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259384+0000", 2024-04-22T18:41:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122052 2024-04-22T18:41:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:78 3.3 3:df7e90d2:::ab0657e24bd96704775de2a378746bc22309c5a7366a5eaf9b4e1f1f2bef0a4c:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.456557480999997, 2024-04-22T18:41:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011974739, 2024-04-22T18:41:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.779 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247586+0000", 2024-04-22T18:41:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247589+0000", 2024-04-22T18:41:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.181e-06 2024-04-22T18:41:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247592+0000", 2024-04-22T18:41:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0230000000000001e-06 2024-04-22T18:41:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247595+0000", 2024-04-22T18:41:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.428e-06 2024-04-22T18:41:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341160+0000", 2024-04-22T18:41:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093564571999999999 2024-04-22T18:41:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341264+0000", 2024-04-22T18:41:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010391999999999999 2024-04-22T18:41:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619269+0000", 2024-04-22T18:41:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27800575100000002 2024-04-22T18:41:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619346+0000", 2024-04-22T18:41:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6459999999999996e-05 2024-04-22T18:41:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.155957+0000", 2024-04-22T18:41:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53661109299999998 2024-04-22T18:41:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156049+0000", 2024-04-22T18:41:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2272e-05 2024-04-22T18:41:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156236+0000", 2024-04-22T18:41:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000186808 2024-04-22T18:41:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.158088+0000", 2024-04-22T18:41:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018522079999999999 2024-04-22T18:41:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259421+0000", 2024-04-22T18:41:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101332718 2024-04-22T18:41:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259460+0000", 2024-04-22T18:41:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8912000000000003e-05 2024-04-22T18:41:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259562+0000", 2024-04-22T18:41:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010175600000000001 2024-04-22T18:41:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.785 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:80 3.3 3:f90f47f2:::e5603f596d699ee3e27e255225eeb90a9f0c1b4a4d099d1c586b19019220aa3f:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.456518404000001, 2024-04-22T18:41:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0121096759999999, 2024-04-22T18:41:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:26.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247625+0000", 2024-04-22T18:41:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247628+0000", 2024-04-22T18:41:26.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6129999999999998e-06 2024-04-22T18:41:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247631+0000", 2024-04-22T18:41:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3869999999999999e-06 2024-04-22T18:41:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247634+0000", 2024-04-22T18:41:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9919999999999999e-06 2024-04-22T18:41:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341550+0000", 2024-04-22T18:41:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093915584999999996 2024-04-22T18:41:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341700+0000", 2024-04-22T18:41:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015000300000000001 2024-04-22T18:41:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620744+0000", 2024-04-22T18:41:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27904399200000002 2024-04-22T18:41:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:26.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620794+0000", 2024-04-22T18:41:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9579000000000001e-05 2024-04-22T18:41:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157597+0000", 2024-04-22T18:41:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53680343799999997 2024-04-22T18:41:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157678+0000", 2024-04-22T18:41:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0829999999999997e-05 2024-04-22T18:41:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:26.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157832+0000", 2024-04-22T18:41:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154184 2024-04-22T18:41:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259572+0000", 2024-04-22T18:41:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10174003600000001 2024-04-22T18:41:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259601+0000", 2024-04-22T18:41:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8841999999999999e-05 2024-04-22T18:41:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:26.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259638+0000", 2024-04-22T18:41:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7308999999999998e-05 2024-04-22T18:41:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259736+0000", 2024-04-22T18:41:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7283999999999999e-05 2024-04-22T18:41:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:135 3.3 3:f89134e5:::3e4e73f5d4a0122957865f0faeac2483204ee98d7d497ef3b54afe2a02b39a3d:head [read 0~802304 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:26.792 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.088703324999997, 2024-04-22T18:41:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.64370653499999997, 2024-04-22T18:41:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615438+0000", 2024-04-22T18:41:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615449+0000", 2024-04-22T18:41:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1219999999999999e-05 2024-04-22T18:41:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615460+0000", 2024-04-22T18:41:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0403e-05 2024-04-22T18:41:26.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.715588+0000", 2024-04-22T18:41:26.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100128881 2024-04-22T18:41:26.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722533+0000", 2024-04-22T18:41:26.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069444010000000002 2024-04-22T18:41:26.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:26.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722658+0000", 2024-04-22T18:41:26.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012560699999999999 2024-04-22T18:41:26.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826437+0000", 2024-04-22T18:41:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103778194 2024-04-22T18:41:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826532+0000", 2024-04-22T18:41:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5111000000000005e-05 2024-04-22T18:41:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258787+0000", 2024-04-22T18:41:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43225529099999999 2024-04-22T18:41:26.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258905+0000", 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118348 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259148+0000", 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000242093 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3890 4.1 4:83c955bf:::smithi08937929-28:head [write 2699879~469889] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:26.797 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:26.798 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.090028480999997, 2024-04-22T18:41:26.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49042297000000001, 2024-04-22T18:41:26.798 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.798 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:26.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.798 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.798 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3890 2024-04-22T18:41:26.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.798 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:26.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:26.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614115+0000", 2024-04-22T18:41:26.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:26.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614496+0000", 2024-04-22T18:41:26.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038181700000000001 2024-04-22T18:41:26.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614509+0000", 2024-04-22T18:41:26.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2381000000000001e-05 2024-04-22T18:41:26.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614517+0000", 2024-04-22T18:41:26.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0879999999999995e-06 2024-04-22T18:41:26.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615610+0000", 2024-04-22T18:41:26.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010933519999999999 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615655+0000", 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4966000000000003e-05 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615769+0000", 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113986 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704882+0000", 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089112825000000007 2024-04-22T18:41:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104488+0000", 2024-04-22T18:41:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39960572999999999 2024-04-22T18:41:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104517+0000", 2024-04-22T18:41:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9006e-05 2024-04-22T18:41:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104539+0000", 2024-04-22T18:41:26.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2073e-05 2024-04-22T18:41:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3891 4.1 4:83c955bf:::smithi08937929-28:head [setxattr _header (56) in=7b,truncate 3630345] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.089620250000003, 2024-04-22T18:41:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59018588699999996, 2024-04-22T18:41:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.803 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3891 2024-04-22T18:41:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614523+0000", 2024-04-22T18:41:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614528+0000", 2024-04-22T18:41:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0379999999999999e-06 2024-04-22T18:41:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614540+0000", 2024-04-22T18:41:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018000000000001e-05 2024-04-22T18:41:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614546+0000", 2024-04-22T18:41:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4990000000000002e-06 2024-04-22T18:41:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704438+0000", 2024-04-22T18:41:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089892044000000004 2024-04-22T18:41:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704504+0000", 2024-04-22T18:41:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6365000000000006e-05 2024-04-22T18:41:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704602+0000", 2024-04-22T18:41:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8048999999999996e-05 2024-04-22T18:41:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.806370+0000", 2024-04-22T18:41:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10176768 2024-04-22T18:41:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204629+0000", 2024-04-22T18:41:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39825917999999999 2024-04-22T18:41:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204673+0000", 2024-04-22T18:41:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3516999999999997e-05 2024-04-22T18:41:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204710+0000", 2024-04-22T18:41:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7391000000000002e-05 2024-04-22T18:41:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3892 4.1 4:83c955bf:::smithi08937929-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:26.808 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 37.089593669000003, 2024-04-22T18:41:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59031281800000002, 2024-04-22T18:41:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3892 2024-04-22T18:41:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:26.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614550+0000", 2024-04-22T18:41:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:26.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614553+0000", 2024-04-22T18:41:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0879999999999999e-06 2024-04-22T18:41:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614559+0000", 2024-04-22T18:41:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9259999999999997e-06 2024-04-22T18:41:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614565+0000", 2024-04-22T18:41:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7950000000000004e-06 2024-04-22T18:41:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704818+0000", 2024-04-22T18:41:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090252624000000004 2024-04-22T18:41:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704866+0000", 2024-04-22T18:41:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8667e-05 2024-04-22T18:41:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204742+0000", 2024-04-22T18:41:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987551899999999 2024-04-22T18:41:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204799+0000", 2024-04-22T18:41:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7299000000000001e-05 2024-04-22T18:41:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204863+0000", 2024-04-22T18:41:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4455000000000005e-05 2024-04-22T18:41:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:26.813 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:26.813 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:26.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:26.815 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:26.816+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:26.916 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:26.917+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:26.959 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:26.959 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:26.959 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:26.968 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:27.052 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:27.054+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:27.111 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:27.111 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:27.111 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:27.117 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:27.119+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:27.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:27.157 INFO:tasks.rados.rados.0.smithi089.stdout:1919: set_chunk oid 2 offset: 413184 length: 441856 target oid cb076f2b64cfcdfafc1ec2cf19f2eaa46006a5a6952a3aeb9c7211f97fca91ae offset: 0 snap -1 2024-04-22T18:41:27.157 INFO:tasks.rados.rados.0.smithi089.stdout:1920: read oid 17 snap -1 2024-04-22T18:41:27.157 INFO:tasks.rados.rados.0.smithi089.stdout:1920: expect (ObjNum 366 snap 0 seq_num 366) 2024-04-22T18:41:27.157 INFO:tasks.rados.rados.0.smithi089.stdout:1921: read oid 9 snap -1 2024-04-22T18:41:27.157 INFO:tasks.rados.rados.0.smithi089.stdout:1921: expect (ObjNum 403 snap 0 seq_num 403) 2024-04-22T18:41:27.157 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:41:27.157 INFO:tasks.rados.rados.0.smithi089.stdout:1905: finishing write tid 3 to smithi08937929-19 2024-04-22T18:41:27.158 INFO:tasks.rados.rados.0.smithi089.stdout:1910: expect (ObjNum 303 snap 0 seq_num 303) 2024-04-22T18:41:27.219 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:27.221+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:27.264 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:27.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 162.41019526299999, 2024-04-22T18:41:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:41:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:41:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:41:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:41:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:41:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:41:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:41:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:41:27.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:41:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:41:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:41:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:41:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:41:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:41:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:41:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:41:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:41:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:41:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:41:27.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:41:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:41:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:41:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 162.41014410099999, 2024-04-22T18:41:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:41:27.270 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:41:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:41:27.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:41:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:41:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:41:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:41:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:41:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:41:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:41:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:41:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:41:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:41:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:41:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:41:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:41:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:41:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:41:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:41:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:41:27.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:41:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:41:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 162.41012887700001, 2024-04-22T18:41:27.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:41:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:41:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:41:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:41:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:41:27.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:41:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:41:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:41:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:41:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:41:27.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:41:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:41:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:41:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:41:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:41:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:41:27.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:41:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:41:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:41:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:41:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:41:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:41:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:41:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 162.4100693, 2024-04-22T18:41:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:41:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.282 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:41:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:27.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:41:27.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:41:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:41:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:41:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:41:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:41:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:41:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:41:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:41:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:41:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:27.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:41:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:41:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:41:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:41:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:41:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:41:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:41:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:41:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:41:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:41:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 162.409986984, 2024-04-22T18:41:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:41:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:41:27.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:41:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:41:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:41:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:41:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:41:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:41:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:41:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:41:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:41:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:41:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:41:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:41:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:41:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:41:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:41:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:41:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:41:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:41:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:41:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:41:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:41:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.295 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 162.409976828, 2024-04-22T18:41:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:41:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:41:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.296 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:41:27.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:41:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:41:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:41:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:41:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:41:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:41:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:41:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:41:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:41:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:41:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:41:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:41:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:41:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:41:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:41:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:41:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:41:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:41:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:41:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:41:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.301 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3163 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.426964389999995, 2024-04-22T18:41:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1009801299999999, 2024-04-22T18:41:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3163 2024-04-22T18:41:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837455+0000", 2024-04-22T18:41:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837464+0000", 2024-04-22T18:41:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5319999999999995e-06 2024-04-22T18:41:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837476+0000", 2024-04-22T18:41:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1759e-05 2024-04-22T18:41:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837488+0000", 2024-04-22T18:41:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1666e-05 2024-04-22T18:41:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837541+0000", 2024-04-22T18:41:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2939999999999998e-05 2024-04-22T18:41:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837547+0000", 2024-04-22T18:41:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6409999999999996e-06 2024-04-22T18:41:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937423+0000", 2024-04-22T18:41:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.099875545 2024-04-22T18:41:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937567+0000", 2024-04-22T18:41:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014363099999999999 2024-04-22T18:41:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938437+0000", 2024-04-22T18:41:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087060399999999997 2024-04-22T18:41:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3164 4.8 4:187adac1:::smithi08937929-36:head [sparse-read 0~2643888] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:27.306 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.426925021000002, 2024-04-22T18:41:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1013670310000001, 2024-04-22T18:41:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3164 2024-04-22T18:41:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:27.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837499+0000", 2024-04-22T18:41:27.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.994e-06 2024-04-22T18:41:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837503+0000", 2024-04-22T18:41:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7249999999999999e-06 2024-04-22T18:41:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837506+0000", 2024-04-22T18:41:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5980000000000001e-06 2024-04-22T18:41:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937641+0000", 2024-04-22T18:41:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10013467700000001 2024-04-22T18:41:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937650+0000", 2024-04-22T18:41:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3489999999999998e-06 2024-04-22T18:41:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938468+0000", 2024-04-22T18:41:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0008176710000001 2024-04-22T18:41:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938582+0000", 2024-04-22T18:41:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114287 2024-04-22T18:41:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938864+0000", 2024-04-22T18:41:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028120899999999998 2024-04-22T18:41:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3165 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:27.311 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.426906252999999, 2024-04-22T18:41:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2022754499999999, 2024-04-22T18:41:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3165 2024-04-22T18:41:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837517+0000", 2024-04-22T18:41:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3369999999999998e-06 2024-04-22T18:41:27.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837523+0000", 2024-04-22T18:41:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5450000000000003e-06 2024-04-22T18:41:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837526+0000", 2024-04-22T18:41:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.354e-06 2024-04-22T18:41:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937682+0000", 2024-04-22T18:41:27.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100155917 2024-04-22T18:41:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937688+0000", 2024-04-22T18:41:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:41:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.038953+0000", 2024-04-22T18:41:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012648110000001 2024-04-22T18:41:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039034+0000", 2024-04-22T18:41:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0969000000000004e-05 2024-04-22T18:41:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039791+0000", 2024-04-22T18:41:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075672900000000004 2024-04-22T18:41:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3183 4.8 4:1bc4668a:::smithi08937929-21:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:27.316 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.228503055000004, 2024-04-22T18:41:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0040478260000001, 2024-04-22T18:41:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3183 2024-04-22T18:41:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:27.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035913+0000", 2024-04-22T18:41:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999943 2024-04-22T18:41:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035935+0000", 2024-04-22T18:41:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2232999999999999e-05 2024-04-22T18:41:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035969+0000", 2024-04-22T18:41:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3969e-05 2024-04-22T18:41:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035987+0000", 2024-04-22T18:41:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8397e-05 2024-04-22T18:41:27.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2916000000000003e-05 2024-04-22T18:41:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:41:27.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0103e-05 2024-04-22T18:41:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039841+0000", 2024-04-22T18:41:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0037508959999999 2024-04-22T18:41:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039966+0000", 2024-04-22T18:41:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012514500000000001 2024-04-22T18:41:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.321 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3184 4.8 4:12d9f826:::smithi08937929-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.228415096000006, 2024-04-22T18:41:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.00402868, 2024-04-22T18:41:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3184 2024-04-22T18:41:27.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036005+0000", 2024-04-22T18:41:27.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:27.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036026+0000", 2024-04-22T18:41:27.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1092999999999999e-05 2024-04-22T18:41:27.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036030+0000", 2024-04-22T18:41:27.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.467e-06 2024-04-22T18:41:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036033+0000", 2024-04-22T18:41:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9780000000000001e-06 2024-04-22T18:41:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036120+0000", 2024-04-22T18:41:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7495000000000002e-05 2024-04-22T18:41:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036125+0000", 2024-04-22T18:41:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8720000000000001e-06 2024-04-22T18:41:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039982+0000", 2024-04-22T18:41:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0038571919999999 2024-04-22T18:41:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040035+0000", 2024-04-22T18:41:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2992999999999999e-05 2024-04-22T18:41:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3185 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.228361183999994, 2024-04-22T18:41:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0041404039999999, 2024-04-22T18:41:27.327 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3185 2024-04-22T18:41:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036062+0000", 2024-04-22T18:41:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9099999999999999e-06 2024-04-22T18:41:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036066+0000", 2024-04-22T18:41:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1699999999999999e-06 2024-04-22T18:41:27.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036077+0000", 2024-04-22T18:41:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0637e-05 2024-04-22T18:41:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036147+0000", 2024-04-22T18:41:27.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0025999999999997e-05 2024-04-22T18:41:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036151+0000", 2024-04-22T18:41:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:41:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040068+0000", 2024-04-22T18:41:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.003916604 2024-04-22T18:41:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040201+0000", 2024-04-22T18:41:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013275400000000001 2024-04-22T18:41:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.333 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3186 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.228341627999995, 2024-04-22T18:41:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2044167510000001, 2024-04-22T18:41:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3186 2024-04-22T18:41:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036079+0000", 2024-04-22T18:41:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036081+0000", 2024-04-22T18:41:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6449999999999999e-06 2024-04-22T18:41:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036084+0000", 2024-04-22T18:41:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9399999999999998e-06 2024-04-22T18:41:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:41:27.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.879e-06 2024-04-22T18:41:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036168+0000", 2024-04-22T18:41:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7657999999999994e-05 2024-04-22T18:41:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036171+0000", 2024-04-22T18:41:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7639999999999999e-06 2024-04-22T18:41:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240345+0000", 2024-04-22T18:41:27.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2041734260000001 2024-04-22T18:41:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240497+0000", 2024-04-22T18:41:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015182800000000001 2024-04-22T18:41:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3724 4.18 4:1bc4668a:::smithi08937929-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 42.204017252, 2024-04-22T18:41:27.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50413114699999995, 2024-04-22T18:41:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3724 2024-04-22T18:41:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060405+0000", 2024-04-22T18:41:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.395e-06 2024-04-22T18:41:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060408+0000", 2024-04-22T18:41:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:41:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060411+0000", 2024-04-22T18:41:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6060000000000001e-06 2024-04-22T18:41:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165799+0000", 2024-04-22T18:41:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105388369 2024-04-22T18:41:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165843+0000", 2024-04-22T18:41:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3891999999999999e-05 2024-04-22T18:41:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564408+0000", 2024-04-22T18:41:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39856466000000002 2024-04-22T18:41:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564461+0000", 2024-04-22T18:41:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3529999999999997e-05 2024-04-22T18:41:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564535+0000", 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4147999999999995e-05 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4413 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.6070372079999995, 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2025380569999999, 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.343 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4413 2024-04-22T18:41:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657377+0000", 2024-04-22T18:41:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999933 2024-04-22T18:41:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657393+0000", 2024-04-22T18:41:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5296999999999999e-05 2024-04-22T18:41:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657411+0000", 2024-04-22T18:41:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8105e-05 2024-04-22T18:41:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657424+0000", 2024-04-22T18:41:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3030000000000001e-05 2024-04-22T18:41:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658313+0000", 2024-04-22T18:41:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00088943099999999999 2024-04-22T18:41:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658324+0000", 2024-04-22T18:41:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0496000000000001e-05 2024-04-22T18:41:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859560+0000", 2024-04-22T18:41:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2012358350000001 2024-04-22T18:41:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859648+0000", 2024-04-22T18:41:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8744000000000003e-05 2024-04-22T18:41:27.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859922+0000", 2024-04-22T18:41:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027396499999999998 2024-04-22T18:41:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4414 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.6069828840000007, 2024-04-22T18:41:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2030255030000001, 2024-04-22T18:41:27.348 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4414 2024-04-22T18:41:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657437+0000", 2024-04-22T18:41:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657443+0000", 2024-04-22T18:41:27.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5899999999999998e-06 2024-04-22T18:41:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657450+0000", 2024-04-22T18:41:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4649999999999997e-06 2024-04-22T18:41:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657457+0000", 2024-04-22T18:41:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6000000000000003e-06 2024-04-22T18:41:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658364+0000", 2024-04-22T18:41:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090735999999999996 2024-04-22T18:41:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658373+0000", 2024-04-22T18:41:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3250000000000008e-06 2024-04-22T18:41:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860258+0000", 2024-04-22T18:41:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018852769999999 2024-04-22T18:41:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860308+0000", 2024-04-22T18:41:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9938000000000002e-05 2024-04-22T18:41:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860464+0000", 2024-04-22T18:41:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015622800000000001 2024-04-22T18:41:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4415 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:27.353 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.6069585600000007, 2024-04-22T18:41:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2038250989999999, 2024-04-22T18:41:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4415 2024-04-22T18:41:27.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657462+0000", 2024-04-22T18:41:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657467+0000", 2024-04-22T18:41:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5399999999999997e-06 2024-04-22T18:41:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657473+0000", 2024-04-22T18:41:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3949999999999998e-06 2024-04-22T18:41:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657480+0000", 2024-04-22T18:41:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2489999999999998e-06 2024-04-22T18:41:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658394+0000", 2024-04-22T18:41:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00091403299999999997 2024-04-22T18:41:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:27.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658398+0000", 2024-04-22T18:41:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7979999999999996e-06 2024-04-22T18:41:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860495+0000", 2024-04-22T18:41:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.202097081 2024-04-22T18:41:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860543+0000", 2024-04-22T18:41:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7907000000000001e-05 2024-04-22T18:41:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861288+0000", 2024-04-22T18:41:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074466500000000004 2024-04-22T18:41:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.359 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4413 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.6042420550000003, 2024-04-22T18:41:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.201491952, 2024-04-22T18:41:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:27.360 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4413 2024-04-22T18:41:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660178+0000", 2024-04-22T18:41:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660185+0000", 2024-04-22T18:41:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4379999999999998e-06 2024-04-22T18:41:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660201+0000", 2024-04-22T18:41:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5252999999999999e-05 2024-04-22T18:41:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660210+0000", 2024-04-22T18:41:27.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1069999999999994e-06 2024-04-22T18:41:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660270+0000", 2024-04-22T18:41:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739000000000002e-05 2024-04-22T18:41:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660281+0000", 2024-04-22T18:41:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0664e-05 2024-04-22T18:41:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861304+0000", 2024-04-22T18:41:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2010225059999999 2024-04-22T18:41:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861342+0000", 2024-04-22T18:41:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8736000000000003e-05 2024-04-22T18:41:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861671+0000", 2024-04-22T18:41:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032915899999999998 2024-04-22T18:41:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4414 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.6041985449999991, 2024-04-22T18:41:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2014303930000001, 2024-04-22T18:41:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.366 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4414 2024-04-22T18:41:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660222+0000", 2024-04-22T18:41:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660227+0000", 2024-04-22T18:41:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6789999999999996e-06 2024-04-22T18:41:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660240+0000", 2024-04-22T18:41:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3485e-05 2024-04-22T18:41:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660245+0000", 2024-04-22T18:41:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8269999999999997e-06 2024-04-22T18:41:27.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660307+0000", 2024-04-22T18:41:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2392999999999997e-05 2024-04-22T18:41:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660316+0000", 2024-04-22T18:41:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7790000000000001e-06 2024-04-22T18:41:27.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861493+0000", 2024-04-22T18:41:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.201177341 2024-04-22T18:41:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861529+0000", 2024-04-22T18:41:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5978e-05 2024-04-22T18:41:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861653+0000", 2024-04-22T18:41:27.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012391799999999999 2024-04-22T18:41:27.372 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.372 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.373 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.373 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4415 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:27.373 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:27.373 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.6041533210000001, 2024-04-22T18:41:27.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2019634779999999, 2024-04-22T18:41:27.374 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:27.374 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:27.374 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:27.374 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:27.374 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.374 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4415 2024-04-22T18:41:27.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.374 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:27.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:27.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:27.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:27.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:27.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:27.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:27.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660267+0000", 2024-04-22T18:41:27.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:27.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:27.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660273+0000", 2024-04-22T18:41:27.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5910000000000003e-06 2024-04-22T18:41:27.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:27.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660280+0000", 2024-04-22T18:41:27.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4000000000000003e-06 2024-04-22T18:41:27.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660288+0000", 2024-04-22T18:41:27.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1769999999999998e-06 2024-04-22T18:41:27.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.760401+0000", 2024-04-22T18:41:27.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100112864 2024-04-22T18:41:27.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:27.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.760408+0000", 2024-04-22T18:41:27.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0299999999999996e-06 2024-04-22T18:41:27.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:27.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861682+0000", 2024-04-22T18:41:27.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012742520000001 2024-04-22T18:41:27.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:27.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861716+0000", 2024-04-22T18:41:27.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3674999999999999e-05 2024-04-22T18:41:27.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:27.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:27.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:27.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.862232+0000", 2024-04-22T18:41:27.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051554300000000003 2024-04-22T18:41:27.378 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.378 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.378 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.378 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:27.379 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:27.379 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:27.380 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:27.381 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:27.380+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:27.408 INFO:tasks.rados.rados.0.smithi089.stdout:1912: got (0) Success 2024-04-22T18:41:27.408 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 72 (ObjNum 398 snap 0 seq_num 398) dirty exists 2024-04-22T18:41:27.408 INFO:tasks.rados.rados.0.smithi089.stdout:1908: finishing write tid 2 to smithi08937929-29 2024-04-22T18:41:27.408 INFO:tasks.rados.rados.0.smithi089.stdout:1908: finishing write tid 3 to smithi08937929-29 2024-04-22T18:41:27.408 INFO:tasks.rados.rados.0.smithi089.stdout:1908: finishing write tid 4 to smithi08937929-29 2024-04-22T18:41:27.408 INFO:tasks.rados.rados.0.smithi089.stdout:1908: oid 29 updating version 0 to 143 2024-04-22T18:41:27.408 INFO:tasks.rados.rados.0.smithi089.stdout:1908: oid 29 updating version 143 to 144 2024-04-22T18:41:27.409 INFO:tasks.rados.rados.0.smithi089.stdout:1908: oid 29 updating version 144 to 145 2024-04-22T18:41:27.409 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 145 (ObjNum 407 snap 0 seq_num 407) dirty exists 2024-04-22T18:41:27.409 INFO:tasks.rados.rados.0.smithi089.stdout:1908: left oid 29 (ObjNum 407 snap 0 seq_num 407) 2024-04-22T18:41:27.409 INFO:tasks.rados.rados.0.smithi089.stdout:1909: finishing set_chunk to oid 48 2024-04-22T18:41:27.409 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 48 set_chunk returned error code -95 offset: 1822720 length: 673792 2024-04-22T18:41:27.409 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 0 (ObjNum 351 snap 0 seq_num 351) dirty exists 2024-04-22T18:41:27.409 INFO:tasks.rados.rados.0.smithi089.stdout:1911: finishing write tid 1 to smithi08937929-3 2024-04-22T18:41:27.409 INFO:tasks.rados.rados.0.smithi089.stdout:1908: done (15 left) 2024-04-22T18:41:27.409 INFO:tasks.rados.rados.0.smithi089.stdout:1909: done (14 left) 2024-04-22T18:41:27.409 INFO:tasks.rados.rados.0.smithi089.stdout:1910: done (13 left) 2024-04-22T18:41:27.409 INFO:tasks.rados.rados.0.smithi089.stdout:1912: done (12 left) 2024-04-22T18:41:27.409 INFO:tasks.rados.rados.0.smithi089.stdout:1922: read oid 26 snap -1 2024-04-22T18:41:27.409 INFO:tasks.rados.rados.0.smithi089.stdout:1922: expect (ObjNum 353 snap 0 seq_num 353) 2024-04-22T18:41:27.410 INFO:tasks.rados.rados.0.smithi089.stdout:1923: set_chunk oid 16 target oid 2024-04-22T18:41:27.410 INFO:tasks.rados.rados.0.smithi089.stdout:1923: set_chunk oid 16 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:41:27.410 INFO:tasks.rados.rados.0.smithi089.stdout:1924: read oid 15 snap -1 2024-04-22T18:41:27.410 INFO:tasks.rados.rados.0.smithi089.stdout:1924: expect (ObjNum 377 snap 0 seq_num 377) 2024-04-22T18:41:27.410 INFO:tasks.rados.rados.0.smithi089.stdout:1925: read oid 13 snap -1 2024-04-22T18:41:27.410 INFO:tasks.rados.rados.0.smithi089.stdout:1925: expect deleted 2024-04-22T18:41:27.410 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:41:27.410 INFO:tasks.rados.rados.0.smithi089.stdout:1905: finishing write tid 4 to smithi08937929-19 2024-04-22T18:41:27.410 INFO:tasks.rados.rados.0.smithi089.stdout:1905: oid 19 updating version 0 to 147 2024-04-22T18:41:27.410 INFO:tasks.rados.rados.0.smithi089.stdout:1905: oid 19 updating version 147 to 149 2024-04-22T18:41:27.410 INFO:tasks.rados.rados.0.smithi089.stdout:1905: oid 19 version 149 is already newer than 148 2024-04-22T18:41:27.410 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 149 (ObjNum 406 snap 0 seq_num 406) dirty exists 2024-04-22T18:41:27.410 INFO:tasks.rados.rados.0.smithi089.stdout:1905: left oid 19 (ObjNum 406 snap 0 seq_num 406) 2024-04-22T18:41:27.410 INFO:tasks.rados.rados.0.smithi089.stdout:1911: finishing write tid 2 to smithi08937929-3 2024-04-22T18:41:27.411 INFO:tasks.rados.rados.0.smithi089.stdout:1911: finishing write tid 3 to smithi08937929-3 2024-04-22T18:41:27.411 INFO:tasks.rados.rados.0.smithi089.stdout:1911: finishing write tid 4 to smithi08937929-3 2024-04-22T18:41:27.411 INFO:tasks.rados.rados.0.smithi089.stdout:1905: done (15 left) 2024-04-22T18:41:27.411 INFO:tasks.rados.rados.0.smithi089.stdout:1926: write (excl) oid 37 current snap is 0 2024-04-22T18:41:27.411 INFO:tasks.rados.rados.0.smithi089.stdout:1911: finishing write tid 5 to smithi08937929-3 2024-04-22T18:41:27.411 INFO:tasks.rados.rados.0.smithi089.stdout:1911: oid 3 updating version 0 to 81 2024-04-22T18:41:27.411 INFO:tasks.rados.rados.0.smithi089.stdout:1911: oid 3 updating version 81 to 82 2024-04-22T18:41:27.411 INFO:tasks.rados.rados.0.smithi089.stdout:1911: oid 3 version 82 is already newer than 79 2024-04-22T18:41:27.411 INFO:tasks.rados.rados.0.smithi089.stdout:1911: oid 3 version 82 is already newer than 80 2024-04-22T18:41:27.411 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 82 (ObjNum 408 snap 0 seq_num 408) dirty exists 2024-04-22T18:41:27.411 INFO:tasks.rados.rados.0.smithi089.stdout:1911: left oid 3 (ObjNum 408 snap 0 seq_num 408) 2024-04-22T18:41:27.411 INFO:tasks.rados.rados.0.smithi089.stdout:1914: expect (ObjNum 399 snap 0 seq_num 399) 2024-04-22T18:41:27.481 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:27.482+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:27.484 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:27.485 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:27.493 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:27.580 INFO:tasks.rados.rados.0.smithi089.stdout:1916: expect (ObjNum 347 snap 0 seq_num 347) 2024-04-22T18:41:27.581 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:27.582+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:27.635 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:27.635 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:27.636 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:27.636 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:27.636 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:27.644 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:27.682 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:27.684+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:27.708 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:27.708+0000 7f1e2dd22640 -1 osd.2 151 handle_osd_map injecting map crc failure 2024-04-22T18:41:27.708 INFO:tasks.rados.rados.0.smithi089.stdout:1913: expect (ObjNum 370 snap 0 seq_num 370) 2024-04-22T18:41:27.782 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:27.783+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:27.790 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:27.790 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:27.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:27.790 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:27.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.790 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:27.790 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 153.334950475, 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:27.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:41:27.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:27.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:41:27.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:41:27.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:41:27.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:41:27.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:41:27.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:41:27.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:41:27.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:41:27.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:41:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:41:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:41:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:41:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:41:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:41:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:41:27.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:41:27.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:27.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:41:27.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:41:27.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:41:27.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:41:27.795 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.795 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.796 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.796 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:27.796 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:27.796 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 144.641893227, 2024-04-22T18:41:27.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:41:27.796 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.796 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.796 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:27.796 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:27.796 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.796 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:41:27.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.796 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:27.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:27.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:41:27.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:41:27.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:41:27.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:41:27.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:41:27.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:41:27.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:41:27.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:41:27.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:41:27.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:41:27.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:41:27.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:27.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:41:27.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:41:27.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:27.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:41:27.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:41:27.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:41:27.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:41:27.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:27.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:41:27.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:41:27.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:41:27.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:41:27.801 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.801 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.801 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.802 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:27.802 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:27.802 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 144.64187475099999, 2024-04-22T18:41:27.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:41:27.802 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.802 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:27.802 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:27.802 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:27.802 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.802 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:41:27.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.802 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:27.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:27.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:27.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:41:27.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:41:27.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:41:27.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:41:27.806 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.806 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.806 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.806 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:41:27.806 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:27.806 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 116.87882643899999, 2024-04-22T18:41:27.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:41:27.807 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.807 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:27.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:27.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.807 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:41:27.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.807 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:27.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:27.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:41:27.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:27.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:41:27.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:41:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:41:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:41:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:41:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:41:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:41:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:41:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:41:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:41:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:41:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:41:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:41:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:41:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:41:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:41:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:41:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:41:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:41:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:41:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:27.812 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 111.349667113, 2024-04-22T18:41:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:41:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:41:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:41:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:41:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:41:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:41:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:41:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:41:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:41:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:41:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:41:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:41:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:41:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:41:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:41:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:41:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:41:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:41:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.817 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:41:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 111.171959718, 2024-04-22T18:41:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:41:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:41:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:41:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:41:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:41:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:41:27.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:41:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:41:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:41:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:41:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:41:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:41:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:41:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:41:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:41:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:41:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:27.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:41:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:41:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:41:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:41:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:41:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:41:27.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:41:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:41:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:41:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 111.171891446, 2024-04-22T18:41:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:41:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:41:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:41:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:41:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:41:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:41:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:41:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:41:27.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:41:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:41:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:41:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:41:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:41:27.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:41:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:41:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:41:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:41:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:41:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:41:27.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.828 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 111.171723605, 2024-04-22T18:41:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:41:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:41:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:41:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:41:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:41:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:41:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:41:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:41:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:41:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:41:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:41:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:41:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:41:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:41:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:41:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.833 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.20639813, 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:41:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:41:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:41:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:41:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:41:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:41:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:41:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:41:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:41:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:41:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:41:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:41:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:41:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:41:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:41:27.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:41:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:41:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:41:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:41:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:41:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:41:27.838 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:41:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.106245943, 2024-04-22T18:41:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:41:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:41:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:41:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:41:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:41:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:41:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:41:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:41:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:41:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:41:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:41:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:41:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:41:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:41:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:41:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:41:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:41:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:41:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:41:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:41:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:41:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:41:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:41:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:41:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:41:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 97.397381422999999, 2024-04-22T18:41:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:41:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:27.844 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:41:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:27.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:41:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:41:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:41:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:41:27.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:41:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:41:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:41:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:41:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:41:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:27.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:41:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:41:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:41:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:41:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:41:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:41:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:41:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:41:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:41:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 80.501343051000006, 2024-04-22T18:41:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:41:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:27.849 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:41:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:41:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:41:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:41:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:41:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:41:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:41:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:41:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:41:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:41:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:41:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:41:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:41:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3330 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-04-22T18:41:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:41:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.766264765000003, 2024-04-22T18:41:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43151673600000001, 2024-04-22T18:41:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:27.853 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3330 2024-04-22T18:41:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:41:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:41:27.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024041+0000", 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024045+0000", 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3950000000000001e-06 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024051+0000", 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0260000000000004e-06 2024-04-22T18:41:27.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024056+0000", 2024-04-22T18:41:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4489999999999998e-06 2024-04-22T18:41:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028383+0000", 2024-04-22T18:41:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004326557 2024-04-22T18:41:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028471+0000", 2024-04-22T18:41:27.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8387999999999995e-05 2024-04-22T18:41:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.455559+0000", 2024-04-22T18:41:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.42708770699999998 2024-04-22T18:41:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3377 4.6 4:6700eab5:::smithi08937929-32:head [read 1812992~307712] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-04-22T18:41:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 57.353939408999999, 2024-04-22T18:41:27.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.52001745799999999, 2024-04-22T18:41:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3377 2024-04-22T18:41:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436364+0000", 2024-04-22T18:41:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436376+0000", 2024-04-22T18:41:27.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2121e-05 2024-04-22T18:41:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436394+0000", 2024-04-22T18:41:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8094e-05 2024-04-22T18:41:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436405+0000", 2024-04-22T18:41:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0189e-05 2024-04-22T18:41:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436471+0000", 2024-04-22T18:41:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6817999999999996e-05 2024-04-22T18:41:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436481+0000", 2024-04-22T18:41:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2720000000000003e-06 2024-04-22T18:41:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956084+0000", 2024-04-22T18:41:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51960320800000004 2024-04-22T18:41:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956232+0000", 2024-04-22T18:41:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014778600000000001 2024-04-22T18:41:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956385+0000", 2024-04-22T18:41:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153394 2024-04-22T18:41:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:131 3.1 3:a0a19997:::ee8ca23a7050e771c77dff761e6ea5a11ed57796f4e8a17cbb5804a7fdc937b9:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:27.862 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.461069813000002, 2024-04-22T18:41:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51449564999999997, 2024-04-22T18:41:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329235+0000", 2024-04-22T18:41:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329244+0000", 2024-04-22T18:41:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6819999999999992e-06 2024-04-22T18:41:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329259+0000", 2024-04-22T18:41:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698000000000001e-05 2024-04-22T18:41:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329265+0000", 2024-04-22T18:41:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1510000000000001e-06 2024-04-22T18:41:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329311+0000", 2024-04-22T18:41:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6096e-05 2024-04-22T18:41:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.332269+0000", 2024-04-22T18:41:27.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029581910000000002 2024-04-22T18:41:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630603+0000", 2024-04-22T18:41:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29833418 2024-04-22T18:41:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630685+0000", 2024-04-22T18:41:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1217999999999994e-05 2024-04-22T18:41:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:27.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630813+0000", 2024-04-22T18:41:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128706 2024-04-22T18:41:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.631447+0000", 2024-04-22T18:41:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063427000000000004 2024-04-22T18:41:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843586+0000", 2024-04-22T18:41:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21213873 2024-04-22T18:41:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843623+0000", 2024-04-22T18:41:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6848999999999999e-05 2024-04-22T18:41:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843733+0000", 2024-04-22T18:41:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010982200000000001 2024-04-22T18:41:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3441 4.b 4:d3068ee3:::smithi08937929-29:head [write 558065~619869] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:27.868 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.872593752, 2024-04-22T18:41:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50272020399999995, 2024-04-22T18:41:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3441 2024-04-22T18:41:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:27.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917711+0000", 2024-04-22T18:41:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917838+0000", 2024-04-22T18:41:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012734899999999999 2024-04-22T18:41:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917854+0000", 2024-04-22T18:41:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6130999999999999e-05 2024-04-22T18:41:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917863+0000", 2024-04-22T18:41:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5969999999999992e-06 2024-04-22T18:41:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992845+0000", 2024-04-22T18:41:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.074982193000000003 2024-04-22T18:41:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992919+0000", 2024-04-22T18:41:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:41:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.993019+0000", 2024-04-22T18:41:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9718999999999999e-05 2024-04-22T18:41:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095554+0000", 2024-04-22T18:41:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102535193 2024-04-22T18:41:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420183+0000", 2024-04-22T18:41:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32462853600000002 2024-04-22T18:41:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420401+0000", 2024-04-22T18:41:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021881400000000001 2024-04-22T18:41:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420433+0000", 2024-04-22T18:41:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079000000000002e-05 2024-04-22T18:41:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3442 4.b 4:d3068ee3:::smithi08937929-29:head [write 1579835~486765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.867907041000002, 2024-04-22T18:41:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49822799099999998, 2024-04-22T18:41:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.874 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3442 2024-04-22T18:41:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922399+0000", 2024-04-22T18:41:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922502+0000", 2024-04-22T18:41:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102784 2024-04-22T18:41:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922513+0000", 2024-04-22T18:41:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1051e-05 2024-04-22T18:41:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922518+0000", 2024-04-22T18:41:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8529999999999997e-06 2024-04-22T18:41:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093158+0000", 2024-04-22T18:41:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.17063958800000001 2024-04-22T18:41:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093230+0000", 2024-04-22T18:41:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1803000000000004e-05 2024-04-22T18:41:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093326+0000", 2024-04-22T18:41:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6454999999999997e-05 2024-04-22T18:41:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095557+0000", 2024-04-22T18:41:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022306399999999999 2024-04-22T18:41:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420484+0000", 2024-04-22T18:41:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32492717700000001 2024-04-22T18:41:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420606+0000", 2024-04-22T18:41:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012225800000000001 2024-04-22T18:41:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420628+0000", 2024-04-22T18:41:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1596000000000001e-05 2024-04-22T18:41:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.879 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3443 4.b 4:d3068ee3:::smithi08937929-29:head [write 2468371~489694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.863304994000003, 2024-04-22T18:41:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49372487199999998, 2024-04-22T18:41:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3443 2024-04-22T18:41:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.880 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927001+0000", 2024-04-22T18:41:27.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927117+0000", 2024-04-22T18:41:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011637 2024-04-22T18:41:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927128+0000", 2024-04-22T18:41:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0696000000000001e-05 2024-04-22T18:41:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927134+0000", 2024-04-22T18:41:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8640000000000001e-06 2024-04-22T18:41:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094288+0000", 2024-04-22T18:41:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16715412900000001 2024-04-22T18:41:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094359+0000", 2024-04-22T18:41:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1639999999999998e-05 2024-04-22T18:41:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094451+0000", 2024-04-22T18:41:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1252e-05 2024-04-22T18:41:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095560+0000", 2024-04-22T18:41:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011095530000000001 2024-04-22T18:41:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420661+0000", 2024-04-22T18:41:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32510105500000003 2024-04-22T18:41:27.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420710+0000", 2024-04-22T18:41:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9018999999999999e-05 2024-04-22T18:41:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420727+0000", 2024-04-22T18:41:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6722e-05 2024-04-22T18:41:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3444 4.b 4:d3068ee3:::smithi08937929-29:head [setxattr _header (56) in=7b,truncate 2958065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.863169276999997, 2024-04-22T18:41:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49366935299999998, 2024-04-22T18:41:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3444 2024-04-22T18:41:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927137+0000", 2024-04-22T18:41:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927141+0000", 2024-04-22T18:41:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3270000000000001e-06 2024-04-22T18:41:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927144+0000", 2024-04-22T18:41:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9730000000000002e-06 2024-04-22T18:41:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927147+0000", 2024-04-22T18:41:27.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.959e-06 2024-04-22T18:41:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095118+0000", 2024-04-22T18:41:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16797115500000001 2024-04-22T18:41:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095169+0000", 2024-04-22T18:41:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1316000000000001e-05 2024-04-22T18:41:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095260+0000", 2024-04-22T18:41:27.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1184e-05 2024-04-22T18:41:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095999+0000", 2024-04-22T18:41:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073868200000000005 2024-04-22T18:41:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420757+0000", 2024-04-22T18:41:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32475783499999999 2024-04-22T18:41:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420781+0000", 2024-04-22T18:41:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4022000000000001e-05 2024-04-22T18:41:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420807+0000", 2024-04-22T18:41:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6366000000000001e-05 2024-04-22T18:41:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3445 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:27.891 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.863158183000003, 2024-04-22T18:41:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.493780049, 2024-04-22T18:41:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3445 2024-04-22T18:41:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:27.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927150+0000", 2024-04-22T18:41:27.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.542e-06 2024-04-22T18:41:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927153+0000", 2024-04-22T18:41:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0520000000000002e-06 2024-04-22T18:41:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027218+0000", 2024-04-22T18:41:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100064604 2024-04-22T18:41:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095501+0000", 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.068283431000000006 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095545+0000", 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4295999999999997e-05 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420833+0000", 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32528718400000001 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420877+0000", 2024-04-22T18:41:27.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4403999999999997e-05 2024-04-22T18:41:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420929+0000", 2024-04-22T18:41:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1848999999999998e-05 2024-04-22T18:41:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:27.896 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:27.896 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:27.898 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:27.898 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:27.898+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:27.900 INFO:tasks.rados.rados.0.smithi089.stdout:1915: expect (ObjNum 392 snap 0 seq_num 392) 2024-04-22T18:41:27.998 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:27.999+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:28.004 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4701 4.1b 4:d923883e:::smithi08937929-16:head [set-chunk in=77b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.408330+0000", 2024-04-22T18:41:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.59717631100000002, 2024-04-22T18:41:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59719699199999998, 2024-04-22T18:41:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for missing object", 2024-04-22T18:41:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4701 2024-04-22T18:41:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408330+0000", 2024-04-22T18:41:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408330+0000", 2024-04-22T18:41:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408326+0000", 2024-04-22T18:41:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999962 2024-04-22T18:41:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408343+0000", 2024-04-22T18:41:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6731999999999998e-05 2024-04-22T18:41:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408366+0000", 2024-04-22T18:41:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3567e-05 2024-04-22T18:41:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408379+0000", 2024-04-22T18:41:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2775999999999999e-05 2024-04-22T18:41:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.557823+0000", 2024-04-22T18:41:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14944426899999999 2024-04-22T18:41:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.557837+0000", 2024-04-22T18:41:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:41:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.858557+0000", 2024-04-22T18:41:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30072007899999997 2024-04-22T18:41:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.858706+0000", 2024-04-22T18:41:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014876000000000001 2024-04-22T18:41:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPushReply(4.7 152/151 [PushReplyOp(4:e117a562:::smithi08937929-33:head)])", 2024-04-22T18:41:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.965382+0000", 2024-04-22T18:41:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.040124112000000003, 2024-04-22T18:41:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.040190788999999998, 2024-04-22T18:41:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:41:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.965382+0000", 2024-04-22T18:41:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.965382+0000", 2024-04-22T18:41:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.965380+0000", 2024-04-22T18:41:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.965389+0000", 2024-04-22T18:41:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6340000000000007e-06 2024-04-22T18:41:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.965403+0000", 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4154999999999999e-05 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.965413+0000", 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4669999999999993e-06 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4692 4.3 4.ee6b2b03 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.74636658300000003, 2024-04-22T18:41:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.74645803200000005, 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4692 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259139+0000", 2024-04-22T18:41:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259145+0000", 2024-04-22T18:41:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4860000000000002e-06 2024-04-22T18:41:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259151+0000", 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1160000000000004e-06 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259156+0000", 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6489999999999999e-06 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259191+0000", 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4737999999999998e-05 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259197+0000", 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9160000000000004e-06 2024-04-22T18:41:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4693 4.3 4.ee6b2b03 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.74634429700000005, 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.74646697699999998, 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4693 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259161+0000", 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259164+0000", 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-06 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259169+0000", 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6260000000000003e-06 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259174+0000", 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7659999999999998e-06 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259231+0000", 2024-04-22T18:41:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6827000000000003e-05 2024-04-22T18:41:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259239+0000", 2024-04-22T18:41:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6889999999999993e-06 2024-04-22T18:41:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4694 4.3 4.ee6b2b03 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.74632801599999998, 2024-04-22T18:41:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.74651140999999999, 2024-04-22T18:41:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4694 2024-04-22T18:41:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259182+0000", 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8700000000000002e-06 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259187+0000", 2024-04-22T18:41:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1669999999999998e-06 2024-04-22T18:41:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259191+0000", 2024-04-22T18:41:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6210000000000003e-06 2024-04-22T18:41:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.359347+0000", 2024-04-22T18:41:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100155518 2024-04-22T18:41:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.359356+0000", 2024-04-22T18:41:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6519999999999995e-06 2024-04-22T18:41:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:41:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 5 2024-04-22T18:41:28.022 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:28.023 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:28.026 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:41:28.044 INFO:tasks.rados.rados.0.smithi089.stdout:1921: expect (ObjNum 403 snap 0 seq_num 403) 2024-04-22T18:41:28.099 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:28.101+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:28.150 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:28.150 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:28.150 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:28.150 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:28.150 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:28.159 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:28.200 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:28.202+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:28.232 INFO:tasks.rados.rados.0.smithi089.stdout:1919: finishing set_chunk to oid 2 2024-04-22T18:41:28.232 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 78 (ObjNum 385 snap 0 seq_num 385) dirty exists 2024-04-22T18:41:28.232 INFO:tasks.rados.rados.0.smithi089.stdout:1922: expect (ObjNum 353 snap 0 seq_num 353) 2024-04-22T18:41:28.266 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:28.266 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:28.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:28.267 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:28.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.267 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:28.267 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:28.267 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 154.07502748900001, 2024-04-22T18:41:28.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:41:28.267 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.267 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.267 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.267 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.267 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.267 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:41:28.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.267 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:28.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:28.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:41:28.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:41:28.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:41:28.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:41:28.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:41:28.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:41:28.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:41:28.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:41:28.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:41:28.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:41:28.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:41:28.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:28.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:41:28.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:41:28.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:28.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:41:28.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:41:28.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:28.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:41:28.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:41:28.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:28.272 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 154.075016414, 2024-04-22T18:41:28.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:41:28.273 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.273 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.273 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.273 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.273 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.273 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:41:28.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.273 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:28.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:28.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:41:28.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:41:28.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:41:28.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:41:28.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:41:28.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:41:28.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:41:28.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:41:28.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:41:28.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:28.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:41:28.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:41:28.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.278 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:308 4.f 4:f0ed3d49:::smithi08937929-9:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:41:28.278 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:41:28.278 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 150.47364691999999, 2024-04-22T18:41:28.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.63730570499999994, 2024-04-22T18:41:28.278 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.278 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.278 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.278 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.278 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.278 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 308 2024-04-22T18:41:28.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.278 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:41:28.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792907+0000", 2024-04-22T18:41:28.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792906+0000", 2024-04-22T18:41:28.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:28.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792913+0000", 2024-04-22T18:41:28.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-04-22T18:41:28.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.792922+0000", 2024-04-22T18:41:28.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3919999999999992e-06 2024-04-22T18:41:28.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.892992+0000", 2024-04-22T18:41:28.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10006997199999999 2024-04-22T18:41:28.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993135+0000", 2024-04-22T18:41:28.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100143041 2024-04-22T18:41:28.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:57.993200+0000", 2024-04-22T18:41:28.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5143999999999998e-05 2024-04-22T18:41:28.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:28.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.429475+0000", 2024-04-22T18:41:28.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43627463700000002 2024-04-22T18:41:28.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:28.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430054+0000", 2024-04-22T18:41:28.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000579077 2024-04-22T18:41:28.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:28.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430159+0000", 2024-04-22T18:41:28.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105184 2024-04-22T18:41:28.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430187+0000", 2024-04-22T18:41:28.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8104000000000001e-05 2024-04-22T18:41:28.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:58.430213+0000", 2024-04-22T18:41:28.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5660000000000002e-05 2024-04-22T18:41:28.283 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.283 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.283 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.283 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3193 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:28.283 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:41:28.283 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 66.223808125000005, 2024-04-22T18:41:28.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69912102700000001, 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3193 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042744+0000", 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043013+0000", 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026906 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043028+0000", 2024-04-22T18:41:28.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5359999999999999e-05 2024-04-22T18:41:28.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043036+0000", 2024-04-22T18:41:28.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8420000000000005e-06 2024-04-22T18:41:28.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043081+0000", 2024-04-22T18:41:28.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5019999999999999e-05 2024-04-22T18:41:28.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043087+0000", 2024-04-22T18:41:28.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5420000000000002e-06 2024-04-22T18:41:28.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741856+0000", 2024-04-22T18:41:28.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69876864500000002 2024-04-22T18:41:28.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741867+0000", 2024-04-22T18:41:28.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.084e-05 2024-04-22T18:41:28.287 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.288 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.288 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.288 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3194 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:28.288 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:41:28.288 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 66.216558144999993, 2024-04-22T18:41:28.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69190137100000004, 2024-04-22T18:41:28.288 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.288 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:28.288 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.288 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.288 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.288 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3194 2024-04-22T18:41:28.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.289 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:41:28.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:41:28.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049991+0000", 2024-04-22T18:41:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999957 2024-04-22T18:41:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050284+0000", 2024-04-22T18:41:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029296699999999999 2024-04-22T18:41:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050304+0000", 2024-04-22T18:41:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.969e-05 2024-04-22T18:41:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050314+0000", 2024-04-22T18:41:28.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0268999999999999e-05 2024-04-22T18:41:28.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150531+0000", 2024-04-22T18:41:28.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100216868 2024-04-22T18:41:28.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150545+0000", 2024-04-22T18:41:28.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4063000000000001e-05 2024-04-22T18:41:28.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741890+0000", 2024-04-22T18:41:28.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59134479100000004 2024-04-22T18:41:28.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741897+0000", 2024-04-22T18:41:28.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1189999999999999e-06 2024-04-22T18:41:28.292 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.292 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3195 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:41:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 66.216225704999999, 2024-04-22T18:41:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69159474200000004, 2024-04-22T18:41:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3195 2024-04-22T18:41:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:41:28.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:41:28.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:41:28.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:28.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050333+0000", 2024-04-22T18:41:28.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0289999999999996e-06 2024-04-22T18:41:28.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050341+0000", 2024-04-22T18:41:28.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0050000000000004e-06 2024-04-22T18:41:28.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050346+0000", 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8329999999999999e-06 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150628+0000", 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100281548 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150672+0000", 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3875999999999998e-05 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741916+0000", 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59124448799999996 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741923+0000", 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7689999999999999e-06 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3196 4.11 4.9f8e4db1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 66.216195585999998, 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69158947400000004, 2024-04-22T18:41:28.297 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.298 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:28.298 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.298 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.298 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.298 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3196 2024-04-22T18:41:28.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.298 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:41:28.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:41:28.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:41:28.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050361+0000", 2024-04-22T18:41:28.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2040000000000002e-06 2024-04-22T18:41:28.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050367+0000", 2024-04-22T18:41:28.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3119999999999999e-06 2024-04-22T18:41:28.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050372+0000", 2024-04-22T18:41:28.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0869999999999997e-06 2024-04-22T18:41:28.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150688+0000", 2024-04-22T18:41:28.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100315381 2024-04-22T18:41:28.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150691+0000", 2024-04-22T18:41:28.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8899999999999999e-06 2024-04-22T18:41:28.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741941+0000", 2024-04-22T18:41:28.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59125044900000001 2024-04-22T18:41:28.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741948+0000", 2024-04-22T18:41:28.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.827e-06 2024-04-22T18:41:28.301 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3504 4.9 4:96737e19:::smithi08937929-4:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.855692410000003, 2024-04-22T18:41:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.951755291, 2024-04-22T18:41:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3504 2024-04-22T18:41:28.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.303 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:28.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:28.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410861+0000", 2024-04-22T18:41:28.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410863+0000", 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.635e-06 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410866+0000", 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8499999999999998e-06 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410868+0000", 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.497e-06 2024-04-22T18:41:28.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615137+0000", 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20426903499999999 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615218+0000", 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0860999999999997e-05 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.362617+0000", 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.74739882700000004 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.305 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4194 4.b 4:d3068ee3:::smithi08937929-29:head [sparse-read 0~1980000,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 25.102433339000001, 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.294164774, 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4194 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:28.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:28.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164120+0000", 2024-04-22T18:41:28.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164125+0000", 2024-04-22T18:41:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2899999999999996e-06 2024-04-22T18:41:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164128+0000", 2024-04-22T18:41:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8700000000000002e-06 2024-04-22T18:41:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164132+0000", 2024-04-22T18:41:28.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.151e-06 2024-04-22T18:41:28.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164188+0000", 2024-04-22T18:41:28.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6912e-05 2024-04-22T18:41:28.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164201+0000", 2024-04-22T18:41:28.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2466000000000001e-05 2024-04-22T18:41:28.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.242879+0000", 2024-04-22T18:41:28.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0786779710000001 2024-04-22T18:41:28.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:28.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243364+0000", 2024-04-22T18:41:28.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000485081 2024-04-22T18:41:28.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456588+0000", 2024-04-22T18:41:28.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21322370900000001 2024-04-22T18:41:28.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456674+0000", 2024-04-22T18:41:28.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6079e-05 2024-04-22T18:41:28.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.458285+0000", 2024-04-22T18:41:28.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001611716 2024-04-22T18:41:28.311 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.311 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.311 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.311 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4195 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:28.311 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:28.311 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 25.102415717, 2024-04-22T18:41:28.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.395681808, 2024-04-22T18:41:28.312 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.312 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.312 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.312 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.312 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.312 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4195 2024-04-22T18:41:28.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.312 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:28.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:28.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:28.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164141+0000", 2024-04-22T18:41:28.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7160000000000001e-06 2024-04-22T18:41:28.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164147+0000", 2024-04-22T18:41:28.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0509999999999996e-06 2024-04-22T18:41:28.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264244+0000", 2024-04-22T18:41:28.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100097063 2024-04-22T18:41:28.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264319+0000", 2024-04-22T18:41:28.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5729000000000006e-05 2024-04-22T18:41:28.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264331+0000", 2024-04-22T18:41:28.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048999999999999e-05 2024-04-22T18:41:28.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243392+0000", 2024-04-22T18:41:28.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97906041099999996 2024-04-22T18:41:28.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:28.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243451+0000", 2024-04-22T18:41:28.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8717e-05 2024-04-22T18:41:28.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558398+0000", 2024-04-22T18:41:28.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31494718500000002 2024-04-22T18:41:28.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558490+0000", 2024-04-22T18:41:28.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2529999999999997e-05 2024-04-22T18:41:28.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559820+0000", 2024-04-22T18:41:28.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013299060000000001 2024-04-22T18:41:28.317 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.317 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.317 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.317 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4196 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:28.317 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:28.317 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 25.00227697, 2024-04-22T18:41:28.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2963829229999999, 2024-04-22T18:41:28.318 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.318 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.318 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.318 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.318 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.318 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4196 2024-04-22T18:41:28.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.318 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:28.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:28.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264275+0000", 2024-04-22T18:41:28.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:28.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264290+0000", 2024-04-22T18:41:28.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5373e-05 2024-04-22T18:41:28.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264300+0000", 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.747e-06 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264314+0000", 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.396e-05 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364450+0000", 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013662500000001 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364459+0000", 2024-04-22T18:41:28.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0710000000000001e-06 2024-04-22T18:41:28.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245690+0000", 2024-04-22T18:41:28.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88123066400000005 2024-04-22T18:41:28.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:28.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245766+0000", 2024-04-22T18:41:28.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5938000000000004e-05 2024-04-22T18:41:28.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559874+0000", 2024-04-22T18:41:28.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31410827499999999 2024-04-22T18:41:28.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559930+0000", 2024-04-22T18:41:28.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5367e-05 2024-04-22T18:41:28.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.560660+0000", 2024-04-22T18:41:28.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073043800000000003 2024-04-22T18:41:28.322 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4182 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.338594125, 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2059652709999999, 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4182 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.323 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927959+0000", 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927962+0000", 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4269999999999998e-06 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927968+0000", 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028037+0000", 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068544 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028078+0000", 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1523999999999997e-05 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028087+0000", 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3140000000000004e-06 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.129951+0000", 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1018649229999999 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.130067+0000", 2024-04-22T18:41:28.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115222 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.133925+0000", 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00385849 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4183 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.238503692999998, 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1063017610000001, 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.327 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.328 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.328 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.328 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4183 2024-04-22T18:41:28.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.328 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:28.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:28.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028049+0000", 2024-04-22T18:41:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028055+0000", 2024-04-22T18:41:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-04-22T18:41:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028064+0000", 2024-04-22T18:41:28.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2830000000000001e-06 2024-04-22T18:41:28.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028071+0000", 2024-04-22T18:41:28.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2740000000000002e-06 2024-04-22T18:41:28.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028126+0000", 2024-04-22T18:41:28.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4771999999999997e-05 2024-04-22T18:41:28.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028134+0000", 2024-04-22T18:41:28.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5599999999999994e-06 2024-04-22T18:41:28.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134109+0000", 2024-04-22T18:41:28.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.105974864 2024-04-22T18:41:28.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134169+0000", 2024-04-22T18:41:28.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0167999999999997e-05 2024-04-22T18:41:28.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134352+0000", 2024-04-22T18:41:28.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018293699999999999 2024-04-22T18:41:28.332 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.332 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.332 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.332 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4190 4.c 4:3227431c:::smithi08937929-35:head [sparse-read 0~1258562,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:28.332 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:28.332 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.238478189999999, 2024-04-22T18:41:28.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1098740030000001, 2024-04-22T18:41:28.332 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.332 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.332 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.332 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.333 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.333 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4190 2024-04-22T18:41:28.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.333 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:28.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:28.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028075+0000", 2024-04-22T18:41:28.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:28.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028080+0000", 2024-04-22T18:41:28.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.561e-06 2024-04-22T18:41:28.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028085+0000", 2024-04-22T18:41:28.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0150000000000003e-06 2024-04-22T18:41:28.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028090+0000", 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.169e-06 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028175+0000", 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4814999999999993e-05 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028181+0000", 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5490000000000003e-06 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134400+0000", 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1062186869999999 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134552+0000", 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015210500000000001 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137950+0000", 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033978210000000001 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.336 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4191 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.238395442000002, 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.110917953, 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4191 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:28.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:28.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028157+0000", 2024-04-22T18:41:28.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:28.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028164+0000", 2024-04-22T18:41:28.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.511e-06 2024-04-22T18:41:28.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028172+0000", 2024-04-22T18:41:28.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0339999999999997e-06 2024-04-22T18:41:28.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028179+0000", 2024-04-22T18:41:28.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8630000000000002e-06 2024-04-22T18:41:28.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028202+0000", 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3635999999999999e-05 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028209+0000", 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4629999999999998e-06 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137970+0000", 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1097613479999999 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.138034+0000", 2024-04-22T18:41:28.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3551000000000002e-05 2024-04-22T18:41:28.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.139077+0000", 2024-04-22T18:41:28.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001042715 2024-04-22T18:41:28.341 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.341 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.341 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.341 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4192 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:28.342 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:28.342 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.238370142000001, 2024-04-22T18:41:28.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.212555542, 2024-04-22T18:41:28.342 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.342 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.342 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.342 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.342 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.342 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4192 2024-04-22T18:41:28.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.342 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:28.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:28.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028183+0000", 2024-04-22T18:41:28.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028187+0000", 2024-04-22T18:41:28.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6119999999999999e-06 2024-04-22T18:41:28.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028192+0000", 2024-04-22T18:41:28.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3489999999999999e-06 2024-04-22T18:41:28.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028197+0000", 2024-04-22T18:41:28.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7489999999999998e-06 2024-04-22T18:41:28.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028228+0000", 2024-04-22T18:41:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1328999999999998e-05 2024-04-22T18:41:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028234+0000", 2024-04-22T18:41:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1129999999999997e-06 2024-04-22T18:41:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239245+0000", 2024-04-22T18:41:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.211011032 2024-04-22T18:41:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239384+0000", 2024-04-22T18:41:28.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013902000000000001 2024-04-22T18:41:28.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.240739+0000", 2024-04-22T18:41:28.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013550389999999999 2024-04-22T18:41:28.346 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.346 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.346 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.346 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4255 4.9 4:905c3da2:::smithi08937929-39:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:28.347 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.705522+0000", 2024-04-22T18:41:28.347 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.561032034, 2024-04-22T18:41:28.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68002623299999998, 2024-04-22T18:41:28.347 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.347 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.347 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.347 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.347 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.347 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4255 2024-04-22T18:41:28.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.347 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705522+0000", 2024-04-22T18:41:28.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705522+0000", 2024-04-22T18:41:28.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705520+0000", 2024-04-22T18:41:28.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:28.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705527+0000", 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2740000000000004e-06 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705533+0000", 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9819999999999996e-06 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705537+0000", 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2869999999999998e-06 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705582+0000", 2024-04-22T18:41:28.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5247000000000002e-05 2024-04-22T18:41:28.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705734+0000", 2024-04-22T18:41:28.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000152029 2024-04-22T18:41:28.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.385548+0000", 2024-04-22T18:41:28.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.67981406899999997 2024-04-22T18:41:28.351 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.351 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.351 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.351 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4692 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:28.351 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:28.351 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.007414263, 2024-04-22T18:41:28.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80332623700000005, 2024-04-22T18:41:28.351 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.351 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.351 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.351 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.351 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.352 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4692 2024-04-22T18:41:28.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.352 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:28.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:28.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259139+0000", 2024-04-22T18:41:28.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:28.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259145+0000", 2024-04-22T18:41:28.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4860000000000002e-06 2024-04-22T18:41:28.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259151+0000", 2024-04-22T18:41:28.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1160000000000004e-06 2024-04-22T18:41:28.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259156+0000", 2024-04-22T18:41:28.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6489999999999999e-06 2024-04-22T18:41:28.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259191+0000", 2024-04-22T18:41:28.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4737999999999998e-05 2024-04-22T18:41:28.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259197+0000", 2024-04-22T18:41:28.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9160000000000004e-06 2024-04-22T18:41:28.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060062+0000", 2024-04-22T18:41:28.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80086472900000005 2024-04-22T18:41:28.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060225+0000", 2024-04-22T18:41:28.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000162771 2024-04-22T18:41:28.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062466+0000", 2024-04-22T18:41:28.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410569999999999 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4693 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.0073919769999999, 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80370012999999996, 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4693 2024-04-22T18:41:28.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259161+0000", 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:28.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259164+0000", 2024-04-22T18:41:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-06 2024-04-22T18:41:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259169+0000", 2024-04-22T18:41:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6260000000000003e-06 2024-04-22T18:41:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259174+0000", 2024-04-22T18:41:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7659999999999998e-06 2024-04-22T18:41:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259231+0000", 2024-04-22T18:41:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6827000000000003e-05 2024-04-22T18:41:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259239+0000", 2024-04-22T18:41:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6889999999999993e-06 2024-04-22T18:41:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062492+0000", 2024-04-22T18:41:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80325246699999997 2024-04-22T18:41:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062573+0000", 2024-04-22T18:41:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1508000000000006e-05 2024-04-22T18:41:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062862+0000", 2024-04-22T18:41:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028893499999999998 2024-04-22T18:41:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4701 4.1b 4:d923883e:::smithi08937929-16:head [set-chunk in=77b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.408330+0000", 2024-04-22T18:41:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.85822399100000002, 2024-04-22T18:41:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.65183720300000003, 2024-04-22T18:41:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4701 2024-04-22T18:41:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408330+0000", 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408330+0000", 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408326+0000", 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999962 2024-04-22T18:41:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408343+0000", 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6731999999999998e-05 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408366+0000", 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3567e-05 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408379+0000", 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2775999999999999e-05 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.557823+0000", 2024-04-22T18:41:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14944426899999999 2024-04-22T18:41:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.557837+0000", 2024-04-22T18:41:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:41:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.858557+0000", 2024-04-22T18:41:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30072007899999997 2024-04-22T18:41:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.858706+0000", 2024-04-22T18:41:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014876000000000001 2024-04-22T18:41:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060052+0000", 2024-04-22T18:41:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.201345943 2024-04-22T18:41:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060167+0000", 2024-04-22T18:41:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115396 2024-04-22T18:41:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.366 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:28.367 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:28.368 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:28.369+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:28.402 INFO:tasks.rados.rados.0.smithi089.stdout:1924: expect (ObjNum 377 snap 0 seq_num 377) 2024-04-22T18:41:28.468 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:28.470+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:28.472 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:28.472 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:28.472 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:28.472 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:28.480 INFO:tasks.rados.rados.0.smithi089.stdout:1923: finishing set_chunk to oid 16 2024-04-22T18:41:28.480 INFO:tasks.rados.rados.0.smithi089.stdout:1923: got expected ENOENT (src dne) 2024-04-22T18:41:28.480 INFO:tasks.rados.rados.0.smithi089.stdout:1920: expect (ObjNum 366 snap 0 seq_num 366) 2024-04-22T18:41:28.481 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:28.569 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:28.571+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:28.611 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:41:28.625 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:28.625 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:28.625 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:28.625 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:28.625 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:28.633 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:28.634 INFO:tasks.rados.rados.0.smithi089.stdout:1887: expect (ObjNum 375 snap 0 seq_num 375) 2024-04-22T18:41:28.669 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:41:28.670 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:28.671+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:28.735 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:28.735 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:28.755 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:28.755 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:28.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:28.755 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:28.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.755 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:41:28.755 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:28.755 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 144.29311604700001, 2024-04-22T18:41:28.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:41:28.756 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.756 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.756 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:28.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:28.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:41:28.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:28.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:41:28.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:41:28.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:41:28.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:41:28.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:41:28.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:41:28.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:41:28.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:41:28.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:41:28.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:41:28.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:41:28.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:41:28.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:41:28.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:41:28.759 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.759 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.760 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.760 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:41:28.760 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:28.760 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 140.416442225, 2024-04-22T18:41:28.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:41:28.760 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.760 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.760 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:41:28.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:41:28.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:41:28.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:41:28.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:41:28.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:41:28.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:41:28.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:41:28.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:41:28.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:41:28.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:41:28.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:41:28.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:41:28.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:41:28.764 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.764 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.764 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.764 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:41:28.764 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:28.764 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 84.211638702000002, 2024-04-22T18:41:28.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:41:28.764 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.765 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.765 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:28.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:28.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:41:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:41:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:41:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:41:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:41:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:41:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:41:28.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3451 4.d 4:bee1184f:::smithi08937929-7:head [read 0~3613505,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.827108375000002, 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20164568099999999, 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.768 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.769 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.769 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.769 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.769 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3451 2024-04-22T18:41:28.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.769 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:28.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:28.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:28.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927183+0000", 2024-04-22T18:41:28.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1110000000000001e-06 2024-04-22T18:41:28.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927188+0000", 2024-04-22T18:41:28.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1569999999999999e-06 2024-04-22T18:41:28.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927192+0000", 2024-04-22T18:41:28.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4710000000000001e-06 2024-04-22T18:41:28.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027382+0000", 2024-04-22T18:41:28.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100189341 2024-04-22T18:41:28.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:41:28.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027387+0000", 2024-04-22T18:41:28.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8400000000000002e-06 2024-04-22T18:41:28.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127638+0000", 2024-04-22T18:41:28.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10025102499999999 2024-04-22T18:41:28.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127740+0000", 2024-04-22T18:41:28.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102749 2024-04-22T18:41:28.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.128825+0000", 2024-04-22T18:41:28.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001084593 2024-04-22T18:41:28.773 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.773 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.773 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.773 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:151 3.0 e114/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 114'213, mlcod=110'210)", 2024-04-22T18:41:28.773 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:28.773 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.236640291999997, 2024-04-22T18:41:28.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20074001299999999, 2024-04-22T18:41:28.773 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.773 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.774 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:28.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:28.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517640+0000", 2024-04-22T18:41:28.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999928 2024-04-22T18:41:28.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517652+0000", 2024-04-22T18:41:28.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1824000000000001e-05 2024-04-22T18:41:28.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517660+0000", 2024-04-22T18:41:28.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1419999999999993e-06 2024-04-22T18:41:28.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617729+0000", 2024-04-22T18:41:28.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068723 2024-04-22T18:41:28.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617770+0000", 2024-04-22T18:41:28.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1391999999999999e-05 2024-04-22T18:41:28.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617793+0000", 2024-04-22T18:41:28.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2807999999999999e-05 2024-04-22T18:41:28.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718361+0000", 2024-04-22T18:41:28.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100567473 2024-04-22T18:41:28.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718387+0000", 2024-04-22T18:41:28.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6826e-05 2024-04-22T18:41:28.777 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.777 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.777 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3862 4.d 4:bee1184f:::smithi08937929-7:head [write 3301257~278667] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.491738022, 2024-04-22T18:41:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301875958, 2024-04-22T18:41:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3862 2024-04-22T18:41:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262548+0000", 2024-04-22T18:41:28.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:28.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262622+0000", 2024-04-22T18:41:28.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3747999999999999e-05 2024-04-22T18:41:28.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262632+0000", 2024-04-22T18:41:28.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0767e-05 2024-04-22T18:41:28.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362705+0000", 2024-04-22T18:41:28.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073157 2024-04-22T18:41:28.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:41:28.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2846999999999997e-05 2024-04-22T18:41:28.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362853+0000", 2024-04-22T18:41:28.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4498000000000002e-05 2024-04-22T18:41:28.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:28.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362992+0000", 2024-04-22T18:41:28.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013879699999999999 2024-04-22T18:41:28.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:28.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.366534+0000", 2024-04-22T18:41:28.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035427969999999999 2024-04-22T18:41:28.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:28.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564353+0000", 2024-04-22T18:41:28.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.197818893 2024-04-22T18:41:28.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564385+0000", 2024-04-22T18:41:28.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1717000000000001e-05 2024-04-22T18:41:28.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564426+0000", 2024-04-22T18:41:28.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0654000000000001e-05 2024-04-22T18:41:28.783 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.783 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.783 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.784 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3863 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3579924] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:28.784 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:28.784 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.391570629999997, 2024-04-22T18:41:28.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20206560900000001, 2024-04-22T18:41:28.784 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.784 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.784 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.784 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.784 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.784 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3863 2024-04-22T18:41:28.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362715+0000", 2024-04-22T18:41:28.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362725+0000", 2024-04-22T18:41:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.006e-05 2024-04-22T18:41:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362732+0000", 2024-04-22T18:41:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6780000000000003e-06 2024-04-22T18:41:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362737+0000", 2024-04-22T18:41:28.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2089999999999996e-06 2024-04-22T18:41:28.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462875+0000", 2024-04-22T18:41:28.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013834100000001 2024-04-22T18:41:28.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462963+0000", 2024-04-22T18:41:28.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7792000000000002e-05 2024-04-22T18:41:28.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:28.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463127+0000", 2024-04-22T18:41:28.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000163688 2024-04-22T18:41:28.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:28.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463692+0000", 2024-04-22T18:41:28.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056466299999999997 2024-04-22T18:41:28.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:28.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564720+0000", 2024-04-22T18:41:28.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10102826199999999 2024-04-22T18:41:28.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564752+0000", 2024-04-22T18:41:28.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1936000000000001e-05 2024-04-22T18:41:28.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564783+0000", 2024-04-22T18:41:28.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0919e-05 2024-04-22T18:41:28.789 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.789 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.789 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.790 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3864 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:28.790 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:28.790 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.391537245999999, 2024-04-22T18:41:28.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20218423599999999, 2024-04-22T18:41:28.790 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.790 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.790 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.790 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.790 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.790 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3864 2024-04-22T18:41:28.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.790 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:28.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:28.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:28.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362753+0000", 2024-04-22T18:41:28.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6989999999999999e-06 2024-04-22T18:41:28.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:41:28.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:41:28.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462871+0000", 2024-04-22T18:41:28.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100113641 2024-04-22T18:41:28.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463385+0000", 2024-04-22T18:41:28.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051307100000000003 2024-04-22T18:41:28.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:28.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463457+0000", 2024-04-22T18:41:28.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2778e-05 2024-04-22T18:41:28.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564814+0000", 2024-04-22T18:41:28.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10135696299999999 2024-04-22T18:41:28.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564868+0000", 2024-04-22T18:41:28.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3872000000000002e-05 2024-04-22T18:41:28.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564935+0000", 2024-04-22T18:41:28.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6545999999999996e-05 2024-04-22T18:41:28.794 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.794 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.794 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.795 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4030 4.d 4:bee1184f:::smithi08937929-7:head [write 615222~472039] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:28.795 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:28.795 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.795088344, 2024-04-22T18:41:28.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109028217, 2024-04-22T18:41:28.795 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.795 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.795 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4030 2024-04-22T18:41:28.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.795 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959177+0000", 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999776 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959331+0000", 2024-04-22T18:41:28.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015334600000000001 2024-04-22T18:41:28.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959358+0000", 2024-04-22T18:41:28.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7220999999999999e-05 2024-04-22T18:41:28.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959371+0000", 2024-04-22T18:41:28.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3489000000000001e-05 2024-04-22T18:41:28.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059523+0000", 2024-04-22T18:41:28.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100152059 2024-04-22T18:41:28.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059588+0000", 2024-04-22T18:41:28.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4493999999999996e-05 2024-04-22T18:41:28.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:28.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059694+0000", 2024-04-22T18:41:28.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010584199999999999 2024-04-22T18:41:28.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:28.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068082+0000", 2024-04-22T18:41:28.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083880589999999998 2024-04-22T18:41:28.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:28.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068138+0000", 2024-04-22T18:41:28.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6196999999999997e-05 2024-04-22T18:41:28.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068185+0000", 2024-04-22T18:41:28.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7318000000000003e-05 2024-04-22T18:41:28.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068228+0000", 2024-04-22T18:41:28.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2376999999999998e-05 2024-04-22T18:41:28.800 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.800 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.800 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.800 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4031 4.d 4:bee1184f:::smithi08937929-7:head [write 1580652~489442] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:28.800 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:28.800 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.789787351999998, 2024-04-22T18:41:28.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20571598299999999, 2024-04-22T18:41:28.801 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.801 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.801 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.801 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.801 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.801 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4031 2024-04-22T18:41:28.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.801 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:28.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:28.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964480+0000", 2024-04-22T18:41:28.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999795 2024-04-22T18:41:28.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964628+0000", 2024-04-22T18:41:28.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148109 2024-04-22T18:41:28.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964651+0000", 2024-04-22T18:41:28.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3215999999999999e-05 2024-04-22T18:41:28.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964664+0000", 2024-04-22T18:41:28.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2673e-05 2024-04-22T18:41:28.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061135+0000", 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096470822999999997 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061228+0000", 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3459000000000001e-05 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061376+0000", 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148128 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:28.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068085+0000", 2024-04-22T18:41:28.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.006708632 2024-04-22T18:41:28.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:28.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170038+0000", 2024-04-22T18:41:28.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101952716 2024-04-22T18:41:28.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170173+0000", 2024-04-22T18:41:28.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013508699999999999 2024-04-22T18:41:28.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170216+0000", 2024-04-22T18:41:28.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3498999999999998e-05 2024-04-22T18:41:28.806 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.806 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.806 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.806 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4032 4.d 4:bee1184f:::smithi08937929-7:head [write 2768826~626587] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:28.806 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:28.806 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.782582245, 2024-04-22T18:41:28.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19941629799999999, 2024-04-22T18:41:28.806 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.806 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.807 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4032 2024-04-22T18:41:28.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.807 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:28.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:28.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971703+0000", 2024-04-22T18:41:28.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:28.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971996+0000", 2024-04-22T18:41:28.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029351300000000002 2024-04-22T18:41:28.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972012+0000", 2024-04-22T18:41:28.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5569e-05 2024-04-22T18:41:28.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972019+0000", 2024-04-22T18:41:28.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7950000000000008e-06 2024-04-22T18:41:28.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.063940+0000", 2024-04-22T18:41:28.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091920162 2024-04-22T18:41:28.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064021+0000", 2024-04-22T18:41:28.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1865000000000003e-05 2024-04-22T18:41:28.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:28.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064145+0000", 2024-04-22T18:41:28.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123893 2024-04-22T18:41:28.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:28.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068317+0000", 2024-04-22T18:41:28.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041720799999999999 2024-04-22T18:41:28.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:28.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171042+0000", 2024-04-22T18:41:28.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102724893 2024-04-22T18:41:28.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171085+0000", 2024-04-22T18:41:28.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2619e-05 2024-04-22T18:41:28.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171122+0000", 2024-04-22T18:41:28.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6838999999999997e-05 2024-04-22T18:41:28.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.812 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.812 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4033 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3815222] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:28.812 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:28.812 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.782258763000002, 2024-04-22T18:41:28.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297586038, 2024-04-22T18:41:28.813 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.813 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.813 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.813 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.813 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.813 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4033 2024-04-22T18:41:28.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.813 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:28.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:28.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972028+0000", 2024-04-22T18:41:28.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972032+0000", 2024-04-22T18:41:28.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8330000000000004e-06 2024-04-22T18:41:28.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972035+0000", 2024-04-22T18:41:28.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:41:28.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972038+0000", 2024-04-22T18:41:28.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9579999999999999e-06 2024-04-22T18:41:28.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067511+0000", 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.095472308000000006 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067607+0000", 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6546999999999996e-05 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067776+0000", 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000169202 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:28.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068470+0000", 2024-04-22T18:41:28.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069403100000000001 2024-04-22T18:41:28.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:28.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269540+0000", 2024-04-22T18:41:28.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20106934000000001 2024-04-22T18:41:28.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269574+0000", 2024-04-22T18:41:28.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4332000000000003e-05 2024-04-22T18:41:28.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269615+0000", 2024-04-22T18:41:28.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0988000000000001e-05 2024-04-22T18:41:28.818 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.818 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.818 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.818 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4034 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:28.818 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:28.818 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.782247126000001, 2024-04-22T18:41:28.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39788621000000002, 2024-04-22T18:41:28.818 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.818 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.818 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.819 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.819 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.819 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4034 2024-04-22T18:41:28.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.819 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:28.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:28.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972040+0000", 2024-04-22T18:41:28.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972042+0000", 2024-04-22T18:41:28.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.519e-06 2024-04-22T18:41:28.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972045+0000", 2024-04-22T18:41:28.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:41:28.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972048+0000", 2024-04-22T18:41:28.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.773e-06 2024-04-22T18:41:28.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068266+0000", 2024-04-22T18:41:28.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096218029999999996 2024-04-22T18:41:28.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:28.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068308+0000", 2024-04-22T18:41:28.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.21e-05 2024-04-22T18:41:28.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369703+0000", 2024-04-22T18:41:28.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301395578 2024-04-22T18:41:28.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369802+0000", 2024-04-22T18:41:28.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8788000000000004e-05 2024-04-22T18:41:28.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369927+0000", 2024-04-22T18:41:28.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012478200000000001 2024-04-22T18:41:28.823 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.823 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.823 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.823 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(4.1 124/123 [PushOp(4:83c955bf:::smithi08937929-28:head, version: 115'157, data_included: [1890685~548185,2699879~469889], data_size: 1018074, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:83c955bf:::smithi08937929-28:head@115'157, size: 3630345, copy_subset: [0~3630345], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3630345, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1890685, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-22T18:41:28.823 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:28.823 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.115679018000002, 2024-04-22T18:41:28.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20293218800000001, 2024-04-22T18:41:28.824 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.824 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.824 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:28.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:28.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638608+0000", 2024-04-22T18:41:28.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:28.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639519+0000", 2024-04-22T18:41:28.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091196199999999999 2024-04-22T18:41:28.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639537+0000", 2024-04-22T18:41:28.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7455000000000001e-05 2024-04-22T18:41:28.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.739639+0000", 2024-04-22T18:41:28.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100101942 2024-04-22T18:41:28.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840271+0000", 2024-04-22T18:41:28.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10063235299999999 2024-04-22T18:41:28.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840281+0000", 2024-04-22T18:41:28.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6509999999999996e-06 2024-04-22T18:41:28.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.841541+0000", 2024-04-22T18:41:28.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012600230000000001 2024-04-22T18:41:28.829 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.829 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.829 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.829 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4205 4.1 4:83c955bf:::smithi08937929-28:head [read 3065344~859648] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:28.829 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:28.830 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.826757177000001, 2024-04-22T18:41:28.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.303433565, 2024-04-22T18:41:28.830 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.830 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:28.830 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:28.830 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:28.830 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:28.830 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4205 2024-04-22T18:41:28.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.830 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:28.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:28.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927528+0000", 2024-04-22T18:41:28.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:28.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927535+0000", 2024-04-22T18:41:28.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1049999999999997e-06 2024-04-22T18:41:28.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927550+0000", 2024-04-22T18:41:28.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4365000000000001e-05 2024-04-22T18:41:28.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927560+0000", 2024-04-22T18:41:28.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0360000000000001e-05 2024-04-22T18:41:28.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028542+0000", 2024-04-22T18:41:28.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100981837 2024-04-22T18:41:28.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:28.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028551+0000", 2024-04-22T18:41:28.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6260000000000001e-06 2024-04-22T18:41:28.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229700+0000", 2024-04-22T18:41:28.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.201149453 2024-04-22T18:41:28.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229861+0000", 2024-04-22T18:41:28.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160605 2024-04-22T18:41:28.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.230964+0000", 2024-04-22T18:41:28.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00110351 2024-04-22T18:41:28.834 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.834 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.834 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.835 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:359 3.0 e133/63 3:351d2474:::ba270d71d9c7ccc8d5c74eded70a7073f0e08c1ce33f7199f7c9db1158888734:head v 133'233, mlcod=132'228)", 2024-04-22T18:41:28.835 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:28.835 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.851962749999998, 2024-04-22T18:41:28.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20122315599999999, 2024-04-22T18:41:28.835 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.835 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.835 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:28.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:28.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902324+0000", 2024-04-22T18:41:28.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:28.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902347+0000", 2024-04-22T18:41:28.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3577000000000001e-05 2024-04-22T18:41:28.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902356+0000", 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8179999999999993e-06 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.002447+0000", 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100090255 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102568+0000", 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100121611 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102592+0000", 2024-04-22T18:41:28.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3515000000000001e-05 2024-04-22T18:41:28.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103514+0000", 2024-04-22T18:41:28.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000922174 2024-04-22T18:41:28.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103548+0000", 2024-04-22T18:41:28.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4316999999999997e-05 2024-04-22T18:41:28.838 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.839 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.839 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.839 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:4289 3.0 e134/63 3:2a3d8f37:::9341e24c4a92f7ec719282d2faa5711f9644dee45a94ba5b4c4c1071ddc0e536:head v 134'235, mlcod=133'233)", 2024-04-22T18:41:28.839 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:28.839 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.907393238000001, 2024-04-22T18:41:28.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10669205499999999, 2024-04-22T18:41:28.839 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.839 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.839 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:28.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:28.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846891+0000", 2024-04-22T18:41:28.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:28.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847884+0000", 2024-04-22T18:41:28.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00099224800000000004 2024-04-22T18:41:28.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847900+0000", 2024-04-22T18:41:28.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6154000000000002e-05 2024-04-22T18:41:28.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.947999+0000", 2024-04-22T18:41:28.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009889199999999 2024-04-22T18:41:28.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948046+0000", 2024-04-22T18:41:28.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7617000000000002e-05 2024-04-22T18:41:28.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948072+0000", 2024-04-22T18:41:28.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5559999999999999e-05 2024-04-22T18:41:28.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953539+0000", 2024-04-22T18:41:28.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054668599999999996 2024-04-22T18:41:28.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953587+0000", 2024-04-22T18:41:28.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7781999999999998e-05 2024-04-22T18:41:28.843 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.843 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.843 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.843 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.7.13:94 3.0 e147/63 3:2812810a:::007b860ed7f2dc72f975d8732f59ce14b4563df44fd56cdf3a173500adaeb557:head v 147'260, mlcod=147'257)", 2024-04-22T18:41:28.843 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:41:28.844 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.3210000319999997, 2024-04-22T18:41:28.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.200869664, 2024-04-22T18:41:28.844 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.844 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.844 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:41:28.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:41:28.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433287+0000", 2024-04-22T18:41:28.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:28.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433295+0000", 2024-04-22T18:41:28.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9929999999999995e-06 2024-04-22T18:41:28.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433302+0000", 2024-04-22T18:41:28.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2740000000000002e-06 2024-04-22T18:41:28.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533368+0000", 2024-04-22T18:41:28.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100066239 2024-04-22T18:41:28.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533415+0000", 2024-04-22T18:41:28.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6669000000000003e-05 2024-04-22T18:41:28.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533442+0000", 2024-04-22T18:41:28.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7143000000000001e-05 2024-04-22T18:41:28.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.634126+0000", 2024-04-22T18:41:28.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10068434900000001 2024-04-22T18:41:28.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.634157+0000", 2024-04-22T18:41:28.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1016000000000003e-05 2024-04-22T18:41:28.847 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.847 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.848 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.848 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:198 3.0 e147/63 3:2be6c676:::11b6849070c44d718872a26a0dc7d7224d7a6a98b28e0fde07a9f66143e32de6:head v 147'266, mlcod=147'257)", 2024-04-22T18:41:28.848 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:41:28.848 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.2202279379999998, 2024-04-22T18:41:28.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106653649, 2024-04-22T18:41:28.848 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.848 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.848 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:41:28.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:41:28.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534059+0000", 2024-04-22T18:41:28.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534082+0000", 2024-04-22T18:41:28.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2291999999999999e-05 2024-04-22T18:41:28.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534086+0000", 2024-04-22T18:41:28.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1019999999999999e-06 2024-04-22T18:41:28.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534088+0000", 2024-04-22T18:41:28.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.525e-06 2024-04-22T18:41:28.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.635447+0000", 2024-04-22T18:41:28.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101358581 2024-04-22T18:41:28.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.635462+0000", 2024-04-22T18:41:28.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5327000000000001e-05 2024-04-22T18:41:28.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.640689+0000", 2024-04-22T18:41:28.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052266329999999996 2024-04-22T18:41:28.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.640713+0000", 2024-04-22T18:41:28.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4630999999999998e-05 2024-04-22T18:41:28.852 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.852 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.852 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.852 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:200 3.0 e147/63 3:19f9e72c:::45a155ef5bb5a968c7feaf289313c887c41e2de64b61f8816d05ad72d02346d9:head v 147'267, mlcod=147'257)", 2024-04-22T18:41:28.852 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:41:28.852 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.2201971809999996, 2024-04-22T18:41:28.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109105273, 2024-04-22T18:41:28.852 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:28.852 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:28.853 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:28.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:28.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:41:28.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:28.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:41:28.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:28.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:28.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534090+0000", 2024-04-22T18:41:28.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:28.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:28.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534228+0000", 2024-04-22T18:41:28.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013768600000000001 2024-04-22T18:41:28.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:28.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534232+0000", 2024-04-22T18:41:28.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4900000000000002e-06 2024-04-22T18:41:28.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:28.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534235+0000", 2024-04-22T18:41:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.61e-06 2024-04-22T18:41:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.641544+0000", 2024-04-22T18:41:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107308717 2024-04-22T18:41:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.641557+0000", 2024-04-22T18:41:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2811e-05 2024-04-22T18:41:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:28.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.643173+0000", 2024-04-22T18:41:28.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00161608 2024-04-22T18:41:28.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:28.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:28.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:28.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.643196+0000", 2024-04-22T18:41:28.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3136e-05 2024-04-22T18:41:28.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.856 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:28.856 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:28.856 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:28.858 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:28.858 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:28.859 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:28.859 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:28.859 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:28.859 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:28.860 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:28.861 INFO:tasks.rados.rados.0.smithi089.stdout:1900: finishing write tid 1 to smithi08937929-50 2024-04-22T18:41:28.862 INFO:tasks.rados.rados.0.smithi089.stdout:1900: finishing write tid 2 to smithi08937929-50 2024-04-22T18:41:28.862 INFO:tasks.rados.rados.0.smithi089.stdout:1926: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2024-04-22T18:41:28.862 INFO:tasks.rados.rados.0.smithi089.stdout:1926: writing smithi08937929-37 from 490870 to 1109669 tid 1 2024-04-22T18:41:28.862 INFO:tasks.rados.rados.0.smithi089.stdout:1926: writing smithi08937929-37 from 1592676 to 2366820 tid 2 2024-04-22T18:41:28.862 INFO:tasks.rados.rados.0.smithi089.stdout:1926: writing smithi08937929-37 from 2793074 to 2890870 tid 3 2024-04-22T18:41:28.862 INFO:tasks.rados.rados.0.smithi089.stdout:1887: done (15 left) 2024-04-22T18:41:28.862 INFO:tasks.rados.rados.0.smithi089.stdout:1911: done (14 left) 2024-04-22T18:41:28.862 INFO:tasks.rados.rados.0.smithi089.stdout:1913: done (13 left) 2024-04-22T18:41:28.862 INFO:tasks.rados.rados.0.smithi089.stdout:1914: done (12 left) 2024-04-22T18:41:28.862 INFO:tasks.rados.rados.0.smithi089.stdout:1915: done (11 left) 2024-04-22T18:41:28.862 INFO:tasks.rados.rados.0.smithi089.stdout:1916: done (10 left) 2024-04-22T18:41:28.862 INFO:tasks.rados.rados.0.smithi089.stdout:1918: done (9 left) 2024-04-22T18:41:28.862 INFO:tasks.rados.rados.0.smithi089.stdout:1919: done (8 left) 2024-04-22T18:41:28.863 INFO:tasks.rados.rados.0.smithi089.stdout:1920: done (7 left) 2024-04-22T18:41:28.863 INFO:tasks.rados.rados.0.smithi089.stdout:1921: done (6 left) 2024-04-22T18:41:28.863 INFO:tasks.rados.rados.0.smithi089.stdout:1922: done (5 left) 2024-04-22T18:41:28.863 INFO:tasks.rados.rados.0.smithi089.stdout:1923: done (4 left) 2024-04-22T18:41:28.863 INFO:tasks.rados.rados.0.smithi089.stdout:1924: done (3 left) 2024-04-22T18:41:28.863 INFO:tasks.rados.rados.0.smithi089.stdout:1925: done (2 left) 2024-04-22T18:41:28.863 INFO:tasks.rados.rados.0.smithi089.stdout:1927: read oid 44 snap -1 2024-04-22T18:41:28.863 INFO:tasks.rados.rados.0.smithi089.stdout:1927: expect (ObjNum 187 snap 0 seq_num 187) 2024-04-22T18:41:28.863 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:28.864 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:28.865 INFO:tasks.rados.rados.0.smithi089.stdout:1928: write (excl) oid 49 current snap is 0 2024-04-22T18:41:28.865 INFO:tasks.rados.rados.0.smithi089.stdout:1900: finishing write tid 3 to smithi08937929-50 2024-04-22T18:41:28.865 INFO:tasks.rados.rados.0.smithi089.stdout:1928: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2024-04-22T18:41:28.866 INFO:tasks.rados.rados.0.smithi089.stdout:1928: writing smithi08937929-49 from 507677 to 918078 tid 1 2024-04-22T18:41:28.866 INFO:tasks.rados.rados.0.smithi089.stdout:1928: writing smithi08937929-49 from 1367511 to 1885313 tid 2 2024-04-22T18:41:28.866 INFO:tasks.rados.rados.0.smithi089.stdout:1928: writing smithi08937929-49 from 2420497 to 2907677 tid 3 2024-04-22T18:41:28.866 INFO:tasks.rados.rados.0.smithi089.stdout:1929: read oid 43 snap -1 2024-04-22T18:41:28.866 INFO:tasks.rados.rados.0.smithi089.stdout:1929: expect (ObjNum 370 snap 0 seq_num 370) 2024-04-22T18:41:28.866 INFO:tasks.rados.rados.0.smithi089.stdout:1930: tier_evict oid 6 2024-04-22T18:41:28.866 INFO:tasks.rados.rados.0.smithi089.stdout:1930: tier_evict oid 6 snap -1 2024-04-22T18:41:28.866 INFO:tasks.rados.rados.0.smithi089.stdout:1900: finishing write tid 4 to smithi08937929-50 2024-04-22T18:41:28.866 INFO:tasks.rados.rados.0.smithi089.stdout:1900: finishing write tid 5 to smithi08937929-50 2024-04-22T18:41:28.866 INFO:tasks.rados.rados.0.smithi089.stdout:1900: oid 50 updating version 0 to 189 2024-04-22T18:41:28.866 INFO:tasks.rados.rados.0.smithi089.stdout:1900: oid 50 version 189 is already newer than 186 2024-04-22T18:41:28.866 INFO:tasks.rados.rados.0.smithi089.stdout:1900: oid 50 version 189 is already newer than 188 2024-04-22T18:41:28.866 INFO:tasks.rados.rados.0.smithi089.stdout:1900: oid 50 version 189 is already newer than 187 2024-04-22T18:41:28.866 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 189 (ObjNum 405 snap 0 seq_num 405) dirty exists 2024-04-22T18:41:28.867 INFO:tasks.rados.rados.0.smithi089.stdout:1900: left oid 50 (ObjNum 405 snap 0 seq_num 405) 2024-04-22T18:41:28.867 INFO:tasks.rados.rados.0.smithi089.stdout:1900: done (5 left) 2024-04-22T18:41:28.867 INFO:tasks.rados.rados.0.smithi089.stdout:1931: read oid 4 snap -1 2024-04-22T18:41:28.867 INFO:tasks.rados.rados.0.smithi089.stdout:1931: expect (ObjNum 361 snap 0 seq_num 361) 2024-04-22T18:41:28.867 INFO:tasks.rados.rados.0.smithi089.stdout:1932: set_chunk oid 27 target oid 2024-04-22T18:41:28.867 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:28.867 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:28.868 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:28.859+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:28.868 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:28.868 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:28.880 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:28.880 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:28.894 INFO:tasks.rados.rados.0.smithi089.stdout:1932: set_chunk oid 27 offset: 773632 length: 1767424 target oid 304e21574303502f8c1c0932da6e2fe7ad9b50f2b8f208238cbe395f2663a9d2 offset: 0 snap -1 2024-04-22T18:41:28.894 INFO:tasks.rados.rados.0.smithi089.stdout:1933: write oid 7 current snap is 0 2024-04-22T18:41:28.894 INFO:tasks.rados.rados.0.smithi089.stdout:1933: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2024-04-22T18:41:28.900 INFO:tasks.rados.rados.0.smithi089.stdout:1933: writing smithi08937929-7 from 524484 to 1010134 tid 1 2024-04-22T18:41:28.907 INFO:tasks.rados.rados.0.smithi089.stdout:1933: writing smithi08937929-7 from 1425993 to 2087453 tid 2 2024-04-22T18:41:28.912 INFO:tasks.rados.rados.0.smithi089.stdout:1933: writing smithi08937929-7 from 2847920 to 2924484 tid 3 2024-04-22T18:41:28.912 INFO:tasks.rados.rados.0.smithi089.stdout:1934: set_chunk oid 9 target oid 2024-04-22T18:41:28.958 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:28.960+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:28.979 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:28.979 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:28.988 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:29.060 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:29.061+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:29.117 INFO:tasks.rados.rados.0.smithi089.stdout:1934: set_chunk oid 9 offset: 683008 length: 928768 target oid 54c554543269af1b5cb0d991174bb8a5a41ea76f3a431da2c624b7469bf44c7b offset: 0 snap -1 2024-04-22T18:41:29.117 INFO:tasks.rados.rados.0.smithi089.stdout:1935: read oid 28 snap -1 2024-04-22T18:41:29.117 INFO:tasks.rados.rados.0.smithi089.stdout:1935: expect (ObjNum 355 snap 0 seq_num 355) 2024-04-22T18:41:29.117 INFO:tasks.rados.rados.0.smithi089.stdout:1936: write (excl) oid 19 current snap is 0 2024-04-22T18:41:29.117 INFO:tasks.rados.rados.0.smithi089.stdout:1930: got (22) Invalid argument 2024-04-22T18:41:29.117 INFO:tasks.rados.rados.0.smithi089.stdout:1931: expect (ObjNum 361 snap 0 seq_num 361) 2024-04-22T18:41:29.132 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:29.132 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:29.141 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:29.161 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:29.162+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:29.249 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:29.249 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:29.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:29.262 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:29.263+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:29.293 INFO:tasks.rados.rados.0.smithi089.stdout:1929: expect (ObjNum 370 snap 0 seq_num 370) 2024-04-22T18:41:29.362 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:29.364+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:29.362 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:29.363 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:29.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:29.464 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:29.465+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:29.477 INFO:tasks.rados.rados.0.smithi089.stdout:1932: finishing set_chunk to oid 27 2024-04-22T18:41:29.477 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 773632 length: 1767424 2024-04-22T18:41:29.477 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 27 v 0 (ObjNum 287 snap 0 seq_num 287) dirty exists 2024-04-22T18:41:29.477 INFO:tasks.rados.rados.0.smithi089.stdout:1934: finishing set_chunk to oid 9 2024-04-22T18:41:29.477 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 9 set_chunk returned error code -95 offset: 683008 length: 928768 2024-04-22T18:41:29.478 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 9 v 0 (ObjNum 403 snap 0 seq_num 403) dirty exists 2024-04-22T18:41:29.478 INFO:tasks.rados.rados.0.smithi089.stdout:1928: finishing write tid 1 to smithi08937929-49 2024-04-22T18:41:29.478 INFO:tasks.rados.rados.0.smithi089.stdout:1933: finishing write tid 1 to smithi08937929-7 2024-04-22T18:41:29.478 INFO:tasks.rados.rados.0.smithi089.stdout:1933: finishing write tid 2 to smithi08937929-7 2024-04-22T18:41:29.478 INFO:tasks.rados.rados.0.smithi089.stdout:1933: finishing write tid 3 to smithi08937929-7 2024-04-22T18:41:29.478 INFO:tasks.rados.rados.0.smithi089.stdout:1935: expect (ObjNum 355 snap 0 seq_num 355) 2024-04-22T18:41:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:29.513 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:29.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:29.566 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:29.568+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:29.624 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:29.633 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:29.665 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:29.666+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:29.688 INFO:tasks.rados.rados.0.smithi089.stdout:1933: finishing write tid 4 to smithi08937929-7 2024-04-22T18:41:29.689 INFO:tasks.rados.rados.0.smithi089.stdout:1933: finishing write tid 5 to smithi08937929-7 2024-04-22T18:41:29.689 INFO:tasks.rados.rados.0.smithi089.stdout:1933: oid 7 updating version 0 to 74 2024-04-22T18:41:29.689 INFO:tasks.rados.rados.0.smithi089.stdout:1933: oid 7 updating version 74 to 77 2024-04-22T18:41:29.689 INFO:tasks.rados.rados.0.smithi089.stdout:1933: oid 7 version 77 is already newer than 76 2024-04-22T18:41:29.689 INFO:tasks.rados.rados.0.smithi089.stdout:1933: oid 7 version 77 is already newer than 75 2024-04-22T18:41:29.689 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 77 (ObjNum 412 snap 0 seq_num 412) dirty exists 2024-04-22T18:41:29.689 INFO:tasks.rados.rados.0.smithi089.stdout:1933: left oid 7 (ObjNum 412 snap 0 seq_num 412) 2024-04-22T18:41:29.689 INFO:tasks.rados.rados.0.smithi089.stdout:1928: finishing write tid 2 to smithi08937929-49 2024-04-22T18:41:29.689 INFO:tasks.rados.rados.0.smithi089.stdout:1928: finishing write tid 3 to smithi08937929-49 2024-04-22T18:41:29.689 INFO:tasks.rados.rados.0.smithi089.stdout:1928: finishing write tid 4 to smithi08937929-49 2024-04-22T18:41:29.689 INFO:tasks.rados.rados.0.smithi089.stdout:1928: finishing write tid 5 to smithi08937929-49 2024-04-22T18:41:29.689 INFO:tasks.rados.rados.0.smithi089.stdout:1928: oid 49 updating version 0 to 147 2024-04-22T18:41:29.689 INFO:tasks.rados.rados.0.smithi089.stdout:1928: oid 49 updating version 147 to 149 2024-04-22T18:41:29.689 INFO:tasks.rados.rados.0.smithi089.stdout:1928: oid 49 version 149 is already newer than 148 2024-04-22T18:41:29.690 INFO:tasks.rados.rados.0.smithi089.stdout:1928: oid 49 version 149 is already newer than 146 2024-04-22T18:41:29.690 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 49 v 149 (ObjNum 411 snap 0 seq_num 411) dirty exists 2024-04-22T18:41:29.690 INFO:tasks.rados.rados.0.smithi089.stdout:1928: left oid 49 (ObjNum 411 snap 0 seq_num 411) 2024-04-22T18:41:29.690 INFO:tasks.rados.rados.0.smithi089.stdout:1936: seq_num 413 ranges {541291=560899,1884475=521471} 2024-04-22T18:41:29.694 INFO:tasks.rados.rados.0.smithi089.stdout:1936: writing smithi08937929-19 from 541291 to 1102190 tid 1 2024-04-22T18:41:29.702 INFO:tasks.rados.rados.0.smithi089.stdout:1936: writing smithi08937929-19 from 1884475 to 2405946 tid 2 2024-04-22T18:41:29.702 INFO:tasks.rados.rados.0.smithi089.stdout:1928: done (10 left) 2024-04-22T18:41:29.702 INFO:tasks.rados.rados.0.smithi089.stdout:1929: done (9 left) 2024-04-22T18:41:29.702 INFO:tasks.rados.rados.0.smithi089.stdout:1930: done (8 left) 2024-04-22T18:41:29.702 INFO:tasks.rados.rados.0.smithi089.stdout:1931: done (7 left) 2024-04-22T18:41:29.702 INFO:tasks.rados.rados.0.smithi089.stdout:1932: done (6 left) 2024-04-22T18:41:29.702 INFO:tasks.rados.rados.0.smithi089.stdout:1933: done (5 left) 2024-04-22T18:41:29.702 INFO:tasks.rados.rados.0.smithi089.stdout:1934: done (4 left) 2024-04-22T18:41:29.702 INFO:tasks.rados.rados.0.smithi089.stdout:1935: done (3 left) 2024-04-22T18:41:29.703 INFO:tasks.rados.rados.0.smithi089.stdout:1937: tier_evict oid 11 2024-04-22T18:41:29.703 INFO:tasks.rados.rados.0.smithi089.stdout:1937: tier_evict oid 11 snap -1 2024-04-22T18:41:29.703 INFO:tasks.rados.rados.0.smithi089.stdout:1938: read oid 28 snap -1 2024-04-22T18:41:29.703 INFO:tasks.rados.rados.0.smithi089.stdout:1938: expect (ObjNum 355 snap 0 seq_num 355) 2024-04-22T18:41:29.703 INFO:tasks.rados.rados.0.smithi089.stdout:1939: read oid 38 snap -1 2024-04-22T18:41:29.703 INFO:tasks.rados.rados.0.smithi089.stdout:1939: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:41:29.703 INFO:tasks.rados.rados.0.smithi089.stdout:1940: read oid 49 snap -1 2024-04-22T18:41:29.703 INFO:tasks.rados.rados.0.smithi089.stdout:1940: expect (ObjNum 411 snap 0 seq_num 411) 2024-04-22T18:41:29.703 INFO:tasks.rados.rados.0.smithi089.stdout:1941: read oid 31 snap -1 2024-04-22T18:41:29.703 INFO:tasks.rados.rados.0.smithi089.stdout:1941: expect deleted 2024-04-22T18:41:29.703 INFO:tasks.rados.rados.0.smithi089.stdout:1942: read oid 21 snap -1 2024-04-22T18:41:29.703 INFO:tasks.rados.rados.0.smithi089.stdout:1942: expect (ObjNum 393 snap 0 seq_num 393) 2024-04-22T18:41:29.703 INFO:tasks.rados.rados.0.smithi089.stdout:1943: read oid 32 snap -1 2024-04-22T18:41:29.703 INFO:tasks.rados.rados.0.smithi089.stdout:1943: expect (ObjNum 371 snap 0 seq_num 371) 2024-04-22T18:41:29.703 INFO:tasks.rados.rados.0.smithi089.stdout:1944: read oid 25 snap -1 2024-04-22T18:41:29.704 INFO:tasks.rados.rados.0.smithi089.stdout:1944: expect (ObjNum 326 snap 0 seq_num 326) 2024-04-22T18:41:29.704 INFO:tasks.rados.rados.0.smithi089.stdout:1945: set_chunk oid 18 target oid 2024-04-22T18:41:29.766 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:29.767+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:29.779 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:29.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:29.867 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:29.868+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:29.892 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:29.900 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:29.968 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:29.969+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:29.997 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:29.999+0000 7fa766d7d640 -1 osd.5 153 handle_osd_map injecting map crc failure 2024-04-22T18:41:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:30.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:30.068 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:30.070+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:30.157 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:30.157 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:30.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:30.169 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:30.171+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:30.244 INFO:tasks.rados.rados.0.smithi089.stdout:1945: set_chunk oid 18 offset: 1472000 length: 1410560 target oid 93622abfc0817c712428832afba48878242609f7563f8044d0a866cd7e3dbe2b offset: 0 snap -1 2024-04-22T18:41:30.244 INFO:tasks.rados.rados.0.smithi089.stdout:1946: read oid 48 snap -1 2024-04-22T18:41:30.244 INFO:tasks.rados.rados.0.smithi089.stdout:1946: expect (ObjNum 351 snap 0 seq_num 351) 2024-04-22T18:41:30.244 INFO:tasks.rados.rados.0.smithi089.stdout:1947: tier_flush oid 50 2024-04-22T18:41:30.245 INFO:tasks.rados.rados.0.smithi089.stdout:1947: tier_flush oid 50 snap -1 2024-04-22T18:41:30.245 INFO:tasks.rados.rados.0.smithi089.stdout:1948: set_chunk oid 35 target oid 2024-04-22T18:41:30.245 INFO:tasks.rados.rados.0.smithi089.stdout:1948: set_chunk oid 35 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:41:30.245 INFO:tasks.rados.rados.0.smithi089.stdout:1949: write (excl) oid 3 current snap is 0 2024-04-22T18:41:30.245 INFO:tasks.rados.rados.0.smithi089.stdout:1937: got (22) Invalid argument 2024-04-22T18:41:30.245 INFO:tasks.rados.rados.0.smithi089.stdout:1940: expect (ObjNum 411 snap 0 seq_num 411) 2024-04-22T18:41:30.270 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:30.271+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:30.279 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:30.371 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:30.372+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:30.383 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:30.383 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:30.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:30.404 INFO:tasks.rados.rados.0.smithi089.stdout:1939: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:41:30.472 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:30.474+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:30.495 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:30.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:30.530 INFO:tasks.rados.rados.0.smithi089.stdout:1943: expect (ObjNum 371 snap 0 seq_num 371) 2024-04-22T18:41:30.588 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:30.590+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:30.601 INFO:tasks.rados.rados.0.smithi089.stdout:1938: expect (ObjNum 355 snap 0 seq_num 355) 2024-04-22T18:41:30.646 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:30.646 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:30.656 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:30.669 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:30.671+0000 7f1e2dd22640 -1 osd.2 154 handle_osd_map injecting map crc failure 2024-04-22T18:41:30.686 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:41:30.697 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:30.699+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:30.774 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:30.776+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:30.798 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:30.798 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:30.807 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:30.822 INFO:tasks.rados.rados.0.smithi089.stdout:1944: expect (ObjNum 326 snap 0 seq_num 326) 2024-04-22T18:41:30.875 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:30.876+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:30.893 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:41:30.913 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:30.913 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:30.923 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:31.006 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:31.008+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:31.017 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:31.017 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:31.027 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 7, 2, 4, 0] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:41:31.027 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:41:31.027 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-04-22T18:41:31.027 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:41:31.035 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:31.053 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:31.054 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:31.075 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:31.076 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:31.076 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:31.076 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:31.078+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:31.085 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:31.086 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:31.086 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:31.104 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:31.104 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:31.108 INFO:tasks.rados.rados.0.smithi089.stdout:1936: finishing write tid 1 to smithi08937929-19 2024-04-22T18:41:31.109 INFO:tasks.rados.rados.0.smithi089.stdout:1942: expect (ObjNum 393 snap 0 seq_num 393) 2024-04-22T18:41:31.136 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:31.136 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:31.157 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:31.157 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:31.177 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:31.178+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:31.184 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:31.184 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:31.196 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:31.196 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:31.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:31.196 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:31.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.196 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:31.196 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:31.196 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 156.74011484600001, 2024-04-22T18:41:31.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:41:31.197 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.197 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:31.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.197 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:41:31.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.197 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:31.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:41:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:41:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:41:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:41:31.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:41:31.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:41:31.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:41:31.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:41:31.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:41:31.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:41:31.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:41:31.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:31.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:41:31.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:41:31.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:41:31.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:41:31.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:31.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:41:31.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:41:31.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:31.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:41:31.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:41:31.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:41:31.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:41:31.202 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.202 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.202 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.202 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:31.202 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:31.202 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 148.04705759800001, 2024-04-22T18:41:31.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:41:31.202 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.202 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:31.202 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.203 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.203 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.203 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:41:31.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.203 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:31.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:31.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:41:31.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:41:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:41:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:41:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:41:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:41:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:41:31.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:41:31.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:41:31.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:41:31.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:41:31.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:31.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:41:31.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:41:31.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:41:31.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:41:31.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:31.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:41:31.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:41:31.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:31.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:41:31.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:41:31.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:41:31.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:41:31.208 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.208 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.208 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.209 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:31.209 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:31.209 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 148.047039122, 2024-04-22T18:41:31.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:41:31.209 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.209 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.209 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.209 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.209 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.209 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:41:31.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.209 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:31.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:31.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:31.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:41:31.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:41:31.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:41:31.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:41:31.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:41:31.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:41:31.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:41:31.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:41:31.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:31.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:41:31.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:41:31.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:41:31.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:41:31.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:41:31.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:41:31.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:41:31.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:41:31.213 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.214 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.214 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.214 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:41:31.214 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:31.214 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 120.28399081000001, 2024-04-22T18:41:31.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:41:31.214 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.214 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:31.214 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.214 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.215 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.215 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:41:31.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.215 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:31.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:31.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:41:31.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:41:31.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:41:31.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:41:31.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:41:31.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:41:31.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:31.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:41:31.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:41:31.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:31.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:41:31.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:41:31.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:41:31.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:41:31.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:31.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:41:31.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:41:31.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:41:31.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:41:31.220 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.220 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.220 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.221 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1919 4.3 4:ca46fde9:::smithi08937929-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:31.221 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:31.221 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 114.75483148399999, 2024-04-22T18:41:31.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43404953899999998, 2024-04-22T18:41:31.222 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.222 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.222 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.222 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.222 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.222 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1919 2024-04-22T18:41:31.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.223 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:31.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:31.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440640+0000", 2024-04-22T18:41:31.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440648+0000", 2024-04-22T18:41:31.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5499999999999995e-06 2024-04-22T18:41:31.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440654+0000", 2024-04-22T18:41:31.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.122e-06 2024-04-22T18:41:31.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.440658+0000", 2024-04-22T18:41:31.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9870000000000001e-06 2024-04-22T18:41:31.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534847+0000", 2024-04-22T18:41:31.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094188487000000001 2024-04-22T18:41:31.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:31.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.534917+0000", 2024-04-22T18:41:31.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0518000000000005e-05 2024-04-22T18:41:31.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874555+0000", 2024-04-22T18:41:31.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.33963728199999998 2024-04-22T18:41:31.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874615+0000", 2024-04-22T18:41:31.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0579999999999999e-05 2024-04-22T18:41:31.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.874689+0000", 2024-04-22T18:41:31.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4369000000000006e-05 2024-04-22T18:41:31.231 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.231 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.231 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.231 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:230 3.1 3:b31cde68:::30efb24c47dbe1df13c2112d242f0fcfa3ee5b97a224de4b875076c5531dfa7e:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:31.231 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:41:31.231 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 114.57712408899999, 2024-04-22T18:41:31.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40970996100000001, 2024-04-22T18:41:31.231 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.232 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:31.232 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:41:31.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618347+0000", 2024-04-22T18:41:31.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618344+0000", 2024-04-22T18:41:31.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:31.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618358+0000", 2024-04-22T18:41:31.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4280999999999999e-05 2024-04-22T18:41:31.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618376+0000", 2024-04-22T18:41:31.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8423e-05 2024-04-22T18:41:31.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618403+0000", 2024-04-22T18:41:31.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6276000000000001e-05 2024-04-22T18:41:31.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618481+0000", 2024-04-22T18:41:31.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8289999999999998e-05 2024-04-22T18:41:31.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618553+0000", 2024-04-22T18:41:31.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1864000000000002e-05 2024-04-22T18:41:31.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618666+0000", 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000112953 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619563+0000", 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00089667499999999999 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028000+0000", 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40843744599999998 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:31.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028032+0000", 2024-04-22T18:41:31.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1798999999999998e-05 2024-04-22T18:41:31.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028057+0000", 2024-04-22T18:41:31.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5499000000000001e-05 2024-04-22T18:41:31.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.237 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.237 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:231 3.1 3:b884ac55:::fabae089d47e7339111bd218d0548c480d5b28f54ac2750e85c6e19b6c8f5cb0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:31.237 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:41:31.237 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 114.577055817, 2024-04-22T18:41:31.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40978218900000002, 2024-04-22T18:41:31.238 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.238 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:31.238 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:41:31.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618416+0000", 2024-04-22T18:41:31.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618415+0000", 2024-04-22T18:41:31.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618418+0000", 2024-04-22T18:41:31.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.134e-06 2024-04-22T18:41:31.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618422+0000", 2024-04-22T18:41:31.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7469999999999999e-06 2024-04-22T18:41:31.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618425+0000", 2024-04-22T18:41:31.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2339999999999999e-06 2024-04-22T18:41:31.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618919+0000", 2024-04-22T18:41:31.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049379100000000004 2024-04-22T18:41:31.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618974+0000", 2024-04-22T18:41:31.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4749e-05 2024-04-22T18:41:31.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:31.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619058+0000", 2024-04-22T18:41:31.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4097000000000005e-05 2024-04-22T18:41:31.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619570+0000", 2024-04-22T18:41:31.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051214499999999998 2024-04-22T18:41:31.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:31.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028110+0000", 2024-04-22T18:41:31.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.408540558 2024-04-22T18:41:31.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:31.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028159+0000", 2024-04-22T18:41:31.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8952999999999997e-05 2024-04-22T18:41:31.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028198+0000", 2024-04-22T18:41:31.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:41:31.243 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.243 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.243 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.243 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:233 3.1 3:b6f79c71:::5d0074a2bc321b12696ca826163e7a043d059a87630dfdb5ca117dc51c80a261:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:31.243 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:31.243 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 114.57688797599999, 2024-04-22T18:41:31.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40967667400000002, 2024-04-22T18:41:31.243 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.243 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618583+0000", 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618586+0000", 2024-04-22T18:41:31.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1949999999999999e-06 2024-04-22T18:41:31.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618591+0000", 2024-04-22T18:41:31.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8380000000000001e-06 2024-04-22T18:41:31.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.618594+0000", 2024-04-22T18:41:31.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0280000000000001e-06 2024-04-22T18:41:31.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619226+0000", 2024-04-22T18:41:31.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063226899999999997 2024-04-22T18:41:31.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619272+0000", 2024-04-22T18:41:31.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5358999999999997e-05 2024-04-22T18:41:31.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:31.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.619352+0000", 2024-04-22T18:41:31.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9862000000000002e-05 2024-04-22T18:41:31.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:36.622265+0000", 2024-04-22T18:41:31.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029137 2024-04-22T18:41:31.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:31.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028222+0000", 2024-04-22T18:41:31.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40595716999999998 2024-04-22T18:41:31.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:31.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028244+0000", 2024-04-22T18:41:31.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1732000000000001e-05 2024-04-22T18:41:31.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:37.028260+0000", 2024-04-22T18:41:31.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6008e-05 2024-04-22T18:41:31.249 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.249 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.249 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.249 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:31.249 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:31.249 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 110.61156250099999, 2024-04-22T18:41:31.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:41:31.249 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.250 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:31.250 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:31.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:31.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:41:31.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:31.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:41:31.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:41:31.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:41:31.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:41:31.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:41:31.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:41:31.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:41:31.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:41:31.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:41:31.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:41:31.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:31.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:41:31.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:41:31.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:41:31.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:41:31.254 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.254 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.254 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.255 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:250 3.1 3:bc71164b:::a65a637424153024af195c308edceda844b95976bd407018840eb800f1da78e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:31.255 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:41:31.255 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 110.511410314, 2024-04-22T18:41:31.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40941283699999997, 2024-04-22T18:41:31.255 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.255 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:31.255 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:41:31.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684061+0000", 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684058+0000", 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684074+0000", 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6337999999999998e-05 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684087+0000", 2024-04-22T18:41:31.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3069e-05 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684096+0000", 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6969999999999999e-06 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684568+0000", 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000472583 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684629+0000", 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0757999999999997e-05 2024-04-22T18:41:31.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684785+0000", 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000155718 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685541+0000", 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075614300000000003 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093424+0000", 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40788337200000002 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093453+0000", 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8271000000000001e-05 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093474+0000", 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1302e-05 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:31.259 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 100.802545794, 2024-04-22T18:41:31.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:41:31.260 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.260 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.260 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.260 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.260 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.260 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:41:31.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.260 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:31.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:31.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:41:31.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:41:31.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:41:31.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:41:31.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:41:31.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:41:31.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:41:31.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:41:31.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:41:31.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:31.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:41:31.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:41:31.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:41:31.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:41:31.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:31.264 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 83.906507422000004, 2024-04-22T18:41:31.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:41:31.265 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.265 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.265 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.265 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.265 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.265 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:41:31.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.265 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:31.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:31.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:31.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:41:31.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:41:31.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:41:31.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:41:31.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:41:31.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:41:31.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:41:31.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:41:31.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:41:31.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:41:31.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:41:31.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:41:31.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.268 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.269 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.269 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3330 4.b 4:d057aa9a:::smithi08937929-49:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-04-22T18:41:31.269 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:41:31.269 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 64.171429136, 2024-04-22T18:41:31.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43151673600000001, 2024-04-22T18:41:31.269 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.269 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.269 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.269 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.269 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.269 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3330 2024-04-22T18:41:31.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024042+0000", 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024041+0000", 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:31.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024045+0000", 2024-04-22T18:41:31.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3950000000000001e-06 2024-04-22T18:41:31.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024051+0000", 2024-04-22T18:41:31.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0260000000000004e-06 2024-04-22T18:41:31.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.024056+0000", 2024-04-22T18:41:31.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4489999999999998e-06 2024-04-22T18:41:31.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028383+0000", 2024-04-22T18:41:31.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004326557 2024-04-22T18:41:31.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.028471+0000", 2024-04-22T18:41:31.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8387999999999995e-05 2024-04-22T18:41:31.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:27.455559+0000", 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.42708770699999998 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3377 4.6 4:6700eab5:::smithi08937929-32:head [read 1812992~307712] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.759103779999997, 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.52001745799999999, 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.273 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.274 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3377 2024-04-22T18:41:31.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.274 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:31.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:31.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436364+0000", 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436376+0000", 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2121e-05 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436394+0000", 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8094e-05 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436405+0000", 2024-04-22T18:41:31.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0189e-05 2024-04-22T18:41:31.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436471+0000", 2024-04-22T18:41:31.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6817999999999996e-05 2024-04-22T18:41:31.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436481+0000", 2024-04-22T18:41:31.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2720000000000003e-06 2024-04-22T18:41:31.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956084+0000", 2024-04-22T18:41:31.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51960320800000004 2024-04-22T18:41:31.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956232+0000", 2024-04-22T18:41:31.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014778600000000001 2024-04-22T18:41:31.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956385+0000", 2024-04-22T18:41:31.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153394 2024-04-22T18:41:31.277 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.278 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.278 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.278 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:131 3.1 3:a0a19997:::ee8ca23a7050e771c77dff761e6ea5a11ed57796f4e8a17cbb5804a7fdc937b9:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:31.278 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:31.278 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.866234184, 2024-04-22T18:41:31.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51449564999999997, 2024-04-22T18:41:31.278 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.278 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:31.278 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:31.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:31.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329235+0000", 2024-04-22T18:41:31.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:31.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329244+0000", 2024-04-22T18:41:31.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6819999999999992e-06 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329259+0000", 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698000000000001e-05 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329265+0000", 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1510000000000001e-06 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329311+0000", 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6096e-05 2024-04-22T18:41:31.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:31.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.332269+0000", 2024-04-22T18:41:31.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029581910000000002 2024-04-22T18:41:31.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630603+0000", 2024-04-22T18:41:31.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29833418 2024-04-22T18:41:31.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630685+0000", 2024-04-22T18:41:31.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1217999999999994e-05 2024-04-22T18:41:31.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630813+0000", 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128706 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.631447+0000", 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063427000000000004 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843586+0000", 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21213873 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843623+0000", 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6848999999999999e-05 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843733+0000", 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010982200000000001 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3441 4.b 4:d3068ee3:::smithi08937929-29:head [write 558065~619869] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.277758122999998, 2024-04-22T18:41:31.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50272020399999995, 2024-04-22T18:41:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3441 2024-04-22T18:41:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:31.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917711+0000", 2024-04-22T18:41:31.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:31.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917838+0000", 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012734899999999999 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917854+0000", 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6130999999999999e-05 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917863+0000", 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5969999999999992e-06 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992845+0000", 2024-04-22T18:41:31.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.074982193000000003 2024-04-22T18:41:31.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992919+0000", 2024-04-22T18:41:31.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:41:31.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:31.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.993019+0000", 2024-04-22T18:41:31.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9718999999999999e-05 2024-04-22T18:41:31.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095554+0000", 2024-04-22T18:41:31.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102535193 2024-04-22T18:41:31.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:31.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420183+0000", 2024-04-22T18:41:31.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32462853600000002 2024-04-22T18:41:31.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:31.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420401+0000", 2024-04-22T18:41:31.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021881400000000001 2024-04-22T18:41:31.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420433+0000", 2024-04-22T18:41:31.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079000000000002e-05 2024-04-22T18:41:31.290 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.290 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.290 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.290 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3442 4.b 4:d3068ee3:::smithi08937929-29:head [write 1579835~486765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:31.290 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:31.290 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.273071412, 2024-04-22T18:41:31.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49822799099999998, 2024-04-22T18:41:31.290 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.290 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:31.290 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.290 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.291 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.291 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3442 2024-04-22T18:41:31.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.291 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:31.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:31.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922399+0000", 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922502+0000", 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102784 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922513+0000", 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1051e-05 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922518+0000", 2024-04-22T18:41:31.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8529999999999997e-06 2024-04-22T18:41:31.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093158+0000", 2024-04-22T18:41:31.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.17063958800000001 2024-04-22T18:41:31.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093230+0000", 2024-04-22T18:41:31.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1803000000000004e-05 2024-04-22T18:41:31.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:31.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093326+0000", 2024-04-22T18:41:31.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6454999999999997e-05 2024-04-22T18:41:31.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095557+0000", 2024-04-22T18:41:31.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022306399999999999 2024-04-22T18:41:31.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:31.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420484+0000", 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32492717700000001 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420606+0000", 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012225800000000001 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420628+0000", 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1596000000000001e-05 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3443 4.b 4:d3068ee3:::smithi08937929-29:head [write 2468371~489694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.268469365000001, 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49372487199999998, 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3443 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927001+0000", 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927117+0000", 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011637 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927128+0000", 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0696000000000001e-05 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927134+0000", 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8640000000000001e-06 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094288+0000", 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16715412900000001 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094359+0000", 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1639999999999998e-05 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094451+0000", 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1252e-05 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095560+0000", 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011095530000000001 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420661+0000", 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32510105500000003 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420710+0000", 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9018999999999999e-05 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420727+0000", 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6722e-05 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.300 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3444 4.b 4:d3068ee3:::smithi08937929-29:head [setxattr _header (56) in=7b,truncate 2958065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.268333648000002, 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49366935299999998, 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3444 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:31.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927137+0000", 2024-04-22T18:41:31.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927141+0000", 2024-04-22T18:41:31.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3270000000000001e-06 2024-04-22T18:41:31.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927144+0000", 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9730000000000002e-06 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927147+0000", 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.959e-06 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095118+0000", 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16797115500000001 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095169+0000", 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1316000000000001e-05 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095260+0000", 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1184e-05 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095999+0000", 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073868200000000005 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:31.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420757+0000", 2024-04-22T18:41:31.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32475783499999999 2024-04-22T18:41:31.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:31.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420781+0000", 2024-04-22T18:41:31.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4022000000000001e-05 2024-04-22T18:41:31.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420807+0000", 2024-04-22T18:41:31.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6366000000000001e-05 2024-04-22T18:41:31.305 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.305 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.306 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.306 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3445 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:31.306 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:31.306 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.268322554000001, 2024-04-22T18:41:31.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.493780049, 2024-04-22T18:41:31.306 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.306 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.306 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.306 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.306 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.306 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3445 2024-04-22T18:41:31.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.306 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:31.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:31.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927150+0000", 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.542e-06 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927153+0000", 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0520000000000002e-06 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027218+0000", 2024-04-22T18:41:31.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100064604 2024-04-22T18:41:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095501+0000", 2024-04-22T18:41:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.068283431000000006 2024-04-22T18:41:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095545+0000", 2024-04-22T18:41:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4295999999999997e-05 2024-04-22T18:41:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420833+0000", 2024-04-22T18:41:31.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32528718400000001 2024-04-22T18:41:31.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420877+0000", 2024-04-22T18:41:31.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4403999999999997e-05 2024-04-22T18:41:31.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420929+0000", 2024-04-22T18:41:31.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1848999999999998e-05 2024-04-22T18:41:31.310 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.310 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.310 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.310 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.311 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.311 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:31.311 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:31.313 INFO:tasks.rados.rados.0.smithi089.stdout:1936: finishing write tid 2 to smithi08937929-19 2024-04-22T18:41:31.313 INFO:tasks.rados.rados.0.smithi089.stdout:1936: finishing write tid 3 to smithi08937929-19 2024-04-22T18:41:31.313 INFO:tasks.rados.rados.0.smithi089.stdout:1949: seq_num 414 ranges {558098=636148,1826604=665129} 2024-04-22T18:41:31.313 INFO:tasks.rados.rados.0.smithi089.stdout:1949: writing smithi08937929-3 from 558098 to 1194246 tid 1 2024-04-22T18:41:31.313 INFO:tasks.rados.rados.0.smithi089.stdout:1949: writing smithi08937929-3 from 1826604 to 2491733 tid 2 2024-04-22T18:41:31.313 INFO:tasks.rados.rados.0.smithi089.stdout:1937: done (15 left) 2024-04-22T18:41:31.313 INFO:tasks.rados.rados.0.smithi089.stdout:1938: done (14 left) 2024-04-22T18:41:31.313 INFO:tasks.rados.rados.0.smithi089.stdout:1939: done (13 left) 2024-04-22T18:41:31.313 INFO:tasks.rados.rados.0.smithi089.stdout:1940: done (12 left) 2024-04-22T18:41:31.313 INFO:tasks.rados.rados.0.smithi089.stdout:1941: done (11 left) 2024-04-22T18:41:31.313 INFO:tasks.rados.rados.0.smithi089.stdout:1942: done (10 left) 2024-04-22T18:41:31.313 INFO:tasks.rados.rados.0.smithi089.stdout:1943: done (9 left) 2024-04-22T18:41:31.313 INFO:tasks.rados.rados.0.smithi089.stdout:1944: done (8 left) 2024-04-22T18:41:31.313 INFO:tasks.rados.rados.0.smithi089.stdout:1950: write (excl) oid 45 current snap is 0 2024-04-22T18:41:31.314 INFO:tasks.rados.rados.0.smithi089.stdout:1950: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2024-04-22T18:41:31.314 INFO:tasks.rados.rados.0.smithi089.stdout:1950: writing smithi08937929-45 from 574905 to 1286302 tid 1 2024-04-22T18:41:31.314 INFO:tasks.rados.rados.0.smithi089.stdout:1950: writing smithi08937929-45 from 1885086 to 2410226 tid 2 2024-04-22T18:41:31.314 INFO:tasks.rados.rados.0.smithi089.stdout:1950: writing smithi08937929-45 from 2930189 to 2974905 tid 3 2024-04-22T18:41:31.314 INFO:tasks.rados.rados.0.smithi089.stdout:1951: set_chunk oid 30 target oid 2024-04-22T18:41:31.327 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:31.328+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:31.359 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:41:31.360 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:41:31.372 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-04-22T18:41:30.991967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":207107252,"num_objects":165,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":3814,"num_read_kb":3432747,"num_write":2227,"num_write_kb":526048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":723,"num_bytes_recovered":292117327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":25,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2548,"ondisk_log_size":2548,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":552156,"kb_used_data":346076,"kb_used_omap":9,"kb_used_meta":198902,"kb_avail":565678884,"statfs":{"total":579820584960,"available":579255177216,"internally_reserved":0,"allocated":354381824,"data_stored":361091219,"data_compressed":12432821,"data_compressed_allocated":13262848,"data_compressed_original":26714112,"omap_allocated":9557,"internal_metadata":203676331},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,0,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8693180,"num_objects":-11,"num_object_clones":0,"num_object_copies":-22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10,"num_whiteouts":0,"num_read":48,"num_read_kb":26288,"num_write":57,"num_write_kb":9817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5557305,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000769"},"pg_stats":[{"pgid":"4.1f","version":"144'78","reported_seq":335,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.589958+0000","last_change":"2024-04-22T18:41:11.171942+0000","last_active":"2024-04-22T18:41:26.589958+0000","last_peered":"2024-04-22T18:41:26.589958+0000","last_clean":"2024-04-22T18:41:26.589958+0000","last_became_active":"2024-04-22T18:41:11.171671+0000","last_became_peered":"2024-04-22T18:41:11.171671+0000","last_unstale":"2024-04-22T18:41:26.589958+0000","last_undegraded":"2024-04-22T18:41:26.589958+0000","last_fullsized":"2024-04-22T18:41:26.589958+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5025392,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":124568,"num_write":42,"num_write_kb":11200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"147'53","reported_seq":60,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:25.000315+0000","last_change":"2024-04-22T18:41:11.253830+0000","last_active":"2024-04-22T18:41:25.000315+0000","last_peered":"2024-04-22T18:41:25.000315+0000","last_clean":"2024-04-22T18:41:25.000315+0000","last_became_active":"2024-04-22T18:41:11.253471+0000","last_became_peered":"2024-04-22T18:41:11.253471+0000","last_unstale":"2024-04-22T18:41:25.000315+0000","last_undegraded":"2024-04-22T18:41:25.000315+0000","last_fullsized":"2024-04-22T18:41:25.000315+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'23","last_scrub_stamp":"2024-04-22T18:40:20.587606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.587606+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10225366599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1840449,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":93930,"num_write":31,"num_write_kb":9939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5574713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"149'73","reported_seq":66,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:25.780529+0000","last_change":"2024-04-22T18:41:11.355276+0000","last_active":"2024-04-22T18:41:25.780529+0000","last_peered":"2024-04-22T18:41:25.780529+0000","last_clean":"2024-04-22T18:41:25.780529+0000","last_became_active":"2024-04-22T18:41:11.354958+0000","last_became_peered":"2024-04-22T18:41:11.354958+0000","last_unstale":"2024-04-22T18:41:25.780529+0000","last_undegraded":"2024-04-22T18:41:25.780529+0000","last_fullsized":"2024-04-22T18:41:25.780529+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"96'46","last_scrub_stamp":"2024-04-22T18:40:34.755106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:34.755106+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40192567099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1386677,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":123,"num_read_kb":129496,"num_write":43,"num_write_kb":12765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8192627,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"132'123","reported_seq":470,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:24.999920+0000","last_change":"2024-04-22T18:41:11.469596+0000","last_active":"2024-04-22T18:41:24.999920+0000","last_peered":"2024-04-22T18:41:24.999920+0000","last_clean":"2024-04-22T18:41:24.999920+0000","last_became_active":"2024-04-22T18:41:11.469282+0000","last_became_peered":"2024-04-22T18:41:11.469282+0000","last_unstale":"2024-04-22T18:41:24.999920+0000","last_undegraded":"2024-04-22T18:41:24.999920+0000","last_fullsized":"2024-04-22T18:41:24.999920+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"121'119","last_scrub_stamp":"2024-04-22T18:40:55.573199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.573199+0000","objects_scrubbed":4,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32976813900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4521133,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":219,"num_read_kb":257792,"num_write":64,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3786236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"unknown","last_fresh":"2024-04-22T18:41:26.009981+0000","last_change":"2024-04-22T18:41:26.481301+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T18:41:26.009981+0000","last_undegraded":"2024-04-22T18:41:26.009981+0000","last_fullsized":"2024-04-22T18:41:26.009981+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'109","last_scrub_stamp":"2024-04-22T18:40:17.658363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:17.658363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:33.907212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"146'112","reported_seq":419,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.011197+0000","last_change":"2024-04-22T18:41:10.956373+0000","last_active":"2024-04-22T18:41:26.011197+0000","last_peered":"2024-04-22T18:41:26.011197+0000","last_clean":"2024-04-22T18:41:26.011197+0000","last_became_active":"2024-04-22T18:41:10.956174+0000","last_became_peered":"2024-04-22T18:41:10.956174+0000","last_unstale":"2024-04-22T18:41:26.011197+0000","last_undegraded":"2024-04-22T18:41:26.011197+0000","last_fullsized":"2024-04-22T18:41:26.011197+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2991663,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":162,"num_read_kb":197185,"num_write":57,"num_write_kb":14924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4554596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"105'114","reported_seq":543,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:25.633344+0000","last_change":"2024-04-22T18:41:18.958159+0000","last_active":"2024-04-22T18:41:25.633344+0000","last_peered":"2024-04-22T18:41:25.633344+0000","last_clean":"2024-04-22T18:41:25.633344+0000","last_became_active":"2024-04-22T18:41:18.957008+0000","last_became_peered":"2024-04-22T18:41:18.957008+0000","last_unstale":"2024-04-22T18:41:25.633344+0000","last_undegraded":"2024-04-22T18:41:25.633344+0000","last_fullsized":"2024-04-22T18:41:25.633344+0000","mapping_epoch":142,"log_start":"84'88","ondisk_log_start":"84'88","created":110,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":26,"log_dups_size":106,"ondisk_log_size":26,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6680865,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":136590,"num_write":19,"num_write_kb":5735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9095252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"148'71","reported_seq":358,"reported_epoch":149,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:41:25.000188+0000","last_change":"2024-04-22T18:41:05.348597+0000","last_active":"2024-04-22T18:41:25.000188+0000","last_peered":"2024-04-22T18:41:25.000188+0000","last_clean":"2024-04-22T18:41:01.854833+0000","last_became_active":"2024-04-22T18:41:05.127081+0000","last_became_peered":"2024-04-22T18:41:05.127081+0000","last_unstale":"2024-04-22T18:41:25.000188+0000","last_undegraded":"2024-04-22T18:41:25.000188+0000","last_fullsized":"2024-04-22T18:41:25.000188+0000","mapping_epoch":130,"log_start":"79'52","ondisk_log_start":"79'52","created":110,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":61,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":857223,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":121,"num_read_kb":168058,"num_write":60,"num_write_kb":17002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7945966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,3],"avail_no_missing":["7","3","6"],"object_location_counts":[{"shards":"3,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"107'52","reported_seq":370,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.007765+0000","last_change":"2024-04-22T18:41:22.602374+0000","last_active":"2024-04-22T18:41:26.007765+0000","last_peered":"2024-04-22T18:41:26.007765+0000","last_clean":"2024-04-22T18:41:26.007765+0000","last_became_active":"2024-04-22T18:41:00.078669+0000","last_became_peered":"2024-04-22T18:41:00.078669+0000","last_unstale":"2024-04-22T18:41:26.007765+0000","last_undegraded":"2024-04-22T18:41:26.007765+0000","last_fullsized":"2024-04-22T18:41:26.007765+0000","mapping_epoch":125,"log_start":"83'49","ondisk_log_start":"83'49","created":110,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":5,"last_scrub":"107'52","last_scrub_stamp":"2024-04-22T18:41:22.602272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:22.602272+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:48.741276+0000","scrub_duration":0.40203006699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"147'96","reported_seq":440,"reported_epoch":151,"state":"active+remapped+backfill_wait","last_fresh":"2024-04-22T18:41:26.958248+0000","last_change":"2024-04-22T18:41:26.958248+0000","last_active":"2024-04-22T18:41:26.958248+0000","last_peered":"2024-04-22T18:41:26.958248+0000","last_clean":"2024-04-22T18:40:53.248441+0000","last_became_active":"2024-04-22T18:41:26.957911+0000","last_became_peered":"2024-04-22T18:41:26.957911+0000","last_unstale":"2024-04-22T18:41:26.958248+0000","last_undegraded":"2024-04-22T18:41:26.958248+0000","last_fullsized":"2024-04-22T18:41:26.958248+0000","mapping_epoch":150,"log_start":"96'71","ondisk_log_start":"96'71","created":110,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":81,"ondisk_log_size":25,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.22461577699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5428711,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":87029,"num_write":27,"num_write_kb":8334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1847758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.15","version":"54'29","reported_seq":91,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:25.000503+0000","last_change":"2024-04-22T18:40:55.328642+0000","last_active":"2024-04-22T18:41:25.000503+0000","last_peered":"2024-04-22T18:41:25.000503+0000","last_clean":"2024-04-22T18:41:25.000503+0000","last_became_active":"2024-04-22T18:40:13.180144+0000","last_became_peered":"2024-04-22T18:40:13.180144+0000","last_unstale":"2024-04-22T18:41:25.000503+0000","last_undegraded":"2024-04-22T18:41:25.000503+0000","last_fullsized":"2024-04-22T18:41:25.000503+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'29","last_scrub_stamp":"2024-04-22T18:40:55.328529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.328529+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:10.967288+0000","scrub_duration":0.30217712000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"51'19","reported_seq":92,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.007740+0000","last_change":"2024-04-22T18:40:57.754351+0000","last_active":"2024-04-22T18:41:26.007740+0000","last_peered":"2024-04-22T18:41:26.007740+0000","last_clean":"2024-04-22T18:41:26.007740+0000","last_became_active":"2024-04-22T18:40:10.978840+0000","last_became_peered":"2024-04-22T18:40:10.978840+0000","last_unstale":"2024-04-22T18:41:26.007740+0000","last_undegraded":"2024-04-22T18:41:26.007740+0000","last_fullsized":"2024-04-22T18:41:26.007740+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"51'19","last_scrub_stamp":"2024-04-22T18:40:57.754261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:57.754261+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:17.927283+0000","scrub_duration":0.59184710600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"148'85","reported_seq":49,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:25.000421+0000","last_change":"2024-04-22T18:41:04.562945+0000","last_active":"2024-04-22T18:41:25.000421+0000","last_peered":"2024-04-22T18:41:25.000421+0000","last_clean":"2024-04-22T18:41:25.000421+0000","last_became_active":"2024-04-22T18:41:04.139159+0000","last_became_peered":"2024-04-22T18:41:04.139159+0000","last_unstale":"2024-04-22T18:41:25.000421+0000","last_undegraded":"2024-04-22T18:41:25.000421+0000","last_fullsized":"2024-04-22T18:41:25.000421+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'75","last_scrub_stamp":"2024-04-22T18:40:22.670296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.670296+0000","objects_scrubbed":2,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:49.126896+0000","scrub_duration":0.20951747300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218623,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":70051,"num_write":7,"num_write_kb":4562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12459808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"140'46","reported_seq":276,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.011317+0000","last_change":"2024-04-22T18:40:59.453830+0000","last_active":"2024-04-22T18:41:26.011317+0000","last_peered":"2024-04-22T18:41:26.011317+0000","last_clean":"2024-04-22T18:41:26.011317+0000","last_became_active":"2024-04-22T18:40:58.107873+0000","last_became_peered":"2024-04-22T18:40:58.107873+0000","last_unstale":"2024-04-22T18:41:26.011317+0000","last_undegraded":"2024-04-22T18:41:26.011317+0000","last_fullsized":"2024-04-22T18:41:26.011317+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361351,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":113,"num_read_kb":106978,"num_write":27,"num_write_kb":7134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2641926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"149'149","reported_seq":472,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.012053+0000","last_change":"2024-04-22T18:40:26.844287+0000","last_active":"2024-04-22T18:41:26.012053+0000","last_peered":"2024-04-22T18:41:26.012053+0000","last_clean":"2024-04-22T18:41:26.012053+0000","last_became_active":"2024-04-22T18:40:26.742351+0000","last_became_peered":"2024-04-22T18:40:26.742351+0000","last_unstale":"2024-04-22T18:41:26.012053+0000","last_undegraded":"2024-04-22T18:41:26.012053+0000","last_fullsized":"2024-04-22T18:41:26.012053+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'113","last_scrub_stamp":"2024-04-22T18:40:20.291988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.291988+0000","objects_scrubbed":2,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10270805500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4840548,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":22760,"num_write":43,"num_write_kb":12200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3940523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"31'5","reported_seq":184,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.090853+0000","last_change":"2024-04-22T18:40:10.212897+0000","last_active":"2024-04-22T18:41:26.090853+0000","last_peered":"2024-04-22T18:41:26.090853+0000","last_clean":"2024-04-22T18:41:26.090853+0000","last_became_active":"2024-04-22T18:39:59.265189+0000","last_became_peered":"2024-04-22T18:39:59.265189+0000","last_unstale":"2024-04-22T18:41:26.090853+0000","last_undegraded":"2024-04-22T18:41:26.090853+0000","last_fullsized":"2024-04-22T18:41:26.090853+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":5,"last_scrub":"31'5","last_scrub_stamp":"2024-04-22T18:40:10.212815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.212815+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015068670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":37,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:25.000614+0000","last_change":"2024-04-22T18:40:58.207745+0000","last_active":"2024-04-22T18:41:25.000614+0000","last_peered":"2024-04-22T18:41:25.000614+0000","last_clean":"2024-04-22T18:41:25.000614+0000","last_became_active":"2024-04-22T18:40:58.207451+0000","last_became_peered":"2024-04-22T18:40:58.207451+0000","last_unstale":"2024-04-22T18:41:25.000614+0000","last_undegraded":"2024-04-22T18:41:25.000614+0000","last_fullsized":"2024-04-22T18:41:25.000614+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:19.136933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:19.136933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:38.724946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"147'185","reported_seq":676,"reported_epoch":150,"state":"peering","last_fresh":"2024-04-22T18:41:26.010258+0000","last_change":"2024-04-22T18:41:26.010382+0000","last_active":"2024-04-22T18:41:23.995395+0000","last_peered":"2024-04-22T18:41:25.202910+0000","last_clean":"2024-04-22T18:41:25.202910+0000","last_became_active":"2024-04-22T18:41:12.069362+0000","last_became_peered":"2024-04-22T18:41:25.101363+0000","last_unstale":"2024-04-22T18:41:26.010258+0000","last_undegraded":"2024-04-22T18:41:26.010258+0000","last_fullsized":"2024-04-22T18:41:26.010258+0000","mapping_epoch":150,"log_start":"147'185","ondisk_log_start":"147'185","created":20,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":0,"log_dups_size":161,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13563546,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":328,"num_read_kb":387906,"num_write":81,"num_write_kb":23008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":43307622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":161,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.013051+0000","last_change":"2024-04-22T18:41:17.658521+0000","last_active":"2024-04-22T18:41:26.013051+0000","last_peered":"2024-04-22T18:41:26.013051+0000","last_clean":"2024-04-22T18:41:26.013051+0000","last_became_active":"2024-04-22T18:41:04.037248+0000","last_became_peered":"2024-04-22T18:41:04.037248+0000","last_unstale":"2024-04-22T18:41:26.013051+0000","last_undegraded":"2024-04-22T18:41:26.013051+0000","last_fullsized":"2024-04-22T18:41:26.013051+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"147'30","reported_seq":245,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.090936+0000","last_change":"2024-04-22T18:41:11.057678+0000","last_active":"2024-04-22T18:41:26.090936+0000","last_peered":"2024-04-22T18:41:26.090936+0000","last_clean":"2024-04-22T18:41:26.090936+0000","last_became_active":"2024-04-22T18:41:11.057432+0000","last_became_peered":"2024-04-22T18:41:11.057432+0000","last_unstale":"2024-04-22T18:41:26.090936+0000","last_undegraded":"2024-04-22T18:41:26.090936+0000","last_fullsized":"2024-04-22T18:41:26.090936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'13","last_scrub_stamp":"2024-04-22T18:39:54.377980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.377980+0000","objects_scrubbed":1,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30260881099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":630312,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":82769,"num_write":13,"num_write_kb":4059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":169,"reported_epoch":150,"state":"peering","last_fresh":"2024-04-22T18:41:26.012843+0000","last_change":"2024-04-22T18:41:26.012968+0000","last_active":"2024-04-22T18:41:26.012843+0000","last_peered":"2024-04-22T18:41:25.100668+0000","last_clean":"2024-04-22T18:41:25.100668+0000","last_became_active":"2024-04-22T18:41:04.427985+0000","last_became_peered":"2024-04-22T18:41:04.427985+0000","last_unstale":"2024-04-22T18:41:26.012843+0000","last_undegraded":"2024-04-22T18:41:26.012843+0000","last_fullsized":"2024-04-22T18:41:26.012843+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:09.538542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:09.538542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:34.189737+0000","scrub_duration":0.35201753200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"137'90","reported_seq":336,"reported_epoch":151,"state":"remapped+peering","last_fresh":"2024-04-22T18:41:26.655859+0000","last_change":"2024-04-22T18:41:26.655859+0000","last_active":"2024-04-22T18:41:26.655859+0000","last_peered":"2024-04-22T18:41:08.944978+0000","last_clean":"2024-04-22T18:41:08.944978+0000","last_became_active":"2024-04-22T18:40:34.910015+0000","last_became_peered":"2024-04-22T18:40:34.910015+0000","last_unstale":"2024-04-22T18:41:26.655859+0000","last_undegraded":"2024-04-22T18:41:26.655859+0000","last_fullsized":"2024-04-22T18:41:26.655859+0000","mapping_epoch":151,"log_start":"137'90","ondisk_log_start":"137'90","created":20,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'90","last_scrub_stamp":"2024-04-22T18:41:22.057812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:22.057812+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":61,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5311111,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":101461,"num_write":21,"num_write_kb":5510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7812071,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":160,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.854653+0000","last_change":"2024-04-22T18:41:26.854653+0000","last_active":"2024-04-22T18:41:26.854653+0000","last_peered":"2024-04-22T18:41:26.854653+0000","last_clean":"2024-04-22T18:41:26.854653+0000","last_became_active":"2024-04-22T18:41:26.854399+0000","last_became_peered":"2024-04-22T18:41:26.854399+0000","last_unstale":"2024-04-22T18:41:26.854653+0000","last_undegraded":"2024-04-22T18:41:26.854653+0000","last_fullsized":"2024-04-22T18:41:26.854653+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"149'42","reported_seq":325,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.210869+0000","last_change":"2024-04-22T18:41:22.856838+0000","last_active":"2024-04-22T18:41:26.210869+0000","last_peered":"2024-04-22T18:41:26.210869+0000","last_clean":"2024-04-22T18:41:26.210869+0000","last_became_active":"2024-04-22T18:41:22.856520+0000","last_became_peered":"2024-04-22T18:41:22.856520+0000","last_unstale":"2024-04-22T18:41:26.210869+0000","last_undegraded":"2024-04-22T18:41:26.210869+0000","last_fullsized":"2024-04-22T18:41:26.210869+0000","mapping_epoch":129,"log_start":"64'38","ondisk_log_start":"64'38","created":20,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":25,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":124,"num_read_kb":177524,"num_write":25,"num_write_kb":6898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":11374836,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"149'352","reported_seq":1197,"reported_epoch":150,"state":"peering","last_fresh":"2024-04-22T18:41:26.010151+0000","last_change":"2024-04-22T18:41:26.010316+0000","last_active":"2024-04-22T18:41:26.010151+0000","last_peered":"2024-04-22T18:41:25.657445+0000","last_clean":"2024-04-22T18:41:25.657445+0000","last_became_active":"2024-04-22T18:41:16.658016+0000","last_became_peered":"2024-04-22T18:41:16.658016+0000","last_unstale":"2024-04-22T18:41:26.010151+0000","last_undegraded":"2024-04-22T18:41:26.010151+0000","last_fullsized":"2024-04-22T18:41:26.010151+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'293","last_scrub_stamp":"2024-04-22T18:40:53.582354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:53.582354+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:19.193770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38670171,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":299,"num_read_kb":4188,"num_write":431,"num_write_kb":106105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":54286740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":121,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.657445+0000","last_change":"2024-04-22T18:41:26.657445+0000","last_active":"2024-04-22T18:41:26.657445+0000","last_peered":"2024-04-22T18:41:26.657445+0000","last_clean":"2024-04-22T18:41:26.657445+0000","last_became_active":"2024-04-22T18:41:26.657206+0000","last_became_peered":"2024-04-22T18:41:26.657206+0000","last_unstale":"2024-04-22T18:41:26.657445+0000","last_undegraded":"2024-04-22T18:41:26.657445+0000","last_fullsized":"2024-04-22T18:41:26.657445+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:55.558446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.558446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:00.817184+0000","scrub_duration":0.56126016300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"115'32","reported_seq":280,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.010593+0000","last_change":"2024-04-22T18:40:21.426510+0000","last_active":"2024-04-22T18:41:26.010593+0000","last_peered":"2024-04-22T18:41:26.010593+0000","last_clean":"2024-04-22T18:41:26.010593+0000","last_became_active":"2024-04-22T18:39:51.943071+0000","last_became_peered":"2024-04-22T18:39:51.943071+0000","last_unstale":"2024-04-22T18:41:26.010593+0000","last_undegraded":"2024-04-22T18:41:26.010593+0000","last_fullsized":"2024-04-22T18:41:26.010593+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-04-22T18:40:21.426350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.426350+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.678685+0000","scrub_duration":0.30334366400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2605118,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":99,"num_read_kb":108445,"num_write":4,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"148'277","reported_seq":652,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.013333+0000","last_change":"2024-04-22T18:40:20.465005+0000","last_active":"2024-04-22T18:41:26.013333+0000","last_peered":"2024-04-22T18:41:26.013333+0000","last_clean":"2024-04-22T18:41:26.013333+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:41:26.013333+0000","last_undegraded":"2024-04-22T18:41:26.013333+0000","last_fullsized":"2024-04-22T18:41:26.013333+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'176","last_scrub_stamp":"2024-04-22T18:40:20.464799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.464799+0000","objects_scrubbed":31,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:45.914514+0000","scrub_duration":0.30629975799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32381109,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":246,"num_read_kb":4944,"num_write":359,"num_write_kb":74266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":109,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:25.564767+0000","last_change":"2024-04-22T18:40:10.403338+0000","last_active":"2024-04-22T18:41:25.564767+0000","last_peered":"2024-04-22T18:41:25.564767+0000","last_clean":"2024-04-22T18:41:25.564767+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:41:25.564767+0000","last_undegraded":"2024-04-22T18:41:25.564767+0000","last_fullsized":"2024-04-22T18:41:25.564767+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:10.403189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.403189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:34.232014+0000","scrub_duration":0.201433005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"147'64","reported_seq":375,"reported_epoch":151,"state":"remapped+peering","last_fresh":"2024-04-22T18:41:26.654647+0000","last_change":"2024-04-22T18:41:26.654804+0000","last_active":"2024-04-22T18:41:26.111478+0000","last_peered":"2024-04-22T18:41:25.989808+0000","last_clean":"2024-04-22T18:41:25.989808+0000","last_became_active":"2024-04-22T18:40:42.750084+0000","last_became_peered":"2024-04-22T18:40:42.750084+0000","last_unstale":"2024-04-22T18:41:26.654647+0000","last_undegraded":"2024-04-22T18:41:26.654647+0000","last_fullsized":"2024-04-22T18:41:26.654647+0000","mapping_epoch":151,"log_start":"83'49","ondisk_log_start":"83'49","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":49,"ondisk_log_size":15,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655547,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":94903,"num_write":29,"num_write_kb":8701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2254765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"unknown","last_fresh":"2024-04-22T18:41:26.008926+0000","last_change":"2024-04-22T18:41:26.481241+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T18:41:26.008926+0000","last_undegraded":"2024-04-22T18:41:26.008926+0000","last_fullsized":"2024-04-22T18:41:26.008926+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'209","last_scrub_stamp":"2024-04-22T18:40:18.683442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:18.683442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:32.858859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":152,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:25.100427+0000","last_change":"2024-04-22T18:40:22.521828+0000","last_active":"2024-04-22T18:41:25.100427+0000","last_peered":"2024-04-22T18:41:25.100427+0000","last_clean":"2024-04-22T18:41:25.100427+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:41:25.100427+0000","last_undegraded":"2024-04-22T18:41:25.100427+0000","last_fullsized":"2024-04-22T18:41:25.100427+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.521638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:43.407910+0000","scrub_duration":0.30190717299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"unknown","last_fresh":"2024-04-22T18:41:26.014310+0000","last_change":"2024-04-22T18:41:26.481320+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T18:41:26.014310+0000","last_undegraded":"2024-04-22T18:41:26.014310+0000","last_fullsized":"2024-04-22T18:41:26.014310+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:29.620751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"149'300","reported_seq":963,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.109109+0000","last_change":"2024-04-22T18:41:19.866530+0000","last_active":"2024-04-22T18:41:26.109109+0000","last_peered":"2024-04-22T18:41:26.109109+0000","last_clean":"2024-04-22T18:41:26.109109+0000","last_became_active":"2024-04-22T18:41:19.765392+0000","last_became_peered":"2024-04-22T18:41:19.765392+0000","last_unstale":"2024-04-22T18:41:26.109109+0000","last_undegraded":"2024-04-22T18:41:26.109109+0000","last_fullsized":"2024-04-22T18:41:26.109109+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'212","last_scrub_stamp":"2024-04-22T18:40:21.870855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.870855+0000","objects_scrubbed":46,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:32.865904+0000","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40865745,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":279,"num_read_kb":12411,"num_write":376,"num_write_kb":83263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":43826500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":164,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:25.464510+0000","last_change":"2024-04-22T18:40:07.420765+0000","last_active":"2024-04-22T18:41:25.464510+0000","last_peered":"2024-04-22T18:41:25.464510+0000","last_clean":"2024-04-22T18:41:25.464510+0000","last_became_active":"2024-04-22T18:40:02.741862+0000","last_became_peered":"2024-04-22T18:40:02.741862+0000","last_unstale":"2024-04-22T18:41:25.464510+0000","last_undegraded":"2024-04-22T18:41:25.464510+0000","last_fullsized":"2024-04-22T18:41:25.464510+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:07.420618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:07.420618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:27.642486+0000","scrub_duration":0.30164867299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":332,"reported_epoch":150,"state":"peering","last_fresh":"2024-04-22T18:41:26.114114+0000","last_change":"2024-04-22T18:41:26.114196+0000","last_active":"2024-04-22T18:41:26.114114+0000","last_peered":"2024-04-22T18:41:25.100513+0000","last_clean":"2024-04-22T18:41:25.100513+0000","last_became_active":"2024-04-22T18:40:57.952303+0000","last_became_peered":"2024-04-22T18:40:57.952303+0000","last_unstale":"2024-04-22T18:41:26.114114+0000","last_undegraded":"2024-04-22T18:41:26.114114+0000","last_fullsized":"2024-04-22T18:41:26.114114+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:39:53.346787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.346787+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20273052499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"137'75","reported_seq":64,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.085833+0000","last_change":"2024-04-22T18:41:04.681901+0000","last_active":"2024-04-22T18:41:26.085833+0000","last_peered":"2024-04-22T18:41:26.085833+0000","last_clean":"2024-04-22T18:41:26.085833+0000","last_became_active":"2024-04-22T18:41:04.227104+0000","last_became_peered":"2024-04-22T18:41:04.227104+0000","last_unstale":"2024-04-22T18:41:26.085833+0000","last_undegraded":"2024-04-22T18:41:26.085833+0000","last_fullsized":"2024-04-22T18:41:26.085833+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'38","last_scrub_stamp":"2024-04-22T18:39:55.746067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:55.746067+0000","objects_scrubbed":1,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60877772299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6168268,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":174457,"num_write":57,"num_write_kb":16481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15782313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"121'57","reported_seq":333,"reported_epoch":149,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:41:26.013511+0000","last_change":"2024-04-22T18:41:04.027077+0000","last_active":"2024-04-22T18:41:26.013511+0000","last_peered":"2024-04-22T18:41:26.013511+0000","last_clean":"2024-04-22T18:40:56.376630+0000","last_became_active":"2024-04-22T18:41:04.026361+0000","last_became_peered":"2024-04-22T18:41:04.026361+0000","last_unstale":"2024-04-22T18:41:26.013511+0000","last_undegraded":"2024-04-22T18:41:26.013511+0000","last_fullsized":"2024-04-22T18:41:26.013511+0000","mapping_epoch":129,"log_start":"79'52","ondisk_log_start":"79'52","created":20,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'57","last_scrub_stamp":"2024-04-22T18:40:56.376497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:56.376497+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":61,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:18.705090+0000","scrub_duration":0.30191299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3832029,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":28073,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"144'145","reported_seq":577,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:25.639418+0000","last_change":"2024-04-22T18:41:17.760985+0000","last_active":"2024-04-22T18:41:25.639418+0000","last_peered":"2024-04-22T18:41:25.639418+0000","last_clean":"2024-04-22T18:41:25.639418+0000","last_became_active":"2024-04-22T18:41:17.759812+0000","last_became_peered":"2024-04-22T18:41:17.759812+0000","last_unstale":"2024-04-22T18:41:25.639418+0000","last_undegraded":"2024-04-22T18:41:25.639418+0000","last_fullsized":"2024-04-22T18:41:25.639418+0000","mapping_epoch":141,"log_start":"84'88","ondisk_log_start":"84'88","created":20,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":57,"log_dups_size":106,"ondisk_log_size":57,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7235562,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":127745,"num_write":53,"num_write_kb":17927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12038689,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"147'116","reported_seq":427,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.211567+0000","last_change":"2024-04-22T18:41:11.054130+0000","last_active":"2024-04-22T18:41:26.211567+0000","last_peered":"2024-04-22T18:41:26.211567+0000","last_clean":"2024-04-22T18:41:26.211567+0000","last_became_active":"2024-04-22T18:41:11.053760+0000","last_became_peered":"2024-04-22T18:41:11.053760+0000","last_unstale":"2024-04-22T18:41:26.211567+0000","last_undegraded":"2024-04-22T18:41:26.211567+0000","last_fullsized":"2024-04-22T18:41:26.211567+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6117830,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":169,"num_read_kb":195188,"num_write":60,"num_write_kb":15942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4554596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":150,"state":"unknown","last_fresh":"2024-04-22T18:41:26.010575+0000","last_change":"2024-04-22T18:41:26.481283+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T18:41:26.010575+0000","last_undegraded":"2024-04-22T18:41:26.010575+0000","last_fullsized":"2024-04-22T18:41:26.010575+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'109","last_scrub_stamp":"2024-04-22T18:40:17.658363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:17.658363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:28.951874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"140'124","reported_seq":491,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:25.766807+0000","last_change":"2024-04-22T18:41:12.783644+0000","last_active":"2024-04-22T18:41:25.766807+0000","last_peered":"2024-04-22T18:41:25.766807+0000","last_clean":"2024-04-22T18:41:25.766807+0000","last_became_active":"2024-04-22T18:41:11.956312+0000","last_became_peered":"2024-04-22T18:41:11.956312+0000","last_unstale":"2024-04-22T18:41:25.766807+0000","last_undegraded":"2024-04-22T18:41:25.766807+0000","last_fullsized":"2024-04-22T18:41:25.766807+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'119","last_scrub_stamp":"2024-04-22T18:40:55.573199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.573199+0000","objects_scrubbed":4,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32976813900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2436998,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":255,"num_read_kb":315526,"num_write":65,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9347897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"133'66","reported_seq":58,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:25.100283+0000","last_change":"2024-04-22T18:41:19.456295+0000","last_active":"2024-04-22T18:41:25.100283+0000","last_peered":"2024-04-22T18:41:25.100283+0000","last_clean":"2024-04-22T18:41:25.100283+0000","last_became_active":"2024-04-22T18:41:11.154301+0000","last_became_peered":"2024-04-22T18:41:11.154301+0000","last_unstale":"2024-04-22T18:41:25.100283+0000","last_undegraded":"2024-04-22T18:41:25.100283+0000","last_fullsized":"2024-04-22T18:41:25.100283+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'66","last_scrub_stamp":"2024-04-22T18:41:19.456160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:19.456160+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:36.925204+0000","scrub_duration":0.20146846400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"145'51","reported_seq":57,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-04-22T18:41:25.000469+0000","last_change":"2024-04-22T18:41:11.353806+0000","last_active":"2024-04-22T18:41:25.000469+0000","last_peered":"2024-04-22T18:41:25.000469+0000","last_clean":"2024-04-22T18:41:25.000469+0000","last_became_active":"2024-04-22T18:41:11.253409+0000","last_became_peered":"2024-04-22T18:41:11.253409+0000","last_unstale":"2024-04-22T18:41:25.000469+0000","last_undegraded":"2024-04-22T18:41:25.000469+0000","last_fullsized":"2024-04-22T18:41:25.000469+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'23","last_scrub_stamp":"2024-04-22T18:40:20.587606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.587606+0000","objects_scrubbed":1,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10225366599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":815123,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":84302,"num_write":27,"num_write_kb":8588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5574713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"149'82","reported_seq":346,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-04-22T18:41:26.010917+0000","last_change":"2024-04-22T18:41:11.069650+0000","last_active":"2024-04-22T18:41:26.010917+0000","last_peered":"2024-04-22T18:41:26.010917+0000","last_clean":"2024-04-22T18:41:26.010917+0000","last_became_active":"2024-04-22T18:41:11.068781+0000","last_became_peered":"2024-04-22T18:41:11.068781+0000","last_unstale":"2024-04-22T18:41:26.010917+0000","last_undegraded":"2024-04-22T18:41:26.010917+0000","last_fullsized":"2024-04-22T18:41:26.010917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3075780,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":146,"num_read_kb":138368,"num_write":47,"num_write_kb":12043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":94599859,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":2874,"num_read_kb":3411104,"num_write":902,"num_write_kb":259998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":172086907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":25,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184209408,"data_stored":184553750,"data_compressed":1247066,"data_compressed_allocated":1335296,"data_compressed_original":2539520,"omap_allocated":0,"internal_metadata":0},"log_size":1502,"ondisk_log_size":1502,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":111917025,"num_objects":129,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":0,"num_read":824,"num_read_kb":21543,"num_write":1166,"num_write_kb":263634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":118972564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":238706688,"data_stored":252146588,"data_compressed":14069349,"data_compressed_allocated":14970880,"data_compressed_original":29057024,"omap_allocated":0,"internal_metadata":0},"log_size":929,"ondisk_log_size":929,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":2115712,"data_compressed":121124,"data_compressed_allocated":163840,"data_compressed_original":2072576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67452,"kb_used_data":35088,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94304388,"statfs":{"total":96636764160,"available":96567693312,"internally_reserved":0,"allocated":35930112,"data_stored":35588981,"data_compressed":211962,"data_compressed_allocated":241664,"data_compressed_original":970752,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":9.8949999999999996,"5min":2.3170000000000002,"15min":1.054},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":139.84299999999999,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.5},{"interface":"front","average":{"1min":13.949,"5min":6.3959999999999999,"15min":5.1369999999999996},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":186.97900000000001,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.42199999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":4.1959999999999997,"15min":4.6420000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":27.027000000000001,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.34300000000000003},{"interface":"front","average":{"1min":14.961,"5min":13.43,"15min":13.175000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":183.16300000000001,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.69099999999999995}]},{"osd":2,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":3.3479999999999999,"15min":3.839},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64100000000000001,"5min":77.320999999999998,"15min":77.320999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47199999999999998,"15min":0.47299999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.54700000000000004}]},{"osd":3,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":3.4079999999999999,"15min":3.903},"min":{"1min":0.32000000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60799999999999998,"5min":77.361000000000004,"15min":77.361000000000004},"last":0.626},{"interface":"front","average":{"1min":0.497,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.30299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.65000000000000002,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.61099999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":5.6710000000000003,"5min":4.7009999999999996,"15min":4.5389999999999997},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":69.097999999999999,"5min":85.509,"15min":85.509},"last":0.52400000000000002},{"interface":"front","average":{"1min":7.9509999999999996,"5min":19.887,"15min":21.876999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":181.97,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.55800000000000005}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.5379999999999998,"15min":2.8929999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67800000000000005,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.67700000000000005},{"interface":"front","average":{"1min":3.3170000000000002,"5min":6.3929999999999998,"15min":6.9050000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":69.063000000000002,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.64200000000000002}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":17.294,"5min":14.079000000000001,"15min":13.542999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":139.428,"5min":139.428,"15min":139.428},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.3580000000000001,"5min":3.5779999999999998,"15min":3.948},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":22.878,"5min":77.405000000000001,"15min":77.405000000000001},"last":0.66400000000000003}]}]},{"osd":6,"up_from":0,"seq":55834574884,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574884,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":158168,"kb_used_data":122448,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94213672,"statfs":{"total":96636764160,"available":96474800128,"internally_reserved":0,"allocated":125386752,"data_stored":128848294,"data_compressed":4890353,"data_compressed_allocated":5210112,"data_compressed_original":9953280,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,0,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":13.022,"5min":15.922000000000001,"15min":16.405999999999999},"min":{"1min":0.27600000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":252.96199999999999,"5min":252.96199999999999,"15min":252.96199999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":20.289000000000001,"5min":10.912000000000001,"15min":9.3490000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":187.148,"5min":187.148,"15min":187.148},"last":0.436}]},{"osd":1,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":9.9629999999999992,"5min":2.327,"15min":1.0549999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":96.486999999999995,"5min":96.486999999999995,"15min":96.486999999999995},"last":0.33800000000000002},{"interface":"front","average":{"1min":19.181999999999999,"5min":8.1449999999999996,"15min":6.3049999999999997},"min":{"1min":0.30199999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":264.68299999999999,"5min":264.68299999999999,"15min":264.68299999999999},"last":0.40300000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":18.23,"5min":10.502000000000001,"15min":9.2140000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":186.94300000000001,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.372},{"interface":"front","average":{"1min":9.7699999999999996,"5min":2.3100000000000001,"15min":1.0669999999999999},"min":{"1min":0.33100000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":96.471999999999994,"5min":96.471999999999994,"15min":96.471999999999994},"last":0.46899999999999997}]},{"osd":3,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":21.619,"5min":7.5410000000000004,"15min":5.1950000000000003},"min":{"1min":0.32200000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":252.91900000000001,"5min":252.91900000000001,"15min":252.91900000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":22.878,"5min":21.350000000000001,"15min":21.094999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":187.11099999999999,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.39500000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":9.75,"5min":2.2559999999999998,"15min":1.0069999999999999},"min":{"1min":0.23100000000000001,"5min":0.218,"15min":0.218},"max":{"1min":96.408000000000001,"5min":96.408000000000001,"15min":96.408000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":3.3159999999999998,"15min":3.7970000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.76500000000000001,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.36199999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":28.026,"5min":16.132999999999999,"15min":14.151},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":197.72999999999999,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":9.7560000000000002,"5min":2.3039999999999998,"15min":1.0620000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":96.438000000000002,"5min":96.438000000000002,"15min":96.438000000000002},"last":0.41799999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:40:37 2024","interfaces":[{"interface":"back","average":{"1min":21.757000000000001,"5min":11.199,"15min":9.4390000000000001},"min":{"1min":0.30099999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":186.79300000000001,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":9.7880000000000003,"5min":2.343,"15min":1.1020000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":96.456999999999994,"5min":96.456999999999994,"15min":96.456999999999994},"last":0.378}]}]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98280,"kb_used_data":61968,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94273560,"statfs":{"total":96636764160,"available":96536125440,"internally_reserved":0,"allocated":63455232,"data_stored":65069842,"data_compressed":3200660,"data_compressed_allocated":3399680,"data_compressed_original":6144000,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":9.9700000000000006,"15min":11.456},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":11.797000000000001,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.26700000000000002,"5min":0.316,"15min":0.32400000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.43099999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.224}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":25.978000000000002,"5min":9.2309999999999999,"15min":6.4400000000000004},"min":{"1min":0.20999999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":197.02099999999999,"5min":197.02099999999999,"15min":197.02099999999999},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.623,"5min":0.59199999999999997,"15min":0.41999999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":16.388999999999999,"5min":16.388999999999999,"15min":16.388999999999999},"last":8.2400000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":6.0090000000000003,"5min":2.0059999999999998,"15min":1.339},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":87.635999999999996,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.51800000000000002,"15min":0.45600000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":11.859999999999999,"5min":11.859999999999999,"15min":11.859999999999999},"last":0.247}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":3.379,"15min":3.8650000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.79100000000000004,"5min":71.998000000000005,"15min":71.998000000000005},"last":8.6950000000000003},{"interface":"front","average":{"1min":6.1719999999999997,"5min":4.5439999999999996,"15min":4.2729999999999997},"min":{"1min":0.34799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":87.724000000000004,"5min":87.724000000000004,"15min":87.724000000000004},"last":8.7100000000000009}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":6.1539999999999999,"5min":2.1760000000000002,"15min":1.5129999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":87.784000000000006,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.34799999999999998},{"interface":"front","average":{"1min":1.038,"5min":1.2130000000000001,"15min":1.242},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":11.911,"5min":13.965,"15min":13.965},"last":0.37}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.52700000000000002,"15min":0.53700000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72999999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.32700000000000001},{"interface":"front","average":{"1min":9.9130000000000003,"5min":2.4079999999999999,"15min":1.157},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":96.140000000000001,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.41399999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":0.69599999999999995,"15min":0.621},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":11.887,"5min":11.887,"15min":11.887},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.55000000000000004,"15min":0.56200000000000006},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75600000000000001,"5min":0.995,"15min":0.995},"last":0.35799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60032,"kb_used_data":28912,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94311808,"statfs":{"total":96636764160,"available":96575291392,"internally_reserved":0,"allocated":29605888,"data_stored":30121766,"data_compressed":1291409,"data_compressed_allocated":1376256,"data_compressed_original":2879488,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":7.4240000000000004,"15min":7.8920000000000003},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":59.002000000000002,"5min":149.03,"15min":149.03},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.308,"5min":1.649,"15min":1.873},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.442,"5min":31.866,"15min":31.866},"last":0.25800000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":3.5579999999999998,"15min":1.8740000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":97.683999999999997,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.374},{"interface":"front","average":{"1min":22.157,"5min":16.338999999999999,"15min":15.369},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":393.42399999999998,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.29599999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":18.393000000000001,"5min":3.9689999999999999,"15min":1.5649999999999999},"min":{"1min":0.252,"5min":0.245,"15min":0.245},"max":{"1min":147.721,"5min":147.721,"15min":147.721},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.317,"5min":0.36899999999999999,"15min":0.377},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.53200000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.28199999999999997}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":3.4060000000000001,"5min":15.698,"15min":17.747},"min":{"1min":0.311,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":52.683,"5min":160.584,"15min":160.584},"last":0.41699999999999998},{"interface":"front","average":{"1min":10.984,"5min":4.1550000000000002,"15min":3.0169999999999999},"min":{"1min":0.32200000000000001,"5min":0.315,"15min":0.315},"max":{"1min":97.887,"5min":97.887,"15min":97.887},"last":9.7940000000000005}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":2.5409999999999999,"15min":2.8809999999999998},"min":{"1min":0.33300000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.74299999999999999,"5min":48.631,"15min":48.631},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":2.8820000000000001,"15min":3.278},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.83499999999999996,"5min":56.18,"15min":56.18},"last":0.38800000000000001}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":10.375,"5min":2.5030000000000001,"15min":1.1910000000000001},"min":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":97.832999999999998,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.436},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.33200000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.42799999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.63800000000000001,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"last":0.36199999999999999},{"interface":"front","average":{"1min":10.396000000000001,"5min":2.5470000000000002,"15min":1.2390000000000001},"min":{"1min":0.38800000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":97.861999999999995,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.35099999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":135420,"kb_used_data":95972,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94236420,"statfs":{"total":96636764160,"available":96498094080,"internally_reserved":0,"allocated":98275328,"data_stored":100689430,"data_compressed":2838437,"data_compressed_allocated":3035136,"data_compressed_original":6766592,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":3.9870000000000001,"5min":7.0839999999999996,"15min":7.5999999999999996},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":66.480999999999995,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.23000000000000001},{"interface":"front","average":{"1min":4.0430000000000001,"5min":7.0990000000000002,"15min":7.6079999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":66.498000000000005,"5min":88.156000000000006,"15min":88.156000000000006},"last":0.29099999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":6.9429999999999996,"15min":8.0470000000000006},"min":{"1min":0.20000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.626,"5min":183.04400000000001,"15min":183.04400000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.27900000000000003}]},{"osd":3,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.33700000000000002,"15min":0.33400000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.35399999999999998,"5min":11.664,"15min":13.548999999999999},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.878,"5min":231.864,"15min":231.864},"last":0.25900000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":7.29,"5min":11.845000000000001,"15min":12.603999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":109.399,"5min":192.42099999999999,"15min":192.42099999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.44900000000000001}]},{"osd":5,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":11.359999999999999,"15min":13.167},"min":{"1min":0.34399999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.052,"5min":138.089,"15min":138.089},"last":0.44},{"interface":"front","average":{"1min":0.48999999999999999,"5min":16.158999999999999,"15min":18.771000000000001},"min":{"1min":0.35099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69899999999999995,"5min":183.08199999999999,"15min":183.08199999999999},"last":0.43099999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":4.8520000000000003,"5min":3.1920000000000002,"15min":2.915},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":78.805000000000007,"5min":78.805000000000007,"15min":78.805000000000007},"last":0.40799999999999997},{"interface":"front","average":{"1min":5.9400000000000004,"5min":14.413,"15min":15.824999999999999},"min":{"1min":0.36599999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":98.968000000000004,"5min":183.10900000000001,"15min":183.10900000000001},"last":0.40200000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.38500000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":2.5350000000000001,"15min":2.871},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.0189999999999999,"5min":51.097000000000001,"15min":51.097000000000001},"last":0.374}]}]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32804,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339036,"statfs":{"total":96636764160,"available":96603172864,"internally_reserved":0,"allocated":1728512,"data_stored":772906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:40:36 2024","interfaces":[{"interface":"back","average":{"1min":17.760999999999999,"5min":8.9600000000000009,"15min":7.4930000000000003},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":99.296000000000006,"5min":133.09899999999999,"15min":133.09899999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":25.457999999999998,"5min":6.2969999999999997,"15min":3.1030000000000002},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.188},"max":{"1min":216.11600000000001,"5min":216.11600000000001,"15min":216.11600000000001},"last":8.2490000000000006}]},{"osd":2,"last update":"Mon Apr 22 18:40:36 2024","interfaces":[{"interface":"back","average":{"1min":2.649,"5min":1.774,"15min":1.6279999999999999},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":39.906999999999996,"5min":39.906999999999996,"15min":39.906999999999996},"last":0.308},{"interface":"front","average":{"1min":9.7650000000000006,"5min":2.2069999999999999,"15min":0.94699999999999995},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":94.820999999999998,"5min":94.820999999999998,"15min":94.820999999999998},"last":0.34699999999999998}]},{"osd":3,"last update":"Mon Apr 22 18:40:36 2024","interfaces":[{"interface":"back","average":{"1min":10.071999999999999,"5min":2.3849999999999998,"15min":1.1040000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":99.332999999999998,"5min":99.332999999999998,"15min":99.332999999999998},"last":0.38},{"interface":"front","average":{"1min":25.184999999999999,"5min":6.2869999999999999,"15min":3.137},"min":{"1min":0.25800000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":297.82100000000003,"5min":297.82100000000003,"15min":297.82100000000003},"last":0.252}]},{"osd":4,"last update":"Mon Apr 22 18:40:36 2024","interfaces":[{"interface":"back","average":{"1min":9.8930000000000007,"5min":2.3820000000000001,"15min":1.1299999999999999},"min":{"1min":0.35999999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":94.599999999999994,"5min":94.599999999999994,"15min":94.599999999999994},"last":0.58299999999999996},{"interface":"front","average":{"1min":13.815,"5min":10.692,"15min":10.172000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":160.02199999999999,"5min":160.02199999999999,"15min":160.02199999999999},"last":0.497}]},{"osd":5,"last update":"Mon Apr 22 18:40:36 2024","interfaces":[{"interface":"back","average":{"1min":5.2169999999999996,"5min":2.4390000000000001,"15min":1.976},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":41.645000000000003,"5min":41.645000000000003,"15min":41.645000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":9.9260000000000002,"5min":13.148999999999999,"15min":13.686},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":94.626000000000005,"5min":282.197,"15min":282.197},"last":0.47599999999999998}]},{"osd":6,"last update":"Mon Apr 22 18:40:36 2024","interfaces":[{"interface":"back","average":{"1min":11.340999999999999,"5min":2.819,"15min":1.399},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":99.355000000000004,"5min":99.355000000000004,"15min":99.355000000000004},"last":0.52900000000000003},{"interface":"front","average":{"1min":8.7260000000000009,"5min":3.1429999999999998,"15min":2.2130000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":52.762,"5min":52.762,"15min":52.762},"last":0.55900000000000005}]},{"osd":7,"last update":"Mon Apr 22 18:40:36 2024","interfaces":[{"interface":"back","average":{"1min":9.9469999999999992,"5min":5.7309999999999999,"15min":5.0289999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":94.662999999999997,"5min":94.662999999999997,"15min":94.662999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":10.285,"5min":2.621,"15min":1.343},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":99.379999999999995,"5min":99.379999999999995,"15min":99.379999999999995},"last":0.57299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45596672,"data_stored":48371571,"data_compressed":2388765,"data_compressed_allocated":2551808,"data_compressed_original":5455872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19222528,"data_stored":20590104,"data_compressed":1239076,"data_compressed_allocated":1318912,"data_compressed_original":2748416,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49733632,"data_stored":52354728,"data_compressed":3200660,"data_compressed_allocated":3399680,"data_compressed_original":6144000,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":76107776,"data_stored":80136195,"data_compressed":4350349,"data_compressed_allocated":4632576,"data_compressed_original":8851456,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":223399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27324416,"data_stored":28849251,"data_compressed":1581247,"data_compressed_allocated":1675264,"data_compressed_original":3284992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":48664576,"data_stored":48792715,"data_compressed":418678,"data_compressed_allocated":442368,"data_compressed_original":786432,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8732672,"data_stored":8766131,"data_compressed":52333,"data_compressed_allocated":57344,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12013568,"data_stored":11942923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45387776,"data_stored":45722582,"data_compressed":540004,"data_compressed_allocated":577536,"data_compressed_original":1101824,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33280000,"data_stored":33118894,"data_compressed":53657,"data_compressed_allocated":57344,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1929216,"data_stored":1913105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":34201600,"data_stored":34297400,"data_compressed":182394,"data_compressed_allocated":200704,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:41:31.375 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-04-22T18:41:31.375 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 36 2024-04-22T18:41:31.396 INFO:tasks.rados.rados.0.smithi089.stdout:1951: set_chunk oid 30 offset: 168448 length: 1486336 target oid c41f25c5c24efa845b2b5eb0e54c58c759b1fa20df6cdc69e9629b2997e13e82 offset: 0 snap -1 2024-04-22T18:41:31.396 INFO:tasks.rados.rados.0.smithi089.stdout:1952: set_chunk oid 47 target oid 2024-04-22T18:41:31.402 INFO:tasks.rados.rados.0.smithi089.stdout:1952: set_chunk oid 47 offset: 2003456 length: 667648 target oid 86f1259191c13db91003e175a32086a847a4d7865f03ec9837bc04f18c7a945f offset: 0 snap -1 2024-04-22T18:41:31.402 INFO:tasks.rados.rados.0.smithi089.stdout:1953: write oid 33 current snap is 0 2024-04-22T18:41:31.402 INFO:tasks.rados.rados.0.smithi089.stdout:1936: finishing write tid 4 to smithi08937929-19 2024-04-22T18:41:31.402 INFO:tasks.rados.rados.0.smithi089.stdout:1936: oid 19 updating version 0 to 152 2024-04-22T18:41:31.402 INFO:tasks.rados.rados.0.smithi089.stdout:1936: oid 19 version 152 is already newer than 151 2024-04-22T18:41:31.402 INFO:tasks.rados.rados.0.smithi089.stdout:1936: oid 19 version 152 is already newer than 150 2024-04-22T18:41:31.402 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 19 v 152 (ObjNum 413 snap 0 seq_num 413) dirty exists 2024-04-22T18:41:31.402 INFO:tasks.rados.rados.0.smithi089.stdout:1936: left oid 19 (ObjNum 413 snap 0 seq_num 413) 2024-04-22T18:41:31.402 INFO:tasks.rados.rados.0.smithi089.stdout:1927: expect (ObjNum 187 snap 0 seq_num 187) 2024-04-22T18:41:31.413 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:31.415+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:31.435 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop_reply(client.4502.0:4777 4.f e155/135)", 2024-04-22T18:41:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.369038+0000", 2024-04-22T18:41:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.067107257000000003, 2024-04-22T18:41:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.067132017000000002, 2024-04-22T18:41:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:41:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.369038+0000", 2024-04-22T18:41:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.369038+0000", 2024-04-22T18:41:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.369036+0000", 2024-04-22T18:41:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.369044+0000", 2024-04-22T18:41:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7020000000000007e-06 2024-04-22T18:41:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.369057+0000", 2024-04-22T18:41:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2978e-05 2024-04-22T18:41:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.369066+0000", 2024-04-22T18:41:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9940000000000006e-06 2024-04-22T18:41:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop_reply(client.4502.0:4778 4.f e155/135)", 2024-04-22T18:41:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.369175+0000", 2024-04-22T18:41:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.066970007999999998, 2024-04-22T18:41:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.067044590000000001, 2024-04-22T18:41:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:41:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.369175+0000", 2024-04-22T18:41:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.369175+0000", 2024-04-22T18:41:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.369174+0000", 2024-04-22T18:41:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.369179+0000", 2024-04-22T18:41:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5399999999999997e-06 2024-04-22T18:41:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.369188+0000", 2024-04-22T18:41:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8519999999999999e-06 2024-04-22T18:41:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.369194+0000", 2024-04-22T18:41:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.801e-06 2024-04-22T18:41:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4790 4.a 4.1f913eca (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.403650+0000", 2024-04-22T18:41:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.032495369000000003, 2024-04-22T18:41:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.032619375999999999, 2024-04-22T18:41:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:41:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4790 2024-04-22T18:41:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.403650+0000", 2024-04-22T18:41:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.403650+0000", 2024-04-22T18:41:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.403640+0000", 2024-04-22T18:41:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999905 2024-04-22T18:41:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.403656+0000", 2024-04-22T18:41:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5807999999999998e-05 2024-04-22T18:41:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.403670+0000", 2024-04-22T18:41:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.402e-05 2024-04-22T18:41:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.403678+0000", 2024-04-22T18:41:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4519999999999997e-06 2024-04-22T18:41:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4777 4.f 4:f2c48a0e:::smithi08937929-3:head [write 1826604~665129 in=665129b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-22T18:41:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.270901+0000", 2024-04-22T18:41:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.165243949, 2024-04-22T18:41:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16543449700000001, 2024-04-22T18:41:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:41:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4777 2024-04-22T18:41:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.270901+0000", 2024-04-22T18:41:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.270901+0000", 2024-04-22T18:41:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.270898+0000", 2024-04-22T18:41:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.271251+0000", 2024-04-22T18:41:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035233999999999998 2024-04-22T18:41:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.271279+0000", 2024-04-22T18:41:31.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8566999999999999e-05 2024-04-22T18:41:31.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.271287+0000", 2024-04-22T18:41:31.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4959999999999999e-06 2024-04-22T18:41:31.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.364252+0000", 2024-04-22T18:41:31.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.092965597999999997 2024-04-22T18:41:31.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.364323+0000", 2024-04-22T18:41:31.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0882000000000003e-05 2024-04-22T18:41:31.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:31.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.364414+0000", 2024-04-22T18:41:31.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1199999999999994e-05 2024-04-22T18:41:31.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.368923+0000", 2024-04-22T18:41:31.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045088159999999997 2024-04-22T18:41:31.466 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.466 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.466 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.467 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4778 4.f 4:f2c48a0e:::smithi08937929-3:head [setxattr _header (55) in=62b,truncate 2958098] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-22T18:41:31.467 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.271294+0000", 2024-04-22T18:41:31.467 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.16485063799999999, 2024-04-22T18:41:31.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16513538, 2024-04-22T18:41:31.468 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.468 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for sub ops", 2024-04-22T18:41:31.468 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.468 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.468 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.468 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4778 2024-04-22T18:41:31.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.469 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.271294+0000", 2024-04-22T18:41:31.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.271294+0000", 2024-04-22T18:41:31.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.271294+0000", 2024-04-22T18:41:31.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:31.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.271316+0000", 2024-04-22T18:41:31.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2228000000000001e-05 2024-04-22T18:41:31.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.271321+0000", 2024-04-22T18:41:31.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6409999999999999e-06 2024-04-22T18:41:31.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.271326+0000", 2024-04-22T18:41:31.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2899999999999996e-06 2024-04-22T18:41:31.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.366209+0000", 2024-04-22T18:41:31.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.094883757999999999 2024-04-22T18:41:31.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.366318+0000", 2024-04-22T18:41:31.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108934 2024-04-22T18:41:31.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:31.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.366438+0000", 2024-04-22T18:41:31.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119405 2024-04-22T18:41:31.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.368931+0000", 2024-04-22T18:41:31.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0024929599999999998 2024-04-22T18:41:31.477 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.477 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.477 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.478 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4779 4.f 4.7051234f (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e154)", 2024-04-22T18:41:31.478 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.271348+0000", 2024-04-22T18:41:31.478 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.16479695599999999, 2024-04-22T18:41:31.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16515234300000001, 2024-04-22T18:41:31.478 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.478 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:41:31.479 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.479 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.479 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.479 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4779 2024-04-22T18:41:31.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.479 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.271348+0000", 2024-04-22T18:41:31.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.271348+0000", 2024-04-22T18:41:31.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.271347+0000", 2024-04-22T18:41:31.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.271350+0000", 2024-04-22T18:41:31.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8329999999999998e-06 2024-04-22T18:41:31.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.271356+0000", 2024-04-22T18:41:31.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5809999999999996e-06 2024-04-22T18:41:31.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.271360+0000", 2024-04-22T18:41:31.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4569999999999998e-06 2024-04-22T18:41:31.484 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.484 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.485 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.485 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.485 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:41:31.485 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 6 2024-04-22T18:41:31.485 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:31.487 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:31.514 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:31.516+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:31.586 INFO:tasks.rados.rados.0.smithi089.stdout:1926: finishing write tid 1 to smithi08937929-37 2024-04-22T18:41:31.586 INFO:tasks.rados.rados.0.smithi089.stdout:1926: finishing write tid 2 to smithi08937929-37 2024-04-22T18:41:31.586 INFO:tasks.rados.rados.0.smithi089.stdout:1926: finishing write tid 3 to smithi08937929-37 2024-04-22T18:41:31.587 INFO:tasks.rados.rados.0.smithi089.stdout:1948: finishing set_chunk to oid 35 2024-04-22T18:41:31.587 INFO:tasks.rados.rados.0.smithi089.stdout:1948: got expected ENOENT (src dne) 2024-04-22T18:41:31.587 INFO:tasks.rados.rados.0.smithi089.stdout:1946: expect (ObjNum 351 snap 0 seq_num 351) 2024-04-22T18:41:31.615 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:31.617+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:31.631 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:31.632 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:31.632 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:31.632 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:31.632 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:31.640 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:31.655 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:41:31.716 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:31.718+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:31.723 INFO:tasks.rados.rados.0.smithi089.stdout:1926: finishing write tid 4 to smithi08937929-37 2024-04-22T18:41:31.723 INFO:tasks.rados.rados.0.smithi089.stdout:1926: finishing write tid 5 to smithi08937929-37 2024-04-22T18:41:31.724 INFO:tasks.rados.rados.0.smithi089.stdout:1926: oid 37 updating version 0 to 154 2024-04-22T18:41:31.724 INFO:tasks.rados.rados.0.smithi089.stdout:1926: oid 37 updating version 154 to 155 2024-04-22T18:41:31.724 INFO:tasks.rados.rados.0.smithi089.stdout:1926: oid 37 version 155 is already newer than 153 2024-04-22T18:41:31.724 INFO:tasks.rados.rados.0.smithi089.stdout:1926: oid 37 version 155 is already newer than 152 2024-04-22T18:41:31.724 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 155 (ObjNum 410 snap 0 seq_num 410) dirty exists 2024-04-22T18:41:31.724 INFO:tasks.rados.rados.0.smithi089.stdout:1926: left oid 37 (ObjNum 410 snap 0 seq_num 410) 2024-04-22T18:41:31.724 INFO:tasks.rados.rados.0.smithi089.stdout:1947: got (0) Success 2024-04-22T18:41:31.724 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 50 v 0 (ObjNum 405 snap 0 seq_num 405) dirty exists 2024-04-22T18:41:31.724 INFO:tasks.rados.rados.0.smithi089.stdout:1945: finishing set_chunk to oid 18 2024-04-22T18:41:31.724 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 18 v 81 (ObjNum 307 snap 0 seq_num 307) dirty exists 2024-04-22T18:41:31.724 INFO:tasks.rados.rados.0.smithi089.stdout:1949: finishing write tid 1 to smithi08937929-3 2024-04-22T18:41:31.724 INFO:tasks.rados.rados.0.smithi089.stdout:1953: seq_num 416 ranges {591712=786646,1943568=668798} 2024-04-22T18:41:31.732 INFO:tasks.rados.rados.0.smithi089.stdout:1953: writing smithi08937929-33 from 591712 to 1378358 tid 1 2024-04-22T18:41:31.740 INFO:tasks.rados.rados.0.smithi089.stdout:1953: writing smithi08937929-33 from 1943568 to 2612366 tid 2 2024-04-22T18:41:31.740 INFO:tasks.rados.rados.0.smithi089.stdout:1926: done (11 left) 2024-04-22T18:41:31.740 INFO:tasks.rados.rados.0.smithi089.stdout:1927: done (10 left) 2024-04-22T18:41:31.740 INFO:tasks.rados.rados.0.smithi089.stdout:1936: done (9 left) 2024-04-22T18:41:31.741 INFO:tasks.rados.rados.0.smithi089.stdout:1945: done (8 left) 2024-04-22T18:41:31.741 INFO:tasks.rados.rados.0.smithi089.stdout:1946: done (7 left) 2024-04-22T18:41:31.741 INFO:tasks.rados.rados.0.smithi089.stdout:1947: done (6 left) 2024-04-22T18:41:31.741 INFO:tasks.rados.rados.0.smithi089.stdout:1948: done (5 left) 2024-04-22T18:41:31.741 INFO:tasks.rados.rados.0.smithi089.stdout:1954: set_chunk oid 2 target oid 2024-04-22T18:41:31.756 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 157.56503416800001, 2024-04-22T18:41:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:41:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:41:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:41:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:41:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:41:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:41:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:41:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:41:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:41:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:41:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:41:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:41:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:41:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:41:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:41:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:41:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:41:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:41:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:41:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:41:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:41:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:41:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:41:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 157.56502309300001, 2024-04-22T18:41:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:41:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:41:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:41:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:41:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:41:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:41:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:41:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:41:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:41:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:41:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:41:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:41:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:41:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:41:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:41:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:41:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:41:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:41:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:41:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3193 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:41:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.713814803999995, 2024-04-22T18:41:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69912102700000001, 2024-04-22T18:41:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3193 2024-04-22T18:41:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:41:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:41:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042744+0000", 2024-04-22T18:41:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043013+0000", 2024-04-22T18:41:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026906 2024-04-22T18:41:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043028+0000", 2024-04-22T18:41:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5359999999999999e-05 2024-04-22T18:41:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043036+0000", 2024-04-22T18:41:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8420000000000005e-06 2024-04-22T18:41:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043081+0000", 2024-04-22T18:41:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5019999999999999e-05 2024-04-22T18:41:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043087+0000", 2024-04-22T18:41:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5420000000000002e-06 2024-04-22T18:41:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741856+0000", 2024-04-22T18:41:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69876864500000002 2024-04-22T18:41:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741867+0000", 2024-04-22T18:41:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.084e-05 2024-04-22T18:41:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3194 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:41:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.706564823999997, 2024-04-22T18:41:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69190137100000004, 2024-04-22T18:41:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3194 2024-04-22T18:41:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:41:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049996+0000", 2024-04-22T18:41:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.049991+0000", 2024-04-22T18:41:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999957 2024-04-22T18:41:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050284+0000", 2024-04-22T18:41:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029296699999999999 2024-04-22T18:41:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050304+0000", 2024-04-22T18:41:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.969e-05 2024-04-22T18:41:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050314+0000", 2024-04-22T18:41:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0268999999999999e-05 2024-04-22T18:41:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150531+0000", 2024-04-22T18:41:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100216868 2024-04-22T18:41:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150545+0000", 2024-04-22T18:41:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4063000000000001e-05 2024-04-22T18:41:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741890+0000", 2024-04-22T18:41:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59134479100000004 2024-04-22T18:41:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741897+0000", 2024-04-22T18:41:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1189999999999999e-06 2024-04-22T18:41:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3195 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.706232384000003, 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69159474200000004, 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3195 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:41:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:41:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050328+0000", 2024-04-22T18:41:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050333+0000", 2024-04-22T18:41:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0289999999999996e-06 2024-04-22T18:41:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050341+0000", 2024-04-22T18:41:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0050000000000004e-06 2024-04-22T18:41:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050346+0000", 2024-04-22T18:41:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8329999999999999e-06 2024-04-22T18:41:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150628+0000", 2024-04-22T18:41:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100281548 2024-04-22T18:41:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150672+0000", 2024-04-22T18:41:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3875999999999998e-05 2024-04-22T18:41:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741916+0000", 2024-04-22T18:41:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59124448799999996 2024-04-22T18:41:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741923+0000", 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7689999999999999e-06 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3196 4.11 4.9f8e4db1 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.706202265000002, 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69158947400000004, 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3196 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050358+0000", 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050361+0000", 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2040000000000002e-06 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050367+0000", 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3119999999999999e-06 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.050372+0000", 2024-04-22T18:41:31.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0869999999999997e-06 2024-04-22T18:41:31.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150688+0000", 2024-04-22T18:41:31.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100315381 2024-04-22T18:41:31.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.150691+0000", 2024-04-22T18:41:31.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8899999999999999e-06 2024-04-22T18:41:31.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741941+0000", 2024-04-22T18:41:31.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.59125044900000001 2024-04-22T18:41:31.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741948+0000", 2024-04-22T18:41:31.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.827e-06 2024-04-22T18:41:31.785 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.785 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.785 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.785 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3504 4.9 4:96737e19:::smithi08937929-4:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:31.785 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:31.785 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.345699089, 2024-04-22T18:41:31.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.951755291, 2024-04-22T18:41:31.786 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.786 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.786 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.786 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.786 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.786 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3504 2024-04-22T18:41:31.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.786 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:31.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:31.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410861+0000", 2024-04-22T18:41:31.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410863+0000", 2024-04-22T18:41:31.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.635e-06 2024-04-22T18:41:31.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410866+0000", 2024-04-22T18:41:31.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8499999999999998e-06 2024-04-22T18:41:31.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410868+0000", 2024-04-22T18:41:31.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.497e-06 2024-04-22T18:41:31.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615137+0000", 2024-04-22T18:41:31.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20426903499999999 2024-04-22T18:41:31.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615218+0000", 2024-04-22T18:41:31.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0860999999999997e-05 2024-04-22T18:41:31.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.362617+0000", 2024-04-22T18:41:31.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.74739882700000004 2024-04-22T18:41:31.791 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.791 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.791 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.791 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4194 4.b 4:d3068ee3:::smithi08937929-29:head [sparse-read 0~1980000,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:31.791 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:31.791 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.592440018000001, 2024-04-22T18:41:31.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.294164774, 2024-04-22T18:41:31.791 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.791 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.791 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.791 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.792 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.792 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4194 2024-04-22T18:41:31.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.792 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:31.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:31.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164120+0000", 2024-04-22T18:41:31.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164125+0000", 2024-04-22T18:41:31.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2899999999999996e-06 2024-04-22T18:41:31.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164128+0000", 2024-04-22T18:41:31.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8700000000000002e-06 2024-04-22T18:41:31.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164132+0000", 2024-04-22T18:41:31.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.151e-06 2024-04-22T18:41:31.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164188+0000", 2024-04-22T18:41:31.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6912e-05 2024-04-22T18:41:31.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164201+0000", 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2466000000000001e-05 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.242879+0000", 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0786779710000001 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243364+0000", 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000485081 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456588+0000", 2024-04-22T18:41:31.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21322370900000001 2024-04-22T18:41:31.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456674+0000", 2024-04-22T18:41:31.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6079e-05 2024-04-22T18:41:31.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.458285+0000", 2024-04-22T18:41:31.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001611716 2024-04-22T18:41:31.796 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.797 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.797 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.797 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4195 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:31.797 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:31.797 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.592422396, 2024-04-22T18:41:31.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.395681808, 2024-04-22T18:41:31.797 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.797 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.797 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.797 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.797 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.798 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4195 2024-04-22T18:41:31.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.798 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:31.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:31.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:31.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164141+0000", 2024-04-22T18:41:31.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7160000000000001e-06 2024-04-22T18:41:31.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164147+0000", 2024-04-22T18:41:31.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0509999999999996e-06 2024-04-22T18:41:31.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264244+0000", 2024-04-22T18:41:31.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100097063 2024-04-22T18:41:31.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264319+0000", 2024-04-22T18:41:31.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5729000000000006e-05 2024-04-22T18:41:31.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264331+0000", 2024-04-22T18:41:31.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048999999999999e-05 2024-04-22T18:41:31.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243392+0000", 2024-04-22T18:41:31.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97906041099999996 2024-04-22T18:41:31.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:31.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243451+0000", 2024-04-22T18:41:31.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8717e-05 2024-04-22T18:41:31.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558398+0000", 2024-04-22T18:41:31.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31494718500000002 2024-04-22T18:41:31.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558490+0000", 2024-04-22T18:41:31.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2529999999999997e-05 2024-04-22T18:41:31.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559820+0000", 2024-04-22T18:41:31.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013299060000000001 2024-04-22T18:41:31.805 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.805 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.805 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.806 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4196 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:31.806 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:31.806 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.492283649000001, 2024-04-22T18:41:31.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2963829229999999, 2024-04-22T18:41:31.807 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.807 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.808 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4196 2024-04-22T18:41:31.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.808 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:31.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:31.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264275+0000", 2024-04-22T18:41:31.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:31.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264290+0000", 2024-04-22T18:41:31.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5373e-05 2024-04-22T18:41:31.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264300+0000", 2024-04-22T18:41:31.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.747e-06 2024-04-22T18:41:31.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264314+0000", 2024-04-22T18:41:31.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.396e-05 2024-04-22T18:41:31.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364450+0000", 2024-04-22T18:41:31.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013662500000001 2024-04-22T18:41:31.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364459+0000", 2024-04-22T18:41:31.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0710000000000001e-06 2024-04-22T18:41:31.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245690+0000", 2024-04-22T18:41:31.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88123066400000005 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245766+0000", 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5938000000000004e-05 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559874+0000", 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31410827499999999 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559930+0000", 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5367e-05 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.560660+0000", 2024-04-22T18:41:31.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073043800000000003 2024-04-22T18:41:31.813 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.813 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.813 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.813 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4182 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:31.813 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:31.813 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.828600804000001, 2024-04-22T18:41:31.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2059652709999999, 2024-04-22T18:41:31.813 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.813 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.813 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.814 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.814 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.814 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4182 2024-04-22T18:41:31.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.814 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:31.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:31.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927959+0000", 2024-04-22T18:41:31.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927962+0000", 2024-04-22T18:41:31.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4269999999999998e-06 2024-04-22T18:41:31.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927968+0000", 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028037+0000", 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068544 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028078+0000", 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1523999999999997e-05 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028087+0000", 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3140000000000004e-06 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.129951+0000", 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1018649229999999 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.130067+0000", 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115222 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.133925+0000", 2024-04-22T18:41:31.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00385849 2024-04-22T18:41:31.818 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.818 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.818 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.818 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4183 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:31.818 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:31.818 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.728510371999999, 2024-04-22T18:41:31.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1063017610000001, 2024-04-22T18:41:31.818 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.818 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.818 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.818 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.818 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.818 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4183 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028049+0000", 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:31.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028055+0000", 2024-04-22T18:41:31.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-04-22T18:41:31.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028064+0000", 2024-04-22T18:41:31.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2830000000000001e-06 2024-04-22T18:41:31.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028071+0000", 2024-04-22T18:41:31.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2740000000000002e-06 2024-04-22T18:41:31.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028126+0000", 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4771999999999997e-05 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028134+0000", 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5599999999999994e-06 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134109+0000", 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.105974864 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134169+0000", 2024-04-22T18:41:31.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0167999999999997e-05 2024-04-22T18:41:31.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134352+0000", 2024-04-22T18:41:31.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018293699999999999 2024-04-22T18:41:31.822 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.822 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.822 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.822 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4190 4.c 4:3227431c:::smithi08937929-35:head [sparse-read 0~1258562,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:31.822 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.728484868999999, 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1098740030000001, 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4190 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028075+0000", 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028080+0000", 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.561e-06 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028085+0000", 2024-04-22T18:41:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0150000000000003e-06 2024-04-22T18:41:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028090+0000", 2024-04-22T18:41:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.169e-06 2024-04-22T18:41:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028175+0000", 2024-04-22T18:41:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4814999999999993e-05 2024-04-22T18:41:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028181+0000", 2024-04-22T18:41:31.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5490000000000003e-06 2024-04-22T18:41:31.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134400+0000", 2024-04-22T18:41:31.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1062186869999999 2024-04-22T18:41:31.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134552+0000", 2024-04-22T18:41:31.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015210500000000001 2024-04-22T18:41:31.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137950+0000", 2024-04-22T18:41:31.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033978210000000001 2024-04-22T18:41:31.827 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.827 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.827 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.827 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4191 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:31.827 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:31.827 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.728402120999998, 2024-04-22T18:41:31.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.110917953, 2024-04-22T18:41:31.827 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.827 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.827 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4191 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028157+0000", 2024-04-22T18:41:31.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:31.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028164+0000", 2024-04-22T18:41:31.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.511e-06 2024-04-22T18:41:31.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028172+0000", 2024-04-22T18:41:31.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0339999999999997e-06 2024-04-22T18:41:31.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028179+0000", 2024-04-22T18:41:31.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8630000000000002e-06 2024-04-22T18:41:31.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028202+0000", 2024-04-22T18:41:31.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3635999999999999e-05 2024-04-22T18:41:31.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028209+0000", 2024-04-22T18:41:31.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4629999999999998e-06 2024-04-22T18:41:31.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137970+0000", 2024-04-22T18:41:31.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1097613479999999 2024-04-22T18:41:31.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.138034+0000", 2024-04-22T18:41:31.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3551000000000002e-05 2024-04-22T18:41:31.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.139077+0000", 2024-04-22T18:41:31.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001042715 2024-04-22T18:41:31.832 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.832 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.832 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.832 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4192 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:31.832 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:31.832 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.728376821000001, 2024-04-22T18:41:31.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.212555542, 2024-04-22T18:41:31.832 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.832 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.832 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.833 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.833 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.833 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4192 2024-04-22T18:41:31.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.833 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:31.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:31.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028183+0000", 2024-04-22T18:41:31.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028187+0000", 2024-04-22T18:41:31.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6119999999999999e-06 2024-04-22T18:41:31.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028192+0000", 2024-04-22T18:41:31.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3489999999999999e-06 2024-04-22T18:41:31.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028197+0000", 2024-04-22T18:41:31.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7489999999999998e-06 2024-04-22T18:41:31.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028228+0000", 2024-04-22T18:41:31.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1328999999999998e-05 2024-04-22T18:41:31.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028234+0000", 2024-04-22T18:41:31.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1129999999999997e-06 2024-04-22T18:41:31.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239245+0000", 2024-04-22T18:41:31.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.211011032 2024-04-22T18:41:31.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239384+0000", 2024-04-22T18:41:31.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013902000000000001 2024-04-22T18:41:31.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.240739+0000", 2024-04-22T18:41:31.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013550389999999999 2024-04-22T18:41:31.837 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.837 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.837 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.837 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4255 4.9 4:905c3da2:::smithi08937929-39:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:31.837 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.705522+0000", 2024-04-22T18:41:31.837 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.051038713000001, 2024-04-22T18:41:31.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68002623299999998, 2024-04-22T18:41:31.838 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.838 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.838 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.838 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.838 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.838 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4255 2024-04-22T18:41:31.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.838 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705522+0000", 2024-04-22T18:41:31.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705522+0000", 2024-04-22T18:41:31.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705520+0000", 2024-04-22T18:41:31.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:31.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705527+0000", 2024-04-22T18:41:31.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2740000000000004e-06 2024-04-22T18:41:31.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705533+0000", 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9819999999999996e-06 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705537+0000", 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2869999999999998e-06 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705582+0000", 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5247000000000002e-05 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.705734+0000", 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000152029 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.385548+0000", 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.67981406899999997 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4692 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.4974209419999998, 2024-04-22T18:41:31.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80332623700000005, 2024-04-22T18:41:31.842 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.842 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.842 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.842 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.842 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.842 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4692 2024-04-22T18:41:31.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.842 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:31.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:31.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259139+0000", 2024-04-22T18:41:31.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:31.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259145+0000", 2024-04-22T18:41:31.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4860000000000002e-06 2024-04-22T18:41:31.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259151+0000", 2024-04-22T18:41:31.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1160000000000004e-06 2024-04-22T18:41:31.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259156+0000", 2024-04-22T18:41:31.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6489999999999999e-06 2024-04-22T18:41:31.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259191+0000", 2024-04-22T18:41:31.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4737999999999998e-05 2024-04-22T18:41:31.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259197+0000", 2024-04-22T18:41:31.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9160000000000004e-06 2024-04-22T18:41:31.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060062+0000", 2024-04-22T18:41:31.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80086472900000005 2024-04-22T18:41:31.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060225+0000", 2024-04-22T18:41:31.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000162771 2024-04-22T18:41:31.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062466+0000", 2024-04-22T18:41:31.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410569999999999 2024-04-22T18:41:31.846 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.846 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.846 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.846 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4693 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:31.846 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:31.846 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.4973986559999997, 2024-04-22T18:41:31.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80370012999999996, 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4693 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:31.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259161+0000", 2024-04-22T18:41:31.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:31.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259164+0000", 2024-04-22T18:41:31.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-06 2024-04-22T18:41:31.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259169+0000", 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6260000000000003e-06 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259174+0000", 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7659999999999998e-06 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259231+0000", 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6827000000000003e-05 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259239+0000", 2024-04-22T18:41:31.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6889999999999993e-06 2024-04-22T18:41:31.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062492+0000", 2024-04-22T18:41:31.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80325246699999997 2024-04-22T18:41:31.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062573+0000", 2024-04-22T18:41:31.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1508000000000006e-05 2024-04-22T18:41:31.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062862+0000", 2024-04-22T18:41:31.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028893499999999998 2024-04-22T18:41:31.851 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.851 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.851 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.851 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4694 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:31.851 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:31.851 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.4973823749999999, 2024-04-22T18:41:31.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.90474754099999999, 2024-04-22T18:41:31.851 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.851 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:31.851 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.851 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.852 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.852 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4694 2024-04-22T18:41:31.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.852 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:31.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:31.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:31.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:31.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259182+0000", 2024-04-22T18:41:31.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8700000000000002e-06 2024-04-22T18:41:31.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259187+0000", 2024-04-22T18:41:31.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1669999999999998e-06 2024-04-22T18:41:31.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259191+0000", 2024-04-22T18:41:31.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6210000000000003e-06 2024-04-22T18:41:31.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.359347+0000", 2024-04-22T18:41:31.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100155518 2024-04-22T18:41:31.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.359356+0000", 2024-04-22T18:41:31.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6519999999999995e-06 2024-04-22T18:41:31.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163126+0000", 2024-04-22T18:41:31.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.803770816 2024-04-22T18:41:31.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:31.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163217+0000", 2024-04-22T18:41:31.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0598000000000002e-05 2024-04-22T18:41:31.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163926+0000", 2024-04-22T18:41:31.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000708856 2024-04-22T18:41:31.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.856 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.856 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4701 4.1b 4:d923883e:::smithi08937929-16:head [set-chunk in=77b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:31.856 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.408330+0000", 2024-04-22T18:41:31.856 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.3482306700000004, 2024-04-22T18:41:31.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.65183720300000003, 2024-04-22T18:41:31.856 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:31.856 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:31.856 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:31.856 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4701 2024-04-22T18:41:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408330+0000", 2024-04-22T18:41:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408330+0000", 2024-04-22T18:41:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408326+0000", 2024-04-22T18:41:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999962 2024-04-22T18:41:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408343+0000", 2024-04-22T18:41:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6731999999999998e-05 2024-04-22T18:41:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408366+0000", 2024-04-22T18:41:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3567e-05 2024-04-22T18:41:31.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:31.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.408379+0000", 2024-04-22T18:41:31.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2775999999999999e-05 2024-04-22T18:41:31.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.557823+0000", 2024-04-22T18:41:31.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.14944426899999999 2024-04-22T18:41:31.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:31.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.557837+0000", 2024-04-22T18:41:31.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:41:31.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.858557+0000", 2024-04-22T18:41:31.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30072007899999997 2024-04-22T18:41:31.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:31.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.858706+0000", 2024-04-22T18:41:31.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014876000000000001 2024-04-22T18:41:31.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:31.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060052+0000", 2024-04-22T18:41:31.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.201345943 2024-04-22T18:41:31.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:31.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:31.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:31.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060167+0000", 2024-04-22T18:41:31.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115396 2024-04-22T18:41:31.861 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.861 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.861 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.862 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:31.862 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:31.862 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:31.863 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:31.864 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:31.865+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:31.964 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:31.966+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:31.969 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:31.969 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:31.969 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:31.969 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:31.978 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:32.065 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:32.066+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:32.081 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:32.081 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:32.081 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:32.081 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:32.081 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:32.090 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:32.173 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:32.175+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:32.234 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:41:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 147.773347237, 2024-04-22T18:41:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:41:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:41:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:41:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:41:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:41:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:41:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:41:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:41:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:41:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:41:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:41:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:41:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:41:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:41:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:41:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:41:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:41:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 143.89667341500001, 2024-04-22T18:41:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:41:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:41:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:41:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:41:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:41:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:41:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:41:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:41:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:41:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:41:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:41:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:41:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:41:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:41:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:41:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:41:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:41:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 87.691869892, 2024-04-22T18:41:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:41:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:41:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:41:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:41:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:41:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:41:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:41:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:41:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:41:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:41:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:41:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:41:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:41:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:41:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:41:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:41:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:41:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3451 4.d 4:bee1184f:::smithi08937929-7:head [read 0~3613505,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.307339564999999, 2024-04-22T18:41:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20164568099999999, 2024-04-22T18:41:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3451 2024-04-22T18:41:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927183+0000", 2024-04-22T18:41:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1110000000000001e-06 2024-04-22T18:41:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927188+0000", 2024-04-22T18:41:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1569999999999999e-06 2024-04-22T18:41:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927192+0000", 2024-04-22T18:41:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4710000000000001e-06 2024-04-22T18:41:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027382+0000", 2024-04-22T18:41:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100189341 2024-04-22T18:41:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:41:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027387+0000", 2024-04-22T18:41:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8400000000000002e-06 2024-04-22T18:41:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127638+0000", 2024-04-22T18:41:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10025102499999999 2024-04-22T18:41:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127740+0000", 2024-04-22T18:41:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102749 2024-04-22T18:41:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.128825+0000", 2024-04-22T18:41:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001084593 2024-04-22T18:41:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:151 3.0 e114/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 114'213, mlcod=110'210)", 2024-04-22T18:41:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.716871482000002, 2024-04-22T18:41:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20074001299999999, 2024-04-22T18:41:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517640+0000", 2024-04-22T18:41:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999928 2024-04-22T18:41:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517652+0000", 2024-04-22T18:41:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1824000000000001e-05 2024-04-22T18:41:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517660+0000", 2024-04-22T18:41:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1419999999999993e-06 2024-04-22T18:41:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617729+0000", 2024-04-22T18:41:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068723 2024-04-22T18:41:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617770+0000", 2024-04-22T18:41:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1391999999999999e-05 2024-04-22T18:41:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617793+0000", 2024-04-22T18:41:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2807999999999999e-05 2024-04-22T18:41:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718361+0000", 2024-04-22T18:41:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100567473 2024-04-22T18:41:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718387+0000", 2024-04-22T18:41:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6826e-05 2024-04-22T18:41:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3862 4.d 4:bee1184f:::smithi08937929-7:head [write 3301257~278667] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.971969211999998, 2024-04-22T18:41:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301875958, 2024-04-22T18:41:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3862 2024-04-22T18:41:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262548+0000", 2024-04-22T18:41:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262622+0000", 2024-04-22T18:41:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3747999999999999e-05 2024-04-22T18:41:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262632+0000", 2024-04-22T18:41:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0767e-05 2024-04-22T18:41:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362705+0000", 2024-04-22T18:41:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073157 2024-04-22T18:41:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:41:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2846999999999997e-05 2024-04-22T18:41:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362853+0000", 2024-04-22T18:41:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4498000000000002e-05 2024-04-22T18:41:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362992+0000", 2024-04-22T18:41:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013879699999999999 2024-04-22T18:41:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.366534+0000", 2024-04-22T18:41:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035427969999999999 2024-04-22T18:41:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564353+0000", 2024-04-22T18:41:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.197818893 2024-04-22T18:41:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564385+0000", 2024-04-22T18:41:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1717000000000001e-05 2024-04-22T18:41:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564426+0000", 2024-04-22T18:41:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0654000000000001e-05 2024-04-22T18:41:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3863 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3579924] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.871801820000002, 2024-04-22T18:41:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20206560900000001, 2024-04-22T18:41:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3863 2024-04-22T18:41:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362715+0000", 2024-04-22T18:41:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362725+0000", 2024-04-22T18:41:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.006e-05 2024-04-22T18:41:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362732+0000", 2024-04-22T18:41:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6780000000000003e-06 2024-04-22T18:41:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362737+0000", 2024-04-22T18:41:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2089999999999996e-06 2024-04-22T18:41:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462875+0000", 2024-04-22T18:41:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013834100000001 2024-04-22T18:41:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462963+0000", 2024-04-22T18:41:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7792000000000002e-05 2024-04-22T18:41:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463127+0000", 2024-04-22T18:41:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000163688 2024-04-22T18:41:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463692+0000", 2024-04-22T18:41:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056466299999999997 2024-04-22T18:41:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564720+0000", 2024-04-22T18:41:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10102826199999999 2024-04-22T18:41:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564752+0000", 2024-04-22T18:41:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1936000000000001e-05 2024-04-22T18:41:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564783+0000", 2024-04-22T18:41:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0919e-05 2024-04-22T18:41:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3864 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.871768435999996, 2024-04-22T18:41:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20218423599999999, 2024-04-22T18:41:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3864 2024-04-22T18:41:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362753+0000", 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6989999999999999e-06 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:41:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:41:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462871+0000", 2024-04-22T18:41:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100113641 2024-04-22T18:41:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463385+0000", 2024-04-22T18:41:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051307100000000003 2024-04-22T18:41:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463457+0000", 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2778e-05 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564814+0000", 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10135696299999999 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564868+0000", 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3872000000000002e-05 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564935+0000", 2024-04-22T18:41:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6545999999999996e-05 2024-04-22T18:41:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4030 4.d 4:bee1184f:::smithi08937929-7:head [write 615222~472039] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.275319533999998, 2024-04-22T18:41:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109028217, 2024-04-22T18:41:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.274 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:32.274 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:32.274 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:32.274 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4030 2024-04-22T18:41:32.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.274 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:32.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:32.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959177+0000", 2024-04-22T18:41:32.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999776 2024-04-22T18:41:32.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959331+0000", 2024-04-22T18:41:32.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015334600000000001 2024-04-22T18:41:32.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959358+0000", 2024-04-22T18:41:32.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7220999999999999e-05 2024-04-22T18:41:32.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959371+0000", 2024-04-22T18:41:32.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3489000000000001e-05 2024-04-22T18:41:32.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059523+0000", 2024-04-22T18:41:32.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100152059 2024-04-22T18:41:32.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059588+0000", 2024-04-22T18:41:32.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4493999999999996e-05 2024-04-22T18:41:32.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:32.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059694+0000", 2024-04-22T18:41:32.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010584199999999999 2024-04-22T18:41:32.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068082+0000", 2024-04-22T18:41:32.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083880589999999998 2024-04-22T18:41:32.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068138+0000", 2024-04-22T18:41:32.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6196999999999997e-05 2024-04-22T18:41:32.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068185+0000", 2024-04-22T18:41:32.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7318000000000003e-05 2024-04-22T18:41:32.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068228+0000", 2024-04-22T18:41:32.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2376999999999998e-05 2024-04-22T18:41:32.279 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.279 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.279 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.279 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4031 4.d 4:bee1184f:::smithi08937929-7:head [write 1580652~489442] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:32.279 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:32.279 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.270018542000003, 2024-04-22T18:41:32.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20571598299999999, 2024-04-22T18:41:32.279 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.279 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.280 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:32.280 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:32.280 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:32.280 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4031 2024-04-22T18:41:32.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.280 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:32.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:32.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964480+0000", 2024-04-22T18:41:32.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999795 2024-04-22T18:41:32.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964628+0000", 2024-04-22T18:41:32.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148109 2024-04-22T18:41:32.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964651+0000", 2024-04-22T18:41:32.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3215999999999999e-05 2024-04-22T18:41:32.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964664+0000", 2024-04-22T18:41:32.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2673e-05 2024-04-22T18:41:32.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061135+0000", 2024-04-22T18:41:32.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096470822999999997 2024-04-22T18:41:32.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061228+0000", 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3459000000000001e-05 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061376+0000", 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148128 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068085+0000", 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.006708632 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170038+0000", 2024-04-22T18:41:32.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101952716 2024-04-22T18:41:32.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170173+0000", 2024-04-22T18:41:32.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013508699999999999 2024-04-22T18:41:32.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170216+0000", 2024-04-22T18:41:32.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3498999999999998e-05 2024-04-22T18:41:32.284 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.285 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.285 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4032 4.d 4:bee1184f:::smithi08937929-7:head [write 2768826~626587] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:32.285 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:32.285 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.262813434999998, 2024-04-22T18:41:32.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19941629799999999, 2024-04-22T18:41:32.285 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.285 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.285 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:32.285 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:32.285 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:32.286 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4032 2024-04-22T18:41:32.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.286 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:32.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:32.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971703+0000", 2024-04-22T18:41:32.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:32.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971996+0000", 2024-04-22T18:41:32.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029351300000000002 2024-04-22T18:41:32.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972012+0000", 2024-04-22T18:41:32.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5569e-05 2024-04-22T18:41:32.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972019+0000", 2024-04-22T18:41:32.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7950000000000008e-06 2024-04-22T18:41:32.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.063940+0000", 2024-04-22T18:41:32.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091920162 2024-04-22T18:41:32.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064021+0000", 2024-04-22T18:41:32.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1865000000000003e-05 2024-04-22T18:41:32.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:32.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064145+0000", 2024-04-22T18:41:32.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123893 2024-04-22T18:41:32.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068317+0000", 2024-04-22T18:41:32.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041720799999999999 2024-04-22T18:41:32.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171042+0000", 2024-04-22T18:41:32.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102724893 2024-04-22T18:41:32.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171085+0000", 2024-04-22T18:41:32.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2619e-05 2024-04-22T18:41:32.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171122+0000", 2024-04-22T18:41:32.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6838999999999997e-05 2024-04-22T18:41:32.291 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.291 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.291 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.291 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4033 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3815222] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:32.291 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:32.291 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.262489952999999, 2024-04-22T18:41:32.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297586038, 2024-04-22T18:41:32.291 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.291 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.291 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:32.291 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:32.291 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:32.292 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4033 2024-04-22T18:41:32.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.292 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:32.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:32.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972028+0000", 2024-04-22T18:41:32.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972032+0000", 2024-04-22T18:41:32.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8330000000000004e-06 2024-04-22T18:41:32.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972035+0000", 2024-04-22T18:41:32.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:41:32.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972038+0000", 2024-04-22T18:41:32.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9579999999999999e-06 2024-04-22T18:41:32.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067511+0000", 2024-04-22T18:41:32.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.095472308000000006 2024-04-22T18:41:32.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067607+0000", 2024-04-22T18:41:32.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6546999999999996e-05 2024-04-22T18:41:32.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:32.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067776+0000", 2024-04-22T18:41:32.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000169202 2024-04-22T18:41:32.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068470+0000", 2024-04-22T18:41:32.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069403100000000001 2024-04-22T18:41:32.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269540+0000", 2024-04-22T18:41:32.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20106934000000001 2024-04-22T18:41:32.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269574+0000", 2024-04-22T18:41:32.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4332000000000003e-05 2024-04-22T18:41:32.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269615+0000", 2024-04-22T18:41:32.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0988000000000001e-05 2024-04-22T18:41:32.296 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.296 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.296 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.296 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4034 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:32.297 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:32.297 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.262478315999999, 2024-04-22T18:41:32.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39788621000000002, 2024-04-22T18:41:32.297 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.297 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:32.297 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:32.297 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:32.297 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:32.297 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4034 2024-04-22T18:41:32.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.297 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:32.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:32.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972040+0000", 2024-04-22T18:41:32.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972042+0000", 2024-04-22T18:41:32.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.519e-06 2024-04-22T18:41:32.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972045+0000", 2024-04-22T18:41:32.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:41:32.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972048+0000", 2024-04-22T18:41:32.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.773e-06 2024-04-22T18:41:32.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068266+0000", 2024-04-22T18:41:32.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096218029999999996 2024-04-22T18:41:32.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:32.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068308+0000", 2024-04-22T18:41:32.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.21e-05 2024-04-22T18:41:32.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369703+0000", 2024-04-22T18:41:32.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301395578 2024-04-22T18:41:32.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369802+0000", 2024-04-22T18:41:32.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8788000000000004e-05 2024-04-22T18:41:32.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369927+0000", 2024-04-22T18:41:32.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012478200000000001 2024-04-22T18:41:32.301 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.301 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.301 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.302 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(4.1 124/123 [PushOp(4:83c955bf:::smithi08937929-28:head, version: 115'157, data_included: [1890685~548185,2699879~469889], data_size: 1018074, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:83c955bf:::smithi08937929-28:head@115'157, size: 3630345, copy_subset: [0~3630345], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3630345, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1890685, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-22T18:41:32.302 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:32.302 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.595910207999999, 2024-04-22T18:41:32.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20293218800000001, 2024-04-22T18:41:32.302 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.302 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:32.302 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:32.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:32.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638608+0000", 2024-04-22T18:41:32.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:32.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639519+0000", 2024-04-22T18:41:32.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091196199999999999 2024-04-22T18:41:32.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639537+0000", 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7455000000000001e-05 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.739639+0000", 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100101942 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840271+0000", 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10063235299999999 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840281+0000", 2024-04-22T18:41:32.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6509999999999996e-06 2024-04-22T18:41:32.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.841541+0000", 2024-04-22T18:41:32.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012600230000000001 2024-04-22T18:41:32.305 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.305 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.305 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.305 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4205 4.1 4:83c955bf:::smithi08937929-28:head [read 3065344~859648] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:32.305 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:32.305 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.306988366999999, 2024-04-22T18:41:32.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.303433565, 2024-04-22T18:41:32.306 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.306 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:32.306 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:32.306 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:32.306 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:32.306 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4205 2024-04-22T18:41:32.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.306 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:32.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927528+0000", 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927535+0000", 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1049999999999997e-06 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927550+0000", 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4365000000000001e-05 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927560+0000", 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0360000000000001e-05 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028542+0000", 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100981837 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:32.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028551+0000", 2024-04-22T18:41:32.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6260000000000001e-06 2024-04-22T18:41:32.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229700+0000", 2024-04-22T18:41:32.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.201149453 2024-04-22T18:41:32.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229861+0000", 2024-04-22T18:41:32.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160605 2024-04-22T18:41:32.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.230964+0000", 2024-04-22T18:41:32.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00110351 2024-04-22T18:41:32.310 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.310 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.310 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.310 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:359 3.0 e133/63 3:351d2474:::ba270d71d9c7ccc8d5c74eded70a7073f0e08c1ce33f7199f7c9db1158888734:head v 133'233, mlcod=132'228)", 2024-04-22T18:41:32.310 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:32.310 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.33219394, 2024-04-22T18:41:32.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20122315599999999, 2024-04-22T18:41:32.311 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.311 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.311 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:32.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:32.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902324+0000", 2024-04-22T18:41:32.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:32.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902347+0000", 2024-04-22T18:41:32.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3577000000000001e-05 2024-04-22T18:41:32.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902356+0000", 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8179999999999993e-06 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.002447+0000", 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100090255 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102568+0000", 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100121611 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102592+0000", 2024-04-22T18:41:32.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3515000000000001e-05 2024-04-22T18:41:32.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103514+0000", 2024-04-22T18:41:32.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000922174 2024-04-22T18:41:32.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103548+0000", 2024-04-22T18:41:32.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4316999999999997e-05 2024-04-22T18:41:32.314 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:4289 3.0 e134/63 3:2a3d8f37:::9341e24c4a92f7ec719282d2faa5711f9644dee45a94ba5b4c4c1071ddc0e536:head v 134'235, mlcod=133'233)", 2024-04-22T18:41:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.387624427999999, 2024-04-22T18:41:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10669205499999999, 2024-04-22T18:41:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:32.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:32.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846891+0000", 2024-04-22T18:41:32.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:32.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847884+0000", 2024-04-22T18:41:32.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00099224800000000004 2024-04-22T18:41:32.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847900+0000", 2024-04-22T18:41:32.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6154000000000002e-05 2024-04-22T18:41:32.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.947999+0000", 2024-04-22T18:41:32.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009889199999999 2024-04-22T18:41:32.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948046+0000", 2024-04-22T18:41:32.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7617000000000002e-05 2024-04-22T18:41:32.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948072+0000", 2024-04-22T18:41:32.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5559999999999999e-05 2024-04-22T18:41:32.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953539+0000", 2024-04-22T18:41:32.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054668599999999996 2024-04-22T18:41:32.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953587+0000", 2024-04-22T18:41:32.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7781999999999998e-05 2024-04-22T18:41:32.319 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.319 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.319 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.319 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.7.13:94 3.0 e147/63 3:2812810a:::007b860ed7f2dc72f975d8732f59ce14b4563df44fd56cdf3a173500adaeb557:head v 147'260, mlcod=147'257)", 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8012312220000002, 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.200869664, 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433287+0000", 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433295+0000", 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9929999999999995e-06 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433302+0000", 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2740000000000002e-06 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533368+0000", 2024-04-22T18:41:32.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100066239 2024-04-22T18:41:32.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533415+0000", 2024-04-22T18:41:32.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6669000000000003e-05 2024-04-22T18:41:32.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533442+0000", 2024-04-22T18:41:32.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7143000000000001e-05 2024-04-22T18:41:32.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.634126+0000", 2024-04-22T18:41:32.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10068434900000001 2024-04-22T18:41:32.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.634157+0000", 2024-04-22T18:41:32.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1016000000000003e-05 2024-04-22T18:41:32.323 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.323 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.323 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.323 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:198 3.0 e147/63 3:2be6c676:::11b6849070c44d718872a26a0dc7d7224d7a6a98b28e0fde07a9f66143e32de6:head v 147'266, mlcod=147'257)", 2024-04-22T18:41:32.324 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:41:32.324 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.7004591280000003, 2024-04-22T18:41:32.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106653649, 2024-04-22T18:41:32.324 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.324 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.324 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:41:32.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:41:32.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534059+0000", 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534082+0000", 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2291999999999999e-05 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534086+0000", 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1019999999999999e-06 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534088+0000", 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.525e-06 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.635447+0000", 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101358581 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.635462+0000", 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5327000000000001e-05 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.640689+0000", 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052266329999999996 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.640713+0000", 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4630999999999998e-05 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:200 3.0 e147/63 3:19f9e72c:::45a155ef5bb5a968c7feaf289313c887c41e2de64b61f8816d05ad72d02346d9:head v 147'267, mlcod=147'257)", 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.7004283709999992, 2024-04-22T18:41:32.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109105273, 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534090+0000", 2024-04-22T18:41:32.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534228+0000", 2024-04-22T18:41:32.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013768600000000001 2024-04-22T18:41:32.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534232+0000", 2024-04-22T18:41:32.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4900000000000002e-06 2024-04-22T18:41:32.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534235+0000", 2024-04-22T18:41:32.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.61e-06 2024-04-22T18:41:32.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.641544+0000", 2024-04-22T18:41:32.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107308717 2024-04-22T18:41:32.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.641557+0000", 2024-04-22T18:41:32.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2811e-05 2024-04-22T18:41:32.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.643173+0000", 2024-04-22T18:41:32.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00161608 2024-04-22T18:41:32.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.643196+0000", 2024-04-22T18:41:32.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3136e-05 2024-04-22T18:41:32.331 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.331 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.331 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.332 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.332 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.332 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:32.333 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:32.334 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:32.334+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:32.433 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:32.434+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:32.477 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:32.477 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:32.485 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:32.570 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:32.571+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:32.627 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:32.627 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:32.628 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:32.628 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:32.628 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:32.635 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:32.636+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:32.636 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:32.658 INFO:teuthology.orchestra.run.smithi089.stderr:set pool 4 pg_num to 36 2024-04-22T18:41:32.736 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:32.737+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:32.762 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:32.762 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:32.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:32.763 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:32.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.763 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:289 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:32.763 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:32.763 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.121299747999998, 2024-04-22T18:41:32.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9176801189999999, 2024-04-22T18:41:32.763 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.763 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.763 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:32.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:32.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640723+0000", 2024-04-22T18:41:32.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:32.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640732+0000", 2024-04-22T18:41:32.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5809999999999997e-06 2024-04-22T18:41:32.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640744+0000", 2024-04-22T18:41:32.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.261e-05 2024-04-22T18:41:32.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640753+0000", 2024-04-22T18:41:32.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7460000000000006e-06 2024-04-22T18:41:32.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642189+0000", 2024-04-22T18:41:32.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001435551 2024-04-22T18:41:32.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:32.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642200+0000", 2024-04-22T18:41:32.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1389999999999999e-05 2024-04-22T18:41:32.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.847015+0000", 2024-04-22T18:41:32.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048149160000001 2024-04-22T18:41:32.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.848217+0000", 2024-04-22T18:41:32.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012020659999999999 2024-04-22T18:41:32.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351823+0000", 2024-04-22T18:41:32.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50360565099999999 2024-04-22T18:41:32.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351907+0000", 2024-04-22T18:41:32.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4141999999999995e-05 2024-04-22T18:41:32.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:32.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352051+0000", 2024-04-22T18:41:32.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014386600000000001 2024-04-22T18:41:32.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453617+0000", 2024-04-22T18:41:32.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156649500000001 2024-04-22T18:41:32.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558297+0000", 2024-04-22T18:41:32.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10468000399999999 2024-04-22T18:41:32.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558331+0000", 2024-04-22T18:41:32.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3518e-05 2024-04-22T18:41:32.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558405+0000", 2024-04-22T18:41:32.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4221000000000006e-05 2024-04-22T18:41:32.769 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.769 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.769 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.770 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:1 3.2 3:6d90b903:::64b5942574fb36ca1b9a2be186c6b740edf58e989056a2f88b83cd280451a438:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:32.770 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:32.770 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 89.589998788000003, 2024-04-22T18:41:32.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.285534422, 2024-04-22T18:41:32.770 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.770 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.770 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:32.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:32.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172024+0000", 2024-04-22T18:41:32.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:32.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173091+0000", 2024-04-22T18:41:32.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010677569999999999 2024-04-22T18:41:32.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173103+0000", 2024-04-22T18:41:32.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1241e-05 2024-04-22T18:41:32.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173109+0000", 2024-04-22T18:41:32.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4300000000000003e-06 2024-04-22T18:41:32.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173168+0000", 2024-04-22T18:41:32.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9011000000000001e-05 2024-04-22T18:41:32.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:32.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173177+0000", 2024-04-22T18:41:32.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-04-22T18:41:32.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948501+0000", 2024-04-22T18:41:32.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77532387199999997 2024-04-22T18:41:32.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948912+0000", 2024-04-22T18:41:32.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041183000000000002 2024-04-22T18:41:32.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950438+0000", 2024-04-22T18:41:32.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015253199999999999 2024-04-22T18:41:32.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950488+0000", 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0285999999999999e-05 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950601+0000", 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011244899999999999 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954048+0000", 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003447313 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457373+0000", 2024-04-22T18:41:32.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50332465299999996 2024-04-22T18:41:32.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457430+0000", 2024-04-22T18:41:32.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7574999999999997e-05 2024-04-22T18:41:32.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457560+0000", 2024-04-22T18:41:32.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013023900000000001 2024-04-22T18:41:32.776 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.776 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:8 3.2 3:5f64590d:::7c47933f331d40689081b2bd8f9d7f7bdae1be1fdccb52d696047a3969e902b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 89.588906418999997, 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3854200569999999, 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:32.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173117+0000", 2024-04-22T18:41:32.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:32.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173393+0000", 2024-04-22T18:41:32.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000275278 2024-04-22T18:41:32.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173399+0000", 2024-04-22T18:41:32.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4590000000000003e-06 2024-04-22T18:41:32.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173406+0000", 2024-04-22T18:41:32.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2450000000000001e-06 2024-04-22T18:41:32.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173795+0000", 2024-04-22T18:41:32.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038912400000000002 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173804+0000", 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.949506+0000", 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77570185700000005 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950166+0000", 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066022800000000001 2024-04-22T18:41:32.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352843+0000", 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40267661799999999 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352911+0000", 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8126000000000005e-05 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.353031+0000", 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119919 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453622+0000", 2024-04-22T18:41:32.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100590793 2024-04-22T18:41:32.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558446+0000", 2024-04-22T18:41:32.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104824073 2024-04-22T18:41:32.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558479+0000", 2024-04-22T18:41:32.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3362999999999999e-05 2024-04-22T18:41:32.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558538+0000", 2024-04-22T18:41:32.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9046999999999999e-05 2024-04-22T18:41:32.783 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.783 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.783 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.783 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:10 3.2 3:518c37f9:::851eca84dafbf2f780c4f07291110c453349bdc9c623c17bdcd26e27badb30ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:32.783 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:32.783 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 89.588612826000002, 2024-04-22T18:41:32.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3852527729999999, 2024-04-22T18:41:32.783 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.783 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.783 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:32.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:32.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173411+0000", 2024-04-22T18:41:32.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:32.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173749+0000", 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000338029 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173755+0000", 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0530000000000003e-06 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173759+0000", 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8190000000000002e-06 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173855+0000", 2024-04-22T18:41:32.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5927999999999994e-05 2024-04-22T18:41:32.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:32.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173862+0000", 2024-04-22T18:41:32.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4449999999999999e-06 2024-04-22T18:41:32.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950835+0000", 2024-04-22T18:41:32.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77697248299999999 2024-04-22T18:41:32.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951632+0000", 2024-04-22T18:41:32.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079752299999999996 2024-04-22T18:41:32.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355407+0000", 2024-04-22T18:41:32.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40377480500000001 2024-04-22T18:41:32.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355462+0000", 2024-04-22T18:41:32.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4577999999999998e-05 2024-04-22T18:41:32.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:32.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355571+0000", 2024-04-22T18:41:32.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108964 2024-04-22T18:41:32.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453625+0000", 2024-04-22T18:41:32.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098054397000000001 2024-04-22T18:41:32.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558574+0000", 2024-04-22T18:41:32.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104948571 2024-04-22T18:41:32.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558602+0000", 2024-04-22T18:41:32.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7966999999999998e-05 2024-04-22T18:41:32.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558665+0000", 2024-04-22T18:41:32.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3032000000000002e-05 2024-04-22T18:41:32.789 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.789 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.789 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.790 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:11 3.2 3:718aa8aa:::9603c4f8e269a436b2f0bf376c5374e77bdd7cf58fccae60e4e65ac4696806d4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:32.790 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:32.790 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 89.588261208000006, 2024-04-22T18:41:32.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4852604440000001, 2024-04-22T18:41:32.790 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.790 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.790 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:32.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:32.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:32.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173813+0000", 2024-04-22T18:41:32.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0655999999999997e-05 2024-04-22T18:41:32.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173819+0000", 2024-04-22T18:41:32.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.468e-06 2024-04-22T18:41:32.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173822+0000", 2024-04-22T18:41:32.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4879999999999999e-06 2024-04-22T18:41:32.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173983+0000", 2024-04-22T18:41:32.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160519 2024-04-22T18:41:32.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:32.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173991+0000", 2024-04-22T18:41:32.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6490000000000008e-06 2024-04-22T18:41:32.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951688+0000", 2024-04-22T18:41:32.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77769743800000002 2024-04-22T18:41:32.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952419+0000", 2024-04-22T18:41:32.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073067499999999997 2024-04-22T18:41:32.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355901+0000", 2024-04-22T18:41:32.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40348237599999998 2024-04-22T18:41:32.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355953+0000", 2024-04-22T18:41:32.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1477000000000002e-05 2024-04-22T18:41:32.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:32.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356060+0000", 2024-04-22T18:41:32.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107114 2024-04-22T18:41:32.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453629+0000", 2024-04-22T18:41:32.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097569127000000005 2024-04-22T18:41:32.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658807+0000", 2024-04-22T18:41:32.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20517829800000001 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658876+0000", 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8454000000000006e-05 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659024+0000", 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148338 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.796 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:12 3.2 3:475e1cd9:::efaab2bd13543cdbb143e869e20eb0ec424f7f5202707859b241ed91d4674936:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:32.797 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:32.797 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 89.588198263999999, 2024-04-22T18:41:32.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4853405259999999, 2024-04-22T18:41:32.797 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.797 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.797 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:32.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:32.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:32.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173940+0000", 2024-04-22T18:41:32.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011465500000000001 2024-04-22T18:41:32.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173946+0000", 2024-04-22T18:41:32.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5289999999999999e-06 2024-04-22T18:41:32.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173950+0000", 2024-04-22T18:41:32.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8290000000000001e-06 2024-04-22T18:41:32.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174277+0000", 2024-04-22T18:41:32.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032761299999999998 2024-04-22T18:41:32.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:32.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174285+0000", 2024-04-22T18:41:32.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.52e-06 2024-04-22T18:41:32.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952455+0000", 2024-04-22T18:41:32.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77817000700000005 2024-04-22T18:41:32.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954035+0000", 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001580078 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356379+0000", 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40234440399999999 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356428+0000", 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8832999999999998e-05 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:32.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356532+0000", 2024-04-22T18:41:32.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103235 2024-04-22T18:41:32.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453632+0000", 2024-04-22T18:41:32.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097100346000000004 2024-04-22T18:41:32.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659069+0000", 2024-04-22T18:41:32.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20543763800000001 2024-04-22T18:41:32.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659105+0000", 2024-04-22T18:41:32.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5240000000000001e-05 2024-04-22T18:41:32.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659167+0000", 2024-04-22T18:41:32.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2207999999999998e-05 2024-04-22T18:41:32.803 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.803 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.803 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.803 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:16 3.2 3:47c76cc8:::c00d18458eb20c0c0aa9c612cf9793a68445ab0ea7b9e650ecd0072c60a96c0f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:32.803 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:32.804 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 89.588070305000002, 2024-04-22T18:41:32.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.485347744, 2024-04-22T18:41:32.804 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.804 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.804 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:32.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:32.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174234+0000", 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028047599999999999 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174240+0000", 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2079999999999997e-06 2024-04-22T18:41:32.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174244+0000", 2024-04-22T18:41:32.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5810000000000001e-06 2024-04-22T18:41:32.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174670+0000", 2024-04-22T18:41:32.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042640000000000001 2024-04-22T18:41:32.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:32.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174675+0000", 2024-04-22T18:41:32.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8709999999999996e-06 2024-04-22T18:41:32.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054168+0000", 2024-04-22T18:41:32.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87949257300000006 2024-04-22T18:41:32.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054877+0000", 2024-04-22T18:41:32.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070884300000000002 2024-04-22T18:41:32.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356836+0000", 2024-04-22T18:41:32.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30195926000000001 2024-04-22T18:41:32.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356887+0000", 2024-04-22T18:41:32.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0767999999999999e-05 2024-04-22T18:41:32.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:32.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356991+0000", 2024-04-22T18:41:32.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104072 2024-04-22T18:41:32.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453636+0000", 2024-04-22T18:41:32.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096644968999999997 2024-04-22T18:41:32.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659207+0000", 2024-04-22T18:41:32.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20557154699999999 2024-04-22T18:41:32.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659240+0000", 2024-04-22T18:41:32.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2665e-05 2024-04-22T18:41:32.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659302+0000", 2024-04-22T18:41:32.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2234000000000001e-05 2024-04-22T18:41:32.810 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.810 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.810 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.810 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:17 3.2 3:6ef31daa:::1cf4ff496c4480e58510f607481d976b47d84868e2c54016273c38baf590f3b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:32.810 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:32.810 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 89.587776024999997, 2024-04-22T18:41:32.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4851703729999999, 2024-04-22T18:41:32.811 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.811 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.811 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:32.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:32.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174248+0000", 2024-04-22T18:41:32.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174626+0000", 2024-04-22T18:41:32.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037839699999999999 2024-04-22T18:41:32.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174633+0000", 2024-04-22T18:41:32.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0990000000000004e-06 2024-04-22T18:41:32.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174636+0000", 2024-04-22T18:41:32.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5540000000000002e-06 2024-04-22T18:41:32.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174790+0000", 2024-04-22T18:41:32.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153807 2024-04-22T18:41:32.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:32.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174793+0000", 2024-04-22T18:41:32.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4970000000000001e-06 2024-04-22T18:41:32.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054902+0000", 2024-04-22T18:41:32.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88010886300000002 2024-04-22T18:41:32.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055366+0000", 2024-04-22T18:41:32.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046336199999999999 2024-04-22T18:41:32.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.452973+0000", 2024-04-22T18:41:32.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39760783999999999 2024-04-22T18:41:32.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453046+0000", 2024-04-22T18:41:32.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2374999999999997e-05 2024-04-22T18:41:32.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:32.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453193+0000", 2024-04-22T18:41:32.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014722499999999999 2024-04-22T18:41:32.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554698+0000", 2024-04-22T18:41:32.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101505096 2024-04-22T18:41:32.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659337+0000", 2024-04-22T18:41:32.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104639236 2024-04-22T18:41:32.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659365+0000", 2024-04-22T18:41:32.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7594e-05 2024-04-22T18:41:32.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659419+0000", 2024-04-22T18:41:32.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4058999999999999e-05 2024-04-22T18:41:32.817 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.817 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.817 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.817 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:18 3.2 3:792a4ed0:::e1c814961b3d1531fb5ed475b3d496a0405c3a819b68a92ff9b2391fe912370f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:32.817 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:32.817 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 89.587379768999995, 2024-04-22T18:41:32.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.585036954, 2024-04-22T18:41:32.818 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.818 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.818 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:32.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:32.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174640+0000", 2024-04-22T18:41:32.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-04-22T18:41:32.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174758+0000", 2024-04-22T18:41:32.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011819099999999999 2024-04-22T18:41:32.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174764+0000", 2024-04-22T18:41:32.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6729999999999997e-06 2024-04-22T18:41:32.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174767+0000", 2024-04-22T18:41:32.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3830000000000001e-06 2024-04-22T18:41:32.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373669+0000", 2024-04-22T18:41:32.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198901989 2024-04-22T18:41:32.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:32.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373686+0000", 2024-04-22T18:41:32.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7286e-05 2024-04-22T18:41:32.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055408+0000", 2024-04-22T18:41:32.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68172133400000001 2024-04-22T18:41:32.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055938+0000", 2024-04-22T18:41:32.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053016599999999999 2024-04-22T18:41:32.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453668+0000", 2024-04-22T18:41:32.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39773030799999998 2024-04-22T18:41:32.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453717+0000", 2024-04-22T18:41:32.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8884000000000002e-05 2024-04-22T18:41:32.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:32.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453823+0000", 2024-04-22T18:41:32.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105791 2024-04-22T18:41:32.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554715+0000", 2024-04-22T18:41:32.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100891628 2024-04-22T18:41:32.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759439+0000", 2024-04-22T18:41:32.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20472459400000001 2024-04-22T18:41:32.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759525+0000", 2024-04-22T18:41:32.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5653000000000001e-05 2024-04-22T18:41:32.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759682+0000", 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015711800000000001 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:75 3.1 3:ab43a94d:::f38e6e97cb24e2b90da964fee5d2d6d8674386870fc552915e0c3fdbefd4b126:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 57.514503251999997, 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60149928399999997, 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247520+0000", 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247527+0000", 2024-04-22T18:41:32.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1670000000000002e-06 2024-04-22T18:41:32.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247539+0000", 2024-04-22T18:41:32.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503e-05 2024-04-22T18:41:32.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247549+0000", 2024-04-22T18:41:32.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0000000000000001e-05 2024-04-22T18:41:32.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247796+0000", 2024-04-22T18:41:32.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024712699999999998 2024-04-22T18:41:32.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:32.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247955+0000", 2024-04-22T18:41:32.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158534 2024-04-22T18:41:32.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.343985+0000", 2024-04-22T18:41:32.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096030344000000004 2024-04-22T18:41:32.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.344035+0000", 2024-04-22T18:41:32.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9969999999999998e-05 2024-04-22T18:41:32.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.745806+0000", 2024-04-22T18:41:32.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.401771029 2024-04-22T18:41:32.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746040+0000", 2024-04-22T18:41:32.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023339300000000001 2024-04-22T18:41:32.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:32.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746198+0000", 2024-04-22T18:41:32.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015793299999999999 2024-04-22T18:41:32.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848737+0000", 2024-04-22T18:41:32.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10253898 2024-04-22T18:41:32.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848751+0000", 2024-04-22T18:41:32.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4694e-05 2024-04-22T18:41:32.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848825+0000", 2024-04-22T18:41:32.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3443999999999997e-05 2024-04-22T18:41:32.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849021+0000", 2024-04-22T18:41:32.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000196047 2024-04-22T18:41:32.830 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.830 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.830 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.831 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:76 3.1 3:b44829a0:::3788286a255d538503614e5f04049b0bb54cacaa317ab102b5dee8f5b640b296:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:32.831 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:32.831 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 57.514464185999998, 2024-04-22T18:41:32.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60168681599999996, 2024-04-22T18:41:32.831 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.831 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.831 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:32.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:32.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:32.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:32.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247564+0000", 2024-04-22T18:41:32.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1150000000000004e-06 2024-04-22T18:41:32.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247570+0000", 2024-04-22T18:41:32.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3940000000000001e-06 2024-04-22T18:41:32.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247576+0000", 2024-04-22T18:41:32.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3619999999999997e-06 2024-04-22T18:41:32.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.340845+0000", 2024-04-22T18:41:32.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.093269199999999997 2024-04-22T18:41:32.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:32.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.341043+0000", 2024-04-22T18:41:32.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019834400000000001 2024-04-22T18:41:32.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345106+0000", 2024-04-22T18:41:32.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040629780000000001 2024-04-22T18:41:32.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345160+0000", 2024-04-22T18:41:32.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4246999999999997e-05 2024-04-22T18:41:32.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747047+0000", 2024-04-22T18:41:32.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40188678300000003 2024-04-22T18:41:32.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747162+0000", 2024-04-22T18:41:32.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011483400000000001 2024-04-22T18:41:32.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747429+0000", 2024-04-22T18:41:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026700299999999998 2024-04-22T18:41:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849032+0000", 2024-04-22T18:41:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101603203 2024-04-22T18:41:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849095+0000", 2024-04-22T18:41:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2597999999999999e-05 2024-04-22T18:41:32.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849136+0000", 2024-04-22T18:41:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1251000000000002e-05 2024-04-22T18:41:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849247+0000", 2024-04-22T18:41:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011127 2024-04-22T18:41:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.837 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3526 4.c 4:314c2fec:::smithi08937929-22:head [setxattr _header (56) in=7b,truncate 3092521] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:41:32.838 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:32.838 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.532827406000003, 2024-04-22T18:41:32.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60463626000000004, 2024-04-22T18:41:32.838 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.838 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.838 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:32.838 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:32.838 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:32.838 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3526 2024-04-22T18:41:32.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.838 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:32.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:32.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:32.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229200+0000", 2024-04-22T18:41:32.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2320000000000001e-06 2024-04-22T18:41:32.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229203+0000", 2024-04-22T18:41:32.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:41:32.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329292+0000", 2024-04-22T18:41:32.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100088891 2024-04-22T18:41:32.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329357+0000", 2024-04-22T18:41:32.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5115000000000003e-05 2024-04-22T18:41:32.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329453+0000", 2024-04-22T18:41:32.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5802000000000002e-05 2024-04-22T18:41:32.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:32.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329594+0000", 2024-04-22T18:41:32.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014157500000000001 2024-04-22T18:41:32.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.330172+0000", 2024-04-22T18:41:32.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057782799999999996 2024-04-22T18:41:32.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833712+0000", 2024-04-22T18:41:32.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50353975299999998 2024-04-22T18:41:32.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833774+0000", 2024-04-22T18:41:32.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1832e-05 2024-04-22T18:41:32.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833834+0000", 2024-04-22T18:41:32.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9809999999999998e-05 2024-04-22T18:41:32.843 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.843 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.843 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.844 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:339 3.2 3:4159b8d1:::bb1f8e997816dd9d2f29d36c4c1859fcaffbf8763b5c8d63c7a818d8d84bac37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:32.844 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:32.844 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.986573647, 2024-04-22T18:41:32.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0092773479999999, 2024-04-22T18:41:32.844 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.844 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.844 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:32.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775450+0000", 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775457+0000", 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2949999999999996e-06 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775467+0000", 2024-04-22T18:41:32.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0180000000000001e-05 2024-04-22T18:41:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775475+0000", 2024-04-22T18:41:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.678e-06 2024-04-22T18:41:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.877787+0000", 2024-04-22T18:41:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102312559 2024-04-22T18:41:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.880686+0000", 2024-04-22T18:41:32.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002898748 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.282867+0000", 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.402180961 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.282945+0000", 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7775999999999995e-05 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283065+0000", 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0001199 2024-04-22T18:41:32.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.285514+0000", 2024-04-22T18:41:32.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0024491729999999998 2024-04-22T18:41:32.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784375+0000", 2024-04-22T18:41:32.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49886154100000002 2024-04-22T18:41:32.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784475+0000", 2024-04-22T18:41:32.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9074999999999996e-05 2024-04-22T18:41:32.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784728+0000", 2024-04-22T18:41:32.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025386799999999998 2024-04-22T18:41:32.849 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.849 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.849 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.849 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:340 3.2 3:4bbc2d06:::e1c179ab87fb8e08c4a1dfc0c7cc725138181c5734d6df18f3f2545b93731e97:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:32.849 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:32.849 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.986541228, 2024-04-22T18:41:32.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009550757, 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:32.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775487+0000", 2024-04-22T18:41:32.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6839999999999998e-06 2024-04-22T18:41:32.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775492+0000", 2024-04-22T18:41:32.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6899999999999997e-06 2024-04-22T18:41:32.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775497+0000", 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8590000000000004e-06 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.880737+0000", 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105240267 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881308+0000", 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057098399999999997 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283810+0000", 2024-04-22T18:41:32.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40250177500000001 2024-04-22T18:41:32.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283883+0000", 2024-04-22T18:41:32.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2430999999999998e-05 2024-04-22T18:41:32.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:32.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284019+0000", 2024-04-22T18:41:32.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013652299999999999 2024-04-22T18:41:32.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.287390+0000", 2024-04-22T18:41:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033710089999999999 2024-04-22T18:41:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784829+0000", 2024-04-22T18:41:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49743879400000002 2024-04-22T18:41:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784893+0000", 2024-04-22T18:41:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4184999999999996e-05 2024-04-22T18:41:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785034+0000", 2024-04-22T18:41:32.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014114600000000001 2024-04-22T18:41:32.855 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.855 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.855 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.855 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:341 3.2 3:5b8bea68:::ab73be59042e3e18c4fe1fdbc2bbcc205779424ddf3bb423a595330be9026399:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:32.855 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:32.855 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.986521876000001, 2024-04-22T18:41:32.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009670804, 2024-04-22T18:41:32.855 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.855 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.855 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:32.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:32.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775502+0000", 2024-04-22T18:41:32.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775505+0000", 2024-04-22T18:41:32.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0809999999999998e-06 2024-04-22T18:41:32.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775511+0000", 2024-04-22T18:41:32.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2039999999999996e-06 2024-04-22T18:41:32.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775515+0000", 2024-04-22T18:41:32.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8790000000000002e-06 2024-04-22T18:41:32.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881376+0000", 2024-04-22T18:41:32.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105861 2024-04-22T18:41:32.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881839+0000", 2024-04-22T18:41:32.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046217200000000001 2024-04-22T18:41:32.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284569+0000", 2024-04-22T18:41:32.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40273058900000003 2024-04-22T18:41:32.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284730+0000", 2024-04-22T18:41:32.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077800000000001 2024-04-22T18:41:32.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:32.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284923+0000", 2024-04-22T18:41:32.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000193253 2024-04-22T18:41:32.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.287593+0000", 2024-04-22T18:41:32.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026699359999999999 2024-04-22T18:41:32.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785074+0000", 2024-04-22T18:41:32.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49748046000000001 2024-04-22T18:41:32.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785103+0000", 2024-04-22T18:41:32.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8935e-05 2024-04-22T18:41:32.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785174+0000", 2024-04-22T18:41:32.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1001999999999995e-05 2024-04-22T18:41:32.861 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.861 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.861 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.862 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:342 3.2 3:634dda9f:::10622dc7de8047e7908c0b5cca7a43d84cdf1b7629e6f7e159b8207feba82b01:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:32.862 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:32.862 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.986503664000001, 2024-04-22T18:41:32.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.022499579, 2024-04-22T18:41:32.862 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.862 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.862 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:32.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:32.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:32.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775524+0000", 2024-04-22T18:41:32.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9629999999999998e-06 2024-04-22T18:41:32.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775529+0000", 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0320000000000003e-06 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775533+0000", 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7380000000000002e-06 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881884+0000", 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106350294 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.882492+0000", 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060872699999999999 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381559+0000", 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49906679199999998 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381677+0000", 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011743699999999999 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:32.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381899+0000", 2024-04-22T18:41:32.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022272599999999999 2024-04-22T18:41:32.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.383440+0000", 2024-04-22T18:41:32.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015407719999999999 2024-04-22T18:41:32.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.797827+0000", 2024-04-22T18:41:32.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.414386741 2024-04-22T18:41:32.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.797887+0000", 2024-04-22T18:41:32.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9889000000000002e-05 2024-04-22T18:41:32.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.798021+0000", 2024-04-22T18:41:32.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000133948 2024-04-22T18:41:32.867 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.867 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.867 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.868 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:343 3.2 3:7db7298e:::00597d8527769aa920807495342cf3ba06efb8f507feae865dd89cd8ad10bda0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:32.868 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:32.868 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.986484708999999, 2024-04-22T18:41:32.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.110995373, 2024-04-22T18:41:32.868 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.868 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.868 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:32.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:32.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:32.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775542+0000", 2024-04-22T18:41:32.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9079999999999999e-06 2024-04-22T18:41:32.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775547+0000", 2024-04-22T18:41:32.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.994e-06 2024-04-22T18:41:32.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775553+0000", 2024-04-22T18:41:32.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2299999999999999e-06 2024-04-22T18:41:32.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.882519+0000", 2024-04-22T18:41:32.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106966453 2024-04-22T18:41:32.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883103+0000", 2024-04-22T18:41:32.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058419400000000001 2024-04-22T18:41:32.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.581957+0000", 2024-04-22T18:41:32.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69885328599999996 2024-04-22T18:41:32.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582023+0000", 2024-04-22T18:41:32.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6210999999999993e-05 2024-04-22T18:41:32.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:32.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582146+0000", 2024-04-22T18:41:32.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123669 2024-04-22T18:41:32.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.682293+0000", 2024-04-22T18:41:32.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10014662100000001 2024-04-22T18:41:32.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886410+0000", 2024-04-22T18:41:32.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20411678899999999 2024-04-22T18:41:32.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886445+0000", 2024-04-22T18:41:32.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5462e-05 2024-04-22T18:41:32.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886535+0000", 2024-04-22T18:41:32.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9995999999999997e-05 2024-04-22T18:41:32.873 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.873 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.874 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.874 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:344 3.1 3:9da67ae4:::04c6b48a110b3f74550f78f0899df697160a1eeab7f41463f6ac18091ca67078:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:32.874 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:32.874 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.986449402000002, 2024-04-22T18:41:32.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60604869800000005, 2024-04-22T18:41:32.874 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.875 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.875 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:32.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:32.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:32.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775578+0000", 2024-04-22T18:41:32.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3440000000000001e-06 2024-04-22T18:41:32.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775584+0000", 2024-04-22T18:41:32.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6969999999999998e-06 2024-04-22T18:41:32.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775589+0000", 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0540000000000002e-06 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775729+0000", 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014061299999999999 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776045+0000", 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000315168 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080584+0000", 2024-04-22T18:41:32.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30453977700000001 2024-04-22T18:41:32.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080635+0000", 2024-04-22T18:41:32.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1007999999999997e-05 2024-04-22T18:41:32.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:32.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080741+0000", 2024-04-22T18:41:32.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105646 2024-04-22T18:41:32.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.180821+0000", 2024-04-22T18:41:32.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10007980900000001 2024-04-22T18:41:32.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381437+0000", 2024-04-22T18:41:32.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20061589399999999 2024-04-22T18:41:32.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381491+0000", 2024-04-22T18:41:32.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4531999999999999e-05 2024-04-22T18:41:32.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381624+0000", 2024-04-22T18:41:32.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000132745 2024-04-22T18:41:32.880 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.880 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.880 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.880 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:345 3.1 3:bd847077:::0e274616605e19dd653fcf05d310933ab02835574414ca18340b3fc325d2da4f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:32.880 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:32.880 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.986425042, 2024-04-22T18:41:32.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.90876140699999997, 2024-04-22T18:41:32.881 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.881 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.881 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:32.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:32.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775599+0000", 2024-04-22T18:41:32.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:32.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775602+0000", 2024-04-22T18:41:32.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.976e-06 2024-04-22T18:41:32.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775607+0000", 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2229999999999998e-06 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775612+0000", 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6539999999999998e-06 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776072+0000", 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046006299999999998 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776256+0000", 2024-04-22T18:41:32.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000183735 2024-04-22T18:41:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081157+0000", 2024-04-22T18:41:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30490169099999997 2024-04-22T18:41:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081203+0000", 2024-04-22T18:41:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5677999999999999e-05 2024-04-22T18:41:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081301+0000", 2024-04-22T18:41:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7893999999999995e-05 2024-04-22T18:41:32.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.180827+0000", 2024-04-22T18:41:32.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099525725999999995 2024-04-22T18:41:32.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684197+0000", 2024-04-22T18:41:32.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50336990699999995 2024-04-22T18:41:32.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684247+0000", 2024-04-22T18:41:32.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0195000000000003e-05 2024-04-22T18:41:32.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684361+0000", 2024-04-22T18:41:32.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114189 2024-04-22T18:41:32.886 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.886 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.886 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.886 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4267 3.2 3:584fd9a2:::cda848272f4ef42ebafc2089aaddeacaf12eabd82aef5e42e32ae93a8a5da0b6:head [write 0~2635776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:32.886 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:32.886 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.906075185999999, 2024-04-22T18:41:32.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.72658671900000005, 2024-04-22T18:41:32.887 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:32.887 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:32.887 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:32.887 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:32.887 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:32.887 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4267 2024-04-22T18:41:32.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.887 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:32.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:32.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:32.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:32.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:32.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:32.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:32.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855946+0000", 2024-04-22T18:41:32.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:32.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:32.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.856984+0000", 2024-04-22T18:41:32.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001037764 2024-04-22T18:41:32.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:32.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.857003+0000", 2024-04-22T18:41:32.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8876e-05 2024-04-22T18:41:32.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:32.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.857016+0000", 2024-04-22T18:41:32.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3337e-05 2024-04-22T18:41:32.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:32.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883170+0000", 2024-04-22T18:41:32.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026154047 2024-04-22T18:41:32.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:32.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883262+0000", 2024-04-22T18:41:32.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1996999999999993e-05 2024-04-22T18:41:32.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:32.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883360+0000", 2024-04-22T18:41:32.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7429000000000005e-05 2024-04-22T18:41:32.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.887277+0000", 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039169449999999998 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582423+0000", 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69514625200000002 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582458+0000", 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5293999999999998e-05 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:32.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:32.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582536+0000", 2024-04-22T18:41:32.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7855000000000006e-05 2024-04-22T18:41:32.892 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.892 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.892 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.892 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:32.892 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:32.892 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:32.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:32.894 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:32.894+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:32.994 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:32.995+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:33.007 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:33.008 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:33.008 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:33.008 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:33.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:33.095 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:33.097+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:33.118 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:33.118 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:33.118 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:33.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:33.196 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:33.198+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:33.229 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:33.229 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:33.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.567156141, 2024-04-22T18:41:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:41:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:41:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.230 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:33.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:41:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:41:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:41:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:41:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:41:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:41:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:41:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:41:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:41:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:41:33.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:41:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:41:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:41:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:41:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:41:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:41:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:41:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:41:33.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:41:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:41:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:33.236 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.46614796399999, 2024-04-22T18:41:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:41:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:41:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:41:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:41:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:41:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:41:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:41:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:41:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:41:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:41:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:41:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:41:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:41:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:41:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:41:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:41:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:41:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:41:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.46613757899999, 2024-04-22T18:41:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:41:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:41:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:33.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:41:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:41:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:41:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:41:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:41:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:41:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:41:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:41:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:41:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:41:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:41:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:41:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:41:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:41:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:41:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:41:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:41:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:41:33.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:41:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:41:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:41:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:41:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 154.12838715199999, 2024-04-22T18:41:33.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:41:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:41:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:41:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:41:33.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:41:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:41:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:41:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:41:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:41:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:41:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:41:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:41:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:41:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:41:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:41:33.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:41:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:41:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:41:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:41:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:41:33.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:41:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:41:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:41:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:41:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 127.093739637, 2024-04-22T18:41:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:41:33.253 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:41:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:41:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:41:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:41:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:41:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:41:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:41:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:41:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.257 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4177 4.19 4.f75b2379 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.064686340000002, 2024-04-22T18:41:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76236081700000002, 2024-04-22T18:41:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4177 2024-04-22T18:41:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164847+0000", 2024-04-22T18:41:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164854+0000", 2024-04-22T18:41:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2620000000000001e-06 2024-04-22T18:41:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164864+0000", 2024-04-22T18:41:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0400999999999999e-05 2024-04-22T18:41:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164876+0000", 2024-04-22T18:41:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1204e-05 2024-04-22T18:41:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164975+0000", 2024-04-22T18:41:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9562999999999995e-05 2024-04-22T18:41:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164980+0000", 2024-04-22T18:41:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7310000000000001e-06 2024-04-22T18:41:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927200+0000", 2024-04-22T18:41:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.762219973 2024-04-22T18:41:33.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927209+0000", 2024-04-22T18:41:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9050000000000003e-06 2024-04-22T18:41:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4182 4.c 4.5345b7ac (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.064655021, 2024-04-22T18:41:33.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76185930999999996, 2024-04-22T18:41:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4182 2024-04-22T18:41:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164881+0000", 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.305e-06 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164885+0000", 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.873e-06 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164890+0000", 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6299999999999997e-06 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225475+0000", 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060585181000000002 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225483+0000", 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6790000000000005e-06 2024-04-22T18:41:33.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926728+0000", 2024-04-22T18:41:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124542099999998 2024-04-22T18:41:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926739+0000", 2024-04-22T18:41:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0669000000000001e-05 2024-04-22T18:41:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.266 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4183 4.c 4.5345b7ac (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.064641765000001, 2024-04-22T18:41:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76187489500000005, 2024-04-22T18:41:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.267 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4183 2024-04-22T18:41:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164892+0000", 2024-04-22T18:41:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164894+0000", 2024-04-22T18:41:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.705e-06 2024-04-22T18:41:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164897+0000", 2024-04-22T18:41:33.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2739999999999999e-06 2024-04-22T18:41:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164901+0000", 2024-04-22T18:41:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.636e-06 2024-04-22T18:41:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225508+0000", 2024-04-22T18:41:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060607564000000003 2024-04-22T18:41:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225515+0000", 2024-04-22T18:41:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1829999999999999e-06 2024-04-22T18:41:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926759+0000", 2024-04-22T18:41:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.701244859 2024-04-22T18:41:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926767+0000", 2024-04-22T18:41:33.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0180000000000001e-06 2024-04-22T18:41:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4190 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.064624843000001, 2024-04-22T18:41:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.761884546, 2024-04-22T18:41:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.272 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4190 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164912+0000", 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6249999999999999e-06 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164918+0000", 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6039999999999998e-06 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164922+0000", 2024-04-22T18:41:33.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9319999999999998e-06 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225536+0000", 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060613725 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225540+0000", 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5030000000000002e-06 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926786+0000", 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124673999999998 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926794+0000", 2024-04-22T18:41:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7689999999999996e-06 2024-04-22T18:41:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4191 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.064609422, 2024-04-22T18:41:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76190695500000005, 2024-04-22T18:41:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:33.276 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4191 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164927+0000", 2024-04-22T18:41:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0959999999999999e-06 2024-04-22T18:41:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164937+0000", 2024-04-22T18:41:33.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0730999999999999e-05 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164944+0000", 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5130000000000002e-06 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225565+0000", 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060621421000000002 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225569+0000", 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.264e-06 2024-04-22T18:41:33.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926813+0000", 2024-04-22T18:41:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124469 2024-04-22T18:41:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926832+0000", 2024-04-22T18:41:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8580000000000002e-05 2024-04-22T18:41:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4192 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.064584814, 2024-04-22T18:41:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76191075399999997, 2024-04-22T18:41:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4192 2024-04-22T18:41:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.281 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:33.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164953+0000", 2024-04-22T18:41:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1099999999999996e-06 2024-04-22T18:41:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164965+0000", 2024-04-22T18:41:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2298e-05 2024-04-22T18:41:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164972+0000", 2024-04-22T18:41:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5230000000000001e-06 2024-04-22T18:41:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225581+0000", 2024-04-22T18:41:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060609356000000003 2024-04-22T18:41:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225584+0000", 2024-04-22T18:41:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.106e-06 2024-04-22T18:41:33.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926852+0000", 2024-04-22T18:41:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70126794999999997 2024-04-22T18:41:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926860+0000", 2024-04-22T18:41:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0919999999999998e-06 2024-04-22T18:41:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4197 4.13 4:ca46fde9:::smithi08937929-14:head [sparse-read 0~1375265,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.064558204000001, 2024-04-22T18:41:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.28429536, 2024-04-22T18:41:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4197 2024-04-22T18:41:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.286 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164975+0000", 2024-04-22T18:41:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164979+0000", 2024-04-22T18:41:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8959999999999996e-06 2024-04-22T18:41:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164983+0000", 2024-04-22T18:41:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.053e-06 2024-04-22T18:41:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164987+0000", 2024-04-22T18:41:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:41:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.264783+0000", 2024-04-22T18:41:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099796088000000005 2024-04-22T18:41:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.264789+0000", 2024-04-22T18:41:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4989999999999999e-06 2024-04-22T18:41:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139375+0000", 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.87458606000000005 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139478+0000", 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010219 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.446961+0000", 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30748385900000003 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:33.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.447101+0000", 2024-04-22T18:41:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001393 2024-04-22T18:41:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.449271+0000", 2024-04-22T18:41:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021707950000000001 2024-04-22T18:41:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4198 4.13 4:ca46fde9:::smithi08937929-14:head [sparse-read 0~1375265] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:33.291 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.064544836, 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2868951370000001, 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4198 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164991+0000", 2024-04-22T18:41:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:41:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164994+0000", 2024-04-22T18:41:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2459999999999999e-06 2024-04-22T18:41:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164997+0000", 2024-04-22T18:41:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1200000000000002e-06 2024-04-22T18:41:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.364948+0000", 2024-04-22T18:41:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19995059200000001 2024-04-22T18:41:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.364969+0000", 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1495999999999999e-05 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139617+0000", 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77464723300000005 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139643+0000", 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6228e-05 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451393+0000", 2024-04-22T18:41:33.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31175035200000001 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451515+0000", 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001214 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451885+0000", 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036992999999999998 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.296 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4199 4.13 4:ca46fde9:::smithi08937929-14:head [read 0~2739541,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.064533813000001, 2024-04-22T18:41:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.291247504, 2024-04-22T18:41:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4199 2024-04-22T18:41:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:33.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165002+0000", 2024-04-22T18:41:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:41:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165006+0000", 2024-04-22T18:41:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:41:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165009+0000", 2024-04-22T18:41:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4599999999999999e-06 2024-04-22T18:41:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365066+0000", 2024-04-22T18:41:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20005682999999999 2024-04-22T18:41:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365078+0000", 2024-04-22T18:41:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2235000000000001e-05 2024-04-22T18:41:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139671+0000", 2024-04-22T18:41:33.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77459301700000005 2024-04-22T18:41:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139702+0000", 2024-04-22T18:41:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0673999999999998e-05 2024-04-22T18:41:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451930+0000", 2024-04-22T18:41:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31222817600000002 2024-04-22T18:41:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:33.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.452092+0000", 2024-04-22T18:41:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001616 2024-04-22T18:41:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.456248+0000", 2024-04-22T18:41:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041563700000000004 2024-04-22T18:41:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.302 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4200 4.13 4:c93c842e:::smithi08937929-25:head [tier-promote] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.064522932999999, 2024-04-22T18:41:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.177728814, 2024-04-22T18:41:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4200 2024-04-22T18:41:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165013+0000", 2024-04-22T18:41:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.635e-06 2024-04-22T18:41:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165018+0000", 2024-04-22T18:41:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2000000000000002e-06 2024-04-22T18:41:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165024+0000", 2024-04-22T18:41:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5910000000000003e-06 2024-04-22T18:41:33.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365110+0000", 2024-04-22T18:41:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20008601400000001 2024-04-22T18:41:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365114+0000", 2024-04-22T18:41:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.651e-06 2024-04-22T18:41:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139723+0000", 2024-04-22T18:41:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77460836899999996 2024-04-22T18:41:33.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139787+0000", 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4707000000000002e-05 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.459491+0000", 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31970372000000002 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.459567+0000", 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6199999999999995e-05 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.562522+0000", 2024-04-22T18:41:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102954925 2024-04-22T18:41:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.562585+0000", 2024-04-22T18:41:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3287000000000006e-05 2024-04-22T18:41:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:05.342740+0000", 2024-04-22T18:41:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.78015481499999995 2024-04-22T18:41:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.308 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.309 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.309 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4205 4.1 4.fdaa93c1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:33.309 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:33.309 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 30.064506782999999, 2024-04-22T18:41:33.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76220996100000005, 2024-04-22T18:41:33.309 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.309 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:33.309 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.309 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.309 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.309 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4205 2024-04-22T18:41:33.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.310 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:33.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:33.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165027+0000", 2024-04-22T18:41:33.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:33.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165029+0000", 2024-04-22T18:41:33.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3549999999999999e-06 2024-04-22T18:41:33.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165035+0000", 2024-04-22T18:41:33.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8549999999999998e-06 2024-04-22T18:41:33.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165040+0000", 2024-04-22T18:41:33.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9200000000000003e-06 2024-04-22T18:41:33.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165116+0000", 2024-04-22T18:41:33.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5834999999999995e-05 2024-04-22T18:41:33.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:33.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165123+0000", 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2139999999999999e-06 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927230+0000", 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76210651100000004 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927237+0000", 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7409999999999992e-06 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.314 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4268 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e133)", 2024-04-22T18:41:33.314 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:41:33.314 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.646799371, 2024-04-22T18:41:33.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70931800300000003, 2024-04-22T18:41:33.314 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.314 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:33.314 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.314 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.314 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.314 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4268 2024-04-22T18:41:33.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.314 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582733+0000", 2024-04-22T18:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582741+0000", 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3329999999999999e-06 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582751+0000", 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0628999999999999e-05 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582759+0000", 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4120000000000004e-06 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582795+0000", 2024-04-22T18:41:33.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6157e-05 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582862+0000", 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7261999999999994e-05 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.987268+0000", 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40440629700000003 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.987771+0000", 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050267400000000005 2024-04-22T18:41:33.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:33.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.291974+0000", 2024-04-22T18:41:33.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30420264499999999 2024-04-22T18:41:33.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:33.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.292008+0000", 2024-04-22T18:41:33.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4011999999999999e-05 2024-04-22T18:41:33.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.292053+0000", 2024-04-22T18:41:33.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5305000000000001e-05 2024-04-22T18:41:33.319 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.319 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.319 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.319 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4272 4.5 4:a6d237d2:::smithi08937929-43:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e133)", 2024-04-22T18:41:33.319 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:33.319 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.443471170999999, 2024-04-22T18:41:33.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60731304699999999, 2024-04-22T18:41:33.319 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.319 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:33.319 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.319 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.319 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4272 2024-04-22T18:41:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:33.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:33.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786068+0000", 2024-04-22T18:41:33.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9910000000000002e-06 2024-04-22T18:41:33.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786079+0000", 2024-04-22T18:41:33.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1287000000000001e-05 2024-04-22T18:41:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786083+0000", 2024-04-22T18:41:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0949999999999998e-06 2024-04-22T18:41:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786121+0000", 2024-04-22T18:41:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8096999999999998e-05 2024-04-22T18:41:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786219+0000", 2024-04-22T18:41:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7429000000000005e-05 2024-04-22T18:41:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.393376+0000", 2024-04-22T18:41:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60715760600000002 2024-04-22T18:41:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4437 4.19 4:9ea6f51f:::smithi08937929-34:head [read 304128~651776] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:41:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 15.313864397, 2024-04-22T18:41:33.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.042388732, 2024-04-22T18:41:33.324 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.324 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:33.324 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.324 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.324 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.324 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4437 2024-04-22T18:41:33.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.324 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:41:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:41:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915669+0000", 2024-04-22T18:41:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915673+0000", 2024-04-22T18:41:33.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6770000000000001e-06 2024-04-22T18:41:33.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915679+0000", 2024-04-22T18:41:33.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.976e-06 2024-04-22T18:41:33.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915684+0000", 2024-04-22T18:41:33.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3430000000000003e-06 2024-04-22T18:41:33.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915725+0000", 2024-04-22T18:41:33.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0751999999999999e-05 2024-04-22T18:41:33.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:33.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915732+0000", 2024-04-22T18:41:33.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6279999999999999e-06 2024-04-22T18:41:33.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.957228+0000", 2024-04-22T18:41:33.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0414961220000001 2024-04-22T18:41:33.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:33.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.957325+0000", 2024-04-22T18:41:33.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7422000000000003e-05 2024-04-22T18:41:33.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.958059+0000", 2024-04-22T18:41:33.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073350499999999996 2024-04-22T18:41:33.328 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.328 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.329 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.329 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4690 3.3 3:ee62d872:::cb076f2b64cfcdfafc1ec2cf19f2eaa46006a5a6952a3aeb9c7211f97fca91ae:head [write 0~441856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-04-22T18:41:33.329 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:26.592341+0000", 2024-04-22T18:41:33.329 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 6.6371931350000004, 2024-04-22T18:41:33.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.56667778400000002, 2024-04-22T18:41:33.329 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:33.329 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:33.329 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:33.329 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:33.329 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:33.330 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4690 2024-04-22T18:41:33.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.330 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:33.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:33.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592341+0000", 2024-04-22T18:41:33.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:33.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592341+0000", 2024-04-22T18:41:33.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:33.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:33.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592340+0000", 2024-04-22T18:41:33.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:33.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:33.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592697+0000", 2024-04-22T18:41:33.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000357334 2024-04-22T18:41:33.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:33.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592709+0000", 2024-04-22T18:41:33.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2211999999999999e-05 2024-04-22T18:41:33.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:33.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592718+0000", 2024-04-22T18:41:33.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0319999999999992e-06 2024-04-22T18:41:33.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.692859+0000", 2024-04-22T18:41:33.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014077 2024-04-22T18:41:33.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:33.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.692870+0000", 2024-04-22T18:41:33.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1225e-05 2024-04-22T18:41:33.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:33.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.995290+0000", 2024-04-22T18:41:33.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242006199999999 2024-04-22T18:41:33.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:33.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.995398+0000", 2024-04-22T18:41:33.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108136 2024-04-22T18:41:33.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.995513+0000", 2024-04-22T18:41:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115047 2024-04-22T18:41:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.996655+0000", 2024-04-22T18:41:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001141162 2024-04-22T18:41:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:27.158858+0000", 2024-04-22T18:41:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16220310299999999 2024-04-22T18:41:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:33.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:27.158901+0000", 2024-04-22T18:41:33.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3667000000000001e-05 2024-04-22T18:41:33.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:33.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:33.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:33.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:27.159019+0000", 2024-04-22T18:41:33.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117619 2024-04-22T18:41:33.335 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.335 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.335 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.335 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:33.335 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:33.335 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:33.336 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:41:33.337 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:33.338+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:33.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:33.438 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:33.438+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:33.440 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:33.440 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:33.440 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:33.440 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:33.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:33.453 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:33.453 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:33.469 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:33.470 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:33.486 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:33.486 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:33.502 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:33.502 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:33.518 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:33.518 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:33.534 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:33.534 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:33.538 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:33.540+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:33.552 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:33.552 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:33.568 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:33.569 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:33.593 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:33.593 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:33.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:33.639 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:33.641+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:33.664 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:33.665+0000 7f3093031640 -1 osd.1 157 handle_osd_map injecting map crc failure 2024-04-22T18:41:33.675 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:41:33.705 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:33.705 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:33.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:33.758 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:33.759+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:33.816 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:33.816 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:33.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:33.841 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:33.842+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:33.942 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:33.943+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:33.967 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:33.967 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:33.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:34.043 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:34.044+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:34.117 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:34.117 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:34.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:34.170 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:34.171+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:34.228 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:34.229 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:34.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:34.279 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:34.280+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:34.346 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:34.347+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:34.379 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:34.379 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:34.388 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:34.447 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:34.448+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:34.491 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:34.491 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:34.500 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:34.548 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:34.549+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:34.607 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:34.607 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:34.616 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:34.649 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:34.650+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:41:34.721 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:34.722 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:34.731 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:34.750 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:34.751+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:34.774 INFO:tasks.rados.rados.0.smithi089.stdout:1954: set_chunk oid 2 offset: 1839616 length: 19456 target oid fa0373de6f5ad7b9f5402c95164479263041a0994795e87be354360e40a6e8df offset: 0 snap -1 2024-04-22T18:41:34.774 INFO:tasks.rados.rados.0.smithi089.stdout:1955: write oid 34 current snap is 0 2024-04-22T18:41:34.774 INFO:tasks.rados.rados.0.smithi089.stdout:1949: finishing write tid 2 to smithi08937929-3 2024-04-22T18:41:34.774 INFO:tasks.rados.rados.0.smithi089.stdout:1955: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-04-22T18:41:34.779 INFO:tasks.rados.rados.0.smithi089.stdout:1955: writing smithi08937929-34 from 608519 to 1070414 tid 1 2024-04-22T18:41:34.785 INFO:tasks.rados.rados.0.smithi089.stdout:1955: writing smithi08937929-34 from 1602050 to 2130859 tid 2 2024-04-22T18:41:34.790 INFO:tasks.rados.rados.0.smithi089.stdout:1955: writing smithi08937929-34 from 2585035 to 3008519 tid 3 2024-04-22T18:41:34.790 INFO:tasks.rados.rados.0.smithi089.stdout:1956: set_chunk oid 4 target oid 2024-04-22T18:41:34.836 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:34.836 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:34.845 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:34.851 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:34.852+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:34.952 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:34.954+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:34.989 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:34.989 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:34.999 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:35.083 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:35.084+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:35.140 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:35.140 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:35.149 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:35.173 INFO:tasks.rados.rados.0.smithi089.stdout:1956: set_chunk oid 4 offset: 580096 length: 351232 target oid 092ea56155b80946e58a464fc8bc1af25fcf8f99300fc293263ec6716b556d3a offset: 0 snap -1 2024-04-22T18:41:35.173 INFO:tasks.rados.rados.0.smithi089.stdout:1957: read oid 32 snap -1 2024-04-22T18:41:35.173 INFO:tasks.rados.rados.0.smithi089.stdout:1957: expect (ObjNum 371 snap 0 seq_num 371) 2024-04-22T18:41:35.173 INFO:tasks.rados.rados.0.smithi089.stdout:1958: write (excl) oid 20 current snap is 0 2024-04-22T18:41:35.173 INFO:tasks.rados.rados.0.smithi089.stdout:1949: finishing write tid 3 to smithi08937929-3 2024-04-22T18:41:35.174 INFO:tasks.rados.rados.0.smithi089.stdout:1949: finishing write tid 4 to smithi08937929-3 2024-04-22T18:41:35.174 INFO:tasks.rados.rados.0.smithi089.stdout:1949: oid 3 updating version 0 to 84 2024-04-22T18:41:35.174 INFO:tasks.rados.rados.0.smithi089.stdout:1949: oid 3 updating version 84 to 85 2024-04-22T18:41:35.174 INFO:tasks.rados.rados.0.smithi089.stdout:1949: oid 3 version 85 is already newer than 83 2024-04-22T18:41:35.174 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 3 v 85 (ObjNum 414 snap 0 seq_num 414) dirty exists 2024-04-22T18:41:35.174 INFO:tasks.rados.rados.0.smithi089.stdout:1949: left oid 3 (ObjNum 414 snap 0 seq_num 414) 2024-04-22T18:41:35.174 INFO:tasks.rados.rados.0.smithi089.stdout:1952: finishing set_chunk to oid 47 2024-04-22T18:41:35.174 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 47 set_chunk returned error code -95 offset: 2003456 length: 667648 2024-04-22T18:41:35.174 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 47 v 0 (ObjNum 323 snap 0 seq_num 323) dirty exists 2024-04-22T18:41:35.174 INFO:tasks.rados.rados.0.smithi089.stdout:1951: finishing set_chunk to oid 30 2024-04-22T18:41:35.174 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 69 (ObjNum 378 snap 0 seq_num 378) dirty exists 2024-04-22T18:41:35.174 INFO:tasks.rados.rados.0.smithi089.stdout:1950: finishing write tid 1 to smithi08937929-45 2024-04-22T18:41:35.174 INFO:tasks.rados.rados.0.smithi089.stdout:1950: finishing write tid 2 to smithi08937929-45 2024-04-22T18:41:35.174 INFO:tasks.rados.rados.0.smithi089.stdout:1950: finishing write tid 3 to smithi08937929-45 2024-04-22T18:41:35.175 INFO:tasks.rados.rados.0.smithi089.stdout:1950: finishing write tid 4 to smithi08937929-45 2024-04-22T18:41:35.175 INFO:tasks.rados.rados.0.smithi089.stdout:1950: finishing write tid 5 to smithi08937929-45 2024-04-22T18:41:35.175 INFO:tasks.rados.rados.0.smithi089.stdout:1950: oid 45 updating version 0 to 104 2024-04-22T18:41:35.175 INFO:tasks.rados.rados.0.smithi089.stdout:1950: oid 45 version 104 is already newer than 103 2024-04-22T18:41:35.175 INFO:tasks.rados.rados.0.smithi089.stdout:1950: oid 45 updating version 104 to 106 2024-04-22T18:41:35.175 INFO:tasks.rados.rados.0.smithi089.stdout:1950: oid 45 version 106 is already newer than 105 2024-04-22T18:41:35.175 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 106 (ObjNum 415 snap 0 seq_num 415) dirty exists 2024-04-22T18:41:35.175 INFO:tasks.rados.rados.0.smithi089.stdout:1950: left oid 45 (ObjNum 415 snap 0 seq_num 415) 2024-04-22T18:41:35.175 INFO:tasks.rados.rados.0.smithi089.stdout:1953: finishing write tid 1 to smithi08937929-33 2024-04-22T18:41:35.175 INFO:tasks.rados.rados.0.smithi089.stdout:1953: finishing write tid 2 to smithi08937929-33 2024-04-22T18:41:35.175 INFO:tasks.rados.rados.0.smithi089.stdout:1953: finishing write tid 3 to smithi08937929-33 2024-04-22T18:41:35.175 INFO:tasks.rados.rados.0.smithi089.stdout:1958: seq_num 418 ranges {625326=537144,1944179=672467} 2024-04-22T18:41:35.176 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:35.173+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:35.178 INFO:tasks.rados.rados.0.smithi089.stdout:1958: writing smithi08937929-20 from 625326 to 1162470 tid 1 2024-04-22T18:41:35.187 INFO:tasks.rados.rados.0.smithi089.stdout:1958: writing smithi08937929-20 from 1944179 to 2616646 tid 2 2024-04-22T18:41:35.187 INFO:tasks.rados.rados.0.smithi089.stdout:1949: done (9 left) 2024-04-22T18:41:35.187 INFO:tasks.rados.rados.0.smithi089.stdout:1950: done (8 left) 2024-04-22T18:41:35.187 INFO:tasks.rados.rados.0.smithi089.stdout:1951: done (7 left) 2024-04-22T18:41:35.187 INFO:tasks.rados.rados.0.smithi089.stdout:1952: done (6 left) 2024-04-22T18:41:35.187 INFO:tasks.rados.rados.0.smithi089.stdout:1959: set_chunk oid 1 target oid 2024-04-22T18:41:35.254 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:35.255+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:35.274 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:35.274 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:35.285 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:35.355 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:35.356+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:35.428 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:35.429 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:35.438 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:35.481 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:35.483+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:35.539 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:35.539 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:35.547 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:35.580 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:41:35.589 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:35.590+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:35.657 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:35.658+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:35.698 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:35.698 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:35.708 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:35.712 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:35.712 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:35.729 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:35.729 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:35.745 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:35.745 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:35.758 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:35.760+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:35.761 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:35.761 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:35.778 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:35.778 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:35.795 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:35.795 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:35.812 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:35.812 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:35.815 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:35.816 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:35.816 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:35.816 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:35.825 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:35.850 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:35.850 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:35.861 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:35.862+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:35.952 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:35.952 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:35.952 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:35.952 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:35.952 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:35.959 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:35.960+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:35.961 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:36.040 INFO:tasks.rados.rados.0.smithi089.stdout:1959: set_chunk oid 1 offset: 16384 length: 551424 target oid be30d8153292cff4296381d63413e8ba639d35e7c4e42d5de95f962864ed4648 offset: 0 snap -1 2024-04-22T18:41:36.040 INFO:tasks.rados.rados.0.smithi089.stdout:1960: read oid 50 snap -1 2024-04-22T18:41:36.040 INFO:tasks.rados.rados.0.smithi089.stdout:1960: expect (ObjNum 405 snap 0 seq_num 405) 2024-04-22T18:41:36.040 INFO:tasks.rados.rados.0.smithi089.stdout:1961: read oid 38 snap -1 2024-04-22T18:41:36.040 INFO:tasks.rados.rados.0.smithi089.stdout:1961: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:41:36.040 INFO:tasks.rados.rados.0.smithi089.stdout:1962: read oid 29 snap -1 2024-04-22T18:41:36.040 INFO:tasks.rados.rados.0.smithi089.stdout:1962: expect (ObjNum 407 snap 0 seq_num 407) 2024-04-22T18:41:36.041 INFO:tasks.rados.rados.0.smithi089.stdout:1963: write oid 37 current snap is 0 2024-04-22T18:41:36.041 INFO:tasks.rados.rados.0.smithi089.stdout:1953: finishing write tid 4 to smithi08937929-33 2024-04-22T18:41:36.041 INFO:tasks.rados.rados.0.smithi089.stdout:1953: oid 33 updating version 0 to 67 2024-04-22T18:41:36.041 INFO:tasks.rados.rados.0.smithi089.stdout:1953: oid 33 version 67 is already newer than 66 2024-04-22T18:41:36.041 INFO:tasks.rados.rados.0.smithi089.stdout:1953: oid 33 version 67 is already newer than 65 2024-04-22T18:41:36.041 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 67 (ObjNum 416 snap 0 seq_num 416) dirty exists 2024-04-22T18:41:36.041 INFO:tasks.rados.rados.0.smithi089.stdout:1953: left oid 33 (ObjNum 416 snap 0 seq_num 416) 2024-04-22T18:41:36.041 INFO:tasks.rados.rados.0.smithi089.stdout:1955: finishing write tid 1 to smithi08937929-34 2024-04-22T18:41:36.041 INFO:tasks.rados.rados.0.smithi089.stdout:1955: finishing write tid 2 to smithi08937929-34 2024-04-22T18:41:36.041 INFO:tasks.rados.rados.0.smithi089.stdout:1955: finishing write tid 3 to smithi08937929-34 2024-04-22T18:41:36.041 INFO:tasks.rados.rados.0.smithi089.stdout:1954: finishing set_chunk to oid 2 2024-04-22T18:41:36.041 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 78 (ObjNum 385 snap 0 seq_num 385) dirty exists 2024-04-22T18:41:36.041 INFO:tasks.rados.rados.0.smithi089.stdout:1955: finishing write tid 4 to smithi08937929-34 2024-04-22T18:41:36.041 INFO:tasks.rados.rados.0.smithi089.stdout:1955: finishing write tid 5 to smithi08937929-34 2024-04-22T18:41:36.042 INFO:tasks.rados.rados.0.smithi089.stdout:1955: oid 34 updating version 0 to 118 2024-04-22T18:41:36.042 INFO:tasks.rados.rados.0.smithi089.stdout:1955: oid 34 version 118 is already newer than 115 2024-04-22T18:41:36.042 INFO:tasks.rados.rados.0.smithi089.stdout:1955: oid 34 version 118 is already newer than 117 2024-04-22T18:41:36.042 INFO:tasks.rados.rados.0.smithi089.stdout:1955: oid 34 version 118 is already newer than 116 2024-04-22T18:41:36.042 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 118 (ObjNum 417 snap 0 seq_num 417) dirty exists 2024-04-22T18:41:36.042 INFO:tasks.rados.rados.0.smithi089.stdout:1955: left oid 34 (ObjNum 417 snap 0 seq_num 417) 2024-04-22T18:41:36.042 INFO:tasks.rados.rados.0.smithi089.stdout:1957: expect (ObjNum 371 snap 0 seq_num 371) 2024-04-22T18:41:36.060 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:36.061+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:36.082 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:289 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 93.441117164000005, 2024-04-22T18:41:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9176801189999999, 2024-04-22T18:41:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640723+0000", 2024-04-22T18:41:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640732+0000", 2024-04-22T18:41:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5809999999999997e-06 2024-04-22T18:41:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640744+0000", 2024-04-22T18:41:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.261e-05 2024-04-22T18:41:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640753+0000", 2024-04-22T18:41:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7460000000000006e-06 2024-04-22T18:41:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642189+0000", 2024-04-22T18:41:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001435551 2024-04-22T18:41:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642200+0000", 2024-04-22T18:41:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1389999999999999e-05 2024-04-22T18:41:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.847015+0000", 2024-04-22T18:41:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048149160000001 2024-04-22T18:41:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.848217+0000", 2024-04-22T18:41:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012020659999999999 2024-04-22T18:41:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351823+0000", 2024-04-22T18:41:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50360565099999999 2024-04-22T18:41:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351907+0000", 2024-04-22T18:41:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4141999999999995e-05 2024-04-22T18:41:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352051+0000", 2024-04-22T18:41:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014386600000000001 2024-04-22T18:41:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453617+0000", 2024-04-22T18:41:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156649500000001 2024-04-22T18:41:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558297+0000", 2024-04-22T18:41:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10468000399999999 2024-04-22T18:41:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558331+0000", 2024-04-22T18:41:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3518e-05 2024-04-22T18:41:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558405+0000", 2024-04-22T18:41:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4221000000000006e-05 2024-04-22T18:41:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:1 3.2 3:6d90b903:::64b5942574fb36ca1b9a2be186c6b740edf58e989056a2f88b83cd280451a438:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.909816203999995, 2024-04-22T18:41:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.285534422, 2024-04-22T18:41:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172024+0000", 2024-04-22T18:41:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173091+0000", 2024-04-22T18:41:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010677569999999999 2024-04-22T18:41:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173103+0000", 2024-04-22T18:41:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1241e-05 2024-04-22T18:41:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173109+0000", 2024-04-22T18:41:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4300000000000003e-06 2024-04-22T18:41:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173168+0000", 2024-04-22T18:41:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9011000000000001e-05 2024-04-22T18:41:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173177+0000", 2024-04-22T18:41:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-04-22T18:41:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948501+0000", 2024-04-22T18:41:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77532387199999997 2024-04-22T18:41:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948912+0000", 2024-04-22T18:41:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041183000000000002 2024-04-22T18:41:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950438+0000", 2024-04-22T18:41:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015253199999999999 2024-04-22T18:41:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950488+0000", 2024-04-22T18:41:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0285999999999999e-05 2024-04-22T18:41:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950601+0000", 2024-04-22T18:41:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011244899999999999 2024-04-22T18:41:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954048+0000", 2024-04-22T18:41:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003447313 2024-04-22T18:41:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457373+0000", 2024-04-22T18:41:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50332465299999996 2024-04-22T18:41:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457430+0000", 2024-04-22T18:41:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7574999999999997e-05 2024-04-22T18:41:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457560+0000", 2024-04-22T18:41:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013023900000000001 2024-04-22T18:41:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:8 3.2 3:5f64590d:::7c47933f331d40689081b2bd8f9d7f7bdae1be1fdccb52d696047a3969e902b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.908723835000004, 2024-04-22T18:41:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3854200569999999, 2024-04-22T18:41:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173117+0000", 2024-04-22T18:41:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173393+0000", 2024-04-22T18:41:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000275278 2024-04-22T18:41:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173399+0000", 2024-04-22T18:41:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4590000000000003e-06 2024-04-22T18:41:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173406+0000", 2024-04-22T18:41:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2450000000000001e-06 2024-04-22T18:41:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173795+0000", 2024-04-22T18:41:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038912400000000002 2024-04-22T18:41:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173804+0000", 2024-04-22T18:41:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-04-22T18:41:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.949506+0000", 2024-04-22T18:41:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77570185700000005 2024-04-22T18:41:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950166+0000", 2024-04-22T18:41:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066022800000000001 2024-04-22T18:41:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352843+0000", 2024-04-22T18:41:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40267661799999999 2024-04-22T18:41:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352911+0000", 2024-04-22T18:41:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8126000000000005e-05 2024-04-22T18:41:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.353031+0000", 2024-04-22T18:41:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119919 2024-04-22T18:41:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453622+0000", 2024-04-22T18:41:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100590793 2024-04-22T18:41:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558446+0000", 2024-04-22T18:41:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104824073 2024-04-22T18:41:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558479+0000", 2024-04-22T18:41:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3362999999999999e-05 2024-04-22T18:41:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558538+0000", 2024-04-22T18:41:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9046999999999999e-05 2024-04-22T18:41:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:10 3.2 3:518c37f9:::851eca84dafbf2f780c4f07291110c453349bdc9c623c17bdcd26e27badb30ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.908430241999994, 2024-04-22T18:41:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3852527729999999, 2024-04-22T18:41:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173411+0000", 2024-04-22T18:41:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173749+0000", 2024-04-22T18:41:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000338029 2024-04-22T18:41:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173755+0000", 2024-04-22T18:41:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0530000000000003e-06 2024-04-22T18:41:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173759+0000", 2024-04-22T18:41:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8190000000000002e-06 2024-04-22T18:41:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173855+0000", 2024-04-22T18:41:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5927999999999994e-05 2024-04-22T18:41:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173862+0000", 2024-04-22T18:41:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4449999999999999e-06 2024-04-22T18:41:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950835+0000", 2024-04-22T18:41:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77697248299999999 2024-04-22T18:41:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951632+0000", 2024-04-22T18:41:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079752299999999996 2024-04-22T18:41:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355407+0000", 2024-04-22T18:41:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40377480500000001 2024-04-22T18:41:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355462+0000", 2024-04-22T18:41:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4577999999999998e-05 2024-04-22T18:41:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355571+0000", 2024-04-22T18:41:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108964 2024-04-22T18:41:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453625+0000", 2024-04-22T18:41:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098054397000000001 2024-04-22T18:41:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558574+0000", 2024-04-22T18:41:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104948571 2024-04-22T18:41:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558602+0000", 2024-04-22T18:41:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7966999999999998e-05 2024-04-22T18:41:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558665+0000", 2024-04-22T18:41:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3032000000000002e-05 2024-04-22T18:41:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:11 3.2 3:718aa8aa:::9603c4f8e269a436b2f0bf376c5374e77bdd7cf58fccae60e4e65ac4696806d4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.908078623999998, 2024-04-22T18:41:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4852604440000001, 2024-04-22T18:41:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173813+0000", 2024-04-22T18:41:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0655999999999997e-05 2024-04-22T18:41:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173819+0000", 2024-04-22T18:41:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.468e-06 2024-04-22T18:41:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173822+0000", 2024-04-22T18:41:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4879999999999999e-06 2024-04-22T18:41:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173983+0000", 2024-04-22T18:41:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160519 2024-04-22T18:41:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173991+0000", 2024-04-22T18:41:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6490000000000008e-06 2024-04-22T18:41:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951688+0000", 2024-04-22T18:41:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77769743800000002 2024-04-22T18:41:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952419+0000", 2024-04-22T18:41:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073067499999999997 2024-04-22T18:41:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355901+0000", 2024-04-22T18:41:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40348237599999998 2024-04-22T18:41:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355953+0000", 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1477000000000002e-05 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356060+0000", 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107114 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453629+0000", 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097569127000000005 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658807+0000", 2024-04-22T18:41:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20517829800000001 2024-04-22T18:41:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658876+0000", 2024-04-22T18:41:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8454000000000006e-05 2024-04-22T18:41:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659024+0000", 2024-04-22T18:41:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148338 2024-04-22T18:41:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:12 3.2 3:475e1cd9:::efaab2bd13543cdbb143e869e20eb0ec424f7f5202707859b241ed91d4674936:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.908015680000005, 2024-04-22T18:41:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4853405259999999, 2024-04-22T18:41:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173940+0000", 2024-04-22T18:41:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011465500000000001 2024-04-22T18:41:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173946+0000", 2024-04-22T18:41:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5289999999999999e-06 2024-04-22T18:41:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173950+0000", 2024-04-22T18:41:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8290000000000001e-06 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174277+0000", 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032761299999999998 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174285+0000", 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.52e-06 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952455+0000", 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77817000700000005 2024-04-22T18:41:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954035+0000", 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001580078 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356379+0000", 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40234440399999999 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356428+0000", 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8832999999999998e-05 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356532+0000", 2024-04-22T18:41:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103235 2024-04-22T18:41:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453632+0000", 2024-04-22T18:41:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097100346000000004 2024-04-22T18:41:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659069+0000", 2024-04-22T18:41:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20543763800000001 2024-04-22T18:41:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659105+0000", 2024-04-22T18:41:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5240000000000001e-05 2024-04-22T18:41:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659167+0000", 2024-04-22T18:41:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2207999999999998e-05 2024-04-22T18:41:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:16 3.2 3:47c76cc8:::c00d18458eb20c0c0aa9c612cf9793a68445ab0ea7b9e650ecd0072c60a96c0f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.907887720999994, 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.485347744, 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174234+0000", 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028047599999999999 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174240+0000", 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2079999999999997e-06 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174244+0000", 2024-04-22T18:41:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5810000000000001e-06 2024-04-22T18:41:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174670+0000", 2024-04-22T18:41:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042640000000000001 2024-04-22T18:41:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174675+0000", 2024-04-22T18:41:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8709999999999996e-06 2024-04-22T18:41:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054168+0000", 2024-04-22T18:41:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87949257300000006 2024-04-22T18:41:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054877+0000", 2024-04-22T18:41:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070884300000000002 2024-04-22T18:41:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356836+0000", 2024-04-22T18:41:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30195926000000001 2024-04-22T18:41:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356887+0000", 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0767999999999999e-05 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356991+0000", 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104072 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453636+0000", 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096644968999999997 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659207+0000", 2024-04-22T18:41:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20557154699999999 2024-04-22T18:41:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659240+0000", 2024-04-22T18:41:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2665e-05 2024-04-22T18:41:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659302+0000", 2024-04-22T18:41:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2234000000000001e-05 2024-04-22T18:41:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:17 3.2 3:6ef31daa:::1cf4ff496c4480e58510f607481d976b47d84868e2c54016273c38baf590f3b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.907593441000003, 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4851703729999999, 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174248+0000", 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174626+0000", 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037839699999999999 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174633+0000", 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0990000000000004e-06 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174636+0000", 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5540000000000002e-06 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174790+0000", 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153807 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174793+0000", 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4970000000000001e-06 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054902+0000", 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88010886300000002 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055366+0000", 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046336199999999999 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.452973+0000", 2024-04-22T18:41:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39760783999999999 2024-04-22T18:41:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453046+0000", 2024-04-22T18:41:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2374999999999997e-05 2024-04-22T18:41:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453193+0000", 2024-04-22T18:41:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014722499999999999 2024-04-22T18:41:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554698+0000", 2024-04-22T18:41:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101505096 2024-04-22T18:41:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659337+0000", 2024-04-22T18:41:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104639236 2024-04-22T18:41:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659365+0000", 2024-04-22T18:41:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7594e-05 2024-04-22T18:41:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659419+0000", 2024-04-22T18:41:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4058999999999999e-05 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:18 3.2 3:792a4ed0:::e1c814961b3d1531fb5ed475b3d496a0405c3a819b68a92ff9b2391fe912370f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.907197185000001, 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.585036954, 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174640+0000", 2024-04-22T18:41:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-04-22T18:41:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174758+0000", 2024-04-22T18:41:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011819099999999999 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174764+0000", 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6729999999999997e-06 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174767+0000", 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3830000000000001e-06 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373669+0000", 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198901989 2024-04-22T18:41:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373686+0000", 2024-04-22T18:41:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7286e-05 2024-04-22T18:41:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055408+0000", 2024-04-22T18:41:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68172133400000001 2024-04-22T18:41:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055938+0000", 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053016599999999999 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453668+0000", 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39773030799999998 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453717+0000", 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8884000000000002e-05 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453823+0000", 2024-04-22T18:41:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105791 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554715+0000", 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100891628 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759439+0000", 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20472459400000001 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759525+0000", 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5653000000000001e-05 2024-04-22T18:41:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759682+0000", 2024-04-22T18:41:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015711800000000001 2024-04-22T18:41:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:75 3.1 3:ab43a94d:::f38e6e97cb24e2b90da964fee5d2d6d8674386870fc552915e0c3fdbefd4b126:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.834320667999997, 2024-04-22T18:41:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60149928399999997, 2024-04-22T18:41:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247520+0000", 2024-04-22T18:41:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247527+0000", 2024-04-22T18:41:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1670000000000002e-06 2024-04-22T18:41:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247539+0000", 2024-04-22T18:41:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503e-05 2024-04-22T18:41:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247549+0000", 2024-04-22T18:41:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0000000000000001e-05 2024-04-22T18:41:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247796+0000", 2024-04-22T18:41:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024712699999999998 2024-04-22T18:41:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247955+0000", 2024-04-22T18:41:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158534 2024-04-22T18:41:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.343985+0000", 2024-04-22T18:41:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096030344000000004 2024-04-22T18:41:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.344035+0000", 2024-04-22T18:41:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9969999999999998e-05 2024-04-22T18:41:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.745806+0000", 2024-04-22T18:41:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.401771029 2024-04-22T18:41:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746040+0000", 2024-04-22T18:41:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023339300000000001 2024-04-22T18:41:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746198+0000", 2024-04-22T18:41:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015793299999999999 2024-04-22T18:41:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848737+0000", 2024-04-22T18:41:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10253898 2024-04-22T18:41:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848751+0000", 2024-04-22T18:41:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4694e-05 2024-04-22T18:41:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848825+0000", 2024-04-22T18:41:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3443999999999997e-05 2024-04-22T18:41:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849021+0000", 2024-04-22T18:41:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000196047 2024-04-22T18:41:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:76 3.1 3:b44829a0:::3788286a255d538503614e5f04049b0bb54cacaa317ab102b5dee8f5b640b296:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.834281601999997, 2024-04-22T18:41:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60168681599999996, 2024-04-22T18:41:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247564+0000", 2024-04-22T18:41:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1150000000000004e-06 2024-04-22T18:41:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247570+0000", 2024-04-22T18:41:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3940000000000001e-06 2024-04-22T18:41:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247576+0000", 2024-04-22T18:41:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3619999999999997e-06 2024-04-22T18:41:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.340845+0000", 2024-04-22T18:41:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.093269199999999997 2024-04-22T18:41:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.341043+0000", 2024-04-22T18:41:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019834400000000001 2024-04-22T18:41:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345106+0000", 2024-04-22T18:41:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040629780000000001 2024-04-22T18:41:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345160+0000", 2024-04-22T18:41:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4246999999999997e-05 2024-04-22T18:41:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747047+0000", 2024-04-22T18:41:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40188678300000003 2024-04-22T18:41:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747162+0000", 2024-04-22T18:41:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011483400000000001 2024-04-22T18:41:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747429+0000", 2024-04-22T18:41:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026700299999999998 2024-04-22T18:41:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849032+0000", 2024-04-22T18:41:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101603203 2024-04-22T18:41:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849095+0000", 2024-04-22T18:41:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2597999999999999e-05 2024-04-22T18:41:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849136+0000", 2024-04-22T18:41:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1251000000000002e-05 2024-04-22T18:41:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849247+0000", 2024-04-22T18:41:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011127 2024-04-22T18:41:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3526 4.c 4:314c2fec:::smithi08937929-22:head [setxattr _header (56) in=7b,truncate 3092521] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.852644822000002, 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60463626000000004, 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3526 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229200+0000", 2024-04-22T18:41:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2320000000000001e-06 2024-04-22T18:41:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229203+0000", 2024-04-22T18:41:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:41:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329292+0000", 2024-04-22T18:41:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100088891 2024-04-22T18:41:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329357+0000", 2024-04-22T18:41:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5115000000000003e-05 2024-04-22T18:41:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329453+0000", 2024-04-22T18:41:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5802000000000002e-05 2024-04-22T18:41:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329594+0000", 2024-04-22T18:41:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014157500000000001 2024-04-22T18:41:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.330172+0000", 2024-04-22T18:41:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057782799999999996 2024-04-22T18:41:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833712+0000", 2024-04-22T18:41:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50353975299999998 2024-04-22T18:41:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833774+0000", 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1832e-05 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833834+0000", 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9809999999999998e-05 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:339 3.2 3:4159b8d1:::bb1f8e997816dd9d2f29d36c4c1859fcaffbf8763b5c8d63c7a818d8d84bac37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.306391063, 2024-04-22T18:41:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0092773479999999, 2024-04-22T18:41:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775450+0000", 2024-04-22T18:41:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775457+0000", 2024-04-22T18:41:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2949999999999996e-06 2024-04-22T18:41:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775467+0000", 2024-04-22T18:41:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0180000000000001e-05 2024-04-22T18:41:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775475+0000", 2024-04-22T18:41:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.678e-06 2024-04-22T18:41:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.877787+0000", 2024-04-22T18:41:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102312559 2024-04-22T18:41:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.880686+0000", 2024-04-22T18:41:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002898748 2024-04-22T18:41:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.282867+0000", 2024-04-22T18:41:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.402180961 2024-04-22T18:41:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.282945+0000", 2024-04-22T18:41:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7775999999999995e-05 2024-04-22T18:41:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283065+0000", 2024-04-22T18:41:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0001199 2024-04-22T18:41:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.285514+0000", 2024-04-22T18:41:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0024491729999999998 2024-04-22T18:41:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784375+0000", 2024-04-22T18:41:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49886154100000002 2024-04-22T18:41:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784475+0000", 2024-04-22T18:41:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9074999999999996e-05 2024-04-22T18:41:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784728+0000", 2024-04-22T18:41:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025386799999999998 2024-04-22T18:41:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:340 3.2 3:4bbc2d06:::e1c179ab87fb8e08c4a1dfc0c7cc725138181c5734d6df18f3f2545b93731e97:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.306358643999999, 2024-04-22T18:41:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009550757, 2024-04-22T18:41:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775487+0000", 2024-04-22T18:41:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6839999999999998e-06 2024-04-22T18:41:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775492+0000", 2024-04-22T18:41:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6899999999999997e-06 2024-04-22T18:41:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775497+0000", 2024-04-22T18:41:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8590000000000004e-06 2024-04-22T18:41:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.880737+0000", 2024-04-22T18:41:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105240267 2024-04-22T18:41:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881308+0000", 2024-04-22T18:41:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057098399999999997 2024-04-22T18:41:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283810+0000", 2024-04-22T18:41:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40250177500000001 2024-04-22T18:41:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283883+0000", 2024-04-22T18:41:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2430999999999998e-05 2024-04-22T18:41:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284019+0000", 2024-04-22T18:41:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013652299999999999 2024-04-22T18:41:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.287390+0000", 2024-04-22T18:41:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033710089999999999 2024-04-22T18:41:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784829+0000", 2024-04-22T18:41:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49743879400000002 2024-04-22T18:41:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784893+0000", 2024-04-22T18:41:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4184999999999996e-05 2024-04-22T18:41:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785034+0000", 2024-04-22T18:41:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014114600000000001 2024-04-22T18:41:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:341 3.2 3:5b8bea68:::ab73be59042e3e18c4fe1fdbc2bbcc205779424ddf3bb423a595330be9026399:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.306339292000001, 2024-04-22T18:41:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009670804, 2024-04-22T18:41:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775502+0000", 2024-04-22T18:41:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775505+0000", 2024-04-22T18:41:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0809999999999998e-06 2024-04-22T18:41:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775511+0000", 2024-04-22T18:41:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2039999999999996e-06 2024-04-22T18:41:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775515+0000", 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8790000000000002e-06 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881376+0000", 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105861 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881839+0000", 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046217200000000001 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284569+0000", 2024-04-22T18:41:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40273058900000003 2024-04-22T18:41:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284730+0000", 2024-04-22T18:41:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077800000000001 2024-04-22T18:41:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284923+0000", 2024-04-22T18:41:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000193253 2024-04-22T18:41:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.287593+0000", 2024-04-22T18:41:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026699359999999999 2024-04-22T18:41:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785074+0000", 2024-04-22T18:41:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49748046000000001 2024-04-22T18:41:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785103+0000", 2024-04-22T18:41:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8935e-05 2024-04-22T18:41:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785174+0000", 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1001999999999995e-05 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:342 3.2 3:634dda9f:::10622dc7de8047e7908c0b5cca7a43d84cdf1b7629e6f7e159b8207feba82b01:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.30632108, 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.022499579, 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775524+0000", 2024-04-22T18:41:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9629999999999998e-06 2024-04-22T18:41:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775529+0000", 2024-04-22T18:41:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0320000000000003e-06 2024-04-22T18:41:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775533+0000", 2024-04-22T18:41:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7380000000000002e-06 2024-04-22T18:41:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881884+0000", 2024-04-22T18:41:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106350294 2024-04-22T18:41:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.882492+0000", 2024-04-22T18:41:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060872699999999999 2024-04-22T18:41:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381559+0000", 2024-04-22T18:41:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49906679199999998 2024-04-22T18:41:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381677+0000", 2024-04-22T18:41:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011743699999999999 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381899+0000", 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022272599999999999 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.383440+0000", 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015407719999999999 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.797827+0000", 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.414386741 2024-04-22T18:41:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.797887+0000", 2024-04-22T18:41:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9889000000000002e-05 2024-04-22T18:41:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.798021+0000", 2024-04-22T18:41:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000133948 2024-04-22T18:41:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:343 3.2 3:7db7298e:::00597d8527769aa920807495342cf3ba06efb8f507feae865dd89cd8ad10bda0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.306302124999998, 2024-04-22T18:41:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.110995373, 2024-04-22T18:41:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775542+0000", 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9079999999999999e-06 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775547+0000", 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.994e-06 2024-04-22T18:41:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775553+0000", 2024-04-22T18:41:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2299999999999999e-06 2024-04-22T18:41:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.882519+0000", 2024-04-22T18:41:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106966453 2024-04-22T18:41:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883103+0000", 2024-04-22T18:41:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058419400000000001 2024-04-22T18:41:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.581957+0000", 2024-04-22T18:41:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69885328599999996 2024-04-22T18:41:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582023+0000", 2024-04-22T18:41:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6210999999999993e-05 2024-04-22T18:41:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582146+0000", 2024-04-22T18:41:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123669 2024-04-22T18:41:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.682293+0000", 2024-04-22T18:41:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10014662100000001 2024-04-22T18:41:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886410+0000", 2024-04-22T18:41:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20411678899999999 2024-04-22T18:41:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886445+0000", 2024-04-22T18:41:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5462e-05 2024-04-22T18:41:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886535+0000", 2024-04-22T18:41:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9995999999999997e-05 2024-04-22T18:41:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:344 3.1 3:9da67ae4:::04c6b48a110b3f74550f78f0899df697160a1eeab7f41463f6ac18091ca67078:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.306266818000001, 2024-04-22T18:41:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60604869800000005, 2024-04-22T18:41:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775578+0000", 2024-04-22T18:41:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3440000000000001e-06 2024-04-22T18:41:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775584+0000", 2024-04-22T18:41:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6969999999999998e-06 2024-04-22T18:41:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775589+0000", 2024-04-22T18:41:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0540000000000002e-06 2024-04-22T18:41:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775729+0000", 2024-04-22T18:41:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014061299999999999 2024-04-22T18:41:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776045+0000", 2024-04-22T18:41:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000315168 2024-04-22T18:41:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080584+0000", 2024-04-22T18:41:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30453977700000001 2024-04-22T18:41:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080635+0000", 2024-04-22T18:41:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1007999999999997e-05 2024-04-22T18:41:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080741+0000", 2024-04-22T18:41:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105646 2024-04-22T18:41:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.180821+0000", 2024-04-22T18:41:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10007980900000001 2024-04-22T18:41:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381437+0000", 2024-04-22T18:41:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20061589399999999 2024-04-22T18:41:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381491+0000", 2024-04-22T18:41:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4531999999999999e-05 2024-04-22T18:41:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381624+0000", 2024-04-22T18:41:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000132745 2024-04-22T18:41:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:345 3.1 3:bd847077:::0e274616605e19dd653fcf05d310933ab02835574414ca18340b3fc325d2da4f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.306242458, 2024-04-22T18:41:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.90876140699999997, 2024-04-22T18:41:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775599+0000", 2024-04-22T18:41:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775602+0000", 2024-04-22T18:41:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.976e-06 2024-04-22T18:41:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775607+0000", 2024-04-22T18:41:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2229999999999998e-06 2024-04-22T18:41:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775612+0000", 2024-04-22T18:41:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6539999999999998e-06 2024-04-22T18:41:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776072+0000", 2024-04-22T18:41:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046006299999999998 2024-04-22T18:41:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776256+0000", 2024-04-22T18:41:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000183735 2024-04-22T18:41:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081157+0000", 2024-04-22T18:41:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30490169099999997 2024-04-22T18:41:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081203+0000", 2024-04-22T18:41:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5677999999999999e-05 2024-04-22T18:41:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081301+0000", 2024-04-22T18:41:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7893999999999995e-05 2024-04-22T18:41:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.180827+0000", 2024-04-22T18:41:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099525725999999995 2024-04-22T18:41:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684197+0000", 2024-04-22T18:41:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50336990699999995 2024-04-22T18:41:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684247+0000", 2024-04-22T18:41:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0195000000000003e-05 2024-04-22T18:41:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684361+0000", 2024-04-22T18:41:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114189 2024-04-22T18:41:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4267 3.2 3:584fd9a2:::cda848272f4ef42ebafc2089aaddeacaf12eabd82aef5e42e32ae93a8a5da0b6:head [write 0~2635776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.225892601999998, 2024-04-22T18:41:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.72658671900000005, 2024-04-22T18:41:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.204 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4267 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855946+0000", 2024-04-22T18:41:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.856984+0000", 2024-04-22T18:41:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001037764 2024-04-22T18:41:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.857003+0000", 2024-04-22T18:41:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8876e-05 2024-04-22T18:41:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.857016+0000", 2024-04-22T18:41:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3337e-05 2024-04-22T18:41:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883170+0000", 2024-04-22T18:41:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026154047 2024-04-22T18:41:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883262+0000", 2024-04-22T18:41:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1996999999999993e-05 2024-04-22T18:41:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883360+0000", 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7429000000000005e-05 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.887277+0000", 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039169449999999998 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582423+0000", 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69514625200000002 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582458+0000", 2024-04-22T18:41:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5293999999999998e-05 2024-04-22T18:41:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582536+0000", 2024-04-22T18:41:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7855000000000006e-05 2024-04-22T18:41:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:36.209 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:36.209 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:36.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:36.212 INFO:tasks.rados.rados.0.smithi089.stdout:1958: finishing write tid 1 to smithi08937929-20 2024-04-22T18:41:36.212 INFO:tasks.rados.rados.0.smithi089.stdout:1956: finishing set_chunk to oid 4 2024-04-22T18:41:36.212 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 4 v 138 (ObjNum 361 snap 0 seq_num 361) dirty exists 2024-04-22T18:41:36.212 INFO:tasks.rados.rados.0.smithi089.stdout:1958: finishing write tid 2 to smithi08937929-20 2024-04-22T18:41:36.212 INFO:tasks.rados.rados.0.smithi089.stdout:1958: finishing write tid 3 to smithi08937929-20 2024-04-22T18:41:36.213 INFO:tasks.rados.rados.0.smithi089.stdout:1963: seq_num 419 ranges {642133=728746,2119014=532478} 2024-04-22T18:41:36.213 INFO:tasks.rados.rados.0.smithi089.stdout:1963: writing smithi08937929-37 from 642133 to 1370879 tid 1 2024-04-22T18:41:36.213 INFO:tasks.rados.rados.0.smithi089.stdout:1963: writing smithi08937929-37 from 2119014 to 2651492 tid 2 2024-04-22T18:41:36.213 INFO:tasks.rados.rados.0.smithi089.stdout:1953: done (10 left) 2024-04-22T18:41:36.213 INFO:tasks.rados.rados.0.smithi089.stdout:1954: done (9 left) 2024-04-22T18:41:36.213 INFO:tasks.rados.rados.0.smithi089.stdout:1955: done (8 left) 2024-04-22T18:41:36.213 INFO:tasks.rados.rados.0.smithi089.stdout:1956: done (7 left) 2024-04-22T18:41:36.213 INFO:tasks.rados.rados.0.smithi089.stdout:1957: done (6 left) 2024-04-22T18:41:36.213 INFO:tasks.rados.rados.0.smithi089.stdout:1964: write (excl) oid 33 current snap is 0 2024-04-22T18:41:36.213 INFO:tasks.rados.rados.0.smithi089.stdout:1958: finishing write tid 4 to smithi08937929-20 2024-04-22T18:41:36.213 INFO:tasks.rados.rados.0.smithi089.stdout:1958: oid 20 updating version 0 to 31 2024-04-22T18:41:36.214 INFO:tasks.rados.rados.0.smithi089.stdout:1958: oid 20 updating version 31 to 32 2024-04-22T18:41:36.214 INFO:tasks.rados.rados.0.smithi089.stdout:1958: oid 20 updating version 32 to 33 2024-04-22T18:41:36.214 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 33 (ObjNum 418 snap 0 seq_num 418) dirty exists 2024-04-22T18:41:36.214 INFO:tasks.rados.rados.0.smithi089.stdout:1958: left oid 20 (ObjNum 418 snap 0 seq_num 418) 2024-04-22T18:41:36.214 INFO:tasks.rados.rados.0.smithi089.stdout:1962: expect (ObjNum 407 snap 0 seq_num 407) 2024-04-22T18:41:36.214 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:36.212+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:36.297 INFO:tasks.rados.rados.0.smithi089.stdout:1959: finishing set_chunk to oid 1 2024-04-22T18:41:36.297 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 57 (ObjNum 347 snap 0 seq_num 347) dirty exists 2024-04-22T18:41:36.297 INFO:tasks.rados.rados.0.smithi089.stdout:1963: finishing write tid 1 to smithi08937929-37 2024-04-22T18:41:36.297 INFO:tasks.rados.rados.0.smithi089.stdout:1960: expect (ObjNum 405 snap 0 seq_num 405) 2024-04-22T18:41:36.311 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:36.312+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:36.327 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:36.328 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:36.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.328 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "MOSDPGRecoveryDeleteReply(3.3 e160,150 3:fd315df5:::bd3bfcf55be7a07c1dc681a4f992f514687e0a7896056eb7190f3cb12c221602:head,83'256)", 2024-04-22T18:41:36.328 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:36.254345+0000", 2024-04-22T18:41:36.328 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.075295505999999998, 2024-04-22T18:41:36.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.075312751999999997, 2024-04-22T18:41:36.328 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.328 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "queued for pg", 2024-04-22T18:41:36.328 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:36.254345+0000", 2024-04-22T18:41:36.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:36.254345+0000", 2024-04-22T18:41:36.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:36.254343+0000", 2024-04-22T18:41:36.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:36.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:36.254354+0000", 2024-04-22T18:41:36.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1202e-05 2024-04-22T18:41:36.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:36.254372+0000", 2024-04-22T18:41:36.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8289000000000001e-05 2024-04-22T18:41:36.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:36.254385+0000", 2024-04-22T18:41:36.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2689e-05 2024-04-22T18:41:36.330 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.330 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.331 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.331 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.331 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-22T18:41:36.331 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 1 2024-04-22T18:41:36.331 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:36.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:36.412 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:36.413+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:36.440 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:36.440 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:36.440 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:36.440 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:36.440 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:36.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:36.494 INFO:tasks.rados.rados.0.smithi089.stdout:1963: finishing write tid 2 to smithi08937929-37 2024-04-22T18:41:36.494 INFO:tasks.rados.rados.0.smithi089.stdout:1963: finishing write tid 3 to smithi08937929-37 2024-04-22T18:41:36.494 INFO:tasks.rados.rados.0.smithi089.stdout:1963: finishing write tid 4 to smithi08937929-37 2024-04-22T18:41:36.494 INFO:tasks.rados.rados.0.smithi089.stdout:1963: oid 37 updating version 0 to 158 2024-04-22T18:41:36.494 INFO:tasks.rados.rados.0.smithi089.stdout:1963: oid 37 version 158 is already newer than 156 2024-04-22T18:41:36.494 INFO:tasks.rados.rados.0.smithi089.stdout:1963: oid 37 version 158 is already newer than 157 2024-04-22T18:41:36.495 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 37 v 158 (ObjNum 419 snap 0 seq_num 419) dirty exists 2024-04-22T18:41:36.495 INFO:tasks.rados.rados.0.smithi089.stdout:1963: left oid 37 (ObjNum 419 snap 0 seq_num 419) 2024-04-22T18:41:36.495 INFO:tasks.rados.rados.0.smithi089.stdout:1961: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:41:36.533 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:36.534+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:36.592 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:36.592 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:36.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:36.592 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:36.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.592 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:36.592 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:36.592 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 161.92974342400001, 2024-04-22T18:41:36.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:41:36.593 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.593 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.593 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.593 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.593 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.593 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:41:36.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.593 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:36.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:36.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:36.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:41:36.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:41:36.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:41:36.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:41:36.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:41:36.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:41:36.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:41:36.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:41:36.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:36.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:41:36.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:41:36.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:36.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:41:36.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:41:36.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:36.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:41:36.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:41:36.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:41:36.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:41:36.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:36.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:41:36.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:41:36.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:41:36.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:41:36.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.598 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.598 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:36.598 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:36.598 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 161.828735247, 2024-04-22T18:41:36.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:41:36.599 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.599 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:36.599 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.599 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.599 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.599 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:41:36.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.599 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:36.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:36.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:36.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:41:36.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:41:36.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:41:36.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:41:36.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:41:36.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:41:36.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:41:36.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:41:36.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:36.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:41:36.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:41:36.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:41:36.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:41:36.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:36.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:41:36.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:41:36.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:41:36.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:41:36.603 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.603 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.603 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.604 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:36.604 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:36.604 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 161.828724862, 2024-04-22T18:41:36.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:41:36.604 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.604 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.604 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.604 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.604 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.604 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:41:36.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.604 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:36.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:36.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:41:36.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:41:36.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:41:36.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:41:36.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:41:36.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:41:36.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:41:36.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:41:36.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:41:36.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:36.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:41:36.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:41:36.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:36.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:41:36.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:41:36.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:36.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:41:36.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:41:36.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:41:36.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:41:36.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:36.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:41:36.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:41:36.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:41:36.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:41:36.610 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.610 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.610 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.610 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:41:36.610 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:36.610 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 157.490974435, 2024-04-22T18:41:36.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:41:36.610 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.611 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.611 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.611 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.611 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.611 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:41:36.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.611 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:36.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:36.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:41:36.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:36.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:41:36.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:41:36.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:41:36.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:41:36.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:41:36.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:41:36.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:41:36.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:41:36.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:36.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:41:36.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:41:36.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:36.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:41:36.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:41:36.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:36.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:41:36.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:41:36.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:41:36.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:41:36.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:36.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:41:36.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:41:36.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:41:36.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:41:36.615 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.616 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.616 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.616 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:41:36.616 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:36.616 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.45632692000001, 2024-04-22T18:41:36.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:41:36.616 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.616 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:36.616 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.616 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.616 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.616 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:41:36.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.617 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:36.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:36.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:41:36.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:41:36.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:41:36.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:41:36.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:41:36.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:41:36.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:41:36.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:41:36.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:41:36.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:36.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:41:36.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:41:36.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:41:36.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:41:36.620 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.620 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.620 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.620 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4177 4.19 4.f75b2379 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:36.620 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:36.620 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.427273622999998, 2024-04-22T18:41:36.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76236081700000002, 2024-04-22T18:41:36.620 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.620 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:36.620 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.621 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.621 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.621 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4177 2024-04-22T18:41:36.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.621 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:36.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:36.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164847+0000", 2024-04-22T18:41:36.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:36.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164854+0000", 2024-04-22T18:41:36.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2620000000000001e-06 2024-04-22T18:41:36.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164864+0000", 2024-04-22T18:41:36.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0400999999999999e-05 2024-04-22T18:41:36.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164876+0000", 2024-04-22T18:41:36.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1204e-05 2024-04-22T18:41:36.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.623 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.623 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164975+0000", 2024-04-22T18:41:36.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9562999999999995e-05 2024-04-22T18:41:36.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164980+0000", 2024-04-22T18:41:36.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7310000000000001e-06 2024-04-22T18:41:36.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927200+0000", 2024-04-22T18:41:36.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.762219973 2024-04-22T18:41:36.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927209+0000", 2024-04-22T18:41:36.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9050000000000003e-06 2024-04-22T18:41:36.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.625 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.625 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.625 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4182 4.c 4.5345b7ac (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:36.625 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:36.625 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.427242304000004, 2024-04-22T18:41:36.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76185930999999996, 2024-04-22T18:41:36.625 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.625 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:36.625 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.625 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.625 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.626 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4182 2024-04-22T18:41:36.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.626 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:36.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:36.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:36.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164881+0000", 2024-04-22T18:41:36.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.305e-06 2024-04-22T18:41:36.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164885+0000", 2024-04-22T18:41:36.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.873e-06 2024-04-22T18:41:36.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164890+0000", 2024-04-22T18:41:36.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6299999999999997e-06 2024-04-22T18:41:36.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225475+0000", 2024-04-22T18:41:36.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060585181000000002 2024-04-22T18:41:36.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225483+0000", 2024-04-22T18:41:36.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6790000000000005e-06 2024-04-22T18:41:36.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926728+0000", 2024-04-22T18:41:36.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124542099999998 2024-04-22T18:41:36.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.629 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.629 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926739+0000", 2024-04-22T18:41:36.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0669000000000001e-05 2024-04-22T18:41:36.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.629 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.629 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.630 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4183 4.c 4.5345b7ac (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:36.630 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:36.630 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.427229048000001, 2024-04-22T18:41:36.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76187489500000005, 2024-04-22T18:41:36.630 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.630 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:36.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.630 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.630 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4183 2024-04-22T18:41:36.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.630 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:36.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:36.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164892+0000", 2024-04-22T18:41:36.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164894+0000", 2024-04-22T18:41:36.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.705e-06 2024-04-22T18:41:36.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164897+0000", 2024-04-22T18:41:36.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2739999999999999e-06 2024-04-22T18:41:36.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164901+0000", 2024-04-22T18:41:36.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.636e-06 2024-04-22T18:41:36.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225508+0000", 2024-04-22T18:41:36.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060607564000000003 2024-04-22T18:41:36.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225515+0000", 2024-04-22T18:41:36.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1829999999999999e-06 2024-04-22T18:41:36.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926759+0000", 2024-04-22T18:41:36.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.701244859 2024-04-22T18:41:36.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926767+0000", 2024-04-22T18:41:36.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0180000000000001e-06 2024-04-22T18:41:36.634 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.634 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.634 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.634 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4190 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:36.634 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:36.634 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.427212126000001, 2024-04-22T18:41:36.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.761884546, 2024-04-22T18:41:36.634 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.634 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:36.634 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.635 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.635 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.635 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4190 2024-04-22T18:41:36.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.635 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:36.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:36.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:36.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164912+0000", 2024-04-22T18:41:36.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6249999999999999e-06 2024-04-22T18:41:36.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164918+0000", 2024-04-22T18:41:36.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6039999999999998e-06 2024-04-22T18:41:36.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164922+0000", 2024-04-22T18:41:36.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9319999999999998e-06 2024-04-22T18:41:36.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225536+0000", 2024-04-22T18:41:36.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060613725 2024-04-22T18:41:36.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225540+0000", 2024-04-22T18:41:36.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5030000000000002e-06 2024-04-22T18:41:36.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926786+0000", 2024-04-22T18:41:36.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124673999999998 2024-04-22T18:41:36.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926794+0000", 2024-04-22T18:41:36.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7689999999999996e-06 2024-04-22T18:41:36.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.639 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.639 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4191 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:36.639 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:36.639 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.427196705, 2024-04-22T18:41:36.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76190695500000005, 2024-04-22T18:41:36.639 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.639 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:36.639 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.639 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.639 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4191 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.640 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:36.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164927+0000", 2024-04-22T18:41:36.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0959999999999999e-06 2024-04-22T18:41:36.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164937+0000", 2024-04-22T18:41:36.641 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0730999999999999e-05 2024-04-22T18:41:36.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.641 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.641 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.641 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164944+0000", 2024-04-22T18:41:36.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5130000000000002e-06 2024-04-22T18:41:36.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225565+0000", 2024-04-22T18:41:36.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060621421000000002 2024-04-22T18:41:36.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.642 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.642 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225569+0000", 2024-04-22T18:41:36.642 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.264e-06 2024-04-22T18:41:36.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.642 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926813+0000", 2024-04-22T18:41:36.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124469 2024-04-22T18:41:36.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.643 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.643 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926832+0000", 2024-04-22T18:41:36.643 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8580000000000002e-05 2024-04-22T18:41:36.643 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.643 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.643 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.643 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.643 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.643 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4192 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:36.644 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:36.644 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.427172097000003, 2024-04-22T18:41:36.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76191075399999997, 2024-04-22T18:41:36.644 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.644 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:36.644 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.644 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.644 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.644 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4192 2024-04-22T18:41:36.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.644 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.644 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.644 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.644 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:36.644 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:36.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:36.645 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.645 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.645 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164953+0000", 2024-04-22T18:41:36.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1099999999999996e-06 2024-04-22T18:41:36.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164965+0000", 2024-04-22T18:41:36.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2298e-05 2024-04-22T18:41:36.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.646 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164972+0000", 2024-04-22T18:41:36.646 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5230000000000001e-06 2024-04-22T18:41:36.646 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.646 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.646 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225581+0000", 2024-04-22T18:41:36.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060609356000000003 2024-04-22T18:41:36.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225584+0000", 2024-04-22T18:41:36.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.106e-06 2024-04-22T18:41:36.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.647 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.647 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926852+0000", 2024-04-22T18:41:36.647 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70126794999999997 2024-04-22T18:41:36.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.647 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.648 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.648 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926860+0000", 2024-04-22T18:41:36.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0919999999999998e-06 2024-04-22T18:41:36.648 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.648 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.648 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.648 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.648 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4197 4.13 4:ca46fde9:::smithi08937929-14:head [sparse-read 0~1375265,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:36.648 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:36.648 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.427145486999997, 2024-04-22T18:41:36.648 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.28429536, 2024-04-22T18:41:36.649 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.649 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:36.649 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.649 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.649 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.649 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4197 2024-04-22T18:41:36.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.649 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.649 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.649 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.649 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:36.649 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.649 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:36.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164975+0000", 2024-04-22T18:41:36.650 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:36.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.650 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.650 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.650 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164979+0000", 2024-04-22T18:41:36.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8959999999999996e-06 2024-04-22T18:41:36.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164983+0000", 2024-04-22T18:41:36.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.053e-06 2024-04-22T18:41:36.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.651 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164987+0000", 2024-04-22T18:41:36.651 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:41:36.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.651 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.264783+0000", 2024-04-22T18:41:36.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099796088000000005 2024-04-22T18:41:36.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.264789+0000", 2024-04-22T18:41:36.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4989999999999999e-06 2024-04-22T18:41:36.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.652 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139375+0000", 2024-04-22T18:41:36.652 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.87458606000000005 2024-04-22T18:41:36.652 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.652 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.652 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139478+0000", 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010219 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.446961+0000", 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30748385900000003 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.447101+0000", 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001393 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.653 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.449271+0000", 2024-04-22T18:41:36.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021707950000000001 2024-04-22T18:41:36.654 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.654 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.654 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.654 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.654 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4198 4.13 4:ca46fde9:::smithi08937929-14:head [sparse-read 0~1375265] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:36.654 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:36.654 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.427132118999999, 2024-04-22T18:41:36.654 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2868951370000001, 2024-04-22T18:41:36.654 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.654 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:36.654 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.654 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.655 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.655 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4198 2024-04-22T18:41:36.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.655 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:36.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.655 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.655 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:36.655 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.655 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:36.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164991+0000", 2024-04-22T18:41:36.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:41:36.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.656 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.656 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.656 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164994+0000", 2024-04-22T18:41:36.656 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2459999999999999e-06 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164997+0000", 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1200000000000002e-06 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.364948+0000", 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19995059200000001 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.364969+0000", 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1495999999999999e-05 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139617+0000", 2024-04-22T18:41:36.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77464723300000005 2024-04-22T18:41:36.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:36.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139643+0000", 2024-04-22T18:41:36.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6228e-05 2024-04-22T18:41:36.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451393+0000", 2024-04-22T18:41:36.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31175035200000001 2024-04-22T18:41:36.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:36.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451515+0000", 2024-04-22T18:41:36.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001214 2024-04-22T18:41:36.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451885+0000", 2024-04-22T18:41:36.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036992999999999998 2024-04-22T18:41:36.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.659 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.660 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4199 4.13 4:ca46fde9:::smithi08937929-14:head [read 0~2739541,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:36.660 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:36.660 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.427121096, 2024-04-22T18:41:36.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.291247504, 2024-04-22T18:41:36.660 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.660 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:36.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.660 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.660 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4199 2024-04-22T18:41:36.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.660 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165002+0000", 2024-04-22T18:41:36.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:41:36.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165006+0000", 2024-04-22T18:41:36.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:41:36.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165009+0000", 2024-04-22T18:41:36.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4599999999999999e-06 2024-04-22T18:41:36.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365066+0000", 2024-04-22T18:41:36.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20005682999999999 2024-04-22T18:41:36.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365078+0000", 2024-04-22T18:41:36.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2235000000000001e-05 2024-04-22T18:41:36.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139671+0000", 2024-04-22T18:41:36.663 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77459301700000005 2024-04-22T18:41:36.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:36.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139702+0000", 2024-04-22T18:41:36.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0673999999999998e-05 2024-04-22T18:41:36.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.664 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.664 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451930+0000", 2024-04-22T18:41:36.664 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31222817600000002 2024-04-22T18:41:36.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.664 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.452092+0000", 2024-04-22T18:41:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001616 2024-04-22T18:41:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.456248+0000", 2024-04-22T18:41:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041563700000000004 2024-04-22T18:41:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.665 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4200 4.13 4:c93c842e:::smithi08937929-25:head [tier-promote] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.427110216000003, 2024-04-22T18:41:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.177728814, 2024-04-22T18:41:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4200 2024-04-22T18:41:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165013+0000", 2024-04-22T18:41:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.635e-06 2024-04-22T18:41:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165018+0000", 2024-04-22T18:41:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2000000000000002e-06 2024-04-22T18:41:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165024+0000", 2024-04-22T18:41:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5910000000000003e-06 2024-04-22T18:41:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365110+0000", 2024-04-22T18:41:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20008601400000001 2024-04-22T18:41:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365114+0000", 2024-04-22T18:41:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.651e-06 2024-04-22T18:41:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139723+0000", 2024-04-22T18:41:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77460836899999996 2024-04-22T18:41:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139787+0000", 2024-04-22T18:41:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4707000000000002e-05 2024-04-22T18:41:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.459491+0000", 2024-04-22T18:41:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31970372000000002 2024-04-22T18:41:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.459567+0000", 2024-04-22T18:41:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6199999999999995e-05 2024-04-22T18:41:36.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.562522+0000", 2024-04-22T18:41:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102954925 2024-04-22T18:41:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.562585+0000", 2024-04-22T18:41:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3287000000000006e-05 2024-04-22T18:41:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:05.342740+0000", 2024-04-22T18:41:36.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.78015481499999995 2024-04-22T18:41:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4205 4.1 4.fdaa93c1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.427094066000002, 2024-04-22T18:41:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76220996100000005, 2024-04-22T18:41:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.673 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4205 2024-04-22T18:41:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165027+0000", 2024-04-22T18:41:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165029+0000", 2024-04-22T18:41:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3549999999999999e-06 2024-04-22T18:41:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165035+0000", 2024-04-22T18:41:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8549999999999998e-06 2024-04-22T18:41:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165040+0000", 2024-04-22T18:41:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9200000000000003e-06 2024-04-22T18:41:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165116+0000", 2024-04-22T18:41:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5834999999999995e-05 2024-04-22T18:41:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165123+0000", 2024-04-22T18:41:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2139999999999999e-06 2024-04-22T18:41:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927230+0000", 2024-04-22T18:41:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76210651100000004 2024-04-22T18:41:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927237+0000", 2024-04-22T18:41:36.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7409999999999992e-06 2024-04-22T18:41:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4268 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e133)", 2024-04-22T18:41:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:41:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 28.009386654, 2024-04-22T18:41:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70931800300000003, 2024-04-22T18:41:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.678 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4268 2024-04-22T18:41:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:41:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:41:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582733+0000", 2024-04-22T18:41:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582741+0000", 2024-04-22T18:41:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3329999999999999e-06 2024-04-22T18:41:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582751+0000", 2024-04-22T18:41:36.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0628999999999999e-05 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582759+0000", 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4120000000000004e-06 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582795+0000", 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6157e-05 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582862+0000", 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7261999999999994e-05 2024-04-22T18:41:36.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.987268+0000", 2024-04-22T18:41:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40440629700000003 2024-04-22T18:41:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.987771+0000", 2024-04-22T18:41:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050267400000000005 2024-04-22T18:41:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.291974+0000", 2024-04-22T18:41:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30420264499999999 2024-04-22T18:41:36.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.292008+0000", 2024-04-22T18:41:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4011999999999999e-05 2024-04-22T18:41:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.292053+0000", 2024-04-22T18:41:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5305000000000001e-05 2024-04-22T18:41:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4272 4.5 4:a6d237d2:::smithi08937929-43:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e133)", 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 27.806058453999999, 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60731304699999999, 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4272 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:36.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786068+0000", 2024-04-22T18:41:36.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9910000000000002e-06 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786079+0000", 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1287000000000001e-05 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786083+0000", 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0949999999999998e-06 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786121+0000", 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8096999999999998e-05 2024-04-22T18:41:36.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786219+0000", 2024-04-22T18:41:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7429000000000005e-05 2024-04-22T18:41:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.393376+0000", 2024-04-22T18:41:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60715760600000002 2024-04-22T18:41:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4437 4.19 4:9ea6f51f:::smithi08937929-34:head [read 304128~651776] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:41:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 18.67645168, 2024-04-22T18:41:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.042388732, 2024-04-22T18:41:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4437 2024-04-22T18:41:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:41:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:41:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915669+0000", 2024-04-22T18:41:36.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915673+0000", 2024-04-22T18:41:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6770000000000001e-06 2024-04-22T18:41:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915679+0000", 2024-04-22T18:41:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.976e-06 2024-04-22T18:41:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915684+0000", 2024-04-22T18:41:36.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3430000000000003e-06 2024-04-22T18:41:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915725+0000", 2024-04-22T18:41:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0751999999999999e-05 2024-04-22T18:41:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915732+0000", 2024-04-22T18:41:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6279999999999999e-06 2024-04-22T18:41:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.957228+0000", 2024-04-22T18:41:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0414961220000001 2024-04-22T18:41:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.957325+0000", 2024-04-22T18:41:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7422000000000003e-05 2024-04-22T18:41:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.958059+0000", 2024-04-22T18:41:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073350499999999996 2024-04-22T18:41:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.692 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4690 3.3 3:ee62d872:::cb076f2b64cfcdfafc1ec2cf19f2eaa46006a5a6952a3aeb9c7211f97fca91ae:head [write 0~441856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-04-22T18:41:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:26.592341+0000", 2024-04-22T18:41:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 9.9997804180000003, 2024-04-22T18:41:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.56667778400000002, 2024-04-22T18:41:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4690 2024-04-22T18:41:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:36.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592341+0000", 2024-04-22T18:41:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592341+0000", 2024-04-22T18:41:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592340+0000", 2024-04-22T18:41:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592697+0000", 2024-04-22T18:41:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000357334 2024-04-22T18:41:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592709+0000", 2024-04-22T18:41:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2211999999999999e-05 2024-04-22T18:41:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592718+0000", 2024-04-22T18:41:36.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0319999999999992e-06 2024-04-22T18:41:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.692859+0000", 2024-04-22T18:41:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014077 2024-04-22T18:41:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.692870+0000", 2024-04-22T18:41:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1225e-05 2024-04-22T18:41:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.995290+0000", 2024-04-22T18:41:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242006199999999 2024-04-22T18:41:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.995398+0000", 2024-04-22T18:41:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108136 2024-04-22T18:41:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.995513+0000", 2024-04-22T18:41:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115047 2024-04-22T18:41:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.996655+0000", 2024-04-22T18:41:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001141162 2024-04-22T18:41:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:27.158858+0000", 2024-04-22T18:41:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16220310299999999 2024-04-22T18:41:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:27.158901+0000", 2024-04-22T18:41:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3667000000000001e-05 2024-04-22T18:41:36.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:27.159019+0000", 2024-04-22T18:41:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117619 2024-04-22T18:41:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:36.699 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:36.700 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:41:36.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:36.702 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:36.701+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:36.703 INFO:tasks.rados.rados.0.smithi089.stdout:1964: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2024-04-22T18:41:36.703 INFO:tasks.rados.rados.0.smithi089.stdout:1964: writing smithi08937929-33 from 658940 to 1062935 tid 1 2024-04-22T18:41:36.703 INFO:tasks.rados.rados.0.smithi089.stdout:1964: writing smithi08937929-33 from 1777496 to 2453632 tid 2 2024-04-22T18:41:36.703 INFO:tasks.rados.rados.0.smithi089.stdout:1964: writing smithi08937929-33 from 2950951 to 3058940 tid 3 2024-04-22T18:41:36.703 INFO:tasks.rados.rados.0.smithi089.stdout:1958: done (6 left) 2024-04-22T18:41:36.704 INFO:tasks.rados.rados.0.smithi089.stdout:1959: done (5 left) 2024-04-22T18:41:36.704 INFO:tasks.rados.rados.0.smithi089.stdout:1960: done (4 left) 2024-04-22T18:41:36.704 INFO:tasks.rados.rados.0.smithi089.stdout:1961: done (3 left) 2024-04-22T18:41:36.704 INFO:tasks.rados.rados.0.smithi089.stdout:1962: done (2 left) 2024-04-22T18:41:36.704 INFO:tasks.rados.rados.0.smithi089.stdout:1963: done (1 left) 2024-04-22T18:41:36.704 INFO:tasks.rados.rados.0.smithi089.stdout:1965: read oid 48 snap -1 2024-04-22T18:41:36.704 INFO:tasks.rados.rados.0.smithi089.stdout:1965: expect (ObjNum 351 snap 0 seq_num 351) 2024-04-22T18:41:36.704 INFO:tasks.rados.rados.0.smithi089.stdout:1966: write oid 35 current snap is 0 2024-04-22T18:41:36.704 INFO:tasks.rados.rados.0.smithi089.stdout:1966: seq_num 421 ranges {675747=479244,1835978=536147} 2024-04-22T18:41:36.704 INFO:tasks.rados.rados.0.smithi089.stdout:1966: writing smithi08937929-35 from 675747 to 1154991 tid 1 2024-04-22T18:41:36.704 INFO:tasks.rados.rados.0.smithi089.stdout:1966: writing smithi08937929-35 from 1835978 to 2372125 tid 2 2024-04-22T18:41:36.705 INFO:tasks.rados.rados.0.smithi089.stdout:1967: set_chunk oid 24 target oid 2024-04-22T18:41:36.801 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:36.802+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:36.845 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:36.845 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:36.845 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:36.845 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:36.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:36.938 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:36.939+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:36.996 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:36.996 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:36.996 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:36.996 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:36.997 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:37.003 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:37.005+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:37.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:37.104 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:37.106+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:37.147 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:37.147 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:37.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:37.147 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:37.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.147 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:41:37.148 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:37.148 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 163.05332474400001, 2024-04-22T18:41:37.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:41:37.148 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.148 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.148 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.148 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.148 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.148 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:41:37.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.148 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:37.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:37.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:41:37.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:37.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:41:37.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:41:37.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:41:37.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:41:37.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:41:37.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:41:37.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:41:37.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:41:37.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:41:37.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:41:37.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:37.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:41:37.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:41:37.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:41:37.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:41:37.153 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.153 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.153 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.153 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:41:37.153 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:37.153 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 159.153826869, 2024-04-22T18:41:37.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:41:37.153 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.153 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.153 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:37.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:37.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:41:37.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:37.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:41:37.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:41:37.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:41:37.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:41:37.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:41:37.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:41:37.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:41:37.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:37.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:41:37.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:41:37.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:41:37.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:41:37.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:41:37.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:41:37.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:41:37.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:41:37.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:41:37.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:41:37.159 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.159 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.159 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.159 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:41:37.159 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:37.159 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.04552707799999, 2024-04-22T18:41:37.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:41:37.159 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.159 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.159 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.159 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.159 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.159 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:41:37.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:37.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:41:37.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:41:37.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:41:37.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:41:37.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:41:37.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:41:37.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:41:37.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:41:37.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:41:37.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:41:37.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:37.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:41:37.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:41:37.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:41:37.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:41:37.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:41:37.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:41:37.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:41:37.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:41:37.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:41:37.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:41:37.164 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.164 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.164 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.164 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:37.164 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:37.165 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.71398345, 2024-04-22T18:41:37.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:41:37.165 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.165 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.165 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:37.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:37.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:41:37.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:37.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:41:37.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:41:37.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:41:37.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:41:37.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:41:37.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:41:37.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:41:37.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:41:37.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:41:37.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:41:37.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:37.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:41:37.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:41:37.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:41:37.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:41:37.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:41:37.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:41:37.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:41:37.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:41:37.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:41:37.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:41:37.169 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.169 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.169 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.170 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:37.170 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:37.170 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.622897642, 2024-04-22T18:41:37.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:41:37.170 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.170 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.170 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.170 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.170 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.170 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:41:37.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.170 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:41:37.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:41:37.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:41:37.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:41:37.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:41:37.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:41:37.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:41:37.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:41:37.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:41:37.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:41:37.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:37.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:41:37.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:41:37.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:41:37.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:41:37.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:41:37.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:41:37.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:41:37.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:41:37.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.175 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.175 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:41:37.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:41:37.175 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.175 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.175 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.175 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:37.175 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:37.175 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.617097382, 2024-04-22T18:41:37.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:41:37.175 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.175 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.175 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.176 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.176 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.176 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:41:37.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.176 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:37.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:37.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:41:37.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:37.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:41:37.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:41:37.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:41:37.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:41:37.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:41:37.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:41:37.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:41:37.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:41:37.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:41:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:41:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:41:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:41:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:41:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:41:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:41:37.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:41:37.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:41:37.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:41:37.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.180 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.180 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:41:37.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:41:37.180 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.180 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.180 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.181 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:37.181 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:37.181 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.61235703899999, 2024-04-22T18:41:37.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:41:37.181 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.181 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.181 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.181 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.181 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.181 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:41:37.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.182 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:37.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:37.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:41:37.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:37.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:41:37.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:41:37.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:41:37.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:41:37.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:41:37.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:41:37.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:41:37.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:41:37.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:41:37.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:41:37.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:37.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:41:37.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:41:37.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:41:37.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:41:37.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:41:37.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:41:37.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:41:37.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:41:37.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:41:37.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:41:37.186 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.186 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.186 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.186 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:37.187 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:37.187 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.61188414099999, 2024-04-22T18:41:37.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:41:37.187 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.187 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.187 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.187 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.187 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.187 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:41:37.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.187 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:37.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:37.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:41:37.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:37.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:41:37.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:41:37.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:41:37.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:41:37.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:41:37.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:41:37.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:41:37.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:41:37.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:41:37.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:41:37.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.190 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:37.190 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:41:37.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:41:37.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:41:37.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:41:37.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:41:37.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:41:37.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:41:37.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:41:37.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:41:37.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:41:37.192 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.192 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.192 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.192 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:37.192 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:37.192 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.61186422, 2024-04-22T18:41:37.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:41:37.192 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.193 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:37.193 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.193 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.193 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.193 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:41:37.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.193 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:37.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:37.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:37.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:37.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:41:37.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:41:37.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:37.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:41:37.196 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.197 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.197 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.197 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:37.197 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:37.197 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 139.30254045500001, 2024-04-22T18:41:37.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:41:37.197 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.197 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.198 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:37.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:37.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:41:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:41:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:41:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:41:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:41:37.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:41:37.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:41:37.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:41:37.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:41:37.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:41:37.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:41:37.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:37.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:41:37.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:41:37.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.201 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:41:37.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:41:37.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.201 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:41:37.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:41:37.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:41:37.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:41:37.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:41:37.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:41:37.202 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.203 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.203 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.203 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:37.203 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:37.203 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 139.30233291900001, 2024-04-22T18:41:37.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:41:37.203 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.203 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.203 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:37.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:37.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:41:37.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:37.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:41:37.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:41:37.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:41:37.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:41:37.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:41:37.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:41:37.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:41:37.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:41:37.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:41:37.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:41:37.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:37.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:41:37.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:41:37.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:41:37.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:41:37.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:41:37.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:41:37.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:41:37.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:41:37.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:41:37.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:41:37.208 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.208 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.208 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.208 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:37.208 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:37.208 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 120.436683186, 2024-04-22T18:41:37.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:41:37.209 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.209 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.209 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:37.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:37.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:41:37.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:37.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:41:37.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:41:37.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:41:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:41:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:41:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:41:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:41:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:41:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:41:37.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:41:37.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:37.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:41:37.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:41:37.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:41:37.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:41:37.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:41:37.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:41:37.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:41:37.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:41:37.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:41:37.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:41:37.213 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.213 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.213 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.214 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:37.214 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:37.214 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 120.22284480099999, 2024-04-22T18:41:37.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:41:37.214 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.214 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.214 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:37.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:37.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:41:37.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:37.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:41:37.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:41:37.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:41:37.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:41:37.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:41:37.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:41:37.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:41:37.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:41:37.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:41:37.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:41:37.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:37.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:41:37.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:41:37.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:41:37.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:41:37.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:41:37.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:41:37.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:41:37.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:41:37.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:41:37.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:41:37.219 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.219 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.219 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.219 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:37.219 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:37.219 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 120.22281139099999, 2024-04-22T18:41:37.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:41:37.220 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.220 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.220 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:37.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:37.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:37.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:37.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:41:37.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:41:37.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:41:37.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:41:37.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:41:37.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:41:37.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:41:37.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:41:37.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.224 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:37.225 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:37.225 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 120.16392873700001, 2024-04-22T18:41:37.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:41:37.225 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.225 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.225 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:37.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:37.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:41:37.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:37.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:41:37.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:41:37.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:41:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:41:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:41:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:41:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:41:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:41:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:41:37.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:41:37.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:37.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:41:37.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:41:37.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:41:37.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:41:37.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:41:37.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:41:37.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:41:37.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:41:37.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:41:37.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:41:37.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.230 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.230 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:37.230 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:37.230 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 120.16379012100001, 2024-04-22T18:41:37.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:41:37.230 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.230 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.230 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:41:37.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:41:37.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:41:37.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:41:37.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:41:37.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:41:37.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:41:37.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:41:37.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:41:37.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:41:37.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:37.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:41:37.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:41:37.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:41:37.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:41:37.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:41:37.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:41:37.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:41:37.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:41:37.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:41:37.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:41:37.235 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.235 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.235 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.235 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:37.235 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:37.235 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 120.163771118, 2024-04-22T18:41:37.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:41:37.235 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.235 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.236 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:37.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:37.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:37.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:37.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:41:37.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:41:37.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:41:37.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:41:37.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:41:37.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:41:37.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:41:37.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:41:37.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:41:37.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:41:37.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:37.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:41:37.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:41:37.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:41:37.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:41:37.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:41:37.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:41:37.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:37.240 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 116.542758585, 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:37.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:41:37.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:37.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:41:37.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:41:37.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:41:37.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:41:37.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:41:37.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:41:37.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:41:37.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:41:37.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:41:37.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:41:37.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:37.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:41:37.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:41:37.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:41:37.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:41:37.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:41:37.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:41:37.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:41:37.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:41:37.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:41:37.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:41:37.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.246 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.246 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:37.246 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:37.246 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 116.542238994, 2024-04-22T18:41:37.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:41:37.246 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.246 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.246 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.246 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.247 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.247 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:41:37.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.247 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:37.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:37.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:41:37.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:37.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:41:37.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:41:37.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:41:37.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:41:37.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:41:37.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:41:37.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:41:37.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:41:37.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:41:37.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:41:37.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:37.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:41:37.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:41:37.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:41:37.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:41:37.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:41:37.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:41:37.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:41:37.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:41:37.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:41:37.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:41:37.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.251 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 116.44208611099999, 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:41:37.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:41:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:41:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:41:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:41:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:41:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:41:37.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:41:37.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:37.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:41:37.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:41:37.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:41:37.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:41:37.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:41:37.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:41:37.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:41:37.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:41:37.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.256 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.256 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.256 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:37.257 INFO:tasks.rados.rados.0.smithi089.stdout:1967: set_chunk oid 24 offset: 601600 length: 292352 target oid bfd4b6f547e9c400c755c93863a377f3f86b25bda07153d93a1b68a5002541f9 offset: 0 snap -1 2024-04-22T18:41:37.258 INFO:tasks.rados.rados.0.smithi089.stdout:1968: set_chunk oid 15 target oid 2024-04-22T18:41:37.258 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:37.259+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:37.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:37.358 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:37.359+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:37.359 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:37.359 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:37.360 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:37.360 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:37.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:37.458 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:37.459+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:37.511 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:37.511 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:37.511 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:37.511 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:37.511 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:37.520 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:37.554 INFO:tasks.rados.rados.0.smithi089.stdout:1968: set_chunk oid 15 offset: 1316352 length: 868864 target oid 53b744013e267047481833251bddb3d56ce1afd2356d386266da7c3ac1dd9f77 offset: 0 snap -1 2024-04-22T18:41:37.554 INFO:tasks.rados.rados.0.smithi089.stdout:1969: set_chunk oid 22 target oid 2024-04-22T18:41:37.559 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:37.560+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:37.658 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:41:37.659 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:37.660+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:37.667 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:37.667 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:37.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:37.667 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:37.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.667 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:37.667 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:37.667 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 163.406190393, 2024-04-22T18:41:37.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:41:37.667 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.668 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.668 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.668 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.668 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.668 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:41:37.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.668 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:37.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:37.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:41:37.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:37.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:41:37.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:41:37.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:41:37.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:41:37.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:41:37.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:41:37.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:41:37.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:41:37.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:41:37.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:41:37.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:37.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:41:37.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:41:37.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:41:37.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:41:37.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:41:37.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:41:37.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.672 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.672 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:41:37.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:41:37.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:41:37.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:41:37.673 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.673 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.673 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.673 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:37.673 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:37.673 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 163.401407963, 2024-04-22T18:41:37.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:41:37.674 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.674 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.674 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.674 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.674 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.674 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:41:37.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.674 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:37.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:37.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:41:37.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:37.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:41:37.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:41:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:41:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:41:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:41:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:41:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:41:37.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:41:37.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:41:37.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:41:37.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:37.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:41:37.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:41:37.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:41:37.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:41:37.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:41:37.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:41:37.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:41:37.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:41:37.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.678 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.678 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 163.40131375999999, 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.679 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:41:37.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.680 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:37.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:37.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:41:37.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:37.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:41:37.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:41:37.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:41:37.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:41:37.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:41:37.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:41:37.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:41:37.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:41:37.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:41:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:41:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:41:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:41:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:41:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:41:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:41:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:41:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:41:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:41:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:41:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:41:37.685 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.685 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.685 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.685 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:37.685 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:37.685 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 163.401292872, 2024-04-22T18:41:37.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:41:37.685 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.685 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:37.685 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.685 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.685 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.685 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:41:37.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.686 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:37.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:37.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:41:37.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:37.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:41:37.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:41:37.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:41:37.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:41:37.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:41:37.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:41:37.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:41:37.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:41:37.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:37.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:41:37.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:41:37.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:41:37.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:41:37.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.689 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:41:37.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:41:37.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.689 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:41:37.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:41:37.690 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.690 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.690 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.690 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:37.690 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:37.690 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 162.61828520899999, 2024-04-22T18:41:37.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:41:37.690 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.690 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.690 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.691 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.691 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.691 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:41:37.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.691 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:37.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:37.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:41:37.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:37.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:41:37.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:41:37.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:41:37.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:41:37.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:41:37.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:41:37.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:41:37.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:41:37.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:41:37.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:41:37.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:37.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:41:37.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:41:37.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:41:37.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:41:37.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:41:37.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:41:37.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:41:37.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:41:37.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:41:37.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:41:37.695 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 162.61045380300001, 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:37.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:37.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:41:37.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:37.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:41:37.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:41:37.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:41:37.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:41:37.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:41:37.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:41:37.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:41:37.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:41:37.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:41:37.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:41:37.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:37.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:41:37.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:41:37.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:41:37.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:41:37.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:41:37.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:41:37.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:41:37.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:41:37.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:41:37.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:41:37.701 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.701 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.701 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.701 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:37.701 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:37.702 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 102.458581173, 2024-04-22T18:41:37.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:41:37.702 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.702 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.702 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.702 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.702 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.702 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:41:37.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.702 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:37.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:37.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:41:37.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:37.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:41:37.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:41:37.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:41:37.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:41:37.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:41:37.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:41:37.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:41:37.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:41:37.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:41:37.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:41:37.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:37.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:41:37.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:41:37.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:41:37.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:41:37.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.706 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:41:37.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:41:37.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:41:37.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:41:37.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:41:37.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:41:37.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.707 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.707 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.708 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:37.708 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:37.708 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 102.45817626500001, 2024-04-22T18:41:37.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:41:37.708 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.708 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:37.708 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.708 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.708 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.708 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:41:37.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.708 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:37.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:37.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:37.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:37.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:41:37.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:41:37.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:41:37.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:41:37.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:41:37.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:41:37.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:41:37.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:41:37.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:37.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:41:37.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:41:37.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.712 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3213 4.a 4:5d2aee44:::smithi08937929-13:head [assert-version v66,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 75.097276377, 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69928701400000004, 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3213 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:37.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:37.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568559+0000", 2024-04-22T18:41:37.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:37.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568567+0000", 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568579+0000", 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.668711+0000", 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100131866 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.768945+0000", 2024-04-22T18:41:37.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100234252 2024-04-22T18:41:37.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769002+0000", 2024-04-22T18:41:37.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7092999999999997e-05 2024-04-22T18:41:37.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:37.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.166404+0000", 2024-04-22T18:41:37.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39740241999999998 2024-04-22T18:41:37.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.191686+0000", 2024-04-22T18:41:37.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025281739000000001 2024-04-22T18:41:37.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267758+0000", 2024-04-22T18:41:37.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076071606999999999 2024-04-22T18:41:37.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267812+0000", 2024-04-22T18:41:37.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4478999999999998e-05 2024-04-22T18:41:37.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267848+0000", 2024-04-22T18:41:37.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6062999999999997e-05 2024-04-22T18:41:37.718 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.718 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.718 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.718 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3319 4.a 4:5d2aee44:::smithi08937929-13:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:41:37.718 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:37.719 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 71.205528626000003, 2024-04-22T18:41:37.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476623401, 2024-04-22T18:41:37.719 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.719 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:37.719 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.719 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.719 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.719 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3319 2024-04-22T18:41:37.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.719 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:37.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:37.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460306+0000", 2024-04-22T18:41:37.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:37.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460318+0000", 2024-04-22T18:41:37.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2896e-05 2024-04-22T18:41:37.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460345+0000", 2024-04-22T18:41:37.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6650000000000001e-05 2024-04-22T18:41:37.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460360+0000", 2024-04-22T18:41:37.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4729000000000001e-05 2024-04-22T18:41:37.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460502+0000", 2024-04-22T18:41:37.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141729 2024-04-22T18:41:37.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460644+0000", 2024-04-22T18:41:37.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142755 2024-04-22T18:41:37.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.936932+0000", 2024-04-22T18:41:37.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476288078 2024-04-22T18:41:37.722 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.723 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.723 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.723 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3412 4.a 4:519c08f7:::smithi08937929-15:head [write 2529878~394573] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:37.723 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:37.723 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 66.523694065000001, 2024-04-22T18:41:37.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48918094899999998, 2024-04-22T18:41:37.723 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.723 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.723 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.723 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.723 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.724 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3412 2024-04-22T18:41:37.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.724 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:37.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:37.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142141+0000", 2024-04-22T18:41:37.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:37.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142496+0000", 2024-04-22T18:41:37.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035444699999999998 2024-04-22T18:41:37.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142510+0000", 2024-04-22T18:41:37.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3841e-05 2024-04-22T18:41:37.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142518+0000", 2024-04-22T18:41:37.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8089999999999993e-06 2024-04-22T18:41:37.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142560+0000", 2024-04-22T18:41:37.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2097e-05 2024-04-22T18:41:37.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142637+0000", 2024-04-22T18:41:37.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6965000000000006e-05 2024-04-22T18:41:37.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:37.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142715+0000", 2024-04-22T18:41:37.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8694000000000002e-05 2024-04-22T18:41:37.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144179+0000", 2024-04-22T18:41:37.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146342 2024-04-22T18:41:37.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631229+0000", 2024-04-22T18:41:37.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48705018500000002 2024-04-22T18:41:37.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631281+0000", 2024-04-22T18:41:37.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2216999999999998e-05 2024-04-22T18:41:37.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631324+0000", 2024-04-22T18:41:37.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3344999999999999e-05 2024-04-22T18:41:37.729 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.729 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.729 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.729 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3413 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2924451] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:37.729 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:37.729 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 66.523312924999999, 2024-04-22T18:41:37.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58904086899999997, 2024-04-22T18:41:37.729 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.729 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.729 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.729 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.729 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.730 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3413 2024-04-22T18:41:37.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.730 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:37.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:37.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142523+0000", 2024-04-22T18:41:37.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:37.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142529+0000", 2024-04-22T18:41:37.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6049999999999997e-06 2024-04-22T18:41:37.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142535+0000", 2024-04-22T18:41:37.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4740000000000002e-06 2024-04-22T18:41:37.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142541+0000", 2024-04-22T18:41:37.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1839999999999998e-06 2024-04-22T18:41:37.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143553+0000", 2024-04-22T18:41:37.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010126689999999999 2024-04-22T18:41:37.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143659+0000", 2024-04-22T18:41:37.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105393 2024-04-22T18:41:37.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:37.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143825+0000", 2024-04-22T18:41:37.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165809 2024-04-22T18:41:37.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144329+0000", 2024-04-22T18:41:37.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050483799999999997 2024-04-22T18:41:37.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731480+0000", 2024-04-22T18:41:37.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58715100600000003 2024-04-22T18:41:37.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731522+0000", 2024-04-22T18:41:37.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2098000000000002e-05 2024-04-22T18:41:37.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731566+0000", 2024-04-22T18:41:37.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3066999999999999e-05 2024-04-22T18:41:37.734 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.734 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.734 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.735 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3414 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:37.735 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:37.735 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 66.523284321000006, 2024-04-22T18:41:37.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58926020199999996, 2024-04-22T18:41:37.735 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.735 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:37.735 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.735 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.735 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.735 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3414 2024-04-22T18:41:37.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.735 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:37.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:37.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142552+0000", 2024-04-22T18:41:37.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:37.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142556+0000", 2024-04-22T18:41:37.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0289999999999999e-06 2024-04-22T18:41:37.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142574+0000", 2024-04-22T18:41:37.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7266999999999999e-05 2024-04-22T18:41:37.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142581+0000", 2024-04-22T18:41:37.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-04-22T18:41:37.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144121+0000", 2024-04-22T18:41:37.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015397480000000001 2024-04-22T18:41:37.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:37.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144170+0000", 2024-04-22T18:41:37.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8803e-05 2024-04-22T18:41:37.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731624+0000", 2024-04-22T18:41:37.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58745365800000005 2024-04-22T18:41:37.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731712+0000", 2024-04-22T18:41:37.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8016999999999994e-05 2024-04-22T18:41:37.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731813+0000", 2024-04-22T18:41:37.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101817 2024-04-22T18:41:37.739 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.739 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:77 3.3 3:dcc8aeec:::3590d5741efdda41a26b62a0982727439225b7b4b51097b313f30d8dc10bf6f6:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 62.418265998999999, 2024-04-22T18:41:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011812878, 2024-04-22T18:41:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:37.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247571+0000", 2024-04-22T18:41:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247574+0000", 2024-04-22T18:41:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:41:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247577+0000", 2024-04-22T18:41:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3069999999999998e-06 2024-04-22T18:41:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247580+0000", 2024-04-22T18:41:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8870000000000001e-06 2024-04-22T18:41:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.340928+0000", 2024-04-22T18:41:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093348678000000004 2024-04-22T18:41:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341122+0000", 2024-04-22T18:41:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000193227 2024-04-22T18:41:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618335+0000", 2024-04-22T18:41:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.277213775 2024-04-22T18:41:37.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:37.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618422+0000", 2024-04-22T18:41:37.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6537000000000002e-05 2024-04-22T18:41:37.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954387+0000", 2024-04-22T18:41:37.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33596509699999999 2024-04-22T18:41:37.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954548+0000", 2024-04-22T18:41:37.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016042199999999999 2024-04-22T18:41:37.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:37.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954858+0000", 2024-04-22T18:41:37.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000310396 2024-04-22T18:41:37.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157019+0000", 2024-04-22T18:41:37.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20216071299999999 2024-04-22T18:41:37.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259213+0000", 2024-04-22T18:41:37.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102194734 2024-04-22T18:41:37.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259262+0000", 2024-04-22T18:41:37.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9052999999999999e-05 2024-04-22T18:41:37.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259384+0000", 2024-04-22T18:41:37.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122052 2024-04-22T18:41:37.746 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.747 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.747 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.747 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:78 3.3 3:df7e90d2:::ab0657e24bd96704775de2a378746bc22309c5a7366a5eaf9b4e1f1f2bef0a4c:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:37.747 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:37.747 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 62.418250802000003, 2024-04-22T18:41:37.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011974739, 2024-04-22T18:41:37.747 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.747 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.747 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:37.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:37.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247586+0000", 2024-04-22T18:41:37.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:37.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247589+0000", 2024-04-22T18:41:37.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.181e-06 2024-04-22T18:41:37.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247592+0000", 2024-04-22T18:41:37.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0230000000000001e-06 2024-04-22T18:41:37.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247595+0000", 2024-04-22T18:41:37.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.428e-06 2024-04-22T18:41:37.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341160+0000", 2024-04-22T18:41:37.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093564571999999999 2024-04-22T18:41:37.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:37.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341264+0000", 2024-04-22T18:41:37.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010391999999999999 2024-04-22T18:41:37.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619269+0000", 2024-04-22T18:41:37.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27800575100000002 2024-04-22T18:41:37.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:37.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619346+0000", 2024-04-22T18:41:37.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6459999999999996e-05 2024-04-22T18:41:37.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.155957+0000", 2024-04-22T18:41:37.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53661109299999998 2024-04-22T18:41:37.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156049+0000", 2024-04-22T18:41:37.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2272e-05 2024-04-22T18:41:37.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:37.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156236+0000", 2024-04-22T18:41:37.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000186808 2024-04-22T18:41:37.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.158088+0000", 2024-04-22T18:41:37.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018522079999999999 2024-04-22T18:41:37.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259421+0000", 2024-04-22T18:41:37.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101332718 2024-04-22T18:41:37.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259460+0000", 2024-04-22T18:41:37.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8912000000000003e-05 2024-04-22T18:41:37.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259562+0000", 2024-04-22T18:41:37.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010175600000000001 2024-04-22T18:41:37.754 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.754 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.754 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.755 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:80 3.3 3:f90f47f2:::e5603f596d699ee3e27e255225eeb90a9f0c1b4a4d099d1c586b19019220aa3f:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:37.755 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:37.755 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 62.418211724999999, 2024-04-22T18:41:37.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0121096759999999, 2024-04-22T18:41:37.755 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.755 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.755 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:37.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:37.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247625+0000", 2024-04-22T18:41:37.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:37.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247628+0000", 2024-04-22T18:41:37.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6129999999999998e-06 2024-04-22T18:41:37.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247631+0000", 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3869999999999999e-06 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247634+0000", 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9919999999999999e-06 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341550+0000", 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093915584999999996 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:37.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341700+0000", 2024-04-22T18:41:37.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015000300000000001 2024-04-22T18:41:37.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620744+0000", 2024-04-22T18:41:37.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27904399200000002 2024-04-22T18:41:37.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:37.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620794+0000", 2024-04-22T18:41:37.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9579000000000001e-05 2024-04-22T18:41:37.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157597+0000", 2024-04-22T18:41:37.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53680343799999997 2024-04-22T18:41:37.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157678+0000", 2024-04-22T18:41:37.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0829999999999997e-05 2024-04-22T18:41:37.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:37.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157832+0000", 2024-04-22T18:41:37.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154184 2024-04-22T18:41:37.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259572+0000", 2024-04-22T18:41:37.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10174003600000001 2024-04-22T18:41:37.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259601+0000", 2024-04-22T18:41:37.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8841999999999999e-05 2024-04-22T18:41:37.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259638+0000", 2024-04-22T18:41:37.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7308999999999998e-05 2024-04-22T18:41:37.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259736+0000", 2024-04-22T18:41:37.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7283999999999999e-05 2024-04-22T18:41:37.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.761 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.761 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:135 3.3 3:f89134e5:::3e4e73f5d4a0122957865f0faeac2483204ee98d7d497ef3b54afe2a02b39a3d:head [read 0~802304 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:37.761 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:37.761 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 62.050396646000003, 2024-04-22T18:41:37.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.64370653499999997, 2024-04-22T18:41:37.761 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.762 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:37.762 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:37.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:37.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615438+0000", 2024-04-22T18:41:37.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:37.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615449+0000", 2024-04-22T18:41:37.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1219999999999999e-05 2024-04-22T18:41:37.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615460+0000", 2024-04-22T18:41:37.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0403e-05 2024-04-22T18:41:37.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.715588+0000", 2024-04-22T18:41:37.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100128881 2024-04-22T18:41:37.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722533+0000", 2024-04-22T18:41:37.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069444010000000002 2024-04-22T18:41:37.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:37.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722658+0000", 2024-04-22T18:41:37.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012560699999999999 2024-04-22T18:41:37.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826437+0000", 2024-04-22T18:41:37.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103778194 2024-04-22T18:41:37.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:37.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826532+0000", 2024-04-22T18:41:37.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5111000000000005e-05 2024-04-22T18:41:37.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258787+0000", 2024-04-22T18:41:37.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43225529099999999 2024-04-22T18:41:37.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258905+0000", 2024-04-22T18:41:37.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118348 2024-04-22T18:41:37.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259148+0000", 2024-04-22T18:41:37.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000242093 2024-04-22T18:41:37.766 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.766 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.766 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.767 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3890 4.1 4:83c955bf:::smithi08937929-28:head [write 2699879~469889] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:37.767 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:37.767 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 48.051721802000003, 2024-04-22T18:41:37.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49042297000000001, 2024-04-22T18:41:37.767 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.767 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.767 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.767 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3890 2024-04-22T18:41:37.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.767 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614115+0000", 2024-04-22T18:41:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614496+0000", 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038181700000000001 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614509+0000", 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2381000000000001e-05 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614517+0000", 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0879999999999995e-06 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615610+0000", 2024-04-22T18:41:37.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010933519999999999 2024-04-22T18:41:37.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615655+0000", 2024-04-22T18:41:37.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4966000000000003e-05 2024-04-22T18:41:37.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:37.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615769+0000", 2024-04-22T18:41:37.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113986 2024-04-22T18:41:37.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704882+0000", 2024-04-22T18:41:37.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089112825000000007 2024-04-22T18:41:37.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104488+0000", 2024-04-22T18:41:37.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39960572999999999 2024-04-22T18:41:37.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104517+0000", 2024-04-22T18:41:37.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9006e-05 2024-04-22T18:41:37.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104539+0000", 2024-04-22T18:41:37.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2073e-05 2024-04-22T18:41:37.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.772 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.773 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3891 4.1 4:83c955bf:::smithi08937929-28:head [setxattr _header (56) in=7b,truncate 3630345] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:37.773 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:37.773 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 48.051313571000001, 2024-04-22T18:41:37.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59018588699999996, 2024-04-22T18:41:37.773 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.773 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:37.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.773 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.773 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3891 2024-04-22T18:41:37.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.773 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:37.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:37.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614523+0000", 2024-04-22T18:41:37.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:37.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614528+0000", 2024-04-22T18:41:37.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0379999999999999e-06 2024-04-22T18:41:37.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614540+0000", 2024-04-22T18:41:37.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018000000000001e-05 2024-04-22T18:41:37.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614546+0000", 2024-04-22T18:41:37.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4990000000000002e-06 2024-04-22T18:41:37.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704438+0000", 2024-04-22T18:41:37.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089892044000000004 2024-04-22T18:41:37.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704504+0000", 2024-04-22T18:41:37.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6365000000000006e-05 2024-04-22T18:41:37.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:37.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704602+0000", 2024-04-22T18:41:37.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8048999999999996e-05 2024-04-22T18:41:37.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:37.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.806370+0000", 2024-04-22T18:41:37.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10176768 2024-04-22T18:41:37.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:37.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204629+0000", 2024-04-22T18:41:37.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39825917999999999 2024-04-22T18:41:37.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:37.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204673+0000", 2024-04-22T18:41:37.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3516999999999997e-05 2024-04-22T18:41:37.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204710+0000", 2024-04-22T18:41:37.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7391000000000002e-05 2024-04-22T18:41:37.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.778 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.778 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3892 4.1 4:83c955bf:::smithi08937929-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:37.779 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:37.779 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 48.051286990000001, 2024-04-22T18:41:37.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59031281800000002, 2024-04-22T18:41:37.779 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:37.779 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:37.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:37.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:37.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:37.779 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3892 2024-04-22T18:41:37.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.779 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:37.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:37.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:37.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:37.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:37.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:37.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:37.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614550+0000", 2024-04-22T18:41:37.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:37.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:37.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614553+0000", 2024-04-22T18:41:37.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0879999999999999e-06 2024-04-22T18:41:37.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:37.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614559+0000", 2024-04-22T18:41:37.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9259999999999997e-06 2024-04-22T18:41:37.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:37.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614565+0000", 2024-04-22T18:41:37.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7950000000000004e-06 2024-04-22T18:41:37.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704818+0000", 2024-04-22T18:41:37.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090252624000000004 2024-04-22T18:41:37.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:37.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704866+0000", 2024-04-22T18:41:37.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8667e-05 2024-04-22T18:41:37.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:37.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204742+0000", 2024-04-22T18:41:37.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987551899999999 2024-04-22T18:41:37.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:37.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204799+0000", 2024-04-22T18:41:37.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7299000000000001e-05 2024-04-22T18:41:37.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:37.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:37.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:37.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204863+0000", 2024-04-22T18:41:37.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4455000000000005e-05 2024-04-22T18:41:37.786 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.786 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.786 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.786 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:37.786 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:37.786 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:37.786 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 7, 2, 4, 0] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:41:37.786 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:41:37.787 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-22T18:41:37.787 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.751539 2024-04-22T18:41:37.787 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 0.7515392619881344 2024-04-22T18:41:37.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:37.790 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:37.789+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:37.862 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:41:37.884 INFO:tasks.rados.rados.0.smithi089.stdout:1969: set_chunk oid 22 offset: 275456 length: 1580544 target oid 6f5eb749380d509f80033386fd68d92a7e209e803991287cc34c4c93162cded2 offset: 0 snap -1 2024-04-22T18:41:37.884 INFO:tasks.rados.rados.0.smithi089.stdout:1970: read oid 45 snap -1 2024-04-22T18:41:37.884 INFO:tasks.rados.rados.0.smithi089.stdout:1970: expect (ObjNum 415 snap 0 seq_num 415) 2024-04-22T18:41:37.885 INFO:tasks.rados.rados.0.smithi089.stdout:1971: write oid 25 current snap is 0 2024-04-22T18:41:37.885 INFO:tasks.rados.rados.0.smithi089.stdout:1971: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2024-04-22T18:41:37.889 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:37.890+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:37.891 INFO:tasks.rados.rados.0.smithi089.stdout:1971: writing smithi08937929-25 from 692554 to 1247047 tid 1 2024-04-22T18:41:37.897 INFO:tasks.rados.rados.0.smithi089.stdout:1971: writing smithi08937929-25 from 1778107 to 2457912 tid 2 2024-04-22T18:41:37.901 INFO:tasks.rados.rados.0.smithi089.stdout:1971: writing smithi08937929-25 from 2889444 to 3092554 tid 3 2024-04-22T18:41:37.901 INFO:tasks.rados.rados.0.smithi089.stdout:1972: tier_evict oid 14 2024-04-22T18:41:37.902 INFO:tasks.rados.rados.0.smithi089.stdout:1972: tier_evict oid 14 snap -1 2024-04-22T18:41:37.902 INFO:tasks.rados.rados.0.smithi089.stdout:1973: read oid 41 snap -1 2024-04-22T18:41:37.902 INFO:tasks.rados.rados.0.smithi089.stdout:1973: expect (ObjNum 358 snap 0 seq_num 358) 2024-04-22T18:41:37.902 INFO:tasks.rados.rados.0.smithi089.stdout:1974: tier_evict oid 7 2024-04-22T18:41:37.902 INFO:tasks.rados.rados.0.smithi089.stdout:1974: tier_evict oid 7 snap -1 2024-04-22T18:41:37.902 INFO:tasks.rados.rados.0.smithi089.stdout:1965: expect (ObjNum 351 snap 0 seq_num 351) 2024-04-22T18:41:37.936 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:37.936 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:37.936 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:37.936 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:37.945 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:38.029 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:38.030+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:38.045 INFO:tasks.rados.rados.0.smithi089.stdout:1966: finishing write tid 1 to smithi08937929-35 2024-04-22T18:41:38.045 INFO:tasks.rados.rados.0.smithi089.stdout:1966: finishing write tid 2 to smithi08937929-35 2024-04-22T18:41:38.045 INFO:tasks.rados.rados.0.smithi089.stdout:1964: finishing write tid 1 to smithi08937929-33 2024-04-22T18:41:38.046 INFO:tasks.rados.rados.0.smithi089.stdout:1964: finishing write tid 2 to smithi08937929-33 2024-04-22T18:41:38.046 INFO:tasks.rados.rados.0.smithi089.stdout:1964: finishing write tid 3 to smithi08937929-33 2024-04-22T18:41:38.046 INFO:tasks.rados.rados.0.smithi089.stdout:1965: done (10 left) 2024-04-22T18:41:38.046 INFO:tasks.rados.rados.0.smithi089.stdout:1975: read oid 8 snap -1 2024-04-22T18:41:38.046 INFO:tasks.rados.rados.0.smithi089.stdout:1975: expect deleted 2024-04-22T18:41:38.046 INFO:tasks.rados.rados.0.smithi089.stdout:1976: read oid 36 snap -1 2024-04-22T18:41:38.046 INFO:tasks.rados.rados.0.smithi089.stdout:1976: expect deleted 2024-04-22T18:41:38.046 INFO:tasks.rados.rados.0.smithi089.stdout:1977: tier_flush oid 20 2024-04-22T18:41:38.046 INFO:tasks.rados.rados.0.smithi089.stdout:1977: tier_flush oid 20 snap -1 2024-04-22T18:41:38.047 INFO:tasks.rados.rados.0.smithi089.stdout:1964: finishing write tid 4 to smithi08937929-33 2024-04-22T18:41:38.047 INFO:tasks.rados.rados.0.smithi089.stdout:1964: finishing write tid 5 to smithi08937929-33 2024-04-22T18:41:38.047 INFO:tasks.rados.rados.0.smithi089.stdout:1964: oid 33 updating version 0 to 70 2024-04-22T18:41:38.047 INFO:tasks.rados.rados.0.smithi089.stdout:1964: oid 33 updating version 70 to 71 2024-04-22T18:41:38.047 INFO:tasks.rados.rados.0.smithi089.stdout:1964: oid 33 version 71 is already newer than 69 2024-04-22T18:41:38.047 INFO:tasks.rados.rados.0.smithi089.stdout:1964: oid 33 version 71 is already newer than 68 2024-04-22T18:41:38.048 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 33 v 71 (ObjNum 420 snap 0 seq_num 420) dirty exists 2024-04-22T18:41:38.048 INFO:tasks.rados.rados.0.smithi089.stdout:1964: left oid 33 (ObjNum 420 snap 0 seq_num 420) 2024-04-22T18:41:38.048 INFO:tasks.rados.rados.0.smithi089.stdout:1966: finishing write tid 3 to smithi08937929-35 2024-04-22T18:41:38.048 INFO:tasks.rados.rados.0.smithi089.stdout:1966: finishing write tid 4 to smithi08937929-35 2024-04-22T18:41:38.048 INFO:tasks.rados.rados.0.smithi089.stdout:1966: oid 35 updating version 0 to 126 2024-04-22T18:41:38.048 INFO:tasks.rados.rados.0.smithi089.stdout:1966: oid 35 version 126 is already newer than 125 2024-04-22T18:41:38.048 INFO:tasks.rados.rados.0.smithi089.stdout:1966: oid 35 updating version 126 to 127 2024-04-22T18:41:38.048 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 35 v 127 (ObjNum 421 snap 0 seq_num 421) dirty exists 2024-04-22T18:41:38.048 INFO:tasks.rados.rados.0.smithi089.stdout:1966: left oid 35 (ObjNum 421 snap 0 seq_num 421) 2024-04-22T18:41:38.048 INFO:tasks.rados.rados.0.smithi089.stdout:1967: finishing set_chunk to oid 24 2024-04-22T18:41:38.048 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 601600 length: 292352 2024-04-22T18:41:38.048 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 0 (ObjNum 384 snap 0 seq_num 384) dirty exists 2024-04-22T18:41:38.049 INFO:tasks.rados.rados.0.smithi089.stdout:1968: finishing set_chunk to oid 15 2024-04-22T18:41:38.049 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 115 (ObjNum 377 snap 0 seq_num 377) dirty exists 2024-04-22T18:41:38.049 INFO:tasks.rados.rados.0.smithi089.stdout:1969: finishing set_chunk to oid 22 2024-04-22T18:41:38.049 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 22 set_chunk returned error code -95 offset: 275456 length: 1580544 2024-04-22T18:41:38.049 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 0 (ObjNum 303 snap 0 seq_num 303) dirty exists 2024-04-22T18:41:38.049 INFO:tasks.rados.rados.0.smithi089.stdout:1970: expect (ObjNum 415 snap 0 seq_num 415) 2024-04-22T18:41:38.089 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:38.089 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:38.089 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:38.089 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:38.089 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:38.091 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:38.092+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:38.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:38.102 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:38.102 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:38.119 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:38.120 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:38.146 INFO:tasks.rados.rados.0.smithi089.stdout:1974: got (22) Invalid argument 2024-04-22T18:41:38.146 INFO:tasks.rados.rados.0.smithi089.stdout:1972: got (22) Invalid argument 2024-04-22T18:41:38.146 INFO:tasks.rados.rados.0.smithi089.stdout:1971: finishing write tid 1 to smithi08937929-25 2024-04-22T18:41:38.146 INFO:tasks.rados.rados.0.smithi089.stdout:1973: expect (ObjNum 358 snap 0 seq_num 358) 2024-04-22T18:41:38.176 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:38.177 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:38.192 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:38.193+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:38.193 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:38.193 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:38.209 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:38.209 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:38.225 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:38.225 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:38.241 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:38.242 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:38.243 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:38.243 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:38.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:38.243 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:38.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.243 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:38.243 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:38.243 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 173.38907153599999, 2024-04-22T18:41:38.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:41:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:41:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:38.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:38.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:38.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:41:38.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:41:38.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:41:38.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:41:38.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:41:38.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:41:38.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:41:38.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:41:38.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:41:38.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:41:38.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:38.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:41:38.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:41:38.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:38.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:41:38.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:41:38.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:38.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:41:38.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:41:38.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:38.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:41:38.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:41:38.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:41:38.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:41:38.249 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.249 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.249 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.249 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:38.249 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:38.249 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 173.38902037400001, 2024-04-22T18:41:38.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:41:38.250 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.250 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:38.250 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.250 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:38.250 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:38.250 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:41:38.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.250 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:38.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:38.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:38.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:38.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:41:38.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:41:38.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:41:38.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:41:38.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:41:38.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:41:38.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:41:38.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:41:38.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:41:38.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:41:38.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:38.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:41:38.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:41:38.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:38.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:41:38.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:41:38.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:38.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:41:38.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:41:38.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:38.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:41:38.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:41:38.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:41:38.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:41:38.255 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.255 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.255 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.255 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:38.255 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:38.255 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 173.38900515, 2024-04-22T18:41:38.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:41:38.255 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.255 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:38.255 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.255 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:38.256 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:38.256 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:41:38.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.256 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:38.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:38.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:41:38.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:38.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:41:38.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:41:38.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.257 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.257 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:41:38.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:41:38.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:41:38.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:41:38.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:41:38.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:41:38.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:41:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:41:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:41:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:41:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:41:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:41:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:41:38.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:41:38.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:38.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:41:38.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:41:38.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:41:38.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:41:38.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.260 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.261 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:38.261 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:38.261 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 173.388945573, 2024-04-22T18:41:38.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:41:38.261 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.261 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:38.261 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.261 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:38.261 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:38.261 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:41:38.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.261 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:38.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:38.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.262 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:38.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:38.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:41:38.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:41:38.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:41:38.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:41:38.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:41:38.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:41:38.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:41:38.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:41:38.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:41:38.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:41:38.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:38.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:41:38.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:41:38.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:38.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:41:38.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:41:38.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:38.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:41:38.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:41:38.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:38.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:41:38.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:41:38.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:41:38.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:41:38.266 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.266 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.266 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.266 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:38.266 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:38.267 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 173.388863257, 2024-04-22T18:41:38.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:41:38.267 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.267 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:38.267 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.267 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:38.267 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:38.267 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:41:38.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.267 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:38.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:41:38.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:41:38.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:41:38.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:41:38.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:41:38.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:41:38.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:41:38.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:41:38.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:38.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:41:38.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:41:38.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:38.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:41:38.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:41:38.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.271 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.272 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.272 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:38.272 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:38.272 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 173.388853101, 2024-04-22T18:41:38.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:41:38.272 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.272 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:38.272 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.272 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:38.272 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:38.272 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:41:38.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.272 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:38.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:38.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:41:38.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:38.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:41:38.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:41:38.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:41:38.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:41:38.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:41:38.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:41:38.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:41:38.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:41:38.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:41:38.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:41:38.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:38.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:41:38.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:41:38.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:38.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:41:38.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:41:38.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:38.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:41:38.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:41:38.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:38.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:41:38.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:41:38.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:41:38.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:41:38.277 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.277 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.277 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.277 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3163 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:38.278 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:38.278 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.405840663000006, 2024-04-22T18:41:38.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1009801299999999, 2024-04-22T18:41:38.278 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.278 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:38.278 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.278 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:38.278 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:38.278 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3163 2024-04-22T18:41:38.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.278 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:38.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:38.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837455+0000", 2024-04-22T18:41:38.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:38.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837464+0000", 2024-04-22T18:41:38.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5319999999999995e-06 2024-04-22T18:41:38.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837476+0000", 2024-04-22T18:41:38.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1759e-05 2024-04-22T18:41:38.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837488+0000", 2024-04-22T18:41:38.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1666e-05 2024-04-22T18:41:38.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837541+0000", 2024-04-22T18:41:38.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2939999999999998e-05 2024-04-22T18:41:38.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:38.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837547+0000", 2024-04-22T18:41:38.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6409999999999996e-06 2024-04-22T18:41:38.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937423+0000", 2024-04-22T18:41:38.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.099875545 2024-04-22T18:41:38.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937567+0000", 2024-04-22T18:41:38.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014363099999999999 2024-04-22T18:41:38.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938437+0000", 2024-04-22T18:41:38.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087060399999999997 2024-04-22T18:41:38.282 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.282 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.282 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.283 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3164 4.8 4:187adac1:::smithi08937929-36:head [sparse-read 0~2643888] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:38.283 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:38.283 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.405801294, 2024-04-22T18:41:38.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1013670310000001, 2024-04-22T18:41:38.283 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.283 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:38.283 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.283 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:38.283 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:38.283 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3164 2024-04-22T18:41:38.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.283 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:38.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:38.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.284 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:38.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:38.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837499+0000", 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.994e-06 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837503+0000", 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7249999999999999e-06 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837506+0000", 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5980000000000001e-06 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937641+0000", 2024-04-22T18:41:38.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10013467700000001 2024-04-22T18:41:38.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:38.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937650+0000", 2024-04-22T18:41:38.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3489999999999998e-06 2024-04-22T18:41:38.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938468+0000", 2024-04-22T18:41:38.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0008176710000001 2024-04-22T18:41:38.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938582+0000", 2024-04-22T18:41:38.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114287 2024-04-22T18:41:38.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938864+0000", 2024-04-22T18:41:38.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028120899999999998 2024-04-22T18:41:38.287 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.287 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.287 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.287 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3165 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:38.288 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:38.288 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.405782525999996, 2024-04-22T18:41:38.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2022754499999999, 2024-04-22T18:41:38.288 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.288 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:38.288 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.288 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:38.288 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:38.288 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3165 2024-04-22T18:41:38.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.288 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:38.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:38.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:38.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:38.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837517+0000", 2024-04-22T18:41:38.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3369999999999998e-06 2024-04-22T18:41:38.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837523+0000", 2024-04-22T18:41:38.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5450000000000003e-06 2024-04-22T18:41:38.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.290 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.290 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837526+0000", 2024-04-22T18:41:38.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.354e-06 2024-04-22T18:41:38.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937682+0000", 2024-04-22T18:41:38.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100155917 2024-04-22T18:41:38.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:38.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937688+0000", 2024-04-22T18:41:38.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:41:38.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.038953+0000", 2024-04-22T18:41:38.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012648110000001 2024-04-22T18:41:38.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039034+0000", 2024-04-22T18:41:38.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0969000000000004e-05 2024-04-22T18:41:38.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039791+0000", 2024-04-22T18:41:38.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075672900000000004 2024-04-22T18:41:38.292 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.292 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.292 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.293 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3183 4.8 4:1bc4668a:::smithi08937929-21:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:38.293 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:38.293 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.207379328000002, 2024-04-22T18:41:38.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0040478260000001, 2024-04-22T18:41:38.293 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.293 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:38.293 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.293 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:38.293 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:38.293 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3183 2024-04-22T18:41:38.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.293 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:38.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:38.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.294 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.294 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035913+0000", 2024-04-22T18:41:38.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999943 2024-04-22T18:41:38.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035935+0000", 2024-04-22T18:41:38.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2232999999999999e-05 2024-04-22T18:41:38.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035969+0000", 2024-04-22T18:41:38.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3969e-05 2024-04-22T18:41:38.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035987+0000", 2024-04-22T18:41:38.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8397e-05 2024-04-22T18:41:38.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:38.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2916000000000003e-05 2024-04-22T18:41:38.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:38.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:41:38.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0103e-05 2024-04-22T18:41:38.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039841+0000", 2024-04-22T18:41:38.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0037508959999999 2024-04-22T18:41:38.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039966+0000", 2024-04-22T18:41:38.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012514500000000001 2024-04-22T18:41:38.297 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.297 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.297 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.297 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3184 4.8 4:12d9f826:::smithi08937929-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:38.297 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:38.297 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.207291369000004, 2024-04-22T18:41:38.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.00402868, 2024-04-22T18:41:38.298 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.298 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:38.298 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.298 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:38.298 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:38.298 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3184 2024-04-22T18:41:38.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.298 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:38.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:38.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036005+0000", 2024-04-22T18:41:38.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:38.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036026+0000", 2024-04-22T18:41:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1092999999999999e-05 2024-04-22T18:41:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036030+0000", 2024-04-22T18:41:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.467e-06 2024-04-22T18:41:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036033+0000", 2024-04-22T18:41:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9780000000000001e-06 2024-04-22T18:41:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036120+0000", 2024-04-22T18:41:38.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7495000000000002e-05 2024-04-22T18:41:38.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:38.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036125+0000", 2024-04-22T18:41:38.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8720000000000001e-06 2024-04-22T18:41:38.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039982+0000", 2024-04-22T18:41:38.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0038571919999999 2024-04-22T18:41:38.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040035+0000", 2024-04-22T18:41:38.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2992999999999999e-05 2024-04-22T18:41:38.302 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.302 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.302 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.302 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3185 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:38.302 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:38.302 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.207237457000005, 2024-04-22T18:41:38.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0041404039999999, 2024-04-22T18:41:38.303 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.303 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:38.303 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.303 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:38.303 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:38.303 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3185 2024-04-22T18:41:38.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.303 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:38.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:38.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:38.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:38.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036062+0000", 2024-04-22T18:41:38.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9099999999999999e-06 2024-04-22T18:41:38.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036066+0000", 2024-04-22T18:41:38.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1699999999999999e-06 2024-04-22T18:41:38.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036077+0000", 2024-04-22T18:41:38.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0637e-05 2024-04-22T18:41:38.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036147+0000", 2024-04-22T18:41:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0025999999999997e-05 2024-04-22T18:41:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036151+0000", 2024-04-22T18:41:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:41:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040068+0000", 2024-04-22T18:41:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.003916604 2024-04-22T18:41:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040201+0000", 2024-04-22T18:41:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013275400000000001 2024-04-22T18:41:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3186 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.207217901000007, 2024-04-22T18:41:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2044167510000001, 2024-04-22T18:41:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:38.308 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:38.308 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3186 2024-04-22T18:41:38.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.308 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:38.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:38.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036079+0000", 2024-04-22T18:41:38.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:38.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036081+0000", 2024-04-22T18:41:38.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6449999999999999e-06 2024-04-22T18:41:38.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036084+0000", 2024-04-22T18:41:38.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9399999999999998e-06 2024-04-22T18:41:38.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:41:38.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.879e-06 2024-04-22T18:41:38.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036168+0000", 2024-04-22T18:41:38.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7657999999999994e-05 2024-04-22T18:41:38.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:38.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036171+0000", 2024-04-22T18:41:38.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7639999999999999e-06 2024-04-22T18:41:38.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240345+0000", 2024-04-22T18:41:38.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2041734260000001 2024-04-22T18:41:38.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240497+0000", 2024-04-22T18:41:38.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015182800000000001 2024-04-22T18:41:38.311 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.311 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.311 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.312 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3724 4.18 4:1bc4668a:::smithi08937929-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:38.312 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:38.312 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 53.182893524999997, 2024-04-22T18:41:38.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50413114699999995, 2024-04-22T18:41:38.312 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.312 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:38.312 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.312 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:38.312 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:38.312 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3724 2024-04-22T18:41:38.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.312 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:38.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:38.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:38.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:38.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060405+0000", 2024-04-22T18:41:38.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.395e-06 2024-04-22T18:41:38.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060408+0000", 2024-04-22T18:41:38.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:41:38.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060411+0000", 2024-04-22T18:41:38.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6060000000000001e-06 2024-04-22T18:41:38.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165799+0000", 2024-04-22T18:41:38.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105388369 2024-04-22T18:41:38.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:38.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165843+0000", 2024-04-22T18:41:38.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3891999999999999e-05 2024-04-22T18:41:38.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564408+0000", 2024-04-22T18:41:38.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39856466000000002 2024-04-22T18:41:38.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564461+0000", 2024-04-22T18:41:38.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3529999999999997e-05 2024-04-22T18:41:38.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564535+0000", 2024-04-22T18:41:38.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4147999999999995e-05 2024-04-22T18:41:38.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.316 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.317 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4413 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:38.317 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:38.317 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.585913480999999, 2024-04-22T18:41:38.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2025380569999999, 2024-04-22T18:41:38.317 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.317 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:38.317 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.317 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:38.317 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:38.317 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4413 2024-04-22T18:41:38.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.317 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:38.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:38.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657377+0000", 2024-04-22T18:41:38.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999933 2024-04-22T18:41:38.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657393+0000", 2024-04-22T18:41:38.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5296999999999999e-05 2024-04-22T18:41:38.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657411+0000", 2024-04-22T18:41:38.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8105e-05 2024-04-22T18:41:38.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657424+0000", 2024-04-22T18:41:38.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3030000000000001e-05 2024-04-22T18:41:38.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658313+0000", 2024-04-22T18:41:38.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00088943099999999999 2024-04-22T18:41:38.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:38.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658324+0000", 2024-04-22T18:41:38.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0496000000000001e-05 2024-04-22T18:41:38.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859560+0000", 2024-04-22T18:41:38.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2012358350000001 2024-04-22T18:41:38.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859648+0000", 2024-04-22T18:41:38.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8744000000000003e-05 2024-04-22T18:41:38.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859922+0000", 2024-04-22T18:41:38.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027396499999999998 2024-04-22T18:41:38.321 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.321 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.322 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.322 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4414 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:38.322 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:38.322 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.585859157000002, 2024-04-22T18:41:38.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2030255030000001, 2024-04-22T18:41:38.322 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.322 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:38.322 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.322 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:38.322 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:38.322 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4414 2024-04-22T18:41:38.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.323 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:38.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:38.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657437+0000", 2024-04-22T18:41:38.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:38.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657443+0000", 2024-04-22T18:41:38.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5899999999999998e-06 2024-04-22T18:41:38.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657450+0000", 2024-04-22T18:41:38.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4649999999999997e-06 2024-04-22T18:41:38.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657457+0000", 2024-04-22T18:41:38.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6000000000000003e-06 2024-04-22T18:41:38.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658364+0000", 2024-04-22T18:41:38.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090735999999999996 2024-04-22T18:41:38.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:38.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658373+0000", 2024-04-22T18:41:38.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3250000000000008e-06 2024-04-22T18:41:38.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860258+0000", 2024-04-22T18:41:38.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018852769999999 2024-04-22T18:41:38.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860308+0000", 2024-04-22T18:41:38.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9938000000000002e-05 2024-04-22T18:41:38.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860464+0000", 2024-04-22T18:41:38.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015622800000000001 2024-04-22T18:41:38.326 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.326 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.327 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.327 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4415 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:38.327 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:38.327 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.585834833, 2024-04-22T18:41:38.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2038250989999999, 2024-04-22T18:41:38.327 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.327 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:38.327 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.327 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:38.327 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:38.327 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4415 2024-04-22T18:41:38.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.327 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:38.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:38.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657462+0000", 2024-04-22T18:41:38.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:38.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657467+0000", 2024-04-22T18:41:38.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5399999999999997e-06 2024-04-22T18:41:38.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657473+0000", 2024-04-22T18:41:38.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3949999999999998e-06 2024-04-22T18:41:38.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657480+0000", 2024-04-22T18:41:38.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2489999999999998e-06 2024-04-22T18:41:38.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658394+0000", 2024-04-22T18:41:38.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00091403299999999997 2024-04-22T18:41:38.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:38.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658398+0000", 2024-04-22T18:41:38.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7979999999999996e-06 2024-04-22T18:41:38.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860495+0000", 2024-04-22T18:41:38.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.202097081 2024-04-22T18:41:38.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860543+0000", 2024-04-22T18:41:38.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7907000000000001e-05 2024-04-22T18:41:38.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861288+0000", 2024-04-22T18:41:38.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074466500000000004 2024-04-22T18:41:38.331 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.331 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.331 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4413 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.583118328000001, 2024-04-22T18:41:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.201491952, 2024-04-22T18:41:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4413 2024-04-22T18:41:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:38.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:38.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660178+0000", 2024-04-22T18:41:38.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:38.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660185+0000", 2024-04-22T18:41:38.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4379999999999998e-06 2024-04-22T18:41:38.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660201+0000", 2024-04-22T18:41:38.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5252999999999999e-05 2024-04-22T18:41:38.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660210+0000", 2024-04-22T18:41:38.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1069999999999994e-06 2024-04-22T18:41:38.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660270+0000", 2024-04-22T18:41:38.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739000000000002e-05 2024-04-22T18:41:38.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:38.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660281+0000", 2024-04-22T18:41:38.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0664e-05 2024-04-22T18:41:38.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861304+0000", 2024-04-22T18:41:38.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2010225059999999 2024-04-22T18:41:38.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861342+0000", 2024-04-22T18:41:38.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8736000000000003e-05 2024-04-22T18:41:38.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861671+0000", 2024-04-22T18:41:38.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032915899999999998 2024-04-22T18:41:38.336 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.336 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.336 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.337 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4414 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:38.337 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:38.337 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.583074818, 2024-04-22T18:41:38.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2014303930000001, 2024-04-22T18:41:38.337 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.337 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:38.337 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.337 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:38.337 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:38.337 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4414 2024-04-22T18:41:38.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.337 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:38.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:38.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660222+0000", 2024-04-22T18:41:38.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:38.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660227+0000", 2024-04-22T18:41:38.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6789999999999996e-06 2024-04-22T18:41:38.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660240+0000", 2024-04-22T18:41:38.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3485e-05 2024-04-22T18:41:38.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660245+0000", 2024-04-22T18:41:38.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8269999999999997e-06 2024-04-22T18:41:38.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660307+0000", 2024-04-22T18:41:38.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2392999999999997e-05 2024-04-22T18:41:38.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:38.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660316+0000", 2024-04-22T18:41:38.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7790000000000001e-06 2024-04-22T18:41:38.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861493+0000", 2024-04-22T18:41:38.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.201177341 2024-04-22T18:41:38.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861529+0000", 2024-04-22T18:41:38.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5978e-05 2024-04-22T18:41:38.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861653+0000", 2024-04-22T18:41:38.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012391799999999999 2024-04-22T18:41:38.341 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.342 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.342 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.342 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4415 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:38.342 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:38.342 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 20.583029593999999, 2024-04-22T18:41:38.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2019634779999999, 2024-04-22T18:41:38.342 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:38.342 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:38.342 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:38.342 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:38.342 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:38.343 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4415 2024-04-22T18:41:38.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.343 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:38.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:38.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:38.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:38.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:38.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:38.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:38.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660267+0000", 2024-04-22T18:41:38.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:38.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:38.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660273+0000", 2024-04-22T18:41:38.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5910000000000003e-06 2024-04-22T18:41:38.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:38.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660280+0000", 2024-04-22T18:41:38.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4000000000000003e-06 2024-04-22T18:41:38.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:38.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660288+0000", 2024-04-22T18:41:38.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1769999999999998e-06 2024-04-22T18:41:38.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.760401+0000", 2024-04-22T18:41:38.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100112864 2024-04-22T18:41:38.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:38.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.760408+0000", 2024-04-22T18:41:38.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0299999999999996e-06 2024-04-22T18:41:38.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:38.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861682+0000", 2024-04-22T18:41:38.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012742520000001 2024-04-22T18:41:38.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:38.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861716+0000", 2024-04-22T18:41:38.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3674999999999999e-05 2024-04-22T18:41:38.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:38.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:38.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:38.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.862232+0000", 2024-04-22T18:41:38.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051554300000000003 2024-04-22T18:41:38.347 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.347 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.347 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.347 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:38.347 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:38.347 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:38.348 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:38.349 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:38.349+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:38.349 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:38.350 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:38.351 INFO:tasks.rados.rados.0.smithi089.stdout:1971: finishing write tid 2 to smithi08937929-25 2024-04-22T18:41:38.351 INFO:tasks.rados.rados.0.smithi089.stdout:1971: finishing write tid 3 to smithi08937929-25 2024-04-22T18:41:38.352 INFO:tasks.rados.rados.0.smithi089.stdout:1971: finishing write tid 4 to smithi08937929-25 2024-04-22T18:41:38.352 INFO:tasks.rados.rados.0.smithi089.stdout:1964: done (12 left) 2024-04-22T18:41:38.352 INFO:tasks.rados.rados.0.smithi089.stdout:1966: done (11 left) 2024-04-22T18:41:38.352 INFO:tasks.rados.rados.0.smithi089.stdout:1967: done (10 left) 2024-04-22T18:41:38.352 INFO:tasks.rados.rados.0.smithi089.stdout:1968: done (9 left) 2024-04-22T18:41:38.352 INFO:tasks.rados.rados.0.smithi089.stdout:1969: done (8 left) 2024-04-22T18:41:38.352 INFO:tasks.rados.rados.0.smithi089.stdout:1970: done (7 left) 2024-04-22T18:41:38.352 INFO:tasks.rados.rados.0.smithi089.stdout:1972: done (6 left) 2024-04-22T18:41:38.352 INFO:tasks.rados.rados.0.smithi089.stdout:1973: done (5 left) 2024-04-22T18:41:38.352 INFO:tasks.rados.rados.0.smithi089.stdout:1974: done (4 left) 2024-04-22T18:41:38.352 INFO:tasks.rados.rados.0.smithi089.stdout:1975: done (3 left) 2024-04-22T18:41:38.352 INFO:tasks.rados.rados.0.smithi089.stdout:1976: done (2 left) 2024-04-22T18:41:38.352 INFO:tasks.rados.rados.0.smithi089.stdout:1978: set_chunk oid 12 target oid 2024-04-22T18:41:38.352 INFO:tasks.rados.rados.0.smithi089.stdout:1978: set_chunk oid 12 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:41:38.352 INFO:tasks.rados.rados.0.smithi089.stdout:1979: read oid 48 snap -1 2024-04-22T18:41:38.353 INFO:tasks.rados.rados.0.smithi089.stdout:1979: expect (ObjNum 351 snap 0 seq_num 351) 2024-04-22T18:41:38.353 INFO:tasks.rados.rados.0.smithi089.stdout:1980: read oid 10 snap -1 2024-04-22T18:41:38.353 INFO:tasks.rados.rados.0.smithi089.stdout:1980: expect deleted 2024-04-22T18:41:38.353 INFO:tasks.rados.rados.0.smithi089.stdout:1981: read oid 18 snap -1 2024-04-22T18:41:38.353 INFO:tasks.rados.rados.0.smithi089.stdout:1981: expect (ObjNum 307 snap 0 seq_num 307) 2024-04-22T18:41:38.353 INFO:tasks.rados.rados.0.smithi089.stdout:1982: delete oid 31 current snap is 0 2024-04-22T18:41:38.353 INFO:tasks.rados.rados.0.smithi089.stdout:1978: finishing set_chunk to oid 12 2024-04-22T18:41:38.353 INFO:tasks.rados.rados.0.smithi089.stdout:1978: got expected ENOENT (src dne) 2024-04-22T18:41:38.353 INFO:tasks.rados.rados.0.smithi089.stdout:1979: expect (ObjNum 351 snap 0 seq_num 351) 2024-04-22T18:41:38.434 INFO:tasks.rados.rados.0.smithi089.stdout:1971: finishing write tid 5 to smithi08937929-25 2024-04-22T18:41:38.434 INFO:tasks.rados.rados.0.smithi089.stdout:1971: oid 25 updating version 0 to 88 2024-04-22T18:41:38.434 INFO:tasks.rados.rados.0.smithi089.stdout:1971: oid 25 version 88 is already newer than 87 2024-04-22T18:41:38.434 INFO:tasks.rados.rados.0.smithi089.stdout:1971: oid 25 updating version 88 to 89 2024-04-22T18:41:38.434 INFO:tasks.rados.rados.0.smithi089.stdout:1971: oid 25 version 89 is already newer than 86 2024-04-22T18:41:38.434 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 89 (ObjNum 422 snap 0 seq_num 422) dirty exists 2024-04-22T18:41:38.434 INFO:tasks.rados.rados.0.smithi089.stdout:1971: left oid 25 (ObjNum 422 snap 0 seq_num 422) 2024-04-22T18:41:38.434 INFO:tasks.rados.rados.0.smithi089.stdout:1981: expect (ObjNum 307 snap 0 seq_num 307) 2024-04-22T18:41:38.449 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:38.450+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:38.493 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:38.493 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:38.502 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:38.587 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:38.588+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:38.636 INFO:tasks.rados.rados.0.smithi089.stdout:1977: got (0) Success 2024-04-22T18:41:38.637 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 0 (ObjNum 418 snap 0 seq_num 418) dirty exists 2024-04-22T18:41:38.637 INFO:tasks.rados.rados.0.smithi089.stdout:1971: done (6 left) 2024-04-22T18:41:38.637 INFO:tasks.rados.rados.0.smithi089.stdout:1977: done (5 left) 2024-04-22T18:41:38.637 INFO:tasks.rados.rados.0.smithi089.stdout:1978: done (4 left) 2024-04-22T18:41:38.637 INFO:tasks.rados.rados.0.smithi089.stdout:1979: done (3 left) 2024-04-22T18:41:38.637 INFO:tasks.rados.rados.0.smithi089.stdout:1980: done (2 left) 2024-04-22T18:41:38.637 INFO:tasks.rados.rados.0.smithi089.stdout:1981: done (1 left) 2024-04-22T18:41:38.637 INFO:tasks.rados.rados.0.smithi089.stdout:1982: done (0 left) 2024-04-22T18:41:38.637 INFO:tasks.rados.rados.0.smithi089.stdout:1983: set_chunk oid 45 target oid 2024-04-22T18:41:38.650 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:38.650 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:38.650 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:38.652+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:38.659 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:38.686 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.2 primary-affinity to 0.751539 (8492522) 2024-04-22T18:41:38.751 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:38.752+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:38.790 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:38.790 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:38.799 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:38.851 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:38.853+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:38.857 INFO:tasks.rados.rados.0.smithi089.stdout:1983: set_chunk oid 45 offset: 340480 length: 2533376 target oid e98e116efa3cd19b11bfb13328287a72c008cc67ba8684c1d3da24a649a23de7 offset: 0 snap -1 2024-04-22T18:41:38.857 INFO:tasks.rados.rados.0.smithi089.stdout:1984: write oid 14 current snap is 0 2024-04-22T18:41:38.857 INFO:tasks.rados.rados.0.smithi089.stdout:1984: seq_num 424 ranges {726168=704991,1895071=683474} 2024-04-22T18:41:38.865 INFO:tasks.rados.rados.0.smithi089.stdout:1984: writing smithi08937929-14 from 726168 to 1431159 tid 1 2024-04-22T18:41:38.872 INFO:tasks.rados.rados.0.smithi089.stdout:1984: writing smithi08937929-14 from 1895071 to 2578545 tid 2 2024-04-22T18:41:38.872 INFO:tasks.rados.rados.0.smithi089.stdout:1985: read oid 34 snap -1 2024-04-22T18:41:38.872 INFO:tasks.rados.rados.0.smithi089.stdout:1985: expect (ObjNum 417 snap 0 seq_num 417) 2024-04-22T18:41:38.872 INFO:tasks.rados.rados.0.smithi089.stdout:1986: write oid 30 current snap is 0 2024-04-22T18:41:38.872 INFO:tasks.rados.rados.0.smithi089.stdout:1986: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2024-04-22T18:41:38.880 INFO:tasks.rados.rados.0.smithi089.stdout:1986: writing smithi08937929-30 from 742975 to 1523215 tid 1 2024-04-22T18:41:38.885 INFO:tasks.rados.rados.0.smithi089.stdout:1986: writing smithi08937929-30 from 1953553 to 2380685 tid 2 2024-04-22T18:41:38.889 INFO:tasks.rados.rados.0.smithi089.stdout:1986: writing smithi08937929-30 from 2971713 to 3142975 tid 3 2024-04-22T18:41:38.889 INFO:tasks.rados.rados.0.smithi089.stdout:1987: read oid 38 snap -1 2024-04-22T18:41:38.890 INFO:tasks.rados.rados.0.smithi089.stdout:1987: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:41:38.890 INFO:tasks.rados.rados.0.smithi089.stdout:1988: tier_flush oid 2 2024-04-22T18:41:38.890 INFO:tasks.rados.rados.0.smithi089.stdout:1988: tier_flush oid 2 snap -1 2024-04-22T18:41:38.890 INFO:tasks.rados.rados.0.smithi089.stdout:1985: expect (ObjNum 417 snap 0 seq_num 417) 2024-04-22T18:41:38.906 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:38.906 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:38.916 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:38.952 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:38.953+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:39.053 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:39.054+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:39.061 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:39.061 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:39.071 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:39.081 INFO:tasks.rados.rados.0.smithi089.stdout:1985: done (5 left) 2024-04-22T18:41:39.081 INFO:tasks.rados.rados.0.smithi089.stdout:1989: set_chunk oid 7 target oid 2024-04-22T18:41:39.154 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:39.156+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:39.172 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:39.172 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:39.181 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:39.186 INFO:tasks.rados.rados.0.smithi089.stdout:1989: set_chunk oid 7 offset: 1571328 length: 462848 target oid ed86750d2bcc2f36ad39f1383a5bd6eea6681ed736fd2decb5665680de08c071 offset: 0 snap -1 2024-04-22T18:41:39.186 INFO:tasks.rados.rados.0.smithi089.stdout:1990: set_chunk oid 24 target oid 2024-04-22T18:41:39.255 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:39.256+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:39.288 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:39.288 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:39.298 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:39.356 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:39.357+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:39.395 INFO:tasks.rados.rados.0.smithi089.stdout:1990: set_chunk oid 24 offset: 137728 length: 143872 target oid 78871d581c84bdd86abb0de5ac75deb85d6a248d4361225f56cd9fec575b4a01 offset: 0 snap -1 2024-04-22T18:41:39.396 INFO:tasks.rados.rados.0.smithi089.stdout:1991: write oid 6 current snap is 0 2024-04-22T18:41:39.396 INFO:tasks.rados.rados.0.smithi089.stdout:1987: expect (ObjNum 230 snap 0 seq_num 230) 2024-04-22T18:41:39.400 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:39.400 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:39.409 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:39.457 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:39.458+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:39.511 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:39.511 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:39.520 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:39.526 INFO:tasks.rados.rados.0.smithi089.stdout:1984: finishing write tid 1 to smithi08937929-14 2024-04-22T18:41:39.526 INFO:tasks.rados.rados.0.smithi089.stdout:1984: finishing write tid 2 to smithi08937929-14 2024-04-22T18:41:39.526 INFO:tasks.rados.rados.0.smithi089.stdout:1991: seq_num 426 ranges {759782=571842,2012035=687143} 2024-04-22T18:41:39.533 INFO:tasks.rados.rados.0.smithi089.stdout:1991: writing smithi08937929-6 from 759782 to 1331624 tid 1 2024-04-22T18:41:39.542 INFO:tasks.rados.rados.0.smithi089.stdout:1991: writing smithi08937929-6 from 2012035 to 2699178 tid 2 2024-04-22T18:41:39.542 INFO:tasks.rados.rados.0.smithi089.stdout:1987: done (7 left) 2024-04-22T18:41:39.542 INFO:tasks.rados.rados.0.smithi089.stdout:1992: read oid 22 snap -1 2024-04-22T18:41:39.542 INFO:tasks.rados.rados.0.smithi089.stdout:1992: expect (ObjNum 303 snap 0 seq_num 303) 2024-04-22T18:41:39.542 INFO:tasks.rados.rados.0.smithi089.stdout:1993: delete oid 9 current snap is 0 2024-04-22T18:41:39.542 INFO:tasks.rados.rados.0.smithi089.stdout:1986: finishing write tid 1 to smithi08937929-30 2024-04-22T18:41:39.542 INFO:tasks.rados.rados.0.smithi089.stdout:1986: finishing write tid 2 to smithi08937929-30 2024-04-22T18:41:39.542 INFO:tasks.rados.rados.0.smithi089.stdout:1984: finishing write tid 3 to smithi08937929-14 2024-04-22T18:41:39.543 INFO:tasks.rados.rados.0.smithi089.stdout:1984: finishing write tid 4 to smithi08937929-14 2024-04-22T18:41:39.543 INFO:tasks.rados.rados.0.smithi089.stdout:1984: oid 14 updating version 0 to 92 2024-04-22T18:41:39.543 INFO:tasks.rados.rados.0.smithi089.stdout:1984: oid 14 version 92 is already newer than 90 2024-04-22T18:41:39.543 INFO:tasks.rados.rados.0.smithi089.stdout:1984: oid 14 version 92 is already newer than 91 2024-04-22T18:41:39.543 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 14 v 92 (ObjNum 424 snap 0 seq_num 424) dirty exists 2024-04-22T18:41:39.543 INFO:tasks.rados.rados.0.smithi089.stdout:1984: left oid 14 (ObjNum 424 snap 0 seq_num 424) 2024-04-22T18:41:39.543 INFO:tasks.rados.rados.0.smithi089.stdout:1986: finishing write tid 3 to smithi08937929-30 2024-04-22T18:41:39.543 INFO:tasks.rados.rados.0.smithi089.stdout:1986: finishing write tid 4 to smithi08937929-30 2024-04-22T18:41:39.543 INFO:tasks.rados.rados.0.smithi089.stdout:1986: finishing write tid 5 to smithi08937929-30 2024-04-22T18:41:39.543 INFO:tasks.rados.rados.0.smithi089.stdout:1986: oid 30 updating version 0 to 82 2024-04-22T18:41:39.543 INFO:tasks.rados.rados.0.smithi089.stdout:1986: oid 30 version 82 is already newer than 81 2024-04-22T18:41:39.543 INFO:tasks.rados.rados.0.smithi089.stdout:1986: oid 30 version 82 is already newer than 79 2024-04-22T18:41:39.543 INFO:tasks.rados.rados.0.smithi089.stdout:1986: oid 30 version 82 is already newer than 80 2024-04-22T18:41:39.543 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 82 (ObjNum 425 snap 0 seq_num 425) dirty exists 2024-04-22T18:41:39.543 INFO:tasks.rados.rados.0.smithi089.stdout:1986: left oid 30 (ObjNum 425 snap 0 seq_num 425) 2024-04-22T18:41:39.544 INFO:tasks.rados.rados.0.smithi089.stdout:1983: finishing set_chunk to oid 45 2024-04-22T18:41:39.544 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 106 (ObjNum 415 snap 0 seq_num 415) dirty exists 2024-04-22T18:41:39.544 INFO:tasks.rados.rados.0.smithi089.stdout:1988: got (0) Success 2024-04-22T18:41:39.544 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 385 snap 0 seq_num 385) dirty exists 2024-04-22T18:41:39.544 INFO:tasks.rados.rados.0.smithi089.stdout:1990: finishing set_chunk to oid 24 2024-04-22T18:41:39.544 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 137728 length: 143872 2024-04-22T18:41:39.544 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 0 (ObjNum 384 snap 0 seq_num 384) dirty exists 2024-04-22T18:41:39.544 INFO:tasks.rados.rados.0.smithi089.stdout:1989: finishing set_chunk to oid 7 2024-04-22T18:41:39.544 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 77 (ObjNum 412 snap 0 seq_num 412) dirty exists 2024-04-22T18:41:39.544 INFO:tasks.rados.rados.0.smithi089.stdout:1991: finishing write tid 1 to smithi08937929-6 2024-04-22T18:41:39.558 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:39.559+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:39.622 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:39.622 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:39.631 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:39.643 INFO:tasks.rados.rados.0.smithi089.stdout:1983: done (8 left) 2024-04-22T18:41:39.644 INFO:tasks.rados.rados.0.smithi089.stdout:1984: done (7 left) 2024-04-22T18:41:39.644 INFO:tasks.rados.rados.0.smithi089.stdout:1986: done (6 left) 2024-04-22T18:41:39.644 INFO:tasks.rados.rados.0.smithi089.stdout:1988: done (5 left) 2024-04-22T18:41:39.644 INFO:tasks.rados.rados.0.smithi089.stdout:1989: done (4 left) 2024-04-22T18:41:39.644 INFO:tasks.rados.rados.0.smithi089.stdout:1990: done (3 left) 2024-04-22T18:41:39.644 INFO:tasks.rados.rados.0.smithi089.stdout:1993: done (2 left) 2024-04-22T18:41:39.644 INFO:tasks.rados.rados.0.smithi089.stdout:1994: write oid 29 current snap is 0 2024-04-22T18:41:39.644 INFO:tasks.rados.rados.0.smithi089.stdout:1994: seq_num 428 ranges {793396=722340,2128999=690812} 2024-04-22T18:41:39.651 INFO:tasks.rados.rados.0.smithi089.stdout:1994: writing smithi08937929-29 from 793396 to 1515736 tid 1 2024-04-22T18:41:39.659 INFO:tasks.rados.rados.0.smithi089.stdout:1994: writing smithi08937929-29 from 2128999 to 2819811 tid 2 2024-04-22T18:41:39.659 INFO:tasks.rados.rados.0.smithi089.stdout:1995: read oid 7 snap -1 2024-04-22T18:41:39.659 INFO:tasks.rados.rados.0.smithi089.stdout:1995: expect (ObjNum 412 snap 0 seq_num 412) 2024-04-22T18:41:39.659 INFO:tasks.rados.rados.0.smithi089.stdout:1996: read oid 43 snap -1 2024-04-22T18:41:39.659 INFO:tasks.rados.rados.0.smithi089.stdout:1996: expect (ObjNum 370 snap 0 seq_num 370) 2024-04-22T18:41:39.659 INFO:tasks.rados.rados.0.smithi089.stdout:1997: read oid 17 snap -1 2024-04-22T18:41:39.660 INFO:tasks.rados.rados.0.smithi089.stdout:1997: expect (ObjNum 366 snap 0 seq_num 366) 2024-04-22T18:41:39.660 INFO:tasks.rados.rados.0.smithi089.stdout:1998: set_chunk oid 30 target oid 2024-04-22T18:41:39.660 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:39.660+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:39.698 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:39.700+0000 7f8607f36640 -1 osd.3 163 handle_osd_map injecting map crc failure 2024-04-22T18:41:39.749 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:39.749 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:39.758 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:39.760 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:39.761+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:39.789 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:41:39.861 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:39.862+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:39.905 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:39.905 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:39.915 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:39.950 INFO:tasks.rados.rados.0.smithi089.stdout:1998: set_chunk oid 30 offset: 1164288 length: 620544 target oid 856960948948aaf36fe0e4440f0ffd016b18d829c69514818ecd277aa9e9b674 offset: 0 snap -1 2024-04-22T18:41:39.950 INFO:tasks.rados.rados.0.smithi089.stdout:1999: set_chunk oid 1 target oid 2024-04-22T18:41:39.962 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:39.963+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:40.024 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:40.024 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:40.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:40.063 INFO:tasks.rados.rados.0.smithi089.stdout:1999: set_chunk oid 1 offset: 1527808 length: 1569280 target oid 54f887cb65b6c619e395e706ff029241f967527acb86c1f9cbf55b41b29f291c offset: 0 snap -1 2024-04-22T18:41:40.063 INFO:tasks.rados.rados.0.smithi089.stdout:2000: write (excl) oid 26 current snap is 0 2024-04-22T18:41:40.063 INFO:tasks.rados.rados.0.smithi089.stdout:2000: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2024-04-22T18:41:40.069 INFO:tasks.rados.rados.0.smithi089.stdout:2000: writing smithi08937929-26 from 410203 to 1207792 tid 1 2024-04-22T18:41:40.074 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:40.075+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:40.074 INFO:tasks.rados.rados.0.smithi089.stdout:2000: writing smithi08937929-26 from 1787481 to 2221951 tid 2 2024-04-22T18:41:40.080 INFO:tasks.rados.rados.0.smithi089.stdout:2000: writing smithi08937929-26 from 2681405 to 3210203 tid 3 2024-04-22T18:41:40.080 INFO:tasks.rados.rados.0.smithi089.stdout:2001: tier_evict oid 33 2024-04-22T18:41:40.080 INFO:tasks.rados.rados.0.smithi089.stdout:2001: tier_evict oid 33 snap -1 2024-04-22T18:41:40.080 INFO:tasks.rados.rados.0.smithi089.stdout:1991: finishing write tid 2 to smithi08937929-6 2024-04-22T18:41:40.080 INFO:tasks.rados.rados.0.smithi089.stdout:1991: finishing write tid 3 to smithi08937929-6 2024-04-22T18:41:40.080 INFO:tasks.rados.rados.0.smithi089.stdout:2002: set_chunk oid 24 target oid 2024-04-22T18:41:40.163 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:40.164+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:40.176 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:40.176 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:40.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:40.264 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:40.266+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:40.325 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:40.326 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:40.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:40.351 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:41:40.395 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:40.396+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:40.435 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:40.435 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:40.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:40.468 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:40.469+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:40.468 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:40.468 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:40.485 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:40.485 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:40.500 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:40.501 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:40.516 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:40.517 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:40.533 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:40.533 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:40.546 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:40.546 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:40.549 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:40.549 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:40.555 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:40.565 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:40.566 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:40.569 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:40.570+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:40.583 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:40.583 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:40.657 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:41:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 166.56335469199999, 2024-04-22T18:41:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:41:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:41:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:41:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:41:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:41:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:41:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:41:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:41:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:41:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:41:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:41:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:41:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:41:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:41:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:41:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:41:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:41:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:41:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:41:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:41:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:41:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:41:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:41:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:41:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 162.66385681700001, 2024-04-22T18:41:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:41:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:41:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:41:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:41:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:41:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:41:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:41:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:41:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:41:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:41:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:41:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:41:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:41:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:41:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:41:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:41:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:41:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:41:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:41:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:41:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:41:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:41:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:41:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:41:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:41:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:41:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:41:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 161.555557026, 2024-04-22T18:41:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:41:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:41:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:41:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:41:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:41:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:41:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:41:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:41:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:41:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:41:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:41:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:41:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:41:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:41:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:41:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:41:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:41:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:41:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:41:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:41:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:41:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:41:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:41:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.22401339800001, 2024-04-22T18:41:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:41:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:41:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:41:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:41:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:41:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:41:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:41:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:41:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:41:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:41:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:41:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:41:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:41:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:41:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:41:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:41:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:41:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.13292759000001, 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:41:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:41:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:41:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:41:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:41:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:41:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:41:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:41:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:41:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:41:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:41:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:41:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.12712733000001, 2024-04-22T18:41:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:41:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:41:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:41:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:41:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:41:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:41:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:41:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:41:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:41:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:41:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:41:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:41:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:41:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:41:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:41:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:41:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.122386987, 2024-04-22T18:41:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:41:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:41:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:41:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:41:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:41:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:41:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:41:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:41:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:41:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:41:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:41:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:41:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:41:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:41:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:41:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:41:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:41:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:41:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:41:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:41:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:41:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:41:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.121914089, 2024-04-22T18:41:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:41:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:41:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:41:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:41:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:41:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:41:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:41:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:41:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:41:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:41:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:41:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:41:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:41:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:41:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:41:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:41:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:41:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:41:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:41:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:41:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:41:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:41:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:41:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.12189416800001, 2024-04-22T18:41:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:41:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:41:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:41:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:41:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:41:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:41:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:41:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:41:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:41:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:41:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:41:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:41:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:41:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:41:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:41:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:41:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:41:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:41:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 142.812570403, 2024-04-22T18:41:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:41:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:41:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:41:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:41:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:41:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:41:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:41:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:41:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:41:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:41:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:41:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:41:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:41:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:41:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:41:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:41:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 142.81236286699999, 2024-04-22T18:41:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:41:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:41:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:41:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:41:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:41:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:41:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:41:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:41:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:41:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:41:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:41:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:41:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:41:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:41:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:41:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 123.94671313400001, 2024-04-22T18:41:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:41:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:41:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:41:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:41:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:41:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:41:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:41:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:41:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:41:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:41:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:41:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:41:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:41:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:41:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:41:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:41:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:41:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:41:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:41:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:41:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:41:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:41:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 123.732874749, 2024-04-22T18:41:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:41:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:41:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:41:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:41:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:41:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:41:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:41:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:41:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:41:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:41:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:41:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:41:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:41:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:41:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:41:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:41:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:41:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:41:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:41:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:41:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:41:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:41:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 123.732841339, 2024-04-22T18:41:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:41:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:41:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:41:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:41:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:41:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:41:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:41:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:41:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:41:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:41:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:41:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 123.673958685, 2024-04-22T18:41:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:41:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:41:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:41:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:41:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:41:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:41:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:41:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:41:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:41:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:41:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:41:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:41:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:41:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:41:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:41:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:41:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:41:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:41:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:41:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:41:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:41:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:41:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 123.673820069, 2024-04-22T18:41:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:41:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:41:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:41:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:41:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:41:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:41:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:41:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:41:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:41:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:41:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:41:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:41:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:41:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:41:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:41:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:41:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:41:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:41:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:41:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:41:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:41:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 123.673801066, 2024-04-22T18:41:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:41:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:41:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:41:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:41:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:41:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:41:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:41:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:41:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:41:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:41:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:41:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:41:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:41:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:41:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:41:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:41:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:41:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:41:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:41:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:41:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:41:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 120.052788533, 2024-04-22T18:41:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:41:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:41:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:41:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:41:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:41:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:41:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:41:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:41:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:41:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:41:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:41:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:41:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:41:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:41:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:41:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:41:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:41:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:41:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 120.052268942, 2024-04-22T18:41:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:41:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:41:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:41:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:41:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:41:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:41:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:41:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:41:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:41:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:41:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:41:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:41:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:41:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:41:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:41:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:41:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:41:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:41:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:41:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:41:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:41:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:41:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:41:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 119.95211605900001, 2024-04-22T18:41:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:41:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:41:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:41:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:41:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:41:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:41:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:41:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:41:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:41:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:41:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:41:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:41:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:41:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:41:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:41:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:41:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:40.770 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:40.771 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:40.703+0000 7f3e7be88640 -1 osd.0 164 handle_osd_map injecting map crc failure 2024-04-22T18:41:40.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:40.773 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:41:40.773 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:40.772+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:40.872 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:40.873+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:40.875 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:40.875 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:40.875 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:40.875 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:40.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:40.973 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:40.974+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:40.984 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:40.984 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:40.984 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:40.984 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:40.985 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:40.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:41.077 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:41.078+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:41.137 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 166.87730571, 2024-04-22T18:41:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:41:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:41:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:41:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:41:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:41:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:41:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:41:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:41:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:41:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:41:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:41:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:41:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:41:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:41:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:41:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:41:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:41:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 166.87252328, 2024-04-22T18:41:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:41:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:41:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:41:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:41:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:41:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:41:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:41:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:41:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:41:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:41:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:41:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:41:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:41:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:41:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:41:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:41:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:41:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:41:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:41:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:41:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:41:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:41:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:41:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 166.87242907699999, 2024-04-22T18:41:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:41:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:41:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:41:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:41:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:41:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:41:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:41:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:41:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:41:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:41:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:41:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:41:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:41:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:41:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:41:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:41:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:41:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:41:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:41:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:41:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:41:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:41:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:41:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 166.872408189, 2024-04-22T18:41:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:41:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:41:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:41:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:41:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:41:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:41:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:41:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:41:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:41:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:41:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:41:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:41:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 166.08940052599999, 2024-04-22T18:41:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:41:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:41:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:41:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:41:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:41:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:41:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:41:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:41:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:41:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:41:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:41:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:41:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:41:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:41:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:41:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:41:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:41:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:41:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:41:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 166.08156912000001, 2024-04-22T18:41:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:41:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:41:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:41:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:41:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:41:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:41:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:41:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:41:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:41:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:41:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:41:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 105.92969649, 2024-04-22T18:41:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:41:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:41:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:41:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:41:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:41:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:41:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:41:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:41:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:41:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:41:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:41:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:41:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:41:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:41:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:41:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:41:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 105.929291582, 2024-04-22T18:41:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:41:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:41:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:41:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:41:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:41:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:41:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:41:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:41:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:41:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:41:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:41:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:41:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:41:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:41:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3213 4.a 4:5d2aee44:::smithi08937929-13:head [assert-version v66,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 78.568391693999999, 2024-04-22T18:41:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69928701400000004, 2024-04-22T18:41:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3213 2024-04-22T18:41:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568559+0000", 2024-04-22T18:41:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568567+0000", 2024-04-22T18:41:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-04-22T18:41:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568579+0000", 2024-04-22T18:41:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-04-22T18:41:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.668711+0000", 2024-04-22T18:41:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100131866 2024-04-22T18:41:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.768945+0000", 2024-04-22T18:41:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100234252 2024-04-22T18:41:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769002+0000", 2024-04-22T18:41:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7092999999999997e-05 2024-04-22T18:41:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.166404+0000", 2024-04-22T18:41:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39740241999999998 2024-04-22T18:41:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.191686+0000", 2024-04-22T18:41:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025281739000000001 2024-04-22T18:41:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267758+0000", 2024-04-22T18:41:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076071606999999999 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267812+0000", 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4478999999999998e-05 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267848+0000", 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6062999999999997e-05 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3319 4.a 4:5d2aee44:::smithi08937929-13:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:41:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 74.676643943000002, 2024-04-22T18:41:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476623401, 2024-04-22T18:41:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3319 2024-04-22T18:41:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460306+0000", 2024-04-22T18:41:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460318+0000", 2024-04-22T18:41:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2896e-05 2024-04-22T18:41:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460345+0000", 2024-04-22T18:41:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6650000000000001e-05 2024-04-22T18:41:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460360+0000", 2024-04-22T18:41:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4729000000000001e-05 2024-04-22T18:41:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460502+0000", 2024-04-22T18:41:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141729 2024-04-22T18:41:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460644+0000", 2024-04-22T18:41:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142755 2024-04-22T18:41:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.936932+0000", 2024-04-22T18:41:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476288078 2024-04-22T18:41:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3412 4.a 4:519c08f7:::smithi08937929-15:head [write 2529878~394573] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.994809382, 2024-04-22T18:41:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48918094899999998, 2024-04-22T18:41:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3412 2024-04-22T18:41:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142141+0000", 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142496+0000", 2024-04-22T18:41:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035444699999999998 2024-04-22T18:41:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142510+0000", 2024-04-22T18:41:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3841e-05 2024-04-22T18:41:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142518+0000", 2024-04-22T18:41:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8089999999999993e-06 2024-04-22T18:41:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142560+0000", 2024-04-22T18:41:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2097e-05 2024-04-22T18:41:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142637+0000", 2024-04-22T18:41:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6965000000000006e-05 2024-04-22T18:41:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142715+0000", 2024-04-22T18:41:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8694000000000002e-05 2024-04-22T18:41:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144179+0000", 2024-04-22T18:41:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146342 2024-04-22T18:41:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631229+0000", 2024-04-22T18:41:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48705018500000002 2024-04-22T18:41:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631281+0000", 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2216999999999998e-05 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631324+0000", 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3344999999999999e-05 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3413 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2924451] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.994428241999998, 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58904086899999997, 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3413 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142523+0000", 2024-04-22T18:41:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142529+0000", 2024-04-22T18:41:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6049999999999997e-06 2024-04-22T18:41:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142535+0000", 2024-04-22T18:41:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4740000000000002e-06 2024-04-22T18:41:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142541+0000", 2024-04-22T18:41:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1839999999999998e-06 2024-04-22T18:41:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143553+0000", 2024-04-22T18:41:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010126689999999999 2024-04-22T18:41:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143659+0000", 2024-04-22T18:41:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105393 2024-04-22T18:41:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143825+0000", 2024-04-22T18:41:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165809 2024-04-22T18:41:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144329+0000", 2024-04-22T18:41:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050483799999999997 2024-04-22T18:41:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731480+0000", 2024-04-22T18:41:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58715100600000003 2024-04-22T18:41:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731522+0000", 2024-04-22T18:41:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2098000000000002e-05 2024-04-22T18:41:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731566+0000", 2024-04-22T18:41:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3066999999999999e-05 2024-04-22T18:41:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3414 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 69.994399638000004, 2024-04-22T18:41:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58926020199999996, 2024-04-22T18:41:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3414 2024-04-22T18:41:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142552+0000", 2024-04-22T18:41:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142556+0000", 2024-04-22T18:41:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0289999999999999e-06 2024-04-22T18:41:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142574+0000", 2024-04-22T18:41:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7266999999999999e-05 2024-04-22T18:41:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142581+0000", 2024-04-22T18:41:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-04-22T18:41:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144121+0000", 2024-04-22T18:41:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015397480000000001 2024-04-22T18:41:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144170+0000", 2024-04-22T18:41:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8803e-05 2024-04-22T18:41:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731624+0000", 2024-04-22T18:41:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58745365800000005 2024-04-22T18:41:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731712+0000", 2024-04-22T18:41:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8016999999999994e-05 2024-04-22T18:41:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731813+0000", 2024-04-22T18:41:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101817 2024-04-22T18:41:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:77 3.3 3:dcc8aeec:::3590d5741efdda41a26b62a0982727439225b7b4b51097b313f30d8dc10bf6f6:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.889381315999998, 2024-04-22T18:41:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011812878, 2024-04-22T18:41:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247571+0000", 2024-04-22T18:41:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247574+0000", 2024-04-22T18:41:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:41:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247577+0000", 2024-04-22T18:41:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3069999999999998e-06 2024-04-22T18:41:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247580+0000", 2024-04-22T18:41:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8870000000000001e-06 2024-04-22T18:41:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.340928+0000", 2024-04-22T18:41:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093348678000000004 2024-04-22T18:41:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341122+0000", 2024-04-22T18:41:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000193227 2024-04-22T18:41:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618335+0000", 2024-04-22T18:41:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.277213775 2024-04-22T18:41:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618422+0000", 2024-04-22T18:41:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6537000000000002e-05 2024-04-22T18:41:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954387+0000", 2024-04-22T18:41:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33596509699999999 2024-04-22T18:41:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954548+0000", 2024-04-22T18:41:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016042199999999999 2024-04-22T18:41:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954858+0000", 2024-04-22T18:41:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000310396 2024-04-22T18:41:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157019+0000", 2024-04-22T18:41:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20216071299999999 2024-04-22T18:41:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259213+0000", 2024-04-22T18:41:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102194734 2024-04-22T18:41:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259262+0000", 2024-04-22T18:41:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9052999999999999e-05 2024-04-22T18:41:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259384+0000", 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122052 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:78 3.3 3:df7e90d2:::ab0657e24bd96704775de2a378746bc22309c5a7366a5eaf9b4e1f1f2bef0a4c:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.889366119000002, 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011974739, 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247586+0000", 2024-04-22T18:41:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247589+0000", 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.181e-06 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247592+0000", 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0230000000000001e-06 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247595+0000", 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.428e-06 2024-04-22T18:41:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341160+0000", 2024-04-22T18:41:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093564571999999999 2024-04-22T18:41:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341264+0000", 2024-04-22T18:41:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010391999999999999 2024-04-22T18:41:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619269+0000", 2024-04-22T18:41:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27800575100000002 2024-04-22T18:41:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619346+0000", 2024-04-22T18:41:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6459999999999996e-05 2024-04-22T18:41:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.155957+0000", 2024-04-22T18:41:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53661109299999998 2024-04-22T18:41:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156049+0000", 2024-04-22T18:41:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2272e-05 2024-04-22T18:41:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156236+0000", 2024-04-22T18:41:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000186808 2024-04-22T18:41:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.158088+0000", 2024-04-22T18:41:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018522079999999999 2024-04-22T18:41:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259421+0000", 2024-04-22T18:41:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101332718 2024-04-22T18:41:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259460+0000", 2024-04-22T18:41:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8912000000000003e-05 2024-04-22T18:41:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259562+0000", 2024-04-22T18:41:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010175600000000001 2024-04-22T18:41:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:80 3.3 3:f90f47f2:::e5603f596d699ee3e27e255225eeb90a9f0c1b4a4d099d1c586b19019220aa3f:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.889327042000005, 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0121096759999999, 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247625+0000", 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247628+0000", 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6129999999999998e-06 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247631+0000", 2024-04-22T18:41:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3869999999999999e-06 2024-04-22T18:41:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247634+0000", 2024-04-22T18:41:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9919999999999999e-06 2024-04-22T18:41:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341550+0000", 2024-04-22T18:41:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093915584999999996 2024-04-22T18:41:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341700+0000", 2024-04-22T18:41:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015000300000000001 2024-04-22T18:41:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620744+0000", 2024-04-22T18:41:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27904399200000002 2024-04-22T18:41:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620794+0000", 2024-04-22T18:41:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9579000000000001e-05 2024-04-22T18:41:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157597+0000", 2024-04-22T18:41:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53680343799999997 2024-04-22T18:41:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157678+0000", 2024-04-22T18:41:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0829999999999997e-05 2024-04-22T18:41:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157832+0000", 2024-04-22T18:41:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154184 2024-04-22T18:41:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259572+0000", 2024-04-22T18:41:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10174003600000001 2024-04-22T18:41:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259601+0000", 2024-04-22T18:41:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8841999999999999e-05 2024-04-22T18:41:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259638+0000", 2024-04-22T18:41:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7308999999999998e-05 2024-04-22T18:41:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259736+0000", 2024-04-22T18:41:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7283999999999999e-05 2024-04-22T18:41:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:135 3.3 3:f89134e5:::3e4e73f5d4a0122957865f0faeac2483204ee98d7d497ef3b54afe2a02b39a3d:head [read 0~802304 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.521511962999995, 2024-04-22T18:41:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.64370653499999997, 2024-04-22T18:41:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615438+0000", 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615449+0000", 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1219999999999999e-05 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615460+0000", 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0403e-05 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.715588+0000", 2024-04-22T18:41:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100128881 2024-04-22T18:41:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722533+0000", 2024-04-22T18:41:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069444010000000002 2024-04-22T18:41:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722658+0000", 2024-04-22T18:41:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012560699999999999 2024-04-22T18:41:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826437+0000", 2024-04-22T18:41:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103778194 2024-04-22T18:41:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826532+0000", 2024-04-22T18:41:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5111000000000005e-05 2024-04-22T18:41:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258787+0000", 2024-04-22T18:41:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43225529099999999 2024-04-22T18:41:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258905+0000", 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118348 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259148+0000", 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000242093 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3890 4.1 4:83c955bf:::smithi08937929-28:head [write 2699879~469889] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.522837119000002, 2024-04-22T18:41:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49042297000000001, 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3890 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614115+0000", 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614496+0000", 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038181700000000001 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614509+0000", 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2381000000000001e-05 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614517+0000", 2024-04-22T18:41:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0879999999999995e-06 2024-04-22T18:41:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615610+0000", 2024-04-22T18:41:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010933519999999999 2024-04-22T18:41:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615655+0000", 2024-04-22T18:41:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4966000000000003e-05 2024-04-22T18:41:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615769+0000", 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113986 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704882+0000", 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089112825000000007 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104488+0000", 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39960572999999999 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104517+0000", 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9006e-05 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104539+0000", 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2073e-05 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3891 4.1 4:83c955bf:::smithi08937929-28:head [setxattr _header (56) in=7b,truncate 3630345] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.522428888, 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59018588699999996, 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3891 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614523+0000", 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614528+0000", 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0379999999999999e-06 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614540+0000", 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018000000000001e-05 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614546+0000", 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4990000000000002e-06 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704438+0000", 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089892044000000004 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704504+0000", 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6365000000000006e-05 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704602+0000", 2024-04-22T18:41:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8048999999999996e-05 2024-04-22T18:41:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.806370+0000", 2024-04-22T18:41:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10176768 2024-04-22T18:41:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204629+0000", 2024-04-22T18:41:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39825917999999999 2024-04-22T18:41:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204673+0000", 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3516999999999997e-05 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204710+0000", 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7391000000000002e-05 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3892 4.1 4:83c955bf:::smithi08937929-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 51.522402307, 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59031281800000002, 2024-04-22T18:41:41.240 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3892 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614550+0000", 2024-04-22T18:41:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614553+0000", 2024-04-22T18:41:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0879999999999999e-06 2024-04-22T18:41:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614559+0000", 2024-04-22T18:41:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9259999999999997e-06 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614565+0000", 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7950000000000004e-06 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704818+0000", 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090252624000000004 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704866+0000", 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8667e-05 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204742+0000", 2024-04-22T18:41:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987551899999999 2024-04-22T18:41:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204799+0000", 2024-04-22T18:41:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7299000000000001e-05 2024-04-22T18:41:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204863+0000", 2024-04-22T18:41:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4455000000000005e-05 2024-04-22T18:41:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.245 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:41.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:41.246 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:41.247+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:41.346 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:41.347+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:41.388 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:41.389 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:41.389 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:41.389 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:41.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:41.447 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:41.448+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:41.509 INFO:tasks.rados.rados.0.smithi089.stdout:2002: set_chunk oid 24 offset: 1211904 length: 396288 target oid a6954558d6d261bd8039fe057a41a32eb57dde7f3e53a34b25ba0607fb91f814 offset: 0 snap -1 2024-04-22T18:41:41.509 INFO:tasks.rados.rados.0.smithi089.stdout:2003: tier_evict oid 28 2024-04-22T18:41:41.509 INFO:tasks.rados.rados.0.smithi089.stdout:2003: tier_evict oid 28 snap -1 2024-04-22T18:41:41.509 INFO:tasks.rados.rados.0.smithi089.stdout:2004: write (excl) oid 48 current snap is 0 2024-04-22T18:41:41.509 INFO:tasks.rados.rados.0.smithi089.stdout:1991: finishing write tid 4 to smithi08937929-6 2024-04-22T18:41:41.509 INFO:tasks.rados.rados.0.smithi089.stdout:1991: oid 6 updating version 0 to 115 2024-04-22T18:41:41.509 INFO:tasks.rados.rados.0.smithi089.stdout:1991: oid 6 version 115 is already newer than 113 2024-04-22T18:41:41.509 INFO:tasks.rados.rados.0.smithi089.stdout:1991: oid 6 version 115 is already newer than 114 2024-04-22T18:41:41.509 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 6 v 115 (ObjNum 426 snap 0 seq_num 426) dirty exists 2024-04-22T18:41:41.509 INFO:tasks.rados.rados.0.smithi089.stdout:1991: left oid 6 (ObjNum 426 snap 0 seq_num 426) 2024-04-22T18:41:41.509 INFO:tasks.rados.rados.0.smithi089.stdout:1997: expect (ObjNum 366 snap 0 seq_num 366) 2024-04-22T18:41:41.538 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:41.539 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:41.539 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:41.539 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:41.539 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:41.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:41.547 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:41.548+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:41.648 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:41.649+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:41.659 INFO:tasks.rados.rados.0.smithi089.stdout:1992: expect (ObjNum 303 snap 0 seq_num 303) 2024-04-22T18:41:41.692 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 176.838681796, 2024-04-22T18:41:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:41:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.693 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:41:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.694 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:41:41.695 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:41:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:41:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:41:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:41:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:41:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.696 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:41:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:41:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:41:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:41:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:41:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:41:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.697 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:41:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:41:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:41:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:41:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:41:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:41:41.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:41:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:41:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 176.838630634, 2024-04-22T18:41:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:41:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:41:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:41.700 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.701 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:41:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:41:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:41:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:41:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:41:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:41:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.702 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:41:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:41:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:41:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:41:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:41:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:41:41.703 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:41:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:41:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:41:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:41:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.704 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:41:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:41:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:41:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:41:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.705 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 176.83861540999999, 2024-04-22T18:41:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:41:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:41:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.706 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:41:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:41:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:41:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:41:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:41:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.708 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:41:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:41:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:41:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:41:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:41:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:41:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:41:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:41:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:41:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:41:41.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:41:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:41:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:41:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:41:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.711 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:41:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:41:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 176.83855583299999, 2024-04-22T18:41:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:41:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.712 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:41:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.713 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:41:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:41:41.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:41:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:41:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:41:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:41:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:41:41.715 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:41:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:41:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:41:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:41:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:41:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.716 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:41:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:41:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:41:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:41:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:41:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:41:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:41:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:41:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 176.83847351700001, 2024-04-22T18:41:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:41:41.718 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:41:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:41:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:41:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:41:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:41:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:41:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:41:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:41:41.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:41:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:41:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:41:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:41:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:41.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:41:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:41:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:41:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:41:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:41:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:41:41.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:41:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:41:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:41:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:41:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.724 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 176.83846336100001, 2024-04-22T18:41:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:41:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:41:41.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:41:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:41:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:41:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:41:41.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:41:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:41:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:41:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:41:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:41:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.728 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:41:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:41:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:41:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:41:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:41:41.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:41:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:41:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:41:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:41:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:41:41.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:41:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:41:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3163 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:41.731 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.855450923000006, 2024-04-22T18:41:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1009801299999999, 2024-04-22T18:41:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3163 2024-04-22T18:41:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:41.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837455+0000", 2024-04-22T18:41:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837464+0000", 2024-04-22T18:41:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5319999999999995e-06 2024-04-22T18:41:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837476+0000", 2024-04-22T18:41:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1759e-05 2024-04-22T18:41:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837488+0000", 2024-04-22T18:41:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1666e-05 2024-04-22T18:41:41.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837541+0000", 2024-04-22T18:41:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2939999999999998e-05 2024-04-22T18:41:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837547+0000", 2024-04-22T18:41:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6409999999999996e-06 2024-04-22T18:41:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937423+0000", 2024-04-22T18:41:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.099875545 2024-04-22T18:41:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937567+0000", 2024-04-22T18:41:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014363099999999999 2024-04-22T18:41:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938437+0000", 2024-04-22T18:41:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087060399999999997 2024-04-22T18:41:41.736 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3164 4.8 4:187adac1:::smithi08937929-36:head [sparse-read 0~2643888] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.855411554, 2024-04-22T18:41:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1013670310000001, 2024-04-22T18:41:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.737 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3164 2024-04-22T18:41:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837499+0000", 2024-04-22T18:41:41.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.994e-06 2024-04-22T18:41:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837503+0000", 2024-04-22T18:41:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7249999999999999e-06 2024-04-22T18:41:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837506+0000", 2024-04-22T18:41:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5980000000000001e-06 2024-04-22T18:41:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937641+0000", 2024-04-22T18:41:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10013467700000001 2024-04-22T18:41:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937650+0000", 2024-04-22T18:41:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3489999999999998e-06 2024-04-22T18:41:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938468+0000", 2024-04-22T18:41:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0008176710000001 2024-04-22T18:41:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938582+0000", 2024-04-22T18:41:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114287 2024-04-22T18:41:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938864+0000", 2024-04-22T18:41:41.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028120899999999998 2024-04-22T18:41:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3165 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.855392785999996, 2024-04-22T18:41:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2022754499999999, 2024-04-22T18:41:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3165 2024-04-22T18:41:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:41.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837517+0000", 2024-04-22T18:41:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3369999999999998e-06 2024-04-22T18:41:41.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837523+0000", 2024-04-22T18:41:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5450000000000003e-06 2024-04-22T18:41:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837526+0000", 2024-04-22T18:41:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.354e-06 2024-04-22T18:41:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937682+0000", 2024-04-22T18:41:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100155917 2024-04-22T18:41:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937688+0000", 2024-04-22T18:41:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:41:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.038953+0000", 2024-04-22T18:41:41.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012648110000001 2024-04-22T18:41:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039034+0000", 2024-04-22T18:41:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0969000000000004e-05 2024-04-22T18:41:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039791+0000", 2024-04-22T18:41:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075672900000000004 2024-04-22T18:41:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3183 4.8 4:1bc4668a:::smithi08937929-21:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.656989588000002, 2024-04-22T18:41:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0040478260000001, 2024-04-22T18:41:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.749 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3183 2024-04-22T18:41:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035913+0000", 2024-04-22T18:41:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999943 2024-04-22T18:41:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035935+0000", 2024-04-22T18:41:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2232999999999999e-05 2024-04-22T18:41:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035969+0000", 2024-04-22T18:41:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3969e-05 2024-04-22T18:41:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035987+0000", 2024-04-22T18:41:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8397e-05 2024-04-22T18:41:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:41.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2916000000000003e-05 2024-04-22T18:41:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:41:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0103e-05 2024-04-22T18:41:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039841+0000", 2024-04-22T18:41:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0037508959999999 2024-04-22T18:41:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039966+0000", 2024-04-22T18:41:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012514500000000001 2024-04-22T18:41:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3184 4.8 4:12d9f826:::smithi08937929-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:41.754 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.656901629000004, 2024-04-22T18:41:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.00402868, 2024-04-22T18:41:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3184 2024-04-22T18:41:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.755 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036005+0000", 2024-04-22T18:41:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036026+0000", 2024-04-22T18:41:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1092999999999999e-05 2024-04-22T18:41:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036030+0000", 2024-04-22T18:41:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.467e-06 2024-04-22T18:41:41.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036033+0000", 2024-04-22T18:41:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9780000000000001e-06 2024-04-22T18:41:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036120+0000", 2024-04-22T18:41:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7495000000000002e-05 2024-04-22T18:41:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036125+0000", 2024-04-22T18:41:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8720000000000001e-06 2024-04-22T18:41:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039982+0000", 2024-04-22T18:41:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0038571919999999 2024-04-22T18:41:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040035+0000", 2024-04-22T18:41:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2992999999999999e-05 2024-04-22T18:41:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3185 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.656847717000005, 2024-04-22T18:41:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0041404039999999, 2024-04-22T18:41:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.760 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3185 2024-04-22T18:41:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036062+0000", 2024-04-22T18:41:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9099999999999999e-06 2024-04-22T18:41:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036066+0000", 2024-04-22T18:41:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1699999999999999e-06 2024-04-22T18:41:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036077+0000", 2024-04-22T18:41:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0637e-05 2024-04-22T18:41:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036147+0000", 2024-04-22T18:41:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0025999999999997e-05 2024-04-22T18:41:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036151+0000", 2024-04-22T18:41:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:41:41.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040068+0000", 2024-04-22T18:41:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.003916604 2024-04-22T18:41:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040201+0000", 2024-04-22T18:41:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013275400000000001 2024-04-22T18:41:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.765 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3186 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.656828160999993, 2024-04-22T18:41:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2044167510000001, 2024-04-22T18:41:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.766 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3186 2024-04-22T18:41:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036079+0000", 2024-04-22T18:41:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036081+0000", 2024-04-22T18:41:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6449999999999999e-06 2024-04-22T18:41:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036084+0000", 2024-04-22T18:41:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9399999999999998e-06 2024-04-22T18:41:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:41:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.879e-06 2024-04-22T18:41:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036168+0000", 2024-04-22T18:41:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7657999999999994e-05 2024-04-22T18:41:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036171+0000", 2024-04-22T18:41:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7639999999999999e-06 2024-04-22T18:41:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240345+0000", 2024-04-22T18:41:41.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2041734260000001 2024-04-22T18:41:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240497+0000", 2024-04-22T18:41:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015182800000000001 2024-04-22T18:41:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.771 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3724 4.18 4:1bc4668a:::smithi08937929-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.632503784999997, 2024-04-22T18:41:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50413114699999995, 2024-04-22T18:41:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3724 2024-04-22T18:41:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060405+0000", 2024-04-22T18:41:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.395e-06 2024-04-22T18:41:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060408+0000", 2024-04-22T18:41:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:41:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060411+0000", 2024-04-22T18:41:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6060000000000001e-06 2024-04-22T18:41:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165799+0000", 2024-04-22T18:41:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105388369 2024-04-22T18:41:41.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165843+0000", 2024-04-22T18:41:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3891999999999999e-05 2024-04-22T18:41:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564408+0000", 2024-04-22T18:41:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39856466000000002 2024-04-22T18:41:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564461+0000", 2024-04-22T18:41:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3529999999999997e-05 2024-04-22T18:41:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564535+0000", 2024-04-22T18:41:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4147999999999995e-05 2024-04-22T18:41:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.777 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4413 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.035523740999999, 2024-04-22T18:41:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2025380569999999, 2024-04-22T18:41:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4413 2024-04-22T18:41:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.778 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657377+0000", 2024-04-22T18:41:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999933 2024-04-22T18:41:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657393+0000", 2024-04-22T18:41:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5296999999999999e-05 2024-04-22T18:41:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657411+0000", 2024-04-22T18:41:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8105e-05 2024-04-22T18:41:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657424+0000", 2024-04-22T18:41:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3030000000000001e-05 2024-04-22T18:41:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658313+0000", 2024-04-22T18:41:41.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00088943099999999999 2024-04-22T18:41:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658324+0000", 2024-04-22T18:41:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0496000000000001e-05 2024-04-22T18:41:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859560+0000", 2024-04-22T18:41:41.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2012358350000001 2024-04-22T18:41:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859648+0000", 2024-04-22T18:41:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8744000000000003e-05 2024-04-22T18:41:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859922+0000", 2024-04-22T18:41:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027396499999999998 2024-04-22T18:41:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.783 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4414 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.035469417000002, 2024-04-22T18:41:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2030255030000001, 2024-04-22T18:41:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.784 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4414 2024-04-22T18:41:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657437+0000", 2024-04-22T18:41:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657443+0000", 2024-04-22T18:41:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5899999999999998e-06 2024-04-22T18:41:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657450+0000", 2024-04-22T18:41:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4649999999999997e-06 2024-04-22T18:41:41.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657457+0000", 2024-04-22T18:41:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6000000000000003e-06 2024-04-22T18:41:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658364+0000", 2024-04-22T18:41:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090735999999999996 2024-04-22T18:41:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:41.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658373+0000", 2024-04-22T18:41:41.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3250000000000008e-06 2024-04-22T18:41:41.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860258+0000", 2024-04-22T18:41:41.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018852769999999 2024-04-22T18:41:41.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860308+0000", 2024-04-22T18:41:41.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9938000000000002e-05 2024-04-22T18:41:41.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860464+0000", 2024-04-22T18:41:41.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015622800000000001 2024-04-22T18:41:41.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.789 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.789 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4415 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:41.790 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:41.790 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.035445093, 2024-04-22T18:41:41.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2038250989999999, 2024-04-22T18:41:41.790 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.790 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.790 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.790 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4415 2024-04-22T18:41:41.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.790 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:41.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:41.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657462+0000", 2024-04-22T18:41:41.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:41.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657467+0000", 2024-04-22T18:41:41.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5399999999999997e-06 2024-04-22T18:41:41.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657473+0000", 2024-04-22T18:41:41.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3949999999999998e-06 2024-04-22T18:41:41.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657480+0000", 2024-04-22T18:41:41.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2489999999999998e-06 2024-04-22T18:41:41.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658394+0000", 2024-04-22T18:41:41.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00091403299999999997 2024-04-22T18:41:41.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:41.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658398+0000", 2024-04-22T18:41:41.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7979999999999996e-06 2024-04-22T18:41:41.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860495+0000", 2024-04-22T18:41:41.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.202097081 2024-04-22T18:41:41.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860543+0000", 2024-04-22T18:41:41.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7907000000000001e-05 2024-04-22T18:41:41.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861288+0000", 2024-04-22T18:41:41.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074466500000000004 2024-04-22T18:41:41.795 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.795 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.795 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.796 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4413 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:41.796 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:41.796 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.032728588000001, 2024-04-22T18:41:41.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.201491952, 2024-04-22T18:41:41.796 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.796 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.796 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.796 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.796 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.796 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4413 2024-04-22T18:41:41.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.797 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:41.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:41.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660178+0000", 2024-04-22T18:41:41.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:41.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660185+0000", 2024-04-22T18:41:41.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4379999999999998e-06 2024-04-22T18:41:41.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660201+0000", 2024-04-22T18:41:41.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5252999999999999e-05 2024-04-22T18:41:41.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660210+0000", 2024-04-22T18:41:41.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1069999999999994e-06 2024-04-22T18:41:41.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660270+0000", 2024-04-22T18:41:41.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739000000000002e-05 2024-04-22T18:41:41.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:41.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660281+0000", 2024-04-22T18:41:41.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0664e-05 2024-04-22T18:41:41.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861304+0000", 2024-04-22T18:41:41.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2010225059999999 2024-04-22T18:41:41.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.800 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.800 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861342+0000", 2024-04-22T18:41:41.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8736000000000003e-05 2024-04-22T18:41:41.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861671+0000", 2024-04-22T18:41:41.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032915899999999998 2024-04-22T18:41:41.801 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.801 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.801 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.801 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4414 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:41.801 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:41.801 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.032685078, 2024-04-22T18:41:41.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2014303930000001, 2024-04-22T18:41:41.802 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.802 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.802 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.802 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.802 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.802 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4414 2024-04-22T18:41:41.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.802 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:41.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:41.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660222+0000", 2024-04-22T18:41:41.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:41.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660227+0000", 2024-04-22T18:41:41.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6789999999999996e-06 2024-04-22T18:41:41.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660240+0000", 2024-04-22T18:41:41.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3485e-05 2024-04-22T18:41:41.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660245+0000", 2024-04-22T18:41:41.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8269999999999997e-06 2024-04-22T18:41:41.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660307+0000", 2024-04-22T18:41:41.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2392999999999997e-05 2024-04-22T18:41:41.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:41.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660316+0000", 2024-04-22T18:41:41.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7790000000000001e-06 2024-04-22T18:41:41.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861493+0000", 2024-04-22T18:41:41.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.201177341 2024-04-22T18:41:41.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861529+0000", 2024-04-22T18:41:41.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5978e-05 2024-04-22T18:41:41.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861653+0000", 2024-04-22T18:41:41.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012391799999999999 2024-04-22T18:41:41.807 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.807 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.807 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.807 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4415 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:41.807 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:41.807 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.032639853999999, 2024-04-22T18:41:41.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2019634779999999, 2024-04-22T18:41:41.807 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:41.807 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:41.807 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:41.808 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:41.808 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:41.808 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4415 2024-04-22T18:41:41.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.808 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:41.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:41.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:41.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:41.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:41.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:41.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:41.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660267+0000", 2024-04-22T18:41:41.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:41.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:41.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660273+0000", 2024-04-22T18:41:41.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5910000000000003e-06 2024-04-22T18:41:41.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:41.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660280+0000", 2024-04-22T18:41:41.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4000000000000003e-06 2024-04-22T18:41:41.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660288+0000", 2024-04-22T18:41:41.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1769999999999998e-06 2024-04-22T18:41:41.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.760401+0000", 2024-04-22T18:41:41.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100112864 2024-04-22T18:41:41.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:41.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.760408+0000", 2024-04-22T18:41:41.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0299999999999996e-06 2024-04-22T18:41:41.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:41.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861682+0000", 2024-04-22T18:41:41.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012742520000001 2024-04-22T18:41:41.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:41.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861716+0000", 2024-04-22T18:41:41.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3674999999999999e-05 2024-04-22T18:41:41.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:41.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:41.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:41.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.862232+0000", 2024-04-22T18:41:41.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051554300000000003 2024-04-22T18:41:41.812 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.812 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:41.813 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:41.813 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:41.814 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:41.815+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:41.815 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:41.910 INFO:tasks.rados.rados.0.smithi089.stdout:1995: expect (ObjNum 412 snap 0 seq_num 412) 2024-04-22T18:41:41.915 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:41.916+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:41.919 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(4.1 166/164 [PushOp(4:81200c09:::smithi08937929-23:head, version: 147'185, data_included: [497573~960365,1515641~88211], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:81200c09:::smithi08937929-23:head@147'185, size: 2554730, copy_subset: [0~2554730], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1603852, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-22T18:41:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:41.815568+0000", 2024-04-22T18:41:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.105521869, 2024-04-22T18:41:41.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105555015, 2024-04-22T18:41:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:41:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.815568+0000", 2024-04-22T18:41:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.815568+0000", 2024-04-22T18:41:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:41.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.815566+0000", 2024-04-22T18:41:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.816439+0000", 2024-04-22T18:41:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087314100000000002 2024-04-22T18:41:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.816454+0000", 2024-04-22T18:41:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4569000000000001e-05 2024-04-22T18:41:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:41.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.816465+0000", 2024-04-22T18:41:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0511e-05 2024-04-22T18:41:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:41:41.922 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 1 2024-04-22T18:41:41.922 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:41.928 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:42.016 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:42.017+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:42.032 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:42.032 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:42.032 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:42.032 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:42.032 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:42.041 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:42.084 INFO:tasks.rados.rados.0.smithi089.stdout:1996: expect (ObjNum 370 snap 0 seq_num 370) 2024-04-22T18:41:42.125 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:42.127+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:42.187 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:42.187 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:42.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:42.187 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:42.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.188 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:42.188 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:42.188 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 167.73178213200001, 2024-04-22T18:41:42.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:41:42.188 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.188 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.188 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.188 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.188 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.188 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:41:42.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.188 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:42.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:42.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:41:42.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:42.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:41:42.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:41:42.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:41:42.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:41:42.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:41:42.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:41:42.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:41:42.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:41:42.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:41:42.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:41:42.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:41:42.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:41:42.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:41:42.193 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.193 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.193 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.193 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:42.193 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:42.193 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 159.038724884, 2024-04-22T18:41:42.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:41:42.194 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.194 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.194 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:41:42.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.194 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:42.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:41:42.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:41:42.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:41:42.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:41:42.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:41:42.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:41:42.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:41:42.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:41:42.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:41:42.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:41:42.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:41:42.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:41:42.198 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.198 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.198 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.199 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:42.199 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:42.199 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 159.038706408, 2024-04-22T18:41:42.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:41:42.199 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.199 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.199 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.199 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.199 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.199 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:41:42.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.199 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:41:42.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:41:42.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:41:42.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:41:42.203 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.203 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.203 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.203 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:41:42.203 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:42.203 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 131.275658096, 2024-04-22T18:41:42.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:41:42.203 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.203 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.204 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.204 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.204 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.204 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:41:42.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.204 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:42.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:42.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:41:42.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:42.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:41:42.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:41:42.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:41:42.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:41:42.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:41:42.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:41:42.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:41:42.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:41:42.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:41:42.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:41:42.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:42.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:41:42.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:41:42.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:41:42.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:41:42.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:41:42.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:41:42.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:41:42.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:41:42.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:41:42.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:41:42.209 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.209 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.209 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.209 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:42.209 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:42.209 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.603229787, 2024-04-22T18:41:42.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:41:42.209 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.209 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.210 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:42.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:42.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:41:42.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:42.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:41:42.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:41:42.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:41:42.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:41:42.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:41:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:41:42.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:41:42.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:41:42.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:41:42.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:41:42.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:41:42.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:41:42.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:41:42.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:41:42.214 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.214 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.214 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.214 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:41:42.214 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:42.214 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 111.79421308000001, 2024-04-22T18:41:42.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:41:42.215 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.215 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.215 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.215 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.215 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.215 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:41:42.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.215 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:42.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:42.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:41:42.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:41:42.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:41:42.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:41:42.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:41:42.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:41:42.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:41:42.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:41:42.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:41:42.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:42.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:41:42.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:41:42.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:41:42.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:41:42.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:41:42.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:41:42.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:41:42.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:41:42.219 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.219 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.219 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.220 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:41:42.220 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:42.220 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 94.898174707999999, 2024-04-22T18:41:42.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:41:42.220 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.220 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.220 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.220 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.220 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.220 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:41:42.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.220 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:42.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:42.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:42.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:41:42.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:41:42.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:41:42.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:41:42.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:41:42.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.223 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.224 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.224 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3377 4.6 4:6700eab5:::smithi08937929-32:head [read 1812992~307712] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-04-22T18:41:42.224 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:42.224 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.750771065999999, 2024-04-22T18:41:42.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.52001745799999999, 2024-04-22T18:41:42.224 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.224 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.224 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.224 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.224 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.224 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3377 2024-04-22T18:41:42.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.224 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:42.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:42.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436364+0000", 2024-04-22T18:41:42.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:42.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436376+0000", 2024-04-22T18:41:42.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2121e-05 2024-04-22T18:41:42.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436394+0000", 2024-04-22T18:41:42.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8094e-05 2024-04-22T18:41:42.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436405+0000", 2024-04-22T18:41:42.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0189e-05 2024-04-22T18:41:42.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436471+0000", 2024-04-22T18:41:42.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6817999999999996e-05 2024-04-22T18:41:42.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436481+0000", 2024-04-22T18:41:42.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2720000000000003e-06 2024-04-22T18:41:42.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956084+0000", 2024-04-22T18:41:42.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51960320800000004 2024-04-22T18:41:42.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956232+0000", 2024-04-22T18:41:42.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014778600000000001 2024-04-22T18:41:42.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956385+0000", 2024-04-22T18:41:42.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153394 2024-04-22T18:41:42.229 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.229 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.229 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.229 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:131 3.1 3:a0a19997:::ee8ca23a7050e771c77dff761e6ea5a11ed57796f4e8a17cbb5804a7fdc937b9:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:42.229 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:42.229 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 70.857901470000002, 2024-04-22T18:41:42.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51449564999999997, 2024-04-22T18:41:42.229 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.229 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.229 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:42.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:42.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329235+0000", 2024-04-22T18:41:42.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:42.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329244+0000", 2024-04-22T18:41:42.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6819999999999992e-06 2024-04-22T18:41:42.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329259+0000", 2024-04-22T18:41:42.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698000000000001e-05 2024-04-22T18:41:42.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329265+0000", 2024-04-22T18:41:42.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1510000000000001e-06 2024-04-22T18:41:42.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329311+0000", 2024-04-22T18:41:42.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6096e-05 2024-04-22T18:41:42.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:42.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.332269+0000", 2024-04-22T18:41:42.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029581910000000002 2024-04-22T18:41:42.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630603+0000", 2024-04-22T18:41:42.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29833418 2024-04-22T18:41:42.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630685+0000", 2024-04-22T18:41:42.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1217999999999994e-05 2024-04-22T18:41:42.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:42.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630813+0000", 2024-04-22T18:41:42.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128706 2024-04-22T18:41:42.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.631447+0000", 2024-04-22T18:41:42.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063427000000000004 2024-04-22T18:41:42.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843586+0000", 2024-04-22T18:41:42.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21213873 2024-04-22T18:41:42.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843623+0000", 2024-04-22T18:41:42.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6848999999999999e-05 2024-04-22T18:41:42.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843733+0000", 2024-04-22T18:41:42.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010982200000000001 2024-04-22T18:41:42.235 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.235 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.235 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.235 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3441 4.b 4:d3068ee3:::smithi08937929-29:head [write 558065~619869] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:42.235 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:42.235 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 70.269425408999993, 2024-04-22T18:41:42.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50272020399999995, 2024-04-22T18:41:42.235 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.235 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.235 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.235 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.236 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.236 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3441 2024-04-22T18:41:42.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.236 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:42.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:42.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917711+0000", 2024-04-22T18:41:42.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:42.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917838+0000", 2024-04-22T18:41:42.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012734899999999999 2024-04-22T18:41:42.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917854+0000", 2024-04-22T18:41:42.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6130999999999999e-05 2024-04-22T18:41:42.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917863+0000", 2024-04-22T18:41:42.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5969999999999992e-06 2024-04-22T18:41:42.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992845+0000", 2024-04-22T18:41:42.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.074982193000000003 2024-04-22T18:41:42.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992919+0000", 2024-04-22T18:41:42.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:41:42.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:42.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.993019+0000", 2024-04-22T18:41:42.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9718999999999999e-05 2024-04-22T18:41:42.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095554+0000", 2024-04-22T18:41:42.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102535193 2024-04-22T18:41:42.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420183+0000", 2024-04-22T18:41:42.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32462853600000002 2024-04-22T18:41:42.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420401+0000", 2024-04-22T18:41:42.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021881400000000001 2024-04-22T18:41:42.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420433+0000", 2024-04-22T18:41:42.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079000000000002e-05 2024-04-22T18:41:42.241 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.241 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.241 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.241 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3442 4.b 4:d3068ee3:::smithi08937929-29:head [write 1579835~486765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:42.241 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:42.241 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 70.264738698000002, 2024-04-22T18:41:42.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49822799099999998, 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3442 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922399+0000", 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922502+0000", 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102784 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922513+0000", 2024-04-22T18:41:42.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1051e-05 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922518+0000", 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8529999999999997e-06 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093158+0000", 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.17063958800000001 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093230+0000", 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1803000000000004e-05 2024-04-22T18:41:42.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093326+0000", 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6454999999999997e-05 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095557+0000", 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022306399999999999 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420484+0000", 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32492717700000001 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420606+0000", 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012225800000000001 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420628+0000", 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1596000000000001e-05 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3443 4.b 4:d3068ee3:::smithi08937929-29:head [write 2468371~489694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 70.260136650999996, 2024-04-22T18:41:42.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49372487199999998, 2024-04-22T18:41:42.247 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.247 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.247 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.247 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.247 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.247 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3443 2024-04-22T18:41:42.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.247 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:42.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:42.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927001+0000", 2024-04-22T18:41:42.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:42.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927117+0000", 2024-04-22T18:41:42.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011637 2024-04-22T18:41:42.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927128+0000", 2024-04-22T18:41:42.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0696000000000001e-05 2024-04-22T18:41:42.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927134+0000", 2024-04-22T18:41:42.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8640000000000001e-06 2024-04-22T18:41:42.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094288+0000", 2024-04-22T18:41:42.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16715412900000001 2024-04-22T18:41:42.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094359+0000", 2024-04-22T18:41:42.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1639999999999998e-05 2024-04-22T18:41:42.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:42.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094451+0000", 2024-04-22T18:41:42.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1252e-05 2024-04-22T18:41:42.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095560+0000", 2024-04-22T18:41:42.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011095530000000001 2024-04-22T18:41:42.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420661+0000", 2024-04-22T18:41:42.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32510105500000003 2024-04-22T18:41:42.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420710+0000", 2024-04-22T18:41:42.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9018999999999999e-05 2024-04-22T18:41:42.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420727+0000", 2024-04-22T18:41:42.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6722e-05 2024-04-22T18:41:42.252 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.252 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.252 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.252 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3444 4.b 4:d3068ee3:::smithi08937929-29:head [setxattr _header (56) in=7b,truncate 2958065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:42.252 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:42.253 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 70.260000934000004, 2024-04-22T18:41:42.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49366935299999998, 2024-04-22T18:41:42.253 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.253 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.253 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.253 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.253 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.253 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3444 2024-04-22T18:41:42.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.253 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:42.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:42.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927137+0000", 2024-04-22T18:41:42.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927141+0000", 2024-04-22T18:41:42.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3270000000000001e-06 2024-04-22T18:41:42.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927144+0000", 2024-04-22T18:41:42.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9730000000000002e-06 2024-04-22T18:41:42.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927147+0000", 2024-04-22T18:41:42.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.959e-06 2024-04-22T18:41:42.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095118+0000", 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16797115500000001 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095169+0000", 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1316000000000001e-05 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095260+0000", 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1184e-05 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095999+0000", 2024-04-22T18:41:42.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073868200000000005 2024-04-22T18:41:42.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420757+0000", 2024-04-22T18:41:42.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32475783499999999 2024-04-22T18:41:42.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420781+0000", 2024-04-22T18:41:42.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4022000000000001e-05 2024-04-22T18:41:42.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420807+0000", 2024-04-22T18:41:42.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6366000000000001e-05 2024-04-22T18:41:42.258 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.258 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.258 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.258 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3445 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:42.258 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:42.258 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 70.259989840000003, 2024-04-22T18:41:42.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.493780049, 2024-04-22T18:41:42.258 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.258 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.258 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.258 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.259 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.259 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3445 2024-04-22T18:41:42.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.259 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:42.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:42.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:42.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927150+0000", 2024-04-22T18:41:42.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.542e-06 2024-04-22T18:41:42.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927153+0000", 2024-04-22T18:41:42.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0520000000000002e-06 2024-04-22T18:41:42.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027218+0000", 2024-04-22T18:41:42.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100064604 2024-04-22T18:41:42.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095501+0000", 2024-04-22T18:41:42.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.068283431000000006 2024-04-22T18:41:42.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:42.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095545+0000", 2024-04-22T18:41:42.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4295999999999997e-05 2024-04-22T18:41:42.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420833+0000", 2024-04-22T18:41:42.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32528718400000001 2024-04-22T18:41:42.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420877+0000", 2024-04-22T18:41:42.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4403999999999997e-05 2024-04-22T18:41:42.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420929+0000", 2024-04-22T18:41:42.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1848999999999998e-05 2024-04-22T18:41:42.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.263 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.263 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4780 4.6 4:630bd8e5:::smithi08937929-45:head [write 574905~711397,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:42.263 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.278066+0000", 2024-04-22T18:41:42.263 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.909072595, 2024-04-22T18:41:42.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88434433999999995, 2024-04-22T18:41:42.263 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4780 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278066+0000", 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278066+0000", 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278063+0000", 2024-04-22T18:41:42.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:42.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278391+0000", 2024-04-22T18:41:42.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032839499999999999 2024-04-22T18:41:42.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278404+0000", 2024-04-22T18:41:42.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3528e-05 2024-04-22T18:41:42.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278415+0000", 2024-04-22T18:41:42.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0295000000000001e-05 2024-04-22T18:41:42.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278454+0000", 2024-04-22T18:41:42.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9551999999999997e-05 2024-04-22T18:41:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278462+0000", 2024-04-22T18:41:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4080000000000001e-06 2024-04-22T18:41:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.959123+0000", 2024-04-22T18:41:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68066109799999996 2024-04-22T18:41:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.959282+0000", 2024-04-22T18:41:42.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000159023 2024-04-22T18:41:42.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:42.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.959448+0000", 2024-04-22T18:41:42.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016666200000000001 2024-04-22T18:41:42.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162299+0000", 2024-04-22T18:41:42.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20285081199999999 2024-04-22T18:41:42.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162316+0000", 2024-04-22T18:41:42.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6354999999999998e-05 2024-04-22T18:41:42.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162347+0000", 2024-04-22T18:41:42.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1427999999999998e-05 2024-04-22T18:41:42.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162410+0000", 2024-04-22T18:41:42.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3350999999999997e-05 2024-04-22T18:41:42.270 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.270 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.270 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.270 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4781 4.6 4:630bd8e5:::smithi08937929-45:head [write 1885086~525140] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:42.270 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.284151+0000", 2024-04-22T18:41:42.270 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.902987755, 2024-04-22T18:41:42.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88746760999999996, 2024-04-22T18:41:42.270 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.270 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.270 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.271 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.271 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.271 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4781 2024-04-22T18:41:42.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.271 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284151+0000", 2024-04-22T18:41:42.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284151+0000", 2024-04-22T18:41:42.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284149+0000", 2024-04-22T18:41:42.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:42.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284385+0000", 2024-04-22T18:41:42.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023556799999999999 2024-04-22T18:41:42.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284396+0000", 2024-04-22T18:41:42.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0842999999999999e-05 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284401+0000", 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3260000000000003e-06 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284444+0000", 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3186000000000003e-05 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284454+0000", 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6160000000000008e-06 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.161315+0000", 2024-04-22T18:41:42.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87686138000000002 2024-04-22T18:41:42.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.161381+0000", 2024-04-22T18:41:42.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5838000000000002e-05 2024-04-22T18:41:42.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:42.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.161470+0000", 2024-04-22T18:41:42.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9332000000000005e-05 2024-04-22T18:41:42.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163245+0000", 2024-04-22T18:41:42.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017744530000000001 2024-04-22T18:41:42.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.171550+0000", 2024-04-22T18:41:42.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083046879999999993 2024-04-22T18:41:42.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.171592+0000", 2024-04-22T18:41:42.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2438999999999998e-05 2024-04-22T18:41:42.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.171619+0000", 2024-04-22T18:41:42.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6517e-05 2024-04-22T18:41:42.276 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.276 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.276 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.276 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4782 4.6 4:630bd8e5:::smithi08937929-45:head [write 2930189~44716] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:42.276 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.287206+0000", 2024-04-22T18:41:42.276 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.899932787999999, 2024-04-22T18:41:42.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98327066299999999, 2024-04-22T18:41:42.276 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.276 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.276 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.277 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.277 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.277 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4782 2024-04-22T18:41:42.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.277 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287206+0000", 2024-04-22T18:41:42.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287206+0000", 2024-04-22T18:41:42.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287205+0000", 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287220+0000", 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5709000000000002e-05 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287231+0000", 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0196e-05 2024-04-22T18:41:42.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287236+0000", 2024-04-22T18:41:42.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3770000000000004e-06 2024-04-22T18:41:42.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287264+0000", 2024-04-22T18:41:42.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8549999999999999e-05 2024-04-22T18:41:42.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287268+0000", 2024-04-22T18:41:42.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7299999999999999e-06 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162435+0000", 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87516707999999999 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162490+0000", 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5210000000000002e-05 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162567+0000", 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6947999999999996e-05 2024-04-22T18:41:42.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163249+0000", 2024-04-22T18:41:42.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00068115299999999999 2024-04-22T18:41:42.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.270414+0000", 2024-04-22T18:41:42.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10716516700000001 2024-04-22T18:41:42.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.270448+0000", 2024-04-22T18:41:42.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4042999999999999e-05 2024-04-22T18:41:42.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.270477+0000", 2024-04-22T18:41:42.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8796e-05 2024-04-22T18:41:42.282 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.282 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.282 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.282 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4783 4.6 4:630bd8e5:::smithi08937929-45:head [setxattr _header (56) in=7b,truncate 2974905] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:42.282 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.287242+0000", 2024-04-22T18:41:42.282 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.899896674000001, 2024-04-22T18:41:42.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98449730599999996, 2024-04-22T18:41:42.283 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.283 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.283 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.283 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.283 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.283 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4783 2024-04-22T18:41:42.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.283 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287242+0000", 2024-04-22T18:41:42.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287242+0000", 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287241+0000", 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287245+0000", 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.997e-06 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287251+0000", 2024-04-22T18:41:42.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4999999999999999e-06 2024-04-22T18:41:42.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387333+0000", 2024-04-22T18:41:42.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100082174 2024-04-22T18:41:42.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387418+0000", 2024-04-22T18:41:42.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5086000000000005e-05 2024-04-22T18:41:42.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387427+0000", 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6829999999999997e-06 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162901+0000", 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77547466899999995 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162942+0000", 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0299999999999997e-05 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163028+0000", 2024-04-22T18:41:42.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6267000000000006e-05 2024-04-22T18:41:42.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163938+0000", 2024-04-22T18:41:42.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091012500000000002 2024-04-22T18:41:42.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271677+0000", 2024-04-22T18:41:42.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107739168 2024-04-22T18:41:42.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271709+0000", 2024-04-22T18:41:42.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1191e-05 2024-04-22T18:41:42.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271739+0000", 2024-04-22T18:41:42.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0814e-05 2024-04-22T18:41:42.288 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.288 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.288 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.288 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4784 4.6 4:630bd8e5:::smithi08937929-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:42.288 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.387355+0000", 2024-04-22T18:41:42.288 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.799783597999999, 2024-04-22T18:41:42.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.884536714, 2024-04-22T18:41:42.289 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.289 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.289 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.289 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.289 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.289 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4784 2024-04-22T18:41:42.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.289 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387355+0000", 2024-04-22T18:41:42.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387355+0000", 2024-04-22T18:41:42.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387352+0000", 2024-04-22T18:41:42.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:42.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387386+0000", 2024-04-22T18:41:42.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3974999999999999e-05 2024-04-22T18:41:42.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387399+0000", 2024-04-22T18:41:42.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2882e-05 2024-04-22T18:41:42.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387408+0000", 2024-04-22T18:41:42.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2310000000000002e-06 2024-04-22T18:41:42.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.487520+0000", 2024-04-22T18:41:42.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100111622 2024-04-22T18:41:42.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.487531+0000", 2024-04-22T18:41:42.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0655999999999999e-05 2024-04-22T18:41:42.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163199+0000", 2024-04-22T18:41:42.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.67566841700000002 2024-04-22T18:41:42.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:42.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163239+0000", 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9749000000000002e-05 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271780+0000", 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10854082600000001 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271830+0000", 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0835999999999999e-05 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271892+0000", 2024-04-22T18:41:42.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1440999999999997e-05 2024-04-22T18:41:42.294 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.294 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.294 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.294 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4940 3.1 3:af0c84c6:::a6954558d6d261bd8039fe057a41a32eb57dde7f3e53a34b25ba0607fb91f814:head [write 0~396288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-04-22T18:41:42.294 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:41.009760+0000", 2024-04-22T18:41:42.294 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.177378925, 2024-04-22T18:41:42.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.500823562, 2024-04-22T18:41:42.294 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.294 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.294 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.294 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.294 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.295 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4940 2024-04-22T18:41:42.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.295 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009760+0000", 2024-04-22T18:41:42.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009760+0000", 2024-04-22T18:41:42.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009758+0000", 2024-04-22T18:41:42.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:42.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009935+0000", 2024-04-22T18:41:42.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017724200000000001 2024-04-22T18:41:42.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009946+0000", 2024-04-22T18:41:42.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0477e-05 2024-04-22T18:41:42.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009952+0000", 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3090000000000001e-06 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.109570+0000", 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099618130999999999 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.109693+0000", 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012241699999999999 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:42.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.109807+0000", 2024-04-22T18:41:42.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011426599999999999 2024-04-22T18:41:42.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.110527+0000", 2024-04-22T18:41:42.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00071964499999999999 2024-04-22T18:41:42.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.510490+0000", 2024-04-22T18:41:42.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39996386499999997 2024-04-22T18:41:42.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.510550+0000", 2024-04-22T18:41:42.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9669000000000001e-05 2024-04-22T18:41:42.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.510583+0000", 2024-04-22T18:41:42.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3281000000000002e-05 2024-04-22T18:41:42.299 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.299 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.299 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.299 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.299 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.299 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:42.301 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:42.303 INFO:tasks.rados.rados.0.smithi089.stdout:1994: finishing write tid 1 to smithi08937929-29 2024-04-22T18:41:42.303 INFO:tasks.rados.rados.0.smithi089.stdout:1994: finishing write tid 2 to smithi08937929-29 2024-04-22T18:41:42.303 INFO:tasks.rados.rados.0.smithi089.stdout:1994: finishing write tid 3 to smithi08937929-29 2024-04-22T18:41:42.304 INFO:tasks.rados.rados.0.smithi089.stdout:2004: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2024-04-22T18:41:42.304 INFO:tasks.rados.rados.0.smithi089.stdout:2004: writing smithi08937929-48 from 427010 to 899848 tid 1 2024-04-22T18:41:42.304 INFO:tasks.rados.rados.0.smithi089.stdout:2004: writing smithi08937929-48 from 1329610 to 2024091 tid 2 2024-04-22T18:41:42.304 INFO:tasks.rados.rados.0.smithi089.stdout:2004: writing smithi08937929-48 from 2708828 to 3227010 tid 3 2024-04-22T18:41:42.304 INFO:tasks.rados.rados.0.smithi089.stdout:1991: done (12 left) 2024-04-22T18:41:42.304 INFO:tasks.rados.rados.0.smithi089.stdout:1992: done (11 left) 2024-04-22T18:41:42.304 INFO:tasks.rados.rados.0.smithi089.stdout:1995: done (10 left) 2024-04-22T18:41:42.304 INFO:tasks.rados.rados.0.smithi089.stdout:1996: done (9 left) 2024-04-22T18:41:42.304 INFO:tasks.rados.rados.0.smithi089.stdout:1997: done (8 left) 2024-04-22T18:41:42.304 INFO:tasks.rados.rados.0.smithi089.stdout:2005: read oid 22 snap -1 2024-04-22T18:41:42.304 INFO:tasks.rados.rados.0.smithi089.stdout:2005: expect (ObjNum 303 snap 0 seq_num 303) 2024-04-22T18:41:42.304 INFO:tasks.rados.rados.0.smithi089.stdout:1994: finishing write tid 4 to smithi08937929-29 2024-04-22T18:41:42.304 INFO:tasks.rados.rados.0.smithi089.stdout:1994: oid 29 updating version 0 to 150 2024-04-22T18:41:42.304 INFO:tasks.rados.rados.0.smithi089.stdout:1994: oid 29 updating version 150 to 152 2024-04-22T18:41:42.305 INFO:tasks.rados.rados.0.smithi089.stdout:1994: oid 29 version 152 is already newer than 151 2024-04-22T18:41:42.305 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 29 v 152 (ObjNum 428 snap 0 seq_num 428) dirty exists 2024-04-22T18:41:42.305 INFO:tasks.rados.rados.0.smithi089.stdout:1994: left oid 29 (ObjNum 428 snap 0 seq_num 428) 2024-04-22T18:41:42.305 INFO:tasks.rados.rados.0.smithi089.stdout:2001: got (22) Invalid argument 2024-04-22T18:41:42.305 INFO:tasks.rados.rados.0.smithi089.stdout:1999: finishing set_chunk to oid 1 2024-04-22T18:41:42.305 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 1 v 57 (ObjNum 347 snap 0 seq_num 347) dirty exists 2024-04-22T18:41:42.305 INFO:tasks.rados.rados.0.smithi089.stdout:2000: finishing write tid 1 to smithi08937929-26 2024-04-22T18:41:42.305 INFO:tasks.rados.rados.0.smithi089.stdout:2000: finishing write tid 2 to smithi08937929-26 2024-04-22T18:41:42.305 INFO:tasks.rados.rados.0.smithi089.stdout:2000: finishing write tid 3 to smithi08937929-26 2024-04-22T18:41:42.305 INFO:tasks.rados.rados.0.smithi089.stdout:2000: finishing write tid 4 to smithi08937929-26 2024-04-22T18:41:42.305 INFO:tasks.rados.rados.0.smithi089.stdout:2006: read oid 36 snap -1 2024-04-22T18:41:42.305 INFO:tasks.rados.rados.0.smithi089.stdout:2006: expect deleted 2024-04-22T18:41:42.305 INFO:tasks.rados.rados.0.smithi089.stdout:1994: done (9 left) 2024-04-22T18:41:42.305 INFO:tasks.rados.rados.0.smithi089.stdout:1999: done (8 left) 2024-04-22T18:41:42.306 INFO:tasks.rados.rados.0.smithi089.stdout:2001: done (7 left) 2024-04-22T18:41:42.306 INFO:tasks.rados.rados.0.smithi089.stdout:2007: write (excl) oid 7 current snap is 0 2024-04-22T18:41:42.306 INFO:tasks.rados.rados.0.smithi089.stdout:2000: finishing write tid 5 to smithi08937929-26 2024-04-22T18:41:42.306 INFO:tasks.rados.rados.0.smithi089.stdout:2000: oid 26 updating version 0 to 148 2024-04-22T18:41:42.306 INFO:tasks.rados.rados.0.smithi089.stdout:2000: oid 26 version 148 is already newer than 147 2024-04-22T18:41:42.306 INFO:tasks.rados.rados.0.smithi089.stdout:2000: oid 26 updating version 148 to 150 2024-04-22T18:41:42.306 INFO:tasks.rados.rados.0.smithi089.stdout:2000: oid 26 version 150 is already newer than 149 2024-04-22T18:41:42.306 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 26 v 150 (ObjNum 429 snap 0 seq_num 429) dirty exists 2024-04-22T18:41:42.306 INFO:tasks.rados.rados.0.smithi089.stdout:2000: left oid 26 (ObjNum 429 snap 0 seq_num 429) 2024-04-22T18:41:42.306 INFO:tasks.rados.rados.0.smithi089.stdout:1998: finishing set_chunk to oid 30 2024-04-22T18:41:42.306 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 30 v 82 (ObjNum 425 snap 0 seq_num 425) dirty exists 2024-04-22T18:41:42.306 INFO:tasks.rados.rados.0.smithi089.stdout:2002: finishing set_chunk to oid 24 2024-04-22T18:41:42.307 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 1211904 length: 396288 2024-04-22T18:41:42.307 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 24 v 0 (ObjNum 384 snap 0 seq_num 384) dirty exists 2024-04-22T18:41:42.307 INFO:tasks.rados.rados.0.smithi089.stdout:2003: got (22) Invalid argument 2024-04-22T18:41:42.307 INFO:tasks.rados.rados.0.smithi089.stdout:2004: finishing write tid 1 to smithi08937929-48 2024-04-22T18:41:42.307 INFO:tasks.rados.rados.0.smithi089.stdout:2007: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2024-04-22T18:41:42.307 INFO:tasks.rados.rados.0.smithi089.stdout:2007: writing smithi08937929-7 from 443817 to 991904 tid 1 2024-04-22T18:41:42.307 INFO:tasks.rados.rados.0.smithi089.stdout:2007: writing smithi08937929-7 from 1788092 to 2226231 tid 2 2024-04-22T18:41:42.310 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:42.301+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:42.310 INFO:tasks.rados.rados.0.smithi089.stdout:2007: writing smithi08937929-7 from 3019898 to 3243817 tid 3 2024-04-22T18:41:42.310 INFO:tasks.rados.rados.0.smithi089.stdout:1998: done (7 left) 2024-04-22T18:41:42.310 INFO:tasks.rados.rados.0.smithi089.stdout:2000: done (6 left) 2024-04-22T18:41:42.310 INFO:tasks.rados.rados.0.smithi089.stdout:2002: done (5 left) 2024-04-22T18:41:42.310 INFO:tasks.rados.rados.0.smithi089.stdout:2003: done (4 left) 2024-04-22T18:41:42.310 INFO:tasks.rados.rados.0.smithi089.stdout:2008: set_chunk oid 2 target oid 2024-04-22T18:41:42.401 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:42.402+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:42.412 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:42.412 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:42.412 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:42.412 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:42.421 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:42.432 INFO:tasks.rados.rados.0.smithi089.stdout:2008: set_chunk oid 2 offset: 1597952 length: 256512 target oid 3ae004fbe3b046121fe96ad01c3acae6b38f234bcc26005bdcfd3ac22f27895f offset: 0 snap -1 2024-04-22T18:41:42.432 INFO:tasks.rados.rados.0.smithi089.stdout:2009: set_chunk oid 25 target oid 2024-04-22T18:41:42.502 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:42.504+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:42.523 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:42.523 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:42.523 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:42.523 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:42.523 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:42.532 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:42.536 INFO:tasks.rados.rados.0.smithi089.stdout:2009: set_chunk oid 25 offset: 350208 length: 312832 target oid 73eb72338d46de163eabe4fb1bac27a9de80e4b395da1f6ed0b848f6f088dac6 offset: 0 snap -1 2024-04-22T18:41:42.536 INFO:tasks.rados.rados.0.smithi089.stdout:2010: write (excl) oid 10 current snap is 0 2024-04-22T18:41:42.536 INFO:tasks.rados.rados.0.smithi089.stdout:2010: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2024-04-22T18:41:42.542 INFO:tasks.rados.rados.0.smithi089.stdout:2010: writing smithi08937929-10 from 460624 to 1083960 tid 1 2024-04-22T18:41:42.550 INFO:tasks.rados.rados.0.smithi089.stdout:2010: writing smithi08937929-10 from 1846574 to 2544724 tid 2 2024-04-22T18:41:42.554 INFO:tasks.rados.rados.0.smithi089.stdout:2010: writing smithi08937929-10 from 3163674 to 3260624 tid 3 2024-04-22T18:41:42.554 INFO:tasks.rados.rados.0.smithi089.stdout:2011: tier_evict oid 50 2024-04-22T18:41:42.554 INFO:tasks.rados.rados.0.smithi089.stdout:2011: tier_evict oid 50 snap -1 2024-04-22T18:41:42.554 INFO:tasks.rados.rados.0.smithi089.stdout:2007: finishing write tid 1 to smithi08937929-7 2024-04-22T18:41:42.554 INFO:tasks.rados.rados.0.smithi089.stdout:2005: expect (ObjNum 303 snap 0 seq_num 303) 2024-04-22T18:41:42.594 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:41:42.603 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:42.604+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:42.643 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:42.643 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:42.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:42.643 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:42.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.643 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:42.643 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:42.643 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 168.451662787, 2024-04-22T18:41:42.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:41:42.643 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:41:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:41:42.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:41:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:41:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:41:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:41:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:41:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:41:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:41:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:41:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:41:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:41:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:41:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:41:42.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:41:42.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:41:42.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:41:42.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:41:42.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:41:42.648 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.648 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.648 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 168.451651712, 2024-04-22T18:41:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:41:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:41:42.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:41:42.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:41:42.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3193 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 80.600443423000002, 2024-04-22T18:41:42.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69912102700000001, 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3193 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042744+0000", 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043013+0000", 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026906 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043028+0000", 2024-04-22T18:41:42.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5359999999999999e-05 2024-04-22T18:41:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043036+0000", 2024-04-22T18:41:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8420000000000005e-06 2024-04-22T18:41:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043081+0000", 2024-04-22T18:41:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5019999999999999e-05 2024-04-22T18:41:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043087+0000", 2024-04-22T18:41:42.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5420000000000002e-06 2024-04-22T18:41:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741856+0000", 2024-04-22T18:41:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69876864500000002 2024-04-22T18:41:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741867+0000", 2024-04-22T18:41:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.084e-05 2024-04-22T18:41:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3504 4.9 4:96737e19:::smithi08937929-4:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 67.232327708, 2024-04-22T18:41:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.951755291, 2024-04-22T18:41:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3504 2024-04-22T18:41:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410861+0000", 2024-04-22T18:41:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410863+0000", 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.635e-06 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410866+0000", 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8499999999999998e-06 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410868+0000", 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.497e-06 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615137+0000", 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20426903499999999 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615218+0000", 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0860999999999997e-05 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.362617+0000", 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.74739882700000004 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4194 4.b 4:d3068ee3:::smithi08937929-29:head [sparse-read 0~1980000,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.479068636999997, 2024-04-22T18:41:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.294164774, 2024-04-22T18:41:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4194 2024-04-22T18:41:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164120+0000", 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164125+0000", 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2899999999999996e-06 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164128+0000", 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8700000000000002e-06 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164132+0000", 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.151e-06 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164188+0000", 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6912e-05 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164201+0000", 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2466000000000001e-05 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.242879+0000", 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0786779710000001 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243364+0000", 2024-04-22T18:41:42.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000485081 2024-04-22T18:41:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456588+0000", 2024-04-22T18:41:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21322370900000001 2024-04-22T18:41:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456674+0000", 2024-04-22T18:41:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6079e-05 2024-04-22T18:41:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.458285+0000", 2024-04-22T18:41:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001611716 2024-04-22T18:41:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4195 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.479051015000003, 2024-04-22T18:41:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.395681808, 2024-04-22T18:41:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.667 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4195 2024-04-22T18:41:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164141+0000", 2024-04-22T18:41:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7160000000000001e-06 2024-04-22T18:41:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164147+0000", 2024-04-22T18:41:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0509999999999996e-06 2024-04-22T18:41:42.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264244+0000", 2024-04-22T18:41:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100097063 2024-04-22T18:41:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264319+0000", 2024-04-22T18:41:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5729000000000006e-05 2024-04-22T18:41:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264331+0000", 2024-04-22T18:41:42.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048999999999999e-05 2024-04-22T18:41:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243392+0000", 2024-04-22T18:41:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97906041099999996 2024-04-22T18:41:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243451+0000", 2024-04-22T18:41:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8717e-05 2024-04-22T18:41:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558398+0000", 2024-04-22T18:41:42.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31494718500000002 2024-04-22T18:41:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558490+0000", 2024-04-22T18:41:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2529999999999997e-05 2024-04-22T18:41:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559820+0000", 2024-04-22T18:41:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013299060000000001 2024-04-22T18:41:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.673 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4196 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:42.673 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:42.673 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.378912268000001, 2024-04-22T18:41:42.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2963829229999999, 2024-04-22T18:41:42.673 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.673 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.673 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.673 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.673 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.673 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4196 2024-04-22T18:41:42.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.673 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:42.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:42.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264275+0000", 2024-04-22T18:41:42.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:42.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264290+0000", 2024-04-22T18:41:42.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5373e-05 2024-04-22T18:41:42.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264300+0000", 2024-04-22T18:41:42.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.747e-06 2024-04-22T18:41:42.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264314+0000", 2024-04-22T18:41:42.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.396e-05 2024-04-22T18:41:42.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364450+0000", 2024-04-22T18:41:42.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013662500000001 2024-04-22T18:41:42.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364459+0000", 2024-04-22T18:41:42.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0710000000000001e-06 2024-04-22T18:41:42.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245690+0000", 2024-04-22T18:41:42.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88123066400000005 2024-04-22T18:41:42.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:42.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245766+0000", 2024-04-22T18:41:42.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5938000000000004e-05 2024-04-22T18:41:42.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559874+0000", 2024-04-22T18:41:42.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31410827499999999 2024-04-22T18:41:42.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559930+0000", 2024-04-22T18:41:42.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5367e-05 2024-04-22T18:41:42.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.560660+0000", 2024-04-22T18:41:42.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073043800000000003 2024-04-22T18:41:42.678 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.678 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.678 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.678 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4182 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:42.678 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:42.678 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.715229422999997, 2024-04-22T18:41:42.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2059652709999999, 2024-04-22T18:41:42.679 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.679 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.679 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.679 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.679 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.679 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4182 2024-04-22T18:41:42.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.679 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:42.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:42.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927959+0000", 2024-04-22T18:41:42.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927962+0000", 2024-04-22T18:41:42.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4269999999999998e-06 2024-04-22T18:41:42.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927968+0000", 2024-04-22T18:41:42.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:41:42.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028037+0000", 2024-04-22T18:41:42.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068544 2024-04-22T18:41:42.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028078+0000", 2024-04-22T18:41:42.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1523999999999997e-05 2024-04-22T18:41:42.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028087+0000", 2024-04-22T18:41:42.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3140000000000004e-06 2024-04-22T18:41:42.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.129951+0000", 2024-04-22T18:41:42.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1018649229999999 2024-04-22T18:41:42.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.130067+0000", 2024-04-22T18:41:42.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115222 2024-04-22T18:41:42.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.133925+0000", 2024-04-22T18:41:42.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00385849 2024-04-22T18:41:42.683 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.683 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.683 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.683 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4183 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:42.683 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:42.683 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.615138991000002, 2024-04-22T18:41:42.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1063017610000001, 2024-04-22T18:41:42.683 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4183 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:42.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028049+0000", 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028055+0000", 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028064+0000", 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2830000000000001e-06 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028071+0000", 2024-04-22T18:41:42.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2740000000000002e-06 2024-04-22T18:41:42.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028126+0000", 2024-04-22T18:41:42.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4771999999999997e-05 2024-04-22T18:41:42.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028134+0000", 2024-04-22T18:41:42.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5599999999999994e-06 2024-04-22T18:41:42.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134109+0000", 2024-04-22T18:41:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.105974864 2024-04-22T18:41:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134169+0000", 2024-04-22T18:41:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0167999999999997e-05 2024-04-22T18:41:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134352+0000", 2024-04-22T18:41:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018293699999999999 2024-04-22T18:41:42.687 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.688 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.688 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.688 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4190 4.c 4:3227431c:::smithi08937929-35:head [sparse-read 0~1258562,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:42.688 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:42.688 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.615113487999999, 2024-04-22T18:41:42.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1098740030000001, 2024-04-22T18:41:42.688 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.688 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.688 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4190 2024-04-22T18:41:42.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.689 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:42.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:42.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028075+0000", 2024-04-22T18:41:42.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:42.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028080+0000", 2024-04-22T18:41:42.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.561e-06 2024-04-22T18:41:42.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028085+0000", 2024-04-22T18:41:42.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0150000000000003e-06 2024-04-22T18:41:42.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028090+0000", 2024-04-22T18:41:42.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.169e-06 2024-04-22T18:41:42.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028175+0000", 2024-04-22T18:41:42.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4814999999999993e-05 2024-04-22T18:41:42.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028181+0000", 2024-04-22T18:41:42.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5490000000000003e-06 2024-04-22T18:41:42.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134400+0000", 2024-04-22T18:41:42.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1062186869999999 2024-04-22T18:41:42.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134552+0000", 2024-04-22T18:41:42.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015210500000000001 2024-04-22T18:41:42.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137950+0000", 2024-04-22T18:41:42.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033978210000000001 2024-04-22T18:41:42.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4191 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.615030740000002, 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.110917953, 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4191 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.693 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028157+0000", 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028164+0000", 2024-04-22T18:41:42.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.511e-06 2024-04-22T18:41:42.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028172+0000", 2024-04-22T18:41:42.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0339999999999997e-06 2024-04-22T18:41:42.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028179+0000", 2024-04-22T18:41:42.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8630000000000002e-06 2024-04-22T18:41:42.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028202+0000", 2024-04-22T18:41:42.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3635999999999999e-05 2024-04-22T18:41:42.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028209+0000", 2024-04-22T18:41:42.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4629999999999998e-06 2024-04-22T18:41:42.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137970+0000", 2024-04-22T18:41:42.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1097613479999999 2024-04-22T18:41:42.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.138034+0000", 2024-04-22T18:41:42.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3551000000000002e-05 2024-04-22T18:41:42.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.139077+0000", 2024-04-22T18:41:42.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001042715 2024-04-22T18:41:42.697 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.697 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.697 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4192 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.615005439999997, 2024-04-22T18:41:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.212555542, 2024-04-22T18:41:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4192 2024-04-22T18:41:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028183+0000", 2024-04-22T18:41:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028187+0000", 2024-04-22T18:41:42.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6119999999999999e-06 2024-04-22T18:41:42.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028192+0000", 2024-04-22T18:41:42.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3489999999999999e-06 2024-04-22T18:41:42.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028197+0000", 2024-04-22T18:41:42.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7489999999999998e-06 2024-04-22T18:41:42.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028228+0000", 2024-04-22T18:41:42.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1328999999999998e-05 2024-04-22T18:41:42.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028234+0000", 2024-04-22T18:41:42.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1129999999999997e-06 2024-04-22T18:41:42.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239245+0000", 2024-04-22T18:41:42.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.211011032 2024-04-22T18:41:42.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239384+0000", 2024-04-22T18:41:42.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013902000000000001 2024-04-22T18:41:42.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.240739+0000", 2024-04-22T18:41:42.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013550389999999999 2024-04-22T18:41:42.702 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.702 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.702 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.703 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4692 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:42.703 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:42.703 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 15.384049560999999, 2024-04-22T18:41:42.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80332623700000005, 2024-04-22T18:41:42.703 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.703 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.703 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.703 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.703 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.703 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4692 2024-04-22T18:41:42.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.703 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259139+0000", 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259145+0000", 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4860000000000002e-06 2024-04-22T18:41:42.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259151+0000", 2024-04-22T18:41:42.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1160000000000004e-06 2024-04-22T18:41:42.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259156+0000", 2024-04-22T18:41:42.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6489999999999999e-06 2024-04-22T18:41:42.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259191+0000", 2024-04-22T18:41:42.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4737999999999998e-05 2024-04-22T18:41:42.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259197+0000", 2024-04-22T18:41:42.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9160000000000004e-06 2024-04-22T18:41:42.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060062+0000", 2024-04-22T18:41:42.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80086472900000005 2024-04-22T18:41:42.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060225+0000", 2024-04-22T18:41:42.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000162771 2024-04-22T18:41:42.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062466+0000", 2024-04-22T18:41:42.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410569999999999 2024-04-22T18:41:42.707 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.707 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.707 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.707 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4693 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:42.707 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:42.707 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 15.384027274999999, 2024-04-22T18:41:42.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80370012999999996, 2024-04-22T18:41:42.707 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.708 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.708 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.708 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.708 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.708 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4693 2024-04-22T18:41:42.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.708 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:42.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:42.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259161+0000", 2024-04-22T18:41:42.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:42.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259164+0000", 2024-04-22T18:41:42.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-06 2024-04-22T18:41:42.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259169+0000", 2024-04-22T18:41:42.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6260000000000003e-06 2024-04-22T18:41:42.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259174+0000", 2024-04-22T18:41:42.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7659999999999998e-06 2024-04-22T18:41:42.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259231+0000", 2024-04-22T18:41:42.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6827000000000003e-05 2024-04-22T18:41:42.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259239+0000", 2024-04-22T18:41:42.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6889999999999993e-06 2024-04-22T18:41:42.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062492+0000", 2024-04-22T18:41:42.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80325246699999997 2024-04-22T18:41:42.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062573+0000", 2024-04-22T18:41:42.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1508000000000006e-05 2024-04-22T18:41:42.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062862+0000", 2024-04-22T18:41:42.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028893499999999998 2024-04-22T18:41:42.712 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.712 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.712 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.712 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4694 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:42.712 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:42.712 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 15.384010994, 2024-04-22T18:41:42.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.90474754099999999, 2024-04-22T18:41:42.712 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.713 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.713 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.713 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.713 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.713 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4694 2024-04-22T18:41:42.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.713 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:42.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:42.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:42.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259182+0000", 2024-04-22T18:41:42.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8700000000000002e-06 2024-04-22T18:41:42.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259187+0000", 2024-04-22T18:41:42.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1669999999999998e-06 2024-04-22T18:41:42.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259191+0000", 2024-04-22T18:41:42.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6210000000000003e-06 2024-04-22T18:41:42.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.359347+0000", 2024-04-22T18:41:42.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100155518 2024-04-22T18:41:42.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.359356+0000", 2024-04-22T18:41:42.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6519999999999995e-06 2024-04-22T18:41:42.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163126+0000", 2024-04-22T18:41:42.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.803770816 2024-04-22T18:41:42.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163217+0000", 2024-04-22T18:41:42.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0598000000000002e-05 2024-04-22T18:41:42.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163926+0000", 2024-04-22T18:41:42.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000708856 2024-04-22T18:41:42.719 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.719 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.719 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.719 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4791 4.7 4:e117a562:::smithi08937929-33:head [write 591712~786646] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.734465+0000", 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.90872454, 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4316495570000001, 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4791 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734465+0000", 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734465+0000", 2024-04-22T18:41:42.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734460+0000", 2024-04-22T18:41:42.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999957 2024-04-22T18:41:42.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734848+0000", 2024-04-22T18:41:42.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000387353 2024-04-22T18:41:42.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734865+0000", 2024-04-22T18:41:42.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7486000000000002e-05 2024-04-22T18:41:42.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734875+0000", 2024-04-22T18:41:42.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-06 2024-04-22T18:41:42.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761489+0000", 2024-04-22T18:41:42.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026613405 2024-04-22T18:41:42.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761495+0000", 2024-04-22T18:41:42.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4350000000000002e-06 2024-04-22T18:41:42.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.961065+0000", 2024-04-22T18:41:42.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.199569608 2024-04-22T18:41:42.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.961155+0000", 2024-04-22T18:41:42.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0109000000000001e-05 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.961254+0000", 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9072000000000003e-05 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.964390+0000", 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031362009999999999 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166027+0000", 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.201636589 2024-04-22T18:41:42.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166075+0000", 2024-04-22T18:41:42.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7998999999999999e-05 2024-04-22T18:41:42.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166114+0000", 2024-04-22T18:41:42.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9638999999999998e-05 2024-04-22T18:41:42.725 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.725 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.725 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.726 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4792 4.7 4:e117a562:::smithi08937929-33:head [write 1943568~668798] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:42.726 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742160+0000", 2024-04-22T18:41:42.726 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.901029426000001, 2024-04-22T18:41:42.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.42404771, 2024-04-22T18:41:42.726 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.726 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.726 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.726 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.726 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.726 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4792 2024-04-22T18:41:42.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.726 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742160+0000", 2024-04-22T18:41:42.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742160+0000", 2024-04-22T18:41:42.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742158+0000", 2024-04-22T18:41:42.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:42.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742473+0000", 2024-04-22T18:41:42.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000315192 2024-04-22T18:41:42.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742485+0000", 2024-04-22T18:41:42.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1795e-05 2024-04-22T18:41:42.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742491+0000", 2024-04-22T18:41:42.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8760000000000002e-06 2024-04-22T18:41:42.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761511+0000", 2024-04-22T18:41:42.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019020760000000001 2024-04-22T18:41:42.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761515+0000", 2024-04-22T18:41:42.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.625e-06 2024-04-22T18:41:42.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.962761+0000", 2024-04-22T18:41:42.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.201246021 2024-04-22T18:41:42.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.962837+0000", 2024-04-22T18:41:42.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6025999999999993e-05 2024-04-22T18:41:42.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:42.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.962944+0000", 2024-04-22T18:41:42.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106949 2024-04-22T18:41:42.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.965124+0000", 2024-04-22T18:41:42.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021803230000000001 2024-04-22T18:41:42.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166154+0000", 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20102966 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166185+0000", 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0713999999999998e-05 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166208+0000", 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2969999999999999e-05 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.731 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.732 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.732 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4793 4.7 4:e117a562:::smithi08937929-33:head [setxattr _header (56) in=7b,truncate 2991712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:42.732 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742495+0000", 2024-04-22T18:41:42.732 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.900694514, 2024-04-22T18:41:42.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5238806949999999, 2024-04-22T18:41:42.732 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.732 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:42.732 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.732 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.732 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.732 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4793 2024-04-22T18:41:42.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.732 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742495+0000", 2024-04-22T18:41:42.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742495+0000", 2024-04-22T18:41:42.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742494+0000", 2024-04-22T18:41:42.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742498+0000", 2024-04-22T18:41:42.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3790000000000002e-06 2024-04-22T18:41:42.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742501+0000", 2024-04-22T18:41:42.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9900000000000002e-06 2024-04-22T18:41:42.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742504+0000", 2024-04-22T18:41:42.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761529+0000", 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019025301000000001 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761532+0000", 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4589999999999998e-06 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.963997+0000", 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2024646560000001 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.964075+0000", 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8015000000000004e-05 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.964176+0000", 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010080499999999999 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.965235+0000", 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00105959 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266295+0000", 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30105968599999999 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266330+0000", 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5219999999999998e-05 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266376+0000", 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5262000000000001e-05 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.738 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4794 4.7 4:e117a562:::smithi08937929-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:42.738 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:41:42.738 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.900683401, 2024-04-22T18:41:42.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.524010123, 2024-04-22T18:41:42.738 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.738 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:42.738 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.738 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.738 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.738 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4794 2024-04-22T18:41:42.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.738 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:41:42.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:41:42.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:41:42.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742507+0000", 2024-04-22T18:41:42.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.637e-06 2024-04-22T18:41:42.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742511+0000", 2024-04-22T18:41:42.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9890000000000003e-06 2024-04-22T18:41:42.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742514+0000", 2024-04-22T18:41:42.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.88e-06 2024-04-22T18:41:42.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.775057+0000", 2024-04-22T18:41:42.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.032543012000000003 2024-04-22T18:41:42.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.775069+0000", 2024-04-22T18:41:42.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2184000000000001e-05 2024-04-22T18:41:42.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.064059+0000", 2024-04-22T18:41:42.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.288990157 2024-04-22T18:41:42.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:42.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.064127+0000", 2024-04-22T18:41:42.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7741000000000001e-05 2024-04-22T18:41:42.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266405+0000", 2024-04-22T18:41:42.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20227735499999999 2024-04-22T18:41:42.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:42.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266458+0000", 2024-04-22T18:41:42.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3430000000000002e-05 2024-04-22T18:41:42.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266516+0000", 2024-04-22T18:41:42.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8038000000000002e-05 2024-04-22T18:41:42.743 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.743 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.743 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.743 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4795 4.1f 4.d9a85c1f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:42.743 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:41:42.744 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.900673031, 2024-04-22T18:41:42.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.118060236, 2024-04-22T18:41:42.744 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:42.744 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:42.744 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:42.744 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:42.744 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:42.744 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4795 2024-04-22T18:41:42.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.744 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:42.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:42.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:41:42.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:42.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:41:42.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:42.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:42.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:41:42.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:42.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:42.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742518+0000", 2024-04-22T18:41:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.584e-06 2024-04-22T18:41:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742520+0000", 2024-04-22T18:41:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7140000000000002e-06 2024-04-22T18:41:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742523+0000", 2024-04-22T18:41:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3609999999999999e-06 2024-04-22T18:41:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761544+0000", 2024-04-22T18:41:42.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019020925000000001 2024-04-22T18:41:42.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:42.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761547+0000", 2024-04-22T18:41:42.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7859999999999998e-06 2024-04-22T18:41:42.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:42.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.860553+0000", 2024-04-22T18:41:42.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0990060859999999 2024-04-22T18:41:42.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:42.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:42.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:42.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.860577+0000", 2024-04-22T18:41:42.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3064000000000001e-05 2024-04-22T18:41:42.748 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.748 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.748 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.748 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:42.748 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:42.748 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:42.749 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:42.750 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:42.659+0000 7f3e7be88640 -1 osd.0 166 handle_osd_map injecting map crc failure 2024-04-22T18:41:42.750 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:42.750 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:42.750 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:42.751 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:42.751 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:42.750+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:42.755 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:42.755 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:42.773 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:42.773 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:42.774 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:41:42.791 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:42.791 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:42.809 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:42.809 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:42.816 INFO:tasks.rados.rados.0.smithi089.stdout:2008: finishing set_chunk to oid 2 2024-04-22T18:41:42.816 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk returned error code -95 offset: 1597952 length: 256512 2024-04-22T18:41:42.816 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 385 snap 0 seq_num 385) dirty exists 2024-04-22T18:41:42.816 INFO:tasks.rados.rados.0.smithi089.stdout:2004: finishing write tid 2 to smithi08937929-48 2024-04-22T18:41:42.816 INFO:tasks.rados.rados.0.smithi089.stdout:2005: done (7 left) 2024-04-22T18:41:42.816 INFO:tasks.rados.rados.0.smithi089.stdout:2006: done (6 left) 2024-04-22T18:41:42.816 INFO:tasks.rados.rados.0.smithi089.stdout:2008: done (5 left) 2024-04-22T18:41:42.817 INFO:tasks.rados.rados.0.smithi089.stdout:2012: write oid 15 current snap is 0 2024-04-22T18:41:42.817 INFO:tasks.rados.rados.0.smithi089.stdout:2004: finishing write tid 3 to smithi08937929-48 2024-04-22T18:41:42.817 INFO:tasks.rados.rados.0.smithi089.stdout:2004: finishing write tid 4 to smithi08937929-48 2024-04-22T18:41:42.817 INFO:tasks.rados.rados.0.smithi089.stdout:2004: finishing write tid 5 to smithi08937929-48 2024-04-22T18:41:42.817 INFO:tasks.rados.rados.0.smithi089.stdout:2004: oid 48 updating version 0 to 98 2024-04-22T18:41:42.817 INFO:tasks.rados.rados.0.smithi089.stdout:2004: oid 48 updating version 98 to 100 2024-04-22T18:41:42.817 INFO:tasks.rados.rados.0.smithi089.stdout:2004: oid 48 version 100 is already newer than 97 2024-04-22T18:41:42.817 INFO:tasks.rados.rados.0.smithi089.stdout:2004: oid 48 version 100 is already newer than 99 2024-04-22T18:41:42.817 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 48 v 100 (ObjNum 430 snap 0 seq_num 430) dirty exists 2024-04-22T18:41:42.817 INFO:tasks.rados.rados.0.smithi089.stdout:2004: left oid 48 (ObjNum 430 snap 0 seq_num 430) 2024-04-22T18:41:42.817 INFO:tasks.rados.rados.0.smithi089.stdout:2007: finishing write tid 2 to smithi08937929-7 2024-04-22T18:41:42.817 INFO:tasks.rados.rados.0.smithi089.stdout:2012: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2024-04-22T18:41:42.823 INFO:tasks.rados.rados.0.smithi089.stdout:2012: writing smithi08937929-15 from 477431 to 1176016 tid 1 2024-04-22T18:41:42.826 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:42.826 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:42.830 INFO:tasks.rados.rados.0.smithi089.stdout:2012: writing smithi08937929-15 from 1905056 to 2346864 tid 2 2024-04-22T18:41:42.836 INFO:tasks.rados.rados.0.smithi089.stdout:2012: writing smithi08937929-15 from 3074744 to 3277431 tid 3 2024-04-22T18:41:42.836 INFO:tasks.rados.rados.0.smithi089.stdout:2004: done (5 left) 2024-04-22T18:41:42.837 INFO:tasks.rados.rados.0.smithi089.stdout:2013: tier_evict oid 32 2024-04-22T18:41:42.837 INFO:tasks.rados.rados.0.smithi089.stdout:2013: tier_evict oid 32 snap -1 2024-04-22T18:41:42.837 INFO:tasks.rados.rados.0.smithi089.stdout:2007: finishing write tid 3 to smithi08937929-7 2024-04-22T18:41:42.837 INFO:tasks.rados.rados.0.smithi089.stdout:2010: finishing write tid 1 to smithi08937929-10 2024-04-22T18:41:42.837 INFO:tasks.rados.rados.0.smithi089.stdout:2007: finishing write tid 4 to smithi08937929-7 2024-04-22T18:41:42.837 INFO:tasks.rados.rados.0.smithi089.stdout:2010: finishing write tid 2 to smithi08937929-10 2024-04-22T18:41:42.837 INFO:tasks.rados.rados.0.smithi089.stdout:2014: write oid 42 current snap is 0 2024-04-22T18:41:42.837 INFO:tasks.rados.rados.0.smithi089.stdout:2010: finishing write tid 3 to smithi08937929-10 2024-04-22T18:41:42.837 INFO:tasks.rados.rados.0.smithi089.stdout:2010: finishing write tid 4 to smithi08937929-10 2024-04-22T18:41:42.837 INFO:tasks.rados.rados.0.smithi089.stdout:2010: finishing write tid 5 to smithi08937929-10 2024-04-22T18:41:42.837 INFO:tasks.rados.rados.0.smithi089.stdout:2010: oid 10 updating version 0 to 50 2024-04-22T18:41:42.837 INFO:tasks.rados.rados.0.smithi089.stdout:2010: oid 10 version 50 is already newer than 48 2024-04-22T18:41:42.837 INFO:tasks.rados.rados.0.smithi089.stdout:2010: oid 10 version 50 is already newer than 47 2024-04-22T18:41:42.837 INFO:tasks.rados.rados.0.smithi089.stdout:2010: oid 10 version 50 is already newer than 49 2024-04-22T18:41:42.838 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 10 v 50 (ObjNum 432 snap 0 seq_num 432) dirty exists 2024-04-22T18:41:42.838 INFO:tasks.rados.rados.0.smithi089.stdout:2010: left oid 10 (ObjNum 432 snap 0 seq_num 432) 2024-04-22T18:41:42.838 INFO:tasks.rados.rados.0.smithi089.stdout:2007: finishing write tid 5 to smithi08937929-7 2024-04-22T18:41:42.838 INFO:tasks.rados.rados.0.smithi089.stdout:2007: oid 7 updating version 0 to 88 2024-04-22T18:41:42.838 INFO:tasks.rados.rados.0.smithi089.stdout:2007: oid 7 version 88 is already newer than 86 2024-04-22T18:41:42.838 INFO:tasks.rados.rados.0.smithi089.stdout:2007: oid 7 version 88 is already newer than 87 2024-04-22T18:41:42.838 INFO:tasks.rados.rados.0.smithi089.stdout:2007: oid 7 version 88 is already newer than 85 2024-04-22T18:41:42.838 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 7 v 88 (ObjNum 431 snap 0 seq_num 431) dirty exists 2024-04-22T18:41:42.838 INFO:tasks.rados.rados.0.smithi089.stdout:2007: left oid 7 (ObjNum 431 snap 0 seq_num 431) 2024-04-22T18:41:42.838 INFO:tasks.rados.rados.0.smithi089.stdout:2014: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2024-04-22T18:41:42.842 INFO:tasks.rados.rados.0.smithi089.stdout:2014: writing smithi08937929-42 from 494238 to 984425 tid 1 2024-04-22T18:41:42.850 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:42.851+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:42.851 INFO:tasks.rados.rados.0.smithi089.stdout:2014: writing smithi08937929-42 from 1563538 to 2265357 tid 2 2024-04-22T18:41:42.858 INFO:tasks.rados.rados.0.smithi089.stdout:2014: writing smithi08937929-42 from 2818520 to 3294238 tid 3 2024-04-22T18:41:42.858 INFO:tasks.rados.rados.0.smithi089.stdout:2007: done (6 left) 2024-04-22T18:41:42.858 INFO:tasks.rados.rados.0.smithi089.stdout:2010: done (5 left) 2024-04-22T18:41:42.858 INFO:tasks.rados.rados.0.smithi089.stdout:2015: set_chunk oid 8 target oid 2024-04-22T18:41:42.858 INFO:tasks.rados.rados.0.smithi089.stdout:2015: set_chunk oid 8 offset: 0 length: 4000000 target oid smithi08937929- offset: 0 snap -1 2024-04-22T18:41:42.858 INFO:tasks.rados.rados.0.smithi089.stdout:2016: read oid 19 snap -1 2024-04-22T18:41:42.858 INFO:tasks.rados.rados.0.smithi089.stdout:2016: expect (ObjNum 413 snap 0 seq_num 413) 2024-04-22T18:41:42.858 INFO:tasks.rados.rados.0.smithi089.stdout:2017: delete oid 49 current snap is 0 2024-04-22T18:41:42.858 INFO:tasks.rados.rados.0.smithi089.stdout:2013: got (22) Invalid argument 2024-04-22T18:41:42.858 INFO:tasks.rados.rados.0.smithi089.stdout:2015: finishing set_chunk to oid 8 2024-04-22T18:41:42.858 INFO:tasks.rados.rados.0.smithi089.stdout:2015: got expected ENOENT (src dne) 2024-04-22T18:41:42.874 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:42.874 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:42.884 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:42.936 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:42.936 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:42.951 INFO:tasks.rados.rados.0.smithi089.stdout:2012: finishing write tid 1 to smithi08937929-15 2024-04-22T18:41:42.952 INFO:tasks.rados.rados.0.smithi089.stdout:2014: finishing write tid 1 to smithi08937929-42 2024-04-22T18:41:42.952 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:42.953+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:42.962 INFO:tasks.rados.rados.0.smithi089.stdout:2009: finishing set_chunk to oid 25 2024-04-22T18:41:42.962 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 25 v 89 (ObjNum 422 snap 0 seq_num 422) dirty exists 2024-04-22T18:41:42.964 INFO:tasks.rados.rados.0.smithi089.stdout:2011: got (22) Invalid argument 2024-04-22T18:41:43.031 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:43.031 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:43.040 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:43.056 INFO:tasks.rados.rados.0.smithi089.stdout:2016: expect (ObjNum 413 snap 0 seq_num 413) 2024-04-22T18:41:43.056 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:43.057+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:43.154 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:43.155+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:43.158 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:43.158 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:43.168 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:43.254 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:43.255+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:43.258 INFO:tasks.rados.rados.0.smithi089.stdout:2009: done (7 left) 2024-04-22T18:41:43.258 INFO:tasks.rados.rados.0.smithi089.stdout:2011: done (6 left) 2024-04-22T18:41:43.258 INFO:tasks.rados.rados.0.smithi089.stdout:2013: done (5 left) 2024-04-22T18:41:43.258 INFO:tasks.rados.rados.0.smithi089.stdout:2015: done (4 left) 2024-04-22T18:41:43.258 INFO:tasks.rados.rados.0.smithi089.stdout:2016: done (3 left) 2024-04-22T18:41:43.258 INFO:tasks.rados.rados.0.smithi089.stdout:2017: done (2 left) 2024-04-22T18:41:43.259 INFO:tasks.rados.rados.0.smithi089.stdout:2018: read oid 13 snap -1 2024-04-22T18:41:43.259 INFO:tasks.rados.rados.0.smithi089.stdout:2018: expect deleted 2024-04-22T18:41:43.259 INFO:tasks.rados.rados.0.smithi089.stdout:2019: tier_evict oid 10 2024-04-22T18:41:43.259 INFO:tasks.rados.rados.0.smithi089.stdout:2019: tier_evict oid 10 snap -1 2024-04-22T18:41:43.259 INFO:tasks.rados.rados.0.smithi089.stdout:2014: finishing write tid 2 to smithi08937929-42 2024-04-22T18:41:43.259 INFO:tasks.rados.rados.0.smithi089.stdout:2012: finishing write tid 2 to smithi08937929-15 2024-04-22T18:41:43.259 INFO:tasks.rados.rados.0.smithi089.stdout:2020: delete oid 17 current snap is 0 2024-04-22T18:41:43.259 INFO:tasks.rados.rados.0.smithi089.stdout:2012: finishing write tid 3 to smithi08937929-15 2024-04-22T18:41:43.259 INFO:tasks.rados.rados.0.smithi089.stdout:2012: finishing write tid 4 to smithi08937929-15 2024-04-22T18:41:43.259 INFO:tasks.rados.rados.0.smithi089.stdout:2012: finishing write tid 5 to smithi08937929-15 2024-04-22T18:41:43.259 INFO:tasks.rados.rados.0.smithi089.stdout:2012: oid 15 updating version 0 to 119 2024-04-22T18:41:43.259 INFO:tasks.rados.rados.0.smithi089.stdout:2012: oid 15 updating version 119 to 120 2024-04-22T18:41:43.259 INFO:tasks.rados.rados.0.smithi089.stdout:2012: oid 15 version 120 is already newer than 118 2024-04-22T18:41:43.260 INFO:tasks.rados.rados.0.smithi089.stdout:2012: oid 15 updating version 120 to 121 2024-04-22T18:41:43.260 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 15 v 121 (ObjNum 433 snap 0 seq_num 433) dirty exists 2024-04-22T18:41:43.260 INFO:tasks.rados.rados.0.smithi089.stdout:2012: left oid 15 (ObjNum 433 snap 0 seq_num 433) 2024-04-22T18:41:43.260 INFO:tasks.rados.rados.0.smithi089.stdout:2014: finishing write tid 3 to smithi08937929-42 2024-04-22T18:41:43.260 INFO:tasks.rados.rados.0.smithi089.stdout:2019: got (22) Invalid argument 2024-04-22T18:41:43.276 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:43.276 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:43.285 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:43.355 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:43.356+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:43.356 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:43.356+0000 7f3cfee88640 -1 osd.6 166 handle_osd_map injecting map crc failure 2024-04-22T18:41:43.360 INFO:tasks.rados.rados.0.smithi089.stdout:2012: done (4 left) 2024-04-22T18:41:43.360 INFO:tasks.rados.rados.0.smithi089.stdout:2018: done (3 left) 2024-04-22T18:41:43.360 INFO:tasks.rados.rados.0.smithi089.stdout:2019: done (2 left) 2024-04-22T18:41:43.360 INFO:tasks.rados.rados.0.smithi089.stdout:2020: done (1 left) 2024-04-22T18:41:43.360 INFO:tasks.rados.rados.0.smithi089.stdout:2021: write (excl) oid 39 current snap is 0 2024-04-22T18:41:43.360 INFO:tasks.rados.rados.0.smithi089.stdout:2021: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2024-04-22T18:41:43.367 INFO:tasks.rados.rados.0.smithi089.stdout:2021: writing smithi08937929-39 from 544659 to 1260593 tid 1 2024-04-22T18:41:43.372 INFO:tasks.rados.rados.0.smithi089.stdout:2021: writing smithi08937929-39 from 1738984 to 2188130 tid 2 2024-04-22T18:41:43.378 INFO:tasks.rados.rados.0.smithi089.stdout:2021: writing smithi08937929-39 from 2784436 to 3344659 tid 3 2024-04-22T18:41:43.378 INFO:tasks.rados.rados.0.smithi089.stdout:2022: read oid 33 snap -1 2024-04-22T18:41:43.379 INFO:tasks.rados.rados.0.smithi089.stdout:2022: expect (ObjNum 420 snap 0 seq_num 420) 2024-04-22T18:41:43.379 INFO:tasks.rados.rados.0.smithi089.stdout:2023: set_chunk oid 23 target oid 2024-04-22T18:41:43.428 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:43.428 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:43.437 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:43.456 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:43.457+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:43.558 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:43.559+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:43.581 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:43.582 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:43.590 INFO:tasks.rados.rados.0.smithi089.stdout:2023: set_chunk oid 23 offset: 259584 length: 1476608 target oid 913e5bb26ee45b02070be5e4a9d54e77ec0f4e6305e6ea3ffbef50dfc41dbed4 offset: 0 snap -1 2024-04-22T18:41:43.590 INFO:tasks.rados.rados.0.smithi089.stdout:2024: set_chunk oid 20 target oid 2024-04-22T18:41:43.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:43.659 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:43.660+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:43.697 INFO:tasks.rados.rados.0.smithi089.stdout:2024: set_chunk oid 20 offset: 670208 length: 945152 target oid 0f599a49a379afd4eecd00b1607c70dfbd6be1d4dccbc013d05b1ccdec8eb5f8 offset: 0 snap -1 2024-04-22T18:41:43.697 INFO:tasks.rados.rados.0.smithi089.stdout:2025: tier_promote oid 34 2024-04-22T18:41:43.697 INFO:tasks.rados.rados.0.smithi089.stdout:2014: finishing write tid 4 to smithi08937929-42 2024-04-22T18:41:43.697 INFO:tasks.rados.rados.0.smithi089.stdout:2014: finishing write tid 5 to smithi08937929-42 2024-04-22T18:41:43.697 INFO:tasks.rados.rados.0.smithi089.stdout:2014: oid 42 updating version 0 to 35 2024-04-22T18:41:43.697 INFO:tasks.rados.rados.0.smithi089.stdout:2014: oid 42 updating version 35 to 36 2024-04-22T18:41:43.697 INFO:tasks.rados.rados.0.smithi089.stdout:2014: oid 42 version 36 is already newer than 34 2024-04-22T18:41:43.697 INFO:tasks.rados.rados.0.smithi089.stdout:2014: oid 42 version 36 is already newer than 33 2024-04-22T18:41:43.697 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 42 v 36 (ObjNum 434 snap 0 seq_num 434) dirty exists 2024-04-22T18:41:43.698 INFO:tasks.rados.rados.0.smithi089.stdout:2014: left oid 42 (ObjNum 434 snap 0 seq_num 434) 2024-04-22T18:41:43.698 INFO:tasks.rados.rados.0.smithi089.stdout:2022: expect (ObjNum 420 snap 0 seq_num 420) 2024-04-22T18:41:43.701 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 7, 2, 4, 0] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:41:43.701 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:41:43.701 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-04-22T18:41:43.701 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:41:43.702 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:43.702 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:43.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:43.711 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:41:43.765 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:43.767+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:43.850 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:43.851+0000 7fa766d7d640 -1 osd.5 167 handle_osd_map injecting map crc failure 2024-04-22T18:41:43.854 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:43.854 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:43.860 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:43.861+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:43.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:43.961 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:43.962+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:43.964 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:43.964 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:43.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:43.974 INFO:tasks.rados.rados.0.smithi089.stdout:2021: finishing write tid 1 to smithi08937929-39 2024-04-22T18:41:43.974 INFO:tasks.rados.rados.0.smithi089.stdout:2021: finishing write tid 2 to smithi08937929-39 2024-04-22T18:41:43.975 INFO:tasks.rados.rados.0.smithi089.stdout:2021: finishing write tid 3 to smithi08937929-39 2024-04-22T18:41:43.975 INFO:tasks.rados.rados.0.smithi089.stdout:2021: finishing write tid 4 to smithi08937929-39 2024-04-22T18:41:43.975 INFO:tasks.rados.rados.0.smithi089.stdout:2021: finishing write tid 5 to smithi08937929-39 2024-04-22T18:41:43.975 INFO:tasks.rados.rados.0.smithi089.stdout:2021: oid 39 updating version 0 to 154 2024-04-22T18:41:43.975 INFO:tasks.rados.rados.0.smithi089.stdout:2021: oid 39 version 154 is already newer than 151 2024-04-22T18:41:43.975 INFO:tasks.rados.rados.0.smithi089.stdout:2021: oid 39 version 154 is already newer than 153 2024-04-22T18:41:43.975 INFO:tasks.rados.rados.0.smithi089.stdout:2021: oid 39 version 154 is already newer than 152 2024-04-22T18:41:43.975 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 39 v 154 (ObjNum 437 snap 0 seq_num 437) dirty exists 2024-04-22T18:41:43.975 INFO:tasks.rados.rados.0.smithi089.stdout:2021: left oid 39 (ObjNum 437 snap 0 seq_num 437) 2024-04-22T18:41:43.975 INFO:tasks.rados.rados.0.smithi089.stdout:2014: done (5 left) 2024-04-22T18:41:43.975 INFO:tasks.rados.rados.0.smithi089.stdout:2021: done (4 left) 2024-04-22T18:41:43.975 INFO:tasks.rados.rados.0.smithi089.stdout:2022: done (3 left) 2024-04-22T18:41:43.975 INFO:tasks.rados.rados.0.smithi089.stdout:2026: delete oid 16 current snap is 0 2024-04-22T18:41:43.975 INFO:tasks.rados.rados.0.smithi089.stdout:2024: finishing set_chunk to oid 20 2024-04-22T18:41:43.975 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 20 set_chunk returned error code -95 offset: 670208 length: 945152 2024-04-22T18:41:43.976 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 20 v 0 (ObjNum 418 snap 0 seq_num 418) dirty exists 2024-04-22T18:41:43.976 INFO:tasks.rados.rados.0.smithi089.stdout:2025: got (0) Success 2024-04-22T18:41:43.976 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 118 (ObjNum 417 snap 0 seq_num 417) dirty exists 2024-04-22T18:41:43.997 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:41:43.998 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:41:44.009 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-04-22T18:41:43.710286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":277467188,"num_objects":253,"num_object_clones":0,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":248,"num_whiteouts":0,"num_read":5183,"num_read_kb":4687732,"num_write":3038,"num_write_kb":733530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1244,"num_bytes_recovered":418812587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":32,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3178,"ondisk_log_size":3178,"up":98,"acting":98,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":607068,"kb_used_data":398440,"kb_used_omap":9,"kb_used_meta":202998,"kb_avail":565623972,"statfs":{"total":579820584960,"available":579198947328,"internally_reserved":0,"allocated":408002560,"data_stored":416560429,"data_compressed":14659377,"data_compressed_allocated":15638528,"data_compressed_original":31833266,"omap_allocated":9557,"internal_metadata":207870635},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8102320,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-33,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":180,"num_read_kb":185627,"num_write":116,"num_write_kb":25555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":20650021,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011467"},"pg_stats":[{"pgid":"4.23","version":"155'91","reported_seq":372,"reported_epoch":166,"state":"remapped+peering","last_fresh":"2024-04-22T18:41:41.761095+0000","last_change":"2024-04-22T18:41:41.760737+0000","last_active":"2024-04-22T18:41:39.865730+0000","last_peered":"2024-04-22T18:41:39.761838+0000","last_clean":"2024-04-22T18:41:08.944978+0000","last_became_active":"2024-04-22T18:41:27.959136+0000","last_became_peered":"2024-04-22T18:41:27.959136+0000","last_unstale":"2024-04-22T18:41:41.761095+0000","last_undegraded":"2024-04-22T18:41:41.761095+0000","last_fullsized":"2024-04-22T18:41:41.761095+0000","mapping_epoch":166,"log_start":"137'90","ondisk_log_start":"137'90","created":164,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":6,"last_scrub":"137'90","last_scrub_stamp":"2024-04-22T18:41:22.057812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:22.057812+0000","objects_scrubbed":2,"log_size":1,"log_dups_size":61,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655556,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":43,"num_read_kb":57909,"num_write":11,"num_write_kb":2755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3906036,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"149'42","reported_seq":348,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-04-22T18:41:41.661089+0000","last_change":"2024-04-22T18:41:40.908387+0000","last_active":"2024-04-22T18:41:41.661089+0000","last_peered":"2024-04-22T18:41:41.661089+0000","last_clean":"2024-04-22T18:41:41.661089+0000","last_became_active":"2024-04-22T18:41:40.908069+0000","last_became_peered":"2024-04-22T18:41:40.908069+0000","last_unstale":"2024-04-22T18:41:41.661089+0000","last_undegraded":"2024-04-22T18:41:41.661089+0000","last_fullsized":"2024-04-22T18:41:41.661089+0000","mapping_epoch":164,"log_start":"64'38","ondisk_log_start":"64'38","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":25,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":88762,"num_write":13,"num_write_kb":3449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5687418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"155'190","reported_seq":726,"reported_epoch":165,"state":"unknown","last_fresh":"2024-04-22T18:41:40.705170+0000","last_change":"2024-04-22T18:41:41.582678+0000","last_active":"2024-04-22T18:41:39.848319+0000","last_peered":"2024-04-22T18:41:39.289418+0000","last_clean":"2024-04-22T18:41:25.202910+0000","last_became_active":"2024-04-22T18:41:27.863994+0000","last_became_peered":"2024-04-22T18:41:27.863994+0000","last_unstale":"2024-04-22T18:41:40.705170+0000","last_undegraded":"2024-04-22T18:41:40.705170+0000","last_fullsized":"2024-04-22T18:41:40.705170+0000","mapping_epoch":164,"log_start":"147'185","ondisk_log_start":"147'185","created":164,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":6,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":5,"log_dups_size":161,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7042282,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":229510,"num_write":43,"num_write_kb":12083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":21653811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"162'34","reported_seq":289,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-04-22T18:41:41.832629+0000","last_change":"2024-04-22T18:41:41.107163+0000","last_active":"2024-04-22T18:41:41.832629+0000","last_peered":"2024-04-22T18:41:41.832629+0000","last_clean":"2024-04-22T18:41:41.832629+0000","last_became_active":"2024-04-22T18:41:41.106971+0000","last_became_peered":"2024-04-22T18:41:41.106971+0000","last_unstale":"2024-04-22T18:41:41.832629+0000","last_undegraded":"2024-04-22T18:41:41.832629+0000","last_fullsized":"2024-04-22T18:41:41.832629+0000","mapping_epoch":164,"log_start":"147'30","ondisk_log_start":"147'30","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'13","last_scrub_stamp":"2024-04-22T18:39:54.377980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.377980+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":30,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30260881099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1512663,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":68466,"num_write":12,"num_write_kb":3428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"163'88","reported_seq":398,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-04-22T18:41:41.661299+0000","last_change":"2024-04-22T18:41:41.008619+0000","last_active":"2024-04-22T18:41:41.661299+0000","last_peered":"2024-04-22T18:41:41.661299+0000","last_clean":"2024-04-22T18:41:41.661299+0000","last_became_active":"2024-04-22T18:41:41.008273+0000","last_became_peered":"2024-04-22T18:41:41.008273+0000","last_unstale":"2024-04-22T18:41:41.661299+0000","last_undegraded":"2024-04-22T18:41:41.661299+0000","last_fullsized":"2024-04-22T18:41:41.661299+0000","mapping_epoch":164,"log_start":"155'85","ondisk_log_start":"155'85","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":89,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2714397,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":136200,"num_write":47,"num_write_kb":12258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"147'53","reported_seq":75,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.749200+0000","last_change":"2024-04-22T18:41:11.253830+0000","last_active":"2024-04-22T18:41:39.749200+0000","last_peered":"2024-04-22T18:41:39.749200+0000","last_clean":"2024-04-22T18:41:39.749200+0000","last_became_active":"2024-04-22T18:41:11.253471+0000","last_became_peered":"2024-04-22T18:41:11.253471+0000","last_unstale":"2024-04-22T18:41:39.749200+0000","last_undegraded":"2024-04-22T18:41:39.749200+0000","last_fullsized":"2024-04-22T18:41:39.749200+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'23","last_scrub_stamp":"2024-04-22T18:40:20.587606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.587606+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10225366599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1840449,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":93930,"num_write":31,"num_write_kb":9939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5574713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"164'84","reported_seq":100,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.277473+0000","last_change":"2024-04-22T18:41:11.355276+0000","last_active":"2024-04-22T18:41:40.277473+0000","last_peered":"2024-04-22T18:41:40.277473+0000","last_clean":"2024-04-22T18:41:40.277473+0000","last_became_active":"2024-04-22T18:41:11.354958+0000","last_became_peered":"2024-04-22T18:41:11.354958+0000","last_unstale":"2024-04-22T18:41:40.277473+0000","last_undegraded":"2024-04-22T18:41:40.277473+0000","last_fullsized":"2024-04-22T18:41:40.277473+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"96'46","last_scrub_stamp":"2024-04-22T18:40:34.755106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:34.755106+0000","objects_scrubbed":2,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40192567099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2411904,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":137,"num_read_kb":145625,"num_write":53,"num_write_kb":15309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8192627,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"132'123","reported_seq":485,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.749134+0000","last_change":"2024-04-22T18:41:11.469596+0000","last_active":"2024-04-22T18:41:39.749134+0000","last_peered":"2024-04-22T18:41:39.749134+0000","last_clean":"2024-04-22T18:41:39.749134+0000","last_became_active":"2024-04-22T18:41:11.469282+0000","last_became_peered":"2024-04-22T18:41:11.469282+0000","last_unstale":"2024-04-22T18:41:39.749134+0000","last_undegraded":"2024-04-22T18:41:39.749134+0000","last_fullsized":"2024-04-22T18:41:39.749134+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"121'119","last_scrub_stamp":"2024-04-22T18:40:55.573199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.573199+0000","objects_scrubbed":4,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32976813900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4521133,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":219,"num_read_kb":257792,"num_write":64,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3786236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"160'158","reported_seq":608,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.805074+0000","last_change":"2024-04-22T18:41:30.033168+0000","last_active":"2024-04-22T18:41:39.805074+0000","last_peered":"2024-04-22T18:41:39.805074+0000","last_clean":"2024-04-22T18:41:39.805074+0000","last_became_active":"2024-04-22T18:41:29.932575+0000","last_became_peered":"2024-04-22T18:41:29.932575+0000","last_unstale":"2024-04-22T18:41:39.805074+0000","last_undegraded":"2024-04-22T18:41:39.805074+0000","last_fullsized":"2024-04-22T18:41:39.805074+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":5,"last_scrub":"85'109","last_scrub_stamp":"2024-04-22T18:40:17.658363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:17.658363+0000","objects_scrubbed":5,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202176033,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4706026,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":305,"num_read_kb":338789,"num_write":99,"num_write_kb":27974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9711724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"163'115","reported_seq":438,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.706890+0000","last_change":"2024-04-22T18:41:10.956373+0000","last_active":"2024-04-22T18:41:40.706890+0000","last_peered":"2024-04-22T18:41:40.706890+0000","last_clean":"2024-04-22T18:41:40.706890+0000","last_became_active":"2024-04-22T18:41:10.956174+0000","last_became_peered":"2024-04-22T18:41:10.956174+0000","last_unstale":"2024-04-22T18:41:40.706890+0000","last_undegraded":"2024-04-22T18:41:40.706890+0000","last_fullsized":"2024-04-22T18:41:40.706890+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2806852,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":163,"num_read_kb":197186,"num_write":61,"num_write_kb":16155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4554596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"159'118","reported_seq":567,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.850311+0000","last_change":"2024-04-22T18:41:18.958159+0000","last_active":"2024-04-22T18:41:39.850311+0000","last_peered":"2024-04-22T18:41:39.850311+0000","last_clean":"2024-04-22T18:41:39.850311+0000","last_became_active":"2024-04-22T18:41:18.957008+0000","last_became_peered":"2024-04-22T18:41:18.957008+0000","last_unstale":"2024-04-22T18:41:39.850311+0000","last_undegraded":"2024-04-22T18:41:39.850311+0000","last_fullsized":"2024-04-22T18:41:39.850311+0000","mapping_epoch":142,"log_start":"84'88","ondisk_log_start":"84'88","created":110,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":30,"log_dups_size":106,"ondisk_log_size":30,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7033845,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":149003,"num_write":24,"num_write_kb":7118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9095252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"148'71","reported_seq":392,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.700460+0000","last_change":"2024-04-22T18:41:37.761338+0000","last_active":"2024-04-22T18:41:39.700460+0000","last_peered":"2024-04-22T18:41:39.700460+0000","last_clean":"2024-04-22T18:41:39.700460+0000","last_became_active":"2024-04-22T18:41:37.760987+0000","last_became_peered":"2024-04-22T18:41:37.760987+0000","last_unstale":"2024-04-22T18:41:39.700460+0000","last_undegraded":"2024-04-22T18:41:39.700460+0000","last_fullsized":"2024-04-22T18:41:39.700460+0000","mapping_epoch":161,"log_start":"79'52","ondisk_log_start":"79'52","created":110,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":61,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":857223,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":130,"num_read_kb":179919,"num_write":60,"num_write_kb":17002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9662508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"107'52","reported_seq":385,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.704755+0000","last_change":"2024-04-22T18:41:22.602374+0000","last_active":"2024-04-22T18:41:40.704755+0000","last_peered":"2024-04-22T18:41:40.704755+0000","last_clean":"2024-04-22T18:41:40.704755+0000","last_became_active":"2024-04-22T18:41:00.078669+0000","last_became_peered":"2024-04-22T18:41:00.078669+0000","last_unstale":"2024-04-22T18:41:40.704755+0000","last_undegraded":"2024-04-22T18:41:40.704755+0000","last_fullsized":"2024-04-22T18:41:40.704755+0000","mapping_epoch":125,"log_start":"83'49","ondisk_log_start":"83'49","created":110,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":5,"last_scrub":"107'52","last_scrub_stamp":"2024-04-22T18:41:22.602272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:22.602272+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:48.741276+0000","scrub_duration":0.40203006699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"147'96","reported_seq":474,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.804816+0000","last_change":"2024-04-22T18:41:30.012836+0000","last_active":"2024-04-22T18:41:39.804816+0000","last_peered":"2024-04-22T18:41:39.804816+0000","last_clean":"2024-04-22T18:41:39.804816+0000","last_became_active":"2024-04-22T18:41:30.012572+0000","last_became_peered":"2024-04-22T18:41:30.012572+0000","last_unstale":"2024-04-22T18:41:39.804816+0000","last_undegraded":"2024-04-22T18:41:39.804816+0000","last_fullsized":"2024-04-22T18:41:39.804816+0000","mapping_epoch":153,"log_start":"96'71","ondisk_log_start":"96'71","created":110,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":81,"ondisk_log_size":25,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.22461577699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5428711,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":132149,"num_write":27,"num_write_kb":8334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3646149,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"54'29","reported_seq":106,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.700380+0000","last_change":"2024-04-22T18:40:55.328642+0000","last_active":"2024-04-22T18:41:39.700380+0000","last_peered":"2024-04-22T18:41:39.700380+0000","last_clean":"2024-04-22T18:41:39.700380+0000","last_became_active":"2024-04-22T18:40:13.180144+0000","last_became_peered":"2024-04-22T18:40:13.180144+0000","last_unstale":"2024-04-22T18:41:39.700380+0000","last_undegraded":"2024-04-22T18:41:39.700380+0000","last_fullsized":"2024-04-22T18:41:39.700380+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'29","last_scrub_stamp":"2024-04-22T18:40:55.328529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.328529+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:10.967288+0000","scrub_duration":0.30217712000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"51'19","reported_seq":107,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.804803+0000","last_change":"2024-04-22T18:40:57.754351+0000","last_active":"2024-04-22T18:41:40.804803+0000","last_peered":"2024-04-22T18:41:40.804803+0000","last_clean":"2024-04-22T18:41:40.804803+0000","last_became_active":"2024-04-22T18:40:10.978840+0000","last_became_peered":"2024-04-22T18:40:10.978840+0000","last_unstale":"2024-04-22T18:41:40.804803+0000","last_undegraded":"2024-04-22T18:41:40.804803+0000","last_fullsized":"2024-04-22T18:41:40.804803+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"51'19","last_scrub_stamp":"2024-04-22T18:40:57.754261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:57.754261+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:17.927283+0000","scrub_duration":0.59184710600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"163'92","reported_seq":76,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.849141+0000","last_change":"2024-04-22T18:41:04.562945+0000","last_active":"2024-04-22T18:41:39.849141+0000","last_peered":"2024-04-22T18:41:39.849141+0000","last_clean":"2024-04-22T18:41:39.849141+0000","last_became_active":"2024-04-22T18:41:04.139159+0000","last_became_peered":"2024-04-22T18:41:04.139159+0000","last_unstale":"2024-04-22T18:41:39.849141+0000","last_undegraded":"2024-04-22T18:41:39.849141+0000","last_fullsized":"2024-04-22T18:41:39.849141+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'75","last_scrub_stamp":"2024-04-22T18:40:22.670296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.670296+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:49.126896+0000","scrub_duration":0.20951747300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218722,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":82401,"num_write":16,"num_write_kb":7324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12459808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"140'46","reported_seq":291,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.706978+0000","last_change":"2024-04-22T18:40:59.453830+0000","last_active":"2024-04-22T18:41:40.706978+0000","last_peered":"2024-04-22T18:41:40.706978+0000","last_clean":"2024-04-22T18:41:40.706978+0000","last_became_active":"2024-04-22T18:40:58.107873+0000","last_became_peered":"2024-04-22T18:40:58.107873+0000","last_unstale":"2024-04-22T18:41:40.706978+0000","last_undegraded":"2024-04-22T18:41:40.706978+0000","last_fullsized":"2024-04-22T18:41:40.706978+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361351,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":113,"num_read_kb":106978,"num_write":27,"num_write_kb":7134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2641926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"153'152","reported_seq":494,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.706367+0000","last_change":"2024-04-22T18:40:26.844287+0000","last_active":"2024-04-22T18:41:40.706367+0000","last_peered":"2024-04-22T18:41:40.706367+0000","last_clean":"2024-04-22T18:41:40.706367+0000","last_became_active":"2024-04-22T18:40:26.742351+0000","last_became_peered":"2024-04-22T18:40:26.742351+0000","last_unstale":"2024-04-22T18:41:40.706367+0000","last_undegraded":"2024-04-22T18:41:40.706367+0000","last_fullsized":"2024-04-22T18:41:40.706367+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'113","last_scrub_stamp":"2024-04-22T18:40:20.291988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.291988+0000","objects_scrubbed":2,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10270805500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4958197,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":29831,"num_write":47,"num_write_kb":13258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3940523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"31'5","reported_seq":200,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:41.145693+0000","last_change":"2024-04-22T18:41:41.832636+0000","last_active":"2024-04-22T18:41:41.145693+0000","last_peered":"2024-04-22T18:41:41.145693+0000","last_clean":"2024-04-22T18:41:41.145693+0000","last_became_active":"2024-04-22T18:39:59.265189+0000","last_became_peered":"2024-04-22T18:39:59.265189+0000","last_unstale":"2024-04-22T18:41:41.145693+0000","last_undegraded":"2024-04-22T18:41:41.145693+0000","last_fullsized":"2024-04-22T18:41:41.145693+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":5,"last_scrub":"31'5","last_scrub_stamp":"2024-04-22T18:40:10.212815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.212815+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015068670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.700425+0000","last_change":"2024-04-22T18:40:58.207745+0000","last_active":"2024-04-22T18:41:39.700425+0000","last_peered":"2024-04-22T18:41:39.700425+0000","last_clean":"2024-04-22T18:41:39.700425+0000","last_became_active":"2024-04-22T18:40:58.207451+0000","last_became_peered":"2024-04-22T18:40:58.207451+0000","last_unstale":"2024-04-22T18:41:39.700425+0000","last_undegraded":"2024-04-22T18:41:39.700425+0000","last_fullsized":"2024-04-22T18:41:39.700425+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:19.136933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:19.136933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"155'190","reported_seq":736,"reported_epoch":165,"state":"active+remapped+backfilling","last_fresh":"2024-04-22T18:41:41.008142+0000","last_change":"2024-04-22T18:41:41.108897+0000","last_active":"2024-04-22T18:41:41.008142+0000","last_peered":"2024-04-22T18:41:41.008142+0000","last_clean":"2024-04-22T18:41:25.202910+0000","last_became_active":"2024-04-22T18:41:41.007236+0000","last_became_peered":"2024-04-22T18:41:41.007236+0000","last_unstale":"2024-04-22T18:41:41.008142+0000","last_undegraded":"2024-04-22T18:41:41.008142+0000","last_fullsized":"2024-04-22T18:41:41.008142+0000","mapping_epoch":151,"log_start":"147'185","ondisk_log_start":"147'185","created":20,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":5,"log_dups_size":161,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7042281,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":189,"num_read_kb":229897,"num_write":43,"num_write_kb":12082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":21653811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":177,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.706852+0000","last_change":"2024-04-22T18:41:33.768149+0000","last_active":"2024-04-22T18:41:40.706852+0000","last_peered":"2024-04-22T18:41:40.706852+0000","last_clean":"2024-04-22T18:41:40.706852+0000","last_became_active":"2024-04-22T18:41:04.037248+0000","last_became_peered":"2024-04-22T18:41:04.037248+0000","last_unstale":"2024-04-22T18:41:40.706852+0000","last_undegraded":"2024-04-22T18:41:40.706852+0000","last_fullsized":"2024-04-22T18:41:40.706852+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"162'34","reported_seq":290,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:41.006962+0000","last_change":"2024-04-22T18:41:41.006485+0000","last_active":"2024-04-22T18:41:41.006962+0000","last_peered":"2024-04-22T18:41:41.006962+0000","last_clean":"2024-04-22T18:41:41.006962+0000","last_became_active":"2024-04-22T18:41:41.006293+0000","last_became_peered":"2024-04-22T18:41:41.006293+0000","last_unstale":"2024-04-22T18:41:41.006962+0000","last_undegraded":"2024-04-22T18:41:41.006962+0000","last_fullsized":"2024-04-22T18:41:41.006962+0000","mapping_epoch":20,"log_start":"147'30","ondisk_log_start":"147'30","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'13","last_scrub_stamp":"2024-04-22T18:39:54.377980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.377980+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":30,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30260881099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":68466,"num_write":11,"num_write_kb":3428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":191,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.700423+0000","last_change":"2024-04-22T18:41:26.962055+0000","last_active":"2024-04-22T18:41:39.700423+0000","last_peered":"2024-04-22T18:41:39.700423+0000","last_clean":"2024-04-22T18:41:39.700423+0000","last_became_active":"2024-04-22T18:41:26.961835+0000","last_became_peered":"2024-04-22T18:41:26.961835+0000","last_unstale":"2024-04-22T18:41:39.700423+0000","last_undegraded":"2024-04-22T18:41:39.700423+0000","last_fullsized":"2024-04-22T18:41:39.700423+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:09.538542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:09.538542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.35201753200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"155'91","reported_seq":360,"reported_epoch":160,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:41:36.147158+0000","last_change":"2024-04-22T18:41:28.162984+0000","last_active":"2024-04-22T18:41:36.147158+0000","last_peered":"2024-04-22T18:41:36.147158+0000","last_clean":"2024-04-22T18:41:08.944978+0000","last_became_active":"2024-04-22T18:41:27.959136+0000","last_became_peered":"2024-04-22T18:41:27.959136+0000","last_unstale":"2024-04-22T18:41:36.147158+0000","last_undegraded":"2024-04-22T18:41:36.147158+0000","last_fullsized":"2024-04-22T18:41:36.147158+0000","mapping_epoch":151,"log_start":"137'90","ondisk_log_start":"137'90","created":20,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'90","last_scrub_stamp":"2024-04-22T18:41:22.057812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:22.057812+0000","objects_scrubbed":2,"log_size":1,"log_dups_size":61,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5311111,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":110041,"num_write":21,"num_write_kb":5510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7812071,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":175,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.807126+0000","last_change":"2024-04-22T18:41:26.854653+0000","last_active":"2024-04-22T18:41:40.807126+0000","last_peered":"2024-04-22T18:41:40.807126+0000","last_clean":"2024-04-22T18:41:40.807126+0000","last_became_active":"2024-04-22T18:41:26.854399+0000","last_became_peered":"2024-04-22T18:41:26.854399+0000","last_unstale":"2024-04-22T18:41:40.807126+0000","last_undegraded":"2024-04-22T18:41:40.807126+0000","last_fullsized":"2024-04-22T18:41:40.807126+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"149'42","reported_seq":349,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.908202+0000","last_change":"2024-04-22T18:41:40.807780+0000","last_active":"2024-04-22T18:41:40.908202+0000","last_peered":"2024-04-22T18:41:40.908202+0000","last_clean":"2024-04-22T18:41:40.908202+0000","last_became_active":"2024-04-22T18:41:40.807410+0000","last_became_peered":"2024-04-22T18:41:40.807410+0000","last_unstale":"2024-04-22T18:41:40.908202+0000","last_undegraded":"2024-04-22T18:41:40.908202+0000","last_fullsized":"2024-04-22T18:41:40.908202+0000","mapping_epoch":129,"log_start":"64'38","ondisk_log_start":"64'38","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":25,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":88762,"num_write":12,"num_write_kb":3449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5687418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"164'376","reported_seq":1496,"reported_epoch":165,"state":"active+recovering","last_fresh":"2024-04-22T18:41:41.208815+0000","last_change":"2024-04-22T18:41:27.461110+0000","last_active":"2024-04-22T18:41:41.208815+0000","last_peered":"2024-04-22T18:41:41.208815+0000","last_clean":"2024-04-22T18:41:25.657445+0000","last_became_active":"2024-04-22T18:41:26.994048+0000","last_became_peered":"2024-04-22T18:41:26.994048+0000","last_unstale":"2024-04-22T18:41:41.208815+0000","last_undegraded":"2024-04-22T18:41:41.208815+0000","last_fullsized":"2024-04-22T18:41:41.208815+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'293","last_scrub_stamp":"2024-04-22T18:40:53.582354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:53.582354+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:14.875876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43365642,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":319,"num_read_kb":4191,"num_write":468,"num_write_kb":112186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":441,"num_bytes_recovered":78594799,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"0,4","objects":52},{"shards":"1,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":136,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.707146+0000","last_change":"2024-04-22T18:41:26.657445+0000","last_active":"2024-04-22T18:41:40.707146+0000","last_peered":"2024-04-22T18:41:40.707146+0000","last_clean":"2024-04-22T18:41:40.707146+0000","last_became_active":"2024-04-22T18:41:26.657206+0000","last_became_peered":"2024-04-22T18:41:26.657206+0000","last_unstale":"2024-04-22T18:41:40.707146+0000","last_undegraded":"2024-04-22T18:41:40.707146+0000","last_fullsized":"2024-04-22T18:41:40.707146+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:55.558446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.558446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:00.817184+0000","scrub_duration":0.56126016300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"115'32","reported_seq":295,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.706989+0000","last_change":"2024-04-22T18:40:21.426510+0000","last_active":"2024-04-22T18:41:40.706989+0000","last_peered":"2024-04-22T18:41:40.706989+0000","last_clean":"2024-04-22T18:41:40.706989+0000","last_became_active":"2024-04-22T18:39:51.943071+0000","last_became_peered":"2024-04-22T18:39:51.943071+0000","last_unstale":"2024-04-22T18:41:40.706989+0000","last_undegraded":"2024-04-22T18:41:40.706989+0000","last_fullsized":"2024-04-22T18:41:40.706989+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-04-22T18:40:21.426350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.426350+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30334366400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2605118,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":99,"num_read_kb":108445,"num_write":4,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"164'300","reported_seq":690,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.806430+0000","last_change":"2024-04-22T18:40:20.465005+0000","last_active":"2024-04-22T18:41:40.806430+0000","last_peered":"2024-04-22T18:41:40.806430+0000","last_clean":"2024-04-22T18:41:40.806430+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:41:40.806430+0000","last_undegraded":"2024-04-22T18:41:40.806430+0000","last_fullsized":"2024-04-22T18:41:40.806430+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'176","last_scrub_stamp":"2024-04-22T18:40:20.464799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.464799+0000","objects_scrubbed":31,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:45.914514+0000","scrub_duration":0.30629975799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32121166,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":264,"num_read_kb":4949,"num_write":392,"num_write_kb":78798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":123,"reported_epoch":165,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T18:41:40.729112+0000","last_change":"2024-04-22T18:41:41.219455+0000","last_active":"2024-04-22T18:41:40.729112+0000","last_peered":"2024-04-22T18:41:40.729112+0000","last_clean":"2024-04-22T18:41:40.729112+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:41:40.729112+0000","last_undegraded":"2024-04-22T18:41:40.729112+0000","last_fullsized":"2024-04-22T18:41:40.729112+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:10.403189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.403189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.201433005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"162'72","reported_seq":424,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.807901+0000","last_change":"2024-04-22T18:41:32.860808+0000","last_active":"2024-04-22T18:41:40.807901+0000","last_peered":"2024-04-22T18:41:40.807901+0000","last_clean":"2024-04-22T18:41:40.807901+0000","last_became_active":"2024-04-22T18:41:32.860455+0000","last_became_peered":"2024-04-22T18:41:32.860455+0000","last_unstale":"2024-04-22T18:41:40.807901+0000","last_undegraded":"2024-04-22T18:41:40.807901+0000","last_fullsized":"2024-04-22T18:41:40.807901+0000","mapping_epoch":156,"log_start":"83'49","ondisk_log_start":"83'49","created":20,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":49,"ondisk_log_size":23,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3008494,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":94905,"num_write":38,"num_write_kb":11286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4519285,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"165'328","reported_seq":1334,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:41.509832+0000","last_change":"2024-04-22T18:41:41.108768+0000","last_active":"2024-04-22T18:41:41.509832+0000","last_peered":"2024-04-22T18:41:41.509832+0000","last_clean":"2024-04-22T18:41:41.509832+0000","last_became_active":"2024-04-22T18:41:41.007340+0000","last_became_peered":"2024-04-22T18:41:41.007340+0000","last_unstale":"2024-04-22T18:41:41.509832+0000","last_undegraded":"2024-04-22T18:41:41.509832+0000","last_fullsized":"2024-04-22T18:41:41.509832+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'209","last_scrub_stamp":"2024-04-22T18:40:18.683442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:18.683442+0000","objects_scrubbed":24,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:45.950082+0000","scrub_duration":0.204344365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37262949,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":285,"num_read_kb":9112,"num_write":405,"num_write_kb":103244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":67954083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":167,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.800703+0000","last_change":"2024-04-22T18:40:22.521828+0000","last_active":"2024-04-22T18:41:39.800703+0000","last_peered":"2024-04-22T18:41:39.800703+0000","last_clean":"2024-04-22T18:41:39.800703+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:41:39.800703+0000","last_undegraded":"2024-04-22T18:41:39.800703+0000","last_fullsized":"2024-04-22T18:41:39.800703+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.521638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:43.407910+0000","scrub_duration":0.30190717299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"163'107","reported_seq":475,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.804844+0000","last_change":"2024-04-22T18:41:32.061113+0000","last_active":"2024-04-22T18:41:39.804844+0000","last_peered":"2024-04-22T18:41:39.804844+0000","last_clean":"2024-04-22T18:41:39.804844+0000","last_became_active":"2024-04-22T18:41:31.858774+0000","last_became_peered":"2024-04-22T18:41:31.858774+0000","last_unstale":"2024-04-22T18:41:39.804844+0000","last_undegraded":"2024-04-22T18:41:39.804844+0000","last_fullsized":"2024-04-22T18:41:39.804844+0000","mapping_epoch":155,"log_start":"96'71","ondisk_log_start":"96'71","created":20,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":81,"ondisk_log_size":36,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.22461577699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5210302,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":109,"num_read_kb":139617,"num_write":33,"num_write_kb":10403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1133573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"163'322","reported_seq":1000,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.047075+0000","last_change":"2024-04-22T18:41:19.866530+0000","last_active":"2024-04-22T18:41:40.047075+0000","last_peered":"2024-04-22T18:41:40.047075+0000","last_clean":"2024-04-22T18:41:40.047075+0000","last_became_active":"2024-04-22T18:41:19.765392+0000","last_became_peered":"2024-04-22T18:41:19.765392+0000","last_unstale":"2024-04-22T18:41:40.047075+0000","last_undegraded":"2024-04-22T18:41:40.047075+0000","last_fullsized":"2024-04-22T18:41:40.047075+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'212","last_scrub_stamp":"2024-04-22T18:40:21.870855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.870855+0000","objects_scrubbed":46,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45114409,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":297,"num_read_kb":12415,"num_write":409,"num_write_kb":89641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":43826500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":188,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.060743+0000","last_change":"2024-04-22T18:41:30.504800+0000","last_active":"2024-04-22T18:41:40.060743+0000","last_peered":"2024-04-22T18:41:40.060743+0000","last_clean":"2024-04-22T18:41:40.060743+0000","last_became_active":"2024-04-22T18:40:02.741862+0000","last_became_peered":"2024-04-22T18:40:02.741862+0000","last_unstale":"2024-04-22T18:41:40.060743+0000","last_undegraded":"2024-04-22T18:41:40.060743+0000","last_fullsized":"2024-04-22T18:41:40.060743+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:30.504686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:30.504686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:57.436870+0000","scrub_duration":0.30171718199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":362,"reported_epoch":165,"state":"active+clean+scrubbing","last_fresh":"2024-04-22T18:41:40.805446+0000","last_change":"2024-04-22T18:41:40.874959+0000","last_active":"2024-04-22T18:41:40.805446+0000","last_peered":"2024-04-22T18:41:40.805446+0000","last_clean":"2024-04-22T18:41:40.805446+0000","last_became_active":"2024-04-22T18:41:26.757958+0000","last_became_peered":"2024-04-22T18:41:26.757958+0000","last_unstale":"2024-04-22T18:41:40.805446+0000","last_undegraded":"2024-04-22T18:41:40.805446+0000","last_fullsized":"2024-04-22T18:41:40.805446+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:39:53.346787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.346787+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20273052499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"137'75","reported_seq":85,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.764221+0000","last_change":"2024-04-22T18:41:04.681901+0000","last_active":"2024-04-22T18:41:39.764221+0000","last_peered":"2024-04-22T18:41:39.764221+0000","last_clean":"2024-04-22T18:41:39.764221+0000","last_became_active":"2024-04-22T18:41:04.227104+0000","last_became_peered":"2024-04-22T18:41:04.227104+0000","last_unstale":"2024-04-22T18:41:39.764221+0000","last_undegraded":"2024-04-22T18:41:39.764221+0000","last_fullsized":"2024-04-22T18:41:39.764221+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'38","last_scrub_stamp":"2024-04-22T18:39:55.746067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:55.746067+0000","objects_scrubbed":1,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60877772299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6168268,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":192624,"num_write":57,"num_write_kb":16481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15782313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"164'59","reported_seq":368,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.167424+0000","last_change":"2024-04-22T18:41:35.835744+0000","last_active":"2024-04-22T18:41:40.167424+0000","last_peered":"2024-04-22T18:41:40.167424+0000","last_clean":"2024-04-22T18:41:40.167424+0000","last_became_active":"2024-04-22T18:41:35.834928+0000","last_became_peered":"2024-04-22T18:41:35.834928+0000","last_unstale":"2024-04-22T18:41:40.167424+0000","last_undegraded":"2024-04-22T18:41:40.167424+0000","last_fullsized":"2024-04-22T18:41:40.167424+0000","mapping_epoch":159,"log_start":"79'52","ondisk_log_start":"79'52","created":20,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'57","last_scrub_stamp":"2024-04-22T18:40:56.376497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:56.376497+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":61,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:57.469237+0000","scrub_duration":0.30191299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3832029,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":30145,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1913105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"164'150","reported_seq":605,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.181743+0000","last_change":"2024-04-22T18:41:17.760985+0000","last_active":"2024-04-22T18:41:40.181743+0000","last_peered":"2024-04-22T18:41:40.181743+0000","last_clean":"2024-04-22T18:41:40.181743+0000","last_became_active":"2024-04-22T18:41:17.759812+0000","last_became_peered":"2024-04-22T18:41:17.759812+0000","last_unstale":"2024-04-22T18:41:40.181743+0000","last_undegraded":"2024-04-22T18:41:40.181743+0000","last_fullsized":"2024-04-22T18:41:40.181743+0000","mapping_epoch":141,"log_start":"84'88","ondisk_log_start":"84'88","created":20,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":62,"log_dups_size":106,"ondisk_log_size":62,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6512894,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":148629,"num_write":58,"num_write_kb":19648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12038689,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"161'117","reported_seq":448,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.707064+0000","last_change":"2024-04-22T18:41:11.054130+0000","last_active":"2024-04-22T18:41:40.707064+0000","last_peered":"2024-04-22T18:41:40.707064+0000","last_clean":"2024-04-22T18:41:40.707064+0000","last_became_active":"2024-04-22T18:41:11.053760+0000","last_became_peered":"2024-04-22T18:41:11.053760+0000","last_unstale":"2024-04-22T18:41:40.707064+0000","last_undegraded":"2024-04-22T18:41:40.707064+0000","last_fullsized":"2024-04-22T18:41:40.707064+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6117830,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":181,"num_read_kb":210385,"num_write":60,"num_write_kb":15942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4554596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"163'152","reported_seq":600,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:40.006771+0000","last_change":"2024-04-22T18:41:31.959627+0000","last_active":"2024-04-22T18:41:40.006771+0000","last_peered":"2024-04-22T18:41:40.006771+0000","last_clean":"2024-04-22T18:41:40.006771+0000","last_became_active":"2024-04-22T18:41:31.959307+0000","last_became_peered":"2024-04-22T18:41:31.959307+0000","last_unstale":"2024-04-22T18:41:40.006771+0000","last_undegraded":"2024-04-22T18:41:40.006771+0000","last_fullsized":"2024-04-22T18:41:40.006771+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'109","last_scrub_stamp":"2024-04-22T18:40:17.658363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:17.658363+0000","objects_scrubbed":5,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202176033,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7580089,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":296,"num_read_kb":338551,"num_write":90,"num_write_kb":26612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":11460299,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"160'127","reported_seq":523,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.749170+0000","last_change":"2024-04-22T18:41:12.783644+0000","last_active":"2024-04-22T18:41:39.749170+0000","last_peered":"2024-04-22T18:41:39.749170+0000","last_clean":"2024-04-22T18:41:39.749170+0000","last_became_active":"2024-04-22T18:41:11.956312+0000","last_became_peered":"2024-04-22T18:41:11.956312+0000","last_unstale":"2024-04-22T18:41:39.749170+0000","last_undegraded":"2024-04-22T18:41:39.749170+0000","last_fullsized":"2024-04-22T18:41:39.749170+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'119","last_scrub_stamp":"2024-04-22T18:40:55.573199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.573199+0000","objects_scrubbed":4,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32976813900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5512745,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":293,"num_read_kb":381546,"num_write":69,"num_write_kb":18972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9347897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"133'66","reported_seq":73,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.849337+0000","last_change":"2024-04-22T18:41:19.456295+0000","last_active":"2024-04-22T18:41:39.849337+0000","last_peered":"2024-04-22T18:41:39.849337+0000","last_clean":"2024-04-22T18:41:39.849337+0000","last_became_active":"2024-04-22T18:41:11.154301+0000","last_became_peered":"2024-04-22T18:41:11.154301+0000","last_unstale":"2024-04-22T18:41:39.849337+0000","last_undegraded":"2024-04-22T18:41:39.849337+0000","last_fullsized":"2024-04-22T18:41:39.849337+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'66","last_scrub_stamp":"2024-04-22T18:41:19.456160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:19.456160+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:36.925204+0000","scrub_duration":0.20146846400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"145'51","reported_seq":72,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-04-22T18:41:39.749276+0000","last_change":"2024-04-22T18:41:11.353806+0000","last_active":"2024-04-22T18:41:39.749276+0000","last_peered":"2024-04-22T18:41:39.749276+0000","last_clean":"2024-04-22T18:41:39.749276+0000","last_became_active":"2024-04-22T18:41:11.253409+0000","last_became_peered":"2024-04-22T18:41:11.253409+0000","last_unstale":"2024-04-22T18:41:39.749276+0000","last_undegraded":"2024-04-22T18:41:39.749276+0000","last_fullsized":"2024-04-22T18:41:39.749276+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'23","last_scrub_stamp":"2024-04-22T18:40:20.587606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.587606+0000","objects_scrubbed":1,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10225366599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":815123,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":84302,"num_write":27,"num_write_kb":8588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5574713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"163'88","reported_seq":398,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-04-22T18:41:41.110305+0000","last_change":"2024-04-22T18:41:41.109970+0000","last_active":"2024-04-22T18:41:41.110305+0000","last_peered":"2024-04-22T18:41:41.110305+0000","last_clean":"2024-04-22T18:41:41.110305+0000","last_became_active":"2024-04-22T18:41:41.109681+0000","last_became_peered":"2024-04-22T18:41:41.109681+0000","last_unstale":"2024-04-22T18:41:41.110305+0000","last_undegraded":"2024-04-22T18:41:41.110305+0000","last_fullsized":"2024-04-22T18:41:41.110305+0000","mapping_epoch":20,"log_start":"155'85","ondisk_log_start":"155'85","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":89,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2714396,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":144,"num_read_kb":136200,"num_write":47,"num_write_kb":12257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":36,"stat_sum":{"num_bytes":119012654,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":3902,"num_read_kb":4656965,"num_write":1205,"num_write_kb":347245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":205991097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":32,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194441216,"data_stored":194573097,"data_compressed":1037892,"data_compressed_allocated":1122304,"data_compressed_original":2198706,"omap_allocated":0,"internal_metadata":0},"log_size":1735,"ondisk_log_size":1735,"up":72,"acting":72,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":157864166,"num_objects":209,"num_object_clones":0,"num_object_copies":418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1165,"num_read_kb":30667,"num_write":1674,"num_write_kb":383869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1133,"num_bytes_recovered":211234706,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":273342464,"data_stored":288749483,"data_compressed":15541397,"data_compressed_allocated":16535552,"data_compressed_original":32813056,"omap_allocated":0,"internal_metadata":0},"log_size":1326,"ondisk_log_size":1326,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":2115712,"data_compressed":122550,"data_compressed_allocated":163840,"data_compressed_original":2084864,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65304,"kb_used_data":33340,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94306536,"statfs":{"total":96636764160,"available":96569892864,"internally_reserved":0,"allocated":34140160,"data_stored":33696608,"data_compressed":211962,"data_compressed_allocated":241664,"data_compressed_original":970752,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":9.8949999999999996,"5min":2.3170000000000002,"15min":1.054},"min":{"1min":0.16500000000000001,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":139.84299999999999,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":13.949,"5min":6.3959999999999999,"15min":5.1369999999999996},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":186.97900000000001,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.247}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":4.1959999999999997,"15min":4.6420000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":27.027000000000001,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.316},{"interface":"front","average":{"1min":14.961,"5min":13.43,"15min":13.175000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":183.16300000000001,"5min":183.16300000000001,"15min":183.16300000000001},"last":100.242}]},{"osd":2,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":3.3479999999999999,"15min":3.839},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64100000000000001,"5min":77.320999999999998,"15min":77.320999999999998},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.47199999999999998,"15min":0.47299999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.61399999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.42499999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":3.4079999999999999,"15min":3.903},"min":{"1min":0.32000000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60799999999999998,"5min":77.361000000000004,"15min":77.361000000000004},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.30299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.65000000000000002,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.45100000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":5.6710000000000003,"5min":4.7009999999999996,"15min":4.5389999999999997},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":69.097999999999999,"5min":85.509,"15min":85.509},"last":0.30399999999999999},{"interface":"front","average":{"1min":7.9509999999999996,"5min":19.887,"15min":21.876999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":181.97,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.46000000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.5379999999999998,"15min":2.8929999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67800000000000005,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":3.3170000000000002,"5min":6.3929999999999998,"15min":6.9050000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":69.063000000000002,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.36699999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":17.294,"5min":14.079000000000001,"15min":13.542999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":139.428,"5min":139.428,"15min":139.428},"last":0.439},{"interface":"front","average":{"1min":1.3580000000000001,"5min":3.5779999999999998,"15min":3.948},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":22.878,"5min":77.405000000000001,"15min":77.405000000000001},"last":0.39800000000000002}]}]},{"osd":6,"up_from":0,"seq":55834574887,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574887,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":160980,"kb_used_data":125768,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94210860,"statfs":{"total":96636764160,"available":96471920640,"internally_reserved":0,"allocated":128786432,"data_stored":132543407,"data_compressed":5223761,"data_compressed_allocated":5570560,"data_compressed_original":10765913,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":13.115,"15min":15.470000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":38.899999999999999,"5min":252.96199999999999,"15min":252.96199999999999},"last":38.899999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":9.282,"15min":8.8059999999999992},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63300000000000001,"5min":187.148,"15min":187.148},"last":0.34300000000000003}]},{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":2.508,"15min":1.115},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":9.141,"5min":96.486999999999995,"15min":96.486999999999995},"last":7.0810000000000004},{"interface":"front","average":{"1min":22.393000000000001,"5min":11.545999999999999,"15min":7.4390000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":310.61500000000001,"5min":310.61500000000001,"15min":310.61500000000001},"last":44.793999999999997}]},{"osd":2,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":8.8989999999999991,"15min":8.6790000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.8330000000000002,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.995,"5min":2.4199999999999999,"15min":1.1040000000000001},"min":{"1min":0.31900000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.1399999999999997,"5min":96.471999999999994,"15min":96.471999999999994},"last":7.1399999999999997}]},{"osd":3,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":11.172000000000001,"5min":8.9710000000000001,"15min":5.6710000000000003},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":122.102,"5min":252.91900000000001,"15min":252.91900000000001},"last":38.863999999999997},{"interface":"front","average":{"1min":0.42999999999999999,"5min":17.242000000000001,"15min":19.725999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59899999999999998,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.30299999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":2.3380000000000001,"15min":1.034},"min":{"1min":0.25800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":7.1559999999999997,"5min":96.408000000000001,"15min":96.408000000000001},"last":7.1559999999999997},{"interface":"front","average":{"1min":2.6960000000000002,"5min":3.048,"15min":3.7080000000000002},"min":{"1min":0.36199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":38.581000000000003,"5min":78.811999999999998,"15min":78.811999999999998},"last":38.581000000000003}]},{"osd":6,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":5.8289999999999997,"5min":14.667,"15min":13.662000000000001},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":87.706999999999994,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.83199999999999996,"5min":2.3820000000000001,"15min":1.0880000000000001},"min":{"1min":0.35699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.1159999999999997,"5min":96.438000000000002,"15min":96.438000000000002},"last":7.1159999999999997}]},{"osd":7,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":9.5749999999999993,"15min":8.8979999999999997},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68000000000000005,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.9790000000000001,"5min":2.6419999999999999,"15min":1.202},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":19.518999999999998,"5min":96.456999999999994,"15min":96.456999999999994},"last":7.1859999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87084,"kb_used_data":51868,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94284756,"statfs":{"total":96636764160,"available":96547590144,"internally_reserved":0,"allocated":53112832,"data_stored":54026836,"data_compressed":2303778,"data_compressed_allocated":2441216,"data_compressed_original":4575232,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":15.041,"5min":10.538,"15min":11.646000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":93.131,"5min":98.957999999999998,"15min":98.957999999999998},"last":80.174999999999997},{"interface":"front","average":{"1min":7.077,"5min":1.6659999999999999,"15min":0.77400000000000002},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":87.305999999999997,"5min":87.305999999999997,"15min":87.305999999999997},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":10.558,"15min":6.8819999999999997},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":78.805000000000007,"5min":197.02099999999999,"15min":197.02099999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":8.9800000000000004,"5min":2.3210000000000002,"15min":0.997},"min":{"1min":0.20999999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":78.393000000000001,"5min":78.393000000000001,"15min":78.393000000000001},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":2.3439999999999999,"15min":1.452},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":47.534999999999997,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.46500000000000002},{"interface":"front","average":{"1min":6.4770000000000003,"5min":1.728,"15min":0.85899999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":78.421999999999997,"5min":78.421999999999997,"15min":78.421999999999997},"last":0.51600000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1800000000000002,"5min":2.9940000000000002,"15min":3.7360000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":26.141999999999999,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.65200000000000002},{"interface":"front","average":{"1min":4.5629999999999997,"5min":4.6289999999999996,"15min":4.3010000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":47.616999999999997,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.71899999999999997}]},{"osd":5,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.8279999999999998,"5min":2.5049999999999999,"15min":1.623},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":47.566000000000003,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.66600000000000004},{"interface":"front","average":{"1min":6.6379999999999999,"5min":2.2890000000000001,"15min":1.601},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.480999999999995,"5min":78.480999999999995,"15min":78.480999999999995},"last":0.73499999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.50900000000000001,"15min":0.53100000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.748,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.748},{"interface":"front","average":{"1min":2.8500000000000001,"5min":2.871,"15min":1.3109999999999999},"min":{"1min":0.377,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":47.590000000000003,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.79000000000000004}]},{"osd":7,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":6.6189999999999998,"5min":1.903,"15min":1.0229999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.453999999999994,"5min":78.453999999999994,"15min":78.453999999999994},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69299999999999995,"5min":0.995,"15min":0.995},"last":0.69299999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574887,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57728,"kb_used_data":26608,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94314112,"statfs":{"total":96636764160,"available":96577650688,"internally_reserved":0,"allocated":27246592,"data_stored":27664660,"data_compressed":1299133,"data_compressed_allocated":1376256,"data_compressed_original":2879488,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":4.6120000000000001,"5min":7.4240000000000004,"15min":7.8920000000000003},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":59.002000000000002,"5min":149.03,"15min":149.03},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.308,"5min":1.649,"15min":1.873},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.442,"5min":31.866,"15min":31.866},"last":0.29799999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":13.661,"5min":3.5579999999999998,"15min":1.8740000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":97.683999999999997,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.34300000000000003},{"interface":"front","average":{"1min":22.157,"5min":16.338999999999999,"15min":15.369},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":393.42399999999998,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.35099999999999998}]},{"osd":3,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":18.393000000000001,"5min":3.9689999999999999,"15min":1.5649999999999999},"min":{"1min":0.252,"5min":0.245,"15min":0.245},"max":{"1min":147.721,"5min":147.721,"15min":147.721},"last":0.314},{"interface":"front","average":{"1min":0.317,"5min":0.36899999999999999,"15min":0.377},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.53200000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.27200000000000002}]},{"osd":4,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":3.4060000000000001,"5min":15.698,"15min":17.747},"min":{"1min":0.311,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":52.683,"5min":160.584,"15min":160.584},"last":0.56499999999999995},{"interface":"front","average":{"1min":10.984,"5min":4.1550000000000002,"15min":3.0169999999999999},"min":{"1min":0.32200000000000001,"5min":0.315,"15min":0.315},"max":{"1min":97.887,"5min":97.887,"15min":97.887},"last":0.47399999999999998}]},{"osd":5,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":2.5409999999999999,"15min":2.8809999999999998},"min":{"1min":0.33300000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.74299999999999999,"5min":48.631,"15min":48.631},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":2.8820000000000001,"15min":3.278},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.83499999999999996,"5min":56.18,"15min":56.18},"last":0.60099999999999998}]},{"osd":6,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":10.375,"5min":2.5030000000000001,"15min":1.1910000000000001},"min":{"1min":0.34000000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":97.832999999999998,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.502},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.33200000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.55100000000000005}]},{"osd":7,"last update":"Mon Apr 22 18:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.63800000000000001,"15min":0.59599999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":7.0759999999999996,"5min":7.0759999999999996,"15min":7.0759999999999996},"last":0.629},{"interface":"front","average":{"1min":10.396000000000001,"5min":2.5470000000000002,"15min":1.2390000000000001},"min":{"1min":0.38800000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":97.861999999999995,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.53000000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":132784,"kb_used_data":92880,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94239056,"statfs":{"total":96636764160,"available":96500793344,"internally_reserved":0,"allocated":95109120,"data_stored":96933447,"data_compressed":2762913,"data_compressed_allocated":2945024,"data_compressed_original":6202969,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":3.9870000000000001,"5min":7.0839999999999996,"15min":7.5999999999999996},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":66.480999999999995,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.20000000000000001},{"interface":"front","average":{"1min":4.0430000000000001,"5min":7.0990000000000002,"15min":7.6079999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":66.498000000000005,"5min":88.156000000000006,"15min":88.156000000000006},"last":0.245}]},{"osd":2,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":6.9429999999999996,"15min":8.0470000000000006},"min":{"1min":0.20000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.626,"5min":183.04400000000001,"15min":183.04400000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.26300000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.33700000000000002,"15min":0.33400000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":11.664,"15min":13.548999999999999},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.878,"5min":231.864,"15min":231.864},"last":0.32500000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":7.29,"5min":11.845000000000001,"15min":12.603999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":109.399,"5min":192.42099999999999,"15min":192.42099999999999},"last":0.316},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.497},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.36399999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":11.359999999999999,"15min":13.167},"min":{"1min":0.34399999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.052,"5min":138.089,"15min":138.089},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":16.158999999999999,"15min":18.771000000000001},"min":{"1min":0.35099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69899999999999995,"5min":183.08199999999999,"15min":183.08199999999999},"last":0.42299999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":4.8520000000000003,"5min":3.1920000000000002,"15min":2.915},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":78.805000000000007,"5min":78.805000000000007,"15min":78.805000000000007},"last":0.35399999999999998},{"interface":"front","average":{"1min":5.9400000000000004,"5min":14.413,"15min":15.824999999999999},"min":{"1min":0.36599999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":98.968000000000004,"5min":183.10900000000001,"15min":183.10900000000001},"last":0.38500000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.38500000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68000000000000005,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":2.5350000000000001,"15min":2.871},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.0189999999999999,"5min":51.097000000000001,"15min":51.097000000000001},"last":0.377}]}]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103188,"kb_used_data":67976,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94268652,"statfs":{"total":96636764160,"available":96531099648,"internally_reserved":0,"allocated":69607424,"data_stored":71695471,"data_compressed":2857830,"data_compressed_allocated":3063808,"data_compressed_original":6438912,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":9.7919999999999998,"5min":9.5660000000000007,"15min":7.6950000000000003},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":127.30500000000001,"5min":133.09899999999999,"15min":133.09899999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":27.553999999999998,"5min":11.506,"15min":4.8399999999999999},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":451.50799999999998,"5min":451.50799999999998,"15min":451.50799999999998},"last":0.27400000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":1.5309999999999999,"15min":1.5469999999999999},"min":{"1min":0.217,"5min":0.185,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":39.906999999999996,"15min":39.906999999999996},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":2.2090000000000001,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.41299999999999998,"5min":94.820999999999998,"15min":94.820999999999998},"last":0.22900000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":6.1859999999999999,"5min":3.5299999999999998,"15min":1.486},"min":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":74.963999999999999,"5min":99.332999999999998,"15min":99.332999999999998},"last":0.29099999999999998},{"interface":"front","average":{"1min":16.207000000000001,"5min":9.2159999999999993,"15min":4.1139999999999999},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":273.65100000000001,"5min":297.82100000000003,"15min":297.82100000000003},"last":0.35199999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":5.9980000000000002,"5min":3.4809999999999999,"15min":1.4970000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":96.528999999999996,"5min":96.528999999999996,"15min":96.528999999999996},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":8.8000000000000007,"15min":9.5410000000000004},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":160.02199999999999,"15min":160.02199999999999},"last":0.45700000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":2.1800000000000002,"15min":1.8899999999999999},"min":{"1min":0.35399999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.60399999999999998,"5min":41.645000000000003,"15min":41.645000000000003},"last":0.38},{"interface":"front","average":{"1min":0.47799999999999998,"5min":10.452999999999999,"15min":12.787000000000001},"min":{"1min":0.32300000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66200000000000003,"5min":282.197,"15min":282.197},"last":0.36399999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.7770000000000001,"15min":1.385},"min":{"1min":0.36199999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58299999999999996,"5min":99.355000000000004,"15min":99.355000000000004},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.499,"5min":2.8929999999999998,"15min":2.129},"min":{"1min":0.35899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.72999999999999998,"5min":52.762,"15min":52.762},"last":0.45000000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":4.8929999999999998,"15min":4.7489999999999997},"min":{"1min":0.34499999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.68700000000000006,"5min":94.662999999999997,"15min":94.662999999999997},"last":0.39500000000000002},{"interface":"front","average":{"1min":9.0709999999999997,"5min":4.2939999999999996,"15min":1.901},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":157.089,"5min":157.089,"15min":157.089},"last":0.49199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50429952,"data_stored":53089284,"data_compressed":2774503,"data_compressed_allocated":2965504,"data_compressed_original":5783552,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45453312,"data_stored":48281475,"data_compressed":2448822,"data_compressed_allocated":2609152,"data_compressed_original":5586944,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":20500008,"data_compressed":1299133,"data_compressed_allocated":1376256,"data_compressed_original":2879488,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28307456,"data_stored":29811017,"data_compressed":1578064,"data_compressed_allocated":1671168,"data_compressed_original":3264512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":82026496,"data_stored":86373709,"data_compressed":4550376,"data_compressed_allocated":4845568,"data_compressed_original":9441280,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":223399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27324416,"data_stored":28849251,"data_compressed":1581247,"data_compressed_allocated":1675264,"data_compressed_original":3284992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18276352,"data_stored":18267810,"data_compressed":52333,"data_compressed_allocated":57344,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47738880,"data_stored":47794054,"data_compressed":314091,"data_compressed_allocated":335872,"data_compressed_original":616025,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6344704,"data_stored":6315634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11837440,"data_stored":11760736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":42819584,"data_stored":43089090,"data_compressed":435417,"data_compressed_allocated":471040,"data_compressed_original":931417,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33280000,"data_stored":33118894,"data_compressed":53657,"data_compressed_allocated":57344,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1929216,"data_stored":1913105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32215040,"data_stored":32313774,"data_compressed":182394,"data_compressed_allocated":200704,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:41:44.012 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 36 2024-04-22T18:41:44.062 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:44.063+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:44.074 INFO:tasks.rados.rados.0.smithi089.stdout:2024: done (3 left) 2024-04-22T18:41:44.074 INFO:tasks.rados.rados.0.smithi089.stdout:2025: done (2 left) 2024-04-22T18:41:44.074 INFO:tasks.rados.rados.0.smithi089.stdout:2026: done (1 left) 2024-04-22T18:41:44.074 INFO:tasks.rados.rados.0.smithi089.stdout:2027: delete oid 30 current snap is 0 2024-04-22T18:41:44.076 INFO:tasks.rados.rados.0.smithi089.stdout:2027: done (1 left) 2024-04-22T18:41:44.076 INFO:tasks.rados.rados.0.smithi089.stdout:2028: write (excl) oid 5 current snap is 0 2024-04-22T18:41:44.076 INFO:tasks.rados.rados.0.smithi089.stdout:2028: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2024-04-22T18:41:44.082 INFO:tasks.rados.rados.0.smithi089.stdout:2028: writing smithi08937929-5 from 595080 to 1136761 tid 1 2024-04-22T18:41:44.089 INFO:tasks.rados.rados.0.smithi089.stdout:2028: writing smithi08937929-5 from 1798077 to 2394550 tid 2 2024-04-22T18:41:44.095 INFO:tasks.rados.rados.0.smithi089.stdout:2028: writing smithi08937929-5 from 3150352 to 3395080 tid 3 2024-04-22T18:41:44.095 INFO:tasks.rados.rados.0.smithi089.stdout:2029: tier_flush oid 34 2024-04-22T18:41:44.095 INFO:tasks.rados.rados.0.smithi089.stdout:2029: tier_flush oid 34 snap -1 2024-04-22T18:41:44.095 INFO:tasks.rados.rados.0.smithi089.stdout:2030: set_chunk oid 2 target oid 2024-04-22T18:41:44.118 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:44.118 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:44.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:44.163 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:44.164+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:44.264 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:44.265+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:44.271 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:44.271 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:44.280 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:44.365 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:44.366+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:44.384 INFO:tasks.rados.rados.0.smithi089.stdout:2030: set_chunk oid 2 offset: 684032 length: 1652224 target oid 84621cd73695d0adb7a0877cc02c3d8ff324f87a886ae88e1d491ca961f989af offset: 0 snap -1 2024-04-22T18:41:44.384 INFO:tasks.rados.rados.0.smithi089.stdout:2031: read oid 8 snap -1 2024-04-22T18:41:44.384 INFO:tasks.rados.rados.0.smithi089.stdout:2031: expect deleted 2024-04-22T18:41:44.384 INFO:tasks.rados.rados.0.smithi089.stdout:2032: read oid 36 snap -1 2024-04-22T18:41:44.384 INFO:tasks.rados.rados.0.smithi089.stdout:2032: expect deleted 2024-04-22T18:41:44.384 INFO:tasks.rados.rados.0.smithi089.stdout:2033: read oid 3 snap -1 2024-04-22T18:41:44.384 INFO:tasks.rados.rados.0.smithi089.stdout:2033: expect (ObjNum 414 snap 0 seq_num 414) 2024-04-22T18:41:44.385 INFO:tasks.rados.rados.0.smithi089.stdout:2034: read oid 41 snap -1 2024-04-22T18:41:44.385 INFO:tasks.rados.rados.0.smithi089.stdout:2034: expect (ObjNum 358 snap 0 seq_num 358) 2024-04-22T18:41:44.385 INFO:tasks.rados.rados.0.smithi089.stdout:2035: read oid 7 snap -1 2024-04-22T18:41:44.385 INFO:tasks.rados.rados.0.smithi089.stdout:2035: expect (ObjNum 431 snap 0 seq_num 431) 2024-04-22T18:41:44.385 INFO:tasks.rados.rados.0.smithi089.stdout:2036: read oid 24 snap -1 2024-04-22T18:41:44.385 INFO:tasks.rados.rados.0.smithi089.stdout:2036: expect (ObjNum 384 snap 0 seq_num 384) 2024-04-22T18:41:44.385 INFO:tasks.rados.rados.0.smithi089.stdout:2037: tier_evict oid 33 2024-04-22T18:41:44.385 INFO:tasks.rados.rados.0.smithi089.stdout:2037: tier_evict oid 33 snap -1 2024-04-22T18:41:44.385 INFO:tasks.rados.rados.0.smithi089.stdout:2038: read oid 37 snap -1 2024-04-22T18:41:44.385 INFO:tasks.rados.rados.0.smithi089.stdout:2038: expect (ObjNum 419 snap 0 seq_num 419) 2024-04-22T18:41:44.385 INFO:tasks.rados.rados.0.smithi089.stdout:2039: write (excl) oid 22 current snap is 0 2024-04-22T18:41:44.385 INFO:tasks.rados.rados.0.smithi089.stdout:2039: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2024-04-22T18:41:44.391 INFO:tasks.rados.rados.0.smithi089.stdout:2039: writing smithi08937929-22 from 611887 to 1345170 tid 1 2024-04-22T18:41:44.398 INFO:tasks.rados.rados.0.smithi089.stdout:2039: writing smithi08937929-22 from 1972912 to 2429396 tid 2 2024-04-22T18:41:44.404 INFO:tasks.rados.rados.0.smithi089.stdout:2039: writing smithi08937929-22 from 2894128 to 3411887 tid 3 2024-04-22T18:41:44.404 INFO:tasks.rados.rados.0.smithi089.stdout:2040: read oid 14 snap -1 2024-04-22T18:41:44.404 INFO:tasks.rados.rados.0.smithi089.stdout:2040: expect (ObjNum 424 snap 0 seq_num 424) 2024-04-22T18:41:44.404 INFO:tasks.rados.rados.0.smithi089.stdout:2041: read oid 42 snap -1 2024-04-22T18:41:44.404 INFO:tasks.rados.rados.0.smithi089.stdout:2041: expect (ObjNum 434 snap 0 seq_num 434) 2024-04-22T18:41:44.404 INFO:tasks.rados.rados.0.smithi089.stdout:2042: read oid 9 snap -1 2024-04-22T18:41:44.404 INFO:tasks.rados.rados.0.smithi089.stdout:2042: expect deleted 2024-04-22T18:41:44.405 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:41:44.405 INFO:tasks.rados.rados.0.smithi089.stdout:2030: finishing set_chunk to oid 2 2024-04-22T18:41:44.405 INFO:tasks.rados.rados.0.smithi089.stdout:Range is overlapped: oid 2 set_chunk returned error code -95 offset: 684032 length: 1652224 2024-04-22T18:41:44.405 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 2 v 0 (ObjNum 385 snap 0 seq_num 385) dirty exists 2024-04-22T18:41:44.405 INFO:tasks.rados.rados.0.smithi089.stdout:2030: done (15 left) 2024-04-22T18:41:44.405 INFO:tasks.rados.rados.0.smithi089.stdout:2043: read oid 1 snap -1 2024-04-22T18:41:44.405 INFO:tasks.rados.rados.0.smithi089.stdout:2043: expect (ObjNum 347 snap 0 seq_num 347) 2024-04-22T18:41:44.405 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 16 2024-04-22T18:41:44.405 INFO:tasks.rados.rados.0.smithi089.stdout:2035: expect (ObjNum 431 snap 0 seq_num 431) 2024-04-22T18:41:44.426 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:44.426 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:44.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:44.479 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:44.480+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:44.541 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:44.541 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:44.550 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:44.591 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:44.592+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:44.627 INFO:tasks.rados.rados.0.smithi089.stdout:2039: finishing write tid 1 to smithi08937929-22 2024-04-22T18:41:44.627 INFO:tasks.rados.rados.0.smithi089.stdout:2041: expect (ObjNum 434 snap 0 seq_num 434) 2024-04-22T18:41:44.667 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:44.668+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:44.691 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:44.691 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:44.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:44.716 INFO:teuthology.orchestra.run.smithi089.stderr:set pool 4 pgp_num to 36 2024-04-22T18:41:44.769 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:44.771+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:44.803 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:44.803 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:44.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:44.869 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:44.870+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:44.911 INFO:tasks.rados.rados.0.smithi089.stdout:2034: expect (ObjNum 358 snap 0 seq_num 358) 2024-04-22T18:41:44.913 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:44.913 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:44.922 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:44.948 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:41:44.970 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:44.971+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:45.024 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:45.024 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:45.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:45.066 INFO:tasks.rados.rados.0.smithi089.stdout:2043: expect (ObjNum 347 snap 0 seq_num 347) 2024-04-22T18:41:45.068 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:45.068 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:45.071 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:45.072+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:45.086 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:45.086 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:45.101 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:45.101 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:45.117 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:45.117 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:45.138 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:45.138 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:45.154 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:45.154 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:45.174 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:45.174 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:45.174 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:45.175+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:45.179 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:45.179 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:45.187 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:45.188 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:45.189 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:45.191 INFO:tasks.rados.rados.0.smithi089.stdout:2040: expect (ObjNum 424 snap 0 seq_num 424) 2024-04-22T18:41:45.275 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:45.276+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:45.291 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:45.291 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:45.291 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:45.300 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:45.376 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:45.377+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:45.381 INFO:tasks.rados.rados.0.smithi089.stdout:2039: finishing write tid 2 to smithi08937929-22 2024-04-22T18:41:45.381 INFO:tasks.rados.rados.0.smithi089.stdout:2039: finishing write tid 3 to smithi08937929-22 2024-04-22T18:41:45.381 INFO:tasks.rados.rados.0.smithi089.stdout:2029: got (0) Success 2024-04-22T18:41:45.381 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 34 v 0 (ObjNum 417 snap 0 seq_num 417) dirty exists 2024-04-22T18:41:45.381 INFO:tasks.rados.rados.0.smithi089.stdout:2033: expect (ObjNum 414 snap 0 seq_num 414) 2024-04-22T18:41:45.402 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:45.402 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:45.402 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:45.411 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:45.476 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:45.477+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:45.516 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:45.517 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:45.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:45.517 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:45.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.517 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:45.517 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:45.517 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 171.06105193400001, 2024-04-22T18:41:45.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:41:45.517 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.517 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:45.517 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.517 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.518 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.518 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:41:45.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.518 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:45.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:45.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:41:45.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:45.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:41:45.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:41:45.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:41:45.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:41:45.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:41:45.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:41:45.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:41:45.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:41:45.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:41:45.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:41:45.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:45.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:41:45.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:41:45.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:45.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:41:45.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:41:45.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:45.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:41:45.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:41:45.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:45.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:41:45.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:41:45.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:41:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:41:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 162.367994686, 2024-04-22T18:41:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:41:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.523 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.524 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:41:45.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.524 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:45.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:45.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:41:45.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:45.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:41:45.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:41:45.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:41:45.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:41:45.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:41:45.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:41:45.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:41:45.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:41:45.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:41:45.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:41:45.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:45.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:41:45.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:41:45.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:41:45.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:41:45.528 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.528 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.528 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.528 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:45.528 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:45.528 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 162.36797620999999, 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:45.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:45.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:45.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:41:45.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:41:45.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:41:45.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:41:45.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:41:45.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:41:45.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:41:45.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:41:45.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:41:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:41:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:41:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:41:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:41:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:41:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:41:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:41:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:41:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 134.604927898, 2024-04-22T18:41:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:41:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:41:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:41:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:41:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:41:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:41:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:41:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:41:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:41:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:41:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:41:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:41:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:41:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:41:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:41:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:41:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:41:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:41:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:41:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:41:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:41:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:41:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:41:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 124.932499589, 2024-04-22T18:41:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:41:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:41:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:41:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:41:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:41:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:41:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:41:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:41:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:41:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:41:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:41:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:41:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:41:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:41:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 115.123482882, 2024-04-22T18:41:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:41:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:41:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:41:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:41:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:41:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:41:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:41:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:41:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:41:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:41:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:41:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:41:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:41:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:41:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:41:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:41:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:41:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:41:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:41:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 98.227444509999998, 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:41:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:41:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:41:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:41:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:41:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:41:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:41:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:41:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:41:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:41:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:41:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:41:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3377 4.6 4:6700eab5:::smithi08937929-32:head [read 1812992~307712] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-04-22T18:41:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 75.080040867999998, 2024-04-22T18:41:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.52001745799999999, 2024-04-22T18:41:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3377 2024-04-22T18:41:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436364+0000", 2024-04-22T18:41:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436376+0000", 2024-04-22T18:41:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2121e-05 2024-04-22T18:41:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436394+0000", 2024-04-22T18:41:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8094e-05 2024-04-22T18:41:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436405+0000", 2024-04-22T18:41:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0189e-05 2024-04-22T18:41:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436471+0000", 2024-04-22T18:41:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6817999999999996e-05 2024-04-22T18:41:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436481+0000", 2024-04-22T18:41:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2720000000000003e-06 2024-04-22T18:41:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956084+0000", 2024-04-22T18:41:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51960320800000004 2024-04-22T18:41:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956232+0000", 2024-04-22T18:41:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014778600000000001 2024-04-22T18:41:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956385+0000", 2024-04-22T18:41:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153394 2024-04-22T18:41:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:131 3.1 3:a0a19997:::ee8ca23a7050e771c77dff761e6ea5a11ed57796f4e8a17cbb5804a7fdc937b9:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.187171272, 2024-04-22T18:41:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51449564999999997, 2024-04-22T18:41:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329235+0000", 2024-04-22T18:41:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329244+0000", 2024-04-22T18:41:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6819999999999992e-06 2024-04-22T18:41:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329259+0000", 2024-04-22T18:41:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698000000000001e-05 2024-04-22T18:41:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329265+0000", 2024-04-22T18:41:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1510000000000001e-06 2024-04-22T18:41:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329311+0000", 2024-04-22T18:41:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6096e-05 2024-04-22T18:41:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.332269+0000", 2024-04-22T18:41:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029581910000000002 2024-04-22T18:41:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630603+0000", 2024-04-22T18:41:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29833418 2024-04-22T18:41:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630685+0000", 2024-04-22T18:41:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1217999999999994e-05 2024-04-22T18:41:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630813+0000", 2024-04-22T18:41:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128706 2024-04-22T18:41:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.631447+0000", 2024-04-22T18:41:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063427000000000004 2024-04-22T18:41:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843586+0000", 2024-04-22T18:41:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21213873 2024-04-22T18:41:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843623+0000", 2024-04-22T18:41:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6848999999999999e-05 2024-04-22T18:41:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843733+0000", 2024-04-22T18:41:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010982200000000001 2024-04-22T18:41:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3441 4.b 4:d3068ee3:::smithi08937929-29:head [write 558065~619869] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 73.598695211000006, 2024-04-22T18:41:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50272020399999995, 2024-04-22T18:41:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3441 2024-04-22T18:41:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917711+0000", 2024-04-22T18:41:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917838+0000", 2024-04-22T18:41:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012734899999999999 2024-04-22T18:41:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917854+0000", 2024-04-22T18:41:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6130999999999999e-05 2024-04-22T18:41:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917863+0000", 2024-04-22T18:41:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5969999999999992e-06 2024-04-22T18:41:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992845+0000", 2024-04-22T18:41:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.074982193000000003 2024-04-22T18:41:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992919+0000", 2024-04-22T18:41:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:41:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.993019+0000", 2024-04-22T18:41:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9718999999999999e-05 2024-04-22T18:41:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095554+0000", 2024-04-22T18:41:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102535193 2024-04-22T18:41:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420183+0000", 2024-04-22T18:41:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32462853600000002 2024-04-22T18:41:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420401+0000", 2024-04-22T18:41:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021881400000000001 2024-04-22T18:41:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420433+0000", 2024-04-22T18:41:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079000000000002e-05 2024-04-22T18:41:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3442 4.b 4:d3068ee3:::smithi08937929-29:head [write 1579835~486765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 73.594008500000001, 2024-04-22T18:41:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49822799099999998, 2024-04-22T18:41:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3442 2024-04-22T18:41:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922399+0000", 2024-04-22T18:41:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922502+0000", 2024-04-22T18:41:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102784 2024-04-22T18:41:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922513+0000", 2024-04-22T18:41:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1051e-05 2024-04-22T18:41:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922518+0000", 2024-04-22T18:41:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8529999999999997e-06 2024-04-22T18:41:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093158+0000", 2024-04-22T18:41:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.17063958800000001 2024-04-22T18:41:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093230+0000", 2024-04-22T18:41:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1803000000000004e-05 2024-04-22T18:41:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093326+0000", 2024-04-22T18:41:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6454999999999997e-05 2024-04-22T18:41:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095557+0000", 2024-04-22T18:41:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022306399999999999 2024-04-22T18:41:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420484+0000", 2024-04-22T18:41:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32492717700000001 2024-04-22T18:41:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420606+0000", 2024-04-22T18:41:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012225800000000001 2024-04-22T18:41:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420628+0000", 2024-04-22T18:41:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1596000000000001e-05 2024-04-22T18:41:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3443 4.b 4:d3068ee3:::smithi08937929-29:head [write 2468371~489694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 73.589406452999995, 2024-04-22T18:41:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49372487199999998, 2024-04-22T18:41:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3443 2024-04-22T18:41:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927001+0000", 2024-04-22T18:41:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927117+0000", 2024-04-22T18:41:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011637 2024-04-22T18:41:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927128+0000", 2024-04-22T18:41:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0696000000000001e-05 2024-04-22T18:41:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927134+0000", 2024-04-22T18:41:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8640000000000001e-06 2024-04-22T18:41:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094288+0000", 2024-04-22T18:41:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16715412900000001 2024-04-22T18:41:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094359+0000", 2024-04-22T18:41:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1639999999999998e-05 2024-04-22T18:41:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094451+0000", 2024-04-22T18:41:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1252e-05 2024-04-22T18:41:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095560+0000", 2024-04-22T18:41:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011095530000000001 2024-04-22T18:41:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420661+0000", 2024-04-22T18:41:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32510105500000003 2024-04-22T18:41:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420710+0000", 2024-04-22T18:41:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9018999999999999e-05 2024-04-22T18:41:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420727+0000", 2024-04-22T18:41:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6722e-05 2024-04-22T18:41:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3444 4.b 4:d3068ee3:::smithi08937929-29:head [setxattr _header (56) in=7b,truncate 2958065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 73.589270736000003, 2024-04-22T18:41:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49366935299999998, 2024-04-22T18:41:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3444 2024-04-22T18:41:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927137+0000", 2024-04-22T18:41:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927141+0000", 2024-04-22T18:41:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3270000000000001e-06 2024-04-22T18:41:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927144+0000", 2024-04-22T18:41:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9730000000000002e-06 2024-04-22T18:41:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927147+0000", 2024-04-22T18:41:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.959e-06 2024-04-22T18:41:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095118+0000", 2024-04-22T18:41:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16797115500000001 2024-04-22T18:41:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095169+0000", 2024-04-22T18:41:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1316000000000001e-05 2024-04-22T18:41:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095260+0000", 2024-04-22T18:41:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1184e-05 2024-04-22T18:41:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095999+0000", 2024-04-22T18:41:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073868200000000005 2024-04-22T18:41:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420757+0000", 2024-04-22T18:41:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32475783499999999 2024-04-22T18:41:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420781+0000", 2024-04-22T18:41:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4022000000000001e-05 2024-04-22T18:41:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420807+0000", 2024-04-22T18:41:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6366000000000001e-05 2024-04-22T18:41:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3445 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 73.589259642000002, 2024-04-22T18:41:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.493780049, 2024-04-22T18:41:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3445 2024-04-22T18:41:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927150+0000", 2024-04-22T18:41:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.542e-06 2024-04-22T18:41:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927153+0000", 2024-04-22T18:41:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0520000000000002e-06 2024-04-22T18:41:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027218+0000", 2024-04-22T18:41:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100064604 2024-04-22T18:41:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095501+0000", 2024-04-22T18:41:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.068283431000000006 2024-04-22T18:41:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095545+0000", 2024-04-22T18:41:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4295999999999997e-05 2024-04-22T18:41:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420833+0000", 2024-04-22T18:41:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32528718400000001 2024-04-22T18:41:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420877+0000", 2024-04-22T18:41:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4403999999999997e-05 2024-04-22T18:41:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420929+0000", 2024-04-22T18:41:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1848999999999998e-05 2024-04-22T18:41:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4780 4.6 4:630bd8e5:::smithi08937929-45:head [write 574905~711397,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.278066+0000", 2024-04-22T18:41:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.238342397, 2024-04-22T18:41:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88434433999999995, 2024-04-22T18:41:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4780 2024-04-22T18:41:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278066+0000", 2024-04-22T18:41:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278066+0000", 2024-04-22T18:41:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278063+0000", 2024-04-22T18:41:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278391+0000", 2024-04-22T18:41:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032839499999999999 2024-04-22T18:41:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278404+0000", 2024-04-22T18:41:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3528e-05 2024-04-22T18:41:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278415+0000", 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0295000000000001e-05 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278454+0000", 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9551999999999997e-05 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278462+0000", 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4080000000000001e-06 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.959123+0000", 2024-04-22T18:41:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68066109799999996 2024-04-22T18:41:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.959282+0000", 2024-04-22T18:41:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000159023 2024-04-22T18:41:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.959448+0000", 2024-04-22T18:41:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016666200000000001 2024-04-22T18:41:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:45.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162299+0000", 2024-04-22T18:41:45.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20285081199999999 2024-04-22T18:41:45.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:45.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162316+0000", 2024-04-22T18:41:45.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6354999999999998e-05 2024-04-22T18:41:45.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:45.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162347+0000", 2024-04-22T18:41:45.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1427999999999998e-05 2024-04-22T18:41:45.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162410+0000", 2024-04-22T18:41:45.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3350999999999997e-05 2024-04-22T18:41:45.598 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.598 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.598 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.598 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4781 4.6 4:630bd8e5:::smithi08937929-45:head [write 1885086~525140] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:45.598 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.284151+0000", 2024-04-22T18:41:45.598 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.232257557000001, 2024-04-22T18:41:45.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88746760999999996, 2024-04-22T18:41:45.598 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4781 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284151+0000", 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284151+0000", 2024-04-22T18:41:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284149+0000", 2024-04-22T18:41:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284385+0000", 2024-04-22T18:41:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023556799999999999 2024-04-22T18:41:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284396+0000", 2024-04-22T18:41:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0842999999999999e-05 2024-04-22T18:41:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284401+0000", 2024-04-22T18:41:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3260000000000003e-06 2024-04-22T18:41:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284444+0000", 2024-04-22T18:41:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3186000000000003e-05 2024-04-22T18:41:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284454+0000", 2024-04-22T18:41:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6160000000000008e-06 2024-04-22T18:41:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.161315+0000", 2024-04-22T18:41:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87686138000000002 2024-04-22T18:41:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.161381+0000", 2024-04-22T18:41:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5838000000000002e-05 2024-04-22T18:41:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.161470+0000", 2024-04-22T18:41:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9332000000000005e-05 2024-04-22T18:41:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163245+0000", 2024-04-22T18:41:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017744530000000001 2024-04-22T18:41:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.171550+0000", 2024-04-22T18:41:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083046879999999993 2024-04-22T18:41:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.171592+0000", 2024-04-22T18:41:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2438999999999998e-05 2024-04-22T18:41:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.171619+0000", 2024-04-22T18:41:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6517e-05 2024-04-22T18:41:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4782 4.6 4:630bd8e5:::smithi08937929-45:head [write 2930189~44716] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.287206+0000", 2024-04-22T18:41:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.22920259, 2024-04-22T18:41:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98327066299999999, 2024-04-22T18:41:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4782 2024-04-22T18:41:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287206+0000", 2024-04-22T18:41:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287206+0000", 2024-04-22T18:41:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287205+0000", 2024-04-22T18:41:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287220+0000", 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5709000000000002e-05 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287231+0000", 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0196e-05 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287236+0000", 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3770000000000004e-06 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287264+0000", 2024-04-22T18:41:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8549999999999999e-05 2024-04-22T18:41:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287268+0000", 2024-04-22T18:41:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7299999999999999e-06 2024-04-22T18:41:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162435+0000", 2024-04-22T18:41:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87516707999999999 2024-04-22T18:41:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162490+0000", 2024-04-22T18:41:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5210000000000002e-05 2024-04-22T18:41:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162567+0000", 2024-04-22T18:41:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6947999999999996e-05 2024-04-22T18:41:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163249+0000", 2024-04-22T18:41:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00068115299999999999 2024-04-22T18:41:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.270414+0000", 2024-04-22T18:41:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10716516700000001 2024-04-22T18:41:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.270448+0000", 2024-04-22T18:41:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4042999999999999e-05 2024-04-22T18:41:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.270477+0000", 2024-04-22T18:41:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8796e-05 2024-04-22T18:41:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4783 4.6 4:630bd8e5:::smithi08937929-45:head [setxattr _header (56) in=7b,truncate 2974905] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.287242+0000", 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.229166476, 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98449730599999996, 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4783 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287242+0000", 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287242+0000", 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287241+0000", 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287245+0000", 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.997e-06 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287251+0000", 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4999999999999999e-06 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387333+0000", 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100082174 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387418+0000", 2024-04-22T18:41:45.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5086000000000005e-05 2024-04-22T18:41:45.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:45.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387427+0000", 2024-04-22T18:41:45.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6829999999999997e-06 2024-04-22T18:41:45.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162901+0000", 2024-04-22T18:41:45.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77547466899999995 2024-04-22T18:41:45.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162942+0000", 2024-04-22T18:41:45.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0299999999999997e-05 2024-04-22T18:41:45.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:45.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163028+0000", 2024-04-22T18:41:45.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6267000000000006e-05 2024-04-22T18:41:45.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:45.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163938+0000", 2024-04-22T18:41:45.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091012500000000002 2024-04-22T18:41:45.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:45.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271677+0000", 2024-04-22T18:41:45.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107739168 2024-04-22T18:41:45.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:45.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271709+0000", 2024-04-22T18:41:45.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1191e-05 2024-04-22T18:41:45.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271739+0000", 2024-04-22T18:41:45.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0814e-05 2024-04-22T18:41:45.616 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.616 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.617 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.617 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4784 4.6 4:630bd8e5:::smithi08937929-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:45.617 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.387355+0000", 2024-04-22T18:41:45.617 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.1290534, 2024-04-22T18:41:45.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.884536714, 2024-04-22T18:41:45.617 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.617 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:45.617 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.617 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.617 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.617 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4784 2024-04-22T18:41:45.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.618 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387355+0000", 2024-04-22T18:41:45.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387355+0000", 2024-04-22T18:41:45.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387352+0000", 2024-04-22T18:41:45.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:45.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387386+0000", 2024-04-22T18:41:45.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3974999999999999e-05 2024-04-22T18:41:45.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387399+0000", 2024-04-22T18:41:45.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2882e-05 2024-04-22T18:41:45.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387408+0000", 2024-04-22T18:41:45.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2310000000000002e-06 2024-04-22T18:41:45.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.487520+0000", 2024-04-22T18:41:45.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100111622 2024-04-22T18:41:45.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:45.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.487531+0000", 2024-04-22T18:41:45.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0655999999999999e-05 2024-04-22T18:41:45.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163199+0000", 2024-04-22T18:41:45.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.67566841700000002 2024-04-22T18:41:45.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:45.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163239+0000", 2024-04-22T18:41:45.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9749000000000002e-05 2024-04-22T18:41:45.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271780+0000", 2024-04-22T18:41:45.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10854082600000001 2024-04-22T18:41:45.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271830+0000", 2024-04-22T18:41:45.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0835999999999999e-05 2024-04-22T18:41:45.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271892+0000", 2024-04-22T18:41:45.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1440999999999997e-05 2024-04-22T18:41:45.622 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.622 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.622 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.622 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4940 3.1 3:af0c84c6:::a6954558d6d261bd8039fe057a41a32eb57dde7f3e53a34b25ba0607fb91f814:head [write 0~396288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-04-22T18:41:45.623 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:41.009760+0000", 2024-04-22T18:41:45.623 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.506648727, 2024-04-22T18:41:45.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.500823562, 2024-04-22T18:41:45.623 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:45.623 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:45.623 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:45.623 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:45.623 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:45.623 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4940 2024-04-22T18:41:45.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.623 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:45.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:45.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009760+0000", 2024-04-22T18:41:45.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:45.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009760+0000", 2024-04-22T18:41:45.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:45.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:45.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009758+0000", 2024-04-22T18:41:45.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:45.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:45.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009935+0000", 2024-04-22T18:41:45.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017724200000000001 2024-04-22T18:41:45.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:45.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009946+0000", 2024-04-22T18:41:45.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0477e-05 2024-04-22T18:41:45.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:45.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009952+0000", 2024-04-22T18:41:45.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3090000000000001e-06 2024-04-22T18:41:45.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:45.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.109570+0000", 2024-04-22T18:41:45.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099618130999999999 2024-04-22T18:41:45.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:45.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.109693+0000", 2024-04-22T18:41:45.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012241699999999999 2024-04-22T18:41:45.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:45.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.109807+0000", 2024-04-22T18:41:45.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011426599999999999 2024-04-22T18:41:45.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:45.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.110527+0000", 2024-04-22T18:41:45.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00071964499999999999 2024-04-22T18:41:45.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:45.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.510490+0000", 2024-04-22T18:41:45.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39996386499999997 2024-04-22T18:41:45.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:45.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.510550+0000", 2024-04-22T18:41:45.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9669000000000001e-05 2024-04-22T18:41:45.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:45.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:45.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:45.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.510583+0000", 2024-04-22T18:41:45.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3281000000000002e-05 2024-04-22T18:41:45.628 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.628 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.628 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.628 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:45.628 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:45.628 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:45.630 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:45.631 INFO:tasks.rados.rados.0.smithi089.stdout:2039: finishing write tid 4 to smithi08937929-22 2024-04-22T18:41:45.631 INFO:tasks.rados.rados.0.smithi089.stdout:2039: finishing write tid 5 to smithi08937929-22 2024-04-22T18:41:45.631 INFO:tasks.rados.rados.0.smithi089.stdout:2039: oid 22 updating version 0 to 129 2024-04-22T18:41:45.631 INFO:tasks.rados.rados.0.smithi089.stdout:2039: oid 22 updating version 129 to 131 2024-04-22T18:41:45.631 INFO:tasks.rados.rados.0.smithi089.stdout:2039: oid 22 version 131 is already newer than 128 2024-04-22T18:41:45.631 INFO:tasks.rados.rados.0.smithi089.stdout:2039: oid 22 version 131 is already newer than 130 2024-04-22T18:41:45.631 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 22 v 131 (ObjNum 441 snap 0 seq_num 441) dirty exists 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2039: left oid 22 (ObjNum 441 snap 0 seq_num 441) 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2029: done (15 left) 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2031: done (14 left) 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2032: done (13 left) 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2033: done (12 left) 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2034: done (11 left) 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2035: done (10 left) 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2039: done (9 left) 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2040: done (8 left) 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2041: done (7 left) 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2043: done (6 left) 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2044: read oid 46 snap -1 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2044: expect (ObjNum 396 snap 0 seq_num 396) 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2045: write oid 45 current snap is 0 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2037: got (22) Invalid argument 2024-04-22T18:41:45.632 INFO:tasks.rados.rados.0.smithi089.stdout:2038: expect (ObjNum 419 snap 0 seq_num 419) 2024-04-22T18:41:45.633 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:45.630+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:45.726 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:41:45.730 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:45.731+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:45.774 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:45.774 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:45.774 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:45.775 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:45.784 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:45.819 INFO:tasks.rados.rados.0.smithi089.stdout:2036: expect (ObjNum 384 snap 0 seq_num 384) 2024-04-22T18:41:45.832 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:45.832+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:45.888 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:45.888 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:45.888 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:45.888 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:45.888 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:45.897 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:45.902 INFO:tasks.rados.rados.0.smithi089.stdout:2028: finishing write tid 1 to smithi08937929-5 2024-04-22T18:41:45.902 INFO:tasks.rados.rados.0.smithi089.stdout:2028: finishing write tid 2 to smithi08937929-5 2024-04-22T18:41:45.902 INFO:tasks.rados.rados.0.smithi089.stdout:2045: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2024-04-22T18:41:45.907 INFO:tasks.rados.rados.0.smithi089.stdout:2045: writing smithi08937929-45 from 628694 to 1037226 tid 1 2024-04-22T18:41:45.914 INFO:tasks.rados.rados.0.smithi089.stdout:2045: writing smithi08937929-45 from 1515041 to 2115183 tid 2 2024-04-22T18:41:45.922 INFO:tasks.rados.rados.0.smithi089.stdout:2045: writing smithi08937929-45 from 2805198 to 3357310 tid 3 2024-04-22T18:41:45.922 INFO:tasks.rados.rados.0.smithi089.stdout:2036: done (7 left) 2024-04-22T18:41:45.922 INFO:tasks.rados.rados.0.smithi089.stdout:2037: done (6 left) 2024-04-22T18:41:45.922 INFO:tasks.rados.rados.0.smithi089.stdout:2038: done (5 left) 2024-04-22T18:41:45.922 INFO:tasks.rados.rados.0.smithi089.stdout:2042: done (4 left) 2024-04-22T18:41:45.922 INFO:tasks.rados.rados.0.smithi089.stdout:2046: read oid 50 snap -1 2024-04-22T18:41:45.922 INFO:tasks.rados.rados.0.smithi089.stdout:2046: expect (ObjNum 405 snap 0 seq_num 405) 2024-04-22T18:41:45.922 INFO:tasks.rados.rados.0.smithi089.stdout:2028: finishing write tid 3 to smithi08937929-5 2024-04-22T18:41:45.923 INFO:tasks.rados.rados.0.smithi089.stdout:2028: finishing write tid 4 to smithi08937929-5 2024-04-22T18:41:45.923 INFO:tasks.rados.rados.0.smithi089.stdout:2028: finishing write tid 5 to smithi08937929-5 2024-04-22T18:41:45.923 INFO:tasks.rados.rados.0.smithi089.stdout:2028: oid 5 updating version 0 to 193 2024-04-22T18:41:45.923 INFO:tasks.rados.rados.0.smithi089.stdout:2028: oid 5 updating version 193 to 194 2024-04-22T18:41:45.923 INFO:tasks.rados.rados.0.smithi089.stdout:2028: oid 5 version 194 is already newer than 191 2024-04-22T18:41:45.923 INFO:tasks.rados.rados.0.smithi089.stdout:2028: oid 5 version 194 is already newer than 192 2024-04-22T18:41:45.923 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 5 v 194 (ObjNum 440 snap 0 seq_num 440) dirty exists 2024-04-22T18:41:45.923 INFO:tasks.rados.rados.0.smithi089.stdout:2028: left oid 5 (ObjNum 440 snap 0 seq_num 440) 2024-04-22T18:41:45.923 INFO:tasks.rados.rados.0.smithi089.stdout:2023: finishing set_chunk to oid 23 2024-04-22T18:41:45.923 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 23 v 185 (ObjNum 390 snap 0 seq_num 390) dirty exists 2024-04-22T18:41:45.923 INFO:tasks.rados.rados.0.smithi089.stdout:2047: read oid 7 snap -1 2024-04-22T18:41:45.923 INFO:tasks.rados.rados.0.smithi089.stdout:2047: expect (ObjNum 431 snap 0 seq_num 431) 2024-04-22T18:41:45.924 INFO:tasks.rados.rados.0.smithi089.stdout:2023: done (5 left) 2024-04-22T18:41:45.924 INFO:tasks.rados.rados.0.smithi089.stdout:2028: done (4 left) 2024-04-22T18:41:45.924 INFO:tasks.rados.rados.0.smithi089.stdout:2048: read oid 12 snap -1 2024-04-22T18:41:45.924 INFO:tasks.rados.rados.0.smithi089.stdout:2048: expect deleted 2024-04-22T18:41:45.924 INFO:tasks.rados.rados.0.smithi089.stdout:2049: delete oid 41 current snap is 0 2024-04-22T18:41:45.924 INFO:tasks.rados.rados.0.smithi089.stdout:2044: expect (ObjNum 396 snap 0 seq_num 396) 2024-04-22T18:41:45.932 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:45.933+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:46.009 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 171.816890973, 2024-04-22T18:41:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:41:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:41:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:41:46.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:41:46.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:41:46.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:41:46.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:41:46.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:46.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:41:46.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:41:46.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:46.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:41:46.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:41:46.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:46.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:41:46.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:41:46.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:41:46.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:41:46.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:41:46.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:41:46.015 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.015 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.016 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.016 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:46.016 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:46.016 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 171.816879898, 2024-04-22T18:41:46.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:41:46.016 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.016 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.016 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.016 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.016 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.016 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:41:46.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.016 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:46.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:46.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:41:46.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:41:46.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:41:46.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:41:46.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:41:46.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:41:46.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:41:46.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:41:46.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:41:46.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:46.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:41:46.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:41:46.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:41:46.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:41:46.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:41:46.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:41:46.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:41:46.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:41:46.021 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.021 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.021 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.021 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3193 4.11 4.9f8e4db1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:46.021 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:41:46.021 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 83.965671608999997, 2024-04-22T18:41:46.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69912102700000001, 2024-04-22T18:41:46.021 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.021 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:46.021 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.021 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.021 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.021 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3193 2024-04-22T18:41:46.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.022 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:41:46.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042746+0000", 2024-04-22T18:41:46.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.042744+0000", 2024-04-22T18:41:46.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:46.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043013+0000", 2024-04-22T18:41:46.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026906 2024-04-22T18:41:46.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043028+0000", 2024-04-22T18:41:46.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5359999999999999e-05 2024-04-22T18:41:46.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043036+0000", 2024-04-22T18:41:46.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8420000000000005e-06 2024-04-22T18:41:46.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043081+0000", 2024-04-22T18:41:46.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5019999999999999e-05 2024-04-22T18:41:46.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.043087+0000", 2024-04-22T18:41:46.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5420000000000002e-06 2024-04-22T18:41:46.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741856+0000", 2024-04-22T18:41:46.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69876864500000002 2024-04-22T18:41:46.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:22.741867+0000", 2024-04-22T18:41:46.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.084e-05 2024-04-22T18:41:46.025 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.025 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.025 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.025 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3504 4.9 4:96737e19:::smithi08937929-4:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:46.025 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:46.026 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 70.597555893999996, 2024-04-22T18:41:46.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.951755291, 2024-04-22T18:41:46.026 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.026 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.026 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.026 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.026 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.026 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3504 2024-04-22T18:41:46.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.026 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:46.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:46.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410861+0000", 2024-04-22T18:41:46.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410863+0000", 2024-04-22T18:41:46.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.635e-06 2024-04-22T18:41:46.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410866+0000", 2024-04-22T18:41:46.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8499999999999998e-06 2024-04-22T18:41:46.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410868+0000", 2024-04-22T18:41:46.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.497e-06 2024-04-22T18:41:46.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615137+0000", 2024-04-22T18:41:46.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20426903499999999 2024-04-22T18:41:46.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615218+0000", 2024-04-22T18:41:46.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0860999999999997e-05 2024-04-22T18:41:46.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.362617+0000", 2024-04-22T18:41:46.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.74739882700000004 2024-04-22T18:41:46.030 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.030 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.030 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.030 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4194 4.b 4:d3068ee3:::smithi08937929-29:head [sparse-read 0~1980000,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:46.030 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:46.030 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.844296823000001, 2024-04-22T18:41:46.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.294164774, 2024-04-22T18:41:46.030 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.030 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.030 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.030 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.030 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.031 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4194 2024-04-22T18:41:46.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.031 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:46.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:46.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164120+0000", 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164125+0000", 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2899999999999996e-06 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164128+0000", 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8700000000000002e-06 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164132+0000", 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.151e-06 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164188+0000", 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6912e-05 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164201+0000", 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2466000000000001e-05 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.242879+0000", 2024-04-22T18:41:46.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0786779710000001 2024-04-22T18:41:46.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:46.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243364+0000", 2024-04-22T18:41:46.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000485081 2024-04-22T18:41:46.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456588+0000", 2024-04-22T18:41:46.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21322370900000001 2024-04-22T18:41:46.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456674+0000", 2024-04-22T18:41:46.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6079e-05 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.458285+0000", 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001611716 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4195 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.844279200999999, 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.395681808, 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.035 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4195 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164141+0000", 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7160000000000001e-06 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164147+0000", 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0509999999999996e-06 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264244+0000", 2024-04-22T18:41:46.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100097063 2024-04-22T18:41:46.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264319+0000", 2024-04-22T18:41:46.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5729000000000006e-05 2024-04-22T18:41:46.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264331+0000", 2024-04-22T18:41:46.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048999999999999e-05 2024-04-22T18:41:46.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243392+0000", 2024-04-22T18:41:46.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97906041099999996 2024-04-22T18:41:46.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:46.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243451+0000", 2024-04-22T18:41:46.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8717e-05 2024-04-22T18:41:46.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558398+0000", 2024-04-22T18:41:46.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31494718500000002 2024-04-22T18:41:46.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558490+0000", 2024-04-22T18:41:46.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2529999999999997e-05 2024-04-22T18:41:46.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559820+0000", 2024-04-22T18:41:46.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013299060000000001 2024-04-22T18:41:46.040 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.040 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.040 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.040 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4196 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:46.041 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:46.041 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.744140453999997, 2024-04-22T18:41:46.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2963829229999999, 2024-04-22T18:41:46.041 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.041 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.041 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.041 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.041 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.041 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4196 2024-04-22T18:41:46.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.041 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:46.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264275+0000", 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264290+0000", 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5373e-05 2024-04-22T18:41:46.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264300+0000", 2024-04-22T18:41:46.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.747e-06 2024-04-22T18:41:46.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264314+0000", 2024-04-22T18:41:46.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.396e-05 2024-04-22T18:41:46.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364450+0000", 2024-04-22T18:41:46.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013662500000001 2024-04-22T18:41:46.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364459+0000", 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0710000000000001e-06 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245690+0000", 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88123066400000005 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245766+0000", 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5938000000000004e-05 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559874+0000", 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31410827499999999 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559930+0000", 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5367e-05 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.560660+0000", 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073043800000000003 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.045 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.046 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4182 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:46.046 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:46.046 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.080457609, 2024-04-22T18:41:46.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2059652709999999, 2024-04-22T18:41:46.046 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.046 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.046 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.046 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.046 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.046 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4182 2024-04-22T18:41:46.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.046 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:46.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:46.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927959+0000", 2024-04-22T18:41:46.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927962+0000", 2024-04-22T18:41:46.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4269999999999998e-06 2024-04-22T18:41:46.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927968+0000", 2024-04-22T18:41:46.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:41:46.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028037+0000", 2024-04-22T18:41:46.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068544 2024-04-22T18:41:46.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028078+0000", 2024-04-22T18:41:46.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1523999999999997e-05 2024-04-22T18:41:46.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028087+0000", 2024-04-22T18:41:46.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3140000000000004e-06 2024-04-22T18:41:46.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.129951+0000", 2024-04-22T18:41:46.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1018649229999999 2024-04-22T18:41:46.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.130067+0000", 2024-04-22T18:41:46.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115222 2024-04-22T18:41:46.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.133925+0000", 2024-04-22T18:41:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00385849 2024-04-22T18:41:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4183 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.980367176999998, 2024-04-22T18:41:46.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1063017610000001, 2024-04-22T18:41:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4183 2024-04-22T18:41:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028049+0000", 2024-04-22T18:41:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028055+0000", 2024-04-22T18:41:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-04-22T18:41:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028064+0000", 2024-04-22T18:41:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2830000000000001e-06 2024-04-22T18:41:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028071+0000", 2024-04-22T18:41:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2740000000000002e-06 2024-04-22T18:41:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028126+0000", 2024-04-22T18:41:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4771999999999997e-05 2024-04-22T18:41:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028134+0000", 2024-04-22T18:41:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5599999999999994e-06 2024-04-22T18:41:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134109+0000", 2024-04-22T18:41:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.105974864 2024-04-22T18:41:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134169+0000", 2024-04-22T18:41:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0167999999999997e-05 2024-04-22T18:41:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134352+0000", 2024-04-22T18:41:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018293699999999999 2024-04-22T18:41:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4190 4.c 4:3227431c:::smithi08937929-35:head [sparse-read 0~1258562,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.980341674000002, 2024-04-22T18:41:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1098740030000001, 2024-04-22T18:41:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4190 2024-04-22T18:41:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028075+0000", 2024-04-22T18:41:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028080+0000", 2024-04-22T18:41:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.561e-06 2024-04-22T18:41:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028085+0000", 2024-04-22T18:41:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0150000000000003e-06 2024-04-22T18:41:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028090+0000", 2024-04-22T18:41:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.169e-06 2024-04-22T18:41:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028175+0000", 2024-04-22T18:41:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4814999999999993e-05 2024-04-22T18:41:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028181+0000", 2024-04-22T18:41:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5490000000000003e-06 2024-04-22T18:41:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134400+0000", 2024-04-22T18:41:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1062186869999999 2024-04-22T18:41:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134552+0000", 2024-04-22T18:41:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015210500000000001 2024-04-22T18:41:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137950+0000", 2024-04-22T18:41:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033978210000000001 2024-04-22T18:41:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4191 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.980258925999998, 2024-04-22T18:41:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.110917953, 2024-04-22T18:41:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4191 2024-04-22T18:41:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028157+0000", 2024-04-22T18:41:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028164+0000", 2024-04-22T18:41:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.511e-06 2024-04-22T18:41:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028172+0000", 2024-04-22T18:41:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0339999999999997e-06 2024-04-22T18:41:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028179+0000", 2024-04-22T18:41:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8630000000000002e-06 2024-04-22T18:41:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028202+0000", 2024-04-22T18:41:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3635999999999999e-05 2024-04-22T18:41:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028209+0000", 2024-04-22T18:41:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4629999999999998e-06 2024-04-22T18:41:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137970+0000", 2024-04-22T18:41:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1097613479999999 2024-04-22T18:41:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.138034+0000", 2024-04-22T18:41:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3551000000000002e-05 2024-04-22T18:41:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.139077+0000", 2024-04-22T18:41:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001042715 2024-04-22T18:41:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4192 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.980233626, 2024-04-22T18:41:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.212555542, 2024-04-22T18:41:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4192 2024-04-22T18:41:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028183+0000", 2024-04-22T18:41:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028187+0000", 2024-04-22T18:41:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6119999999999999e-06 2024-04-22T18:41:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028192+0000", 2024-04-22T18:41:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3489999999999999e-06 2024-04-22T18:41:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028197+0000", 2024-04-22T18:41:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7489999999999998e-06 2024-04-22T18:41:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028228+0000", 2024-04-22T18:41:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1328999999999998e-05 2024-04-22T18:41:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028234+0000", 2024-04-22T18:41:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1129999999999997e-06 2024-04-22T18:41:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239245+0000", 2024-04-22T18:41:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.211011032 2024-04-22T18:41:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239384+0000", 2024-04-22T18:41:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013902000000000001 2024-04-22T18:41:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.240739+0000", 2024-04-22T18:41:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013550389999999999 2024-04-22T18:41:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4692 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.749277747000001, 2024-04-22T18:41:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80332623700000005, 2024-04-22T18:41:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4692 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259139+0000", 2024-04-22T18:41:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259145+0000", 2024-04-22T18:41:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4860000000000002e-06 2024-04-22T18:41:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259151+0000", 2024-04-22T18:41:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1160000000000004e-06 2024-04-22T18:41:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259156+0000", 2024-04-22T18:41:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6489999999999999e-06 2024-04-22T18:41:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259191+0000", 2024-04-22T18:41:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4737999999999998e-05 2024-04-22T18:41:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259197+0000", 2024-04-22T18:41:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9160000000000004e-06 2024-04-22T18:41:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060062+0000", 2024-04-22T18:41:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80086472900000005 2024-04-22T18:41:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060225+0000", 2024-04-22T18:41:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000162771 2024-04-22T18:41:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062466+0000", 2024-04-22T18:41:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410569999999999 2024-04-22T18:41:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4693 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.749255461000001, 2024-04-22T18:41:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80370012999999996, 2024-04-22T18:41:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4693 2024-04-22T18:41:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259161+0000", 2024-04-22T18:41:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259164+0000", 2024-04-22T18:41:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-06 2024-04-22T18:41:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259169+0000", 2024-04-22T18:41:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6260000000000003e-06 2024-04-22T18:41:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259174+0000", 2024-04-22T18:41:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7659999999999998e-06 2024-04-22T18:41:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259231+0000", 2024-04-22T18:41:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6827000000000003e-05 2024-04-22T18:41:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259239+0000", 2024-04-22T18:41:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6889999999999993e-06 2024-04-22T18:41:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062492+0000", 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80325246699999997 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062573+0000", 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1508000000000006e-05 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062862+0000", 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028893499999999998 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4694 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.74923918, 2024-04-22T18:41:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.90474754099999999, 2024-04-22T18:41:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4694 2024-04-22T18:41:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259182+0000", 2024-04-22T18:41:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8700000000000002e-06 2024-04-22T18:41:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259187+0000", 2024-04-22T18:41:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1669999999999998e-06 2024-04-22T18:41:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259191+0000", 2024-04-22T18:41:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6210000000000003e-06 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.359347+0000", 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100155518 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.359356+0000", 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6519999999999995e-06 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163126+0000", 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.803770816 2024-04-22T18:41:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163217+0000", 2024-04-22T18:41:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0598000000000002e-05 2024-04-22T18:41:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163926+0000", 2024-04-22T18:41:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000708856 2024-04-22T18:41:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4791 4.7 4:e117a562:::smithi08937929-33:head [write 591712~786646] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.734465+0000", 2024-04-22T18:41:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.273952725999999, 2024-04-22T18:41:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4316495570000001, 2024-04-22T18:41:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4791 2024-04-22T18:41:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734465+0000", 2024-04-22T18:41:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734465+0000", 2024-04-22T18:41:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734460+0000", 2024-04-22T18:41:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999957 2024-04-22T18:41:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734848+0000", 2024-04-22T18:41:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000387353 2024-04-22T18:41:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734865+0000", 2024-04-22T18:41:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7486000000000002e-05 2024-04-22T18:41:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734875+0000", 2024-04-22T18:41:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-06 2024-04-22T18:41:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761489+0000", 2024-04-22T18:41:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026613405 2024-04-22T18:41:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761495+0000", 2024-04-22T18:41:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4350000000000002e-06 2024-04-22T18:41:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.961065+0000", 2024-04-22T18:41:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.199569608 2024-04-22T18:41:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.961155+0000", 2024-04-22T18:41:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0109000000000001e-05 2024-04-22T18:41:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.961254+0000", 2024-04-22T18:41:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9072000000000003e-05 2024-04-22T18:41:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.964390+0000", 2024-04-22T18:41:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031362009999999999 2024-04-22T18:41:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166027+0000", 2024-04-22T18:41:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.201636589 2024-04-22T18:41:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166075+0000", 2024-04-22T18:41:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7998999999999999e-05 2024-04-22T18:41:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166114+0000", 2024-04-22T18:41:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9638999999999998e-05 2024-04-22T18:41:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4792 4.7 4:e117a562:::smithi08937929-33:head [write 1943568~668798] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742160+0000", 2024-04-22T18:41:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.266257612, 2024-04-22T18:41:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.42404771, 2024-04-22T18:41:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4792 2024-04-22T18:41:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742160+0000", 2024-04-22T18:41:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742160+0000", 2024-04-22T18:41:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742158+0000", 2024-04-22T18:41:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742473+0000", 2024-04-22T18:41:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000315192 2024-04-22T18:41:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742485+0000", 2024-04-22T18:41:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1795e-05 2024-04-22T18:41:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742491+0000", 2024-04-22T18:41:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8760000000000002e-06 2024-04-22T18:41:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761511+0000", 2024-04-22T18:41:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019020760000000001 2024-04-22T18:41:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761515+0000", 2024-04-22T18:41:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.625e-06 2024-04-22T18:41:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.962761+0000", 2024-04-22T18:41:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.201246021 2024-04-22T18:41:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.962837+0000", 2024-04-22T18:41:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6025999999999993e-05 2024-04-22T18:41:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.962944+0000", 2024-04-22T18:41:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106949 2024-04-22T18:41:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.965124+0000", 2024-04-22T18:41:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021803230000000001 2024-04-22T18:41:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166154+0000", 2024-04-22T18:41:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20102966 2024-04-22T18:41:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166185+0000", 2024-04-22T18:41:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0713999999999998e-05 2024-04-22T18:41:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166208+0000", 2024-04-22T18:41:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2969999999999999e-05 2024-04-22T18:41:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4793 4.7 4:e117a562:::smithi08937929-33:head [setxattr _header (56) in=7b,truncate 2991712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742495+0000", 2024-04-22T18:41:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.265922700000001, 2024-04-22T18:41:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5238806949999999, 2024-04-22T18:41:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4793 2024-04-22T18:41:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742495+0000", 2024-04-22T18:41:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742495+0000", 2024-04-22T18:41:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742494+0000", 2024-04-22T18:41:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742498+0000", 2024-04-22T18:41:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3790000000000002e-06 2024-04-22T18:41:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742501+0000", 2024-04-22T18:41:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9900000000000002e-06 2024-04-22T18:41:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742504+0000", 2024-04-22T18:41:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:41:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761529+0000", 2024-04-22T18:41:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019025301000000001 2024-04-22T18:41:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761532+0000", 2024-04-22T18:41:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4589999999999998e-06 2024-04-22T18:41:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.963997+0000", 2024-04-22T18:41:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2024646560000001 2024-04-22T18:41:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.964075+0000", 2024-04-22T18:41:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8015000000000004e-05 2024-04-22T18:41:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.964176+0000", 2024-04-22T18:41:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010080499999999999 2024-04-22T18:41:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.965235+0000", 2024-04-22T18:41:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00105959 2024-04-22T18:41:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266295+0000", 2024-04-22T18:41:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30105968599999999 2024-04-22T18:41:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266330+0000", 2024-04-22T18:41:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5219999999999998e-05 2024-04-22T18:41:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266376+0000", 2024-04-22T18:41:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5262000000000001e-05 2024-04-22T18:41:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4794 4.7 4:e117a562:::smithi08937929-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:41:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.265911587, 2024-04-22T18:41:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.524010123, 2024-04-22T18:41:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4794 2024-04-22T18:41:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:41:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:41:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:41:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742507+0000", 2024-04-22T18:41:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.637e-06 2024-04-22T18:41:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742511+0000", 2024-04-22T18:41:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9890000000000003e-06 2024-04-22T18:41:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742514+0000", 2024-04-22T18:41:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.88e-06 2024-04-22T18:41:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.775057+0000", 2024-04-22T18:41:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.032543012000000003 2024-04-22T18:41:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.775069+0000", 2024-04-22T18:41:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2184000000000001e-05 2024-04-22T18:41:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.064059+0000", 2024-04-22T18:41:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.288990157 2024-04-22T18:41:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.064127+0000", 2024-04-22T18:41:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7741000000000001e-05 2024-04-22T18:41:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266405+0000", 2024-04-22T18:41:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20227735499999999 2024-04-22T18:41:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266458+0000", 2024-04-22T18:41:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3430000000000002e-05 2024-04-22T18:41:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266516+0000", 2024-04-22T18:41:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8038000000000002e-05 2024-04-22T18:41:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4795 4.1f 4.d9a85c1f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:41:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.265901217, 2024-04-22T18:41:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.118060236, 2024-04-22T18:41:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4795 2024-04-22T18:41:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:41:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:41:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:41:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742518+0000", 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.584e-06 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742520+0000", 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7140000000000002e-06 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742523+0000", 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3609999999999999e-06 2024-04-22T18:41:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761544+0000", 2024-04-22T18:41:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019020925000000001 2024-04-22T18:41:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761547+0000", 2024-04-22T18:41:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7859999999999998e-06 2024-04-22T18:41:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.860553+0000", 2024-04-22T18:41:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0990060859999999 2024-04-22T18:41:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.860577+0000", 2024-04-22T18:41:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3064000000000001e-05 2024-04-22T18:41:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.117 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:46.119 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:46.120 INFO:tasks.rados.rados.0.smithi089.stdout:2045: finishing write tid 1 to smithi08937929-45 2024-04-22T18:41:46.120 INFO:tasks.rados.rados.0.smithi089.stdout:2044: done (5 left) 2024-04-22T18:41:46.120 INFO:tasks.rados.rados.0.smithi089.stdout:2049: done (4 left) 2024-04-22T18:41:46.120 INFO:tasks.rados.rados.0.smithi089.stdout: waiting on 4 2024-04-22T18:41:46.121 INFO:tasks.rados.rados.0.smithi089.stdout:2046: expect (ObjNum 405 snap 0 seq_num 405) 2024-04-22T18:41:46.121 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:46.120+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:46.219 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:46.221+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:46.223 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:46.223 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:46.223 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:46.223 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:46.232 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:46.320 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:46.321+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:46.322 INFO:tasks.rados.rados.0.smithi089.stdout:2047: expect (ObjNum 431 snap 0 seq_num 431) 2024-04-22T18:41:46.375 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:46.375 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:46.375 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:46.375 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:46.375 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:46.384 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:46.425 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:46.426+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:46.522 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:46.523+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:46.527 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:46.527 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:46.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:46.527 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:46.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.528 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:41:46.528 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:46.528 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 162.06621081700001, 2024-04-22T18:41:46.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:41:46.528 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.528 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.528 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:46.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:46.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:41:46.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:46.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:41:46.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:41:46.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:41:46.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:41:46.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:41:46.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:41:46.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:41:46.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:41:46.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:41:46.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:41:46.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:41:46.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:41:46.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:41:46.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:41:46.532 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.532 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.532 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.532 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:41:46.532 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:46.532 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 158.189536995, 2024-04-22T18:41:46.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:41:46.532 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.532 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.532 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:46.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:46.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:41:46.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:41:46.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:41:46.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:41:46.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:41:46.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:41:46.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:41:46.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:41:46.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:41:46.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:41:46.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:41:46.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:41:46.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:41:46.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:41:46.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:41:46.536 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.536 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.536 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.536 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:41:46.536 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:46.536 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 101.984733472, 2024-04-22T18:41:46.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:41:46.537 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.537 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.537 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:46.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:46.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:41:46.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:41:46.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:41:46.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:41:46.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:41:46.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:41:46.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:41:46.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:41:46.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:41:46.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:41:46.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:41:46.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:41:46.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:41:46.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:41:46.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:41:46.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:41:46.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.540 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.540 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3451 4.d 4:bee1184f:::smithi08937929-7:head [read 0~3613505,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:46.541 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:46.541 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.600203144999995, 2024-04-22T18:41:46.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20164568099999999, 2024-04-22T18:41:46.541 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.541 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.541 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.541 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.541 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.541 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3451 2024-04-22T18:41:46.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.541 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:46.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:46.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:46.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927183+0000", 2024-04-22T18:41:46.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1110000000000001e-06 2024-04-22T18:41:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927188+0000", 2024-04-22T18:41:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1569999999999999e-06 2024-04-22T18:41:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927192+0000", 2024-04-22T18:41:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4710000000000001e-06 2024-04-22T18:41:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027382+0000", 2024-04-22T18:41:46.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100189341 2024-04-22T18:41:46.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:41:46.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027387+0000", 2024-04-22T18:41:46.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8400000000000002e-06 2024-04-22T18:41:46.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127638+0000", 2024-04-22T18:41:46.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10025102499999999 2024-04-22T18:41:46.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127740+0000", 2024-04-22T18:41:46.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102749 2024-04-22T18:41:46.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.128825+0000", 2024-04-22T18:41:46.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001084593 2024-04-22T18:41:46.545 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.545 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.545 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.545 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:151 3.0 e114/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 114'213, mlcod=110'210)", 2024-04-22T18:41:46.545 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:46.545 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.009735061999997, 2024-04-22T18:41:46.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20074001299999999, 2024-04-22T18:41:46.545 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.546 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.546 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:46.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:46.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517640+0000", 2024-04-22T18:41:46.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999928 2024-04-22T18:41:46.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517652+0000", 2024-04-22T18:41:46.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1824000000000001e-05 2024-04-22T18:41:46.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517660+0000", 2024-04-22T18:41:46.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1419999999999993e-06 2024-04-22T18:41:46.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617729+0000", 2024-04-22T18:41:46.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068723 2024-04-22T18:41:46.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617770+0000", 2024-04-22T18:41:46.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1391999999999999e-05 2024-04-22T18:41:46.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617793+0000", 2024-04-22T18:41:46.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2807999999999999e-05 2024-04-22T18:41:46.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718361+0000", 2024-04-22T18:41:46.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100567473 2024-04-22T18:41:46.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718387+0000", 2024-04-22T18:41:46.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6826e-05 2024-04-22T18:41:46.549 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.549 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.550 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.550 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3862 4.d 4:bee1184f:::smithi08937929-7:head [write 3301257~278667] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:46.550 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:46.550 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 57.264832792, 2024-04-22T18:41:46.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301875958, 2024-04-22T18:41:46.550 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.550 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.550 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.550 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.550 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.550 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3862 2024-04-22T18:41:46.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.551 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:46.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:46.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262548+0000", 2024-04-22T18:41:46.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:46.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262622+0000", 2024-04-22T18:41:46.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3747999999999999e-05 2024-04-22T18:41:46.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262632+0000", 2024-04-22T18:41:46.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0767e-05 2024-04-22T18:41:46.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362705+0000", 2024-04-22T18:41:46.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073157 2024-04-22T18:41:46.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:41:46.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2846999999999997e-05 2024-04-22T18:41:46.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362853+0000", 2024-04-22T18:41:46.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4498000000000002e-05 2024-04-22T18:41:46.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:46.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362992+0000", 2024-04-22T18:41:46.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013879699999999999 2024-04-22T18:41:46.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:46.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.366534+0000", 2024-04-22T18:41:46.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035427969999999999 2024-04-22T18:41:46.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:46.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564353+0000", 2024-04-22T18:41:46.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.197818893 2024-04-22T18:41:46.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564385+0000", 2024-04-22T18:41:46.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1717000000000001e-05 2024-04-22T18:41:46.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564426+0000", 2024-04-22T18:41:46.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0654000000000001e-05 2024-04-22T18:41:46.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.555 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.555 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3863 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3579924] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:46.556 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:46.556 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 57.164665399999997, 2024-04-22T18:41:46.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20206560900000001, 2024-04-22T18:41:46.556 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.556 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.556 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.556 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.556 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.556 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3863 2024-04-22T18:41:46.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.556 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:46.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:46.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362715+0000", 2024-04-22T18:41:46.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:46.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362725+0000", 2024-04-22T18:41:46.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.006e-05 2024-04-22T18:41:46.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362732+0000", 2024-04-22T18:41:46.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6780000000000003e-06 2024-04-22T18:41:46.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362737+0000", 2024-04-22T18:41:46.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2089999999999996e-06 2024-04-22T18:41:46.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462875+0000", 2024-04-22T18:41:46.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013834100000001 2024-04-22T18:41:46.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462963+0000", 2024-04-22T18:41:46.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7792000000000002e-05 2024-04-22T18:41:46.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:46.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463127+0000", 2024-04-22T18:41:46.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000163688 2024-04-22T18:41:46.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:46.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463692+0000", 2024-04-22T18:41:46.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056466299999999997 2024-04-22T18:41:46.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:46.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564720+0000", 2024-04-22T18:41:46.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10102826199999999 2024-04-22T18:41:46.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564752+0000", 2024-04-22T18:41:46.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1936000000000001e-05 2024-04-22T18:41:46.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564783+0000", 2024-04-22T18:41:46.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0919e-05 2024-04-22T18:41:46.561 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.561 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.561 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.561 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3864 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:46.561 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:46.562 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 57.164632015999999, 2024-04-22T18:41:46.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20218423599999999, 2024-04-22T18:41:46.562 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.562 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.562 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.562 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.562 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.562 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3864 2024-04-22T18:41:46.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.562 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:46.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:46.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:46.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362753+0000", 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6989999999999999e-06 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462871+0000", 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100113641 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463385+0000", 2024-04-22T18:41:46.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051307100000000003 2024-04-22T18:41:46.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:46.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463457+0000", 2024-04-22T18:41:46.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2778e-05 2024-04-22T18:41:46.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564814+0000", 2024-04-22T18:41:46.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10135696299999999 2024-04-22T18:41:46.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564868+0000", 2024-04-22T18:41:46.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3872000000000002e-05 2024-04-22T18:41:46.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564935+0000", 2024-04-22T18:41:46.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6545999999999996e-05 2024-04-22T18:41:46.566 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.566 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.566 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.566 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4030 4.d 4:bee1184f:::smithi08937929-7:head [write 615222~472039] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:46.566 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:46.566 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.568183114, 2024-04-22T18:41:46.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109028217, 2024-04-22T18:41:46.567 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.567 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.567 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.567 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.567 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.567 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4030 2024-04-22T18:41:46.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.567 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:46.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:46.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959177+0000", 2024-04-22T18:41:46.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999776 2024-04-22T18:41:46.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959331+0000", 2024-04-22T18:41:46.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015334600000000001 2024-04-22T18:41:46.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959358+0000", 2024-04-22T18:41:46.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7220999999999999e-05 2024-04-22T18:41:46.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959371+0000", 2024-04-22T18:41:46.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3489000000000001e-05 2024-04-22T18:41:46.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059523+0000", 2024-04-22T18:41:46.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100152059 2024-04-22T18:41:46.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059588+0000", 2024-04-22T18:41:46.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4493999999999996e-05 2024-04-22T18:41:46.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:46.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059694+0000", 2024-04-22T18:41:46.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010584199999999999 2024-04-22T18:41:46.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:46.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068082+0000", 2024-04-22T18:41:46.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083880589999999998 2024-04-22T18:41:46.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:46.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068138+0000", 2024-04-22T18:41:46.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6196999999999997e-05 2024-04-22T18:41:46.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068185+0000", 2024-04-22T18:41:46.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7318000000000003e-05 2024-04-22T18:41:46.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068228+0000", 2024-04-22T18:41:46.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2376999999999998e-05 2024-04-22T18:41:46.572 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.572 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.572 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.572 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4031 4.d 4:bee1184f:::smithi08937929-7:head [write 1580652~489442] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:46.572 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:46.572 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.562882121999998, 2024-04-22T18:41:46.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20571598299999999, 2024-04-22T18:41:46.572 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.572 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.572 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.572 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.573 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.573 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4031 2024-04-22T18:41:46.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.573 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:46.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:46.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964480+0000", 2024-04-22T18:41:46.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999795 2024-04-22T18:41:46.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964628+0000", 2024-04-22T18:41:46.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148109 2024-04-22T18:41:46.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964651+0000", 2024-04-22T18:41:46.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3215999999999999e-05 2024-04-22T18:41:46.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964664+0000", 2024-04-22T18:41:46.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2673e-05 2024-04-22T18:41:46.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061135+0000", 2024-04-22T18:41:46.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096470822999999997 2024-04-22T18:41:46.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061228+0000", 2024-04-22T18:41:46.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3459000000000001e-05 2024-04-22T18:41:46.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:46.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061376+0000", 2024-04-22T18:41:46.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148128 2024-04-22T18:41:46.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:46.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068085+0000", 2024-04-22T18:41:46.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.006708632 2024-04-22T18:41:46.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:46.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170038+0000", 2024-04-22T18:41:46.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101952716 2024-04-22T18:41:46.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170173+0000", 2024-04-22T18:41:46.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013508699999999999 2024-04-22T18:41:46.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170216+0000", 2024-04-22T18:41:46.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3498999999999998e-05 2024-04-22T18:41:46.577 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.577 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.577 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.578 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4032 4.d 4:bee1184f:::smithi08937929-7:head [write 2768826~626587] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:46.578 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:46.578 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.555677015000001, 2024-04-22T18:41:46.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19941629799999999, 2024-04-22T18:41:46.578 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.578 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.578 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.578 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.578 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.578 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4032 2024-04-22T18:41:46.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.578 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:46.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:46.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971703+0000", 2024-04-22T18:41:46.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:46.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971996+0000", 2024-04-22T18:41:46.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029351300000000002 2024-04-22T18:41:46.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972012+0000", 2024-04-22T18:41:46.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5569e-05 2024-04-22T18:41:46.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972019+0000", 2024-04-22T18:41:46.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7950000000000008e-06 2024-04-22T18:41:46.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.063940+0000", 2024-04-22T18:41:46.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091920162 2024-04-22T18:41:46.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064021+0000", 2024-04-22T18:41:46.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1865000000000003e-05 2024-04-22T18:41:46.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:46.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064145+0000", 2024-04-22T18:41:46.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123893 2024-04-22T18:41:46.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:46.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068317+0000", 2024-04-22T18:41:46.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041720799999999999 2024-04-22T18:41:46.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:46.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171042+0000", 2024-04-22T18:41:46.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102724893 2024-04-22T18:41:46.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171085+0000", 2024-04-22T18:41:46.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2619e-05 2024-04-22T18:41:46.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171122+0000", 2024-04-22T18:41:46.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6838999999999997e-05 2024-04-22T18:41:46.583 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.583 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.583 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.583 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4033 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3815222] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:46.583 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:46.583 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.555353533000002, 2024-04-22T18:41:46.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297586038, 2024-04-22T18:41:46.584 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.584 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.584 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.584 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.584 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.584 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4033 2024-04-22T18:41:46.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.584 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:46.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:46.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972028+0000", 2024-04-22T18:41:46.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972032+0000", 2024-04-22T18:41:46.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8330000000000004e-06 2024-04-22T18:41:46.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972035+0000", 2024-04-22T18:41:46.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:41:46.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972038+0000", 2024-04-22T18:41:46.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9579999999999999e-06 2024-04-22T18:41:46.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067511+0000", 2024-04-22T18:41:46.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.095472308000000006 2024-04-22T18:41:46.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067607+0000", 2024-04-22T18:41:46.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6546999999999996e-05 2024-04-22T18:41:46.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:46.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067776+0000", 2024-04-22T18:41:46.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000169202 2024-04-22T18:41:46.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:46.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068470+0000", 2024-04-22T18:41:46.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069403100000000001 2024-04-22T18:41:46.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:46.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269540+0000", 2024-04-22T18:41:46.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20106934000000001 2024-04-22T18:41:46.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269574+0000", 2024-04-22T18:41:46.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4332000000000003e-05 2024-04-22T18:41:46.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269615+0000", 2024-04-22T18:41:46.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0988000000000001e-05 2024-04-22T18:41:46.589 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.589 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.589 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.589 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4034 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:46.589 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:46.589 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.555341896000002, 2024-04-22T18:41:46.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39788621000000002, 2024-04-22T18:41:46.589 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.589 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.589 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.589 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.589 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.590 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4034 2024-04-22T18:41:46.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.590 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:46.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:46.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972040+0000", 2024-04-22T18:41:46.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972042+0000", 2024-04-22T18:41:46.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.519e-06 2024-04-22T18:41:46.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972045+0000", 2024-04-22T18:41:46.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:41:46.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972048+0000", 2024-04-22T18:41:46.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.773e-06 2024-04-22T18:41:46.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068266+0000", 2024-04-22T18:41:46.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096218029999999996 2024-04-22T18:41:46.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:46.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068308+0000", 2024-04-22T18:41:46.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.21e-05 2024-04-22T18:41:46.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369703+0000", 2024-04-22T18:41:46.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301395578 2024-04-22T18:41:46.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369802+0000", 2024-04-22T18:41:46.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8788000000000004e-05 2024-04-22T18:41:46.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369927+0000", 2024-04-22T18:41:46.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012478200000000001 2024-04-22T18:41:46.593 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.594 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.594 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.594 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(4.1 124/123 [PushOp(4:83c955bf:::smithi08937929-28:head, version: 115'157, data_included: [1890685~548185,2699879~469889], data_size: 1018074, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:83c955bf:::smithi08937929-28:head@115'157, size: 3630345, copy_subset: [0~3630345], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3630345, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1890685, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-22T18:41:46.594 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:46.594 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 47.888773788000002, 2024-04-22T18:41:46.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20293218800000001, 2024-04-22T18:41:46.594 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.594 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.594 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638608+0000", 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639519+0000", 2024-04-22T18:41:46.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091196199999999999 2024-04-22T18:41:46.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639537+0000", 2024-04-22T18:41:46.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7455000000000001e-05 2024-04-22T18:41:46.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.739639+0000", 2024-04-22T18:41:46.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100101942 2024-04-22T18:41:46.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840271+0000", 2024-04-22T18:41:46.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10063235299999999 2024-04-22T18:41:46.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840281+0000", 2024-04-22T18:41:46.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6509999999999996e-06 2024-04-22T18:41:46.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.841541+0000", 2024-04-22T18:41:46.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012600230000000001 2024-04-22T18:41:46.597 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.597 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.597 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.598 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4205 4.1 4:83c955bf:::smithi08937929-28:head [read 3065344~859648] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:46.598 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:46.598 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.599851946999998, 2024-04-22T18:41:46.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.303433565, 2024-04-22T18:41:46.598 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.598 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:46.598 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:46.598 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:46.599 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:46.599 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4205 2024-04-22T18:41:46.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.599 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927528+0000", 2024-04-22T18:41:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927535+0000", 2024-04-22T18:41:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1049999999999997e-06 2024-04-22T18:41:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927550+0000", 2024-04-22T18:41:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4365000000000001e-05 2024-04-22T18:41:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927560+0000", 2024-04-22T18:41:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0360000000000001e-05 2024-04-22T18:41:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028542+0000", 2024-04-22T18:41:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100981837 2024-04-22T18:41:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028551+0000", 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6260000000000001e-06 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229700+0000", 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.201149453 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229861+0000", 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160605 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.230964+0000", 2024-04-22T18:41:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00110351 2024-04-22T18:41:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:359 3.0 e133/63 3:351d2474:::ba270d71d9c7ccc8d5c74eded70a7073f0e08c1ce33f7199f7c9db1158888734:head v 133'233, mlcod=132'228)", 2024-04-22T18:41:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.625057519999999, 2024-04-22T18:41:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20122315599999999, 2024-04-22T18:41:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902324+0000", 2024-04-22T18:41:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902347+0000", 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3577000000000001e-05 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902356+0000", 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8179999999999993e-06 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.002447+0000", 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100090255 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102568+0000", 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100121611 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102592+0000", 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3515000000000001e-05 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103514+0000", 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000922174 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103548+0000", 2024-04-22T18:41:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4316999999999997e-05 2024-04-22T18:41:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:4289 3.0 e134/63 3:2a3d8f37:::9341e24c4a92f7ec719282d2faa5711f9644dee45a94ba5b4c4c1071ddc0e536:head v 134'235, mlcod=133'233)", 2024-04-22T18:41:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.680488007999998, 2024-04-22T18:41:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10669205499999999, 2024-04-22T18:41:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846891+0000", 2024-04-22T18:41:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847884+0000", 2024-04-22T18:41:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00099224800000000004 2024-04-22T18:41:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847900+0000", 2024-04-22T18:41:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6154000000000002e-05 2024-04-22T18:41:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.947999+0000", 2024-04-22T18:41:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009889199999999 2024-04-22T18:41:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948046+0000", 2024-04-22T18:41:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7617000000000002e-05 2024-04-22T18:41:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948072+0000", 2024-04-22T18:41:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5559999999999999e-05 2024-04-22T18:41:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953539+0000", 2024-04-22T18:41:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054668599999999996 2024-04-22T18:41:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953587+0000", 2024-04-22T18:41:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7781999999999998e-05 2024-04-22T18:41:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.7.13:94 3.0 e147/63 3:2812810a:::007b860ed7f2dc72f975d8732f59ce14b4563df44fd56cdf3a173500adaeb557:head v 147'260, mlcod=147'257)", 2024-04-22T18:41:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:41:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.094094802000001, 2024-04-22T18:41:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.200869664, 2024-04-22T18:41:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:41:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:41:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433287+0000", 2024-04-22T18:41:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433295+0000", 2024-04-22T18:41:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9929999999999995e-06 2024-04-22T18:41:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433302+0000", 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2740000000000002e-06 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533368+0000", 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100066239 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533415+0000", 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6669000000000003e-05 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533442+0000", 2024-04-22T18:41:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7143000000000001e-05 2024-04-22T18:41:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.634126+0000", 2024-04-22T18:41:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10068434900000001 2024-04-22T18:41:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.634157+0000", 2024-04-22T18:41:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1016000000000003e-05 2024-04-22T18:41:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:198 3.0 e147/63 3:2be6c676:::11b6849070c44d718872a26a0dc7d7224d7a6a98b28e0fde07a9f66143e32de6:head v 147'266, mlcod=147'257)", 2024-04-22T18:41:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:41:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.993322708000001, 2024-04-22T18:41:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106653649, 2024-04-22T18:41:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:41:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:41:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534059+0000", 2024-04-22T18:41:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534082+0000", 2024-04-22T18:41:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2291999999999999e-05 2024-04-22T18:41:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534086+0000", 2024-04-22T18:41:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1019999999999999e-06 2024-04-22T18:41:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534088+0000", 2024-04-22T18:41:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.525e-06 2024-04-22T18:41:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.635447+0000", 2024-04-22T18:41:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101358581 2024-04-22T18:41:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.635462+0000", 2024-04-22T18:41:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5327000000000001e-05 2024-04-22T18:41:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.640689+0000", 2024-04-22T18:41:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052266329999999996 2024-04-22T18:41:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.640713+0000", 2024-04-22T18:41:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4630999999999998e-05 2024-04-22T18:41:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:200 3.0 e147/63 3:19f9e72c:::45a155ef5bb5a968c7feaf289313c887c41e2de64b61f8816d05ad72d02346d9:head v 147'267, mlcod=147'257)", 2024-04-22T18:41:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:41:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.993291951, 2024-04-22T18:41:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109105273, 2024-04-22T18:41:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:41:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:41:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534090+0000", 2024-04-22T18:41:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534228+0000", 2024-04-22T18:41:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013768600000000001 2024-04-22T18:41:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534232+0000", 2024-04-22T18:41:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4900000000000002e-06 2024-04-22T18:41:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534235+0000", 2024-04-22T18:41:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.61e-06 2024-04-22T18:41:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.641544+0000", 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107308717 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.641557+0000", 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2811e-05 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.643173+0000", 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00161608 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:46.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:46.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.643196+0000", 2024-04-22T18:41:46.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3136e-05 2024-04-22T18:41:46.626 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.626 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.626 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.626 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:46.626 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:46.626 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:46.628 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:46.628 INFO:tasks.rados.rados.0.smithi089.stdout:2045: finishing write tid 2 to smithi08937929-45 2024-04-22T18:41:46.628 INFO:tasks.rados.rados.0.smithi089.stdout:2045: finishing write tid 3 to smithi08937929-45 2024-04-22T18:41:46.628 INFO:tasks.rados.rados.0.smithi089.stdout:2045: finishing write tid 4 to smithi08937929-45 2024-04-22T18:41:46.628 INFO:tasks.rados.rados.0.smithi089.stdout:2045: finishing write tid 5 to smithi08937929-45 2024-04-22T18:41:46.629 INFO:tasks.rados.rados.0.smithi089.stdout:2045: oid 45 updating version 0 to 108 2024-04-22T18:41:46.629 INFO:tasks.rados.rados.0.smithi089.stdout:2045: oid 45 updating version 108 to 109 2024-04-22T18:41:46.629 INFO:tasks.rados.rados.0.smithi089.stdout:2045: oid 45 updating version 109 to 110 2024-04-22T18:41:46.629 INFO:tasks.rados.rados.0.smithi089.stdout:2045: oid 45 updating version 110 to 111 2024-04-22T18:41:46.629 INFO:tasks.rados.rados.0.smithi089.stdout:update_object_version oid 45 v 111 (ObjNum 442 snap 0 seq_num 442) dirty exists 2024-04-22T18:41:46.629 INFO:tasks.rados.rados.0.smithi089.stdout:2045: left oid 45 (ObjNum 442 snap 0 seq_num 442) 2024-04-22T18:41:46.629 INFO:tasks.rados.rados.0.smithi089.stdout:2045: done (3 left) 2024-04-22T18:41:46.629 INFO:tasks.rados.rados.0.smithi089.stdout:2046: done (2 left) 2024-04-22T18:41:46.629 INFO:tasks.rados.rados.0.smithi089.stdout:2047: done (1 left) 2024-04-22T18:41:46.629 INFO:tasks.rados.rados.0.smithi089.stdout:2048: done (0 left) 2024-04-22T18:41:46.629 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:46.629+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:46.657 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:41:46.728 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:46.730+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:46.753 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:46.754+0000 7fec57bee640 -1 osd.7 169 handle_osd_map injecting map crc failure 2024-04-22T18:41:46.762 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:46.763 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:46.763 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:46.763 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:46.772 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:46.855 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:46.857+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:46.913 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:46.914 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:46.914 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:46.914 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:46.914 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:46.922 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:46.964 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:46.965+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:47.031 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:47.032+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:47.066 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:47.369 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:47.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:47.370 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:47.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.370 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:289 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:47.370 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:47.370 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.425069147, 2024-04-22T18:41:47.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9176801189999999, 2024-04-22T18:41:47.370 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.370 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.370 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:47.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:47.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640723+0000", 2024-04-22T18:41:47.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:47.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640732+0000", 2024-04-22T18:41:47.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5809999999999997e-06 2024-04-22T18:41:47.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640744+0000", 2024-04-22T18:41:47.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.261e-05 2024-04-22T18:41:47.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640753+0000", 2024-04-22T18:41:47.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7460000000000006e-06 2024-04-22T18:41:47.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642189+0000", 2024-04-22T18:41:47.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001435551 2024-04-22T18:41:47.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:47.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642200+0000", 2024-04-22T18:41:47.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1389999999999999e-05 2024-04-22T18:41:47.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.847015+0000", 2024-04-22T18:41:47.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048149160000001 2024-04-22T18:41:47.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.848217+0000", 2024-04-22T18:41:47.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012020659999999999 2024-04-22T18:41:47.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351823+0000", 2024-04-22T18:41:47.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50360565099999999 2024-04-22T18:41:47.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351907+0000", 2024-04-22T18:41:47.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4141999999999995e-05 2024-04-22T18:41:47.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:47.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352051+0000", 2024-04-22T18:41:47.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014386600000000001 2024-04-22T18:41:47.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453617+0000", 2024-04-22T18:41:47.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156649500000001 2024-04-22T18:41:47.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558297+0000", 2024-04-22T18:41:47.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10468000399999999 2024-04-22T18:41:47.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558331+0000", 2024-04-22T18:41:47.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3518e-05 2024-04-22T18:41:47.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558405+0000", 2024-04-22T18:41:47.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4221000000000006e-05 2024-04-22T18:41:47.376 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.376 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.376 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.376 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:1 3.2 3:6d90b903:::64b5942574fb36ca1b9a2be186c6b740edf58e989056a2f88b83cd280451a438:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:47.377 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:47.377 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 103.89376818700001, 2024-04-22T18:41:47.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.285534422, 2024-04-22T18:41:47.377 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.377 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.377 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:47.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:47.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172024+0000", 2024-04-22T18:41:47.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:47.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173091+0000", 2024-04-22T18:41:47.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010677569999999999 2024-04-22T18:41:47.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173103+0000", 2024-04-22T18:41:47.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1241e-05 2024-04-22T18:41:47.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173109+0000", 2024-04-22T18:41:47.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4300000000000003e-06 2024-04-22T18:41:47.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173168+0000", 2024-04-22T18:41:47.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9011000000000001e-05 2024-04-22T18:41:47.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:47.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173177+0000", 2024-04-22T18:41:47.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-04-22T18:41:47.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948501+0000", 2024-04-22T18:41:47.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77532387199999997 2024-04-22T18:41:47.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948912+0000", 2024-04-22T18:41:47.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041183000000000002 2024-04-22T18:41:47.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950438+0000", 2024-04-22T18:41:47.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015253199999999999 2024-04-22T18:41:47.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950488+0000", 2024-04-22T18:41:47.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0285999999999999e-05 2024-04-22T18:41:47.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:47.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950601+0000", 2024-04-22T18:41:47.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011244899999999999 2024-04-22T18:41:47.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954048+0000", 2024-04-22T18:41:47.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003447313 2024-04-22T18:41:47.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457373+0000", 2024-04-22T18:41:47.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50332465299999996 2024-04-22T18:41:47.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457430+0000", 2024-04-22T18:41:47.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7574999999999997e-05 2024-04-22T18:41:47.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457560+0000", 2024-04-22T18:41:47.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013023900000000001 2024-04-22T18:41:47.383 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.383 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.383 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.383 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:8 3.2 3:5f64590d:::7c47933f331d40689081b2bd8f9d7f7bdae1be1fdccb52d696047a3969e902b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:47.383 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:47.383 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 103.892675818, 2024-04-22T18:41:47.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3854200569999999, 2024-04-22T18:41:47.383 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.383 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.384 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:47.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:47.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173117+0000", 2024-04-22T18:41:47.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:47.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173393+0000", 2024-04-22T18:41:47.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000275278 2024-04-22T18:41:47.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173399+0000", 2024-04-22T18:41:47.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4590000000000003e-06 2024-04-22T18:41:47.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173406+0000", 2024-04-22T18:41:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2450000000000001e-06 2024-04-22T18:41:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173795+0000", 2024-04-22T18:41:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038912400000000002 2024-04-22T18:41:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173804+0000", 2024-04-22T18:41:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-04-22T18:41:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.949506+0000", 2024-04-22T18:41:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77570185700000005 2024-04-22T18:41:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950166+0000", 2024-04-22T18:41:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066022800000000001 2024-04-22T18:41:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352843+0000", 2024-04-22T18:41:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40267661799999999 2024-04-22T18:41:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352911+0000", 2024-04-22T18:41:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8126000000000005e-05 2024-04-22T18:41:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.353031+0000", 2024-04-22T18:41:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119919 2024-04-22T18:41:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453622+0000", 2024-04-22T18:41:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100590793 2024-04-22T18:41:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558446+0000", 2024-04-22T18:41:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104824073 2024-04-22T18:41:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558479+0000", 2024-04-22T18:41:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3362999999999999e-05 2024-04-22T18:41:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558538+0000", 2024-04-22T18:41:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9046999999999999e-05 2024-04-22T18:41:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.390 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:10 3.2 3:518c37f9:::851eca84dafbf2f780c4f07291110c453349bdc9c623c17bdcd26e27badb30ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:47.390 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:47.390 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 103.89238222500001, 2024-04-22T18:41:47.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3852527729999999, 2024-04-22T18:41:47.390 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.390 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.390 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:47.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:47.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173411+0000", 2024-04-22T18:41:47.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:47.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173749+0000", 2024-04-22T18:41:47.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000338029 2024-04-22T18:41:47.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173755+0000", 2024-04-22T18:41:47.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0530000000000003e-06 2024-04-22T18:41:47.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173759+0000", 2024-04-22T18:41:47.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8190000000000002e-06 2024-04-22T18:41:47.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173855+0000", 2024-04-22T18:41:47.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5927999999999994e-05 2024-04-22T18:41:47.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:47.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173862+0000", 2024-04-22T18:41:47.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4449999999999999e-06 2024-04-22T18:41:47.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950835+0000", 2024-04-22T18:41:47.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77697248299999999 2024-04-22T18:41:47.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951632+0000", 2024-04-22T18:41:47.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079752299999999996 2024-04-22T18:41:47.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355407+0000", 2024-04-22T18:41:47.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40377480500000001 2024-04-22T18:41:47.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355462+0000", 2024-04-22T18:41:47.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4577999999999998e-05 2024-04-22T18:41:47.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:47.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355571+0000", 2024-04-22T18:41:47.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108964 2024-04-22T18:41:47.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453625+0000", 2024-04-22T18:41:47.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098054397000000001 2024-04-22T18:41:47.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558574+0000", 2024-04-22T18:41:47.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104948571 2024-04-22T18:41:47.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558602+0000", 2024-04-22T18:41:47.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7966999999999998e-05 2024-04-22T18:41:47.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558665+0000", 2024-04-22T18:41:47.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3032000000000002e-05 2024-04-22T18:41:47.396 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.396 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.396 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.396 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:11 3.2 3:718aa8aa:::9603c4f8e269a436b2f0bf376c5374e77bdd7cf58fccae60e4e65ac4696806d4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:47.396 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:47.396 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 103.892030607, 2024-04-22T18:41:47.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4852604440000001, 2024-04-22T18:41:47.396 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.396 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:47.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173813+0000", 2024-04-22T18:41:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0655999999999997e-05 2024-04-22T18:41:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173819+0000", 2024-04-22T18:41:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.468e-06 2024-04-22T18:41:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173822+0000", 2024-04-22T18:41:47.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4879999999999999e-06 2024-04-22T18:41:47.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173983+0000", 2024-04-22T18:41:47.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160519 2024-04-22T18:41:47.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:47.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173991+0000", 2024-04-22T18:41:47.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6490000000000008e-06 2024-04-22T18:41:47.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951688+0000", 2024-04-22T18:41:47.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77769743800000002 2024-04-22T18:41:47.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952419+0000", 2024-04-22T18:41:47.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073067499999999997 2024-04-22T18:41:47.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355901+0000", 2024-04-22T18:41:47.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40348237599999998 2024-04-22T18:41:47.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355953+0000", 2024-04-22T18:41:47.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1477000000000002e-05 2024-04-22T18:41:47.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:47.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356060+0000", 2024-04-22T18:41:47.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107114 2024-04-22T18:41:47.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453629+0000", 2024-04-22T18:41:47.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097569127000000005 2024-04-22T18:41:47.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658807+0000", 2024-04-22T18:41:47.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20517829800000001 2024-04-22T18:41:47.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658876+0000", 2024-04-22T18:41:47.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8454000000000006e-05 2024-04-22T18:41:47.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659024+0000", 2024-04-22T18:41:47.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148338 2024-04-22T18:41:47.402 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.402 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.402 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.403 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:12 3.2 3:475e1cd9:::efaab2bd13543cdbb143e869e20eb0ec424f7f5202707859b241ed91d4674936:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:47.403 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:47.403 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 103.891967663, 2024-04-22T18:41:47.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4853405259999999, 2024-04-22T18:41:47.403 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.403 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.403 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:47.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:47.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:47.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:47.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173940+0000", 2024-04-22T18:41:47.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011465500000000001 2024-04-22T18:41:47.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173946+0000", 2024-04-22T18:41:47.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5289999999999999e-06 2024-04-22T18:41:47.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173950+0000", 2024-04-22T18:41:47.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8290000000000001e-06 2024-04-22T18:41:47.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174277+0000", 2024-04-22T18:41:47.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032761299999999998 2024-04-22T18:41:47.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:47.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174285+0000", 2024-04-22T18:41:47.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.52e-06 2024-04-22T18:41:47.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952455+0000", 2024-04-22T18:41:47.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77817000700000005 2024-04-22T18:41:47.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954035+0000", 2024-04-22T18:41:47.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001580078 2024-04-22T18:41:47.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356379+0000", 2024-04-22T18:41:47.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40234440399999999 2024-04-22T18:41:47.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356428+0000", 2024-04-22T18:41:47.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8832999999999998e-05 2024-04-22T18:41:47.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:47.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356532+0000", 2024-04-22T18:41:47.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103235 2024-04-22T18:41:47.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453632+0000", 2024-04-22T18:41:47.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097100346000000004 2024-04-22T18:41:47.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659069+0000", 2024-04-22T18:41:47.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20543763800000001 2024-04-22T18:41:47.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659105+0000", 2024-04-22T18:41:47.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5240000000000001e-05 2024-04-22T18:41:47.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659167+0000", 2024-04-22T18:41:47.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2207999999999998e-05 2024-04-22T18:41:47.409 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.409 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.409 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.409 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:16 3.2 3:47c76cc8:::c00d18458eb20c0c0aa9c612cf9793a68445ab0ea7b9e650ecd0072c60a96c0f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:47.409 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:47.409 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 103.89183970400001, 2024-04-22T18:41:47.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.485347744, 2024-04-22T18:41:47.410 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.410 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.410 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:47.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:47.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:47.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:47.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174234+0000", 2024-04-22T18:41:47.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028047599999999999 2024-04-22T18:41:47.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174240+0000", 2024-04-22T18:41:47.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2079999999999997e-06 2024-04-22T18:41:47.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174244+0000", 2024-04-22T18:41:47.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5810000000000001e-06 2024-04-22T18:41:47.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174670+0000", 2024-04-22T18:41:47.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042640000000000001 2024-04-22T18:41:47.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:47.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174675+0000", 2024-04-22T18:41:47.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8709999999999996e-06 2024-04-22T18:41:47.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054168+0000", 2024-04-22T18:41:47.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87949257300000006 2024-04-22T18:41:47.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054877+0000", 2024-04-22T18:41:47.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070884300000000002 2024-04-22T18:41:47.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356836+0000", 2024-04-22T18:41:47.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30195926000000001 2024-04-22T18:41:47.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356887+0000", 2024-04-22T18:41:47.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0767999999999999e-05 2024-04-22T18:41:47.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:47.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356991+0000", 2024-04-22T18:41:47.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104072 2024-04-22T18:41:47.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453636+0000", 2024-04-22T18:41:47.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096644968999999997 2024-04-22T18:41:47.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659207+0000", 2024-04-22T18:41:47.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20557154699999999 2024-04-22T18:41:47.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659240+0000", 2024-04-22T18:41:47.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2665e-05 2024-04-22T18:41:47.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659302+0000", 2024-04-22T18:41:47.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2234000000000001e-05 2024-04-22T18:41:47.416 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.416 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.416 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.416 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:17 3.2 3:6ef31daa:::1cf4ff496c4480e58510f607481d976b47d84868e2c54016273c38baf590f3b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:47.416 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:47.416 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 103.891545424, 2024-04-22T18:41:47.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4851703729999999, 2024-04-22T18:41:47.416 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.416 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.416 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:47.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:47.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174248+0000", 2024-04-22T18:41:47.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:47.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174626+0000", 2024-04-22T18:41:47.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037839699999999999 2024-04-22T18:41:47.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174633+0000", 2024-04-22T18:41:47.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0990000000000004e-06 2024-04-22T18:41:47.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174636+0000", 2024-04-22T18:41:47.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5540000000000002e-06 2024-04-22T18:41:47.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174790+0000", 2024-04-22T18:41:47.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153807 2024-04-22T18:41:47.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:47.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174793+0000", 2024-04-22T18:41:47.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4970000000000001e-06 2024-04-22T18:41:47.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054902+0000", 2024-04-22T18:41:47.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88010886300000002 2024-04-22T18:41:47.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055366+0000", 2024-04-22T18:41:47.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046336199999999999 2024-04-22T18:41:47.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.452973+0000", 2024-04-22T18:41:47.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39760783999999999 2024-04-22T18:41:47.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453046+0000", 2024-04-22T18:41:47.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2374999999999997e-05 2024-04-22T18:41:47.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:47.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453193+0000", 2024-04-22T18:41:47.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014722499999999999 2024-04-22T18:41:47.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554698+0000", 2024-04-22T18:41:47.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101505096 2024-04-22T18:41:47.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659337+0000", 2024-04-22T18:41:47.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104639236 2024-04-22T18:41:47.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659365+0000", 2024-04-22T18:41:47.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7594e-05 2024-04-22T18:41:47.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659419+0000", 2024-04-22T18:41:47.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4058999999999999e-05 2024-04-22T18:41:47.422 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.422 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.422 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.423 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:18 3.2 3:792a4ed0:::e1c814961b3d1531fb5ed475b3d496a0405c3a819b68a92ff9b2391fe912370f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:47.423 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:47.423 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 103.891149168, 2024-04-22T18:41:47.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.585036954, 2024-04-22T18:41:47.423 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.423 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.423 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:47.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:47.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174640+0000", 2024-04-22T18:41:47.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-04-22T18:41:47.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174758+0000", 2024-04-22T18:41:47.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011819099999999999 2024-04-22T18:41:47.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174764+0000", 2024-04-22T18:41:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6729999999999997e-06 2024-04-22T18:41:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174767+0000", 2024-04-22T18:41:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3830000000000001e-06 2024-04-22T18:41:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373669+0000", 2024-04-22T18:41:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198901989 2024-04-22T18:41:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:47.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373686+0000", 2024-04-22T18:41:47.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7286e-05 2024-04-22T18:41:47.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055408+0000", 2024-04-22T18:41:47.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68172133400000001 2024-04-22T18:41:47.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055938+0000", 2024-04-22T18:41:47.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053016599999999999 2024-04-22T18:41:47.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453668+0000", 2024-04-22T18:41:47.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39773030799999998 2024-04-22T18:41:47.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453717+0000", 2024-04-22T18:41:47.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8884000000000002e-05 2024-04-22T18:41:47.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:47.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453823+0000", 2024-04-22T18:41:47.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105791 2024-04-22T18:41:47.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554715+0000", 2024-04-22T18:41:47.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100891628 2024-04-22T18:41:47.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759439+0000", 2024-04-22T18:41:47.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20472459400000001 2024-04-22T18:41:47.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759525+0000", 2024-04-22T18:41:47.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5653000000000001e-05 2024-04-22T18:41:47.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759682+0000", 2024-04-22T18:41:47.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015711800000000001 2024-04-22T18:41:47.429 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.429 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.429 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.429 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:75 3.1 3:ab43a94d:::f38e6e97cb24e2b90da964fee5d2d6d8674386870fc552915e0c3fdbefd4b126:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:47.429 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:47.429 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.818272651000001, 2024-04-22T18:41:47.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60149928399999997, 2024-04-22T18:41:47.430 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.430 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.430 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:47.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:47.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247520+0000", 2024-04-22T18:41:47.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:47.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247527+0000", 2024-04-22T18:41:47.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1670000000000002e-06 2024-04-22T18:41:47.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247539+0000", 2024-04-22T18:41:47.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503e-05 2024-04-22T18:41:47.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247549+0000", 2024-04-22T18:41:47.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0000000000000001e-05 2024-04-22T18:41:47.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247796+0000", 2024-04-22T18:41:47.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024712699999999998 2024-04-22T18:41:47.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:47.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247955+0000", 2024-04-22T18:41:47.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158534 2024-04-22T18:41:47.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.343985+0000", 2024-04-22T18:41:47.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096030344000000004 2024-04-22T18:41:47.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.344035+0000", 2024-04-22T18:41:47.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9969999999999998e-05 2024-04-22T18:41:47.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.745806+0000", 2024-04-22T18:41:47.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.401771029 2024-04-22T18:41:47.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746040+0000", 2024-04-22T18:41:47.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023339300000000001 2024-04-22T18:41:47.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:47.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746198+0000", 2024-04-22T18:41:47.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015793299999999999 2024-04-22T18:41:47.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848737+0000", 2024-04-22T18:41:47.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10253898 2024-04-22T18:41:47.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848751+0000", 2024-04-22T18:41:47.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4694e-05 2024-04-22T18:41:47.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848825+0000", 2024-04-22T18:41:47.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3443999999999997e-05 2024-04-22T18:41:47.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849021+0000", 2024-04-22T18:41:47.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000196047 2024-04-22T18:41:47.436 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.436 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.436 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.436 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:76 3.1 3:b44829a0:::3788286a255d538503614e5f04049b0bb54cacaa317ab102b5dee8f5b640b296:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:47.436 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:47.436 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.818233585000002, 2024-04-22T18:41:47.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60168681599999996, 2024-04-22T18:41:47.436 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.436 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:47.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247564+0000", 2024-04-22T18:41:47.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1150000000000004e-06 2024-04-22T18:41:47.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247570+0000", 2024-04-22T18:41:47.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3940000000000001e-06 2024-04-22T18:41:47.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247576+0000", 2024-04-22T18:41:47.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3619999999999997e-06 2024-04-22T18:41:47.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.340845+0000", 2024-04-22T18:41:47.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.093269199999999997 2024-04-22T18:41:47.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:47.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.341043+0000", 2024-04-22T18:41:47.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019834400000000001 2024-04-22T18:41:47.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345106+0000", 2024-04-22T18:41:47.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040629780000000001 2024-04-22T18:41:47.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345160+0000", 2024-04-22T18:41:47.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4246999999999997e-05 2024-04-22T18:41:47.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747047+0000", 2024-04-22T18:41:47.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40188678300000003 2024-04-22T18:41:47.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747162+0000", 2024-04-22T18:41:47.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011483400000000001 2024-04-22T18:41:47.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:47.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747429+0000", 2024-04-22T18:41:47.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026700299999999998 2024-04-22T18:41:47.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849032+0000", 2024-04-22T18:41:47.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101603203 2024-04-22T18:41:47.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849095+0000", 2024-04-22T18:41:47.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2597999999999999e-05 2024-04-22T18:41:47.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849136+0000", 2024-04-22T18:41:47.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1251000000000002e-05 2024-04-22T18:41:47.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849247+0000", 2024-04-22T18:41:47.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011127 2024-04-22T18:41:47.442 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.442 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.442 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.443 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3526 4.c 4:314c2fec:::smithi08937929-22:head [setxattr _header (56) in=7b,truncate 3092521] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:41:47.443 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:47.443 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 70.836596804999999, 2024-04-22T18:41:47.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60463626000000004, 2024-04-22T18:41:47.443 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.443 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.443 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:47.443 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:47.443 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:47.443 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3526 2024-04-22T18:41:47.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.443 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:47.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:47.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:47.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:47.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229200+0000", 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2320000000000001e-06 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229203+0000", 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329292+0000", 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100088891 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329357+0000", 2024-04-22T18:41:47.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5115000000000003e-05 2024-04-22T18:41:47.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329453+0000", 2024-04-22T18:41:47.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5802000000000002e-05 2024-04-22T18:41:47.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:47.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329594+0000", 2024-04-22T18:41:47.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014157500000000001 2024-04-22T18:41:47.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.330172+0000", 2024-04-22T18:41:47.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057782799999999996 2024-04-22T18:41:47.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833712+0000", 2024-04-22T18:41:47.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50353975299999998 2024-04-22T18:41:47.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833774+0000", 2024-04-22T18:41:47.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1832e-05 2024-04-22T18:41:47.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833834+0000", 2024-04-22T18:41:47.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9809999999999998e-05 2024-04-22T18:41:47.448 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.448 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.448 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.448 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:339 3.2 3:4159b8d1:::bb1f8e997816dd9d2f29d36c4c1859fcaffbf8763b5c8d63c7a818d8d84bac37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:47.448 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:47.448 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.290343045999997, 2024-04-22T18:41:47.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0092773479999999, 2024-04-22T18:41:47.448 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.448 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.448 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:47.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:47.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775450+0000", 2024-04-22T18:41:47.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:47.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775457+0000", 2024-04-22T18:41:47.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2949999999999996e-06 2024-04-22T18:41:47.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775467+0000", 2024-04-22T18:41:47.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0180000000000001e-05 2024-04-22T18:41:47.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775475+0000", 2024-04-22T18:41:47.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.678e-06 2024-04-22T18:41:47.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.877787+0000", 2024-04-22T18:41:47.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102312559 2024-04-22T18:41:47.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.880686+0000", 2024-04-22T18:41:47.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002898748 2024-04-22T18:41:47.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.282867+0000", 2024-04-22T18:41:47.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.402180961 2024-04-22T18:41:47.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.282945+0000", 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7775999999999995e-05 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283065+0000", 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0001199 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.285514+0000", 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0024491729999999998 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784375+0000", 2024-04-22T18:41:47.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49886154100000002 2024-04-22T18:41:47.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784475+0000", 2024-04-22T18:41:47.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9074999999999996e-05 2024-04-22T18:41:47.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784728+0000", 2024-04-22T18:41:47.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025386799999999998 2024-04-22T18:41:47.453 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.454 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.454 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.454 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:340 3.2 3:4bbc2d06:::e1c179ab87fb8e08c4a1dfc0c7cc725138181c5734d6df18f3f2545b93731e97:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:47.454 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:47.454 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.290310626999997, 2024-04-22T18:41:47.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009550757, 2024-04-22T18:41:47.454 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.454 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.454 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:47.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:47.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:47.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:47.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775487+0000", 2024-04-22T18:41:47.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6839999999999998e-06 2024-04-22T18:41:47.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775492+0000", 2024-04-22T18:41:47.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6899999999999997e-06 2024-04-22T18:41:47.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775497+0000", 2024-04-22T18:41:47.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8590000000000004e-06 2024-04-22T18:41:47.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.880737+0000", 2024-04-22T18:41:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105240267 2024-04-22T18:41:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881308+0000", 2024-04-22T18:41:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057098399999999997 2024-04-22T18:41:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283810+0000", 2024-04-22T18:41:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40250177500000001 2024-04-22T18:41:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283883+0000", 2024-04-22T18:41:47.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2430999999999998e-05 2024-04-22T18:41:47.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:47.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284019+0000", 2024-04-22T18:41:47.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013652299999999999 2024-04-22T18:41:47.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.287390+0000", 2024-04-22T18:41:47.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033710089999999999 2024-04-22T18:41:47.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784829+0000", 2024-04-22T18:41:47.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49743879400000002 2024-04-22T18:41:47.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784893+0000", 2024-04-22T18:41:47.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4184999999999996e-05 2024-04-22T18:41:47.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785034+0000", 2024-04-22T18:41:47.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014114600000000001 2024-04-22T18:41:47.459 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.459 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.459 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.460 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:341 3.2 3:5b8bea68:::ab73be59042e3e18c4fe1fdbc2bbcc205779424ddf3bb423a595330be9026399:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:47.460 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:47.460 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.290291275000001, 2024-04-22T18:41:47.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009670804, 2024-04-22T18:41:47.460 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.460 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.460 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:47.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:47.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775502+0000", 2024-04-22T18:41:47.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:47.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775505+0000", 2024-04-22T18:41:47.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0809999999999998e-06 2024-04-22T18:41:47.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775511+0000", 2024-04-22T18:41:47.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2039999999999996e-06 2024-04-22T18:41:47.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775515+0000", 2024-04-22T18:41:47.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8790000000000002e-06 2024-04-22T18:41:47.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881376+0000", 2024-04-22T18:41:47.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105861 2024-04-22T18:41:47.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881839+0000", 2024-04-22T18:41:47.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046217200000000001 2024-04-22T18:41:47.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284569+0000", 2024-04-22T18:41:47.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40273058900000003 2024-04-22T18:41:47.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284730+0000", 2024-04-22T18:41:47.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077800000000001 2024-04-22T18:41:47.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:47.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284923+0000", 2024-04-22T18:41:47.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000193253 2024-04-22T18:41:47.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.287593+0000", 2024-04-22T18:41:47.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026699359999999999 2024-04-22T18:41:47.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785074+0000", 2024-04-22T18:41:47.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49748046000000001 2024-04-22T18:41:47.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785103+0000", 2024-04-22T18:41:47.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8935e-05 2024-04-22T18:41:47.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785174+0000", 2024-04-22T18:41:47.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1001999999999995e-05 2024-04-22T18:41:47.466 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.466 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.466 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.466 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:342 3.2 3:634dda9f:::10622dc7de8047e7908c0b5cca7a43d84cdf1b7629e6f7e159b8207feba82b01:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:47.466 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:47.466 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.290273063000001, 2024-04-22T18:41:47.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.022499579, 2024-04-22T18:41:47.466 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.466 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.466 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:47.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:47.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:47.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:47.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775524+0000", 2024-04-22T18:41:47.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9629999999999998e-06 2024-04-22T18:41:47.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775529+0000", 2024-04-22T18:41:47.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0320000000000003e-06 2024-04-22T18:41:47.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775533+0000", 2024-04-22T18:41:47.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7380000000000002e-06 2024-04-22T18:41:47.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881884+0000", 2024-04-22T18:41:47.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106350294 2024-04-22T18:41:47.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.882492+0000", 2024-04-22T18:41:47.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060872699999999999 2024-04-22T18:41:47.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381559+0000", 2024-04-22T18:41:47.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49906679199999998 2024-04-22T18:41:47.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381677+0000", 2024-04-22T18:41:47.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011743699999999999 2024-04-22T18:41:47.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:47.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381899+0000", 2024-04-22T18:41:47.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022272599999999999 2024-04-22T18:41:47.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.383440+0000", 2024-04-22T18:41:47.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015407719999999999 2024-04-22T18:41:47.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.797827+0000", 2024-04-22T18:41:47.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.414386741 2024-04-22T18:41:47.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.797887+0000", 2024-04-22T18:41:47.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9889000000000002e-05 2024-04-22T18:41:47.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.798021+0000", 2024-04-22T18:41:47.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000133948 2024-04-22T18:41:47.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.472 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.472 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.472 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:343 3.2 3:7db7298e:::00597d8527769aa920807495342cf3ba06efb8f507feae865dd89cd8ad10bda0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:47.472 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:47.472 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.290254107999999, 2024-04-22T18:41:47.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.110995373, 2024-04-22T18:41:47.472 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.472 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.472 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:47.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:47.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:47.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:47.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775542+0000", 2024-04-22T18:41:47.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9079999999999999e-06 2024-04-22T18:41:47.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775547+0000", 2024-04-22T18:41:47.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.994e-06 2024-04-22T18:41:47.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775553+0000", 2024-04-22T18:41:47.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2299999999999999e-06 2024-04-22T18:41:47.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.882519+0000", 2024-04-22T18:41:47.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106966453 2024-04-22T18:41:47.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883103+0000", 2024-04-22T18:41:47.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058419400000000001 2024-04-22T18:41:47.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.581957+0000", 2024-04-22T18:41:47.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69885328599999996 2024-04-22T18:41:47.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582023+0000", 2024-04-22T18:41:47.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6210999999999993e-05 2024-04-22T18:41:47.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:47.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582146+0000", 2024-04-22T18:41:47.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123669 2024-04-22T18:41:47.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.682293+0000", 2024-04-22T18:41:47.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10014662100000001 2024-04-22T18:41:47.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886410+0000", 2024-04-22T18:41:47.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20411678899999999 2024-04-22T18:41:47.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886445+0000", 2024-04-22T18:41:47.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5462e-05 2024-04-22T18:41:47.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886535+0000", 2024-04-22T18:41:47.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9995999999999997e-05 2024-04-22T18:41:47.477 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.477 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.478 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.478 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:344 3.1 3:9da67ae4:::04c6b48a110b3f74550f78f0899df697160a1eeab7f41463f6ac18091ca67078:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:47.478 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:47.478 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.290218801000002, 2024-04-22T18:41:47.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60604869800000005, 2024-04-22T18:41:47.478 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.478 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.478 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:47.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:47.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:47.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:47.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775578+0000", 2024-04-22T18:41:47.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3440000000000001e-06 2024-04-22T18:41:47.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775584+0000", 2024-04-22T18:41:47.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6969999999999998e-06 2024-04-22T18:41:47.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775589+0000", 2024-04-22T18:41:47.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0540000000000002e-06 2024-04-22T18:41:47.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775729+0000", 2024-04-22T18:41:47.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014061299999999999 2024-04-22T18:41:47.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776045+0000", 2024-04-22T18:41:47.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000315168 2024-04-22T18:41:47.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080584+0000", 2024-04-22T18:41:47.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30453977700000001 2024-04-22T18:41:47.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080635+0000", 2024-04-22T18:41:47.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1007999999999997e-05 2024-04-22T18:41:47.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:47.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080741+0000", 2024-04-22T18:41:47.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105646 2024-04-22T18:41:47.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.180821+0000", 2024-04-22T18:41:47.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10007980900000001 2024-04-22T18:41:47.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381437+0000", 2024-04-22T18:41:47.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20061589399999999 2024-04-22T18:41:47.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381491+0000", 2024-04-22T18:41:47.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4531999999999999e-05 2024-04-22T18:41:47.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381624+0000", 2024-04-22T18:41:47.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000132745 2024-04-22T18:41:47.483 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.483 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.483 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.484 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:345 3.1 3:bd847077:::0e274616605e19dd653fcf05d310933ab02835574414ca18340b3fc325d2da4f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:47.484 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:47.484 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.290194440999997, 2024-04-22T18:41:47.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.90876140699999997, 2024-04-22T18:41:47.484 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.484 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.484 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:47.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:47.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775599+0000", 2024-04-22T18:41:47.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:47.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775602+0000", 2024-04-22T18:41:47.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.976e-06 2024-04-22T18:41:47.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775607+0000", 2024-04-22T18:41:47.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2229999999999998e-06 2024-04-22T18:41:47.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775612+0000", 2024-04-22T18:41:47.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6539999999999998e-06 2024-04-22T18:41:47.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776072+0000", 2024-04-22T18:41:47.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046006299999999998 2024-04-22T18:41:47.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:47.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776256+0000", 2024-04-22T18:41:47.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000183735 2024-04-22T18:41:47.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081157+0000", 2024-04-22T18:41:47.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30490169099999997 2024-04-22T18:41:47.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081203+0000", 2024-04-22T18:41:47.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5677999999999999e-05 2024-04-22T18:41:47.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:47.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081301+0000", 2024-04-22T18:41:47.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7893999999999995e-05 2024-04-22T18:41:47.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.180827+0000", 2024-04-22T18:41:47.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099525725999999995 2024-04-22T18:41:47.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684197+0000", 2024-04-22T18:41:47.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50336990699999995 2024-04-22T18:41:47.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684247+0000", 2024-04-22T18:41:47.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0195000000000003e-05 2024-04-22T18:41:47.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684361+0000", 2024-04-22T18:41:47.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114189 2024-04-22T18:41:47.489 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.489 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.489 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.489 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4267 3.2 3:584fd9a2:::cda848272f4ef42ebafc2089aaddeacaf12eabd82aef5e42e32ae93a8a5da0b6:head [write 0~2635776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:47.490 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:47.490 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.209844584999999, 2024-04-22T18:41:47.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.72658671900000005, 2024-04-22T18:41:47.490 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:47.490 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:47.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:47.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:47.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:47.490 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4267 2024-04-22T18:41:47.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.490 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:47.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:47.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:47.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:47.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:47.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:47.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:47.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855946+0000", 2024-04-22T18:41:47.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:47.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:47.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.856984+0000", 2024-04-22T18:41:47.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001037764 2024-04-22T18:41:47.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:47.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.857003+0000", 2024-04-22T18:41:47.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8876e-05 2024-04-22T18:41:47.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:47.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.857016+0000", 2024-04-22T18:41:47.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3337e-05 2024-04-22T18:41:47.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:47.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883170+0000", 2024-04-22T18:41:47.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026154047 2024-04-22T18:41:47.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:47.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883262+0000", 2024-04-22T18:41:47.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1996999999999993e-05 2024-04-22T18:41:47.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:47.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883360+0000", 2024-04-22T18:41:47.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7429000000000005e-05 2024-04-22T18:41:47.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:47.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.887277+0000", 2024-04-22T18:41:47.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039169449999999998 2024-04-22T18:41:47.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:47.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582423+0000", 2024-04-22T18:41:47.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69514625200000002 2024-04-22T18:41:47.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:47.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582458+0000", 2024-04-22T18:41:47.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5293999999999998e-05 2024-04-22T18:41:47.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:47.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:47.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:47.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582536+0000", 2024-04-22T18:41:47.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7855000000000006e-05 2024-04-22T18:41:47.495 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.495 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.495 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.495 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:47.495 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:47.495 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:47.496 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:41:47.497 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:47.497+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:47.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:47.597 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:47.599+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:47.611 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:47.611 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:47.611 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:47.611 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:47.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:47.654 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:47.654 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:47.670 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:47.670 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:47.686 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:47.686 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:47.698 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:47.700+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:47.702 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:47.702 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:47.718 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:47.718 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:47.724 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:47.725 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:47.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:47.736 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:47.736 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:47.753 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:47.753 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:47.770 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:47.770 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:47.821 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:47.823+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:47.835 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:47.836 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:47.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:47.900 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:47.901+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:47.948 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:47.948 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:47.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:48.041 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:48.042+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:48.098 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:48.099 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:48.101 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:48.102+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:48.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:48.201 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:48.202+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:48.249 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:48.250 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:48.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:48.342 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:48.343+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:48.400 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:48.400 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:48.403 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:48.404+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:48.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:48.505 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:48.506+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:48.552 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:48.552 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:48.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:48.606 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:48.607+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:48.661 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:48.661 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:48.670 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:48.673 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:41:48.717 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:48.718+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:48.772 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:48.772 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:48.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:48.807 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:48.808+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:48.827 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:48.828+0000 7fa766d7d640 -1 osd.5 170 handle_osd_map injecting map crc failure 2024-04-22T18:41:48.884 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:48.884 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:48.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:48.908 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:48.909+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:48.993 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:48.994 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:49.002 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:49.009 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:49.010+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:49.105 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:49.106 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:49.111 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:49.112+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:49.114 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:49.212 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:49.213+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:49.217 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:49.217 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:49.226 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:49.313 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:49.313+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:49.327 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:49.328 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:49.337 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:49.413 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:49.414+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:49.438 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:49.438 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:49.447 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:49.514 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:49.515+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:49.549 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:49.549 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:49.558 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:49.642 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:49.643+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:49.699 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:49.699 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:49.708 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:49.730 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 7, 2, 4, 0] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:41:49.730 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:41:49.730 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-22T18:41:49.730 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-22T18:41:49.735 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:41:49.736 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:49.736+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:49.781 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:41:49.817 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:49.818+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:49.841 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:49.841 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:49.851 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:49.918 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:49.919+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:49.926 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:49.926 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:49.944 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:49.944 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:49.957 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:49.957 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:49.962 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:49.962 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:49.967 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:49.980 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:49.980 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:49.996 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:49.997 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:50.014 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:50.014 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:50.018 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:50.019+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:50.031 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:50.047 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:41:50.047 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:50.047 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 173, 2024-04-22T18:41:50.047 INFO:teuthology.orchestra.run.smithi089.stdout: "fsid": "9e25744c-eff1-4a4a-8d47-0dda406b2c35", 2024-04-22T18:41:50.047 INFO:teuthology.orchestra.run.smithi089.stdout: "created": "2024-04-22T18:38:26.841687+0000", 2024-04-22T18:41:50.047 INFO:teuthology.orchestra.run.smithi089.stdout: "modified": "2024-04-22T18:41:49.729332+0000", 2024-04-22T18:41:50.047 INFO:teuthology.orchestra.run.smithi089.stdout: "last_up_change": "2024-04-22T18:38:35.971820+0000", 2024-04-22T18:41:50.047 INFO:teuthology.orchestra.run.smithi089.stdout: "last_in_change": "2024-04-22T18:41:26.001892+0000", 2024-04-22T18:41:50.047 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-22T18:41:50.047 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_num": 5806080, 2024-04-22T18:41:50.048 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_set": [ 2024-04-22T18:41:50.048 INFO:teuthology.orchestra.run.smithi089.stdout: "nodeep-scrub", 2024-04-22T18:41:50.048 INFO:teuthology.orchestra.run.smithi089.stdout: "noscrub", 2024-04-22T18:41:50.048 INFO:teuthology.orchestra.run.smithi089.stdout: "pglog_hardlimit", 2024-04-22T18:41:50.048 INFO:teuthology.orchestra.run.smithi089.stdout: "purged_snapdirs", 2024-04-22T18:41:50.048 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_deletes", 2024-04-22T18:41:50.048 INFO:teuthology.orchestra.run.smithi089.stdout: "sortbitwise" 2024-04-22T18:41:50.048 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:41:50.048 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_version": 5, 2024-04-22T18:41:50.048 INFO:teuthology.orchestra.run.smithi089.stdout: "full_ratio": 0.94999998807907104, 2024-04-22T18:41:50.048 INFO:teuthology.orchestra.run.smithi089.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-22T18:41:50.048 INFO:teuthology.orchestra.run.smithi089.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-22T18:41:50.048 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_snapshot": "", 2024-04-22T18:41:50.048 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_max": 4, 2024-04-22T18:41:50.048 INFO:teuthology.orchestra.run.smithi089.stdout: "max_osd": 8, 2024-04-22T18:41:50.049 INFO:teuthology.orchestra.run.smithi089.stdout: "require_min_compat_client": "luminous", 2024-04-22T18:41:50.049 INFO:teuthology.orchestra.run.smithi089.stdout: "min_compat_client": "luminous", 2024-04-22T18:41:50.049 INFO:teuthology.orchestra.run.smithi089.stdout: "require_osd_release": "reef", 2024-04-22T18:41:50.049 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_crimson": false, 2024-04-22T18:41:50.049 INFO:teuthology.orchestra.run.smithi089.stdout: "pools": [ 2024-04-22T18:41:50.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.049 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 1, 2024-04-22T18:41:50.049 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": ".mgr", 2024-04-22T18:41:50.049 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:36.991610+0000", 2024-04-22T18:41:50.049 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:41:50.049 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:41:50.049 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:41:50.049 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:41:50.049 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:41:50.049 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:41:50.050 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:41:50.050 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:41:50.050 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:41:50.050 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:41:50.050 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:41:50.050 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:41:50.050 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 1, 2024-04-22T18:41:50.050 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 1, 2024-04-22T18:41:50.050 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 1, 2024-04-22T18:41:50.050 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 1, 2024-04-22T18:41:50.050 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 1, 2024-04-22T18:41:50.050 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:41:50.050 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:41:50.050 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:41:50.050 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:41:50.051 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:41:50.051 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:41:50.051 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:41:50.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.051 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "16", 2024-04-22T18:41:50.051 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:41:50.051 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:41:50.051 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-04-22T18:41:50.051 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:41:50.051 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:41:50.051 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:41:50.051 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:41:50.051 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:41:50.051 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:41:50.052 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:41:50.052 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:41:50.052 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:41:50.052 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:41:50.052 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:41:50.052 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:41:50.052 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:41:50.052 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:41:50.052 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:41:50.052 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:41:50.052 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:41:50.052 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:41:50.052 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:41:50.052 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:41:50.052 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:41:50.053 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:41:50.053 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:41:50.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.053 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:41:50.053 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:41:50.053 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:41:50.053 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:41:50.053 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:41:50.053 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:41:50.053 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:41:50.053 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:41:50.053 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:41:50.053 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:41:50.053 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:41:50.053 INFO:teuthology.orchestra.run.smithi089.stdout: "options": { 2024-04-22T18:41:50.054 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_max": 32, 2024-04-22T18:41:50.054 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_min": 1 2024-04-22T18:41:50.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.054 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-04-22T18:41:50.054 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": {} 2024-04-22T18:41:50.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.054 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:41:50.054 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 8, 2024-04-22T18:41:50.054 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 8, 2024-04-22T18:41:50.054 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.25, 2024-04-22T18:41:50.054 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2, 2024-04-22T18:41:50.054 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2, 2024-04-22T18:41:50.054 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.94993972778320312, 2024-04-22T18:41:50.054 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-22T18:41:50.054 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 2, 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "rbd", 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:37.622620+0000", 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:41:50.055 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:41:50.056 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:41:50.056 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:41:50.056 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 8, 2024-04-22T18:41:50.056 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 8, 2024-04-22T18:41:50.056 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 8, 2024-04-22T18:41:50.056 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 8, 2024-04-22T18:41:50.056 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 8, 2024-04-22T18:41:50.056 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:41:50.056 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:41:50.056 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:41:50.056 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:41:50.056 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:41:50.056 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:41:50.056 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:41:50.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "17", 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:41:50.057 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:41:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:41:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:41:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:41:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:41:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:41:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:41:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:41:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:41:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:41:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:41:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:41:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:41:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:41:50.058 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:41:50.059 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:41:50.059 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:41:50.059 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:41:50.059 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:41:50.059 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:41:50.059 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:41:50.059 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:41:50.059 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:41:50.059 INFO:teuthology.orchestra.run.smithi089.stdout: "options": {}, 2024-04-22T18:41:50.059 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-04-22T18:41:50.059 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd": {} 2024-04-22T18:41:50.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.059 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:41:50.059 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 3, 2024-04-22T18:41:50.060 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 3, 2024-04-22T18:41:50.060 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.75, 2024-04-22T18:41:50.060 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2.25, 2024-04-22T18:41:50.060 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2.25, 2024-04-22T18:41:50.060 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.94993972778320312, 2024-04-22T18:41:50.060 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-22T18:41:50.060 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-22T18:41:50.060 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.060 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 3, 2024-04-22T18:41:50.060 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "low_tier", 2024-04-22T18:41:50.060 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:46.038005+0000", 2024-04-22T18:41:50.060 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:41:50.060 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 4, 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 4, 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 4, 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 4, 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 4, 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:41:50.061 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-04-22T18:41:50.062 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-04-22T18:41:50.062 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-04-22T18:41:50.062 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-04-22T18:41:50.062 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-04-22T18:41:50.062 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-04-22T18:41:50.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.062 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "21", 2024-04-22T18:41:50.062 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:41:50.062 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-22T18:41:50.062 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-04-22T18:41:50.062 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:41:50.062 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:41:50.062 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:41:50.062 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:41:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:41:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:41:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:41:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:41:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:41:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:41:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:41:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:41:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:41:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:41:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:41:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:41:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:41:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:41:50.064 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: "options": {}, 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": {}, 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 3.9700000286102295, 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 3.9700000286102295, 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.62999999523162842, 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2.5, 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2.5, 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.94993972778320312, 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.065 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 4, 2024-04-22T18:41:50.066 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "unique_pool_0", 2024-04-22T18:41:50.066 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-04-22T18:38:46.483369+0000", 2024-04-22T18:41:50.066 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-04-22T18:41:50.066 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-04-22T18:41:50.066 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-04-22T18:41:50.066 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-04-22T18:41:50.066 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 2, 2024-04-22T18:41:50.066 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-04-22T18:41:50.066 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-04-22T18:41:50.066 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-04-22T18:41:50.066 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-04-22T18:41:50.066 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-22T18:41:50.066 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-04-22T18:41:50.066 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-04-22T18:41:50.067 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 36, 2024-04-22T18:41:50.067 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 26, 2024-04-22T18:41:50.067 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 36, 2024-04-22T18:41:50.067 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 36, 2024-04-22T18:41:50.067 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 36, 2024-04-22T18:41:50.067 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-04-22T18:41:50.067 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "4.1f", 2024-04-22T18:41:50.067 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 158, 2024-04-22T18:41:50.067 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 157, 2024-04-22T18:41:50.067 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 157, 2024-04-22T18:41:50.067 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "151'79", 2024-04-22T18:41:50.067 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "155'85" 2024-04-22T18:41:50.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.069 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "173", 2024-04-22T18:41:50.069 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-04-22T18:41:50.069 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "158", 2024-04-22T18:41:50.069 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "164", 2024-04-22T18:41:50.069 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-04-22T18:41:50.070 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-04-22T18:41:50.070 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-04-22T18:41:50.070 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-04-22T18:41:50.070 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-04-22T18:41:50.070 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-04-22T18:41:50.070 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-04-22T18:41:50.070 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-04-22T18:41:50.070 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-04-22T18:41:50.070 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-04-22T18:41:50.070 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-04-22T18:41:50.070 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-04-22T18:41:50.070 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-04-22T18:41:50.070 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-04-22T18:41:50.071 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-04-22T18:41:50.071 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-22T18:41:50.071 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-22T18:41:50.071 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-22T18:41:50.071 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-04-22T18:41:50.071 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-04-22T18:41:50.071 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-04-22T18:41:50.071 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-04-22T18:41:50.071 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-04-22T18:41:50.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.071 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-04-22T18:41:50.071 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-04-22T18:41:50.071 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-04-22T18:41:50.071 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-04-22T18:41:50.071 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-04-22T18:41:50.072 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-04-22T18:41:50.072 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-04-22T18:41:50.072 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-04-22T18:41:50.072 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-04-22T18:41:50.072 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-04-22T18:41:50.072 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-04-22T18:41:50.072 INFO:teuthology.orchestra.run.smithi089.stdout: "options": { 2024-04-22T18:41:50.072 INFO:teuthology.orchestra.run.smithi089.stdout: "dedup_cdc_chunk_size": 131072, 2024-04-22T18:41:50.072 INFO:teuthology.orchestra.run.smithi089.stdout: "dedup_chunk_algorithm": "fastcdc", 2024-04-22T18:41:50.072 INFO:teuthology.orchestra.run.smithi089.stdout: "dedup_tier": 3, 2024-04-22T18:41:50.072 INFO:teuthology.orchestra.run.smithi089.stdout: "fingerprint_algorithm": "sha256" 2024-04-22T18:41:50.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.072 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-04-22T18:41:50.072 INFO:teuthology.orchestra.run.smithi089.stdout: "rados": {} 2024-04-22T18:41:50.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-04-22T18:41:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 2.6700000762939453, 2024-04-22T18:41:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 2.6700000762939453, 2024-04-22T18:41:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.75, 2024-04-22T18:41:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2, 2024-04-22T18:41:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2, 2024-04-22T18:41:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.94993972778320312, 2024-04-22T18:41:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-22T18:41:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-22T18:41:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:41:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "osds": [ 2024-04-22T18:41:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.074 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-04-22T18:41:50.074 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "bbf33cfb-0f07-4df6-b90f-83e9edb039a6", 2024-04-22T18:41:50.074 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:41:50.074 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:41:50.074 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:41:50.074 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:41:50.074 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:41:50.074 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:41:50.074 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:41:50.074 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 168, 2024-04-22T18:41:50.074 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:41:50.074 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:41:50.074 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:41:50.074 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.075 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.075 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6800", 2024-04-22T18:41:50.075 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:41:50.075 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.075 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.075 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:41:50.075 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.075 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.075 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6801", 2024-04-22T18:41:50.075 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:41:50.075 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.076 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.076 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:41:50.076 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.076 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.076 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6803", 2024-04-22T18:41:50.076 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:41:50.076 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.076 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.076 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:41:50.076 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.077 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.077 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6802", 2024-04-22T18:41:50.077 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31890 2024-04-22T18:41:50.077 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.077 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.077 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6800/31890", 2024-04-22T18:41:50.077 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6801/31890", 2024-04-22T18:41:50.077 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6803/31890", 2024-04-22T18:41:50.077 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6802/31890", 2024-04-22T18:41:50.077 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:41:50.077 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:41:50.077 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:41:50.077 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.078 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-04-22T18:41:50.078 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "30415a8e-9f47-416f-9fd0-716c2cb3f562", 2024-04-22T18:41:50.078 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:41:50.078 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:41:50.078 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:41:50.078 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:41:50.078 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:41:50.078 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:41:50.078 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:41:50.078 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 167, 2024-04-22T18:41:50.078 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:41:50.078 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:41:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:41:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6804", 2024-04-22T18:41:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:41:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:41:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6805", 2024-04-22T18:41:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:41:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:41:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6807", 2024-04-22T18:41:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:41:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:41:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6806", 2024-04-22T18:41:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31899 2024-04-22T18:41:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6804/31899", 2024-04-22T18:41:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6805/31899", 2024-04-22T18:41:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6807/31899", 2024-04-22T18:41:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6806/31899", 2024-04-22T18:41:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:41:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:41:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:41:50.081 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.082 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-04-22T18:41:50.082 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "b8da6b88-8355-4225-930d-bfed881089b7", 2024-04-22T18:41:50.082 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:41:50.082 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:41:50.082 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:41:50.082 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 0.75152587890625, 2024-04-22T18:41:50.082 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:41:50.082 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:41:50.082 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:41:50.082 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 135, 2024-04-22T18:41:50.082 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:41:50.082 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:41:50.082 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:41:50.083 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.083 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.083 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6812", 2024-04-22T18:41:50.083 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:41:50.083 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.083 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.083 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:41:50.083 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.083 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.083 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6813", 2024-04-22T18:41:50.083 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:41:50.084 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.084 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.084 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:41:50.084 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.084 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.084 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6815", 2024-04-22T18:41:50.084 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:41:50.084 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.084 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.084 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:41:50.084 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.085 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.085 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6814", 2024-04-22T18:41:50.085 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31903 2024-04-22T18:41:50.085 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.085 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.085 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6812/31903", 2024-04-22T18:41:50.085 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6813/31903", 2024-04-22T18:41:50.085 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6815/31903", 2024-04-22T18:41:50.085 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6814/31903", 2024-04-22T18:41:50.085 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:41:50.085 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:41:50.085 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:41:50.085 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.086 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-04-22T18:41:50.086 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "3eaec913-9d7d-4b03-a28f-0c334e7c39f1", 2024-04-22T18:41:50.086 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:41:50.086 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:41:50.086 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:41:50.086 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 0.847991943359375, 2024-04-22T18:41:50.086 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:41:50.086 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:41:50.086 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:41:50.086 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 164, 2024-04-22T18:41:50.086 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:41:50.086 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:41:50.086 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:41:50.086 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6808", 2024-04-22T18:41:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:41:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:41:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6809", 2024-04-22T18:41:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:41:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.087 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.088 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:41:50.088 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.088 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.088 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6811", 2024-04-22T18:41:50.088 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:41:50.088 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.088 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.088 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:41:50.088 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.088 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.089 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6810", 2024-04-22T18:41:50.089 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31913 2024-04-22T18:41:50.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.089 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.089 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6808/31913", 2024-04-22T18:41:50.089 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6809/31913", 2024-04-22T18:41:50.089 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6811/31913", 2024-04-22T18:41:50.089 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6810/31913", 2024-04-22T18:41:50.089 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:41:50.089 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:41:50.089 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:41:50.089 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.090 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-04-22T18:41:50.090 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "887acc3b-6f6b-49eb-a039-13050a2c7d64", 2024-04-22T18:41:50.090 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:41:50.090 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:41:50.090 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:41:50.090 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:41:50.090 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:41:50.090 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:41:50.090 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:41:50.090 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 172, 2024-04-22T18:41:50.090 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:41:50.090 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:41:50.090 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:41:50.091 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.091 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.091 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6808", 2024-04-22T18:41:50.091 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:41:50.091 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.091 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.091 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:41:50.091 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.091 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.091 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6809", 2024-04-22T18:41:50.091 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:41:50.092 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.092 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.092 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:41:50.092 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.092 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.092 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6811", 2024-04-22T18:41:50.092 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:41:50.092 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.092 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.092 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:41:50.092 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.093 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.093 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6810", 2024-04-22T18:41:50.093 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31922 2024-04-22T18:41:50.093 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.093 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.093 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6808/31922", 2024-04-22T18:41:50.093 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6809/31922", 2024-04-22T18:41:50.093 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6811/31922", 2024-04-22T18:41:50.093 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6810/31922", 2024-04-22T18:41:50.093 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:41:50.093 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:41:50.093 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:41:50.093 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.094 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-04-22T18:41:50.094 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "dcaddfd0-f8d1-4c28-bda2-d17eed606b60", 2024-04-22T18:41:50.094 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:41:50.094 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 0, 2024-04-22T18:41:50.094 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 0, 2024-04-22T18:41:50.094 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:41:50.094 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:41:50.094 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:41:50.094 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:41:50.094 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 63, 2024-04-22T18:41:50.094 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:41:50.094 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:41:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:41:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6804", 2024-04-22T18:41:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:41:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:41:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6805", 2024-04-22T18:41:50.095 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:41:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:41:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6807", 2024-04-22T18:41:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:41:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:41:50.096 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6806", 2024-04-22T18:41:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31921 2024-04-22T18:41:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6804/31921", 2024-04-22T18:41:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6805/31921", 2024-04-22T18:41:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6807/31921", 2024-04-22T18:41:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6806/31921", 2024-04-22T18:41:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:41:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:41:50.097 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:41:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-04-22T18:41:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "acf0527b-3020-4b0a-b64c-5068d21a5781", 2024-04-22T18:41:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:41:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 0, 2024-04-22T18:41:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 0, 2024-04-22T18:41:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:41:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:41:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:41:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:41:50.098 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 103, 2024-04-22T18:41:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:41:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:41:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:41:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6800", 2024-04-22T18:41:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:41:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:41:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6801", 2024-04-22T18:41:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:41:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:41:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6803", 2024-04-22T18:41:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:41:50.100 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:41:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6802", 2024-04-22T18:41:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31920 2024-04-22T18:41:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6800/31920", 2024-04-22T18:41:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6801/31920", 2024-04-22T18:41:50.101 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6803/31920", 2024-04-22T18:41:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6802/31920", 2024-04-22T18:41:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:41:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:41:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:41:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-04-22T18:41:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "305394bc-be3d-420b-88d7-d273670540b1", 2024-04-22T18:41:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-04-22T18:41:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-04-22T18:41:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-04-22T18:41:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-04-22T18:41:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-04-22T18:41:50.102 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-04-22T18:41:50.103 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-04-22T18:41:50.103 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 161, 2024-04-22T18:41:50.103 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-04-22T18:41:50.103 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-04-22T18:41:50.103 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-04-22T18:41:50.103 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.103 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.103 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6812", 2024-04-22T18:41:50.103 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:41:50.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.103 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.103 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-04-22T18:41:50.104 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.104 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.104 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6813", 2024-04-22T18:41:50.104 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:41:50.104 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.104 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.104 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-04-22T18:41:50.104 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.104 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.104 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6815", 2024-04-22T18:41:50.104 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:41:50.105 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.105 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.105 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-04-22T18:41:50.105 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-04-22T18:41:50.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.105 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-04-22T18:41:50.105 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.121:6814", 2024-04-22T18:41:50.105 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 31939 2024-04-22T18:41:50.105 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.105 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.105 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.121:6812/31939", 2024-04-22T18:41:50.105 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.121:6813/31939", 2024-04-22T18:41:50.105 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.121:6815/31939", 2024-04-22T18:41:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.121:6814/31939", 2024-04-22T18:41:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-04-22T18:41:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-04-22T18:41:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-04-22T18:41:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:41:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_xinfo": [ 2024-04-22T18:41:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-04-22T18:41:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:41:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:41:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:41:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:41:50.106 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:41:50.107 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:40:59.259212+0000", 2024-04-22T18:41:50.107 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:41:50.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.107 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-04-22T18:41:50.107 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:41:50.107 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:41:50.107 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:41:50.107 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:41:50.107 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:41:50.107 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:41:05.370813+0000", 2024-04-22T18:41:50.107 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:41:50.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.108 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-04-22T18:41:50.108 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:41:50.108 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:41:50.108 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:41:50.108 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:41:50.108 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:41:50.108 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:41:20.388547+0000", 2024-04-22T18:41:50.108 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:41:50.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.108 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-04-22T18:41:50.108 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:41:50.108 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:41:50.108 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:41:50.108 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:41:50.109 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:41:50.109 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:41:13.858167+0000", 2024-04-22T18:41:50.109 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:41:50.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.109 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-04-22T18:41:50.109 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:41:50.109 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:41:50.109 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:41:50.109 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:41:50.109 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:41:50.109 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:41:02.314614+0000", 2024-04-22T18:41:50.109 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:41:50.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.110 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-04-22T18:41:50.110 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:41:50.110 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:41:50.110 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:41:50.110 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:41:50.110 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 65536, 2024-04-22T18:41:50.110 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:41:10.687865+0000", 2024-04-22T18:41:50.110 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:41:50.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.110 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-04-22T18:41:50.110 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:41:50.110 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:41:50.111 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:41:50.111 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:41:50.111 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 65536, 2024-04-22T18:41:50.111 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:40:50.978238+0000", 2024-04-22T18:41:50.111 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:41:50.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.111 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-04-22T18:41:50.111 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-04-22T18:41:50.111 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-04-22T18:41:50.111 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-04-22T18:41:50.111 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-04-22T18:41:50.111 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-04-22T18:41:50.111 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-04-22T18:41:00.313214+0000", 2024-04-22T18:41:50.112 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-04-22T18:41:50.112 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.112 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:41:50.112 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap": [], 2024-04-22T18:41:50.112 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_items": [ 2024-04-22T18:41:50.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.112 INFO:teuthology.orchestra.run.smithi089.stdout: "pgid": "4.11", 2024-04-22T18:41:50.112 INFO:teuthology.orchestra.run.smithi089.stdout: "mappings": [ 2024-04-22T18:41:50.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.112 INFO:teuthology.orchestra.run.smithi089.stdout: "from": 3, 2024-04-22T18:41:50.112 INFO:teuthology.orchestra.run.smithi089.stdout: "to": 1 2024-04-22T18:41:50.112 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.112 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.112 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.112 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:41:50.113 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_primaries": [], 2024-04-22T18:41:50.113 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_temp": [ 2024-04-22T18:41:50.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.113 INFO:teuthology.orchestra.run.smithi089.stdout: "pgid": "4.18", 2024-04-22T18:41:50.113 INFO:teuthology.orchestra.run.smithi089.stdout: "osds": [ 2024-04-22T18:41:50.113 INFO:teuthology.orchestra.run.smithi089.stdout: 4, 2024-04-22T18:41:50.113 INFO:teuthology.orchestra.run.smithi089.stdout: 7 2024-04-22T18:41:50.113 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.113 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.113 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:41:50.113 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_temp": [], 2024-04-22T18:41:50.113 INFO:teuthology.orchestra.run.smithi089.stdout: "blocklist": {}, 2024-04-22T18:41:50.113 INFO:teuthology.orchestra.run.smithi089.stdout: "range_blocklist": {}, 2024-04-22T18:41:50.113 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profiles": { 2024-04-22T18:41:50.114 INFO:teuthology.orchestra.run.smithi089.stdout: "default": { 2024-04-22T18:41:50.114 INFO:teuthology.orchestra.run.smithi089.stdout: "crush-failure-domain": "osd", 2024-04-22T18:41:50.114 INFO:teuthology.orchestra.run.smithi089.stdout: "k": "2", 2024-04-22T18:41:50.114 INFO:teuthology.orchestra.run.smithi089.stdout: "m": "1", 2024-04-22T18:41:50.114 INFO:teuthology.orchestra.run.smithi089.stdout: "plugin": "jerasure", 2024-04-22T18:41:50.114 INFO:teuthology.orchestra.run.smithi089.stdout: "technique": "reed_sol_van" 2024-04-22T18:41:50.114 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.114 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps_queue": [], 2024-04-22T18:41:50.114 INFO:teuthology.orchestra.run.smithi089.stdout: "new_removed_snaps": [], 2024-04-22T18:41:50.114 INFO:teuthology.orchestra.run.smithi089.stdout: "new_purged_snaps": [], 2024-04-22T18:41:50.114 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_node_flags": {}, 2024-04-22T18:41:50.114 INFO:teuthology.orchestra.run.smithi089.stdout: "device_class_flags": {}, 2024-04-22T18:41:50.115 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode": { 2024-04-22T18:41:50.115 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_enabled": false, 2024-04-22T18:41:50.115 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_bucket_count": 0, 2024-04-22T18:41:50.115 INFO:teuthology.orchestra.run.smithi089.stdout: "degraded_stretch_mode": 0, 2024-04-22T18:41:50.115 INFO:teuthology.orchestra.run.smithi089.stdout: "recovering_stretch_mode": 0, 2024-04-22T18:41:50.115 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_bucket": 0 2024-04-22T18:41:50.115 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.115 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:50.115 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:50.116 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:50.118 INFO:tasks.thrashosds.thrasher:j is {'epoch': 173, 'fsid': '9e25744c-eff1-4a4a-8d47-0dda406b2c35', 'created': '2024-04-22T18:38:26.841687+0000', 'modified': '2024-04-22T18:41:49.729332+0000', 'last_up_change': '2024-04-22T18:38:35.971820+0000', 'last_in_change': '2024-04-22T18:41:26.001892+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T18:38:36.991610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9499397277832031, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-22T18:38:37.622620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9499397277832031, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-04-22T18:38:46.038005+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_acting': 3.9700000286102295, 'score_stable': 3.9700000286102295, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9499397277832031, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-22T18:38:46.483369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 36, 'pg_placement_num': 26, 'pg_placement_num_target': 36, 'pg_num_target': 36, 'pg_num_pending': 36, 'last_pg_merge_meta': {'source_pgid': '4.1f', 'ready_epoch': 158, 'last_epoch_started': 157, 'last_epoch_clean': 157, 'source_version': "151'79", 'target_version': "155'85"}, 'last_change': '173', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '158', 'last_force_op_resend_preluminous': '164', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'dedup_cdc_chunk_size': 131072, 'dedup_chunk_algorithm': 'fastcdc', 'dedup_tier': 3, 'fingerprint_algorithm': 'sha256'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9499397277832031, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': 'bbf33cfb-0f07-4df6-b90f-83e9edb039a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 168, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6800', 'nonce': 31890}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6801', 'nonce': 31890}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6803', 'nonce': 31890}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6802', 'nonce': 31890}]}, 'public_addr': '172.21.15.89:6800/31890', 'cluster_addr': '172.21.15.89:6801/31890', 'heartbeat_back_addr': '172.21.15.89:6803/31890', 'heartbeat_front_addr': '172.21.15.89:6802/31890', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '30415a8e-9f47-416f-9fd0-716c2cb3f562', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 167, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6804', 'nonce': 31899}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6805', 'nonce': 31899}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6807', 'nonce': 31899}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6806', 'nonce': 31899}]}, 'public_addr': '172.21.15.89:6804/31899', 'cluster_addr': '172.21.15.89:6805/31899', 'heartbeat_back_addr': '172.21.15.89:6807/31899', 'heartbeat_front_addr': '172.21.15.89:6806/31899', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b8da6b88-8355-4225-930d-bfed881089b7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.75152587890625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 135, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6812', 'nonce': 31903}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6813', 'nonce': 31903}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6815', 'nonce': 31903}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6814', 'nonce': 31903}]}, 'public_addr': '172.21.15.89:6812/31903', 'cluster_addr': '172.21.15.89:6813/31903', 'heartbeat_back_addr': '172.21.15.89:6815/31903', 'heartbeat_front_addr': '172.21.15.89:6814/31903', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3eaec913-9d7d-4b03-a28f-0c334e7c39f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.847991943359375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 164, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6808', 'nonce': 31913}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6809', 'nonce': 31913}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6811', 'nonce': 31913}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.89:6810', 'nonce': 31913}]}, 'public_addr': '172.21.15.89:6808/31913', 'cluster_addr': '172.21.15.89:6809/31913', 'heartbeat_back_addr': '172.21.15.89:6811/31913', 'heartbeat_front_addr': '172.21.15.89:6810/31913', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '887acc3b-6f6b-49eb-a039-13050a2c7d64', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6808', 'nonce': 31922}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6809', 'nonce': 31922}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6811', 'nonce': 31922}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6810', 'nonce': 31922}]}, 'public_addr': '172.21.15.121:6808/31922', 'cluster_addr': '172.21.15.121:6809/31922', 'heartbeat_back_addr': '172.21.15.121:6811/31922', 'heartbeat_front_addr': '172.21.15.121:6810/31922', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'dcaddfd0-f8d1-4c28-bda2-d17eed606b60', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 63, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6804', 'nonce': 31921}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6805', 'nonce': 31921}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6807', 'nonce': 31921}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6806', 'nonce': 31921}]}, 'public_addr': '172.21.15.121:6804/31921', 'cluster_addr': '172.21.15.121:6805/31921', 'heartbeat_back_addr': '172.21.15.121:6807/31921', 'heartbeat_front_addr': '172.21.15.121:6806/31921', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'acf0527b-3020-4b0a-b64c-5068d21a5781', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6800', 'nonce': 31920}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6801', 'nonce': 31920}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6803', 'nonce': 31920}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6802', 'nonce': 31920}]}, 'public_addr': '172.21.15.121:6800/31920', 'cluster_addr': '172.21.15.121:6801/31920', 'heartbeat_back_addr': '172.21.15.121:6803/31920', 'heartbeat_front_addr': '172.21.15.121:6802/31920', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '305394bc-be3d-420b-88d7-d273670540b1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 161, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6812', 'nonce': 31939}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6813', 'nonce': 31939}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6815', 'nonce': 31939}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.121:6814', 'nonce': 31939}]}, 'public_addr': '172.21.15.121:6812/31939', 'cluster_addr': '172.21.15.121:6813/31939', 'heartbeat_back_addr': '172.21.15.121:6815/31939', 'heartbeat_front_addr': '172.21.15.121:6814/31939', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:40:59.259212+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:41:05.370813+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:41:20.388547+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:41:13.858167+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:41:02.314614+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-22T18:41:10.687865+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-22T18:40:50.978238+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-22T18:41:00.313214+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.11', 'mappings': [{'from': 3, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '4.18', 'osds': [4, 7]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-22T18:41:50.118 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:41:50.119 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:50.119 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:41:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 165.62943259900001, 2024-04-22T18:41:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:41:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:41:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:41:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:41:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:41:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:41:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:41:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:41:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:41:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:41:50.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:41:50.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:41:50.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:41:50.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:41:50.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:41:50.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:41:50.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:41:50.124 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.124 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.124 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.124 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:41:50.124 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:50.124 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 161.752758777, 2024-04-22T18:41:50.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:41:50.124 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.124 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.124 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:50.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:41:50.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:41:50.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:41:50.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:41:50.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:41:50.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:41:50.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:41:50.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:41:50.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:41:50.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:41:50.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:41:50.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:41:50.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:41:50.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:41:50.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:41:50.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:41:50.128 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.128 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.128 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.128 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:41:50.128 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:50.128 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 105.547955254, 2024-04-22T18:41:50.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:41:50.128 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.129 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.129 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:50.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:41:50.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:41:50.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:41:50.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:41:50.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:41:50.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:41:50.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:41:50.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:41:50.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:41:50.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:41:50.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:41:50.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:41:50.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:41:50.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:41:50.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:41:50.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:41:50.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:41:50.132 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.132 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.132 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.132 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3451 4.d 4:bee1184f:::smithi08937929-7:head [read 0~3613505,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:50.132 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:50.133 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.163424926999994, 2024-04-22T18:41:50.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20164568099999999, 2024-04-22T18:41:50.133 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.133 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:50.133 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:50.133 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:50.133 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:50.133 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3451 2024-04-22T18:41:50.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.133 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:50.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:50.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:41:50.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927183+0000", 2024-04-22T18:41:50.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1110000000000001e-06 2024-04-22T18:41:50.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927188+0000", 2024-04-22T18:41:50.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1569999999999999e-06 2024-04-22T18:41:50.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927192+0000", 2024-04-22T18:41:50.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4710000000000001e-06 2024-04-22T18:41:50.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027382+0000", 2024-04-22T18:41:50.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100189341 2024-04-22T18:41:50.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:41:50.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027387+0000", 2024-04-22T18:41:50.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8400000000000002e-06 2024-04-22T18:41:50.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127638+0000", 2024-04-22T18:41:50.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10025102499999999 2024-04-22T18:41:50.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127740+0000", 2024-04-22T18:41:50.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102749 2024-04-22T18:41:50.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.128825+0000", 2024-04-22T18:41:50.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001084593 2024-04-22T18:41:50.137 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.137 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.137 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.137 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:151 3.0 e114/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 114'213, mlcod=110'210)", 2024-04-22T18:41:50.137 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:50.137 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 63.572956843999997, 2024-04-22T18:41:50.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20074001299999999, 2024-04-22T18:41:50.138 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.138 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.138 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:50.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:41:50.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517640+0000", 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999928 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517652+0000", 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1824000000000001e-05 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517660+0000", 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1419999999999993e-06 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617729+0000", 2024-04-22T18:41:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068723 2024-04-22T18:41:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617770+0000", 2024-04-22T18:41:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1391999999999999e-05 2024-04-22T18:41:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617793+0000", 2024-04-22T18:41:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2807999999999999e-05 2024-04-22T18:41:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718361+0000", 2024-04-22T18:41:50.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100567473 2024-04-22T18:41:50.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718387+0000", 2024-04-22T18:41:50.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6826e-05 2024-04-22T18:41:50.141 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.141 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.141 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.141 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3862 4.d 4:bee1184f:::smithi08937929-7:head [write 3301257~278667] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:50.141 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:50.141 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.828054573999999, 2024-04-22T18:41:50.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301875958, 2024-04-22T18:41:50.142 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.142 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.142 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:50.142 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:50.142 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:50.142 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3862 2024-04-22T18:41:50.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.142 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:50.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262548+0000", 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262622+0000", 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3747999999999999e-05 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262632+0000", 2024-04-22T18:41:50.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0767e-05 2024-04-22T18:41:50.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362705+0000", 2024-04-22T18:41:50.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073157 2024-04-22T18:41:50.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:41:50.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2846999999999997e-05 2024-04-22T18:41:50.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362853+0000", 2024-04-22T18:41:50.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4498000000000002e-05 2024-04-22T18:41:50.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:50.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362992+0000", 2024-04-22T18:41:50.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013879699999999999 2024-04-22T18:41:50.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.366534+0000", 2024-04-22T18:41:50.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035427969999999999 2024-04-22T18:41:50.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564353+0000", 2024-04-22T18:41:50.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.197818893 2024-04-22T18:41:50.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564385+0000", 2024-04-22T18:41:50.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1717000000000001e-05 2024-04-22T18:41:50.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564426+0000", 2024-04-22T18:41:50.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0654000000000001e-05 2024-04-22T18:41:50.147 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.147 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.147 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.147 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3863 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3579924] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:50.147 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:50.147 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.727887182000003, 2024-04-22T18:41:50.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20206560900000001, 2024-04-22T18:41:50.147 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.147 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.147 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:50.147 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:50.147 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:50.148 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3863 2024-04-22T18:41:50.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.148 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:50.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:41:50.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362715+0000", 2024-04-22T18:41:50.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:50.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362725+0000", 2024-04-22T18:41:50.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.006e-05 2024-04-22T18:41:50.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362732+0000", 2024-04-22T18:41:50.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6780000000000003e-06 2024-04-22T18:41:50.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362737+0000", 2024-04-22T18:41:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2089999999999996e-06 2024-04-22T18:41:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462875+0000", 2024-04-22T18:41:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013834100000001 2024-04-22T18:41:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462963+0000", 2024-04-22T18:41:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7792000000000002e-05 2024-04-22T18:41:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:50.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463127+0000", 2024-04-22T18:41:50.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000163688 2024-04-22T18:41:50.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463692+0000", 2024-04-22T18:41:50.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056466299999999997 2024-04-22T18:41:50.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564720+0000", 2024-04-22T18:41:50.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10102826199999999 2024-04-22T18:41:50.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564752+0000", 2024-04-22T18:41:50.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1936000000000001e-05 2024-04-22T18:41:50.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564783+0000", 2024-04-22T18:41:50.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0919e-05 2024-04-22T18:41:50.152 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.152 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.152 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.152 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3864 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:50.152 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:50.153 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.727853797999998, 2024-04-22T18:41:50.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20218423599999999, 2024-04-22T18:41:50.153 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.153 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:50.153 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:50.153 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:50.153 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:50.153 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3864 2024-04-22T18:41:50.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.153 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:50.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:50.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:41:50.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362753+0000", 2024-04-22T18:41:50.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6989999999999999e-06 2024-04-22T18:41:50.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:41:50.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:41:50.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462871+0000", 2024-04-22T18:41:50.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100113641 2024-04-22T18:41:50.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463385+0000", 2024-04-22T18:41:50.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051307100000000003 2024-04-22T18:41:50.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:50.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463457+0000", 2024-04-22T18:41:50.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2778e-05 2024-04-22T18:41:50.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564814+0000", 2024-04-22T18:41:50.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10135696299999999 2024-04-22T18:41:50.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564868+0000", 2024-04-22T18:41:50.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3872000000000002e-05 2024-04-22T18:41:50.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564935+0000", 2024-04-22T18:41:50.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6545999999999996e-05 2024-04-22T18:41:50.157 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.157 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.157 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.157 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4030 4.d 4:bee1184f:::smithi08937929-7:head [write 615222~472039] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:50.157 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:50.157 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.131404895999999, 2024-04-22T18:41:50.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109028217, 2024-04-22T18:41:50.157 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.158 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.158 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:50.158 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:50.158 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:50.158 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4030 2024-04-22T18:41:50.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.158 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:50.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:41:50.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959177+0000", 2024-04-22T18:41:50.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999776 2024-04-22T18:41:50.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959331+0000", 2024-04-22T18:41:50.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015334600000000001 2024-04-22T18:41:50.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959358+0000", 2024-04-22T18:41:50.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7220999999999999e-05 2024-04-22T18:41:50.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959371+0000", 2024-04-22T18:41:50.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3489000000000001e-05 2024-04-22T18:41:50.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059523+0000", 2024-04-22T18:41:50.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100152059 2024-04-22T18:41:50.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059588+0000", 2024-04-22T18:41:50.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4493999999999996e-05 2024-04-22T18:41:50.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:50.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059694+0000", 2024-04-22T18:41:50.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010584199999999999 2024-04-22T18:41:50.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068082+0000", 2024-04-22T18:41:50.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083880589999999998 2024-04-22T18:41:50.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068138+0000", 2024-04-22T18:41:50.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6196999999999997e-05 2024-04-22T18:41:50.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068185+0000", 2024-04-22T18:41:50.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7318000000000003e-05 2024-04-22T18:41:50.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068228+0000", 2024-04-22T18:41:50.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2376999999999998e-05 2024-04-22T18:41:50.162 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.162 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.163 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.163 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4031 4.d 4:bee1184f:::smithi08937929-7:head [write 1580652~489442] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:50.163 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:50.163 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.126103903999997, 2024-04-22T18:41:50.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20571598299999999, 2024-04-22T18:41:50.163 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.163 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.163 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:50.163 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:50.163 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:50.163 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4031 2024-04-22T18:41:50.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.164 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:50.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:41:50.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964480+0000", 2024-04-22T18:41:50.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999795 2024-04-22T18:41:50.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964628+0000", 2024-04-22T18:41:50.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148109 2024-04-22T18:41:50.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964651+0000", 2024-04-22T18:41:50.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3215999999999999e-05 2024-04-22T18:41:50.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964664+0000", 2024-04-22T18:41:50.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2673e-05 2024-04-22T18:41:50.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061135+0000", 2024-04-22T18:41:50.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096470822999999997 2024-04-22T18:41:50.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061228+0000", 2024-04-22T18:41:50.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3459000000000001e-05 2024-04-22T18:41:50.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:50.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061376+0000", 2024-04-22T18:41:50.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148128 2024-04-22T18:41:50.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068085+0000", 2024-04-22T18:41:50.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.006708632 2024-04-22T18:41:50.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170038+0000", 2024-04-22T18:41:50.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101952716 2024-04-22T18:41:50.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170173+0000", 2024-04-22T18:41:50.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013508699999999999 2024-04-22T18:41:50.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170216+0000", 2024-04-22T18:41:50.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3498999999999998e-05 2024-04-22T18:41:50.168 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.168 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.168 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.168 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4032 4.d 4:bee1184f:::smithi08937929-7:head [write 2768826~626587] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:50.168 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:50.168 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.118898797, 2024-04-22T18:41:50.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19941629799999999, 2024-04-22T18:41:50.169 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.169 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.169 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:50.169 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:50.169 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:50.169 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4032 2024-04-22T18:41:50.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.169 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:50.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:41:50.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971703+0000", 2024-04-22T18:41:50.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:50.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971996+0000", 2024-04-22T18:41:50.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029351300000000002 2024-04-22T18:41:50.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972012+0000", 2024-04-22T18:41:50.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5569e-05 2024-04-22T18:41:50.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972019+0000", 2024-04-22T18:41:50.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7950000000000008e-06 2024-04-22T18:41:50.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.063940+0000", 2024-04-22T18:41:50.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091920162 2024-04-22T18:41:50.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064021+0000", 2024-04-22T18:41:50.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1865000000000003e-05 2024-04-22T18:41:50.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:50.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064145+0000", 2024-04-22T18:41:50.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123893 2024-04-22T18:41:50.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068317+0000", 2024-04-22T18:41:50.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041720799999999999 2024-04-22T18:41:50.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171042+0000", 2024-04-22T18:41:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102724893 2024-04-22T18:41:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171085+0000", 2024-04-22T18:41:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2619e-05 2024-04-22T18:41:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171122+0000", 2024-04-22T18:41:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6838999999999997e-05 2024-04-22T18:41:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4033 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3815222] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.118575315000001, 2024-04-22T18:41:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297586038, 2024-04-22T18:41:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4033 2024-04-22T18:41:50.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.175 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:50.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:41:50.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972028+0000", 2024-04-22T18:41:50.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972032+0000", 2024-04-22T18:41:50.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8330000000000004e-06 2024-04-22T18:41:50.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972035+0000", 2024-04-22T18:41:50.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:41:50.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972038+0000", 2024-04-22T18:41:50.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9579999999999999e-06 2024-04-22T18:41:50.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067511+0000", 2024-04-22T18:41:50.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.095472308000000006 2024-04-22T18:41:50.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067607+0000", 2024-04-22T18:41:50.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6546999999999996e-05 2024-04-22T18:41:50.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:50.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067776+0000", 2024-04-22T18:41:50.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000169202 2024-04-22T18:41:50.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068470+0000", 2024-04-22T18:41:50.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069403100000000001 2024-04-22T18:41:50.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269540+0000", 2024-04-22T18:41:50.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20106934000000001 2024-04-22T18:41:50.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269574+0000", 2024-04-22T18:41:50.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4332000000000003e-05 2024-04-22T18:41:50.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269615+0000", 2024-04-22T18:41:50.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0988000000000001e-05 2024-04-22T18:41:50.179 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.179 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.179 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.179 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4034 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:41:50.179 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:50.180 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.118563678000001, 2024-04-22T18:41:50.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39788621000000002, 2024-04-22T18:41:50.180 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.180 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:50.180 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:50.180 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:50.180 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:50.180 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4034 2024-04-22T18:41:50.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.180 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:50.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:41:50.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972040+0000", 2024-04-22T18:41:50.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972042+0000", 2024-04-22T18:41:50.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.519e-06 2024-04-22T18:41:50.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972045+0000", 2024-04-22T18:41:50.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:41:50.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972048+0000", 2024-04-22T18:41:50.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.773e-06 2024-04-22T18:41:50.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068266+0000", 2024-04-22T18:41:50.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096218029999999996 2024-04-22T18:41:50.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:50.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068308+0000", 2024-04-22T18:41:50.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.21e-05 2024-04-22T18:41:50.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369703+0000", 2024-04-22T18:41:50.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301395578 2024-04-22T18:41:50.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369802+0000", 2024-04-22T18:41:50.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8788000000000004e-05 2024-04-22T18:41:50.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369927+0000", 2024-04-22T18:41:50.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012478200000000001 2024-04-22T18:41:50.184 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.184 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.184 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.184 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(4.1 124/123 [PushOp(4:83c955bf:::smithi08937929-28:head, version: 115'157, data_included: [1890685~548185,2699879~469889], data_size: 1018074, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:83c955bf:::smithi08937929-28:head@115'157, size: 3630345, copy_subset: [0~3630345], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3630345, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1890685, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-22T18:41:50.184 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:50.184 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.451995570000001, 2024-04-22T18:41:50.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20293218800000001, 2024-04-22T18:41:50.184 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.185 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:50.185 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:50.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:41:50.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638608+0000", 2024-04-22T18:41:50.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:50.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639519+0000", 2024-04-22T18:41:50.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091196199999999999 2024-04-22T18:41:50.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639537+0000", 2024-04-22T18:41:50.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7455000000000001e-05 2024-04-22T18:41:50.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.739639+0000", 2024-04-22T18:41:50.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100101942 2024-04-22T18:41:50.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840271+0000", 2024-04-22T18:41:50.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10063235299999999 2024-04-22T18:41:50.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840281+0000", 2024-04-22T18:41:50.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6509999999999996e-06 2024-04-22T18:41:50.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.841541+0000", 2024-04-22T18:41:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012600230000000001 2024-04-22T18:41:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4205 4.1 4:83c955bf:::smithi08937929-28:head [read 3065344~859648] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.163073728999997, 2024-04-22T18:41:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.303433565, 2024-04-22T18:41:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:50.188 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:50.189 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:50.189 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4205 2024-04-22T18:41:50.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.189 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:50.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:41:50.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927528+0000", 2024-04-22T18:41:50.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:50.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927535+0000", 2024-04-22T18:41:50.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1049999999999997e-06 2024-04-22T18:41:50.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927550+0000", 2024-04-22T18:41:50.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4365000000000001e-05 2024-04-22T18:41:50.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927560+0000", 2024-04-22T18:41:50.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0360000000000001e-05 2024-04-22T18:41:50.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028542+0000", 2024-04-22T18:41:50.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100981837 2024-04-22T18:41:50.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:50.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028551+0000", 2024-04-22T18:41:50.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6260000000000001e-06 2024-04-22T18:41:50.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229700+0000", 2024-04-22T18:41:50.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.201149453 2024-04-22T18:41:50.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229861+0000", 2024-04-22T18:41:50.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160605 2024-04-22T18:41:50.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.230964+0000", 2024-04-22T18:41:50.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00110351 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:359 3.0 e133/63 3:351d2474:::ba270d71d9c7ccc8d5c74eded70a7073f0e08c1ce33f7199f7c9db1158888734:head v 133'233, mlcod=132'228)", 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.188279301999998, 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20122315599999999, 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:50.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:41:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902324+0000", 2024-04-22T18:41:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902347+0000", 2024-04-22T18:41:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3577000000000001e-05 2024-04-22T18:41:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902356+0000", 2024-04-22T18:41:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8179999999999993e-06 2024-04-22T18:41:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.002447+0000", 2024-04-22T18:41:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100090255 2024-04-22T18:41:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102568+0000", 2024-04-22T18:41:50.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100121611 2024-04-22T18:41:50.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102592+0000", 2024-04-22T18:41:50.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3515000000000001e-05 2024-04-22T18:41:50.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103514+0000", 2024-04-22T18:41:50.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000922174 2024-04-22T18:41:50.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103548+0000", 2024-04-22T18:41:50.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4316999999999997e-05 2024-04-22T18:41:50.197 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.197 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.197 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.197 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:4289 3.0 e134/63 3:2a3d8f37:::9341e24c4a92f7ec719282d2faa5711f9644dee45a94ba5b4c4c1071ddc0e536:head v 134'235, mlcod=133'233)", 2024-04-22T18:41:50.197 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:50.197 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.243709789999997, 2024-04-22T18:41:50.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10669205499999999, 2024-04-22T18:41:50.197 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.197 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.197 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:50.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:41:50.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846891+0000", 2024-04-22T18:41:50.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:50.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847884+0000", 2024-04-22T18:41:50.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00099224800000000004 2024-04-22T18:41:50.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847900+0000", 2024-04-22T18:41:50.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6154000000000002e-05 2024-04-22T18:41:50.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.947999+0000", 2024-04-22T18:41:50.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009889199999999 2024-04-22T18:41:50.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948046+0000", 2024-04-22T18:41:50.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7617000000000002e-05 2024-04-22T18:41:50.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948072+0000", 2024-04-22T18:41:50.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5559999999999999e-05 2024-04-22T18:41:50.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953539+0000", 2024-04-22T18:41:50.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054668599999999996 2024-04-22T18:41:50.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953587+0000", 2024-04-22T18:41:50.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7781999999999998e-05 2024-04-22T18:41:50.201 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.201 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.201 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.201 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.7.13:94 3.0 e147/63 3:2812810a:::007b860ed7f2dc72f975d8732f59ce14b4563df44fd56cdf3a173500adaeb557:head v 147'260, mlcod=147'257)", 2024-04-22T18:41:50.201 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:41:50.201 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.657316584, 2024-04-22T18:41:50.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.200869664, 2024-04-22T18:41:50.201 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.202 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.202 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:41:50.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:41:50.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433287+0000", 2024-04-22T18:41:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433295+0000", 2024-04-22T18:41:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9929999999999995e-06 2024-04-22T18:41:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433302+0000", 2024-04-22T18:41:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2740000000000002e-06 2024-04-22T18:41:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533368+0000", 2024-04-22T18:41:50.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100066239 2024-04-22T18:41:50.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533415+0000", 2024-04-22T18:41:50.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6669000000000003e-05 2024-04-22T18:41:50.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533442+0000", 2024-04-22T18:41:50.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7143000000000001e-05 2024-04-22T18:41:50.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.634126+0000", 2024-04-22T18:41:50.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10068434900000001 2024-04-22T18:41:50.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.634157+0000", 2024-04-22T18:41:50.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1016000000000003e-05 2024-04-22T18:41:50.205 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.205 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.205 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.205 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:198 3.0 e147/63 3:2be6c676:::11b6849070c44d718872a26a0dc7d7224d7a6a98b28e0fde07a9f66143e32de6:head v 147'266, mlcod=147'257)", 2024-04-22T18:41:50.205 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:41:50.205 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.55654449, 2024-04-22T18:41:50.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106653649, 2024-04-22T18:41:50.206 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.206 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.206 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:41:50.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:41:50.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534059+0000", 2024-04-22T18:41:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534082+0000", 2024-04-22T18:41:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2291999999999999e-05 2024-04-22T18:41:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534086+0000", 2024-04-22T18:41:50.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1019999999999999e-06 2024-04-22T18:41:50.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534088+0000", 2024-04-22T18:41:50.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.525e-06 2024-04-22T18:41:50.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.635447+0000", 2024-04-22T18:41:50.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101358581 2024-04-22T18:41:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.635462+0000", 2024-04-22T18:41:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5327000000000001e-05 2024-04-22T18:41:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.640689+0000", 2024-04-22T18:41:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052266329999999996 2024-04-22T18:41:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.640713+0000", 2024-04-22T18:41:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4630999999999998e-05 2024-04-22T18:41:50.210 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.210 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.210 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.210 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:200 3.0 e147/63 3:19f9e72c:::45a155ef5bb5a968c7feaf289313c887c41e2de64b61f8816d05ad72d02346d9:head v 147'267, mlcod=147'257)", 2024-04-22T18:41:50.210 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:41:50.210 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.556513732999999, 2024-04-22T18:41:50.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109105273, 2024-04-22T18:41:50.210 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.210 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.210 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:41:50.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:41:50.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534090+0000", 2024-04-22T18:41:50.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534228+0000", 2024-04-22T18:41:50.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013768600000000001 2024-04-22T18:41:50.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534232+0000", 2024-04-22T18:41:50.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4900000000000002e-06 2024-04-22T18:41:50.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534235+0000", 2024-04-22T18:41:50.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.61e-06 2024-04-22T18:41:50.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.641544+0000", 2024-04-22T18:41:50.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107308717 2024-04-22T18:41:50.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.641557+0000", 2024-04-22T18:41:50.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2811e-05 2024-04-22T18:41:50.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.643173+0000", 2024-04-22T18:41:50.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00161608 2024-04-22T18:41:50.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.643196+0000", 2024-04-22T18:41:50.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3136e-05 2024-04-22T18:41:50.214 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.214 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.214 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.214 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.214 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.214 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:50.215 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:50.216 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:50.217+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:50.317 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:50.318+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:50.321 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:5189 4.19 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e173)", 2024-04-22T18:41:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:50.139380+0000", 2024-04-22T18:41:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.18366419, 2024-04-22T18:41:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.183686179, 2024-04-22T18:41:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for peered", 2024-04-22T18:41:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5189 2024-04-22T18:41:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139380+0000", 2024-04-22T18:41:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139380+0000", 2024-04-22T18:41:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139376+0000", 2024-04-22T18:41:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139390+0000", 2024-04-22T18:41:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3441999999999999e-05 2024-04-22T18:41:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139403+0000", 2024-04-22T18:41:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3845999999999999e-05 2024-04-22T18:41:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139414+0000", 2024-04-22T18:41:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0351e-05 2024-04-22T18:41:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139471+0000", 2024-04-22T18:41:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7444000000000001e-05 2024-04-22T18:41:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139479+0000", 2024-04-22T18:41:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8699999999999992e-06 2024-04-22T18:41:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:41:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 1 2024-04-22T18:41:50.325 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:50.331 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:50.418 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:50.419+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:50.433 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:50.433 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:50.433 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:50.442 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:50.507 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:41:50.508 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:41:50.519 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:50.520+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:50.520 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-04-22T18:41:49.734423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":269277700,"num_objects":262,"num_object_clones":0,"num_object_copies":524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":255,"num_whiteouts":0,"num_read":5097,"num_read_kb":4519656,"num_write":3054,"num_write_kb":729651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1260,"num_bytes_recovered":422250058,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":26,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3255,"ondisk_log_size":3255,"up":98,"acting":98,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":565772,"kb_used_data":357288,"kb_used_omap":9,"kb_used_meta":202998,"kb_avail":565665268,"statfs":{"total":579820584960,"available":579241234432,"internally_reserved":0,"allocated":365862912,"data_stored":372620140,"data_compressed":12780311,"data_compressed_allocated":13647872,"data_compressed_original":28150962,"omap_allocated":9557,"internal_metadata":207870635},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8189488,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":-86,"num_read_kb":-168076,"num_write":16,"num_write_kb":-3879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":3437471,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.739858"},"pg_stats":[{"pgid":"4.23","version":"155'91","reported_seq":393,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:44.918810+0000","last_change":"2024-04-22T18:41:44.918810+0000","last_active":"2024-04-22T18:41:44.918810+0000","last_peered":"2024-04-22T18:41:44.918810+0000","last_clean":"2024-04-22T18:41:44.918810+0000","last_became_active":"2024-04-22T18:41:44.918345+0000","last_became_peered":"2024-04-22T18:41:44.918345+0000","last_unstale":"2024-04-22T18:41:44.918810+0000","last_undegraded":"2024-04-22T18:41:44.918810+0000","last_fullsized":"2024-04-22T18:41:44.918810+0000","mapping_epoch":168,"log_start":"137'90","ondisk_log_start":"137'90","created":164,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":6,"last_scrub":"137'90","last_scrub_stamp":"2024-04-22T18:41:22.057812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:22.057812+0000","objects_scrubbed":2,"log_size":1,"log_dups_size":61,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655556,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":43,"num_read_kb":57909,"num_write":11,"num_write_kb":2755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5875215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.22","version":"149'42","reported_seq":352,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.300989+0000","last_change":"2024-04-22T18:41:40.908387+0000","last_active":"2024-04-22T18:41:45.300989+0000","last_peered":"2024-04-22T18:41:45.300989+0000","last_clean":"2024-04-22T18:41:45.300989+0000","last_became_active":"2024-04-22T18:41:40.908069+0000","last_became_peered":"2024-04-22T18:41:40.908069+0000","last_unstale":"2024-04-22T18:41:45.300989+0000","last_undegraded":"2024-04-22T18:41:45.300989+0000","last_fullsized":"2024-04-22T18:41:45.300989+0000","mapping_epoch":164,"log_start":"64'38","ondisk_log_start":"64'38","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":25,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":88762,"num_write":13,"num_write_kb":3449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5687418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"155'190","reported_seq":750,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:44.820690+0000","last_change":"2024-04-22T18:41:44.820690+0000","last_active":"2024-04-22T18:41:44.820690+0000","last_peered":"2024-04-22T18:41:44.820690+0000","last_clean":"2024-04-22T18:41:44.820690+0000","last_became_active":"2024-04-22T18:41:44.820321+0000","last_became_peered":"2024-04-22T18:41:44.820321+0000","last_unstale":"2024-04-22T18:41:44.820690+0000","last_undegraded":"2024-04-22T18:41:44.820690+0000","last_fullsized":"2024-04-22T18:41:44.820690+0000","mapping_epoch":168,"log_start":"147'185","ondisk_log_start":"147'185","created":164,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":6,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":5,"log_dups_size":161,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7042282,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":229510,"num_write":43,"num_write_kb":12083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":21653811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"162'34","reported_seq":293,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:44.717184+0000","last_change":"2024-04-22T18:41:41.107163+0000","last_active":"2024-04-22T18:41:44.717184+0000","last_peered":"2024-04-22T18:41:44.717184+0000","last_clean":"2024-04-22T18:41:44.717184+0000","last_became_active":"2024-04-22T18:41:41.106971+0000","last_became_peered":"2024-04-22T18:41:41.106971+0000","last_unstale":"2024-04-22T18:41:44.717184+0000","last_undegraded":"2024-04-22T18:41:44.717184+0000","last_fullsized":"2024-04-22T18:41:44.717184+0000","mapping_epoch":164,"log_start":"147'30","ondisk_log_start":"147'30","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'13","last_scrub_stamp":"2024-04-22T18:39:54.377980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.377980+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":30,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30260881099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1512663,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":68466,"num_write":12,"num_write_kb":3428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"163'88","reported_seq":404,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.201214+0000","last_change":"2024-04-22T18:41:41.008619+0000","last_active":"2024-04-22T18:41:45.201214+0000","last_peered":"2024-04-22T18:41:45.201214+0000","last_clean":"2024-04-22T18:41:45.201214+0000","last_became_active":"2024-04-22T18:41:41.008273+0000","last_became_peered":"2024-04-22T18:41:41.008273+0000","last_unstale":"2024-04-22T18:41:45.201214+0000","last_undegraded":"2024-04-22T18:41:45.201214+0000","last_fullsized":"2024-04-22T18:41:45.201214+0000","mapping_epoch":164,"log_start":"155'85","ondisk_log_start":"155'85","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":89,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2714397,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":146,"num_read_kb":138065,"num_write":47,"num_write_kb":12258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"147'53","reported_seq":82,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.611653+0000","last_change":"2024-04-22T18:41:11.253830+0000","last_active":"2024-04-22T18:41:45.611653+0000","last_peered":"2024-04-22T18:41:45.611653+0000","last_clean":"2024-04-22T18:41:45.611653+0000","last_became_active":"2024-04-22T18:41:11.253471+0000","last_became_peered":"2024-04-22T18:41:11.253471+0000","last_unstale":"2024-04-22T18:41:45.611653+0000","last_undegraded":"2024-04-22T18:41:45.611653+0000","last_fullsized":"2024-04-22T18:41:45.611653+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'23","last_scrub_stamp":"2024-04-22T18:40:20.587606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.587606+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10225366599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1840449,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":93,"num_read_kb":105860,"num_write":31,"num_write_kb":9939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5574713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"168'89","reported_seq":116,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:46.016571+0000","last_change":"2024-04-22T18:41:11.355276+0000","last_active":"2024-04-22T18:41:46.016571+0000","last_peered":"2024-04-22T18:41:46.016571+0000","last_clean":"2024-04-22T18:41:46.016571+0000","last_became_active":"2024-04-22T18:41:11.354958+0000","last_became_peered":"2024-04-22T18:41:11.354958+0000","last_unstale":"2024-04-22T18:41:46.016571+0000","last_undegraded":"2024-04-22T18:41:46.016571+0000","last_fullsized":"2024-04-22T18:41:46.016571+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"96'46","last_scrub_stamp":"2024-04-22T18:40:34.755106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:34.755106+0000","objects_scrubbed":2,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40192567099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":155,"num_read_kb":168254,"num_write":59,"num_write_kb":16492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8192627,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"132'123","reported_seq":489,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:43.817449+0000","last_change":"2024-04-22T18:41:11.469596+0000","last_active":"2024-04-22T18:41:43.817449+0000","last_peered":"2024-04-22T18:41:43.817449+0000","last_clean":"2024-04-22T18:41:43.817449+0000","last_became_active":"2024-04-22T18:41:11.469282+0000","last_became_peered":"2024-04-22T18:41:11.469282+0000","last_unstale":"2024-04-22T18:41:43.817449+0000","last_undegraded":"2024-04-22T18:41:43.817449+0000","last_fullsized":"2024-04-22T18:41:43.817449+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"121'119","last_scrub_stamp":"2024-04-22T18:40:55.573199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.573199+0000","objects_scrubbed":4,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32976813900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4521133,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":219,"num_read_kb":257792,"num_write":64,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3786236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"168'159","reported_seq":616,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:44.690179+0000","last_change":"2024-04-22T18:41:30.033168+0000","last_active":"2024-04-22T18:41:44.690179+0000","last_peered":"2024-04-22T18:41:44.690179+0000","last_clean":"2024-04-22T18:41:44.690179+0000","last_became_active":"2024-04-22T18:41:29.932575+0000","last_became_peered":"2024-04-22T18:41:29.932575+0000","last_unstale":"2024-04-22T18:41:44.690179+0000","last_undegraded":"2024-04-22T18:41:44.690179+0000","last_fullsized":"2024-04-22T18:41:44.690179+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":5,"last_scrub":"85'109","last_scrub_stamp":"2024-04-22T18:40:17.658363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:17.658363+0000","objects_scrubbed":5,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202176033,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4706026,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":314,"num_read_kb":352810,"num_write":99,"num_write_kb":27974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9711724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"163'115","reported_seq":442,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.200972+0000","last_change":"2024-04-22T18:41:10.956373+0000","last_active":"2024-04-22T18:41:45.200972+0000","last_peered":"2024-04-22T18:41:45.200972+0000","last_clean":"2024-04-22T18:41:45.200972+0000","last_became_active":"2024-04-22T18:41:10.956174+0000","last_became_peered":"2024-04-22T18:41:10.956174+0000","last_unstale":"2024-04-22T18:41:45.200972+0000","last_undegraded":"2024-04-22T18:41:45.200972+0000","last_fullsized":"2024-04-22T18:41:45.200972+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2806852,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":163,"num_read_kb":197186,"num_write":61,"num_write_kb":16155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4554596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"168'120","reported_seq":573,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:44.683355+0000","last_change":"2024-04-22T18:41:18.958159+0000","last_active":"2024-04-22T18:41:44.683355+0000","last_peered":"2024-04-22T18:41:44.683355+0000","last_clean":"2024-04-22T18:41:44.683355+0000","last_became_active":"2024-04-22T18:41:18.957008+0000","last_became_peered":"2024-04-22T18:41:18.957008+0000","last_unstale":"2024-04-22T18:41:44.683355+0000","last_undegraded":"2024-04-22T18:41:44.683355+0000","last_fullsized":"2024-04-22T18:41:44.683355+0000","mapping_epoch":142,"log_start":"84'88","ondisk_log_start":"84'88","created":110,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":32,"log_dups_size":106,"ondisk_log_size":32,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7033845,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":122,"num_read_kb":149003,"num_write":24,"num_write_kb":7118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9095252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"148'71","reported_seq":396,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:43.714434+0000","last_change":"2024-04-22T18:41:37.761338+0000","last_active":"2024-04-22T18:41:43.714434+0000","last_peered":"2024-04-22T18:41:43.714434+0000","last_clean":"2024-04-22T18:41:43.714434+0000","last_became_active":"2024-04-22T18:41:37.760987+0000","last_became_peered":"2024-04-22T18:41:37.760987+0000","last_unstale":"2024-04-22T18:41:43.714434+0000","last_undegraded":"2024-04-22T18:41:43.714434+0000","last_fullsized":"2024-04-22T18:41:43.714434+0000","mapping_epoch":161,"log_start":"79'52","ondisk_log_start":"79'52","created":110,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":61,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":857223,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":130,"num_read_kb":179919,"num_write":60,"num_write_kb":17002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9662508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"107'52","reported_seq":389,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:44.717209+0000","last_change":"2024-04-22T18:41:22.602374+0000","last_active":"2024-04-22T18:41:44.717209+0000","last_peered":"2024-04-22T18:41:44.717209+0000","last_clean":"2024-04-22T18:41:44.717209+0000","last_became_active":"2024-04-22T18:41:00.078669+0000","last_became_peered":"2024-04-22T18:41:00.078669+0000","last_unstale":"2024-04-22T18:41:44.717209+0000","last_undegraded":"2024-04-22T18:41:44.717209+0000","last_fullsized":"2024-04-22T18:41:44.717209+0000","mapping_epoch":125,"log_start":"83'49","ondisk_log_start":"83'49","created":110,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":5,"last_scrub":"107'52","last_scrub_stamp":"2024-04-22T18:41:22.602272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:22.602272+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:48.741276+0000","scrub_duration":0.40203006699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"166'100","reported_seq":495,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.814548+0000","last_change":"2024-04-22T18:41:45.814689+0000","last_active":"2024-04-22T18:41:45.814548+0000","last_peered":"2024-04-22T18:41:45.814548+0000","last_clean":"2024-04-22T18:41:45.814548+0000","last_became_active":"2024-04-22T18:41:30.012572+0000","last_became_peered":"2024-04-22T18:41:30.012572+0000","last_unstale":"2024-04-22T18:41:45.814548+0000","last_undegraded":"2024-04-22T18:41:45.814548+0000","last_fullsized":"2024-04-22T18:41:45.814548+0000","mapping_epoch":153,"log_start":"96'71","ondisk_log_start":"96'71","created":110,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":5,"last_scrub":"166'100","last_scrub_stamp":"2024-04-22T18:41:45.814522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:45.814522+0000","objects_scrubbed":1,"log_size":29,"log_dups_size":81,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:06.282120+0000","scrub_duration":0.40262363400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3227010,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"54'29","reported_seq":110,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:43.714456+0000","last_change":"2024-04-22T18:40:55.328642+0000","last_active":"2024-04-22T18:41:43.714456+0000","last_peered":"2024-04-22T18:41:43.714456+0000","last_clean":"2024-04-22T18:41:43.714456+0000","last_became_active":"2024-04-22T18:40:13.180144+0000","last_became_peered":"2024-04-22T18:40:13.180144+0000","last_unstale":"2024-04-22T18:41:43.714456+0000","last_undegraded":"2024-04-22T18:41:43.714456+0000","last_fullsized":"2024-04-22T18:41:43.714456+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'29","last_scrub_stamp":"2024-04-22T18:40:55.328529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.328529+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:10.967288+0000","scrub_duration":0.30217712000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"51'19","reported_seq":111,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:44.717208+0000","last_change":"2024-04-22T18:40:57.754351+0000","last_active":"2024-04-22T18:41:44.717208+0000","last_peered":"2024-04-22T18:41:44.717208+0000","last_clean":"2024-04-22T18:41:44.717208+0000","last_became_active":"2024-04-22T18:40:10.978840+0000","last_became_peered":"2024-04-22T18:40:10.978840+0000","last_unstale":"2024-04-22T18:41:44.717208+0000","last_undegraded":"2024-04-22T18:41:44.717208+0000","last_fullsized":"2024-04-22T18:41:44.717208+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"51'19","last_scrub_stamp":"2024-04-22T18:40:57.754261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:57.754261+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:17.927283+0000","scrub_duration":0.59184710600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"166'93","reported_seq":85,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:44.509438+0000","last_change":"2024-04-22T18:41:04.562945+0000","last_active":"2024-04-22T18:41:44.509438+0000","last_peered":"2024-04-22T18:41:44.509438+0000","last_clean":"2024-04-22T18:41:44.509438+0000","last_became_active":"2024-04-22T18:41:04.139159+0000","last_became_peered":"2024-04-22T18:41:04.139159+0000","last_unstale":"2024-04-22T18:41:44.509438+0000","last_undegraded":"2024-04-22T18:41:44.509438+0000","last_fullsized":"2024-04-22T18:41:44.509438+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'75","last_scrub_stamp":"2024-04-22T18:40:22.670296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.670296+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:49.126896+0000","scrub_duration":0.20951747300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218722,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":96410,"num_write":16,"num_write_kb":7324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12459808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"166'50","reported_seq":300,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.301151+0000","last_change":"2024-04-22T18:40:59.453830+0000","last_active":"2024-04-22T18:41:45.301151+0000","last_peered":"2024-04-22T18:41:45.301151+0000","last_clean":"2024-04-22T18:41:45.301151+0000","last_became_active":"2024-04-22T18:40:58.107873+0000","last_became_peered":"2024-04-22T18:40:58.107873+0000","last_unstale":"2024-04-22T18:41:45.301151+0000","last_undegraded":"2024-04-22T18:41:45.301151+0000","last_fullsized":"2024-04-22T18:41:45.301151+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3621975,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":115,"num_read_kb":106979,"num_write":32,"num_write_kb":8520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2641926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"169'153","reported_seq":505,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.925003+0000","last_change":"2024-04-22T18:40:26.844287+0000","last_active":"2024-04-22T18:41:45.925003+0000","last_peered":"2024-04-22T18:41:45.925003+0000","last_clean":"2024-04-22T18:41:45.925003+0000","last_became_active":"2024-04-22T18:40:26.742351+0000","last_became_peered":"2024-04-22T18:40:26.742351+0000","last_unstale":"2024-04-22T18:41:45.925003+0000","last_undegraded":"2024-04-22T18:41:45.925003+0000","last_fullsized":"2024-04-22T18:41:45.925003+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'113","last_scrub_stamp":"2024-04-22T18:40:20.291988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.291988+0000","objects_scrubbed":2,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10270805500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941291,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":63,"num_read_kb":42947,"num_write":48,"num_write_kb":13258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3940523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"31'5","reported_seq":204,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:44.817398+0000","last_change":"2024-04-22T18:41:41.832636+0000","last_active":"2024-04-22T18:41:44.817398+0000","last_peered":"2024-04-22T18:41:44.817398+0000","last_clean":"2024-04-22T18:41:44.817398+0000","last_became_active":"2024-04-22T18:39:59.265189+0000","last_became_peered":"2024-04-22T18:39:59.265189+0000","last_unstale":"2024-04-22T18:41:44.817398+0000","last_undegraded":"2024-04-22T18:41:44.817398+0000","last_fullsized":"2024-04-22T18:41:44.817398+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":5,"last_scrub":"31'5","last_scrub_stamp":"2024-04-22T18:40:10.212815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.212815+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015068670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":56,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:43.714515+0000","last_change":"2024-04-22T18:40:58.207745+0000","last_active":"2024-04-22T18:41:43.714515+0000","last_peered":"2024-04-22T18:41:43.714515+0000","last_clean":"2024-04-22T18:41:43.714515+0000","last_became_active":"2024-04-22T18:40:58.207451+0000","last_became_peered":"2024-04-22T18:40:58.207451+0000","last_unstale":"2024-04-22T18:41:43.714515+0000","last_undegraded":"2024-04-22T18:41:43.714515+0000","last_fullsized":"2024-04-22T18:41:43.714515+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:19.136933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:19.136933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"169'195","reported_seq":774,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:46.015465+0000","last_change":"2024-04-22T18:41:44.821952+0000","last_active":"2024-04-22T18:41:46.015465+0000","last_peered":"2024-04-22T18:41:46.015465+0000","last_clean":"2024-04-22T18:41:46.015465+0000","last_became_active":"2024-04-22T18:41:44.821076+0000","last_became_peered":"2024-04-22T18:41:44.821076+0000","last_unstale":"2024-04-22T18:41:46.015465+0000","last_undegraded":"2024-04-22T18:41:46.015465+0000","last_fullsized":"2024-04-22T18:41:46.015465+0000","mapping_epoch":168,"log_start":"147'185","ondisk_log_start":"147'185","created":20,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":10,"log_dups_size":161,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8134736,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":208,"num_read_kb":247495,"num_write":48,"num_write_kb":13433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":31242513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":181,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.401438+0000","last_change":"2024-04-22T18:41:33.768149+0000","last_active":"2024-04-22T18:41:45.401438+0000","last_peered":"2024-04-22T18:41:45.401438+0000","last_clean":"2024-04-22T18:41:45.401438+0000","last_became_active":"2024-04-22T18:41:04.037248+0000","last_became_peered":"2024-04-22T18:41:04.037248+0000","last_unstale":"2024-04-22T18:41:45.401438+0000","last_undegraded":"2024-04-22T18:41:45.401438+0000","last_fullsized":"2024-04-22T18:41:45.401438+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"162'34","reported_seq":307,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.590948+0000","last_change":"2024-04-22T18:41:45.590948+0000","last_active":"2024-04-22T18:41:45.590948+0000","last_peered":"2024-04-22T18:41:45.590948+0000","last_clean":"2024-04-22T18:41:45.590948+0000","last_became_active":"2024-04-22T18:41:41.006293+0000","last_became_peered":"2024-04-22T18:41:41.006293+0000","last_unstale":"2024-04-22T18:41:45.590948+0000","last_undegraded":"2024-04-22T18:41:45.590948+0000","last_fullsized":"2024-04-22T18:41:45.590948+0000","mapping_epoch":20,"log_start":"147'30","ondisk_log_start":"147'30","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'34","last_scrub_stamp":"2024-04-22T18:41:45.590834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:45.590834+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":30,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:48.626740+0000","scrub_duration":0.40273239100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025326,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":196,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:43.814617+0000","last_change":"2024-04-22T18:41:43.814617+0000","last_active":"2024-04-22T18:41:43.814617+0000","last_peered":"2024-04-22T18:41:43.814617+0000","last_clean":"2024-04-22T18:41:43.814617+0000","last_became_active":"2024-04-22T18:41:26.961835+0000","last_became_peered":"2024-04-22T18:41:26.961835+0000","last_unstale":"2024-04-22T18:41:43.814617+0000","last_undegraded":"2024-04-22T18:41:43.814617+0000","last_fullsized":"2024-04-22T18:41:43.814617+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:09.538542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:09.538542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.35201753200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"167'92","reported_seq":394,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:43.812039+0000","last_change":"2024-04-22T18:41:42.764452+0000","last_active":"2024-04-22T18:41:43.812039+0000","last_peered":"2024-04-22T18:41:43.812039+0000","last_clean":"2024-04-22T18:41:43.812039+0000","last_became_active":"2024-04-22T18:41:42.764184+0000","last_became_peered":"2024-04-22T18:41:42.764184+0000","last_unstale":"2024-04-22T18:41:43.812039+0000","last_undegraded":"2024-04-22T18:41:43.812039+0000","last_fullsized":"2024-04-22T18:41:43.812039+0000","mapping_epoch":166,"log_start":"137'90","ondisk_log_start":"137'90","created":20,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'90","last_scrub_stamp":"2024-04-22T18:41:22.057812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:22.057812+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":61,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":504193,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":57908,"num_write":11,"num_write_kb":2755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5439115,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":179,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.301224+0000","last_change":"2024-04-22T18:41:26.854653+0000","last_active":"2024-04-22T18:41:45.301224+0000","last_peered":"2024-04-22T18:41:45.301224+0000","last_clean":"2024-04-22T18:41:45.301224+0000","last_became_active":"2024-04-22T18:41:26.854399+0000","last_became_peered":"2024-04-22T18:41:26.854399+0000","last_unstale":"2024-04-22T18:41:45.301224+0000","last_undegraded":"2024-04-22T18:41:45.301224+0000","last_fullsized":"2024-04-22T18:41:45.301224+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"149'42","reported_seq":353,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.301159+0000","last_change":"2024-04-22T18:41:40.807780+0000","last_active":"2024-04-22T18:41:45.301159+0000","last_peered":"2024-04-22T18:41:45.301159+0000","last_clean":"2024-04-22T18:41:45.301159+0000","last_became_active":"2024-04-22T18:41:40.807410+0000","last_became_peered":"2024-04-22T18:41:40.807410+0000","last_unstale":"2024-04-22T18:41:45.301159+0000","last_undegraded":"2024-04-22T18:41:45.301159+0000","last_fullsized":"2024-04-22T18:41:45.301159+0000","mapping_epoch":129,"log_start":"64'38","ondisk_log_start":"64'38","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":25,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":88762,"num_write":12,"num_write_kb":3449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5687418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"169'384","reported_seq":1543,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:44.719819+0000","last_change":"2024-04-22T18:41:43.266638+0000","last_active":"2024-04-22T18:41:44.719819+0000","last_peered":"2024-04-22T18:41:44.719819+0000","last_clean":"2024-04-22T18:41:44.719819+0000","last_became_active":"2024-04-22T18:41:26.994048+0000","last_became_peered":"2024-04-22T18:41:26.994048+0000","last_unstale":"2024-04-22T18:41:44.719819+0000","last_undegraded":"2024-04-22T18:41:44.719819+0000","last_fullsized":"2024-04-22T18:41:44.719819+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'293","last_scrub_stamp":"2024-04-22T18:40:53.582354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:53.582354+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:14.875876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41123046,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":325,"num_read_kb":4195,"num_write":477,"num_write_kb":113967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":82068207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":140,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.301230+0000","last_change":"2024-04-22T18:41:26.657445+0000","last_active":"2024-04-22T18:41:45.301230+0000","last_peered":"2024-04-22T18:41:45.301230+0000","last_clean":"2024-04-22T18:41:45.301230+0000","last_became_active":"2024-04-22T18:41:26.657206+0000","last_became_peered":"2024-04-22T18:41:26.657206+0000","last_unstale":"2024-04-22T18:41:45.301230+0000","last_undegraded":"2024-04-22T18:41:45.301230+0000","last_fullsized":"2024-04-22T18:41:45.301230+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:55.558446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.558446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:00.817184+0000","scrub_duration":0.56126016300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"167'36","reported_seq":307,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.301199+0000","last_change":"2024-04-22T18:40:21.426510+0000","last_active":"2024-04-22T18:41:45.301199+0000","last_peered":"2024-04-22T18:41:45.301199+0000","last_clean":"2024-04-22T18:41:45.301199+0000","last_became_active":"2024-04-22T18:39:51.943071+0000","last_became_peered":"2024-04-22T18:39:51.943071+0000","last_unstale":"2024-04-22T18:41:45.301199+0000","last_undegraded":"2024-04-22T18:41:45.301199+0000","last_fullsized":"2024-04-22T18:41:45.301199+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-04-22T18:40:21.426350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.426350+0000","objects_scrubbed":1,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30334366400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294238,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":109,"num_read_kb":123704,"num_write":9,"num_write_kb":2718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"169'309","reported_seq":708,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-04-22T18:41:47.060495+0000","last_change":"2024-04-22T18:40:20.465005+0000","last_active":"2024-04-22T18:41:47.060495+0000","last_peered":"2024-04-22T18:41:47.060495+0000","last_clean":"2024-04-22T18:41:47.060495+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:41:47.060495+0000","last_undegraded":"2024-04-22T18:41:47.060495+0000","last_fullsized":"2024-04-22T18:41:47.060495+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'176","last_scrub_stamp":"2024-04-22T18:40:20.464799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.464799+0000","objects_scrubbed":31,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30629975799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382624,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":276,"num_read_kb":4957,"num_write":404,"num_write_kb":80545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":137,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:43.853482+0000","last_change":"2024-04-22T18:41:41.421087+0000","last_active":"2024-04-22T18:41:43.853482+0000","last_peered":"2024-04-22T18:41:43.853482+0000","last_clean":"2024-04-22T18:41:43.853482+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:41:43.853482+0000","last_undegraded":"2024-04-22T18:41:43.853482+0000","last_fullsized":"2024-04-22T18:41:43.853482+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:41.420996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:41.420996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:57.844851+0000","scrub_duration":0.20141637900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"162'72","reported_seq":431,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.301372+0000","last_change":"2024-04-22T18:41:32.860808+0000","last_active":"2024-04-22T18:41:45.301372+0000","last_peered":"2024-04-22T18:41:45.301372+0000","last_clean":"2024-04-22T18:41:45.301372+0000","last_became_active":"2024-04-22T18:41:32.860455+0000","last_became_peered":"2024-04-22T18:41:32.860455+0000","last_unstale":"2024-04-22T18:41:45.301372+0000","last_undegraded":"2024-04-22T18:41:45.301372+0000","last_fullsized":"2024-04-22T18:41:45.301372+0000","mapping_epoch":156,"log_start":"83'49","ondisk_log_start":"83'49","created":20,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":49,"ondisk_log_size":23,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3008494,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":95,"num_read_kb":112837,"num_write":38,"num_write_kb":11286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4519285,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"168'337","reported_seq":1347,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:44.582157+0000","last_change":"2024-04-22T18:41:41.108768+0000","last_active":"2024-04-22T18:41:44.582157+0000","last_peered":"2024-04-22T18:41:44.582157+0000","last_clean":"2024-04-22T18:41:44.582157+0000","last_became_active":"2024-04-22T18:41:41.007340+0000","last_became_peered":"2024-04-22T18:41:41.007340+0000","last_unstale":"2024-04-22T18:41:44.582157+0000","last_undegraded":"2024-04-22T18:41:44.582157+0000","last_fullsized":"2024-04-22T18:41:44.582157+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'209","last_scrub_stamp":"2024-04-22T18:40:18.683442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:18.683442+0000","objects_scrubbed":24,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:45.950082+0000","scrub_duration":0.204344365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39035424,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":294,"num_read_kb":9112,"num_write":423,"num_write_kb":104979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":67954083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":171,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:43.714535+0000","last_change":"2024-04-22T18:40:22.521828+0000","last_active":"2024-04-22T18:41:43.714535+0000","last_peered":"2024-04-22T18:41:43.714535+0000","last_clean":"2024-04-22T18:41:43.714535+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:41:43.714535+0000","last_undegraded":"2024-04-22T18:41:43.714535+0000","last_fullsized":"2024-04-22T18:41:43.714535+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.521638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30190717299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"169'111","reported_seq":484,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:46.326489+0000","last_change":"2024-04-22T18:41:32.061113+0000","last_active":"2024-04-22T18:41:46.326489+0000","last_peered":"2024-04-22T18:41:46.326489+0000","last_clean":"2024-04-22T18:41:46.326489+0000","last_became_active":"2024-04-22T18:41:31.858774+0000","last_became_peered":"2024-04-22T18:41:31.858774+0000","last_unstale":"2024-04-22T18:41:46.326489+0000","last_undegraded":"2024-04-22T18:41:46.326489+0000","last_fullsized":"2024-04-22T18:41:46.326489+0000","mapping_epoch":155,"log_start":"96'71","ondisk_log_start":"96'71","created":20,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'66","last_scrub_stamp":"2024-04-22T18:40:08.763512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:08.763512+0000","objects_scrubbed":2,"log_size":40,"log_dups_size":81,"ondisk_log_size":40,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.22461577699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5664091,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":139618,"num_write":38,"num_write_kb":11929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1133573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"169'328","reported_seq":1010,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:44.719816+0000","last_change":"2024-04-22T18:41:19.866530+0000","last_active":"2024-04-22T18:41:44.719816+0000","last_peered":"2024-04-22T18:41:44.719816+0000","last_clean":"2024-04-22T18:41:44.719816+0000","last_became_active":"2024-04-22T18:41:19.765392+0000","last_became_peered":"2024-04-22T18:41:19.765392+0000","last_unstale":"2024-04-22T18:41:44.719816+0000","last_undegraded":"2024-04-22T18:41:44.719816+0000","last_fullsized":"2024-04-22T18:41:44.719816+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'212","last_scrub_stamp":"2024-04-22T18:40:21.870855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.870855+0000","objects_scrubbed":46,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43117026,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":302,"num_read_kb":12417,"num_write":418,"num_write_kb":91859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":43826500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":192,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:43.853488+0000","last_change":"2024-04-22T18:41:30.504800+0000","last_active":"2024-04-22T18:41:43.853488+0000","last_peered":"2024-04-22T18:41:43.853488+0000","last_clean":"2024-04-22T18:41:43.853488+0000","last_became_active":"2024-04-22T18:40:02.741862+0000","last_became_peered":"2024-04-22T18:40:02.741862+0000","last_unstale":"2024-04-22T18:41:43.853488+0000","last_undegraded":"2024-04-22T18:41:43.853488+0000","last_fullsized":"2024-04-22T18:41:43.853488+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:30.504686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:30.504686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:57.436870+0000","scrub_duration":0.30171718199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":366,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:43.714552+0000","last_change":"2024-04-22T18:41:41.759987+0000","last_active":"2024-04-22T18:41:43.714552+0000","last_peered":"2024-04-22T18:41:43.714552+0000","last_clean":"2024-04-22T18:41:43.714552+0000","last_became_active":"2024-04-22T18:41:26.757958+0000","last_became_peered":"2024-04-22T18:41:26.757958+0000","last_unstale":"2024-04-22T18:41:43.714552+0000","last_undegraded":"2024-04-22T18:41:43.714552+0000","last_fullsized":"2024-04-22T18:41:43.714552+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:39:53.346787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:39:53.346787+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20273052499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"137'75","reported_seq":89,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:43.717430+0000","last_change":"2024-04-22T18:41:04.681901+0000","last_active":"2024-04-22T18:41:43.717430+0000","last_peered":"2024-04-22T18:41:43.717430+0000","last_clean":"2024-04-22T18:41:43.717430+0000","last_became_active":"2024-04-22T18:41:04.227104+0000","last_became_peered":"2024-04-22T18:41:04.227104+0000","last_unstale":"2024-04-22T18:41:43.717430+0000","last_undegraded":"2024-04-22T18:41:43.717430+0000","last_fullsized":"2024-04-22T18:41:43.717430+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'38","last_scrub_stamp":"2024-04-22T18:39:55.746067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:55.746067+0000","objects_scrubbed":1,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60877772299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6168268,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":192624,"num_write":57,"num_write_kb":16481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15782313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"164'59","reported_seq":375,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:44.489542+0000","last_change":"2024-04-22T18:41:35.835744+0000","last_active":"2024-04-22T18:41:44.489542+0000","last_peered":"2024-04-22T18:41:44.489542+0000","last_clean":"2024-04-22T18:41:44.489542+0000","last_became_active":"2024-04-22T18:41:35.834928+0000","last_became_peered":"2024-04-22T18:41:35.834928+0000","last_unstale":"2024-04-22T18:41:44.489542+0000","last_undegraded":"2024-04-22T18:41:44.489542+0000","last_fullsized":"2024-04-22T18:41:44.489542+0000","mapping_epoch":159,"log_start":"79'52","ondisk_log_start":"79'52","created":20,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'57","last_scrub_stamp":"2024-04-22T18:40:56.376497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:56.376497+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":61,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:57.469237+0000","scrub_duration":0.30191299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3832029,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":46990,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1913105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"167'154","reported_seq":614,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:43.719147+0000","last_change":"2024-04-22T18:41:17.760985+0000","last_active":"2024-04-22T18:41:43.719147+0000","last_peered":"2024-04-22T18:41:43.719147+0000","last_clean":"2024-04-22T18:41:43.719147+0000","last_became_active":"2024-04-22T18:41:17.759812+0000","last_became_peered":"2024-04-22T18:41:17.759812+0000","last_unstale":"2024-04-22T18:41:43.719147+0000","last_undegraded":"2024-04-22T18:41:43.719147+0000","last_fullsized":"2024-04-22T18:41:43.719147+0000","mapping_epoch":141,"log_start":"84'88","ondisk_log_start":"84'88","created":20,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":66,"log_dups_size":106,"ondisk_log_size":66,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7420472,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":148630,"num_write":63,"num_write_kb":21335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12038689,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"167'121","reported_seq":457,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.301068+0000","last_change":"2024-04-22T18:41:11.054130+0000","last_active":"2024-04-22T18:41:45.301068+0000","last_peered":"2024-04-22T18:41:45.301068+0000","last_clean":"2024-04-22T18:41:45.301068+0000","last_became_active":"2024-04-22T18:41:11.053760+0000","last_became_peered":"2024-04-22T18:41:11.053760+0000","last_unstale":"2024-04-22T18:41:45.301068+0000","last_undegraded":"2024-04-22T18:41:45.301068+0000","last_fullsized":"2024-04-22T18:41:45.301068+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7059022,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":182,"num_read_kb":210386,"num_write":65,"num_write_kb":17255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4554596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"167'153","reported_seq":605,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:43.812122+0000","last_change":"2024-04-22T18:41:31.959627+0000","last_active":"2024-04-22T18:41:43.812122+0000","last_peered":"2024-04-22T18:41:43.812122+0000","last_clean":"2024-04-22T18:41:43.812122+0000","last_became_active":"2024-04-22T18:41:31.959307+0000","last_became_peered":"2024-04-22T18:41:31.959307+0000","last_unstale":"2024-04-22T18:41:43.812122+0000","last_undegraded":"2024-04-22T18:41:43.812122+0000","last_fullsized":"2024-04-22T18:41:43.812122+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'109","last_scrub_stamp":"2024-04-22T18:40:17.658363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:17.658363+0000","objects_scrubbed":5,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202176033,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4672412,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":297,"num_read_kb":338551,"num_write":91,"num_write_kb":26612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":11460299,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"168'131","reported_seq":535,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:44.611153+0000","last_change":"2024-04-22T18:41:12.783644+0000","last_active":"2024-04-22T18:41:44.611153+0000","last_peered":"2024-04-22T18:41:44.611153+0000","last_clean":"2024-04-22T18:41:44.611153+0000","last_became_active":"2024-04-22T18:41:11.956312+0000","last_became_peered":"2024-04-22T18:41:11.956312+0000","last_unstale":"2024-04-22T18:41:44.611153+0000","last_undegraded":"2024-04-22T18:41:44.611153+0000","last_fullsized":"2024-04-22T18:41:44.611153+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'119","last_scrub_stamp":"2024-04-22T18:40:55.573199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.573199+0000","objects_scrubbed":4,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32976813900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832111,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":304,"num_read_kb":395814,"num_write":74,"num_write_kb":20641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9347897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"133'66","reported_seq":77,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-22T18:41:43.817557+0000","last_change":"2024-04-22T18:41:19.456295+0000","last_active":"2024-04-22T18:41:43.817557+0000","last_peered":"2024-04-22T18:41:43.817557+0000","last_clean":"2024-04-22T18:41:43.817557+0000","last_became_active":"2024-04-22T18:41:11.154301+0000","last_became_peered":"2024-04-22T18:41:11.154301+0000","last_unstale":"2024-04-22T18:41:43.817557+0000","last_undegraded":"2024-04-22T18:41:43.817557+0000","last_fullsized":"2024-04-22T18:41:43.817557+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'66","last_scrub_stamp":"2024-04-22T18:41:19.456160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:19.456160+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:36.925204+0000","scrub_duration":0.20146846400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"145'51","reported_seq":88,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.675139+0000","last_change":"2024-04-22T18:41:45.675260+0000","last_active":"2024-04-22T18:41:45.675139+0000","last_peered":"2024-04-22T18:41:45.675139+0000","last_clean":"2024-04-22T18:41:45.675139+0000","last_became_active":"2024-04-22T18:41:11.253409+0000","last_became_peered":"2024-04-22T18:41:11.253409+0000","last_unstale":"2024-04-22T18:41:45.675139+0000","last_undegraded":"2024-04-22T18:41:45.675139+0000","last_fullsized":"2024-04-22T18:41:45.675139+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'51","last_scrub_stamp":"2024-04-22T18:41:45.675122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:45.675122+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:09.124003+0000","scrub_duration":0.40271887499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"163'88","reported_seq":405,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-22T18:41:45.201220+0000","last_change":"2024-04-22T18:41:41.109970+0000","last_active":"2024-04-22T18:41:45.201220+0000","last_peered":"2024-04-22T18:41:45.201220+0000","last_clean":"2024-04-22T18:41:45.201220+0000","last_became_active":"2024-04-22T18:41:41.109681+0000","last_became_peered":"2024-04-22T18:41:41.109681+0000","last_unstale":"2024-04-22T18:41:45.201220+0000","last_undegraded":"2024-04-22T18:41:45.201220+0000","last_fullsized":"2024-04-22T18:41:45.201220+0000","mapping_epoch":20,"log_start":"155'85","ondisk_log_start":"155'85","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":89,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2714396,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":152,"num_read_kb":145684,"num_write":47,"num_write_kb":12257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":36,"stat_sum":{"num_bytes":113029212,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":3784,"num_read_kb":4488875,"num_write":1173,"num_write_kb":335885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":205955160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":26,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":187736064,"data_stored":187891771,"data_compressed":1037892,"data_compressed_allocated":1122304,"data_compressed_original":2198706,"omap_allocated":0,"internal_metadata":0},"log_size":1780,"ondisk_log_size":1780,"up":72,"acting":72,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":155658120,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":1197,"num_read_kb":30681,"num_write":1722,"num_write_kb":391350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1149,"num_bytes_recovered":214708114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249724928,"data_stored":264136596,"data_compressed":14626013,"data_compressed_allocated":15568896,"data_compressed_original":30834688,"omap_allocated":0,"internal_metadata":0},"log_size":1358,"ondisk_log_size":1358,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":2115712,"data_compressed":122550,"data_compressed_allocated":163840,"data_compressed_original":2084864,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53344,"kb_used_data":21404,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94318496,"statfs":{"total":96636764160,"available":96582139904,"internally_reserved":0,"allocated":21917696,"data_stored":21505306,"data_compressed":211962,"data_compressed_allocated":241664,"data_compressed_original":970752,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":2.5870000000000002,"15min":1.1439999999999999},"min":{"1min":0.182,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":26.568000000000001,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":6.3600000000000003,"5min":6.766,"15min":5.2599999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":95.918999999999997,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.36699999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":3.8050000000000002,"5min":3.984,"15min":4.5709999999999997},"min":{"1min":0.29299999999999998,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":66.814999999999998,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":6.6040000000000001,"5min":12.141,"15min":12.744999999999999},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":100.242,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7130000000000001,"5min":4.0730000000000004,"15min":4.0810000000000004},"min":{"1min":0.27400000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":97.662000000000006,"5min":97.662000000000006,"15min":97.662000000000006},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.47399999999999998}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7560000000000002,"5min":4.1289999999999996,"15min":4.1429999999999998},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":97.686999999999998,"5min":97.686999999999998,"15min":97.686999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.53500000000000003}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":8.2089999999999996,"5min":5.4509999999999996,"15min":4.7889999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":78.149000000000001,"5min":85.509,"15min":85.509},"last":0.41299999999999998},{"interface":"front","average":{"1min":7.7130000000000001,"5min":16.856000000000002,"15min":20.866},"min":{"1min":0.26000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":97.590000000000003,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.48499999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":2.0270000000000001,"15min":2.7229999999999999},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.4730000000000001,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.6679999999999999,"5min":5.2939999999999996,"15min":6.5389999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":23.007999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.501}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":11.756,"15min":12.769},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":23.027000000000001,"5min":139.428,"15min":139.428},"last":0.46000000000000002},{"interface":"front","average":{"1min":7.7800000000000002,"5min":4.3070000000000004,"15min":4.1909999999999998},"min":{"1min":0.29199999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":97.635999999999996,"5min":97.635999999999996,"15min":97.635999999999996},"last":0.52300000000000002}]}]},{"osd":6,"up_from":0,"seq":55834574888,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574888,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":160136,"kb_used_data":122696,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94211704,"statfs":{"total":96636764160,"available":96472784896,"internally_reserved":0,"allocated":125640704,"data_stored":129298115,"data_compressed":4963321,"data_compressed_allocated":5300224,"data_compressed_original":10430041,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":13.115,"15min":15.470000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":38.899999999999999,"5min":252.96199999999999,"15min":252.96199999999999},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":9.282,"15min":8.8059999999999992},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63300000000000001,"5min":187.148,"15min":187.148},"last":0.38}]},{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":2.508,"15min":1.115},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":9.141,"5min":96.486999999999995,"15min":96.486999999999995},"last":0.28399999999999997},{"interface":"front","average":{"1min":22.393000000000001,"5min":11.545999999999999,"15min":7.4390000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":310.61500000000001,"5min":310.61500000000001,"15min":310.61500000000001},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":8.8989999999999991,"15min":8.6790000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.8330000000000002,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.995,"5min":2.4199999999999999,"15min":1.1040000000000001},"min":{"1min":0.31900000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.1399999999999997,"5min":96.471999999999994,"15min":96.471999999999994},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":11.172000000000001,"5min":8.9710000000000001,"15min":5.6710000000000003},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":122.102,"5min":252.91900000000001,"15min":252.91900000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":17.242000000000001,"15min":19.725999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59899999999999998,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.40999999999999998}]},{"osd":5,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":2.3380000000000001,"15min":1.034},"min":{"1min":0.25800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":7.1559999999999997,"5min":96.408000000000001,"15min":96.408000000000001},"last":0.318},{"interface":"front","average":{"1min":2.6960000000000002,"5min":3.048,"15min":3.7080000000000002},"min":{"1min":0.36199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":38.581000000000003,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.30199999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":5.8289999999999997,"5min":14.667,"15min":13.662000000000001},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":87.706999999999994,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.83199999999999996,"5min":2.3820000000000001,"15min":1.0880000000000001},"min":{"1min":0.35699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.1159999999999997,"5min":96.438000000000002,"15min":96.438000000000002},"last":0.34599999999999997}]},{"osd":7,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":9.5749999999999993,"15min":8.8979999999999997},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68000000000000005,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.9790000000000001,"5min":2.6419999999999999,"15min":1.202},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":19.518999999999998,"5min":96.456999999999994,"15min":96.456999999999994},"last":0.38800000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76424,"kb_used_data":41216,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94295416,"statfs":{"total":96636764160,"available":96558505984,"internally_reserved":0,"allocated":42205184,"data_stored":42545458,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":15.041,"5min":10.538,"15min":11.646000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":93.131,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.23400000000000001},{"interface":"front","average":{"1min":7.077,"5min":1.6659999999999999,"15min":0.77400000000000002},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":87.305999999999997,"5min":87.305999999999997,"15min":87.305999999999997},"last":0.26900000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":10.558,"15min":6.8819999999999997},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":78.805000000000007,"5min":197.02099999999999,"15min":197.02099999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":8.9800000000000004,"5min":2.3210000000000002,"15min":0.997},"min":{"1min":0.20999999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":78.393000000000001,"5min":78.393000000000001,"15min":78.393000000000001},"last":0.34200000000000003}]},{"osd":2,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":2.3439999999999999,"15min":1.452},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":47.534999999999997,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":6.4770000000000003,"5min":1.728,"15min":0.85899999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":78.421999999999997,"5min":78.421999999999997,"15min":78.421999999999997},"last":0.29099999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1800000000000002,"5min":2.9940000000000002,"15min":3.7360000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":26.141999999999999,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.623},{"interface":"front","average":{"1min":4.5629999999999997,"5min":4.6289999999999996,"15min":4.3010000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":47.616999999999997,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.61099999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.8279999999999998,"5min":2.5049999999999999,"15min":1.623},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":47.566000000000003,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.64400000000000002},{"interface":"front","average":{"1min":6.6379999999999999,"5min":2.2890000000000001,"15min":1.601},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.480999999999995,"5min":78.480999999999995,"15min":78.480999999999995},"last":0.59699999999999998}]},{"osd":6,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.50900000000000001,"15min":0.53100000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.748,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":2.8500000000000001,"5min":2.871,"15min":1.3109999999999999},"min":{"1min":0.377,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":47.590000000000003,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.67000000000000004}]},{"osd":7,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":6.6189999999999998,"5min":1.903,"15min":1.0229999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.453999999999994,"5min":78.453999999999994,"15min":78.453999999999994},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69299999999999995,"5min":0.995,"15min":0.995},"last":0.56299999999999994}]}]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58224,"kb_used_data":27108,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94313616,"statfs":{"total":96636764160,"available":96577142784,"internally_reserved":0,"allocated":27758592,"data_stored":28204744,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":6.7439999999999998,"15min":7.6660000000000004},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":84.338999999999999,"5min":149.03,"15min":149.03},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.28299999999999997,"5min":1.3089999999999999,"15min":1.7589999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.62,"5min":31.866,"15min":31.866},"last":0.32600000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":9.4849999999999994,"5min":5.2489999999999997,"15min":2.4380000000000002},"min":{"1min":0.216,"5min":0.193,"15min":0.193},"max":{"1min":91.912000000000006,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.24299999999999999},{"interface":"front","average":{"1min":12.752000000000001,"5min":15.912000000000001,"15min":15.227},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":126.42,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.28000000000000003}]},{"osd":3,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":6.2830000000000004,"15min":2.3359999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":129.041,"5min":147.721,"15min":147.721},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.26800000000000002}]},{"osd":4,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":8.1280000000000001,"5min":13.57,"15min":17.038},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":90.882999999999996,"5min":160.584,"15min":160.584},"last":0.44800000000000001},{"interface":"front","average":{"1min":5.9349999999999996,"5min":4.8520000000000003,"15min":3.2490000000000001},"min":{"1min":0.32500000000000001,"5min":0.315,"15min":0.315},"max":{"1min":93.962999999999994,"5min":97.887,"15min":97.887},"last":0.38}]},{"osd":5,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":2.0339999999999998,"15min":2.7120000000000002},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.879,"5min":48.631,"15min":48.631},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.50900000000000001,"5min":2.2890000000000001,"15min":3.0800000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.91800000000000004,"5min":56.18,"15min":56.18},"last":0.37}]},{"osd":6,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.3540000000000001,"5min":3.4670000000000001,"15min":1.5129999999999999},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":92.028000000000006,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.436}]},{"osd":7,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.29,"5min":1.5800000000000001,"15min":0.91000000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":90.944999999999993,"5min":90.944999999999993,"15min":90.944999999999993},"last":0.40799999999999997},{"interface":"front","average":{"1min":7.4640000000000004,"5min":3.923,"15min":1.6970000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":92.048000000000002,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.42599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":99748,"kb_used_data":63872,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94272092,"statfs":{"total":96636764160,"available":96534622208,"internally_reserved":0,"allocated":65404928,"data_stored":65860936,"data_compressed":1653944,"data_compressed_allocated":1757184,"data_compressed_original":3606105,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":5.883,"15min":7.2000000000000002},"min":{"1min":0.20000000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":27.116,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.222},{"interface":"front","average":{"1min":5.8040000000000003,"5min":6.6870000000000003,"15min":7.4710000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":94.046000000000006,"5min":94.046000000000006,"15min":94.046000000000006},"last":0.26900000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.679000000000002,"5min":13.159000000000001,"15min":10.119},"min":{"1min":0.23000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":406.68799999999999,"5min":406.68799999999999,"15min":406.68799999999999},"last":0.307},{"interface":"front","average":{"1min":1.8700000000000001,"5min":0.61899999999999999,"15min":0.41099999999999998},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19},"max":{"1min":27.094999999999999,"5min":27.094999999999999,"15min":27.094999999999999},"last":0.29099999999999998}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":15.968,"5min":3.464,"15min":1.3759999999999999},"min":{"1min":0.29099999999999998,"5min":0.216,"15min":0.216},"max":{"1min":228.90100000000001,"5min":228.90100000000001,"15min":228.90100000000001},"last":10.741},{"interface":"front","average":{"1min":21.481999999999999,"5min":13.061999999999999,"15min":14.015000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":196.82900000000001,"5min":231.864,"15min":231.864},"last":0.32200000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.823,"5min":17.213000000000001,"15min":14.393000000000001},"min":{"1min":0.316,"5min":0.309,"15min":0.309},"max":{"1min":406.71199999999999,"5min":406.71199999999999,"15min":406.71199999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":25.027000000000001,"5min":5.3979999999999997,"15min":2.1320000000000001},"min":{"1min":0.34499999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":186.89500000000001,"5min":186.89500000000001,"15min":186.89500000000001},"last":186.89500000000001}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":9.7430000000000003,"15min":12.628},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":94.183000000000007,"5min":138.089,"15min":138.089},"last":0.41399999999999998},{"interface":"front","average":{"1min":39.863999999999997,"5min":20.117000000000001,"15min":20.09},"min":{"1min":0.35499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":406.72899999999998,"5min":406.72899999999998,"15min":406.72899999999998},"last":0.46899999999999997}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":6.0010000000000003,"5min":3.8359999999999999,"15min":3.1299999999999999},"min":{"1min":0.34000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":94.302999999999997,"5min":94.302999999999997,"15min":94.302999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":43.036000000000001,"5min":19.713999999999999,"15min":17.591999999999999},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":427.69299999999998,"5min":427.69299999999998,"15min":427.69299999999998},"last":0.50700000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":2.0720000000000001,"5min":0.82799999999999996,"15min":0.626},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":27.253,"5min":27.253,"15min":27.253},"last":0.45600000000000002},{"interface":"front","average":{"1min":6.0090000000000003,"5min":3.129,"15min":3.069},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":94.260999999999996,"5min":94.260999999999996,"15min":94.260999999999996},"last":0.44400000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117896,"kb_used_data":80992,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94253944,"statfs":{"total":96636764160,"available":96516038656,"internally_reserved":0,"allocated":82935808,"data_stored":85205581,"data_compressed":3073787,"data_compressed_allocated":3297280,"data_compressed_original":6955008,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":9.7919999999999998,"5min":9.5660000000000007,"15min":7.6950000000000003},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":127.30500000000001,"5min":133.09899999999999,"15min":133.09899999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":27.553999999999998,"5min":11.506,"15min":4.8399999999999999},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":451.50799999999998,"5min":451.50799999999998,"15min":451.50799999999998},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":1.5309999999999999,"15min":1.5469999999999999},"min":{"1min":0.217,"5min":0.185,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":39.906999999999996,"15min":39.906999999999996},"last":0.443},{"interface":"front","average":{"1min":0.32600000000000001,"5min":2.2090000000000001,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.41299999999999998,"5min":94.820999999999998,"15min":94.820999999999998},"last":0.373}]},{"osd":3,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":6.1859999999999999,"5min":3.5299999999999998,"15min":1.486},"min":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":74.963999999999999,"5min":99.332999999999998,"15min":99.332999999999998},"last":0.42199999999999999},{"interface":"front","average":{"1min":16.207000000000001,"5min":9.2159999999999993,"15min":4.1139999999999999},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":273.65100000000001,"5min":297.82100000000003,"15min":297.82100000000003},"last":0.32900000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":5.9980000000000002,"5min":3.4809999999999999,"15min":1.4970000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":96.528999999999996,"5min":96.528999999999996,"15min":96.528999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.45100000000000001,"5min":8.8000000000000007,"15min":9.5410000000000004},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":160.02199999999999,"15min":160.02199999999999},"last":0.46500000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":2.1800000000000002,"15min":1.8899999999999999},"min":{"1min":0.35399999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.60399999999999998,"5min":41.645000000000003,"15min":41.645000000000003},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":10.452999999999999,"15min":12.787000000000001},"min":{"1min":0.32300000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66200000000000003,"5min":282.197,"15min":282.197},"last":0.58999999999999997}]},{"osd":6,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.7770000000000001,"15min":1.385},"min":{"1min":0.36199999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58299999999999996,"5min":99.355000000000004,"15min":99.355000000000004},"last":0.495},{"interface":"front","average":{"1min":0.499,"5min":2.8929999999999998,"15min":2.129},"min":{"1min":0.35899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.72999999999999998,"5min":52.762,"15min":52.762},"last":0.68100000000000005}]},{"osd":7,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":4.8929999999999998,"15min":4.7489999999999997},"min":{"1min":0.34499999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.68700000000000006,"5min":94.662999999999997,"15min":94.662999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":9.0709999999999997,"5min":4.2939999999999996,"15min":1.901},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":157.089,"5min":157.089,"15min":157.089},"last":0.66200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":81391616,"data_stored":85790877,"data_compressed":4527904,"data_compressed_allocated":4829184,"data_compressed_original":9498624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":223399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27324416,"data_stored":28849251,"data_compressed":1581247,"data_compressed_allocated":1675264,"data_compressed_original":3284992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26755072,"data_stored":26700452,"data_compressed":52333,"data_compressed_allocated":57344,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43991040,"data_stored":44050507,"data_compressed":314091,"data_compressed_allocated":335872,"data_compressed_original":616025,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6438912,"data_stored":6403215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13045760,"data_stored":12971930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":42352640,"data_stored":42633281,"data_compressed":435417,"data_compressed_allocated":471040,"data_compressed_original":931417,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33280000,"data_stored":33118894,"data_compressed":53657,"data_compressed_allocated":57344,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1929216,"data_stored":1913105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":19943424,"data_stored":20100387,"data_compressed":182394,"data_compressed_allocated":200704,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:41:50.523 INFO:tasks.thrashosds.thrasher:Setting 4.22 to [6, 7] 2024-04-22T18:41:50.523 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.22', '6', '7'] 2024-04-22T18:41:50.523 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 4.22 6 7 2024-04-22T18:41:50.548 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:50.548 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:50.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:50.548 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:50.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.548 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:289 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:50.549 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:50.549 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.907206887, 2024-04-22T18:41:50.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9176801189999999, 2024-04-22T18:41:50.549 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.549 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.549 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:50.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:41:50.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640723+0000", 2024-04-22T18:41:50.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:50.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640732+0000", 2024-04-22T18:41:50.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5809999999999997e-06 2024-04-22T18:41:50.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640744+0000", 2024-04-22T18:41:50.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.261e-05 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640753+0000", 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7460000000000006e-06 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642189+0000", 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001435551 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642200+0000", 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1389999999999999e-05 2024-04-22T18:41:50.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.847015+0000", 2024-04-22T18:41:50.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048149160000001 2024-04-22T18:41:50.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.848217+0000", 2024-04-22T18:41:50.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012020659999999999 2024-04-22T18:41:50.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351823+0000", 2024-04-22T18:41:50.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50360565099999999 2024-04-22T18:41:50.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351907+0000", 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4141999999999995e-05 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352051+0000", 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014386600000000001 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453617+0000", 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156649500000001 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558297+0000", 2024-04-22T18:41:50.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10468000399999999 2024-04-22T18:41:50.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558331+0000", 2024-04-22T18:41:50.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3518e-05 2024-04-22T18:41:50.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558405+0000", 2024-04-22T18:41:50.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4221000000000006e-05 2024-04-22T18:41:50.554 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.554 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.555 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:1 3.2 3:6d90b903:::64b5942574fb36ca1b9a2be186c6b740edf58e989056a2f88b83cd280451a438:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:50.555 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:50.555 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.37590592700001, 2024-04-22T18:41:50.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.285534422, 2024-04-22T18:41:50.555 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.555 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.555 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:50.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172024+0000", 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173091+0000", 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010677569999999999 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173103+0000", 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1241e-05 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173109+0000", 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4300000000000003e-06 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173168+0000", 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9011000000000001e-05 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:50.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173177+0000", 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948501+0000", 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77532387199999997 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948912+0000", 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041183000000000002 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950438+0000", 2024-04-22T18:41:50.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015253199999999999 2024-04-22T18:41:50.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950488+0000", 2024-04-22T18:41:50.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0285999999999999e-05 2024-04-22T18:41:50.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:50.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950601+0000", 2024-04-22T18:41:50.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011244899999999999 2024-04-22T18:41:50.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954048+0000", 2024-04-22T18:41:50.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003447313 2024-04-22T18:41:50.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457373+0000", 2024-04-22T18:41:50.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50332465299999996 2024-04-22T18:41:50.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457430+0000", 2024-04-22T18:41:50.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7574999999999997e-05 2024-04-22T18:41:50.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457560+0000", 2024-04-22T18:41:50.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013023900000000001 2024-04-22T18:41:50.561 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.561 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.561 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.561 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:8 3.2 3:5f64590d:::7c47933f331d40689081b2bd8f9d7f7bdae1be1fdccb52d696047a3969e902b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:50.561 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:50.561 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.374813558, 2024-04-22T18:41:50.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3854200569999999, 2024-04-22T18:41:50.561 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.561 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.561 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:50.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:41:50.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173117+0000", 2024-04-22T18:41:50.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:50.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173393+0000", 2024-04-22T18:41:50.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000275278 2024-04-22T18:41:50.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173399+0000", 2024-04-22T18:41:50.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4590000000000003e-06 2024-04-22T18:41:50.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173406+0000", 2024-04-22T18:41:50.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2450000000000001e-06 2024-04-22T18:41:50.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173795+0000", 2024-04-22T18:41:50.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038912400000000002 2024-04-22T18:41:50.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:50.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173804+0000", 2024-04-22T18:41:50.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-04-22T18:41:50.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.949506+0000", 2024-04-22T18:41:50.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77570185700000005 2024-04-22T18:41:50.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950166+0000", 2024-04-22T18:41:50.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066022800000000001 2024-04-22T18:41:50.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352843+0000", 2024-04-22T18:41:50.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40267661799999999 2024-04-22T18:41:50.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352911+0000", 2024-04-22T18:41:50.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8126000000000005e-05 2024-04-22T18:41:50.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:50.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.353031+0000", 2024-04-22T18:41:50.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119919 2024-04-22T18:41:50.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453622+0000", 2024-04-22T18:41:50.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100590793 2024-04-22T18:41:50.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558446+0000", 2024-04-22T18:41:50.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104824073 2024-04-22T18:41:50.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558479+0000", 2024-04-22T18:41:50.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3362999999999999e-05 2024-04-22T18:41:50.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558538+0000", 2024-04-22T18:41:50.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9046999999999999e-05 2024-04-22T18:41:50.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.567 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.567 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:10 3.2 3:518c37f9:::851eca84dafbf2f780c4f07291110c453349bdc9c623c17bdcd26e27badb30ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:50.567 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:50.567 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.374519965, 2024-04-22T18:41:50.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3852527729999999, 2024-04-22T18:41:50.568 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.568 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.568 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:50.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:41:50.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173411+0000", 2024-04-22T18:41:50.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:50.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173749+0000", 2024-04-22T18:41:50.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000338029 2024-04-22T18:41:50.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173755+0000", 2024-04-22T18:41:50.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0530000000000003e-06 2024-04-22T18:41:50.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173759+0000", 2024-04-22T18:41:50.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8190000000000002e-06 2024-04-22T18:41:50.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173855+0000", 2024-04-22T18:41:50.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5927999999999994e-05 2024-04-22T18:41:50.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:50.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173862+0000", 2024-04-22T18:41:50.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4449999999999999e-06 2024-04-22T18:41:50.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950835+0000", 2024-04-22T18:41:50.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77697248299999999 2024-04-22T18:41:50.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951632+0000", 2024-04-22T18:41:50.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079752299999999996 2024-04-22T18:41:50.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355407+0000", 2024-04-22T18:41:50.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40377480500000001 2024-04-22T18:41:50.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355462+0000", 2024-04-22T18:41:50.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4577999999999998e-05 2024-04-22T18:41:50.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:50.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355571+0000", 2024-04-22T18:41:50.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108964 2024-04-22T18:41:50.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453625+0000", 2024-04-22T18:41:50.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098054397000000001 2024-04-22T18:41:50.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558574+0000", 2024-04-22T18:41:50.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104948571 2024-04-22T18:41:50.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558602+0000", 2024-04-22T18:41:50.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7966999999999998e-05 2024-04-22T18:41:50.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558665+0000", 2024-04-22T18:41:50.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3032000000000002e-05 2024-04-22T18:41:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:11 3.2 3:718aa8aa:::9603c4f8e269a436b2f0bf376c5374e77bdd7cf58fccae60e4e65ac4696806d4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.37416834699999, 2024-04-22T18:41:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4852604440000001, 2024-04-22T18:41:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:50.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:50.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:41:50.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173813+0000", 2024-04-22T18:41:50.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0655999999999997e-05 2024-04-22T18:41:50.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173819+0000", 2024-04-22T18:41:50.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.468e-06 2024-04-22T18:41:50.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173822+0000", 2024-04-22T18:41:50.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4879999999999999e-06 2024-04-22T18:41:50.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173983+0000", 2024-04-22T18:41:50.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160519 2024-04-22T18:41:50.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:50.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173991+0000", 2024-04-22T18:41:50.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6490000000000008e-06 2024-04-22T18:41:50.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951688+0000", 2024-04-22T18:41:50.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77769743800000002 2024-04-22T18:41:50.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952419+0000", 2024-04-22T18:41:50.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073067499999999997 2024-04-22T18:41:50.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355901+0000", 2024-04-22T18:41:50.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40348237599999998 2024-04-22T18:41:50.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355953+0000", 2024-04-22T18:41:50.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1477000000000002e-05 2024-04-22T18:41:50.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:50.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356060+0000", 2024-04-22T18:41:50.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107114 2024-04-22T18:41:50.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453629+0000", 2024-04-22T18:41:50.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097569127000000005 2024-04-22T18:41:50.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658807+0000", 2024-04-22T18:41:50.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20517829800000001 2024-04-22T18:41:50.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658876+0000", 2024-04-22T18:41:50.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8454000000000006e-05 2024-04-22T18:41:50.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659024+0000", 2024-04-22T18:41:50.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148338 2024-04-22T18:41:50.580 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.580 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.580 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.581 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:12 3.2 3:475e1cd9:::efaab2bd13543cdbb143e869e20eb0ec424f7f5202707859b241ed91d4674936:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:50.581 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:50.581 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.374105403, 2024-04-22T18:41:50.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4853405259999999, 2024-04-22T18:41:50.581 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.581 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.581 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:50.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:50.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:41:50.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173940+0000", 2024-04-22T18:41:50.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011465500000000001 2024-04-22T18:41:50.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173946+0000", 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5289999999999999e-06 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173950+0000", 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8290000000000001e-06 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174277+0000", 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032761299999999998 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:50.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174285+0000", 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.52e-06 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952455+0000", 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77817000700000005 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954035+0000", 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001580078 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356379+0000", 2024-04-22T18:41:50.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40234440399999999 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356428+0000", 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8832999999999998e-05 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356532+0000", 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103235 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453632+0000", 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097100346000000004 2024-04-22T18:41:50.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659069+0000", 2024-04-22T18:41:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20543763800000001 2024-04-22T18:41:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659105+0000", 2024-04-22T18:41:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5240000000000001e-05 2024-04-22T18:41:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659167+0000", 2024-04-22T18:41:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2207999999999998e-05 2024-04-22T18:41:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:16 3.2 3:47c76cc8:::c00d18458eb20c0c0aa9c612cf9793a68445ab0ea7b9e650ecd0072c60a96c0f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.373977444, 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.485347744, 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:50.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:41:50.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:50.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174234+0000", 2024-04-22T18:41:50.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028047599999999999 2024-04-22T18:41:50.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174240+0000", 2024-04-22T18:41:50.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2079999999999997e-06 2024-04-22T18:41:50.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174244+0000", 2024-04-22T18:41:50.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5810000000000001e-06 2024-04-22T18:41:50.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174670+0000", 2024-04-22T18:41:50.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042640000000000001 2024-04-22T18:41:50.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:50.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174675+0000", 2024-04-22T18:41:50.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8709999999999996e-06 2024-04-22T18:41:50.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054168+0000", 2024-04-22T18:41:50.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87949257300000006 2024-04-22T18:41:50.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054877+0000", 2024-04-22T18:41:50.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070884300000000002 2024-04-22T18:41:50.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356836+0000", 2024-04-22T18:41:50.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30195926000000001 2024-04-22T18:41:50.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356887+0000", 2024-04-22T18:41:50.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0767999999999999e-05 2024-04-22T18:41:50.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:50.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356991+0000", 2024-04-22T18:41:50.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104072 2024-04-22T18:41:50.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453636+0000", 2024-04-22T18:41:50.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096644968999999997 2024-04-22T18:41:50.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659207+0000", 2024-04-22T18:41:50.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20557154699999999 2024-04-22T18:41:50.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659240+0000", 2024-04-22T18:41:50.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2665e-05 2024-04-22T18:41:50.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659302+0000", 2024-04-22T18:41:50.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2234000000000001e-05 2024-04-22T18:41:50.593 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.593 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.593 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.593 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:17 3.2 3:6ef31daa:::1cf4ff496c4480e58510f607481d976b47d84868e2c54016273c38baf590f3b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:50.593 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:50.593 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.373683164, 2024-04-22T18:41:50.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4851703729999999, 2024-04-22T18:41:50.594 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.594 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.594 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:50.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:41:50.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174248+0000", 2024-04-22T18:41:50.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174626+0000", 2024-04-22T18:41:50.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037839699999999999 2024-04-22T18:41:50.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174633+0000", 2024-04-22T18:41:50.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0990000000000004e-06 2024-04-22T18:41:50.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174636+0000", 2024-04-22T18:41:50.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5540000000000002e-06 2024-04-22T18:41:50.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174790+0000", 2024-04-22T18:41:50.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153807 2024-04-22T18:41:50.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:50.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174793+0000", 2024-04-22T18:41:50.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4970000000000001e-06 2024-04-22T18:41:50.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054902+0000", 2024-04-22T18:41:50.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88010886300000002 2024-04-22T18:41:50.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055366+0000", 2024-04-22T18:41:50.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046336199999999999 2024-04-22T18:41:50.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.452973+0000", 2024-04-22T18:41:50.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39760783999999999 2024-04-22T18:41:50.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453046+0000", 2024-04-22T18:41:50.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2374999999999997e-05 2024-04-22T18:41:50.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:50.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453193+0000", 2024-04-22T18:41:50.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014722499999999999 2024-04-22T18:41:50.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554698+0000", 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101505096 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659337+0000", 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104639236 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659365+0000", 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7594e-05 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659419+0000", 2024-04-22T18:41:50.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4058999999999999e-05 2024-04-22T18:41:50.600 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.600 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.600 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.600 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:18 3.2 3:792a4ed0:::e1c814961b3d1531fb5ed475b3d496a0405c3a819b68a92ff9b2391fe912370f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:41:50.600 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:50.600 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 107.373286908, 2024-04-22T18:41:50.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.585036954, 2024-04-22T18:41:50.600 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.600 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.600 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:50.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:41:50.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174640+0000", 2024-04-22T18:41:50.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-04-22T18:41:50.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174758+0000", 2024-04-22T18:41:50.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011819099999999999 2024-04-22T18:41:50.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174764+0000", 2024-04-22T18:41:50.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6729999999999997e-06 2024-04-22T18:41:50.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174767+0000", 2024-04-22T18:41:50.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3830000000000001e-06 2024-04-22T18:41:50.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373669+0000", 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198901989 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373686+0000", 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7286e-05 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055408+0000", 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68172133400000001 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055938+0000", 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053016599999999999 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453668+0000", 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39773030799999998 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453717+0000", 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8884000000000002e-05 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453823+0000", 2024-04-22T18:41:50.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105791 2024-04-22T18:41:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554715+0000", 2024-04-22T18:41:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100891628 2024-04-22T18:41:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759439+0000", 2024-04-22T18:41:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20472459400000001 2024-04-22T18:41:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759525+0000", 2024-04-22T18:41:50.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5653000000000001e-05 2024-04-22T18:41:50.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759682+0000", 2024-04-22T18:41:50.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015711800000000001 2024-04-22T18:41:50.606 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.606 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.606 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.606 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:75 3.1 3:ab43a94d:::f38e6e97cb24e2b90da964fee5d2d6d8674386870fc552915e0c3fdbefd4b126:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:50.606 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:50.606 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 75.300410391, 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60149928399999997, 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247521+0000", 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247520+0000", 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247527+0000", 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1670000000000002e-06 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247539+0000", 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503e-05 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247549+0000", 2024-04-22T18:41:50.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0000000000000001e-05 2024-04-22T18:41:50.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247796+0000", 2024-04-22T18:41:50.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024712699999999998 2024-04-22T18:41:50.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:50.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247955+0000", 2024-04-22T18:41:50.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158534 2024-04-22T18:41:50.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.343985+0000", 2024-04-22T18:41:50.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096030344000000004 2024-04-22T18:41:50.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.344035+0000", 2024-04-22T18:41:50.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9969999999999998e-05 2024-04-22T18:41:50.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.745806+0000", 2024-04-22T18:41:50.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.401771029 2024-04-22T18:41:50.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746040+0000", 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023339300000000001 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.746198+0000", 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015793299999999999 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848737+0000", 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10253898 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848751+0000", 2024-04-22T18:41:50.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4694e-05 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.848825+0000", 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3443999999999997e-05 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849021+0000", 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000196047 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.612 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:76 3.1 3:b44829a0:::3788286a255d538503614e5f04049b0bb54cacaa317ab102b5dee8f5b640b296:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 75.300371325, 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60168681599999996, 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:41:50.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:50.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247564+0000", 2024-04-22T18:41:50.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1150000000000004e-06 2024-04-22T18:41:50.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247570+0000", 2024-04-22T18:41:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3940000000000001e-06 2024-04-22T18:41:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247576+0000", 2024-04-22T18:41:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3619999999999997e-06 2024-04-22T18:41:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.340845+0000", 2024-04-22T18:41:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.093269199999999997 2024-04-22T18:41:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.341043+0000", 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019834400000000001 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345106+0000", 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040629780000000001 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345160+0000", 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4246999999999997e-05 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747047+0000", 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40188678300000003 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747162+0000", 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011483400000000001 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747429+0000", 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026700299999999998 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849032+0000", 2024-04-22T18:41:50.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101603203 2024-04-22T18:41:50.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849095+0000", 2024-04-22T18:41:50.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2597999999999999e-05 2024-04-22T18:41:50.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849136+0000", 2024-04-22T18:41:50.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1251000000000002e-05 2024-04-22T18:41:50.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849247+0000", 2024-04-22T18:41:50.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011127 2024-04-22T18:41:50.619 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.619 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.619 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.619 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3526 4.c 4:314c2fec:::smithi08937929-22:head [setxattr _header (56) in=7b,truncate 3092521] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:41:50.619 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:50.619 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.318734544999998, 2024-04-22T18:41:50.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60463626000000004, 2024-04-22T18:41:50.619 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.619 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.619 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:50.619 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:50.619 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:50.620 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3526 2024-04-22T18:41:50.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.620 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:50.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:50.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229200+0000", 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2320000000000001e-06 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229203+0000", 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329292+0000", 2024-04-22T18:41:50.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100088891 2024-04-22T18:41:50.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329357+0000", 2024-04-22T18:41:50.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5115000000000003e-05 2024-04-22T18:41:50.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329453+0000", 2024-04-22T18:41:50.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5802000000000002e-05 2024-04-22T18:41:50.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:41:50.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329594+0000", 2024-04-22T18:41:50.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014157500000000001 2024-04-22T18:41:50.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.330172+0000", 2024-04-22T18:41:50.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057782799999999996 2024-04-22T18:41:50.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833712+0000", 2024-04-22T18:41:50.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50353975299999998 2024-04-22T18:41:50.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833774+0000", 2024-04-22T18:41:50.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1832e-05 2024-04-22T18:41:50.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833834+0000", 2024-04-22T18:41:50.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9809999999999998e-05 2024-04-22T18:41:50.624 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.624 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.624 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.624 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:339 3.2 3:4159b8d1:::bb1f8e997816dd9d2f29d36c4c1859fcaffbf8763b5c8d63c7a818d8d84bac37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:50.624 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.772480786000003, 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0092773479999999, 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775450+0000", 2024-04-22T18:41:50.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:50.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775457+0000", 2024-04-22T18:41:50.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2949999999999996e-06 2024-04-22T18:41:50.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775467+0000", 2024-04-22T18:41:50.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0180000000000001e-05 2024-04-22T18:41:50.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775475+0000", 2024-04-22T18:41:50.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.678e-06 2024-04-22T18:41:50.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.877787+0000", 2024-04-22T18:41:50.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102312559 2024-04-22T18:41:50.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.880686+0000", 2024-04-22T18:41:50.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002898748 2024-04-22T18:41:50.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.282867+0000", 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.402180961 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.282945+0000", 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7775999999999995e-05 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283065+0000", 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0001199 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.285514+0000", 2024-04-22T18:41:50.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0024491729999999998 2024-04-22T18:41:50.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784375+0000", 2024-04-22T18:41:50.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49886154100000002 2024-04-22T18:41:50.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784475+0000", 2024-04-22T18:41:50.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9074999999999996e-05 2024-04-22T18:41:50.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784728+0000", 2024-04-22T18:41:50.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025386799999999998 2024-04-22T18:41:50.630 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.630 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.630 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.630 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:340 3.2 3:4bbc2d06:::e1c179ab87fb8e08c4a1dfc0c7cc725138181c5734d6df18f3f2545b93731e97:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:50.630 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:50.630 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.772448367000003, 2024-04-22T18:41:50.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009550757, 2024-04-22T18:41:50.630 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.630 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.630 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:50.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:50.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:41:50.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775487+0000", 2024-04-22T18:41:50.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6839999999999998e-06 2024-04-22T18:41:50.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775492+0000", 2024-04-22T18:41:50.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6899999999999997e-06 2024-04-22T18:41:50.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775497+0000", 2024-04-22T18:41:50.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8590000000000004e-06 2024-04-22T18:41:50.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.880737+0000", 2024-04-22T18:41:50.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105240267 2024-04-22T18:41:50.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881308+0000", 2024-04-22T18:41:50.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057098399999999997 2024-04-22T18:41:50.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283810+0000", 2024-04-22T18:41:50.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40250177500000001 2024-04-22T18:41:50.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283883+0000", 2024-04-22T18:41:50.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2430999999999998e-05 2024-04-22T18:41:50.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:50.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284019+0000", 2024-04-22T18:41:50.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013652299999999999 2024-04-22T18:41:50.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.287390+0000", 2024-04-22T18:41:50.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033710089999999999 2024-04-22T18:41:50.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784829+0000", 2024-04-22T18:41:50.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49743879400000002 2024-04-22T18:41:50.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784893+0000", 2024-04-22T18:41:50.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4184999999999996e-05 2024-04-22T18:41:50.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785034+0000", 2024-04-22T18:41:50.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014114600000000001 2024-04-22T18:41:50.636 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.636 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.636 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.636 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:341 3.2 3:5b8bea68:::ab73be59042e3e18c4fe1fdbc2bbcc205779424ddf3bb423a595330be9026399:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:50.636 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:50.636 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.772429015, 2024-04-22T18:41:50.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009670804, 2024-04-22T18:41:50.636 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.636 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.636 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:50.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:41:50.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775502+0000", 2024-04-22T18:41:50.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775505+0000", 2024-04-22T18:41:50.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0809999999999998e-06 2024-04-22T18:41:50.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775511+0000", 2024-04-22T18:41:50.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2039999999999996e-06 2024-04-22T18:41:50.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775515+0000", 2024-04-22T18:41:50.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8790000000000002e-06 2024-04-22T18:41:50.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881376+0000", 2024-04-22T18:41:50.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105861 2024-04-22T18:41:50.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881839+0000", 2024-04-22T18:41:50.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046217200000000001 2024-04-22T18:41:50.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284569+0000", 2024-04-22T18:41:50.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40273058900000003 2024-04-22T18:41:50.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284730+0000", 2024-04-22T18:41:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077800000000001 2024-04-22T18:41:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284923+0000", 2024-04-22T18:41:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000193253 2024-04-22T18:41:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.287593+0000", 2024-04-22T18:41:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026699359999999999 2024-04-22T18:41:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785074+0000", 2024-04-22T18:41:50.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49748046000000001 2024-04-22T18:41:50.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785103+0000", 2024-04-22T18:41:50.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8935e-05 2024-04-22T18:41:50.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785174+0000", 2024-04-22T18:41:50.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1001999999999995e-05 2024-04-22T18:41:50.641 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.642 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.642 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.642 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:342 3.2 3:634dda9f:::10622dc7de8047e7908c0b5cca7a43d84cdf1b7629e6f7e159b8207feba82b01:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:50.642 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:50.642 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.772410803, 2024-04-22T18:41:50.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.022499579, 2024-04-22T18:41:50.642 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.642 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.642 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:50.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:50.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:41:50.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775524+0000", 2024-04-22T18:41:50.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9629999999999998e-06 2024-04-22T18:41:50.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775529+0000", 2024-04-22T18:41:50.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0320000000000003e-06 2024-04-22T18:41:50.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775533+0000", 2024-04-22T18:41:50.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7380000000000002e-06 2024-04-22T18:41:50.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881884+0000", 2024-04-22T18:41:50.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106350294 2024-04-22T18:41:50.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.882492+0000", 2024-04-22T18:41:50.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060872699999999999 2024-04-22T18:41:50.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381559+0000", 2024-04-22T18:41:50.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49906679199999998 2024-04-22T18:41:50.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381677+0000", 2024-04-22T18:41:50.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011743699999999999 2024-04-22T18:41:50.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:50.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381899+0000", 2024-04-22T18:41:50.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022272599999999999 2024-04-22T18:41:50.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.383440+0000", 2024-04-22T18:41:50.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015407719999999999 2024-04-22T18:41:50.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.797827+0000", 2024-04-22T18:41:50.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.414386741 2024-04-22T18:41:50.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.797887+0000", 2024-04-22T18:41:50.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9889000000000002e-05 2024-04-22T18:41:50.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.798021+0000", 2024-04-22T18:41:50.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000133948 2024-04-22T18:41:50.647 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.647 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.647 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:343 3.2 3:7db7298e:::00597d8527769aa920807495342cf3ba06efb8f507feae865dd89cd8ad10bda0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.772391847999998, 2024-04-22T18:41:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.110995373, 2024-04-22T18:41:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:50.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:41:50.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775542+0000", 2024-04-22T18:41:50.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9079999999999999e-06 2024-04-22T18:41:50.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775547+0000", 2024-04-22T18:41:50.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.994e-06 2024-04-22T18:41:50.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775553+0000", 2024-04-22T18:41:50.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2299999999999999e-06 2024-04-22T18:41:50.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.882519+0000", 2024-04-22T18:41:50.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106966453 2024-04-22T18:41:50.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883103+0000", 2024-04-22T18:41:50.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058419400000000001 2024-04-22T18:41:50.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.581957+0000", 2024-04-22T18:41:50.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69885328599999996 2024-04-22T18:41:50.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582023+0000", 2024-04-22T18:41:50.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6210999999999993e-05 2024-04-22T18:41:50.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:50.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582146+0000", 2024-04-22T18:41:50.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123669 2024-04-22T18:41:50.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.682293+0000", 2024-04-22T18:41:50.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10014662100000001 2024-04-22T18:41:50.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886410+0000", 2024-04-22T18:41:50.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20411678899999999 2024-04-22T18:41:50.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886445+0000", 2024-04-22T18:41:50.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5462e-05 2024-04-22T18:41:50.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886535+0000", 2024-04-22T18:41:50.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9995999999999997e-05 2024-04-22T18:41:50.653 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.653 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.653 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.653 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:344 3.1 3:9da67ae4:::04c6b48a110b3f74550f78f0899df697160a1eeab7f41463f6ac18091ca67078:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:50.653 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:50.654 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.772356541000001, 2024-04-22T18:41:50.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60604869800000005, 2024-04-22T18:41:50.654 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.654 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.654 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:50.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:50.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:41:50.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775578+0000", 2024-04-22T18:41:50.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3440000000000001e-06 2024-04-22T18:41:50.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775584+0000", 2024-04-22T18:41:50.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6969999999999998e-06 2024-04-22T18:41:50.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775589+0000", 2024-04-22T18:41:50.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0540000000000002e-06 2024-04-22T18:41:50.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775729+0000", 2024-04-22T18:41:50.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014061299999999999 2024-04-22T18:41:50.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776045+0000", 2024-04-22T18:41:50.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000315168 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080584+0000", 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30453977700000001 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080635+0000", 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1007999999999997e-05 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080741+0000", 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105646 2024-04-22T18:41:50.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.180821+0000", 2024-04-22T18:41:50.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10007980900000001 2024-04-22T18:41:50.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381437+0000", 2024-04-22T18:41:50.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20061589399999999 2024-04-22T18:41:50.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381491+0000", 2024-04-22T18:41:50.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4531999999999999e-05 2024-04-22T18:41:50.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381624+0000", 2024-04-22T18:41:50.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000132745 2024-04-22T18:41:50.659 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.659 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.659 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.659 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:345 3.1 3:bd847077:::0e274616605e19dd653fcf05d310933ab02835574414ca18340b3fc325d2da4f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:50.659 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:50.659 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.772332181000003, 2024-04-22T18:41:50.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.90876140699999997, 2024-04-22T18:41:50.659 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.660 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.660 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:50.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:41:50.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775599+0000", 2024-04-22T18:41:50.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:50.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775602+0000", 2024-04-22T18:41:50.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.976e-06 2024-04-22T18:41:50.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775607+0000", 2024-04-22T18:41:50.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2229999999999998e-06 2024-04-22T18:41:50.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775612+0000", 2024-04-22T18:41:50.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6539999999999998e-06 2024-04-22T18:41:50.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776072+0000", 2024-04-22T18:41:50.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046006299999999998 2024-04-22T18:41:50.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:50.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776256+0000", 2024-04-22T18:41:50.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000183735 2024-04-22T18:41:50.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081157+0000", 2024-04-22T18:41:50.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30490169099999997 2024-04-22T18:41:50.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081203+0000", 2024-04-22T18:41:50.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5677999999999999e-05 2024-04-22T18:41:50.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:50.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081301+0000", 2024-04-22T18:41:50.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7893999999999995e-05 2024-04-22T18:41:50.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.180827+0000", 2024-04-22T18:41:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099525725999999995 2024-04-22T18:41:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684197+0000", 2024-04-22T18:41:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50336990699999995 2024-04-22T18:41:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684247+0000", 2024-04-22T18:41:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0195000000000003e-05 2024-04-22T18:41:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684361+0000", 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114189 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4267 3.2 3:584fd9a2:::cda848272f4ef42ebafc2089aaddeacaf12eabd82aef5e42e32ae93a8a5da0b6:head [write 0~2635776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.691982324999998, 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.72658671900000005, 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:50.665 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:50.666 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:50.666 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4267 2024-04-22T18:41:50.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.666 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:50.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:50.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:50.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:50.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:41:50.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:50.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855946+0000", 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.856984+0000", 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001037764 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.857003+0000", 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8876e-05 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:50.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.857016+0000", 2024-04-22T18:41:50.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3337e-05 2024-04-22T18:41:50.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:50.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883170+0000", 2024-04-22T18:41:50.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026154047 2024-04-22T18:41:50.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:50.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883262+0000", 2024-04-22T18:41:50.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1996999999999993e-05 2024-04-22T18:41:50.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:50.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883360+0000", 2024-04-22T18:41:50.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7429000000000005e-05 2024-04-22T18:41:50.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:50.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.887277+0000", 2024-04-22T18:41:50.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039169449999999998 2024-04-22T18:41:50.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:50.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582423+0000", 2024-04-22T18:41:50.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69514625200000002 2024-04-22T18:41:50.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:50.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582458+0000", 2024-04-22T18:41:50.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5293999999999998e-05 2024-04-22T18:41:50.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:50.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:50.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:50.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582536+0000", 2024-04-22T18:41:50.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7855000000000006e-05 2024-04-22T18:41:50.670 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.670 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.670 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.670 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:50.670 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:50.670 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:50.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:50.673 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:50.672+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:50.772 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:50.773+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:50.786 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:50.786 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:50.787 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:50.787 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:50.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:50.873 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:50.875+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:50.898 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:50.898 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:50.898 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:50.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:50.974 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:50.974+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:51.010 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:51.010 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 176.34830861, 2024-04-22T18:41:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:41:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:41:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:41:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:41:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:41:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:41:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:41:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:41:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:41:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:41:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:41:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:41:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:41:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:41:51.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:41:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:41:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:41:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:41:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:41:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:41:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:41:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:41:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:41:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 176.24730043299999, 2024-04-22T18:41:51.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:41:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:41:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:41:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:41:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:41:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:41:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:41:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:41:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:41:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:41:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:41:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:41:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:41:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:41:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:41:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:41:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:41:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:41:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:41:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 176.247290048, 2024-04-22T18:41:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:41:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.021 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:41:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:41:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:41:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:41:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:41:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:41:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:41:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:41:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:41:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:41:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:41:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:41:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:41:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:41:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:41:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:41:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:41:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:41:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:41:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:41:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:41:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:41:51.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:41:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:41:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 171.90953962099999, 2024-04-22T18:41:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:41:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.027 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:41:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:41:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:41:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:41:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:41:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:41:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:41:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:41:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:41:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:41:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:41:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:41:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:41:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:51.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:41:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:41:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:41:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:41:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:41:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:41:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:41:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:41:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:41:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:41:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:41:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:51.032 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.874892106, 2024-04-22T18:41:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:41:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:41:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:41:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:41:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:41:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:41:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:41:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:41:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:41:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:41:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:41:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:41:51.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:41:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:41:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:41:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:41:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4177 4.19 4.f75b2379 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.845838809, 2024-04-22T18:41:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76236081700000002, 2024-04-22T18:41:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4177 2024-04-22T18:41:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:41:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164847+0000", 2024-04-22T18:41:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164854+0000", 2024-04-22T18:41:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2620000000000001e-06 2024-04-22T18:41:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164864+0000", 2024-04-22T18:41:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0400999999999999e-05 2024-04-22T18:41:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164876+0000", 2024-04-22T18:41:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1204e-05 2024-04-22T18:41:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164975+0000", 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9562999999999995e-05 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164980+0000", 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7310000000000001e-06 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927200+0000", 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.762219973 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927209+0000", 2024-04-22T18:41:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9050000000000003e-06 2024-04-22T18:41:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4182 4.c 4.5345b7ac (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.845807489999999, 2024-04-22T18:41:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76185930999999996, 2024-04-22T18:41:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.041 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4182 2024-04-22T18:41:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:41:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164881+0000", 2024-04-22T18:41:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.305e-06 2024-04-22T18:41:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164885+0000", 2024-04-22T18:41:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.873e-06 2024-04-22T18:41:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164890+0000", 2024-04-22T18:41:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6299999999999997e-06 2024-04-22T18:41:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225475+0000", 2024-04-22T18:41:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060585181000000002 2024-04-22T18:41:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225483+0000", 2024-04-22T18:41:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6790000000000005e-06 2024-04-22T18:41:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926728+0000", 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124542099999998 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926739+0000", 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0669000000000001e-05 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4183 4.c 4.5345b7ac (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.845794234000003, 2024-04-22T18:41:51.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76187489500000005, 2024-04-22T18:41:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4183 2024-04-22T18:41:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:41:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164892+0000", 2024-04-22T18:41:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164894+0000", 2024-04-22T18:41:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.705e-06 2024-04-22T18:41:51.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164897+0000", 2024-04-22T18:41:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2739999999999999e-06 2024-04-22T18:41:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164901+0000", 2024-04-22T18:41:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.636e-06 2024-04-22T18:41:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225508+0000", 2024-04-22T18:41:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060607564000000003 2024-04-22T18:41:51.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225515+0000", 2024-04-22T18:41:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1829999999999999e-06 2024-04-22T18:41:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926759+0000", 2024-04-22T18:41:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.701244859 2024-04-22T18:41:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926767+0000", 2024-04-22T18:41:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0180000000000001e-06 2024-04-22T18:41:51.049 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4190 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.845777312000003, 2024-04-22T18:41:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.761884546, 2024-04-22T18:41:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4190 2024-04-22T18:41:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:41:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164912+0000", 2024-04-22T18:41:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6249999999999999e-06 2024-04-22T18:41:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164918+0000", 2024-04-22T18:41:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6039999999999998e-06 2024-04-22T18:41:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164922+0000", 2024-04-22T18:41:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9319999999999998e-06 2024-04-22T18:41:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225536+0000", 2024-04-22T18:41:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060613725 2024-04-22T18:41:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225540+0000", 2024-04-22T18:41:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5030000000000002e-06 2024-04-22T18:41:51.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926786+0000", 2024-04-22T18:41:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124673999999998 2024-04-22T18:41:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926794+0000", 2024-04-22T18:41:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7689999999999996e-06 2024-04-22T18:41:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4191 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.845761891000002, 2024-04-22T18:41:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76190695500000005, 2024-04-22T18:41:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4191 2024-04-22T18:41:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:41:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164927+0000", 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0959999999999999e-06 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164937+0000", 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0730999999999999e-05 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164944+0000", 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5130000000000002e-06 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225565+0000", 2024-04-22T18:41:51.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060621421000000002 2024-04-22T18:41:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225569+0000", 2024-04-22T18:41:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.264e-06 2024-04-22T18:41:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926813+0000", 2024-04-22T18:41:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124469 2024-04-22T18:41:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926832+0000", 2024-04-22T18:41:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8580000000000002e-05 2024-04-22T18:41:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4192 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.845737282999998, 2024-04-22T18:41:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76191075399999997, 2024-04-22T18:41:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.059 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4192 2024-04-22T18:41:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:41:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164953+0000", 2024-04-22T18:41:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1099999999999996e-06 2024-04-22T18:41:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164965+0000", 2024-04-22T18:41:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2298e-05 2024-04-22T18:41:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164972+0000", 2024-04-22T18:41:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5230000000000001e-06 2024-04-22T18:41:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225581+0000", 2024-04-22T18:41:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060609356000000003 2024-04-22T18:41:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225584+0000", 2024-04-22T18:41:51.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.106e-06 2024-04-22T18:41:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926852+0000", 2024-04-22T18:41:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70126794999999997 2024-04-22T18:41:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926860+0000", 2024-04-22T18:41:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0919999999999998e-06 2024-04-22T18:41:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4197 4.13 4:ca46fde9:::smithi08937929-14:head [sparse-read 0~1375265,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.845710672999999, 2024-04-22T18:41:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.28429536, 2024-04-22T18:41:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4197 2024-04-22T18:41:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:41:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164975+0000", 2024-04-22T18:41:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164979+0000", 2024-04-22T18:41:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8959999999999996e-06 2024-04-22T18:41:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164983+0000", 2024-04-22T18:41:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.053e-06 2024-04-22T18:41:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164987+0000", 2024-04-22T18:41:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:41:51.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.264783+0000", 2024-04-22T18:41:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099796088000000005 2024-04-22T18:41:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.264789+0000", 2024-04-22T18:41:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4989999999999999e-06 2024-04-22T18:41:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139375+0000", 2024-04-22T18:41:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.87458606000000005 2024-04-22T18:41:51.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139478+0000", 2024-04-22T18:41:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010219 2024-04-22T18:41:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.446961+0000", 2024-04-22T18:41:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30748385900000003 2024-04-22T18:41:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.447101+0000", 2024-04-22T18:41:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001393 2024-04-22T18:41:51.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.449271+0000", 2024-04-22T18:41:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021707950000000001 2024-04-22T18:41:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4198 4.13 4:ca46fde9:::smithi08937929-14:head [sparse-read 0~1375265] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.845697305000002, 2024-04-22T18:41:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2868951370000001, 2024-04-22T18:41:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.069 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4198 2024-04-22T18:41:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:41:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164991+0000", 2024-04-22T18:41:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:41:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164994+0000", 2024-04-22T18:41:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2459999999999999e-06 2024-04-22T18:41:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164997+0000", 2024-04-22T18:41:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1200000000000002e-06 2024-04-22T18:41:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.364948+0000", 2024-04-22T18:41:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19995059200000001 2024-04-22T18:41:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.364969+0000", 2024-04-22T18:41:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1495999999999999e-05 2024-04-22T18:41:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139617+0000", 2024-04-22T18:41:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77464723300000005 2024-04-22T18:41:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139643+0000", 2024-04-22T18:41:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6228e-05 2024-04-22T18:41:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451393+0000", 2024-04-22T18:41:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31175035200000001 2024-04-22T18:41:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451515+0000", 2024-04-22T18:41:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001214 2024-04-22T18:41:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451885+0000", 2024-04-22T18:41:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036992999999999998 2024-04-22T18:41:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.074 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4199 4.13 4:ca46fde9:::smithi08937929-14:head [read 0~2739541,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.845686282000003, 2024-04-22T18:41:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.291247504, 2024-04-22T18:41:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4199 2024-04-22T18:41:51.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:41:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165002+0000", 2024-04-22T18:41:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:41:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165006+0000", 2024-04-22T18:41:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:41:51.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165009+0000", 2024-04-22T18:41:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4599999999999999e-06 2024-04-22T18:41:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365066+0000", 2024-04-22T18:41:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20005682999999999 2024-04-22T18:41:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365078+0000", 2024-04-22T18:41:51.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2235000000000001e-05 2024-04-22T18:41:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139671+0000", 2024-04-22T18:41:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77459301700000005 2024-04-22T18:41:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139702+0000", 2024-04-22T18:41:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0673999999999998e-05 2024-04-22T18:41:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451930+0000", 2024-04-22T18:41:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31222817600000002 2024-04-22T18:41:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.452092+0000", 2024-04-22T18:41:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001616 2024-04-22T18:41:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.456248+0000", 2024-04-22T18:41:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041563700000000004 2024-04-22T18:41:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4200 4.13 4:c93c842e:::smithi08937929-25:head [tier-promote] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.845675401999998, 2024-04-22T18:41:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.177728814, 2024-04-22T18:41:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4200 2024-04-22T18:41:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:41:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165013+0000", 2024-04-22T18:41:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.635e-06 2024-04-22T18:41:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165018+0000", 2024-04-22T18:41:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2000000000000002e-06 2024-04-22T18:41:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165024+0000", 2024-04-22T18:41:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5910000000000003e-06 2024-04-22T18:41:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365110+0000", 2024-04-22T18:41:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20008601400000001 2024-04-22T18:41:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365114+0000", 2024-04-22T18:41:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.651e-06 2024-04-22T18:41:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139723+0000", 2024-04-22T18:41:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77460836899999996 2024-04-22T18:41:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139787+0000", 2024-04-22T18:41:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4707000000000002e-05 2024-04-22T18:41:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.459491+0000", 2024-04-22T18:41:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31970372000000002 2024-04-22T18:41:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.459567+0000", 2024-04-22T18:41:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6199999999999995e-05 2024-04-22T18:41:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.562522+0000", 2024-04-22T18:41:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102954925 2024-04-22T18:41:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.562585+0000", 2024-04-22T18:41:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3287000000000006e-05 2024-04-22T18:41:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:05.342740+0000", 2024-04-22T18:41:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.78015481499999995 2024-04-22T18:41:51.086 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4205 4.1 4.fdaa93c1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:41:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.845659251999997, 2024-04-22T18:41:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76220996100000005, 2024-04-22T18:41:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4205 2024-04-22T18:41:51.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:41:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165027+0000", 2024-04-22T18:41:51.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165029+0000", 2024-04-22T18:41:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3549999999999999e-06 2024-04-22T18:41:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165035+0000", 2024-04-22T18:41:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8549999999999998e-06 2024-04-22T18:41:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165040+0000", 2024-04-22T18:41:51.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9200000000000003e-06 2024-04-22T18:41:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165116+0000", 2024-04-22T18:41:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5834999999999995e-05 2024-04-22T18:41:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165123+0000", 2024-04-22T18:41:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2139999999999999e-06 2024-04-22T18:41:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927230+0000", 2024-04-22T18:41:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76210651100000004 2024-04-22T18:41:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927237+0000", 2024-04-22T18:41:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7409999999999992e-06 2024-04-22T18:41:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4268 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e133)", 2024-04-22T18:41:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:41:51.091 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 42.427951839999999, 2024-04-22T18:41:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70931800300000003, 2024-04-22T18:41:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4268 2024-04-22T18:41:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:41:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:41:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582733+0000", 2024-04-22T18:41:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582741+0000", 2024-04-22T18:41:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3329999999999999e-06 2024-04-22T18:41:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582751+0000", 2024-04-22T18:41:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0628999999999999e-05 2024-04-22T18:41:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582759+0000", 2024-04-22T18:41:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4120000000000004e-06 2024-04-22T18:41:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582795+0000", 2024-04-22T18:41:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6157e-05 2024-04-22T18:41:51.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582862+0000", 2024-04-22T18:41:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7261999999999994e-05 2024-04-22T18:41:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.987268+0000", 2024-04-22T18:41:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40440629700000003 2024-04-22T18:41:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.987771+0000", 2024-04-22T18:41:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050267400000000005 2024-04-22T18:41:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.291974+0000", 2024-04-22T18:41:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30420264499999999 2024-04-22T18:41:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.292008+0000", 2024-04-22T18:41:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4011999999999999e-05 2024-04-22T18:41:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.292053+0000", 2024-04-22T18:41:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5305000000000001e-05 2024-04-22T18:41:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4272 4.5 4:a6d237d2:::smithi08937929-43:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e133)", 2024-04-22T18:41:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 42.224623639999997, 2024-04-22T18:41:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60731304699999999, 2024-04-22T18:41:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:51.097 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4272 2024-04-22T18:41:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:41:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786068+0000", 2024-04-22T18:41:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9910000000000002e-06 2024-04-22T18:41:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786079+0000", 2024-04-22T18:41:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1287000000000001e-05 2024-04-22T18:41:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786083+0000", 2024-04-22T18:41:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0949999999999998e-06 2024-04-22T18:41:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786121+0000", 2024-04-22T18:41:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8096999999999998e-05 2024-04-22T18:41:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786219+0000", 2024-04-22T18:41:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7429000000000005e-05 2024-04-22T18:41:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.393376+0000", 2024-04-22T18:41:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60715760600000002 2024-04-22T18:41:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.101 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4437 4.19 4:9ea6f51f:::smithi08937929-34:head [read 304128~651776] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:41:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 33.095016866000002, 2024-04-22T18:41:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.042388732, 2024-04-22T18:41:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4437 2024-04-22T18:41:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:41:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:41:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915669+0000", 2024-04-22T18:41:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915673+0000", 2024-04-22T18:41:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6770000000000001e-06 2024-04-22T18:41:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915679+0000", 2024-04-22T18:41:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.976e-06 2024-04-22T18:41:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915684+0000", 2024-04-22T18:41:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3430000000000003e-06 2024-04-22T18:41:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915725+0000", 2024-04-22T18:41:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0751999999999999e-05 2024-04-22T18:41:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915732+0000", 2024-04-22T18:41:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6279999999999999e-06 2024-04-22T18:41:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.957228+0000", 2024-04-22T18:41:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0414961220000001 2024-04-22T18:41:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.957325+0000", 2024-04-22T18:41:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7422000000000003e-05 2024-04-22T18:41:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.958059+0000", 2024-04-22T18:41:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073350499999999996 2024-04-22T18:41:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4690 3.3 3:ee62d872:::cb076f2b64cfcdfafc1ec2cf19f2eaa46006a5a6952a3aeb9c7211f97fca91ae:head [write 0~441856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-04-22T18:41:51.106 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:26.592341+0000", 2024-04-22T18:41:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 24.418345603999999, 2024-04-22T18:41:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.56667778400000002, 2024-04-22T18:41:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4690 2024-04-22T18:41:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592341+0000", 2024-04-22T18:41:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592341+0000", 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592340+0000", 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592697+0000", 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000357334 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592709+0000", 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2211999999999999e-05 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592718+0000", 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0319999999999992e-06 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.692859+0000", 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014077 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.692870+0000", 2024-04-22T18:41:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1225e-05 2024-04-22T18:41:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.995290+0000", 2024-04-22T18:41:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242006199999999 2024-04-22T18:41:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.995398+0000", 2024-04-22T18:41:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108136 2024-04-22T18:41:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.995513+0000", 2024-04-22T18:41:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115047 2024-04-22T18:41:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.996655+0000", 2024-04-22T18:41:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001141162 2024-04-22T18:41:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:27.158858+0000", 2024-04-22T18:41:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16220310299999999 2024-04-22T18:41:51.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:27.158901+0000", 2024-04-22T18:41:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3667000000000001e-05 2024-04-22T18:41:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:27.159019+0000", 2024-04-22T18:41:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117619 2024-04-22T18:41:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.112 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.112 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:51.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:51.114 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:51.114+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:51.214 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:51.215+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:51.256 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:51.256 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:51.256 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:51.256 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:51.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:51.315 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:51.316+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:51.366 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:51.366 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:51.366 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:51.366 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:51.366 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:51.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:51.416 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:51.417+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:51.518 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:51.519+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:51.519 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:51.520 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:51.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:51.520 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:51.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.520 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:41:51.520 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:51.520 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 177.42562531799999, 2024-04-22T18:41:51.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:41:51.520 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:41:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:41:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:41:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:41:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:41:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:41:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:41:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:41:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:41:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:41:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:41:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:41:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:41:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:41:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:41:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:41:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:41:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:41:51.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:41:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:41:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:41:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:41:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:41:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 173.52612744300001, 2024-04-22T18:41:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:41:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:41:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:41:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:41:51.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:41:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:41:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:41:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:41:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:41:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:41:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:41:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:41:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:41:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:41:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:41:51.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:41:51.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:51.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:41:51.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:41:51.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:41:51.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:41:51.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:41:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:41:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:41:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:41:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:41:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:41:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.535 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:41:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 172.417827652, 2024-04-22T18:41:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:41:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:41:51.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:41:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:41:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:41:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:41:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:41:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:41:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:41:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:41:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:41:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:41:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:41:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:41:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:41:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:41:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:41:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:41:51.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:41:51.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:41:51.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:41:51.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:41:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:41:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 159.08628402400001, 2024-04-22T18:41:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:41:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:41:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:41:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:41:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:41:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:41:51.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:41:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:41:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:41:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:41:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:41:51.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:41:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:41:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:41:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:41:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:41:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.546 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:41:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:41:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:41:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:41:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:41:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:41:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.547 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.99519821600001, 2024-04-22T18:41:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:41:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:41:51.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:41:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:41:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:41:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:41:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:41:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:41:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:41:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:41:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:41:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:41:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:41:51.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:41:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:41:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:41:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:41:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:41:51.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:41:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:41:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:41:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:41:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:41:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.989397956, 2024-04-22T18:41:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:41:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:41:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.554 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:41:51.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:41:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:41:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:41:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:41:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.557 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:41:51.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:41:51.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:41:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:41:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:41:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:41:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:41:51.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:41:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:41:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:41:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:41:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:41:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:41:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:41:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:41:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:41:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:51.560 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.984657613, 2024-04-22T18:41:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:41:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:41:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:51.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:41:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.562 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:41:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:41:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:41:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:41:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:41:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:41:51.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:41:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:41:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:41:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:41:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:51.564 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:41:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:41:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:41:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:41:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:41:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:41:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:41:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:41:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:41:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:41:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.984184715, 2024-04-22T18:41:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:41:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:41:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:41:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:41:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:41:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:41:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:41:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:41:51.569 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:41:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:41:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:41:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:41:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:41:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.570 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:41:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:41:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:41:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:41:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:41:51.571 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:41:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:41:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:41:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:41:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:41:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.98416479400001, 2024-04-22T18:41:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:41:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:41:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:41:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:41:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:41:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:41:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.575 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:41:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:41:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:41:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:41:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:41:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:41:51.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:41:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:41:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:41:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:41:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:41:51.577 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:41:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 153.67484102899999, 2024-04-22T18:41:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:41:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:41:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:51.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:41:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:41:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:41:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:41:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.580 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:41:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:41:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:41:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:41:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:41:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:41:51.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:41:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:41:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:41:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:41:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.582 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:41:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:41:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:41:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:41:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:41:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:41:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.583 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.584 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:51.584 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:51.585 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 153.67463349299999, 2024-04-22T18:41:51.585 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:41:51.585 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.585 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.585 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.585 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:41:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:51.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:41:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:41:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:41:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:41:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:41:51.587 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:41:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:41:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:41:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:41:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:41:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:51.588 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:41:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:41:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:41:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:41:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:41:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:41:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.589 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:41:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:41:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:41:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:41:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.590 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.80898375999999, 2024-04-22T18:41:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:41:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:51.591 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:41:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:41:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:41:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.592 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:41:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:41:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:41:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:41:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:41:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:41:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.593 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:41:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:41:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:41:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:41:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:41:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:41:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.594 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:41:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:41:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:41:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:41:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:41:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:41:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.595 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.59514537499999, 2024-04-22T18:41:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:41:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:41:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:41:51.597 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:41:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:41:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:41:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:41:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:41:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.598 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:41:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:41:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:41:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:41:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:41:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:41:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.599 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:41:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:41:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:41:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:41:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:41:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:41:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:41:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:41:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.595111965, 2024-04-22T18:41:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:41:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.601 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.602 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:41:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:41:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:41:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:41:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:41:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:41:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:41:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:41:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:41:51.604 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:41:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:41:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:41:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:41:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:41:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.605 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:41:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:41:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:41:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:41:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:41:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:41:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.606 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.536229311, 2024-04-22T18:41:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:41:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:41:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.608 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:41:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:41:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:41:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:41:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:41:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:41:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.609 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:41:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:41:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:41:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:41:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:41:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:41:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.610 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:41:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:41:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:41:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:41:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:41:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:41:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:41:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:41:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.53609069500001, 2024-04-22T18:41:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:41:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.612 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.613 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:41:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:41:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:41:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:41:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.614 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:41:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:41:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:41:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:41:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:41:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:41:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.615 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:41:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:41:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:41:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:41:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:41:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:41:51.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:41:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:41:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:41:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:41:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.53607169200001, 2024-04-22T18:41:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:41:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.618 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:41:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:41:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.619 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:41:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:41:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:41:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:41:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:41:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:41:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:41:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:41:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:41:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:41:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:41:51.621 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:41:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:41:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:41:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:41:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:41:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.622 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:41:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:41:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.91505915900001, 2024-04-22T18:41:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:41:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.623 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:41:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:41:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:41:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:41:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:41:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:41:51.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:41:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:41:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:41:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:41:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:41:51.626 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:41:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:41:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:41:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:41:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:41:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:41:51.627 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:41:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:41:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:41:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:41:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:41:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.628 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.91453956800001, 2024-04-22T18:41:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:41:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:41:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.629 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:41:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:41:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:41:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:41:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:41:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:41:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:41:51.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.632 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:41:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:41:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:41:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:41:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:41:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:41:51.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.634 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.634 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:41:51.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:41:51.634 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.634 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.634 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.634 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.634 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:51.634 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:51.634 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.81438668499999, 2024-04-22T18:41:51.634 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:41:51.634 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:41:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:51.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:51.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:51.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:41:51.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:51.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:51.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:41:51.636 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:41:51.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.636 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.636 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:51.636 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:41:51.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:41:51.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:51.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:41:51.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:41:51.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:41:51.637 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:41:51.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.637 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.637 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:51.637 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:41:51.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:41:51.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:51.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:41:51.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:41:51.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:51.638 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:41:51.638 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:41:51.638 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:51.638 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:51.638 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:51.639 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:41:51.639 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:41:51.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.639 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:51.639 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:51.639 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:51.641 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:51.642+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:51.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:51.742 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:51.743+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:51.743 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:51.743 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:51.743 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:51.744 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:51.745+0000 7fec57bee640 -1 osd.7 174 handle_osd_map injecting map crc failure 2024-04-22T18:41:51.744 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:51.746+0000 7f3e7be88640 -1 osd.0 174 handle_osd_map injecting map crc failure 2024-04-22T18:41:51.747 INFO:teuthology.orchestra.run.smithi089.stderr:set 4.22 pg_upmap mapping to [6,7] 2024-04-22T18:41:51.749 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:41:51.750 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:51.751+0000 7f8605f32640 -1 osd.3 174 handle_osd_map injecting map crc failure 2024-04-22T18:41:51.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:51.842 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:51.844+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:51.896 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:51.896 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:51.896 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:51.896 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:51.896 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:51.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:51.943 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:51.944+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:52.009 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:52.009 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:52.009 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:52.009 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:52.009 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.009 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:52.010 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:52.010 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 177.74972970799999, 2024-04-22T18:41:52.010 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:41:52.010 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.010 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:52.010 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.010 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.010 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.010 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:41:52.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.010 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.010 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.010 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:52.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:52.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.011 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.011 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:41:52.011 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:52.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.011 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:41:52.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:41:52.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:41:52.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:41:52.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.012 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.012 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.012 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:41:52.012 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:41:52.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:41:52.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:41:52.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:41:52.013 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:41:52.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.013 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.013 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:52.013 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:41:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:41:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:41:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:41:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:41:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:41:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.014 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:52.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:41:52.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:41:52.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.015 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.015 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.015 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:41:52.015 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:41:52.015 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.015 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.015 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.016 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.016 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:52.016 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:52.016 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 177.74494727800001, 2024-04-22T18:41:52.016 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:41:52.016 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.017 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:52.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.017 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.017 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:41:52.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.017 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.017 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:52.017 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.017 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.017 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:52.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:41:52.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:52.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.018 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.018 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:41:52.018 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:41:52.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.018 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:41:52.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:41:52.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:41:52.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:41:52.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.019 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.019 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.019 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:41:52.019 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:41:52.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:41:52.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:41:52.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:52.020 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:41:52.020 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:41:52.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.020 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.020 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:52.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:41:52.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:41:52.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:52.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:41:52.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:41:52.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.021 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:52.021 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:41:52.021 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:41:52.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.021 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.022 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.022 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:41:52.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:41:52.022 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.022 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.022 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.022 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.022 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:52.022 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:52.022 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 177.74485307500001, 2024-04-22T18:41:52.022 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:41:52.022 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.022 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:52.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.023 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.023 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:41:52.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.023 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:52.023 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.023 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.023 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:52.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:41:52.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:52.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.024 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:41:52.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:41:52.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.024 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.024 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:41:52.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:41:52.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:41:52.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:41:52.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.025 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.025 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:41:52.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:41:52.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:41:52.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:41:52.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:52.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:41:52.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:41:52.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:52.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:41:52.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:41:52.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:52.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:41:52.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:41:52.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:52.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:41:52.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:41:52.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:41:52.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:41:52.028 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.028 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.028 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.028 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:52.028 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:52.028 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 177.74483218699999, 2024-04-22T18:41:52.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:41:52.028 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.028 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:52.028 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.028 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.029 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.029 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:41:52.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.029 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:52.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:52.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:41:52.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:52.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:41:52.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:41:52.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:41:52.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:41:52.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:41:52.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:41:52.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:41:52.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:41:52.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:52.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:41:52.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:41:52.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:41:52.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:41:52.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:41:52.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:41:52.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:41:52.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:41:52.034 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.034 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.034 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.034 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:52.034 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:52.034 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 176.96182452400001, 2024-04-22T18:41:52.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:41:52.035 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.035 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:52.035 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.035 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.035 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.035 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:41:52.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.035 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:52.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:52.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.036 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:41:52.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:52.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:41:52.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:41:52.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:41:52.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:41:52.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.037 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.037 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:41:52.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:41:52.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:41:52.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:41:52.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:41:52.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:41:52.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:52.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:41:52.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:41:52.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:52.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:41:52.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:41:52.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:52.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:41:52.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:41:52.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:52.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:41:52.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:41:52.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:41:52.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:41:52.040 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.041 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.041 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.041 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:52.041 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:52.041 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 176.953993118, 2024-04-22T18:41:52.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:41:52.041 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.041 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:52.041 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.041 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.041 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.041 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:41:52.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.042 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:52.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:52.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:41:52.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:52.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.043 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.043 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:41:52.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:41:52.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:41:52.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:41:52.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:41:52.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:41:52.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:41:52.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:41:52.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:41:52.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:41:52.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:52.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:41:52.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:41:52.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:52.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:41:52.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:41:52.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:52.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:41:52.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:41:52.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:52.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:41:52.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:41:52.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:41:52.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:41:52.047 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.047 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.047 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.047 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:52.048 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:52.048 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 116.802120488, 2024-04-22T18:41:52.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:41:52.048 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.048 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:52.048 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.048 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.048 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.048 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:41:52.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.048 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:52.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:52.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:41:52.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:52.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:41:52.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:41:52.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:41:52.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:41:52.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:41:52.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:41:52.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:41:52.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:41:52.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:41:52.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:41:52.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:52.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:41:52.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:41:52.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:52.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:41:52.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:41:52.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:52.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:41:52.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:41:52.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:52.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:41:52.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:41:52.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:41:52.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:41:52.054 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.054 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.054 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.054 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:52.054 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:52.054 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 116.80171558000001, 2024-04-22T18:41:52.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:41:52.054 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.054 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:41:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:41:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:41:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:41:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:41:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:41:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:41:52.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:41:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:41:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:41:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:41:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:41:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:41:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:41:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:41:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:41:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:41:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3213 4.a 4:5d2aee44:::smithi08937929-13:head [assert-version v66,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 89.440815692000001, 2024-04-22T18:41:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69928701400000004, 2024-04-22T18:41:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.060 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3213 2024-04-22T18:41:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568559+0000", 2024-04-22T18:41:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568567+0000", 2024-04-22T18:41:52.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-04-22T18:41:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568579+0000", 2024-04-22T18:41:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-04-22T18:41:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.668711+0000", 2024-04-22T18:41:52.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100131866 2024-04-22T18:41:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.768945+0000", 2024-04-22T18:41:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100234252 2024-04-22T18:41:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769002+0000", 2024-04-22T18:41:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7092999999999997e-05 2024-04-22T18:41:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.166404+0000", 2024-04-22T18:41:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39740241999999998 2024-04-22T18:41:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.191686+0000", 2024-04-22T18:41:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025281739000000001 2024-04-22T18:41:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:52.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267758+0000", 2024-04-22T18:41:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076071606999999999 2024-04-22T18:41:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267812+0000", 2024-04-22T18:41:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4478999999999998e-05 2024-04-22T18:41:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267848+0000", 2024-04-22T18:41:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6062999999999997e-05 2024-04-22T18:41:52.066 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3319 4.a 4:5d2aee44:::smithi08937929-13:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:41:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 85.549067941000004, 2024-04-22T18:41:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476623401, 2024-04-22T18:41:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.067 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3319 2024-04-22T18:41:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460306+0000", 2024-04-22T18:41:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460318+0000", 2024-04-22T18:41:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2896e-05 2024-04-22T18:41:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460345+0000", 2024-04-22T18:41:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6650000000000001e-05 2024-04-22T18:41:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460360+0000", 2024-04-22T18:41:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4729000000000001e-05 2024-04-22T18:41:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460502+0000", 2024-04-22T18:41:52.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141729 2024-04-22T18:41:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460644+0000", 2024-04-22T18:41:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142755 2024-04-22T18:41:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.936932+0000", 2024-04-22T18:41:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476288078 2024-04-22T18:41:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.071 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3412 4.a 4:519c08f7:::smithi08937929-15:head [write 2529878~394573] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 80.867233380000002, 2024-04-22T18:41:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48918094899999998, 2024-04-22T18:41:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.072 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3412 2024-04-22T18:41:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142141+0000", 2024-04-22T18:41:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142496+0000", 2024-04-22T18:41:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035444699999999998 2024-04-22T18:41:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142510+0000", 2024-04-22T18:41:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3841e-05 2024-04-22T18:41:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142518+0000", 2024-04-22T18:41:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8089999999999993e-06 2024-04-22T18:41:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142560+0000", 2024-04-22T18:41:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2097e-05 2024-04-22T18:41:52.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142637+0000", 2024-04-22T18:41:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6965000000000006e-05 2024-04-22T18:41:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142715+0000", 2024-04-22T18:41:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8694000000000002e-05 2024-04-22T18:41:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144179+0000", 2024-04-22T18:41:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146342 2024-04-22T18:41:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631229+0000", 2024-04-22T18:41:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48705018500000002 2024-04-22T18:41:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631281+0000", 2024-04-22T18:41:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2216999999999998e-05 2024-04-22T18:41:52.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631324+0000", 2024-04-22T18:41:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3344999999999999e-05 2024-04-22T18:41:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3413 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2924451] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:52.078 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 80.86685224, 2024-04-22T18:41:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58904086899999997, 2024-04-22T18:41:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3413 2024-04-22T18:41:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:52.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142523+0000", 2024-04-22T18:41:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142529+0000", 2024-04-22T18:41:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6049999999999997e-06 2024-04-22T18:41:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142535+0000", 2024-04-22T18:41:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4740000000000002e-06 2024-04-22T18:41:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142541+0000", 2024-04-22T18:41:52.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1839999999999998e-06 2024-04-22T18:41:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143553+0000", 2024-04-22T18:41:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010126689999999999 2024-04-22T18:41:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143659+0000", 2024-04-22T18:41:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105393 2024-04-22T18:41:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143825+0000", 2024-04-22T18:41:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165809 2024-04-22T18:41:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144329+0000", 2024-04-22T18:41:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050483799999999997 2024-04-22T18:41:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731480+0000", 2024-04-22T18:41:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58715100600000003 2024-04-22T18:41:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731522+0000", 2024-04-22T18:41:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2098000000000002e-05 2024-04-22T18:41:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731566+0000", 2024-04-22T18:41:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3066999999999999e-05 2024-04-22T18:41:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3414 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 80.866823636000007, 2024-04-22T18:41:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58926020199999996, 2024-04-22T18:41:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.085 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3414 2024-04-22T18:41:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142552+0000", 2024-04-22T18:41:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142556+0000", 2024-04-22T18:41:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0289999999999999e-06 2024-04-22T18:41:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142574+0000", 2024-04-22T18:41:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7266999999999999e-05 2024-04-22T18:41:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142581+0000", 2024-04-22T18:41:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-04-22T18:41:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144121+0000", 2024-04-22T18:41:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015397480000000001 2024-04-22T18:41:52.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144170+0000", 2024-04-22T18:41:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8803e-05 2024-04-22T18:41:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731624+0000", 2024-04-22T18:41:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58745365800000005 2024-04-22T18:41:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731712+0000", 2024-04-22T18:41:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8016999999999994e-05 2024-04-22T18:41:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731813+0000", 2024-04-22T18:41:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101817 2024-04-22T18:41:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.090 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:77 3.3 3:dcc8aeec:::3590d5741efdda41a26b62a0982727439225b7b4b51097b313f30d8dc10bf6f6:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.761805314, 2024-04-22T18:41:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011812878, 2024-04-22T18:41:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247571+0000", 2024-04-22T18:41:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247574+0000", 2024-04-22T18:41:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:41:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247577+0000", 2024-04-22T18:41:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3069999999999998e-06 2024-04-22T18:41:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247580+0000", 2024-04-22T18:41:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8870000000000001e-06 2024-04-22T18:41:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.340928+0000", 2024-04-22T18:41:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093348678000000004 2024-04-22T18:41:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341122+0000", 2024-04-22T18:41:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000193227 2024-04-22T18:41:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618335+0000", 2024-04-22T18:41:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.277213775 2024-04-22T18:41:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618422+0000", 2024-04-22T18:41:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6537000000000002e-05 2024-04-22T18:41:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954387+0000", 2024-04-22T18:41:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33596509699999999 2024-04-22T18:41:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954548+0000", 2024-04-22T18:41:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016042199999999999 2024-04-22T18:41:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954858+0000", 2024-04-22T18:41:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000310396 2024-04-22T18:41:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:52.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157019+0000", 2024-04-22T18:41:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20216071299999999 2024-04-22T18:41:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259213+0000", 2024-04-22T18:41:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102194734 2024-04-22T18:41:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259262+0000", 2024-04-22T18:41:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9052999999999999e-05 2024-04-22T18:41:52.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259384+0000", 2024-04-22T18:41:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122052 2024-04-22T18:41:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:78 3.3 3:df7e90d2:::ab0657e24bd96704775de2a378746bc22309c5a7366a5eaf9b4e1f1f2bef0a4c:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:52.098 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.761790117000004, 2024-04-22T18:41:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011974739, 2024-04-22T18:41:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247586+0000", 2024-04-22T18:41:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247589+0000", 2024-04-22T18:41:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.181e-06 2024-04-22T18:41:52.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247592+0000", 2024-04-22T18:41:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0230000000000001e-06 2024-04-22T18:41:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247595+0000", 2024-04-22T18:41:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.428e-06 2024-04-22T18:41:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341160+0000", 2024-04-22T18:41:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093564571999999999 2024-04-22T18:41:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341264+0000", 2024-04-22T18:41:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010391999999999999 2024-04-22T18:41:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619269+0000", 2024-04-22T18:41:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27800575100000002 2024-04-22T18:41:52.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619346+0000", 2024-04-22T18:41:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6459999999999996e-05 2024-04-22T18:41:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.155957+0000", 2024-04-22T18:41:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53661109299999998 2024-04-22T18:41:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156049+0000", 2024-04-22T18:41:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2272e-05 2024-04-22T18:41:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156236+0000", 2024-04-22T18:41:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000186808 2024-04-22T18:41:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.158088+0000", 2024-04-22T18:41:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018522079999999999 2024-04-22T18:41:52.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259421+0000", 2024-04-22T18:41:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101332718 2024-04-22T18:41:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259460+0000", 2024-04-22T18:41:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8912000000000003e-05 2024-04-22T18:41:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259562+0000", 2024-04-22T18:41:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010175600000000001 2024-04-22T18:41:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:80 3.3 3:f90f47f2:::e5603f596d699ee3e27e255225eeb90a9f0c1b4a4d099d1c586b19019220aa3f:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.761751040000007, 2024-04-22T18:41:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0121096759999999, 2024-04-22T18:41:52.106 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247625+0000", 2024-04-22T18:41:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247628+0000", 2024-04-22T18:41:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6129999999999998e-06 2024-04-22T18:41:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247631+0000", 2024-04-22T18:41:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3869999999999999e-06 2024-04-22T18:41:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247634+0000", 2024-04-22T18:41:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9919999999999999e-06 2024-04-22T18:41:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341550+0000", 2024-04-22T18:41:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093915584999999996 2024-04-22T18:41:52.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341700+0000", 2024-04-22T18:41:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015000300000000001 2024-04-22T18:41:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620744+0000", 2024-04-22T18:41:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27904399200000002 2024-04-22T18:41:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620794+0000", 2024-04-22T18:41:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9579000000000001e-05 2024-04-22T18:41:52.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157597+0000", 2024-04-22T18:41:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53680343799999997 2024-04-22T18:41:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157678+0000", 2024-04-22T18:41:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0829999999999997e-05 2024-04-22T18:41:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157832+0000", 2024-04-22T18:41:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154184 2024-04-22T18:41:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259572+0000", 2024-04-22T18:41:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10174003600000001 2024-04-22T18:41:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259601+0000", 2024-04-22T18:41:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8841999999999999e-05 2024-04-22T18:41:52.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259638+0000", 2024-04-22T18:41:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7308999999999998e-05 2024-04-22T18:41:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259736+0000", 2024-04-22T18:41:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7283999999999999e-05 2024-04-22T18:41:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:135 3.3 3:f89134e5:::3e4e73f5d4a0122957865f0faeac2483204ee98d7d497ef3b54afe2a02b39a3d:head [read 0~802304 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.393935960999997, 2024-04-22T18:41:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.64370653499999997, 2024-04-22T18:41:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615438+0000", 2024-04-22T18:41:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:52.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615449+0000", 2024-04-22T18:41:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1219999999999999e-05 2024-04-22T18:41:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615460+0000", 2024-04-22T18:41:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0403e-05 2024-04-22T18:41:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.715588+0000", 2024-04-22T18:41:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100128881 2024-04-22T18:41:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722533+0000", 2024-04-22T18:41:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069444010000000002 2024-04-22T18:41:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722658+0000", 2024-04-22T18:41:52.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012560699999999999 2024-04-22T18:41:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826437+0000", 2024-04-22T18:41:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103778194 2024-04-22T18:41:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826532+0000", 2024-04-22T18:41:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5111000000000005e-05 2024-04-22T18:41:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258787+0000", 2024-04-22T18:41:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43225529099999999 2024-04-22T18:41:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258905+0000", 2024-04-22T18:41:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118348 2024-04-22T18:41:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259148+0000", 2024-04-22T18:41:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000242093 2024-04-22T18:41:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3890 4.1 4:83c955bf:::smithi08937929-28:head [write 2699879~469889] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 62.395261116999997, 2024-04-22T18:41:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49042297000000001, 2024-04-22T18:41:52.123 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3890 2024-04-22T18:41:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614115+0000", 2024-04-22T18:41:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614496+0000", 2024-04-22T18:41:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038181700000000001 2024-04-22T18:41:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614509+0000", 2024-04-22T18:41:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2381000000000001e-05 2024-04-22T18:41:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614517+0000", 2024-04-22T18:41:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0879999999999995e-06 2024-04-22T18:41:52.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615610+0000", 2024-04-22T18:41:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010933519999999999 2024-04-22T18:41:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615655+0000", 2024-04-22T18:41:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4966000000000003e-05 2024-04-22T18:41:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615769+0000", 2024-04-22T18:41:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113986 2024-04-22T18:41:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704882+0000", 2024-04-22T18:41:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089112825000000007 2024-04-22T18:41:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.128 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104488+0000", 2024-04-22T18:41:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39960572999999999 2024-04-22T18:41:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104517+0000", 2024-04-22T18:41:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9006e-05 2024-04-22T18:41:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104539+0000", 2024-04-22T18:41:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2073e-05 2024-04-22T18:41:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3891 4.1 4:83c955bf:::smithi08937929-28:head [setxattr _header (56) in=7b,truncate 3630345] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 62.394852886000002, 2024-04-22T18:41:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59018588699999996, 2024-04-22T18:41:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:52.130 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3891 2024-04-22T18:41:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.131 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614523+0000", 2024-04-22T18:41:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614528+0000", 2024-04-22T18:41:52.132 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0379999999999999e-06 2024-04-22T18:41:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614540+0000", 2024-04-22T18:41:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018000000000001e-05 2024-04-22T18:41:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614546+0000", 2024-04-22T18:41:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4990000000000002e-06 2024-04-22T18:41:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.133 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704438+0000", 2024-04-22T18:41:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089892044000000004 2024-04-22T18:41:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704504+0000", 2024-04-22T18:41:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6365000000000006e-05 2024-04-22T18:41:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704602+0000", 2024-04-22T18:41:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8048999999999996e-05 2024-04-22T18:41:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.806370+0000", 2024-04-22T18:41:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10176768 2024-04-22T18:41:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204629+0000", 2024-04-22T18:41:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39825917999999999 2024-04-22T18:41:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.135 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204673+0000", 2024-04-22T18:41:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3516999999999997e-05 2024-04-22T18:41:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204710+0000", 2024-04-22T18:41:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7391000000000002e-05 2024-04-22T18:41:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3892 4.1 4:83c955bf:::smithi08937929-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 62.394826305000002, 2024-04-22T18:41:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59031281800000002, 2024-04-22T18:41:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3892 2024-04-22T18:41:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.137 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.138 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614550+0000", 2024-04-22T18:41:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614553+0000", 2024-04-22T18:41:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0879999999999999e-06 2024-04-22T18:41:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:52.139 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614559+0000", 2024-04-22T18:41:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9259999999999997e-06 2024-04-22T18:41:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614565+0000", 2024-04-22T18:41:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7950000000000004e-06 2024-04-22T18:41:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704818+0000", 2024-04-22T18:41:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090252624000000004 2024-04-22T18:41:52.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704866+0000", 2024-04-22T18:41:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8667e-05 2024-04-22T18:41:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204742+0000", 2024-04-22T18:41:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987551899999999 2024-04-22T18:41:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:52.141 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204799+0000", 2024-04-22T18:41:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7299000000000001e-05 2024-04-22T18:41:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204863+0000", 2024-04-22T18:41:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4455000000000005e-05 2024-04-22T18:41:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:52.142 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:52.142 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:52.143 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:41:52.144 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:52.145+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:52.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:52.244 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:52.245+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:52.248 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:52.249 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:52.249 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:52.249 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:52.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:52.260 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:52.261 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:52.277 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:52.277 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:52.292 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:52.292 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:52.308 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:52.324 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:52.324 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:52.340 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:52.341 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:52.345 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:52.346+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:52.357 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:52.357 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:52.373 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:52.374 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:52.400 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:52.400 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:52.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:52.445 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:52.447+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:52.510 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:52.510 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:52.521 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:52.546 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:52.547+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:52.624 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:52.624 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:52.633 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:52.647 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:52.647+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:52.734 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:52.734 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:52.743 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:52.747 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:41:52.748 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:52.749+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:52.750 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:52.750+0000 7f7f44bb3640 -1 osd.4 175 handle_osd_map injecting map crc failure 2024-04-22T18:41:52.845 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:52.845 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:52.848 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:52.849+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:52.855 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:52.949 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:52.950+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:52.956 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:52.956 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:52.965 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:53.050 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:53.051+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:53.067 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:53.067 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:53.077 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:53.151 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:53.152+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:53.178 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:53.178 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:53.187 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:53.252 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:53.253+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:53.288 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:53.288 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:53.297 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:53.353 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:53.354+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:53.438 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:53.439 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:53.448 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:53.491 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:53.493+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:53.550 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:53.550 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:53.555 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:53.556+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:53.559 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:53.656 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:53.658+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:53.701 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:53.701 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:53.710 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:53.795 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:53.796+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:53.856 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:53.856 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:53.857 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:53.859+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:53.866 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:53.958 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:53.960+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:54.009 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:54.009 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:54.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:54.063 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:54.064+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:54.125 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:54.125 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:54.133 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:54.174 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:54.175+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:54.262 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:54.263+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:54.275 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:54.275 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:54.284 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:54.362 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:54.364+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:54.385 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:41:54.385 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:54.385 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:54.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:54.463 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:54.464+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:54.497 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:54.497 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:54.502 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:54.503 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:54.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:54.519 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:54.519 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:54.535 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:54.535 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:54.550 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:54.550 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:54.563 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:54.565+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:54.567 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:54.567 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:54.583 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:54.583 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:54.599 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:54.600 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:54.615 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:54.616 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:54.648 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:54.649 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:54.649 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:54.649 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:54.649 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:54.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:54.668 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:54.669+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:54.759 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:54.760+0000 7f7f44bb3640 -1 osd.4 177 handle_osd_map injecting map crc failure 2024-04-22T18:41:54.763 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:41:54.764 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:54.764 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:54.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:54.765 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:54.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.765 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:41:54.765 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:54.765 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 180.66988858299999, 2024-04-22T18:41:54.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:41:54.765 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.765 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.765 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:54.765 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:54.765 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:54.765 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:41:54.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.766 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:54.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:41:54.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.766 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.766 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:41:54.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:54.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:41:54.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:41:54.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:41:54.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:41:54.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:41:54.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:41:54.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:41:54.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:41:54.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:41:54.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:41:54.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.770 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:54.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:41:54.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:41:54.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:41:54.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:41:54.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:41:54.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:41:54.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:41:54.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:41:54.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:41:54.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:41:54.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.772 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.772 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:41:54.772 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:54.772 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 176.77039070800001, 2024-04-22T18:41:54.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:41:54.773 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.773 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.773 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:54.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:41:54.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:41:54.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:54.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:41:54.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:41:54.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:41:54.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:41:54.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:41:54.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:41:54.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:41:54.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:41:54.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:54.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:41:54.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:41:54.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:41:54.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:41:54.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:41:54.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:41:54.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:41:54.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:41:54.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:41:54.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:41:54.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.778 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.778 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.778 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:41:54.778 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:54.778 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 175.662090917, 2024-04-22T18:41:54.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:41:54.778 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:41:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:41:54.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:41:54.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:54.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:41:54.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:41:54.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:41:54.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:41:54.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:41:54.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:41:54.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:41:54.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:41:54.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:41:54.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:41:54.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:54.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:41:54.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:41:54.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:41:54.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:41:54.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:41:54.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:41:54.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:41:54.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:41:54.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:41:54.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:41:54.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.783 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.784 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.784 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:54.784 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:54.784 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 162.33054728900001, 2024-04-22T18:41:54.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:41:54.784 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.784 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.784 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:54.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:41:54.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:41:54.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:54.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:41:54.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:41:54.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:41:54.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:41:54.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:41:54.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:41:54.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:41:54.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:41:54.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:41:54.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:41:54.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:54.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:41:54.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:41:54.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:41:54.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:41:54.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:41:54.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:41:54.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:41:54.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:41:54.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.788 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.788 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:41:54.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:41:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 162.23946148100001, 2024-04-22T18:41:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:41:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:54.789 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:41:54.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.790 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:54.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:41:54.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:41:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:41:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:41:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:41:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:41:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.791 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:41:54.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:41:54.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:41:54.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:41:54.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.792 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.792 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:41:54.792 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:41:54.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.792 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:54.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:41:54.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:41:54.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:41:54.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:41:54.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:41:54.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:41:54.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:41:54.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:41:54.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:41:54.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:41:54.794 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.794 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.794 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.794 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 162.23366122100001, 2024-04-22T18:41:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:41:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:41:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:54.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:41:54.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:41:54.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:54.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:41:54.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:41:54.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:41:54.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:41:54.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:41:54.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:41:54.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:41:54.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:41:54.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:41:54.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:41:54.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:54.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:41:54.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:41:54.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:41:54.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:41:54.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:41:54.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:41:54.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:41:54.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:41:54.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:41:54.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:41:54.800 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.800 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.800 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.800 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:54.800 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:54.800 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 162.228920878, 2024-04-22T18:41:54.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:41:54.800 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.800 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.800 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:54.800 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:54.800 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:54.800 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:41:54.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.801 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:54.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:41:54.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:41:54.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:54.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:41:54.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:41:54.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:41:54.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:41:54.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:54.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:41:54.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:41:54.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:41:54.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:41:54.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:41:54.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:41:54.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:41:54.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:41:54.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:41:54.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:41:54.805 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.805 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.805 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.805 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:54.805 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:54.805 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 162.22844798, 2024-04-22T18:41:54.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:41:54.806 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.806 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.806 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:54.806 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:54.806 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:54.806 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:41:54.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.806 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:54.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:41:54.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:41:54.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:54.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:41:54.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:41:54.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:41:54.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:41:54.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:41:54.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:41:54.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:41:54.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:41:54.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:41:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:41:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:41:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:41:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:41:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:41:54.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:41:54.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:41:54.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:41:54.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:41:54.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:41:54.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:41:54.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.811 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.811 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:41:54.811 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:54.811 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 162.22842805900001, 2024-04-22T18:41:54.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:41:54.811 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.811 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:54.811 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:54.811 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:54.811 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:54.811 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:41:54.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.812 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:54.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:54.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:41:54.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:54.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:41:54.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:41:54.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:41:54.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:41:54.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:41:54.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:41:54.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:41:54.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:41:54.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:54.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:41:54.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:41:54.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.814 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.814 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:41:54.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:41:54.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:41:54.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:41:54.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:41:54.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:41:54.815 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.815 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.815 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.816 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:54.816 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:54.816 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 156.91910429399999, 2024-04-22T18:41:54.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:41:54.816 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.816 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.816 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:54.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:41:54.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:41:54.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:54.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:41:54.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:41:54.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:41:54.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:41:54.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:41:54.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:41:54.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:41:54.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:41:54.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:41:54.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:41:54.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:54.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:41:54.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:41:54.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.819 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.819 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.819 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:41:54.819 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:41:54.819 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:41:54.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:41:54.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:41:54.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:41:54.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.820 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.820 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.820 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:41:54.820 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:41:54.820 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.821 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.821 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.821 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:41:54.821 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:54.821 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 156.91889675799999, 2024-04-22T18:41:54.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:41:54.821 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.821 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.821 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.821 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.821 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.821 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:54.821 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:41:54.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:41:54.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:54.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.822 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.822 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.822 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:41:54.822 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:41:54.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:41:54.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:41:54.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:41:54.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:41:54.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.823 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.823 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.823 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:41:54.823 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:41:54.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:41:54.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:41:54.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:54.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:41:54.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:41:54.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.824 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.824 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.824 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:41:54.824 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:41:54.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:41:54.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:41:54.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:41:54.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:41:54.825 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.825 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.825 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.825 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:41:54.825 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:41:54.826 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.826 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.826 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.826 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:54.826 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:54.826 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 138.05324702499999, 2024-04-22T18:41:54.826 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:41:54.826 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.826 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.826 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.826 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.826 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.827 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:41:54.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:41:54.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:41:54.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:41:54.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:41:54.828 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:41:54.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.828 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.828 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.828 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:41:54.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:41:54.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:41:54.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:41:54.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:54.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:41:54.829 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:41:54.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.829 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.829 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.829 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:41:54.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:41:54.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:41:54.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:41:54.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.830 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:41:54.830 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:41:54.830 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.830 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.830 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.831 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:41:54.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:41:54.831 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.831 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.831 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.831 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.831 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:54.831 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:54.831 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 137.83940863999999, 2024-04-22T18:41:54.831 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:41:54.831 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.831 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.831 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:54.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:41:54.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.832 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.832 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.832 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:41:54.832 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:54.832 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:41:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:41:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:41:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:41:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:41:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:41:54.833 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:41:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:41:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:41:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:41:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:41:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:41:54.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:41:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:41:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:41:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:41:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:41:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:41:54.835 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:41:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:41:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 137.83937523, 2024-04-22T18:41:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:41:54.836 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.837 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.837 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:54.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:54.837 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.837 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.837 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.837 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:41:54.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:54.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:41:54.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:41:54.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:41:54.838 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:41:54.838 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.838 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.838 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.838 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:41:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:41:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:41:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:41:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:41:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:41:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:54.839 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:41:54.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:41:54.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:41:54.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:41:54.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.840 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:41:54.840 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:41:54.840 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.840 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.840 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:41:54.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:41:54.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.841 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.841 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:41:54.841 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:41:54.841 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.841 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.841 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.841 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.841 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:54.842 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:54.842 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 137.780492576, 2024-04-22T18:41:54.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:41:54.842 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.842 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.842 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:54.842 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.842 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.842 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.842 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:41:54.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:41:54.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:54.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:41:54.843 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:41:54.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.843 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.843 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.843 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:41:54.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:41:54.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:41:54.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:41:54.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.844 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:41:54.844 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:41:54.844 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.844 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.844 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:41:54.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:41:54.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:54.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:41:54.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:41:54.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.845 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:41:54.845 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:41:54.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.845 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.845 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:41:54.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:41:54.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:41:54.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:41:54.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.846 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.846 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:41:54.846 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:41:54.846 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.846 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.846 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.847 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:54.847 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:54.847 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 137.78035396000001, 2024-04-22T18:41:54.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:41:54.847 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.847 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.847 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.847 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.847 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:54.847 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:54.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:41:54.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:54.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.848 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.848 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:41:54.848 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:41:54.848 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:41:54.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:41:54.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:41:54.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:41:54.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.849 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.849 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.849 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:41:54.849 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:41:54.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:41:54.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:41:54.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:54.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:41:54.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:41:54.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.850 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.850 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.850 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:41:54.850 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:41:54.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:41:54.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:41:54.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:41:54.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:41:54.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.851 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.851 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:41:54.851 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:41:54.852 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.852 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.852 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.852 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:41:54.852 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:54.852 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 137.78033495700001, 2024-04-22T18:41:54.852 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:41:54.852 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.852 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.852 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.852 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.852 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.852 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:54.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:54.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:41:54.853 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:54.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.853 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.853 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.853 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:41:54.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:41:54.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:41:54.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:41:54.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:41:54.854 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:41:54.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.854 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.854 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.854 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:41:54.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:41:54.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:41:54.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:41:54.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:54.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:41:54.855 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:41:54.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.855 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.855 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.855 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:41:54.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:41:54.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:41:54.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:41:54.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.856 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:41:54.856 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:41:54.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.856 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.856 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:41:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:41:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.15932242400001, 2024-04-22T18:41:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:41:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:54.857 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:54.858 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:54.858 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:41:54.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.858 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:54.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.858 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.858 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.858 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:41:54.858 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:41:54.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:54.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:41:54.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:41:54.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.859 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.859 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.859 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:41:54.859 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:41:54.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:41:54.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:41:54.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:41:54.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:41:54.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.860 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.860 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.860 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:41:54.860 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:41:54.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:54.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:41:54.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:41:54.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:41:54.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:41:54.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.861 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.861 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.861 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:41:54.861 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:41:54.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:41:54.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:41:54.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.862 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.862 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.862 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:41:54.862 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:41:54.862 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.862 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.862 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.863 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:54.863 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:54.863 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.15880283300001, 2024-04-22T18:41:54.863 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:41:54.863 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.863 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:54.863 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:54.863 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:54.863 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:54.863 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:41:54.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.863 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.864 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:41:54.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:41:54.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:41:54.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:41:54.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.865 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.865 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.865 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:41:54.865 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:41:54.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:41:54.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:41:54.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:41:54.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:41:54.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.866 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.866 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:54.866 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:41:54.866 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:41:54.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:54.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:41:54.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:41:54.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:54.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:41:54.867 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:41:54.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.867 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.867 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:54.867 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 134.05864994999999, 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:41:54.868 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:54.869 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:54.869 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:54.869 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:54.869 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:54.869 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:41:54.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.869 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:54.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:54.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:54.869 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.869 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.869 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:54.869 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:41:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:41:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:41:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:41:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:54.870 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:41:54.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:41:54.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:54.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:41:54.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:41:54.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.871 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:41:54.871 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:41:54.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.871 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.871 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:54.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:41:54.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:41:54.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:54.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:41:54.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:41:54.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:54.872 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:41:54.872 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:41:54.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:54.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:54.872 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:54.873 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:41:54.873 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:41:54.873 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.873 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.873 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.873 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:54.873 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:54.873 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:54.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:54.875 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:54.875+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:54.977 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:54.976+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:55.018 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:55.019 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:55.019 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:55.019 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:55.027 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:55.076 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:55.077+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:55.128 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:55.128 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:55.129 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:55.129 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:55.129 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:55.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:55.181 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:55.182+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:55.239 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:55.239 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 180.979963258, 2024-04-22T18:41:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:41:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:41:55.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:41:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.241 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:41:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:41:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:41:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:41:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:41:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.242 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:41:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:41:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:41:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:41:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:41:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:41:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.243 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:41:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:41:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:41:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:41:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:41:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:41:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.244 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:41:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:41:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:41:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:41:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:55.245 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 180.97518082799999, 2024-04-22T18:41:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:41:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:41:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:55.246 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:41:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:41:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.247 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:41:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:41:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:41:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:41:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:41:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:41:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.248 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:41:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:41:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:41:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:41:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:41:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:41:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.249 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:41:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:41:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:41:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:41:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:41:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:41:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:41:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:41:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 180.97508662499999, 2024-04-22T18:41:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:41:55.251 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:41:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.252 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:41:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:41:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:41:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:41:55.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:41:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:41:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:41:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:41:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:41:55.254 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:41:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:41:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:41:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:41:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:41:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.255 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:41:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:41:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:41:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:41:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:41:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:41:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.256 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:41:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:41:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 180.97506573699999, 2024-04-22T18:41:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:41:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.257 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:41:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:41:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:41:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:41:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:41:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.259 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:41:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:41:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:41:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:41:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:41:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:41:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:55.260 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:41:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:41:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:41:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:41:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:41:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:41:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.261 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:41:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:41:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 180.19205807399999, 2024-04-22T18:41:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:41:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.262 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:41:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:41:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.263 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:41:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:41:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:41:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:41:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:41:55.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:41:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:41:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:41:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:41:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:41:55.265 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:41:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:41:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:41:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:41:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:41:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.266 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:41:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:41:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:41:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:41:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:41:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:41:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.267 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 180.18422666800001, 2024-04-22T18:41:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:41:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:41:55.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:41:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.269 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:41:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:41:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:41:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:41:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:41:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.270 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:41:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:41:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:41:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:41:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:41:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:41:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.271 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:41:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:41:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:41:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:41:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:41:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:41:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.272 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:41:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:41:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:41:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:41:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:55.273 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 120.03235403799999, 2024-04-22T18:41:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:41:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:41:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:55.274 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:41:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:41:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.275 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:41:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:41:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:41:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:41:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:41:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:41:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.276 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:41:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:41:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:41:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:41:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:41:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:41:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.277 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:41:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:41:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:41:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:41:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:41:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:41:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:41:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:41:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:41:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 120.03194913, 2024-04-22T18:41:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:41:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.279 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:41:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:55.280 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:41:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:41:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:41:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:41:55.281 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:41:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:41:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:41:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:41:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:41:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:55.282 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:41:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:41:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:41:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:41:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:41:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:41:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.283 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:41:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:41:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3213 4.a 4:5d2aee44:::smithi08937929-13:head [assert-version v66,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 92.671049241999995, 2024-04-22T18:41:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69928701400000004, 2024-04-22T18:41:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3213 2024-04-22T18:41:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:41:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568559+0000", 2024-04-22T18:41:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568567+0000", 2024-04-22T18:41:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-04-22T18:41:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568579+0000", 2024-04-22T18:41:55.286 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-04-22T18:41:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.668711+0000", 2024-04-22T18:41:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100131866 2024-04-22T18:41:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.768945+0000", 2024-04-22T18:41:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100234252 2024-04-22T18:41:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.287 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769002+0000", 2024-04-22T18:41:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7092999999999997e-05 2024-04-22T18:41:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.166404+0000", 2024-04-22T18:41:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39740241999999998 2024-04-22T18:41:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.191686+0000", 2024-04-22T18:41:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025281739000000001 2024-04-22T18:41:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.288 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267758+0000", 2024-04-22T18:41:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076071606999999999 2024-04-22T18:41:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267812+0000", 2024-04-22T18:41:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4478999999999998e-05 2024-04-22T18:41:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267848+0000", 2024-04-22T18:41:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6062999999999997e-05 2024-04-22T18:41:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3319 4.a 4:5d2aee44:::smithi08937929-13:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:41:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 88.779301490999998, 2024-04-22T18:41:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476623401, 2024-04-22T18:41:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3319 2024-04-22T18:41:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.290 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:41:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460306+0000", 2024-04-22T18:41:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:55.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460318+0000", 2024-04-22T18:41:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2896e-05 2024-04-22T18:41:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460345+0000", 2024-04-22T18:41:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6650000000000001e-05 2024-04-22T18:41:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460360+0000", 2024-04-22T18:41:55.292 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4729000000000001e-05 2024-04-22T18:41:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460502+0000", 2024-04-22T18:41:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141729 2024-04-22T18:41:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460644+0000", 2024-04-22T18:41:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142755 2024-04-22T18:41:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.293 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.936932+0000", 2024-04-22T18:41:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476288078 2024-04-22T18:41:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3412 4.a 4:519c08f7:::smithi08937929-15:head [write 2529878~394573] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.097466929999996, 2024-04-22T18:41:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48918094899999998, 2024-04-22T18:41:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.294 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3412 2024-04-22T18:41:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:41:55.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142141+0000", 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142496+0000", 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035444699999999998 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142510+0000", 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3841e-05 2024-04-22T18:41:55.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142518+0000", 2024-04-22T18:41:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8089999999999993e-06 2024-04-22T18:41:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142560+0000", 2024-04-22T18:41:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2097e-05 2024-04-22T18:41:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142637+0000", 2024-04-22T18:41:55.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6965000000000006e-05 2024-04-22T18:41:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142715+0000", 2024-04-22T18:41:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8694000000000002e-05 2024-04-22T18:41:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144179+0000", 2024-04-22T18:41:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146342 2024-04-22T18:41:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631229+0000", 2024-04-22T18:41:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48705018500000002 2024-04-22T18:41:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631281+0000", 2024-04-22T18:41:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2216999999999998e-05 2024-04-22T18:41:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631324+0000", 2024-04-22T18:41:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3344999999999999e-05 2024-04-22T18:41:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.299 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3413 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2924451] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.097085789999994, 2024-04-22T18:41:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58904086899999997, 2024-04-22T18:41:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3413 2024-04-22T18:41:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.300 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:41:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142523+0000", 2024-04-22T18:41:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:55.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142529+0000", 2024-04-22T18:41:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6049999999999997e-06 2024-04-22T18:41:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142535+0000", 2024-04-22T18:41:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4740000000000002e-06 2024-04-22T18:41:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142541+0000", 2024-04-22T18:41:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1839999999999998e-06 2024-04-22T18:41:55.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143553+0000", 2024-04-22T18:41:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010126689999999999 2024-04-22T18:41:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143659+0000", 2024-04-22T18:41:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105393 2024-04-22T18:41:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143825+0000", 2024-04-22T18:41:55.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165809 2024-04-22T18:41:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144329+0000", 2024-04-22T18:41:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050483799999999997 2024-04-22T18:41:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731480+0000", 2024-04-22T18:41:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58715100600000003 2024-04-22T18:41:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731522+0000", 2024-04-22T18:41:55.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2098000000000002e-05 2024-04-22T18:41:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731566+0000", 2024-04-22T18:41:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3066999999999999e-05 2024-04-22T18:41:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3414 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 84.097057186000001, 2024-04-22T18:41:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58926020199999996, 2024-04-22T18:41:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3414 2024-04-22T18:41:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:41:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142552+0000", 2024-04-22T18:41:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142556+0000", 2024-04-22T18:41:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0289999999999999e-06 2024-04-22T18:41:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142574+0000", 2024-04-22T18:41:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7266999999999999e-05 2024-04-22T18:41:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142581+0000", 2024-04-22T18:41:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-04-22T18:41:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144121+0000", 2024-04-22T18:41:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015397480000000001 2024-04-22T18:41:55.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:55.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144170+0000", 2024-04-22T18:41:55.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8803e-05 2024-04-22T18:41:55.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731624+0000", 2024-04-22T18:41:55.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58745365800000005 2024-04-22T18:41:55.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731712+0000", 2024-04-22T18:41:55.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8016999999999994e-05 2024-04-22T18:41:55.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731813+0000", 2024-04-22T18:41:55.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101817 2024-04-22T18:41:55.310 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.310 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.310 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.310 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:77 3.3 3:dcc8aeec:::3590d5741efdda41a26b62a0982727439225b7b4b51097b313f30d8dc10bf6f6:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:55.310 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:55.310 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.992038863999994, 2024-04-22T18:41:55.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011812878, 2024-04-22T18:41:55.310 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247571+0000", 2024-04-22T18:41:55.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247574+0000", 2024-04-22T18:41:55.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:41:55.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247577+0000", 2024-04-22T18:41:55.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3069999999999998e-06 2024-04-22T18:41:55.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247580+0000", 2024-04-22T18:41:55.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8870000000000001e-06 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.340928+0000", 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093348678000000004 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341122+0000", 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000193227 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618335+0000", 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.277213775 2024-04-22T18:41:55.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:55.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618422+0000", 2024-04-22T18:41:55.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6537000000000002e-05 2024-04-22T18:41:55.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954387+0000", 2024-04-22T18:41:55.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33596509699999999 2024-04-22T18:41:55.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954548+0000", 2024-04-22T18:41:55.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016042199999999999 2024-04-22T18:41:55.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:55.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954858+0000", 2024-04-22T18:41:55.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000310396 2024-04-22T18:41:55.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157019+0000", 2024-04-22T18:41:55.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20216071299999999 2024-04-22T18:41:55.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259213+0000", 2024-04-22T18:41:55.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102194734 2024-04-22T18:41:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259262+0000", 2024-04-22T18:41:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9052999999999999e-05 2024-04-22T18:41:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259384+0000", 2024-04-22T18:41:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122052 2024-04-22T18:41:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:78 3.3 3:df7e90d2:::ab0657e24bd96704775de2a378746bc22309c5a7366a5eaf9b4e1f1f2bef0a4c:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.992023666999998, 2024-04-22T18:41:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011974739, 2024-04-22T18:41:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:41:55.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247586+0000", 2024-04-22T18:41:55.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.318 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247589+0000", 2024-04-22T18:41:55.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.181e-06 2024-04-22T18:41:55.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.318 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247592+0000", 2024-04-22T18:41:55.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0230000000000001e-06 2024-04-22T18:41:55.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247595+0000", 2024-04-22T18:41:55.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.428e-06 2024-04-22T18:41:55.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341160+0000", 2024-04-22T18:41:55.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093564571999999999 2024-04-22T18:41:55.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:55.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341264+0000", 2024-04-22T18:41:55.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010391999999999999 2024-04-22T18:41:55.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619269+0000", 2024-04-22T18:41:55.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27800575100000002 2024-04-22T18:41:55.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:55.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619346+0000", 2024-04-22T18:41:55.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6459999999999996e-05 2024-04-22T18:41:55.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.155957+0000", 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53661109299999998 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156049+0000", 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2272e-05 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156236+0000", 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000186808 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.158088+0000", 2024-04-22T18:41:55.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018522079999999999 2024-04-22T18:41:55.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259421+0000", 2024-04-22T18:41:55.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101332718 2024-04-22T18:41:55.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259460+0000", 2024-04-22T18:41:55.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8912000000000003e-05 2024-04-22T18:41:55.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259562+0000", 2024-04-22T18:41:55.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010175600000000001 2024-04-22T18:41:55.323 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.323 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.323 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.323 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:80 3.3 3:f90f47f2:::e5603f596d699ee3e27e255225eeb90a9f0c1b4a4d099d1c586b19019220aa3f:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:55.323 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:55.323 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.991984590000001, 2024-04-22T18:41:55.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0121096759999999, 2024-04-22T18:41:55.323 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.323 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.323 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:55.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:41:55.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247625+0000", 2024-04-22T18:41:55.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247628+0000", 2024-04-22T18:41:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6129999999999998e-06 2024-04-22T18:41:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247631+0000", 2024-04-22T18:41:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3869999999999999e-06 2024-04-22T18:41:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247634+0000", 2024-04-22T18:41:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9919999999999999e-06 2024-04-22T18:41:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341550+0000", 2024-04-22T18:41:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093915584999999996 2024-04-22T18:41:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341700+0000", 2024-04-22T18:41:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015000300000000001 2024-04-22T18:41:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620744+0000", 2024-04-22T18:41:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27904399200000002 2024-04-22T18:41:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620794+0000", 2024-04-22T18:41:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9579000000000001e-05 2024-04-22T18:41:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157597+0000", 2024-04-22T18:41:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53680343799999997 2024-04-22T18:41:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157678+0000", 2024-04-22T18:41:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0829999999999997e-05 2024-04-22T18:41:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157832+0000", 2024-04-22T18:41:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154184 2024-04-22T18:41:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259572+0000", 2024-04-22T18:41:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10174003600000001 2024-04-22T18:41:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259601+0000", 2024-04-22T18:41:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8841999999999999e-05 2024-04-22T18:41:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259638+0000", 2024-04-22T18:41:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7308999999999998e-05 2024-04-22T18:41:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259736+0000", 2024-04-22T18:41:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7283999999999999e-05 2024-04-22T18:41:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:135 3.3 3:f89134e5:::3e4e73f5d4a0122957865f0faeac2483204ee98d7d497ef3b54afe2a02b39a3d:head [read 0~802304 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 79.624169511000005, 2024-04-22T18:41:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.64370653499999997, 2024-04-22T18:41:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615438+0000", 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615449+0000", 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1219999999999999e-05 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615460+0000", 2024-04-22T18:41:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0403e-05 2024-04-22T18:41:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.715588+0000", 2024-04-22T18:41:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100128881 2024-04-22T18:41:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722533+0000", 2024-04-22T18:41:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069444010000000002 2024-04-22T18:41:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:41:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722658+0000", 2024-04-22T18:41:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012560699999999999 2024-04-22T18:41:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826437+0000", 2024-04-22T18:41:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103778194 2024-04-22T18:41:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826532+0000", 2024-04-22T18:41:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5111000000000005e-05 2024-04-22T18:41:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258787+0000", 2024-04-22T18:41:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43225529099999999 2024-04-22T18:41:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258905+0000", 2024-04-22T18:41:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118348 2024-04-22T18:41:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259148+0000", 2024-04-22T18:41:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000242093 2024-04-22T18:41:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3890 4.1 4:83c955bf:::smithi08937929-28:head [write 2699879~469889] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.625494666999998, 2024-04-22T18:41:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49042297000000001, 2024-04-22T18:41:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3890 2024-04-22T18:41:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:41:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614115+0000", 2024-04-22T18:41:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614496+0000", 2024-04-22T18:41:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038181700000000001 2024-04-22T18:41:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614509+0000", 2024-04-22T18:41:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2381000000000001e-05 2024-04-22T18:41:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614517+0000", 2024-04-22T18:41:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0879999999999995e-06 2024-04-22T18:41:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615610+0000", 2024-04-22T18:41:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010933519999999999 2024-04-22T18:41:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615655+0000", 2024-04-22T18:41:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4966000000000003e-05 2024-04-22T18:41:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615769+0000", 2024-04-22T18:41:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113986 2024-04-22T18:41:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704882+0000", 2024-04-22T18:41:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089112825000000007 2024-04-22T18:41:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104488+0000", 2024-04-22T18:41:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39960572999999999 2024-04-22T18:41:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104517+0000", 2024-04-22T18:41:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9006e-05 2024-04-22T18:41:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104539+0000", 2024-04-22T18:41:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2073e-05 2024-04-22T18:41:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3891 4.1 4:83c955bf:::smithi08937929-28:head [setxattr _header (56) in=7b,truncate 3630345] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.625086436000004, 2024-04-22T18:41:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59018588699999996, 2024-04-22T18:41:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3891 2024-04-22T18:41:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:41:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614523+0000", 2024-04-22T18:41:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614528+0000", 2024-04-22T18:41:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0379999999999999e-06 2024-04-22T18:41:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614540+0000", 2024-04-22T18:41:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018000000000001e-05 2024-04-22T18:41:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614546+0000", 2024-04-22T18:41:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4990000000000002e-06 2024-04-22T18:41:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704438+0000", 2024-04-22T18:41:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089892044000000004 2024-04-22T18:41:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704504+0000", 2024-04-22T18:41:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6365000000000006e-05 2024-04-22T18:41:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:41:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704602+0000", 2024-04-22T18:41:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8048999999999996e-05 2024-04-22T18:41:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.806370+0000", 2024-04-22T18:41:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10176768 2024-04-22T18:41:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204629+0000", 2024-04-22T18:41:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39825917999999999 2024-04-22T18:41:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204673+0000", 2024-04-22T18:41:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3516999999999997e-05 2024-04-22T18:41:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204710+0000", 2024-04-22T18:41:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7391000000000002e-05 2024-04-22T18:41:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3892 4.1 4:83c955bf:::smithi08937929-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:41:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 65.625059855000003, 2024-04-22T18:41:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59031281800000002, 2024-04-22T18:41:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3892 2024-04-22T18:41:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:41:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614550+0000", 2024-04-22T18:41:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614553+0000", 2024-04-22T18:41:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0879999999999999e-06 2024-04-22T18:41:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614559+0000", 2024-04-22T18:41:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9259999999999997e-06 2024-04-22T18:41:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614565+0000", 2024-04-22T18:41:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7950000000000004e-06 2024-04-22T18:41:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704818+0000", 2024-04-22T18:41:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090252624000000004 2024-04-22T18:41:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704866+0000", 2024-04-22T18:41:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8667e-05 2024-04-22T18:41:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204742+0000", 2024-04-22T18:41:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987551899999999 2024-04-22T18:41:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204799+0000", 2024-04-22T18:41:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7299000000000001e-05 2024-04-22T18:41:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204863+0000", 2024-04-22T18:41:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4455000000000005e-05 2024-04-22T18:41:55.351 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.351 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.351 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.351 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.351 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.351 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:55.352 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:55.353 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:55.353+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:55.453 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:55.455+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:55.496 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:55.496 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:55.496 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:41:55.496 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:55.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:55.554 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:55.555+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:55.605 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:55.605 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:55.605 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:55.605 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:55.605 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:55.613 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:55.657 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:55.658+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:55.716 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:55.716 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:55.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:55.716 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:55.716 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.716 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:55.716 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:55.716 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 190.862056913, 2024-04-22T18:41:55.716 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:41:55.717 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.717 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.717 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.717 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:55.717 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:55.717 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:41:55.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.717 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.717 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.717 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:55.717 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.717 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:55.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:55.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.718 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.718 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.718 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:41:55.718 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:41:55.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:41:55.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:41:55.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:41:55.719 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:41:55.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.719 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.719 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.719 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:41:55.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:41:55.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:41:55.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:41:55.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:55.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:41:55.720 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:41:55.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.720 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.720 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:41:55.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:41:55.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:41:55.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:41:55.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.721 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:41:55.721 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:41:55.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.721 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.722 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:41:55.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:41:55.722 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.722 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.722 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.722 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:55.722 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:55.722 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 190.862005751, 2024-04-22T18:41:55.722 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:41:55.722 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.722 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.723 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.723 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:55.723 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:55.723 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:41:55.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.723 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:55.723 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.723 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.723 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:55.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:55.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:41:55.724 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:41:55.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.724 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.724 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:41:55.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:41:55.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:41:55.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:41:55.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.725 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:41:55.725 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:41:55.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.725 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:41:55.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:41:55.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:55.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:41:55.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:41:55.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.726 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.726 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:41:55.726 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:41:55.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:41:55.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:41:55.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:41:55.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:41:55.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.727 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.727 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:41:55.727 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:41:55.727 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.728 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.728 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.728 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:55.728 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:55.728 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 190.86199052699999, 2024-04-22T18:41:55.728 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:41:55.728 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.728 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.728 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.728 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:55.728 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:55.728 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:41:55.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.729 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:55.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.729 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:55.729 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.729 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:41:55.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:41:55.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:41:55.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.730 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.730 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:41:55.730 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:41:55.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:41:55.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:41:55.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:41:55.731 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:41:55.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.731 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.731 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:41:55.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:41:55.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:55.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:41:55.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:41:55.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.732 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:41:55.732 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:41:55.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.732 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:41:55.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:41:55.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:41:55.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:41:55.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.733 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.733 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:41:55.733 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:41:55.733 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.733 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.733 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.734 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:55.734 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:55.734 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 190.86193094999999, 2024-04-22T18:41:55.734 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:41:55.734 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.734 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.734 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.734 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:55.734 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:55.734 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:41:55.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.734 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:55.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:55.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.735 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.735 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:55.735 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:41:55.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:41:55.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:41:55.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:41:55.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.736 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.736 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:41:55.736 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:41:55.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:41:55.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:41:55.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:41:55.737 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:41:55.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.737 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:55.737 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:41:55.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:41:55.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:41:55.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:41:55.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.738 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.738 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:41:55.738 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:41:55.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:41:55.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:41:55.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.739 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.739 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:41:55.739 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:41:55.739 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.739 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.739 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.740 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:55.740 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:55.740 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 190.86184863400001, 2024-04-22T18:41:55.740 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:41:55.740 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.740 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.740 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.740 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:55.740 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:55.740 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:41:55.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.740 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:55.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:55.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.741 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.741 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:41:55.741 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:41:55.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:41:55.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:41:55.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:41:55.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.742 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.742 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:41:55.742 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:41:55.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:41:55.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:41:55.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:41:55.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:41:55.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.743 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:55.743 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:41:55.743 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:41:55.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:41:55.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:41:55.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:41:55.744 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:41:55.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.744 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.744 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.744 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:41:55.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:41:55.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.745 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.745 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:41:55.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:41:55.745 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.745 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.745 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.745 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.745 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:55.745 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:55.745 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 190.86183847800001, 2024-04-22T18:41:55.745 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:41:55.746 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.746 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:55.746 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.746 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:55.746 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:55.746 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:41:55.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.746 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.746 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.746 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:55.746 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.746 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.747 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.747 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:55.747 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:41:55.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.748 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.748 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.748 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:41:55.748 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:41:55.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:41:55.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:41:55.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:41:55.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:41:55.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.749 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.749 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:41:55.749 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:41:55.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:41:55.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:41:55.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:55.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:41:55.750 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:41:55.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.750 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:55.750 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:41:55.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:41:55.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:55.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:41:55.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:41:55.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:55.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:41:55.751 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:41:55.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.751 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.751 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:41:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:41:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3163 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 93.878826040000007, 2024-04-22T18:41:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1009801299999999, 2024-04-22T18:41:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.752 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3163 2024-04-22T18:41:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.753 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837455+0000", 2024-04-22T18:41:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837464+0000", 2024-04-22T18:41:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5319999999999995e-06 2024-04-22T18:41:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837476+0000", 2024-04-22T18:41:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1759e-05 2024-04-22T18:41:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.754 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837488+0000", 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1666e-05 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837541+0000", 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2939999999999998e-05 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837547+0000", 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6409999999999996e-06 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.755 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937423+0000", 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.099875545 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937567+0000", 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014363099999999999 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938437+0000", 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087060399999999997 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.757 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3164 4.8 4:187adac1:::smithi08937929-36:head [sparse-read 0~2643888] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:55.757 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:55.757 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 93.878786671, 2024-04-22T18:41:55.757 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1013670310000001, 2024-04-22T18:41:55.757 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.757 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.757 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.757 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.757 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.757 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3164 2024-04-22T18:41:55.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.757 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.757 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.757 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:55.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:55.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:55.758 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.758 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.758 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837499+0000", 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.994e-06 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837503+0000", 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7249999999999999e-06 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837506+0000", 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5980000000000001e-06 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937641+0000", 2024-04-22T18:41:55.759 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10013467700000001 2024-04-22T18:41:55.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:55.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937650+0000", 2024-04-22T18:41:55.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3489999999999998e-06 2024-04-22T18:41:55.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938468+0000", 2024-04-22T18:41:55.760 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0008176710000001 2024-04-22T18:41:55.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.760 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.760 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938582+0000", 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114287 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938864+0000", 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028120899999999998 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3165 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 93.878767902999996, 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2022754499999999, 2024-04-22T18:41:55.761 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.762 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.762 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3165 2024-04-22T18:41:55.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.762 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:55.762 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.762 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.762 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:55.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:55.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837517+0000", 2024-04-22T18:41:55.763 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3369999999999998e-06 2024-04-22T18:41:55.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.763 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.763 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837523+0000", 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5450000000000003e-06 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837526+0000", 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.354e-06 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937682+0000", 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100155917 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937688+0000", 2024-04-22T18:41:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.038953+0000", 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012648110000001 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039034+0000", 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0969000000000004e-05 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039791+0000", 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075672900000000004 2024-04-22T18:41:55.765 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.766 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.766 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.766 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3183 4.8 4:1bc4668a:::smithi08937929-21:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:55.766 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:55.766 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 93.680364705000002, 2024-04-22T18:41:55.766 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0040478260000001, 2024-04-22T18:41:55.766 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.766 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:55.766 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.766 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.766 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.766 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3183 2024-04-22T18:41:55.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035913+0000", 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999943 2024-04-22T18:41:55.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035935+0000", 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2232999999999999e-05 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035969+0000", 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3969e-05 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035987+0000", 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8397e-05 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2916000000000003e-05 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0103e-05 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039841+0000", 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0037508959999999 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.769 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.770 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039966+0000", 2024-04-22T18:41:55.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012514500000000001 2024-04-22T18:41:55.770 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.770 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.770 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.770 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3184 4.8 4:12d9f826:::smithi08937929-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:55.770 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:55.770 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 93.680276746000004, 2024-04-22T18:41:55.770 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.00402868, 2024-04-22T18:41:55.770 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.770 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:55.771 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.771 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.771 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.771 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3184 2024-04-22T18:41:55.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.771 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:55.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.771 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.771 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:55.771 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036005+0000", 2024-04-22T18:41:55.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:55.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.772 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036026+0000", 2024-04-22T18:41:55.772 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1092999999999999e-05 2024-04-22T18:41:55.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.772 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036030+0000", 2024-04-22T18:41:55.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.467e-06 2024-04-22T18:41:55.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036033+0000", 2024-04-22T18:41:55.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9780000000000001e-06 2024-04-22T18:41:55.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.773 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036120+0000", 2024-04-22T18:41:55.773 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7495000000000002e-05 2024-04-22T18:41:55.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.773 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:55.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036125+0000", 2024-04-22T18:41:55.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8720000000000001e-06 2024-04-22T18:41:55.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039982+0000", 2024-04-22T18:41:55.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0038571919999999 2024-04-22T18:41:55.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.774 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.774 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040035+0000", 2024-04-22T18:41:55.774 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2992999999999999e-05 2024-04-22T18:41:55.774 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.774 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.774 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3185 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 93.680222834000006, 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0041404039999999, 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3185 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.775 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.776 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036062+0000", 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9099999999999999e-06 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036066+0000", 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1699999999999999e-06 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036077+0000", 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0637e-05 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036147+0000", 2024-04-22T18:41:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0025999999999997e-05 2024-04-22T18:41:55.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:55.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036151+0000", 2024-04-22T18:41:55.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:41:55.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.778 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040068+0000", 2024-04-22T18:41:55.778 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.003916604 2024-04-22T18:41:55.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.778 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.778 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.779 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040201+0000", 2024-04-22T18:41:55.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013275400000000001 2024-04-22T18:41:55.779 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.779 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.779 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.779 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.779 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3186 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:55.779 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:55.779 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 93.680203277999993, 2024-04-22T18:41:55.779 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2044167510000001, 2024-04-22T18:41:55.779 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.779 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:55.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.779 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3186 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.780 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036079+0000", 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036081+0000", 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6449999999999999e-06 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036084+0000", 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9399999999999998e-06 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:41:55.781 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.879e-06 2024-04-22T18:41:55.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036168+0000", 2024-04-22T18:41:55.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7657999999999994e-05 2024-04-22T18:41:55.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:55.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036171+0000", 2024-04-22T18:41:55.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7639999999999999e-06 2024-04-22T18:41:55.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.782 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.782 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.782 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240345+0000", 2024-04-22T18:41:55.782 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2041734260000001 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240497+0000", 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015182800000000001 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3724 4.18 4:1bc4668a:::smithi08937929-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 70.655878901999998, 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50413114699999995, 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.783 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3724 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060405+0000", 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.395e-06 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060408+0000", 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.785 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060411+0000", 2024-04-22T18:41:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6060000000000001e-06 2024-04-22T18:41:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165799+0000", 2024-04-22T18:41:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105388369 2024-04-22T18:41:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165843+0000", 2024-04-22T18:41:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3891999999999999e-05 2024-04-22T18:41:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.786 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564408+0000", 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39856466000000002 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564461+0000", 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3529999999999997e-05 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564535+0000", 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4147999999999995e-05 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.788 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4413 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:55.788 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:55.788 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 38.058898857999999, 2024-04-22T18:41:55.788 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2025380569999999, 2024-04-22T18:41:55.788 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.788 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.788 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.788 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4413 2024-04-22T18:41:55.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.788 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.788 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657377+0000", 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999933 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.789 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657393+0000", 2024-04-22T18:41:55.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5296999999999999e-05 2024-04-22T18:41:55.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657411+0000", 2024-04-22T18:41:55.790 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8105e-05 2024-04-22T18:41:55.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.790 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.790 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.790 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657424+0000", 2024-04-22T18:41:55.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3030000000000001e-05 2024-04-22T18:41:55.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.793 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658313+0000", 2024-04-22T18:41:55.793 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00088943099999999999 2024-04-22T18:41:55.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.793 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.793 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:55.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658324+0000", 2024-04-22T18:41:55.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0496000000000001e-05 2024-04-22T18:41:55.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859560+0000", 2024-04-22T18:41:55.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2012358350000001 2024-04-22T18:41:55.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.794 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859648+0000", 2024-04-22T18:41:55.794 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8744000000000003e-05 2024-04-22T18:41:55.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.794 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.794 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.795 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859922+0000", 2024-04-22T18:41:55.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027396499999999998 2024-04-22T18:41:55.795 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.795 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.795 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.795 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.795 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4414 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:55.795 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:55.795 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 38.058844534000002, 2024-04-22T18:41:55.795 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2030255030000001, 2024-04-22T18:41:55.795 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.795 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.795 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4414 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.796 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657437+0000", 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657443+0000", 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5899999999999998e-06 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657450+0000", 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4649999999999997e-06 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.797 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657457+0000", 2024-04-22T18:41:55.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6000000000000003e-06 2024-04-22T18:41:55.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658364+0000", 2024-04-22T18:41:55.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090735999999999996 2024-04-22T18:41:55.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.798 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:55.798 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658373+0000", 2024-04-22T18:41:55.798 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3250000000000008e-06 2024-04-22T18:41:55.798 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.798 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860258+0000", 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018852769999999 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860308+0000", 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9938000000000002e-05 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860464+0000", 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015622800000000001 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.799 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.800 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4415 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:55.800 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:55.800 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 38.05882021, 2024-04-22T18:41:55.800 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2038250989999999, 2024-04-22T18:41:55.800 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.800 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.800 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.800 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.800 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.800 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4415 2024-04-22T18:41:55.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.800 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:55.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:55.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.801 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.801 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657462+0000", 2024-04-22T18:41:55.801 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:55.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657467+0000", 2024-04-22T18:41:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5399999999999997e-06 2024-04-22T18:41:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657473+0000", 2024-04-22T18:41:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3949999999999998e-06 2024-04-22T18:41:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657480+0000", 2024-04-22T18:41:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2489999999999998e-06 2024-04-22T18:41:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658394+0000", 2024-04-22T18:41:55.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00091403299999999997 2024-04-22T18:41:55.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:55.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658398+0000", 2024-04-22T18:41:55.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7979999999999996e-06 2024-04-22T18:41:55.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.803 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.803 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860495+0000", 2024-04-22T18:41:55.803 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.202097081 2024-04-22T18:41:55.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860543+0000", 2024-04-22T18:41:55.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7907000000000001e-05 2024-04-22T18:41:55.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.804 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.804 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861288+0000", 2024-04-22T18:41:55.804 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074466500000000004 2024-04-22T18:41:55.804 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.804 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.804 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.804 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4413 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:55.804 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:55.805 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 38.056103704999998, 2024-04-22T18:41:55.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.201491952, 2024-04-22T18:41:55.805 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.805 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.805 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.805 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.805 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.805 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4413 2024-04-22T18:41:55.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.805 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.805 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.805 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:55.805 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:55.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660178+0000", 2024-04-22T18:41:55.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:55.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.806 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.806 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660185+0000", 2024-04-22T18:41:55.806 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4379999999999998e-06 2024-04-22T18:41:55.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660201+0000", 2024-04-22T18:41:55.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5252999999999999e-05 2024-04-22T18:41:55.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660210+0000", 2024-04-22T18:41:55.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1069999999999994e-06 2024-04-22T18:41:55.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.807 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.807 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660270+0000", 2024-04-22T18:41:55.807 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739000000000002e-05 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660281+0000", 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0664e-05 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861304+0000", 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2010225059999999 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861342+0000", 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8736000000000003e-05 2024-04-22T18:41:55.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.809 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.809 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861671+0000", 2024-04-22T18:41:55.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032915899999999998 2024-04-22T18:41:55.809 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.809 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.809 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.809 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4414 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:55.809 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:55.809 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 38.056060195000001, 2024-04-22T18:41:55.809 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2014303930000001, 2024-04-22T18:41:55.809 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.809 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.810 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.810 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.810 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.810 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4414 2024-04-22T18:41:55.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.810 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:55.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.810 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.810 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:55.810 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660222+0000", 2024-04-22T18:41:55.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:55.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660227+0000", 2024-04-22T18:41:55.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6789999999999996e-06 2024-04-22T18:41:55.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.811 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.811 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660240+0000", 2024-04-22T18:41:55.811 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3485e-05 2024-04-22T18:41:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660245+0000", 2024-04-22T18:41:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8269999999999997e-06 2024-04-22T18:41:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660307+0000", 2024-04-22T18:41:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2392999999999997e-05 2024-04-22T18:41:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:55.812 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660316+0000", 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7790000000000001e-06 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861493+0000", 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.201177341 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861529+0000", 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5978e-05 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861653+0000", 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012391799999999999 2024-04-22T18:41:55.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4415 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 38.056014971000003, 2024-04-22T18:41:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2019634779999999, 2024-04-22T18:41:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4415 2024-04-22T18:41:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.815 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:55.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:55.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:55.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:55.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:55.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:55.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.815 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.815 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:55.815 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660267+0000", 2024-04-22T18:41:55.815 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:55.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:55.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660273+0000", 2024-04-22T18:41:55.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5910000000000003e-06 2024-04-22T18:41:55.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:55.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660280+0000", 2024-04-22T18:41:55.816 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4000000000000003e-06 2024-04-22T18:41:55.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.816 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.816 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:55.816 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660288+0000", 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1769999999999998e-06 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.760401+0000", 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100112864 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.760408+0000", 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0299999999999996e-06 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861682+0000", 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012742520000001 2024-04-22T18:41:55.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:55.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861716+0000", 2024-04-22T18:41:55.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3674999999999999e-05 2024-04-22T18:41:55.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:55.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:55.818 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:55.818 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.862232+0000", 2024-04-22T18:41:55.818 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051554300000000003 2024-04-22T18:41:55.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.818 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:55.818 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:55.818 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:41:55.820 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:55.821+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:55.821 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:55.920 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:55.921+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:55.965 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:55.965 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:55.965 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:55.965 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:55.974 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:56.022 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:56.023+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:56.075 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:56.075 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:56.075 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:56.085 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:56.127 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:56.129+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:56.188 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:56.188 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:56.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:56.188 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:56.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.188 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:56.188 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:56.188 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 181.73248694899999, 2024-04-22T18:41:56.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:41:56.188 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.188 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.188 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.189 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.189 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.189 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:41:56.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.189 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:56.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:56.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:41:56.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:56.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:41:56.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:41:56.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:41:56.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:41:56.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:41:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:41:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:41:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:41:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.193 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 173.03942970099999, 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:56.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:41:56.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:41:56.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:41:56.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:41:56.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:41:56.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:41:56.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:41:56.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:41:56.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:41:56.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:41:56.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:41:56.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:56.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:41:56.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:41:56.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:41:56.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:41:56.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:41:56.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:41:56.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:41:56.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:41:56.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:41:56.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:41:56.199 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.199 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.199 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.199 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:56.199 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:56.199 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 173.03941122500001, 2024-04-22T18:41:56.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:41:56.199 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.201 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.201 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.201 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.201 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.202 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:41:56.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.202 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:56.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:56.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:56.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:41:56.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:41:56.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:41:56.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:41:56.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:41:56.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:41:56.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:41:56.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:41:56.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:56.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:41:56.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:41:56.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:41:56.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:41:56.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:41:56.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:41:56.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:41:56.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:41:56.205 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.206 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.206 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.206 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:41:56.206 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:56.206 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 145.27636291300001, 2024-04-22T18:41:56.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:41:56.206 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.206 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.206 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.206 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.206 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:41:56.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:56.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:41:56.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:41:56.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:41:56.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:41:56.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:41:56.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:41:56.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:41:56.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:41:56.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:41:56.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:41:56.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:56.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:41:56.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:41:56.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:41:56.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:41:56.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:41:56.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:41:56.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 135.60393460399999, 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:41:56.211 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:41:56.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:41:56.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:41:56.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:41:56.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:41:56.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:41:56.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:56.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:41:56.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:41:56.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:41:56.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:41:56.216 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.216 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.216 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.216 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:41:56.216 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:56.216 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 125.794917897, 2024-04-22T18:41:56.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:41:56.216 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.216 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.216 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:41:56.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:41:56.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:41:56.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:41:56.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:41:56.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.898879525, 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:56.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:56.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:56.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:41:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.224 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3377 4.6 4:6700eab5:::smithi08937929-32:head [read 1812992~307712] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 85.751475882999998, 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.52001745799999999, 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3377 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:56.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436364+0000", 2024-04-22T18:41:56.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:56.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436376+0000", 2024-04-22T18:41:56.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2121e-05 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436394+0000", 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8094e-05 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436405+0000", 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0189e-05 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436471+0000", 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6817999999999996e-05 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436481+0000", 2024-04-22T18:41:56.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2720000000000003e-06 2024-04-22T18:41:56.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956084+0000", 2024-04-22T18:41:56.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51960320800000004 2024-04-22T18:41:56.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956232+0000", 2024-04-22T18:41:56.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014778600000000001 2024-04-22T18:41:56.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956385+0000", 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153394 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:131 3.1 3:a0a19997:::ee8ca23a7050e771c77dff761e6ea5a11ed57796f4e8a17cbb5804a7fdc937b9:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 84.858606287000001, 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51449564999999997, 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:56.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:56.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329235+0000", 2024-04-22T18:41:56.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:56.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329244+0000", 2024-04-22T18:41:56.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6819999999999992e-06 2024-04-22T18:41:56.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329259+0000", 2024-04-22T18:41:56.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698000000000001e-05 2024-04-22T18:41:56.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329265+0000", 2024-04-22T18:41:56.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1510000000000001e-06 2024-04-22T18:41:56.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329311+0000", 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6096e-05 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.332269+0000", 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029581910000000002 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630603+0000", 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29833418 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630685+0000", 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1217999999999994e-05 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630813+0000", 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128706 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.631447+0000", 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063427000000000004 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843586+0000", 2024-04-22T18:41:56.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21213873 2024-04-22T18:41:56.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843623+0000", 2024-04-22T18:41:56.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6848999999999999e-05 2024-04-22T18:41:56.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843733+0000", 2024-04-22T18:41:56.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010982200000000001 2024-04-22T18:41:56.234 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.234 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.234 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3441 4.b 4:d3068ee3:::smithi08937929-29:head [write 558065~619869] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 84.270130226000006, 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50272020399999995, 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3441 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917711+0000", 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917838+0000", 2024-04-22T18:41:56.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012734899999999999 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917854+0000", 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6130999999999999e-05 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917863+0000", 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5969999999999992e-06 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992845+0000", 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.074982193000000003 2024-04-22T18:41:56.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992919+0000", 2024-04-22T18:41:56.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:41:56.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:56.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.993019+0000", 2024-04-22T18:41:56.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9718999999999999e-05 2024-04-22T18:41:56.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095554+0000", 2024-04-22T18:41:56.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102535193 2024-04-22T18:41:56.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420183+0000", 2024-04-22T18:41:56.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32462853600000002 2024-04-22T18:41:56.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420401+0000", 2024-04-22T18:41:56.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021881400000000001 2024-04-22T18:41:56.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420433+0000", 2024-04-22T18:41:56.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079000000000002e-05 2024-04-22T18:41:56.239 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.240 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.240 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.240 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3442 4.b 4:d3068ee3:::smithi08937929-29:head [write 1579835~486765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:56.240 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:56.240 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 84.265443515000001, 2024-04-22T18:41:56.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49822799099999998, 2024-04-22T18:41:56.240 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.240 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.240 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.240 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.240 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.240 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3442 2024-04-22T18:41:56.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.241 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:56.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:56.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922399+0000", 2024-04-22T18:41:56.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:56.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922502+0000", 2024-04-22T18:41:56.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102784 2024-04-22T18:41:56.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922513+0000", 2024-04-22T18:41:56.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1051e-05 2024-04-22T18:41:56.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922518+0000", 2024-04-22T18:41:56.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8529999999999997e-06 2024-04-22T18:41:56.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093158+0000", 2024-04-22T18:41:56.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.17063958800000001 2024-04-22T18:41:56.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093230+0000", 2024-04-22T18:41:56.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1803000000000004e-05 2024-04-22T18:41:56.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093326+0000", 2024-04-22T18:41:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6454999999999997e-05 2024-04-22T18:41:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095557+0000", 2024-04-22T18:41:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022306399999999999 2024-04-22T18:41:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420484+0000", 2024-04-22T18:41:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32492717700000001 2024-04-22T18:41:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420606+0000", 2024-04-22T18:41:56.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012225800000000001 2024-04-22T18:41:56.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420628+0000", 2024-04-22T18:41:56.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1596000000000001e-05 2024-04-22T18:41:56.245 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.245 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.245 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.245 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3443 4.b 4:d3068ee3:::smithi08937929-29:head [write 2468371~489694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:56.245 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:56.245 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 84.260841467999995, 2024-04-22T18:41:56.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49372487199999998, 2024-04-22T18:41:56.246 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.246 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.246 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.246 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.246 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.246 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3443 2024-04-22T18:41:56.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.246 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:56.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:56.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927001+0000", 2024-04-22T18:41:56.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:56.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927117+0000", 2024-04-22T18:41:56.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011637 2024-04-22T18:41:56.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927128+0000", 2024-04-22T18:41:56.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0696000000000001e-05 2024-04-22T18:41:56.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927134+0000", 2024-04-22T18:41:56.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8640000000000001e-06 2024-04-22T18:41:56.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094288+0000", 2024-04-22T18:41:56.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16715412900000001 2024-04-22T18:41:56.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094359+0000", 2024-04-22T18:41:56.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1639999999999998e-05 2024-04-22T18:41:56.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:56.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094451+0000", 2024-04-22T18:41:56.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1252e-05 2024-04-22T18:41:56.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095560+0000", 2024-04-22T18:41:56.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011095530000000001 2024-04-22T18:41:56.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420661+0000", 2024-04-22T18:41:56.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32510105500000003 2024-04-22T18:41:56.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420710+0000", 2024-04-22T18:41:56.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9018999999999999e-05 2024-04-22T18:41:56.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420727+0000", 2024-04-22T18:41:56.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6722e-05 2024-04-22T18:41:56.251 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.251 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.251 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.251 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3444 4.b 4:d3068ee3:::smithi08937929-29:head [setxattr _header (56) in=7b,truncate 2958065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:56.251 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:56.251 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 84.260705751000003, 2024-04-22T18:41:56.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49366935299999998, 2024-04-22T18:41:56.251 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.251 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.251 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.251 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.252 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.252 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3444 2024-04-22T18:41:56.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.252 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:56.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:56.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927137+0000", 2024-04-22T18:41:56.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927141+0000", 2024-04-22T18:41:56.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3270000000000001e-06 2024-04-22T18:41:56.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927144+0000", 2024-04-22T18:41:56.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9730000000000002e-06 2024-04-22T18:41:56.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927147+0000", 2024-04-22T18:41:56.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.959e-06 2024-04-22T18:41:56.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095118+0000", 2024-04-22T18:41:56.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16797115500000001 2024-04-22T18:41:56.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095169+0000", 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1316000000000001e-05 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095260+0000", 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1184e-05 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095999+0000", 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073868200000000005 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420757+0000", 2024-04-22T18:41:56.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32475783499999999 2024-04-22T18:41:56.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420781+0000", 2024-04-22T18:41:56.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4022000000000001e-05 2024-04-22T18:41:56.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420807+0000", 2024-04-22T18:41:56.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6366000000000001e-05 2024-04-22T18:41:56.256 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.256 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.256 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.257 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3445 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:56.257 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:56.257 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 84.260694657000002, 2024-04-22T18:41:56.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.493780049, 2024-04-22T18:41:56.257 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.257 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.257 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.257 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.257 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.257 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3445 2024-04-22T18:41:56.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.257 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:56.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:56.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:56.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927150+0000", 2024-04-22T18:41:56.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.542e-06 2024-04-22T18:41:56.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927153+0000", 2024-04-22T18:41:56.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0520000000000002e-06 2024-04-22T18:41:56.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027218+0000", 2024-04-22T18:41:56.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100064604 2024-04-22T18:41:56.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095501+0000", 2024-04-22T18:41:56.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.068283431000000006 2024-04-22T18:41:56.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:56.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095545+0000", 2024-04-22T18:41:56.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4295999999999997e-05 2024-04-22T18:41:56.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420833+0000", 2024-04-22T18:41:56.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32528718400000001 2024-04-22T18:41:56.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420877+0000", 2024-04-22T18:41:56.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4403999999999997e-05 2024-04-22T18:41:56.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420929+0000", 2024-04-22T18:41:56.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1848999999999998e-05 2024-04-22T18:41:56.261 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.261 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.261 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.261 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4780 4.6 4:630bd8e5:::smithi08937929-45:head [write 574905~711397,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:56.262 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.278066+0000", 2024-04-22T18:41:56.262 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.909777412, 2024-04-22T18:41:56.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88434433999999995, 2024-04-22T18:41:56.262 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.262 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.262 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.262 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.262 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.262 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4780 2024-04-22T18:41:56.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.262 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278066+0000", 2024-04-22T18:41:56.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278066+0000", 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278063+0000", 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278391+0000", 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032839499999999999 2024-04-22T18:41:56.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278404+0000", 2024-04-22T18:41:56.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3528e-05 2024-04-22T18:41:56.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278415+0000", 2024-04-22T18:41:56.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0295000000000001e-05 2024-04-22T18:41:56.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278454+0000", 2024-04-22T18:41:56.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9551999999999997e-05 2024-04-22T18:41:56.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278462+0000", 2024-04-22T18:41:56.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4080000000000001e-06 2024-04-22T18:41:56.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.959123+0000", 2024-04-22T18:41:56.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68066109799999996 2024-04-22T18:41:56.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.959282+0000", 2024-04-22T18:41:56.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000159023 2024-04-22T18:41:56.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:56.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.959448+0000", 2024-04-22T18:41:56.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016666200000000001 2024-04-22T18:41:56.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162299+0000", 2024-04-22T18:41:56.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20285081199999999 2024-04-22T18:41:56.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162316+0000", 2024-04-22T18:41:56.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6354999999999998e-05 2024-04-22T18:41:56.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162347+0000", 2024-04-22T18:41:56.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1427999999999998e-05 2024-04-22T18:41:56.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162410+0000", 2024-04-22T18:41:56.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3350999999999997e-05 2024-04-22T18:41:56.267 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.267 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.267 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4781 4.6 4:630bd8e5:::smithi08937929-45:head [write 1885086~525140] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.284151+0000", 2024-04-22T18:41:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.903692572000001, 2024-04-22T18:41:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88746760999999996, 2024-04-22T18:41:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4781 2024-04-22T18:41:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284151+0000", 2024-04-22T18:41:56.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284151+0000", 2024-04-22T18:41:56.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284149+0000", 2024-04-22T18:41:56.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:56.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284385+0000", 2024-04-22T18:41:56.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023556799999999999 2024-04-22T18:41:56.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284396+0000", 2024-04-22T18:41:56.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0842999999999999e-05 2024-04-22T18:41:56.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284401+0000", 2024-04-22T18:41:56.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3260000000000003e-06 2024-04-22T18:41:56.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284444+0000", 2024-04-22T18:41:56.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3186000000000003e-05 2024-04-22T18:41:56.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284454+0000", 2024-04-22T18:41:56.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6160000000000008e-06 2024-04-22T18:41:56.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.161315+0000", 2024-04-22T18:41:56.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87686138000000002 2024-04-22T18:41:56.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.161381+0000", 2024-04-22T18:41:56.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5838000000000002e-05 2024-04-22T18:41:56.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:56.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.161470+0000", 2024-04-22T18:41:56.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9332000000000005e-05 2024-04-22T18:41:56.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163245+0000", 2024-04-22T18:41:56.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017744530000000001 2024-04-22T18:41:56.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.171550+0000", 2024-04-22T18:41:56.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083046879999999993 2024-04-22T18:41:56.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.171592+0000", 2024-04-22T18:41:56.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2438999999999998e-05 2024-04-22T18:41:56.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.171619+0000", 2024-04-22T18:41:56.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6517e-05 2024-04-22T18:41:56.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.274 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.274 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.274 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4782 4.6 4:630bd8e5:::smithi08937929-45:head [write 2930189~44716] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:56.274 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.287206+0000", 2024-04-22T18:41:56.274 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.900637605, 2024-04-22T18:41:56.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98327066299999999, 2024-04-22T18:41:56.274 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.274 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.274 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.274 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.274 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.274 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4782 2024-04-22T18:41:56.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.275 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287206+0000", 2024-04-22T18:41:56.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287206+0000", 2024-04-22T18:41:56.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287205+0000", 2024-04-22T18:41:56.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:56.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287220+0000", 2024-04-22T18:41:56.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5709000000000002e-05 2024-04-22T18:41:56.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287231+0000", 2024-04-22T18:41:56.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0196e-05 2024-04-22T18:41:56.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287236+0000", 2024-04-22T18:41:56.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3770000000000004e-06 2024-04-22T18:41:56.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287264+0000", 2024-04-22T18:41:56.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8549999999999999e-05 2024-04-22T18:41:56.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287268+0000", 2024-04-22T18:41:56.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7299999999999999e-06 2024-04-22T18:41:56.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162435+0000", 2024-04-22T18:41:56.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87516707999999999 2024-04-22T18:41:56.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162490+0000", 2024-04-22T18:41:56.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5210000000000002e-05 2024-04-22T18:41:56.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:56.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162567+0000", 2024-04-22T18:41:56.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6947999999999996e-05 2024-04-22T18:41:56.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163249+0000", 2024-04-22T18:41:56.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00068115299999999999 2024-04-22T18:41:56.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.270414+0000", 2024-04-22T18:41:56.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10716516700000001 2024-04-22T18:41:56.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.270448+0000", 2024-04-22T18:41:56.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4042999999999999e-05 2024-04-22T18:41:56.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.270477+0000", 2024-04-22T18:41:56.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8796e-05 2024-04-22T18:41:56.280 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.280 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.280 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.280 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4783 4.6 4:630bd8e5:::smithi08937929-45:head [setxattr _header (56) in=7b,truncate 2974905] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:56.280 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.287242+0000", 2024-04-22T18:41:56.280 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.900601491, 2024-04-22T18:41:56.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98449730599999996, 2024-04-22T18:41:56.280 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.280 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.280 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.281 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.281 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.281 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4783 2024-04-22T18:41:56.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.281 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287242+0000", 2024-04-22T18:41:56.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287242+0000", 2024-04-22T18:41:56.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287241+0000", 2024-04-22T18:41:56.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287245+0000", 2024-04-22T18:41:56.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.997e-06 2024-04-22T18:41:56.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287251+0000", 2024-04-22T18:41:56.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4999999999999999e-06 2024-04-22T18:41:56.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387333+0000", 2024-04-22T18:41:56.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100082174 2024-04-22T18:41:56.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387418+0000", 2024-04-22T18:41:56.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5086000000000005e-05 2024-04-22T18:41:56.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387427+0000", 2024-04-22T18:41:56.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6829999999999997e-06 2024-04-22T18:41:56.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162901+0000", 2024-04-22T18:41:56.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77547466899999995 2024-04-22T18:41:56.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162942+0000", 2024-04-22T18:41:56.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0299999999999997e-05 2024-04-22T18:41:56.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:56.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163028+0000", 2024-04-22T18:41:56.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6267000000000006e-05 2024-04-22T18:41:56.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163938+0000", 2024-04-22T18:41:56.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091012500000000002 2024-04-22T18:41:56.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271677+0000", 2024-04-22T18:41:56.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107739168 2024-04-22T18:41:56.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271709+0000", 2024-04-22T18:41:56.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1191e-05 2024-04-22T18:41:56.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271739+0000", 2024-04-22T18:41:56.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0814e-05 2024-04-22T18:41:56.286 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.286 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.286 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.286 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4784 4.6 4:630bd8e5:::smithi08937929-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:56.286 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.387355+0000", 2024-04-22T18:41:56.287 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.800488415, 2024-04-22T18:41:56.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.884536714, 2024-04-22T18:41:56.287 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.287 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.287 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.287 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.287 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.287 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4784 2024-04-22T18:41:56.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.287 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387355+0000", 2024-04-22T18:41:56.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387355+0000", 2024-04-22T18:41:56.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387352+0000", 2024-04-22T18:41:56.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:56.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387386+0000", 2024-04-22T18:41:56.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3974999999999999e-05 2024-04-22T18:41:56.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387399+0000", 2024-04-22T18:41:56.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2882e-05 2024-04-22T18:41:56.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387408+0000", 2024-04-22T18:41:56.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2310000000000002e-06 2024-04-22T18:41:56.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.487520+0000", 2024-04-22T18:41:56.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100111622 2024-04-22T18:41:56.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.487531+0000", 2024-04-22T18:41:56.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0655999999999999e-05 2024-04-22T18:41:56.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163199+0000", 2024-04-22T18:41:56.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.67566841700000002 2024-04-22T18:41:56.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:56.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163239+0000", 2024-04-22T18:41:56.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9749000000000002e-05 2024-04-22T18:41:56.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271780+0000", 2024-04-22T18:41:56.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10854082600000001 2024-04-22T18:41:56.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271830+0000", 2024-04-22T18:41:56.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0835999999999999e-05 2024-04-22T18:41:56.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271892+0000", 2024-04-22T18:41:56.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1440999999999997e-05 2024-04-22T18:41:56.292 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.292 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.292 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.292 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4940 3.1 3:af0c84c6:::a6954558d6d261bd8039fe057a41a32eb57dde7f3e53a34b25ba0607fb91f814:head [write 0~396288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-04-22T18:41:56.292 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:41.009760+0000", 2024-04-22T18:41:56.292 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 15.178083742, 2024-04-22T18:41:56.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.500823562, 2024-04-22T18:41:56.292 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.292 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.292 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.292 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.292 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.293 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4940 2024-04-22T18:41:56.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.293 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009760+0000", 2024-04-22T18:41:56.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009760+0000", 2024-04-22T18:41:56.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009758+0000", 2024-04-22T18:41:56.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:56.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009935+0000", 2024-04-22T18:41:56.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017724200000000001 2024-04-22T18:41:56.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009946+0000", 2024-04-22T18:41:56.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0477e-05 2024-04-22T18:41:56.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009952+0000", 2024-04-22T18:41:56.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3090000000000001e-06 2024-04-22T18:41:56.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.109570+0000", 2024-04-22T18:41:56.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099618130999999999 2024-04-22T18:41:56.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.109693+0000", 2024-04-22T18:41:56.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012241699999999999 2024-04-22T18:41:56.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:56.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.109807+0000", 2024-04-22T18:41:56.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011426599999999999 2024-04-22T18:41:56.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.110527+0000", 2024-04-22T18:41:56.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00071964499999999999 2024-04-22T18:41:56.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.510490+0000", 2024-04-22T18:41:56.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39996386499999997 2024-04-22T18:41:56.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.510550+0000", 2024-04-22T18:41:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9669000000000001e-05 2024-04-22T18:41:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.510583+0000", 2024-04-22T18:41:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3281000000000002e-05 2024-04-22T18:41:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.297 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.297 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:56.299 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:56.299 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:56.300+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:56.399 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:56.401+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:56.442 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:56.443 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:56.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.443 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:5327 4.19 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e179)", 2024-04-22T18:41:56.443 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:56.401435+0000", 2024-04-22T18:41:56.443 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.043070139, 2024-04-22T18:41:56.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043088815000000003, 2024-04-22T18:41:56.443 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.443 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-04-22T18:41:56.443 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.443 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.443 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.443 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5327 2024-04-22T18:41:56.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.443 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:56.401435+0000", 2024-04-22T18:41:56.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:56.401435+0000", 2024-04-22T18:41:56.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:56.401432+0000", 2024-04-22T18:41:56.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:56.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:56.401440+0000", 2024-04-22T18:41:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1729999999999998e-06 2024-04-22T18:41:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:56.401451+0000", 2024-04-22T18:41:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1473e-05 2024-04-22T18:41:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:56.401457+0000", 2024-04-22T18:41:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2330000000000003e-06 2024-04-22T18:41:56.445 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.446 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.446 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.446 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.446 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-04-22T18:41:56.446 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 1 2024-04-22T18:41:56.446 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:56.452 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:41:56.535 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:56.537+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:56.593 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:56.593 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:56.602 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:41:56.602 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:56.602+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:56.627 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:41:56.701 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:56.703+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:56.702 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:56.703+0000 7f3e7be88640 -1 osd.0 179 handle_osd_map injecting map crc failure 2024-04-22T18:41:56.751 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 182.55995196000001, 2024-04-22T18:41:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:41:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:41:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:41:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:41:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:41:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:41:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:41:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:41:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:41:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:41:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 182.559940885, 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:41:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:41:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:41:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:41:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:41:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:41:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3504 4.9 4:96737e19:::smithi08937929-4:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 81.340616881000003, 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.951755291, 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3504 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410861+0000", 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410863+0000", 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.635e-06 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410866+0000", 2024-04-22T18:41:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8499999999999998e-06 2024-04-22T18:41:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410868+0000", 2024-04-22T18:41:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.497e-06 2024-04-22T18:41:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615137+0000", 2024-04-22T18:41:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20426903499999999 2024-04-22T18:41:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615218+0000", 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0860999999999997e-05 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.362617+0000", 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.74739882700000004 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4194 4.b 4:d3068ee3:::smithi08937929-29:head [sparse-read 0~1980000,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 53.58735781, 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.294164774, 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4194 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:41:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164120+0000", 2024-04-22T18:41:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164125+0000", 2024-04-22T18:41:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2899999999999996e-06 2024-04-22T18:41:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164128+0000", 2024-04-22T18:41:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8700000000000002e-06 2024-04-22T18:41:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164132+0000", 2024-04-22T18:41:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.151e-06 2024-04-22T18:41:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164188+0000", 2024-04-22T18:41:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6912e-05 2024-04-22T18:41:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164201+0000", 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2466000000000001e-05 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.242879+0000", 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0786779710000001 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243364+0000", 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000485081 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456588+0000", 2024-04-22T18:41:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21322370900000001 2024-04-22T18:41:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456674+0000", 2024-04-22T18:41:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6079e-05 2024-04-22T18:41:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.458285+0000", 2024-04-22T18:41:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001611716 2024-04-22T18:41:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4195 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 53.587340187999999, 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.395681808, 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4195 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164141+0000", 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7160000000000001e-06 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164147+0000", 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0509999999999996e-06 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264244+0000", 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100097063 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264319+0000", 2024-04-22T18:41:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5729000000000006e-05 2024-04-22T18:41:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264331+0000", 2024-04-22T18:41:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048999999999999e-05 2024-04-22T18:41:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243392+0000", 2024-04-22T18:41:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97906041099999996 2024-04-22T18:41:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243451+0000", 2024-04-22T18:41:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8717e-05 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558398+0000", 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31494718500000002 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558490+0000", 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2529999999999997e-05 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559820+0000", 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013299060000000001 2024-04-22T18:41:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4196 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:41:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 53.487201441000003, 2024-04-22T18:41:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2963829229999999, 2024-04-22T18:41:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4196 2024-04-22T18:41:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264275+0000", 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264290+0000", 2024-04-22T18:41:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5373e-05 2024-04-22T18:41:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264300+0000", 2024-04-22T18:41:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.747e-06 2024-04-22T18:41:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264314+0000", 2024-04-22T18:41:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.396e-05 2024-04-22T18:41:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364450+0000", 2024-04-22T18:41:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013662500000001 2024-04-22T18:41:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364459+0000", 2024-04-22T18:41:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0710000000000001e-06 2024-04-22T18:41:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245690+0000", 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88123066400000005 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245766+0000", 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5938000000000004e-05 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559874+0000", 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31410827499999999 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559930+0000", 2024-04-22T18:41:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5367e-05 2024-04-22T18:41:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.560660+0000", 2024-04-22T18:41:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073043800000000003 2024-04-22T18:41:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4182 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.823518596, 2024-04-22T18:41:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2059652709999999, 2024-04-22T18:41:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4182 2024-04-22T18:41:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:41:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927959+0000", 2024-04-22T18:41:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927962+0000", 2024-04-22T18:41:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4269999999999998e-06 2024-04-22T18:41:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927968+0000", 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028037+0000", 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068544 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028078+0000", 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1523999999999997e-05 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028087+0000", 2024-04-22T18:41:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3140000000000004e-06 2024-04-22T18:41:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.129951+0000", 2024-04-22T18:41:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1018649229999999 2024-04-22T18:41:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.130067+0000", 2024-04-22T18:41:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115222 2024-04-22T18:41:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.133925+0000", 2024-04-22T18:41:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00385849 2024-04-22T18:41:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4183 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.723428163999998, 2024-04-22T18:41:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1063017610000001, 2024-04-22T18:41:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4183 2024-04-22T18:41:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:41:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028049+0000", 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028055+0000", 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028064+0000", 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2830000000000001e-06 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028071+0000", 2024-04-22T18:41:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2740000000000002e-06 2024-04-22T18:41:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028126+0000", 2024-04-22T18:41:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4771999999999997e-05 2024-04-22T18:41:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028134+0000", 2024-04-22T18:41:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5599999999999994e-06 2024-04-22T18:41:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134109+0000", 2024-04-22T18:41:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.105974864 2024-04-22T18:41:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134169+0000", 2024-04-22T18:41:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0167999999999997e-05 2024-04-22T18:41:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134352+0000", 2024-04-22T18:41:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018293699999999999 2024-04-22T18:41:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4190 4.c 4:3227431c:::smithi08937929-35:head [sparse-read 0~1258562,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.723402661000002, 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1098740030000001, 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4190 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028075+0000", 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028080+0000", 2024-04-22T18:41:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.561e-06 2024-04-22T18:41:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028085+0000", 2024-04-22T18:41:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0150000000000003e-06 2024-04-22T18:41:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028090+0000", 2024-04-22T18:41:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.169e-06 2024-04-22T18:41:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028175+0000", 2024-04-22T18:41:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4814999999999993e-05 2024-04-22T18:41:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028181+0000", 2024-04-22T18:41:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5490000000000003e-06 2024-04-22T18:41:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134400+0000", 2024-04-22T18:41:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1062186869999999 2024-04-22T18:41:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134552+0000", 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015210500000000001 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137950+0000", 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033978210000000001 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4191 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.723319912999997, 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.110917953, 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4191 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028157+0000", 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028164+0000", 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.511e-06 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028172+0000", 2024-04-22T18:41:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0339999999999997e-06 2024-04-22T18:41:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028179+0000", 2024-04-22T18:41:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8630000000000002e-06 2024-04-22T18:41:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028202+0000", 2024-04-22T18:41:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3635999999999999e-05 2024-04-22T18:41:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028209+0000", 2024-04-22T18:41:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4629999999999998e-06 2024-04-22T18:41:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137970+0000", 2024-04-22T18:41:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1097613479999999 2024-04-22T18:41:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.138034+0000", 2024-04-22T18:41:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3551000000000002e-05 2024-04-22T18:41:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.139077+0000", 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001042715 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4192 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 52.723294613, 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.212555542, 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4192 2024-04-22T18:41:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:41:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028183+0000", 2024-04-22T18:41:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028187+0000", 2024-04-22T18:41:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6119999999999999e-06 2024-04-22T18:41:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028192+0000", 2024-04-22T18:41:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3489999999999999e-06 2024-04-22T18:41:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028197+0000", 2024-04-22T18:41:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7489999999999998e-06 2024-04-22T18:41:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028228+0000", 2024-04-22T18:41:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1328999999999998e-05 2024-04-22T18:41:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028234+0000", 2024-04-22T18:41:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1129999999999997e-06 2024-04-22T18:41:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239245+0000", 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.211011032 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239384+0000", 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013902000000000001 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.240739+0000", 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013550389999999999 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4692 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 29.492338734, 2024-04-22T18:41:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80332623700000005, 2024-04-22T18:41:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4692 2024-04-22T18:41:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:41:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259139+0000", 2024-04-22T18:41:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259145+0000", 2024-04-22T18:41:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4860000000000002e-06 2024-04-22T18:41:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259151+0000", 2024-04-22T18:41:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1160000000000004e-06 2024-04-22T18:41:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259156+0000", 2024-04-22T18:41:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6489999999999999e-06 2024-04-22T18:41:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259191+0000", 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4737999999999998e-05 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259197+0000", 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9160000000000004e-06 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060062+0000", 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80086472900000005 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060225+0000", 2024-04-22T18:41:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000162771 2024-04-22T18:41:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062466+0000", 2024-04-22T18:41:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410569999999999 2024-04-22T18:41:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4693 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 29.492316448, 2024-04-22T18:41:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80370012999999996, 2024-04-22T18:41:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4693 2024-04-22T18:41:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:41:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259161+0000", 2024-04-22T18:41:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259164+0000", 2024-04-22T18:41:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-06 2024-04-22T18:41:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259169+0000", 2024-04-22T18:41:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6260000000000003e-06 2024-04-22T18:41:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259174+0000", 2024-04-22T18:41:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7659999999999998e-06 2024-04-22T18:41:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259231+0000", 2024-04-22T18:41:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6827000000000003e-05 2024-04-22T18:41:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259239+0000", 2024-04-22T18:41:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6889999999999993e-06 2024-04-22T18:41:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062492+0000", 2024-04-22T18:41:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80325246699999997 2024-04-22T18:41:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062573+0000", 2024-04-22T18:41:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1508000000000006e-05 2024-04-22T18:41:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062862+0000", 2024-04-22T18:41:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028893499999999998 2024-04-22T18:41:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4694 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:41:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 29.492300167, 2024-04-22T18:41:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.90474754099999999, 2024-04-22T18:41:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4694 2024-04-22T18:41:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:41:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259182+0000", 2024-04-22T18:41:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8700000000000002e-06 2024-04-22T18:41:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259187+0000", 2024-04-22T18:41:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1669999999999998e-06 2024-04-22T18:41:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259191+0000", 2024-04-22T18:41:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6210000000000003e-06 2024-04-22T18:41:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.359347+0000", 2024-04-22T18:41:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100155518 2024-04-22T18:41:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.359356+0000", 2024-04-22T18:41:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6519999999999995e-06 2024-04-22T18:41:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163126+0000", 2024-04-22T18:41:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.803770816 2024-04-22T18:41:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163217+0000", 2024-04-22T18:41:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0598000000000002e-05 2024-04-22T18:41:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163926+0000", 2024-04-22T18:41:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000708856 2024-04-22T18:41:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4791 4.7 4:e117a562:::smithi08937929-33:head [write 591712~786646] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.734465+0000", 2024-04-22T18:41:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 25.017013713000001, 2024-04-22T18:41:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4316495570000001, 2024-04-22T18:41:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4791 2024-04-22T18:41:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734465+0000", 2024-04-22T18:41:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734465+0000", 2024-04-22T18:41:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734460+0000", 2024-04-22T18:41:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999957 2024-04-22T18:41:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734848+0000", 2024-04-22T18:41:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000387353 2024-04-22T18:41:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734865+0000", 2024-04-22T18:41:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7486000000000002e-05 2024-04-22T18:41:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734875+0000", 2024-04-22T18:41:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-06 2024-04-22T18:41:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761489+0000", 2024-04-22T18:41:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026613405 2024-04-22T18:41:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761495+0000", 2024-04-22T18:41:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4350000000000002e-06 2024-04-22T18:41:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.961065+0000", 2024-04-22T18:41:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.199569608 2024-04-22T18:41:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.961155+0000", 2024-04-22T18:41:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0109000000000001e-05 2024-04-22T18:41:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.961254+0000", 2024-04-22T18:41:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9072000000000003e-05 2024-04-22T18:41:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.964390+0000", 2024-04-22T18:41:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031362009999999999 2024-04-22T18:41:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166027+0000", 2024-04-22T18:41:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.201636589 2024-04-22T18:41:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166075+0000", 2024-04-22T18:41:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7998999999999999e-05 2024-04-22T18:41:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166114+0000", 2024-04-22T18:41:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9638999999999998e-05 2024-04-22T18:41:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4792 4.7 4:e117a562:::smithi08937929-33:head [write 1943568~668798] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742160+0000", 2024-04-22T18:41:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 25.009318599, 2024-04-22T18:41:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.42404771, 2024-04-22T18:41:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4792 2024-04-22T18:41:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742160+0000", 2024-04-22T18:41:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742160+0000", 2024-04-22T18:41:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742158+0000", 2024-04-22T18:41:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742473+0000", 2024-04-22T18:41:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000315192 2024-04-22T18:41:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742485+0000", 2024-04-22T18:41:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1795e-05 2024-04-22T18:41:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742491+0000", 2024-04-22T18:41:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8760000000000002e-06 2024-04-22T18:41:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761511+0000", 2024-04-22T18:41:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019020760000000001 2024-04-22T18:41:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761515+0000", 2024-04-22T18:41:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.625e-06 2024-04-22T18:41:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.962761+0000", 2024-04-22T18:41:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.201246021 2024-04-22T18:41:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.962837+0000", 2024-04-22T18:41:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6025999999999993e-05 2024-04-22T18:41:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.962944+0000", 2024-04-22T18:41:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106949 2024-04-22T18:41:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.965124+0000", 2024-04-22T18:41:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021803230000000001 2024-04-22T18:41:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166154+0000", 2024-04-22T18:41:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20102966 2024-04-22T18:41:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166185+0000", 2024-04-22T18:41:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0713999999999998e-05 2024-04-22T18:41:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166208+0000", 2024-04-22T18:41:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2969999999999999e-05 2024-04-22T18:41:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4793 4.7 4:e117a562:::smithi08937929-33:head [setxattr _header (56) in=7b,truncate 2991712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742495+0000", 2024-04-22T18:41:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 25.008983687000001, 2024-04-22T18:41:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5238806949999999, 2024-04-22T18:41:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4793 2024-04-22T18:41:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742495+0000", 2024-04-22T18:41:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742495+0000", 2024-04-22T18:41:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742494+0000", 2024-04-22T18:41:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742498+0000", 2024-04-22T18:41:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3790000000000002e-06 2024-04-22T18:41:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742501+0000", 2024-04-22T18:41:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9900000000000002e-06 2024-04-22T18:41:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742504+0000", 2024-04-22T18:41:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:41:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761529+0000", 2024-04-22T18:41:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019025301000000001 2024-04-22T18:41:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761532+0000", 2024-04-22T18:41:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4589999999999998e-06 2024-04-22T18:41:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.963997+0000", 2024-04-22T18:41:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2024646560000001 2024-04-22T18:41:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.964075+0000", 2024-04-22T18:41:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8015000000000004e-05 2024-04-22T18:41:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.964176+0000", 2024-04-22T18:41:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010080499999999999 2024-04-22T18:41:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.965235+0000", 2024-04-22T18:41:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00105959 2024-04-22T18:41:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266295+0000", 2024-04-22T18:41:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30105968599999999 2024-04-22T18:41:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266330+0000", 2024-04-22T18:41:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5219999999999998e-05 2024-04-22T18:41:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266376+0000", 2024-04-22T18:41:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5262000000000001e-05 2024-04-22T18:41:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4794 4.7 4:e117a562:::smithi08937929-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:41:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 25.008972574000001, 2024-04-22T18:41:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.524010123, 2024-04-22T18:41:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4794 2024-04-22T18:41:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:41:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:41:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:41:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742507+0000", 2024-04-22T18:41:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.637e-06 2024-04-22T18:41:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742511+0000", 2024-04-22T18:41:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9890000000000003e-06 2024-04-22T18:41:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742514+0000", 2024-04-22T18:41:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.88e-06 2024-04-22T18:41:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.775057+0000", 2024-04-22T18:41:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.032543012000000003 2024-04-22T18:41:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.775069+0000", 2024-04-22T18:41:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2184000000000001e-05 2024-04-22T18:41:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.064059+0000", 2024-04-22T18:41:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.288990157 2024-04-22T18:41:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.064127+0000", 2024-04-22T18:41:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7741000000000001e-05 2024-04-22T18:41:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266405+0000", 2024-04-22T18:41:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20227735499999999 2024-04-22T18:41:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266458+0000", 2024-04-22T18:41:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3430000000000002e-05 2024-04-22T18:41:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266516+0000", 2024-04-22T18:41:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8038000000000002e-05 2024-04-22T18:41:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4795 4.1f 4.d9a85c1f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:41:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 25.008962203999999, 2024-04-22T18:41:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.118060236, 2024-04-22T18:41:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:41:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4795 2024-04-22T18:41:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:41:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:41:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:41:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742518+0000", 2024-04-22T18:41:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.584e-06 2024-04-22T18:41:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742520+0000", 2024-04-22T18:41:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7140000000000002e-06 2024-04-22T18:41:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742523+0000", 2024-04-22T18:41:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3609999999999999e-06 2024-04-22T18:41:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761544+0000", 2024-04-22T18:41:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019020925000000001 2024-04-22T18:41:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761547+0000", 2024-04-22T18:41:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7859999999999998e-06 2024-04-22T18:41:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.860553+0000", 2024-04-22T18:41:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0990060859999999 2024-04-22T18:41:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.860577+0000", 2024-04-22T18:41:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3064000000000001e-05 2024-04-22T18:41:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:5189 4.19 4:9ea6f51f:::smithi08937929-34:head [call cas.references_chunk in=19b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e174)", 2024-04-22T18:41:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:50.743728+0000", 2024-04-22T18:41:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.0077502819999999, 2024-04-22T18:41:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5121397000000001, 2024-04-22T18:41:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5189 2024-04-22T18:41:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743728+0000", 2024-04-22T18:41:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743728+0000", 2024-04-22T18:41:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743726+0000", 2024-04-22T18:41:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743736+0000", 2024-04-22T18:41:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9890000000000004e-06 2024-04-22T18:41:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743750+0000", 2024-04-22T18:41:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3533000000000001e-05 2024-04-22T18:41:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743760+0000", 2024-04-22T18:41:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0121e-05 2024-04-22T18:41:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743807+0000", 2024-04-22T18:41:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6972999999999999e-05 2024-04-22T18:41:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:41:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743812+0000", 2024-04-22T18:41:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:41:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.845187+0000", 2024-04-22T18:41:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101374929 2024-04-22T18:41:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.845194+0000", 2024-04-22T18:41:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7719999999999997e-06 2024-04-22T18:41:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:52.255561+0000", 2024-04-22T18:41:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4103678770000001 2024-04-22T18:41:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:52.255766+0000", 2024-04-22T18:41:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000204093 2024-04-22T18:41:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:52.255868+0000", 2024-04-22T18:41:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102332 2024-04-22T18:41:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:56.856 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:56.856 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 7, 2, 4, 0] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:41:56.856 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:41:56.857 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-04-22T18:41:56.857 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:41:56.859 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:41:56.859 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:56.859 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:56.859 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:56.860 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:56.860 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:56.860 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:56.861 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:56.861 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:56.861 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:56.861 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:56.862 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:56.862 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:56.863 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:56.859+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:56.863 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:56.863 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:56.879 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:56.879 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:56.906 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:56.907+0000 7f8607f36640 -1 osd.3 179 handle_osd_map injecting map crc failure 2024-04-22T18:41:56.959 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:56.960+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:56.968 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:56.968 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:56.979 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:41:57.063 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:57.064+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:57.122 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:57.122 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:57.131 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:41:57.156 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:41:57.157 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:41:57.168 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-04-22T18:41:56.997092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":272017324,"num_objects":263,"num_object_clones":0,"num_object_copies":526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":4261,"num_read_kb":3462598,"num_write":2756,"num_write_kb":643889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1243,"num_bytes_recovered":388595110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":26,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3255,"ondisk_log_size":3255,"up":98,"acting":98,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":563204,"kb_used_data":360100,"kb_used_omap":9,"kb_used_meta":202998,"kb_avail":565667836,"statfs":{"total":579820584960,"available":579243864064,"internally_reserved":0,"allocated":368742400,"data_stored":375175296,"data_compressed":12780311,"data_compressed_allocated":13647872,"data_compressed_original":28150962,"omap_allocated":9557,"internal_metadata":207870635},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2739624,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":-874,"num_read_kb":-1057096,"num_write":-298,"num_write_kb":-85762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-17,"num_bytes_recovered":-33654948,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000617"},"pg_stats":[{"pgid":"4.23","version":"155'91","reported_seq":405,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.763837+0000","last_change":"2024-04-22T18:41:44.918810+0000","last_active":"2024-04-22T18:41:54.763837+0000","last_peered":"2024-04-22T18:41:54.763837+0000","last_clean":"2024-04-22T18:41:54.763837+0000","last_became_active":"2024-04-22T18:41:44.918345+0000","last_became_peered":"2024-04-22T18:41:44.918345+0000","last_unstale":"2024-04-22T18:41:54.763837+0000","last_undegraded":"2024-04-22T18:41:54.763837+0000","last_fullsized":"2024-04-22T18:41:54.763837+0000","mapping_epoch":168,"log_start":"137'90","ondisk_log_start":"137'90","created":164,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":6,"last_scrub":"137'90","last_scrub_stamp":"2024-04-22T18:41:22.057812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:22.057812+0000","objects_scrubbed":2,"log_size":1,"log_dups_size":61,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655556,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":43,"num_read_kb":57909,"num_write":11,"num_write_kb":2755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5875215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.22","version":"149'42","reported_seq":357,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.844758+0000","last_change":"2024-04-22T18:41:40.908387+0000","last_active":"2024-04-22T18:41:50.844758+0000","last_peered":"2024-04-22T18:41:50.844758+0000","last_clean":"2024-04-22T18:41:50.844758+0000","last_became_active":"2024-04-22T18:41:40.908069+0000","last_became_peered":"2024-04-22T18:41:40.908069+0000","last_unstale":"2024-04-22T18:41:50.844758+0000","last_undegraded":"2024-04-22T18:41:50.844758+0000","last_fullsized":"2024-04-22T18:41:50.844758+0000","mapping_epoch":164,"log_start":"64'38","ondisk_log_start":"64'38","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":25,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":88762,"num_write":13,"num_write_kb":3449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5687418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"155'190","reported_seq":760,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.764526+0000","last_change":"2024-04-22T18:41:44.820690+0000","last_active":"2024-04-22T18:41:54.764526+0000","last_peered":"2024-04-22T18:41:54.764526+0000","last_clean":"2024-04-22T18:41:54.764526+0000","last_became_active":"2024-04-22T18:41:44.820321+0000","last_became_peered":"2024-04-22T18:41:44.820321+0000","last_unstale":"2024-04-22T18:41:54.764526+0000","last_undegraded":"2024-04-22T18:41:54.764526+0000","last_fullsized":"2024-04-22T18:41:54.764526+0000","mapping_epoch":168,"log_start":"147'185","ondisk_log_start":"147'185","created":164,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":6,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":5,"log_dups_size":161,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7042282,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":229510,"num_write":43,"num_write_kb":12083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":21653811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"162'34","reported_seq":298,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.843049+0000","last_change":"2024-04-22T18:41:41.107163+0000","last_active":"2024-04-22T18:41:50.843049+0000","last_peered":"2024-04-22T18:41:50.843049+0000","last_clean":"2024-04-22T18:41:50.843049+0000","last_became_active":"2024-04-22T18:41:41.106971+0000","last_became_peered":"2024-04-22T18:41:41.106971+0000","last_unstale":"2024-04-22T18:41:50.843049+0000","last_undegraded":"2024-04-22T18:41:50.843049+0000","last_fullsized":"2024-04-22T18:41:50.843049+0000","mapping_epoch":164,"log_start":"147'30","ondisk_log_start":"147'30","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'13","last_scrub_stamp":"2024-04-22T18:39:54.377980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.377980+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":30,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30260881099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1512663,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":68466,"num_write":12,"num_write_kb":3428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"163'88","reported_seq":410,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.744879+0000","last_change":"2024-04-22T18:41:41.008619+0000","last_active":"2024-04-22T18:41:50.744879+0000","last_peered":"2024-04-22T18:41:50.744879+0000","last_clean":"2024-04-22T18:41:50.744879+0000","last_became_active":"2024-04-22T18:41:41.008273+0000","last_became_peered":"2024-04-22T18:41:41.008273+0000","last_unstale":"2024-04-22T18:41:50.744879+0000","last_undegraded":"2024-04-22T18:41:50.744879+0000","last_fullsized":"2024-04-22T18:41:50.744879+0000","mapping_epoch":164,"log_start":"155'85","ondisk_log_start":"155'85","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":89,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2714397,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":146,"num_read_kb":138065,"num_write":47,"num_write_kb":12258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"147'53","reported_seq":92,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.764743+0000","last_change":"2024-04-22T18:41:11.253830+0000","last_active":"2024-04-22T18:41:54.764743+0000","last_peered":"2024-04-22T18:41:54.764743+0000","last_clean":"2024-04-22T18:41:54.764743+0000","last_became_active":"2024-04-22T18:41:11.253471+0000","last_became_peered":"2024-04-22T18:41:11.253471+0000","last_unstale":"2024-04-22T18:41:54.764743+0000","last_undegraded":"2024-04-22T18:41:54.764743+0000","last_fullsized":"2024-04-22T18:41:54.764743+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'23","last_scrub_stamp":"2024-04-22T18:40:20.587606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.587606+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10225366599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1840449,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":93,"num_read_kb":105860,"num_write":31,"num_write_kb":9939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5574713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"168'89","reported_seq":138,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.764717+0000","last_change":"2024-04-22T18:41:53.569917+0000","last_active":"2024-04-22T18:41:54.764717+0000","last_peered":"2024-04-22T18:41:54.764717+0000","last_clean":"2024-04-22T18:41:54.764717+0000","last_became_active":"2024-04-22T18:41:11.354958+0000","last_became_peered":"2024-04-22T18:41:11.354958+0000","last_unstale":"2024-04-22T18:41:54.764717+0000","last_undegraded":"2024-04-22T18:41:54.764717+0000","last_fullsized":"2024-04-22T18:41:54.764717+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"168'89","last_scrub_stamp":"2024-04-22T18:41:53.569764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:53.569764+0000","objects_scrubbed":1,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:55.086845+0000","scrub_duration":0.34939699600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243817,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"132'123","reported_seq":499,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.764680+0000","last_change":"2024-04-22T18:41:11.469596+0000","last_active":"2024-04-22T18:41:54.764680+0000","last_peered":"2024-04-22T18:41:54.764680+0000","last_clean":"2024-04-22T18:41:54.764680+0000","last_became_active":"2024-04-22T18:41:11.469282+0000","last_became_peered":"2024-04-22T18:41:11.469282+0000","last_unstale":"2024-04-22T18:41:54.764680+0000","last_undegraded":"2024-04-22T18:41:54.764680+0000","last_fullsized":"2024-04-22T18:41:54.764680+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"121'119","last_scrub_stamp":"2024-04-22T18:40:55.573199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.573199+0000","objects_scrubbed":4,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32976813900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4521133,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":219,"num_read_kb":257792,"num_write":64,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3786236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"168'159","reported_seq":638,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.864067+0000","last_change":"2024-04-22T18:41:53.557266+0000","last_active":"2024-04-22T18:41:54.864067+0000","last_peered":"2024-04-22T18:41:54.864067+0000","last_clean":"2024-04-22T18:41:54.864067+0000","last_became_active":"2024-04-22T18:41:29.932575+0000","last_became_peered":"2024-04-22T18:41:29.932575+0000","last_unstale":"2024-04-22T18:41:54.864067+0000","last_undegraded":"2024-04-22T18:41:54.864067+0000","last_fullsized":"2024-04-22T18:41:54.864067+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":5,"last_scrub":"168'159","last_scrub_stamp":"2024-04-22T18:41:53.557137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:53.557137+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:02.040915+0000","scrub_duration":0.101410045,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185042,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"163'115","reported_seq":460,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-04-22T18:41:56.070857+0000","last_change":"2024-04-22T18:41:56.070499+0000","last_active":"2024-04-22T18:41:56.070857+0000","last_peered":"2024-04-22T18:41:56.070857+0000","last_clean":"2024-04-22T18:41:56.070857+0000","last_became_active":"2024-04-22T18:41:56.070223+0000","last_became_peered":"2024-04-22T18:41:56.070223+0000","last_unstale":"2024-04-22T18:41:56.070857+0000","last_undegraded":"2024-04-22T18:41:56.070857+0000","last_fullsized":"2024-04-22T18:41:56.070857+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2806852,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":163,"num_read_kb":197186,"num_write":61,"num_write_kb":16155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":6836986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"168'120","reported_seq":576,"reported_epoch":175,"state":"activating+remapped","last_fresh":"2024-04-22T18:41:51.948728+0000","last_change":"2024-04-22T18:41:51.751130+0000","last_active":"2024-04-22T18:41:50.744903+0000","last_peered":"2024-04-22T18:41:43.719042+0000","last_clean":"2024-04-22T18:41:43.719042+0000","last_became_active":"2024-04-22T18:41:18.957008+0000","last_became_peered":"2024-04-22T18:41:18.957008+0000","last_unstale":"2024-04-22T18:41:51.948728+0000","last_undegraded":"2024-04-22T18:41:51.948728+0000","last_fullsized":"2024-04-22T18:41:51.948728+0000","mapping_epoch":174,"log_start":"84'88","ondisk_log_start":"84'88","created":110,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":32,"log_dups_size":106,"ondisk_log_size":32,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7033845,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":122,"num_read_kb":149003,"num_write":24,"num_write_kb":7118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9095252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"148'71","reported_seq":396,"reported_epoch":178,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:41:54.764544+0000","last_change":"2024-04-22T18:41:49.936581+0000","last_active":"2024-04-22T18:41:54.764544+0000","last_peered":"2024-04-22T18:41:54.764544+0000","last_clean":"2024-04-22T18:41:01.854833+0000","last_became_active":"2024-04-22T18:41:49.835532+0000","last_became_peered":"2024-04-22T18:41:49.835532+0000","last_unstale":"2024-04-22T18:41:54.764544+0000","last_undegraded":"2024-04-22T18:41:54.764544+0000","last_fullsized":"2024-04-22T18:41:54.764544+0000","mapping_epoch":172,"log_start":"79'52","ondisk_log_start":"79'52","created":110,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":61,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":857223,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":130,"num_read_kb":179919,"num_write":60,"num_write_kb":17002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9662508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"107'52","reported_seq":394,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.743162+0000","last_change":"2024-04-22T18:41:22.602374+0000","last_active":"2024-04-22T18:41:50.743162+0000","last_peered":"2024-04-22T18:41:50.743162+0000","last_clean":"2024-04-22T18:41:50.743162+0000","last_became_active":"2024-04-22T18:41:00.078669+0000","last_became_peered":"2024-04-22T18:41:00.078669+0000","last_unstale":"2024-04-22T18:41:50.743162+0000","last_undegraded":"2024-04-22T18:41:50.743162+0000","last_fullsized":"2024-04-22T18:41:50.743162+0000","mapping_epoch":125,"log_start":"83'49","ondisk_log_start":"83'49","created":110,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":5,"last_scrub":"107'52","last_scrub_stamp":"2024-04-22T18:41:22.602272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:22.602272+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:48.741276+0000","scrub_duration":0.40203006699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"166'100","reported_seq":505,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.763835+0000","last_change":"2024-04-22T18:41:45.814689+0000","last_active":"2024-04-22T18:41:54.763835+0000","last_peered":"2024-04-22T18:41:54.763835+0000","last_clean":"2024-04-22T18:41:54.763835+0000","last_became_active":"2024-04-22T18:41:30.012572+0000","last_became_peered":"2024-04-22T18:41:30.012572+0000","last_unstale":"2024-04-22T18:41:54.763835+0000","last_undegraded":"2024-04-22T18:41:54.763835+0000","last_fullsized":"2024-04-22T18:41:54.763835+0000","mapping_epoch":153,"log_start":"96'71","ondisk_log_start":"96'71","created":110,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":5,"last_scrub":"166'100","last_scrub_stamp":"2024-04-22T18:41:45.814522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:45.814522+0000","objects_scrubbed":1,"log_size":29,"log_dups_size":81,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:06.282120+0000","scrub_duration":0.40262363400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3227010,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"54'29","reported_seq":120,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.866602+0000","last_change":"2024-04-22T18:40:55.328642+0000","last_active":"2024-04-22T18:41:54.866602+0000","last_peered":"2024-04-22T18:41:54.866602+0000","last_clean":"2024-04-22T18:41:54.866602+0000","last_became_active":"2024-04-22T18:40:13.180144+0000","last_became_peered":"2024-04-22T18:40:13.180144+0000","last_unstale":"2024-04-22T18:41:54.866602+0000","last_undegraded":"2024-04-22T18:41:54.866602+0000","last_fullsized":"2024-04-22T18:41:54.866602+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'29","last_scrub_stamp":"2024-04-22T18:40:55.328529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.328529+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:10.967288+0000","scrub_duration":0.30217712000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"51'19","reported_seq":116,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.743037+0000","last_change":"2024-04-22T18:40:57.754351+0000","last_active":"2024-04-22T18:41:50.743037+0000","last_peered":"2024-04-22T18:41:50.743037+0000","last_clean":"2024-04-22T18:41:50.743037+0000","last_became_active":"2024-04-22T18:40:10.978840+0000","last_became_peered":"2024-04-22T18:40:10.978840+0000","last_unstale":"2024-04-22T18:41:50.743037+0000","last_undegraded":"2024-04-22T18:41:50.743037+0000","last_fullsized":"2024-04-22T18:41:50.743037+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"51'19","last_scrub_stamp":"2024-04-22T18:40:57.754261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:57.754261+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:17.927283+0000","scrub_duration":0.59184710600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"166'93","reported_seq":96,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.764661+0000","last_change":"2024-04-22T18:41:04.562945+0000","last_active":"2024-04-22T18:41:54.764661+0000","last_peered":"2024-04-22T18:41:54.764661+0000","last_clean":"2024-04-22T18:41:54.764661+0000","last_became_active":"2024-04-22T18:41:04.139159+0000","last_became_peered":"2024-04-22T18:41:04.139159+0000","last_unstale":"2024-04-22T18:41:54.764661+0000","last_undegraded":"2024-04-22T18:41:54.764661+0000","last_fullsized":"2024-04-22T18:41:54.764661+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'75","last_scrub_stamp":"2024-04-22T18:40:22.670296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.670296+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20951747300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218722,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":96410,"num_write":16,"num_write_kb":7324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12459808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"166'50","reported_seq":305,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.844855+0000","last_change":"2024-04-22T18:40:59.453830+0000","last_active":"2024-04-22T18:41:50.844855+0000","last_peered":"2024-04-22T18:41:50.844855+0000","last_clean":"2024-04-22T18:41:50.844855+0000","last_became_active":"2024-04-22T18:40:58.107873+0000","last_became_peered":"2024-04-22T18:40:58.107873+0000","last_unstale":"2024-04-22T18:41:50.844855+0000","last_undegraded":"2024-04-22T18:41:50.844855+0000","last_fullsized":"2024-04-22T18:41:50.844855+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3621975,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":115,"num_read_kb":106979,"num_write":32,"num_write_kb":8520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2641926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"169'153","reported_seq":510,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.945433+0000","last_change":"2024-04-22T18:40:26.844287+0000","last_active":"2024-04-22T18:41:50.945433+0000","last_peered":"2024-04-22T18:41:50.945433+0000","last_clean":"2024-04-22T18:41:50.945433+0000","last_became_active":"2024-04-22T18:40:26.742351+0000","last_became_peered":"2024-04-22T18:40:26.742351+0000","last_unstale":"2024-04-22T18:41:50.945433+0000","last_undegraded":"2024-04-22T18:41:50.945433+0000","last_fullsized":"2024-04-22T18:41:50.945433+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'113","last_scrub_stamp":"2024-04-22T18:40:20.291988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.291988+0000","objects_scrubbed":2,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10270805500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941291,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":63,"num_read_kb":42947,"num_write":48,"num_write_kb":13258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3940523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"31'5","reported_seq":209,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.843049+0000","last_change":"2024-04-22T18:41:41.832636+0000","last_active":"2024-04-22T18:41:50.843049+0000","last_peered":"2024-04-22T18:41:50.843049+0000","last_clean":"2024-04-22T18:41:50.843049+0000","last_became_active":"2024-04-22T18:39:59.265189+0000","last_became_peered":"2024-04-22T18:39:59.265189+0000","last_unstale":"2024-04-22T18:41:50.843049+0000","last_undegraded":"2024-04-22T18:41:50.843049+0000","last_fullsized":"2024-04-22T18:41:50.843049+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":5,"last_scrub":"31'5","last_scrub_stamp":"2024-04-22T18:40:10.212815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:10.212815+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015068670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":66,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.866609+0000","last_change":"2024-04-22T18:40:58.207745+0000","last_active":"2024-04-22T18:41:54.866609+0000","last_peered":"2024-04-22T18:41:54.866609+0000","last_clean":"2024-04-22T18:41:54.866609+0000","last_became_active":"2024-04-22T18:40:58.207451+0000","last_became_peered":"2024-04-22T18:40:58.207451+0000","last_unstale":"2024-04-22T18:41:54.866609+0000","last_undegraded":"2024-04-22T18:41:54.866609+0000","last_fullsized":"2024-04-22T18:41:54.866609+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:19.136933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:19.136933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"169'195","reported_seq":802,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-04-22T18:41:55.968213+0000","last_change":"2024-04-22T18:41:44.821952+0000","last_active":"2024-04-22T18:41:55.968213+0000","last_peered":"2024-04-22T18:41:55.968213+0000","last_clean":"2024-04-22T18:41:55.968213+0000","last_became_active":"2024-04-22T18:41:44.821076+0000","last_became_peered":"2024-04-22T18:41:44.821076+0000","last_unstale":"2024-04-22T18:41:55.968213+0000","last_undegraded":"2024-04-22T18:41:55.968213+0000","last_fullsized":"2024-04-22T18:41:55.968213+0000","mapping_epoch":168,"log_start":"147'185","ondisk_log_start":"147'185","created":20,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":10,"log_dups_size":161,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8134736,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":208,"num_read_kb":247495,"num_write":48,"num_write_kb":13433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":31242513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":186,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.744876+0000","last_change":"2024-04-22T18:41:33.768149+0000","last_active":"2024-04-22T18:41:50.744876+0000","last_peered":"2024-04-22T18:41:50.744876+0000","last_clean":"2024-04-22T18:41:50.744876+0000","last_became_active":"2024-04-22T18:41:04.037248+0000","last_became_peered":"2024-04-22T18:41:04.037248+0000","last_unstale":"2024-04-22T18:41:50.744876+0000","last_undegraded":"2024-04-22T18:41:50.744876+0000","last_fullsized":"2024-04-22T18:41:50.744876+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"162'34","reported_seq":317,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.943216+0000","last_change":"2024-04-22T18:41:45.590948+0000","last_active":"2024-04-22T18:41:50.943216+0000","last_peered":"2024-04-22T18:41:50.943216+0000","last_clean":"2024-04-22T18:41:50.943216+0000","last_became_active":"2024-04-22T18:41:41.006293+0000","last_became_peered":"2024-04-22T18:41:41.006293+0000","last_unstale":"2024-04-22T18:41:50.943216+0000","last_undegraded":"2024-04-22T18:41:50.943216+0000","last_fullsized":"2024-04-22T18:41:50.943216+0000","mapping_epoch":20,"log_start":"147'30","ondisk_log_start":"147'30","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'34","last_scrub_stamp":"2024-04-22T18:41:45.590834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:45.590834+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":30,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:48.626740+0000","scrub_duration":0.40273239100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025326,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":206,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.766594+0000","last_change":"2024-04-22T18:41:43.814617+0000","last_active":"2024-04-22T18:41:54.766594+0000","last_peered":"2024-04-22T18:41:54.766594+0000","last_clean":"2024-04-22T18:41:54.766594+0000","last_became_active":"2024-04-22T18:41:26.961835+0000","last_became_peered":"2024-04-22T18:41:26.961835+0000","last_unstale":"2024-04-22T18:41:54.766594+0000","last_undegraded":"2024-04-22T18:41:54.766594+0000","last_fullsized":"2024-04-22T18:41:54.766594+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:09.538542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:09.538542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.35201753200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"167'92","reported_seq":416,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-04-22T18:41:56.587252+0000","last_change":"2024-04-22T18:41:56.587360+0000","last_active":"2024-04-22T18:41:56.587252+0000","last_peered":"2024-04-22T18:41:56.587252+0000","last_clean":"2024-04-22T18:41:56.587252+0000","last_became_active":"2024-04-22T18:41:42.764184+0000","last_became_peered":"2024-04-22T18:41:42.764184+0000","last_unstale":"2024-04-22T18:41:56.587252+0000","last_undegraded":"2024-04-22T18:41:56.587252+0000","last_fullsized":"2024-04-22T18:41:56.587252+0000","mapping_epoch":166,"log_start":"137'90","ondisk_log_start":"137'90","created":20,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'92","last_scrub_stamp":"2024-04-22T18:41:56.587238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:56.587238+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":61,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:09.467131+0000","scrub_duration":0.101671527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":184,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.744780+0000","last_change":"2024-04-22T18:41:26.854653+0000","last_active":"2024-04-22T18:41:50.744780+0000","last_peered":"2024-04-22T18:41:50.744780+0000","last_clean":"2024-04-22T18:41:50.744780+0000","last_became_active":"2024-04-22T18:41:26.854399+0000","last_became_peered":"2024-04-22T18:41:26.854399+0000","last_unstale":"2024-04-22T18:41:50.744780+0000","last_undegraded":"2024-04-22T18:41:50.744780+0000","last_fullsized":"2024-04-22T18:41:50.744780+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"149'42","reported_seq":358,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.844952+0000","last_change":"2024-04-22T18:41:40.807780+0000","last_active":"2024-04-22T18:41:50.844952+0000","last_peered":"2024-04-22T18:41:50.844952+0000","last_clean":"2024-04-22T18:41:50.844952+0000","last_became_active":"2024-04-22T18:41:40.807410+0000","last_became_peered":"2024-04-22T18:41:40.807410+0000","last_unstale":"2024-04-22T18:41:50.844952+0000","last_undegraded":"2024-04-22T18:41:50.844952+0000","last_fullsized":"2024-04-22T18:41:50.844952+0000","mapping_epoch":129,"log_start":"64'38","ondisk_log_start":"64'38","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":25,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":88762,"num_write":12,"num_write_kb":3449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5687418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"169'384","reported_seq":1553,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.764753+0000","last_change":"2024-04-22T18:41:43.266638+0000","last_active":"2024-04-22T18:41:54.764753+0000","last_peered":"2024-04-22T18:41:54.764753+0000","last_clean":"2024-04-22T18:41:54.764753+0000","last_became_active":"2024-04-22T18:41:26.994048+0000","last_became_peered":"2024-04-22T18:41:26.994048+0000","last_unstale":"2024-04-22T18:41:54.764753+0000","last_undegraded":"2024-04-22T18:41:54.764753+0000","last_fullsized":"2024-04-22T18:41:54.764753+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'293","last_scrub_stamp":"2024-04-22T18:40:53.582354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:53.582354+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:14.875876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41123046,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":325,"num_read_kb":4195,"num_write":477,"num_write_kb":113967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":82068207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":145,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.844945+0000","last_change":"2024-04-22T18:41:26.657445+0000","last_active":"2024-04-22T18:41:50.844945+0000","last_peered":"2024-04-22T18:41:50.844945+0000","last_clean":"2024-04-22T18:41:50.844945+0000","last_became_active":"2024-04-22T18:41:26.657206+0000","last_became_peered":"2024-04-22T18:41:26.657206+0000","last_unstale":"2024-04-22T18:41:50.844945+0000","last_undegraded":"2024-04-22T18:41:50.844945+0000","last_fullsized":"2024-04-22T18:41:50.844945+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:55.558446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.558446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:00.817184+0000","scrub_duration":0.56126016300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"167'36","reported_seq":312,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.744779+0000","last_change":"2024-04-22T18:40:21.426510+0000","last_active":"2024-04-22T18:41:50.744779+0000","last_peered":"2024-04-22T18:41:50.744779+0000","last_clean":"2024-04-22T18:41:50.744779+0000","last_became_active":"2024-04-22T18:39:51.943071+0000","last_became_peered":"2024-04-22T18:39:51.943071+0000","last_unstale":"2024-04-22T18:41:50.744779+0000","last_undegraded":"2024-04-22T18:41:50.744779+0000","last_fullsized":"2024-04-22T18:41:50.744779+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-04-22T18:40:21.426350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.426350+0000","objects_scrubbed":1,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30334366400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294238,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":109,"num_read_kb":123704,"num_write":9,"num_write_kb":2718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"169'309","reported_seq":743,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.744602+0000","last_change":"2024-04-22T18:40:20.465005+0000","last_active":"2024-04-22T18:41:50.744602+0000","last_peered":"2024-04-22T18:41:50.744602+0000","last_clean":"2024-04-22T18:41:50.744602+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:41:50.744602+0000","last_undegraded":"2024-04-22T18:41:50.744602+0000","last_fullsized":"2024-04-22T18:41:50.744602+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'176","last_scrub_stamp":"2024-04-22T18:40:20.464799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.464799+0000","objects_scrubbed":31,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30629975799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382624,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":306,"num_read_kb":4987,"num_write":404,"num_write_kb":80545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":147,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-04-22T18:41:55.833438+0000","last_change":"2024-04-22T18:41:41.421087+0000","last_active":"2024-04-22T18:41:55.833438+0000","last_peered":"2024-04-22T18:41:55.833438+0000","last_clean":"2024-04-22T18:41:55.833438+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:41:55.833438+0000","last_undegraded":"2024-04-22T18:41:55.833438+0000","last_fullsized":"2024-04-22T18:41:55.833438+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:41.420996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:41.420996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:57.844851+0000","scrub_duration":0.20141637900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"162'72","reported_seq":436,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.744945+0000","last_change":"2024-04-22T18:41:32.860808+0000","last_active":"2024-04-22T18:41:50.744945+0000","last_peered":"2024-04-22T18:41:50.744945+0000","last_clean":"2024-04-22T18:41:50.744945+0000","last_became_active":"2024-04-22T18:41:32.860455+0000","last_became_peered":"2024-04-22T18:41:32.860455+0000","last_unstale":"2024-04-22T18:41:50.744945+0000","last_undegraded":"2024-04-22T18:41:50.744945+0000","last_fullsized":"2024-04-22T18:41:50.744945+0000","mapping_epoch":156,"log_start":"83'49","ondisk_log_start":"83'49","created":20,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":49,"ondisk_log_size":23,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3008494,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":95,"num_read_kb":112837,"num_write":38,"num_write_kb":11286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4519285,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"168'337","reported_seq":1378,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-04-22T18:41:56.400527+0000","last_change":"2024-04-22T18:41:41.108768+0000","last_active":"2024-04-22T18:41:56.400527+0000","last_peered":"2024-04-22T18:41:56.400527+0000","last_clean":"2024-04-22T18:41:56.400527+0000","last_became_active":"2024-04-22T18:41:41.007340+0000","last_became_peered":"2024-04-22T18:41:41.007340+0000","last_unstale":"2024-04-22T18:41:56.400527+0000","last_undegraded":"2024-04-22T18:41:56.400527+0000","last_fullsized":"2024-04-22T18:41:56.400527+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'209","last_scrub_stamp":"2024-04-22T18:40:18.683442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:18.683442+0000","objects_scrubbed":24,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204344365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39035424,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":315,"num_read_kb":9133,"num_write":423,"num_write_kb":104979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":67954083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":181,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.766653+0000","last_change":"2024-04-22T18:40:22.521828+0000","last_active":"2024-04-22T18:41:54.766653+0000","last_peered":"2024-04-22T18:41:54.766653+0000","last_clean":"2024-04-22T18:41:54.766653+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:41:54.766653+0000","last_undegraded":"2024-04-22T18:41:54.766653+0000","last_fullsized":"2024-04-22T18:41:54.766653+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.521638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30190717299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"169'111","reported_seq":507,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.980019+0000","last_change":"2024-04-22T18:41:54.980019+0000","last_active":"2024-04-22T18:41:54.980019+0000","last_peered":"2024-04-22T18:41:54.980019+0000","last_clean":"2024-04-22T18:41:54.980019+0000","last_became_active":"2024-04-22T18:41:31.858774+0000","last_became_peered":"2024-04-22T18:41:31.858774+0000","last_unstale":"2024-04-22T18:41:54.980019+0000","last_undegraded":"2024-04-22T18:41:54.980019+0000","last_fullsized":"2024-04-22T18:41:54.980019+0000","mapping_epoch":155,"log_start":"96'71","ondisk_log_start":"96'71","created":20,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'111","last_scrub_stamp":"2024-04-22T18:41:54.979825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:54.979825+0000","objects_scrubbed":2,"log_size":40,"log_dups_size":81,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:06.600955+0000","scrub_duration":0.39464260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5664091,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"169'328","reported_seq":1052,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.765275+0000","last_change":"2024-04-22T18:41:19.866530+0000","last_active":"2024-04-22T18:41:54.765275+0000","last_peered":"2024-04-22T18:41:54.765275+0000","last_clean":"2024-04-22T18:41:54.765275+0000","last_became_active":"2024-04-22T18:41:19.765392+0000","last_became_peered":"2024-04-22T18:41:19.765392+0000","last_unstale":"2024-04-22T18:41:54.765275+0000","last_undegraded":"2024-04-22T18:41:54.765275+0000","last_fullsized":"2024-04-22T18:41:54.765275+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'212","last_scrub_stamp":"2024-04-22T18:40:21.870855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.870855+0000","objects_scrubbed":46,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43117026,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":334,"num_read_kb":12449,"num_write":418,"num_write_kb":91859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":43826500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":201,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.968108+0000","last_change":"2024-04-22T18:41:30.504800+0000","last_active":"2024-04-22T18:41:54.968108+0000","last_peered":"2024-04-22T18:41:54.968108+0000","last_clean":"2024-04-22T18:41:54.968108+0000","last_became_active":"2024-04-22T18:40:02.741862+0000","last_became_peered":"2024-04-22T18:40:02.741862+0000","last_unstale":"2024-04-22T18:41:54.968108+0000","last_undegraded":"2024-04-22T18:41:54.968108+0000","last_fullsized":"2024-04-22T18:41:54.968108+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:30.504686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:30.504686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:57.436870+0000","scrub_duration":0.30171718199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":387,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-04-22T18:41:56.270666+0000","last_change":"2024-04-22T18:41:56.270751+0000","last_active":"2024-04-22T18:41:56.270666+0000","last_peered":"2024-04-22T18:41:56.270666+0000","last_clean":"2024-04-22T18:41:56.270666+0000","last_became_active":"2024-04-22T18:41:26.757958+0000","last_became_peered":"2024-04-22T18:41:26.757958+0000","last_unstale":"2024-04-22T18:41:56.270666+0000","last_undegraded":"2024-04-22T18:41:56.270666+0000","last_fullsized":"2024-04-22T18:41:56.270666+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:41:56.270650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:41:56.270650+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:12.138524+0000","scrub_duration":0.202387964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"137'75","reported_seq":105,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:55.390323+0000","last_change":"2024-04-22T18:41:04.681901+0000","last_active":"2024-04-22T18:41:55.390323+0000","last_peered":"2024-04-22T18:41:55.390323+0000","last_clean":"2024-04-22T18:41:55.390323+0000","last_became_active":"2024-04-22T18:41:04.227104+0000","last_became_peered":"2024-04-22T18:41:04.227104+0000","last_unstale":"2024-04-22T18:41:55.390323+0000","last_undegraded":"2024-04-22T18:41:55.390323+0000","last_fullsized":"2024-04-22T18:41:55.390323+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'38","last_scrub_stamp":"2024-04-22T18:39:55.746067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:55.746067+0000","objects_scrubbed":1,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60877772299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6168268,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":192624,"num_write":57,"num_write_kb":16481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15782313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"164'59","reported_seq":387,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.766512+0000","last_change":"2024-04-22T18:41:35.835744+0000","last_active":"2024-04-22T18:41:54.766512+0000","last_peered":"2024-04-22T18:41:54.766512+0000","last_clean":"2024-04-22T18:41:54.766512+0000","last_became_active":"2024-04-22T18:41:35.834928+0000","last_became_peered":"2024-04-22T18:41:35.834928+0000","last_unstale":"2024-04-22T18:41:54.766512+0000","last_undegraded":"2024-04-22T18:41:54.766512+0000","last_fullsized":"2024-04-22T18:41:54.766512+0000","mapping_epoch":159,"log_start":"79'52","ondisk_log_start":"79'52","created":20,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'57","last_scrub_stamp":"2024-04-22T18:40:56.376497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:56.376497+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":61,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:41:57.469237+0000","scrub_duration":0.30191299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3832029,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":46990,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1913105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"167'154","reported_seq":627,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.764705+0000","last_change":"2024-04-22T18:41:17.760985+0000","last_active":"2024-04-22T18:41:54.764705+0000","last_peered":"2024-04-22T18:41:54.764705+0000","last_clean":"2024-04-22T18:41:54.764705+0000","last_became_active":"2024-04-22T18:41:17.759812+0000","last_became_peered":"2024-04-22T18:41:17.759812+0000","last_unstale":"2024-04-22T18:41:54.764705+0000","last_undegraded":"2024-04-22T18:41:54.764705+0000","last_fullsized":"2024-04-22T18:41:54.764705+0000","mapping_epoch":141,"log_start":"84'88","ondisk_log_start":"84'88","created":20,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":66,"log_dups_size":106,"ondisk_log_size":66,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7420472,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":148630,"num_write":63,"num_write_kb":21335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12038689,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"167'121","reported_seq":462,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.844861+0000","last_change":"2024-04-22T18:41:11.054130+0000","last_active":"2024-04-22T18:41:50.844861+0000","last_peered":"2024-04-22T18:41:50.844861+0000","last_clean":"2024-04-22T18:41:50.844861+0000","last_became_active":"2024-04-22T18:41:11.053760+0000","last_became_peered":"2024-04-22T18:41:11.053760+0000","last_unstale":"2024-04-22T18:41:50.844861+0000","last_undegraded":"2024-04-22T18:41:50.844861+0000","last_fullsized":"2024-04-22T18:41:50.844861+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7059022,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":182,"num_read_kb":210386,"num_write":65,"num_write_kb":17255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4554596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"167'153","reported_seq":627,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:55.669111+0000","last_change":"2024-04-22T18:41:55.669111+0000","last_active":"2024-04-22T18:41:55.669111+0000","last_peered":"2024-04-22T18:41:55.669111+0000","last_clean":"2024-04-22T18:41:55.669111+0000","last_became_active":"2024-04-22T18:41:31.959307+0000","last_became_peered":"2024-04-22T18:41:31.959307+0000","last_unstale":"2024-04-22T18:41:55.669111+0000","last_undegraded":"2024-04-22T18:41:55.669111+0000","last_fullsized":"2024-04-22T18:41:55.669111+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'153","last_scrub_stamp":"2024-04-22T18:41:55.669014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:55.669014+0000","objects_scrubbed":1,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:18.053733+0000","scrub_duration":0.201688442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3193396,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"168'131","reported_seq":546,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:55.185846+0000","last_change":"2024-04-22T18:41:12.783644+0000","last_active":"2024-04-22T18:41:55.185846+0000","last_peered":"2024-04-22T18:41:55.185846+0000","last_clean":"2024-04-22T18:41:55.185846+0000","last_became_active":"2024-04-22T18:41:11.956312+0000","last_became_peered":"2024-04-22T18:41:11.956312+0000","last_unstale":"2024-04-22T18:41:55.185846+0000","last_undegraded":"2024-04-22T18:41:55.185846+0000","last_fullsized":"2024-04-22T18:41:55.185846+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'119","last_scrub_stamp":"2024-04-22T18:40:55.573199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.573199+0000","objects_scrubbed":4,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32976813900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832111,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":304,"num_read_kb":395814,"num_write":74,"num_write_kb":20641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9347897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"133'66","reported_seq":87,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.764806+0000","last_change":"2024-04-22T18:41:19.456295+0000","last_active":"2024-04-22T18:41:54.764806+0000","last_peered":"2024-04-22T18:41:54.764806+0000","last_clean":"2024-04-22T18:41:54.764806+0000","last_became_active":"2024-04-22T18:41:11.154301+0000","last_became_peered":"2024-04-22T18:41:11.154301+0000","last_unstale":"2024-04-22T18:41:54.764806+0000","last_undegraded":"2024-04-22T18:41:54.764806+0000","last_fullsized":"2024-04-22T18:41:54.764806+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'66","last_scrub_stamp":"2024-04-22T18:41:19.456160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:19.456160+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:36.925204+0000","scrub_duration":0.20146846400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"145'51","reported_seq":98,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-22T18:41:54.864916+0000","last_change":"2024-04-22T18:41:45.675260+0000","last_active":"2024-04-22T18:41:54.864916+0000","last_peered":"2024-04-22T18:41:54.864916+0000","last_clean":"2024-04-22T18:41:54.864916+0000","last_became_active":"2024-04-22T18:41:11.253409+0000","last_became_peered":"2024-04-22T18:41:11.253409+0000","last_unstale":"2024-04-22T18:41:54.864916+0000","last_undegraded":"2024-04-22T18:41:54.864916+0000","last_fullsized":"2024-04-22T18:41:54.864916+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'51","last_scrub_stamp":"2024-04-22T18:41:45.675122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:45.675122+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:09.124003+0000","scrub_duration":0.40271887499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"163'88","reported_seq":410,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-04-22T18:41:50.744879+0000","last_change":"2024-04-22T18:41:41.109970+0000","last_active":"2024-04-22T18:41:50.744879+0000","last_peered":"2024-04-22T18:41:50.744879+0000","last_clean":"2024-04-22T18:41:50.744879+0000","last_became_active":"2024-04-22T18:41:41.109681+0000","last_became_peered":"2024-04-22T18:41:41.109681+0000","last_unstale":"2024-04-22T18:41:50.744879+0000","last_undegraded":"2024-04-22T18:41:50.744879+0000","last_fullsized":"2024-04-22T18:41:50.744879+0000","mapping_epoch":20,"log_start":"155'85","ondisk_log_start":"155'85","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":89,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2714396,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":152,"num_read_kb":145684,"num_write":47,"num_write_kb":12257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":36,"stat_sum":{"num_bytes":115768836,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":2865,"num_read_kb":3431734,"num_write":875,"num_write_kb":250123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":172300212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":26,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":187736064,"data_stored":187891771,"data_compressed":1037892,"data_compressed_allocated":1122304,"data_compressed_original":2198706,"omap_allocated":0,"internal_metadata":0},"log_size":1780,"ondisk_log_size":1780,"up":72,"acting":72,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":155658120,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":1280,"num_read_kb":30764,"num_write":1722,"num_write_kb":391350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1149,"num_bytes_recovered":214708114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249724928,"data_stored":264136596,"data_compressed":14626013,"data_compressed_allocated":15568896,"data_compressed_original":30834688,"omap_allocated":0,"internal_metadata":0},"log_size":1358,"ondisk_log_size":1358,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":2115712,"data_compressed":122550,"data_compressed_allocated":163840,"data_compressed_original":2084864,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52632,"kb_used_data":21512,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94319208,"statfs":{"total":96636764160,"available":96582868992,"internally_reserved":0,"allocated":22028288,"data_stored":21559336,"data_compressed":211962,"data_compressed_allocated":241664,"data_compressed_original":970752,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":2.5870000000000002,"15min":1.1439999999999999},"min":{"1min":0.182,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":26.568000000000001,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":6.3600000000000003,"5min":6.766,"15min":5.2599999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":95.918999999999997,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.35499999999999998}]},{"osd":1,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":3.8050000000000002,"5min":3.984,"15min":4.5709999999999997},"min":{"1min":0.29299999999999998,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":66.814999999999998,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":6.6040000000000001,"5min":12.141,"15min":12.744999999999999},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":100.242,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.58999999999999997}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7130000000000001,"5min":4.0730000000000004,"15min":4.0810000000000004},"min":{"1min":0.27400000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":97.662000000000006,"5min":97.662000000000006,"15min":97.662000000000006},"last":0.378},{"interface":"front","average":{"1min":0.497,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7560000000000002,"5min":4.1289999999999996,"15min":4.1429999999999998},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":97.686999999999998,"5min":97.686999999999998,"15min":97.686999999999998},"last":0.505},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.48399999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":8.2089999999999996,"5min":5.4509999999999996,"15min":4.7889999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":78.149000000000001,"5min":85.509,"15min":85.509},"last":0.55400000000000005},{"interface":"front","average":{"1min":7.7130000000000001,"5min":16.856000000000002,"15min":20.866},"min":{"1min":0.26000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":97.590000000000003,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.33700000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":2.0270000000000001,"15min":2.7229999999999999},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.4730000000000001,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.6679999999999999,"5min":5.2939999999999996,"15min":6.5389999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":23.007999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.60899999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":11.756,"15min":12.769},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":23.027000000000001,"5min":139.428,"15min":139.428},"last":0.44},{"interface":"front","average":{"1min":7.7800000000000002,"5min":4.3070000000000004,"15min":4.1909999999999998},"min":{"1min":0.29199999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":97.635999999999996,"5min":97.635999999999996,"15min":97.635999999999996},"last":0.53900000000000003}]}]},{"osd":6,"up_from":0,"seq":55834574890,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574890,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":160264,"kb_used_data":125052,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94211576,"statfs":{"total":96636764160,"available":96472653824,"internally_reserved":0,"allocated":128053248,"data_stored":131635188,"data_compressed":4963321,"data_compressed_allocated":5300224,"data_compressed_original":10430041,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":13.115,"15min":15.470000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":38.899999999999999,"5min":252.96199999999999,"15min":252.96199999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":9.282,"15min":8.8059999999999992},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63300000000000001,"5min":187.148,"15min":187.148},"last":0.46999999999999997}]},{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":2.508,"15min":1.115},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":9.141,"5min":96.486999999999995,"15min":96.486999999999995},"last":0.42599999999999999},{"interface":"front","average":{"1min":22.393000000000001,"5min":11.545999999999999,"15min":7.4390000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":310.61500000000001,"5min":310.61500000000001,"15min":310.61500000000001},"last":0.35699999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":8.8989999999999991,"15min":8.6790000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.8330000000000002,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.501},{"interface":"front","average":{"1min":0.995,"5min":2.4199999999999999,"15min":1.1040000000000001},"min":{"1min":0.31900000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.1399999999999997,"5min":96.471999999999994,"15min":96.471999999999994},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":11.172000000000001,"5min":8.9710000000000001,"15min":5.6710000000000003},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":122.102,"5min":252.91900000000001,"15min":252.91900000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":17.242000000000001,"15min":19.725999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59899999999999998,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.52200000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":2.3380000000000001,"15min":1.034},"min":{"1min":0.25800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":7.1559999999999997,"5min":96.408000000000001,"15min":96.408000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":2.6960000000000002,"5min":3.048,"15min":3.7080000000000002},"min":{"1min":0.36199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":38.581000000000003,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.48499999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":5.8289999999999997,"5min":14.667,"15min":13.662000000000001},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":87.706999999999994,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.83199999999999996,"5min":2.3820000000000001,"15min":1.0880000000000001},"min":{"1min":0.35699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.1159999999999997,"5min":96.438000000000002,"15min":96.438000000000002},"last":0.33300000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":9.5749999999999993,"15min":8.8979999999999997},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68000000000000005,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.44},{"interface":"front","average":{"1min":1.9790000000000001,"5min":2.6419999999999999,"15min":1.202},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":19.518999999999998,"5min":96.456999999999994,"15min":96.456999999999994},"last":0.377}]}]},{"osd":3,"up_from":13,"seq":55834574889,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76496,"kb_used_data":41276,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94295344,"statfs":{"total":96636764160,"available":96558432256,"internally_reserved":0,"allocated":42266624,"data_stored":42573074,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":15.041,"5min":10.538,"15min":11.646000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":93.131,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.498},{"interface":"front","average":{"1min":7.077,"5min":1.6659999999999999,"15min":0.77400000000000002},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":87.305999999999997,"5min":87.305999999999997,"15min":87.305999999999997},"last":0.26600000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":10.558,"15min":6.8819999999999997},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":78.805000000000007,"5min":197.02099999999999,"15min":197.02099999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":8.9800000000000004,"5min":2.3210000000000002,"15min":0.997},"min":{"1min":0.20999999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":78.393000000000001,"5min":78.393000000000001,"15min":78.393000000000001},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":2.3439999999999999,"15min":1.452},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":47.534999999999997,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.42199999999999999},{"interface":"front","average":{"1min":6.4770000000000003,"5min":1.728,"15min":0.85899999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":78.421999999999997,"5min":78.421999999999997,"15min":78.421999999999997},"last":0.379}]},{"osd":4,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1800000000000002,"5min":2.9940000000000002,"15min":3.7360000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":26.141999999999999,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":4.5629999999999997,"5min":4.6289999999999996,"15min":4.3010000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":47.616999999999997,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.8279999999999998,"5min":2.5049999999999999,"15min":1.623},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":47.566000000000003,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.54700000000000004},{"interface":"front","average":{"1min":6.6379999999999999,"5min":2.2890000000000001,"15min":1.601},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.480999999999995,"5min":78.480999999999995,"15min":78.480999999999995},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.50900000000000001,"15min":0.53100000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.748,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.52400000000000002},{"interface":"front","average":{"1min":2.8500000000000001,"5min":2.871,"15min":1.3109999999999999},"min":{"1min":0.377,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":47.590000000000003,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.46000000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":6.6189999999999998,"5min":1.903,"15min":1.0229999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.453999999999994,"5min":78.453999999999994,"15min":78.453999999999994},"last":0.629},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69299999999999995,"5min":0.995,"15min":0.995},"last":0.65900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58340,"kb_used_data":27220,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94313500,"statfs":{"total":96636764160,"available":96577024000,"internally_reserved":0,"allocated":27873280,"data_stored":28259018,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":6.7439999999999998,"15min":7.6660000000000004},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":84.338999999999999,"5min":149.03,"15min":149.03},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.28299999999999997,"5min":1.3089999999999999,"15min":1.7589999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.62,"5min":31.866,"15min":31.866},"last":0.34899999999999998}]},{"osd":1,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":9.4849999999999994,"5min":5.2489999999999997,"15min":2.4380000000000002},"min":{"1min":0.216,"5min":0.193,"15min":0.193},"max":{"1min":91.912000000000006,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.32800000000000001},{"interface":"front","average":{"1min":12.752000000000001,"5min":15.912000000000001,"15min":15.227},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":126.42,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.312}]},{"osd":3,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":6.2830000000000004,"15min":2.3359999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":129.041,"5min":147.721,"15min":147.721},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.29499999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":8.1280000000000001,"5min":13.57,"15min":17.038},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":90.882999999999996,"5min":160.584,"15min":160.584},"last":0.48799999999999999},{"interface":"front","average":{"1min":5.9349999999999996,"5min":4.8520000000000003,"15min":3.2490000000000001},"min":{"1min":0.32500000000000001,"5min":0.315,"15min":0.315},"max":{"1min":93.962999999999994,"5min":97.887,"15min":97.887},"last":0.42399999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":2.0339999999999998,"15min":2.7120000000000002},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.879,"5min":48.631,"15min":48.631},"last":0.434},{"interface":"front","average":{"1min":0.50900000000000001,"5min":2.2890000000000001,"15min":3.0800000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.91800000000000004,"5min":56.18,"15min":56.18},"last":0.39300000000000002}]},{"osd":6,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.3540000000000001,"5min":3.4670000000000001,"15min":1.5129999999999999},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":92.028000000000006,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.53300000000000003}]},{"osd":7,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.29,"5min":1.5800000000000001,"15min":0.91000000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":90.944999999999993,"5min":90.944999999999993,"15min":90.944999999999993},"last":0.52200000000000002},{"interface":"front","average":{"1min":7.4640000000000004,"5min":3.923,"15min":1.6970000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":92.048000000000002,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.40500000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":99152,"kb_used_data":63932,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94272688,"statfs":{"total":96636764160,"available":96535232512,"internally_reserved":0,"allocated":65466368,"data_stored":65888552,"data_compressed":1653944,"data_compressed_allocated":1757184,"data_compressed_original":3606105,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":5.883,"15min":7.2000000000000002},"min":{"1min":0.20000000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":27.116,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.374},{"interface":"front","average":{"1min":5.8040000000000003,"5min":6.6870000000000003,"15min":7.4710000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":94.046000000000006,"5min":94.046000000000006,"15min":94.046000000000006},"last":0.40500000000000003}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.679000000000002,"5min":13.159000000000001,"15min":10.119},"min":{"1min":0.23000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":406.68799999999999,"5min":406.68799999999999,"15min":406.68799999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":1.8700000000000001,"5min":0.61899999999999999,"15min":0.41099999999999998},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19},"max":{"1min":27.094999999999999,"5min":27.094999999999999,"15min":27.094999999999999},"last":0.308}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":15.968,"5min":3.464,"15min":1.3759999999999999},"min":{"1min":0.29099999999999998,"5min":0.216,"15min":0.216},"max":{"1min":228.90100000000001,"5min":228.90100000000001,"15min":228.90100000000001},"last":0.436},{"interface":"front","average":{"1min":21.481999999999999,"5min":13.061999999999999,"15min":14.015000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":196.82900000000001,"5min":231.864,"15min":231.864},"last":0.32800000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.823,"5min":17.213000000000001,"15min":14.393000000000001},"min":{"1min":0.316,"5min":0.309,"15min":0.309},"max":{"1min":406.71199999999999,"5min":406.71199999999999,"15min":406.71199999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":25.027000000000001,"5min":5.3979999999999997,"15min":2.1320000000000001},"min":{"1min":0.34499999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":186.89500000000001,"5min":186.89500000000001,"15min":186.89500000000001},"last":0.53000000000000003}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":9.7430000000000003,"15min":12.628},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":94.183000000000007,"5min":138.089,"15min":138.089},"last":0.499},{"interface":"front","average":{"1min":39.863999999999997,"5min":20.117000000000001,"15min":20.09},"min":{"1min":0.35499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":406.72899999999998,"5min":406.72899999999998,"15min":406.72899999999998},"last":0.55500000000000005}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":6.0010000000000003,"5min":3.8359999999999999,"15min":3.1299999999999999},"min":{"1min":0.34000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":94.302999999999997,"5min":94.302999999999997,"15min":94.302999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":43.036000000000001,"5min":19.713999999999999,"15min":17.591999999999999},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":427.69299999999998,"5min":427.69299999999998,"15min":427.69299999999998},"last":0.45700000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":2.0720000000000001,"5min":0.82799999999999996,"15min":0.626},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":27.253,"5min":27.253,"15min":27.253},"last":0.59799999999999998},{"interface":"front","average":{"1min":6.0090000000000003,"5min":3.129,"15min":3.069},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":94.260999999999996,"5min":94.260999999999996,"15min":94.260999999999996},"last":0.54500000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116320,"kb_used_data":81108,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94255520,"statfs":{"total":96636764160,"available":96517652480,"internally_reserved":0,"allocated":83054592,"data_stored":85260128,"data_compressed":3073787,"data_compressed_allocated":3297280,"data_compressed_original":6955008,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":9.7919999999999998,"5min":9.5660000000000007,"15min":7.6950000000000003},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":127.30500000000001,"5min":133.09899999999999,"15min":133.09899999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":27.553999999999998,"5min":11.506,"15min":4.8399999999999999},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":451.50799999999998,"5min":451.50799999999998,"15min":451.50799999999998},"last":0.378}]},{"osd":2,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":1.5309999999999999,"15min":1.5469999999999999},"min":{"1min":0.217,"5min":0.185,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":39.906999999999996,"15min":39.906999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.32600000000000001,"5min":2.2090000000000001,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.41299999999999998,"5min":94.820999999999998,"15min":94.820999999999998},"last":0.36099999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":6.1859999999999999,"5min":3.5299999999999998,"15min":1.486},"min":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":74.963999999999999,"5min":99.332999999999998,"15min":99.332999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":16.207000000000001,"5min":9.2159999999999993,"15min":4.1139999999999999},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":273.65100000000001,"5min":297.82100000000003,"15min":297.82100000000003},"last":0.438}]},{"osd":4,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":5.9980000000000002,"5min":3.4809999999999999,"15min":1.4970000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":96.528999999999996,"5min":96.528999999999996,"15min":96.528999999999996},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.45100000000000001,"5min":8.8000000000000007,"15min":9.5410000000000004},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":160.02199999999999,"15min":160.02199999999999},"last":0.72399999999999998}]},{"osd":5,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":2.1800000000000002,"15min":1.8899999999999999},"min":{"1min":0.35399999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.60399999999999998,"5min":41.645000000000003,"15min":41.645000000000003},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":10.452999999999999,"15min":12.787000000000001},"min":{"1min":0.32300000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66200000000000003,"5min":282.197,"15min":282.197},"last":0.755}]},{"osd":6,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.7770000000000001,"15min":1.385},"min":{"1min":0.36199999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58299999999999996,"5min":99.355000000000004,"15min":99.355000000000004},"last":0.748},{"interface":"front","average":{"1min":0.499,"5min":2.8929999999999998,"15min":2.129},"min":{"1min":0.35899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.72999999999999998,"5min":52.762,"15min":52.762},"last":0.68600000000000005}]},{"osd":7,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":4.8929999999999998,"15min":4.7489999999999997},"min":{"1min":0.34499999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.68700000000000006,"5min":94.662999999999997,"15min":94.662999999999997},"last":0.70699999999999996},{"interface":"front","average":{"1min":9.0709999999999997,"5min":4.2939999999999996,"15min":1.901},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":157.089,"5min":157.089,"15min":157.089},"last":0.64700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":81391616,"data_stored":85790877,"data_compressed":4527904,"data_compressed_allocated":4829184,"data_compressed_original":9498624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":223399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27324416,"data_stored":28849251,"data_compressed":1581247,"data_compressed_allocated":1675264,"data_compressed_original":3284992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26755072,"data_stored":26700452,"data_compressed":52333,"data_compressed_allocated":57344,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43991040,"data_stored":44050507,"data_compressed":314091,"data_compressed_allocated":335872,"data_compressed_original":616025,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6438912,"data_stored":6403215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13045760,"data_stored":12971930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":44650496,"data_stored":44915671,"data_compressed":435417,"data_compressed_allocated":471040,"data_compressed_original":931417,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33280000,"data_stored":33118894,"data_compressed":53657,"data_compressed_allocated":57344,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1929216,"data_stored":1913105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":17645568,"data_stored":17817997,"data_compressed":182394,"data_compressed_allocated":200704,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:41:57.172 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 36 2024-04-22T18:41:57.172 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:57.169+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:57.258 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:57.259 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:57.262 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:57.263+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:57.269 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:41:57.363 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:57.364+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:57.412 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:57.412 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:57.421 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:41:57.464 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:57.465+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:57.563 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:57.563 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:57.564 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:57.565+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:57.572 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:41:57.664 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:57.665+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:57.713 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:57.713 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:57.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:41:57.766 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:57.767+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:57.770 INFO:teuthology.orchestra.run.smithi089.stderr:set pool 4 pgp_num to 36 2024-04-22T18:41:57.821 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-22T18:41:57.836 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:57.836 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:57.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:41:57.886 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:57.887+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:57.967 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:57.968+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:41:57.987 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:57.987 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:57.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:41:58.068 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:58.068+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:58.137 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:58.137 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:58.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:41:58.168 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:58.169+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:58.246 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:58.246 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:58.255 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:41:58.298 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:58.299+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:58.355 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:58.356 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:58.364 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:41:58.405 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:58.406+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:58.471 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:58.472+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:58.506 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:58.507 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:58.516 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:41:58.573 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:58.574+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:58.658 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:58.659 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:58.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:41:58.674 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:58.674+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:58.768 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:58.768 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:58.774 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:58.775+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:58.778 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:41:58.780 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:58.781+0000 7f3e7be88640 -1 osd.0 181 handle_osd_map injecting map crc failure 2024-04-22T18:41:58.782 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-04-22T18:41:58.853 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:41:58.854+0000 7fa766d7d640 -1 osd.5 180 handle_osd_map injecting map crc failure 2024-04-22T18:41:58.875 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:58.876+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:58.892 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:41:58.921 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:58.921 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:58.930 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:41:58.976 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:58.977+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:59.012 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:41:59.012 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:41:59.028 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:41:59.029 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:41:59.044 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:41:59.044 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:41:59.061 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:41:59.061 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:41:59.072 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:41:59.072 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:41:59.076 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:41:59.077 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:41:59.077 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:59.077+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:41:59.081 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:41:59.093 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:41:59.093 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:41:59.109 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:41:59.109 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:41:59.125 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:41:59.125 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:41:59.178 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:41:59.179+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:41:59.183 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:41:59.183 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:41:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:41:59.184 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:59.192 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:41:59.279 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:59.279+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:59.295 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:41:59.295 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:41:59.295 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:41:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:41:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 194.44141286300001, 2024-04-22T18:41:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:41:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:41:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.296 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:41:59.297 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:41:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:41:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:41:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:41:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.298 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:41:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:41:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:41:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:41:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:41:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:41:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.299 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:41:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:41:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:41:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:41:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:41:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:41:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.300 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:41:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:41:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:41:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:41:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:59.301 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 194.44136170100001, 2024-04-22T18:41:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:41:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:41:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:41:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:41:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:41:59.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:41:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:41:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:41:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:41:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:41:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:41:59.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:41:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:41:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:41:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:41:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:41:59.305 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:41:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:41:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:41:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:41:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:41:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.306 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:41:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:41:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 194.441346477, 2024-04-22T18:41:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:41:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:59.307 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:41:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:41:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.308 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:41:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:41:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:41:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:41:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:41:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.309 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:41:59.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:41:59.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:41:59.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:41:59.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.310 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.310 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:41:59.310 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:41:59.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:59.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:41:59.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:41:59.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:59.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:41:59.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:41:59.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.311 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.311 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:41:59.311 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:41:59.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:59.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:41:59.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:41:59.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.312 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.312 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:41:59.312 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:41:59.312 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.312 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.312 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.312 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:59.312 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:59.313 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 194.44128689999999, 2024-04-22T18:41:59.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:41:59.313 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.313 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.313 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.313 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:59.313 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:59.313 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:41:59.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.313 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.313 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.313 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:59.313 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:59.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:41:59.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.314 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.314 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:41:59.314 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:41:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:41:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:41:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:41:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:41:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:41:59.315 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:41:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:41:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:41:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:41:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:41:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:41:59.316 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:41:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:41:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:41:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:41:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:41:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:41:59.317 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:41:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 194.44120458399999, 2024-04-22T18:41:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:41:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:59.318 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:41:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:41:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.319 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:41:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:41:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:41:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:41:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:41:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.320 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:41:59.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:41:59.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:41:59.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:41:59.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.321 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.321 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:41:59.321 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:41:59.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:59.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:41:59.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:41:59.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:59.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:41:59.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:41:59.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.322 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.322 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:41:59.322 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:41:59.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:59.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:41:59.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:41:59.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.323 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.323 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:41:59.323 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:41:59.323 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.323 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.323 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.323 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:41:59.324 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:59.324 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 194.44119442799999, 2024-04-22T18:41:59.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:41:59.324 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.324 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.324 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.324 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:41:59.324 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:41:59.324 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:41:59.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.324 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.324 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.324 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:59.324 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:41:59.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:41:59.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.325 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.325 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:41:59.325 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:41:59.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:41:59.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:41:59.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:41:59.326 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:41:59.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.326 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.326 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:41:59.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:41:59.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:41:59.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:41:59.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:41:59.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:41:59.327 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:41:59.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.327 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:41:59.327 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:41:59.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:41:59.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:41:59.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:41:59.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:41:59.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:41:59.328 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:41:59.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.328 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.328 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:41:59.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:41:59.329 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.329 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.329 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.329 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3163 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:59.329 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:59.329 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.45818199, 2024-04-22T18:41:59.329 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1009801299999999, 2024-04-22T18:41:59.329 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.329 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:59.329 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.329 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:59.329 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.330 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3163 2024-04-22T18:41:59.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.330 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:59.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.330 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.330 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:41:59.330 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837455+0000", 2024-04-22T18:41:59.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:59.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837464+0000", 2024-04-22T18:41:59.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5319999999999995e-06 2024-04-22T18:41:59.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.331 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.331 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837476+0000", 2024-04-22T18:41:59.331 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1759e-05 2024-04-22T18:41:59.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837488+0000", 2024-04-22T18:41:59.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1666e-05 2024-04-22T18:41:59.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837541+0000", 2024-04-22T18:41:59.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2939999999999998e-05 2024-04-22T18:41:59.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.332 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.332 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837547+0000", 2024-04-22T18:41:59.332 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6409999999999996e-06 2024-04-22T18:41:59.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937423+0000", 2024-04-22T18:41:59.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.099875545 2024-04-22T18:41:59.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937567+0000", 2024-04-22T18:41:59.333 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014363099999999999 2024-04-22T18:41:59.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.333 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.333 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938437+0000", 2024-04-22T18:41:59.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087060399999999997 2024-04-22T18:41:59.334 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.334 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.334 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.334 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3164 4.8 4:187adac1:::smithi08937929-36:head [sparse-read 0~2643888] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:59.334 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:59.334 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.458142620999993, 2024-04-22T18:41:59.334 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1013670310000001, 2024-04-22T18:41:59.334 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.334 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:59.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.334 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:59.335 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.335 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3164 2024-04-22T18:41:59.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.335 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:59.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.335 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.335 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:59.335 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:41:59.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837499+0000", 2024-04-22T18:41:59.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.994e-06 2024-04-22T18:41:59.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.336 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.336 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837503+0000", 2024-04-22T18:41:59.336 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7249999999999999e-06 2024-04-22T18:41:59.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837506+0000", 2024-04-22T18:41:59.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5980000000000001e-06 2024-04-22T18:41:59.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937641+0000", 2024-04-22T18:41:59.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10013467700000001 2024-04-22T18:41:59.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.337 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.337 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937650+0000", 2024-04-22T18:41:59.337 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3489999999999998e-06 2024-04-22T18:41:59.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938468+0000", 2024-04-22T18:41:59.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0008176710000001 2024-04-22T18:41:59.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938582+0000", 2024-04-22T18:41:59.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114287 2024-04-22T18:41:59.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.338 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.338 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938864+0000", 2024-04-22T18:41:59.338 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028120899999999998 2024-04-22T18:41:59.338 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.339 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.339 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.339 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3165 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:59.339 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:59.339 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.458123853000004, 2024-04-22T18:41:59.339 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2022754499999999, 2024-04-22T18:41:59.339 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.339 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:59.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:59.339 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.339 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3165 2024-04-22T18:41:59.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:41:59.340 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837517+0000", 2024-04-22T18:41:59.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3369999999999998e-06 2024-04-22T18:41:59.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837523+0000", 2024-04-22T18:41:59.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5450000000000003e-06 2024-04-22T18:41:59.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.341 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.341 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.341 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837526+0000", 2024-04-22T18:41:59.341 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.354e-06 2024-04-22T18:41:59.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937682+0000", 2024-04-22T18:41:59.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100155917 2024-04-22T18:41:59.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937688+0000", 2024-04-22T18:41:59.342 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:41:59.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.342 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.342 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.038953+0000", 2024-04-22T18:41:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012648110000001 2024-04-22T18:41:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039034+0000", 2024-04-22T18:41:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0969000000000004e-05 2024-04-22T18:41:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039791+0000", 2024-04-22T18:41:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075672900000000004 2024-04-22T18:41:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.344 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3183 4.8 4:1bc4668a:::smithi08937929-21:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:59.344 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:59.344 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.259720654999995, 2024-04-22T18:41:59.344 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0040478260000001, 2024-04-22T18:41:59.344 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.344 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:59.344 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.344 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:59.344 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.344 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3183 2024-04-22T18:41:59.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.344 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.344 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:59.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:41:59.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.345 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035913+0000", 2024-04-22T18:41:59.345 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999943 2024-04-22T18:41:59.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.345 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035935+0000", 2024-04-22T18:41:59.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2232999999999999e-05 2024-04-22T18:41:59.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035969+0000", 2024-04-22T18:41:59.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3969e-05 2024-04-22T18:41:59.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.346 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035987+0000", 2024-04-22T18:41:59.346 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8397e-05 2024-04-22T18:41:59.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.346 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:59.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2916000000000003e-05 2024-04-22T18:41:59.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:41:59.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0103e-05 2024-04-22T18:41:59.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.347 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039841+0000", 2024-04-22T18:41:59.347 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0037508959999999 2024-04-22T18:41:59.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.347 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.348 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039966+0000", 2024-04-22T18:41:59.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012514500000000001 2024-04-22T18:41:59.348 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.348 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.348 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.348 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3184 4.8 4:12d9f826:::smithi08937929-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:59.348 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:59.348 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.259632695999997, 2024-04-22T18:41:59.348 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.00402868, 2024-04-22T18:41:59.348 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.348 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:59.348 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.349 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:59.349 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.349 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3184 2024-04-22T18:41:59.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.349 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:59.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.349 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.349 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:41:59.349 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036005+0000", 2024-04-22T18:41:59.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:59.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036026+0000", 2024-04-22T18:41:59.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1092999999999999e-05 2024-04-22T18:41:59.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.350 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.350 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036030+0000", 2024-04-22T18:41:59.350 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.467e-06 2024-04-22T18:41:59.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036033+0000", 2024-04-22T18:41:59.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9780000000000001e-06 2024-04-22T18:41:59.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036120+0000", 2024-04-22T18:41:59.351 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7495000000000002e-05 2024-04-22T18:41:59.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.351 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.351 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036125+0000", 2024-04-22T18:41:59.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8720000000000001e-06 2024-04-22T18:41:59.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039982+0000", 2024-04-22T18:41:59.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0038571919999999 2024-04-22T18:41:59.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.352 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.352 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040035+0000", 2024-04-22T18:41:59.352 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2992999999999999e-05 2024-04-22T18:41:59.352 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.352 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.352 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.353 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3185 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:59.353 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:59.353 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.259578783999999, 2024-04-22T18:41:59.353 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0041404039999999, 2024-04-22T18:41:59.353 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.353 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:59.353 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.353 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:59.353 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.353 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3185 2024-04-22T18:41:59.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.353 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.353 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:59.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:59.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.354 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:41:59.354 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.354 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036062+0000", 2024-04-22T18:41:59.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9099999999999999e-06 2024-04-22T18:41:59.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036066+0000", 2024-04-22T18:41:59.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1699999999999999e-06 2024-04-22T18:41:59.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.355 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036077+0000", 2024-04-22T18:41:59.355 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0637e-05 2024-04-22T18:41:59.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.355 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036147+0000", 2024-04-22T18:41:59.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0025999999999997e-05 2024-04-22T18:41:59.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036151+0000", 2024-04-22T18:41:59.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:41:59.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.356 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.356 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040068+0000", 2024-04-22T18:41:59.356 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.003916604 2024-04-22T18:41:59.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040201+0000", 2024-04-22T18:41:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013275400000000001 2024-04-22T18:41:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3186 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:41:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 97.259559228000001, 2024-04-22T18:41:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2044167510000001, 2024-04-22T18:41:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:41:59.357 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.358 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:59.358 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.358 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3186 2024-04-22T18:41:59.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.358 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:59.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.358 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.358 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:41:59.358 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036079+0000", 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036081+0000", 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6449999999999999e-06 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036084+0000", 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9399999999999998e-06 2024-04-22T18:41:59.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:41:59.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.879e-06 2024-04-22T18:41:59.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036168+0000", 2024-04-22T18:41:59.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7657999999999994e-05 2024-04-22T18:41:59.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.360 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.360 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036171+0000", 2024-04-22T18:41:59.360 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7639999999999999e-06 2024-04-22T18:41:59.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240345+0000", 2024-04-22T18:41:59.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2041734260000001 2024-04-22T18:41:59.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.361 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.361 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240497+0000", 2024-04-22T18:41:59.361 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015182800000000001 2024-04-22T18:41:59.361 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.361 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.361 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.362 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3724 4.18 4:1bc4668a:::smithi08937929-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:41:59.362 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:59.362 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 74.235234852000005, 2024-04-22T18:41:59.362 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50413114699999995, 2024-04-22T18:41:59.362 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.362 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:59.362 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.362 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:59.362 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.362 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3724 2024-04-22T18:41:59.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.362 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.362 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.362 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:59.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:59.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.363 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:41:59.363 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.363 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060405+0000", 2024-04-22T18:41:59.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.395e-06 2024-04-22T18:41:59.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060408+0000", 2024-04-22T18:41:59.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:41:59.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.364 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060411+0000", 2024-04-22T18:41:59.364 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6060000000000001e-06 2024-04-22T18:41:59.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.364 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165799+0000", 2024-04-22T18:41:59.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105388369 2024-04-22T18:41:59.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:59.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165843+0000", 2024-04-22T18:41:59.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3891999999999999e-05 2024-04-22T18:41:59.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.365 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564408+0000", 2024-04-22T18:41:59.365 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39856466000000002 2024-04-22T18:41:59.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.365 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.365 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564461+0000", 2024-04-22T18:41:59.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3529999999999997e-05 2024-04-22T18:41:59.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.366 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.366 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564535+0000", 2024-04-22T18:41:59.366 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4147999999999995e-05 2024-04-22T18:41:59.366 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.366 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.366 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.366 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.366 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4413 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:59.366 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:59.366 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.638254807999999, 2024-04-22T18:41:59.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2025380569999999, 2024-04-22T18:41:59.367 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.367 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:59.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:59.367 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.367 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4413 2024-04-22T18:41:59.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.367 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.367 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.367 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:59.367 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.367 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:41:59.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657377+0000", 2024-04-22T18:41:59.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999933 2024-04-22T18:41:59.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.368 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.368 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657393+0000", 2024-04-22T18:41:59.368 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5296999999999999e-05 2024-04-22T18:41:59.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.368 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657411+0000", 2024-04-22T18:41:59.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8105e-05 2024-04-22T18:41:59.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657424+0000", 2024-04-22T18:41:59.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3030000000000001e-05 2024-04-22T18:41:59.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.369 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.369 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658313+0000", 2024-04-22T18:41:59.369 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00088943099999999999 2024-04-22T18:41:59.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.369 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658324+0000", 2024-04-22T18:41:59.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0496000000000001e-05 2024-04-22T18:41:59.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859560+0000", 2024-04-22T18:41:59.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2012358350000001 2024-04-22T18:41:59.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.370 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.370 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859648+0000", 2024-04-22T18:41:59.370 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8744000000000003e-05 2024-04-22T18:41:59.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859922+0000", 2024-04-22T18:41:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027396499999999998 2024-04-22T18:41:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4414 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.638200484000002, 2024-04-22T18:41:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2030255030000001, 2024-04-22T18:41:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.372 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:59.372 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.372 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4414 2024-04-22T18:41:59.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.372 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:59.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.372 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.372 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.372 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:41:59.372 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657437+0000", 2024-04-22T18:41:59.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:59.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657443+0000", 2024-04-22T18:41:59.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5899999999999998e-06 2024-04-22T18:41:59.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.373 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.373 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657450+0000", 2024-04-22T18:41:59.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4649999999999997e-06 2024-04-22T18:41:59.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657457+0000", 2024-04-22T18:41:59.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6000000000000003e-06 2024-04-22T18:41:59.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.374 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.374 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.374 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658364+0000", 2024-04-22T18:41:59.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090735999999999996 2024-04-22T18:41:59.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658373+0000", 2024-04-22T18:41:59.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3250000000000008e-06 2024-04-22T18:41:59.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860258+0000", 2024-04-22T18:41:59.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018852769999999 2024-04-22T18:41:59.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860308+0000", 2024-04-22T18:41:59.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9938000000000002e-05 2024-04-22T18:41:59.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860464+0000", 2024-04-22T18:41:59.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015622800000000001 2024-04-22T18:41:59.377 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.377 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.377 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.378 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4415 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:41:59.378 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:59.378 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.63817616, 2024-04-22T18:41:59.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2038250989999999, 2024-04-22T18:41:59.378 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.378 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:59.378 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.379 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:59.379 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.379 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4415 2024-04-22T18:41:59.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.379 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:59.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:41:59.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657462+0000", 2024-04-22T18:41:59.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657467+0000", 2024-04-22T18:41:59.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5399999999999997e-06 2024-04-22T18:41:59.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657473+0000", 2024-04-22T18:41:59.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3949999999999998e-06 2024-04-22T18:41:59.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657480+0000", 2024-04-22T18:41:59.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2489999999999998e-06 2024-04-22T18:41:59.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658394+0000", 2024-04-22T18:41:59.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00091403299999999997 2024-04-22T18:41:59.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658398+0000", 2024-04-22T18:41:59.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7979999999999996e-06 2024-04-22T18:41:59.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860495+0000", 2024-04-22T18:41:59.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.202097081 2024-04-22T18:41:59.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860543+0000", 2024-04-22T18:41:59.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7907000000000001e-05 2024-04-22T18:41:59.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861288+0000", 2024-04-22T18:41:59.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074466500000000004 2024-04-22T18:41:59.383 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.383 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.383 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.383 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4413 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:59.383 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:59.383 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.635459654999998, 2024-04-22T18:41:59.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.201491952, 2024-04-22T18:41:59.383 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4413 2024-04-22T18:41:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:41:59.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660178+0000", 2024-04-22T18:41:59.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:59.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660185+0000", 2024-04-22T18:41:59.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4379999999999998e-06 2024-04-22T18:41:59.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660201+0000", 2024-04-22T18:41:59.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5252999999999999e-05 2024-04-22T18:41:59.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660210+0000", 2024-04-22T18:41:59.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1069999999999994e-06 2024-04-22T18:41:59.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660270+0000", 2024-04-22T18:41:59.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739000000000002e-05 2024-04-22T18:41:59.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660281+0000", 2024-04-22T18:41:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0664e-05 2024-04-22T18:41:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861304+0000", 2024-04-22T18:41:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2010225059999999 2024-04-22T18:41:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861342+0000", 2024-04-22T18:41:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8736000000000003e-05 2024-04-22T18:41:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.388 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861671+0000", 2024-04-22T18:41:59.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032915899999999998 2024-04-22T18:41:59.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.388 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.388 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4414 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:59.388 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:59.388 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.635416145000001, 2024-04-22T18:41:59.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2014303930000001, 2024-04-22T18:41:59.388 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.388 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:59.388 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.388 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:59.389 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.389 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4414 2024-04-22T18:41:59.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.389 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:59.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:41:59.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660222+0000", 2024-04-22T18:41:59.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:59.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660227+0000", 2024-04-22T18:41:59.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6789999999999996e-06 2024-04-22T18:41:59.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660240+0000", 2024-04-22T18:41:59.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3485e-05 2024-04-22T18:41:59.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660245+0000", 2024-04-22T18:41:59.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8269999999999997e-06 2024-04-22T18:41:59.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660307+0000", 2024-04-22T18:41:59.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2392999999999997e-05 2024-04-22T18:41:59.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660316+0000", 2024-04-22T18:41:59.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7790000000000001e-06 2024-04-22T18:41:59.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861493+0000", 2024-04-22T18:41:59.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.201177341 2024-04-22T18:41:59.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861529+0000", 2024-04-22T18:41:59.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5978e-05 2024-04-22T18:41:59.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861653+0000", 2024-04-22T18:41:59.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012391799999999999 2024-04-22T18:41:59.392 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4415 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:41:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 41.635370921000003, 2024-04-22T18:41:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2019634779999999, 2024-04-22T18:41:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:41:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:41:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:41:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:41:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4415 2024-04-22T18:41:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.394 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:41:59.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:41:59.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:59.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:41:59.394 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:41:59.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:41:59.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.394 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:41:59.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660267+0000", 2024-04-22T18:41:59.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:41:59.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:41:59.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660273+0000", 2024-04-22T18:41:59.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5910000000000003e-06 2024-04-22T18:41:59.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:41:59.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660280+0000", 2024-04-22T18:41:59.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4000000000000003e-06 2024-04-22T18:41:59.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660288+0000", 2024-04-22T18:41:59.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1769999999999998e-06 2024-04-22T18:41:59.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.760401+0000", 2024-04-22T18:41:59.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100112864 2024-04-22T18:41:59.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.760408+0000", 2024-04-22T18:41:59.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0299999999999996e-06 2024-04-22T18:41:59.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:41:59.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861682+0000", 2024-04-22T18:41:59.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012742520000001 2024-04-22T18:41:59.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:41:59.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861716+0000", 2024-04-22T18:41:59.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3674999999999999e-05 2024-04-22T18:41:59.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:41:59.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:41:59.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:41:59.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.862232+0000", 2024-04-22T18:41:59.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051554300000000003 2024-04-22T18:41:59.399 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.399 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.399 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.400 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:41:59.400 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:41:59.400 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:41:59.402 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:41:59.402+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:41:59.402 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:41:59.502 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:41:59.503+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:41:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:59.504 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:59.504 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:41:59.504 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:59.513 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:41:59.604 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:41:59.604+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:41:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:41:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:41:59.615 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:41:59.615 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:59.624 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:41:59.708 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:41:59.708+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:41:59.767 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:41:59.767 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:41:59.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:41:59.768 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:41:59.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.768 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:66 4.3 4:c0d4d677:::smithi08937929-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:41:59.768 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:59.768 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 185.31220337900001, 2024-04-22T18:41:59.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50374043300000004, 2024-04-22T18:41:59.768 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.768 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.768 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.768 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.768 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.768 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 66 2024-04-22T18:41:59.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.769 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:59.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455357+0000", 2024-04-22T18:41:59.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455355+0000", 2024-04-22T18:41:59.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:41:59.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455441+0000", 2024-04-22T18:41:59.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5446999999999997e-05 2024-04-22T18:41:59.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455448+0000", 2024-04-22T18:41:59.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8020000000000003e-06 2024-04-22T18:41:59.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.455453+0000", 2024-04-22T18:41:59.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1769999999999997e-06 2024-04-22T18:41:59.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756319+0000", 2024-04-22T18:41:59.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30086636999999999 2024-04-22T18:41:59.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756381+0000", 2024-04-22T18:41:59.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1687999999999996e-05 2024-04-22T18:41:59.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-04-22T18:41:59.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.756472+0000", 2024-04-22T18:41:59.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1189999999999999e-05 2024-04-22T18:41:59.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:59.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.759091+0000", 2024-04-22T18:41:59.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002618591 2024-04-22T18:41:59.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959045+0000", 2024-04-22T18:41:59.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19995481200000001 2024-04-22T18:41:59.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:59.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959075+0000", 2024-04-22T18:41:59.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9354999999999999e-05 2024-04-22T18:41:59.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.959097+0000", 2024-04-22T18:41:59.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2166000000000001e-05 2024-04-22T18:41:59.773 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.773 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.773 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.773 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:775 4.b 4:defe92fb:::smithi08937929-44:head [setxattr _header (56) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:59.774 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:59.774 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 176.61914613100001, 2024-04-22T18:41:59.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50543532700000005, 2024-04-22T18:41:59.774 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.774 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.774 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.774 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.774 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.774 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 775 2024-04-22T18:41:59.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.774 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:59.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148414+0000", 2024-04-22T18:41:59.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148413+0000", 2024-04-22T18:41:59.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148417+0000", 2024-04-22T18:41:59.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8039999999999999e-06 2024-04-22T18:41:59.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148423+0000", 2024-04-22T18:41:59.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2609999999999999e-06 2024-04-22T18:41:59.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148428+0000", 2024-04-22T18:41:59.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7709999999999997e-06 2024-04-22T18:41:59.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252523+0000", 2024-04-22T18:41:59.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104094913 2024-04-22T18:41:59.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252605+0000", 2024-04-22T18:41:59.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1866999999999994e-05 2024-04-22T18:41:59.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:59.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252742+0000", 2024-04-22T18:41:59.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137479 2024-04-22T18:41:59.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:59.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.256154+0000", 2024-04-22T18:41:59.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00341129 2024-04-22T18:41:59.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653738+0000", 2024-04-22T18:41:59.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39758473999999999 2024-04-22T18:41:59.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:59.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653790+0000", 2024-04-22T18:41:59.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1641000000000002e-05 2024-04-22T18:41:59.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.653849+0000", 2024-04-22T18:41:59.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9197999999999998e-05 2024-04-22T18:41:59.779 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.779 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.779 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.779 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:776 4.b 4:defe92fb:::smithi08937929-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-22T18:41:59.779 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:59.779 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 176.619127655, 2024-04-22T18:41:59.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60565048300000002, 2024-04-22T18:41:59.780 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.780 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:59.780 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.780 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.780 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.780 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 776 2024-04-22T18:41:59.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.780 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:59.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148432+0000", 2024-04-22T18:41:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148434+0000", 2024-04-22T18:41:59.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4719999999999998e-06 2024-04-22T18:41:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148441+0000", 2024-04-22T18:41:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5189999999999998e-06 2024-04-22T18:41:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.148448+0000", 2024-04-22T18:41:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-04-22T18:41:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.252977+0000", 2024-04-22T18:41:59.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104528695 2024-04-22T18:41:59.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:59.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.253023+0000", 2024-04-22T18:41:59.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6347000000000002e-05 2024-04-22T18:41:59.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.753955+0000", 2024-04-22T18:41:59.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50093195300000004 2024-04-22T18:41:59.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754009+0000", 2024-04-22T18:41:59.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4002000000000002e-05 2024-04-22T18:41:59.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:03.754083+0000", 2024-04-22T18:41:59.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3837000000000005e-05 2024-04-22T18:41:59.784 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.784 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.784 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.784 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:1690 4.b 4:dc800c01:::smithi08937929-37:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-04-22T18:41:59.784 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:59.784 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 148.856079343, 2024-04-22T18:41:59.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.55448709299999999, 2024-04-22T18:41:59.785 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.785 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.785 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.785 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.785 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.785 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1690 2024-04-22T18:41:59.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.785 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:59.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911481+0000", 2024-04-22T18:41:59.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911479+0000", 2024-04-22T18:41:59.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:59.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911487+0000", 2024-04-22T18:41:59.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-04-22T18:41:59.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911500+0000", 2024-04-22T18:41:59.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2751999999999999e-05 2024-04-22T18:41:59.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:30.911507+0000", 2024-04-22T18:41:59.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6559999999999992e-06 2024-04-22T18:41:59.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011615+0000", 2024-04-22T18:41:59.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100107635 2024-04-22T18:41:59.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.011706+0000", 2024-04-22T18:41:59.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0867999999999998e-05 2024-04-22T18:41:59.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:59.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.372511+0000", 2024-04-22T18:41:59.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.360805231 2024-04-22T18:41:59.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.461308+0000", 2024-04-22T18:41:59.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.088796781000000005 2024-04-22T18:41:59.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:59.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465927+0000", 2024-04-22T18:41:59.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196299999999996 2024-04-22T18:41:59.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:59.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465950+0000", 2024-04-22T18:41:59.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2920000000000001e-05 2024-04-22T18:41:59.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:31.465968+0000", 2024-04-22T18:41:59.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7459e-05 2024-04-22T18:41:59.790 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.790 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.790 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.790 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:249 3.1 3:b5e2b342:::dc8adfa0f92e2d59188cf539148947abc7f152783b0d839b3e4fceb86ffd5ef4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:41:59.790 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:59.790 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 139.18365103400001, 2024-04-22T18:41:59.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50948474700000002, 2024-04-22T18:41:59.790 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.790 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.790 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:59.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583909+0000", 2024-04-22T18:41:59.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583906+0000", 2024-04-22T18:41:59.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:59.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583920+0000", 2024-04-22T18:41:59.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698999999999999e-05 2024-04-22T18:41:59.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.583937+0000", 2024-04-22T18:41:59.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6827e-05 2024-04-22T18:41:59.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684030+0000", 2024-04-22T18:41:59.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100092732 2024-04-22T18:41:59.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684111+0000", 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0945999999999994e-05 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684191+0000", 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9767999999999999e-05 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.684301+0000", 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011001300000000001 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:59.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:40.685143+0000", 2024-04-22T18:41:59.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084245900000000005 2024-04-22T18:41:59.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093331+0000", 2024-04-22T18:41:59.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40818738999999998 2024-04-22T18:41:59.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:59.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093368+0000", 2024-04-22T18:41:59.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7029000000000001e-05 2024-04-22T18:41:59.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:41.093394+0000", 2024-04-22T18:41:59.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6123e-05 2024-04-22T18:41:59.795 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.795 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.795 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.795 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2388 4.13 4:c93c842e:::smithi08937929-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-04-22T18:41:59.795 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:59.795 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 129.374634327, 2024-04-22T18:41:59.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43630956999999998, 2024-04-22T18:41:59.795 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.795 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:59.796 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.796 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.796 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.796 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2388 2024-04-22T18:41:59.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.796 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:59.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392926+0000", 2024-04-22T18:41:59.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392925+0000", 2024-04-22T18:41:59.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392928+0000", 2024-04-22T18:41:59.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7709999999999998e-06 2024-04-22T18:41:59.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392931+0000", 2024-04-22T18:41:59.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0910000000000001e-06 2024-04-22T18:41:59.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.392934+0000", 2024-04-22T18:41:59.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:41:59.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.585962+0000", 2024-04-22T18:41:59.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19302776699999999 2024-04-22T18:41:59.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:59.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.586025+0000", 2024-04-22T18:41:59.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3572999999999996e-05 2024-04-22T18:41:59.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829101+0000", 2024-04-22T18:41:59.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.24307578599999999 2024-04-22T18:41:59.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829155+0000", 2024-04-22T18:41:59.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3792000000000003e-05 2024-04-22T18:41:59.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:50.829235+0000", 2024-04-22T18:41:59.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0192999999999996e-05 2024-04-22T18:41:59.800 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.800 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.800 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.800 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:2827 4.6 4:6700eab5:::smithi08937929-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-04-22T18:41:59.800 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:59.800 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 112.478595955, 2024-04-22T18:41:59.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.43818224300000003, 2024-04-22T18:41:59.800 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.800 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:59.800 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.801 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.801 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.801 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 2827 2024-04-22T18:41:59.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.801 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:59.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:59.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288964+0000", 2024-04-22T18:41:59.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288966+0000", 2024-04-22T18:41:59.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:41:59.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288970+0000", 2024-04-22T18:41:59.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.194e-06 2024-04-22T18:41:59.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.288973+0000", 2024-04-22T18:41:59.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1659999999999998e-06 2024-04-22T18:41:59.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289002+0000", 2024-04-22T18:41:59.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8739999999999999e-05 2024-04-22T18:41:59.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.289097+0000", 2024-04-22T18:41:59.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5511000000000001e-05 2024-04-22T18:41:59.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:07.727146+0000", 2024-04-22T18:41:59.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.438048832 2024-04-22T18:41:59.804 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.804 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.804 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.804 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3377 4.6 4:6700eab5:::smithi08937929-32:head [read 1812992~307712] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-04-22T18:41:59.804 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:59.804 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 89.331192313000003, 2024-04-22T18:41:59.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.52001745799999999, 2024-04-22T18:41:59.804 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.804 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3377 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436368+0000", 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436364+0000", 2024-04-22T18:41:59.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:59.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436376+0000", 2024-04-22T18:41:59.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2121e-05 2024-04-22T18:41:59.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436394+0000", 2024-04-22T18:41:59.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8094e-05 2024-04-22T18:41:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436405+0000", 2024-04-22T18:41:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0189e-05 2024-04-22T18:41:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436471+0000", 2024-04-22T18:41:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6817999999999996e-05 2024-04-22T18:41:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.436481+0000", 2024-04-22T18:41:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2720000000000003e-06 2024-04-22T18:41:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956084+0000", 2024-04-22T18:41:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51960320800000004 2024-04-22T18:41:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956232+0000", 2024-04-22T18:41:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014778600000000001 2024-04-22T18:41:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:30.956385+0000", 2024-04-22T18:41:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153394 2024-04-22T18:41:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.1.13:131 3.1 3:a0a19997:::ee8ca23a7050e771c77dff761e6ea5a11ed57796f4e8a17cbb5804a7fdc937b9:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:41:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 88.438322717000005, 2024-04-22T18:41:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51449564999999997, 2024-04-22T18:41:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329237+0000", 2024-04-22T18:41:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329235+0000", 2024-04-22T18:41:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:59.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329244+0000", 2024-04-22T18:41:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6819999999999992e-06 2024-04-22T18:41:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329259+0000", 2024-04-22T18:41:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4698000000000001e-05 2024-04-22T18:41:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329265+0000", 2024-04-22T18:41:59.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1510000000000001e-06 2024-04-22T18:41:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.329311+0000", 2024-04-22T18:41:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6096e-05 2024-04-22T18:41:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:41:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.332269+0000", 2024-04-22T18:41:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029581910000000002 2024-04-22T18:41:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630603+0000", 2024-04-22T18:41:59.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.29833418 2024-04-22T18:41:59.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630685+0000", 2024-04-22T18:41:59.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1217999999999994e-05 2024-04-22T18:41:59.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:41:59.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.630813+0000", 2024-04-22T18:41:59.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128706 2024-04-22T18:41:59.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:59.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.631447+0000", 2024-04-22T18:41:59.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063427000000000004 2024-04-22T18:41:59.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843586+0000", 2024-04-22T18:41:59.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21213873 2024-04-22T18:41:59.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:59.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843623+0000", 2024-04-22T18:41:59.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6848999999999999e-05 2024-04-22T18:41:59.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.843733+0000", 2024-04-22T18:41:59.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010982200000000001 2024-04-22T18:41:59.815 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.815 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.815 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.815 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3441 4.b 4:d3068ee3:::smithi08937929-29:head [write 558065~619869] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:59.815 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:59.815 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 87.849846655999997, 2024-04-22T18:41:59.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50272020399999995, 2024-04-22T18:41:59.815 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.815 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.815 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.815 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.816 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3441 2024-04-22T18:41:59.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.816 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:59.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917713+0000", 2024-04-22T18:41:59.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917711+0000", 2024-04-22T18:41:59.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:41:59.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917838+0000", 2024-04-22T18:41:59.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012734899999999999 2024-04-22T18:41:59.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917854+0000", 2024-04-22T18:41:59.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6130999999999999e-05 2024-04-22T18:41:59.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.917863+0000", 2024-04-22T18:41:59.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5969999999999992e-06 2024-04-22T18:41:59.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992845+0000", 2024-04-22T18:41:59.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.074982193000000003 2024-04-22T18:41:59.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.992919+0000", 2024-04-22T18:41:59.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4023999999999995e-05 2024-04-22T18:41:59.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:59.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.993019+0000", 2024-04-22T18:41:59.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9718999999999999e-05 2024-04-22T18:41:59.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:59.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095554+0000", 2024-04-22T18:41:59.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102535193 2024-04-22T18:41:59.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420183+0000", 2024-04-22T18:41:59.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32462853600000002 2024-04-22T18:41:59.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:59.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420401+0000", 2024-04-22T18:41:59.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021881400000000001 2024-04-22T18:41:59.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420433+0000", 2024-04-22T18:41:59.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2079000000000002e-05 2024-04-22T18:41:59.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.820 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.820 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3442 4.b 4:d3068ee3:::smithi08937929-29:head [write 1579835~486765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:59.821 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:59.821 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 87.845159945000006, 2024-04-22T18:41:59.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49822799099999998, 2024-04-22T18:41:59.821 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.821 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.821 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3442 2024-04-22T18:41:59.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.821 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:59.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922400+0000", 2024-04-22T18:41:59.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922399+0000", 2024-04-22T18:41:59.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:59.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922502+0000", 2024-04-22T18:41:59.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102784 2024-04-22T18:41:59.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922513+0000", 2024-04-22T18:41:59.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1051e-05 2024-04-22T18:41:59.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.922518+0000", 2024-04-22T18:41:59.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8529999999999997e-06 2024-04-22T18:41:59.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093158+0000", 2024-04-22T18:41:59.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.17063958800000001 2024-04-22T18:41:59.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093230+0000", 2024-04-22T18:41:59.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1803000000000004e-05 2024-04-22T18:41:59.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:59.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.093326+0000", 2024-04-22T18:41:59.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6454999999999997e-05 2024-04-22T18:41:59.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:59.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095557+0000", 2024-04-22T18:41:59.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022306399999999999 2024-04-22T18:41:59.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420484+0000", 2024-04-22T18:41:59.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32492717700000001 2024-04-22T18:41:59.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:59.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420606+0000", 2024-04-22T18:41:59.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012225800000000001 2024-04-22T18:41:59.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420628+0000", 2024-04-22T18:41:59.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1596000000000001e-05 2024-04-22T18:41:59.826 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.826 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.826 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.826 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3443 4.b 4:d3068ee3:::smithi08937929-29:head [write 2468371~489694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:59.826 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:59.826 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 87.840557898, 2024-04-22T18:41:59.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49372487199999998, 2024-04-22T18:41:59.826 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.826 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.827 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.827 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.827 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.827 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3443 2024-04-22T18:41:59.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.827 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:59.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927002+0000", 2024-04-22T18:41:59.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927001+0000", 2024-04-22T18:41:59.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:59.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927117+0000", 2024-04-22T18:41:59.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011637 2024-04-22T18:41:59.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927128+0000", 2024-04-22T18:41:59.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0696000000000001e-05 2024-04-22T18:41:59.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927134+0000", 2024-04-22T18:41:59.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8640000000000001e-06 2024-04-22T18:41:59.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094288+0000", 2024-04-22T18:41:59.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16715412900000001 2024-04-22T18:41:59.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094359+0000", 2024-04-22T18:41:59.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1639999999999998e-05 2024-04-22T18:41:59.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:59.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.094451+0000", 2024-04-22T18:41:59.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1252e-05 2024-04-22T18:41:59.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:59.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095560+0000", 2024-04-22T18:41:59.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011095530000000001 2024-04-22T18:41:59.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420661+0000", 2024-04-22T18:41:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32510105500000003 2024-04-22T18:41:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420710+0000", 2024-04-22T18:41:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9018999999999999e-05 2024-04-22T18:41:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420727+0000", 2024-04-22T18:41:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6722e-05 2024-04-22T18:41:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3444 4.b 4:d3068ee3:::smithi08937929-29:head [setxattr _header (56) in=7b,truncate 2958065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 87.840422180999994, 2024-04-22T18:41:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49366935299999998, 2024-04-22T18:41:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3444 2024-04-22T18:41:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:59.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927138+0000", 2024-04-22T18:41:59.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927137+0000", 2024-04-22T18:41:59.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927141+0000", 2024-04-22T18:41:59.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3270000000000001e-06 2024-04-22T18:41:59.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927144+0000", 2024-04-22T18:41:59.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9730000000000002e-06 2024-04-22T18:41:59.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927147+0000", 2024-04-22T18:41:59.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.959e-06 2024-04-22T18:41:59.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095118+0000", 2024-04-22T18:41:59.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.16797115500000001 2024-04-22T18:41:59.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095169+0000", 2024-04-22T18:41:59.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1316000000000001e-05 2024-04-22T18:41:59.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:59.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095260+0000", 2024-04-22T18:41:59.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1184e-05 2024-04-22T18:41:59.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:59.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095999+0000", 2024-04-22T18:41:59.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073868200000000005 2024-04-22T18:41:59.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420757+0000", 2024-04-22T18:41:59.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32475783499999999 2024-04-22T18:41:59.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:59.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420781+0000", 2024-04-22T18:41:59.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4022000000000001e-05 2024-04-22T18:41:59.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420807+0000", 2024-04-22T18:41:59.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6366000000000001e-05 2024-04-22T18:41:59.837 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.837 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.837 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.837 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3445 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:41:59.837 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:59.837 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 87.840411087000007, 2024-04-22T18:41:59.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.493780049, 2024-04-22T18:41:59.837 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.837 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:59.837 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.838 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.838 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.838 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3445 2024-04-22T18:41:59.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.838 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:59.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:59.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927149+0000", 2024-04-22T18:41:59.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927150+0000", 2024-04-22T18:41:59.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.542e-06 2024-04-22T18:41:59.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927153+0000", 2024-04-22T18:41:59.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0520000000000002e-06 2024-04-22T18:41:59.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027218+0000", 2024-04-22T18:41:59.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100064604 2024-04-22T18:41:59.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095501+0000", 2024-04-22T18:41:59.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.068283431000000006 2024-04-22T18:41:59.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:59.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.095545+0000", 2024-04-22T18:41:59.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4295999999999997e-05 2024-04-22T18:41:59.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420833+0000", 2024-04-22T18:41:59.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.32528718400000001 2024-04-22T18:41:59.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420877+0000", 2024-04-22T18:41:59.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4403999999999997e-05 2024-04-22T18:41:59.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.420929+0000", 2024-04-22T18:41:59.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1848999999999998e-05 2024-04-22T18:41:59.842 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.842 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.842 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.842 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4780 4.6 4:630bd8e5:::smithi08937929-45:head [write 574905~711397,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:59.842 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.278066+0000", 2024-04-22T18:41:59.842 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.489493842000002, 2024-04-22T18:41:59.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88434433999999995, 2024-04-22T18:41:59.842 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.842 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.842 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.842 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.842 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.842 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4780 2024-04-22T18:41:59.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.843 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278066+0000", 2024-04-22T18:41:59.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278066+0000", 2024-04-22T18:41:59.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278063+0000", 2024-04-22T18:41:59.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:41:59.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278391+0000", 2024-04-22T18:41:59.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032839499999999999 2024-04-22T18:41:59.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278404+0000", 2024-04-22T18:41:59.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3528e-05 2024-04-22T18:41:59.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278415+0000", 2024-04-22T18:41:59.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0295000000000001e-05 2024-04-22T18:41:59.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278454+0000", 2024-04-22T18:41:59.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9551999999999997e-05 2024-04-22T18:41:59.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.278462+0000", 2024-04-22T18:41:59.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4080000000000001e-06 2024-04-22T18:41:59.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.959123+0000", 2024-04-22T18:41:59.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68066109799999996 2024-04-22T18:41:59.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.959282+0000", 2024-04-22T18:41:59.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000159023 2024-04-22T18:41:59.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:59.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.959448+0000", 2024-04-22T18:41:59.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016666200000000001 2024-04-22T18:41:59.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162299+0000", 2024-04-22T18:41:59.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20285081199999999 2024-04-22T18:41:59.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:59.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162316+0000", 2024-04-22T18:41:59.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6354999999999998e-05 2024-04-22T18:41:59.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:59.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162347+0000", 2024-04-22T18:41:59.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1427999999999998e-05 2024-04-22T18:41:59.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162410+0000", 2024-04-22T18:41:59.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3350999999999997e-05 2024-04-22T18:41:59.848 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.848 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.848 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.848 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4781 4.6 4:630bd8e5:::smithi08937929-45:head [write 1885086~525140] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:59.848 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.284151+0000", 2024-04-22T18:41:59.848 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.483409001999998, 2024-04-22T18:41:59.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88746760999999996, 2024-04-22T18:41:59.848 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.848 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.848 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.848 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.849 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.849 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4781 2024-04-22T18:41:59.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.849 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284151+0000", 2024-04-22T18:41:59.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284151+0000", 2024-04-22T18:41:59.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284149+0000", 2024-04-22T18:41:59.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:59.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284385+0000", 2024-04-22T18:41:59.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00023556799999999999 2024-04-22T18:41:59.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284396+0000", 2024-04-22T18:41:59.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0842999999999999e-05 2024-04-22T18:41:59.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284401+0000", 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3260000000000003e-06 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284444+0000", 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3186000000000003e-05 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.284454+0000", 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6160000000000008e-06 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.161315+0000", 2024-04-22T18:41:59.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87686138000000002 2024-04-22T18:41:59.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.161381+0000", 2024-04-22T18:41:59.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5838000000000002e-05 2024-04-22T18:41:59.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:59.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.161470+0000", 2024-04-22T18:41:59.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9332000000000005e-05 2024-04-22T18:41:59.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163245+0000", 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017744530000000001 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.171550+0000", 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083046879999999993 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.171592+0000", 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2438999999999998e-05 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.171619+0000", 2024-04-22T18:41:59.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6517e-05 2024-04-22T18:41:59.854 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.854 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.854 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.854 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4782 4.6 4:630bd8e5:::smithi08937929-45:head [write 2930189~44716] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:59.854 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.287206+0000", 2024-04-22T18:41:59.854 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.480354035000001, 2024-04-22T18:41:59.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98327066299999999, 2024-04-22T18:41:59.855 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.855 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.855 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.855 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.855 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.855 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4782 2024-04-22T18:41:59.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.855 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287206+0000", 2024-04-22T18:41:59.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287206+0000", 2024-04-22T18:41:59.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287205+0000", 2024-04-22T18:41:59.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:41:59.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287220+0000", 2024-04-22T18:41:59.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5709000000000002e-05 2024-04-22T18:41:59.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287231+0000", 2024-04-22T18:41:59.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0196e-05 2024-04-22T18:41:59.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287236+0000", 2024-04-22T18:41:59.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3770000000000004e-06 2024-04-22T18:41:59.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287264+0000", 2024-04-22T18:41:59.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8549999999999999e-05 2024-04-22T18:41:59.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287268+0000", 2024-04-22T18:41:59.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7299999999999999e-06 2024-04-22T18:41:59.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162435+0000", 2024-04-22T18:41:59.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87516707999999999 2024-04-22T18:41:59.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162490+0000", 2024-04-22T18:41:59.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5210000000000002e-05 2024-04-22T18:41:59.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:59.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162567+0000", 2024-04-22T18:41:59.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6947999999999996e-05 2024-04-22T18:41:59.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:59.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163249+0000", 2024-04-22T18:41:59.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00068115299999999999 2024-04-22T18:41:59.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.270414+0000", 2024-04-22T18:41:59.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10716516700000001 2024-04-22T18:41:59.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:59.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.270448+0000", 2024-04-22T18:41:59.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4042999999999999e-05 2024-04-22T18:41:59.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.270477+0000", 2024-04-22T18:41:59.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8796e-05 2024-04-22T18:41:59.860 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.860 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.860 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.860 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4783 4.6 4:630bd8e5:::smithi08937929-45:head [setxattr _header (56) in=7b,truncate 2974905] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:59.861 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.287242+0000", 2024-04-22T18:41:59.861 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.480317921000001, 2024-04-22T18:41:59.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98449730599999996, 2024-04-22T18:41:59.861 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.861 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.861 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.861 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.861 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.861 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4783 2024-04-22T18:41:59.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.861 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287242+0000", 2024-04-22T18:41:59.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287242+0000", 2024-04-22T18:41:59.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287241+0000", 2024-04-22T18:41:59.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:41:59.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287245+0000", 2024-04-22T18:41:59.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.997e-06 2024-04-22T18:41:59.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.287251+0000", 2024-04-22T18:41:59.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4999999999999999e-06 2024-04-22T18:41:59.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387333+0000", 2024-04-22T18:41:59.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100082174 2024-04-22T18:41:59.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387418+0000", 2024-04-22T18:41:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5086000000000005e-05 2024-04-22T18:41:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387427+0000", 2024-04-22T18:41:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6829999999999997e-06 2024-04-22T18:41:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162901+0000", 2024-04-22T18:41:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77547466899999995 2024-04-22T18:41:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.162942+0000", 2024-04-22T18:41:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0299999999999997e-05 2024-04-22T18:41:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:41:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163028+0000", 2024-04-22T18:41:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6267000000000006e-05 2024-04-22T18:41:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163938+0000", 2024-04-22T18:41:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091012500000000002 2024-04-22T18:41:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271677+0000", 2024-04-22T18:41:59.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107739168 2024-04-22T18:41:59.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:59.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271709+0000", 2024-04-22T18:41:59.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1191e-05 2024-04-22T18:41:59.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271739+0000", 2024-04-22T18:41:59.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0814e-05 2024-04-22T18:41:59.866 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.866 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.866 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.867 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4784 4.6 4:630bd8e5:::smithi08937929-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:41:59.867 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.387355+0000", 2024-04-22T18:41:59.867 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.380204845000002, 2024-04-22T18:41:59.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.884536714, 2024-04-22T18:41:59.867 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.867 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:41:59.867 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.867 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.867 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.867 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4784 2024-04-22T18:41:59.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.867 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387355+0000", 2024-04-22T18:41:59.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387355+0000", 2024-04-22T18:41:59.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387352+0000", 2024-04-22T18:41:59.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:41:59.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387386+0000", 2024-04-22T18:41:59.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3974999999999999e-05 2024-04-22T18:41:59.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387399+0000", 2024-04-22T18:41:59.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2882e-05 2024-04-22T18:41:59.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.387408+0000", 2024-04-22T18:41:59.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2310000000000002e-06 2024-04-22T18:41:59.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.487520+0000", 2024-04-22T18:41:59.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100111622 2024-04-22T18:41:59.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:41:59.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.487531+0000", 2024-04-22T18:41:59.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0655999999999999e-05 2024-04-22T18:41:59.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163199+0000", 2024-04-22T18:41:59.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.67566841700000002 2024-04-22T18:41:59.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:41:59.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.163239+0000", 2024-04-22T18:41:59.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9749000000000002e-05 2024-04-22T18:41:59.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271780+0000", 2024-04-22T18:41:59.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10854082600000001 2024-04-22T18:41:59.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271830+0000", 2024-04-22T18:41:59.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0835999999999999e-05 2024-04-22T18:41:59.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.271892+0000", 2024-04-22T18:41:59.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1440999999999997e-05 2024-04-22T18:41:59.875 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.875 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.875 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.875 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4940 3.1 3:af0c84c6:::a6954558d6d261bd8039fe057a41a32eb57dde7f3e53a34b25ba0607fb91f814:head [write 0~396288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-04-22T18:41:59.875 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:41.009760+0000", 2024-04-22T18:41:59.875 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.757800172, 2024-04-22T18:41:59.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.500823562, 2024-04-22T18:41:59.875 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:41:59.875 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:41:59.875 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:41:59.876 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:41:59.876 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:41:59.876 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4940 2024-04-22T18:41:59.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.876 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:41:59.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:41:59.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009760+0000", 2024-04-22T18:41:59.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:41:59.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009760+0000", 2024-04-22T18:41:59.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:41:59.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:41:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009758+0000", 2024-04-22T18:41:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:41:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:41:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009935+0000", 2024-04-22T18:41:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017724200000000001 2024-04-22T18:41:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:41:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009946+0000", 2024-04-22T18:41:59.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0477e-05 2024-04-22T18:41:59.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:41:59.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.009952+0000", 2024-04-22T18:41:59.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3090000000000001e-06 2024-04-22T18:41:59.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:41:59.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.109570+0000", 2024-04-22T18:41:59.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099618130999999999 2024-04-22T18:41:59.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:41:59.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.109693+0000", 2024-04-22T18:41:59.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012241699999999999 2024-04-22T18:41:59.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:41:59.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.109807+0000", 2024-04-22T18:41:59.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011426599999999999 2024-04-22T18:41:59.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:41:59.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.110527+0000", 2024-04-22T18:41:59.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00071964499999999999 2024-04-22T18:41:59.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:41:59.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.510490+0000", 2024-04-22T18:41:59.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39996386499999997 2024-04-22T18:41:59.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:41:59.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.510550+0000", 2024-04-22T18:41:59.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9669000000000001e-05 2024-04-22T18:41:59.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:41:59.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:41:59.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:41:59.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:41.510583+0000", 2024-04-22T18:41:59.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3281000000000002e-05 2024-04-22T18:41:59.880 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.880 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.880 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.880 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:41:59.880 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:41:59.881 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:41:59.882 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:41:59.882 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:41:59.882+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:41:59.982 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:41:59.982+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:42:00.024 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:00.024 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:42:00.024 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:42:00.025 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:00.033 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:42:00.083 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:42:00.084+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:42:00.174 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:00.175 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:42:00.175 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:42:00.175 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:42:00.175 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:00.184 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:42:00.184 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:42:00.185+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:42:00.284 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:42:00.285+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:42:00.287 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:00.287 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:42:00.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:42:00.287 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:42:00.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.287 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:42:00.287 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:42:00.287 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 186.095551406, 2024-04-22T18:42:00.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:42:00.287 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.287 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.287 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.288 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.288 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.288 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:42:00.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.288 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:42:00.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:42:00.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:42:00.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:42:00.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:42:00.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:42:00.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:42:00.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:42:00.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:42:00.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:42:00.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:42:00.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:42:00.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:42:00.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:42:00.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:42:00.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:42:00.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:00.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:42:00.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:42:00.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:00.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:42:00.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:42:00.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:42:00.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:42:00.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:42:00.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:42:00.292 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.292 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.292 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.293 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:42:00.293 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:42:00.293 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 186.095540331, 2024-04-22T18:42:00.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:42:00.293 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.293 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.293 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.293 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.293 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.293 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:42:00.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.293 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:42:00.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:42:00.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:42:00.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:42:00.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:42:00.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:42:00.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:42:00.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:42:00.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:42:00.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:42:00.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:42:00.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:00.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:42:00.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:42:00.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:42:00.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:42:00.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:42:00.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:42:00.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:42:00.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:42:00.297 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.297 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.297 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.297 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3504 4.9 4:96737e19:::smithi08937929-4:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:42:00.298 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:42:00.298 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 84.876216326999995, 2024-04-22T18:42:00.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.951755291, 2024-04-22T18:42:00.298 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.298 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.298 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.298 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.298 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.298 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3504 2024-04-22T18:42:00.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.298 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:42:00.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:42:00.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410861+0000", 2024-04-22T18:42:00.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410863+0000", 2024-04-22T18:42:00.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.635e-06 2024-04-22T18:42:00.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410866+0000", 2024-04-22T18:42:00.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8499999999999998e-06 2024-04-22T18:42:00.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410868+0000", 2024-04-22T18:42:00.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.497e-06 2024-04-22T18:42:00.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615137+0000", 2024-04-22T18:42:00.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20426903499999999 2024-04-22T18:42:00.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615218+0000", 2024-04-22T18:42:00.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0860999999999997e-05 2024-04-22T18:42:00.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.362617+0000", 2024-04-22T18:42:00.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.74739882700000004 2024-04-22T18:42:00.301 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.301 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.301 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.302 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4194 4.b 4:d3068ee3:::smithi08937929-29:head [sparse-read 0~1980000,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:42:00.302 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:42:00.302 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 57.122957255999999, 2024-04-22T18:42:00.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.294164774, 2024-04-22T18:42:00.302 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.302 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.302 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.302 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.302 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.302 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4194 2024-04-22T18:42:00.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.302 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:42:00.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:42:00.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164120+0000", 2024-04-22T18:42:00.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164125+0000", 2024-04-22T18:42:00.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2899999999999996e-06 2024-04-22T18:42:00.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164128+0000", 2024-04-22T18:42:00.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8700000000000002e-06 2024-04-22T18:42:00.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164132+0000", 2024-04-22T18:42:00.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.151e-06 2024-04-22T18:42:00.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164188+0000", 2024-04-22T18:42:00.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6912e-05 2024-04-22T18:42:00.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164201+0000", 2024-04-22T18:42:00.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2466000000000001e-05 2024-04-22T18:42:00.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.242879+0000", 2024-04-22T18:42:00.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0786779710000001 2024-04-22T18:42:00.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:00.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243364+0000", 2024-04-22T18:42:00.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000485081 2024-04-22T18:42:00.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456588+0000", 2024-04-22T18:42:00.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21322370900000001 2024-04-22T18:42:00.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456674+0000", 2024-04-22T18:42:00.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6079e-05 2024-04-22T18:42:00.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.458285+0000", 2024-04-22T18:42:00.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001611716 2024-04-22T18:42:00.307 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.307 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.307 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.307 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4195 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:42:00.307 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:42:00.307 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 57.122939633999998, 2024-04-22T18:42:00.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.395681808, 2024-04-22T18:42:00.307 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.307 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.308 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.308 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.308 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.308 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4195 2024-04-22T18:42:00.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.308 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:42:00.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:42:00.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:42:00.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164141+0000", 2024-04-22T18:42:00.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7160000000000001e-06 2024-04-22T18:42:00.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164147+0000", 2024-04-22T18:42:00.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0509999999999996e-06 2024-04-22T18:42:00.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264244+0000", 2024-04-22T18:42:00.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100097063 2024-04-22T18:42:00.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264319+0000", 2024-04-22T18:42:00.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5729000000000006e-05 2024-04-22T18:42:00.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264331+0000", 2024-04-22T18:42:00.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048999999999999e-05 2024-04-22T18:42:00.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243392+0000", 2024-04-22T18:42:00.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97906041099999996 2024-04-22T18:42:00.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:00.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243451+0000", 2024-04-22T18:42:00.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8717e-05 2024-04-22T18:42:00.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558398+0000", 2024-04-22T18:42:00.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31494718500000002 2024-04-22T18:42:00.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558490+0000", 2024-04-22T18:42:00.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2529999999999997e-05 2024-04-22T18:42:00.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559820+0000", 2024-04-22T18:42:00.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013299060000000001 2024-04-22T18:42:00.312 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.312 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.312 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.313 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4196 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:42:00.313 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:42:00.313 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 57.022800887000002, 2024-04-22T18:42:00.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2963829229999999, 2024-04-22T18:42:00.313 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.313 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.313 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.313 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.313 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.313 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4196 2024-04-22T18:42:00.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.313 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:42:00.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:42:00.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264275+0000", 2024-04-22T18:42:00.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:42:00.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264290+0000", 2024-04-22T18:42:00.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5373e-05 2024-04-22T18:42:00.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264300+0000", 2024-04-22T18:42:00.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.747e-06 2024-04-22T18:42:00.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264314+0000", 2024-04-22T18:42:00.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.396e-05 2024-04-22T18:42:00.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364450+0000", 2024-04-22T18:42:00.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013662500000001 2024-04-22T18:42:00.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364459+0000", 2024-04-22T18:42:00.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0710000000000001e-06 2024-04-22T18:42:00.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245690+0000", 2024-04-22T18:42:00.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88123066400000005 2024-04-22T18:42:00.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:00.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245766+0000", 2024-04-22T18:42:00.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5938000000000004e-05 2024-04-22T18:42:00.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559874+0000", 2024-04-22T18:42:00.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31410827499999999 2024-04-22T18:42:00.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559930+0000", 2024-04-22T18:42:00.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5367e-05 2024-04-22T18:42:00.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.560660+0000", 2024-04-22T18:42:00.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073043800000000003 2024-04-22T18:42:00.318 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.318 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.318 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.318 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4182 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:42:00.318 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:42:00.318 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.359118041999999, 2024-04-22T18:42:00.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2059652709999999, 2024-04-22T18:42:00.318 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.318 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.319 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.319 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.319 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.319 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4182 2024-04-22T18:42:00.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.319 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:42:00.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:42:00.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927959+0000", 2024-04-22T18:42:00.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927962+0000", 2024-04-22T18:42:00.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4269999999999998e-06 2024-04-22T18:42:00.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927968+0000", 2024-04-22T18:42:00.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:42:00.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028037+0000", 2024-04-22T18:42:00.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068544 2024-04-22T18:42:00.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028078+0000", 2024-04-22T18:42:00.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1523999999999997e-05 2024-04-22T18:42:00.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028087+0000", 2024-04-22T18:42:00.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3140000000000004e-06 2024-04-22T18:42:00.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.129951+0000", 2024-04-22T18:42:00.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1018649229999999 2024-04-22T18:42:00.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.130067+0000", 2024-04-22T18:42:00.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115222 2024-04-22T18:42:00.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.133925+0000", 2024-04-22T18:42:00.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00385849 2024-04-22T18:42:00.323 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.323 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.323 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.323 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4183 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:42:00.323 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:42:00.323 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.259027609999997, 2024-04-22T18:42:00.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1063017610000001, 2024-04-22T18:42:00.323 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.323 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.323 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.323 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.324 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.324 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4183 2024-04-22T18:42:00.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.324 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:42:00.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:42:00.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028049+0000", 2024-04-22T18:42:00.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:00.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028055+0000", 2024-04-22T18:42:00.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-04-22T18:42:00.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028064+0000", 2024-04-22T18:42:00.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2830000000000001e-06 2024-04-22T18:42:00.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028071+0000", 2024-04-22T18:42:00.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2740000000000002e-06 2024-04-22T18:42:00.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028126+0000", 2024-04-22T18:42:00.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4771999999999997e-05 2024-04-22T18:42:00.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028134+0000", 2024-04-22T18:42:00.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5599999999999994e-06 2024-04-22T18:42:00.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134109+0000", 2024-04-22T18:42:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.105974864 2024-04-22T18:42:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134169+0000", 2024-04-22T18:42:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0167999999999997e-05 2024-04-22T18:42:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134352+0000", 2024-04-22T18:42:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018293699999999999 2024-04-22T18:42:00.327 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.328 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.328 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.328 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4190 4.c 4:3227431c:::smithi08937929-35:head [sparse-read 0~1258562,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:42:00.328 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:42:00.328 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.259002107000001, 2024-04-22T18:42:00.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1098740030000001, 2024-04-22T18:42:00.328 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.328 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.328 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.328 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.328 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.328 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4190 2024-04-22T18:42:00.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.329 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:42:00.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:42:00.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028075+0000", 2024-04-22T18:42:00.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:00.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028080+0000", 2024-04-22T18:42:00.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.561e-06 2024-04-22T18:42:00.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028085+0000", 2024-04-22T18:42:00.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0150000000000003e-06 2024-04-22T18:42:00.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028090+0000", 2024-04-22T18:42:00.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.169e-06 2024-04-22T18:42:00.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028175+0000", 2024-04-22T18:42:00.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4814999999999993e-05 2024-04-22T18:42:00.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028181+0000", 2024-04-22T18:42:00.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5490000000000003e-06 2024-04-22T18:42:00.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134400+0000", 2024-04-22T18:42:00.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1062186869999999 2024-04-22T18:42:00.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134552+0000", 2024-04-22T18:42:00.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015210500000000001 2024-04-22T18:42:00.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137950+0000", 2024-04-22T18:42:00.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033978210000000001 2024-04-22T18:42:00.332 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.332 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.332 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.333 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4191 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:42:00.333 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:42:00.333 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.258919358999997, 2024-04-22T18:42:00.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.110917953, 2024-04-22T18:42:00.333 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.333 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.333 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.333 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.333 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.333 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4191 2024-04-22T18:42:00.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.333 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:42:00.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:42:00.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028157+0000", 2024-04-22T18:42:00.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:00.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028164+0000", 2024-04-22T18:42:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.511e-06 2024-04-22T18:42:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028172+0000", 2024-04-22T18:42:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0339999999999997e-06 2024-04-22T18:42:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028179+0000", 2024-04-22T18:42:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8630000000000002e-06 2024-04-22T18:42:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028202+0000", 2024-04-22T18:42:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3635999999999999e-05 2024-04-22T18:42:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028209+0000", 2024-04-22T18:42:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4629999999999998e-06 2024-04-22T18:42:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137970+0000", 2024-04-22T18:42:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1097613479999999 2024-04-22T18:42:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.138034+0000", 2024-04-22T18:42:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3551000000000002e-05 2024-04-22T18:42:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.139077+0000", 2024-04-22T18:42:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001042715 2024-04-22T18:42:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4192 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:42:00.337 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:42:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.258894058999999, 2024-04-22T18:42:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.212555542, 2024-04-22T18:42:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4192 2024-04-22T18:42:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:42:00.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:42:00.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028183+0000", 2024-04-22T18:42:00.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028187+0000", 2024-04-22T18:42:00.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6119999999999999e-06 2024-04-22T18:42:00.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028192+0000", 2024-04-22T18:42:00.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3489999999999999e-06 2024-04-22T18:42:00.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028197+0000", 2024-04-22T18:42:00.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7489999999999998e-06 2024-04-22T18:42:00.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028228+0000", 2024-04-22T18:42:00.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1328999999999998e-05 2024-04-22T18:42:00.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028234+0000", 2024-04-22T18:42:00.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1129999999999997e-06 2024-04-22T18:42:00.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239245+0000", 2024-04-22T18:42:00.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.211011032 2024-04-22T18:42:00.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239384+0000", 2024-04-22T18:42:00.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013902000000000001 2024-04-22T18:42:00.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.240739+0000", 2024-04-22T18:42:00.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013550389999999999 2024-04-22T18:42:00.342 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.342 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.342 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.342 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4692 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:42:00.342 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:42:00.343 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.02793818, 2024-04-22T18:42:00.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80332623700000005, 2024-04-22T18:42:00.343 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.343 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.343 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.343 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.343 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.343 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4692 2024-04-22T18:42:00.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.343 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:42:00.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:42:00.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259139+0000", 2024-04-22T18:42:00.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:00.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259145+0000", 2024-04-22T18:42:00.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4860000000000002e-06 2024-04-22T18:42:00.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259151+0000", 2024-04-22T18:42:00.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1160000000000004e-06 2024-04-22T18:42:00.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259156+0000", 2024-04-22T18:42:00.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6489999999999999e-06 2024-04-22T18:42:00.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259191+0000", 2024-04-22T18:42:00.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4737999999999998e-05 2024-04-22T18:42:00.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259197+0000", 2024-04-22T18:42:00.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9160000000000004e-06 2024-04-22T18:42:00.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060062+0000", 2024-04-22T18:42:00.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80086472900000005 2024-04-22T18:42:00.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060225+0000", 2024-04-22T18:42:00.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000162771 2024-04-22T18:42:00.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062466+0000", 2024-04-22T18:42:00.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410569999999999 2024-04-22T18:42:00.347 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.347 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.347 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.347 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4693 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:42:00.347 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:42:00.347 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.027915894000003, 2024-04-22T18:42:00.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80370012999999996, 2024-04-22T18:42:00.347 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.348 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.348 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.348 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.348 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.348 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4693 2024-04-22T18:42:00.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.348 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:42:00.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:42:00.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259161+0000", 2024-04-22T18:42:00.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:00.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259164+0000", 2024-04-22T18:42:00.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-06 2024-04-22T18:42:00.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259169+0000", 2024-04-22T18:42:00.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6260000000000003e-06 2024-04-22T18:42:00.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259174+0000", 2024-04-22T18:42:00.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7659999999999998e-06 2024-04-22T18:42:00.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259231+0000", 2024-04-22T18:42:00.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6827000000000003e-05 2024-04-22T18:42:00.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259239+0000", 2024-04-22T18:42:00.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6889999999999993e-06 2024-04-22T18:42:00.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062492+0000", 2024-04-22T18:42:00.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80325246699999997 2024-04-22T18:42:00.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062573+0000", 2024-04-22T18:42:00.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1508000000000006e-05 2024-04-22T18:42:00.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062862+0000", 2024-04-22T18:42:00.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028893499999999998 2024-04-22T18:42:00.352 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.352 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.352 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.352 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4694 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:42:00.352 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:42:00.352 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.027899613000002, 2024-04-22T18:42:00.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.90474754099999999, 2024-04-22T18:42:00.352 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.352 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.352 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.353 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.353 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.353 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4694 2024-04-22T18:42:00.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.353 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:42:00.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:42:00.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:42:00.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259182+0000", 2024-04-22T18:42:00.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8700000000000002e-06 2024-04-22T18:42:00.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259187+0000", 2024-04-22T18:42:00.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1669999999999998e-06 2024-04-22T18:42:00.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259191+0000", 2024-04-22T18:42:00.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6210000000000003e-06 2024-04-22T18:42:00.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.359347+0000", 2024-04-22T18:42:00.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100155518 2024-04-22T18:42:00.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.359356+0000", 2024-04-22T18:42:00.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6519999999999995e-06 2024-04-22T18:42:00.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163126+0000", 2024-04-22T18:42:00.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.803770816 2024-04-22T18:42:00.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163217+0000", 2024-04-22T18:42:00.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0598000000000002e-05 2024-04-22T18:42:00.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163926+0000", 2024-04-22T18:42:00.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000708856 2024-04-22T18:42:00.357 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.357 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.357 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.357 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4791 4.7 4:e117a562:::smithi08937929-33:head [write 591712~786646] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:42:00.357 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.734465+0000", 2024-04-22T18:42:00.357 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.552613159, 2024-04-22T18:42:00.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4316495570000001, 2024-04-22T18:42:00.357 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.357 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.357 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.357 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.357 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.357 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4791 2024-04-22T18:42:00.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.358 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734465+0000", 2024-04-22T18:42:00.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734465+0000", 2024-04-22T18:42:00.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734460+0000", 2024-04-22T18:42:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999957 2024-04-22T18:42:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734848+0000", 2024-04-22T18:42:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000387353 2024-04-22T18:42:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734865+0000", 2024-04-22T18:42:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7486000000000002e-05 2024-04-22T18:42:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734875+0000", 2024-04-22T18:42:00.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-06 2024-04-22T18:42:00.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761489+0000", 2024-04-22T18:42:00.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026613405 2024-04-22T18:42:00.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761495+0000", 2024-04-22T18:42:00.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4350000000000002e-06 2024-04-22T18:42:00.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.961065+0000", 2024-04-22T18:42:00.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.199569608 2024-04-22T18:42:00.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.961155+0000", 2024-04-22T18:42:00.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0109000000000001e-05 2024-04-22T18:42:00.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:42:00.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.961254+0000", 2024-04-22T18:42:00.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9072000000000003e-05 2024-04-22T18:42:00.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:00.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.964390+0000", 2024-04-22T18:42:00.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031362009999999999 2024-04-22T18:42:00.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:00.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166027+0000", 2024-04-22T18:42:00.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.201636589 2024-04-22T18:42:00.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166075+0000", 2024-04-22T18:42:00.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7998999999999999e-05 2024-04-22T18:42:00.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166114+0000", 2024-04-22T18:42:00.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9638999999999998e-05 2024-04-22T18:42:00.363 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.363 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.363 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.363 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4792 4.7 4:e117a562:::smithi08937929-33:head [write 1943568~668798] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:42:00.363 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742160+0000", 2024-04-22T18:42:00.363 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.544918044999999, 2024-04-22T18:42:00.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.42404771, 2024-04-22T18:42:00.363 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.364 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.364 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4792 2024-04-22T18:42:00.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.364 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742160+0000", 2024-04-22T18:42:00.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742160+0000", 2024-04-22T18:42:00.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742158+0000", 2024-04-22T18:42:00.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:42:00.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742473+0000", 2024-04-22T18:42:00.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000315192 2024-04-22T18:42:00.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742485+0000", 2024-04-22T18:42:00.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1795e-05 2024-04-22T18:42:00.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742491+0000", 2024-04-22T18:42:00.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8760000000000002e-06 2024-04-22T18:42:00.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761511+0000", 2024-04-22T18:42:00.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019020760000000001 2024-04-22T18:42:00.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761515+0000", 2024-04-22T18:42:00.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.625e-06 2024-04-22T18:42:00.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.962761+0000", 2024-04-22T18:42:00.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.201246021 2024-04-22T18:42:00.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.962837+0000", 2024-04-22T18:42:00.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6025999999999993e-05 2024-04-22T18:42:00.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:42:00.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.962944+0000", 2024-04-22T18:42:00.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106949 2024-04-22T18:42:00.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:00.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.965124+0000", 2024-04-22T18:42:00.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021803230000000001 2024-04-22T18:42:00.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:00.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166154+0000", 2024-04-22T18:42:00.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20102966 2024-04-22T18:42:00.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166185+0000", 2024-04-22T18:42:00.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0713999999999998e-05 2024-04-22T18:42:00.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166208+0000", 2024-04-22T18:42:00.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2969999999999999e-05 2024-04-22T18:42:00.369 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.369 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.369 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.369 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4793 4.7 4:e117a562:::smithi08937929-33:head [setxattr _header (56) in=7b,truncate 2991712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:42:00.369 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742495+0000", 2024-04-22T18:42:00.370 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.544583133, 2024-04-22T18:42:00.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5238806949999999, 2024-04-22T18:42:00.370 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.370 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.370 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.370 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.370 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.370 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4793 2024-04-22T18:42:00.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.370 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742495+0000", 2024-04-22T18:42:00.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742495+0000", 2024-04-22T18:42:00.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742494+0000", 2024-04-22T18:42:00.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742498+0000", 2024-04-22T18:42:00.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3790000000000002e-06 2024-04-22T18:42:00.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742501+0000", 2024-04-22T18:42:00.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9900000000000002e-06 2024-04-22T18:42:00.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742504+0000", 2024-04-22T18:42:00.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:42:00.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761529+0000", 2024-04-22T18:42:00.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019025301000000001 2024-04-22T18:42:00.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761532+0000", 2024-04-22T18:42:00.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4589999999999998e-06 2024-04-22T18:42:00.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.963997+0000", 2024-04-22T18:42:00.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2024646560000001 2024-04-22T18:42:00.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.964075+0000", 2024-04-22T18:42:00.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8015000000000004e-05 2024-04-22T18:42:00.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:42:00.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.964176+0000", 2024-04-22T18:42:00.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010080499999999999 2024-04-22T18:42:00.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:00.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.965235+0000", 2024-04-22T18:42:00.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00105959 2024-04-22T18:42:00.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:00.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266295+0000", 2024-04-22T18:42:00.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30105968599999999 2024-04-22T18:42:00.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266330+0000", 2024-04-22T18:42:00.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5219999999999998e-05 2024-04-22T18:42:00.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266376+0000", 2024-04-22T18:42:00.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5262000000000001e-05 2024-04-22T18:42:00.375 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.375 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.375 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.376 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4794 4.7 4:e117a562:::smithi08937929-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:42:00.376 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:42:00.376 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.54457202, 2024-04-22T18:42:00.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.524010123, 2024-04-22T18:42:00.376 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.376 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.376 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.376 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.376 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.376 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4794 2024-04-22T18:42:00.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.376 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:42:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:42:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:42:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742507+0000", 2024-04-22T18:42:00.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.637e-06 2024-04-22T18:42:00.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742511+0000", 2024-04-22T18:42:00.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9890000000000003e-06 2024-04-22T18:42:00.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742514+0000", 2024-04-22T18:42:00.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.88e-06 2024-04-22T18:42:00.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.775057+0000", 2024-04-22T18:42:00.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.032543012000000003 2024-04-22T18:42:00.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.775069+0000", 2024-04-22T18:42:00.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2184000000000001e-05 2024-04-22T18:42:00.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.064059+0000", 2024-04-22T18:42:00.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.288990157 2024-04-22T18:42:00.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:00.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.064127+0000", 2024-04-22T18:42:00.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7741000000000001e-05 2024-04-22T18:42:00.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266405+0000", 2024-04-22T18:42:00.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20227735499999999 2024-04-22T18:42:00.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266458+0000", 2024-04-22T18:42:00.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3430000000000002e-05 2024-04-22T18:42:00.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266516+0000", 2024-04-22T18:42:00.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8038000000000002e-05 2024-04-22T18:42:00.381 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.381 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.381 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.381 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4795 4.1f 4.d9a85c1f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:42:00.381 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:42:00.381 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.544561649999999, 2024-04-22T18:42:00.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.118060236, 2024-04-22T18:42:00.381 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.381 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:42:00.381 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.381 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.382 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.382 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4795 2024-04-22T18:42:00.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.382 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:42:00.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:42:00.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:42:00.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742518+0000", 2024-04-22T18:42:00.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.584e-06 2024-04-22T18:42:00.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742520+0000", 2024-04-22T18:42:00.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7140000000000002e-06 2024-04-22T18:42:00.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742523+0000", 2024-04-22T18:42:00.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3609999999999999e-06 2024-04-22T18:42:00.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761544+0000", 2024-04-22T18:42:00.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019020925000000001 2024-04-22T18:42:00.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761547+0000", 2024-04-22T18:42:00.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7859999999999998e-06 2024-04-22T18:42:00.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.860553+0000", 2024-04-22T18:42:00.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0990060859999999 2024-04-22T18:42:00.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.860577+0000", 2024-04-22T18:42:00.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3064000000000001e-05 2024-04-22T18:42:00.385 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.385 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.385 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.386 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:5189 4.19 4:9ea6f51f:::smithi08937929-34:head [call cas.references_chunk in=19b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e174)", 2024-04-22T18:42:00.386 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:50.743728+0000", 2024-04-22T18:42:00.386 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.5433497280000008, 2024-04-22T18:42:00.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5121397000000001, 2024-04-22T18:42:00.386 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.386 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.386 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.386 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.386 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.386 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5189 2024-04-22T18:42:00.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.386 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743728+0000", 2024-04-22T18:42:00.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743728+0000", 2024-04-22T18:42:00.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743726+0000", 2024-04-22T18:42:00.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:42:00.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743736+0000", 2024-04-22T18:42:00.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9890000000000004e-06 2024-04-22T18:42:00.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743750+0000", 2024-04-22T18:42:00.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3533000000000001e-05 2024-04-22T18:42:00.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743760+0000", 2024-04-22T18:42:00.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0121e-05 2024-04-22T18:42:00.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743807+0000", 2024-04-22T18:42:00.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6972999999999999e-05 2024-04-22T18:42:00.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:42:00.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743812+0000", 2024-04-22T18:42:00.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:42:00.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.845187+0000", 2024-04-22T18:42:00.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101374929 2024-04-22T18:42:00.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.845194+0000", 2024-04-22T18:42:00.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7719999999999997e-06 2024-04-22T18:42:00.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:52.255561+0000", 2024-04-22T18:42:00.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4103678770000001 2024-04-22T18:42:00.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:52.255766+0000", 2024-04-22T18:42:00.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000204093 2024-04-22T18:42:00.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:52.255868+0000", 2024-04-22T18:42:00.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102332 2024-04-22T18:42:00.391 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.391 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.391 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.391 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.391 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.391 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:00.392 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:42:00.393 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:42:00.393+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:42:00.493 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:42:00.493+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:42:00.494 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:00.494 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:42:00.494 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:42:00.494 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:00.503 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:42:00.593 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:42:00.594+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:42:00.644 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:00.644 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:42:00.644 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:42:00.644 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:42:00.644 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:00.653 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:42:00.694 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:42:00.694+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:42:00.779 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:42:00.779+0000 7f8607f36640 -1 osd.3 183 handle_osd_map injecting map crc failure 2024-04-22T18:42:00.780 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:42:00.779+0000 7f3e7be88640 -1 osd.0 183 handle_osd_map injecting map crc failure 2024-04-22T18:42:00.795 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:42:00.795+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:42:00.797 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-22T18:42:00.799 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:00.799 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:42:00.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:42:00.799 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:42:00.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.799 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:42:00.799 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:42:00.799 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 176.33796170599999, 2024-04-22T18:42:00.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:42:00.799 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.799 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.799 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:42:00.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:42:00.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:42:00.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:00.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:42:00.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:42:00.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:42:00.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:42:00.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:42:00.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:42:00.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:42:00.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:42:00.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:42:00.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:42:00.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:42:00.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:42:00.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:42:00.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:42:00.803 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.803 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.803 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.803 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:42:00.803 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:42:00.803 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 172.461287884, 2024-04-22T18:42:00.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:42:00.804 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.804 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.804 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:42:00.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:42:00.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:42:00.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:42:00.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:42:00.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:42:00.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:42:00.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:42:00.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:42:00.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:42:00.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:42:00.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:42:00.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:42:00.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:42:00.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:42:00.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:42:00.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:42:00.807 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.807 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.807 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.807 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:42:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:42:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 116.25648436100001, 2024-04-22T18:42:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:42:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:42:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:42:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:42:00.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:42:00.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:42:00.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:42:00.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:42:00.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:42:00.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:42:00.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:42:00.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:42:00.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:42:00.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:42:00.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:42:00.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:42:00.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:42:00.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:42:00.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:42:00.811 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.811 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.811 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.812 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3451 4.d 4:bee1184f:::smithi08937929-7:head [read 0~3613505,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:42:00.812 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:42:00.812 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 88.871954033999998, 2024-04-22T18:42:00.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20164568099999999, 2024-04-22T18:42:00.812 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.812 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.812 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.812 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.812 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.812 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3451 2024-04-22T18:42:00.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.812 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:42:00.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:42:00.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:42:00.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927183+0000", 2024-04-22T18:42:00.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1110000000000001e-06 2024-04-22T18:42:00.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927188+0000", 2024-04-22T18:42:00.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1569999999999999e-06 2024-04-22T18:42:00.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927192+0000", 2024-04-22T18:42:00.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4710000000000001e-06 2024-04-22T18:42:00.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027382+0000", 2024-04-22T18:42:00.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100189341 2024-04-22T18:42:00.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:42:00.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027387+0000", 2024-04-22T18:42:00.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8400000000000002e-06 2024-04-22T18:42:00.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127638+0000", 2024-04-22T18:42:00.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10025102499999999 2024-04-22T18:42:00.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127740+0000", 2024-04-22T18:42:00.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102749 2024-04-22T18:42:00.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.128825+0000", 2024-04-22T18:42:00.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001084593 2024-04-22T18:42:00.816 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.816 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.816 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.817 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:151 3.0 e114/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 114'213, mlcod=110'210)", 2024-04-22T18:42:00.817 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:42:00.817 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.281485950999993, 2024-04-22T18:42:00.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20074001299999999, 2024-04-22T18:42:00.817 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.817 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.817 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:42:00.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:42:00.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517640+0000", 2024-04-22T18:42:00.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999928 2024-04-22T18:42:00.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517652+0000", 2024-04-22T18:42:00.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1824000000000001e-05 2024-04-22T18:42:00.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517660+0000", 2024-04-22T18:42:00.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1419999999999993e-06 2024-04-22T18:42:00.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617729+0000", 2024-04-22T18:42:00.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068723 2024-04-22T18:42:00.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617770+0000", 2024-04-22T18:42:00.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1391999999999999e-05 2024-04-22T18:42:00.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617793+0000", 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2807999999999999e-05 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718361+0000", 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100567473 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718387+0000", 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6826e-05 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.821 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3862 4.d 4:bee1184f:::smithi08937929-7:head [write 3301257~278667] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:42:00.821 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:42:00.821 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.536583680999996, 2024-04-22T18:42:00.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301875958, 2024-04-22T18:42:00.821 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.821 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.821 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3862 2024-04-22T18:42:00.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.821 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262548+0000", 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262622+0000", 2024-04-22T18:42:00.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3747999999999999e-05 2024-04-22T18:42:00.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262632+0000", 2024-04-22T18:42:00.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0767e-05 2024-04-22T18:42:00.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362705+0000", 2024-04-22T18:42:00.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073157 2024-04-22T18:42:00.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:42:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2846999999999997e-05 2024-04-22T18:42:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362853+0000", 2024-04-22T18:42:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4498000000000002e-05 2024-04-22T18:42:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362992+0000", 2024-04-22T18:42:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013879699999999999 2024-04-22T18:42:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:00.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.366534+0000", 2024-04-22T18:42:00.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035427969999999999 2024-04-22T18:42:00.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:00.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564353+0000", 2024-04-22T18:42:00.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.197818893 2024-04-22T18:42:00.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564385+0000", 2024-04-22T18:42:00.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1717000000000001e-05 2024-04-22T18:42:00.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564426+0000", 2024-04-22T18:42:00.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0654000000000001e-05 2024-04-22T18:42:00.826 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.826 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.826 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.826 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3863 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3579924] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:42:00.826 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:42:00.826 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.436416288999993, 2024-04-22T18:42:00.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20206560900000001, 2024-04-22T18:42:00.826 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3863 2024-04-22T18:42:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:42:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:42:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362715+0000", 2024-04-22T18:42:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362725+0000", 2024-04-22T18:42:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.006e-05 2024-04-22T18:42:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362732+0000", 2024-04-22T18:42:00.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6780000000000003e-06 2024-04-22T18:42:00.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362737+0000", 2024-04-22T18:42:00.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2089999999999996e-06 2024-04-22T18:42:00.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462875+0000", 2024-04-22T18:42:00.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013834100000001 2024-04-22T18:42:00.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462963+0000", 2024-04-22T18:42:00.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7792000000000002e-05 2024-04-22T18:42:00.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:00.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463127+0000", 2024-04-22T18:42:00.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000163688 2024-04-22T18:42:00.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:00.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463692+0000", 2024-04-22T18:42:00.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056466299999999997 2024-04-22T18:42:00.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:00.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564720+0000", 2024-04-22T18:42:00.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10102826199999999 2024-04-22T18:42:00.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564752+0000", 2024-04-22T18:42:00.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1936000000000001e-05 2024-04-22T18:42:00.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564783+0000", 2024-04-22T18:42:00.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0919e-05 2024-04-22T18:42:00.832 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.832 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.832 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.832 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3864 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:42:00.832 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:42:00.832 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.436382905000002, 2024-04-22T18:42:00.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20218423599999999, 2024-04-22T18:42:00.832 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.832 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.832 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.832 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.832 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.832 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3864 2024-04-22T18:42:00.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.833 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:42:00.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:42:00.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:42:00.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362753+0000", 2024-04-22T18:42:00.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6989999999999999e-06 2024-04-22T18:42:00.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:42:00.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:42:00.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462871+0000", 2024-04-22T18:42:00.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100113641 2024-04-22T18:42:00.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463385+0000", 2024-04-22T18:42:00.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051307100000000003 2024-04-22T18:42:00.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:00.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463457+0000", 2024-04-22T18:42:00.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2778e-05 2024-04-22T18:42:00.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564814+0000", 2024-04-22T18:42:00.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10135696299999999 2024-04-22T18:42:00.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564868+0000", 2024-04-22T18:42:00.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3872000000000002e-05 2024-04-22T18:42:00.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564935+0000", 2024-04-22T18:42:00.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6545999999999996e-05 2024-04-22T18:42:00.836 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.836 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.836 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.837 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4030 4.d 4:bee1184f:::smithi08937929-7:head [write 615222~472039] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:42:00.837 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:42:00.837 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.839934002999996, 2024-04-22T18:42:00.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109028217, 2024-04-22T18:42:00.837 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.837 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.837 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.837 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.837 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.837 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4030 2024-04-22T18:42:00.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.837 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:42:00.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:42:00.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959177+0000", 2024-04-22T18:42:00.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999776 2024-04-22T18:42:00.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959331+0000", 2024-04-22T18:42:00.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015334600000000001 2024-04-22T18:42:00.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959358+0000", 2024-04-22T18:42:00.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7220999999999999e-05 2024-04-22T18:42:00.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959371+0000", 2024-04-22T18:42:00.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3489000000000001e-05 2024-04-22T18:42:00.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059523+0000", 2024-04-22T18:42:00.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100152059 2024-04-22T18:42:00.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059588+0000", 2024-04-22T18:42:00.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4493999999999996e-05 2024-04-22T18:42:00.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:00.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059694+0000", 2024-04-22T18:42:00.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010584199999999999 2024-04-22T18:42:00.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:00.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068082+0000", 2024-04-22T18:42:00.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083880589999999998 2024-04-22T18:42:00.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:00.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068138+0000", 2024-04-22T18:42:00.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6196999999999997e-05 2024-04-22T18:42:00.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068185+0000", 2024-04-22T18:42:00.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7318000000000003e-05 2024-04-22T18:42:00.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068228+0000", 2024-04-22T18:42:00.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2376999999999998e-05 2024-04-22T18:42:00.842 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.842 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.842 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.842 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4031 4.d 4:bee1184f:::smithi08937929-7:head [write 1580652~489442] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:42:00.842 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:42:00.843 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.834633010999994, 2024-04-22T18:42:00.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20571598299999999, 2024-04-22T18:42:00.843 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.843 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.843 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.843 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.843 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.843 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4031 2024-04-22T18:42:00.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.843 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:42:00.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:42:00.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964480+0000", 2024-04-22T18:42:00.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999795 2024-04-22T18:42:00.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964628+0000", 2024-04-22T18:42:00.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148109 2024-04-22T18:42:00.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964651+0000", 2024-04-22T18:42:00.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3215999999999999e-05 2024-04-22T18:42:00.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964664+0000", 2024-04-22T18:42:00.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2673e-05 2024-04-22T18:42:00.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061135+0000", 2024-04-22T18:42:00.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096470822999999997 2024-04-22T18:42:00.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061228+0000", 2024-04-22T18:42:00.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3459000000000001e-05 2024-04-22T18:42:00.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:00.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061376+0000", 2024-04-22T18:42:00.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148128 2024-04-22T18:42:00.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:00.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068085+0000", 2024-04-22T18:42:00.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.006708632 2024-04-22T18:42:00.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:00.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170038+0000", 2024-04-22T18:42:00.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101952716 2024-04-22T18:42:00.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170173+0000", 2024-04-22T18:42:00.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013508699999999999 2024-04-22T18:42:00.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170216+0000", 2024-04-22T18:42:00.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3498999999999998e-05 2024-04-22T18:42:00.848 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.848 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.848 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.848 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4032 4.d 4:bee1184f:::smithi08937929-7:head [write 2768826~626587] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:42:00.848 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:42:00.848 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.827427904000004, 2024-04-22T18:42:00.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19941629799999999, 2024-04-22T18:42:00.848 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.848 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.848 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.848 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4032 2024-04-22T18:42:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:42:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:42:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971703+0000", 2024-04-22T18:42:00.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:42:00.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971996+0000", 2024-04-22T18:42:00.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029351300000000002 2024-04-22T18:42:00.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972012+0000", 2024-04-22T18:42:00.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5569e-05 2024-04-22T18:42:00.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972019+0000", 2024-04-22T18:42:00.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7950000000000008e-06 2024-04-22T18:42:00.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.063940+0000", 2024-04-22T18:42:00.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091920162 2024-04-22T18:42:00.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064021+0000", 2024-04-22T18:42:00.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1865000000000003e-05 2024-04-22T18:42:00.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:00.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064145+0000", 2024-04-22T18:42:00.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123893 2024-04-22T18:42:00.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:00.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068317+0000", 2024-04-22T18:42:00.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041720799999999999 2024-04-22T18:42:00.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:00.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171042+0000", 2024-04-22T18:42:00.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102724893 2024-04-22T18:42:00.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171085+0000", 2024-04-22T18:42:00.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2619e-05 2024-04-22T18:42:00.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171122+0000", 2024-04-22T18:42:00.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6838999999999997e-05 2024-04-22T18:42:00.853 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.853 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.853 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.853 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4033 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3815222] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:42:00.854 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:42:00.854 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.827104422000005, 2024-04-22T18:42:00.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297586038, 2024-04-22T18:42:00.854 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.854 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.854 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.854 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.854 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.854 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4033 2024-04-22T18:42:00.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.854 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:42:00.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:42:00.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972028+0000", 2024-04-22T18:42:00.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972032+0000", 2024-04-22T18:42:00.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8330000000000004e-06 2024-04-22T18:42:00.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972035+0000", 2024-04-22T18:42:00.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:42:00.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972038+0000", 2024-04-22T18:42:00.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9579999999999999e-06 2024-04-22T18:42:00.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067511+0000", 2024-04-22T18:42:00.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.095472308000000006 2024-04-22T18:42:00.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067607+0000", 2024-04-22T18:42:00.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6546999999999996e-05 2024-04-22T18:42:00.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:00.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067776+0000", 2024-04-22T18:42:00.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000169202 2024-04-22T18:42:00.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:00.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068470+0000", 2024-04-22T18:42:00.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069403100000000001 2024-04-22T18:42:00.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:00.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269540+0000", 2024-04-22T18:42:00.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20106934000000001 2024-04-22T18:42:00.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269574+0000", 2024-04-22T18:42:00.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4332000000000003e-05 2024-04-22T18:42:00.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269615+0000", 2024-04-22T18:42:00.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0988000000000001e-05 2024-04-22T18:42:00.859 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.859 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.859 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.859 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4034 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:42:00.859 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:42:00.859 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.827092785000005, 2024-04-22T18:42:00.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39788621000000002, 2024-04-22T18:42:00.859 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.859 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.859 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4034 2024-04-22T18:42:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:42:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:42:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972040+0000", 2024-04-22T18:42:00.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972042+0000", 2024-04-22T18:42:00.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.519e-06 2024-04-22T18:42:00.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972045+0000", 2024-04-22T18:42:00.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:42:00.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972048+0000", 2024-04-22T18:42:00.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.773e-06 2024-04-22T18:42:00.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068266+0000", 2024-04-22T18:42:00.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096218029999999996 2024-04-22T18:42:00.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:00.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068308+0000", 2024-04-22T18:42:00.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.21e-05 2024-04-22T18:42:00.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369703+0000", 2024-04-22T18:42:00.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301395578 2024-04-22T18:42:00.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369802+0000", 2024-04-22T18:42:00.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8788000000000004e-05 2024-04-22T18:42:00.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369927+0000", 2024-04-22T18:42:00.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012478200000000001 2024-04-22T18:42:00.864 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.864 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.864 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.864 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(4.1 124/123 [PushOp(4:83c955bf:::smithi08937929-28:head, version: 115'157, data_included: [1890685~548185,2699879~469889], data_size: 1018074, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:83c955bf:::smithi08937929-28:head@115'157, size: 3630345, copy_subset: [0~3630345], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3630345, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1890685, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-22T18:42:00.864 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:42:00.864 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 62.160524676999998, 2024-04-22T18:42:00.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20293218800000001, 2024-04-22T18:42:00.864 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.864 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.865 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:42:00.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:42:00.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638608+0000", 2024-04-22T18:42:00.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:00.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639519+0000", 2024-04-22T18:42:00.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091196199999999999 2024-04-22T18:42:00.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639537+0000", 2024-04-22T18:42:00.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7455000000000001e-05 2024-04-22T18:42:00.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.739639+0000", 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100101942 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840271+0000", 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10063235299999999 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840281+0000", 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6509999999999996e-06 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.841541+0000", 2024-04-22T18:42:00.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012600230000000001 2024-04-22T18:42:00.868 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.868 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.868 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.868 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4205 4.1 4:83c955bf:::smithi08937929-28:head [read 3065344~859648] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:42:00.868 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:42:00.868 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.871602836000001, 2024-04-22T18:42:00.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.303433565, 2024-04-22T18:42:00.868 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.868 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:00.868 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:00.868 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:00.868 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:00.868 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4205 2024-04-22T18:42:00.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.869 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:42:00.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:42:00.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927528+0000", 2024-04-22T18:42:00.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:42:00.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927535+0000", 2024-04-22T18:42:00.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1049999999999997e-06 2024-04-22T18:42:00.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927550+0000", 2024-04-22T18:42:00.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4365000000000001e-05 2024-04-22T18:42:00.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927560+0000", 2024-04-22T18:42:00.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0360000000000001e-05 2024-04-22T18:42:00.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028542+0000", 2024-04-22T18:42:00.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100981837 2024-04-22T18:42:00.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:00.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028551+0000", 2024-04-22T18:42:00.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6260000000000001e-06 2024-04-22T18:42:00.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229700+0000", 2024-04-22T18:42:00.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.201149453 2024-04-22T18:42:00.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229861+0000", 2024-04-22T18:42:00.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160605 2024-04-22T18:42:00.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.230964+0000", 2024-04-22T18:42:00.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00110351 2024-04-22T18:42:00.872 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.872 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.873 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.873 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:359 3.0 e133/63 3:351d2474:::ba270d71d9c7ccc8d5c74eded70a7073f0e08c1ce33f7199f7c9db1158888734:head v 133'233, mlcod=132'228)", 2024-04-22T18:42:00.873 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:42:00.873 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.896808409000002, 2024-04-22T18:42:00.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20122315599999999, 2024-04-22T18:42:00.873 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.873 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.873 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:42:00.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:42:00.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902324+0000", 2024-04-22T18:42:00.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:00.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902347+0000", 2024-04-22T18:42:00.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3577000000000001e-05 2024-04-22T18:42:00.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902356+0000", 2024-04-22T18:42:00.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8179999999999993e-06 2024-04-22T18:42:00.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.002447+0000", 2024-04-22T18:42:00.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100090255 2024-04-22T18:42:00.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102568+0000", 2024-04-22T18:42:00.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100121611 2024-04-22T18:42:00.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102592+0000", 2024-04-22T18:42:00.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3515000000000001e-05 2024-04-22T18:42:00.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103514+0000", 2024-04-22T18:42:00.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000922174 2024-04-22T18:42:00.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103548+0000", 2024-04-22T18:42:00.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4316999999999997e-05 2024-04-22T18:42:00.876 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.877 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.877 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.877 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:4289 3.0 e134/63 3:2a3d8f37:::9341e24c4a92f7ec719282d2faa5711f9644dee45a94ba5b4c4c1071ddc0e536:head v 134'235, mlcod=133'233)", 2024-04-22T18:42:00.877 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:42:00.877 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.952238897000001, 2024-04-22T18:42:00.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10669205499999999, 2024-04-22T18:42:00.877 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.877 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.877 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:42:00.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:42:00.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846891+0000", 2024-04-22T18:42:00.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:42:00.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847884+0000", 2024-04-22T18:42:00.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00099224800000000004 2024-04-22T18:42:00.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847900+0000", 2024-04-22T18:42:00.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6154000000000002e-05 2024-04-22T18:42:00.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.947999+0000", 2024-04-22T18:42:00.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009889199999999 2024-04-22T18:42:00.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948046+0000", 2024-04-22T18:42:00.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7617000000000002e-05 2024-04-22T18:42:00.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948072+0000", 2024-04-22T18:42:00.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5559999999999999e-05 2024-04-22T18:42:00.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953539+0000", 2024-04-22T18:42:00.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054668599999999996 2024-04-22T18:42:00.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953587+0000", 2024-04-22T18:42:00.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7781999999999998e-05 2024-04-22T18:42:00.881 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.881 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.881 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.881 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.7.13:94 3.0 e147/63 3:2812810a:::007b860ed7f2dc72f975d8732f59ce14b4563df44fd56cdf3a173500adaeb557:head v 147'260, mlcod=147'257)", 2024-04-22T18:42:00.881 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:42:00.881 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.365845690999997, 2024-04-22T18:42:00.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.200869664, 2024-04-22T18:42:00.881 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.881 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.881 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:42:00.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:42:00.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433287+0000", 2024-04-22T18:42:00.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:00.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433295+0000", 2024-04-22T18:42:00.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9929999999999995e-06 2024-04-22T18:42:00.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433302+0000", 2024-04-22T18:42:00.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2740000000000002e-06 2024-04-22T18:42:00.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533368+0000", 2024-04-22T18:42:00.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100066239 2024-04-22T18:42:00.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533415+0000", 2024-04-22T18:42:00.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6669000000000003e-05 2024-04-22T18:42:00.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533442+0000", 2024-04-22T18:42:00.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7143000000000001e-05 2024-04-22T18:42:00.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.634126+0000", 2024-04-22T18:42:00.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10068434900000001 2024-04-22T18:42:00.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.634157+0000", 2024-04-22T18:42:00.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1016000000000003e-05 2024-04-22T18:42:00.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.885 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.885 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:198 3.0 e147/63 3:2be6c676:::11b6849070c44d718872a26a0dc7d7224d7a6a98b28e0fde07a9f66143e32de6:head v 147'266, mlcod=147'257)", 2024-04-22T18:42:00.885 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:42:00.885 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.265073596999997, 2024-04-22T18:42:00.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106653649, 2024-04-22T18:42:00.885 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.886 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.886 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:42:00.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:42:00.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534059+0000", 2024-04-22T18:42:00.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534082+0000", 2024-04-22T18:42:00.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2291999999999999e-05 2024-04-22T18:42:00.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534086+0000", 2024-04-22T18:42:00.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1019999999999999e-06 2024-04-22T18:42:00.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534088+0000", 2024-04-22T18:42:00.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.525e-06 2024-04-22T18:42:00.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.635447+0000", 2024-04-22T18:42:00.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101358581 2024-04-22T18:42:00.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.635462+0000", 2024-04-22T18:42:00.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5327000000000001e-05 2024-04-22T18:42:00.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.640689+0000", 2024-04-22T18:42:00.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052266329999999996 2024-04-22T18:42:00.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.640713+0000", 2024-04-22T18:42:00.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4630999999999998e-05 2024-04-22T18:42:00.889 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.889 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.889 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.889 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:200 3.0 e147/63 3:19f9e72c:::45a155ef5bb5a968c7feaf289313c887c41e2de64b61f8816d05ad72d02346d9:head v 147'267, mlcod=147'257)", 2024-04-22T18:42:00.889 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:42:00.889 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.26504284, 2024-04-22T18:42:00.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109105273, 2024-04-22T18:42:00.890 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:00.890 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:00.890 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:00.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:00.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:42:00.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:00.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:42:00.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:00.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:00.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534090+0000", 2024-04-22T18:42:00.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:00.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:00.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534228+0000", 2024-04-22T18:42:00.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013768600000000001 2024-04-22T18:42:00.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:00.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534232+0000", 2024-04-22T18:42:00.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4900000000000002e-06 2024-04-22T18:42:00.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:00.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534235+0000", 2024-04-22T18:42:00.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.61e-06 2024-04-22T18:42:00.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:00.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.641544+0000", 2024-04-22T18:42:00.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107308717 2024-04-22T18:42:00.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:00.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.641557+0000", 2024-04-22T18:42:00.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2811e-05 2024-04-22T18:42:00.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:00.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.643173+0000", 2024-04-22T18:42:00.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00161608 2024-04-22T18:42:00.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:00.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:00.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:00.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.643196+0000", 2024-04-22T18:42:00.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3136e-05 2024-04-22T18:42:00.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.893 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:00.893 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:00.893 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:00.895 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:42:00.936 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:42:00.936+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:42:00.997 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:42:00.997+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:42:01.039 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:01.039 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:42:01.039 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:42:01.040 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:01.049 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:42:01.133 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:42:01.134+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:42:01.137 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T18:42:01.194 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:01.194 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:42:01.194 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:42:01.194 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:42:01.194 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:01.198 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:42:01.199+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:42:01.203 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:42:01.257 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:42:01.257 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:42:01.274 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:42:01.274 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:42:01.290 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:42:01.290 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:42:01.299 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:42:01.299+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:42:01.306 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:42:01.306 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:42:01.309 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:01.309 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:01.318 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:42:01.323 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:42:01.324 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:42:01.340 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:42:01.340 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:42:01.357 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:42:01.357 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:42:01.374 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:42:01.374 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:42:01.425 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:42:01.426+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:42:01.432 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:01.432 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:01.441 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:42:01.500 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:42:01.501+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:42:01.544 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:01.544 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:01.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:42:01.601 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:42:01.602+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:42:01.653 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:01.654 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:01.662 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:42:01.706 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:42:01.707+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:42:01.764 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:01.764 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:01.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:42:01.786 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-04-22T18:42:01.842 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:42:01.842+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:42:01.903 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:42:01.904+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:42:01.914 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:01.914 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:01.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:42:02.005 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:42:02.005+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:42:02.066 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:02.067 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:02.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:42:02.122 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:42:02.123+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:42:02.176 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:02.177 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:02.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:42:02.226 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:42:02.227+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:42:02.308 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:42:02.309+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:42:02.326 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:02.326 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:02.335 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:42:02.345 INFO:tasks.rados.rados.0.smithi089.stderr:0 errors. 2024-04-22T18:42:02.345 INFO:tasks.rados.rados.0.smithi089.stderr: 2024-04-22T18:42:02.348 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T18:42:02.433 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:42:02.433+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:42:02.477 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:02.478 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:02.486 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:42:02.530 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:42:02.531+0000 7f7f4e1b9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31909) UID: 0 2024-04-22T18:42:02.586 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:02.586 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:02.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:42:02.609 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:42:02.610+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:42:02.680 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:42:02.680 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":185,"fsid":"9e25744c-eff1-4a4a-8d47-0dda406b2c35","created":"2024-04-22T18:38:26.841687+0000","modified":"2024-04-22T18:42:01.781178+0000","last_up_change":"2024-04-22T18:38:35.971820+0000","last_in_change":"2024-04-22T18:41:26.001892+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T18:38:36.991610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.94993972778320312,"average_primary_affinity":0.94999998807907104,"average_primary_affinity_weighted":0.94999998807907104}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T18:38:37.622620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":3,"score_stable":3,"optimal_score":0.75,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.94993972778320312,"average_primary_affinity":0.94999998807907104,"average_primary_affinity_weighted":0.94999998807907104}},{"pool":3,"pool_name":"low_tier","create_time":"2024-04-22T18:38:46.038005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":0.94993972778320312,"average_primary_affinity":0.94999998807907104,"average_primary_affinity_weighted":0.94999998807907104}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-04-22T18:38:46.483369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":36,"pg_placement_num":29,"pg_placement_num_target":36,"pg_num_target":36,"pg_num_pending":36,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":158,"last_epoch_started":157,"last_epoch_clean":157,"source_version":"151'79","target_version":"155'85"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"158","last_force_op_resend_preluminous":"164","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_acting":2.6700000762939453,"score_stable":2.8900001049041748,"optimal_score":0.75,"raw_score_acting":2,"raw_score_stable":2.1700000762939453,"primary_affinity_weighted":0.94993972778320312,"average_primary_affinity":0.94999998807907104,"average_primary_affinity_weighted":0.94999998807907104}}],"osds":[{"osd":0,"uuid":"bbf33cfb-0f07-4df6-b90f-83e9edb039a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6800","nonce":31890}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6801","nonce":31890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6803","nonce":31890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6802","nonce":31890}]},"public_addr":"172.21.15.89:6800/31890","cluster_addr":"172.21.15.89:6801/31890","heartbeat_back_addr":"172.21.15.89:6803/31890","heartbeat_front_addr":"172.21.15.89:6802/31890","state":["exists","up"]},{"osd":1,"uuid":"30415a8e-9f47-416f-9fd0-716c2cb3f562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6804","nonce":31899}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6805","nonce":31899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6807","nonce":31899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6806","nonce":31899}]},"public_addr":"172.21.15.89:6804/31899","cluster_addr":"172.21.15.89:6805/31899","heartbeat_back_addr":"172.21.15.89:6807/31899","heartbeat_front_addr":"172.21.15.89:6806/31899","state":["exists","up"]},{"osd":2,"uuid":"b8da6b88-8355-4225-930d-bfed881089b7","up":1,"in":1,"weight":1,"primary_affinity":0.75152587890625,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6812","nonce":31903}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6813","nonce":31903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6815","nonce":31903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6814","nonce":31903}]},"public_addr":"172.21.15.89:6812/31903","cluster_addr":"172.21.15.89:6813/31903","heartbeat_back_addr":"172.21.15.89:6815/31903","heartbeat_front_addr":"172.21.15.89:6814/31903","state":["exists","up"]},{"osd":3,"uuid":"3eaec913-9d7d-4b03-a28f-0c334e7c39f1","up":1,"in":1,"weight":1,"primary_affinity":0.847991943359375,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6808","nonce":31913}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6809","nonce":31913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6811","nonce":31913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6810","nonce":31913}]},"public_addr":"172.21.15.89:6808/31913","cluster_addr":"172.21.15.89:6809/31913","heartbeat_back_addr":"172.21.15.89:6811/31913","heartbeat_front_addr":"172.21.15.89:6810/31913","state":["exists","up"]},{"osd":4,"uuid":"887acc3b-6f6b-49eb-a039-13050a2c7d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":31922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":31922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":31922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":31922}]},"public_addr":"172.21.15.121:6808/31922","cluster_addr":"172.21.15.121:6809/31922","heartbeat_back_addr":"172.21.15.121:6811/31922","heartbeat_front_addr":"172.21.15.121:6810/31922","state":["exists","up"]},{"osd":5,"uuid":"dcaddfd0-f8d1-4c28-bda2-d17eed606b60","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":31921}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":31921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":31921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":31921}]},"public_addr":"172.21.15.121:6804/31921","cluster_addr":"172.21.15.121:6805/31921","heartbeat_back_addr":"172.21.15.121:6807/31921","heartbeat_front_addr":"172.21.15.121:6806/31921","state":["exists","up"]},{"osd":6,"uuid":"acf0527b-3020-4b0a-b64c-5068d21a5781","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":31920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":31920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":31920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":31920}]},"public_addr":"172.21.15.121:6800/31920","cluster_addr":"172.21.15.121:6801/31920","heartbeat_back_addr":"172.21.15.121:6803/31920","heartbeat_front_addr":"172.21.15.121:6802/31920","state":["exists","up"]},{"osd":7,"uuid":"305394bc-be3d-420b-88d7-d273670540b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":31939}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":31939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":31939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":31939}]},"public_addr":"172.21.15.121:6812/31939","cluster_addr":"172.21.15.121:6813/31939","heartbeat_back_addr":"172.21.15.121:6815/31939","heartbeat_front_addr":"172.21.15.121:6814/31939","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:40:59.259212+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:41:05.370813+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:41:20.388547+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:41:13.858167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:41:02.314614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-22T18:41:10.687865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-22T18:41:56.085558+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:41:00.313214+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.22","osds":[6,7]}],"pg_upmap_items":[{"pgid":"4.11","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.18","osds":[4,7]},{"pgid":"4.19","osds":[1,4]},{"pgid":"4.1b","osds":[0,3]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T18:42:02.693 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:42:02.710 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:42:02.711+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:42:02.738 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:02.738 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:02.745 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:42:02.784 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 7, 2, 4, 0] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-22T18:42:02.784 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T18:42:02.784 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-04-22T18:42:02.784 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-22T18:42:02.784 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-22T18:42:02.784 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-04-22T18:42:02.788 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:42:02.787+0000 7fec57bee640 -1 osd.7 184 handle_osd_map injecting map crc failure 2024-04-22T18:42:02.829 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-22T18:42:02.830+0000 7fec651fc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 31918) UID: 0 2024-04-22T18:42:02.886 INFO:teuthology.orchestra.run.smithi121.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:02.886 INFO:teuthology.orchestra.run.smithi121.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:02.889 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:02.889 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-22T18:42:02.889 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:02.895 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T18:42:02.940 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:42:02.940+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:42:02.997 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:02.997 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:03.006 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-22T18:42:03.012 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:42:03.013+0000 7f3d0a492640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 31913) UID: 0 2024-04-22T18:42:03.025 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:42:03.026 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:42:03.037 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-04-22T18:42:02.998228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":271319784,"num_objects":262,"num_object_clones":0,"num_object_copies":524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":4159,"num_read_kb":3288306,"num_write":2713,"num_write_kb":630522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1240,"num_bytes_recovered":383020397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":25,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3255,"ondisk_log_size":3255,"up":98,"acting":98,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":102,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":565472,"kb_used_data":358268,"kb_used_omap":9,"kb_used_meta":207094,"kb_avail":565665568,"statfs":{"total":579820584960,"available":579241541632,"internally_reserved":0,"allocated":366866432,"data_stored":373101646,"data_compressed":12780311,"data_compressed_allocated":13647872,"data_compressed_original":28150962,"omap_allocated":9557,"internal_metadata":212064939},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-697540,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-102,"num_read_kb":-174292,"num_write":-43,"num_write_kb":-13367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":-5574713,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000839"},"pg_stats":[{"pgid":"4.23","version":"155'91","reported_seq":411,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.881590+0000","last_change":"2024-04-22T18:41:44.918810+0000","last_active":"2024-04-22T18:41:59.881590+0000","last_peered":"2024-04-22T18:41:59.881590+0000","last_clean":"2024-04-22T18:41:59.881590+0000","last_became_active":"2024-04-22T18:41:44.918345+0000","last_became_peered":"2024-04-22T18:41:44.918345+0000","last_unstale":"2024-04-22T18:41:59.881590+0000","last_undegraded":"2024-04-22T18:41:59.881590+0000","last_fullsized":"2024-04-22T18:41:59.881590+0000","mapping_epoch":168,"log_start":"137'90","ondisk_log_start":"137'90","created":164,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":6,"last_scrub":"137'90","last_scrub_stamp":"2024-04-22T18:41:22.057812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:22.057812+0000","objects_scrubbed":2,"log_size":1,"log_dups_size":61,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20189616099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655556,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":43,"num_read_kb":57909,"num_write":11,"num_write_kb":2755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5875215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.22","version":"149'42","reported_seq":367,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.886408+0000","last_change":"2024-04-22T18:41:40.908387+0000","last_active":"2024-04-22T18:42:00.886408+0000","last_peered":"2024-04-22T18:42:00.886408+0000","last_clean":"2024-04-22T18:42:00.886408+0000","last_became_active":"2024-04-22T18:41:40.908069+0000","last_became_peered":"2024-04-22T18:41:40.908069+0000","last_unstale":"2024-04-22T18:42:00.886408+0000","last_undegraded":"2024-04-22T18:42:00.886408+0000","last_fullsized":"2024-04-22T18:42:00.886408+0000","mapping_epoch":164,"log_start":"64'38","ondisk_log_start":"64'38","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":25,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":88762,"num_write":13,"num_write_kb":3449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5687418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"155'190","reported_seq":765,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.781945+0000","last_change":"2024-04-22T18:41:44.820690+0000","last_active":"2024-04-22T18:41:59.781945+0000","last_peered":"2024-04-22T18:41:59.781945+0000","last_clean":"2024-04-22T18:41:59.781945+0000","last_became_active":"2024-04-22T18:41:44.820321+0000","last_became_peered":"2024-04-22T18:41:44.820321+0000","last_unstale":"2024-04-22T18:41:59.781945+0000","last_undegraded":"2024-04-22T18:41:59.781945+0000","last_fullsized":"2024-04-22T18:41:59.781945+0000","mapping_epoch":168,"log_start":"147'185","ondisk_log_start":"147'185","created":164,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":6,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":5,"log_dups_size":161,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7042282,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":229510,"num_write":43,"num_write_kb":12083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":21653811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"162'34","reported_seq":320,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.884653+0000","last_change":"2024-04-22T18:41:58.656105+0000","last_active":"2024-04-22T18:42:00.884653+0000","last_peered":"2024-04-22T18:42:00.884653+0000","last_clean":"2024-04-22T18:42:00.884653+0000","last_became_active":"2024-04-22T18:41:41.106971+0000","last_became_peered":"2024-04-22T18:41:41.106971+0000","last_unstale":"2024-04-22T18:42:00.884653+0000","last_undegraded":"2024-04-22T18:42:00.884653+0000","last_fullsized":"2024-04-22T18:42:00.884653+0000","mapping_epoch":164,"log_start":"147'30","ondisk_log_start":"147'30","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":6,"last_scrub":"162'34","last_scrub_stamp":"2024-04-22T18:41:58.656008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:58.656008+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":30,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:09.431914+0000","scrub_duration":0.30219005700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"163'88","reported_seq":430,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.886562+0000","last_change":"2024-04-22T18:41:41.008619+0000","last_active":"2024-04-22T18:42:00.886562+0000","last_peered":"2024-04-22T18:42:00.886562+0000","last_clean":"2024-04-22T18:42:00.886562+0000","last_became_active":"2024-04-22T18:41:41.008273+0000","last_became_peered":"2024-04-22T18:41:41.008273+0000","last_unstale":"2024-04-22T18:42:00.886562+0000","last_undegraded":"2024-04-22T18:42:00.886562+0000","last_fullsized":"2024-04-22T18:42:00.886562+0000","mapping_epoch":164,"log_start":"155'85","ondisk_log_start":"155'85","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":89,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2714397,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":146,"num_read_kb":138065,"num_write":47,"num_write_kb":12258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"147'53","reported_seq":109,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.882414+0000","last_change":"2024-04-22T18:41:57.495745+0000","last_active":"2024-04-22T18:41:59.882414+0000","last_peered":"2024-04-22T18:41:59.882414+0000","last_clean":"2024-04-22T18:41:59.882414+0000","last_became_active":"2024-04-22T18:41:11.253471+0000","last_became_peered":"2024-04-22T18:41:11.253471+0000","last_unstale":"2024-04-22T18:41:59.882414+0000","last_undegraded":"2024-04-22T18:41:59.882414+0000","last_fullsized":"2024-04-22T18:41:59.882414+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'53","last_scrub_stamp":"2024-04-22T18:41:57.495600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:57.495600+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:20.349122+0000","scrub_duration":0.30282076499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655572,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"168'89","reported_seq":143,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.882404+0000","last_change":"2024-04-22T18:41:53.569917+0000","last_active":"2024-04-22T18:41:59.882404+0000","last_peered":"2024-04-22T18:41:59.882404+0000","last_clean":"2024-04-22T18:41:59.882404+0000","last_became_active":"2024-04-22T18:41:11.354958+0000","last_became_peered":"2024-04-22T18:41:11.354958+0000","last_unstale":"2024-04-22T18:41:59.882404+0000","last_undegraded":"2024-04-22T18:41:59.882404+0000","last_fullsized":"2024-04-22T18:41:59.882404+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"168'89","last_scrub_stamp":"2024-04-22T18:41:53.569764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:53.569764+0000","objects_scrubbed":1,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:55.086845+0000","scrub_duration":0.34939699600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243817,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"132'123","reported_seq":504,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.782092+0000","last_change":"2024-04-22T18:41:11.469596+0000","last_active":"2024-04-22T18:41:59.782092+0000","last_peered":"2024-04-22T18:41:59.782092+0000","last_clean":"2024-04-22T18:41:59.782092+0000","last_became_active":"2024-04-22T18:41:11.469282+0000","last_became_peered":"2024-04-22T18:41:11.469282+0000","last_unstale":"2024-04-22T18:41:59.782092+0000","last_undegraded":"2024-04-22T18:41:59.782092+0000","last_fullsized":"2024-04-22T18:41:59.782092+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"121'119","last_scrub_stamp":"2024-04-22T18:40:55.573199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.573199+0000","objects_scrubbed":4,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32976813900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4521133,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":219,"num_read_kb":257792,"num_write":64,"num_write_kb":17979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3786236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"168'159","reported_seq":644,"reported_epoch":182,"state":"remapped+peering","last_fresh":"2024-04-22T18:41:59.788468+0000","last_change":"2024-04-22T18:42:00.884904+0000","last_active":"2024-04-22T18:42:00.884904+0000","last_peered":"2024-04-22T18:41:59.788468+0000","last_clean":"2024-04-22T18:41:59.788468+0000","last_became_active":"2024-04-22T18:41:29.932575+0000","last_became_peered":"2024-04-22T18:41:29.932575+0000","last_unstale":"2024-04-22T18:41:59.788468+0000","last_undegraded":"2024-04-22T18:41:59.788468+0000","last_fullsized":"2024-04-22T18:41:59.788468+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":5,"last_scrub":"168'159","last_scrub_stamp":"2024-04-22T18:41:53.557137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:53.557137+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:54.988213+0000","scrub_duration":0.101410045,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185042,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3,4],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"163'115","reported_seq":465,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.782040+0000","last_change":"2024-04-22T18:41:56.070499+0000","last_active":"2024-04-22T18:41:59.782040+0000","last_peered":"2024-04-22T18:41:59.782040+0000","last_clean":"2024-04-22T18:41:59.782040+0000","last_became_active":"2024-04-22T18:41:56.070223+0000","last_became_peered":"2024-04-22T18:41:56.070223+0000","last_unstale":"2024-04-22T18:41:59.782040+0000","last_undegraded":"2024-04-22T18:41:59.782040+0000","last_fullsized":"2024-04-22T18:41:59.782040+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2806852,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":163,"num_read_kb":197186,"num_write":61,"num_write_kb":16155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":6836986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"168'120","reported_seq":611,"reported_epoch":184,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:42:00.886412+0000","last_change":"2024-04-22T18:41:52.356620+0000","last_active":"2024-04-22T18:42:00.886412+0000","last_peered":"2024-04-22T18:42:00.886412+0000","last_clean":"2024-04-22T18:41:43.719042+0000","last_became_active":"2024-04-22T18:41:52.255112+0000","last_became_peered":"2024-04-22T18:41:52.255112+0000","last_unstale":"2024-04-22T18:42:00.886412+0000","last_undegraded":"2024-04-22T18:42:00.886412+0000","last_fullsized":"2024-04-22T18:42:00.886412+0000","mapping_epoch":174,"log_start":"84'88","ondisk_log_start":"84'88","created":110,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":32,"log_dups_size":106,"ondisk_log_size":32,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7033845,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":122,"num_read_kb":149003,"num_write":24,"num_write_kb":7118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9095252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"148'71","reported_seq":403,"reported_epoch":183,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-22T18:42:00.394876+0000","last_change":"2024-04-22T18:41:49.936581+0000","last_active":"2024-04-22T18:42:00.394876+0000","last_peered":"2024-04-22T18:42:00.394876+0000","last_clean":"2024-04-22T18:41:01.854833+0000","last_became_active":"2024-04-22T18:41:49.835532+0000","last_became_peered":"2024-04-22T18:41:49.835532+0000","last_unstale":"2024-04-22T18:42:00.394876+0000","last_undegraded":"2024-04-22T18:42:00.394876+0000","last_fullsized":"2024-04-22T18:42:00.394876+0000","mapping_epoch":172,"log_start":"79'52","ondisk_log_start":"79'52","created":110,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":61,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":857223,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":130,"num_read_kb":179919,"num_write":60,"num_write_kb":17002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9662508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"107'52","reported_seq":404,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.885088+0000","last_change":"2024-04-22T18:41:22.602374+0000","last_active":"2024-04-22T18:42:00.885088+0000","last_peered":"2024-04-22T18:42:00.885088+0000","last_clean":"2024-04-22T18:42:00.885088+0000","last_became_active":"2024-04-22T18:41:00.078669+0000","last_became_peered":"2024-04-22T18:41:00.078669+0000","last_unstale":"2024-04-22T18:42:00.885088+0000","last_undegraded":"2024-04-22T18:42:00.885088+0000","last_fullsized":"2024-04-22T18:42:00.885088+0000","mapping_epoch":125,"log_start":"83'49","ondisk_log_start":"83'49","created":110,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":5,"last_scrub":"107'52","last_scrub_stamp":"2024-04-22T18:41:22.602272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:22.602272+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:48.741276+0000","scrub_duration":0.40203006699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"166'100","reported_seq":510,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.781596+0000","last_change":"2024-04-22T18:41:45.814689+0000","last_active":"2024-04-22T18:41:59.781596+0000","last_peered":"2024-04-22T18:41:59.781596+0000","last_clean":"2024-04-22T18:41:59.781596+0000","last_became_active":"2024-04-22T18:41:30.012572+0000","last_became_peered":"2024-04-22T18:41:30.012572+0000","last_unstale":"2024-04-22T18:41:59.781596+0000","last_undegraded":"2024-04-22T18:41:59.781596+0000","last_fullsized":"2024-04-22T18:41:59.781596+0000","mapping_epoch":153,"log_start":"96'71","ondisk_log_start":"96'71","created":110,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":5,"last_scrub":"166'100","last_scrub_stamp":"2024-04-22T18:41:45.814522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:45.814522+0000","objects_scrubbed":1,"log_size":29,"log_dups_size":81,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:06.282120+0000","scrub_duration":0.40262363400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3227010,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"54'29","reported_seq":124,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.074244+0000","last_change":"2024-04-22T18:40:55.328642+0000","last_active":"2024-04-22T18:42:00.074244+0000","last_peered":"2024-04-22T18:42:00.074244+0000","last_clean":"2024-04-22T18:42:00.074244+0000","last_became_active":"2024-04-22T18:40:13.180144+0000","last_became_peered":"2024-04-22T18:40:13.180144+0000","last_unstale":"2024-04-22T18:42:00.074244+0000","last_undegraded":"2024-04-22T18:42:00.074244+0000","last_fullsized":"2024-04-22T18:42:00.074244+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"54'29","last_scrub_stamp":"2024-04-22T18:40:55.328529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.328529+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:10.967288+0000","scrub_duration":0.30217712000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"51'19","reported_seq":126,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.884746+0000","last_change":"2024-04-22T18:40:57.754351+0000","last_active":"2024-04-22T18:42:00.884746+0000","last_peered":"2024-04-22T18:42:00.884746+0000","last_clean":"2024-04-22T18:42:00.884746+0000","last_became_active":"2024-04-22T18:40:10.978840+0000","last_became_peered":"2024-04-22T18:40:10.978840+0000","last_unstale":"2024-04-22T18:42:00.884746+0000","last_undegraded":"2024-04-22T18:42:00.884746+0000","last_fullsized":"2024-04-22T18:42:00.884746+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"51'19","last_scrub_stamp":"2024-04-22T18:40:57.754261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:57.754261+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:17.927283+0000","scrub_duration":0.59184710600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"166'93","reported_seq":101,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.882417+0000","last_change":"2024-04-22T18:41:04.562945+0000","last_active":"2024-04-22T18:41:59.882417+0000","last_peered":"2024-04-22T18:41:59.882417+0000","last_clean":"2024-04-22T18:41:59.882417+0000","last_became_active":"2024-04-22T18:41:04.139159+0000","last_became_peered":"2024-04-22T18:41:04.139159+0000","last_unstale":"2024-04-22T18:41:59.882417+0000","last_undegraded":"2024-04-22T18:41:59.882417+0000","last_fullsized":"2024-04-22T18:41:59.882417+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'75","last_scrub_stamp":"2024-04-22T18:40:22.670296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.670296+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20951747300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218722,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":96410,"num_write":16,"num_write_kb":7324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12459808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"166'50","reported_seq":315,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.986512+0000","last_change":"2024-04-22T18:40:59.453830+0000","last_active":"2024-04-22T18:42:00.986512+0000","last_peered":"2024-04-22T18:42:00.986512+0000","last_clean":"2024-04-22T18:42:00.986512+0000","last_became_active":"2024-04-22T18:40:58.107873+0000","last_became_peered":"2024-04-22T18:40:58.107873+0000","last_unstale":"2024-04-22T18:42:00.986512+0000","last_undegraded":"2024-04-22T18:42:00.986512+0000","last_fullsized":"2024-04-22T18:42:00.986512+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3621975,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":115,"num_read_kb":106979,"num_write":32,"num_write_kb":8520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2641926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"169'153","reported_seq":520,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.886387+0000","last_change":"2024-04-22T18:40:26.844287+0000","last_active":"2024-04-22T18:42:00.886387+0000","last_peered":"2024-04-22T18:42:00.886387+0000","last_clean":"2024-04-22T18:42:00.886387+0000","last_became_active":"2024-04-22T18:40:26.742351+0000","last_became_peered":"2024-04-22T18:40:26.742351+0000","last_unstale":"2024-04-22T18:42:00.886387+0000","last_undegraded":"2024-04-22T18:42:00.886387+0000","last_fullsized":"2024-04-22T18:42:00.886387+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'113","last_scrub_stamp":"2024-04-22T18:40:20.291988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.291988+0000","objects_scrubbed":2,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10270805500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941291,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":63,"num_read_kb":42947,"num_write":48,"num_write_kb":13258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3940523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"31'5","reported_seq":230,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.884739+0000","last_change":"2024-04-22T18:41:57.706938+0000","last_active":"2024-04-22T18:42:00.884739+0000","last_peered":"2024-04-22T18:42:00.884739+0000","last_clean":"2024-04-22T18:42:00.884739+0000","last_became_active":"2024-04-22T18:39:59.265189+0000","last_became_peered":"2024-04-22T18:39:59.265189+0000","last_unstale":"2024-04-22T18:42:00.884739+0000","last_undegraded":"2024-04-22T18:42:00.884739+0000","last_fullsized":"2024-04-22T18:42:00.884739+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":5,"last_scrub":"31'5","last_scrub_stamp":"2024-04-22T18:41:57.706814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:57.706814+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:18.447978+0000","scrub_duration":0.40209979499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":70,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.174487+0000","last_change":"2024-04-22T18:40:58.207745+0000","last_active":"2024-04-22T18:42:00.174487+0000","last_peered":"2024-04-22T18:42:00.174487+0000","last_clean":"2024-04-22T18:42:00.174487+0000","last_became_active":"2024-04-22T18:40:58.207451+0000","last_became_peered":"2024-04-22T18:40:58.207451+0000","last_unstale":"2024-04-22T18:42:00.174487+0000","last_undegraded":"2024-04-22T18:42:00.174487+0000","last_fullsized":"2024-04-22T18:42:00.174487+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:19.136933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:19.136933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"169'195","reported_seq":819,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.092307+0000","last_change":"2024-04-22T18:41:44.821952+0000","last_active":"2024-04-22T18:42:00.092307+0000","last_peered":"2024-04-22T18:42:00.092307+0000","last_clean":"2024-04-22T18:42:00.092307+0000","last_became_active":"2024-04-22T18:41:44.821076+0000","last_became_peered":"2024-04-22T18:41:44.821076+0000","last_unstale":"2024-04-22T18:42:00.092307+0000","last_undegraded":"2024-04-22T18:42:00.092307+0000","last_fullsized":"2024-04-22T18:42:00.092307+0000","mapping_epoch":168,"log_start":"147'185","ondisk_log_start":"147'185","created":20,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'110","last_scrub_stamp":"2024-04-22T18:39:54.381616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:54.381616+0000","objects_scrubbed":4,"log_size":10,"log_dups_size":161,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.202319315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8134736,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":208,"num_read_kb":247495,"num_write":48,"num_write_kb":13433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":31242513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":207,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.886655+0000","last_change":"2024-04-22T18:41:58.562039+0000","last_active":"2024-04-22T18:42:00.886655+0000","last_peered":"2024-04-22T18:42:00.886655+0000","last_clean":"2024-04-22T18:42:00.886655+0000","last_became_active":"2024-04-22T18:41:04.037248+0000","last_became_peered":"2024-04-22T18:41:04.037248+0000","last_unstale":"2024-04-22T18:42:00.886655+0000","last_undegraded":"2024-04-22T18:42:00.886655+0000","last_fullsized":"2024-04-22T18:42:00.886655+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:58.561808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:58.561808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:06.700692+0000","scrub_duration":0.301876478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"162'34","reported_seq":334,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.884822+0000","last_change":"2024-04-22T18:41:45.590948+0000","last_active":"2024-04-22T18:42:00.884822+0000","last_peered":"2024-04-22T18:42:00.884822+0000","last_clean":"2024-04-22T18:42:00.884822+0000","last_became_active":"2024-04-22T18:41:41.006293+0000","last_became_peered":"2024-04-22T18:41:41.006293+0000","last_unstale":"2024-04-22T18:42:00.884822+0000","last_undegraded":"2024-04-22T18:42:00.884822+0000","last_fullsized":"2024-04-22T18:42:00.884822+0000","mapping_epoch":20,"log_start":"147'30","ondisk_log_start":"147'30","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'34","last_scrub_stamp":"2024-04-22T18:41:45.590834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:45.590834+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":30,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:48.626740+0000","scrub_duration":0.40273239100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025326,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":210,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.174462+0000","last_change":"2024-04-22T18:41:43.814617+0000","last_active":"2024-04-22T18:42:00.174462+0000","last_peered":"2024-04-22T18:42:00.174462+0000","last_clean":"2024-04-22T18:42:00.174462+0000","last_became_active":"2024-04-22T18:41:26.961835+0000","last_became_peered":"2024-04-22T18:41:26.961835+0000","last_unstale":"2024-04-22T18:42:00.174462+0000","last_undegraded":"2024-04-22T18:42:00.174462+0000","last_fullsized":"2024-04-22T18:42:00.174462+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:09.538542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:09.538542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.35201753200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"167'92","reported_seq":421,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.788600+0000","last_change":"2024-04-22T18:41:56.587360+0000","last_active":"2024-04-22T18:41:59.788600+0000","last_peered":"2024-04-22T18:41:59.788600+0000","last_clean":"2024-04-22T18:41:59.788600+0000","last_became_active":"2024-04-22T18:41:42.764184+0000","last_became_peered":"2024-04-22T18:41:42.764184+0000","last_unstale":"2024-04-22T18:41:59.788600+0000","last_undegraded":"2024-04-22T18:41:59.788600+0000","last_fullsized":"2024-04-22T18:41:59.788600+0000","mapping_epoch":166,"log_start":"137'90","ondisk_log_start":"137'90","created":20,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'92","last_scrub_stamp":"2024-04-22T18:41:56.587238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:56.587238+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":61,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:09.467131+0000","scrub_duration":0.101671527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":194,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.986586+0000","last_change":"2024-04-22T18:41:26.854653+0000","last_active":"2024-04-22T18:42:00.986586+0000","last_peered":"2024-04-22T18:42:00.986586+0000","last_clean":"2024-04-22T18:42:00.986586+0000","last_became_active":"2024-04-22T18:41:26.854399+0000","last_became_peered":"2024-04-22T18:41:26.854399+0000","last_unstale":"2024-04-22T18:42:00.986586+0000","last_undegraded":"2024-04-22T18:42:00.986586+0000","last_fullsized":"2024-04-22T18:42:00.986586+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:38:37.997225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"149'42","reported_seq":368,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.886598+0000","last_change":"2024-04-22T18:41:40.807780+0000","last_active":"2024-04-22T18:42:00.886598+0000","last_peered":"2024-04-22T18:42:00.886598+0000","last_clean":"2024-04-22T18:42:00.886598+0000","last_became_active":"2024-04-22T18:41:40.807410+0000","last_became_peered":"2024-04-22T18:41:40.807410+0000","last_unstale":"2024-04-22T18:42:00.886598+0000","last_undegraded":"2024-04-22T18:42:00.886598+0000","last_fullsized":"2024-04-22T18:42:00.886598+0000","mapping_epoch":129,"log_start":"64'38","ondisk_log_start":"64'38","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":25,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":88762,"num_write":12,"num_write_kb":3449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5687418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"169'384","reported_seq":1585,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.395382+0000","last_change":"2024-04-22T18:41:43.266638+0000","last_active":"2024-04-22T18:42:00.395382+0000","last_peered":"2024-04-22T18:42:00.395382+0000","last_clean":"2024-04-22T18:42:00.395382+0000","last_became_active":"2024-04-22T18:41:26.994048+0000","last_became_peered":"2024-04-22T18:41:26.994048+0000","last_unstale":"2024-04-22T18:42:00.395382+0000","last_undegraded":"2024-04-22T18:42:00.395382+0000","last_fullsized":"2024-04-22T18:42:00.395382+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'293","last_scrub_stamp":"2024-04-22T18:40:53.582354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:53.582354+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:14.875876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41123046,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":352,"num_read_kb":4222,"num_write":477,"num_write_kb":113967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":82068207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":155,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.886703+0000","last_change":"2024-04-22T18:41:26.657445+0000","last_active":"2024-04-22T18:42:00.886703+0000","last_peered":"2024-04-22T18:42:00.886703+0000","last_clean":"2024-04-22T18:42:00.886703+0000","last_became_active":"2024-04-22T18:41:26.657206+0000","last_became_peered":"2024-04-22T18:41:26.657206+0000","last_unstale":"2024-04-22T18:42:00.886703+0000","last_undegraded":"2024-04-22T18:42:00.886703+0000","last_fullsized":"2024-04-22T18:42:00.886703+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:55.558446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.558446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.56126016300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"167'36","reported_seq":322,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.886501+0000","last_change":"2024-04-22T18:40:21.426510+0000","last_active":"2024-04-22T18:42:00.886501+0000","last_peered":"2024-04-22T18:42:00.886501+0000","last_clean":"2024-04-22T18:42:00.886501+0000","last_became_active":"2024-04-22T18:39:51.943071+0000","last_became_peered":"2024-04-22T18:39:51.943071+0000","last_unstale":"2024-04-22T18:42:00.886501+0000","last_undegraded":"2024-04-22T18:42:00.886501+0000","last_fullsized":"2024-04-22T18:42:00.886501+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-04-22T18:40:21.426350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.426350+0000","objects_scrubbed":1,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30334366400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294238,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":109,"num_read_kb":123704,"num_write":9,"num_write_kb":2718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"169'309","reported_seq":753,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.886341+0000","last_change":"2024-04-22T18:40:20.465005+0000","last_active":"2024-04-22T18:42:00.886341+0000","last_peered":"2024-04-22T18:42:00.886341+0000","last_clean":"2024-04-22T18:42:00.886341+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:42:00.886341+0000","last_undegraded":"2024-04-22T18:42:00.886341+0000","last_fullsized":"2024-04-22T18:42:00.886341+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'176","last_scrub_stamp":"2024-04-22T18:40:20.464799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:20.464799+0000","objects_scrubbed":31,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30629975799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382624,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":306,"num_read_kb":4987,"num_write":404,"num_write_kb":80545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":151,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.881535+0000","last_change":"2024-04-22T18:41:41.421087+0000","last_active":"2024-04-22T18:41:59.881535+0000","last_peered":"2024-04-22T18:41:59.881535+0000","last_clean":"2024-04-22T18:41:59.881535+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:41:59.881535+0000","last_undegraded":"2024-04-22T18:41:59.881535+0000","last_fullsized":"2024-04-22T18:41:59.881535+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:41.420996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:41.420996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:57.844851+0000","scrub_duration":0.20141637900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"162'72","reported_seq":446,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.886729+0000","last_change":"2024-04-22T18:41:32.860808+0000","last_active":"2024-04-22T18:42:00.886729+0000","last_peered":"2024-04-22T18:42:00.886729+0000","last_clean":"2024-04-22T18:42:00.886729+0000","last_became_active":"2024-04-22T18:41:32.860455+0000","last_became_peered":"2024-04-22T18:41:32.860455+0000","last_unstale":"2024-04-22T18:42:00.886729+0000","last_undegraded":"2024-04-22T18:42:00.886729+0000","last_fullsized":"2024-04-22T18:42:00.886729+0000","mapping_epoch":156,"log_start":"83'49","ondisk_log_start":"83'49","created":20,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'39","last_scrub_stamp":"2024-04-22T18:39:57.812202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:57.812202+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":49,"ondisk_log_size":23,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55542838699999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3008494,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":95,"num_read_kb":112837,"num_write":38,"num_write_kb":11286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4519285,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"168'337","reported_seq":1390,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.781657+0000","last_change":"2024-04-22T18:41:41.108768+0000","last_active":"2024-04-22T18:41:59.781657+0000","last_peered":"2024-04-22T18:41:59.781657+0000","last_clean":"2024-04-22T18:41:59.781657+0000","last_became_active":"2024-04-22T18:41:41.007340+0000","last_became_peered":"2024-04-22T18:41:41.007340+0000","last_unstale":"2024-04-22T18:41:59.781657+0000","last_undegraded":"2024-04-22T18:41:59.781657+0000","last_fullsized":"2024-04-22T18:41:59.781657+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'209","last_scrub_stamp":"2024-04-22T18:40:18.683442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:18.683442+0000","objects_scrubbed":24,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204344365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39035424,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":322,"num_read_kb":9140,"num_write":423,"num_write_kb":104979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":67954083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":185,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.074281+0000","last_change":"2024-04-22T18:40:22.521828+0000","last_active":"2024-04-22T18:42:00.074281+0000","last_peered":"2024-04-22T18:42:00.074281+0000","last_clean":"2024-04-22T18:42:00.074281+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:42:00.074281+0000","last_undegraded":"2024-04-22T18:42:00.074281+0000","last_fullsized":"2024-04-22T18:42:00.074281+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:40:22.521638+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.521638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30190717299999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"169'111","reported_seq":512,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.781606+0000","last_change":"2024-04-22T18:41:54.980019+0000","last_active":"2024-04-22T18:41:59.781606+0000","last_peered":"2024-04-22T18:41:59.781606+0000","last_clean":"2024-04-22T18:41:59.781606+0000","last_became_active":"2024-04-22T18:41:31.858774+0000","last_became_peered":"2024-04-22T18:41:31.858774+0000","last_unstale":"2024-04-22T18:41:59.781606+0000","last_undegraded":"2024-04-22T18:41:59.781606+0000","last_fullsized":"2024-04-22T18:41:59.781606+0000","mapping_epoch":155,"log_start":"96'71","ondisk_log_start":"96'71","created":20,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'111","last_scrub_stamp":"2024-04-22T18:41:54.979825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:54.979825+0000","objects_scrubbed":2,"log_size":40,"log_dups_size":81,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:06.600955+0000","scrub_duration":0.39464260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5664091,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"169'328","reported_seq":1057,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.882332+0000","last_change":"2024-04-22T18:41:19.866530+0000","last_active":"2024-04-22T18:41:59.882332+0000","last_peered":"2024-04-22T18:41:59.882332+0000","last_clean":"2024-04-22T18:41:59.882332+0000","last_became_active":"2024-04-22T18:41:19.765392+0000","last_became_peered":"2024-04-22T18:41:19.765392+0000","last_unstale":"2024-04-22T18:41:59.882332+0000","last_undegraded":"2024-04-22T18:41:59.882332+0000","last_fullsized":"2024-04-22T18:41:59.882332+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'212","last_scrub_stamp":"2024-04-22T18:40:21.870855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:40:21.870855+0000","objects_scrubbed":46,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43117026,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":334,"num_read_kb":12449,"num_write":418,"num_write_kb":91859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":43826500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":205,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.981533+0000","last_change":"2024-04-22T18:41:30.504800+0000","last_active":"2024-04-22T18:41:59.981533+0000","last_peered":"2024-04-22T18:41:59.981533+0000","last_clean":"2024-04-22T18:41:59.981533+0000","last_became_active":"2024-04-22T18:40:02.741862+0000","last_became_peered":"2024-04-22T18:40:02.741862+0000","last_unstale":"2024-04-22T18:41:59.981533+0000","last_undegraded":"2024-04-22T18:41:59.981533+0000","last_fullsized":"2024-04-22T18:41:59.981533+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:30.504686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:30.504686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:57.436870+0000","scrub_duration":0.30171718199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":391,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.074319+0000","last_change":"2024-04-22T18:41:56.270751+0000","last_active":"2024-04-22T18:42:00.074319+0000","last_peered":"2024-04-22T18:42:00.074319+0000","last_clean":"2024-04-22T18:42:00.074319+0000","last_became_active":"2024-04-22T18:41:26.757958+0000","last_became_peered":"2024-04-22T18:41:26.757958+0000","last_unstale":"2024-04-22T18:42:00.074319+0000","last_undegraded":"2024-04-22T18:42:00.074319+0000","last_fullsized":"2024-04-22T18:42:00.074319+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:41:56.270650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:41:56.270650+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:12.138524+0000","scrub_duration":0.202387964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"137'75","reported_seq":110,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.782152+0000","last_change":"2024-04-22T18:41:04.681901+0000","last_active":"2024-04-22T18:41:59.782152+0000","last_peered":"2024-04-22T18:41:59.782152+0000","last_clean":"2024-04-22T18:41:59.782152+0000","last_became_active":"2024-04-22T18:41:04.227104+0000","last_became_peered":"2024-04-22T18:41:04.227104+0000","last_unstale":"2024-04-22T18:41:59.782152+0000","last_undegraded":"2024-04-22T18:41:59.782152+0000","last_fullsized":"2024-04-22T18:41:59.782152+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'38","last_scrub_stamp":"2024-04-22T18:39:55.746067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:39:55.746067+0000","objects_scrubbed":1,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60877772299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6168268,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":192624,"num_write":57,"num_write_kb":16481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15782313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"164'59","reported_seq":391,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.074079+0000","last_change":"2024-04-22T18:41:35.835744+0000","last_active":"2024-04-22T18:42:00.074079+0000","last_peered":"2024-04-22T18:42:00.074079+0000","last_clean":"2024-04-22T18:42:00.074079+0000","last_became_active":"2024-04-22T18:41:35.834928+0000","last_became_peered":"2024-04-22T18:41:35.834928+0000","last_unstale":"2024-04-22T18:42:00.074079+0000","last_undegraded":"2024-04-22T18:42:00.074079+0000","last_fullsized":"2024-04-22T18:42:00.074079+0000","mapping_epoch":159,"log_start":"79'52","ondisk_log_start":"79'52","created":20,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'57","last_scrub_stamp":"2024-04-22T18:40:56.376497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:56.376497+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":61,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30191299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3832029,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":46990,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1913105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"167'154","reported_seq":632,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.782191+0000","last_change":"2024-04-22T18:41:17.760985+0000","last_active":"2024-04-22T18:41:59.782191+0000","last_peered":"2024-04-22T18:41:59.782191+0000","last_clean":"2024-04-22T18:41:59.782191+0000","last_became_active":"2024-04-22T18:41:17.759812+0000","last_became_peered":"2024-04-22T18:41:17.759812+0000","last_unstale":"2024-04-22T18:41:59.782191+0000","last_undegraded":"2024-04-22T18:41:59.782191+0000","last_fullsized":"2024-04-22T18:41:59.782191+0000","mapping_epoch":141,"log_start":"84'88","ondisk_log_start":"84'88","created":20,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'91","last_scrub_stamp":"2024-04-22T18:40:05.576719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:05.576719+0000","objects_scrubbed":3,"log_size":66,"log_dups_size":106,"ondisk_log_size":66,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20299785100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7420472,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":148630,"num_write":63,"num_write_kb":21335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12038689,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"167'121","reported_seq":473,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.886519+0000","last_change":"2024-04-22T18:41:11.054130+0000","last_active":"2024-04-22T18:42:00.886519+0000","last_peered":"2024-04-22T18:42:00.886519+0000","last_clean":"2024-04-22T18:42:00.886519+0000","last_became_active":"2024-04-22T18:41:11.053760+0000","last_became_peered":"2024-04-22T18:41:11.053760+0000","last_unstale":"2024-04-22T18:42:00.886519+0000","last_undegraded":"2024-04-22T18:42:00.886519+0000","last_fullsized":"2024-04-22T18:42:00.886519+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'66","last_scrub_stamp":"2024-04-22T18:40:22.140629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:22.140629+0000","objects_scrubbed":3,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20459416499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7059022,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":182,"num_read_kb":210386,"num_write":65,"num_write_kb":17255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4554596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"167'153","reported_seq":632,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.788674+0000","last_change":"2024-04-22T18:41:55.669111+0000","last_active":"2024-04-22T18:41:59.788674+0000","last_peered":"2024-04-22T18:41:59.788674+0000","last_clean":"2024-04-22T18:41:59.788674+0000","last_became_active":"2024-04-22T18:41:31.959307+0000","last_became_peered":"2024-04-22T18:41:31.959307+0000","last_unstale":"2024-04-22T18:41:59.788674+0000","last_undegraded":"2024-04-22T18:41:59.788674+0000","last_fullsized":"2024-04-22T18:41:59.788674+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'153","last_scrub_stamp":"2024-04-22T18:41:55.669014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:55.669014+0000","objects_scrubbed":1,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:18.053733+0000","scrub_duration":0.201688442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3193396,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"168'131","reported_seq":552,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.782095+0000","last_change":"2024-04-22T18:41:12.783644+0000","last_active":"2024-04-22T18:41:59.782095+0000","last_peered":"2024-04-22T18:41:59.782095+0000","last_clean":"2024-04-22T18:41:59.782095+0000","last_became_active":"2024-04-22T18:41:11.956312+0000","last_became_peered":"2024-04-22T18:41:11.956312+0000","last_unstale":"2024-04-22T18:41:59.782095+0000","last_undegraded":"2024-04-22T18:41:59.782095+0000","last_fullsized":"2024-04-22T18:41:59.782095+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'119","last_scrub_stamp":"2024-04-22T18:40:55.573199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.573199+0000","objects_scrubbed":4,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32976813900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832111,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":304,"num_read_kb":395814,"num_write":74,"num_write_kb":20641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9347897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"133'66","reported_seq":92,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.782255+0000","last_change":"2024-04-22T18:41:19.456295+0000","last_active":"2024-04-22T18:41:59.782255+0000","last_peered":"2024-04-22T18:41:59.782255+0000","last_clean":"2024-04-22T18:41:59.782255+0000","last_became_active":"2024-04-22T18:41:11.154301+0000","last_became_peered":"2024-04-22T18:41:11.154301+0000","last_unstale":"2024-04-22T18:41:59.782255+0000","last_undegraded":"2024-04-22T18:41:59.782255+0000","last_fullsized":"2024-04-22T18:41:59.782255+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'66","last_scrub_stamp":"2024-04-22T18:41:19.456160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:19.456160+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:36.925204+0000","scrub_duration":0.20146846400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"145'51","reported_seq":103,"reported_epoch":183,"state":"active+clean","last_fresh":"2024-04-22T18:41:59.782243+0000","last_change":"2024-04-22T18:41:45.675260+0000","last_active":"2024-04-22T18:41:59.782243+0000","last_peered":"2024-04-22T18:41:59.782243+0000","last_clean":"2024-04-22T18:41:59.782243+0000","last_became_active":"2024-04-22T18:41:11.253409+0000","last_became_peered":"2024-04-22T18:41:11.253409+0000","last_unstale":"2024-04-22T18:41:59.782243+0000","last_undegraded":"2024-04-22T18:41:59.782243+0000","last_fullsized":"2024-04-22T18:41:59.782243+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'51","last_scrub_stamp":"2024-04-22T18:41:45.675122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:41:45.675122+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:09.124003+0000","scrub_duration":0.40271887499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"163'88","reported_seq":420,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-22T18:42:00.886650+0000","last_change":"2024-04-22T18:41:41.109970+0000","last_active":"2024-04-22T18:42:00.886650+0000","last_peered":"2024-04-22T18:42:00.886650+0000","last_clean":"2024-04-22T18:42:00.886650+0000","last_became_active":"2024-04-22T18:41:41.109681+0000","last_became_peered":"2024-04-22T18:41:41.109681+0000","last_unstale":"2024-04-22T18:42:00.886650+0000","last_undegraded":"2024-04-22T18:42:00.886650+0000","last_fullsized":"2024-04-22T18:42:00.886650+0000","mapping_epoch":20,"log_start":"155'85","ondisk_log_start":"155'85","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:47.063013+0000","last_clean_scrub_stamp":"2024-04-22T18:38:47.063013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":89,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2714396,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":152,"num_read_kb":145684,"num_write":47,"num_write_kb":12257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":36,"stat_sum":{"num_bytes":115071296,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":2729,"num_read_kb":3257408,"num_write":832,"num_write_kb":236756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":166725499,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":25,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":187736064,"data_stored":187891771,"data_compressed":1037892,"data_compressed_allocated":1122304,"data_compressed_original":2198706,"omap_allocated":0,"internal_metadata":0},"log_size":1780,"ondisk_log_size":1780,"up":72,"acting":72,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":155658120,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":1314,"num_read_kb":30798,"num_write":1722,"num_write_kb":391350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1149,"num_bytes_recovered":214708114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249724928,"data_stored":264136596,"data_compressed":14626013,"data_compressed_allocated":15568896,"data_compressed_original":30834688,"omap_allocated":0,"internal_metadata":0},"log_size":1358,"ondisk_log_size":1358,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":2115712,"data_compressed":122550,"data_compressed_allocated":163840,"data_compressed_original":2084864,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50440,"kb_used_data":19316,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94321400,"statfs":{"total":96636764160,"available":96585113600,"internally_reserved":0,"allocated":19779584,"data_stored":19299442,"data_compressed":211962,"data_compressed_allocated":241664,"data_compressed_original":970752,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":2.5870000000000002,"15min":1.1439999999999999},"min":{"1min":0.182,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":26.568000000000001,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":6.3600000000000003,"5min":6.766,"15min":5.2599999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":95.918999999999997,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.371}]},{"osd":1,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":3.8050000000000002,"5min":3.984,"15min":4.5709999999999997},"min":{"1min":0.29299999999999998,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":66.814999999999998,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.433},{"interface":"front","average":{"1min":6.6040000000000001,"5min":12.141,"15min":12.744999999999999},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":100.242,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.32400000000000001}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7130000000000001,"5min":4.0730000000000004,"15min":4.0810000000000004},"min":{"1min":0.27400000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":97.662000000000006,"5min":97.662000000000006,"15min":97.662000000000006},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.33900000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7560000000000002,"5min":4.1289999999999996,"15min":4.1429999999999998},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":97.686999999999998,"5min":97.686999999999998,"15min":97.686999999999998},"last":0.314},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.45500000000000002}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":8.2089999999999996,"5min":5.4509999999999996,"15min":4.7889999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":78.149000000000001,"5min":85.509,"15min":85.509},"last":0.46200000000000002},{"interface":"front","average":{"1min":7.7130000000000001,"5min":16.856000000000002,"15min":20.866},"min":{"1min":0.26000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":97.590000000000003,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.35399999999999998}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":2.0270000000000001,"15min":2.7229999999999999},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.4730000000000001,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.6679999999999999,"5min":5.2939999999999996,"15min":6.5389999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":23.007999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.44400000000000001}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":11.756,"15min":12.769},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":23.027000000000001,"5min":139.428,"15min":139.428},"last":0.41299999999999998},{"interface":"front","average":{"1min":7.7800000000000002,"5min":4.3070000000000004,"15min":4.1909999999999998},"min":{"1min":0.29199999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":97.635999999999996,"5min":97.635999999999996,"15min":97.635999999999996},"last":0.42299999999999999}]}]},{"osd":6,"up_from":0,"seq":55834574891,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574891,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":160332,"kb_used_data":125112,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94211508,"statfs":{"total":96636764160,"available":96472584192,"internally_reserved":0,"allocated":128114688,"data_stored":131663157,"data_compressed":4963321,"data_compressed_allocated":5300224,"data_compressed_original":10430041,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":13.115,"15min":15.470000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":38.899999999999999,"5min":252.96199999999999,"15min":252.96199999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":9.282,"15min":8.8059999999999992},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63300000000000001,"5min":187.148,"15min":187.148},"last":0.45500000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":2.508,"15min":1.115},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":9.141,"5min":96.486999999999995,"15min":96.486999999999995},"last":0.63900000000000001},{"interface":"front","average":{"1min":22.393000000000001,"5min":11.545999999999999,"15min":7.4390000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":310.61500000000001,"5min":310.61500000000001,"15min":310.61500000000001},"last":0.47699999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":8.8989999999999991,"15min":8.6790000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.8330000000000002,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.995,"5min":2.4199999999999999,"15min":1.1040000000000001},"min":{"1min":0.31900000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.1399999999999997,"5min":96.471999999999994,"15min":96.471999999999994},"last":0.52100000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":11.172000000000001,"5min":8.9710000000000001,"15min":5.6710000000000003},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":122.102,"5min":252.91900000000001,"15min":252.91900000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.42999999999999999,"5min":17.242000000000001,"15min":19.725999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59899999999999998,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.70399999999999996}]},{"osd":5,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":2.3380000000000001,"15min":1.034},"min":{"1min":0.25800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":7.1559999999999997,"5min":96.408000000000001,"15min":96.408000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":2.6960000000000002,"5min":3.048,"15min":3.7080000000000002},"min":{"1min":0.36199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":38.581000000000003,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.749}]},{"osd":6,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":5.8289999999999997,"5min":14.667,"15min":13.662000000000001},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":87.706999999999994,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.83199999999999996,"5min":2.3820000000000001,"15min":1.0880000000000001},"min":{"1min":0.35699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.1159999999999997,"5min":96.438000000000002,"15min":96.438000000000002},"last":0.71599999999999997}]},{"osd":7,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":9.5749999999999993,"15min":8.8979999999999997},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68000000000000005,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.9790000000000001,"5min":2.6419999999999999,"15min":1.202},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":19.518999999999998,"5min":96.456999999999994,"15min":96.456999999999994},"last":0.78600000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78832,"kb_used_data":43620,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94293008,"statfs":{"total":96636764160,"available":96556040192,"internally_reserved":0,"allocated":44666880,"data_stored":44909272,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":15.041,"5min":10.538,"15min":11.646000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":93.131,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.26600000000000001},{"interface":"front","average":{"1min":7.077,"5min":1.6659999999999999,"15min":0.77400000000000002},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":87.305999999999997,"5min":87.305999999999997,"15min":87.305999999999997},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":10.558,"15min":6.8819999999999997},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":78.805000000000007,"5min":197.02099999999999,"15min":197.02099999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":8.9800000000000004,"5min":2.3210000000000002,"15min":0.997},"min":{"1min":0.20999999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":78.393000000000001,"5min":78.393000000000001,"15min":78.393000000000001},"last":0.32200000000000001}]},{"osd":2,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":2.3439999999999999,"15min":1.452},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":47.534999999999997,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.32900000000000001},{"interface":"front","average":{"1min":6.4770000000000003,"5min":1.728,"15min":0.85899999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":78.421999999999997,"5min":78.421999999999997,"15min":78.421999999999997},"last":0.30399999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1800000000000002,"5min":2.9940000000000002,"15min":3.7360000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":26.141999999999999,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.503},{"interface":"front","average":{"1min":4.5629999999999997,"5min":4.6289999999999996,"15min":4.3010000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":47.616999999999997,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.8279999999999998,"5min":2.5049999999999999,"15min":1.623},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":47.566000000000003,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.52000000000000002},{"interface":"front","average":{"1min":6.6379999999999999,"5min":2.2890000000000001,"15min":1.601},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.480999999999995,"5min":78.480999999999995,"15min":78.480999999999995},"last":0.48499999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.50900000000000001,"15min":0.53100000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.748,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.41399999999999998},{"interface":"front","average":{"1min":2.8500000000000001,"5min":2.871,"15min":1.3109999999999999},"min":{"1min":0.377,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":47.590000000000003,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.45400000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":6.6189999999999998,"5min":1.903,"15min":1.0229999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.453999999999994,"5min":78.453999999999994,"15min":78.453999999999994},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69299999999999995,"5min":0.995,"15min":0.995},"last":0.47299999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58384,"kb_used_data":27264,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94313456,"statfs":{"total":96636764160,"available":96576978944,"internally_reserved":0,"allocated":27918336,"data_stored":28281270,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":6.7439999999999998,"15min":7.6660000000000004},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":84.338999999999999,"5min":149.03,"15min":149.03},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.28299999999999997,"5min":1.3089999999999999,"15min":1.7589999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.62,"5min":31.866,"15min":31.866},"last":0.35999999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":9.4849999999999994,"5min":5.2489999999999997,"15min":2.4380000000000002},"min":{"1min":0.216,"5min":0.193,"15min":0.193},"max":{"1min":91.912000000000006,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":12.752000000000001,"5min":15.912000000000001,"15min":15.227},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":126.42,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.39500000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":6.2830000000000004,"15min":2.3359999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":129.041,"5min":147.721,"15min":147.721},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.33100000000000002}]},{"osd":4,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":8.1280000000000001,"5min":13.57,"15min":17.038},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":90.882999999999996,"5min":160.584,"15min":160.584},"last":0.70999999999999996},{"interface":"front","average":{"1min":5.9349999999999996,"5min":4.8520000000000003,"15min":3.2490000000000001},"min":{"1min":0.32500000000000001,"5min":0.315,"15min":0.315},"max":{"1min":93.962999999999994,"5min":97.887,"15min":97.887},"last":0.53600000000000003}]},{"osd":5,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":2.0339999999999998,"15min":2.7120000000000002},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.879,"5min":48.631,"15min":48.631},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":2.2890000000000001,"15min":3.0800000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.91800000000000004,"5min":56.18,"15min":56.18},"last":0.58899999999999997}]},{"osd":6,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.3540000000000001,"5min":3.4670000000000001,"15min":1.5129999999999999},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":92.028000000000006,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.61599999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.29,"5min":1.5800000000000001,"15min":0.91000000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":90.944999999999993,"5min":90.944999999999993,"15min":90.944999999999993},"last":0.65500000000000003},{"interface":"front","average":{"1min":7.4640000000000004,"5min":3.923,"15min":1.6970000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":92.048000000000002,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.69999999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101108,"kb_used_data":61800,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94270732,"statfs":{"total":96636764160,"available":96533229568,"internally_reserved":0,"allocated":63283200,"data_stored":63661227,"data_compressed":1653944,"data_compressed_allocated":1757184,"data_compressed_original":3606105,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":5.883,"15min":7.2000000000000002},"min":{"1min":0.20000000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":27.116,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.24099999999999999},{"interface":"front","average":{"1min":5.8040000000000003,"5min":6.6870000000000003,"15min":7.4710000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":94.046000000000006,"5min":94.046000000000006,"15min":94.046000000000006},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.679000000000002,"5min":13.159000000000001,"15min":10.119},"min":{"1min":0.23000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":406.68799999999999,"5min":406.68799999999999,"15min":406.68799999999999},"last":0.223},{"interface":"front","average":{"1min":1.8700000000000001,"5min":0.61899999999999999,"15min":0.41099999999999998},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19},"max":{"1min":27.094999999999999,"5min":27.094999999999999,"15min":27.094999999999999},"last":0.27600000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":15.968,"5min":3.464,"15min":1.3759999999999999},"min":{"1min":0.29099999999999998,"5min":0.216,"15min":0.216},"max":{"1min":228.90100000000001,"5min":228.90100000000001,"15min":228.90100000000001},"last":0.316},{"interface":"front","average":{"1min":21.481999999999999,"5min":13.061999999999999,"15min":14.015000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":196.82900000000001,"5min":231.864,"15min":231.864},"last":0.25600000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.823,"5min":17.213000000000001,"15min":14.393000000000001},"min":{"1min":0.316,"5min":0.309,"15min":0.309},"max":{"1min":406.71199999999999,"5min":406.71199999999999,"15min":406.71199999999999},"last":85.793999999999997},{"interface":"front","average":{"1min":25.027000000000001,"5min":5.3979999999999997,"15min":2.1320000000000001},"min":{"1min":0.34499999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":186.89500000000001,"5min":186.89500000000001,"15min":186.89500000000001},"last":0.34599999999999997}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":9.7430000000000003,"15min":12.628},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":94.183000000000007,"5min":138.089,"15min":138.089},"last":0.39500000000000002},{"interface":"front","average":{"1min":39.863999999999997,"5min":20.117000000000001,"15min":20.09},"min":{"1min":0.35499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":406.72899999999998,"5min":406.72899999999998,"15min":406.72899999999998},"last":0.33700000000000002}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":6.0010000000000003,"5min":3.8359999999999999,"15min":3.1299999999999999},"min":{"1min":0.34000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":94.302999999999997,"5min":94.302999999999997,"15min":94.302999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":43.036000000000001,"5min":19.713999999999999,"15min":17.591999999999999},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":427.69299999999998,"5min":427.69299999999998,"15min":427.69299999999998},"last":0.38500000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":2.0720000000000001,"5min":0.82799999999999996,"15min":0.626},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":27.253,"5min":27.253,"15min":27.253},"last":0.374},{"interface":"front","average":{"1min":6.0090000000000003,"5min":3.129,"15min":3.069},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":94.260999999999996,"5min":94.260999999999996,"15min":94.260999999999996},"last":0.46000000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116376,"kb_used_data":81156,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94255464,"statfs":{"total":96636764160,"available":96517595136,"internally_reserved":0,"allocated":83103744,"data_stored":85287278,"data_compressed":3073787,"data_compressed_allocated":3297280,"data_compressed_original":6955008,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":9.7919999999999998,"5min":9.5660000000000007,"15min":7.6950000000000003},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":127.30500000000001,"5min":133.09899999999999,"15min":133.09899999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":27.553999999999998,"5min":11.506,"15min":4.8399999999999999},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":451.50799999999998,"5min":451.50799999999998,"15min":451.50799999999998},"last":0.73299999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":1.5309999999999999,"15min":1.5469999999999999},"min":{"1min":0.217,"5min":0.185,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":39.906999999999996,"15min":39.906999999999996},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.32600000000000001,"5min":2.2090000000000001,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.41299999999999998,"5min":94.820999999999998,"15min":94.820999999999998},"last":0.42599999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":6.1859999999999999,"5min":3.5299999999999998,"15min":1.486},"min":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":74.963999999999999,"5min":99.332999999999998,"15min":99.332999999999998},"last":0.71199999999999997},{"interface":"front","average":{"1min":16.207000000000001,"5min":9.2159999999999993,"15min":4.1139999999999999},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":273.65100000000001,"5min":297.82100000000003,"15min":297.82100000000003},"last":0.64900000000000002}]},{"osd":4,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":5.9980000000000002,"5min":3.4809999999999999,"15min":1.4970000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":96.528999999999996,"5min":96.528999999999996,"15min":96.528999999999996},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":8.8000000000000007,"15min":9.5410000000000004},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":160.02199999999999,"15min":160.02199999999999},"last":0.91400000000000003}]},{"osd":5,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":2.1800000000000002,"15min":1.8899999999999999},"min":{"1min":0.35399999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.60399999999999998,"5min":41.645000000000003,"15min":41.645000000000003},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.47799999999999998,"5min":10.452999999999999,"15min":12.787000000000001},"min":{"1min":0.32300000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66200000000000003,"5min":282.197,"15min":282.197},"last":0.749}]},{"osd":6,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.7770000000000001,"15min":1.385},"min":{"1min":0.36199999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58299999999999996,"5min":99.355000000000004,"15min":99.355000000000004},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.499,"5min":2.8929999999999998,"15min":2.129},"min":{"1min":0.35899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.72999999999999998,"5min":52.762,"15min":52.762},"last":0.95599999999999996}]},{"osd":7,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":4.8929999999999998,"15min":4.7489999999999997},"min":{"1min":0.34499999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.68700000000000006,"5min":94.662999999999997,"15min":94.662999999999997},"last":1.0489999999999999},{"interface":"front","average":{"1min":9.0709999999999997,"5min":4.2939999999999996,"15min":1.901},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":157.089,"5min":157.089,"15min":157.089},"last":0.94099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":81391616,"data_stored":85790877,"data_compressed":4527904,"data_compressed_allocated":4829184,"data_compressed_original":9498624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":223399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27324416,"data_stored":28849251,"data_compressed":1581247,"data_compressed_allocated":1675264,"data_compressed_original":3284992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26755072,"data_stored":26700452,"data_compressed":52333,"data_compressed_allocated":57344,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41693184,"data_stored":41768117,"data_compressed":314091,"data_compressed_allocated":335872,"data_compressed_original":616025,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6438912,"data_stored":6403215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":15343616,"data_stored":15254320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":44650496,"data_stored":44915671,"data_compressed":435417,"data_compressed_allocated":471040,"data_compressed_original":931417,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33280000,"data_stored":33118894,"data_compressed":53657,"data_compressed_allocated":57344,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1929216,"data_stored":1913105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":17645568,"data_stored":17817997,"data_compressed":182394,"data_compressed_allocated":200704,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:42:03.040 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-22T18:42:03.040 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-22T18:42:03.122 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:42:03.122+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:42:03.127 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:03.127 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:03.137 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-22T18:42:03.214 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:42:03.215+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:42:03.280 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:03.280 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:03.289 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T18:42:03.332 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:42:03.333+0000 7f1e3932c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31878) UID: 0 2024-04-22T18:42:03.385 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T18:42:03.392 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-04-22T18:42:03.392 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-22T18:42:03.401 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-22T18:42:03.415 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:42:03.416+0000 7fa772387640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31914) UID: 0 2024-04-22T18:42:03.503 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-04-22T18:42:03.504 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-22T18:42:03.517 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:42:03.517+0000 7f3e89496640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31874) UID: 0 2024-04-22T18:42:03.522 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-04-22T18:42:03.522 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-22T18:42:03.538 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-04-22T18:42:03.538 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-22T18:42:03.548 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:42:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:42:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:42:03.548 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:03.554 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-04-22T18:42:03.554 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-22T18:42:03.558 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-22T18:42:03.571 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-04-22T18:42:03.571 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-22T18:42:03.587 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-04-22T18:42:03.587 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-22T18:42:03.604 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-04-22T18:42:03.604 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-22T18:42:03.617 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:42:03.618+0000 7f30a063f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31881) UID: 0 2024-04-22T18:42:03.621 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-04-22T18:42:03.621 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-22T18:42:03.674 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:42:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:42:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:42:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:35 4.f 4:f0ed3d49:::smithi08937929-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:42:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:42:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 189.483373903, 2024-04-22T18:42:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.81977801900000002, 2024-04-22T18:42:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.675 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 35 2024-04-22T18:42:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:42:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191527+0000", 2024-04-22T18:42:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191526+0000", 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191530+0000", 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5580000000000001e-06 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191532+0000", 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8260000000000001e-06 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191536+0000", 2024-04-22T18:42:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-04-22T18:42:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517843+0000", 2024-04-22T18:42:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.326307668 2024-04-22T18:42:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.517913+0000", 2024-04-22T18:42:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0261999999999999e-05 2024-04-22T18:42:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518029+0000", 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115982 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.518413+0000", 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038371400000000001 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011229+0000", 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.492815683 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:03.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011265+0000", 2024-04-22T18:42:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6310999999999998e-05 2024-04-22T18:42:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011305+0000", 2024-04-22T18:42:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9384000000000001e-05 2024-04-22T18:42:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:36 4.f 4:f0ed3d49:::smithi08937929-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:42:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:42:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 189.483362828, 2024-04-22T18:42:03.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.82002925299999996, 2024-04-22T18:42:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 36 2024-04-22T18:42:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:42:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191538+0000", 2024-04-22T18:42:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191537+0000", 2024-04-22T18:42:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191539+0000", 2024-04-22T18:42:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.474e-06 2024-04-22T18:42:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191542+0000", 2024-04-22T18:42:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.971e-06 2024-04-22T18:42:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.191544+0000", 2024-04-22T18:42:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.559e-06 2024-04-22T18:42:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609519+0000", 2024-04-22T18:42:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.41797419600000002 2024-04-22T18:42:03.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:54.609590+0000", 2024-04-22T18:42:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1482000000000004e-05 2024-04-22T18:42:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011357+0000", 2024-04-22T18:42:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40176710100000002 2024-04-22T18:42:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011453+0000", 2024-04-22T18:42:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5717000000000005e-05 2024-04-22T18:42:03.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:38:55.011567+0000", 2024-04-22T18:42:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114027 2024-04-22T18:42:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3504 4.9 4:96737e19:::smithi08937929-4:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:42:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:42:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 88.264038823999996, 2024-04-22T18:42:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.951755291, 2024-04-22T18:42:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.685 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3504 2024-04-22T18:42:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:42:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410862+0000", 2024-04-22T18:42:03.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410861+0000", 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410863+0000", 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.635e-06 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410866+0000", 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8499999999999998e-06 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.410868+0000", 2024-04-22T18:42:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.497e-06 2024-04-22T18:42:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615137+0000", 2024-04-22T18:42:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20426903499999999 2024-04-22T18:42:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.615218+0000", 2024-04-22T18:42:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0860999999999997e-05 2024-04-22T18:42:03.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.362617+0000", 2024-04-22T18:42:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.74739882700000004 2024-04-22T18:42:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4194 4.b 4:d3068ee3:::smithi08937929-29:head [sparse-read 0~1980000,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:42:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:42:03.689 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.510779753000001, 2024-04-22T18:42:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.294164774, 2024-04-22T18:42:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4194 2024-04-22T18:42:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:42:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164121+0000", 2024-04-22T18:42:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164120+0000", 2024-04-22T18:42:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164125+0000", 2024-04-22T18:42:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2899999999999996e-06 2024-04-22T18:42:03.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164128+0000", 2024-04-22T18:42:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8700000000000002e-06 2024-04-22T18:42:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164132+0000", 2024-04-22T18:42:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.151e-06 2024-04-22T18:42:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164188+0000", 2024-04-22T18:42:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6912e-05 2024-04-22T18:42:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164201+0000", 2024-04-22T18:42:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2466000000000001e-05 2024-04-22T18:42:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.242879+0000", 2024-04-22T18:42:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0786779710000001 2024-04-22T18:42:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243364+0000", 2024-04-22T18:42:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000485081 2024-04-22T18:42:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456588+0000", 2024-04-22T18:42:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.21322370900000001 2024-04-22T18:42:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.456674+0000", 2024-04-22T18:42:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6079e-05 2024-04-22T18:42:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.458285+0000", 2024-04-22T18:42:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001611716 2024-04-22T18:42:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4195 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:42:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:42:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.510762131, 2024-04-22T18:42:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.395681808, 2024-04-22T18:42:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.695 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4195 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164138+0000", 2024-04-22T18:42:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164141+0000", 2024-04-22T18:42:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7160000000000001e-06 2024-04-22T18:42:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.164147+0000", 2024-04-22T18:42:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0509999999999996e-06 2024-04-22T18:42:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264244+0000", 2024-04-22T18:42:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100097063 2024-04-22T18:42:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264319+0000", 2024-04-22T18:42:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5729000000000006e-05 2024-04-22T18:42:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264331+0000", 2024-04-22T18:42:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048999999999999e-05 2024-04-22T18:42:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243392+0000", 2024-04-22T18:42:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97906041099999996 2024-04-22T18:42:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.243451+0000", 2024-04-22T18:42:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8717e-05 2024-04-22T18:42:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558398+0000", 2024-04-22T18:42:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31494718500000002 2024-04-22T18:42:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.558490+0000", 2024-04-22T18:42:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2529999999999997e-05 2024-04-22T18:42:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559820+0000", 2024-04-22T18:42:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013299060000000001 2024-04-22T18:42:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4196 4.b 4:d3068ee3:::smithi08937929-29:head [read 0~2958065,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-04-22T18:42:03.700 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:42:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.410623383999997, 2024-04-22T18:42:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2963829229999999, 2024-04-22T18:42:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4196 2024-04-22T18:42:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:42:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264277+0000", 2024-04-22T18:42:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264275+0000", 2024-04-22T18:42:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:42:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264290+0000", 2024-04-22T18:42:03.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5373e-05 2024-04-22T18:42:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264300+0000", 2024-04-22T18:42:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.747e-06 2024-04-22T18:42:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.264314+0000", 2024-04-22T18:42:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.396e-05 2024-04-22T18:42:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364450+0000", 2024-04-22T18:42:03.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013662500000001 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.364459+0000", 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0710000000000001e-06 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245690+0000", 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88123066400000005 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.245766+0000", 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5938000000000004e-05 2024-04-22T18:42:03.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559874+0000", 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.31410827499999999 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.559930+0000", 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5367e-05 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.560660+0000", 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073043800000000003 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.705 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4182 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:42:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:42:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.746940539000001, 2024-04-22T18:42:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2059652709999999, 2024-04-22T18:42:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4182 2024-04-22T18:42:03.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:42:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927960+0000", 2024-04-22T18:42:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927959+0000", 2024-04-22T18:42:03.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:03.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927962+0000", 2024-04-22T18:42:03.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4269999999999998e-06 2024-04-22T18:42:03.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927968+0000", 2024-04-22T18:42:03.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.387e-06 2024-04-22T18:42:03.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028037+0000", 2024-04-22T18:42:03.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068544 2024-04-22T18:42:03.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028078+0000", 2024-04-22T18:42:03.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1523999999999997e-05 2024-04-22T18:42:03.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028087+0000", 2024-04-22T18:42:03.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3140000000000004e-06 2024-04-22T18:42:03.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.129951+0000", 2024-04-22T18:42:03.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1018649229999999 2024-04-22T18:42:03.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.130067+0000", 2024-04-22T18:42:03.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000115222 2024-04-22T18:42:03.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.133925+0000", 2024-04-22T18:42:03.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00385849 2024-04-22T18:42:03.710 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.710 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.710 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.711 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4183 4.c 4:35eda2ca:::smithi08937929-12:head [sparse-read 0~1697474] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:42:03.711 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:42:03.711 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.646850106999999, 2024-04-22T18:42:03.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1063017610000001, 2024-04-22T18:42:03.711 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.711 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:03.711 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.711 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.711 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.711 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4183 2024-04-22T18:42:03.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.711 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:42:03.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028050+0000", 2024-04-22T18:42:03.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028049+0000", 2024-04-22T18:42:03.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:03.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028055+0000", 2024-04-22T18:42:03.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-04-22T18:42:03.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028064+0000", 2024-04-22T18:42:03.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2830000000000001e-06 2024-04-22T18:42:03.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028071+0000", 2024-04-22T18:42:03.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2740000000000002e-06 2024-04-22T18:42:03.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028126+0000", 2024-04-22T18:42:03.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4771999999999997e-05 2024-04-22T18:42:03.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028134+0000", 2024-04-22T18:42:03.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5599999999999994e-06 2024-04-22T18:42:03.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134109+0000", 2024-04-22T18:42:03.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.105974864 2024-04-22T18:42:03.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134169+0000", 2024-04-22T18:42:03.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0167999999999997e-05 2024-04-22T18:42:03.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134352+0000", 2024-04-22T18:42:03.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018293699999999999 2024-04-22T18:42:03.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.715 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4190 4.c 4:3227431c:::smithi08937929-35:head [sparse-read 0~1258562,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.646824604000003, 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1098740030000001, 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4190 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028076+0000", 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028075+0000", 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028080+0000", 2024-04-22T18:42:03.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.561e-06 2024-04-22T18:42:03.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028085+0000", 2024-04-22T18:42:03.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0150000000000003e-06 2024-04-22T18:42:03.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028090+0000", 2024-04-22T18:42:03.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.169e-06 2024-04-22T18:42:03.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028175+0000", 2024-04-22T18:42:03.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4814999999999993e-05 2024-04-22T18:42:03.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028181+0000", 2024-04-22T18:42:03.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5490000000000003e-06 2024-04-22T18:42:03.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134400+0000", 2024-04-22T18:42:03.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1062186869999999 2024-04-22T18:42:03.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.134552+0000", 2024-04-22T18:42:03.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015210500000000001 2024-04-22T18:42:03.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137950+0000", 2024-04-22T18:42:03.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033978210000000001 2024-04-22T18:42:03.720 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.720 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.720 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.720 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4191 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:42:03.720 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:42:03.720 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.646741855999998, 2024-04-22T18:42:03.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.110917953, 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4191 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028159+0000", 2024-04-22T18:42:03.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028157+0000", 2024-04-22T18:42:03.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:03.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028164+0000", 2024-04-22T18:42:03.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.511e-06 2024-04-22T18:42:03.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028172+0000", 2024-04-22T18:42:03.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0339999999999997e-06 2024-04-22T18:42:03.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028179+0000", 2024-04-22T18:42:03.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8630000000000002e-06 2024-04-22T18:42:03.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028202+0000", 2024-04-22T18:42:03.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3635999999999999e-05 2024-04-22T18:42:03.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028209+0000", 2024-04-22T18:42:03.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4629999999999998e-06 2024-04-22T18:42:03.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.137970+0000", 2024-04-22T18:42:03.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1097613479999999 2024-04-22T18:42:03.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.138034+0000", 2024-04-22T18:42:03.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3551000000000002e-05 2024-04-22T18:42:03.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.139077+0000", 2024-04-22T18:42:03.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001042715 2024-04-22T18:42:03.725 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.725 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.725 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.725 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4192 4.c 4:3227431c:::smithi08937929-35:head [read 0~3109328,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:42:03.725 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:42:03.725 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.646716556000001, 2024-04-22T18:42:03.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.212555542, 2024-04-22T18:42:03.725 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.726 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:03.726 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.726 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.726 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.726 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4192 2024-04-22T18:42:03.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.726 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:42:03.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028184+0000", 2024-04-22T18:42:03.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028183+0000", 2024-04-22T18:42:03.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:03.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028187+0000", 2024-04-22T18:42:03.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6119999999999999e-06 2024-04-22T18:42:03.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028192+0000", 2024-04-22T18:42:03.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3489999999999999e-06 2024-04-22T18:42:03.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028197+0000", 2024-04-22T18:42:03.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7489999999999998e-06 2024-04-22T18:42:03.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028228+0000", 2024-04-22T18:42:03.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1328999999999998e-05 2024-04-22T18:42:03.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028234+0000", 2024-04-22T18:42:03.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1129999999999997e-06 2024-04-22T18:42:03.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239245+0000", 2024-04-22T18:42:03.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.211011032 2024-04-22T18:42:03.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.239384+0000", 2024-04-22T18:42:03.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013902000000000001 2024-04-22T18:42:03.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.240739+0000", 2024-04-22T18:42:03.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0013550389999999999 2024-04-22T18:42:03.730 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.730 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.730 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.730 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4692 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:42:03.730 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:42:03.730 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.415760677000002, 2024-04-22T18:42:03.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80332623700000005, 2024-04-22T18:42:03.730 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.730 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:03.731 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.731 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.731 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.731 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4692 2024-04-22T18:42:03.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.731 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:42:03.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259140+0000", 2024-04-22T18:42:03.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259139+0000", 2024-04-22T18:42:03.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:03.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259145+0000", 2024-04-22T18:42:03.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4860000000000002e-06 2024-04-22T18:42:03.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259151+0000", 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1160000000000004e-06 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259156+0000", 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6489999999999999e-06 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259191+0000", 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4737999999999998e-05 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259197+0000", 2024-04-22T18:42:03.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9160000000000004e-06 2024-04-22T18:42:03.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060062+0000", 2024-04-22T18:42:03.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80086472900000005 2024-04-22T18:42:03.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.060225+0000", 2024-04-22T18:42:03.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000162771 2024-04-22T18:42:03.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062466+0000", 2024-04-22T18:42:03.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410569999999999 2024-04-22T18:42:03.735 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.735 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.735 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.735 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4693 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:42:03.735 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:42:03.735 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.415738390999998, 2024-04-22T18:42:03.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80370012999999996, 2024-04-22T18:42:03.735 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.735 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:03.735 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.735 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.735 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.735 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4693 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259162+0000", 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259161+0000", 2024-04-22T18:42:03.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259164+0000", 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1109999999999999e-06 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259169+0000", 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6260000000000003e-06 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259174+0000", 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7659999999999998e-06 2024-04-22T18:42:03.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259231+0000", 2024-04-22T18:42:03.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6827000000000003e-05 2024-04-22T18:42:03.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259239+0000", 2024-04-22T18:42:03.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6889999999999993e-06 2024-04-22T18:42:03.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062492+0000", 2024-04-22T18:42:03.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.80325246699999997 2024-04-22T18:42:03.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062573+0000", 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1508000000000006e-05 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.062862+0000", 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028893499999999998 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4694 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e151)", 2024-04-22T18:42:03.739 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:42:03.740 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.415722109999997, 2024-04-22T18:42:03.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.90474754099999999, 2024-04-22T18:42:03.740 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.740 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:03.740 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.740 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.740 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.740 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4694 2024-04-22T18:42:03.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.740 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:42:03.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:42:03.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259178+0000", 2024-04-22T18:42:03.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:03.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259182+0000", 2024-04-22T18:42:03.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8700000000000002e-06 2024-04-22T18:42:03.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259187+0000", 2024-04-22T18:42:03.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1669999999999998e-06 2024-04-22T18:42:03.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.259191+0000", 2024-04-22T18:42:03.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6210000000000003e-06 2024-04-22T18:42:03.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.359347+0000", 2024-04-22T18:42:03.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100155518 2024-04-22T18:42:03.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:27.359356+0000", 2024-04-22T18:42:03.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6519999999999995e-06 2024-04-22T18:42:03.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163126+0000", 2024-04-22T18:42:03.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.803770816 2024-04-22T18:42:03.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163217+0000", 2024-04-22T18:42:03.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0598000000000002e-05 2024-04-22T18:42:03.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:28.163926+0000", 2024-04-22T18:42:03.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000708856 2024-04-22T18:42:03.744 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.744 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.744 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.744 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4791 4.7 4:e117a562:::smithi08937929-33:head [write 591712~786646] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:42:03.744 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.734465+0000", 2024-04-22T18:42:03.745 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.940435655999998, 2024-04-22T18:42:03.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4316495570000001, 2024-04-22T18:42:03.745 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.745 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:03.745 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.745 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.745 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.745 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4791 2024-04-22T18:42:03.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.745 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734465+0000", 2024-04-22T18:42:03.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734465+0000", 2024-04-22T18:42:03.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734460+0000", 2024-04-22T18:42:03.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999957 2024-04-22T18:42:03.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734848+0000", 2024-04-22T18:42:03.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000387353 2024-04-22T18:42:03.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734865+0000", 2024-04-22T18:42:03.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7486000000000002e-05 2024-04-22T18:42:03.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.734875+0000", 2024-04-22T18:42:03.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-06 2024-04-22T18:42:03.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761489+0000", 2024-04-22T18:42:03.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026613405 2024-04-22T18:42:03.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761495+0000", 2024-04-22T18:42:03.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4350000000000002e-06 2024-04-22T18:42:03.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.961065+0000", 2024-04-22T18:42:03.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.199569608 2024-04-22T18:42:03.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.961155+0000", 2024-04-22T18:42:03.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0109000000000001e-05 2024-04-22T18:42:03.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:42:03.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.961254+0000", 2024-04-22T18:42:03.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9072000000000003e-05 2024-04-22T18:42:03.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:03.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.964390+0000", 2024-04-22T18:42:03.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031362009999999999 2024-04-22T18:42:03.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:03.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166027+0000", 2024-04-22T18:42:03.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.201636589 2024-04-22T18:42:03.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:03.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166075+0000", 2024-04-22T18:42:03.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7998999999999999e-05 2024-04-22T18:42:03.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166114+0000", 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9638999999999998e-05 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4792 4.7 4:e117a562:::smithi08937929-33:head [write 1943568~668798] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742160+0000", 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.932740542000001, 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.42404771, 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.751 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.752 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.752 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4792 2024-04-22T18:42:03.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.752 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742160+0000", 2024-04-22T18:42:03.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742160+0000", 2024-04-22T18:42:03.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742158+0000", 2024-04-22T18:42:03.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:42:03.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742473+0000", 2024-04-22T18:42:03.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000315192 2024-04-22T18:42:03.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742485+0000", 2024-04-22T18:42:03.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1795e-05 2024-04-22T18:42:03.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742491+0000", 2024-04-22T18:42:03.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8760000000000002e-06 2024-04-22T18:42:03.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761511+0000", 2024-04-22T18:42:03.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019020760000000001 2024-04-22T18:42:03.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761515+0000", 2024-04-22T18:42:03.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.625e-06 2024-04-22T18:42:03.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.962761+0000", 2024-04-22T18:42:03.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.201246021 2024-04-22T18:42:03.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.962837+0000", 2024-04-22T18:42:03.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6025999999999993e-05 2024-04-22T18:42:03.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:42:03.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.962944+0000", 2024-04-22T18:42:03.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106949 2024-04-22T18:42:03.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:03.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.965124+0000", 2024-04-22T18:42:03.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021803230000000001 2024-04-22T18:42:03.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:03.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166154+0000", 2024-04-22T18:42:03.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20102966 2024-04-22T18:42:03.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:03.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166185+0000", 2024-04-22T18:42:03.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0713999999999998e-05 2024-04-22T18:42:03.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.166208+0000", 2024-04-22T18:42:03.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2969999999999999e-05 2024-04-22T18:42:03.757 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.757 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.757 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.757 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4793 4.7 4:e117a562:::smithi08937929-33:head [setxattr _header (56) in=7b,truncate 2991712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:42:03.757 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742495+0000", 2024-04-22T18:42:03.758 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.932405630000002, 2024-04-22T18:42:03.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5238806949999999, 2024-04-22T18:42:03.758 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.758 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:03.758 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.758 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.758 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.758 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4793 2024-04-22T18:42:03.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.758 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742495+0000", 2024-04-22T18:42:03.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742495+0000", 2024-04-22T18:42:03.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742494+0000", 2024-04-22T18:42:03.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:03.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742498+0000", 2024-04-22T18:42:03.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3790000000000002e-06 2024-04-22T18:42:03.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742501+0000", 2024-04-22T18:42:03.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9900000000000002e-06 2024-04-22T18:42:03.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742504+0000", 2024-04-22T18:42:03.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8600000000000001e-06 2024-04-22T18:42:03.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761529+0000", 2024-04-22T18:42:03.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019025301000000001 2024-04-22T18:42:03.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761532+0000", 2024-04-22T18:42:03.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4589999999999998e-06 2024-04-22T18:42:03.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.963997+0000", 2024-04-22T18:42:03.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2024646560000001 2024-04-22T18:42:03.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.964075+0000", 2024-04-22T18:42:03.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8015000000000004e-05 2024-04-22T18:42:03.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-04-22T18:42:03.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.964176+0000", 2024-04-22T18:42:03.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010080499999999999 2024-04-22T18:42:03.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:03.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.965235+0000", 2024-04-22T18:42:03.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00105959 2024-04-22T18:42:03.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:03.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266295+0000", 2024-04-22T18:42:03.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30105968599999999 2024-04-22T18:42:03.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:03.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266330+0000", 2024-04-22T18:42:03.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5219999999999998e-05 2024-04-22T18:42:03.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266376+0000", 2024-04-22T18:42:03.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5262000000000001e-05 2024-04-22T18:42:03.764 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.764 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.764 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.764 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4794 4.7 4:e117a562:::smithi08937929-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:42:03.764 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:42:03.764 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.932394516999999, 2024-04-22T18:42:03.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.524010123, 2024-04-22T18:42:03.765 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.765 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:03.765 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.765 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.765 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.765 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4794 2024-04-22T18:42:03.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.765 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:42:03.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:42:03.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742506+0000", 2024-04-22T18:42:03.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:03.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742507+0000", 2024-04-22T18:42:03.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.637e-06 2024-04-22T18:42:03.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742511+0000", 2024-04-22T18:42:03.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9890000000000003e-06 2024-04-22T18:42:03.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742514+0000", 2024-04-22T18:42:03.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.88e-06 2024-04-22T18:42:03.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.775057+0000", 2024-04-22T18:42:03.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.032543012000000003 2024-04-22T18:42:03.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.775069+0000", 2024-04-22T18:42:03.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2184000000000001e-05 2024-04-22T18:42:03.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.064059+0000", 2024-04-22T18:42:03.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.288990157 2024-04-22T18:42:03.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:03.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.064127+0000", 2024-04-22T18:42:03.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7741000000000001e-05 2024-04-22T18:42:03.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266405+0000", 2024-04-22T18:42:03.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20227735499999999 2024-04-22T18:42:03.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266458+0000", 2024-04-22T18:42:03.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3430000000000002e-05 2024-04-22T18:42:03.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:33.266516+0000", 2024-04-22T18:42:03.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8038000000000002e-05 2024-04-22T18:42:03.770 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.770 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.770 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.770 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4795 4.1f 4.d9a85c1f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e155)", 2024-04-22T18:42:03.770 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:42:03.770 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.932384147, 2024-04-22T18:42:03.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.118060236, 2024-04-22T18:42:03.771 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.771 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:42:03.771 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.771 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.771 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.771 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4795 2024-04-22T18:42:03.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.771 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:42:03.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742516+0000", 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742518+0000", 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.584e-06 2024-04-22T18:42:03.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742520+0000", 2024-04-22T18:42:03.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7140000000000002e-06 2024-04-22T18:42:03.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.742523+0000", 2024-04-22T18:42:03.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3609999999999999e-06 2024-04-22T18:42:03.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761544+0000", 2024-04-22T18:42:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.019020925000000001 2024-04-22T18:42:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:31.761547+0000", 2024-04-22T18:42:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7859999999999998e-06 2024-04-22T18:42:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.860553+0000", 2024-04-22T18:42:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0990060859999999 2024-04-22T18:42:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:32.860577+0000", 2024-04-22T18:42:03.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3064000000000001e-05 2024-04-22T18:42:03.775 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.775 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.775 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.775 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:5189 4.19 4:9ea6f51f:::smithi08937929-34:head [call cas.references_chunk in=19b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e174)", 2024-04-22T18:42:03.775 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:50.743728+0000", 2024-04-22T18:42:03.775 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.931172224999999, 2024-04-22T18:42:03.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5121397000000001, 2024-04-22T18:42:03.775 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:03.775 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5189 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743728+0000", 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743728+0000", 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:03.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:03.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743726+0000", 2024-04-22T18:42:03.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:42:03.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:03.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743736+0000", 2024-04-22T18:42:03.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9890000000000004e-06 2024-04-22T18:42:03.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:03.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743750+0000", 2024-04-22T18:42:03.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3533000000000001e-05 2024-04-22T18:42:03.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743760+0000", 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0121e-05 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743807+0000", 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6972999999999999e-05 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743812+0000", 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.845187+0000", 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101374929 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.845194+0000", 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7719999999999997e-06 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:52.255561+0000", 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4103678770000001 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:03.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:52.255766+0000", 2024-04-22T18:42:03.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000204093 2024-04-22T18:42:03.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:03.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:03.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:03.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:52.255868+0000", 2024-04-22T18:42:03.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102332 2024-04-22T18:42:03.780 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.780 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.780 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.780 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:03.780 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:03.780 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:03.782 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T18:42:03.782 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:42:03.782+0000 7f8615544640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31883) UID: 0 2024-04-22T18:42:03.795 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'unique_pool_0' removed 2024-04-22T18:42:03.801 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:42:03.809 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-22T18:42:03.819 INFO:tasks.thrashosds:joining thrashosds 2024-04-22T18:42:03.900 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:03.900 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:42:03.900 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:42:03.900 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:03.910 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-22T18:42:04.016 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:04.017 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:42:04.017 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:42:04.017 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:42:04.017 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:04.026 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-22T18:42:04.132 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:42:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:42:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:42:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:80 3.0 e34/19 3:31abd154:::501e9513bf6610b47bff18678a440034cffd586d64073d7151824cd8f9c525f9:head v 34'35, mlcod=31'29)", 2024-04-22T18:42:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:42:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 179.67069227299999, 2024-04-22T18:42:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20011964400000001, 2024-04-22T18:42:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:42:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461172+0000", 2024-04-22T18:42:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461171+0000", 2024-04-22T18:42:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461266+0000", 2024-04-22T18:42:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4707e-05 2024-04-22T18:42:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461276+0000", 2024-04-22T18:42:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0171e-05 2024-04-22T18:42:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.461282+0000", 2024-04-22T18:42:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0059999999999998e-06 2024-04-22T18:42:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660588+0000", 2024-04-22T18:42:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19930647000000001 2024-04-22T18:42:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.660605+0000", 2024-04-22T18:42:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7214000000000002e-05 2024-04-22T18:42:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661263+0000", 2024-04-22T18:42:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065747199999999998 2024-04-22T18:42:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:04.661291+0000", 2024-04-22T18:42:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-04-22T18:42:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:947 3.0 e37/19 3:16f69222:::746f89d4f500443be9a73f62a7bd842dd869addcd0600a43d66971e3a94789b9:head v 37'41, mlcod=34'39)", 2024-04-22T18:42:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:42:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 175.794018451, 2024-04-22T18:42:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20348606299999999, 2024-04-22T18:42:04.136 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337846+0000", 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.337845+0000", 2024-04-22T18:42:04.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.338990+0000", 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00114474 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.339004+0000", 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4357e-05 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.439075+0000", 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100070595 2024-04-22T18:42:04.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539295+0000", 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100220538 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.539339+0000", 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.384e-05 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541302+0000", 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019629920000000002 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:39:08.541332+0000", 2024-04-22T18:42:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9485999999999999e-05 2024-04-22T18:42:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:2744 4.4 e74/63 4:24717d05:::smithi08937929-42:head v 74'27, mlcod=74'25)", 2024-04-22T18:42:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:42:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 119.589214928, 2024-04-22T18:42:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20251403700000001, 2024-04-22T18:42:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.140 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542649+0000", 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542638+0000", 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999989 2024-04-22T18:42:04.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542956+0000", 2024-04-22T18:42:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031814400000000002 2024-04-22T18:42:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.542976+0000", 2024-04-22T18:42:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0234e-05 2024-04-22T18:42:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.643056+0000", 2024-04-22T18:42:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100079906 2024-04-22T18:42:04.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743189+0000", 2024-04-22T18:42:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100133152 2024-04-22T18:42:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.743216+0000", 2024-04-22T18:42:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7036999999999999e-05 2024-04-22T18:42:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745117+0000", 2024-04-22T18:42:04.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001900742 2024-04-22T18:42:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.745163+0000", 2024-04-22T18:42:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5874999999999997e-05 2024-04-22T18:42:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3451 4.d 4:bee1184f:::smithi08937929-7:head [read 0~3613505,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e101)", 2024-04-22T18:42:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:42:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.204684600999997, 2024-04-22T18:42:04.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20164568099999999, 2024-04-22T18:42:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3451 2024-04-22T18:42:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:42:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:42:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927179+0000", 2024-04-22T18:42:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927183+0000", 2024-04-22T18:42:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1110000000000001e-06 2024-04-22T18:42:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927188+0000", 2024-04-22T18:42:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1569999999999999e-06 2024-04-22T18:42:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:31.927192+0000", 2024-04-22T18:42:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4710000000000001e-06 2024-04-22T18:42:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027382+0000", 2024-04-22T18:42:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100189341 2024-04-22T18:42:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting_for_map not empty", 2024-04-22T18:42:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.027387+0000", 2024-04-22T18:42:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8400000000000002e-06 2024-04-22T18:42:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127638+0000", 2024-04-22T18:42:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10025102499999999 2024-04-22T18:42:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.127740+0000", 2024-04-22T18:42:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102749 2024-04-22T18:42:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:32.128825+0000", 2024-04-22T18:42:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001084593 2024-04-22T18:42:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:151 3.0 e114/63 3:28e0209b:::8e39c7491c7cfedbbdb99e4a5c28b7f9dc3617c5c76b829cb1e3f80718625fdf:head v 114'213, mlcod=110'210)", 2024-04-22T18:42:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:42:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 77.614216518000006, 2024-04-22T18:42:04.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20074001299999999, 2024-04-22T18:42:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:42:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517647+0000", 2024-04-22T18:42:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517640+0000", 2024-04-22T18:42:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999928 2024-04-22T18:42:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517652+0000", 2024-04-22T18:42:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1824000000000001e-05 2024-04-22T18:42:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.517660+0000", 2024-04-22T18:42:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1419999999999993e-06 2024-04-22T18:42:04.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617729+0000", 2024-04-22T18:42:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100068723 2024-04-22T18:42:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617770+0000", 2024-04-22T18:42:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1391999999999999e-05 2024-04-22T18:42:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.617793+0000", 2024-04-22T18:42:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2807999999999999e-05 2024-04-22T18:42:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718361+0000", 2024-04-22T18:42:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100567473 2024-04-22T18:42:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:46.718387+0000", 2024-04-22T18:42:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6826e-05 2024-04-22T18:42:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.153 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3862 4.d 4:bee1184f:::smithi08937929-7:head [write 3301257~278667] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:42:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:42:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.869314247999995, 2024-04-22T18:42:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301875958, 2024-04-22T18:42:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3862 2024-04-22T18:42:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.154 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:42:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262550+0000", 2024-04-22T18:42:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262548+0000", 2024-04-22T18:42:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:04.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262622+0000", 2024-04-22T18:42:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3747999999999999e-05 2024-04-22T18:42:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.262632+0000", 2024-04-22T18:42:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0767e-05 2024-04-22T18:42:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362705+0000", 2024-04-22T18:42:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100073157 2024-04-22T18:42:04.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2846999999999997e-05 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362853+0000", 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4498000000000002e-05 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362992+0000", 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013879699999999999 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.366534+0000", 2024-04-22T18:42:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035427969999999999 2024-04-22T18:42:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564353+0000", 2024-04-22T18:42:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.197818893 2024-04-22T18:42:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564385+0000", 2024-04-22T18:42:04.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1717000000000001e-05 2024-04-22T18:42:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564426+0000", 2024-04-22T18:42:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0654000000000001e-05 2024-04-22T18:42:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3863 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3579924] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:42:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:42:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.769146856000006, 2024-04-22T18:42:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20206560900000001, 2024-04-22T18:42:04.159 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3863 2024-04-22T18:42:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:42:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362717+0000", 2024-04-22T18:42:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362715+0000", 2024-04-22T18:42:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362725+0000", 2024-04-22T18:42:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.006e-05 2024-04-22T18:42:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362732+0000", 2024-04-22T18:42:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6780000000000003e-06 2024-04-22T18:42:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362737+0000", 2024-04-22T18:42:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2089999999999996e-06 2024-04-22T18:42:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462875+0000", 2024-04-22T18:42:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10013834100000001 2024-04-22T18:42:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462963+0000", 2024-04-22T18:42:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7792000000000002e-05 2024-04-22T18:42:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463127+0000", 2024-04-22T18:42:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000163688 2024-04-22T18:42:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463692+0000", 2024-04-22T18:42:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00056466299999999997 2024-04-22T18:42:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564720+0000", 2024-04-22T18:42:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10102826199999999 2024-04-22T18:42:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564752+0000", 2024-04-22T18:42:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1936000000000001e-05 2024-04-22T18:42:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564783+0000", 2024-04-22T18:42:04.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0919e-05 2024-04-22T18:42:04.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.167 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.167 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3864 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:42:04.167 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:42:04.167 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.769113472000001, 2024-04-22T18:42:04.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20218423599999999, 2024-04-22T18:42:04.167 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.167 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:04.167 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:04.167 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3864 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362750+0000", 2024-04-22T18:42:04.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362753+0000", 2024-04-22T18:42:04.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6989999999999999e-06 2024-04-22T18:42:04.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.362758+0000", 2024-04-22T18:42:04.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1959999999999999e-06 2024-04-22T18:42:04.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.462871+0000", 2024-04-22T18:42:04.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100113641 2024-04-22T18:42:04.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463385+0000", 2024-04-22T18:42:04.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051307100000000003 2024-04-22T18:42:04.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:04.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.463457+0000", 2024-04-22T18:42:04.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2778e-05 2024-04-22T18:42:04.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564814+0000", 2024-04-22T18:42:04.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10135696299999999 2024-04-22T18:42:04.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564868+0000", 2024-04-22T18:42:04.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3872000000000002e-05 2024-04-22T18:42:04.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:49.564935+0000", 2024-04-22T18:42:04.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6545999999999996e-05 2024-04-22T18:42:04.172 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.172 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.172 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.172 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4030 4.d 4:bee1184f:::smithi08937929-7:head [write 615222~472039] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:42:04.172 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:42:04.172 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.172664569999995, 2024-04-22T18:42:04.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109028217, 2024-04-22T18:42:04.172 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.172 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.173 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:04.173 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:04.173 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:04.173 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4030 2024-04-22T18:42:04.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.173 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:42:04.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959199+0000", 2024-04-22T18:42:04.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959177+0000", 2024-04-22T18:42:04.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999776 2024-04-22T18:42:04.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959331+0000", 2024-04-22T18:42:04.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015334600000000001 2024-04-22T18:42:04.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959358+0000", 2024-04-22T18:42:04.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7220999999999999e-05 2024-04-22T18:42:04.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.959371+0000", 2024-04-22T18:42:04.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3489000000000001e-05 2024-04-22T18:42:04.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059523+0000", 2024-04-22T18:42:04.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100152059 2024-04-22T18:42:04.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059588+0000", 2024-04-22T18:42:04.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4493999999999996e-05 2024-04-22T18:42:04.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:04.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.059694+0000", 2024-04-22T18:42:04.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010584199999999999 2024-04-22T18:42:04.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068082+0000", 2024-04-22T18:42:04.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083880589999999998 2024-04-22T18:42:04.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068138+0000", 2024-04-22T18:42:04.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6196999999999997e-05 2024-04-22T18:42:04.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068185+0000", 2024-04-22T18:42:04.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7318000000000003e-05 2024-04-22T18:42:04.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068228+0000", 2024-04-22T18:42:04.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2376999999999998e-05 2024-04-22T18:42:04.178 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.178 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.178 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.178 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4031 4.d 4:bee1184f:::smithi08937929-7:head [write 1580652~489442] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:42:04.178 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:42:04.178 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.167363578000007, 2024-04-22T18:42:04.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20571598299999999, 2024-04-22T18:42:04.178 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.179 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.179 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:04.179 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:04.179 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:04.179 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4031 2024-04-22T18:42:04.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.179 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:42:04.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964500+0000", 2024-04-22T18:42:04.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964480+0000", 2024-04-22T18:42:04.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999795 2024-04-22T18:42:04.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964628+0000", 2024-04-22T18:42:04.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148109 2024-04-22T18:42:04.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964651+0000", 2024-04-22T18:42:04.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3215999999999999e-05 2024-04-22T18:42:04.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.964664+0000", 2024-04-22T18:42:04.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2673e-05 2024-04-22T18:42:04.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061135+0000", 2024-04-22T18:42:04.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096470822999999997 2024-04-22T18:42:04.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061228+0000", 2024-04-22T18:42:04.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3459000000000001e-05 2024-04-22T18:42:04.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:04.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.061376+0000", 2024-04-22T18:42:04.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148128 2024-04-22T18:42:04.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068085+0000", 2024-04-22T18:42:04.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.006708632 2024-04-22T18:42:04.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170038+0000", 2024-04-22T18:42:04.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101952716 2024-04-22T18:42:04.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170173+0000", 2024-04-22T18:42:04.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013508699999999999 2024-04-22T18:42:04.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.170216+0000", 2024-04-22T18:42:04.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3498999999999998e-05 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4032 4.d 4:bee1184f:::smithi08937929-7:head [write 2768826~626587] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.160158471000003, 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.19941629799999999, 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4032 2024-04-22T18:42:04.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.185 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:42:04.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971705+0000", 2024-04-22T18:42:04.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971703+0000", 2024-04-22T18:42:04.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:42:04.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.971996+0000", 2024-04-22T18:42:04.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029351300000000002 2024-04-22T18:42:04.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972012+0000", 2024-04-22T18:42:04.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5569e-05 2024-04-22T18:42:04.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972019+0000", 2024-04-22T18:42:04.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7950000000000008e-06 2024-04-22T18:42:04.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.063940+0000", 2024-04-22T18:42:04.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.091920162 2024-04-22T18:42:04.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064021+0000", 2024-04-22T18:42:04.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1865000000000003e-05 2024-04-22T18:42:04.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:04.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.064145+0000", 2024-04-22T18:42:04.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123893 2024-04-22T18:42:04.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068317+0000", 2024-04-22T18:42:04.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041720799999999999 2024-04-22T18:42:04.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171042+0000", 2024-04-22T18:42:04.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102724893 2024-04-22T18:42:04.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171085+0000", 2024-04-22T18:42:04.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2619e-05 2024-04-22T18:42:04.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.171122+0000", 2024-04-22T18:42:04.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6838999999999997e-05 2024-04-22T18:42:04.189 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.190 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.190 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.190 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4033 4.d 4:bee1184f:::smithi08937929-7:head [setxattr _header (55) in=7b,truncate 3815222] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:42:04.190 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:42:04.190 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.159834989000004, 2024-04-22T18:42:04.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.297586038, 2024-04-22T18:42:04.190 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.190 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.190 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:04.191 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:04.191 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:04.191 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4033 2024-04-22T18:42:04.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.191 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:42:04.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972029+0000", 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972028+0000", 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972032+0000", 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8330000000000004e-06 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972035+0000", 2024-04-22T18:42:04.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0139999999999999e-06 2024-04-22T18:42:04.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972038+0000", 2024-04-22T18:42:04.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9579999999999999e-06 2024-04-22T18:42:04.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067511+0000", 2024-04-22T18:42:04.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.095472308000000006 2024-04-22T18:42:04.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067607+0000", 2024-04-22T18:42:04.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6546999999999996e-05 2024-04-22T18:42:04.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:04.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.067776+0000", 2024-04-22T18:42:04.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000169202 2024-04-22T18:42:04.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068470+0000", 2024-04-22T18:42:04.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069403100000000001 2024-04-22T18:42:04.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269540+0000", 2024-04-22T18:42:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20106934000000001 2024-04-22T18:42:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269574+0000", 2024-04-22T18:42:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4332000000000003e-05 2024-04-22T18:42:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.269615+0000", 2024-04-22T18:42:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0988000000000001e-05 2024-04-22T18:42:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4034 4.d 4:bee1184f:::smithi08937929-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2024-04-22T18:42:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:42:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.159823352000004, 2024-04-22T18:42:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39788621000000002, 2024-04-22T18:42:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4034 2024-04-22T18:42:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:42:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972041+0000", 2024-04-22T18:42:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972040+0000", 2024-04-22T18:42:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972042+0000", 2024-04-22T18:42:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.519e-06 2024-04-22T18:42:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972045+0000", 2024-04-22T18:42:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0060000000000001e-06 2024-04-22T18:42:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:54.972048+0000", 2024-04-22T18:42:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.773e-06 2024-04-22T18:42:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068266+0000", 2024-04-22T18:42:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096218029999999996 2024-04-22T18:42:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.068308+0000", 2024-04-22T18:42:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.21e-05 2024-04-22T18:42:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369703+0000", 2024-04-22T18:42:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.301395578 2024-04-22T18:42:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369802+0000", 2024-04-22T18:42:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8788000000000004e-05 2024-04-22T18:42:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:55.369927+0000", 2024-04-22T18:42:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012478200000000001 2024-04-22T18:42:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(4.1 124/123 [PushOp(4:83c955bf:::smithi08937929-28:head, version: 115'157, data_included: [1890685~548185,2699879~469889], data_size: 1018074, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:83c955bf:::smithi08937929-28:head@115'157, size: 3630345, copy_subset: [0~3630345], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3630345, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1890685, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-22T18:42:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:42:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.493255243999997, 2024-04-22T18:42:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20293218800000001, 2024-04-22T18:42:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:42:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638609+0000", 2024-04-22T18:42:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.638608+0000", 2024-04-22T18:42:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639519+0000", 2024-04-22T18:42:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091196199999999999 2024-04-22T18:42:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.639537+0000", 2024-04-22T18:42:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7455000000000001e-05 2024-04-22T18:42:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.739639+0000", 2024-04-22T18:42:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100101942 2024-04-22T18:42:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840271+0000", 2024-04-22T18:42:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10063235299999999 2024-04-22T18:42:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.840281+0000", 2024-04-22T18:42:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6509999999999996e-06 2024-04-22T18:42:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:58.841541+0000", 2024-04-22T18:42:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012600230000000001 2024-04-22T18:42:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4205 4.1 4:83c955bf:::smithi08937929-28:head [read 3065344~859648] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e130)", 2024-04-22T18:42:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:42:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.204333403, 2024-04-22T18:42:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.303433565, 2024-04-22T18:42:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-04-22T18:42:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4205 2024-04-22T18:42:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:42:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927531+0000", 2024-04-22T18:42:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927528+0000", 2024-04-22T18:42:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:42:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927535+0000", 2024-04-22T18:42:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1049999999999997e-06 2024-04-22T18:42:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927550+0000", 2024-04-22T18:42:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4365000000000001e-05 2024-04-22T18:42:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:03.927560+0000", 2024-04-22T18:42:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0360000000000001e-05 2024-04-22T18:42:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028542+0000", 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100981837 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:04.028551+0000", 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6260000000000001e-06 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229700+0000", 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.201149453 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.229861+0000", 2024-04-22T18:42:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160605 2024-04-22T18:42:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:05.230964+0000", 2024-04-22T18:42:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00110351 2024-04-22T18:42:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.13:359 3.0 e133/63 3:351d2474:::ba270d71d9c7ccc8d5c74eded70a7073f0e08c1ce33f7199f7c9db1158888734:head v 133'233, mlcod=132'228)", 2024-04-22T18:42:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:42:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.229538976000001, 2024-04-22T18:42:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20122315599999999, 2024-04-22T18:42:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:42:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902325+0000", 2024-04-22T18:42:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902324+0000", 2024-04-22T18:42:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902347+0000", 2024-04-22T18:42:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3577000000000001e-05 2024-04-22T18:42:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.902356+0000", 2024-04-22T18:42:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8179999999999993e-06 2024-04-22T18:42:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.002447+0000", 2024-04-22T18:42:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100090255 2024-04-22T18:42:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102568+0000", 2024-04-22T18:42:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100121611 2024-04-22T18:42:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.102592+0000", 2024-04-22T18:42:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3515000000000001e-05 2024-04-22T18:42:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103514+0000", 2024-04-22T18:42:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000922174 2024-04-22T18:42:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.103548+0000", 2024-04-22T18:42:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4316999999999997e-05 2024-04-22T18:42:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4502.0:4289 3.0 e134/63 3:2a3d8f37:::9341e24c4a92f7ec719282d2faa5711f9644dee45a94ba5b4c4c1071ddc0e536:head v 134'235, mlcod=133'233)", 2024-04-22T18:42:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:42:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 54.284969464, 2024-04-22T18:42:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10669205499999999, 2024-04-22T18:42:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:42:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846894+0000", 2024-04-22T18:42:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.846891+0000", 2024-04-22T18:42:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:42:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847884+0000", 2024-04-22T18:42:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00099224800000000004 2024-04-22T18:42:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.847900+0000", 2024-04-22T18:42:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6154000000000002e-05 2024-04-22T18:42:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.947999+0000", 2024-04-22T18:42:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10009889199999999 2024-04-22T18:42:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948046+0000", 2024-04-22T18:42:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7617000000000002e-05 2024-04-22T18:42:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.948072+0000", 2024-04-22T18:42:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5559999999999999e-05 2024-04-22T18:42:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953539+0000", 2024-04-22T18:42:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054668599999999996 2024-04-22T18:42:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:09.953587+0000", 2024-04-22T18:42:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7781999999999998e-05 2024-04-22T18:42:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.7.13:94 3.0 e147/63 3:2812810a:::007b860ed7f2dc72f975d8732f59ce14b4563df44fd56cdf3a173500adaeb557:head v 147'260, mlcod=147'257)", 2024-04-22T18:42:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:42:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.698576258000003, 2024-04-22T18:42:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.200869664, 2024-04-22T18:42:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:42:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433288+0000", 2024-04-22T18:42:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433287+0000", 2024-04-22T18:42:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433295+0000", 2024-04-22T18:42:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9929999999999995e-06 2024-04-22T18:42:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.433302+0000", 2024-04-22T18:42:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2740000000000002e-06 2024-04-22T18:42:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533368+0000", 2024-04-22T18:42:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100066239 2024-04-22T18:42:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533415+0000", 2024-04-22T18:42:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6669000000000003e-05 2024-04-22T18:42:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.533442+0000", 2024-04-22T18:42:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7143000000000001e-05 2024-04-22T18:42:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.634126+0000", 2024-04-22T18:42:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10068434900000001 2024-04-22T18:42:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.634157+0000", 2024-04-22T18:42:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1016000000000003e-05 2024-04-22T18:42:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:198 3.0 e147/63 3:2be6c676:::11b6849070c44d718872a26a0dc7d7224d7a6a98b28e0fde07a9f66143e32de6:head v 147'266, mlcod=147'257)", 2024-04-22T18:42:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:42:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.597804164000003, 2024-04-22T18:42:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106653649, 2024-04-22T18:42:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:42:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534060+0000", 2024-04-22T18:42:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534059+0000", 2024-04-22T18:42:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534082+0000", 2024-04-22T18:42:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2291999999999999e-05 2024-04-22T18:42:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534086+0000", 2024-04-22T18:42:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1019999999999999e-06 2024-04-22T18:42:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534088+0000", 2024-04-22T18:42:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.525e-06 2024-04-22T18:42:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.635447+0000", 2024-04-22T18:42:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101358581 2024-04-22T18:42:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.635462+0000", 2024-04-22T18:42:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5327000000000001e-05 2024-04-22T18:42:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.640689+0000", 2024-04-22T18:42:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052266329999999996 2024-04-22T18:42:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.640713+0000", 2024-04-22T18:42:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4630999999999998e-05 2024-04-22T18:42:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.13:200 3.0 e147/63 3:19f9e72c:::45a155ef5bb5a968c7feaf289313c887c41e2de64b61f8816d05ad72d02346d9:head v 147'267, mlcod=147'257)", 2024-04-22T18:42:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:42:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.597773406999998, 2024-04-22T18:42:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.109105273, 2024-04-22T18:42:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:42:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534091+0000", 2024-04-22T18:42:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534090+0000", 2024-04-22T18:42:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534228+0000", 2024-04-22T18:42:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013768600000000001 2024-04-22T18:42:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534232+0000", 2024-04-22T18:42:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4900000000000002e-06 2024-04-22T18:42:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.534235+0000", 2024-04-22T18:42:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.61e-06 2024-04-22T18:42:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.641544+0000", 2024-04-22T18:42:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.107308717 2024-04-22T18:42:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.641557+0000", 2024-04-22T18:42:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2811e-05 2024-04-22T18:42:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.643173+0000", 2024-04-22T18:42:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00161608 2024-04-22T18:42:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:23.643196+0000", 2024-04-22T18:42:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3136e-05 2024-04-22T18:42:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.233 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:04.233 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:42:04.154+0000 7f3cfee88640 -1 osd.6 186 handle_osd_map injecting map crc failure 2024-04-22T18:42:04.234 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T18:42:04.336 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:04.336 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:42:04.336 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-04-22T18:42:04.336 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:04.345 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-22T18:42:04.446 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:04.446 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-04-22T18:42:04.446 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-04-22T18:42:04.446 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-04-22T18:42:04.446 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:04.454 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-22T18:42:04.559 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-04-22T18:42:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-04-22T18:42:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-04-22T18:42:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:289 3.2 3:5f0623ce:::smithi08937929-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:42:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:42:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.91828017900001, 2024-04-22T18:42:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9176801189999999, 2024-04-22T18:42:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:42:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640725+0000", 2024-04-22T18:42:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640723+0000", 2024-04-22T18:42:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640732+0000", 2024-04-22T18:42:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5809999999999997e-06 2024-04-22T18:42:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640744+0000", 2024-04-22T18:42:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.261e-05 2024-04-22T18:42:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.640753+0000", 2024-04-22T18:42:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7460000000000006e-06 2024-04-22T18:42:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642189+0000", 2024-04-22T18:42:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001435551 2024-04-22T18:42:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:02.642200+0000", 2024-04-22T18:42:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1389999999999999e-05 2024-04-22T18:42:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.847015+0000", 2024-04-22T18:42:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2048149160000001 2024-04-22T18:42:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.848217+0000", 2024-04-22T18:42:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012020659999999999 2024-04-22T18:42:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351823+0000", 2024-04-22T18:42:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50360565099999999 2024-04-22T18:42:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.351907+0000", 2024-04-22T18:42:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4141999999999995e-05 2024-04-22T18:42:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:42:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352051+0000", 2024-04-22T18:42:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014386600000000001 2024-04-22T18:42:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453617+0000", 2024-04-22T18:42:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10156649500000001 2024-04-22T18:42:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558297+0000", 2024-04-22T18:42:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10468000399999999 2024-04-22T18:42:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558331+0000", 2024-04-22T18:42:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3518e-05 2024-04-22T18:42:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558405+0000", 2024-04-22T18:42:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4221000000000006e-05 2024-04-22T18:42:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:1 3.2 3:6d90b903:::64b5942574fb36ca1b9a2be186c6b740edf58e989056a2f88b83cd280451a438:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:42:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:42:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.386979219, 2024-04-22T18:42:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.285534422, 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172026+0000", 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.172024+0000", 2024-04-22T18:42:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-04-22T18:42:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173091+0000", 2024-04-22T18:42:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010677569999999999 2024-04-22T18:42:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173103+0000", 2024-04-22T18:42:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1241e-05 2024-04-22T18:42:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173109+0000", 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4300000000000003e-06 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173168+0000", 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9011000000000001e-05 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173177+0000", 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948501+0000", 2024-04-22T18:42:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77532387199999997 2024-04-22T18:42:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.948912+0000", 2024-04-22T18:42:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041183000000000002 2024-04-22T18:42:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950438+0000", 2024-04-22T18:42:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015253199999999999 2024-04-22T18:42:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950488+0000", 2024-04-22T18:42:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0285999999999999e-05 2024-04-22T18:42:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:42:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950601+0000", 2024-04-22T18:42:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011244899999999999 2024-04-22T18:42:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954048+0000", 2024-04-22T18:42:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003447313 2024-04-22T18:42:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457373+0000", 2024-04-22T18:42:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50332465299999996 2024-04-22T18:42:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457430+0000", 2024-04-22T18:42:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7574999999999997e-05 2024-04-22T18:42:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.457560+0000", 2024-04-22T18:42:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013023900000000001 2024-04-22T18:42:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:8 3.2 3:5f64590d:::7c47933f331d40689081b2bd8f9d7f7bdae1be1fdccb52d696047a3969e902b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:42:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:42:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.38588685000001, 2024-04-22T18:42:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3854200569999999, 2024-04-22T18:42:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:42:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173118+0000", 2024-04-22T18:42:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173117+0000", 2024-04-22T18:42:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173393+0000", 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000275278 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173399+0000", 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4590000000000003e-06 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173406+0000", 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2450000000000001e-06 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173795+0000", 2024-04-22T18:42:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038912400000000002 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173804+0000", 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.949506+0000", 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77570185700000005 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950166+0000", 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066022800000000001 2024-04-22T18:42:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352843+0000", 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40267661799999999 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.352911+0000", 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8126000000000005e-05 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.353031+0000", 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119919 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453622+0000", 2024-04-22T18:42:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100590793 2024-04-22T18:42:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558446+0000", 2024-04-22T18:42:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104824073 2024-04-22T18:42:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558479+0000", 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3362999999999999e-05 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558538+0000", 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9046999999999999e-05 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:10 3.2 3:518c37f9:::851eca84dafbf2f780c4f07291110c453349bdc9c623c17bdcd26e27badb30ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.385593257, 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3852527729999999, 2024-04-22T18:42:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173412+0000", 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173411+0000", 2024-04-22T18:42:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173749+0000", 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000338029 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173755+0000", 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0530000000000003e-06 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173759+0000", 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8190000000000002e-06 2024-04-22T18:42:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173855+0000", 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5927999999999994e-05 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173862+0000", 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4449999999999999e-06 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.950835+0000", 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77697248299999999 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951632+0000", 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079752299999999996 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355407+0000", 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40377480500000001 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355462+0000", 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4577999999999998e-05 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:42:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355571+0000", 2024-04-22T18:42:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000108964 2024-04-22T18:42:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453625+0000", 2024-04-22T18:42:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.098054397000000001 2024-04-22T18:42:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558574+0000", 2024-04-22T18:42:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104948571 2024-04-22T18:42:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558602+0000", 2024-04-22T18:42:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7966999999999998e-05 2024-04-22T18:42:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.558665+0000", 2024-04-22T18:42:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3032000000000002e-05 2024-04-22T18:42:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:11 3.2 3:718aa8aa:::9603c4f8e269a436b2f0bf376c5374e77bdd7cf58fccae60e4e65ac4696806d4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:42:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:42:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.385241639, 2024-04-22T18:42:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4852604440000001, 2024-04-22T18:42:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:42:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:42:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173763+0000", 2024-04-22T18:42:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173813+0000", 2024-04-22T18:42:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0655999999999997e-05 2024-04-22T18:42:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173819+0000", 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.468e-06 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173822+0000", 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4879999999999999e-06 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173983+0000", 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000160519 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173991+0000", 2024-04-22T18:42:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6490000000000008e-06 2024-04-22T18:42:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.951688+0000", 2024-04-22T18:42:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77769743800000002 2024-04-22T18:42:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952419+0000", 2024-04-22T18:42:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073067499999999997 2024-04-22T18:42:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355901+0000", 2024-04-22T18:42:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40348237599999998 2024-04-22T18:42:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.355953+0000", 2024-04-22T18:42:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1477000000000002e-05 2024-04-22T18:42:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:42:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356060+0000", 2024-04-22T18:42:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107114 2024-04-22T18:42:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453629+0000", 2024-04-22T18:42:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097569127000000005 2024-04-22T18:42:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658807+0000", 2024-04-22T18:42:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20517829800000001 2024-04-22T18:42:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.658876+0000", 2024-04-22T18:42:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8454000000000006e-05 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659024+0000", 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000148338 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:12 3.2 3:475e1cd9:::efaab2bd13543cdbb143e869e20eb0ec424f7f5202707859b241ed91d4674936:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.38517869499999, 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4853405259999999, 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:42:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:42:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173826+0000", 2024-04-22T18:42:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173940+0000", 2024-04-22T18:42:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011465500000000001 2024-04-22T18:42:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173946+0000", 2024-04-22T18:42:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5289999999999999e-06 2024-04-22T18:42:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173950+0000", 2024-04-22T18:42:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8290000000000001e-06 2024-04-22T18:42:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174277+0000", 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032761299999999998 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174285+0000", 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.52e-06 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.952455+0000", 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.77817000700000005 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.954035+0000", 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001580078 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356379+0000", 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40234440399999999 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356428+0000", 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8832999999999998e-05 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:42:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356532+0000", 2024-04-22T18:42:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103235 2024-04-22T18:42:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453632+0000", 2024-04-22T18:42:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.097100346000000004 2024-04-22T18:42:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659069+0000", 2024-04-22T18:42:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20543763800000001 2024-04-22T18:42:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659105+0000", 2024-04-22T18:42:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5240000000000001e-05 2024-04-22T18:42:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659167+0000", 2024-04-22T18:42:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2207999999999998e-05 2024-04-22T18:42:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:16 3.2 3:47c76cc8:::c00d18458eb20c0c0aa9c612cf9793a68445ab0ea7b9e650ecd0072c60a96c0f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:42:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.385050736, 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.485347744, 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.173954+0000", 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174234+0000", 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028047599999999999 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174240+0000", 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2079999999999997e-06 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174244+0000", 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5810000000000001e-06 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174670+0000", 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042640000000000001 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174675+0000", 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8709999999999996e-06 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054168+0000", 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.87949257300000006 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054877+0000", 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070884300000000002 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356836+0000", 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30195926000000001 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356887+0000", 2024-04-22T18:42:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0767999999999999e-05 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.356991+0000", 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104072 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453636+0000", 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.096644968999999997 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659207+0000", 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20557154699999999 2024-04-22T18:42:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659240+0000", 2024-04-22T18:42:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2665e-05 2024-04-22T18:42:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659302+0000", 2024-04-22T18:42:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2234000000000001e-05 2024-04-22T18:42:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:17 3.2 3:6ef31daa:::1cf4ff496c4480e58510f607481d976b47d84868e2c54016273c38baf590f3b0:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:42:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:42:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.38475645600001, 2024-04-22T18:42:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4851703729999999, 2024-04-22T18:42:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:42:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174249+0000", 2024-04-22T18:42:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174248+0000", 2024-04-22T18:42:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174626+0000", 2024-04-22T18:42:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00037839699999999999 2024-04-22T18:42:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174633+0000", 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0990000000000004e-06 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174636+0000", 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5540000000000002e-06 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174790+0000", 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000153807 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174793+0000", 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4970000000000001e-06 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.054902+0000", 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.88010886300000002 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055366+0000", 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046336199999999999 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.452973+0000", 2024-04-22T18:42:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39760783999999999 2024-04-22T18:42:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453046+0000", 2024-04-22T18:42:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2374999999999997e-05 2024-04-22T18:42:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:42:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453193+0000", 2024-04-22T18:42:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014722499999999999 2024-04-22T18:42:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554698+0000", 2024-04-22T18:42:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101505096 2024-04-22T18:42:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659337+0000", 2024-04-22T18:42:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.104639236 2024-04-22T18:42:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659365+0000", 2024-04-22T18:42:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7594e-05 2024-04-22T18:42:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.659419+0000", 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4058999999999999e-05 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.7.13:18 3.2 3:792a4ed0:::e1c814961b3d1531fb5ed475b3d496a0405c3a819b68a92ff9b2391fe912370f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 121.3843602, 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.585036954, 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174645+0000", 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174640+0000", 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174758+0000", 2024-04-22T18:42:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011819099999999999 2024-04-22T18:42:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174764+0000", 2024-04-22T18:42:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6729999999999997e-06 2024-04-22T18:42:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.174767+0000", 2024-04-22T18:42:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3830000000000001e-06 2024-04-22T18:42:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373669+0000", 2024-04-22T18:42:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.198901989 2024-04-22T18:42:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:03.373686+0000", 2024-04-22T18:42:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7286e-05 2024-04-22T18:42:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055408+0000", 2024-04-22T18:42:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.68172133400000001 2024-04-22T18:42:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.055938+0000", 2024-04-22T18:42:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053016599999999999 2024-04-22T18:42:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453668+0000", 2024-04-22T18:42:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.39773030799999998 2024-04-22T18:42:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453717+0000", 2024-04-22T18:42:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8884000000000002e-05 2024-04-22T18:42:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.453823+0000", 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105791 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.554715+0000", 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100891628 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759439+0000", 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20472459400000001 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759525+0000", 2024-04-22T18:42:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5653000000000001e-05 2024-04-22T18:42:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:04.759682+0000", 2024-04-22T18:42:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015711800000000001 2024-04-22T18:42:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.6.13:76 3.1 3:b44829a0:::3788286a255d538503614e5f04049b0bb54cacaa317ab102b5dee8f5b640b296:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:42:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:42:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 89.311444617000006, 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60168681599999996, 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247560+0000", 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247564+0000", 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1150000000000004e-06 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247570+0000", 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3940000000000001e-06 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.247576+0000", 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3619999999999997e-06 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.340845+0000", 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.093269199999999997 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.341043+0000", 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019834400000000001 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345106+0000", 2024-04-22T18:42:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040629780000000001 2024-04-22T18:42:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.345160+0000", 2024-04-22T18:42:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4246999999999997e-05 2024-04-22T18:42:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747047+0000", 2024-04-22T18:42:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40188678300000003 2024-04-22T18:42:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747162+0000", 2024-04-22T18:42:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011483400000000001 2024-04-22T18:42:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-04-22T18:42:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.747429+0000", 2024-04-22T18:42:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026700299999999998 2024-04-22T18:42:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849032+0000", 2024-04-22T18:42:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.101603203 2024-04-22T18:42:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849095+0000", 2024-04-22T18:42:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2597999999999999e-05 2024-04-22T18:42:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849136+0000", 2024-04-22T18:42:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1251000000000002e-05 2024-04-22T18:42:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:35.849247+0000", 2024-04-22T18:42:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011127 2024-04-22T18:42:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:3526 4.c 4:314c2fec:::smithi08937929-22:head [setxattr _header (56) in=7b,truncate 3092521] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 88.329807837000004, 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60463626000000004, 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 3526 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:42:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:42:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229197+0000", 2024-04-22T18:42:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229200+0000", 2024-04-22T18:42:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2320000000000001e-06 2024-04-22T18:42:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.229203+0000", 2024-04-22T18:42:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:42:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329292+0000", 2024-04-22T18:42:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.100088891 2024-04-22T18:42:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329357+0000", 2024-04-22T18:42:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5115000000000003e-05 2024-04-22T18:42:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329453+0000", 2024-04-22T18:42:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5802000000000002e-05 2024-04-22T18:42:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-04-22T18:42:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.329594+0000", 2024-04-22T18:42:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014157500000000001 2024-04-22T18:42:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.330172+0000", 2024-04-22T18:42:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057782799999999996 2024-04-22T18:42:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833712+0000", 2024-04-22T18:42:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50353975299999998 2024-04-22T18:42:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833774+0000", 2024-04-22T18:42:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1832e-05 2024-04-22T18:42:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:40:36.833834+0000", 2024-04-22T18:42:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9809999999999998e-05 2024-04-22T18:42:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:339 3.2 3:4159b8d1:::bb1f8e997816dd9d2f29d36c4c1859fcaffbf8763b5c8d63c7a818d8d84bac37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:42:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:42:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.783554078000002, 2024-04-22T18:42:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0092773479999999, 2024-04-22T18:42:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:42:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775451+0000", 2024-04-22T18:42:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775450+0000", 2024-04-22T18:42:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775457+0000", 2024-04-22T18:42:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2949999999999996e-06 2024-04-22T18:42:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775467+0000", 2024-04-22T18:42:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0180000000000001e-05 2024-04-22T18:42:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775475+0000", 2024-04-22T18:42:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.678e-06 2024-04-22T18:42:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.877787+0000", 2024-04-22T18:42:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.102312559 2024-04-22T18:42:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.880686+0000", 2024-04-22T18:42:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002898748 2024-04-22T18:42:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.282867+0000", 2024-04-22T18:42:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.402180961 2024-04-22T18:42:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.282945+0000", 2024-04-22T18:42:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7775999999999995e-05 2024-04-22T18:42:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:42:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283065+0000", 2024-04-22T18:42:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0001199 2024-04-22T18:42:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.285514+0000", 2024-04-22T18:42:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0024491729999999998 2024-04-22T18:42:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784375+0000", 2024-04-22T18:42:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49886154100000002 2024-04-22T18:42:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784475+0000", 2024-04-22T18:42:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9074999999999996e-05 2024-04-22T18:42:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784728+0000", 2024-04-22T18:42:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025386799999999998 2024-04-22T18:42:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:340 3.2 3:4bbc2d06:::e1c179ab87fb8e08c4a1dfc0c7cc725138181c5734d6df18f3f2545b93731e97:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:42:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:42:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.783521659000002, 2024-04-22T18:42:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009550757, 2024-04-22T18:42:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:42:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:42:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775483+0000", 2024-04-22T18:42:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775487+0000", 2024-04-22T18:42:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6839999999999998e-06 2024-04-22T18:42:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775492+0000", 2024-04-22T18:42:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6899999999999997e-06 2024-04-22T18:42:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775497+0000", 2024-04-22T18:42:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8590000000000004e-06 2024-04-22T18:42:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.880737+0000", 2024-04-22T18:42:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105240267 2024-04-22T18:42:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881308+0000", 2024-04-22T18:42:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057098399999999997 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283810+0000", 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40250177500000001 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.283883+0000", 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2430999999999998e-05 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284019+0000", 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013652299999999999 2024-04-22T18:42:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.287390+0000", 2024-04-22T18:42:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033710089999999999 2024-04-22T18:42:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784829+0000", 2024-04-22T18:42:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49743879400000002 2024-04-22T18:42:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.784893+0000", 2024-04-22T18:42:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4184999999999996e-05 2024-04-22T18:42:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785034+0000", 2024-04-22T18:42:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014114600000000001 2024-04-22T18:42:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:341 3.2 3:5b8bea68:::ab73be59042e3e18c4fe1fdbc2bbcc205779424ddf3bb423a595330be9026399:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:42:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:42:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.783502306999999, 2024-04-22T18:42:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009670804, 2024-04-22T18:42:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:42:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775503+0000", 2024-04-22T18:42:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775502+0000", 2024-04-22T18:42:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775505+0000", 2024-04-22T18:42:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0809999999999998e-06 2024-04-22T18:42:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775511+0000", 2024-04-22T18:42:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2039999999999996e-06 2024-04-22T18:42:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775515+0000", 2024-04-22T18:42:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8790000000000002e-06 2024-04-22T18:42:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881376+0000", 2024-04-22T18:42:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.105861 2024-04-22T18:42:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881839+0000", 2024-04-22T18:42:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046217200000000001 2024-04-22T18:42:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284569+0000", 2024-04-22T18:42:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.40273058900000003 2024-04-22T18:42:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284730+0000", 2024-04-22T18:42:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077800000000001 2024-04-22T18:42:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:42:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.284923+0000", 2024-04-22T18:42:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000193253 2024-04-22T18:42:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.287593+0000", 2024-04-22T18:42:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026699359999999999 2024-04-22T18:42:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785074+0000", 2024-04-22T18:42:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49748046000000001 2024-04-22T18:42:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785103+0000", 2024-04-22T18:42:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8935e-05 2024-04-22T18:42:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.785174+0000", 2024-04-22T18:42:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1001999999999995e-05 2024-04-22T18:42:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:342 3.2 3:634dda9f:::10622dc7de8047e7908c0b5cca7a43d84cdf1b7629e6f7e159b8207feba82b01:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:42:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:42:04.648 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.783484094999999, 2024-04-22T18:42:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.022499579, 2024-04-22T18:42:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:42:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:42:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775521+0000", 2024-04-22T18:42:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775524+0000", 2024-04-22T18:42:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9629999999999998e-06 2024-04-22T18:42:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775529+0000", 2024-04-22T18:42:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0320000000000003e-06 2024-04-22T18:42:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775533+0000", 2024-04-22T18:42:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7380000000000002e-06 2024-04-22T18:42:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.881884+0000", 2024-04-22T18:42:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106350294 2024-04-22T18:42:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.882492+0000", 2024-04-22T18:42:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060872699999999999 2024-04-22T18:42:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381559+0000", 2024-04-22T18:42:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.49906679199999998 2024-04-22T18:42:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381677+0000", 2024-04-22T18:42:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011743699999999999 2024-04-22T18:42:04.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:42:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381899+0000", 2024-04-22T18:42:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022272599999999999 2024-04-22T18:42:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.383440+0000", 2024-04-22T18:42:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015407719999999999 2024-04-22T18:42:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.797827+0000", 2024-04-22T18:42:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.414386741 2024-04-22T18:42:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.797887+0000", 2024-04-22T18:42:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9889000000000002e-05 2024-04-22T18:42:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.798021+0000", 2024-04-22T18:42:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000133948 2024-04-22T18:42:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.654 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:343 3.2 3:7db7298e:::00597d8527769aa920807495342cf3ba06efb8f507feae865dd89cd8ad10bda0:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:42:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:42:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.783465139999997, 2024-04-22T18:42:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.110995373, 2024-04-22T18:42:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:42:04.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:42:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775540+0000", 2024-04-22T18:42:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775542+0000", 2024-04-22T18:42:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9079999999999999e-06 2024-04-22T18:42:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775547+0000", 2024-04-22T18:42:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.994e-06 2024-04-22T18:42:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775553+0000", 2024-04-22T18:42:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2299999999999999e-06 2024-04-22T18:42:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.882519+0000", 2024-04-22T18:42:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.106966453 2024-04-22T18:42:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883103+0000", 2024-04-22T18:42:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058419400000000001 2024-04-22T18:42:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.581957+0000", 2024-04-22T18:42:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69885328599999996 2024-04-22T18:42:04.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582023+0000", 2024-04-22T18:42:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6210999999999993e-05 2024-04-22T18:42:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:42:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582146+0000", 2024-04-22T18:42:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000123669 2024-04-22T18:42:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.682293+0000", 2024-04-22T18:42:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10014662100000001 2024-04-22T18:42:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886410+0000", 2024-04-22T18:42:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20411678899999999 2024-04-22T18:42:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886445+0000", 2024-04-22T18:42:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5462e-05 2024-04-22T18:42:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.886535+0000", 2024-04-22T18:42:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9995999999999997e-05 2024-04-22T18:42:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:344 3.1 3:9da67ae4:::04c6b48a110b3f74550f78f0899df697160a1eeab7f41463f6ac18091ca67078:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:42:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:42:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.783429833, 2024-04-22T18:42:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60604869800000005, 2024-04-22T18:42:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:42:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:42:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775575+0000", 2024-04-22T18:42:04.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775578+0000", 2024-04-22T18:42:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3440000000000001e-06 2024-04-22T18:42:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775584+0000", 2024-04-22T18:42:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6969999999999998e-06 2024-04-22T18:42:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775589+0000", 2024-04-22T18:42:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0540000000000002e-06 2024-04-22T18:42:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775729+0000", 2024-04-22T18:42:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014061299999999999 2024-04-22T18:42:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776045+0000", 2024-04-22T18:42:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000315168 2024-04-22T18:42:04.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080584+0000", 2024-04-22T18:42:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30453977700000001 2024-04-22T18:42:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080635+0000", 2024-04-22T18:42:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1007999999999997e-05 2024-04-22T18:42:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:42:04.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.080741+0000", 2024-04-22T18:42:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105646 2024-04-22T18:42:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.180821+0000", 2024-04-22T18:42:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.10007980900000001 2024-04-22T18:42:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381437+0000", 2024-04-22T18:42:04.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20061589399999999 2024-04-22T18:42:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381491+0000", 2024-04-22T18:42:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4531999999999999e-05 2024-04-22T18:42:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.381624+0000", 2024-04-22T18:42:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000132745 2024-04-22T18:42:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.667 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(osd.4.13:345 3.1 3:bd847077:::0e274616605e19dd653fcf05d310933ab02835574414ca18340b3fc325d2da4f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:42:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:42:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.783405473000002, 2024-04-22T18:42:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.90876140699999997, 2024-04-22T18:42:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:42:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775600+0000", 2024-04-22T18:42:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775599+0000", 2024-04-22T18:42:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775602+0000", 2024-04-22T18:42:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.976e-06 2024-04-22T18:42:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775607+0000", 2024-04-22T18:42:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2229999999999998e-06 2024-04-22T18:42:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.775612+0000", 2024-04-22T18:42:04.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6539999999999998e-06 2024-04-22T18:42:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776072+0000", 2024-04-22T18:42:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046006299999999998 2024-04-22T18:42:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.776256+0000", 2024-04-22T18:42:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000183735 2024-04-22T18:42:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081157+0000", 2024-04-22T18:42:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.30490169099999997 2024-04-22T18:42:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081203+0000", 2024-04-22T18:42:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5677999999999999e-05 2024-04-22T18:42:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:42:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.081301+0000", 2024-04-22T18:42:04.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7893999999999995e-05 2024-04-22T18:42:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.180827+0000", 2024-04-22T18:42:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.099525725999999995 2024-04-22T18:42:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684197+0000", 2024-04-22T18:42:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.50336990699999995 2024-04-22T18:42:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684247+0000", 2024-04-22T18:42:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0195000000000003e-05 2024-04-22T18:42:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.684361+0000", 2024-04-22T18:42:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114189 2024-04-22T18:42:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:4267 3.2 3:584fd9a2:::cda848272f4ef42ebafc2089aaddeacaf12eabd82aef5e42e32ae93a8a5da0b6:head [write 0~2635776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-22T18:42:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:42:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.703055616999997, 2024-04-22T18:42:04.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.72658671900000005, 2024-04-22T18:42:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 4267 2024-04-22T18:42:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:42:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855949+0000", 2024-04-22T18:42:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.855946+0000", 2024-04-22T18:42:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-04-22T18:42:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.856984+0000", 2024-04-22T18:42:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001037764 2024-04-22T18:42:04.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.857003+0000", 2024-04-22T18:42:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8876e-05 2024-04-22T18:42:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.857016+0000", 2024-04-22T18:42:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3337e-05 2024-04-22T18:42:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883170+0000", 2024-04-22T18:42:04.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026154047 2024-04-22T18:42:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-04-22T18:42:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883262+0000", 2024-04-22T18:42:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1996999999999993e-05 2024-04-22T18:42:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-04-22T18:42:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.883360+0000", 2024-04-22T18:42:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7429000000000005e-05 2024-04-22T18:42:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-04-22T18:42:04.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:07.887277+0000", 2024-04-22T18:42:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039169449999999998 2024-04-22T18:42:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582423+0000", 2024-04-22T18:42:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.69514625200000002 2024-04-22T18:42:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-04-22T18:42:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582458+0000", 2024-04-22T18:42:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5293999999999998e-05 2024-04-22T18:42:04.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:08.582536+0000", 2024-04-22T18:42:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7855000000000006e-05 2024-04-22T18:42:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4502.0:5189 4.19 4.f8af6579 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e173)", 2024-04-22T18:42:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-04-22T18:41:50.139380+0000", 2024-04-22T18:42:04.680 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.419625506999999, 2024-04-22T18:42:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60416509100000004, 2024-04-22T18:42:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-04-22T18:42:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-04-22T18:42:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-04-22T18:42:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4502", 2024-04-22T18:42:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5189 2024-04-22T18:42:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-04-22T18:42:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-04-22T18:42:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139380+0000", 2024-04-22T18:42:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-04-22T18:42:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139380+0000", 2024-04-22T18:42:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-04-22T18:42:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-04-22T18:42:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139376+0000", 2024-04-22T18:42:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999967 2024-04-22T18:42:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-04-22T18:42:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139390+0000", 2024-04-22T18:42:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3441999999999999e-05 2024-04-22T18:42:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-04-22T18:42:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139403+0000", 2024-04-22T18:42:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3845999999999999e-05 2024-04-22T18:42:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-04-22T18:42:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139414+0000", 2024-04-22T18:42:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0351e-05 2024-04-22T18:42:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139471+0000", 2024-04-22T18:42:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7444000000000001e-05 2024-04-22T18:42:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-04-22T18:42:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.139479+0000", 2024-04-22T18:42:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8699999999999992e-06 2024-04-22T18:42:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-04-22T18:42:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743532+0000", 2024-04-22T18:42:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.60405305200000003 2024-04-22T18:42:04.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-04-22T18:42:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-04-22T18:42:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-04-22T18:42:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-04-22T18:41:50.743545+0000", 2024-04-22T18:42:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2493e-05 2024-04-22T18:42:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-04-22T18:42:04.685 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-04-22T18:42:04.685 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:04.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-22T18:42:04.792 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:42:04.792+0000 7f8607f36640 -1 osd.3 187 handle_osd_map injecting map crc failure 2024-04-22T18:42:04.793 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:42:04.793+0000 7f3e7be88640 -1 osd.0 187 handle_osd_map injecting map crc failure 2024-04-22T18:42:04.795 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:42:04.799 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:42:04.800 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:42:04.800 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:42:04.800 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:42:04.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-22T18:42:04.912 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:42:04.912 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:42:04.912 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:42:04.912 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:42:04.912 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:42:04.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-22T18:42:05.024 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:42:05.024 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:42:05.024 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:42:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:42:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:89 4.1 4:81200c09:::smithi08937929-23:head [write 769754~688184,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:42:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:42:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 190.36226959000001, 2024-04-22T18:42:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55518307600000005, 2024-04-22T18:42:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 89 2024-04-22T18:42:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.025 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:42:05.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.026 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.026 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.026 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:42:05.026 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.662378+0000", 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663122+0000", 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000744759 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.663136+0000", 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3611000000000001e-05 2024-04-22T18:42:05.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763206+0000", 2024-04-22T18:42:05.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10007049799999999 2024-04-22T18:42:05.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763272+0000", 2024-04-22T18:42:05.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5407999999999994e-05 2024-04-22T18:42:05.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.028 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.028 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.028 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763369+0000", 2024-04-22T18:42:05.028 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6906999999999999e-05 2024-04-22T18:42:05.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:42:05.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763479+0000", 2024-04-22T18:42:05.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000110618 2024-04-22T18:42:05.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.870555+0000", 2024-04-22T18:42:05.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.107076002 2024-04-22T18:42:05.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.029 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.029 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.029 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217496+0000", 2024-04-22T18:42:05.029 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.346940847 2024-04-22T18:42:05.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217531+0000", 2024-04-22T18:42:05.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4654999999999999e-05 2024-04-22T18:42:05.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.030 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.030 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.217561+0000", 2024-04-22T18:42:05.030 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0371000000000001e-05 2024-04-22T18:42:05.030 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.030 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.030 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.030 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.031 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:92 4.1 4:81200c09:::smithi08937929-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:42:05.031 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:42:05.031 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 190.261261413, 2024-04-22T18:42:05.031 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55451593600000004, 2024-04-22T18:42:05.031 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.031 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.031 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.031 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.031 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.031 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 92 2024-04-22T18:42:05.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.031 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.031 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.031 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.031 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:42:05.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.032 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.032 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:42:05.032 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763386+0000", 2024-04-22T18:42:05.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763388+0000", 2024-04-22T18:42:05.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.505e-06 2024-04-22T18:42:05.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.033 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.033 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763391+0000", 2024-04-22T18:42:05.033 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3909999999999998e-06 2024-04-22T18:42:05.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763395+0000", 2024-04-22T18:42:05.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6650000000000001e-06 2024-04-22T18:42:05.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768099+0000", 2024-04-22T18:42:05.034 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.004704209 2024-04-22T18:42:05.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.034 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:05.034 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.768176+0000", 2024-04-22T18:42:05.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6715e-05 2024-04-22T18:42:05.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317781+0000", 2024-04-22T18:42:05.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.54960516500000001 2024-04-22T18:42:05.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.035 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317834+0000", 2024-04-22T18:42:05.035 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3109000000000003e-05 2024-04-22T18:42:05.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.035 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.036 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317902+0000", 2024-04-22T18:42:05.036 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8473000000000007e-05 2024-04-22T18:42:05.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.036 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.037 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:93 4.1 4:821263ac:::smithi08937929-24:head [write 786561~479786,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:42:05.037 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:42:05.037 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 190.261251028, 2024-04-22T18:42:05.037 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55458529899999998, 2024-04-22T18:42:05.037 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.037 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.037 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.038 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.038 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.038 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 93 2024-04-22T18:42:05.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.038 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:42:05.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.038 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.038 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763397+0000", 2024-04-22T18:42:05.038 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.763396+0000", 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764304+0000", 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090711800000000005 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764312+0000", 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7630000000000006e-06 2024-04-22T18:42:05.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.764321+0000", 2024-04-22T18:42:05.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3790000000000006e-06 2024-04-22T18:42:05.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868276+0000", 2024-04-22T18:42:05.040 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10395552399999999 2024-04-22T18:42:05.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.040 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.040 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868386+0000", 2024-04-22T18:42:05.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010959399999999999 2024-04-22T18:42:05.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 5", 2024-04-22T18:42:05.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.868502+0000", 2024-04-22T18:42:05.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000116653 2024-04-22T18:42:05.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.871064+0000", 2024-04-22T18:42:05.041 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0025612590000000002 2024-04-22T18:42:05.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.041 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.041 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317932+0000", 2024-04-22T18:42:05.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44686778799999999 2024-04-22T18:42:05.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317959+0000", 2024-04-22T18:42:05.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.709e-05 2024-04-22T18:42:05.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.042 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.042 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.317982+0000", 2024-04-22T18:42:05.042 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3417e-05 2024-04-22T18:42:05.042 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.042 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.043 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.043 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:388 4.9 4:905c3da2:::smithi08937929-39:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:42:05.043 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:42:05.043 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 185.923500601, 2024-04-22T18:42:05.043 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80352940900000003, 2024-04-22T18:42:05.043 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.043 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.043 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.043 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.043 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.043 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 388 2024-04-22T18:42:05.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.044 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:42:05.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101147+0000", 2024-04-22T18:42:05.044 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.044 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.044 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101146+0000", 2024-04-22T18:42:05.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:42:05.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101151+0000", 2024-04-22T18:42:05.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.763e-06 2024-04-22T18:42:05.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101161+0000", 2024-04-22T18:42:05.045 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4059999999999994e-06 2024-04-22T18:42:05.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.045 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.045 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101167+0000", 2024-04-22T18:42:05.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739999999999998e-06 2024-04-22T18:42:05.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101283+0000", 2024-04-22T18:42:05.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011666 2024-04-22T18:42:05.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101357+0000", 2024-04-22T18:42:05.046 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3266e-05 2024-04-22T18:42:05.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.046 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:05.046 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.603130+0000", 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50177313599999995 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.703227+0000", 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100096874 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904602+0000", 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.201375893 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904631+0000", 2024-04-22T18:42:05.047 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.866e-05 2024-04-22T18:42:05.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.048 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.048 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.904677+0000", 2024-04-22T18:42:05.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5460000000000002e-05 2024-04-22T18:42:05.048 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.048 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.048 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.048 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1535 4.9 4:9ea6f51f:::smithi08937929-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e45)", 2024-04-22T18:42:05.048 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:42:05.048 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 158.88885308600001, 2024-04-22T18:42:05.048 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.57952267199999996, 2024-04-22T18:42:05.049 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.049 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.049 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.049 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.049 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.049 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1535 2024-04-22T18:42:05.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.049 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.049 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.049 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:42:05.049 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135795+0000", 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135794+0000", 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135796+0000", 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0930000000000001e-06 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.050 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135800+0000", 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9509999999999999e-06 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.135805+0000", 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.391e-06 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416706+0000", 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28090107800000003 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.051 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.416760+0000", 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4082999999999997e-05 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:26.715317+0000", 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29855743899999998 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4177 4.19 4.f75b2379 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.859799789, 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76236081700000002, 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.052 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:42:05.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.053 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.053 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4177 2024-04-22T18:42:05.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.053 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:42:05.053 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.053 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.053 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164848+0000", 2024-04-22T18:42:05.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164847+0000", 2024-04-22T18:42:05.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:05.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164854+0000", 2024-04-22T18:42:05.054 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2620000000000001e-06 2024-04-22T18:42:05.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.054 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.054 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164864+0000", 2024-04-22T18:42:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0400999999999999e-05 2024-04-22T18:42:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164876+0000", 2024-04-22T18:42:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1204e-05 2024-04-22T18:42:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164975+0000", 2024-04-22T18:42:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9562999999999995e-05 2024-04-22T18:42:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164980+0000", 2024-04-22T18:42:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7310000000000001e-06 2024-04-22T18:42:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927200+0000", 2024-04-22T18:42:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.762219973 2024-04-22T18:42:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927209+0000", 2024-04-22T18:42:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.9050000000000003e-06 2024-04-22T18:42:05.056 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4182 4.c 4.5345b7ac (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:42:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:42:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.859768469999999, 2024-04-22T18:42:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76185930999999996, 2024-04-22T18:42:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:42:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.057 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4182 2024-04-22T18:42:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:42:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:42:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.058 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164879+0000", 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164881+0000", 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.305e-06 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164885+0000", 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.873e-06 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164890+0000", 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6299999999999997e-06 2024-04-22T18:42:05.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225475+0000", 2024-04-22T18:42:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060585181000000002 2024-04-22T18:42:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225483+0000", 2024-04-22T18:42:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6790000000000005e-06 2024-04-22T18:42:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926728+0000", 2024-04-22T18:42:05.060 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124542099999998 2024-04-22T18:42:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926739+0000", 2024-04-22T18:42:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0669000000000001e-05 2024-04-22T18:42:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4183 4.c 4.5345b7ac (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:42:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:42:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.859755214000003, 2024-04-22T18:42:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76187489500000005, 2024-04-22T18:42:05.061 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4183 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164893+0000", 2024-04-22T18:42:05.062 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164892+0000", 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164894+0000", 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.705e-06 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164897+0000", 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2739999999999999e-06 2024-04-22T18:42:05.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164901+0000", 2024-04-22T18:42:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.636e-06 2024-04-22T18:42:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225508+0000", 2024-04-22T18:42:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060607564000000003 2024-04-22T18:42:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225515+0000", 2024-04-22T18:42:05.064 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1829999999999999e-06 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926759+0000", 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.701244859 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926767+0000", 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0180000000000001e-06 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4190 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:42:05.065 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:42:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.859738292000003, 2024-04-22T18:42:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.761884546, 2024-04-22T18:42:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:42:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4190 2024-04-22T18:42:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:42:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.066 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164909+0000", 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164912+0000", 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6249999999999999e-06 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.067 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164918+0000", 2024-04-22T18:42:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6039999999999998e-06 2024-04-22T18:42:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164922+0000", 2024-04-22T18:42:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.9319999999999998e-06 2024-04-22T18:42:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225536+0000", 2024-04-22T18:42:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060613725 2024-04-22T18:42:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.068 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225540+0000", 2024-04-22T18:42:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5030000000000002e-06 2024-04-22T18:42:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926786+0000", 2024-04-22T18:42:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124673999999998 2024-04-22T18:42:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926794+0000", 2024-04-22T18:42:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7689999999999996e-06 2024-04-22T18:42:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.069 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4191 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:42:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:42:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.859722871000002, 2024-04-22T18:42:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76190695500000005, 2024-04-22T18:42:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:42:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4191 2024-04-22T18:42:05.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164925+0000", 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.071 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164927+0000", 2024-04-22T18:42:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0959999999999999e-06 2024-04-22T18:42:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164937+0000", 2024-04-22T18:42:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0730999999999999e-05 2024-04-22T18:42:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164944+0000", 2024-04-22T18:42:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5130000000000002e-06 2024-04-22T18:42:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225565+0000", 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060621421000000002 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225569+0000", 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.264e-06 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926813+0000", 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70124469 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.073 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926832+0000", 2024-04-22T18:42:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8580000000000002e-05 2024-04-22T18:42:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4192 4.c 4.38c2e44c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:42:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:42:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.859698262999999, 2024-04-22T18:42:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76191075399999997, 2024-04-22T18:42:05.074 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4192 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:42:05.075 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164949+0000", 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164953+0000", 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1099999999999996e-06 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164965+0000", 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2298e-05 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164972+0000", 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5230000000000001e-06 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225581+0000", 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.060609356000000003 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.225584+0000", 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.106e-06 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.077 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926852+0000", 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70126794999999997 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.926860+0000", 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0919999999999998e-06 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4197 4.13 4:ca46fde9:::smithi08937929-14:head [sparse-read 0~1375265,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.859671652999999, 2024-04-22T18:42:05.078 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.28429536, 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4197 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.079 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164976+0000", 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164975+0000", 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164979+0000", 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8959999999999996e-06 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164983+0000", 2024-04-22T18:42:05.080 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.053e-06 2024-04-22T18:42:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164987+0000", 2024-04-22T18:42:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3670000000000001e-06 2024-04-22T18:42:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.264783+0000", 2024-04-22T18:42:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099796088000000005 2024-04-22T18:42:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.264789+0000", 2024-04-22T18:42:05.081 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4989999999999999e-06 2024-04-22T18:42:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139375+0000", 2024-04-22T18:42:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.87458606000000005 2024-04-22T18:42:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:42:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139478+0000", 2024-04-22T18:42:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010219 2024-04-22T18:42:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.082 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.446961+0000", 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30748385900000003 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.447101+0000", 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001393 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.449271+0000", 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021707950000000001 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4198 4.13 4:ca46fde9:::smithi08937929-14:head [sparse-read 0~1375265] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.859658285000002, 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2868951370000001, 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4198 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:42:05.084 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164989+0000", 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164991+0000", 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:42:05.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164994+0000", 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2459999999999999e-06 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.164997+0000", 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1200000000000002e-06 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.364948+0000", 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.19995059200000001 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.364969+0000", 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1495999999999999e-05 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139617+0000", 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77464723300000005 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139643+0000", 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6228e-05 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.087 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451393+0000", 2024-04-22T18:42:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31175035200000001 2024-04-22T18:42:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451515+0000", 2024-04-22T18:42:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001214 2024-04-22T18:42:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451885+0000", 2024-04-22T18:42:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00036992999999999998 2024-04-22T18:42:05.088 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4199 4.13 4:ca46fde9:::smithi08937929-14:head [read 0~2739541,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:42:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:42:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.859647262000003, 2024-04-22T18:42:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.291247504, 2024-04-22T18:42:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4199 2024-04-22T18:42:05.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:42:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:42:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.090 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165000+0000", 2024-04-22T18:42:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165002+0000", 2024-04-22T18:42:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2649999999999999e-06 2024-04-22T18:42:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165006+0000", 2024-04-22T18:42:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1599999999999998e-06 2024-04-22T18:42:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.091 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165009+0000", 2024-04-22T18:42:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4599999999999999e-06 2024-04-22T18:42:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365066+0000", 2024-04-22T18:42:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20005682999999999 2024-04-22T18:42:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365078+0000", 2024-04-22T18:42:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2235000000000001e-05 2024-04-22T18:42:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.092 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139671+0000", 2024-04-22T18:42:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77459301700000005 2024-04-22T18:42:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:42:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139702+0000", 2024-04-22T18:42:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0673999999999998e-05 2024-04-22T18:42:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.451930+0000", 2024-04-22T18:42:05.093 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31222817600000002 2024-04-22T18:42:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.452092+0000", 2024-04-22T18:42:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0001616 2024-04-22T18:42:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.456248+0000", 2024-04-22T18:42:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0041563700000000004 2024-04-22T18:42:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4200 4.13 4:c93c842e:::smithi08937929-25:head [tier-promote] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:42:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:42:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.859636381999998, 2024-04-22T18:42:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.177728814, 2024-04-22T18:42:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4200 2024-04-22T18:42:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.095 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:42:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:42:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165011+0000", 2024-04-22T18:42:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165013+0000", 2024-04-22T18:42:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.635e-06 2024-04-22T18:42:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165018+0000", 2024-04-22T18:42:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2000000000000002e-06 2024-04-22T18:42:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165024+0000", 2024-04-22T18:42:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5910000000000003e-06 2024-04-22T18:42:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365110+0000", 2024-04-22T18:42:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20008601400000001 2024-04-22T18:42:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.365114+0000", 2024-04-22T18:42:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.651e-06 2024-04-22T18:42:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.098 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139723+0000", 2024-04-22T18:42:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.77460836899999996 2024-04-22T18:42:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:42:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.139787+0000", 2024-04-22T18:42:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4707000000000002e-05 2024-04-22T18:42:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.459491+0000", 2024-04-22T18:42:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31970372000000002 2024-04-22T18:42:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.459567+0000", 2024-04-22T18:42:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6199999999999995e-05 2024-04-22T18:42:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.562522+0000", 2024-04-22T18:42:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102954925 2024-04-22T18:42:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:04.562585+0000", 2024-04-22T18:42:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3287000000000006e-05 2024-04-22T18:42:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:05.342740+0000", 2024-04-22T18:42:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.78015481499999995 2024-04-22T18:42:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4205 4.1 4.fdaa93c1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e129)", 2024-04-22T18:42:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:42:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 61.859620231999997, 2024-04-22T18:42:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76220996100000005, 2024-04-22T18:42:05.101 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4205 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165028+0000", 2024-04-22T18:42:05.102 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165027+0000", 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165029+0000", 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3549999999999999e-06 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165035+0000", 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8549999999999998e-06 2024-04-22T18:42:05.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165040+0000", 2024-04-22T18:42:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9200000000000003e-06 2024-04-22T18:42:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165116+0000", 2024-04-22T18:42:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5834999999999995e-05 2024-04-22T18:42:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:05.104 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.165123+0000", 2024-04-22T18:42:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.2139999999999999e-06 2024-04-22T18:42:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927230+0000", 2024-04-22T18:42:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.76210651100000004 2024-04-22T18:42:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:03.927237+0000", 2024-04-22T18:42:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7409999999999992e-06 2024-04-22T18:42:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.105 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4268 4.5 4:a4a14dff:::smithi08937929-40:head [assert-version v65,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e133)", 2024-04-22T18:42:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:42:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.441912819999999, 2024-04-22T18:42:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70931800300000003, 2024-04-22T18:42:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4268 2024-04-22T18:42:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.106 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582735+0000", 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582733+0000", 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.107 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582741+0000", 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3329999999999999e-06 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582751+0000", 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0628999999999999e-05 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582759+0000", 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4120000000000004e-06 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582795+0000", 2024-04-22T18:42:05.108 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6157e-05 2024-04-22T18:42:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.582862+0000", 2024-04-22T18:42:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7261999999999994e-05 2024-04-22T18:42:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:42:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.987268+0000", 2024-04-22T18:42:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40440629700000003 2024-04-22T18:42:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.987771+0000", 2024-04-22T18:42:05.109 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050267400000000005 2024-04-22T18:42:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.291974+0000", 2024-04-22T18:42:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30420264499999999 2024-04-22T18:42:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.292008+0000", 2024-04-22T18:42:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4011999999999999e-05 2024-04-22T18:42:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.110 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.292053+0000", 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5305000000000001e-05 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4272 4.5 4:a6d237d2:::smithi08937929-43:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e133)", 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 56.238584619999997, 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60731304699999999, 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4272 2024-04-22T18:42:05.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786063+0000", 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786068+0000", 2024-04-22T18:42:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9910000000000002e-06 2024-04-22T18:42:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786079+0000", 2024-04-22T18:42:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1287000000000001e-05 2024-04-22T18:42:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786083+0000", 2024-04-22T18:42:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0949999999999998e-06 2024-04-22T18:42:05.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786121+0000", 2024-04-22T18:42:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8096999999999998e-05 2024-04-22T18:42:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:08.786219+0000", 2024-04-22T18:42:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7429000000000005e-05 2024-04-22T18:42:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.114 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:09.393376+0000", 2024-04-22T18:42:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.60715760600000002 2024-04-22T18:42:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4437 4.19 4:9ea6f51f:::smithi08937929-34:head [read 304128~651776] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:42:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:42:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 47.108977846000002, 2024-04-22T18:42:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.042388732, 2024-04-22T18:42:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.115 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4437 2024-04-22T18:42:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:42:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915670+0000", 2024-04-22T18:42:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915669+0000", 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915673+0000", 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6770000000000001e-06 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915679+0000", 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.976e-06 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.117 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915684+0000", 2024-04-22T18:42:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3430000000000003e-06 2024-04-22T18:42:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915725+0000", 2024-04-22T18:42:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0751999999999999e-05 2024-04-22T18:42:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.915732+0000", 2024-04-22T18:42:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6279999999999999e-06 2024-04-22T18:42:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.118 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.957228+0000", 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0414961220000001 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.957325+0000", 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7422000000000003e-05 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.958059+0000", 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073350499999999996 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4690 3.3 3:ee62d872:::cb076f2b64cfcdfafc1ec2cf19f2eaa46006a5a6952a3aeb9c7211f97fca91ae:head [write 0~441856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:26.592341+0000", 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 38.432306584000003, 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.56667778400000002, 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4690 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592341+0000", 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592341+0000", 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592340+0000", 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592697+0000", 2024-04-22T18:42:05.121 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000357334 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592709+0000", 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2211999999999999e-05 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.592718+0000", 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0319999999999992e-06 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.692859+0000", 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10014077 2024-04-22T18:42:05.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.692870+0000", 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1225e-05 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.995290+0000", 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242006199999999 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.995398+0000", 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000108136 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.995513+0000", 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000115047 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:26.996655+0000", 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001141162 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:27.158858+0000", 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.16220310299999999 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.124 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:27.158901+0000", 2024-04-22T18:42:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3667000000000001e-05 2024-04-22T18:42:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:27.159019+0000", 2024-04-22T18:42:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117619 2024-04-22T18:42:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.125 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.125 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:42:05.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-22T18:42:05.228 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:42:05.229 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:42:05.229 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:42:05.229 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:42:05.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-22T18:42:05.339 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:42:05.339 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:42:05.339 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:42:05.339 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:42:05.339 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:42:05.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-22T18:42:05.450 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:42:05.450 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:42:05.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:42:05.450 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:42:05.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.450 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1 4.8 4:12d9f826:::smithi08937929-1:head [write 416807~475249 in=475249b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-22T18:42:05.450 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:42:05.450 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 191.35640282099999, 2024-04-22T18:42:05.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30440807600000003, 2024-04-22T18:42:05.451 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.451 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.451 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1 2024-04-22T18:42:05.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.452 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:42:05.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093860+0000", 2024-04-22T18:42:05.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.093858+0000", 2024-04-22T18:42:05.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:05.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094427+0000", 2024-04-22T18:42:05.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00056913299999999999 2024-04-22T18:42:05.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.094436+0000", 2024-04-22T18:42:05.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2119999999999992e-06 2024-04-22T18:42:05.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.194532+0000", 2024-04-22T18:42:05.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10009557099999999 2024-04-22T18:42:05.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295132+0000", 2024-04-22T18:42:05.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10059979600000001 2024-04-22T18:42:05.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295224+0000", 2024-04-22T18:42:05.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.221e-05 2024-04-22T18:42:05.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:42:05.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.295332+0000", 2024-04-22T18:42:05.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010833200000000001 2024-04-22T18:42:05.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.297104+0000", 2024-04-22T18:42:05.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.001771838 2024-04-22T18:42:05.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398202+0000", 2024-04-22T18:42:05.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10109789700000001 2024-04-22T18:42:05.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398239+0000", 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7058000000000003e-05 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.398268+0000", 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8781000000000001e-05 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:19 3.2 3:7730addc:::smithi08937929-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e29)", 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:42:05.456 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 187.45690494600001, 2024-04-22T18:42:05.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43597650500000001, 2024-04-22T18:42:05.457 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.457 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.457 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:42:05.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993358+0000", 2024-04-22T18:42:05.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993355+0000", 2024-04-22T18:42:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:42:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993375+0000", 2024-04-22T18:42:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9519999999999999e-05 2024-04-22T18:42:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993387+0000", 2024-04-22T18:42:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2478e-05 2024-04-22T18:42:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993394+0000", 2024-04-22T18:42:05.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.793e-06 2024-04-22T18:42:05.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993446+0000", 2024-04-22T18:42:05.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1685999999999999e-05 2024-04-22T18:42:05.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:05.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:57.993458+0000", 2024-04-22T18:42:05.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2497000000000001e-05 2024-04-22T18:42:05.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227829+0000", 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.23437034100000001 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.227905+0000", 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6299999999999998e-05 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.228023+0000", 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000117713 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.229248+0000", 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00122568 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429185+0000", 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.199937109 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429244+0000", 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8264000000000003e-05 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.461 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:58.429334+0000", 2024-04-22T18:42:05.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0297000000000006e-05 2024-04-22T18:42:05.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.462 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.462 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:400 4.8 4:12d9f826:::smithi08937929-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2024-04-22T18:42:05.462 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:42:05.462 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 186.348605155, 2024-04-22T18:42:05.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30078007699999998, 2024-04-22T18:42:05.462 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.462 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.462 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.463 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 400 2024-04-22T18:42:05.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.463 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:42:05.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101657+0000", 2024-04-22T18:42:05.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101656+0000", 2024-04-22T18:42:05.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:05.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101665+0000", 2024-04-22T18:42:05.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7490000000000004e-06 2024-04-22T18:42:05.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101674+0000", 2024-04-22T18:42:05.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1130000000000007e-06 2024-04-22T18:42:05.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.101681+0000", 2024-04-22T18:42:05.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0469999999999996e-06 2024-04-22T18:42:05.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201784+0000", 2024-04-22T18:42:05.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10010329799999999 2024-04-22T18:42:05.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.201848+0000", 2024-04-22T18:42:05.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3440000000000002e-05 2024-04-22T18:42:05.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:42:05.466 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.303862+0000", 2024-04-22T18:42:05.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102014191 2024-04-22T18:42:05.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.304315+0000", 2024-04-22T18:42:05.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045359600000000001 2024-04-22T18:42:05.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402396+0000", 2024-04-22T18:42:05.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.098080991000000006 2024-04-22T18:42:05.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402419+0000", 2024-04-22T18:42:05.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198999999999999e-05 2024-04-22T18:42:05.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:59.402437+0000", 2024-04-22T18:42:05.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8780999999999999e-05 2024-04-22T18:42:05.469 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.469 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.469 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.469 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:53 3.2 3:77b9e1eb:::smithi08937929-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:42:05.470 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:42:05.470 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 173.01706152700001, 2024-04-22T18:42:05.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30242701500000002, 2024-04-22T18:42:05.470 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.470 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.470 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:42:05.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433201+0000", 2024-04-22T18:42:05.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433200+0000", 2024-04-22T18:42:05.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433203+0000", 2024-04-22T18:42:05.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6510000000000001e-06 2024-04-22T18:42:05.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433210+0000", 2024-04-22T18:42:05.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.917e-06 2024-04-22T18:42:05.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.433213+0000", 2024-04-22T18:42:05.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2440000000000002e-06 2024-04-22T18:42:05.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534458+0000", 2024-04-22T18:42:05.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101245165 2024-04-22T18:42:05.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534515+0000", 2024-04-22T18:42:05.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6079000000000003e-05 2024-04-22T18:42:05.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:42:05.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534606+0000", 2024-04-22T18:42:05.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1773000000000004e-05 2024-04-22T18:42:05.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.633838+0000", 2024-04-22T18:42:05.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.099231458999999994 2024-04-22T18:42:05.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735549+0000", 2024-04-22T18:42:05.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10171074400000001 2024-04-22T18:42:05.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735590+0000", 2024-04-22T18:42:05.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1854999999999998e-05 2024-04-22T18:42:05.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.735628+0000", 2024-04-22T18:42:05.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7614000000000003e-05 2024-04-22T18:42:05.475 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.475 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.475 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.475 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1083 4.8 4:187adac1:::smithi08937929-36:head [write 544560~783450,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:42:05.475 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:42:05.475 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 172.92597571900001, 2024-04-22T18:42:05.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30392126200000003, 2024-04-22T18:42:05.475 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.476 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.476 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.476 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.476 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.476 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1083 2024-04-22T18:42:05.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.476 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.476 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:42:05.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.476 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.476 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524287+0000", 2024-04-22T18:42:05.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.524283+0000", 2024-04-22T18:42:05.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:42:05.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525215+0000", 2024-04-22T18:42:05.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093154900000000005 2024-04-22T18:42:05.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525234+0000", 2024-04-22T18:42:05.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9351000000000002e-05 2024-04-22T18:42:05.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525246+0000", 2024-04-22T18:42:05.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2075999999999999e-05 2024-04-22T18:42:05.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525291+0000", 2024-04-22T18:42:05.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4397000000000002e-05 2024-04-22T18:42:05.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525360+0000", 2024-04-22T18:42:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9614000000000001e-05 2024-04-22T18:42:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:42:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.525471+0000", 2024-04-22T18:42:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000111187 2024-04-22T18:42:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.527288+0000", 2024-04-22T18:42:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00181671 2024-04-22T18:42:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828112+0000", 2024-04-22T18:42:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30082353000000001 2024-04-22T18:42:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828165+0000", 2024-04-22T18:42:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3171999999999998e-05 2024-04-22T18:42:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.828208+0000", 2024-04-22T18:42:05.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3228e-05 2024-04-22T18:42:05.481 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.481 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.481 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.481 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.481 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1084 4.8 4:187adac1:::smithi08937929-36:head [write 1915030~481348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:42:05.481 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:42:05.481 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 172.92017545900001, 2024-04-22T18:42:05.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39833184900000002, 2024-04-22T18:42:05.481 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.481 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.481 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.481 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1084 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530087+0000", 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.482 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530086+0000", 2024-04-22T18:42:05.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:05.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530490+0000", 2024-04-22T18:42:05.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00040457299999999998 2024-04-22T18:42:05.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.483 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.483 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530500+0000", 2024-04-22T18:42:05.483 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9979999999999998e-06 2024-04-22T18:42:05.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.483 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530506+0000", 2024-04-22T18:42:05.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8649999999999998e-06 2024-04-22T18:42:05.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530549+0000", 2024-04-22T18:42:05.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2746e-05 2024-04-22T18:42:05.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.484 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.484 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530632+0000", 2024-04-22T18:42:05.484 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3343000000000005e-05 2024-04-22T18:42:05.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.484 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.530766+0000", 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013331899999999999 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535005+0000", 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0042390090000000002 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928327+0000", 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39332255599999999 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.485 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928377+0000", 2024-04-22T18:42:05.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0356999999999999e-05 2024-04-22T18:42:05.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.486 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.486 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928419+0000", 2024-04-22T18:42:05.486 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1440999999999998e-05 2024-04-22T18:42:05.486 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.486 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.486 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.486 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.486 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1085 4.8 4:187adac1:::smithi08937929-36:head [write 2797604~479761] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:42:05.486 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:42:05.487 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 172.915435116, 2024-04-22T18:42:05.487 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39369363499999999, 2024-04-22T18:42:05.487 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.487 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.487 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.487 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.487 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.487 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1085 2024-04-22T18:42:05.487 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.487 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.487 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.487 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:42:05.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534827+0000", 2024-04-22T18:42:05.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.488 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.488 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.534826+0000", 2024-04-22T18:42:05.488 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:05.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.488 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535279+0000", 2024-04-22T18:42:05.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00045375100000000001 2024-04-22T18:42:05.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535289+0000", 2024-04-22T18:42:05.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6579999999999997e-06 2024-04-22T18:42:05.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.489 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.489 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.489 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535295+0000", 2024-04-22T18:42:05.489 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5280000000000003e-06 2024-04-22T18:42:05.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535348+0000", 2024-04-22T18:42:05.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2972e-05 2024-04-22T18:42:05.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535390+0000", 2024-04-22T18:42:05.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1987999999999998e-05 2024-04-22T18:42:05.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.490 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.490 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:42:05.490 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535457+0000", 2024-04-22T18:42:05.490 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.7731999999999995e-05 2024-04-22T18:42:05.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537116+0000", 2024-04-22T18:42:05.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016589280000000001 2024-04-22T18:42:05.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928470+0000", 2024-04-22T18:42:05.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.391353644 2024-04-22T18:42:05.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.491 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.491 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.491 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928499+0000", 2024-04-22T18:42:05.491 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8867e-05 2024-04-22T18:42:05.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.492 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.492 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928521+0000", 2024-04-22T18:42:05.492 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2198e-05 2024-04-22T18:42:05.492 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.492 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.492 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.492 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.492 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1086 4.8 4:187adac1:::smithi08937929-36:head [setxattr _header (56) in=7b,truncate 3344560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:42:05.492 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:42:05.492 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 172.914962218, 2024-04-22T18:42:05.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39334296699999999, 2024-04-22T18:42:05.493 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.493 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.493 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.493 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.493 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.493 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1086 2024-04-22T18:42:05.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.493 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.493 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.493 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:42:05.493 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.493 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535300+0000", 2024-04-22T18:42:05.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535299+0000", 2024-04-22T18:42:05.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:42:05.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.494 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.494 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.494 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535306+0000", 2024-04-22T18:42:05.494 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.251e-06 2024-04-22T18:42:05.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535311+0000", 2024-04-22T18:42:05.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9790000000000001e-06 2024-04-22T18:42:05.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535315+0000", 2024-04-22T18:42:05.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8799999999999999e-06 2024-04-22T18:42:05.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.495 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.495 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.495 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536402+0000", 2024-04-22T18:42:05.495 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010864329999999999 2024-04-22T18:42:05.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536453+0000", 2024-04-22T18:42:05.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1533000000000003e-05 2024-04-22T18:42:05.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:42:05.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.536528+0000", 2024-04-22T18:42:05.496 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4496e-05 2024-04-22T18:42:05.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.496 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.496 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.496 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.537120+0000", 2024-04-22T18:42:05.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00059160799999999998 2024-04-22T18:42:05.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928557+0000", 2024-04-22T18:42:05.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39143739700000002 2024-04-22T18:42:05.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.497 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.497 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928597+0000", 2024-04-22T18:42:05.497 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0414000000000003e-05 2024-04-22T18:42:05.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.497 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.498 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.498 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928643+0000", 2024-04-22T18:42:05.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5911000000000001e-05 2024-04-22T18:42:05.498 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.498 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.498 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.498 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.498 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.498 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:1087 4.8 4:187adac1:::smithi08937929-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-22T18:42:05.498 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:42:05.498 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 172.91494229700001, 2024-04-22T18:42:05.498 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39344613299999998, 2024-04-22T18:42:05.498 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.498 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.499 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.499 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.499 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.499 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 1087 2024-04-22T18:42:05.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.499 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:42:05.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.499 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.499 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.499 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:42:05.499 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535320+0000", 2024-04-22T18:42:05.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535322+0000", 2024-04-22T18:42:05.500 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6819999999999999e-06 2024-04-22T18:42:05.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.500 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.500 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.500 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535327+0000", 2024-04-22T18:42:05.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.181e-06 2024-04-22T18:42:05.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.535333+0000", 2024-04-22T18:42:05.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7189999999999998e-06 2024-04-22T18:42:05.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635396+0000", 2024-04-22T18:42:05.501 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100062333 2024-04-22T18:42:05.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.501 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.501 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:05.501 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.635453+0000", 2024-04-22T18:42:05.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6983e-05 2024-04-22T18:42:05.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928661+0000", 2024-04-22T18:42:05.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29320895000000002 2024-04-22T18:42:05.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.502 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928711+0000", 2024-04-22T18:42:05.502 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9849000000000003e-05 2024-04-22T18:42:05.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.502 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.502 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.503 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:12.928766+0000", 2024-04-22T18:42:05.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5028000000000002e-05 2024-04-22T18:42:05.503 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.503 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.503 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.503 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.503 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:115 3.2 3:58045128:::b6aec59b24ac72144b32648c22fbe9fe410c37e40a3ad71dd7d382a103a9c104:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:42:05.503 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:42:05.503 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 167.60561853199999, 2024-04-22T18:42:05.503 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22648539100000001, 2024-04-22T18:42:05.503 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.503 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.503 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844644+0000", 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844642+0000", 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.504 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844810+0000", 2024-04-22T18:42:05.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016835600000000001 2024-04-22T18:42:05.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844827+0000", 2024-04-22T18:42:05.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6279000000000001e-05 2024-04-22T18:42:05.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.505 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.505 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.505 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844839+0000", 2024-04-22T18:42:05.505 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2356e-05 2024-04-22T18:42:05.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844899+0000", 2024-04-22T18:42:05.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9406e-05 2024-04-22T18:42:05.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845040+0000", 2024-04-22T18:42:05.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014105699999999999 2024-04-22T18:42:05.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.506 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.506 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:42:05.506 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845282+0000", 2024-04-22T18:42:05.506 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00024262200000000001 2024-04-22T18:42:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947847+0000", 2024-04-22T18:42:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10256459599999999 2024-04-22T18:42:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071019+0000", 2024-04-22T18:42:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.12317204499999999 2024-04-22T18:42:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071103+0000", 2024-04-22T18:42:05.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3908999999999999e-05 2024-04-22T18:42:05.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.508 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.508 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071129+0000", 2024-04-22T18:42:05.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6585e-05 2024-04-22T18:42:05.508 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.508 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.508 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.508 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.508 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:116 3.2 3:563a65d6:::42ba5e2647a3ff7a062c6cff68778032e0d8203536bf4d232ff6ac53f20780f9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-04-22T18:42:05.508 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:42:05.508 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 167.60541099599999, 2024-04-22T18:42:05.508 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.22637526199999999, 2024-04-22T18:42:05.509 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.509 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.509 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:42:05.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.509 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.509 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844851+0000", 2024-04-22T18:42:05.509 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.509 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.509 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844850+0000", 2024-04-22T18:42:05.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:42:05.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844925+0000", 2024-04-22T18:42:05.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4799e-05 2024-04-22T18:42:05.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.510 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.510 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.510 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844933+0000", 2024-04-22T18:42:05.510 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7670000000000003e-06 2024-04-22T18:42:05.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.844939+0000", 2024-04-22T18:42:05.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4389999999999997e-06 2024-04-22T18:42:05.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845728+0000", 2024-04-22T18:42:05.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00078844799999999997 2024-04-22T18:42:05.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.511 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.511 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.511 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.845854+0000", 2024-04-22T18:42:05.511 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012643299999999999 2024-04-22T18:42:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:42:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.846063+0000", 2024-04-22T18:42:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000208469 2024-04-22T18:42:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:17.947934+0000", 2024-04-22T18:42:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101870746 2024-04-22T18:42:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071189+0000", 2024-04-22T18:42:05.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.123255643 2024-04-22T18:42:05.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071210+0000", 2024-04-22T18:42:05.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.0823999999999999e-05 2024-04-22T18:42:05.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.513 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.513 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.513 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:18.071227+0000", 2024-04-22T18:42:05.513 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6728999999999999e-05 2024-04-22T18:42:05.513 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.513 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.513 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.514 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:229 3.2 3:5be209fc:::25051470fbcff6d2c4eb584796058598071a2406340389489f1b466fdd194c8a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:42:05.514 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:42:05.514 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.73976126299999, 2024-04-22T18:42:05.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.41626064699999998, 2024-04-22T18:42:05.514 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.514 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.514 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.514 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.514 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.514 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:42:05.514 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.514 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710501+0000", 2024-04-22T18:42:05.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710499+0000", 2024-04-22T18:42:05.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:05.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.515 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.515 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.515 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710507+0000", 2024-04-22T18:42:05.515 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3900000000000004e-06 2024-04-22T18:42:05.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710529+0000", 2024-04-22T18:42:05.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2313000000000001e-05 2024-04-22T18:42:05.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.710537+0000", 2024-04-22T18:42:05.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5560000000000002e-06 2024-04-22T18:42:05.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.516 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.516 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.516 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810716+0000", 2024-04-22T18:42:05.516 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017970700000001 2024-04-22T18:42:05.516 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810840+0000", 2024-04-22T18:42:05.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012369700000000001 2024-04-22T18:42:05.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:42:05.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.810930+0000", 2024-04-22T18:42:05.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0346999999999994e-05 2024-04-22T18:42:05.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.517 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.517 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.517 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.811405+0000", 2024-04-22T18:42:05.517 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00047492699999999999 2024-04-22T18:42:05.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126702+0000", 2024-04-22T18:42:05.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.31529641800000002 2024-04-22T18:42:05.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126735+0000", 2024-04-22T18:42:05.518 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3222999999999997e-05 2024-04-22T18:42:05.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.518 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.518 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.518 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.126762+0000", 2024-04-22T18:42:05.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6931999999999999e-05 2024-04-22T18:42:05.519 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.519 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.519 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.519 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:235 3.2 3:56877a58:::b6b6a1f39d772fee7e562b8b2f3fda689c97b97550ea37e33b34fd1c1e0cfa37:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:42:05.519 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:42:05.519 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.52592287799999, 2024-04-22T18:42:05.519 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302611401, 2024-04-22T18:42:05.519 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.519 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.519 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.519 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:42:05.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924340+0000", 2024-04-22T18:42:05.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.520 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.520 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924338+0000", 2024-04-22T18:42:05.520 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:05.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.520 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924345+0000", 2024-04-22T18:42:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.498e-06 2024-04-22T18:42:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924358+0000", 2024-04-22T18:42:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2859e-05 2024-04-22T18:42:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924365+0000", 2024-04-22T18:42:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1590000000000004e-06 2024-04-22T18:42:05.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924417+0000", 2024-04-22T18:42:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1314999999999999e-05 2024-04-22T18:42:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924495+0000", 2024-04-22T18:42:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8778999999999999e-05 2024-04-22T18:42:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:42:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924665+0000", 2024-04-22T18:42:05.522 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016940299999999999 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925385+0000", 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00072032599999999997 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226890+0000", 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30150469099999999 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226919+0000", 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8979000000000002e-05 2024-04-22T18:42:05.523 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.524 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.524 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.226951+0000", 2024-04-22T18:42:05.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2292000000000002e-05 2024-04-22T18:42:05.524 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.524 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.524 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.524 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.524 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.4.13:238 3.2 3:7858d359:::29d7ef230dd5abaea06fe773982fed77049cae5796ba4a9e5ffe42fa25ca8972:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:42:05.524 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:42:05.524 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.525889468, 2024-04-22T18:42:05.524 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.302738067, 2024-04-22T18:42:05.524 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924373+0000", 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924378+0000", 2024-04-22T18:42:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5509999999999999e-06 2024-04-22T18:42:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924383+0000", 2024-04-22T18:42:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4089999999999996e-06 2024-04-22T18:42:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924386+0000", 2024-04-22T18:42:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2839999999999998e-06 2024-04-22T18:42:05.526 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924915+0000", 2024-04-22T18:42:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00052879200000000004 2024-04-22T18:42:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.924978+0000", 2024-04-22T18:42:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3268999999999993e-05 2024-04-22T18:42:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:42:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925131+0000", 2024-04-22T18:42:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153308 2024-04-22T18:42:05.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.925954+0000", 2024-04-22T18:42:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00082311699999999995 2024-04-22T18:42:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227024+0000", 2024-04-22T18:42:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30106928500000002 2024-04-22T18:42:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227077+0000", 2024-04-22T18:42:05.528 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3013999999999998e-05 2024-04-22T18:42:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227111+0000", 2024-04-22T18:42:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4520000000000002e-05 2024-04-22T18:42:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:118 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:42:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:42:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.467006814, 2024-04-22T18:42:05.529 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24396172699999999, 2024-04-22T18:42:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:42:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983256+0000", 2024-04-22T18:42:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.530 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983254+0000", 2024-04-22T18:42:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983260+0000", 2024-04-22T18:42:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0120000000000002e-06 2024-04-22T18:42:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983272+0000", 2024-04-22T18:42:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.166e-05 2024-04-22T18:42:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.531 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983286+0000", 2024-04-22T18:42:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.394e-05 2024-04-22T18:42:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983313+0000", 2024-04-22T18:42:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6832e-05 2024-04-22T18:42:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983364+0000", 2024-04-22T18:42:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1375999999999997e-05 2024-04-22T18:42:05.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:42:05.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983461+0000", 2024-04-22T18:42:05.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6572999999999998e-05 2024-04-22T18:42:05.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983977+0000", 2024-04-22T18:42:05.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051574399999999995 2024-04-22T18:42:05.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.533 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.533 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.533 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227161+0000", 2024-04-22T18:42:05.533 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24318442000000001 2024-04-22T18:42:05.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227194+0000", 2024-04-22T18:42:05.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3305e-05 2024-04-22T18:42:05.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.534 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.534 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227217+0000", 2024-04-22T18:42:05.534 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3059000000000001e-05 2024-04-22T18:42:05.534 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.534 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.534 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.534 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.534 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.535 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:119 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:42:05.535 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:42:05.535 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.46686819799999, 2024-04-22T18:42:05.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243916152, 2024-04-22T18:42:05.535 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.535 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.535 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.535 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:42:05.535 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.535 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.535 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:42:05.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983394+0000", 2024-04-22T18:42:05.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.536 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.536 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.536 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983398+0000", 2024-04-22T18:42:05.536 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5700000000000003e-06 2024-04-22T18:42:05.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983404+0000", 2024-04-22T18:42:05.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8340000000000004e-06 2024-04-22T18:42:05.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.537 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.537 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983408+0000", 2024-04-22T18:42:05.537 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1559999999999997e-06 2024-04-22T18:42:05.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.537 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983645+0000", 2024-04-22T18:42:05.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000236765 2024-04-22T18:42:05.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.538 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983698+0000", 2024-04-22T18:42:05.538 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.342e-05 2024-04-22T18:42:05.538 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.538 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.538 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:42:05.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983798+0000", 2024-04-22T18:42:05.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.9585000000000003e-05 2024-04-22T18:42:05.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984441+0000", 2024-04-22T18:42:05.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00064350599999999996 2024-04-22T18:42:05.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.539 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.539 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227252+0000", 2024-04-22T18:42:05.539 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.242810786 2024-04-22T18:42:05.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.539 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227281+0000", 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.826e-05 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227310+0000", 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9916999999999998e-05 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.0.13:120 3.2 3:43d85947:::smithi08937929-18 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:42:05.540 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 148.46684919500001, 2024-04-22T18:42:05.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.243982272, 2024-04-22T18:42:05.541 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.541 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.541 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:42:05.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.541 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.541 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:42:05.541 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.541 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983413+0000", 2024-04-22T18:42:05.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983416+0000", 2024-04-22T18:42:05.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.084e-06 2024-04-22T18:42:05.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.542 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.542 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.542 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983420+0000", 2024-04-22T18:42:05.542 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0929999999999996e-06 2024-04-22T18:42:05.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.983424+0000", 2024-04-22T18:42:05.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.4699999999999998e-06 2024-04-22T18:42:05.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984004+0000", 2024-04-22T18:42:05.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00058033399999999995 2024-04-22T18:42:05.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.543 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.543 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.543 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984092+0000", 2024-04-22T18:42:05.543 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7929999999999993e-05 2024-04-22T18:42:05.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 3", 2024-04-22T18:42:05.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984246+0000", 2024-04-22T18:42:05.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000153714 2024-04-22T18:42:05.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:36.984642+0000", 2024-04-22T18:42:05.544 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00039653900000000001 2024-04-22T18:42:05.544 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.544 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.544 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.544 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227351+0000", 2024-04-22T18:42:05.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.24270841500000001 2024-04-22T18:42:05.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227377+0000", 2024-04-22T18:42:05.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6139000000000002e-05 2024-04-22T18:42:05.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.545 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.545 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:37.227396+0000", 2024-04-22T18:42:05.545 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8875000000000001e-05 2024-04-22T18:42:05.545 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.545 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.545 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.546 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.546 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2073 4.8 4:12d9f826:::smithi08937929-1:head [write 1849419~592048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:42:05.546 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:42:05.546 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.84583666200001, 2024-04-22T18:42:05.546 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36759219399999998, 2024-04-22T18:42:05.546 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.546 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.546 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.546 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2073 2024-04-22T18:42:05.546 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.546 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:42:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604426+0000", 2024-04-22T18:42:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604424+0000", 2024-04-22T18:42:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604914+0000", 2024-04-22T18:42:05.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00049043800000000005 2024-04-22T18:42:05.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604927+0000", 2024-04-22T18:42:05.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2627e-05 2024-04-22T18:42:05.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.548 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.548 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.548 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604935+0000", 2024-04-22T18:42:05.548 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1280000000000008e-06 2024-04-22T18:42:05.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604978+0000", 2024-04-22T18:42:05.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3424999999999998e-05 2024-04-22T18:42:05.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605047+0000", 2024-04-22T18:42:05.549 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.8485000000000006e-05 2024-04-22T18:42:05.549 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.549 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.549 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:42:05.549 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.605153+0000", 2024-04-22T18:42:05.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000106455 2024-04-22T18:42:05.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.607394+0000", 2024-04-22T18:42:05.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0022407630000000002 2024-04-22T18:42:05.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971857+0000", 2024-04-22T18:42:05.550 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36446299799999998 2024-04-22T18:42:05.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.550 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.550 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.550 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.971991+0000", 2024-04-22T18:42:05.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013360499999999999 2024-04-22T18:42:05.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.551 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.551 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.972018+0000", 2024-04-22T18:42:05.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7280999999999999e-05 2024-04-22T18:42:05.551 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.551 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.551 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.551 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.551 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2074 4.8 4:12d9f826:::smithi08937929-1:head [setxattr _header (55) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:42:05.551 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:42:05.551 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.84531707100001, 2024-04-22T18:42:05.551 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.46754099599999999, 2024-04-22T18:42:05.552 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.552 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.552 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.552 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.552 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.552 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2074 2024-04-22T18:42:05.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.552 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.552 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.552 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.552 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:42:05.552 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604945+0000", 2024-04-22T18:42:05.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604944+0000", 2024-04-22T18:42:05.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:42:05.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.553 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.553 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.553 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604950+0000", 2024-04-22T18:42:05.553 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0549999999999996e-06 2024-04-22T18:42:05.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.604963+0000", 2024-04-22T18:42:05.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2693e-05 2024-04-22T18:42:05.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705062+0000", 2024-04-22T18:42:05.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100099201 2024-04-22T18:42:05.554 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.554 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.554 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.554 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705188+0000", 2024-04-22T18:42:05.554 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000125723 2024-04-22T18:42:05.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705264+0000", 2024-04-22T18:42:05.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6342999999999997e-05 2024-04-22T18:42:05.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:42:05.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705365+0000", 2024-04-22T18:42:05.555 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010050700000000001 2024-04-22T18:42:05.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.555 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.555 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.805677+0000", 2024-04-22T18:42:05.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10031206600000001 2024-04-22T18:42:05.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072360+0000", 2024-04-22T18:42:05.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.26668324399999999 2024-04-22T18:42:05.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.556 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072444+0000", 2024-04-22T18:42:05.556 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3387000000000006e-05 2024-04-22T18:42:05.556 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.556 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.556 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.557 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072486+0000", 2024-04-22T18:42:05.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2910000000000001e-05 2024-04-22T18:42:05.557 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.557 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.557 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.557 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.557 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2075 4.8 4:12d9f826:::smithi08937929-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-04-22T18:42:05.557 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:42:05.557 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 144.74516418799999, 2024-04-22T18:42:05.557 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.36757730199999999, 2024-04-22T18:42:05.557 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.557 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.557 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.558 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.558 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.558 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2075 2024-04-22T18:42:05.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.558 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:42:05.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.558 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.558 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.558 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705098+0000", 2024-04-22T18:42:05.558 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.558 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705096+0000", 2024-04-22T18:42:05.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:05.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705104+0000", 2024-04-22T18:42:05.559 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1049999999999995e-06 2024-04-22T18:42:05.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.559 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.559 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.559 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705112+0000", 2024-04-22T18:42:05.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7579999999999992e-06 2024-04-22T18:42:05.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.705118+0000", 2024-04-22T18:42:05.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2349999999999996e-06 2024-04-22T18:42:05.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706058+0000", 2024-04-22T18:42:05.560 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00093942500000000003 2024-04-22T18:42:05.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.560 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.560 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:05.560 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:40.706127+0000", 2024-04-22T18:42:05.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9062999999999999e-05 2024-04-22T18:42:05.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072534+0000", 2024-04-22T18:42:05.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.366407123 2024-04-22T18:42:05.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.561 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072604+0000", 2024-04-22T18:42:05.561 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0165000000000003e-05 2024-04-22T18:42:05.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.561 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.561 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.562 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:41.072676+0000", 2024-04-22T18:42:05.562 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.1373999999999998e-05 2024-04-22T18:42:05.562 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.562 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.562 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.562 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.562 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.562 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:42:05.563 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-22T18:42:05.666 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:42:05.666 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:42:05.666 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:42:05.666 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:42:05.675 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-22T18:42:05.776 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:42:05.776 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:42:05.776 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:42:05.776 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:42:05.776 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:42:05.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-22T18:42:05.800 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:42:05.801+0000 7f7f44bb3640 -1 osd.4 188 handle_osd_map injecting map crc failure 2024-04-22T18:42:05.890 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:42:05.890 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:42:05.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:42:05.890 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:42:05.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.890 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:57 4.a 4:519c08f7:::smithi08937929-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:42:05.890 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:42:05.890 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 191.63028005499999, 2024-04-22T18:42:05.890 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48882900499999998, 2024-04-22T18:42:05.890 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.890 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.891 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 57 2024-04-22T18:42:05.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.891 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:42:05.891 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.891 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.891 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259647+0000", 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.259645+0000", 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260455+0000", 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00081010900000000005 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260465+0000", 2024-04-22T18:42:05.892 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5550000000000008e-06 2024-04-22T18:42:05.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.260471+0000", 2024-04-22T18:42:05.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3980000000000004e-06 2024-04-22T18:42:05.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363831+0000", 2024-04-22T18:42:05.893 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103359906 2024-04-22T18:42:05.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.893 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.893 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.363941+0000", 2024-04-22T18:42:05.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00011034100000000001 2024-04-22T18:42:05.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:05.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.364095+0000", 2024-04-22T18:42:05.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154017 2024-04-22T18:42:05.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.894 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.894 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466848+0000", 2024-04-22T18:42:05.894 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10275267 2024-04-22T18:42:05.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748305+0000", 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.28145755 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748429+0000", 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012335499999999999 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.748476+0000", 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7395999999999998e-05 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.895 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.896 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:58 4.a 4:519c08f7:::smithi08937929-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:42:05.896 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:42:05.896 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 191.62549762500001, 2024-04-22T18:42:05.896 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58431225499999995, 2024-04-22T18:42:05.896 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.896 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.896 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.896 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.896 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.896 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 58 2024-04-22T18:42:05.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.896 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264430+0000", 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264427+0000", 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264501+0000", 2024-04-22T18:42:05.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3509000000000004e-05 2024-04-22T18:42:05.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264511+0000", 2024-04-22T18:42:05.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0516000000000001e-05 2024-04-22T18:42:05.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.898 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.898 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264518+0000", 2024-04-22T18:42:05.898 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0360000000000001e-06 2024-04-22T18:42:05.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465460+0000", 2024-04-22T18:42:05.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20094114299999999 2024-04-22T18:42:05.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465555+0000", 2024-04-22T18:42:05.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5487000000000002e-05 2024-04-22T18:42:05.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.899 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:05.899 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.465697+0000", 2024-04-22T18:42:05.899 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014247500000000001 2024-04-22T18:42:05.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466854+0000", 2024-04-22T18:42:05.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00115605 2024-04-22T18:42:05.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848653+0000", 2024-04-22T18:42:05.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38179925999999997 2024-04-22T18:42:05.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.900 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.900 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848704+0000", 2024-04-22T18:42:05.900 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1079999999999999e-05 2024-04-22T18:42:05.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848742+0000", 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8025999999999998e-05 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:59 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 191.62540342200001, 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58434748599999997, 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.901 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.902 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.902 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.902 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 59 2024-04-22T18:42:05.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.902 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:42:05.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.902 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.902 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.902 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264524+0000", 2024-04-22T18:42:05.902 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264523+0000", 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264529+0000", 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7629999999999997e-06 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264535+0000", 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.1879999999999999e-06 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.903 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264540+0000", 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0529999999999997e-06 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466291+0000", 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20175077799999999 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466356+0000", 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.5035000000000003e-05 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.904 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:05.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466485+0000", 2024-04-22T18:42:05.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012924900000000001 2024-04-22T18:42:05.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466869+0000", 2024-04-22T18:42:05.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038427399999999999 2024-04-22T18:42:05.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.905 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848810+0000", 2024-04-22T18:42:05.905 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38194095500000003 2024-04-22T18:42:05.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.905 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.905 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848840+0000", 2024-04-22T18:42:05.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9312999999999999e-05 2024-04-22T18:42:05.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.906 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.906 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848871+0000", 2024-04-22T18:42:05.906 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.1761999999999998e-05 2024-04-22T18:42:05.906 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.906 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.906 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.906 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.906 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:60 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:42:05.906 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:42:05.907 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 191.62538253400001, 2024-04-22T18:42:05.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58445202500000004, 2024-04-22T18:42:05.907 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.907 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.907 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.907 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.907 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.907 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 60 2024-04-22T18:42:05.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.907 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.907 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.907 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.907 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:42:05.907 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264545+0000", 2024-04-22T18:42:05.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264544+0000", 2024-04-22T18:42:05.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:42:05.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.908 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.908 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.908 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264548+0000", 2024-04-22T18:42:05.908 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.737e-06 2024-04-22T18:42:05.908 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264553+0000", 2024-04-22T18:42:05.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4909999999999996e-06 2024-04-22T18:42:05.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.264559+0000", 2024-04-22T18:42:05.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3000000000000001e-06 2024-04-22T18:42:05.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.909 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.909 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.909 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466773+0000", 2024-04-22T18:42:05.909 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20221408099999999 2024-04-22T18:42:05.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:05.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.466836+0000", 2024-04-22T18:42:05.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3303e-05 2024-04-22T18:42:05.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848895+0000", 2024-04-22T18:42:05.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.38205874899999998 2024-04-22T18:42:05.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.910 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.910 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.910 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848942+0000", 2024-04-22T18:42:05.910 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7329e-05 2024-04-22T18:42:05.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.911 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.911 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:54.848997+0000", 2024-04-22T18:42:05.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4795e-05 2024-04-22T18:42:05.911 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.911 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.911 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.911 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.911 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:157 4.5 4:a4a14dff:::smithi08937929-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:42:05.911 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:42:05.911 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 190.842374871, 2024-04-22T18:42:05.911 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55875614699999998, 2024-04-22T18:42:05.911 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 157 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047552+0000", 2024-04-22T18:42:05.912 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.047550+0000", 2024-04-22T18:42:05.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:42:05.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048287+0000", 2024-04-22T18:42:05.913 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00073718799999999995 2024-04-22T18:42:05.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.913 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.913 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.913 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048300+0000", 2024-04-22T18:42:05.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3183999999999999e-05 2024-04-22T18:42:05.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048308+0000", 2024-04-22T18:42:05.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.3499999999999999e-06 2024-04-22T18:42:05.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.914 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048357+0000", 2024-04-22T18:42:05.914 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9601000000000002e-05 2024-04-22T18:42:05.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.914 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.914 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048493+0000", 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013593200000000001 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.048632+0000", 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013825900000000001 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.050240+0000", 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0016080599999999999 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.915 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606259+0000", 2024-04-22T18:42:05.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.556019245 2024-04-22T18:42:05.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606285+0000", 2024-04-22T18:42:05.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.5743000000000001e-05 2024-04-22T18:42:05.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.916 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.916 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.916 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606309+0000", 2024-04-22T18:42:05.916 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3929000000000001e-05 2024-04-22T18:42:05.916 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.916 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.916 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.917 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.917 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:158 4.5 4:a4a14dff:::smithi08937929-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-22T18:42:05.917 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:42:05.917 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 190.834543465, 2024-04-22T18:42:05.917 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.55104249100000002, 2024-04-22T18:42:05.917 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.917 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.917 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.917 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.917 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.917 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 158 2024-04-22T18:42:05.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:42:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055384+0000", 2024-04-22T18:42:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.918 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055381+0000", 2024-04-22T18:42:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:42:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055959+0000", 2024-04-22T18:42:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00057859800000000005 2024-04-22T18:42:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055979+0000", 2024-04-22T18:42:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9635000000000001e-05 2024-04-22T18:42:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.919 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.055989+0000", 2024-04-22T18:42:05.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0149999999999999e-05 2024-04-22T18:42:05.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056064+0000", 2024-04-22T18:42:05.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5230999999999999e-05 2024-04-22T18:42:05.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.920 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056139+0000", 2024-04-22T18:42:05.920 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.5128000000000003e-05 2024-04-22T18:42:05.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.920 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.920 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:42:05.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.056234+0000", 2024-04-22T18:42:05.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.4571e-05 2024-04-22T18:42:05.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.156637+0000", 2024-04-22T18:42:05.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10040314 2024-04-22T18:42:05.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.921 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.921 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606355+0000", 2024-04-22T18:42:05.921 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.44971787400000002 2024-04-22T18:42:05.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.921 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606393+0000", 2024-04-22T18:42:05.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8350999999999999e-05 2024-04-22T18:42:05.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.922 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.922 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:55.606426+0000", 2024-04-22T18:42:05.922 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3123000000000001e-05 2024-04-22T18:42:05.922 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.922 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.922 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.922 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2504 4.5 4:a6d237d2:::smithi08937929-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.68267083500001, 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43734996199999998, 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2504 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207256+0000", 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207254+0000", 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207625+0000", 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00037060499999999998 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207638+0000", 2024-04-22T18:42:05.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3018e-05 2024-04-22T18:42:05.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207644+0000", 2024-04-22T18:42:05.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3459999999999999e-06 2024-04-22T18:42:05.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.925 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.925 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.241986+0000", 2024-04-22T18:42:05.925 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.034341597000000001 2024-04-22T18:42:05.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242127+0000", 2024-04-22T18:42:05.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014064700000000001 2024-04-22T18:42:05.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 4", 2024-04-22T18:42:05.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.242288+0000", 2024-04-22T18:42:05.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016162899999999999 2024-04-22T18:42:05.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.926 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.926 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.297439+0000", 2024-04-22T18:42:05.926 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.055150953000000003 2024-04-22T18:42:05.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644438+0000", 2024-04-22T18:42:05.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.34699853000000003 2024-04-22T18:42:05.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644519+0000", 2024-04-22T18:42:05.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1309000000000004e-05 2024-04-22T18:42:05.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.927 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.927 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644606+0000", 2024-04-22T18:42:05.927 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7326999999999999e-05 2024-04-22T18:42:05.927 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:2506 4.5 4:a6d237d2:::smithi08937929-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e65)", 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 130.682265927, 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43719112199999999, 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 2506 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:42:05.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:42:05.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.929 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.929 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207661+0000", 2024-04-22T18:42:05.929 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207663+0000", 2024-04-22T18:42:05.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8840000000000001e-06 2024-04-22T18:42:05.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207666+0000", 2024-04-22T18:42:05.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.117e-06 2024-04-22T18:42:05.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.930 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.930 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.207669+0000", 2024-04-22T18:42:05.930 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9390000000000002e-06 2024-04-22T18:42:05.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243523+0000", 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.035853548999999998 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.243571+0000", 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7981e-05 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644747+0000", 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40117610300000001 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.931 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644792+0000", 2024-04-22T18:42:05.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4990999999999997e-05 2024-04-22T18:42:05.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.932 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.932 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:39:55.644852+0000", 2024-04-22T18:42:05.932 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0877e-05 2024-04-22T18:42:05.932 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.932 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.932 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.932 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3213 4.a 4:5d2aee44:::smithi08937929-13:head [assert-version v66,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:42:05.932 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:42:05.932 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 103.321366039, 2024-04-22T18:42:05.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.69928701400000004, 2024-04-22T18:42:05.933 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.933 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.933 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.933 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.933 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.933 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3213 2024-04-22T18:42:05.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.933 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.933 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.933 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:42:05.933 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.933 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568561+0000", 2024-04-22T18:42:05.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568559+0000", 2024-04-22T18:42:05.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:05.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.934 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.934 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568567+0000", 2024-04-22T18:42:05.934 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6399999999999997e-06 2024-04-22T18:42:05.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.568579+0000", 2024-04-22T18:42:05.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1634000000000001e-05 2024-04-22T18:42:05.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.668711+0000", 2024-04-22T18:42:05.935 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100131866 2024-04-22T18:42:05.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.935 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.935 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.768945+0000", 2024-04-22T18:42:05.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100234252 2024-04-22T18:42:05.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.769002+0000", 2024-04-22T18:42:05.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7092999999999997e-05 2024-04-22T18:42:05.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:05.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.166404+0000", 2024-04-22T18:42:05.936 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39740241999999998 2024-04-22T18:42:05.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.936 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.936 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.191686+0000", 2024-04-22T18:42:05.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.025281739000000001 2024-04-22T18:42:05.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267758+0000", 2024-04-22T18:42:05.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.076071606999999999 2024-04-22T18:42:05.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.937 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267812+0000", 2024-04-22T18:42:05.937 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4478999999999998e-05 2024-04-22T18:42:05.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.937 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.938 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.267848+0000", 2024-04-22T18:42:05.938 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6062999999999997e-05 2024-04-22T18:42:05.938 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.938 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.938 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.938 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3319 4.a 4:5d2aee44:::smithi08937929-13:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-04-22T18:42:05.938 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:42:05.938 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 99.429618288, 2024-04-22T18:42:05.939 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476623401, 2024-04-22T18:42:05.939 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.939 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.939 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.939 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.939 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.939 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3319 2024-04-22T18:42:05.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.939 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.939 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.939 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:42:05.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460309+0000", 2024-04-22T18:42:05.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.940 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460306+0000", 2024-04-22T18:42:05.940 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999967 2024-04-22T18:42:05.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.940 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460318+0000", 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2896e-05 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460345+0000", 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6650000000000001e-05 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460360+0000", 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4729000000000001e-05 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.941 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460502+0000", 2024-04-22T18:42:05.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000141729 2024-04-22T18:42:05.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.460644+0000", 2024-04-22T18:42:05.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000142755 2024-04-22T18:42:05.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.942 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.942 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:26.936932+0000", 2024-04-22T18:42:05.942 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.476288078 2024-04-22T18:42:05.942 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.942 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.943 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.943 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3412 4.a 4:519c08f7:::smithi08937929-15:head [write 2529878~394573] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:42:05.943 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:42:05.943 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.747783726999998, 2024-04-22T18:42:05.943 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48918094899999998, 2024-04-22T18:42:05.943 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.943 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.943 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.943 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.943 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.943 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3412 2024-04-22T18:42:05.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:42:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142144+0000", 2024-04-22T18:42:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142141+0000", 2024-04-22T18:42:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999981 2024-04-22T18:42:05.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142496+0000", 2024-04-22T18:42:05.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00035444699999999998 2024-04-22T18:42:05.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.945 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.945 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142510+0000", 2024-04-22T18:42:05.945 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3841e-05 2024-04-22T18:42:05.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.945 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142518+0000", 2024-04-22T18:42:05.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8089999999999993e-06 2024-04-22T18:42:05.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142560+0000", 2024-04-22T18:42:05.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2097e-05 2024-04-22T18:42:05.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.946 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.946 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.946 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142637+0000", 2024-04-22T18:42:05.946 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6965000000000006e-05 2024-04-22T18:42:05.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:05.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142715+0000", 2024-04-22T18:42:05.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.8694000000000002e-05 2024-04-22T18:42:05.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144179+0000", 2024-04-22T18:42:05.947 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00146342 2024-04-22T18:42:05.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.947 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.947 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.947 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631229+0000", 2024-04-22T18:42:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.48705018500000002 2024-04-22T18:42:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631281+0000", 2024-04-22T18:42:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2216999999999998e-05 2024-04-22T18:42:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.631324+0000", 2024-04-22T18:42:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3344999999999999e-05 2024-04-22T18:42:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.949 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3413 4.a 4:519c08f7:::smithi08937929-15:head [setxattr _header (56) in=7b,truncate 2924451] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:42:05.949 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:42:05.949 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.747402586999996, 2024-04-22T18:42:05.949 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58904086899999997, 2024-04-22T18:42:05.949 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.949 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.949 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.949 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.949 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.949 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3413 2024-04-22T18:42:05.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.949 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.949 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:42:05.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142525+0000", 2024-04-22T18:42:05.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.950 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.950 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.950 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142523+0000", 2024-04-22T18:42:05.950 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:05.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142529+0000", 2024-04-22T18:42:05.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.6049999999999997e-06 2024-04-22T18:42:05.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142535+0000", 2024-04-22T18:42:05.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4740000000000002e-06 2024-04-22T18:42:05.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.951 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.951 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.951 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142541+0000", 2024-04-22T18:42:05.951 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.1839999999999998e-06 2024-04-22T18:42:05.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143553+0000", 2024-04-22T18:42:05.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010126689999999999 2024-04-22T18:42:05.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143659+0000", 2024-04-22T18:42:05.952 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000105393 2024-04-22T18:42:05.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.952 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:05.952 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.143825+0000", 2024-04-22T18:42:05.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000165809 2024-04-22T18:42:05.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144329+0000", 2024-04-22T18:42:05.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00050483799999999997 2024-04-22T18:42:05.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.953 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731480+0000", 2024-04-22T18:42:05.953 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58715100600000003 2024-04-22T18:42:05.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.953 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.953 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731522+0000", 2024-04-22T18:42:05.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2098000000000002e-05 2024-04-22T18:42:05.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.954 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.954 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731566+0000", 2024-04-22T18:42:05.954 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3066999999999999e-05 2024-04-22T18:42:05.954 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.954 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.954 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.954 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3414 4.a 4:519c08f7:::smithi08937929-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 94.747373983000003, 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58926020199999996, 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3414 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:42:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142553+0000", 2024-04-22T18:42:05.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142552+0000", 2024-04-22T18:42:05.956 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:42:05.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.956 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.956 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142556+0000", 2024-04-22T18:42:05.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0289999999999999e-06 2024-04-22T18:42:05.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142574+0000", 2024-04-22T18:42:05.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.7266999999999999e-05 2024-04-22T18:42:05.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.957 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.142581+0000", 2024-04-22T18:42:05.957 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7109999999999995e-06 2024-04-22T18:42:05.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.957 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144121+0000", 2024-04-22T18:42:05.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0015397480000000001 2024-04-22T18:42:05.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:05.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.144170+0000", 2024-04-22T18:42:05.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8803e-05 2024-04-22T18:42:05.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.958 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.958 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731624+0000", 2024-04-22T18:42:05.958 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.58745365800000005 2024-04-22T18:42:05.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731712+0000", 2024-04-22T18:42:05.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8016999999999994e-05 2024-04-22T18:42:05.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.959 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.959 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:31.731813+0000", 2024-04-22T18:42:05.959 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000101817 2024-04-22T18:42:05.959 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.959 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.959 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.959 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:77 3.3 3:dcc8aeec:::3590d5741efdda41a26b62a0982727439225b7b4b51097b313f30d8dc10bf6f6:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:42:05.959 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:42:05.960 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 90.642355660999996, 2024-04-22T18:42:05.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011812878, 2024-04-22T18:42:05.960 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.960 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.960 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:42:05.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.960 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.960 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247572+0000", 2024-04-22T18:42:05.960 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247571+0000", 2024-04-22T18:42:05.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247574+0000", 2024-04-22T18:42:05.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.491e-06 2024-04-22T18:42:05.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.961 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.961 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247577+0000", 2024-04-22T18:42:05.961 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3069999999999998e-06 2024-04-22T18:42:05.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247580+0000", 2024-04-22T18:42:05.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8870000000000001e-06 2024-04-22T18:42:05.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.340928+0000", 2024-04-22T18:42:05.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093348678000000004 2024-04-22T18:42:05.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.962 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:42:05.962 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341122+0000", 2024-04-22T18:42:05.962 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000193227 2024-04-22T18:42:05.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618335+0000", 2024-04-22T18:42:05.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.277213775 2024-04-22T18:42:05.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:05.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.618422+0000", 2024-04-22T18:42:05.963 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.6537000000000002e-05 2024-04-22T18:42:05.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.963 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.963 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954387+0000", 2024-04-22T18:42:05.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.33596509699999999 2024-04-22T18:42:05.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954548+0000", 2024-04-22T18:42:05.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016042199999999999 2024-04-22T18:42:05.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:05.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.954858+0000", 2024-04-22T18:42:05.964 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000310396 2024-04-22T18:42:05.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.964 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.964 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157019+0000", 2024-04-22T18:42:05.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.20216071299999999 2024-04-22T18:42:05.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259213+0000", 2024-04-22T18:42:05.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.102194734 2024-04-22T18:42:05.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259262+0000", 2024-04-22T18:42:05.965 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9052999999999999e-05 2024-04-22T18:42:05.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.965 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.965 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259384+0000", 2024-04-22T18:42:05.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000122052 2024-04-22T18:42:05.966 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.966 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.966 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.966 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:78 3.3 3:df7e90d2:::ab0657e24bd96704775de2a378746bc22309c5a7366a5eaf9b4e1f1f2bef0a4c:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:42:05.966 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:42:05.966 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 90.642340464, 2024-04-22T18:42:05.966 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.011974739, 2024-04-22T18:42:05.966 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.966 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.966 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:42:05.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247587+0000", 2024-04-22T18:42:05.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.967 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.967 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247586+0000", 2024-04-22T18:42:05.967 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247589+0000", 2024-04-22T18:42:05.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.181e-06 2024-04-22T18:42:05.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247592+0000", 2024-04-22T18:42:05.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0230000000000001e-06 2024-04-22T18:42:05.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.968 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.968 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247595+0000", 2024-04-22T18:42:05.968 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.428e-06 2024-04-22T18:42:05.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341160+0000", 2024-04-22T18:42:05.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093564571999999999 2024-04-22T18:42:05.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:42:05.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341264+0000", 2024-04-22T18:42:05.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010391999999999999 2024-04-22T18:42:05.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.969 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.969 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619269+0000", 2024-04-22T18:42:05.969 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27800575100000002 2024-04-22T18:42:05.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:05.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.619346+0000", 2024-04-22T18:42:05.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.6459999999999996e-05 2024-04-22T18:42:05.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.155957+0000", 2024-04-22T18:42:05.970 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53661109299999998 2024-04-22T18:42:05.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.970 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.970 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.970 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156049+0000", 2024-04-22T18:42:05.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2272e-05 2024-04-22T18:42:05.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:05.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.156236+0000", 2024-04-22T18:42:05.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000186808 2024-04-22T18:42:05.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.971 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.158088+0000", 2024-04-22T18:42:05.971 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0018522079999999999 2024-04-22T18:42:05.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.971 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.971 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259421+0000", 2024-04-22T18:42:05.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.101332718 2024-04-22T18:42:05.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259460+0000", 2024-04-22T18:42:05.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8912000000000003e-05 2024-04-22T18:42:05.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.972 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.972 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.972 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259562+0000", 2024-04-22T18:42:05.972 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00010175600000000001 2024-04-22T18:42:05.972 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.973 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.973 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.973 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.6.13:80 3.3 3:f90f47f2:::e5603f596d699ee3e27e255225eeb90a9f0c1b4a4d099d1c586b19019220aa3f:head [call cas.chunk_put_ref in=78b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:42:05.973 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:42:05.973 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 90.642301387000003, 2024-04-22T18:42:05.973 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0121096759999999, 2024-04-22T18:42:05.973 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.973 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.973 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.973 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.973 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.973 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:42:05.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247626+0000", 2024-04-22T18:42:05.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.974 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.974 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247625+0000", 2024-04-22T18:42:05.974 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:05.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.974 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247628+0000", 2024-04-22T18:42:05.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6129999999999998e-06 2024-04-22T18:42:05.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247631+0000", 2024-04-22T18:42:05.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3869999999999999e-06 2024-04-22T18:42:05.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.975 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.975 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.975 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.247634+0000", 2024-04-22T18:42:05.975 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9919999999999999e-06 2024-04-22T18:42:05.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341550+0000", 2024-04-22T18:42:05.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.093915584999999996 2024-04-22T18:42:05.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:42:05.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.341700+0000", 2024-04-22T18:42:05.976 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015000300000000001 2024-04-22T18:42:05.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.976 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.976 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.976 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620744+0000", 2024-04-22T18:42:05.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.27904399200000002 2024-04-22T18:42:05.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:05.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.620794+0000", 2024-04-22T18:42:05.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9579000000000001e-05 2024-04-22T18:42:05.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157597+0000", 2024-04-22T18:42:05.977 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.53680343799999997 2024-04-22T18:42:05.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.977 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.977 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.977 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157678+0000", 2024-04-22T18:42:05.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0829999999999997e-05 2024-04-22T18:42:05.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 1", 2024-04-22T18:42:05.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.157832+0000", 2024-04-22T18:42:05.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000154184 2024-04-22T18:42:05.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.978 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259572+0000", 2024-04-22T18:42:05.978 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10174003600000001 2024-04-22T18:42:05.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.978 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.978 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259601+0000", 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8841999999999999e-05 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259638+0000", 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7308999999999998e-05 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259736+0000", 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7283999999999999e-05 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(osd.1.13:135 3.3 3:f89134e5:::3e4e73f5d4a0122957865f0faeac2483204ee98d7d497ef3b54afe2a02b39a3d:head [read 0~802304 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e104)", 2024-04-22T18:42:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:42:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 90.274486307999993, 2024-04-22T18:42:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.64370653499999997, 2024-04-22T18:42:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:42:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.980 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615441+0000", 2024-04-22T18:42:05.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615438+0000", 2024-04-22T18:42:05.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999971 2024-04-22T18:42:05.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.981 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.981 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615449+0000", 2024-04-22T18:42:05.981 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1219999999999999e-05 2024-04-22T18:42:05.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.981 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.615460+0000", 2024-04-22T18:42:05.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0403e-05 2024-04-22T18:42:05.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.715588+0000", 2024-04-22T18:42:05.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100128881 2024-04-22T18:42:05.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.982 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.982 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.982 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722533+0000", 2024-04-22T18:42:05.982 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0069444010000000002 2024-04-22T18:42:05.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for missing object", 2024-04-22T18:42:05.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.722658+0000", 2024-04-22T18:42:05.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012560699999999999 2024-04-22T18:42:05.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826437+0000", 2024-04-22T18:42:05.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.103778194 2024-04-22T18:42:05.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.983 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.983 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for degraded object", 2024-04-22T18:42:05.983 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:35.826532+0000", 2024-04-22T18:42:05.983 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5111000000000005e-05 2024-04-22T18:42:05.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258787+0000", 2024-04-22T18:42:05.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.43225529099999999 2024-04-22T18:42:05.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.258905+0000", 2024-04-22T18:42:05.984 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000118348 2024-04-22T18:42:05.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.984 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.984 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.984 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:36.259148+0000", 2024-04-22T18:42:05.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000242093 2024-04-22T18:42:05.985 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.985 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.985 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.985 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.985 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3890 4.1 4:83c955bf:::smithi08937929-28:head [write 2699879~469889] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:42:05.985 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:42:05.985 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.275811464, 2024-04-22T18:42:05.985 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49042297000000001, 2024-04-22T18:42:05.985 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.985 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.985 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.985 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.985 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.986 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3890 2024-04-22T18:42:05.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.986 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:42:05.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.986 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.986 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.986 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614116+0000", 2024-04-22T18:42:05.986 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614115+0000", 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614496+0000", 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00038181700000000001 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614509+0000", 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2381000000000001e-05 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.987 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614517+0000", 2024-04-22T18:42:05.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0879999999999995e-06 2024-04-22T18:42:05.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615610+0000", 2024-04-22T18:42:05.988 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010933519999999999 2024-04-22T18:42:05.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.988 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.988 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.988 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615655+0000", 2024-04-22T18:42:05.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4966000000000003e-05 2024-04-22T18:42:05.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:42:05.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.615769+0000", 2024-04-22T18:42:05.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000113986 2024-04-22T18:42:05.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.989 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704882+0000", 2024-04-22T18:42:05.989 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089112825000000007 2024-04-22T18:42:05.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.989 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.989 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104488+0000", 2024-04-22T18:42:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39960572999999999 2024-04-22T18:42:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104517+0000", 2024-04-22T18:42:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9006e-05 2024-04-22T18:42:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.104539+0000", 2024-04-22T18:42:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2073e-05 2024-04-22T18:42:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.990 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.991 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.991 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.991 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3891 4.1 4:83c955bf:::smithi08937929-28:head [setxattr _header (56) in=7b,truncate 3630345] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:42:05.991 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:42:05.991 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.275403233000006, 2024-04-22T18:42:05.991 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59018588699999996, 2024-04-22T18:42:05.991 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.991 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:05.991 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.991 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.991 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.991 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3891 2024-04-22T18:42:05.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.991 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:05.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:05.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:42:05.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:05.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614524+0000", 2024-04-22T18:42:05.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:05.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.992 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.992 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:05.992 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614523+0000", 2024-04-22T18:42:05.992 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:42:05.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:05.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614528+0000", 2024-04-22T18:42:05.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0379999999999999e-06 2024-04-22T18:42:05.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:05.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614540+0000", 2024-04-22T18:42:05.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018000000000001e-05 2024-04-22T18:42:05.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.993 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.993 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:05.993 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614546+0000", 2024-04-22T18:42:05.993 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.4990000000000002e-06 2024-04-22T18:42:05.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:05.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704438+0000", 2024-04-22T18:42:05.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.089892044000000004 2024-04-22T18:42:05.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.994 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.994 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:05.994 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704504+0000", 2024-04-22T18:42:05.994 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6365000000000006e-05 2024-04-22T18:42:05.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.995 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 7", 2024-04-22T18:42:05.995 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704602+0000", 2024-04-22T18:42:05.995 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.8048999999999996e-05 2024-04-22T18:42:05.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.995 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:05.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.806370+0000", 2024-04-22T18:42:05.996 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10176768 2024-04-22T18:42:05.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.996 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.996 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:05.996 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204629+0000", 2024-04-22T18:42:05.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39825917999999999 2024-04-22T18:42:05.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.997 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.997 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:05.997 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204673+0000", 2024-04-22T18:42:05.997 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3516999999999997e-05 2024-04-22T18:42:05.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.998 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:05.998 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204710+0000", 2024-04-22T18:42:05.998 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7391000000000002e-05 2024-04-22T18:42:05.998 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.998 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:05.998 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:05.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:05.998 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:05.998 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3892 4.1 4:83c955bf:::smithi08937929-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e115)", 2024-04-22T18:42:05.999 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:42:05.999 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 76.275376652000006, 2024-04-22T18:42:05.999 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.59031281800000002, 2024-04-22T18:42:05.999 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:05.999 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:05.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:05.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:05.999 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:05.999 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3892 2024-04-22T18:42:06.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.000 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.000 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.000 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.000 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:42:06.000 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.001 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.001 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614551+0000", 2024-04-22T18:42:06.001 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.001 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614550+0000", 2024-04-22T18:42:06.002 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:06.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.002 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.002 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.002 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614553+0000", 2024-04-22T18:42:06.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0879999999999999e-06 2024-04-22T18:42:06.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614559+0000", 2024-04-22T18:42:06.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.9259999999999997e-06 2024-04-22T18:42:06.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.003 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.003 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.003 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.614565+0000", 2024-04-22T18:42:06.003 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7950000000000004e-06 2024-04-22T18:42:06.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704818+0000", 2024-04-22T18:42:06.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.090252624000000004 2024-04-22T18:42:06.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:06.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:49.704866+0000", 2024-04-22T18:42:06.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8667e-05 2024-04-22T18:42:06.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.004 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.004 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.004 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204742+0000", 2024-04-22T18:42:06.004 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.49987551899999999 2024-04-22T18:42:06.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204799+0000", 2024-04-22T18:42:06.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7299000000000001e-05 2024-04-22T18:42:06.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.005 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.005 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.005 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:50.204863+0000", 2024-04-22T18:42:06.005 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.4455000000000005e-05 2024-04-22T18:42:06.005 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.005 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.005 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.005 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.005 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.006 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:42:06.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-22T18:42:06.150 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:42:06.150 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:42:06.150 INFO:teuthology.orchestra.run.smithi121.stdout: "num_ops": 0 2024-04-22T18:42:06.150 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:42:06.159 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-22T18:42:06.260 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:42:06.260 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [], 2024-04-22T18:42:06.260 INFO:teuthology.orchestra.run.smithi121.stdout: "complaint_time": 30, 2024-04-22T18:42:06.260 INFO:teuthology.orchestra.run.smithi121.stdout: "num_blocked_ops": 0 2024-04-22T18:42:06.260 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:42:06.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-22T18:42:06.372 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-22T18:42:06.373 INFO:teuthology.orchestra.run.smithi121.stdout: "size": 20, 2024-04-22T18:42:06.373 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 600, 2024-04-22T18:42:06.373 INFO:teuthology.orchestra.run.smithi121.stdout: "ops": [ 2024-04-22T18:42:06.373 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.373 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:42:06.373 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:42:06.373 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 201.51808054, 2024-04-22T18:42:06.374 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.407731185, 2024-04-22T18:42:06.374 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.374 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:06.374 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.374 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:42:06.374 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:42:06.374 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 112 2024-04-22T18:42:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:42:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.375 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:42:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.376 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854226+0000", 2024-04-22T18:42:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.377 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854264+0000", 2024-04-22T18:42:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8346999999999997e-05 2024-04-22T18:42:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854269+0000", 2024-04-22T18:42:06.378 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.101e-06 2024-04-22T18:42:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854274+0000", 2024-04-22T18:42:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9690000000000002e-06 2024-04-22T18:42:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.379 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159704+0000", 2024-04-22T18:42:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.305430015 2024-04-22T18:42:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159801+0000", 2024-04-22T18:42:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.6775999999999996e-05 2024-04-22T18:42:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:42:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.159963+0000", 2024-04-22T18:42:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016158000000000001 2024-04-22T18:42:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.161040+0000", 2024-04-22T18:42:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0010771909999999999 2024-04-22T18:42:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261875+0000", 2024-04-22T18:42:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10083496 2024-04-22T18:42:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261920+0000", 2024-04-22T18:42:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.4805000000000002e-05 2024-04-22T18:42:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261957+0000", 2024-04-22T18:42:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7916000000000001e-05 2024-04-22T18:42:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:42:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:42:06.382 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 201.51802937799999, 2024-04-22T18:42:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40782637700000002, 2024-04-22T18:42:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:42:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:42:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 113 2024-04-22T18:42:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:42:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:42:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854277+0000", 2024-04-22T18:42:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854280+0000", 2024-04-22T18:42:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8389999999999998e-06 2024-04-22T18:42:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854283+0000", 2024-04-22T18:42:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.2109999999999998e-06 2024-04-22T18:42:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854286+0000", 2024-04-22T18:42:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0249999999999998e-06 2024-04-22T18:42:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160388+0000", 2024-04-22T18:42:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.30610178100000002 2024-04-22T18:42:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160479+0000", 2024-04-22T18:42:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1368999999999999e-05 2024-04-22T18:42:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:42:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.160615+0000", 2024-04-22T18:42:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013542900000000001 2024-04-22T18:42:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:06.386 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261979+0000", 2024-04-22T18:42:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10136474600000001 2024-04-22T18:42:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262030+0000", 2024-04-22T18:42:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.0691e-05 2024-04-22T18:42:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262060+0000", 2024-04-22T18:42:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.0142e-05 2024-04-22T18:42:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.387 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262104+0000", 2024-04-22T18:42:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3655000000000001e-05 2024-04-22T18:42:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:42:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:42:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 201.51801415400001, 2024-04-22T18:42:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50801962700000003, 2024-04-22T18:42:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 114 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854293+0000", 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.389 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854292+0000", 2024-04-22T18:42:06.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:06.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.390 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854296+0000", 2024-04-22T18:42:06.390 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1520000000000002e-06 2024-04-22T18:42:06.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.390 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.390 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854302+0000", 2024-04-22T18:42:06.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.13e-06 2024-04-22T18:42:06.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854316+0000", 2024-04-22T18:42:06.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.4212999999999999e-05 2024-04-22T18:42:06.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.391 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.391 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259473+0000", 2024-04-22T18:42:06.391 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40515696299999998 2024-04-22T18:42:06.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.391 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259706+0000", 2024-04-22T18:42:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00023342 2024-04-22T18:42:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:42:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.259870+0000", 2024-04-22T18:42:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00016372899999999999 2024-04-22T18:42:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261987+0000", 2024-04-22T18:42:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0021169600000000002 2024-04-22T18:42:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.392 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:06.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362158+0000", 2024-04-22T18:42:06.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10017116399999999 2024-04-22T18:42:06.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:06.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362249+0000", 2024-04-22T18:42:06.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.0629000000000003e-05 2024-04-22T18:42:06.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.393 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.393 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.393 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.362312+0000", 2024-04-22T18:42:06.393 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3639000000000005e-05 2024-04-22T18:42:06.393 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.393 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.394 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.394 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.394 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:42:06.394 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:42:06.394 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 201.51795457700001, 2024-04-22T18:42:06.394 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70817331100000003, 2024-04-22T18:42:06.394 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.394 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:06.394 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.394 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:42:06.394 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:42:06.394 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 115 2024-04-22T18:42:06.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.395 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:42:06.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:42:06.395 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.395 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.395 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.395 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854352+0000", 2024-04-22T18:42:06.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:06.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854421+0000", 2024-04-22T18:42:06.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.9097999999999995e-05 2024-04-22T18:42:06.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854426+0000", 2024-04-22T18:42:06.396 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.8080000000000001e-06 2024-04-22T18:42:06.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.396 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.396 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.396 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854431+0000", 2024-04-22T18:42:06.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:42:06.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261138+0000", 2024-04-22T18:42:06.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40670637100000001 2024-04-22T18:42:06.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.397 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.397 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.397 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261195+0000", 2024-04-22T18:42:06.397 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.7370000000000001e-05 2024-04-22T18:42:06.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:42:06.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.261293+0000", 2024-04-22T18:42:06.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.7697999999999999e-05 2024-04-22T18:42:06.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.398 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:06.398 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262661+0000", 2024-04-22T18:42:06.398 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0013679600000000001 2024-04-22T18:42:06.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.398 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:06.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562442+0000", 2024-04-22T18:42:06.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29978149599999998 2024-04-22T18:42:06.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:06.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562489+0000", 2024-04-22T18:42:06.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7126000000000002e-05 2024-04-22T18:42:06.399 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.399 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.399 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.399 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562526+0000", 2024-04-22T18:42:06.399 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.6349e-05 2024-04-22T18:42:06.399 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.399 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.400 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.400 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.400 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:42:06.400 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:42:06.400 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 201.51787226100001, 2024-04-22T18:42:06.400 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.70825728700000001, 2024-04-22T18:42:06.400 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.400 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:06.400 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.400 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:42:06.400 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:42:06.400 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 116 2024-04-22T18:42:06.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.400 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:42:06.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854435+0000", 2024-04-22T18:42:06.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.401 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.401 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.401 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854434+0000", 2024-04-22T18:42:06.401 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:06.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854437+0000", 2024-04-22T18:42:06.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6089999999999999e-06 2024-04-22T18:42:06.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854440+0000", 2024-04-22T18:42:06.402 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9840000000000001e-06 2024-04-22T18:42:06.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.402 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.402 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.402 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854442+0000", 2024-04-22T18:42:06.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.7970000000000001e-06 2024-04-22T18:42:06.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262151+0000", 2024-04-22T18:42:06.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40770863699999998 2024-04-22T18:42:06.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.403 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.403 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262234+0000", 2024-04-22T18:42:06.403 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.2729e-05 2024-04-22T18:42:06.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.403 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:42:06.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262362+0000", 2024-04-22T18:42:06.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012842299999999999 2024-04-22T18:42:06.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:06.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.264842+0000", 2024-04-22T18:42:06.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.002479382 2024-04-22T18:42:06.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.404 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:06.404 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562609+0000", 2024-04-22T18:42:06.404 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.29776706800000002 2024-04-22T18:42:06.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.404 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:06.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562650+0000", 2024-04-22T18:42:06.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.0874999999999997e-05 2024-04-22T18:42:06.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.405 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.405 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.562692+0000", 2024-04-22T18:42:06.405 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.2281999999999999e-05 2024-04-22T18:42:06.405 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.405 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.405 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.405 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.405 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4199.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-04-22T18:42:06.405 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:42:06.406 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 201.51786210500001, 2024-04-22T18:42:06.406 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.80855886799999999, 2024-04-22T18:42:06.406 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.406 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-22T18:42:06.406 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.406 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4199", 2024-04-22T18:42:06.406 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/4168002553", 2024-04-22T18:42:06.406 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 117 2024-04-22T18:42:06.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.406 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.406 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.406 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.406 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:42:06.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854445+0000", 2024-04-22T18:42:06.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854444+0000", 2024-04-22T18:42:06.407 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:06.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.407 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.407 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.407 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854446+0000", 2024-04-22T18:42:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8899999999999999e-06 2024-04-22T18:42:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854449+0000", 2024-04-22T18:42:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.128e-06 2024-04-22T18:42:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:44.854452+0000", 2024-04-22T18:42:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6570000000000001e-06 2024-04-22T18:42:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262678+0000", 2024-04-22T18:42:06.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.40822625800000001 2024-04-22T18:42:06.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262731+0000", 2024-04-22T18:42:06.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2404999999999997e-05 2024-04-22T18:42:06.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.409 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for subops from 0", 2024-04-22T18:42:06.409 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.262818+0000", 2024-04-22T18:42:06.409 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7522999999999995e-05 2024-04-22T18:42:06.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "op_commit", 2024-04-22T18:42:06.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.265176+0000", 2024-04-22T18:42:06.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.0023573819999999999 2024-04-22T18:42:06.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "sub_op_commit_rec", 2024-04-22T18:42:06.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662919+0000", 2024-04-22T18:42:06.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39774317300000001 2024-04-22T18:42:06.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.410 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "commit_sent", 2024-04-22T18:42:06.410 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.662956+0000", 2024-04-22T18:42:06.410 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7613000000000001e-05 2024-04-22T18:42:06.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.411 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.411 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:38:45.663004+0000", 2024-04-22T18:42:06.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7129999999999998e-05 2024-04-22T18:42:06.411 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.411 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.411 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.411 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.411 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3163 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:42:06.411 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:42:06.411 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.534849667, 2024-04-22T18:42:06.411 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1009801299999999, 2024-04-22T18:42:06.411 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3163 2024-04-22T18:42:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:42:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.412 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837457+0000", 2024-04-22T18:42:06.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.413 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.413 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837455+0000", 2024-04-22T18:42:06.413 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999976 2024-04-22T18:42:06.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837464+0000", 2024-04-22T18:42:06.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.5319999999999995e-06 2024-04-22T18:42:06.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.414 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.414 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837476+0000", 2024-04-22T18:42:06.414 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1759e-05 2024-04-22T18:42:06.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.414 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837488+0000", 2024-04-22T18:42:06.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1666e-05 2024-04-22T18:42:06.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837541+0000", 2024-04-22T18:42:06.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2939999999999998e-05 2024-04-22T18:42:06.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.415 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:06.415 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837547+0000", 2024-04-22T18:42:06.415 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6409999999999996e-06 2024-04-22T18:42:06.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937423+0000", 2024-04-22T18:42:06.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.099875545 2024-04-22T18:42:06.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.937567+0000", 2024-04-22T18:42:06.416 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00014363099999999999 2024-04-22T18:42:06.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.416 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.416 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938437+0000", 2024-04-22T18:42:06.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00087060399999999997 2024-04-22T18:42:06.417 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.417 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.417 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.417 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.417 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3164 4.8 4:187adac1:::smithi08937929-36:head [sparse-read 0~2643888] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:42:06.417 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:42:06.417 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.534810298, 2024-04-22T18:42:06.417 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1013670310000001, 2024-04-22T18:42:06.417 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.417 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:06.417 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.417 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:06.417 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:06.418 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3164 2024-04-22T18:42:06.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.418 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:42:06.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.418 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.418 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:42:06.418 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837496+0000", 2024-04-22T18:42:06.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:06.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.419 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.419 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837499+0000", 2024-04-22T18:42:06.419 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.994e-06 2024-04-22T18:42:06.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837503+0000", 2024-04-22T18:42:06.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7249999999999999e-06 2024-04-22T18:42:06.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.420 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.420 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837506+0000", 2024-04-22T18:42:06.420 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5980000000000001e-06 2024-04-22T18:42:06.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937641+0000", 2024-04-22T18:42:06.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.10013467700000001 2024-04-22T18:42:06.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:06.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937650+0000", 2024-04-22T18:42:06.421 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.3489999999999998e-06 2024-04-22T18:42:06.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.421 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.421 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938468+0000", 2024-04-22T18:42:06.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0008176710000001 2024-04-22T18:42:06.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938582+0000", 2024-04-22T18:42:06.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.000114287 2024-04-22T18:42:06.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.422 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.422 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.938864+0000", 2024-04-22T18:42:06.422 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00028120899999999998 2024-04-22T18:42:06.422 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.422 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.423 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.423 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3165 4.8 4:187adac1:::smithi08937929-36:head [read 0~3495856,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:42:06.423 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:42:06.423 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.53479153000001, 2024-04-22T18:42:06.423 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2022754499999999, 2024-04-22T18:42:06.423 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.423 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:06.423 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.423 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:06.423 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:06.423 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3165 2024-04-22T18:42:06.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.424 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:42:06.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:42:06.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.424 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.424 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837515+0000", 2024-04-22T18:42:06.424 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:06.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837517+0000", 2024-04-22T18:42:06.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.3369999999999998e-06 2024-04-22T18:42:06.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837523+0000", 2024-04-22T18:42:06.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5450000000000003e-06 2024-04-22T18:42:06.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.425 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.425 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.837526+0000", 2024-04-22T18:42:06.425 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.354e-06 2024-04-22T18:42:06.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937682+0000", 2024-04-22T18:42:06.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100155917 2024-04-22T18:42:06.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:06.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:21.937688+0000", 2024-04-22T18:42:06.426 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2230000000000004e-06 2024-04-22T18:42:06.426 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.426 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.426 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.038953+0000", 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012648110000001 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039034+0000", 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.0969000000000004e-05 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039791+0000", 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00075672900000000004 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.427 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.428 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3183 4.8 4:1bc4668a:::smithi08937929-21:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:42:06.428 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:42:06.428 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.336388332, 2024-04-22T18:42:06.428 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0040478260000001, 2024-04-22T18:42:06.428 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.428 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:42:06.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:06.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:06.428 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3183 2024-04-22T18:42:06.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.428 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:42:06.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035918+0000", 2024-04-22T18:42:06.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.429 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.429 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035913+0000", 2024-04-22T18:42:06.429 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999943 2024-04-22T18:42:06.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035935+0000", 2024-04-22T18:42:06.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.2232999999999999e-05 2024-04-22T18:42:06.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035969+0000", 2024-04-22T18:42:06.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3969e-05 2024-04-22T18:42:06.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.430 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.430 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.035987+0000", 2024-04-22T18:42:06.430 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8397e-05 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.2916000000000003e-05 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0103e-05 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039841+0000", 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0037508959999999 2024-04-22T18:42:06.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.432 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.432 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039966+0000", 2024-04-22T18:42:06.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012514500000000001 2024-04-22T18:42:06.432 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.432 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.432 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.432 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3184 4.8 4:12d9f826:::smithi08937929-1:head [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:42:06.432 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:42:06.432 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.336300373, 2024-04-22T18:42:06.432 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.00402868, 2024-04-22T18:42:06.432 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.433 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:42:06.433 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.433 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:06.433 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:06.433 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3184 2024-04-22T18:42:06.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.433 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:42:06.433 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.433 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.433 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036006+0000", 2024-04-22T18:42:06.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036005+0000", 2024-04-22T18:42:06.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:06.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.434 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.434 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036026+0000", 2024-04-22T18:42:06.434 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.1092999999999999e-05 2024-04-22T18:42:06.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036030+0000", 2024-04-22T18:42:06.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.467e-06 2024-04-22T18:42:06.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036033+0000", 2024-04-22T18:42:06.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9780000000000001e-06 2024-04-22T18:42:06.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.435 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.435 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036120+0000", 2024-04-22T18:42:06.435 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7495000000000002e-05 2024-04-22T18:42:06.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036125+0000", 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8720000000000001e-06 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.039982+0000", 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0038571919999999 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040035+0000", 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.2992999999999999e-05 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.436 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.437 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.437 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3185 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:42:06.437 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:42:06.437 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.336246461, 2024-04-22T18:42:06.437 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0041404039999999, 2024-04-22T18:42:06.437 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.437 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:42:06.437 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.437 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:06.437 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:06.437 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3185 2024-04-22T18:42:06.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.438 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:42:06.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:42:06.438 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.438 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.438 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036060+0000", 2024-04-22T18:42:06.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:06.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036062+0000", 2024-04-22T18:42:06.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.9099999999999999e-06 2024-04-22T18:42:06.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.439 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036066+0000", 2024-04-22T18:42:06.439 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.1699999999999999e-06 2024-04-22T18:42:06.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.439 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036077+0000", 2024-04-22T18:42:06.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0637e-05 2024-04-22T18:42:06.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036147+0000", 2024-04-22T18:42:06.440 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0025999999999997e-05 2024-04-22T18:42:06.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.440 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:06.440 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036151+0000", 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6859999999999997e-06 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040068+0000", 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.003916604 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.040201+0000", 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00013275400000000001 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.442 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3186 4.8 4:12d9f826:::smithi08937929-1:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-04-22T18:42:06.442 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:42:06.442 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 104.336226905, 2024-04-22T18:42:06.442 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2044167510000001, 2024-04-22T18:42:06.442 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.442 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "reached pg", 2024-04-22T18:42:06.442 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.442 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:06.442 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:06.442 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3186 2024-04-22T18:42:06.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.442 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.442 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.442 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.442 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:42:06.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036080+0000", 2024-04-22T18:42:06.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.443 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036079+0000", 2024-04-22T18:42:06.443 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:06.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.443 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.443 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036081+0000", 2024-04-22T18:42:06.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.6449999999999999e-06 2024-04-22T18:42:06.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036084+0000", 2024-04-22T18:42:06.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.9399999999999998e-06 2024-04-22T18:42:06.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.444 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036090+0000", 2024-04-22T18:42:06.444 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.879e-06 2024-04-22T18:42:06.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.444 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.444 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036168+0000", 2024-04-22T18:42:06.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.7657999999999994e-05 2024-04-22T18:42:06.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:06.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:22.036171+0000", 2024-04-22T18:42:06.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.7639999999999999e-06 2024-04-22T18:42:06.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.445 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240345+0000", 2024-04-22T18:42:06.445 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2041734260000001 2024-04-22T18:42:06.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.445 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.446 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:23.240497+0000", 2024-04-22T18:42:06.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015182800000000001 2024-04-22T18:42:06.446 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.446 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.446 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.446 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.446 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:3724 4.18 4:1bc4668a:::smithi08937929-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e113)", 2024-04-22T18:42:06.446 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:42:06.446 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 81.311902528999994, 2024-04-22T18:42:06.446 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.50413114699999995, 2024-04-22T18:42:06.446 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.446 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:06.446 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.446 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:06.447 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:06.447 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 3724 2024-04-22T18:42:06.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.447 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:42:06.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.447 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.447 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:42:06.447 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.447 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060404+0000", 2024-04-22T18:42:06.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:06.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060405+0000", 2024-04-22T18:42:06.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.395e-06 2024-04-22T18:42:06.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.448 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.448 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060408+0000", 2024-04-22T18:42:06.448 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.8629999999999999e-06 2024-04-22T18:42:06.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.448 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.060411+0000", 2024-04-22T18:42:06.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 2.6060000000000001e-06 2024-04-22T18:42:06.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165799+0000", 2024-04-22T18:42:06.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.105388369 2024-04-22T18:42:06.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.449 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for rw locks", 2024-04-22T18:42:06.449 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.165843+0000", 2024-04-22T18:42:06.449 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.3891999999999999e-05 2024-04-22T18:42:06.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.449 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564408+0000", 2024-04-22T18:42:06.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.39856466000000002 2024-04-22T18:42:06.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564461+0000", 2024-04-22T18:42:06.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.3529999999999997e-05 2024-04-22T18:42:06.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.450 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.450 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.450 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:40:45.564535+0000", 2024-04-22T18:42:06.450 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4147999999999995e-05 2024-04-22T18:42:06.450 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.451 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.451 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.451 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.451 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4413 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:42:06.451 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:42:06.451 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 48.714922485000002, 2024-04-22T18:42:06.451 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2025380569999999, 2024-04-22T18:42:06.451 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.451 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:06.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:06.451 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:06.451 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4413 2024-04-22T18:42:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:42:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657384+0000", 2024-04-22T18:42:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.452 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657377+0000", 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999933 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657393+0000", 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5296999999999999e-05 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657411+0000", 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.8105e-05 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657424+0000", 2024-04-22T18:42:06.453 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3030000000000001e-05 2024-04-22T18:42:06.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658313+0000", 2024-04-22T18:42:06.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00088943099999999999 2024-04-22T18:42:06.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:06.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658324+0000", 2024-04-22T18:42:06.454 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0496000000000001e-05 2024-04-22T18:42:06.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.454 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.454 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859560+0000", 2024-04-22T18:42:06.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2012358350000001 2024-04-22T18:42:06.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859648+0000", 2024-04-22T18:42:06.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.8744000000000003e-05 2024-04-22T18:42:06.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.455 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.455 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.859922+0000", 2024-04-22T18:42:06.455 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00027396499999999998 2024-04-22T18:42:06.455 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.455 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.455 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.456 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4414 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:42:06.456 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:42:06.456 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 48.714868160999998, 2024-04-22T18:42:06.456 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2030255030000001, 2024-04-22T18:42:06.456 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.456 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:06.456 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.456 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:06.456 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:06.456 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4414 2024-04-22T18:42:06.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.456 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:42:06.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657439+0000", 2024-04-22T18:42:06.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.457 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.457 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657437+0000", 2024-04-22T18:42:06.457 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:06.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657443+0000", 2024-04-22T18:42:06.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5899999999999998e-06 2024-04-22T18:42:06.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657450+0000", 2024-04-22T18:42:06.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4649999999999997e-06 2024-04-22T18:42:06.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.458 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.458 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657457+0000", 2024-04-22T18:42:06.458 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.6000000000000003e-06 2024-04-22T18:42:06.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658364+0000", 2024-04-22T18:42:06.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00090735999999999996 2024-04-22T18:42:06.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:06.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658373+0000", 2024-04-22T18:42:06.459 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.3250000000000008e-06 2024-04-22T18:42:06.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.459 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.459 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860258+0000", 2024-04-22T18:42:06.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2018852769999999 2024-04-22T18:42:06.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860308+0000", 2024-04-22T18:42:06.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.9938000000000002e-05 2024-04-22T18:42:06.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.460 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.460 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860464+0000", 2024-04-22T18:42:06.460 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00015622800000000001 2024-04-22T18:42:06.460 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.460 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.460 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4415 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e141)", 2024-04-22T18:42:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:42:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 48.714843836999997, 2024-04-22T18:42:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2038250989999999, 2024-04-22T18:42:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4415 2024-04-22T18:42:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:42:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657463+0000", 2024-04-22T18:42:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657462+0000", 2024-04-22T18:42:06.462 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999995 2024-04-22T18:42:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657467+0000", 2024-04-22T18:42:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.5399999999999997e-06 2024-04-22T18:42:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657473+0000", 2024-04-22T18:42:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.3949999999999998e-06 2024-04-22T18:42:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.657480+0000", 2024-04-22T18:42:06.463 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2489999999999998e-06 2024-04-22T18:42:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658394+0000", 2024-04-22T18:42:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00091403299999999997 2024-04-22T18:42:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.658398+0000", 2024-04-22T18:42:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7979999999999996e-06 2024-04-22T18:42:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860495+0000", 2024-04-22T18:42:06.464 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.202097081 2024-04-22T18:42:06.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.860543+0000", 2024-04-22T18:42:06.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.7907000000000001e-05 2024-04-22T18:42:06.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.465 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.465 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861288+0000", 2024-04-22T18:42:06.465 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00074466500000000004 2024-04-22T18:42:06.465 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.465 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.465 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.466 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4413 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:42:06.466 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:42:06.466 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 48.712127332000001, 2024-04-22T18:42:06.466 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.201491952, 2024-04-22T18:42:06.466 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.466 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:06.466 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.466 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:06.466 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:06.466 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4413 2024-04-22T18:42:06.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.466 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.466 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:42:06.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660179+0000", 2024-04-22T18:42:06.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.467 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660178+0000", 2024-04-22T18:42:06.467 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.9999986 2024-04-22T18:42:06.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.467 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.467 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660185+0000", 2024-04-22T18:42:06.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4379999999999998e-06 2024-04-22T18:42:06.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660201+0000", 2024-04-22T18:42:06.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.5252999999999999e-05 2024-04-22T18:42:06.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.468 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660210+0000", 2024-04-22T18:42:06.468 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 9.1069999999999994e-06 2024-04-22T18:42:06.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.468 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.468 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660270+0000", 2024-04-22T18:42:06.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.0739000000000002e-05 2024-04-22T18:42:06.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:06.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660281+0000", 2024-04-22T18:42:06.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.0664e-05 2024-04-22T18:42:06.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.469 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.469 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861304+0000", 2024-04-22T18:42:06.469 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2010225059999999 2024-04-22T18:42:06.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.469 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861342+0000", 2024-04-22T18:42:06.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.8736000000000003e-05 2024-04-22T18:42:06.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.470 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.470 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861671+0000", 2024-04-22T18:42:06.470 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00032915899999999998 2024-04-22T18:42:06.470 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.470 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.470 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.470 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.470 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4414 4.3 4:c0d4d677:::smithi08937929-17:head [sparse-read 0~1533080] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:42:06.471 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:42:06.471 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 48.712083821999997, 2024-04-22T18:42:06.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2014303930000001, 2024-04-22T18:42:06.471 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.471 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:06.471 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.471 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:06.471 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:06.471 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4414 2024-04-22T18:42:06.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.471 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.471 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.471 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.471 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:42:06.471 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660223+0000", 2024-04-22T18:42:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660222+0000", 2024-04-22T18:42:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:42:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660227+0000", 2024-04-22T18:42:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.6789999999999996e-06 2024-04-22T18:42:06.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660240+0000", 2024-04-22T18:42:06.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.3485e-05 2024-04-22T18:42:06.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660245+0000", 2024-04-22T18:42:06.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4.8269999999999997e-06 2024-04-22T18:42:06.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.473 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.473 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.473 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660307+0000", 2024-04-22T18:42:06.473 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 6.2392999999999997e-05 2024-04-22T18:42:06.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:06.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660316+0000", 2024-04-22T18:42:06.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.7790000000000001e-06 2024-04-22T18:42:06.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861493+0000", 2024-04-22T18:42:06.474 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.201177341 2024-04-22T18:42:06.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.474 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.474 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.474 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861529+0000", 2024-04-22T18:42:06.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.5978e-05 2024-04-22T18:42:06.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.475 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.475 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861653+0000", 2024-04-22T18:42:06.475 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00012391799999999999 2024-04-22T18:42:06.475 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.475 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.475 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.475 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.475 INFO:teuthology.orchestra.run.smithi121.stdout: "description": "osd_op(client.4502.0:4415 4.3 4:c0d4d677:::smithi08937929-17:head [read 0~2151362,checksum] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e142)", 2024-04-22T18:42:06.475 INFO:teuthology.orchestra.run.smithi121.stdout: "initiated_at": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:42:06.475 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 48.712038597999999, 2024-04-22T18:42:06.476 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.2019634779999999, 2024-04-22T18:42:06.476 INFO:teuthology.orchestra.run.smithi121.stdout: "type_data": { 2024-04-22T18:42:06.476 INFO:teuthology.orchestra.run.smithi121.stdout: "flag_point": "started", 2024-04-22T18:42:06.476 INFO:teuthology.orchestra.run.smithi121.stdout: "client_info": { 2024-04-22T18:42:06.476 INFO:teuthology.orchestra.run.smithi121.stdout: "client": "client.4502", 2024-04-22T18:42:06.476 INFO:teuthology.orchestra.run.smithi121.stdout: "client_addr": "v1:172.21.15.89:0/1443091552", 2024-04-22T18:42:06.476 INFO:teuthology.orchestra.run.smithi121.stdout: "tid": 4415 2024-04-22T18:42:06.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.476 INFO:teuthology.orchestra.run.smithi121.stdout: "events": [ 2024-04-22T18:42:06.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "initiated", 2024-04-22T18:42:06.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:42:06.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "throttled", 2024-04-22T18:42:06.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660268+0000", 2024-04-22T18:42:06.477 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0 2024-04-22T18:42:06.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.477 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.477 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "header_read", 2024-04-22T18:42:06.477 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660267+0000", 2024-04-22T18:42:06.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 4294967295.999999 2024-04-22T18:42:06.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "all_read", 2024-04-22T18:42:06.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660273+0000", 2024-04-22T18:42:06.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 5.5910000000000003e-06 2024-04-22T18:42:06.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "dispatched", 2024-04-22T18:42:06.478 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660280+0000", 2024-04-22T18:42:06.478 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.4000000000000003e-06 2024-04-22T18:42:06.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.478 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.478 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "queued_for_pg", 2024-04-22T18:42:06.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.660288+0000", 2024-04-22T18:42:06.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 8.1769999999999998e-06 2024-04-22T18:42:06.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.760401+0000", 2024-04-22T18:42:06.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.100112864 2024-04-22T18:42:06.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.479 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "waiting for peered", 2024-04-22T18:42:06.479 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:17.760408+0000", 2024-04-22T18:42:06.479 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 7.0299999999999996e-06 2024-04-22T18:42:06.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.479 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "reached_pg", 2024-04-22T18:42:06.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861682+0000", 2024-04-22T18:42:06.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 1.1012742520000001 2024-04-22T18:42:06.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "started", 2024-04-22T18:42:06.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.861716+0000", 2024-04-22T18:42:06.480 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 3.3674999999999999e-05 2024-04-22T18:42:06.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-22T18:42:06.480 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-04-22T18:42:06.480 INFO:teuthology.orchestra.run.smithi121.stdout: "event": "done", 2024-04-22T18:42:06.480 INFO:teuthology.orchestra.run.smithi121.stdout: "time": "2024-04-22T18:41:18.862232+0000", 2024-04-22T18:42:06.481 INFO:teuthology.orchestra.run.smithi121.stdout: "duration": 0.00051554300000000003 2024-04-22T18:42:06.481 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.481 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.481 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.481 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-22T18:42:06.481 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-22T18:42:06.481 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-22T18:42:06.809 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-22T18:42:07.810 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-04-22T18:42:07.824 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-22T18:42:07.899 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-22T18:42:07.899 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-04-22T18:42:07.899 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-04-22T18:42:08.220 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-22T18:42:08.220+0000 7f9b4bbc8640 -1 mon.a@0(leader).osd e191 definitely_dead 0 2024-04-22T18:42:08.812 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:42:08.812+0000 7f3e7be88640 -1 osd.0 191 handle_osd_map injecting map crc failure 2024-04-22T18:42:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-04-22T18:42:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:marked in osd.5. 2024-04-22T18:42:08.829 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-04-22T18:42:08.830 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-22T18:42:08.830 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-04-22T18:42:08.830 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 6 2024-04-22T18:42:09.159 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-04-22T18:42:09.160+0000 7f9b4bbc8640 -1 mon.a@0(leader).osd e192 definitely_dead 0 2024-04-22T18:42:09.823 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:42:09.823+0000 7f3e7be88640 -1 osd.0 192 handle_osd_map injecting map crc failure 2024-04-22T18:42:09.825 INFO:teuthology.orchestra.run.smithi089.stderr:marked in osd.6. 2024-04-22T18:42:09.841 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-04-22T18:42:09.841 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-22T18:42:09.954 INFO:teuthology.orchestra.run.smithi089.stdout:mon.a: {} 2024-04-22T18:42:09.955 INFO:teuthology.orchestra.run.smithi089.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-22T18:42:09.964 INFO:teuthology.orchestra.run.smithi089.stdout:mon.b: {} 2024-04-22T18:42:09.964 INFO:teuthology.orchestra.run.smithi089.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-22T18:42:09.972 INFO:teuthology.orchestra.run.smithi089.stdout:mon.c: {} 2024-04-22T18:42:09.972 INFO:teuthology.orchestra.run.smithi089.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-22T18:42:09.983 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-22T18:42:10.827 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.0 to 1 (10000) 2024-04-22T18:42:10.841 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-04-22T18:42:11.837 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-22T18:42:11.851 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-22T18:42:12.837 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:42:12.837+0000 7f8607f36640 -1 osd.3 194 handle_osd_map injecting map crc failure 2024-04-22T18:42:12.840 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.1 to 1 (10000) 2024-04-22T18:42:12.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-22T18:42:12.863 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-22T18:42:12.864+0000 7fa766d7d640 -1 osd.5 194 handle_osd_map injecting map crc failure 2024-04-22T18:42:13.845 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-22T18:42:13.861 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-22T18:42:14.851 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.2 to 1 (10000) 2024-04-22T18:42:14.866 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-04-22T18:42:15.856 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-22T18:42:15.871 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-22T18:42:16.663 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.3 to 1 (10000) 2024-04-22T18:42:16.676 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-04-22T18:42:17.861 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:42:17.861+0000 7f7f44bb3640 -1 osd.4 200 handle_osd_map injecting map crc failure 2024-04-22T18:42:17.862 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-22T18:42:17.878 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-22T18:42:18.874 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.4 to 1 (10000) 2024-04-22T18:42:18.889 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-04-22T18:42:19.876 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-22T18:42:19.891 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-22T18:42:19.963 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:42:19.964+0000 7f3cfee88640 -1 osd.6 202 handle_osd_map injecting map crc failure 2024-04-22T18:42:20.878 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:42:20.879+0000 7f8607f36640 -1 osd.3 203 handle_osd_map injecting map crc failure 2024-04-22T18:42:20.880 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.5 to 1 (10000) 2024-04-22T18:42:20.895 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-04-22T18:42:21.887 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-22T18:42:21.887+0000 7f7f44bb3640 -1 osd.4 204 handle_osd_map injecting map crc failure 2024-04-22T18:42:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-22T18:42:21.904 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-22T18:42:22.891 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.6 to 1 (10000) 2024-04-22T18:42:22.906 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-04-22T18:42:23.355 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-04-22T18:42:23.356+0000 7f309102d640 -1 osd.1 205 handle_osd_map injecting map crc failure 2024-04-22T18:42:23.894 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:42:23.895+0000 7f3e7be88640 -1 osd.0 206 handle_osd_map injecting map crc failure 2024-04-22T18:42:23.895 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-22T18:42:23.912 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-22T18:42:24.904 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.7 to 1 (10000) 2024-04-22T18:42:24.921 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-04-22T18:42:25.915 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-22T18:42:25.932 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-22T18:42:25.932 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-22T18:42:25.932 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-22T18:42:25.932 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-22T18:42:25.932 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-22T18:42:25.933 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T18:42:26.219 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:42:26.219 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":209,"fsid":"9e25744c-eff1-4a4a-8d47-0dda406b2c35","created":"2024-04-22T18:38:26.841687+0000","modified":"2024-04-22T18:42:25.909781+0000","last_up_change":"2024-04-22T18:38:35.971820+0000","last_in_change":"2024-04-22T18:42:09.816570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T18:38:36.991610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T18:38:37.622620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-04-22T18:38:46.038005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bbf33cfb-0f07-4df6-b90f-83e9edb039a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6800","nonce":31890}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6801","nonce":31890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6803","nonce":31890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6802","nonce":31890}]},"public_addr":"172.21.15.89:6800/31890","cluster_addr":"172.21.15.89:6801/31890","heartbeat_back_addr":"172.21.15.89:6803/31890","heartbeat_front_addr":"172.21.15.89:6802/31890","state":["exists","up"]},{"osd":1,"uuid":"30415a8e-9f47-416f-9fd0-716c2cb3f562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6804","nonce":31899}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6805","nonce":31899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6807","nonce":31899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6806","nonce":31899}]},"public_addr":"172.21.15.89:6804/31899","cluster_addr":"172.21.15.89:6805/31899","heartbeat_back_addr":"172.21.15.89:6807/31899","heartbeat_front_addr":"172.21.15.89:6806/31899","state":["exists","up"]},{"osd":2,"uuid":"b8da6b88-8355-4225-930d-bfed881089b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6812","nonce":31903}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6813","nonce":31903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6815","nonce":31903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6814","nonce":31903}]},"public_addr":"172.21.15.89:6812/31903","cluster_addr":"172.21.15.89:6813/31903","heartbeat_back_addr":"172.21.15.89:6815/31903","heartbeat_front_addr":"172.21.15.89:6814/31903","state":["exists","up"]},{"osd":3,"uuid":"3eaec913-9d7d-4b03-a28f-0c334e7c39f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6808","nonce":31913}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6809","nonce":31913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6811","nonce":31913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6810","nonce":31913}]},"public_addr":"172.21.15.89:6808/31913","cluster_addr":"172.21.15.89:6809/31913","heartbeat_back_addr":"172.21.15.89:6811/31913","heartbeat_front_addr":"172.21.15.89:6810/31913","state":["exists","up"]},{"osd":4,"uuid":"887acc3b-6f6b-49eb-a039-13050a2c7d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":31922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":31922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":31922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":31922}]},"public_addr":"172.21.15.121:6808/31922","cluster_addr":"172.21.15.121:6809/31922","heartbeat_back_addr":"172.21.15.121:6811/31922","heartbeat_front_addr":"172.21.15.121:6810/31922","state":["exists","up"]},{"osd":5,"uuid":"dcaddfd0-f8d1-4c28-bda2-d17eed606b60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":31921}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":31921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":31921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":31921}]},"public_addr":"172.21.15.121:6804/31921","cluster_addr":"172.21.15.121:6805/31921","heartbeat_back_addr":"172.21.15.121:6807/31921","heartbeat_front_addr":"172.21.15.121:6806/31921","state":["exists","up"]},{"osd":6,"uuid":"acf0527b-3020-4b0a-b64c-5068d21a5781","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":31920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":31920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":31920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":31920}]},"public_addr":"172.21.15.121:6800/31920","cluster_addr":"172.21.15.121:6801/31920","heartbeat_back_addr":"172.21.15.121:6803/31920","heartbeat_front_addr":"172.21.15.121:6802/31920","state":["exists","up"]},{"osd":7,"uuid":"305394bc-be3d-420b-88d7-d273670540b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":31939}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":31939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":31939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":31939}]},"public_addr":"172.21.15.121:6812/31939","cluster_addr":"172.21.15.121:6813/31939","heartbeat_back_addr":"172.21.15.121:6815/31939","heartbeat_front_addr":"172.21.15.121:6814/31939","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:42:11.065407+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:42:19.663078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:41:20.388547+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:41:13.858167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:42:16.196556+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:41:10.687865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:41:56.085558+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:42:05.294143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T18:42:26.232 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-22T18:42:26.232 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T18:42:26.256 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:42:26.257+0000 7f3cfee88640 -1 osd.6 208 handle_osd_map injecting map crc failure 2024-04-22T18:42:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:42:26.510 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":209,"fsid":"9e25744c-eff1-4a4a-8d47-0dda406b2c35","created":"2024-04-22T18:38:26.841687+0000","modified":"2024-04-22T18:42:25.909781+0000","last_up_change":"2024-04-22T18:38:35.971820+0000","last_in_change":"2024-04-22T18:42:09.816570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T18:38:36.991610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T18:38:37.622620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-04-22T18:38:46.038005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bbf33cfb-0f07-4df6-b90f-83e9edb039a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6800","nonce":31890}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6801","nonce":31890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6803","nonce":31890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6802","nonce":31890}]},"public_addr":"172.21.15.89:6800/31890","cluster_addr":"172.21.15.89:6801/31890","heartbeat_back_addr":"172.21.15.89:6803/31890","heartbeat_front_addr":"172.21.15.89:6802/31890","state":["exists","up"]},{"osd":1,"uuid":"30415a8e-9f47-416f-9fd0-716c2cb3f562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6804","nonce":31899}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6805","nonce":31899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6807","nonce":31899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6806","nonce":31899}]},"public_addr":"172.21.15.89:6804/31899","cluster_addr":"172.21.15.89:6805/31899","heartbeat_back_addr":"172.21.15.89:6807/31899","heartbeat_front_addr":"172.21.15.89:6806/31899","state":["exists","up"]},{"osd":2,"uuid":"b8da6b88-8355-4225-930d-bfed881089b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6812","nonce":31903}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6813","nonce":31903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6815","nonce":31903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6814","nonce":31903}]},"public_addr":"172.21.15.89:6812/31903","cluster_addr":"172.21.15.89:6813/31903","heartbeat_back_addr":"172.21.15.89:6815/31903","heartbeat_front_addr":"172.21.15.89:6814/31903","state":["exists","up"]},{"osd":3,"uuid":"3eaec913-9d7d-4b03-a28f-0c334e7c39f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6808","nonce":31913}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6809","nonce":31913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6811","nonce":31913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6810","nonce":31913}]},"public_addr":"172.21.15.89:6808/31913","cluster_addr":"172.21.15.89:6809/31913","heartbeat_back_addr":"172.21.15.89:6811/31913","heartbeat_front_addr":"172.21.15.89:6810/31913","state":["exists","up"]},{"osd":4,"uuid":"887acc3b-6f6b-49eb-a039-13050a2c7d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":31922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":31922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":31922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":31922}]},"public_addr":"172.21.15.121:6808/31922","cluster_addr":"172.21.15.121:6809/31922","heartbeat_back_addr":"172.21.15.121:6811/31922","heartbeat_front_addr":"172.21.15.121:6810/31922","state":["exists","up"]},{"osd":5,"uuid":"dcaddfd0-f8d1-4c28-bda2-d17eed606b60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":31921}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":31921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":31921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":31921}]},"public_addr":"172.21.15.121:6804/31921","cluster_addr":"172.21.15.121:6805/31921","heartbeat_back_addr":"172.21.15.121:6807/31921","heartbeat_front_addr":"172.21.15.121:6806/31921","state":["exists","up"]},{"osd":6,"uuid":"acf0527b-3020-4b0a-b64c-5068d21a5781","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":31920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":31920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":31920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":31920}]},"public_addr":"172.21.15.121:6800/31920","cluster_addr":"172.21.15.121:6801/31920","heartbeat_back_addr":"172.21.15.121:6803/31920","heartbeat_front_addr":"172.21.15.121:6802/31920","state":["exists","up"]},{"osd":7,"uuid":"305394bc-be3d-420b-88d7-d273670540b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":31939}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":31939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":31939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":31939}]},"public_addr":"172.21.15.121:6812/31939","cluster_addr":"172.21.15.121:6813/31939","heartbeat_back_addr":"172.21.15.121:6815/31939","heartbeat_front_addr":"172.21.15.121:6814/31939","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:42:11.065407+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:42:19.663078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:41:20.388547+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:41:13.858167+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:42:16.196556+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:41:10.687865+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:41:56.085558+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:42:05.294143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T18:42:26.523 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T18:42:26.523 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-22T18:42:26.524 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-22T18:42:26.524 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-22T18:42:26.525 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-22T18:42:26.525 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-22T18:42:26.525 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-22T18:42:26.526 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-22T18:42:26.673 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-04-22T18:42:26.674+0000 7f3e7be88640 -1 osd.0 209 handle_osd_map injecting map crc failure 2024-04-22T18:42:26.699 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:42:26.699+0000 7f3cfee88640 -1 osd.6 209 handle_osd_map injecting map crc failure 2024-04-22T18:42:26.699 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-04-22T18:42:26.700+0000 7f1e2dd22640 -1 osd.2 209 handle_osd_map injecting map crc failure 2024-04-22T18:42:26.761 INFO:teuthology.orchestra.run.smithi089.stdout:55834574897 2024-04-22T18:42:26.761 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T18:42:26.762 INFO:teuthology.orchestra.run.smithi089.stdout:55834574897 2024-04-22T18:42:26.763 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-22T18:42:26.766 INFO:teuthology.orchestra.run.smithi089.stdout:55834574896 2024-04-22T18:42:26.767 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-22T18:42:26.770 INFO:teuthology.orchestra.run.smithi089.stdout:55834574897 2024-04-22T18:42:26.770 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-22T18:42:26.772 INFO:teuthology.orchestra.run.smithi089.stdout:55834574897 2024-04-22T18:42:26.773 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-22T18:42:26.774 INFO:teuthology.orchestra.run.smithi089.stdout:55834574897 2024-04-22T18:42:26.775 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T18:42:26.784 INFO:teuthology.orchestra.run.smithi089.stdout:55834574896 2024-04-22T18:42:26.784 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T18:42:26.821 INFO:teuthology.orchestra.run.smithi089.stdout:55834574897 2024-04-22T18:42:26.821 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-22T18:42:27.340 INFO:teuthology.orchestra.run.smithi089.stdout:55834574895 2024-04-22T18:42:27.360 INFO:teuthology.orchestra.run.smithi089.stdout:55834574895 2024-04-22T18:42:27.361 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574895 for osd.5 2024-04-22T18:42:27.369 INFO:teuthology.orchestra.run.smithi089.stdout:55834574895 2024-04-22T18:42:27.378 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574895 for osd.2 2024-04-22T18:42:27.383 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574896 got 55834574895 for osd.3 2024-04-22T18:42:27.384 INFO:teuthology.orchestra.run.smithi089.stdout:55834574895 2024-04-22T18:42:27.387 INFO:teuthology.orchestra.run.smithi089.stdout:55834574895 2024-04-22T18:42:27.394 INFO:teuthology.orchestra.run.smithi089.stdout:55834574895 2024-04-22T18:42:27.399 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574895 for osd.0 2024-04-22T18:42:27.399 INFO:teuthology.orchestra.run.smithi089.stdout:55834574895 2024-04-22T18:42:27.407 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574895 for osd.6 2024-04-22T18:42:27.414 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574896 got 55834574895 for osd.1 2024-04-22T18:42:27.415 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574895 for osd.4 2024-04-22T18:42:27.434 INFO:teuthology.orchestra.run.smithi089.stdout:55834574895 2024-04-22T18:42:27.446 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574895 for osd.7 2024-04-22T18:42:28.362 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-22T18:42:28.379 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T18:42:28.383 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-22T18:42:28.399 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T18:42:28.408 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-22T18:42:28.415 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T18:42:28.416 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-22T18:42:28.449 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-22T18:42:28.909 INFO:teuthology.orchestra.run.smithi089.stdout:55834574897 2024-04-22T18:42:28.936 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574897 for osd.5 2024-04-22T18:42:28.936 DEBUG:teuthology.parallel:result is None 2024-04-22T18:42:28.958 INFO:teuthology.orchestra.run.smithi089.stdout:55834574897 2024-04-22T18:42:28.966 INFO:teuthology.orchestra.run.smithi089.stdout:55834574896 2024-04-22T18:42:28.972 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574897 for osd.0 2024-04-22T18:42:28.972 DEBUG:teuthology.parallel:result is None 2024-04-22T18:42:28.974 INFO:teuthology.orchestra.run.smithi089.stdout:55834574897 2024-04-22T18:42:28.987 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574896 got 55834574896 for osd.3 2024-04-22T18:42:28.987 DEBUG:teuthology.parallel:result is None 2024-04-22T18:42:28.989 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574897 for osd.2 2024-04-22T18:42:28.989 DEBUG:teuthology.parallel:result is None 2024-04-22T18:42:29.009 INFO:teuthology.orchestra.run.smithi089.stdout:55834574897 2024-04-22T18:42:29.011 INFO:teuthology.orchestra.run.smithi089.stdout:55834574897 2024-04-22T18:42:29.024 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574897 for osd.6 2024-04-22T18:42:29.024 DEBUG:teuthology.parallel:result is None 2024-04-22T18:42:29.025 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574897 for osd.4 2024-04-22T18:42:29.025 DEBUG:teuthology.parallel:result is None 2024-04-22T18:42:29.048 INFO:teuthology.orchestra.run.smithi089.stdout:55834574897 2024-04-22T18:42:29.054 INFO:teuthology.orchestra.run.smithi089.stdout:55834574896 2024-04-22T18:42:29.062 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574897 for osd.7 2024-04-22T18:42:29.062 DEBUG:teuthology.parallel:result is None 2024-04-22T18:42:29.067 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574896 got 55834574896 for osd.1 2024-04-22T18:42:29.067 DEBUG:teuthology.parallel:result is None 2024-04-22T18:42:29.067 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-22T18:42:29.067 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:42:29.265 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-22T18:42:29.266+0000 7f3cfee88640 -1 osd.6 210 handle_osd_map injecting map crc failure 2024-04-22T18:42:29.387 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:42:29.387 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:42:29.399 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-04-22T18:42:29.002414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":156248488,"num_objects":224,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1404,"num_read_kb":30872,"num_write":1881,"num_write_kb":393766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1298,"num_bytes_recovered":243128123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1528,"ondisk_log_size":1528,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":502328,"kb_used_data":232908,"kb_used_omap":12,"kb_used_meta":269299,"kb_avail":754472392,"statfs":{"total":773094113280,"available":772579729408,"internally_reserved":0,"allocated":238497792,"data_stored":245878802,"data_compressed":13126867,"data_compressed_allocated":13996032,"data_compressed_original":28819456,"omap_allocated":12744,"internal_metadata":275762744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-28,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":8818863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.098522"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":35,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.260024+0000","last_change":"2024-04-22T18:42:11.391813+0000","last_active":"2024-04-22T18:42:26.260024+0000","last_peered":"2024-04-22T18:42:26.260024+0000","last_clean":"2024-04-22T18:42:26.260024+0000","last_became_active":"2024-04-22T18:42:11.030639+0000","last_became_peered":"2024-04-22T18:42:11.030639+0000","last_unstale":"2024-04-22T18:42:26.260024+0000","last_undegraded":"2024-04-22T18:42:26.260024+0000","last_fullsized":"2024-04-22T18:42:26.260024+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.391633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.391633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:18.616280+0000","scrub_duration":0.102099431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":239,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.258799+0000","last_change":"2024-04-22T18:42:11.029787+0000","last_active":"2024-04-22T18:42:26.258799+0000","last_peered":"2024-04-22T18:42:26.258799+0000","last_clean":"2024-04-22T18:42:26.258799+0000","last_became_active":"2024-04-22T18:42:11.029453+0000","last_became_peered":"2024-04-22T18:42:11.029453+0000","last_unstale":"2024-04-22T18:42:26.258799+0000","last_undegraded":"2024-04-22T18:42:26.258799+0000","last_fullsized":"2024-04-22T18:42:26.258799+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:58.561808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:58.561808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:02.874625+0000","scrub_duration":0.301876478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":245,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.259270+0000","last_change":"2024-04-22T18:42:11.428432+0000","last_active":"2024-04-22T18:42:26.259270+0000","last_peered":"2024-04-22T18:42:26.259270+0000","last_clean":"2024-04-22T18:42:26.259270+0000","last_became_active":"2024-04-22T18:41:26.961835+0000","last_became_peered":"2024-04-22T18:41:26.961835+0000","last_unstale":"2024-04-22T18:42:26.259270+0000","last_undegraded":"2024-04-22T18:42:26.259270+0000","last_fullsized":"2024-04-22T18:42:26.259270+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.428262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.428262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:11.658873+0000","scrub_duration":0.30251553399999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":229,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.258805+0000","last_change":"2024-04-22T18:42:12.520547+0000","last_active":"2024-04-22T18:42:26.258805+0000","last_peered":"2024-04-22T18:42:26.258805+0000","last_clean":"2024-04-22T18:42:26.258805+0000","last_became_active":"2024-04-22T18:41:26.854399+0000","last_became_peered":"2024-04-22T18:41:26.854399+0000","last_unstale":"2024-04-22T18:42:26.258805+0000","last_undegraded":"2024-04-22T18:42:26.258805+0000","last_fullsized":"2024-04-22T18:42:26.258805+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:12.520456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:12.520456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:42.305803+0000","scrub_duration":0.20146086999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"169'384","reported_seq":1626,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-04-22T18:42:25.136623+0000","last_change":"2024-04-22T18:42:17.393680+0000","last_active":"2024-04-22T18:42:25.136623+0000","last_peered":"2024-04-22T18:42:25.136623+0000","last_clean":"2024-04-22T18:42:25.136623+0000","last_became_active":"2024-04-22T18:41:26.994048+0000","last_became_peered":"2024-04-22T18:41:26.994048+0000","last_unstale":"2024-04-22T18:42:25.136623+0000","last_undegraded":"2024-04-22T18:42:25.136623+0000","last_fullsized":"2024-04-22T18:42:25.136623+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'384","last_scrub_stamp":"2024-04-22T18:42:17.393461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.393461+0000","objects_scrubbed":54,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:29.216372+0000","scrub_duration":0.111477038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41123046,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":358,"num_read_kb":4228,"num_write":477,"num_write_kb":113967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":82068207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.219113+0000","last_change":"2024-04-22T18:42:09.930934+0000","last_active":"2024-04-22T18:42:26.219113+0000","last_peered":"2024-04-22T18:42:26.219113+0000","last_clean":"2024-04-22T18:42:26.219113+0000","last_became_active":"2024-04-22T18:42:09.929115+0000","last_became_peered":"2024-04-22T18:42:09.929115+0000","last_unstale":"2024-04-22T18:42:26.219113+0000","last_undegraded":"2024-04-22T18:42:26.219113+0000","last_fullsized":"2024-04-22T18:42:26.219113+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:55.558446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.558446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.56126016300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"200'362","reported_seq":846,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.358870+0000","last_change":"2024-04-22T18:42:17.640545+0000","last_active":"2024-04-22T18:42:26.358870+0000","last_peered":"2024-04-22T18:42:26.358870+0000","last_clean":"2024-04-22T18:42:26.358870+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:42:26.358870+0000","last_undegraded":"2024-04-22T18:42:26.358870+0000","last_fullsized":"2024-04-22T18:42:26.358870+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'362","last_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_deep_scrub":"200'362","last_deep_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.640441+0000","objects_scrubbed":53,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:47.115778+0000","scrub_duration":3.4448620060000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382624,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":306,"num_read_kb":4987,"num_write":404,"num_write_kb":80545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":176,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.358947+0000","last_change":"2024-04-22T18:41:41.421087+0000","last_active":"2024-04-22T18:42:26.358947+0000","last_peered":"2024-04-22T18:42:26.358947+0000","last_clean":"2024-04-22T18:42:26.358947+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:42:26.358947+0000","last_undegraded":"2024-04-22T18:42:26.358947+0000","last_fullsized":"2024-04-22T18:42:26.358947+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:41.420996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:41.420996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:57.844851+0000","scrub_duration":0.20141637900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"168'337","reported_seq":1427,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:25.917451+0000","last_change":"2024-04-22T18:42:09.682366+0000","last_active":"2024-04-22T18:42:25.917451+0000","last_peered":"2024-04-22T18:42:25.917451+0000","last_clean":"2024-04-22T18:42:25.917451+0000","last_became_active":"2024-04-22T18:41:41.007340+0000","last_became_peered":"2024-04-22T18:41:41.007340+0000","last_unstale":"2024-04-22T18:42:25.917451+0000","last_undegraded":"2024-04-22T18:42:25.917451+0000","last_fullsized":"2024-04-22T18:42:25.917451+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'337","last_scrub_stamp":"2024-04-22T18:42:09.682178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:09.682178+0000","objects_scrubbed":54,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:19.217879+0000","scrub_duration":0.20999784699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39035424,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":322,"num_read_kb":9140,"num_write":423,"num_write_kb":104979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":67954083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":220,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.359433+0000","last_change":"2024-04-22T18:42:13.608651+0000","last_active":"2024-04-22T18:42:26.359433+0000","last_peered":"2024-04-22T18:42:26.359433+0000","last_clean":"2024-04-22T18:42:26.359433+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:42:26.359433+0000","last_undegraded":"2024-04-22T18:42:26.359433+0000","last_fullsized":"2024-04-22T18:42:26.359433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_clean_scrub_stamp":"2024-04-22T18:42:13.608479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:21.626020+0000","scrub_duration":0.43950688900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"169'328","reported_seq":1326,"reported_epoch":210,"state":"active+recovering","last_fresh":"2024-04-22T18:42:27.005294+0000","last_change":"2024-04-22T18:42:19.385009+0000","last_active":"2024-04-22T18:42:27.005294+0000","last_peered":"2024-04-22T18:42:27.005294+0000","last_clean":"2024-04-22T18:41:44.719816+0000","last_became_active":"2024-04-22T18:42:18.879845+0000","last_became_peered":"2024-04-22T18:42:18.879845+0000","last_unstale":"2024-04-22T18:42:27.005294+0000","last_undegraded":"2024-04-22T18:42:27.005294+0000","last_fullsized":"2024-04-22T18:42:27.005294+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'328","last_scrub_stamp":"2024-04-22T18:42:06.716200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:06.716200+0000","objects_scrubbed":46,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:35.844741+0000","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43117026,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":302,"num_read_kb":12417,"num_write":418,"num_write_kb":91859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":367,"num_bytes_recovered":70659725,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":4},{"shards":"3,5","objects":57}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.118901+0000","last_change":"2024-04-22T18:42:10.136809+0000","last_active":"2024-04-22T18:42:26.118901+0000","last_peered":"2024-04-22T18:42:26.118901+0000","last_clean":"2024-04-22T18:42:26.118901+0000","last_became_active":"2024-04-22T18:42:10.036239+0000","last_became_peered":"2024-04-22T18:42:10.036239+0000","last_unstale":"2024-04-22T18:42:26.118901+0000","last_undegraded":"2024-04-22T18:42:26.118901+0000","last_fullsized":"2024-04-22T18:42:26.118901+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:30.504686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:30.504686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:45.796059+0000","scrub_duration":0.30171718199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":415,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.259297+0000","last_change":"2024-04-22T18:41:56.270751+0000","last_active":"2024-04-22T18:42:26.259297+0000","last_peered":"2024-04-22T18:42:26.259297+0000","last_clean":"2024-04-22T18:42:26.259297+0000","last_became_active":"2024-04-22T18:41:26.757958+0000","last_became_peered":"2024-04-22T18:41:26.757958+0000","last_unstale":"2024-04-22T18:42:26.259297+0000","last_undegraded":"2024-04-22T18:42:26.259297+0000","last_fullsized":"2024-04-22T18:42:26.259297+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:41:56.270650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:41:56.270650+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:12.138524+0000","scrub_duration":0.202387964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":155658120,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":1288,"num_read_kb":30772,"num_write":1722,"num_write_kb":391350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1292,"num_bytes_recovered":241541339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274956288,"data_stored":290746422,"data_compressed":16015051,"data_compressed_allocated":17043456,"data_compressed_original":33771520,"omap_allocated":0,"internal_metadata":0},"log_size":1411,"ondisk_log_size":1411,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":2115712,"data_compressed":122550,"data_compressed_allocated":163840,"data_compressed_original":2084864,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574897,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1585152,"data_stored":1228441,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":2.5870000000000002,"15min":1.1439999999999999},"min":{"1min":0.182,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":26.568000000000001,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":6.3600000000000003,"5min":6.766,"15min":5.2599999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":95.918999999999997,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.41499999999999998}]},{"osd":1,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":3.8050000000000002,"5min":3.984,"15min":4.5709999999999997},"min":{"1min":0.29299999999999998,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":66.814999999999998,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.33100000000000002},{"interface":"front","average":{"1min":6.6040000000000001,"5min":12.141,"15min":12.744999999999999},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":100.242,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7130000000000001,"5min":4.0730000000000004,"15min":4.0810000000000004},"min":{"1min":0.27400000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":97.662000000000006,"5min":97.662000000000006,"15min":97.662000000000006},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.48599999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7560000000000002,"5min":4.1289999999999996,"15min":4.1429999999999998},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":97.686999999999998,"5min":97.686999999999998,"15min":97.686999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.36499999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":8.2089999999999996,"5min":5.4509999999999996,"15min":4.7889999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":78.149000000000001,"5min":85.509,"15min":85.509},"last":0.31},{"interface":"front","average":{"1min":7.7130000000000001,"5min":16.856000000000002,"15min":20.866},"min":{"1min":0.26000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":97.590000000000003,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.47499999999999998}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":2.0270000000000001,"15min":2.7229999999999999},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.4730000000000001,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.6679999999999999,"5min":5.2939999999999996,"15min":6.5389999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":23.007999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.55300000000000005}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":11.756,"15min":12.769},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":23.027000000000001,"5min":139.428,"15min":139.428},"last":0.503},{"interface":"front","average":{"1min":7.7800000000000002,"5min":4.3070000000000004,"15min":4.1909999999999998},"min":{"1min":0.29199999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":97.635999999999996,"5min":97.635999999999996,"15min":97.635999999999996},"last":0.51700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1503232,"data_stored":698051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.5030000000000001,"5min":3.0579999999999998,"15min":2.2149999999999999},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":70.436000000000007,"5min":70.436000000000007,"15min":70.436000000000007},"last":0.307},{"interface":"front","average":{"1min":0.41599999999999998,"5min":16.753,"15min":22.789000000000001},"min":{"1min":0.26700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.58599999999999997,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":3.577,"15min":2.5510000000000002},"min":{"1min":0.254,"5min":0.215,"15min":0.215},"max":{"1min":0.54000000000000004,"5min":199.047,"15min":199.047},"last":0.38500000000000001},{"interface":"front","average":{"1min":11.327999999999999,"5min":8.6519999999999992,"15min":9.6120000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":192.279,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.68799999999999994,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":12.56,"15min":16.722000000000001},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.59099999999999997,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.41199999999999998}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":14.815,"15min":19.283999999999999},"min":{"1min":0.29599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65200000000000002,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":1.335,"15min":0.74299999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.875,"5min":77.097999999999999,"15min":77.097999999999999},"last":0.44400000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":2.052,"5min":5.0499999999999998,"15min":6.7519999999999998},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":29.978999999999999,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":10.577,"5min":16.312000000000001,"15min":18.321999999999999},"min":{"1min":0.27900000000000003,"5min":0.218,"15min":0.218},"max":{"1min":168.44,"5min":168.44,"15min":168.44},"last":0.53700000000000003}]},{"osd":5,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.252,"15min":0.252},"max":{"1min":0.61199999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.28199999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.51200000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":5.6289999999999996,"5min":14.432,"15min":17.378},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":90.649000000000001,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.165,"5min":0.60099999999999998,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.077,"5min":13.077,"15min":13.077},"last":0.496}]}]},{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56432,"kb_used_data":25316,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94315408,"statfs":{"total":96636764160,"available":96578977792,"internally_reserved":0,"allocated":25923584,"data_stored":26446986,"data_compressed":1330791,"data_compressed_allocated":1413120,"data_compressed_original":2805760,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3580000000000001,"5min":9.875,"15min":12.218999999999999},"min":{"1min":0.34999999999999998,"5min":0.192,"15min":0.192},"max":{"1min":58.421999999999997,"5min":74.019999999999996,"15min":74.019999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.438,"5min":0.71199999999999997,"15min":0.84099999999999997},"min":{"1min":0.34000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.52400000000000002,"5min":8.3919999999999995,"15min":8.3919999999999995},"last":0.52900000000000003}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.0259999999999998,"5min":11.034000000000001,"15min":10.702999999999999},"min":{"1min":0.28100000000000003,"5min":0.224,"15min":0.224},"max":{"1min":91.911000000000001,"5min":202.13800000000001,"15min":202.13800000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.3440000000000001,"15min":0.72699999999999998},"min":{"1min":0.34399999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.669,"5min":68.346999999999994,"15min":68.346999999999994},"last":0.61799999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.496,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":3.339,"15min":4.4720000000000004},"min":{"1min":0.377,"5min":0.308,"15min":0.308},"max":{"1min":0.57399999999999995,"5min":78.296999999999997,"15min":78.296999999999997},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":4.5499999999999998,"15min":5.3490000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.018,"5min":89.688000000000002,"15min":89.688000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":6.5380000000000003,"15min":9.0890000000000004},"min":{"1min":0.34899999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60899999999999999,"5min":89.718999999999994,"15min":89.718999999999994},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.4809999999999999,"5min":15.465999999999999,"15min":16.969000000000001},"min":{"1min":0.249,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":100.544,"5min":202.983,"15min":202.983},"last":0.40100000000000002},{"interface":"front","average":{"1min":9.5050000000000008,"5min":14.603999999999999,"15min":16.722999999999999},"min":{"1min":0.314,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":100.56,"5min":202.999,"15min":202.999},"last":0.48699999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":4.3739999999999997,"15min":4.8159999999999998},"min":{"1min":0.27100000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.59099999999999997,"5min":79.945999999999998,"15min":79.945999999999998},"last":0.378},{"interface":"front","average":{"1min":0.40799999999999997,"5min":5.274,"15min":5.9770000000000003},"min":{"1min":0.251,"5min":0.249,"15min":0.249},"max":{"1min":0.57299999999999995,"5min":100.54900000000001,"15min":100.54900000000001},"last":0.435}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":1.149,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":71.602999999999994,"5min":71.602999999999994,"15min":71.602999999999994},"last":0.41799999999999998},{"interface":"front","average":{"1min":4.508,"5min":4.173,"15min":4.7770000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":75.436000000000007,"5min":78.340999999999994,"15min":78.340999999999994},"last":0.45000000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116212,"kb_used_data":81000,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94255628,"statfs":{"total":96636764160,"available":96517763072,"internally_reserved":0,"allocated":82944000,"data_stored":86498190,"data_compressed":4527904,"data_compressed_allocated":4829184,"data_compressed_original":9498624,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":13.115,"15min":15.470000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":38.899999999999999,"5min":252.96199999999999,"15min":252.96199999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":9.282,"15min":8.8059999999999992},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63300000000000001,"5min":187.148,"15min":187.148},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":2.508,"15min":1.115},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":9.141,"5min":96.486999999999995,"15min":96.486999999999995},"last":0.41599999999999998},{"interface":"front","average":{"1min":22.393000000000001,"5min":11.545999999999999,"15min":7.4390000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":310.61500000000001,"5min":310.61500000000001,"15min":310.61500000000001},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":8.8989999999999991,"15min":8.6790000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.8330000000000002,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.995,"5min":2.4199999999999999,"15min":1.1040000000000001},"min":{"1min":0.31900000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.1399999999999997,"5min":96.471999999999994,"15min":96.471999999999994},"last":0.61799999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":11.172000000000001,"5min":8.9710000000000001,"15min":5.6710000000000003},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":122.102,"5min":252.91900000000001,"15min":252.91900000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":17.242000000000001,"15min":19.725999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59899999999999998,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.745}]},{"osd":5,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":2.3380000000000001,"15min":1.034},"min":{"1min":0.25800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":7.1559999999999997,"5min":96.408000000000001,"15min":96.408000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":2.6960000000000002,"5min":3.048,"15min":3.7080000000000002},"min":{"1min":0.36199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":38.581000000000003,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.69399999999999995}]},{"osd":6,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":5.8289999999999997,"5min":14.667,"15min":13.662000000000001},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":87.706999999999994,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.83199999999999996,"5min":2.3820000000000001,"15min":1.0880000000000001},"min":{"1min":0.35699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.1159999999999997,"5min":96.438000000000002,"15min":96.438000000000002},"last":0.76800000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":9.5749999999999993,"15min":8.8979999999999997},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68000000000000005,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.9790000000000001,"5min":2.6419999999999999,"15min":1.202},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":19.518999999999998,"5min":96.456999999999994,"15min":96.456999999999994},"last":0.72299999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574897,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63312,"kb_used_data":28096,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94308528,"statfs":{"total":96636764160,"available":96571932672,"internally_reserved":0,"allocated":28770304,"data_stored":29401156,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":15.041,"5min":10.538,"15min":11.646000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":93.131,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":7.077,"5min":1.6659999999999999,"15min":0.77400000000000002},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":87.305999999999997,"5min":87.305999999999997,"15min":87.305999999999997},"last":0.28000000000000003}]},{"osd":1,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":10.558,"15min":6.8819999999999997},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":78.805000000000007,"5min":197.02099999999999,"15min":197.02099999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":8.9800000000000004,"5min":2.3210000000000002,"15min":0.997},"min":{"1min":0.20999999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":78.393000000000001,"5min":78.393000000000001,"15min":78.393000000000001},"last":0.26700000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":2.3439999999999999,"15min":1.452},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":47.534999999999997,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.37},{"interface":"front","average":{"1min":6.4770000000000003,"5min":1.728,"15min":0.85899999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":78.421999999999997,"5min":78.421999999999997,"15min":78.421999999999997},"last":0.34999999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1800000000000002,"5min":2.9940000000000002,"15min":3.7360000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":26.141999999999999,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.59199999999999997},{"interface":"front","average":{"1min":4.5629999999999997,"5min":4.6289999999999996,"15min":4.3010000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":47.616999999999997,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.66200000000000003}]},{"osd":5,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.8279999999999998,"5min":2.5049999999999999,"15min":1.623},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":47.566000000000003,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.61799999999999999},{"interface":"front","average":{"1min":6.6379999999999999,"5min":2.2890000000000001,"15min":1.601},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.480999999999995,"5min":78.480999999999995,"15min":78.480999999999995},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.50900000000000001,"15min":0.53100000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.748,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":2.8500000000000001,"5min":2.871,"15min":1.3109999999999999},"min":{"1min":0.377,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":47.590000000000003,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":6.6189999999999998,"5min":1.903,"15min":1.0229999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.453999999999994,"5min":78.453999999999994,"15min":78.453999999999994},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69299999999999995,"5min":0.995,"15min":0.995},"last":0.49399999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51588,"kb_used_data":20472,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94320252,"statfs":{"total":96636764160,"available":96583938048,"internally_reserved":0,"allocated":20963328,"data_stored":21629172,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":6.7439999999999998,"15min":7.6660000000000004},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":84.338999999999999,"5min":149.03,"15min":149.03},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.28299999999999997,"5min":1.3089999999999999,"15min":1.7589999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.62,"5min":31.866,"15min":31.866},"last":0.42899999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":9.4849999999999994,"5min":5.2489999999999997,"15min":2.4380000000000002},"min":{"1min":0.216,"5min":0.193,"15min":0.193},"max":{"1min":91.912000000000006,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":12.752000000000001,"5min":15.912000000000001,"15min":15.227},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":126.42,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.53300000000000003}]},{"osd":3,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":6.2830000000000004,"15min":2.3359999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":129.041,"5min":147.721,"15min":147.721},"last":1.179},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":101.43300000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":8.1280000000000001,"5min":13.57,"15min":17.038},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":90.882999999999996,"5min":160.584,"15min":160.584},"last":0.50700000000000001},{"interface":"front","average":{"1min":5.9349999999999996,"5min":4.8520000000000003,"15min":3.2490000000000001},"min":{"1min":0.32500000000000001,"5min":0.315,"15min":0.315},"max":{"1min":93.962999999999994,"5min":97.887,"15min":97.887},"last":0.68100000000000005}]},{"osd":5,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":2.0339999999999998,"15min":2.7120000000000002},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.879,"5min":48.631,"15min":48.631},"last":0.624},{"interface":"front","average":{"1min":0.50900000000000001,"5min":2.2890000000000001,"15min":3.0800000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.91800000000000004,"5min":56.18,"15min":56.18},"last":0.56599999999999995}]},{"osd":6,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.3540000000000001,"5min":3.4670000000000001,"15min":1.5129999999999999},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":92.028000000000006,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59799999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.29,"5min":1.5800000000000001,"15min":0.91000000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":90.944999999999993,"5min":90.944999999999993,"15min":90.944999999999993},"last":0.47599999999999998},{"interface":"front","average":{"1min":7.4640000000000004,"5min":3.923,"15min":1.6970000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":92.048000000000002,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.72899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59852,"kb_used_data":20536,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94311988,"statfs":{"total":96636764160,"available":96575475712,"internally_reserved":0,"allocated":21028864,"data_stored":21644126,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":5.883,"15min":7.2000000000000002},"min":{"1min":0.20000000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":27.116,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.83899999999999997},{"interface":"front","average":{"1min":5.8040000000000003,"5min":6.6870000000000003,"15min":7.4710000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":94.046000000000006,"5min":94.046000000000006,"15min":94.046000000000006},"last":0.64400000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.679000000000002,"5min":13.159000000000001,"15min":10.119},"min":{"1min":0.23000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":406.68799999999999,"5min":406.68799999999999,"15min":406.68799999999999},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.8700000000000001,"5min":0.61899999999999999,"15min":0.41099999999999998},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19},"max":{"1min":27.094999999999999,"5min":27.094999999999999,"15min":27.094999999999999},"last":0.90900000000000003}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":15.968,"5min":3.464,"15min":1.3759999999999999},"min":{"1min":0.29099999999999998,"5min":0.216,"15min":0.216},"max":{"1min":228.90100000000001,"5min":228.90100000000001,"15min":228.90100000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":21.481999999999999,"5min":13.061999999999999,"15min":14.015000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":196.82900000000001,"5min":231.864,"15min":231.864},"last":0.69099999999999995}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.823,"5min":17.213000000000001,"15min":14.393000000000001},"min":{"1min":0.316,"5min":0.309,"15min":0.309},"max":{"1min":406.71199999999999,"5min":406.71199999999999,"15min":406.71199999999999},"last":1.228},{"interface":"front","average":{"1min":25.027000000000001,"5min":5.3979999999999997,"15min":2.1320000000000001},"min":{"1min":0.34499999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":186.89500000000001,"5min":186.89500000000001,"15min":186.89500000000001},"last":0.80800000000000005}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":9.7430000000000003,"15min":12.628},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":94.183000000000007,"5min":138.089,"15min":138.089},"last":0.60899999999999999},{"interface":"front","average":{"1min":39.863999999999997,"5min":20.117000000000001,"15min":20.09},"min":{"1min":0.35499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":406.72899999999998,"5min":406.72899999999998,"15min":406.72899999999998},"last":1.198}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":6.0010000000000003,"5min":3.8359999999999999,"15min":3.1299999999999999},"min":{"1min":0.34000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":94.302999999999997,"5min":94.302999999999997,"15min":94.302999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":43.036000000000001,"5min":19.713999999999999,"15min":17.591999999999999},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":427.69299999999998,"5min":427.69299999999998,"15min":427.69299999999998},"last":1.2549999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":2.0720000000000001,"5min":0.82799999999999996,"15min":0.626},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":27.253,"5min":27.253,"15min":27.253},"last":0.77900000000000003},{"interface":"front","average":{"1min":6.0090000000000003,"5min":3.129,"15min":3.069},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":94.260999999999996,"5min":94.260999999999996,"15min":94.260999999999996},"last":0.47599999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574897,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89684,"kb_used_data":54472,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282156,"statfs":{"total":96636764160,"available":96544927744,"internally_reserved":0,"allocated":55779328,"data_stored":58332680,"data_compressed":3021454,"data_compressed_allocated":3239936,"data_compressed_original":6823936,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":9.7919999999999998,"5min":9.5660000000000007,"15min":7.6950000000000003},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":127.30500000000001,"5min":133.09899999999999,"15min":133.09899999999999},"last":0.314},{"interface":"front","average":{"1min":27.553999999999998,"5min":11.506,"15min":4.8399999999999999},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":451.50799999999998,"5min":451.50799999999998,"15min":451.50799999999998},"last":0.214}]},{"osd":2,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":1.5309999999999999,"15min":1.5469999999999999},"min":{"1min":0.217,"5min":0.185,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":39.906999999999996,"15min":39.906999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.32600000000000001,"5min":2.2090000000000001,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.41299999999999998,"5min":94.820999999999998,"15min":94.820999999999998},"last":0.32600000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":6.1859999999999999,"5min":3.5299999999999998,"15min":1.486},"min":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":74.963999999999999,"5min":99.332999999999998,"15min":99.332999999999998},"last":0.28299999999999997},{"interface":"front","average":{"1min":16.207000000000001,"5min":9.2159999999999993,"15min":4.1139999999999999},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":273.65100000000001,"5min":297.82100000000003,"15min":297.82100000000003},"last":0.36699999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":5.9980000000000002,"5min":3.4809999999999999,"15min":1.4970000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":96.528999999999996,"5min":96.528999999999996,"15min":96.528999999999996},"last":0.495},{"interface":"front","average":{"1min":0.45100000000000001,"5min":8.8000000000000007,"15min":9.5410000000000004},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":160.02199999999999,"15min":160.02199999999999},"last":0.63100000000000001}]},{"osd":5,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":2.1800000000000002,"15min":1.8899999999999999},"min":{"1min":0.35399999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.60399999999999998,"5min":41.645000000000003,"15min":41.645000000000003},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":10.452999999999999,"15min":12.787000000000001},"min":{"1min":0.32300000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66200000000000003,"5min":282.197,"15min":282.197},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.7770000000000001,"15min":1.385},"min":{"1min":0.36199999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58299999999999996,"5min":99.355000000000004,"15min":99.355000000000004},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.499,"5min":2.8929999999999998,"15min":2.129},"min":{"1min":0.35899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.72999999999999998,"5min":52.762,"15min":52.762},"last":0.65800000000000003}]},{"osd":7,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":4.8929999999999998,"15min":4.7489999999999997},"min":{"1min":0.34499999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.68700000000000006,"5min":94.662999999999997,"15min":94.662999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":9.0709999999999997,"5min":4.2939999999999996,"15min":1.901},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":157.089,"5min":157.089,"15min":157.089},"last":0.753}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":81391616,"data_stored":85790877,"data_compressed":4527904,"data_compressed_allocated":4829184,"data_compressed_original":9498624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25460736,"data_stored":26833225,"data_compressed":1389038,"data_compressed_allocated":1474560,"data_compressed_original":2936832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27324416,"data_stored":28849251,"data_compressed":1581247,"data_compressed_allocated":1675264,"data_compressed_original":3284992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:42:29.401 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:42:29.679 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:42:29.680 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:42:29.691 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-04-22T18:42:29.002414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":156248488,"num_objects":224,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1404,"num_read_kb":30872,"num_write":1881,"num_write_kb":393766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1298,"num_bytes_recovered":243128123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1528,"ondisk_log_size":1528,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":502328,"kb_used_data":232908,"kb_used_omap":12,"kb_used_meta":269299,"kb_avail":754472392,"statfs":{"total":773094113280,"available":772579729408,"internally_reserved":0,"allocated":238497792,"data_stored":245878802,"data_compressed":13126867,"data_compressed_allocated":13996032,"data_compressed_original":28819456,"omap_allocated":12744,"internal_metadata":275762744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-28,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":8818863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.098522"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":35,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.260024+0000","last_change":"2024-04-22T18:42:11.391813+0000","last_active":"2024-04-22T18:42:26.260024+0000","last_peered":"2024-04-22T18:42:26.260024+0000","last_clean":"2024-04-22T18:42:26.260024+0000","last_became_active":"2024-04-22T18:42:11.030639+0000","last_became_peered":"2024-04-22T18:42:11.030639+0000","last_unstale":"2024-04-22T18:42:26.260024+0000","last_undegraded":"2024-04-22T18:42:26.260024+0000","last_fullsized":"2024-04-22T18:42:26.260024+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.391633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.391633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:18.616280+0000","scrub_duration":0.102099431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":239,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.258799+0000","last_change":"2024-04-22T18:42:11.029787+0000","last_active":"2024-04-22T18:42:26.258799+0000","last_peered":"2024-04-22T18:42:26.258799+0000","last_clean":"2024-04-22T18:42:26.258799+0000","last_became_active":"2024-04-22T18:42:11.029453+0000","last_became_peered":"2024-04-22T18:42:11.029453+0000","last_unstale":"2024-04-22T18:42:26.258799+0000","last_undegraded":"2024-04-22T18:42:26.258799+0000","last_fullsized":"2024-04-22T18:42:26.258799+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:58.561808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:58.561808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:02.874625+0000","scrub_duration":0.301876478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":245,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.259270+0000","last_change":"2024-04-22T18:42:11.428432+0000","last_active":"2024-04-22T18:42:26.259270+0000","last_peered":"2024-04-22T18:42:26.259270+0000","last_clean":"2024-04-22T18:42:26.259270+0000","last_became_active":"2024-04-22T18:41:26.961835+0000","last_became_peered":"2024-04-22T18:41:26.961835+0000","last_unstale":"2024-04-22T18:42:26.259270+0000","last_undegraded":"2024-04-22T18:42:26.259270+0000","last_fullsized":"2024-04-22T18:42:26.259270+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.428262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.428262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:11.658873+0000","scrub_duration":0.30251553399999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":229,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.258805+0000","last_change":"2024-04-22T18:42:12.520547+0000","last_active":"2024-04-22T18:42:26.258805+0000","last_peered":"2024-04-22T18:42:26.258805+0000","last_clean":"2024-04-22T18:42:26.258805+0000","last_became_active":"2024-04-22T18:41:26.854399+0000","last_became_peered":"2024-04-22T18:41:26.854399+0000","last_unstale":"2024-04-22T18:42:26.258805+0000","last_undegraded":"2024-04-22T18:42:26.258805+0000","last_fullsized":"2024-04-22T18:42:26.258805+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:12.520456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:12.520456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:42.305803+0000","scrub_duration":0.20146086999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"169'384","reported_seq":1626,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-04-22T18:42:25.136623+0000","last_change":"2024-04-22T18:42:17.393680+0000","last_active":"2024-04-22T18:42:25.136623+0000","last_peered":"2024-04-22T18:42:25.136623+0000","last_clean":"2024-04-22T18:42:25.136623+0000","last_became_active":"2024-04-22T18:41:26.994048+0000","last_became_peered":"2024-04-22T18:41:26.994048+0000","last_unstale":"2024-04-22T18:42:25.136623+0000","last_undegraded":"2024-04-22T18:42:25.136623+0000","last_fullsized":"2024-04-22T18:42:25.136623+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'384","last_scrub_stamp":"2024-04-22T18:42:17.393461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.393461+0000","objects_scrubbed":54,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:29.216372+0000","scrub_duration":0.111477038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41123046,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":358,"num_read_kb":4228,"num_write":477,"num_write_kb":113967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":82068207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.219113+0000","last_change":"2024-04-22T18:42:09.930934+0000","last_active":"2024-04-22T18:42:26.219113+0000","last_peered":"2024-04-22T18:42:26.219113+0000","last_clean":"2024-04-22T18:42:26.219113+0000","last_became_active":"2024-04-22T18:42:09.929115+0000","last_became_peered":"2024-04-22T18:42:09.929115+0000","last_unstale":"2024-04-22T18:42:26.219113+0000","last_undegraded":"2024-04-22T18:42:26.219113+0000","last_fullsized":"2024-04-22T18:42:26.219113+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:55.558446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.558446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.56126016300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"200'362","reported_seq":846,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.358870+0000","last_change":"2024-04-22T18:42:17.640545+0000","last_active":"2024-04-22T18:42:26.358870+0000","last_peered":"2024-04-22T18:42:26.358870+0000","last_clean":"2024-04-22T18:42:26.358870+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:42:26.358870+0000","last_undegraded":"2024-04-22T18:42:26.358870+0000","last_fullsized":"2024-04-22T18:42:26.358870+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'362","last_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_deep_scrub":"200'362","last_deep_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.640441+0000","objects_scrubbed":53,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:47.115778+0000","scrub_duration":3.4448620060000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382624,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":306,"num_read_kb":4987,"num_write":404,"num_write_kb":80545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":176,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.358947+0000","last_change":"2024-04-22T18:41:41.421087+0000","last_active":"2024-04-22T18:42:26.358947+0000","last_peered":"2024-04-22T18:42:26.358947+0000","last_clean":"2024-04-22T18:42:26.358947+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:42:26.358947+0000","last_undegraded":"2024-04-22T18:42:26.358947+0000","last_fullsized":"2024-04-22T18:42:26.358947+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:41.420996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:41.420996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:57.844851+0000","scrub_duration":0.20141637900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"168'337","reported_seq":1427,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:25.917451+0000","last_change":"2024-04-22T18:42:09.682366+0000","last_active":"2024-04-22T18:42:25.917451+0000","last_peered":"2024-04-22T18:42:25.917451+0000","last_clean":"2024-04-22T18:42:25.917451+0000","last_became_active":"2024-04-22T18:41:41.007340+0000","last_became_peered":"2024-04-22T18:41:41.007340+0000","last_unstale":"2024-04-22T18:42:25.917451+0000","last_undegraded":"2024-04-22T18:42:25.917451+0000","last_fullsized":"2024-04-22T18:42:25.917451+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'337","last_scrub_stamp":"2024-04-22T18:42:09.682178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:09.682178+0000","objects_scrubbed":54,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:19.217879+0000","scrub_duration":0.20999784699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39035424,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":322,"num_read_kb":9140,"num_write":423,"num_write_kb":104979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":67954083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":220,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.359433+0000","last_change":"2024-04-22T18:42:13.608651+0000","last_active":"2024-04-22T18:42:26.359433+0000","last_peered":"2024-04-22T18:42:26.359433+0000","last_clean":"2024-04-22T18:42:26.359433+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:42:26.359433+0000","last_undegraded":"2024-04-22T18:42:26.359433+0000","last_fullsized":"2024-04-22T18:42:26.359433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_clean_scrub_stamp":"2024-04-22T18:42:13.608479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:21.626020+0000","scrub_duration":0.43950688900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"169'328","reported_seq":1326,"reported_epoch":210,"state":"active+recovering","last_fresh":"2024-04-22T18:42:27.005294+0000","last_change":"2024-04-22T18:42:19.385009+0000","last_active":"2024-04-22T18:42:27.005294+0000","last_peered":"2024-04-22T18:42:27.005294+0000","last_clean":"2024-04-22T18:41:44.719816+0000","last_became_active":"2024-04-22T18:42:18.879845+0000","last_became_peered":"2024-04-22T18:42:18.879845+0000","last_unstale":"2024-04-22T18:42:27.005294+0000","last_undegraded":"2024-04-22T18:42:27.005294+0000","last_fullsized":"2024-04-22T18:42:27.005294+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'328","last_scrub_stamp":"2024-04-22T18:42:06.716200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:06.716200+0000","objects_scrubbed":46,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:35.844741+0000","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43117026,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":302,"num_read_kb":12417,"num_write":418,"num_write_kb":91859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":367,"num_bytes_recovered":70659725,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":4},{"shards":"3,5","objects":57}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.118901+0000","last_change":"2024-04-22T18:42:10.136809+0000","last_active":"2024-04-22T18:42:26.118901+0000","last_peered":"2024-04-22T18:42:26.118901+0000","last_clean":"2024-04-22T18:42:26.118901+0000","last_became_active":"2024-04-22T18:42:10.036239+0000","last_became_peered":"2024-04-22T18:42:10.036239+0000","last_unstale":"2024-04-22T18:42:26.118901+0000","last_undegraded":"2024-04-22T18:42:26.118901+0000","last_fullsized":"2024-04-22T18:42:26.118901+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:30.504686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:30.504686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:45.796059+0000","scrub_duration":0.30171718199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":415,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.259297+0000","last_change":"2024-04-22T18:41:56.270751+0000","last_active":"2024-04-22T18:42:26.259297+0000","last_peered":"2024-04-22T18:42:26.259297+0000","last_clean":"2024-04-22T18:42:26.259297+0000","last_became_active":"2024-04-22T18:41:26.757958+0000","last_became_peered":"2024-04-22T18:41:26.757958+0000","last_unstale":"2024-04-22T18:42:26.259297+0000","last_undegraded":"2024-04-22T18:42:26.259297+0000","last_fullsized":"2024-04-22T18:42:26.259297+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:41:56.270650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:41:56.270650+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:12.138524+0000","scrub_duration":0.202387964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":155658120,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":1288,"num_read_kb":30772,"num_write":1722,"num_write_kb":391350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1292,"num_bytes_recovered":241541339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274956288,"data_stored":290746422,"data_compressed":16015051,"data_compressed_allocated":17043456,"data_compressed_original":33771520,"omap_allocated":0,"internal_metadata":0},"log_size":1411,"ondisk_log_size":1411,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":2115712,"data_compressed":122550,"data_compressed_allocated":163840,"data_compressed_original":2084864,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574897,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1585152,"data_stored":1228441,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":2.5870000000000002,"15min":1.1439999999999999},"min":{"1min":0.182,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":26.568000000000001,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":6.3600000000000003,"5min":6.766,"15min":5.2599999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":95.918999999999997,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.41499999999999998}]},{"osd":1,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":3.8050000000000002,"5min":3.984,"15min":4.5709999999999997},"min":{"1min":0.29299999999999998,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":66.814999999999998,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.33100000000000002},{"interface":"front","average":{"1min":6.6040000000000001,"5min":12.141,"15min":12.744999999999999},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":100.242,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7130000000000001,"5min":4.0730000000000004,"15min":4.0810000000000004},"min":{"1min":0.27400000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":97.662000000000006,"5min":97.662000000000006,"15min":97.662000000000006},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.48599999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7560000000000002,"5min":4.1289999999999996,"15min":4.1429999999999998},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":97.686999999999998,"5min":97.686999999999998,"15min":97.686999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.36499999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":8.2089999999999996,"5min":5.4509999999999996,"15min":4.7889999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":78.149000000000001,"5min":85.509,"15min":85.509},"last":0.31},{"interface":"front","average":{"1min":7.7130000000000001,"5min":16.856000000000002,"15min":20.866},"min":{"1min":0.26000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":97.590000000000003,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.47499999999999998}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":2.0270000000000001,"15min":2.7229999999999999},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.4730000000000001,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.6679999999999999,"5min":5.2939999999999996,"15min":6.5389999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":23.007999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.55300000000000005}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":11.756,"15min":12.769},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":23.027000000000001,"5min":139.428,"15min":139.428},"last":0.503},{"interface":"front","average":{"1min":7.7800000000000002,"5min":4.3070000000000004,"15min":4.1909999999999998},"min":{"1min":0.29199999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":97.635999999999996,"5min":97.635999999999996,"15min":97.635999999999996},"last":0.51700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1503232,"data_stored":698051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.5030000000000001,"5min":3.0579999999999998,"15min":2.2149999999999999},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":70.436000000000007,"5min":70.436000000000007,"15min":70.436000000000007},"last":0.307},{"interface":"front","average":{"1min":0.41599999999999998,"5min":16.753,"15min":22.789000000000001},"min":{"1min":0.26700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.58599999999999997,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":3.577,"15min":2.5510000000000002},"min":{"1min":0.254,"5min":0.215,"15min":0.215},"max":{"1min":0.54000000000000004,"5min":199.047,"15min":199.047},"last":0.38500000000000001},{"interface":"front","average":{"1min":11.327999999999999,"5min":8.6519999999999992,"15min":9.6120000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":192.279,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.68799999999999994,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":12.56,"15min":16.722000000000001},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.59099999999999997,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.41199999999999998}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":14.815,"15min":19.283999999999999},"min":{"1min":0.29599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65200000000000002,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":1.335,"15min":0.74299999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.875,"5min":77.097999999999999,"15min":77.097999999999999},"last":0.44400000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":2.052,"5min":5.0499999999999998,"15min":6.7519999999999998},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":29.978999999999999,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":10.577,"5min":16.312000000000001,"15min":18.321999999999999},"min":{"1min":0.27900000000000003,"5min":0.218,"15min":0.218},"max":{"1min":168.44,"5min":168.44,"15min":168.44},"last":0.53700000000000003}]},{"osd":5,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.252,"15min":0.252},"max":{"1min":0.61199999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.28199999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.51200000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":5.6289999999999996,"5min":14.432,"15min":17.378},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":90.649000000000001,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.165,"5min":0.60099999999999998,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.077,"5min":13.077,"15min":13.077},"last":0.496}]}]},{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56432,"kb_used_data":25316,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94315408,"statfs":{"total":96636764160,"available":96578977792,"internally_reserved":0,"allocated":25923584,"data_stored":26446986,"data_compressed":1330791,"data_compressed_allocated":1413120,"data_compressed_original":2805760,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3580000000000001,"5min":9.875,"15min":12.218999999999999},"min":{"1min":0.34999999999999998,"5min":0.192,"15min":0.192},"max":{"1min":58.421999999999997,"5min":74.019999999999996,"15min":74.019999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.438,"5min":0.71199999999999997,"15min":0.84099999999999997},"min":{"1min":0.34000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.52400000000000002,"5min":8.3919999999999995,"15min":8.3919999999999995},"last":0.52900000000000003}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.0259999999999998,"5min":11.034000000000001,"15min":10.702999999999999},"min":{"1min":0.28100000000000003,"5min":0.224,"15min":0.224},"max":{"1min":91.911000000000001,"5min":202.13800000000001,"15min":202.13800000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.3440000000000001,"15min":0.72699999999999998},"min":{"1min":0.34399999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.669,"5min":68.346999999999994,"15min":68.346999999999994},"last":0.61799999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.496,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":3.339,"15min":4.4720000000000004},"min":{"1min":0.377,"5min":0.308,"15min":0.308},"max":{"1min":0.57399999999999995,"5min":78.296999999999997,"15min":78.296999999999997},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":4.5499999999999998,"15min":5.3490000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.018,"5min":89.688000000000002,"15min":89.688000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":6.5380000000000003,"15min":9.0890000000000004},"min":{"1min":0.34899999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60899999999999999,"5min":89.718999999999994,"15min":89.718999999999994},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.4809999999999999,"5min":15.465999999999999,"15min":16.969000000000001},"min":{"1min":0.249,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":100.544,"5min":202.983,"15min":202.983},"last":0.40100000000000002},{"interface":"front","average":{"1min":9.5050000000000008,"5min":14.603999999999999,"15min":16.722999999999999},"min":{"1min":0.314,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":100.56,"5min":202.999,"15min":202.999},"last":0.48699999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":4.3739999999999997,"15min":4.8159999999999998},"min":{"1min":0.27100000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.59099999999999997,"5min":79.945999999999998,"15min":79.945999999999998},"last":0.378},{"interface":"front","average":{"1min":0.40799999999999997,"5min":5.274,"15min":5.9770000000000003},"min":{"1min":0.251,"5min":0.249,"15min":0.249},"max":{"1min":0.57299999999999995,"5min":100.54900000000001,"15min":100.54900000000001},"last":0.435}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":1.149,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":71.602999999999994,"5min":71.602999999999994,"15min":71.602999999999994},"last":0.41799999999999998},{"interface":"front","average":{"1min":4.508,"5min":4.173,"15min":4.7770000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":75.436000000000007,"5min":78.340999999999994,"15min":78.340999999999994},"last":0.45000000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116212,"kb_used_data":81000,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94255628,"statfs":{"total":96636764160,"available":96517763072,"internally_reserved":0,"allocated":82944000,"data_stored":86498190,"data_compressed":4527904,"data_compressed_allocated":4829184,"data_compressed_original":9498624,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":13.115,"15min":15.470000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":38.899999999999999,"5min":252.96199999999999,"15min":252.96199999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":9.282,"15min":8.8059999999999992},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63300000000000001,"5min":187.148,"15min":187.148},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":2.508,"15min":1.115},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":9.141,"5min":96.486999999999995,"15min":96.486999999999995},"last":0.41599999999999998},{"interface":"front","average":{"1min":22.393000000000001,"5min":11.545999999999999,"15min":7.4390000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":310.61500000000001,"5min":310.61500000000001,"15min":310.61500000000001},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":8.8989999999999991,"15min":8.6790000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.8330000000000002,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.995,"5min":2.4199999999999999,"15min":1.1040000000000001},"min":{"1min":0.31900000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.1399999999999997,"5min":96.471999999999994,"15min":96.471999999999994},"last":0.61799999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":11.172000000000001,"5min":8.9710000000000001,"15min":5.6710000000000003},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":122.102,"5min":252.91900000000001,"15min":252.91900000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":17.242000000000001,"15min":19.725999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59899999999999998,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.745}]},{"osd":5,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":2.3380000000000001,"15min":1.034},"min":{"1min":0.25800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":7.1559999999999997,"5min":96.408000000000001,"15min":96.408000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":2.6960000000000002,"5min":3.048,"15min":3.7080000000000002},"min":{"1min":0.36199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":38.581000000000003,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.69399999999999995}]},{"osd":6,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":5.8289999999999997,"5min":14.667,"15min":13.662000000000001},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":87.706999999999994,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.83199999999999996,"5min":2.3820000000000001,"15min":1.0880000000000001},"min":{"1min":0.35699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.1159999999999997,"5min":96.438000000000002,"15min":96.438000000000002},"last":0.76800000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":9.5749999999999993,"15min":8.8979999999999997},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68000000000000005,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.9790000000000001,"5min":2.6419999999999999,"15min":1.202},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":19.518999999999998,"5min":96.456999999999994,"15min":96.456999999999994},"last":0.72299999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574897,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63312,"kb_used_data":28096,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94308528,"statfs":{"total":96636764160,"available":96571932672,"internally_reserved":0,"allocated":28770304,"data_stored":29401156,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":15.041,"5min":10.538,"15min":11.646000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":93.131,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":7.077,"5min":1.6659999999999999,"15min":0.77400000000000002},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":87.305999999999997,"5min":87.305999999999997,"15min":87.305999999999997},"last":0.28000000000000003}]},{"osd":1,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":10.558,"15min":6.8819999999999997},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":78.805000000000007,"5min":197.02099999999999,"15min":197.02099999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":8.9800000000000004,"5min":2.3210000000000002,"15min":0.997},"min":{"1min":0.20999999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":78.393000000000001,"5min":78.393000000000001,"15min":78.393000000000001},"last":0.26700000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":2.3439999999999999,"15min":1.452},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":47.534999999999997,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.37},{"interface":"front","average":{"1min":6.4770000000000003,"5min":1.728,"15min":0.85899999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":78.421999999999997,"5min":78.421999999999997,"15min":78.421999999999997},"last":0.34999999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1800000000000002,"5min":2.9940000000000002,"15min":3.7360000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":26.141999999999999,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.59199999999999997},{"interface":"front","average":{"1min":4.5629999999999997,"5min":4.6289999999999996,"15min":4.3010000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":47.616999999999997,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.66200000000000003}]},{"osd":5,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.8279999999999998,"5min":2.5049999999999999,"15min":1.623},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":47.566000000000003,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.61799999999999999},{"interface":"front","average":{"1min":6.6379999999999999,"5min":2.2890000000000001,"15min":1.601},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.480999999999995,"5min":78.480999999999995,"15min":78.480999999999995},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.50900000000000001,"15min":0.53100000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.748,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":2.8500000000000001,"5min":2.871,"15min":1.3109999999999999},"min":{"1min":0.377,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":47.590000000000003,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":6.6189999999999998,"5min":1.903,"15min":1.0229999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.453999999999994,"5min":78.453999999999994,"15min":78.453999999999994},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69299999999999995,"5min":0.995,"15min":0.995},"last":0.49399999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51588,"kb_used_data":20472,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94320252,"statfs":{"total":96636764160,"available":96583938048,"internally_reserved":0,"allocated":20963328,"data_stored":21629172,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":6.7439999999999998,"15min":7.6660000000000004},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":84.338999999999999,"5min":149.03,"15min":149.03},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.28299999999999997,"5min":1.3089999999999999,"15min":1.7589999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.62,"5min":31.866,"15min":31.866},"last":0.42899999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":9.4849999999999994,"5min":5.2489999999999997,"15min":2.4380000000000002},"min":{"1min":0.216,"5min":0.193,"15min":0.193},"max":{"1min":91.912000000000006,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":12.752000000000001,"5min":15.912000000000001,"15min":15.227},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":126.42,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.53300000000000003}]},{"osd":3,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":6.2830000000000004,"15min":2.3359999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":129.041,"5min":147.721,"15min":147.721},"last":1.179},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":101.43300000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":8.1280000000000001,"5min":13.57,"15min":17.038},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":90.882999999999996,"5min":160.584,"15min":160.584},"last":0.50700000000000001},{"interface":"front","average":{"1min":5.9349999999999996,"5min":4.8520000000000003,"15min":3.2490000000000001},"min":{"1min":0.32500000000000001,"5min":0.315,"15min":0.315},"max":{"1min":93.962999999999994,"5min":97.887,"15min":97.887},"last":0.68100000000000005}]},{"osd":5,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":2.0339999999999998,"15min":2.7120000000000002},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.879,"5min":48.631,"15min":48.631},"last":0.624},{"interface":"front","average":{"1min":0.50900000000000001,"5min":2.2890000000000001,"15min":3.0800000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.91800000000000004,"5min":56.18,"15min":56.18},"last":0.56599999999999995}]},{"osd":6,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.3540000000000001,"5min":3.4670000000000001,"15min":1.5129999999999999},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":92.028000000000006,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59799999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.29,"5min":1.5800000000000001,"15min":0.91000000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":90.944999999999993,"5min":90.944999999999993,"15min":90.944999999999993},"last":0.47599999999999998},{"interface":"front","average":{"1min":7.4640000000000004,"5min":3.923,"15min":1.6970000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":92.048000000000002,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.72899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59852,"kb_used_data":20536,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94311988,"statfs":{"total":96636764160,"available":96575475712,"internally_reserved":0,"allocated":21028864,"data_stored":21644126,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":5.883,"15min":7.2000000000000002},"min":{"1min":0.20000000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":27.116,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.83899999999999997},{"interface":"front","average":{"1min":5.8040000000000003,"5min":6.6870000000000003,"15min":7.4710000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":94.046000000000006,"5min":94.046000000000006,"15min":94.046000000000006},"last":0.64400000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.679000000000002,"5min":13.159000000000001,"15min":10.119},"min":{"1min":0.23000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":406.68799999999999,"5min":406.68799999999999,"15min":406.68799999999999},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.8700000000000001,"5min":0.61899999999999999,"15min":0.41099999999999998},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19},"max":{"1min":27.094999999999999,"5min":27.094999999999999,"15min":27.094999999999999},"last":0.90900000000000003}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":15.968,"5min":3.464,"15min":1.3759999999999999},"min":{"1min":0.29099999999999998,"5min":0.216,"15min":0.216},"max":{"1min":228.90100000000001,"5min":228.90100000000001,"15min":228.90100000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":21.481999999999999,"5min":13.061999999999999,"15min":14.015000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":196.82900000000001,"5min":231.864,"15min":231.864},"last":0.69099999999999995}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.823,"5min":17.213000000000001,"15min":14.393000000000001},"min":{"1min":0.316,"5min":0.309,"15min":0.309},"max":{"1min":406.71199999999999,"5min":406.71199999999999,"15min":406.71199999999999},"last":1.228},{"interface":"front","average":{"1min":25.027000000000001,"5min":5.3979999999999997,"15min":2.1320000000000001},"min":{"1min":0.34499999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":186.89500000000001,"5min":186.89500000000001,"15min":186.89500000000001},"last":0.80800000000000005}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":9.7430000000000003,"15min":12.628},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":94.183000000000007,"5min":138.089,"15min":138.089},"last":0.60899999999999999},{"interface":"front","average":{"1min":39.863999999999997,"5min":20.117000000000001,"15min":20.09},"min":{"1min":0.35499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":406.72899999999998,"5min":406.72899999999998,"15min":406.72899999999998},"last":1.198}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":6.0010000000000003,"5min":3.8359999999999999,"15min":3.1299999999999999},"min":{"1min":0.34000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":94.302999999999997,"5min":94.302999999999997,"15min":94.302999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":43.036000000000001,"5min":19.713999999999999,"15min":17.591999999999999},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":427.69299999999998,"5min":427.69299999999998,"15min":427.69299999999998},"last":1.2549999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":2.0720000000000001,"5min":0.82799999999999996,"15min":0.626},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":27.253,"5min":27.253,"15min":27.253},"last":0.77900000000000003},{"interface":"front","average":{"1min":6.0090000000000003,"5min":3.129,"15min":3.069},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":94.260999999999996,"5min":94.260999999999996,"15min":94.260999999999996},"last":0.47599999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574897,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89684,"kb_used_data":54472,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282156,"statfs":{"total":96636764160,"available":96544927744,"internally_reserved":0,"allocated":55779328,"data_stored":58332680,"data_compressed":3021454,"data_compressed_allocated":3239936,"data_compressed_original":6823936,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":9.7919999999999998,"5min":9.5660000000000007,"15min":7.6950000000000003},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":127.30500000000001,"5min":133.09899999999999,"15min":133.09899999999999},"last":0.314},{"interface":"front","average":{"1min":27.553999999999998,"5min":11.506,"15min":4.8399999999999999},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":451.50799999999998,"5min":451.50799999999998,"15min":451.50799999999998},"last":0.214}]},{"osd":2,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":1.5309999999999999,"15min":1.5469999999999999},"min":{"1min":0.217,"5min":0.185,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":39.906999999999996,"15min":39.906999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.32600000000000001,"5min":2.2090000000000001,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.41299999999999998,"5min":94.820999999999998,"15min":94.820999999999998},"last":0.32600000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":6.1859999999999999,"5min":3.5299999999999998,"15min":1.486},"min":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":74.963999999999999,"5min":99.332999999999998,"15min":99.332999999999998},"last":0.28299999999999997},{"interface":"front","average":{"1min":16.207000000000001,"5min":9.2159999999999993,"15min":4.1139999999999999},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":273.65100000000001,"5min":297.82100000000003,"15min":297.82100000000003},"last":0.36699999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":5.9980000000000002,"5min":3.4809999999999999,"15min":1.4970000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":96.528999999999996,"5min":96.528999999999996,"15min":96.528999999999996},"last":0.495},{"interface":"front","average":{"1min":0.45100000000000001,"5min":8.8000000000000007,"15min":9.5410000000000004},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":160.02199999999999,"15min":160.02199999999999},"last":0.63100000000000001}]},{"osd":5,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":2.1800000000000002,"15min":1.8899999999999999},"min":{"1min":0.35399999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.60399999999999998,"5min":41.645000000000003,"15min":41.645000000000003},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":10.452999999999999,"15min":12.787000000000001},"min":{"1min":0.32300000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66200000000000003,"5min":282.197,"15min":282.197},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.7770000000000001,"15min":1.385},"min":{"1min":0.36199999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58299999999999996,"5min":99.355000000000004,"15min":99.355000000000004},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.499,"5min":2.8929999999999998,"15min":2.129},"min":{"1min":0.35899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.72999999999999998,"5min":52.762,"15min":52.762},"last":0.65800000000000003}]},{"osd":7,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":4.8929999999999998,"15min":4.7489999999999997},"min":{"1min":0.34499999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.68700000000000006,"5min":94.662999999999997,"15min":94.662999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":9.0709999999999997,"5min":4.2939999999999996,"15min":1.901},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":157.089,"5min":157.089,"15min":157.089},"last":0.753}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":81391616,"data_stored":85790877,"data_compressed":4527904,"data_compressed_allocated":4829184,"data_compressed_original":9498624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25460736,"data_stored":26833225,"data_compressed":1389038,"data_compressed_allocated":1474560,"data_compressed_original":2936832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27324416,"data_stored":28849251,"data_compressed":1581247,"data_compressed_allocated":1675264,"data_compressed_original":3284992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:42:29.693 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T18:42:30.034 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:42:30.034 INFO:teuthology.orchestra.run.smithi089.stdout:{"fsid":"9e25744c-eff1-4a4a-8d47-0dda406b2c35","health":{"status":"HEALTH_WARN","checks":{"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":243,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":211,"num_osds":8,"num_up_osds":8,"osd_up_since":1713811115,"num_in_osds":8,"osd_in_since":1713811329,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovering","count":1}],"num_pgs":13,"num_pools":3,"num_objects":224,"data_bytes":156248488,"bytes_used":514383872,"bytes_avail":772579729408,"bytes_total":773094113280,"misplaced_objects":4,"misplaced_total":448,"misplaced_ratio":0.0089285714285714281,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":2151717,"recovering_keys_per_sec":0,"num_objects_recovered":45,"num_bytes_recovered":8818863,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-04-22T18:39:36.971077+0000","services":{}},"progress_events":{}} 2024-04-22T18:42:30.047 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-04-22T18:42:30.047 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:42:30.321 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:42:30.322 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:42:30.333 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-04-22T18:42:29.002414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":156248488,"num_objects":224,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1404,"num_read_kb":30872,"num_write":1881,"num_write_kb":393766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1298,"num_bytes_recovered":243128123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1528,"ondisk_log_size":1528,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":502328,"kb_used_data":232908,"kb_used_omap":12,"kb_used_meta":269299,"kb_avail":754472392,"statfs":{"total":773094113280,"available":772579729408,"internally_reserved":0,"allocated":238497792,"data_stored":245878802,"data_compressed":13126867,"data_compressed_allocated":13996032,"data_compressed_original":28819456,"omap_allocated":12744,"internal_metadata":275762744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-28,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":8818863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.098522"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":35,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.260024+0000","last_change":"2024-04-22T18:42:11.391813+0000","last_active":"2024-04-22T18:42:26.260024+0000","last_peered":"2024-04-22T18:42:26.260024+0000","last_clean":"2024-04-22T18:42:26.260024+0000","last_became_active":"2024-04-22T18:42:11.030639+0000","last_became_peered":"2024-04-22T18:42:11.030639+0000","last_unstale":"2024-04-22T18:42:26.260024+0000","last_undegraded":"2024-04-22T18:42:26.260024+0000","last_fullsized":"2024-04-22T18:42:26.260024+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.391633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.391633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:18.616280+0000","scrub_duration":0.102099431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":239,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.258799+0000","last_change":"2024-04-22T18:42:11.029787+0000","last_active":"2024-04-22T18:42:26.258799+0000","last_peered":"2024-04-22T18:42:26.258799+0000","last_clean":"2024-04-22T18:42:26.258799+0000","last_became_active":"2024-04-22T18:42:11.029453+0000","last_became_peered":"2024-04-22T18:42:11.029453+0000","last_unstale":"2024-04-22T18:42:26.258799+0000","last_undegraded":"2024-04-22T18:42:26.258799+0000","last_fullsized":"2024-04-22T18:42:26.258799+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:58.561808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:58.561808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:02.874625+0000","scrub_duration":0.301876478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":245,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.259270+0000","last_change":"2024-04-22T18:42:11.428432+0000","last_active":"2024-04-22T18:42:26.259270+0000","last_peered":"2024-04-22T18:42:26.259270+0000","last_clean":"2024-04-22T18:42:26.259270+0000","last_became_active":"2024-04-22T18:41:26.961835+0000","last_became_peered":"2024-04-22T18:41:26.961835+0000","last_unstale":"2024-04-22T18:42:26.259270+0000","last_undegraded":"2024-04-22T18:42:26.259270+0000","last_fullsized":"2024-04-22T18:42:26.259270+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.428262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.428262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:11.658873+0000","scrub_duration":0.30251553399999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":229,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.258805+0000","last_change":"2024-04-22T18:42:12.520547+0000","last_active":"2024-04-22T18:42:26.258805+0000","last_peered":"2024-04-22T18:42:26.258805+0000","last_clean":"2024-04-22T18:42:26.258805+0000","last_became_active":"2024-04-22T18:41:26.854399+0000","last_became_peered":"2024-04-22T18:41:26.854399+0000","last_unstale":"2024-04-22T18:42:26.258805+0000","last_undegraded":"2024-04-22T18:42:26.258805+0000","last_fullsized":"2024-04-22T18:42:26.258805+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:12.520456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:12.520456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:42.305803+0000","scrub_duration":0.20146086999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"169'384","reported_seq":1626,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-04-22T18:42:25.136623+0000","last_change":"2024-04-22T18:42:17.393680+0000","last_active":"2024-04-22T18:42:25.136623+0000","last_peered":"2024-04-22T18:42:25.136623+0000","last_clean":"2024-04-22T18:42:25.136623+0000","last_became_active":"2024-04-22T18:41:26.994048+0000","last_became_peered":"2024-04-22T18:41:26.994048+0000","last_unstale":"2024-04-22T18:42:25.136623+0000","last_undegraded":"2024-04-22T18:42:25.136623+0000","last_fullsized":"2024-04-22T18:42:25.136623+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'384","last_scrub_stamp":"2024-04-22T18:42:17.393461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.393461+0000","objects_scrubbed":54,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:29.216372+0000","scrub_duration":0.111477038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41123046,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":358,"num_read_kb":4228,"num_write":477,"num_write_kb":113967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":82068207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.219113+0000","last_change":"2024-04-22T18:42:09.930934+0000","last_active":"2024-04-22T18:42:26.219113+0000","last_peered":"2024-04-22T18:42:26.219113+0000","last_clean":"2024-04-22T18:42:26.219113+0000","last_became_active":"2024-04-22T18:42:09.929115+0000","last_became_peered":"2024-04-22T18:42:09.929115+0000","last_unstale":"2024-04-22T18:42:26.219113+0000","last_undegraded":"2024-04-22T18:42:26.219113+0000","last_fullsized":"2024-04-22T18:42:26.219113+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:40:55.558446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:40:55.558446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.56126016300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"200'362","reported_seq":846,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.358870+0000","last_change":"2024-04-22T18:42:17.640545+0000","last_active":"2024-04-22T18:42:26.358870+0000","last_peered":"2024-04-22T18:42:26.358870+0000","last_clean":"2024-04-22T18:42:26.358870+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:42:26.358870+0000","last_undegraded":"2024-04-22T18:42:26.358870+0000","last_fullsized":"2024-04-22T18:42:26.358870+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'362","last_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_deep_scrub":"200'362","last_deep_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.640441+0000","objects_scrubbed":53,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:47.115778+0000","scrub_duration":3.4448620060000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382624,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":306,"num_read_kb":4987,"num_write":404,"num_write_kb":80545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":176,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.358947+0000","last_change":"2024-04-22T18:41:41.421087+0000","last_active":"2024-04-22T18:42:26.358947+0000","last_peered":"2024-04-22T18:42:26.358947+0000","last_clean":"2024-04-22T18:42:26.358947+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:42:26.358947+0000","last_undegraded":"2024-04-22T18:42:26.358947+0000","last_fullsized":"2024-04-22T18:42:26.358947+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:41.420996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:41.420996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:57.844851+0000","scrub_duration":0.20141637900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"168'337","reported_seq":1427,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:25.917451+0000","last_change":"2024-04-22T18:42:09.682366+0000","last_active":"2024-04-22T18:42:25.917451+0000","last_peered":"2024-04-22T18:42:25.917451+0000","last_clean":"2024-04-22T18:42:25.917451+0000","last_became_active":"2024-04-22T18:41:41.007340+0000","last_became_peered":"2024-04-22T18:41:41.007340+0000","last_unstale":"2024-04-22T18:42:25.917451+0000","last_undegraded":"2024-04-22T18:42:25.917451+0000","last_fullsized":"2024-04-22T18:42:25.917451+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'337","last_scrub_stamp":"2024-04-22T18:42:09.682178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:09.682178+0000","objects_scrubbed":54,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:19.217879+0000","scrub_duration":0.20999784699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39035424,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":322,"num_read_kb":9140,"num_write":423,"num_write_kb":104979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":67954083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":220,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.359433+0000","last_change":"2024-04-22T18:42:13.608651+0000","last_active":"2024-04-22T18:42:26.359433+0000","last_peered":"2024-04-22T18:42:26.359433+0000","last_clean":"2024-04-22T18:42:26.359433+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:42:26.359433+0000","last_undegraded":"2024-04-22T18:42:26.359433+0000","last_fullsized":"2024-04-22T18:42:26.359433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_clean_scrub_stamp":"2024-04-22T18:42:13.608479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:21.626020+0000","scrub_duration":0.43950688900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"169'328","reported_seq":1326,"reported_epoch":210,"state":"active+recovering","last_fresh":"2024-04-22T18:42:27.005294+0000","last_change":"2024-04-22T18:42:19.385009+0000","last_active":"2024-04-22T18:42:27.005294+0000","last_peered":"2024-04-22T18:42:27.005294+0000","last_clean":"2024-04-22T18:41:44.719816+0000","last_became_active":"2024-04-22T18:42:18.879845+0000","last_became_peered":"2024-04-22T18:42:18.879845+0000","last_unstale":"2024-04-22T18:42:27.005294+0000","last_undegraded":"2024-04-22T18:42:27.005294+0000","last_fullsized":"2024-04-22T18:42:27.005294+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'328","last_scrub_stamp":"2024-04-22T18:42:06.716200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:06.716200+0000","objects_scrubbed":46,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:35.844741+0000","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43117026,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":302,"num_read_kb":12417,"num_write":418,"num_write_kb":91859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":367,"num_bytes_recovered":70659725,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":4},{"shards":"3,5","objects":57}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.118901+0000","last_change":"2024-04-22T18:42:10.136809+0000","last_active":"2024-04-22T18:42:26.118901+0000","last_peered":"2024-04-22T18:42:26.118901+0000","last_clean":"2024-04-22T18:42:26.118901+0000","last_became_active":"2024-04-22T18:42:10.036239+0000","last_became_peered":"2024-04-22T18:42:10.036239+0000","last_unstale":"2024-04-22T18:42:26.118901+0000","last_undegraded":"2024-04-22T18:42:26.118901+0000","last_fullsized":"2024-04-22T18:42:26.118901+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:30.504686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:30.504686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:45.796059+0000","scrub_duration":0.30171718199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":415,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.259297+0000","last_change":"2024-04-22T18:41:56.270751+0000","last_active":"2024-04-22T18:42:26.259297+0000","last_peered":"2024-04-22T18:42:26.259297+0000","last_clean":"2024-04-22T18:42:26.259297+0000","last_became_active":"2024-04-22T18:41:26.757958+0000","last_became_peered":"2024-04-22T18:41:26.757958+0000","last_unstale":"2024-04-22T18:42:26.259297+0000","last_undegraded":"2024-04-22T18:42:26.259297+0000","last_fullsized":"2024-04-22T18:42:26.259297+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:41:56.270650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:41:56.270650+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:12.138524+0000","scrub_duration":0.202387964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":155658120,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":1288,"num_read_kb":30772,"num_write":1722,"num_write_kb":391350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1292,"num_bytes_recovered":241541339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274956288,"data_stored":290746422,"data_compressed":16015051,"data_compressed_allocated":17043456,"data_compressed_original":33771520,"omap_allocated":0,"internal_metadata":0},"log_size":1411,"ondisk_log_size":1411,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":2115712,"data_compressed":122550,"data_compressed_allocated":163840,"data_compressed_original":2084864,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574897,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1585152,"data_stored":1228441,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":2.5870000000000002,"15min":1.1439999999999999},"min":{"1min":0.182,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":26.568000000000001,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":6.3600000000000003,"5min":6.766,"15min":5.2599999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":95.918999999999997,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.41499999999999998}]},{"osd":1,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":3.8050000000000002,"5min":3.984,"15min":4.5709999999999997},"min":{"1min":0.29299999999999998,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":66.814999999999998,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.33100000000000002},{"interface":"front","average":{"1min":6.6040000000000001,"5min":12.141,"15min":12.744999999999999},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":100.242,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7130000000000001,"5min":4.0730000000000004,"15min":4.0810000000000004},"min":{"1min":0.27400000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":97.662000000000006,"5min":97.662000000000006,"15min":97.662000000000006},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.48599999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7560000000000002,"5min":4.1289999999999996,"15min":4.1429999999999998},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":97.686999999999998,"5min":97.686999999999998,"15min":97.686999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.36499999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":8.2089999999999996,"5min":5.4509999999999996,"15min":4.7889999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":78.149000000000001,"5min":85.509,"15min":85.509},"last":0.31},{"interface":"front","average":{"1min":7.7130000000000001,"5min":16.856000000000002,"15min":20.866},"min":{"1min":0.26000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":97.590000000000003,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.47499999999999998}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":2.0270000000000001,"15min":2.7229999999999999},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.4730000000000001,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.6679999999999999,"5min":5.2939999999999996,"15min":6.5389999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":23.007999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.55300000000000005}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":11.756,"15min":12.769},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":23.027000000000001,"5min":139.428,"15min":139.428},"last":0.503},{"interface":"front","average":{"1min":7.7800000000000002,"5min":4.3070000000000004,"15min":4.1909999999999998},"min":{"1min":0.29199999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":97.635999999999996,"5min":97.635999999999996,"15min":97.635999999999996},"last":0.51700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1503232,"data_stored":698051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.5030000000000001,"5min":3.0579999999999998,"15min":2.2149999999999999},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":70.436000000000007,"5min":70.436000000000007,"15min":70.436000000000007},"last":0.307},{"interface":"front","average":{"1min":0.41599999999999998,"5min":16.753,"15min":22.789000000000001},"min":{"1min":0.26700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.58599999999999997,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":3.577,"15min":2.5510000000000002},"min":{"1min":0.254,"5min":0.215,"15min":0.215},"max":{"1min":0.54000000000000004,"5min":199.047,"15min":199.047},"last":0.38500000000000001},{"interface":"front","average":{"1min":11.327999999999999,"5min":8.6519999999999992,"15min":9.6120000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":192.279,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.68799999999999994,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":12.56,"15min":16.722000000000001},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.59099999999999997,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.41199999999999998}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":14.815,"15min":19.283999999999999},"min":{"1min":0.29599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65200000000000002,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":1.335,"15min":0.74299999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.875,"5min":77.097999999999999,"15min":77.097999999999999},"last":0.44400000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":2.052,"5min":5.0499999999999998,"15min":6.7519999999999998},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":29.978999999999999,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":10.577,"5min":16.312000000000001,"15min":18.321999999999999},"min":{"1min":0.27900000000000003,"5min":0.218,"15min":0.218},"max":{"1min":168.44,"5min":168.44,"15min":168.44},"last":0.53700000000000003}]},{"osd":5,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.252,"15min":0.252},"max":{"1min":0.61199999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.28199999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.51200000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":5.6289999999999996,"5min":14.432,"15min":17.378},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":90.649000000000001,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.165,"5min":0.60099999999999998,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.077,"5min":13.077,"15min":13.077},"last":0.496}]}]},{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56432,"kb_used_data":25316,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94315408,"statfs":{"total":96636764160,"available":96578977792,"internally_reserved":0,"allocated":25923584,"data_stored":26446986,"data_compressed":1330791,"data_compressed_allocated":1413120,"data_compressed_original":2805760,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3580000000000001,"5min":9.875,"15min":12.218999999999999},"min":{"1min":0.34999999999999998,"5min":0.192,"15min":0.192},"max":{"1min":58.421999999999997,"5min":74.019999999999996,"15min":74.019999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.438,"5min":0.71199999999999997,"15min":0.84099999999999997},"min":{"1min":0.34000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.52400000000000002,"5min":8.3919999999999995,"15min":8.3919999999999995},"last":0.52900000000000003}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.0259999999999998,"5min":11.034000000000001,"15min":10.702999999999999},"min":{"1min":0.28100000000000003,"5min":0.224,"15min":0.224},"max":{"1min":91.911000000000001,"5min":202.13800000000001,"15min":202.13800000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.3440000000000001,"15min":0.72699999999999998},"min":{"1min":0.34399999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.669,"5min":68.346999999999994,"15min":68.346999999999994},"last":0.61799999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.496,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":3.339,"15min":4.4720000000000004},"min":{"1min":0.377,"5min":0.308,"15min":0.308},"max":{"1min":0.57399999999999995,"5min":78.296999999999997,"15min":78.296999999999997},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":4.5499999999999998,"15min":5.3490000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.018,"5min":89.688000000000002,"15min":89.688000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":6.5380000000000003,"15min":9.0890000000000004},"min":{"1min":0.34899999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60899999999999999,"5min":89.718999999999994,"15min":89.718999999999994},"last":0.60299999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.4809999999999999,"5min":15.465999999999999,"15min":16.969000000000001},"min":{"1min":0.249,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":100.544,"5min":202.983,"15min":202.983},"last":0.40100000000000002},{"interface":"front","average":{"1min":9.5050000000000008,"5min":14.603999999999999,"15min":16.722999999999999},"min":{"1min":0.314,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":100.56,"5min":202.999,"15min":202.999},"last":0.48699999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":4.3739999999999997,"15min":4.8159999999999998},"min":{"1min":0.27100000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.59099999999999997,"5min":79.945999999999998,"15min":79.945999999999998},"last":0.378},{"interface":"front","average":{"1min":0.40799999999999997,"5min":5.274,"15min":5.9770000000000003},"min":{"1min":0.251,"5min":0.249,"15min":0.249},"max":{"1min":0.57299999999999995,"5min":100.54900000000001,"15min":100.54900000000001},"last":0.435}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":1.149,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":71.602999999999994,"5min":71.602999999999994,"15min":71.602999999999994},"last":0.41799999999999998},{"interface":"front","average":{"1min":4.508,"5min":4.173,"15min":4.7770000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":75.436000000000007,"5min":78.340999999999994,"15min":78.340999999999994},"last":0.45000000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116212,"kb_used_data":81000,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94255628,"statfs":{"total":96636764160,"available":96517763072,"internally_reserved":0,"allocated":82944000,"data_stored":86498190,"data_compressed":4527904,"data_compressed_allocated":4829184,"data_compressed_original":9498624,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":13.115,"15min":15.470000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":38.899999999999999,"5min":252.96199999999999,"15min":252.96199999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":9.282,"15min":8.8059999999999992},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63300000000000001,"5min":187.148,"15min":187.148},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":2.508,"15min":1.115},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":9.141,"5min":96.486999999999995,"15min":96.486999999999995},"last":0.41599999999999998},{"interface":"front","average":{"1min":22.393000000000001,"5min":11.545999999999999,"15min":7.4390000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":310.61500000000001,"5min":310.61500000000001,"15min":310.61500000000001},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":8.8989999999999991,"15min":8.6790000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.8330000000000002,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.995,"5min":2.4199999999999999,"15min":1.1040000000000001},"min":{"1min":0.31900000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.1399999999999997,"5min":96.471999999999994,"15min":96.471999999999994},"last":0.61799999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":11.172000000000001,"5min":8.9710000000000001,"15min":5.6710000000000003},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":122.102,"5min":252.91900000000001,"15min":252.91900000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":17.242000000000001,"15min":19.725999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59899999999999998,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.745}]},{"osd":5,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":2.3380000000000001,"15min":1.034},"min":{"1min":0.25800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":7.1559999999999997,"5min":96.408000000000001,"15min":96.408000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":2.6960000000000002,"5min":3.048,"15min":3.7080000000000002},"min":{"1min":0.36199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":38.581000000000003,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.69399999999999995}]},{"osd":6,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":5.8289999999999997,"5min":14.667,"15min":13.662000000000001},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":87.706999999999994,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.83199999999999996,"5min":2.3820000000000001,"15min":1.0880000000000001},"min":{"1min":0.35699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.1159999999999997,"5min":96.438000000000002,"15min":96.438000000000002},"last":0.76800000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":9.5749999999999993,"15min":8.8979999999999997},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68000000000000005,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.9790000000000001,"5min":2.6419999999999999,"15min":1.202},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":19.518999999999998,"5min":96.456999999999994,"15min":96.456999999999994},"last":0.72299999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574897,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63312,"kb_used_data":28096,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94308528,"statfs":{"total":96636764160,"available":96571932672,"internally_reserved":0,"allocated":28770304,"data_stored":29401156,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":15.041,"5min":10.538,"15min":11.646000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":93.131,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":7.077,"5min":1.6659999999999999,"15min":0.77400000000000002},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":87.305999999999997,"5min":87.305999999999997,"15min":87.305999999999997},"last":0.28000000000000003}]},{"osd":1,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":10.558,"15min":6.8819999999999997},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":78.805000000000007,"5min":197.02099999999999,"15min":197.02099999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":8.9800000000000004,"5min":2.3210000000000002,"15min":0.997},"min":{"1min":0.20999999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":78.393000000000001,"5min":78.393000000000001,"15min":78.393000000000001},"last":0.26700000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":2.3439999999999999,"15min":1.452},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":47.534999999999997,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.37},{"interface":"front","average":{"1min":6.4770000000000003,"5min":1.728,"15min":0.85899999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":78.421999999999997,"5min":78.421999999999997,"15min":78.421999999999997},"last":0.34999999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1800000000000002,"5min":2.9940000000000002,"15min":3.7360000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":26.141999999999999,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.59199999999999997},{"interface":"front","average":{"1min":4.5629999999999997,"5min":4.6289999999999996,"15min":4.3010000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":47.616999999999997,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.66200000000000003}]},{"osd":5,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.8279999999999998,"5min":2.5049999999999999,"15min":1.623},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":47.566000000000003,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.61799999999999999},{"interface":"front","average":{"1min":6.6379999999999999,"5min":2.2890000000000001,"15min":1.601},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.480999999999995,"5min":78.480999999999995,"15min":78.480999999999995},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.50900000000000001,"15min":0.53100000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.748,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":2.8500000000000001,"5min":2.871,"15min":1.3109999999999999},"min":{"1min":0.377,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":47.590000000000003,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":6.6189999999999998,"5min":1.903,"15min":1.0229999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.453999999999994,"5min":78.453999999999994,"15min":78.453999999999994},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69299999999999995,"5min":0.995,"15min":0.995},"last":0.49399999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51588,"kb_used_data":20472,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94320252,"statfs":{"total":96636764160,"available":96583938048,"internally_reserved":0,"allocated":20963328,"data_stored":21629172,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":6.7439999999999998,"15min":7.6660000000000004},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":84.338999999999999,"5min":149.03,"15min":149.03},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.28299999999999997,"5min":1.3089999999999999,"15min":1.7589999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.62,"5min":31.866,"15min":31.866},"last":0.42899999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":9.4849999999999994,"5min":5.2489999999999997,"15min":2.4380000000000002},"min":{"1min":0.216,"5min":0.193,"15min":0.193},"max":{"1min":91.912000000000006,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":12.752000000000001,"5min":15.912000000000001,"15min":15.227},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":126.42,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.53300000000000003}]},{"osd":3,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":6.2830000000000004,"15min":2.3359999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":129.041,"5min":147.721,"15min":147.721},"last":1.179},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":101.43300000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":8.1280000000000001,"5min":13.57,"15min":17.038},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":90.882999999999996,"5min":160.584,"15min":160.584},"last":0.50700000000000001},{"interface":"front","average":{"1min":5.9349999999999996,"5min":4.8520000000000003,"15min":3.2490000000000001},"min":{"1min":0.32500000000000001,"5min":0.315,"15min":0.315},"max":{"1min":93.962999999999994,"5min":97.887,"15min":97.887},"last":0.68100000000000005}]},{"osd":5,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":2.0339999999999998,"15min":2.7120000000000002},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.879,"5min":48.631,"15min":48.631},"last":0.624},{"interface":"front","average":{"1min":0.50900000000000001,"5min":2.2890000000000001,"15min":3.0800000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.91800000000000004,"5min":56.18,"15min":56.18},"last":0.56599999999999995}]},{"osd":6,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.3540000000000001,"5min":3.4670000000000001,"15min":1.5129999999999999},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":92.028000000000006,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59799999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.29,"5min":1.5800000000000001,"15min":0.91000000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":90.944999999999993,"5min":90.944999999999993,"15min":90.944999999999993},"last":0.47599999999999998},{"interface":"front","average":{"1min":7.4640000000000004,"5min":3.923,"15min":1.6970000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":92.048000000000002,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.72899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59852,"kb_used_data":20536,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94311988,"statfs":{"total":96636764160,"available":96575475712,"internally_reserved":0,"allocated":21028864,"data_stored":21644126,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":5.883,"15min":7.2000000000000002},"min":{"1min":0.20000000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":27.116,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.83899999999999997},{"interface":"front","average":{"1min":5.8040000000000003,"5min":6.6870000000000003,"15min":7.4710000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":94.046000000000006,"5min":94.046000000000006,"15min":94.046000000000006},"last":0.64400000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.679000000000002,"5min":13.159000000000001,"15min":10.119},"min":{"1min":0.23000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":406.68799999999999,"5min":406.68799999999999,"15min":406.68799999999999},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.8700000000000001,"5min":0.61899999999999999,"15min":0.41099999999999998},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19},"max":{"1min":27.094999999999999,"5min":27.094999999999999,"15min":27.094999999999999},"last":0.90900000000000003}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":15.968,"5min":3.464,"15min":1.3759999999999999},"min":{"1min":0.29099999999999998,"5min":0.216,"15min":0.216},"max":{"1min":228.90100000000001,"5min":228.90100000000001,"15min":228.90100000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":21.481999999999999,"5min":13.061999999999999,"15min":14.015000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":196.82900000000001,"5min":231.864,"15min":231.864},"last":0.69099999999999995}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.823,"5min":17.213000000000001,"15min":14.393000000000001},"min":{"1min":0.316,"5min":0.309,"15min":0.309},"max":{"1min":406.71199999999999,"5min":406.71199999999999,"15min":406.71199999999999},"last":1.228},{"interface":"front","average":{"1min":25.027000000000001,"5min":5.3979999999999997,"15min":2.1320000000000001},"min":{"1min":0.34499999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":186.89500000000001,"5min":186.89500000000001,"15min":186.89500000000001},"last":0.80800000000000005}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":9.7430000000000003,"15min":12.628},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":94.183000000000007,"5min":138.089,"15min":138.089},"last":0.60899999999999999},{"interface":"front","average":{"1min":39.863999999999997,"5min":20.117000000000001,"15min":20.09},"min":{"1min":0.35499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":406.72899999999998,"5min":406.72899999999998,"15min":406.72899999999998},"last":1.198}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":6.0010000000000003,"5min":3.8359999999999999,"15min":3.1299999999999999},"min":{"1min":0.34000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":94.302999999999997,"5min":94.302999999999997,"15min":94.302999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":43.036000000000001,"5min":19.713999999999999,"15min":17.591999999999999},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":427.69299999999998,"5min":427.69299999999998,"15min":427.69299999999998},"last":1.2549999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":2.0720000000000001,"5min":0.82799999999999996,"15min":0.626},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":27.253,"5min":27.253,"15min":27.253},"last":0.77900000000000003},{"interface":"front","average":{"1min":6.0090000000000003,"5min":3.129,"15min":3.069},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":94.260999999999996,"5min":94.260999999999996,"15min":94.260999999999996},"last":0.47599999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574897,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89684,"kb_used_data":54472,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282156,"statfs":{"total":96636764160,"available":96544927744,"internally_reserved":0,"allocated":55779328,"data_stored":58332680,"data_compressed":3021454,"data_compressed_allocated":3239936,"data_compressed_original":6823936,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":9.7919999999999998,"5min":9.5660000000000007,"15min":7.6950000000000003},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":127.30500000000001,"5min":133.09899999999999,"15min":133.09899999999999},"last":0.314},{"interface":"front","average":{"1min":27.553999999999998,"5min":11.506,"15min":4.8399999999999999},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":451.50799999999998,"5min":451.50799999999998,"15min":451.50799999999998},"last":0.214}]},{"osd":2,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":1.5309999999999999,"15min":1.5469999999999999},"min":{"1min":0.217,"5min":0.185,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":39.906999999999996,"15min":39.906999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.32600000000000001,"5min":2.2090000000000001,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.41299999999999998,"5min":94.820999999999998,"15min":94.820999999999998},"last":0.32600000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":6.1859999999999999,"5min":3.5299999999999998,"15min":1.486},"min":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":74.963999999999999,"5min":99.332999999999998,"15min":99.332999999999998},"last":0.28299999999999997},{"interface":"front","average":{"1min":16.207000000000001,"5min":9.2159999999999993,"15min":4.1139999999999999},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":273.65100000000001,"5min":297.82100000000003,"15min":297.82100000000003},"last":0.36699999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":5.9980000000000002,"5min":3.4809999999999999,"15min":1.4970000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":96.528999999999996,"5min":96.528999999999996,"15min":96.528999999999996},"last":0.495},{"interface":"front","average":{"1min":0.45100000000000001,"5min":8.8000000000000007,"15min":9.5410000000000004},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":160.02199999999999,"15min":160.02199999999999},"last":0.63100000000000001}]},{"osd":5,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":2.1800000000000002,"15min":1.8899999999999999},"min":{"1min":0.35399999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.60399999999999998,"5min":41.645000000000003,"15min":41.645000000000003},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":10.452999999999999,"15min":12.787000000000001},"min":{"1min":0.32300000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66200000000000003,"5min":282.197,"15min":282.197},"last":0.68600000000000005}]},{"osd":6,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.7770000000000001,"15min":1.385},"min":{"1min":0.36199999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58299999999999996,"5min":99.355000000000004,"15min":99.355000000000004},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.499,"5min":2.8929999999999998,"15min":2.129},"min":{"1min":0.35899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.72999999999999998,"5min":52.762,"15min":52.762},"last":0.65800000000000003}]},{"osd":7,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":4.8929999999999998,"15min":4.7489999999999997},"min":{"1min":0.34499999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.68700000000000006,"5min":94.662999999999997,"15min":94.662999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":9.0709999999999997,"5min":4.2939999999999996,"15min":1.901},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":157.089,"5min":157.089,"15min":157.089},"last":0.753}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":81391616,"data_stored":85790877,"data_compressed":4527904,"data_compressed_allocated":4829184,"data_compressed_original":9498624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":25460736,"data_stored":26833225,"data_compressed":1389038,"data_compressed_allocated":1474560,"data_compressed_original":2936832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27324416,"data_stored":28849251,"data_compressed":1581247,"data_compressed_allocated":1675264,"data_compressed_original":3284992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:42:32.361 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:42:32.362+0000 7f8605f32640 -1 osd.3 211 handle_osd_map injecting map crc failure 2024-04-22T18:42:33.336 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:42:33.626 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:42:33.626 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:42:33.638 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-04-22T18:42:33.003110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":156248488,"num_objects":224,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1404,"num_read_kb":30872,"num_write":1881,"num_write_kb":393766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1303,"num_bytes_recovered":244989684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1528,"ondisk_log_size":1528,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":482772,"kb_used_data":209264,"kb_used_omap":12,"kb_used_meta":273395,"kb_avail":754491948,"statfs":{"total":773094113280,"available":772599754752,"internally_reserved":0,"allocated":214286336,"data_stored":220224565,"data_compressed":11796076,"data_compressed_allocated":12582912,"data_compressed_original":26013696,"omap_allocated":12744,"internal_metadata":279957048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-32,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":10680424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.331024"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":36,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.703285+0000","last_change":"2024-04-22T18:42:11.391813+0000","last_active":"2024-04-22T18:42:26.703285+0000","last_peered":"2024-04-22T18:42:26.703285+0000","last_clean":"2024-04-22T18:42:26.703285+0000","last_became_active":"2024-04-22T18:42:11.030639+0000","last_became_peered":"2024-04-22T18:42:11.030639+0000","last_unstale":"2024-04-22T18:42:26.703285+0000","last_undegraded":"2024-04-22T18:42:26.703285+0000","last_fullsized":"2024-04-22T18:42:26.703285+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.391633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.391633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:18.616280+0000","scrub_duration":0.102099431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":241,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.930536+0000","last_change":"2024-04-22T18:42:11.029787+0000","last_active":"2024-04-22T18:42:27.930536+0000","last_peered":"2024-04-22T18:42:27.930536+0000","last_clean":"2024-04-22T18:42:27.930536+0000","last_became_active":"2024-04-22T18:42:11.029453+0000","last_became_peered":"2024-04-22T18:42:11.029453+0000","last_unstale":"2024-04-22T18:42:27.930536+0000","last_undegraded":"2024-04-22T18:42:27.930536+0000","last_fullsized":"2024-04-22T18:42:27.930536+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:58.561808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:58.561808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:02.874625+0000","scrub_duration":0.301876478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":246,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.702491+0000","last_change":"2024-04-22T18:42:11.428432+0000","last_active":"2024-04-22T18:42:26.702491+0000","last_peered":"2024-04-22T18:42:26.702491+0000","last_clean":"2024-04-22T18:42:26.702491+0000","last_became_active":"2024-04-22T18:41:26.961835+0000","last_became_peered":"2024-04-22T18:41:26.961835+0000","last_unstale":"2024-04-22T18:42:26.702491+0000","last_undegraded":"2024-04-22T18:42:26.702491+0000","last_fullsized":"2024-04-22T18:42:26.702491+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.428262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.428262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:11.658873+0000","scrub_duration":0.30251553399999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":231,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.930444+0000","last_change":"2024-04-22T18:42:12.520547+0000","last_active":"2024-04-22T18:42:27.930444+0000","last_peered":"2024-04-22T18:42:27.930444+0000","last_clean":"2024-04-22T18:42:27.930444+0000","last_became_active":"2024-04-22T18:41:26.854399+0000","last_became_peered":"2024-04-22T18:41:26.854399+0000","last_unstale":"2024-04-22T18:42:27.930444+0000","last_undegraded":"2024-04-22T18:42:27.930444+0000","last_fullsized":"2024-04-22T18:42:27.930444+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:12.520456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:12.520456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:42.305803+0000","scrub_duration":0.20146086999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"169'384","reported_seq":1628,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.868428+0000","last_change":"2024-04-22T18:42:17.393680+0000","last_active":"2024-04-22T18:42:26.868428+0000","last_peered":"2024-04-22T18:42:26.868428+0000","last_clean":"2024-04-22T18:42:26.868428+0000","last_became_active":"2024-04-22T18:41:26.994048+0000","last_became_peered":"2024-04-22T18:41:26.994048+0000","last_unstale":"2024-04-22T18:42:26.868428+0000","last_undegraded":"2024-04-22T18:42:26.868428+0000","last_fullsized":"2024-04-22T18:42:26.868428+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'384","last_scrub_stamp":"2024-04-22T18:42:17.393461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.393461+0000","objects_scrubbed":54,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:29.216372+0000","scrub_duration":0.111477038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41123046,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":358,"num_read_kb":4228,"num_write":477,"num_write_kb":113967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":82068207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":37,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:28.388274+0000","last_change":"2024-04-22T18:42:28.388472+0000","last_active":"2024-04-22T18:42:28.388274+0000","last_peered":"2024-04-22T18:42:28.388274+0000","last_clean":"2024-04-22T18:42:28.388274+0000","last_became_active":"2024-04-22T18:42:09.929115+0000","last_became_peered":"2024-04-22T18:42:09.929115+0000","last_unstale":"2024-04-22T18:42:28.388274+0000","last_undegraded":"2024-04-22T18:42:28.388274+0000","last_fullsized":"2024-04-22T18:42:28.388274+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:28.388238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:28.388238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:47.486386+0000","scrub_duration":0.203509897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"200'362","reported_seq":848,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.930444+0000","last_change":"2024-04-22T18:42:17.640545+0000","last_active":"2024-04-22T18:42:27.930444+0000","last_peered":"2024-04-22T18:42:27.930444+0000","last_clean":"2024-04-22T18:42:27.930444+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:42:27.930444+0000","last_undegraded":"2024-04-22T18:42:27.930444+0000","last_fullsized":"2024-04-22T18:42:27.930444+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'362","last_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_deep_scrub":"200'362","last_deep_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.640441+0000","objects_scrubbed":53,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:47.115778+0000","scrub_duration":3.4448620060000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382624,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":306,"num_read_kb":4987,"num_write":404,"num_write_kb":80545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":177,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.708851+0000","last_change":"2024-04-22T18:41:41.421087+0000","last_active":"2024-04-22T18:42:26.708851+0000","last_peered":"2024-04-22T18:42:26.708851+0000","last_clean":"2024-04-22T18:42:26.708851+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:42:26.708851+0000","last_undegraded":"2024-04-22T18:42:26.708851+0000","last_fullsized":"2024-04-22T18:42:26.708851+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:41.420996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:41.420996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:57.844851+0000","scrub_duration":0.20141637900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"168'337","reported_seq":1428,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.694108+0000","last_change":"2024-04-22T18:42:09.682366+0000","last_active":"2024-04-22T18:42:26.694108+0000","last_peered":"2024-04-22T18:42:26.694108+0000","last_clean":"2024-04-22T18:42:26.694108+0000","last_became_active":"2024-04-22T18:41:41.007340+0000","last_became_peered":"2024-04-22T18:41:41.007340+0000","last_unstale":"2024-04-22T18:42:26.694108+0000","last_undegraded":"2024-04-22T18:42:26.694108+0000","last_fullsized":"2024-04-22T18:42:26.694108+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'337","last_scrub_stamp":"2024-04-22T18:42:09.682178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:09.682178+0000","objects_scrubbed":54,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:19.217879+0000","scrub_duration":0.20999784699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39035424,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":322,"num_read_kb":9140,"num_write":423,"num_write_kb":104979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":67954083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":221,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.702504+0000","last_change":"2024-04-22T18:42:13.608651+0000","last_active":"2024-04-22T18:42:26.702504+0000","last_peered":"2024-04-22T18:42:26.702504+0000","last_clean":"2024-04-22T18:42:26.702504+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:42:26.702504+0000","last_undegraded":"2024-04-22T18:42:26.702504+0000","last_fullsized":"2024-04-22T18:42:26.702504+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_clean_scrub_stamp":"2024-04-22T18:42:13.608479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:21.626020+0000","scrub_duration":0.43950688900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"169'328","reported_seq":1340,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.324501+0000","last_change":"2024-04-22T18:42:27.323551+0000","last_active":"2024-04-22T18:42:27.324501+0000","last_peered":"2024-04-22T18:42:27.324501+0000","last_clean":"2024-04-22T18:42:27.324501+0000","last_became_active":"2024-04-22T18:42:18.879845+0000","last_became_peered":"2024-04-22T18:42:18.879845+0000","last_unstale":"2024-04-22T18:42:27.324501+0000","last_undegraded":"2024-04-22T18:42:27.324501+0000","last_fullsized":"2024-04-22T18:42:27.324501+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'328","last_scrub_stamp":"2024-04-22T18:42:06.716200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:06.716200+0000","objects_scrubbed":46,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:35.844741+0000","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43117026,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":302,"num_read_kb":12417,"num_write":418,"num_write_kb":91859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":72521286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.702021+0000","last_change":"2024-04-22T18:42:10.136809+0000","last_active":"2024-04-22T18:42:26.702021+0000","last_peered":"2024-04-22T18:42:26.702021+0000","last_clean":"2024-04-22T18:42:26.702021+0000","last_became_active":"2024-04-22T18:42:10.036239+0000","last_became_peered":"2024-04-22T18:42:10.036239+0000","last_unstale":"2024-04-22T18:42:26.702021+0000","last_undegraded":"2024-04-22T18:42:26.702021+0000","last_fullsized":"2024-04-22T18:42:26.702021+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:30.504686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:30.504686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:45.796059+0000","scrub_duration":0.30171718199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":416,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.802807+0000","last_change":"2024-04-22T18:41:56.270751+0000","last_active":"2024-04-22T18:42:26.802807+0000","last_peered":"2024-04-22T18:42:26.802807+0000","last_clean":"2024-04-22T18:42:26.802807+0000","last_became_active":"2024-04-22T18:41:26.757958+0000","last_became_peered":"2024-04-22T18:41:26.757958+0000","last_unstale":"2024-04-22T18:42:26.802807+0000","last_undegraded":"2024-04-22T18:42:26.802807+0000","last_fullsized":"2024-04-22T18:42:26.802807+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:41:56.270650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:41:56.270650+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:12.138524+0000","scrub_duration":0.202387964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":155658120,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":1288,"num_read_kb":30772,"num_write":1722,"num_write_kb":391350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1297,"num_bytes_recovered":243402900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249495552,"data_stored":263913197,"data_compressed":14626013,"data_compressed_allocated":15568896,"data_compressed_original":30834688,"omap_allocated":0,"internal_metadata":0},"log_size":1411,"ondisk_log_size":1411,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":2115712,"data_compressed":122550,"data_compressed_allocated":163840,"data_compressed_original":2084864,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36796,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335044,"statfs":{"total":96636764160,"available":96599085056,"internally_reserved":0,"allocated":1622016,"data_stored":1243070,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":2.5870000000000002,"15min":1.1439999999999999},"min":{"1min":0.182,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":26.568000000000001,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":6.3600000000000003,"5min":6.766,"15min":5.2599999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":95.918999999999997,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.49199999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":3.8050000000000002,"5min":3.984,"15min":4.5709999999999997},"min":{"1min":0.29299999999999998,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":66.814999999999998,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":6.6040000000000001,"5min":12.141,"15min":12.744999999999999},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":100.242,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7130000000000001,"5min":4.0730000000000004,"15min":4.0810000000000004},"min":{"1min":0.27400000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":97.662000000000006,"5min":97.662000000000006,"15min":97.662000000000006},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.441}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7560000000000002,"5min":4.1289999999999996,"15min":4.1429999999999998},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":97.686999999999998,"5min":97.686999999999998,"15min":97.686999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.59799999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":8.2089999999999996,"5min":5.4509999999999996,"15min":4.7889999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":78.149000000000001,"5min":85.509,"15min":85.509},"last":0.58799999999999997},{"interface":"front","average":{"1min":7.7130000000000001,"5min":16.856000000000002,"15min":20.866},"min":{"1min":0.26000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":97.590000000000003,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.61599999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":2.0270000000000001,"15min":2.7229999999999999},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.4730000000000001,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.6679999999999999,"5min":5.2939999999999996,"15min":6.5389999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":23.007999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.63}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":11.756,"15min":12.769},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":23.027000000000001,"5min":139.428,"15min":139.428},"last":0.55800000000000005},{"interface":"front","average":{"1min":7.7800000000000002,"5min":4.3070000000000004,"15min":4.1909999999999998},"min":{"1min":0.29199999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":97.635999999999996,"5min":97.635999999999996,"15min":97.635999999999996},"last":0.57099999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1527808,"data_stored":711890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.5030000000000001,"5min":3.0579999999999998,"15min":2.2149999999999999},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":70.436000000000007,"5min":70.436000000000007,"15min":70.436000000000007},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":16.753,"15min":22.789000000000001},"min":{"1min":0.26700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.58599999999999997,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.38100000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":3.577,"15min":2.5510000000000002},"min":{"1min":0.254,"5min":0.215,"15min":0.215},"max":{"1min":0.54000000000000004,"5min":199.047,"15min":199.047},"last":0.35699999999999998},{"interface":"front","average":{"1min":11.327999999999999,"5min":8.6519999999999992,"15min":9.6120000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":192.279,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.68799999999999994,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.442},{"interface":"front","average":{"1min":0.44600000000000001,"5min":12.56,"15min":16.722000000000001},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.59099999999999997,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.315}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":14.815,"15min":19.283999999999999},"min":{"1min":0.29599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65200000000000002,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":1.335,"15min":0.74299999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.875,"5min":77.097999999999999,"15min":77.097999999999999},"last":0.42999999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":2.052,"5min":5.0499999999999998,"15min":6.7519999999999998},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":29.978999999999999,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":10.577,"5min":16.312000000000001,"15min":18.321999999999999},"min":{"1min":0.27900000000000003,"5min":0.218,"15min":0.218},"max":{"1min":168.44,"5min":168.44,"15min":168.44},"last":0.45600000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.252,"15min":0.252},"max":{"1min":0.61199999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.28199999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":5.6289999999999996,"5min":14.432,"15min":17.378},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":90.649000000000001,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.165,"5min":0.60099999999999998,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.077,"5min":13.077,"15min":13.077},"last":0.47199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59188,"kb_used_data":28072,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94312652,"statfs":{"total":96636764160,"available":96576155648,"internally_reserved":0,"allocated":28745728,"data_stored":29405870,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3580000000000001,"5min":9.875,"15min":12.218999999999999},"min":{"1min":0.34999999999999998,"5min":0.192,"15min":0.192},"max":{"1min":58.421999999999997,"5min":74.019999999999996,"15min":74.019999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.71199999999999997,"15min":0.84099999999999997},"min":{"1min":0.34000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.52400000000000002,"5min":8.3919999999999995,"15min":8.3919999999999995},"last":0.52100000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.0259999999999998,"5min":11.034000000000001,"15min":10.702999999999999},"min":{"1min":0.28100000000000003,"5min":0.224,"15min":0.224},"max":{"1min":91.911000000000001,"5min":202.13800000000001,"15min":202.13800000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.3440000000000001,"15min":0.72699999999999998},"min":{"1min":0.34399999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.669,"5min":68.346999999999994,"15min":68.346999999999994},"last":0.48499999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.496,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":3.339,"15min":4.4720000000000004},"min":{"1min":0.377,"5min":0.308,"15min":0.308},"max":{"1min":0.57399999999999995,"5min":78.296999999999997,"15min":78.296999999999997},"last":0.50700000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":4.5499999999999998,"15min":5.3490000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.018,"5min":89.688000000000002,"15min":89.688000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46999999999999997,"5min":6.5380000000000003,"15min":9.0890000000000004},"min":{"1min":0.34899999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60899999999999999,"5min":89.718999999999994,"15min":89.718999999999994},"last":0.439}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.4809999999999999,"5min":15.465999999999999,"15min":16.969000000000001},"min":{"1min":0.249,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":100.544,"5min":202.983,"15min":202.983},"last":0.42499999999999999},{"interface":"front","average":{"1min":9.5050000000000008,"5min":14.603999999999999,"15min":16.722999999999999},"min":{"1min":0.314,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":100.56,"5min":202.999,"15min":202.999},"last":0.57699999999999996}]},{"osd":6,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":4.3739999999999997,"15min":4.8159999999999998},"min":{"1min":0.27100000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.59099999999999997,"5min":79.945999999999998,"15min":79.945999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.40799999999999997,"5min":5.274,"15min":5.9770000000000003},"min":{"1min":0.251,"5min":0.249,"15min":0.249},"max":{"1min":0.57299999999999995,"5min":100.54900000000001,"15min":100.54900000000001},"last":0.61199999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":1.149,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":71.602999999999994,"5min":71.602999999999994,"15min":71.602999999999994},"last":0.45500000000000002},{"interface":"front","average":{"1min":4.508,"5min":4.173,"15min":4.7770000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":75.436000000000007,"5min":78.340999999999994,"15min":78.340999999999994},"last":0.56100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89648,"kb_used_data":54440,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282192,"statfs":{"total":96636764160,"available":96544964608,"internally_reserved":0,"allocated":55746560,"data_stored":57813168,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":13.115,"15min":15.470000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":38.899999999999999,"5min":252.96199999999999,"15min":252.96199999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":9.282,"15min":8.8059999999999992},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63300000000000001,"5min":187.148,"15min":187.148},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":2.508,"15min":1.115},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":9.141,"5min":96.486999999999995,"15min":96.486999999999995},"last":0.48099999999999998},{"interface":"front","average":{"1min":22.393000000000001,"5min":11.545999999999999,"15min":7.4390000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":310.61500000000001,"5min":310.61500000000001,"15min":310.61500000000001},"last":0.53900000000000003}]},{"osd":2,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":8.8989999999999991,"15min":8.6790000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.8330000000000002,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.995,"5min":2.4199999999999999,"15min":1.1040000000000001},"min":{"1min":0.31900000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.1399999999999997,"5min":96.471999999999994,"15min":96.471999999999994},"last":0.57699999999999996}]},{"osd":3,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":11.172000000000001,"5min":8.9710000000000001,"15min":5.6710000000000003},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":122.102,"5min":252.91900000000001,"15min":252.91900000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.42999999999999999,"5min":17.242000000000001,"15min":19.725999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59899999999999998,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.79700000000000004}]},{"osd":5,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":2.3380000000000001,"15min":1.034},"min":{"1min":0.25800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":7.1559999999999997,"5min":96.408000000000001,"15min":96.408000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":2.6960000000000002,"5min":3.048,"15min":3.7080000000000002},"min":{"1min":0.36199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":38.581000000000003,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.73899999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":5.8289999999999997,"5min":14.667,"15min":13.662000000000001},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":87.706999999999994,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.83199999999999996,"5min":2.3820000000000001,"15min":1.0880000000000001},"min":{"1min":0.35699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.1159999999999997,"5min":96.438000000000002,"15min":96.438000000000002},"last":0.88100000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":9.5749999999999993,"15min":8.8979999999999997},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68000000000000005,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.9790000000000001,"5min":2.6419999999999999,"15min":1.202},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":19.518999999999998,"5min":96.456999999999994,"15min":96.456999999999994},"last":0.84499999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63332,"kb_used_data":28120,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94308508,"statfs":{"total":96636764160,"available":96571912192,"internally_reserved":0,"allocated":28794880,"data_stored":29410920,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":15.041,"5min":10.538,"15min":11.646000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":93.131,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.221},{"interface":"front","average":{"1min":7.077,"5min":1.6659999999999999,"15min":0.77400000000000002},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":87.305999999999997,"5min":87.305999999999997,"15min":87.305999999999997},"last":0.26700000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":10.558,"15min":6.8819999999999997},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":78.805000000000007,"5min":197.02099999999999,"15min":197.02099999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":8.9800000000000004,"5min":2.3210000000000002,"15min":0.997},"min":{"1min":0.20999999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":78.393000000000001,"5min":78.393000000000001,"15min":78.393000000000001},"last":0.317}]},{"osd":2,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":2.3439999999999999,"15min":1.452},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":47.534999999999997,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":6.4770000000000003,"5min":1.728,"15min":0.85899999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":78.421999999999997,"5min":78.421999999999997,"15min":78.421999999999997},"last":0.28699999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1800000000000002,"5min":2.9940000000000002,"15min":3.7360000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":26.141999999999999,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":4.5629999999999997,"5min":4.6289999999999996,"15min":4.3010000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":47.616999999999997,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.8279999999999998,"5min":2.5049999999999999,"15min":1.623},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":47.566000000000003,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.53100000000000003},{"interface":"front","average":{"1min":6.6379999999999999,"5min":2.2890000000000001,"15min":1.601},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.480999999999995,"5min":78.480999999999995,"15min":78.480999999999995},"last":0.54200000000000004}]},{"osd":6,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.50900000000000001,"15min":0.53100000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.748,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.501},{"interface":"front","average":{"1min":2.8500000000000001,"5min":2.871,"15min":1.3109999999999999},"min":{"1min":0.377,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":47.590000000000003,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.46000000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":6.6189999999999998,"5min":1.903,"15min":1.0229999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.453999999999994,"5min":78.453999999999994,"15min":78.453999999999994},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69299999999999995,"5min":0.995,"15min":0.995},"last":0.46999999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51620,"kb_used_data":20504,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94320220,"statfs":{"total":96636764160,"available":96583905280,"internally_reserved":0,"allocated":20996096,"data_stored":21643557,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":6.7439999999999998,"15min":7.6660000000000004},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":84.338999999999999,"5min":149.03,"15min":149.03},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.28299999999999997,"5min":1.3089999999999999,"15min":1.7589999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.62,"5min":31.866,"15min":31.866},"last":0.42899999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":9.4849999999999994,"5min":5.2489999999999997,"15min":2.4380000000000002},"min":{"1min":0.216,"5min":0.193,"15min":0.193},"max":{"1min":91.912000000000006,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":12.752000000000001,"5min":15.912000000000001,"15min":15.227},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":126.42,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.53300000000000003}]},{"osd":3,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":6.2830000000000004,"15min":2.3359999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":129.041,"5min":147.721,"15min":147.721},"last":1.179},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":101.43300000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":8.1280000000000001,"5min":13.57,"15min":17.038},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":90.882999999999996,"5min":160.584,"15min":160.584},"last":0.50700000000000001},{"interface":"front","average":{"1min":5.9349999999999996,"5min":4.8520000000000003,"15min":3.2490000000000001},"min":{"1min":0.32500000000000001,"5min":0.315,"15min":0.315},"max":{"1min":93.962999999999994,"5min":97.887,"15min":97.887},"last":0.68100000000000005}]},{"osd":5,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":2.0339999999999998,"15min":2.7120000000000002},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.879,"5min":48.631,"15min":48.631},"last":0.624},{"interface":"front","average":{"1min":0.50900000000000001,"5min":2.2890000000000001,"15min":3.0800000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.91800000000000004,"5min":56.18,"15min":56.18},"last":0.56599999999999995}]},{"osd":6,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.3540000000000001,"5min":3.4670000000000001,"15min":1.5129999999999999},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":92.028000000000006,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59799999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.29,"5min":1.5800000000000001,"15min":0.91000000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":90.944999999999993,"5min":90.944999999999993,"15min":90.944999999999993},"last":0.47599999999999998},{"interface":"front","average":{"1min":7.4640000000000004,"5min":3.923,"15min":1.6970000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":92.048000000000002,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.72899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59868,"kb_used_data":20560,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94311972,"statfs":{"total":96636764160,"available":96575459328,"internally_reserved":0,"allocated":21053440,"data_stored":21653890,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":5.883,"15min":7.2000000000000002},"min":{"1min":0.20000000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":27.116,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.28199999999999997},{"interface":"front","average":{"1min":5.8040000000000003,"5min":6.6870000000000003,"15min":7.4710000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":94.046000000000006,"5min":94.046000000000006,"15min":94.046000000000006},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.679000000000002,"5min":13.159000000000001,"15min":10.119},"min":{"1min":0.23000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":406.68799999999999,"5min":406.68799999999999,"15min":406.68799999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.8700000000000001,"5min":0.61899999999999999,"15min":0.41099999999999998},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19},"max":{"1min":27.094999999999999,"5min":27.094999999999999,"15min":27.094999999999999},"last":0.31}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":15.968,"5min":3.464,"15min":1.3759999999999999},"min":{"1min":0.29099999999999998,"5min":0.216,"15min":0.216},"max":{"1min":228.90100000000001,"5min":228.90100000000001,"15min":228.90100000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":21.481999999999999,"5min":13.061999999999999,"15min":14.015000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":196.82900000000001,"5min":231.864,"15min":231.864},"last":0.32100000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.823,"5min":17.213000000000001,"15min":14.393000000000001},"min":{"1min":0.316,"5min":0.309,"15min":0.309},"max":{"1min":406.71199999999999,"5min":406.71199999999999,"15min":406.71199999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":25.027000000000001,"5min":5.3979999999999997,"15min":2.1320000000000001},"min":{"1min":0.34499999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":186.89500000000001,"5min":186.89500000000001,"15min":186.89500000000001},"last":0.372}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":9.7430000000000003,"15min":12.628},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":94.183000000000007,"5min":138.089,"15min":138.089},"last":0.38},{"interface":"front","average":{"1min":39.863999999999997,"5min":20.117000000000001,"15min":20.09},"min":{"1min":0.35499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":406.72899999999998,"5min":406.72899999999998,"15min":406.72899999999998},"last":0.45500000000000002}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":6.0010000000000003,"5min":3.8359999999999999,"15min":3.1299999999999999},"min":{"1min":0.34000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":94.302999999999997,"5min":94.302999999999997,"15min":94.302999999999997},"last":0.48399999999999999},{"interface":"front","average":{"1min":43.036000000000001,"5min":19.713999999999999,"15min":17.591999999999999},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":427.69299999999998,"5min":427.69299999999998,"15min":427.69299999999998},"last":0.39600000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":2.0720000000000001,"5min":0.82799999999999996,"15min":0.626},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":27.253,"5min":27.253,"15min":27.253},"last":0.51000000000000001},{"interface":"front","average":{"1min":6.0090000000000003,"5min":3.129,"15min":3.069},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":94.260999999999996,"5min":94.260999999999996,"15min":94.260999999999996},"last":0.41699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89712,"kb_used_data":54492,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282128,"statfs":{"total":96636764160,"available":96544899072,"internally_reserved":0,"allocated":55799808,"data_stored":58342200,"data_compressed":3021454,"data_compressed_allocated":3239936,"data_compressed_original":6823936,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":9.7919999999999998,"5min":9.5660000000000007,"15min":7.6950000000000003},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":127.30500000000001,"5min":133.09899999999999,"15min":133.09899999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":27.553999999999998,"5min":11.506,"15min":4.8399999999999999},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":451.50799999999998,"5min":451.50799999999998,"15min":451.50799999999998},"last":0.376}]},{"osd":2,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":1.5309999999999999,"15min":1.5469999999999999},"min":{"1min":0.217,"5min":0.185,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":39.906999999999996,"15min":39.906999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.32600000000000001,"5min":2.2090000000000001,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.41299999999999998,"5min":94.820999999999998,"15min":94.820999999999998},"last":0.31900000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":6.1859999999999999,"5min":3.5299999999999998,"15min":1.486},"min":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":74.963999999999999,"5min":99.332999999999998,"15min":99.332999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":16.207000000000001,"5min":9.2159999999999993,"15min":4.1139999999999999},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":273.65100000000001,"5min":297.82100000000003,"15min":297.82100000000003},"last":0.42999999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":5.9980000000000002,"5min":3.4809999999999999,"15min":1.4970000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":96.528999999999996,"5min":96.528999999999996,"15min":96.528999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.45100000000000001,"5min":8.8000000000000007,"15min":9.5410000000000004},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":160.02199999999999,"15min":160.02199999999999},"last":0.71499999999999997}]},{"osd":5,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":2.1800000000000002,"15min":1.8899999999999999},"min":{"1min":0.35399999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.60399999999999998,"5min":41.645000000000003,"15min":41.645000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":10.452999999999999,"15min":12.787000000000001},"min":{"1min":0.32300000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66200000000000003,"5min":282.197,"15min":282.197},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.7770000000000001,"15min":1.385},"min":{"1min":0.36199999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58299999999999996,"5min":99.355000000000004,"15min":99.355000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.499,"5min":2.8929999999999998,"15min":2.129},"min":{"1min":0.35899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.72999999999999998,"5min":52.762,"15min":52.762},"last":0.55900000000000005}]},{"osd":7,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":4.8929999999999998,"15min":4.7489999999999997},"min":{"1min":0.34499999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.68700000000000006,"5min":94.662999999999997,"15min":94.662999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":9.0709999999999997,"5min":4.2939999999999996,"15min":1.901},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":157.089,"5min":157.089,"15min":157.089},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27324416,"data_stored":28849251,"data_compressed":1581247,"data_compressed_allocated":1675264,"data_compressed_original":3284992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:42:33.640 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-22T18:42:33.640 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-22T18:42:33.649 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T18:42:33.649 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:42:33.926 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:42:33.926 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:42:33.938 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-04-22T18:42:33.003110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":156248488,"num_objects":224,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1404,"num_read_kb":30872,"num_write":1881,"num_write_kb":393766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1303,"num_bytes_recovered":244989684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1528,"ondisk_log_size":1528,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":482772,"kb_used_data":209264,"kb_used_omap":12,"kb_used_meta":273395,"kb_avail":754491948,"statfs":{"total":773094113280,"available":772599754752,"internally_reserved":0,"allocated":214286336,"data_stored":220224565,"data_compressed":11796076,"data_compressed_allocated":12582912,"data_compressed_original":26013696,"omap_allocated":12744,"internal_metadata":279957048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-32,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":10680424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.331024"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":36,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.703285+0000","last_change":"2024-04-22T18:42:11.391813+0000","last_active":"2024-04-22T18:42:26.703285+0000","last_peered":"2024-04-22T18:42:26.703285+0000","last_clean":"2024-04-22T18:42:26.703285+0000","last_became_active":"2024-04-22T18:42:11.030639+0000","last_became_peered":"2024-04-22T18:42:11.030639+0000","last_unstale":"2024-04-22T18:42:26.703285+0000","last_undegraded":"2024-04-22T18:42:26.703285+0000","last_fullsized":"2024-04-22T18:42:26.703285+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.391633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.391633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:18.616280+0000","scrub_duration":0.102099431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":241,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.930536+0000","last_change":"2024-04-22T18:42:11.029787+0000","last_active":"2024-04-22T18:42:27.930536+0000","last_peered":"2024-04-22T18:42:27.930536+0000","last_clean":"2024-04-22T18:42:27.930536+0000","last_became_active":"2024-04-22T18:42:11.029453+0000","last_became_peered":"2024-04-22T18:42:11.029453+0000","last_unstale":"2024-04-22T18:42:27.930536+0000","last_undegraded":"2024-04-22T18:42:27.930536+0000","last_fullsized":"2024-04-22T18:42:27.930536+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:58.561808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:58.561808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:02.874625+0000","scrub_duration":0.301876478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":246,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.702491+0000","last_change":"2024-04-22T18:42:11.428432+0000","last_active":"2024-04-22T18:42:26.702491+0000","last_peered":"2024-04-22T18:42:26.702491+0000","last_clean":"2024-04-22T18:42:26.702491+0000","last_became_active":"2024-04-22T18:41:26.961835+0000","last_became_peered":"2024-04-22T18:41:26.961835+0000","last_unstale":"2024-04-22T18:42:26.702491+0000","last_undegraded":"2024-04-22T18:42:26.702491+0000","last_fullsized":"2024-04-22T18:42:26.702491+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.428262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.428262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:11.658873+0000","scrub_duration":0.30251553399999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":231,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.930444+0000","last_change":"2024-04-22T18:42:12.520547+0000","last_active":"2024-04-22T18:42:27.930444+0000","last_peered":"2024-04-22T18:42:27.930444+0000","last_clean":"2024-04-22T18:42:27.930444+0000","last_became_active":"2024-04-22T18:41:26.854399+0000","last_became_peered":"2024-04-22T18:41:26.854399+0000","last_unstale":"2024-04-22T18:42:27.930444+0000","last_undegraded":"2024-04-22T18:42:27.930444+0000","last_fullsized":"2024-04-22T18:42:27.930444+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:12.520456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:12.520456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:42.305803+0000","scrub_duration":0.20146086999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"169'384","reported_seq":1628,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.868428+0000","last_change":"2024-04-22T18:42:17.393680+0000","last_active":"2024-04-22T18:42:26.868428+0000","last_peered":"2024-04-22T18:42:26.868428+0000","last_clean":"2024-04-22T18:42:26.868428+0000","last_became_active":"2024-04-22T18:41:26.994048+0000","last_became_peered":"2024-04-22T18:41:26.994048+0000","last_unstale":"2024-04-22T18:42:26.868428+0000","last_undegraded":"2024-04-22T18:42:26.868428+0000","last_fullsized":"2024-04-22T18:42:26.868428+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'384","last_scrub_stamp":"2024-04-22T18:42:17.393461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.393461+0000","objects_scrubbed":54,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:29.216372+0000","scrub_duration":0.111477038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41123046,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":358,"num_read_kb":4228,"num_write":477,"num_write_kb":113967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":82068207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":37,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:28.388274+0000","last_change":"2024-04-22T18:42:28.388472+0000","last_active":"2024-04-22T18:42:28.388274+0000","last_peered":"2024-04-22T18:42:28.388274+0000","last_clean":"2024-04-22T18:42:28.388274+0000","last_became_active":"2024-04-22T18:42:09.929115+0000","last_became_peered":"2024-04-22T18:42:09.929115+0000","last_unstale":"2024-04-22T18:42:28.388274+0000","last_undegraded":"2024-04-22T18:42:28.388274+0000","last_fullsized":"2024-04-22T18:42:28.388274+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:28.388238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:28.388238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:47.486386+0000","scrub_duration":0.203509897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"200'362","reported_seq":848,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.930444+0000","last_change":"2024-04-22T18:42:17.640545+0000","last_active":"2024-04-22T18:42:27.930444+0000","last_peered":"2024-04-22T18:42:27.930444+0000","last_clean":"2024-04-22T18:42:27.930444+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:42:27.930444+0000","last_undegraded":"2024-04-22T18:42:27.930444+0000","last_fullsized":"2024-04-22T18:42:27.930444+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'362","last_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_deep_scrub":"200'362","last_deep_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.640441+0000","objects_scrubbed":53,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:47.115778+0000","scrub_duration":3.4448620060000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382624,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":306,"num_read_kb":4987,"num_write":404,"num_write_kb":80545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":177,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.708851+0000","last_change":"2024-04-22T18:41:41.421087+0000","last_active":"2024-04-22T18:42:26.708851+0000","last_peered":"2024-04-22T18:42:26.708851+0000","last_clean":"2024-04-22T18:42:26.708851+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:42:26.708851+0000","last_undegraded":"2024-04-22T18:42:26.708851+0000","last_fullsized":"2024-04-22T18:42:26.708851+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:41.420996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:41.420996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:57.844851+0000","scrub_duration":0.20141637900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"168'337","reported_seq":1428,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.694108+0000","last_change":"2024-04-22T18:42:09.682366+0000","last_active":"2024-04-22T18:42:26.694108+0000","last_peered":"2024-04-22T18:42:26.694108+0000","last_clean":"2024-04-22T18:42:26.694108+0000","last_became_active":"2024-04-22T18:41:41.007340+0000","last_became_peered":"2024-04-22T18:41:41.007340+0000","last_unstale":"2024-04-22T18:42:26.694108+0000","last_undegraded":"2024-04-22T18:42:26.694108+0000","last_fullsized":"2024-04-22T18:42:26.694108+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'337","last_scrub_stamp":"2024-04-22T18:42:09.682178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:09.682178+0000","objects_scrubbed":54,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:19.217879+0000","scrub_duration":0.20999784699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39035424,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":322,"num_read_kb":9140,"num_write":423,"num_write_kb":104979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":67954083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":221,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.702504+0000","last_change":"2024-04-22T18:42:13.608651+0000","last_active":"2024-04-22T18:42:26.702504+0000","last_peered":"2024-04-22T18:42:26.702504+0000","last_clean":"2024-04-22T18:42:26.702504+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:42:26.702504+0000","last_undegraded":"2024-04-22T18:42:26.702504+0000","last_fullsized":"2024-04-22T18:42:26.702504+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_clean_scrub_stamp":"2024-04-22T18:42:13.608479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:21.626020+0000","scrub_duration":0.43950688900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"169'328","reported_seq":1340,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.324501+0000","last_change":"2024-04-22T18:42:27.323551+0000","last_active":"2024-04-22T18:42:27.324501+0000","last_peered":"2024-04-22T18:42:27.324501+0000","last_clean":"2024-04-22T18:42:27.324501+0000","last_became_active":"2024-04-22T18:42:18.879845+0000","last_became_peered":"2024-04-22T18:42:18.879845+0000","last_unstale":"2024-04-22T18:42:27.324501+0000","last_undegraded":"2024-04-22T18:42:27.324501+0000","last_fullsized":"2024-04-22T18:42:27.324501+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'328","last_scrub_stamp":"2024-04-22T18:42:06.716200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:06.716200+0000","objects_scrubbed":46,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:35.844741+0000","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43117026,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":302,"num_read_kb":12417,"num_write":418,"num_write_kb":91859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":72521286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.702021+0000","last_change":"2024-04-22T18:42:10.136809+0000","last_active":"2024-04-22T18:42:26.702021+0000","last_peered":"2024-04-22T18:42:26.702021+0000","last_clean":"2024-04-22T18:42:26.702021+0000","last_became_active":"2024-04-22T18:42:10.036239+0000","last_became_peered":"2024-04-22T18:42:10.036239+0000","last_unstale":"2024-04-22T18:42:26.702021+0000","last_undegraded":"2024-04-22T18:42:26.702021+0000","last_fullsized":"2024-04-22T18:42:26.702021+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:30.504686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:30.504686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:45.796059+0000","scrub_duration":0.30171718199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":416,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.802807+0000","last_change":"2024-04-22T18:41:56.270751+0000","last_active":"2024-04-22T18:42:26.802807+0000","last_peered":"2024-04-22T18:42:26.802807+0000","last_clean":"2024-04-22T18:42:26.802807+0000","last_became_active":"2024-04-22T18:41:26.757958+0000","last_became_peered":"2024-04-22T18:41:26.757958+0000","last_unstale":"2024-04-22T18:42:26.802807+0000","last_undegraded":"2024-04-22T18:42:26.802807+0000","last_fullsized":"2024-04-22T18:42:26.802807+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:41:56.270650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:41:56.270650+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:12.138524+0000","scrub_duration":0.202387964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":155658120,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":1288,"num_read_kb":30772,"num_write":1722,"num_write_kb":391350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1297,"num_bytes_recovered":243402900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249495552,"data_stored":263913197,"data_compressed":14626013,"data_compressed_allocated":15568896,"data_compressed_original":30834688,"omap_allocated":0,"internal_metadata":0},"log_size":1411,"ondisk_log_size":1411,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":2115712,"data_compressed":122550,"data_compressed_allocated":163840,"data_compressed_original":2084864,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36796,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335044,"statfs":{"total":96636764160,"available":96599085056,"internally_reserved":0,"allocated":1622016,"data_stored":1243070,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":2.5870000000000002,"15min":1.1439999999999999},"min":{"1min":0.182,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":26.568000000000001,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":6.3600000000000003,"5min":6.766,"15min":5.2599999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":95.918999999999997,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.49199999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":3.8050000000000002,"5min":3.984,"15min":4.5709999999999997},"min":{"1min":0.29299999999999998,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":66.814999999999998,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":6.6040000000000001,"5min":12.141,"15min":12.744999999999999},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":100.242,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7130000000000001,"5min":4.0730000000000004,"15min":4.0810000000000004},"min":{"1min":0.27400000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":97.662000000000006,"5min":97.662000000000006,"15min":97.662000000000006},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.441}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7560000000000002,"5min":4.1289999999999996,"15min":4.1429999999999998},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":97.686999999999998,"5min":97.686999999999998,"15min":97.686999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.59799999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":8.2089999999999996,"5min":5.4509999999999996,"15min":4.7889999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":78.149000000000001,"5min":85.509,"15min":85.509},"last":0.58799999999999997},{"interface":"front","average":{"1min":7.7130000000000001,"5min":16.856000000000002,"15min":20.866},"min":{"1min":0.26000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":97.590000000000003,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.61599999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":2.0270000000000001,"15min":2.7229999999999999},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.4730000000000001,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.6679999999999999,"5min":5.2939999999999996,"15min":6.5389999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":23.007999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.63}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":11.756,"15min":12.769},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":23.027000000000001,"5min":139.428,"15min":139.428},"last":0.55800000000000005},{"interface":"front","average":{"1min":7.7800000000000002,"5min":4.3070000000000004,"15min":4.1909999999999998},"min":{"1min":0.29199999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":97.635999999999996,"5min":97.635999999999996,"15min":97.635999999999996},"last":0.57099999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1527808,"data_stored":711890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.5030000000000001,"5min":3.0579999999999998,"15min":2.2149999999999999},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":70.436000000000007,"5min":70.436000000000007,"15min":70.436000000000007},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":16.753,"15min":22.789000000000001},"min":{"1min":0.26700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.58599999999999997,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.38100000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":3.577,"15min":2.5510000000000002},"min":{"1min":0.254,"5min":0.215,"15min":0.215},"max":{"1min":0.54000000000000004,"5min":199.047,"15min":199.047},"last":0.35699999999999998},{"interface":"front","average":{"1min":11.327999999999999,"5min":8.6519999999999992,"15min":9.6120000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":192.279,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.68799999999999994,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.442},{"interface":"front","average":{"1min":0.44600000000000001,"5min":12.56,"15min":16.722000000000001},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.59099999999999997,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.315}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":14.815,"15min":19.283999999999999},"min":{"1min":0.29599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65200000000000002,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":1.335,"15min":0.74299999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.875,"5min":77.097999999999999,"15min":77.097999999999999},"last":0.42999999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":2.052,"5min":5.0499999999999998,"15min":6.7519999999999998},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":29.978999999999999,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":10.577,"5min":16.312000000000001,"15min":18.321999999999999},"min":{"1min":0.27900000000000003,"5min":0.218,"15min":0.218},"max":{"1min":168.44,"5min":168.44,"15min":168.44},"last":0.45600000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.252,"15min":0.252},"max":{"1min":0.61199999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.28199999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":5.6289999999999996,"5min":14.432,"15min":17.378},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":90.649000000000001,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.165,"5min":0.60099999999999998,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.077,"5min":13.077,"15min":13.077},"last":0.47199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59188,"kb_used_data":28072,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94312652,"statfs":{"total":96636764160,"available":96576155648,"internally_reserved":0,"allocated":28745728,"data_stored":29405870,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3580000000000001,"5min":9.875,"15min":12.218999999999999},"min":{"1min":0.34999999999999998,"5min":0.192,"15min":0.192},"max":{"1min":58.421999999999997,"5min":74.019999999999996,"15min":74.019999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.71199999999999997,"15min":0.84099999999999997},"min":{"1min":0.34000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.52400000000000002,"5min":8.3919999999999995,"15min":8.3919999999999995},"last":0.52100000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.0259999999999998,"5min":11.034000000000001,"15min":10.702999999999999},"min":{"1min":0.28100000000000003,"5min":0.224,"15min":0.224},"max":{"1min":91.911000000000001,"5min":202.13800000000001,"15min":202.13800000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.3440000000000001,"15min":0.72699999999999998},"min":{"1min":0.34399999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.669,"5min":68.346999999999994,"15min":68.346999999999994},"last":0.48499999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.496,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":3.339,"15min":4.4720000000000004},"min":{"1min":0.377,"5min":0.308,"15min":0.308},"max":{"1min":0.57399999999999995,"5min":78.296999999999997,"15min":78.296999999999997},"last":0.50700000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":4.5499999999999998,"15min":5.3490000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.018,"5min":89.688000000000002,"15min":89.688000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46999999999999997,"5min":6.5380000000000003,"15min":9.0890000000000004},"min":{"1min":0.34899999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60899999999999999,"5min":89.718999999999994,"15min":89.718999999999994},"last":0.439}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.4809999999999999,"5min":15.465999999999999,"15min":16.969000000000001},"min":{"1min":0.249,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":100.544,"5min":202.983,"15min":202.983},"last":0.42499999999999999},{"interface":"front","average":{"1min":9.5050000000000008,"5min":14.603999999999999,"15min":16.722999999999999},"min":{"1min":0.314,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":100.56,"5min":202.999,"15min":202.999},"last":0.57699999999999996}]},{"osd":6,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":4.3739999999999997,"15min":4.8159999999999998},"min":{"1min":0.27100000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.59099999999999997,"5min":79.945999999999998,"15min":79.945999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.40799999999999997,"5min":5.274,"15min":5.9770000000000003},"min":{"1min":0.251,"5min":0.249,"15min":0.249},"max":{"1min":0.57299999999999995,"5min":100.54900000000001,"15min":100.54900000000001},"last":0.61199999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":1.149,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":71.602999999999994,"5min":71.602999999999994,"15min":71.602999999999994},"last":0.45500000000000002},{"interface":"front","average":{"1min":4.508,"5min":4.173,"15min":4.7770000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":75.436000000000007,"5min":78.340999999999994,"15min":78.340999999999994},"last":0.56100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89648,"kb_used_data":54440,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282192,"statfs":{"total":96636764160,"available":96544964608,"internally_reserved":0,"allocated":55746560,"data_stored":57813168,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":13.115,"15min":15.470000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":38.899999999999999,"5min":252.96199999999999,"15min":252.96199999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":9.282,"15min":8.8059999999999992},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63300000000000001,"5min":187.148,"15min":187.148},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":2.508,"15min":1.115},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":9.141,"5min":96.486999999999995,"15min":96.486999999999995},"last":0.48099999999999998},{"interface":"front","average":{"1min":22.393000000000001,"5min":11.545999999999999,"15min":7.4390000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":310.61500000000001,"5min":310.61500000000001,"15min":310.61500000000001},"last":0.53900000000000003}]},{"osd":2,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":8.8989999999999991,"15min":8.6790000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.8330000000000002,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.995,"5min":2.4199999999999999,"15min":1.1040000000000001},"min":{"1min":0.31900000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.1399999999999997,"5min":96.471999999999994,"15min":96.471999999999994},"last":0.57699999999999996}]},{"osd":3,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":11.172000000000001,"5min":8.9710000000000001,"15min":5.6710000000000003},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":122.102,"5min":252.91900000000001,"15min":252.91900000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.42999999999999999,"5min":17.242000000000001,"15min":19.725999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59899999999999998,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.79700000000000004}]},{"osd":5,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":2.3380000000000001,"15min":1.034},"min":{"1min":0.25800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":7.1559999999999997,"5min":96.408000000000001,"15min":96.408000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":2.6960000000000002,"5min":3.048,"15min":3.7080000000000002},"min":{"1min":0.36199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":38.581000000000003,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.73899999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":5.8289999999999997,"5min":14.667,"15min":13.662000000000001},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":87.706999999999994,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.83199999999999996,"5min":2.3820000000000001,"15min":1.0880000000000001},"min":{"1min":0.35699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.1159999999999997,"5min":96.438000000000002,"15min":96.438000000000002},"last":0.88100000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":9.5749999999999993,"15min":8.8979999999999997},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68000000000000005,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.9790000000000001,"5min":2.6419999999999999,"15min":1.202},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":19.518999999999998,"5min":96.456999999999994,"15min":96.456999999999994},"last":0.84499999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63332,"kb_used_data":28120,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94308508,"statfs":{"total":96636764160,"available":96571912192,"internally_reserved":0,"allocated":28794880,"data_stored":29410920,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":15.041,"5min":10.538,"15min":11.646000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":93.131,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.221},{"interface":"front","average":{"1min":7.077,"5min":1.6659999999999999,"15min":0.77400000000000002},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":87.305999999999997,"5min":87.305999999999997,"15min":87.305999999999997},"last":0.26700000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":10.558,"15min":6.8819999999999997},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":78.805000000000007,"5min":197.02099999999999,"15min":197.02099999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":8.9800000000000004,"5min":2.3210000000000002,"15min":0.997},"min":{"1min":0.20999999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":78.393000000000001,"5min":78.393000000000001,"15min":78.393000000000001},"last":0.317}]},{"osd":2,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":2.3439999999999999,"15min":1.452},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":47.534999999999997,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":6.4770000000000003,"5min":1.728,"15min":0.85899999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":78.421999999999997,"5min":78.421999999999997,"15min":78.421999999999997},"last":0.28699999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1800000000000002,"5min":2.9940000000000002,"15min":3.7360000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":26.141999999999999,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":4.5629999999999997,"5min":4.6289999999999996,"15min":4.3010000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":47.616999999999997,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.8279999999999998,"5min":2.5049999999999999,"15min":1.623},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":47.566000000000003,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.53100000000000003},{"interface":"front","average":{"1min":6.6379999999999999,"5min":2.2890000000000001,"15min":1.601},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.480999999999995,"5min":78.480999999999995,"15min":78.480999999999995},"last":0.54200000000000004}]},{"osd":6,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.50900000000000001,"15min":0.53100000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.748,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.501},{"interface":"front","average":{"1min":2.8500000000000001,"5min":2.871,"15min":1.3109999999999999},"min":{"1min":0.377,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":47.590000000000003,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.46000000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":6.6189999999999998,"5min":1.903,"15min":1.0229999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.453999999999994,"5min":78.453999999999994,"15min":78.453999999999994},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69299999999999995,"5min":0.995,"15min":0.995},"last":0.46999999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51620,"kb_used_data":20504,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94320220,"statfs":{"total":96636764160,"available":96583905280,"internally_reserved":0,"allocated":20996096,"data_stored":21643557,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":6.7439999999999998,"15min":7.6660000000000004},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":84.338999999999999,"5min":149.03,"15min":149.03},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.28299999999999997,"5min":1.3089999999999999,"15min":1.7589999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.62,"5min":31.866,"15min":31.866},"last":0.42899999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":9.4849999999999994,"5min":5.2489999999999997,"15min":2.4380000000000002},"min":{"1min":0.216,"5min":0.193,"15min":0.193},"max":{"1min":91.912000000000006,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":12.752000000000001,"5min":15.912000000000001,"15min":15.227},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":126.42,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.53300000000000003}]},{"osd":3,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":6.2830000000000004,"15min":2.3359999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":129.041,"5min":147.721,"15min":147.721},"last":1.179},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":101.43300000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":8.1280000000000001,"5min":13.57,"15min":17.038},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":90.882999999999996,"5min":160.584,"15min":160.584},"last":0.50700000000000001},{"interface":"front","average":{"1min":5.9349999999999996,"5min":4.8520000000000003,"15min":3.2490000000000001},"min":{"1min":0.32500000000000001,"5min":0.315,"15min":0.315},"max":{"1min":93.962999999999994,"5min":97.887,"15min":97.887},"last":0.68100000000000005}]},{"osd":5,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":2.0339999999999998,"15min":2.7120000000000002},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.879,"5min":48.631,"15min":48.631},"last":0.624},{"interface":"front","average":{"1min":0.50900000000000001,"5min":2.2890000000000001,"15min":3.0800000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.91800000000000004,"5min":56.18,"15min":56.18},"last":0.56599999999999995}]},{"osd":6,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.3540000000000001,"5min":3.4670000000000001,"15min":1.5129999999999999},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":92.028000000000006,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59799999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.29,"5min":1.5800000000000001,"15min":0.91000000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":90.944999999999993,"5min":90.944999999999993,"15min":90.944999999999993},"last":0.47599999999999998},{"interface":"front","average":{"1min":7.4640000000000004,"5min":3.923,"15min":1.6970000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":92.048000000000002,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.72899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59868,"kb_used_data":20560,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94311972,"statfs":{"total":96636764160,"available":96575459328,"internally_reserved":0,"allocated":21053440,"data_stored":21653890,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":5.883,"15min":7.2000000000000002},"min":{"1min":0.20000000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":27.116,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.28199999999999997},{"interface":"front","average":{"1min":5.8040000000000003,"5min":6.6870000000000003,"15min":7.4710000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":94.046000000000006,"5min":94.046000000000006,"15min":94.046000000000006},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.679000000000002,"5min":13.159000000000001,"15min":10.119},"min":{"1min":0.23000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":406.68799999999999,"5min":406.68799999999999,"15min":406.68799999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.8700000000000001,"5min":0.61899999999999999,"15min":0.41099999999999998},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19},"max":{"1min":27.094999999999999,"5min":27.094999999999999,"15min":27.094999999999999},"last":0.31}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":15.968,"5min":3.464,"15min":1.3759999999999999},"min":{"1min":0.29099999999999998,"5min":0.216,"15min":0.216},"max":{"1min":228.90100000000001,"5min":228.90100000000001,"15min":228.90100000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":21.481999999999999,"5min":13.061999999999999,"15min":14.015000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":196.82900000000001,"5min":231.864,"15min":231.864},"last":0.32100000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.823,"5min":17.213000000000001,"15min":14.393000000000001},"min":{"1min":0.316,"5min":0.309,"15min":0.309},"max":{"1min":406.71199999999999,"5min":406.71199999999999,"15min":406.71199999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":25.027000000000001,"5min":5.3979999999999997,"15min":2.1320000000000001},"min":{"1min":0.34499999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":186.89500000000001,"5min":186.89500000000001,"15min":186.89500000000001},"last":0.372}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":9.7430000000000003,"15min":12.628},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":94.183000000000007,"5min":138.089,"15min":138.089},"last":0.38},{"interface":"front","average":{"1min":39.863999999999997,"5min":20.117000000000001,"15min":20.09},"min":{"1min":0.35499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":406.72899999999998,"5min":406.72899999999998,"15min":406.72899999999998},"last":0.45500000000000002}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":6.0010000000000003,"5min":3.8359999999999999,"15min":3.1299999999999999},"min":{"1min":0.34000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":94.302999999999997,"5min":94.302999999999997,"15min":94.302999999999997},"last":0.48399999999999999},{"interface":"front","average":{"1min":43.036000000000001,"5min":19.713999999999999,"15min":17.591999999999999},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":427.69299999999998,"5min":427.69299999999998,"15min":427.69299999999998},"last":0.39600000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":2.0720000000000001,"5min":0.82799999999999996,"15min":0.626},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":27.253,"5min":27.253,"15min":27.253},"last":0.51000000000000001},{"interface":"front","average":{"1min":6.0090000000000003,"5min":3.129,"15min":3.069},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":94.260999999999996,"5min":94.260999999999996,"15min":94.260999999999996},"last":0.41699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89712,"kb_used_data":54492,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282128,"statfs":{"total":96636764160,"available":96544899072,"internally_reserved":0,"allocated":55799808,"data_stored":58342200,"data_compressed":3021454,"data_compressed_allocated":3239936,"data_compressed_original":6823936,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":9.7919999999999998,"5min":9.5660000000000007,"15min":7.6950000000000003},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":127.30500000000001,"5min":133.09899999999999,"15min":133.09899999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":27.553999999999998,"5min":11.506,"15min":4.8399999999999999},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":451.50799999999998,"5min":451.50799999999998,"15min":451.50799999999998},"last":0.376}]},{"osd":2,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":1.5309999999999999,"15min":1.5469999999999999},"min":{"1min":0.217,"5min":0.185,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":39.906999999999996,"15min":39.906999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.32600000000000001,"5min":2.2090000000000001,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.41299999999999998,"5min":94.820999999999998,"15min":94.820999999999998},"last":0.31900000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":6.1859999999999999,"5min":3.5299999999999998,"15min":1.486},"min":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":74.963999999999999,"5min":99.332999999999998,"15min":99.332999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":16.207000000000001,"5min":9.2159999999999993,"15min":4.1139999999999999},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":273.65100000000001,"5min":297.82100000000003,"15min":297.82100000000003},"last":0.42999999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":5.9980000000000002,"5min":3.4809999999999999,"15min":1.4970000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":96.528999999999996,"5min":96.528999999999996,"15min":96.528999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.45100000000000001,"5min":8.8000000000000007,"15min":9.5410000000000004},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":160.02199999999999,"15min":160.02199999999999},"last":0.71499999999999997}]},{"osd":5,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":2.1800000000000002,"15min":1.8899999999999999},"min":{"1min":0.35399999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.60399999999999998,"5min":41.645000000000003,"15min":41.645000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":10.452999999999999,"15min":12.787000000000001},"min":{"1min":0.32300000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66200000000000003,"5min":282.197,"15min":282.197},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.7770000000000001,"15min":1.385},"min":{"1min":0.36199999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58299999999999996,"5min":99.355000000000004,"15min":99.355000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.499,"5min":2.8929999999999998,"15min":2.129},"min":{"1min":0.35899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.72999999999999998,"5min":52.762,"15min":52.762},"last":0.55900000000000005}]},{"osd":7,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":4.8929999999999998,"15min":4.7489999999999997},"min":{"1min":0.34499999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.68700000000000006,"5min":94.662999999999997,"15min":94.662999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":9.0709999999999997,"5min":4.2939999999999996,"15min":1.901},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":157.089,"5min":157.089,"15min":157.089},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27324416,"data_stored":28849251,"data_compressed":1581247,"data_compressed_allocated":1675264,"data_compressed_original":3284992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:42:33.940 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:42:34.215 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:42:34.216 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:42:34.227 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-04-22T18:42:33.003110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":156248488,"num_objects":224,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1404,"num_read_kb":30872,"num_write":1881,"num_write_kb":393766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1303,"num_bytes_recovered":244989684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1528,"ondisk_log_size":1528,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":482772,"kb_used_data":209264,"kb_used_omap":12,"kb_used_meta":273395,"kb_avail":754491948,"statfs":{"total":773094113280,"available":772599754752,"internally_reserved":0,"allocated":214286336,"data_stored":220224565,"data_compressed":11796076,"data_compressed_allocated":12582912,"data_compressed_original":26013696,"omap_allocated":12744,"internal_metadata":279957048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-32,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":10680424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.331024"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":36,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.703285+0000","last_change":"2024-04-22T18:42:11.391813+0000","last_active":"2024-04-22T18:42:26.703285+0000","last_peered":"2024-04-22T18:42:26.703285+0000","last_clean":"2024-04-22T18:42:26.703285+0000","last_became_active":"2024-04-22T18:42:11.030639+0000","last_became_peered":"2024-04-22T18:42:11.030639+0000","last_unstale":"2024-04-22T18:42:26.703285+0000","last_undegraded":"2024-04-22T18:42:26.703285+0000","last_fullsized":"2024-04-22T18:42:26.703285+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.391633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.391633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:18.616280+0000","scrub_duration":0.102099431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":241,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.930536+0000","last_change":"2024-04-22T18:42:11.029787+0000","last_active":"2024-04-22T18:42:27.930536+0000","last_peered":"2024-04-22T18:42:27.930536+0000","last_clean":"2024-04-22T18:42:27.930536+0000","last_became_active":"2024-04-22T18:42:11.029453+0000","last_became_peered":"2024-04-22T18:42:11.029453+0000","last_unstale":"2024-04-22T18:42:27.930536+0000","last_undegraded":"2024-04-22T18:42:27.930536+0000","last_fullsized":"2024-04-22T18:42:27.930536+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:58.561808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:58.561808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:02.874625+0000","scrub_duration":0.301876478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":246,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.702491+0000","last_change":"2024-04-22T18:42:11.428432+0000","last_active":"2024-04-22T18:42:26.702491+0000","last_peered":"2024-04-22T18:42:26.702491+0000","last_clean":"2024-04-22T18:42:26.702491+0000","last_became_active":"2024-04-22T18:41:26.961835+0000","last_became_peered":"2024-04-22T18:41:26.961835+0000","last_unstale":"2024-04-22T18:42:26.702491+0000","last_undegraded":"2024-04-22T18:42:26.702491+0000","last_fullsized":"2024-04-22T18:42:26.702491+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.428262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.428262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:11.658873+0000","scrub_duration":0.30251553399999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":231,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.930444+0000","last_change":"2024-04-22T18:42:12.520547+0000","last_active":"2024-04-22T18:42:27.930444+0000","last_peered":"2024-04-22T18:42:27.930444+0000","last_clean":"2024-04-22T18:42:27.930444+0000","last_became_active":"2024-04-22T18:41:26.854399+0000","last_became_peered":"2024-04-22T18:41:26.854399+0000","last_unstale":"2024-04-22T18:42:27.930444+0000","last_undegraded":"2024-04-22T18:42:27.930444+0000","last_fullsized":"2024-04-22T18:42:27.930444+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:12.520456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:12.520456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:42.305803+0000","scrub_duration":0.20146086999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"169'384","reported_seq":1628,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.868428+0000","last_change":"2024-04-22T18:42:17.393680+0000","last_active":"2024-04-22T18:42:26.868428+0000","last_peered":"2024-04-22T18:42:26.868428+0000","last_clean":"2024-04-22T18:42:26.868428+0000","last_became_active":"2024-04-22T18:41:26.994048+0000","last_became_peered":"2024-04-22T18:41:26.994048+0000","last_unstale":"2024-04-22T18:42:26.868428+0000","last_undegraded":"2024-04-22T18:42:26.868428+0000","last_fullsized":"2024-04-22T18:42:26.868428+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'384","last_scrub_stamp":"2024-04-22T18:42:17.393461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.393461+0000","objects_scrubbed":54,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:29.216372+0000","scrub_duration":0.111477038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41123046,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":358,"num_read_kb":4228,"num_write":477,"num_write_kb":113967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":82068207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":37,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:28.388274+0000","last_change":"2024-04-22T18:42:28.388472+0000","last_active":"2024-04-22T18:42:28.388274+0000","last_peered":"2024-04-22T18:42:28.388274+0000","last_clean":"2024-04-22T18:42:28.388274+0000","last_became_active":"2024-04-22T18:42:09.929115+0000","last_became_peered":"2024-04-22T18:42:09.929115+0000","last_unstale":"2024-04-22T18:42:28.388274+0000","last_undegraded":"2024-04-22T18:42:28.388274+0000","last_fullsized":"2024-04-22T18:42:28.388274+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:28.388238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:28.388238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:47.486386+0000","scrub_duration":0.203509897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"200'362","reported_seq":848,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.930444+0000","last_change":"2024-04-22T18:42:17.640545+0000","last_active":"2024-04-22T18:42:27.930444+0000","last_peered":"2024-04-22T18:42:27.930444+0000","last_clean":"2024-04-22T18:42:27.930444+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:42:27.930444+0000","last_undegraded":"2024-04-22T18:42:27.930444+0000","last_fullsized":"2024-04-22T18:42:27.930444+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'362","last_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_deep_scrub":"200'362","last_deep_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.640441+0000","objects_scrubbed":53,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:47.115778+0000","scrub_duration":3.4448620060000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382624,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":306,"num_read_kb":4987,"num_write":404,"num_write_kb":80545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":177,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.708851+0000","last_change":"2024-04-22T18:41:41.421087+0000","last_active":"2024-04-22T18:42:26.708851+0000","last_peered":"2024-04-22T18:42:26.708851+0000","last_clean":"2024-04-22T18:42:26.708851+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:42:26.708851+0000","last_undegraded":"2024-04-22T18:42:26.708851+0000","last_fullsized":"2024-04-22T18:42:26.708851+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:41.420996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:41.420996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:57.844851+0000","scrub_duration":0.20141637900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"168'337","reported_seq":1428,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.694108+0000","last_change":"2024-04-22T18:42:09.682366+0000","last_active":"2024-04-22T18:42:26.694108+0000","last_peered":"2024-04-22T18:42:26.694108+0000","last_clean":"2024-04-22T18:42:26.694108+0000","last_became_active":"2024-04-22T18:41:41.007340+0000","last_became_peered":"2024-04-22T18:41:41.007340+0000","last_unstale":"2024-04-22T18:42:26.694108+0000","last_undegraded":"2024-04-22T18:42:26.694108+0000","last_fullsized":"2024-04-22T18:42:26.694108+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'337","last_scrub_stamp":"2024-04-22T18:42:09.682178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:09.682178+0000","objects_scrubbed":54,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:19.217879+0000","scrub_duration":0.20999784699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39035424,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":322,"num_read_kb":9140,"num_write":423,"num_write_kb":104979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":67954083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":221,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.702504+0000","last_change":"2024-04-22T18:42:13.608651+0000","last_active":"2024-04-22T18:42:26.702504+0000","last_peered":"2024-04-22T18:42:26.702504+0000","last_clean":"2024-04-22T18:42:26.702504+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:42:26.702504+0000","last_undegraded":"2024-04-22T18:42:26.702504+0000","last_fullsized":"2024-04-22T18:42:26.702504+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_clean_scrub_stamp":"2024-04-22T18:42:13.608479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:21.626020+0000","scrub_duration":0.43950688900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"169'328","reported_seq":1340,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.324501+0000","last_change":"2024-04-22T18:42:27.323551+0000","last_active":"2024-04-22T18:42:27.324501+0000","last_peered":"2024-04-22T18:42:27.324501+0000","last_clean":"2024-04-22T18:42:27.324501+0000","last_became_active":"2024-04-22T18:42:18.879845+0000","last_became_peered":"2024-04-22T18:42:18.879845+0000","last_unstale":"2024-04-22T18:42:27.324501+0000","last_undegraded":"2024-04-22T18:42:27.324501+0000","last_fullsized":"2024-04-22T18:42:27.324501+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'328","last_scrub_stamp":"2024-04-22T18:42:06.716200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:06.716200+0000","objects_scrubbed":46,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:35.844741+0000","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43117026,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":302,"num_read_kb":12417,"num_write":418,"num_write_kb":91859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":72521286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.702021+0000","last_change":"2024-04-22T18:42:10.136809+0000","last_active":"2024-04-22T18:42:26.702021+0000","last_peered":"2024-04-22T18:42:26.702021+0000","last_clean":"2024-04-22T18:42:26.702021+0000","last_became_active":"2024-04-22T18:42:10.036239+0000","last_became_peered":"2024-04-22T18:42:10.036239+0000","last_unstale":"2024-04-22T18:42:26.702021+0000","last_undegraded":"2024-04-22T18:42:26.702021+0000","last_fullsized":"2024-04-22T18:42:26.702021+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:30.504686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:30.504686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:45.796059+0000","scrub_duration":0.30171718199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":416,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.802807+0000","last_change":"2024-04-22T18:41:56.270751+0000","last_active":"2024-04-22T18:42:26.802807+0000","last_peered":"2024-04-22T18:42:26.802807+0000","last_clean":"2024-04-22T18:42:26.802807+0000","last_became_active":"2024-04-22T18:41:26.757958+0000","last_became_peered":"2024-04-22T18:41:26.757958+0000","last_unstale":"2024-04-22T18:42:26.802807+0000","last_undegraded":"2024-04-22T18:42:26.802807+0000","last_fullsized":"2024-04-22T18:42:26.802807+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:41:56.270650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:41:56.270650+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:12.138524+0000","scrub_duration":0.202387964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":155658120,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":1288,"num_read_kb":30772,"num_write":1722,"num_write_kb":391350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1297,"num_bytes_recovered":243402900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249495552,"data_stored":263913197,"data_compressed":14626013,"data_compressed_allocated":15568896,"data_compressed_original":30834688,"omap_allocated":0,"internal_metadata":0},"log_size":1411,"ondisk_log_size":1411,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":2115712,"data_compressed":122550,"data_compressed_allocated":163840,"data_compressed_original":2084864,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36796,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335044,"statfs":{"total":96636764160,"available":96599085056,"internally_reserved":0,"allocated":1622016,"data_stored":1243070,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":2.5870000000000002,"15min":1.1439999999999999},"min":{"1min":0.182,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":26.568000000000001,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":6.3600000000000003,"5min":6.766,"15min":5.2599999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":95.918999999999997,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.49199999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":3.8050000000000002,"5min":3.984,"15min":4.5709999999999997},"min":{"1min":0.29299999999999998,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":66.814999999999998,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":6.6040000000000001,"5min":12.141,"15min":12.744999999999999},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":100.242,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7130000000000001,"5min":4.0730000000000004,"15min":4.0810000000000004},"min":{"1min":0.27400000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":97.662000000000006,"5min":97.662000000000006,"15min":97.662000000000006},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.441}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7560000000000002,"5min":4.1289999999999996,"15min":4.1429999999999998},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":97.686999999999998,"5min":97.686999999999998,"15min":97.686999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.59799999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":8.2089999999999996,"5min":5.4509999999999996,"15min":4.7889999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":78.149000000000001,"5min":85.509,"15min":85.509},"last":0.58799999999999997},{"interface":"front","average":{"1min":7.7130000000000001,"5min":16.856000000000002,"15min":20.866},"min":{"1min":0.26000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":97.590000000000003,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.61599999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":2.0270000000000001,"15min":2.7229999999999999},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.4730000000000001,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.6679999999999999,"5min":5.2939999999999996,"15min":6.5389999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":23.007999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.63}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":11.756,"15min":12.769},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":23.027000000000001,"5min":139.428,"15min":139.428},"last":0.55800000000000005},{"interface":"front","average":{"1min":7.7800000000000002,"5min":4.3070000000000004,"15min":4.1909999999999998},"min":{"1min":0.29199999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":97.635999999999996,"5min":97.635999999999996,"15min":97.635999999999996},"last":0.57099999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1527808,"data_stored":711890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.5030000000000001,"5min":3.0579999999999998,"15min":2.2149999999999999},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":70.436000000000007,"5min":70.436000000000007,"15min":70.436000000000007},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":16.753,"15min":22.789000000000001},"min":{"1min":0.26700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.58599999999999997,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.38100000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":3.577,"15min":2.5510000000000002},"min":{"1min":0.254,"5min":0.215,"15min":0.215},"max":{"1min":0.54000000000000004,"5min":199.047,"15min":199.047},"last":0.35699999999999998},{"interface":"front","average":{"1min":11.327999999999999,"5min":8.6519999999999992,"15min":9.6120000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":192.279,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.68799999999999994,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.442},{"interface":"front","average":{"1min":0.44600000000000001,"5min":12.56,"15min":16.722000000000001},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.59099999999999997,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.315}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":14.815,"15min":19.283999999999999},"min":{"1min":0.29599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65200000000000002,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":1.335,"15min":0.74299999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.875,"5min":77.097999999999999,"15min":77.097999999999999},"last":0.42999999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":2.052,"5min":5.0499999999999998,"15min":6.7519999999999998},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":29.978999999999999,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":10.577,"5min":16.312000000000001,"15min":18.321999999999999},"min":{"1min":0.27900000000000003,"5min":0.218,"15min":0.218},"max":{"1min":168.44,"5min":168.44,"15min":168.44},"last":0.45600000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.252,"15min":0.252},"max":{"1min":0.61199999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.28199999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":5.6289999999999996,"5min":14.432,"15min":17.378},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":90.649000000000001,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.165,"5min":0.60099999999999998,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.077,"5min":13.077,"15min":13.077},"last":0.47199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59188,"kb_used_data":28072,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94312652,"statfs":{"total":96636764160,"available":96576155648,"internally_reserved":0,"allocated":28745728,"data_stored":29405870,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3580000000000001,"5min":9.875,"15min":12.218999999999999},"min":{"1min":0.34999999999999998,"5min":0.192,"15min":0.192},"max":{"1min":58.421999999999997,"5min":74.019999999999996,"15min":74.019999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.71199999999999997,"15min":0.84099999999999997},"min":{"1min":0.34000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.52400000000000002,"5min":8.3919999999999995,"15min":8.3919999999999995},"last":0.52100000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.0259999999999998,"5min":11.034000000000001,"15min":10.702999999999999},"min":{"1min":0.28100000000000003,"5min":0.224,"15min":0.224},"max":{"1min":91.911000000000001,"5min":202.13800000000001,"15min":202.13800000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.3440000000000001,"15min":0.72699999999999998},"min":{"1min":0.34399999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.669,"5min":68.346999999999994,"15min":68.346999999999994},"last":0.48499999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.496,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":3.339,"15min":4.4720000000000004},"min":{"1min":0.377,"5min":0.308,"15min":0.308},"max":{"1min":0.57399999999999995,"5min":78.296999999999997,"15min":78.296999999999997},"last":0.50700000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":4.5499999999999998,"15min":5.3490000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.018,"5min":89.688000000000002,"15min":89.688000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46999999999999997,"5min":6.5380000000000003,"15min":9.0890000000000004},"min":{"1min":0.34899999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60899999999999999,"5min":89.718999999999994,"15min":89.718999999999994},"last":0.439}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.4809999999999999,"5min":15.465999999999999,"15min":16.969000000000001},"min":{"1min":0.249,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":100.544,"5min":202.983,"15min":202.983},"last":0.42499999999999999},{"interface":"front","average":{"1min":9.5050000000000008,"5min":14.603999999999999,"15min":16.722999999999999},"min":{"1min":0.314,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":100.56,"5min":202.999,"15min":202.999},"last":0.57699999999999996}]},{"osd":6,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":4.3739999999999997,"15min":4.8159999999999998},"min":{"1min":0.27100000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.59099999999999997,"5min":79.945999999999998,"15min":79.945999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.40799999999999997,"5min":5.274,"15min":5.9770000000000003},"min":{"1min":0.251,"5min":0.249,"15min":0.249},"max":{"1min":0.57299999999999995,"5min":100.54900000000001,"15min":100.54900000000001},"last":0.61199999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":1.149,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":71.602999999999994,"5min":71.602999999999994,"15min":71.602999999999994},"last":0.45500000000000002},{"interface":"front","average":{"1min":4.508,"5min":4.173,"15min":4.7770000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":75.436000000000007,"5min":78.340999999999994,"15min":78.340999999999994},"last":0.56100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89648,"kb_used_data":54440,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282192,"statfs":{"total":96636764160,"available":96544964608,"internally_reserved":0,"allocated":55746560,"data_stored":57813168,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":13.115,"15min":15.470000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":38.899999999999999,"5min":252.96199999999999,"15min":252.96199999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":9.282,"15min":8.8059999999999992},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63300000000000001,"5min":187.148,"15min":187.148},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":2.508,"15min":1.115},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":9.141,"5min":96.486999999999995,"15min":96.486999999999995},"last":0.48099999999999998},{"interface":"front","average":{"1min":22.393000000000001,"5min":11.545999999999999,"15min":7.4390000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":310.61500000000001,"5min":310.61500000000001,"15min":310.61500000000001},"last":0.53900000000000003}]},{"osd":2,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":8.8989999999999991,"15min":8.6790000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.8330000000000002,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.995,"5min":2.4199999999999999,"15min":1.1040000000000001},"min":{"1min":0.31900000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.1399999999999997,"5min":96.471999999999994,"15min":96.471999999999994},"last":0.57699999999999996}]},{"osd":3,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":11.172000000000001,"5min":8.9710000000000001,"15min":5.6710000000000003},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":122.102,"5min":252.91900000000001,"15min":252.91900000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.42999999999999999,"5min":17.242000000000001,"15min":19.725999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59899999999999998,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.79700000000000004}]},{"osd":5,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":2.3380000000000001,"15min":1.034},"min":{"1min":0.25800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":7.1559999999999997,"5min":96.408000000000001,"15min":96.408000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":2.6960000000000002,"5min":3.048,"15min":3.7080000000000002},"min":{"1min":0.36199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":38.581000000000003,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.73899999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":5.8289999999999997,"5min":14.667,"15min":13.662000000000001},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":87.706999999999994,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.83199999999999996,"5min":2.3820000000000001,"15min":1.0880000000000001},"min":{"1min":0.35699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.1159999999999997,"5min":96.438000000000002,"15min":96.438000000000002},"last":0.88100000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":9.5749999999999993,"15min":8.8979999999999997},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68000000000000005,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.9790000000000001,"5min":2.6419999999999999,"15min":1.202},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":19.518999999999998,"5min":96.456999999999994,"15min":96.456999999999994},"last":0.84499999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63332,"kb_used_data":28120,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94308508,"statfs":{"total":96636764160,"available":96571912192,"internally_reserved":0,"allocated":28794880,"data_stored":29410920,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":15.041,"5min":10.538,"15min":11.646000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":93.131,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.221},{"interface":"front","average":{"1min":7.077,"5min":1.6659999999999999,"15min":0.77400000000000002},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":87.305999999999997,"5min":87.305999999999997,"15min":87.305999999999997},"last":0.26700000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":10.558,"15min":6.8819999999999997},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":78.805000000000007,"5min":197.02099999999999,"15min":197.02099999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":8.9800000000000004,"5min":2.3210000000000002,"15min":0.997},"min":{"1min":0.20999999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":78.393000000000001,"5min":78.393000000000001,"15min":78.393000000000001},"last":0.317}]},{"osd":2,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":2.3439999999999999,"15min":1.452},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":47.534999999999997,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":6.4770000000000003,"5min":1.728,"15min":0.85899999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":78.421999999999997,"5min":78.421999999999997,"15min":78.421999999999997},"last":0.28699999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1800000000000002,"5min":2.9940000000000002,"15min":3.7360000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":26.141999999999999,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":4.5629999999999997,"5min":4.6289999999999996,"15min":4.3010000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":47.616999999999997,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.8279999999999998,"5min":2.5049999999999999,"15min":1.623},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":47.566000000000003,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.53100000000000003},{"interface":"front","average":{"1min":6.6379999999999999,"5min":2.2890000000000001,"15min":1.601},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.480999999999995,"5min":78.480999999999995,"15min":78.480999999999995},"last":0.54200000000000004}]},{"osd":6,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.50900000000000001,"15min":0.53100000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.748,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.501},{"interface":"front","average":{"1min":2.8500000000000001,"5min":2.871,"15min":1.3109999999999999},"min":{"1min":0.377,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":47.590000000000003,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.46000000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":6.6189999999999998,"5min":1.903,"15min":1.0229999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.453999999999994,"5min":78.453999999999994,"15min":78.453999999999994},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69299999999999995,"5min":0.995,"15min":0.995},"last":0.46999999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51620,"kb_used_data":20504,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94320220,"statfs":{"total":96636764160,"available":96583905280,"internally_reserved":0,"allocated":20996096,"data_stored":21643557,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":6.7439999999999998,"15min":7.6660000000000004},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":84.338999999999999,"5min":149.03,"15min":149.03},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.28299999999999997,"5min":1.3089999999999999,"15min":1.7589999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.62,"5min":31.866,"15min":31.866},"last":0.42899999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":9.4849999999999994,"5min":5.2489999999999997,"15min":2.4380000000000002},"min":{"1min":0.216,"5min":0.193,"15min":0.193},"max":{"1min":91.912000000000006,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":12.752000000000001,"5min":15.912000000000001,"15min":15.227},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":126.42,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.53300000000000003}]},{"osd":3,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":6.2830000000000004,"15min":2.3359999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":129.041,"5min":147.721,"15min":147.721},"last":1.179},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":101.43300000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":8.1280000000000001,"5min":13.57,"15min":17.038},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":90.882999999999996,"5min":160.584,"15min":160.584},"last":0.50700000000000001},{"interface":"front","average":{"1min":5.9349999999999996,"5min":4.8520000000000003,"15min":3.2490000000000001},"min":{"1min":0.32500000000000001,"5min":0.315,"15min":0.315},"max":{"1min":93.962999999999994,"5min":97.887,"15min":97.887},"last":0.68100000000000005}]},{"osd":5,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":2.0339999999999998,"15min":2.7120000000000002},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.879,"5min":48.631,"15min":48.631},"last":0.624},{"interface":"front","average":{"1min":0.50900000000000001,"5min":2.2890000000000001,"15min":3.0800000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.91800000000000004,"5min":56.18,"15min":56.18},"last":0.56599999999999995}]},{"osd":6,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.3540000000000001,"5min":3.4670000000000001,"15min":1.5129999999999999},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":92.028000000000006,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59799999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.29,"5min":1.5800000000000001,"15min":0.91000000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":90.944999999999993,"5min":90.944999999999993,"15min":90.944999999999993},"last":0.47599999999999998},{"interface":"front","average":{"1min":7.4640000000000004,"5min":3.923,"15min":1.6970000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":92.048000000000002,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.72899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59868,"kb_used_data":20560,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94311972,"statfs":{"total":96636764160,"available":96575459328,"internally_reserved":0,"allocated":21053440,"data_stored":21653890,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":5.883,"15min":7.2000000000000002},"min":{"1min":0.20000000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":27.116,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.28199999999999997},{"interface":"front","average":{"1min":5.8040000000000003,"5min":6.6870000000000003,"15min":7.4710000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":94.046000000000006,"5min":94.046000000000006,"15min":94.046000000000006},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.679000000000002,"5min":13.159000000000001,"15min":10.119},"min":{"1min":0.23000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":406.68799999999999,"5min":406.68799999999999,"15min":406.68799999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.8700000000000001,"5min":0.61899999999999999,"15min":0.41099999999999998},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19},"max":{"1min":27.094999999999999,"5min":27.094999999999999,"15min":27.094999999999999},"last":0.31}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":15.968,"5min":3.464,"15min":1.3759999999999999},"min":{"1min":0.29099999999999998,"5min":0.216,"15min":0.216},"max":{"1min":228.90100000000001,"5min":228.90100000000001,"15min":228.90100000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":21.481999999999999,"5min":13.061999999999999,"15min":14.015000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":196.82900000000001,"5min":231.864,"15min":231.864},"last":0.32100000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.823,"5min":17.213000000000001,"15min":14.393000000000001},"min":{"1min":0.316,"5min":0.309,"15min":0.309},"max":{"1min":406.71199999999999,"5min":406.71199999999999,"15min":406.71199999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":25.027000000000001,"5min":5.3979999999999997,"15min":2.1320000000000001},"min":{"1min":0.34499999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":186.89500000000001,"5min":186.89500000000001,"15min":186.89500000000001},"last":0.372}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":9.7430000000000003,"15min":12.628},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":94.183000000000007,"5min":138.089,"15min":138.089},"last":0.38},{"interface":"front","average":{"1min":39.863999999999997,"5min":20.117000000000001,"15min":20.09},"min":{"1min":0.35499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":406.72899999999998,"5min":406.72899999999998,"15min":406.72899999999998},"last":0.45500000000000002}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":6.0010000000000003,"5min":3.8359999999999999,"15min":3.1299999999999999},"min":{"1min":0.34000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":94.302999999999997,"5min":94.302999999999997,"15min":94.302999999999997},"last":0.48399999999999999},{"interface":"front","average":{"1min":43.036000000000001,"5min":19.713999999999999,"15min":17.591999999999999},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":427.69299999999998,"5min":427.69299999999998,"15min":427.69299999999998},"last":0.39600000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":2.0720000000000001,"5min":0.82799999999999996,"15min":0.626},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":27.253,"5min":27.253,"15min":27.253},"last":0.51000000000000001},{"interface":"front","average":{"1min":6.0090000000000003,"5min":3.129,"15min":3.069},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":94.260999999999996,"5min":94.260999999999996,"15min":94.260999999999996},"last":0.41699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89712,"kb_used_data":54492,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282128,"statfs":{"total":96636764160,"available":96544899072,"internally_reserved":0,"allocated":55799808,"data_stored":58342200,"data_compressed":3021454,"data_compressed_allocated":3239936,"data_compressed_original":6823936,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":9.7919999999999998,"5min":9.5660000000000007,"15min":7.6950000000000003},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":127.30500000000001,"5min":133.09899999999999,"15min":133.09899999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":27.553999999999998,"5min":11.506,"15min":4.8399999999999999},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":451.50799999999998,"5min":451.50799999999998,"15min":451.50799999999998},"last":0.376}]},{"osd":2,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":1.5309999999999999,"15min":1.5469999999999999},"min":{"1min":0.217,"5min":0.185,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":39.906999999999996,"15min":39.906999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.32600000000000001,"5min":2.2090000000000001,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.41299999999999998,"5min":94.820999999999998,"15min":94.820999999999998},"last":0.31900000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":6.1859999999999999,"5min":3.5299999999999998,"15min":1.486},"min":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":74.963999999999999,"5min":99.332999999999998,"15min":99.332999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":16.207000000000001,"5min":9.2159999999999993,"15min":4.1139999999999999},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":273.65100000000001,"5min":297.82100000000003,"15min":297.82100000000003},"last":0.42999999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":5.9980000000000002,"5min":3.4809999999999999,"15min":1.4970000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":96.528999999999996,"5min":96.528999999999996,"15min":96.528999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.45100000000000001,"5min":8.8000000000000007,"15min":9.5410000000000004},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":160.02199999999999,"15min":160.02199999999999},"last":0.71499999999999997}]},{"osd":5,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":2.1800000000000002,"15min":1.8899999999999999},"min":{"1min":0.35399999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.60399999999999998,"5min":41.645000000000003,"15min":41.645000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":10.452999999999999,"15min":12.787000000000001},"min":{"1min":0.32300000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66200000000000003,"5min":282.197,"15min":282.197},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.7770000000000001,"15min":1.385},"min":{"1min":0.36199999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58299999999999996,"5min":99.355000000000004,"15min":99.355000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.499,"5min":2.8929999999999998,"15min":2.129},"min":{"1min":0.35899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.72999999999999998,"5min":52.762,"15min":52.762},"last":0.55900000000000005}]},{"osd":7,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":4.8929999999999998,"15min":4.7489999999999997},"min":{"1min":0.34499999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.68700000000000006,"5min":94.662999999999997,"15min":94.662999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":9.0709999999999997,"5min":4.2939999999999996,"15min":1.901},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":157.089,"5min":157.089,"15min":157.089},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27324416,"data_stored":28849251,"data_compressed":1581247,"data_compressed_allocated":1675264,"data_compressed_original":3284992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:42:34.229 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T18:42:34.229 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:42:34.508 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:42:34.509 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:42:34.520 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-04-22T18:42:33.003110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":156248488,"num_objects":224,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1404,"num_read_kb":30872,"num_write":1881,"num_write_kb":393766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1303,"num_bytes_recovered":244989684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1528,"ondisk_log_size":1528,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":482772,"kb_used_data":209264,"kb_used_omap":12,"kb_used_meta":273395,"kb_avail":754491948,"statfs":{"total":773094113280,"available":772599754752,"internally_reserved":0,"allocated":214286336,"data_stored":220224565,"data_compressed":11796076,"data_compressed_allocated":12582912,"data_compressed_original":26013696,"omap_allocated":12744,"internal_metadata":279957048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-32,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":10680424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.331024"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":36,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.703285+0000","last_change":"2024-04-22T18:42:11.391813+0000","last_active":"2024-04-22T18:42:26.703285+0000","last_peered":"2024-04-22T18:42:26.703285+0000","last_clean":"2024-04-22T18:42:26.703285+0000","last_became_active":"2024-04-22T18:42:11.030639+0000","last_became_peered":"2024-04-22T18:42:11.030639+0000","last_unstale":"2024-04-22T18:42:26.703285+0000","last_undegraded":"2024-04-22T18:42:26.703285+0000","last_fullsized":"2024-04-22T18:42:26.703285+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.391633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.391633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:18.616280+0000","scrub_duration":0.102099431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":241,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.930536+0000","last_change":"2024-04-22T18:42:11.029787+0000","last_active":"2024-04-22T18:42:27.930536+0000","last_peered":"2024-04-22T18:42:27.930536+0000","last_clean":"2024-04-22T18:42:27.930536+0000","last_became_active":"2024-04-22T18:42:11.029453+0000","last_became_peered":"2024-04-22T18:42:11.029453+0000","last_unstale":"2024-04-22T18:42:27.930536+0000","last_undegraded":"2024-04-22T18:42:27.930536+0000","last_fullsized":"2024-04-22T18:42:27.930536+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:58.561808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:58.561808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:02.874625+0000","scrub_duration":0.301876478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":246,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.702491+0000","last_change":"2024-04-22T18:42:11.428432+0000","last_active":"2024-04-22T18:42:26.702491+0000","last_peered":"2024-04-22T18:42:26.702491+0000","last_clean":"2024-04-22T18:42:26.702491+0000","last_became_active":"2024-04-22T18:41:26.961835+0000","last_became_peered":"2024-04-22T18:41:26.961835+0000","last_unstale":"2024-04-22T18:42:26.702491+0000","last_undegraded":"2024-04-22T18:42:26.702491+0000","last_fullsized":"2024-04-22T18:42:26.702491+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.428262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.428262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:11.658873+0000","scrub_duration":0.30251553399999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":231,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.930444+0000","last_change":"2024-04-22T18:42:12.520547+0000","last_active":"2024-04-22T18:42:27.930444+0000","last_peered":"2024-04-22T18:42:27.930444+0000","last_clean":"2024-04-22T18:42:27.930444+0000","last_became_active":"2024-04-22T18:41:26.854399+0000","last_became_peered":"2024-04-22T18:41:26.854399+0000","last_unstale":"2024-04-22T18:42:27.930444+0000","last_undegraded":"2024-04-22T18:42:27.930444+0000","last_fullsized":"2024-04-22T18:42:27.930444+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:12.520456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:12.520456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:42.305803+0000","scrub_duration":0.20146086999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"169'384","reported_seq":1628,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.868428+0000","last_change":"2024-04-22T18:42:17.393680+0000","last_active":"2024-04-22T18:42:26.868428+0000","last_peered":"2024-04-22T18:42:26.868428+0000","last_clean":"2024-04-22T18:42:26.868428+0000","last_became_active":"2024-04-22T18:41:26.994048+0000","last_became_peered":"2024-04-22T18:41:26.994048+0000","last_unstale":"2024-04-22T18:42:26.868428+0000","last_undegraded":"2024-04-22T18:42:26.868428+0000","last_fullsized":"2024-04-22T18:42:26.868428+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'384","last_scrub_stamp":"2024-04-22T18:42:17.393461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.393461+0000","objects_scrubbed":54,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:29.216372+0000","scrub_duration":0.111477038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41123046,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":358,"num_read_kb":4228,"num_write":477,"num_write_kb":113967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":82068207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":37,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:28.388274+0000","last_change":"2024-04-22T18:42:28.388472+0000","last_active":"2024-04-22T18:42:28.388274+0000","last_peered":"2024-04-22T18:42:28.388274+0000","last_clean":"2024-04-22T18:42:28.388274+0000","last_became_active":"2024-04-22T18:42:09.929115+0000","last_became_peered":"2024-04-22T18:42:09.929115+0000","last_unstale":"2024-04-22T18:42:28.388274+0000","last_undegraded":"2024-04-22T18:42:28.388274+0000","last_fullsized":"2024-04-22T18:42:28.388274+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:28.388238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:28.388238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:47.486386+0000","scrub_duration":0.203509897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"200'362","reported_seq":848,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.930444+0000","last_change":"2024-04-22T18:42:17.640545+0000","last_active":"2024-04-22T18:42:27.930444+0000","last_peered":"2024-04-22T18:42:27.930444+0000","last_clean":"2024-04-22T18:42:27.930444+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:42:27.930444+0000","last_undegraded":"2024-04-22T18:42:27.930444+0000","last_fullsized":"2024-04-22T18:42:27.930444+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'362","last_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_deep_scrub":"200'362","last_deep_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.640441+0000","objects_scrubbed":53,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:47.115778+0000","scrub_duration":3.4448620060000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382624,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":306,"num_read_kb":4987,"num_write":404,"num_write_kb":80545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":177,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.708851+0000","last_change":"2024-04-22T18:41:41.421087+0000","last_active":"2024-04-22T18:42:26.708851+0000","last_peered":"2024-04-22T18:42:26.708851+0000","last_clean":"2024-04-22T18:42:26.708851+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:42:26.708851+0000","last_undegraded":"2024-04-22T18:42:26.708851+0000","last_fullsized":"2024-04-22T18:42:26.708851+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:41.420996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:41.420996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:57.844851+0000","scrub_duration":0.20141637900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"168'337","reported_seq":1428,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.694108+0000","last_change":"2024-04-22T18:42:09.682366+0000","last_active":"2024-04-22T18:42:26.694108+0000","last_peered":"2024-04-22T18:42:26.694108+0000","last_clean":"2024-04-22T18:42:26.694108+0000","last_became_active":"2024-04-22T18:41:41.007340+0000","last_became_peered":"2024-04-22T18:41:41.007340+0000","last_unstale":"2024-04-22T18:42:26.694108+0000","last_undegraded":"2024-04-22T18:42:26.694108+0000","last_fullsized":"2024-04-22T18:42:26.694108+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'337","last_scrub_stamp":"2024-04-22T18:42:09.682178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:09.682178+0000","objects_scrubbed":54,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:19.217879+0000","scrub_duration":0.20999784699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39035424,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":322,"num_read_kb":9140,"num_write":423,"num_write_kb":104979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":67954083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":221,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.702504+0000","last_change":"2024-04-22T18:42:13.608651+0000","last_active":"2024-04-22T18:42:26.702504+0000","last_peered":"2024-04-22T18:42:26.702504+0000","last_clean":"2024-04-22T18:42:26.702504+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:42:26.702504+0000","last_undegraded":"2024-04-22T18:42:26.702504+0000","last_fullsized":"2024-04-22T18:42:26.702504+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_clean_scrub_stamp":"2024-04-22T18:42:13.608479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:21.626020+0000","scrub_duration":0.43950688900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"169'328","reported_seq":1340,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:27.324501+0000","last_change":"2024-04-22T18:42:27.323551+0000","last_active":"2024-04-22T18:42:27.324501+0000","last_peered":"2024-04-22T18:42:27.324501+0000","last_clean":"2024-04-22T18:42:27.324501+0000","last_became_active":"2024-04-22T18:42:18.879845+0000","last_became_peered":"2024-04-22T18:42:18.879845+0000","last_unstale":"2024-04-22T18:42:27.324501+0000","last_undegraded":"2024-04-22T18:42:27.324501+0000","last_fullsized":"2024-04-22T18:42:27.324501+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'328","last_scrub_stamp":"2024-04-22T18:42:06.716200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:06.716200+0000","objects_scrubbed":46,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:35.844741+0000","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43117026,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":302,"num_read_kb":12417,"num_write":418,"num_write_kb":91859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":72521286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.702021+0000","last_change":"2024-04-22T18:42:10.136809+0000","last_active":"2024-04-22T18:42:26.702021+0000","last_peered":"2024-04-22T18:42:26.702021+0000","last_clean":"2024-04-22T18:42:26.702021+0000","last_became_active":"2024-04-22T18:42:10.036239+0000","last_became_peered":"2024-04-22T18:42:10.036239+0000","last_unstale":"2024-04-22T18:42:26.702021+0000","last_undegraded":"2024-04-22T18:42:26.702021+0000","last_fullsized":"2024-04-22T18:42:26.702021+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:30.504686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:30.504686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:45.796059+0000","scrub_duration":0.30171718199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":416,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-04-22T18:42:26.802807+0000","last_change":"2024-04-22T18:41:56.270751+0000","last_active":"2024-04-22T18:42:26.802807+0000","last_peered":"2024-04-22T18:42:26.802807+0000","last_clean":"2024-04-22T18:42:26.802807+0000","last_became_active":"2024-04-22T18:41:26.757958+0000","last_became_peered":"2024-04-22T18:41:26.757958+0000","last_unstale":"2024-04-22T18:42:26.802807+0000","last_undegraded":"2024-04-22T18:42:26.802807+0000","last_fullsized":"2024-04-22T18:42:26.802807+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:41:56.270650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:41:56.270650+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:12.138524+0000","scrub_duration":0.202387964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":155658120,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":1288,"num_read_kb":30772,"num_write":1722,"num_write_kb":391350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1297,"num_bytes_recovered":243402900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249495552,"data_stored":263913197,"data_compressed":14626013,"data_compressed_allocated":15568896,"data_compressed_original":30834688,"omap_allocated":0,"internal_metadata":0},"log_size":1411,"ondisk_log_size":1411,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":2115712,"data_compressed":122550,"data_compressed_allocated":163840,"data_compressed_original":2084864,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36796,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335044,"statfs":{"total":96636764160,"available":96599085056,"internally_reserved":0,"allocated":1622016,"data_stored":1243070,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":2.5870000000000002,"15min":1.1439999999999999},"min":{"1min":0.182,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":26.568000000000001,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":6.3600000000000003,"5min":6.766,"15min":5.2599999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":95.918999999999997,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.49199999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":3.8050000000000002,"5min":3.984,"15min":4.5709999999999997},"min":{"1min":0.29299999999999998,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":66.814999999999998,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":6.6040000000000001,"5min":12.141,"15min":12.744999999999999},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":100.242,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7130000000000001,"5min":4.0730000000000004,"15min":4.0810000000000004},"min":{"1min":0.27400000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":97.662000000000006,"5min":97.662000000000006,"15min":97.662000000000006},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.441}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7560000000000002,"5min":4.1289999999999996,"15min":4.1429999999999998},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":97.686999999999998,"5min":97.686999999999998,"15min":97.686999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.59799999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":8.2089999999999996,"5min":5.4509999999999996,"15min":4.7889999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":78.149000000000001,"5min":85.509,"15min":85.509},"last":0.58799999999999997},{"interface":"front","average":{"1min":7.7130000000000001,"5min":16.856000000000002,"15min":20.866},"min":{"1min":0.26000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":97.590000000000003,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.61599999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":2.0270000000000001,"15min":2.7229999999999999},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.4730000000000001,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.6679999999999999,"5min":5.2939999999999996,"15min":6.5389999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":23.007999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.63}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":11.756,"15min":12.769},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":23.027000000000001,"5min":139.428,"15min":139.428},"last":0.55800000000000005},{"interface":"front","average":{"1min":7.7800000000000002,"5min":4.3070000000000004,"15min":4.1909999999999998},"min":{"1min":0.29199999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":97.635999999999996,"5min":97.635999999999996,"15min":97.635999999999996},"last":0.57099999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1527808,"data_stored":711890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.5030000000000001,"5min":3.0579999999999998,"15min":2.2149999999999999},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":70.436000000000007,"5min":70.436000000000007,"15min":70.436000000000007},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":16.753,"15min":22.789000000000001},"min":{"1min":0.26700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.58599999999999997,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.38100000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":3.577,"15min":2.5510000000000002},"min":{"1min":0.254,"5min":0.215,"15min":0.215},"max":{"1min":0.54000000000000004,"5min":199.047,"15min":199.047},"last":0.35699999999999998},{"interface":"front","average":{"1min":11.327999999999999,"5min":8.6519999999999992,"15min":9.6120000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":192.279,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.68799999999999994,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.442},{"interface":"front","average":{"1min":0.44600000000000001,"5min":12.56,"15min":16.722000000000001},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.59099999999999997,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.315}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":14.815,"15min":19.283999999999999},"min":{"1min":0.29599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65200000000000002,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":1.335,"15min":0.74299999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.875,"5min":77.097999999999999,"15min":77.097999999999999},"last":0.42999999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":2.052,"5min":5.0499999999999998,"15min":6.7519999999999998},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":29.978999999999999,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":10.577,"5min":16.312000000000001,"15min":18.321999999999999},"min":{"1min":0.27900000000000003,"5min":0.218,"15min":0.218},"max":{"1min":168.44,"5min":168.44,"15min":168.44},"last":0.45600000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.252,"15min":0.252},"max":{"1min":0.61199999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.28199999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":5.6289999999999996,"5min":14.432,"15min":17.378},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":90.649000000000001,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.165,"5min":0.60099999999999998,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.077,"5min":13.077,"15min":13.077},"last":0.47199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59188,"kb_used_data":28072,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94312652,"statfs":{"total":96636764160,"available":96576155648,"internally_reserved":0,"allocated":28745728,"data_stored":29405870,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3580000000000001,"5min":9.875,"15min":12.218999999999999},"min":{"1min":0.34999999999999998,"5min":0.192,"15min":0.192},"max":{"1min":58.421999999999997,"5min":74.019999999999996,"15min":74.019999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.71199999999999997,"15min":0.84099999999999997},"min":{"1min":0.34000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.52400000000000002,"5min":8.3919999999999995,"15min":8.3919999999999995},"last":0.52100000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.0259999999999998,"5min":11.034000000000001,"15min":10.702999999999999},"min":{"1min":0.28100000000000003,"5min":0.224,"15min":0.224},"max":{"1min":91.911000000000001,"5min":202.13800000000001,"15min":202.13800000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.3440000000000001,"15min":0.72699999999999998},"min":{"1min":0.34399999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.669,"5min":68.346999999999994,"15min":68.346999999999994},"last":0.48499999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.496,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":3.339,"15min":4.4720000000000004},"min":{"1min":0.377,"5min":0.308,"15min":0.308},"max":{"1min":0.57399999999999995,"5min":78.296999999999997,"15min":78.296999999999997},"last":0.50700000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":4.5499999999999998,"15min":5.3490000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.018,"5min":89.688000000000002,"15min":89.688000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46999999999999997,"5min":6.5380000000000003,"15min":9.0890000000000004},"min":{"1min":0.34899999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60899999999999999,"5min":89.718999999999994,"15min":89.718999999999994},"last":0.439}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.4809999999999999,"5min":15.465999999999999,"15min":16.969000000000001},"min":{"1min":0.249,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":100.544,"5min":202.983,"15min":202.983},"last":0.42499999999999999},{"interface":"front","average":{"1min":9.5050000000000008,"5min":14.603999999999999,"15min":16.722999999999999},"min":{"1min":0.314,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":100.56,"5min":202.999,"15min":202.999},"last":0.57699999999999996}]},{"osd":6,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":4.3739999999999997,"15min":4.8159999999999998},"min":{"1min":0.27100000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.59099999999999997,"5min":79.945999999999998,"15min":79.945999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.40799999999999997,"5min":5.274,"15min":5.9770000000000003},"min":{"1min":0.251,"5min":0.249,"15min":0.249},"max":{"1min":0.57299999999999995,"5min":100.54900000000001,"15min":100.54900000000001},"last":0.61199999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":1.149,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":71.602999999999994,"5min":71.602999999999994,"15min":71.602999999999994},"last":0.45500000000000002},{"interface":"front","average":{"1min":4.508,"5min":4.173,"15min":4.7770000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":75.436000000000007,"5min":78.340999999999994,"15min":78.340999999999994},"last":0.56100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89648,"kb_used_data":54440,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282192,"statfs":{"total":96636764160,"available":96544964608,"internally_reserved":0,"allocated":55746560,"data_stored":57813168,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":13.115,"15min":15.470000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":38.899999999999999,"5min":252.96199999999999,"15min":252.96199999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":9.282,"15min":8.8059999999999992},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63300000000000001,"5min":187.148,"15min":187.148},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":2.508,"15min":1.115},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":9.141,"5min":96.486999999999995,"15min":96.486999999999995},"last":0.48099999999999998},{"interface":"front","average":{"1min":22.393000000000001,"5min":11.545999999999999,"15min":7.4390000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":310.61500000000001,"5min":310.61500000000001,"15min":310.61500000000001},"last":0.53900000000000003}]},{"osd":2,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":8.8989999999999991,"15min":8.6790000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.8330000000000002,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.995,"5min":2.4199999999999999,"15min":1.1040000000000001},"min":{"1min":0.31900000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.1399999999999997,"5min":96.471999999999994,"15min":96.471999999999994},"last":0.57699999999999996}]},{"osd":3,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":11.172000000000001,"5min":8.9710000000000001,"15min":5.6710000000000003},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":122.102,"5min":252.91900000000001,"15min":252.91900000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.42999999999999999,"5min":17.242000000000001,"15min":19.725999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59899999999999998,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.79700000000000004}]},{"osd":5,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":2.3380000000000001,"15min":1.034},"min":{"1min":0.25800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":7.1559999999999997,"5min":96.408000000000001,"15min":96.408000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":2.6960000000000002,"5min":3.048,"15min":3.7080000000000002},"min":{"1min":0.36199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":38.581000000000003,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.73899999999999999}]},{"osd":6,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":5.8289999999999997,"5min":14.667,"15min":13.662000000000001},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":87.706999999999994,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.83199999999999996,"5min":2.3820000000000001,"15min":1.0880000000000001},"min":{"1min":0.35699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.1159999999999997,"5min":96.438000000000002,"15min":96.438000000000002},"last":0.88100000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":9.5749999999999993,"15min":8.8979999999999997},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68000000000000005,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.9790000000000001,"5min":2.6419999999999999,"15min":1.202},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":19.518999999999998,"5min":96.456999999999994,"15min":96.456999999999994},"last":0.84499999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63332,"kb_used_data":28120,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94308508,"statfs":{"total":96636764160,"available":96571912192,"internally_reserved":0,"allocated":28794880,"data_stored":29410920,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":15.041,"5min":10.538,"15min":11.646000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":93.131,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.221},{"interface":"front","average":{"1min":7.077,"5min":1.6659999999999999,"15min":0.77400000000000002},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":87.305999999999997,"5min":87.305999999999997,"15min":87.305999999999997},"last":0.26700000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":10.558,"15min":6.8819999999999997},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":78.805000000000007,"5min":197.02099999999999,"15min":197.02099999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":8.9800000000000004,"5min":2.3210000000000002,"15min":0.997},"min":{"1min":0.20999999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":78.393000000000001,"5min":78.393000000000001,"15min":78.393000000000001},"last":0.317}]},{"osd":2,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":2.3439999999999999,"15min":1.452},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":47.534999999999997,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":6.4770000000000003,"5min":1.728,"15min":0.85899999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":78.421999999999997,"5min":78.421999999999997,"15min":78.421999999999997},"last":0.28699999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1800000000000002,"5min":2.9940000000000002,"15min":3.7360000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":26.141999999999999,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":4.5629999999999997,"5min":4.6289999999999996,"15min":4.3010000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":47.616999999999997,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.8279999999999998,"5min":2.5049999999999999,"15min":1.623},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":47.566000000000003,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.53100000000000003},{"interface":"front","average":{"1min":6.6379999999999999,"5min":2.2890000000000001,"15min":1.601},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.480999999999995,"5min":78.480999999999995,"15min":78.480999999999995},"last":0.54200000000000004}]},{"osd":6,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.50900000000000001,"15min":0.53100000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.748,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.501},{"interface":"front","average":{"1min":2.8500000000000001,"5min":2.871,"15min":1.3109999999999999},"min":{"1min":0.377,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":47.590000000000003,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.46000000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":6.6189999999999998,"5min":1.903,"15min":1.0229999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.453999999999994,"5min":78.453999999999994,"15min":78.453999999999994},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69299999999999995,"5min":0.995,"15min":0.995},"last":0.46999999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51620,"kb_used_data":20504,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94320220,"statfs":{"total":96636764160,"available":96583905280,"internally_reserved":0,"allocated":20996096,"data_stored":21643557,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":6.7439999999999998,"15min":7.6660000000000004},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":84.338999999999999,"5min":149.03,"15min":149.03},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.28299999999999997,"5min":1.3089999999999999,"15min":1.7589999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.62,"5min":31.866,"15min":31.866},"last":0.42899999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":9.4849999999999994,"5min":5.2489999999999997,"15min":2.4380000000000002},"min":{"1min":0.216,"5min":0.193,"15min":0.193},"max":{"1min":91.912000000000006,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":12.752000000000001,"5min":15.912000000000001,"15min":15.227},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":126.42,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.53300000000000003}]},{"osd":3,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":6.2830000000000004,"15min":2.3359999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":129.041,"5min":147.721,"15min":147.721},"last":1.179},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":101.43300000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":8.1280000000000001,"5min":13.57,"15min":17.038},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":90.882999999999996,"5min":160.584,"15min":160.584},"last":0.50700000000000001},{"interface":"front","average":{"1min":5.9349999999999996,"5min":4.8520000000000003,"15min":3.2490000000000001},"min":{"1min":0.32500000000000001,"5min":0.315,"15min":0.315},"max":{"1min":93.962999999999994,"5min":97.887,"15min":97.887},"last":0.68100000000000005}]},{"osd":5,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":2.0339999999999998,"15min":2.7120000000000002},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.879,"5min":48.631,"15min":48.631},"last":0.624},{"interface":"front","average":{"1min":0.50900000000000001,"5min":2.2890000000000001,"15min":3.0800000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.91800000000000004,"5min":56.18,"15min":56.18},"last":0.56599999999999995}]},{"osd":6,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.3540000000000001,"5min":3.4670000000000001,"15min":1.5129999999999999},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":92.028000000000006,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59799999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.29,"5min":1.5800000000000001,"15min":0.91000000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":90.944999999999993,"5min":90.944999999999993,"15min":90.944999999999993},"last":0.47599999999999998},{"interface":"front","average":{"1min":7.4640000000000004,"5min":3.923,"15min":1.6970000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":92.048000000000002,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.72899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59868,"kb_used_data":20560,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94311972,"statfs":{"total":96636764160,"available":96575459328,"internally_reserved":0,"allocated":21053440,"data_stored":21653890,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":5.883,"15min":7.2000000000000002},"min":{"1min":0.20000000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":27.116,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.28199999999999997},{"interface":"front","average":{"1min":5.8040000000000003,"5min":6.6870000000000003,"15min":7.4710000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":94.046000000000006,"5min":94.046000000000006,"15min":94.046000000000006},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.679000000000002,"5min":13.159000000000001,"15min":10.119},"min":{"1min":0.23000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":406.68799999999999,"5min":406.68799999999999,"15min":406.68799999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.8700000000000001,"5min":0.61899999999999999,"15min":0.41099999999999998},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19},"max":{"1min":27.094999999999999,"5min":27.094999999999999,"15min":27.094999999999999},"last":0.31}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":15.968,"5min":3.464,"15min":1.3759999999999999},"min":{"1min":0.29099999999999998,"5min":0.216,"15min":0.216},"max":{"1min":228.90100000000001,"5min":228.90100000000001,"15min":228.90100000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":21.481999999999999,"5min":13.061999999999999,"15min":14.015000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":196.82900000000001,"5min":231.864,"15min":231.864},"last":0.32100000000000001}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.823,"5min":17.213000000000001,"15min":14.393000000000001},"min":{"1min":0.316,"5min":0.309,"15min":0.309},"max":{"1min":406.71199999999999,"5min":406.71199999999999,"15min":406.71199999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":25.027000000000001,"5min":5.3979999999999997,"15min":2.1320000000000001},"min":{"1min":0.34499999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":186.89500000000001,"5min":186.89500000000001,"15min":186.89500000000001},"last":0.372}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":9.7430000000000003,"15min":12.628},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":94.183000000000007,"5min":138.089,"15min":138.089},"last":0.38},{"interface":"front","average":{"1min":39.863999999999997,"5min":20.117000000000001,"15min":20.09},"min":{"1min":0.35499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":406.72899999999998,"5min":406.72899999999998,"15min":406.72899999999998},"last":0.45500000000000002}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":6.0010000000000003,"5min":3.8359999999999999,"15min":3.1299999999999999},"min":{"1min":0.34000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":94.302999999999997,"5min":94.302999999999997,"15min":94.302999999999997},"last":0.48399999999999999},{"interface":"front","average":{"1min":43.036000000000001,"5min":19.713999999999999,"15min":17.591999999999999},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":427.69299999999998,"5min":427.69299999999998,"15min":427.69299999999998},"last":0.39600000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":2.0720000000000001,"5min":0.82799999999999996,"15min":0.626},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":27.253,"5min":27.253,"15min":27.253},"last":0.51000000000000001},{"interface":"front","average":{"1min":6.0090000000000003,"5min":3.129,"15min":3.069},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":94.260999999999996,"5min":94.260999999999996,"15min":94.260999999999996},"last":0.41699999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89712,"kb_used_data":54492,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282128,"statfs":{"total":96636764160,"available":96544899072,"internally_reserved":0,"allocated":55799808,"data_stored":58342200,"data_compressed":3021454,"data_compressed_allocated":3239936,"data_compressed_original":6823936,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":9.7919999999999998,"5min":9.5660000000000007,"15min":7.6950000000000003},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":127.30500000000001,"5min":133.09899999999999,"15min":133.09899999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":27.553999999999998,"5min":11.506,"15min":4.8399999999999999},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":451.50799999999998,"5min":451.50799999999998,"15min":451.50799999999998},"last":0.376}]},{"osd":2,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":1.5309999999999999,"15min":1.5469999999999999},"min":{"1min":0.217,"5min":0.185,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":39.906999999999996,"15min":39.906999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.32600000000000001,"5min":2.2090000000000001,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.41299999999999998,"5min":94.820999999999998,"15min":94.820999999999998},"last":0.31900000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":6.1859999999999999,"5min":3.5299999999999998,"15min":1.486},"min":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":74.963999999999999,"5min":99.332999999999998,"15min":99.332999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":16.207000000000001,"5min":9.2159999999999993,"15min":4.1139999999999999},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":273.65100000000001,"5min":297.82100000000003,"15min":297.82100000000003},"last":0.42999999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":5.9980000000000002,"5min":3.4809999999999999,"15min":1.4970000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":96.528999999999996,"5min":96.528999999999996,"15min":96.528999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.45100000000000001,"5min":8.8000000000000007,"15min":9.5410000000000004},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":160.02199999999999,"15min":160.02199999999999},"last":0.71499999999999997}]},{"osd":5,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":2.1800000000000002,"15min":1.8899999999999999},"min":{"1min":0.35399999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.60399999999999998,"5min":41.645000000000003,"15min":41.645000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":10.452999999999999,"15min":12.787000000000001},"min":{"1min":0.32300000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66200000000000003,"5min":282.197,"15min":282.197},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.7770000000000001,"15min":1.385},"min":{"1min":0.36199999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58299999999999996,"5min":99.355000000000004,"15min":99.355000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.499,"5min":2.8929999999999998,"15min":2.129},"min":{"1min":0.35899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.72999999999999998,"5min":52.762,"15min":52.762},"last":0.55900000000000005}]},{"osd":7,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":4.8929999999999998,"15min":4.7489999999999997},"min":{"1min":0.34499999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.68700000000000006,"5min":94.662999999999997,"15min":94.662999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":9.0709999999999997,"5min":4.2939999999999996,"15min":1.901},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":157.089,"5min":157.089,"15min":157.089},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27324416,"data_stored":28849251,"data_compressed":1581247,"data_compressed_allocated":1675264,"data_compressed_original":3284992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:42:34.522 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T18:42:34.805 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:42:34.805 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":213,"fsid":"9e25744c-eff1-4a4a-8d47-0dda406b2c35","created":"2024-04-22T18:38:26.841687+0000","modified":"2024-04-22T18:42:34.671910+0000","last_up_change":"2024-04-22T18:38:35.971820+0000","last_in_change":"2024-04-22T18:42:09.816570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T18:38:36.991610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T18:38:37.622620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-04-22T18:38:46.038005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bbf33cfb-0f07-4df6-b90f-83e9edb039a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":186,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6800","nonce":31890}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6801","nonce":31890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6803","nonce":31890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6802","nonce":31890}]},"public_addr":"172.21.15.89:6800/31890","cluster_addr":"172.21.15.89:6801/31890","heartbeat_back_addr":"172.21.15.89:6803/31890","heartbeat_front_addr":"172.21.15.89:6802/31890","state":["exists","up"]},{"osd":1,"uuid":"30415a8e-9f47-416f-9fd0-716c2cb3f562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6804","nonce":31899}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6805","nonce":31899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6807","nonce":31899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6806","nonce":31899}]},"public_addr":"172.21.15.89:6804/31899","cluster_addr":"172.21.15.89:6805/31899","heartbeat_back_addr":"172.21.15.89:6807/31899","heartbeat_front_addr":"172.21.15.89:6806/31899","state":["exists","up"]},{"osd":2,"uuid":"b8da6b88-8355-4225-930d-bfed881089b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6812","nonce":31903}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6813","nonce":31903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6815","nonce":31903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6814","nonce":31903}]},"public_addr":"172.21.15.89:6812/31903","cluster_addr":"172.21.15.89:6813/31903","heartbeat_back_addr":"172.21.15.89:6815/31903","heartbeat_front_addr":"172.21.15.89:6814/31903","state":["exists","up"]},{"osd":3,"uuid":"3eaec913-9d7d-4b03-a28f-0c334e7c39f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6808","nonce":31913}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6809","nonce":31913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6811","nonce":31913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.89:6810","nonce":31913}]},"public_addr":"172.21.15.89:6808/31913","cluster_addr":"172.21.15.89:6809/31913","heartbeat_back_addr":"172.21.15.89:6811/31913","heartbeat_front_addr":"172.21.15.89:6810/31913","state":["exists","up"]},{"osd":4,"uuid":"887acc3b-6f6b-49eb-a039-13050a2c7d64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":183,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":31922}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":31922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":31922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":31922}]},"public_addr":"172.21.15.121:6808/31922","cluster_addr":"172.21.15.121:6809/31922","heartbeat_back_addr":"172.21.15.121:6811/31922","heartbeat_front_addr":"172.21.15.121:6810/31922","state":["exists","up"]},{"osd":5,"uuid":"dcaddfd0-f8d1-4c28-bda2-d17eed606b60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":31921}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":31921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":31921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":31921}]},"public_addr":"172.21.15.121:6804/31921","cluster_addr":"172.21.15.121:6805/31921","heartbeat_back_addr":"172.21.15.121:6807/31921","heartbeat_front_addr":"172.21.15.121:6806/31921","state":["exists","up"]},{"osd":6,"uuid":"acf0527b-3020-4b0a-b64c-5068d21a5781","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":31920}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":31920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":31920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":31920}]},"public_addr":"172.21.15.121:6800/31920","cluster_addr":"172.21.15.121:6801/31920","heartbeat_back_addr":"172.21.15.121:6803/31920","heartbeat_front_addr":"172.21.15.121:6802/31920","state":["exists","up"]},{"osd":7,"uuid":"305394bc-be3d-420b-88d7-d273670540b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6812","nonce":31939}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6813","nonce":31939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6815","nonce":31939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6814","nonce":31939}]},"public_addr":"172.21.15.121:6812/31939","cluster_addr":"172.21.15.121:6813/31939","heartbeat_back_addr":"172.21.15.121:6815/31939","heartbeat_front_addr":"172.21.15.121:6814/31939","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:42:11.065407+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:42:19.663078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:41:20.388547+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:42:34.228108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:42:16.196556+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:42:27.141369+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:41:56.085558+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T18:42:05.294143+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T18:42:35.819 INFO:tasks.ceph:Scrubbing osd.0 2024-04-22T18:42:35.819 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-22T18:42:35.942 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:35.942 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T18:42:35.942 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:35.953 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-22T18:42:36.229 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 0 to deep-scrub 2024-04-22T18:42:36.241 INFO:tasks.ceph:Scrubbing osd.1 2024-04-22T18:42:36.242 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-22T18:42:36.360 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:36.360 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T18:42:36.360 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:36.370 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-22T18:42:36.646 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 1 to deep-scrub 2024-04-22T18:42:36.659 INFO:tasks.ceph:Scrubbing osd.2 2024-04-22T18:42:36.659 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-22T18:42:36.667 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-04-22T18:42:36.668+0000 7f8607f36640 -1 osd.3 213 handle_osd_map injecting map crc failure 2024-04-22T18:42:36.779 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:36.779 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T18:42:36.779 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:36.790 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-22T18:42:37.109 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 2 to deep-scrub 2024-04-22T18:42:37.122 INFO:tasks.ceph:Scrubbing osd.3 2024-04-22T18:42:37.123 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-22T18:42:37.241 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:37.241 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T18:42:37.241 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:37.252 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-22T18:42:37.528 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 3 to deep-scrub 2024-04-22T18:42:37.541 INFO:tasks.ceph:Scrubbing osd.4 2024-04-22T18:42:37.542 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-22T18:42:37.660 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:37.660 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T18:42:37.660 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:37.671 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-22T18:42:37.949 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 4 to deep-scrub 2024-04-22T18:42:37.962 INFO:tasks.ceph:Scrubbing osd.5 2024-04-22T18:42:37.963 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-22T18:42:38.083 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:38.084 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T18:42:38.084 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:38.094 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-22T18:42:38.375 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 5 to deep-scrub 2024-04-22T18:42:38.388 INFO:tasks.ceph:Scrubbing osd.6 2024-04-22T18:42:38.388 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-22T18:42:38.510 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:38.511 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T18:42:38.511 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:38.521 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-22T18:42:38.801 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 6 to deep-scrub 2024-04-22T18:42:38.814 INFO:tasks.ceph:Scrubbing osd.7 2024-04-22T18:42:38.815 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-22T18:42:38.934 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-04-22T18:42:38.934 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-22T18:42:38.934 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-04-22T18:42:38.945 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-22T18:42:39.223 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 7 to deep-scrub 2024-04-22T18:42:39.236 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:42:39.516 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:42:39.517 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:42:39.528 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-04-22T18:42:39.004164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":156248488,"num_objects":224,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1404,"num_read_kb":30872,"num_write":1881,"num_write_kb":393766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1303,"num_bytes_recovered":244989684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1528,"ondisk_log_size":1528,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":482944,"kb_used_data":209440,"kb_used_omap":12,"kb_used_meta":273395,"kb_avail":754491776,"statfs":{"total":773094113280,"available":772599578624,"internally_reserved":0,"allocated":214466560,"data_stored":220297576,"data_compressed":11796076,"data_compressed_allocated":12582912,"data_compressed_original":26013696,"omap_allocated":12744,"internal_metadata":279957048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1861561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.334593"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":38,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-22T18:42:31.669567+0000","last_change":"2024-04-22T18:42:11.391813+0000","last_active":"2024-04-22T18:42:31.669567+0000","last_peered":"2024-04-22T18:42:31.669567+0000","last_clean":"2024-04-22T18:42:31.669567+0000","last_became_active":"2024-04-22T18:42:11.030639+0000","last_became_peered":"2024-04-22T18:42:11.030639+0000","last_unstale":"2024-04-22T18:42:31.669567+0000","last_undegraded":"2024-04-22T18:42:31.669567+0000","last_fullsized":"2024-04-22T18:42:31.669567+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.391633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.391633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:18.616280+0000","scrub_duration":0.102099431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":243,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-22T18:42:35.060103+0000","last_change":"2024-04-22T18:42:11.029787+0000","last_active":"2024-04-22T18:42:35.060103+0000","last_peered":"2024-04-22T18:42:35.060103+0000","last_clean":"2024-04-22T18:42:35.060103+0000","last_became_active":"2024-04-22T18:42:11.029453+0000","last_became_peered":"2024-04-22T18:42:11.029453+0000","last_unstale":"2024-04-22T18:42:35.060103+0000","last_undegraded":"2024-04-22T18:42:35.060103+0000","last_fullsized":"2024-04-22T18:42:35.060103+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:58.561808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:58.561808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.301876478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":248,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-22T18:42:31.772975+0000","last_change":"2024-04-22T18:42:11.428432+0000","last_active":"2024-04-22T18:42:31.772975+0000","last_peered":"2024-04-22T18:42:31.772975+0000","last_clean":"2024-04-22T18:42:31.772975+0000","last_became_active":"2024-04-22T18:41:26.961835+0000","last_became_peered":"2024-04-22T18:41:26.961835+0000","last_unstale":"2024-04-22T18:42:31.772975+0000","last_undegraded":"2024-04-22T18:42:31.772975+0000","last_fullsized":"2024-04-22T18:42:31.772975+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:11.428262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:11.428262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:11.658873+0000","scrub_duration":0.30251553399999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":233,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-22T18:42:35.060102+0000","last_change":"2024-04-22T18:42:12.520547+0000","last_active":"2024-04-22T18:42:35.060102+0000","last_peered":"2024-04-22T18:42:35.060102+0000","last_clean":"2024-04-22T18:42:35.060102+0000","last_became_active":"2024-04-22T18:41:26.854399+0000","last_became_peered":"2024-04-22T18:41:26.854399+0000","last_unstale":"2024-04-22T18:42:35.060102+0000","last_undegraded":"2024-04-22T18:42:35.060102+0000","last_fullsized":"2024-04-22T18:42:35.060102+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:12.520456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:12.520456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.20146086999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"169'384","reported_seq":1631,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-22T18:42:35.160526+0000","last_change":"2024-04-22T18:42:17.393680+0000","last_active":"2024-04-22T18:42:35.160526+0000","last_peered":"2024-04-22T18:42:35.160526+0000","last_clean":"2024-04-22T18:42:35.160526+0000","last_became_active":"2024-04-22T18:41:26.994048+0000","last_became_peered":"2024-04-22T18:41:26.994048+0000","last_unstale":"2024-04-22T18:42:35.160526+0000","last_undegraded":"2024-04-22T18:42:35.160526+0000","last_fullsized":"2024-04-22T18:42:35.160526+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'384","last_scrub_stamp":"2024-04-22T18:42:17.393461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.393461+0000","objects_scrubbed":54,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:29.216372+0000","scrub_duration":0.111477038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41123046,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":358,"num_read_kb":4228,"num_write":477,"num_write_kb":113967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":82068207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":39,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-22T18:42:32.464695+0000","last_change":"2024-04-22T18:42:28.388472+0000","last_active":"2024-04-22T18:42:32.464695+0000","last_peered":"2024-04-22T18:42:32.464695+0000","last_clean":"2024-04-22T18:42:32.464695+0000","last_became_active":"2024-04-22T18:42:09.929115+0000","last_became_peered":"2024-04-22T18:42:09.929115+0000","last_unstale":"2024-04-22T18:42:32.464695+0000","last_undegraded":"2024-04-22T18:42:32.464695+0000","last_fullsized":"2024-04-22T18:42:32.464695+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:28.388238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:42:28.388238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:47.486386+0000","scrub_duration":0.203509897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"200'362","reported_seq":850,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-22T18:42:35.160231+0000","last_change":"2024-04-22T18:42:17.640545+0000","last_active":"2024-04-22T18:42:35.160231+0000","last_peered":"2024-04-22T18:42:35.160231+0000","last_clean":"2024-04-22T18:42:35.160231+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:42:35.160231+0000","last_undegraded":"2024-04-22T18:42:35.160231+0000","last_fullsized":"2024-04-22T18:42:35.160231+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'362","last_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_deep_scrub":"200'362","last_deep_scrub_stamp":"2024-04-22T18:42:17.640441+0000","last_clean_scrub_stamp":"2024-04-22T18:42:17.640441+0000","objects_scrubbed":53,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for deep scrub","scrub_duration":3.4448620060000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382624,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":306,"num_read_kb":4987,"num_write":404,"num_write_kb":80545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":179,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-22T18:42:31.938369+0000","last_change":"2024-04-22T18:41:41.421087+0000","last_active":"2024-04-22T18:42:31.938369+0000","last_peered":"2024-04-22T18:42:31.938369+0000","last_clean":"2024-04-22T18:42:31.938369+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:42:31.938369+0000","last_undegraded":"2024-04-22T18:42:31.938369+0000","last_fullsized":"2024-04-22T18:42:31.938369+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:41.420996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:41.420996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:57.844851+0000","scrub_duration":0.20141637900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"168'337","reported_seq":1431,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-22T18:42:34.778699+0000","last_change":"2024-04-22T18:42:09.682366+0000","last_active":"2024-04-22T18:42:34.778699+0000","last_peered":"2024-04-22T18:42:34.778699+0000","last_clean":"2024-04-22T18:42:34.778699+0000","last_became_active":"2024-04-22T18:41:41.007340+0000","last_became_peered":"2024-04-22T18:41:41.007340+0000","last_unstale":"2024-04-22T18:42:34.778699+0000","last_undegraded":"2024-04-22T18:42:34.778699+0000","last_fullsized":"2024-04-22T18:42:34.778699+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'337","last_scrub_stamp":"2024-04-22T18:42:09.682178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:09.682178+0000","objects_scrubbed":54,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.20999784699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39035424,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":322,"num_read_kb":9140,"num_write":423,"num_write_kb":104979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":67954083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":223,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-22T18:42:31.772957+0000","last_change":"2024-04-22T18:42:13.608651+0000","last_active":"2024-04-22T18:42:31.772957+0000","last_peered":"2024-04-22T18:42:31.772957+0000","last_clean":"2024-04-22T18:42:31.772957+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:42:31.772957+0000","last_undegraded":"2024-04-22T18:42:31.772957+0000","last_fullsized":"2024-04-22T18:42:31.772957+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:13.608479+0000","last_clean_scrub_stamp":"2024-04-22T18:42:13.608479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:21.626020+0000","scrub_duration":0.43950688900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"169'328","reported_seq":1343,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-22T18:42:34.980213+0000","last_change":"2024-04-22T18:42:27.323551+0000","last_active":"2024-04-22T18:42:34.980213+0000","last_peered":"2024-04-22T18:42:34.980213+0000","last_clean":"2024-04-22T18:42:34.980213+0000","last_became_active":"2024-04-22T18:42:18.879845+0000","last_became_peered":"2024-04-22T18:42:18.879845+0000","last_unstale":"2024-04-22T18:42:34.980213+0000","last_undegraded":"2024-04-22T18:42:34.980213+0000","last_fullsized":"2024-04-22T18:42:34.980213+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'328","last_scrub_stamp":"2024-04-22T18:42:06.716200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:46.054802+0000","last_clean_scrub_stamp":"2024-04-22T18:42:06.716200+0000","objects_scrubbed":46,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:35.844741+0000","scrub_duration":0.61663019600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43117026,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":302,"num_read_kb":12417,"num_write":418,"num_write_kb":91859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":72521286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":28,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-22T18:42:32.564729+0000","last_change":"2024-04-22T18:42:10.136809+0000","last_active":"2024-04-22T18:42:32.564729+0000","last_peered":"2024-04-22T18:42:32.564729+0000","last_clean":"2024-04-22T18:42:32.564729+0000","last_became_active":"2024-04-22T18:42:10.036239+0000","last_became_peered":"2024-04-22T18:42:10.036239+0000","last_unstale":"2024-04-22T18:42:32.564729+0000","last_undegraded":"2024-04-22T18:42:32.564729+0000","last_fullsized":"2024-04-22T18:42:32.564729+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:41:30.504686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:37.997225+0000","last_clean_scrub_stamp":"2024-04-22T18:41:30.504686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:42:45.796059+0000","scrub_duration":0.30171718199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'117","reported_seq":418,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-04-22T18:42:31.772993+0000","last_change":"2024-04-22T18:41:56.270751+0000","last_active":"2024-04-22T18:42:31.772993+0000","last_peered":"2024-04-22T18:42:31.772993+0000","last_clean":"2024-04-22T18:42:31.772993+0000","last_became_active":"2024-04-22T18:41:26.757958+0000","last_became_peered":"2024-04-22T18:41:26.757958+0000","last_unstale":"2024-04-22T18:42:31.772993+0000","last_undegraded":"2024-04-22T18:42:31.772993+0000","last_fullsized":"2024-04-22T18:42:31.772993+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-04-22T18:41:56.270650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:38:36.991764+0000","last_clean_scrub_stamp":"2024-04-22T18:41:56.270650+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:12.138524+0000","scrub_duration":0.202387964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":155658120,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":1288,"num_read_kb":30772,"num_write":1722,"num_write_kb":391350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1297,"num_bytes_recovered":243402900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249495552,"data_stored":263913197,"data_compressed":14626013,"data_compressed_allocated":15568896,"data_compressed_original":30834688,"omap_allocated":0,"internal_metadata":0},"log_size":1411,"ondisk_log_size":1411,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":2115712,"data_compressed":122550,"data_compressed_allocated":163840,"data_compressed_original":2084864,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36820,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335020,"statfs":{"total":96636764160,"available":96599060480,"internally_reserved":0,"allocated":1646592,"data_stored":1252834,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":2.5870000000000002,"15min":1.1439999999999999},"min":{"1min":0.182,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":26.568000000000001,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":6.3600000000000003,"5min":6.766,"15min":5.2599999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":95.918999999999997,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.44800000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":3.8050000000000002,"5min":3.984,"15min":4.5709999999999997},"min":{"1min":0.29299999999999998,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":66.814999999999998,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.38900000000000001},{"interface":"front","average":{"1min":6.6040000000000001,"5min":12.141,"15min":12.744999999999999},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":100.242,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.54700000000000004}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7130000000000001,"5min":4.0730000000000004,"15min":4.0810000000000004},"min":{"1min":0.27400000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":97.662000000000006,"5min":97.662000000000006,"15min":97.662000000000006},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.42999999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":7.7560000000000002,"5min":4.1289999999999996,"15min":4.1429999999999998},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":97.686999999999998,"5min":97.686999999999998,"15min":97.686999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.47499999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":8.2089999999999996,"5min":5.4509999999999996,"15min":4.7889999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":78.149000000000001,"5min":85.509,"15min":85.509},"last":0.51700000000000002},{"interface":"front","average":{"1min":7.7130000000000001,"5min":16.856000000000002,"15min":20.866},"min":{"1min":0.26000000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":97.590000000000003,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.58499999999999996}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":2.0270000000000001,"15min":2.7229999999999999},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.4730000000000001,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.6679999999999999,"5min":5.2939999999999996,"15min":6.5389999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":23.007999999999999,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.59399999999999997}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":11.756,"15min":12.769},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":23.027000000000001,"5min":139.428,"15min":139.428},"last":0.57499999999999996},{"interface":"front","average":{"1min":7.7800000000000002,"5min":4.3070000000000004,"15min":4.1909999999999998},"min":{"1min":0.29199999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":97.635999999999996,"5min":97.635999999999996,"15min":97.635999999999996},"last":0.60499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574899,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32632,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1552384,"data_stored":721654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.5030000000000001,"5min":3.0579999999999998,"15min":2.2149999999999999},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":70.436000000000007,"5min":70.436000000000007,"15min":70.436000000000007},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":16.753,"15min":22.789000000000001},"min":{"1min":0.26700000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.58599999999999997,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.58799999999999997}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":3.577,"15min":2.5510000000000002},"min":{"1min":0.254,"5min":0.215,"15min":0.215},"max":{"1min":0.54000000000000004,"5min":199.047,"15min":199.047},"last":0.60299999999999998},{"interface":"front","average":{"1min":11.327999999999999,"5min":8.6519999999999992,"15min":9.6120000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":192.279,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.61899999999999999}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.44900000000000001,"15min":0.45600000000000002},"min":{"1min":0.33400000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.68799999999999994,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.44600000000000001,"5min":12.56,"15min":16.722000000000001},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.59099999999999997,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.45800000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":14.815,"15min":19.283999999999999},"min":{"1min":0.29599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65200000000000002,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":1.335,"15min":0.74299999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.875,"5min":77.097999999999999,"15min":77.097999999999999},"last":0.54000000000000004}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":2.052,"5min":5.0499999999999998,"15min":6.7519999999999998},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":29.978999999999999,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.30199999999999999},{"interface":"front","average":{"1min":10.577,"5min":16.312000000000001,"15min":18.321999999999999},"min":{"1min":0.27900000000000003,"5min":0.218,"15min":0.218},"max":{"1min":168.44,"5min":168.44,"15min":168.44},"last":0.35199999999999998}]},{"osd":5,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39300000000000002,"15min":0.39700000000000002},"min":{"1min":0.254,"5min":0.252,"15min":0.252},"max":{"1min":0.61199999999999999,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.28199999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.42799999999999999}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":5.6289999999999996,"5min":14.432,"15min":17.378},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":90.649000000000001,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.502},{"interface":"front","average":{"1min":1.165,"5min":0.60099999999999998,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.077,"5min":13.077,"15min":13.077},"last":0.48199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574899,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59212,"kb_used_data":28096,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94312628,"statfs":{"total":96636764160,"available":96576131072,"internally_reserved":0,"allocated":28770304,"data_stored":29415634,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3580000000000001,"5min":9.875,"15min":12.218999999999999},"min":{"1min":0.34999999999999998,"5min":0.192,"15min":0.192},"max":{"1min":58.421999999999997,"5min":74.019999999999996,"15min":74.019999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.71199999999999997,"15min":0.84099999999999997},"min":{"1min":0.34000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.52400000000000002,"5min":8.3919999999999995,"15min":8.3919999999999995},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.0259999999999998,"5min":11.034000000000001,"15min":10.702999999999999},"min":{"1min":0.28100000000000003,"5min":0.224,"15min":0.224},"max":{"1min":91.911000000000001,"5min":202.13800000000001,"15min":202.13800000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.3440000000000001,"15min":0.72699999999999998},"min":{"1min":0.34399999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.669,"5min":68.346999999999994,"15min":68.346999999999994},"last":0.44700000000000001}]},{"osd":2,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.496,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":3.339,"15min":4.4720000000000004},"min":{"1min":0.377,"5min":0.308,"15min":0.308},"max":{"1min":0.57399999999999995,"5min":78.296999999999997,"15min":78.296999999999997},"last":0.51300000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":4.5499999999999998,"15min":5.3490000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.018,"5min":89.688000000000002,"15min":89.688000000000002},"last":0.501},{"interface":"front","average":{"1min":0.46999999999999997,"5min":6.5380000000000003,"15min":9.0890000000000004},"min":{"1min":0.34899999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60899999999999999,"5min":89.718999999999994,"15min":89.718999999999994},"last":0.53500000000000003}]},{"osd":4,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":9.4809999999999999,"5min":15.465999999999999,"15min":16.969000000000001},"min":{"1min":0.249,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":100.544,"5min":202.983,"15min":202.983},"last":0.33300000000000002},{"interface":"front","average":{"1min":9.5050000000000008,"5min":14.603999999999999,"15min":16.722999999999999},"min":{"1min":0.314,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":100.56,"5min":202.999,"15min":202.999},"last":0.45700000000000002}]},{"osd":6,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":4.3739999999999997,"15min":4.8159999999999998},"min":{"1min":0.27100000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.59099999999999997,"5min":79.945999999999998,"15min":79.945999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":5.274,"15min":5.9770000000000003},"min":{"1min":0.251,"5min":0.249,"15min":0.249},"max":{"1min":0.57299999999999995,"5min":100.54900000000001,"15min":100.54900000000001},"last":0.40899999999999997}]},{"osd":7,"last update":"Mon Apr 22 18:41:43 2024","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":1.149,"15min":0.68700000000000006},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":71.602999999999994,"5min":71.602999999999994,"15min":71.602999999999994},"last":0.311},{"interface":"front","average":{"1min":4.508,"5min":4.173,"15min":4.7770000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":75.436000000000007,"5min":78.340999999999994,"15min":78.340999999999994},"last":0.48199999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574899,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89676,"kb_used_data":54464,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282164,"statfs":{"total":96636764160,"available":96544935936,"internally_reserved":0,"allocated":55771136,"data_stored":57822932,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":13.115,"15min":15.470000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":38.899999999999999,"5min":252.96199999999999,"15min":252.96199999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":9.282,"15min":8.8059999999999992},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63300000000000001,"5min":187.148,"15min":187.148},"last":0.40500000000000003}]},{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":2.508,"15min":1.115},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":9.141,"5min":96.486999999999995,"15min":96.486999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":22.393000000000001,"5min":11.545999999999999,"15min":7.4390000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":310.61500000000001,"5min":310.61500000000001,"15min":310.61500000000001},"last":0.47299999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":8.8989999999999991,"15min":8.6790000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.8330000000000002,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.995,"5min":2.4199999999999999,"15min":1.1040000000000001},"min":{"1min":0.31900000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":7.1399999999999997,"5min":96.471999999999994,"15min":96.471999999999994},"last":0.48999999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":11.172000000000001,"5min":8.9710000000000001,"15min":5.6710000000000003},"min":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":122.102,"5min":252.91900000000001,"15min":252.91900000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":17.242000000000001,"15min":19.725999999999999},"min":{"1min":0.30299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59899999999999998,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.45800000000000002}]},{"osd":5,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":2.3380000000000001,"15min":1.034},"min":{"1min":0.25800000000000001,"5min":0.218,"15min":0.218},"max":{"1min":7.1559999999999997,"5min":96.408000000000001,"15min":96.408000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":2.6960000000000002,"5min":3.048,"15min":3.7080000000000002},"min":{"1min":0.36199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":38.581000000000003,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.59599999999999997}]},{"osd":6,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":5.8289999999999997,"5min":14.667,"15min":13.662000000000001},"min":{"1min":0.25700000000000001,"5min":0.247,"15min":0.247},"max":{"1min":87.706999999999994,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.62},{"interface":"front","average":{"1min":0.83199999999999996,"5min":2.3820000000000001,"15min":1.0880000000000001},"min":{"1min":0.35699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":7.1159999999999997,"5min":96.438000000000002,"15min":96.438000000000002},"last":0.54900000000000004}]},{"osd":7,"last update":"Mon Apr 22 18:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":9.5749999999999993,"15min":8.8979999999999997},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68000000000000005,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.9790000000000001,"5min":2.6419999999999999,"15min":1.202},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":19.518999999999998,"5min":96.456999999999994,"15min":96.456999999999994},"last":0.63400000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574899,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63352,"kb_used_data":28140,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94308488,"statfs":{"total":96636764160,"available":96571891712,"internally_reserved":0,"allocated":28815360,"data_stored":29420440,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":15.041,"5min":10.538,"15min":11.646000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":93.131,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.253},{"interface":"front","average":{"1min":7.077,"5min":1.6659999999999999,"15min":0.77400000000000002},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":87.305999999999997,"5min":87.305999999999997,"15min":87.305999999999997},"last":0.36299999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":11.677,"5min":10.558,"15min":6.8819999999999997},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":78.805000000000007,"5min":197.02099999999999,"15min":197.02099999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":8.9800000000000004,"5min":2.3210000000000002,"15min":0.997},"min":{"1min":0.20999999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":78.393000000000001,"5min":78.393000000000001,"15min":78.393000000000001},"last":0.28199999999999997}]},{"osd":2,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.6960000000000002,"5min":2.3439999999999999,"15min":1.452},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":47.534999999999997,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.26600000000000001},{"interface":"front","average":{"1min":6.4770000000000003,"5min":1.728,"15min":0.85899999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":78.421999999999997,"5min":78.421999999999997,"15min":78.421999999999997},"last":0.34399999999999997}]},{"osd":4,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1800000000000002,"5min":2.9940000000000002,"15min":3.7360000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":26.141999999999999,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.57399999999999995},{"interface":"front","average":{"1min":4.5629999999999997,"5min":4.6289999999999996,"15min":4.3010000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":47.616999999999997,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.48799999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":2.8279999999999998,"5min":2.5049999999999999,"15min":1.623},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":47.566000000000003,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.56599999999999995},{"interface":"front","average":{"1min":6.6379999999999999,"5min":2.2890000000000001,"15min":1.601},"min":{"1min":0.35699999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.480999999999995,"5min":78.480999999999995,"15min":78.480999999999995},"last":0.47899999999999998}]},{"osd":6,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.50900000000000001,"15min":0.53100000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.748,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.63800000000000001},{"interface":"front","average":{"1min":2.8500000000000001,"5min":2.871,"15min":1.3109999999999999},"min":{"1min":0.377,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":47.590000000000003,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.51000000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:41:40 2024","interfaces":[{"interface":"back","average":{"1min":6.6189999999999998,"5min":1.903,"15min":1.0229999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":78.453999999999994,"5min":78.453999999999994,"15min":78.453999999999994},"last":0.5},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.55800000000000005},"min":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69299999999999995,"5min":0.995,"15min":0.995},"last":0.61099999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51644,"kb_used_data":20528,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94320196,"statfs":{"total":96636764160,"available":96583880704,"internally_reserved":0,"allocated":21020672,"data_stored":21653321,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":6.7439999999999998,"15min":7.6660000000000004},"min":{"1min":0.20399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":84.338999999999999,"5min":149.03,"15min":149.03},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.28299999999999997,"5min":1.3089999999999999,"15min":1.7589999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.62,"5min":31.866,"15min":31.866},"last":0.27700000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":9.4849999999999994,"5min":5.2489999999999997,"15min":2.4380000000000002},"min":{"1min":0.216,"5min":0.193,"15min":0.193},"max":{"1min":91.912000000000006,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.29199999999999998},{"interface":"front","average":{"1min":12.752000000000001,"5min":15.912000000000001,"15min":15.227},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":126.42,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.31900000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":11.936999999999999,"5min":6.2830000000000004,"15min":2.3359999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":129.041,"5min":147.721,"15min":147.721},"last":0.372},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.53300000000000003}]},{"osd":4,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":8.1280000000000001,"5min":13.57,"15min":17.038},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":90.882999999999996,"5min":160.584,"15min":160.584},"last":0.46600000000000003},{"interface":"front","average":{"1min":5.9349999999999996,"5min":4.8520000000000003,"15min":3.2490000000000001},"min":{"1min":0.32500000000000001,"5min":0.315,"15min":0.315},"max":{"1min":93.962999999999994,"5min":97.887,"15min":97.887},"last":0.56299999999999994}]},{"osd":5,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":2.0339999999999998,"15min":2.7120000000000002},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.879,"5min":48.631,"15min":48.631},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":2.2890000000000001,"15min":3.0800000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.91800000000000004,"5min":56.18,"15min":56.18},"last":0.503}]},{"osd":6,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.3540000000000001,"5min":3.4670000000000001,"15min":1.5129999999999999},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":92.028000000000006,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.68000000000000005}]},{"osd":7,"last update":"Mon Apr 22 18:41:41 2024","interfaces":[{"interface":"back","average":{"1min":5.29,"5min":1.5800000000000001,"15min":0.91000000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":90.944999999999993,"5min":90.944999999999993,"15min":90.944999999999993},"last":0.54600000000000004},{"interface":"front","average":{"1min":7.4640000000000004,"5min":3.923,"15min":1.6970000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":92.048000000000002,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.63}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59880,"kb_used_data":20572,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94311960,"statfs":{"total":96636764160,"available":96575447040,"internally_reserved":0,"allocated":21065728,"data_stored":21658797,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":5.883,"15min":7.2000000000000002},"min":{"1min":0.20000000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":27.116,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.34999999999999998},{"interface":"front","average":{"1min":5.8040000000000003,"5min":6.6870000000000003,"15min":7.4710000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":94.046000000000006,"5min":94.046000000000006,"15min":94.046000000000006},"last":0.33700000000000002}]},{"osd":2,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.679000000000002,"5min":13.159000000000001,"15min":10.119},"min":{"1min":0.23000000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":406.68799999999999,"5min":406.68799999999999,"15min":406.68799999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.8700000000000001,"5min":0.61899999999999999,"15min":0.41099999999999998},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19},"max":{"1min":27.094999999999999,"5min":27.094999999999999,"15min":27.094999999999999},"last":0.25600000000000001}]},{"osd":3,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":15.968,"5min":3.464,"15min":1.3759999999999999},"min":{"1min":0.29099999999999998,"5min":0.216,"15min":0.216},"max":{"1min":228.90100000000001,"5min":228.90100000000001,"15min":228.90100000000001},"last":0.314},{"interface":"front","average":{"1min":21.481999999999999,"5min":13.061999999999999,"15min":14.015000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":196.82900000000001,"5min":231.864,"15min":231.864},"last":0.28799999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":39.823,"5min":17.213000000000001,"15min":14.393000000000001},"min":{"1min":0.316,"5min":0.309,"15min":0.309},"max":{"1min":406.71199999999999,"5min":406.71199999999999,"15min":406.71199999999999},"last":0.626},{"interface":"front","average":{"1min":25.027000000000001,"5min":5.3979999999999997,"15min":2.1320000000000001},"min":{"1min":0.34499999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":186.89500000000001,"5min":186.89500000000001,"15min":186.89500000000001},"last":0.501}]},{"osd":5,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":9.7430000000000003,"15min":12.628},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":94.183000000000007,"5min":138.089,"15min":138.089},"last":0.67600000000000005},{"interface":"front","average":{"1min":39.863999999999997,"5min":20.117000000000001,"15min":20.09},"min":{"1min":0.35499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":406.72899999999998,"5min":406.72899999999998,"15min":406.72899999999998},"last":0.65400000000000003}]},{"osd":6,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":6.0010000000000003,"5min":3.8359999999999999,"15min":3.1299999999999999},"min":{"1min":0.34000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":94.302999999999997,"5min":94.302999999999997,"15min":94.302999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":43.036000000000001,"5min":19.713999999999999,"15min":17.591999999999999},"min":{"1min":0.38500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":427.69299999999998,"5min":427.69299999999998,"15min":427.69299999999998},"last":0.50700000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:41:44 2024","interfaces":[{"interface":"back","average":{"1min":2.0720000000000001,"5min":0.82799999999999996,"15min":0.626},"min":{"1min":0.373,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":27.253,"5min":27.253,"15min":27.253},"last":0.66600000000000004},{"interface":"front","average":{"1min":6.0090000000000003,"5min":3.129,"15min":3.069},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":94.260999999999996,"5min":94.260999999999996,"15min":94.260999999999996},"last":0.64400000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89728,"kb_used_data":54516,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282112,"statfs":{"total":96636764160,"available":96544882688,"internally_reserved":0,"allocated":55824384,"data_stored":58351964,"data_compressed":3021454,"data_compressed_allocated":3239936,"data_compressed_original":6823936,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":9.7919999999999998,"5min":9.5660000000000007,"15min":7.6950000000000003},"min":{"1min":0.185,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":127.30500000000001,"5min":133.09899999999999,"15min":133.09899999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":27.553999999999998,"5min":11.506,"15min":4.8399999999999999},"min":{"1min":0.20300000000000001,"5min":0.188,"15min":0.188},"max":{"1min":451.50799999999998,"5min":451.50799999999998,"15min":451.50799999999998},"last":0.35199999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":1.5309999999999999,"15min":1.5469999999999999},"min":{"1min":0.217,"5min":0.185,"15min":0.185},"max":{"1min":0.55900000000000005,"5min":39.906999999999996,"15min":39.906999999999996},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.32600000000000001,"5min":2.2090000000000001,"15min":0.94799999999999995},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.41299999999999998,"5min":94.820999999999998,"15min":94.820999999999998},"last":0.42999999999999999}]},{"osd":3,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":6.1859999999999999,"5min":3.5299999999999998,"15min":1.486},"min":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":74.963999999999999,"5min":99.332999999999998,"15min":99.332999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":16.207000000000001,"5min":9.2159999999999993,"15min":4.1139999999999999},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":273.65100000000001,"5min":297.82100000000003,"15min":297.82100000000003},"last":0.41699999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":5.9980000000000002,"5min":3.4809999999999999,"15min":1.4970000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":96.528999999999996,"5min":96.528999999999996,"15min":96.528999999999996},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.45100000000000001,"5min":8.8000000000000007,"15min":9.5410000000000004},"min":{"1min":0.316,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65800000000000003,"5min":160.02199999999999,"15min":160.02199999999999},"last":0.55600000000000005}]},{"osd":5,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":2.1800000000000002,"15min":1.8899999999999999},"min":{"1min":0.35399999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.60399999999999998,"5min":41.645000000000003,"15min":41.645000000000003},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":10.452999999999999,"15min":12.787000000000001},"min":{"1min":0.32300000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66200000000000003,"5min":282.197,"15min":282.197},"last":0.65100000000000002}]},{"osd":6,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.7770000000000001,"15min":1.385},"min":{"1min":0.36199999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.58299999999999996,"5min":99.355000000000004,"15min":99.355000000000004},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.499,"5min":2.8929999999999998,"15min":2.129},"min":{"1min":0.35899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.72999999999999998,"5min":52.762,"15min":52.762},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Apr 22 18:41:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":4.8929999999999998,"15min":4.7489999999999997},"min":{"1min":0.34499999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.68700000000000006,"5min":94.662999999999997,"15min":94.662999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":9.0709999999999997,"5min":4.2939999999999996,"15min":1.901},"min":{"1min":0.42299999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":157.089,"5min":157.089,"15min":157.089},"last":0.66000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27324416,"data_stored":28849251,"data_compressed":1581247,"data_compressed_allocated":1675264,"data_compressed_original":3284992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:42:39.533 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-22T18:42:11.391633+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=11, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=34, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T18:42:39.533 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-22T18:41:58.561808+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=41, tm_sec=58, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=34, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T18:42:39.534 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-22T18:42:11.428262+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=11, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=34, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T18:42:39.534 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-22T18:42:12.520456+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=12, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=34, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T18:42:39.534 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-22T18:42:17.393461+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=17, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=34, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T18:42:39.534 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-22T18:42:28.388238+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=28, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=34, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T18:42:39.534 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-22T18:42:17.640441+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=17, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=34, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T18:42:39.534 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-22T18:41:41.420996+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=41, tm_sec=41, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=34, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T18:42:39.534 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-22T18:42:09.682178+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=9, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=34, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T18:42:39.534 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-22T18:42:13.608479+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=13, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=34, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T18:42:39.534 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-22T18:42:06.716200+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=6, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=34, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T18:42:39.535 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-22T18:41:30.504686+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=41, tm_sec=30, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=34, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T18:42:39.535 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-22T18:41:56.270650+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=41, tm_sec=56, tm_wday=0, tm_yday=113, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=22, tm_hour=18, tm_min=42, tm_sec=34, tm_wday=0, tm_yday=113, tm_isdst=0) 2024-04-22T18:42:39.535 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-22T18:42:59.536 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T18:42:59.831 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-04-22T18:42:59.832 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-04-22T18:42:59.843 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-04-22T18:42:59.008603+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":156248488,"num_objects":224,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1404,"num_read_kb":30872,"num_write":1881,"num_write_kb":393766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1303,"num_bytes_recovered":244989684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1699,"ondisk_log_size":1699,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485488,"kb_used_data":206884,"kb_used_omap":12,"kb_used_meta":277491,"kb_avail":754489232,"statfs":{"total":773094113280,"available":772596973568,"internally_reserved":0,"allocated":211849216,"data_stored":219066644,"data_compressed":11796076,"data_compressed_allocated":12582912,"data_compressed_original":26013696,"omap_allocated":12744,"internal_metadata":284151352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002709"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-04-22T18:42:41.436964+0000","last_change":"2024-04-22T18:42:41.436964+0000","last_active":"2024-04-22T18:42:41.436964+0000","last_peered":"2024-04-22T18:42:41.436964+0000","last_clean":"2024-04-22T18:42:41.436964+0000","last_became_active":"2024-04-22T18:42:11.030639+0000","last_became_peered":"2024-04-22T18:42:11.030639+0000","last_unstale":"2024-04-22T18:42:41.436964+0000","last_undegraded":"2024-04-22T18:42:41.436964+0000","last_fullsized":"2024-04-22T18:42:41.436964+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:41.436825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:41.436825+0000","last_clean_scrub_stamp":"2024-04-22T18:42:41.436825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:48.398583+0000","scrub_duration":0.30279683699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":257,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-22T18:42:50.317130+0000","last_change":"2024-04-22T18:42:50.317130+0000","last_active":"2024-04-22T18:42:50.317130+0000","last_peered":"2024-04-22T18:42:50.317130+0000","last_clean":"2024-04-22T18:42:50.317130+0000","last_became_active":"2024-04-22T18:42:11.029453+0000","last_became_peered":"2024-04-22T18:42:11.029453+0000","last_unstale":"2024-04-22T18:42:50.317130+0000","last_undegraded":"2024-04-22T18:42:50.317130+0000","last_fullsized":"2024-04-22T18:42:50.317130+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:50.317004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:50.317004+0000","last_clean_scrub_stamp":"2024-04-22T18:42:50.317004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:44:07.601740+0000","scrub_duration":0.202757199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":263,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-22T18:42:44.630518+0000","last_change":"2024-04-22T18:42:44.630518+0000","last_active":"2024-04-22T18:42:44.630518+0000","last_peered":"2024-04-22T18:42:44.630518+0000","last_clean":"2024-04-22T18:42:44.630518+0000","last_became_active":"2024-04-22T18:41:26.961835+0000","last_became_peered":"2024-04-22T18:41:26.961835+0000","last_unstale":"2024-04-22T18:42:44.630518+0000","last_undegraded":"2024-04-22T18:42:44.630518+0000","last_fullsized":"2024-04-22T18:42:44.630518+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:44.630349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:44.630349+0000","last_clean_scrub_stamp":"2024-04-22T18:42:44.630349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:44:13.641296+0000","scrub_duration":0.50336294699999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":247,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-22T18:42:42.485630+0000","last_change":"2024-04-22T18:42:42.485630+0000","last_active":"2024-04-22T18:42:42.485630+0000","last_peered":"2024-04-22T18:42:42.485630+0000","last_clean":"2024-04-22T18:42:42.485630+0000","last_became_active":"2024-04-22T18:41:26.854399+0000","last_became_peered":"2024-04-22T18:41:26.854399+0000","last_unstale":"2024-04-22T18:42:42.485630+0000","last_undegraded":"2024-04-22T18:42:42.485630+0000","last_fullsized":"2024-04-22T18:42:42.485630+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:42.485465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:42.485465+0000","last_clean_scrub_stamp":"2024-04-22T18:42:42.485465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:44:01.843899+0000","scrub_duration":0.20436565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"215'438","reported_seq":1703,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-22T18:42:47.115008+0000","last_change":"2024-04-22T18:42:47.115194+0000","last_active":"2024-04-22T18:42:47.115008+0000","last_peered":"2024-04-22T18:42:47.115008+0000","last_clean":"2024-04-22T18:42:47.115008+0000","last_became_active":"2024-04-22T18:41:26.994048+0000","last_became_peered":"2024-04-22T18:41:26.994048+0000","last_unstale":"2024-04-22T18:42:47.115008+0000","last_undegraded":"2024-04-22T18:42:47.115008+0000","last_fullsized":"2024-04-22T18:42:47.115008+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'438","last_scrub_stamp":"2024-04-22T18:42:47.114982+0000","last_deep_scrub":"215'438","last_deep_scrub_stamp":"2024-04-22T18:42:47.114982+0000","last_clean_scrub_stamp":"2024-04-22T18:42:47.114982+0000","objects_scrubbed":54,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:44:11.290916+0000","scrub_duration":3.9860218490000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41123046,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":358,"num_read_kb":4228,"num_write":477,"num_write_kb":113967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":82068207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":53,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-04-22T18:42:39.210665+0000","last_change":"2024-04-22T18:42:39.210665+0000","last_active":"2024-04-22T18:42:39.210665+0000","last_peered":"2024-04-22T18:42:39.210665+0000","last_clean":"2024-04-22T18:42:39.210665+0000","last_became_active":"2024-04-22T18:42:09.929115+0000","last_became_peered":"2024-04-22T18:42:09.929115+0000","last_unstale":"2024-04-22T18:42:39.210665+0000","last_undegraded":"2024-04-22T18:42:39.210665+0000","last_fullsized":"2024-04-22T18:42:39.210665+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:39.210519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:39.210519+0000","last_clean_scrub_stamp":"2024-04-22T18:42:39.210519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:44:08.999688+0000","scrub_duration":0.20164906399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"200'362","reported_seq":867,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-22T18:42:46.094177+0000","last_change":"2024-04-22T18:42:46.094298+0000","last_active":"2024-04-22T18:42:46.094177+0000","last_peered":"2024-04-22T18:42:46.094177+0000","last_clean":"2024-04-22T18:42:46.094177+0000","last_became_active":"2024-04-22T18:39:52.046702+0000","last_became_peered":"2024-04-22T18:39:52.046702+0000","last_unstale":"2024-04-22T18:42:46.094177+0000","last_undegraded":"2024-04-22T18:42:46.094177+0000","last_fullsized":"2024-04-22T18:42:46.094177+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'362","last_scrub_stamp":"2024-04-22T18:42:46.094154+0000","last_deep_scrub":"200'362","last_deep_scrub_stamp":"2024-04-22T18:42:46.094154+0000","last_clean_scrub_stamp":"2024-04-22T18:42:46.094154+0000","objects_scrubbed":53,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:52.510269+0000","scrub_duration":1.8603612360000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382624,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":306,"num_read_kb":4987,"num_write":404,"num_write_kb":80545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":20859324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":193,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-04-22T18:42:38.408897+0000","last_change":"2024-04-22T18:42:38.408897+0000","last_active":"2024-04-22T18:42:38.408897+0000","last_peered":"2024-04-22T18:42:38.408897+0000","last_clean":"2024-04-22T18:42:38.408897+0000","last_became_active":"2024-04-22T18:39:52.029562+0000","last_became_peered":"2024-04-22T18:39:52.029562+0000","last_unstale":"2024-04-22T18:42:38.408897+0000","last_undegraded":"2024-04-22T18:42:38.408897+0000","last_fullsized":"2024-04-22T18:42:38.408897+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:38.408722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:38.408722+0000","last_clean_scrub_stamp":"2024-04-22T18:42:38.408722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:58.495278+0000","scrub_duration":0.30157335299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"214'391","reported_seq":1502,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-04-22T18:42:41.120449+0000","last_change":"2024-04-22T18:42:41.120449+0000","last_active":"2024-04-22T18:42:41.120449+0000","last_peered":"2024-04-22T18:42:41.120449+0000","last_clean":"2024-04-22T18:42:41.120449+0000","last_became_active":"2024-04-22T18:41:41.007340+0000","last_became_peered":"2024-04-22T18:41:41.007340+0000","last_unstale":"2024-04-22T18:42:41.120449+0000","last_undegraded":"2024-04-22T18:42:41.120449+0000","last_fullsized":"2024-04-22T18:42:41.120449+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'391","last_scrub_stamp":"2024-04-22T18:42:41.120260+0000","last_deep_scrub":"214'391","last_deep_scrub_stamp":"2024-04-22T18:42:41.120260+0000","last_clean_scrub_stamp":"2024-04-22T18:42:41.120260+0000","objects_scrubbed":54,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:49.709884+0000","scrub_duration":3.2875705389999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39035424,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":322,"num_read_kb":9140,"num_write":423,"num_write_kb":104979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":67954083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":238,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-04-22T18:42:41.449820+0000","last_change":"2024-04-22T18:42:40.074437+0000","last_active":"2024-04-22T18:42:41.449820+0000","last_peered":"2024-04-22T18:42:41.449820+0000","last_clean":"2024-04-22T18:42:41.449820+0000","last_became_active":"2024-04-22T18:38:39.125423+0000","last_became_peered":"2024-04-22T18:38:39.125423+0000","last_unstale":"2024-04-22T18:42:41.449820+0000","last_undegraded":"2024-04-22T18:42:41.449820+0000","last_fullsized":"2024-04-22T18:42:41.449820+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:40.074319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:40.074319+0000","last_clean_scrub_stamp":"2024-04-22T18:42:40.074319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:51.350937+0000","scrub_duration":0.0017873800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"215'389","reported_seq":1422,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-22T18:42:49.289515+0000","last_change":"2024-04-22T18:42:49.289655+0000","last_active":"2024-04-22T18:42:49.289515+0000","last_peered":"2024-04-22T18:42:49.289515+0000","last_clean":"2024-04-22T18:42:49.289515+0000","last_became_active":"2024-04-22T18:42:18.879845+0000","last_became_peered":"2024-04-22T18:42:18.879845+0000","last_unstale":"2024-04-22T18:42:49.289515+0000","last_undegraded":"2024-04-22T18:42:49.289515+0000","last_fullsized":"2024-04-22T18:42:49.289515+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'389","last_scrub_stamp":"2024-04-22T18:42:49.289489+0000","last_deep_scrub":"215'389","last_deep_scrub_stamp":"2024-04-22T18:42:49.289489+0000","last_clean_scrub_stamp":"2024-04-22T18:42:49.289489+0000","objects_scrubbed":61,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:44:16.753753+0000","scrub_duration":3.6878648900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43117026,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":302,"num_read_kb":12417,"num_write":418,"num_write_kb":91859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":72521286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":42,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-04-22T18:42:40.124449+0000","last_change":"2024-04-22T18:42:40.124449+0000","last_active":"2024-04-22T18:42:40.124449+0000","last_peered":"2024-04-22T18:42:40.124449+0000","last_clean":"2024-04-22T18:42:40.124449+0000","last_became_active":"2024-04-22T18:42:10.036239+0000","last_became_peered":"2024-04-22T18:42:10.036239+0000","last_unstale":"2024-04-22T18:42:40.124449+0000","last_undegraded":"2024-04-22T18:42:40.124449+0000","last_fullsized":"2024-04-22T18:42:40.124449+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T18:42:40.124278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T18:42:40.124278+0000","last_clean_scrub_stamp":"2024-04-22T18:42:40.124278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:51.610152+0000","scrub_duration":0.101908919,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"215'119","reported_seq":435,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-04-22T18:42:43.310468+0000","last_change":"2024-04-22T18:42:43.310622+0000","last_active":"2024-04-22T18:42:43.310468+0000","last_peered":"2024-04-22T18:42:43.310468+0000","last_clean":"2024-04-22T18:42:43.310468+0000","last_became_active":"2024-04-22T18:41:26.757958+0000","last_became_peered":"2024-04-22T18:41:26.757958+0000","last_unstale":"2024-04-22T18:42:43.310468+0000","last_undegraded":"2024-04-22T18:42:43.310468+0000","last_fullsized":"2024-04-22T18:42:43.310468+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'119","last_scrub_stamp":"2024-04-22T18:42:43.310446+0000","last_deep_scrub":"215'119","last_deep_scrub_stamp":"2024-04-22T18:42:43.310446+0000","last_clean_scrub_stamp":"2024-04-22T18:42:43.310446+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:43:59.911713+0000","scrub_duration":0.30623825700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":155658120,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":1288,"num_read_kb":30772,"num_write":1722,"num_write_kb":391350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1297,"num_bytes_recovered":243402900,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249495552,"data_stored":263913197,"data_compressed":14626013,"data_compressed_allocated":15568896,"data_compressed_original":30834688,"omap_allocated":0,"internal_metadata":0},"log_size":1580,"ondisk_log_size":1580,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":2115712,"data_compressed":122550,"data_compressed_allocated":163840,"data_compressed_original":2084864,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36840,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94335000,"statfs":{"total":96636764160,"available":96599040000,"internally_reserved":0,"allocated":1318912,"data_stored":1099209,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":2.597,"15min":1.147},"min":{"1min":0.29199999999999998,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.65100000000000002,"5min":139.84299999999999,"15min":139.84299999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.49299999999999999,"5min":5.9630000000000001,"15min":4.9930000000000003},"min":{"1min":0.315,"5min":0.247,"15min":0.247},"max":{"1min":0.70099999999999996,"5min":186.97900000000001,"15min":186.97900000000001},"last":0.44900000000000001}]},{"osd":1,"last update":"Mon Apr 22 18:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":3.1059999999999999,"15min":4.2779999999999996},"min":{"1min":0.251,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.627,"5min":79.927999999999997,"15min":79.927999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":9.6280000000000001,"15min":11.907999999999999},"min":{"1min":0.32400000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.64800000000000002,"5min":183.16300000000001,"15min":183.16300000000001},"last":0.69999999999999996}]},{"osd":2,"last update":"Mon Apr 22 18:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":3.3519999999999999,"15min":3.8399999999999999},"min":{"1min":0.28599999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.66000000000000003,"5min":97.662000000000006,"15min":97.662000000000006},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.48099999999999998,"15min":0.47599999999999998},"min":{"1min":0.32500000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68100000000000005,"5min":1.599,"15min":1.599},"last":0.59799999999999998}]},{"osd":3,"last update":"Mon Apr 22 18:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":3.4020000000000001,"15min":3.9009999999999998},"min":{"1min":0.314,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71799999999999997,"5min":97.686999999999998,"15min":97.686999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52200000000000002,"15min":0.52600000000000002},"min":{"1min":0.36399999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64700000000000002,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.624}]},{"osd":4,"last update":"Mon Apr 22 18:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":4.6470000000000002,"15min":4.5209999999999999},"min":{"1min":0.247,"5min":0.218,"15min":0.218},"max":{"1min":0.59999999999999998,"5min":85.509,"15min":85.509},"last":0.314},{"interface":"front","average":{"1min":0.45200000000000001,"5min":12.372,"15min":19.372},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61599999999999999,"5min":327.77300000000002,"15min":327.77300000000002},"last":0.36899999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":1.506,"15min":2.5489999999999999},"min":{"1min":0.25900000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69499999999999995,"5min":55.116999999999997,"15min":55.116999999999997},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":3.9710000000000001,"15min":6.0979999999999999},"min":{"1min":0.33400000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.68300000000000005,"5min":85.475999999999999,"15min":85.475999999999999},"last":0.40999999999999998}]},{"osd":6,"last update":"Mon Apr 22 18:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":9.1980000000000004,"15min":11.916},"min":{"1min":0.309,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.65900000000000003,"5min":139.428,"15min":139.428},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":3.5790000000000002,"15min":3.948},"min":{"1min":0.33200000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.63100000000000001,"5min":97.635999999999996,"15min":97.635999999999996},"last":0.42999999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574903,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32652,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339188,"statfs":{"total":96636764160,"available":96603328512,"internally_reserved":0,"allocated":1220608,"data_stored":567376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":2.802,"15min":2.1299999999999999},"min":{"1min":0.307,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.70699999999999996,"5min":70.436000000000007,"15min":70.436000000000007},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":11.691000000000001,"15min":21.102},"min":{"1min":0.36399999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.77700000000000002,"5min":166.85499999999999,"15min":166.85499999999999},"last":0.57799999999999996}]},{"osd":1,"last update":"Mon Apr 22 18:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":3.2650000000000001,"15min":2.4470000000000001},"min":{"1min":0.33700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.67200000000000004,"5min":199.047,"15min":199.047},"last":0.60199999999999998},{"interface":"front","average":{"1min":3.6509999999999998,"5min":7.3639999999999999,"15min":9.1820000000000004},"min":{"1min":0.34799999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":49.021000000000001,"5min":222.55199999999999,"15min":222.55199999999999},"last":0.51600000000000001}]},{"osd":2,"last update":"Mon Apr 22 18:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.46300000000000002,"15min":0.46100000000000002},"min":{"1min":0.40799999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69599999999999995,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.497,"5min":8.8989999999999991,"15min":15.502000000000001},"min":{"1min":0.315,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.80500000000000005,"5min":166.81800000000001,"15min":166.81800000000001},"last":0.53000000000000003}]},{"osd":3,"last update":"Mon Apr 22 18:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":10.614000000000001,"15min":17.884},"min":{"1min":0.32300000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73799999999999999,"5min":171.99100000000001,"15min":171.99100000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.502,"5min":1.3460000000000001,"15min":0.747},"min":{"1min":0.38,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.72899999999999998,"5min":77.097999999999999,"15min":77.097999999999999},"last":0.54300000000000004}]},{"osd":4,"last update":"Mon Apr 22 18:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":3.617,"15min":6.274},"min":{"1min":0.27800000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.66100000000000003,"5min":81.697000000000003,"15min":81.697000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":2.5390000000000001,"5min":12.954000000000001,"15min":17.202999999999999},"min":{"1min":0.251,"5min":0.218,"15min":0.218},"max":{"1min":40.317,"5min":168.44,"15min":168.44},"last":0.432}]},{"osd":5,"last update":"Mon Apr 22 18:42:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":0.622,"15min":0.47399999999999998},"min":{"1min":0.26900000000000002,"5min":0.252,"15min":0.252},"max":{"1min":22.190999999999999,"5min":22.190999999999999,"15min":22.190999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.45200000000000001,"15min":0.44800000000000001},"min":{"1min":0.312,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.64800000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.59199999999999997}]},{"osd":7,"last update":"Mon Apr 22 18:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":10.755000000000001,"15min":16.152000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.63700000000000001,"5min":166.79400000000001,"15min":166.79400000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.59999999999999998,"15min":0.51000000000000001},"min":{"1min":0.30199999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.627,"5min":13.077,"15min":13.077},"last":0.63}]}]},{"osd":5,"up_from":13,"seq":55834574903,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58900,"kb_used_data":27788,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94312940,"statfs":{"total":96636764160,"available":96576450560,"internally_reserved":0,"allocated":28454912,"data_stored":29263174,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:42:47 2024","interfaces":[{"interface":"back","average":{"1min":1.72,"5min":7.54,"15min":11.441000000000001},"min":{"1min":0.35099999999999998,"5min":0.192,"15min":0.192},"max":{"1min":25.673999999999999,"5min":74.019999999999996,"15min":74.019999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":4.6609999999999996,"5min":1.4630000000000001,"15min":1.091},"min":{"1min":0.28299999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":88.903999999999996,"5min":88.903999999999996,"15min":88.903999999999996},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Apr 22 18:42:47 2024","interfaces":[{"interface":"back","average":{"1min":9.4120000000000008,"5min":10.808999999999999,"15min":10.628},"min":{"1min":0.314,"5min":0.224,"15min":0.224},"max":{"1min":187.739,"5min":202.13800000000001,"15min":202.13800000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":1.3620000000000001,"15min":0.73299999999999998},"min":{"1min":0.33000000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.91600000000000004,"5min":68.346999999999994,"15min":68.346999999999994},"last":0.46899999999999997}]},{"osd":2,"last update":"Mon Apr 22 18:42:47 2024","interfaces":[{"interface":"back","average":{"1min":4.6890000000000001,"5min":1.266,"15min":0.69599999999999995},"min":{"1min":0.34999999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":88.924999999999997,"5min":88.924999999999997,"15min":88.924999999999997},"last":0.498},{"interface":"front","average":{"1min":0.49399999999999999,"5min":2.4300000000000002,"15min":4.1689999999999996},"min":{"1min":0.34000000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.74299999999999999,"5min":78.296999999999997,"15min":78.296999999999997},"last":0.53400000000000003}]},{"osd":3,"last update":"Mon Apr 22 18:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":3.5059999999999998,"15min":5.0010000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.88600000000000001,"5min":89.688000000000002,"15min":89.688000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":4.5709999999999997,"15min":8.4329999999999998},"min":{"1min":0.376,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.67700000000000005,"5min":89.718999999999994,"15min":89.718999999999994},"last":0.48499999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:42:47 2024","interfaces":[{"interface":"back","average":{"1min":7.6420000000000003,"5min":13.451000000000001,"15min":16.297000000000001},"min":{"1min":0.29099999999999998,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":90.509,"5min":202.983,"15min":202.983},"last":0.41099999999999998},{"interface":"front","average":{"1min":11.9,"5min":13.428000000000001,"15min":16.331},"min":{"1min":0.33000000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":90.543000000000006,"5min":202.999,"15min":202.999},"last":0.57799999999999996}]},{"osd":6,"last update":"Mon Apr 22 18:42:47 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":3.4550000000000001,"15min":4.5090000000000003},"min":{"1min":0.26700000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.64700000000000002,"5min":79.945999999999998,"15min":79.945999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":4.1020000000000003,"15min":5.5860000000000003},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.96099999999999997,"5min":100.54900000000001,"15min":100.54900000000001},"last":0.51500000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:42:47 2024","interfaces":[{"interface":"back","average":{"1min":4.6470000000000002,"5min":1.9870000000000001,"15min":0.96599999999999997},"min":{"1min":0.27900000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":88.843999999999994,"5min":88.843999999999994,"15min":88.843999999999994},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.47999999999999998,"5min":3.2530000000000001,"15min":4.4710000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.70599999999999996,"5min":78.340999999999994,"15min":78.340999999999994},"last":0.59099999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93452,"kb_used_data":54148,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94278388,"statfs":{"total":96636764160,"available":96541069312,"internally_reserved":0,"allocated":55447552,"data_stored":57669195,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:42:38 2024","interfaces":[{"interface":"back","average":{"1min":1.9410000000000001,"5min":10.173999999999999,"15min":14.49},"min":{"1min":0.23400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":30.177,"5min":252.96199999999999,"15min":252.96199999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":2.262,"5min":8.0210000000000008,"15min":8.3849999999999998},"min":{"1min":0.29799999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":36.042000000000002,"5min":187.148,"15min":187.148},"last":0.60299999999999998}]},{"osd":1,"last update":"Mon Apr 22 18:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":2.5190000000000001,"15min":1.119},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73399999999999999,"5min":96.486999999999995,"15min":96.486999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.498,"5min":10.569000000000001,"15min":7.1130000000000004},"min":{"1min":0.29899999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72499999999999998,"5min":310.61500000000001,"15min":310.61500000000001},"last":0.62}]},{"osd":2,"last update":"Mon Apr 22 18:42:38 2024","interfaces":[{"interface":"back","average":{"1min":2.3359999999999999,"5min":7.6520000000000001,"15min":8.2639999999999993},"min":{"1min":0.35699999999999998,"5min":0.254,"15min":0.254},"max":{"1min":36.021999999999998,"5min":186.94300000000001,"15min":186.94300000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":2.4329999999999998,"15min":1.1080000000000001},"min":{"1min":0.317,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.83199999999999996,"5min":96.471999999999994,"15min":96.471999999999994},"last":0.58599999999999997}]},{"osd":3,"last update":"Mon Apr 22 18:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":8.2739999999999991,"15min":5.4390000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.74299999999999999,"5min":252.91900000000001,"15min":252.91900000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":2.3340000000000001,"5min":13.515000000000001,"15min":18.483000000000001},"min":{"1min":0.35499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":36.067999999999998,"5min":227.41200000000001,"15min":227.41200000000001},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Apr 22 18:42:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.355,"15min":1.04},"min":{"1min":0.26500000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.76900000000000002,"5min":96.408000000000001,"15min":96.408000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":2.3570000000000002,"15min":3.4769999999999999},"min":{"1min":0.30199999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.81399999999999995,"5min":78.811999999999998,"15min":78.811999999999998},"last":0.38900000000000001}]},{"osd":6,"last update":"Mon Apr 22 18:42:38 2024","interfaces":[{"interface":"back","average":{"1min":2.3319999999999999,"5min":12.500999999999999,"15min":12.94},"min":{"1min":0.32300000000000001,"5min":0.247,"15min":0.247},"max":{"1min":35.969000000000001,"5min":197.72999999999999,"15min":197.72999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":2.4060000000000001,"15min":1.0960000000000001},"min":{"1min":0.33300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.88100000000000001,"5min":96.438000000000002,"15min":96.438000000000002},"last":0.33100000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:42:38 2024","interfaces":[{"interface":"back","average":{"1min":2.379,"5min":8.3379999999999992,"15min":8.4860000000000007},"min":{"1min":0.37,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":35.996000000000002,"5min":186.79300000000001,"15min":186.79300000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.79800000000000004,"5min":2.7050000000000001,"15min":1.2230000000000001},"min":{"1min":0.377,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.21,"5min":96.456999999999994,"15min":96.456999999999994},"last":0.373}]}]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63032,"kb_used_data":27824,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94308808,"statfs":{"total":96636764160,"available":96572219392,"internally_reserved":0,"allocated":28491776,"data_stored":29267059,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:42:43 2024","interfaces":[{"interface":"back","average":{"1min":1.931,"5min":8.484,"15min":10.961},"min":{"1min":0.20399999999999999,"5min":0.182,"15min":0.182},"max":{"1min":31.172000000000001,"5min":98.957999999999998,"15min":98.957999999999998},"last":0.28299999999999997},{"interface":"front","average":{"1min":3.7109999999999999,"5min":2.3420000000000001,"15min":1},"min":{"1min":0.20100000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":65.275999999999996,"5min":87.305999999999997,"15min":87.305999999999997},"last":0.36499999999999999}]},{"osd":1,"last update":"Mon Apr 22 18:42:40 2024","interfaces":[{"interface":"back","average":{"1min":2.593,"5min":10.067,"15min":6.7190000000000003},"min":{"1min":0.23300000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":41.572000000000003,"5min":197.02099999999999,"15min":197.02099999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.31900000000000001,"5min":2.3180000000000001,"15min":0.996},"min":{"1min":0.26500000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47199999999999998,"5min":78.393000000000001,"15min":78.393000000000001},"last":0.40600000000000003}]},{"osd":2,"last update":"Mon Apr 22 18:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":2.2130000000000001,"15min":1.4079999999999999},"min":{"1min":0.26600000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.53800000000000003,"5min":87.635999999999996,"15min":87.635999999999996},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":1.7130000000000001,"15min":0.85399999999999998},"min":{"1min":0.26600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55200000000000005,"5min":78.421999999999997,"15min":78.421999999999997},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Apr 22 18:42:40 2024","interfaces":[{"interface":"back","average":{"1min":4.1369999999999996,"5min":3,"15min":3.738},"min":{"1min":0.34699999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":65.483999999999995,"5min":71.998000000000005,"15min":71.998000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":3.9079999999999999,"15min":4.0609999999999999},"min":{"1min":0.376,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71599999999999997,"5min":87.724000000000004,"15min":87.724000000000004},"last":0.66500000000000004}]},{"osd":5,"last update":"Mon Apr 22 18:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":2.3769999999999998,"15min":1.5800000000000001},"min":{"1min":0.39300000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.82999999999999996,"5min":87.784000000000006,"15min":87.784000000000006},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":2.1429999999999998,"15min":1.552},"min":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.74199999999999999,"5min":78.480999999999995,"15min":78.480999999999995},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Apr 22 18:42:40 2024","interfaces":[{"interface":"back","average":{"1min":4.0970000000000004,"5min":1.22,"15min":0.76800000000000002},"min":{"1min":0.377,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":65.501000000000005,"5min":65.501000000000005,"15min":65.501000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.52200000000000002,"5min":2.8690000000000002,"15min":1.3109999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":96.140000000000001,"15min":96.140000000000001},"last":0.59899999999999998}]},{"osd":7,"last update":"Mon Apr 22 18:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.889,"15min":1.0189999999999999},"min":{"1min":0.39900000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.75800000000000001,"5min":78.453999999999994,"15min":78.453999999999994},"last":0.64700000000000002},{"interface":"front","average":{"1min":4.1200000000000001,"5min":1.246,"15min":0.79400000000000004},"min":{"1min":0.36799999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":65.442999999999998,"5min":65.442999999999998,"15min":65.442999999999998},"last":0.58799999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574903,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51664,"kb_used_data":20208,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94320176,"statfs":{"total":96636764160,"available":96583860224,"internally_reserved":0,"allocated":20692992,"data_stored":21499369,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":5.1879999999999997,"15min":7.1470000000000002},"min":{"1min":0.17999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.64900000000000002,"5min":149.03,"15min":149.03},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.98399999999999999,"15min":1.651},"min":{"1min":0.223,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.63500000000000001,"5min":31.866,"15min":31.866},"last":0.40500000000000003}]},{"osd":1,"last update":"Mon Apr 22 18:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":5.109,"15min":2.391},"min":{"1min":0.24299999999999999,"5min":0.193,"15min":0.193},"max":{"1min":0.43099999999999999,"5min":97.683999999999997,"15min":97.683999999999997},"last":0.435},{"interface":"front","average":{"1min":0.35999999999999999,"5min":13.007,"15min":14.259},"min":{"1min":0.26800000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.53300000000000003,"5min":393.42399999999998,"15min":393.42399999999998},"last":0.46100000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:42:44 2024","interfaces":[{"interface":"back","average":{"1min":4.125,"5min":7.0359999999999996,"15min":2.5870000000000002},"min":{"1min":0.251,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":74.870999999999995,"5min":147.721,"15min":147.721},"last":0.51700000000000002},{"interface":"front","average":{"1min":5.4089999999999998,"5min":1.363,"15min":0.70899999999999996},"min":{"1min":0.26800000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":101.43300000000001,"5min":101.43300000000001,"15min":101.43300000000001},"last":0.47799999999999998}]},{"osd":4,"last update":"Mon Apr 22 18:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":9.9220000000000006,"15min":15.821999999999999},"min":{"1min":0.35799999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72999999999999998,"5min":160.584,"15min":160.584},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":4.4740000000000002,"15min":3.1230000000000002},"min":{"1min":0.38,"5min":0.315,"15min":0.315},"max":{"1min":0.751,"5min":97.887,"15min":97.887},"last":0.71499999999999997}]},{"osd":5,"last update":"Mon Apr 22 18:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":1.5249999999999999,"15min":2.5430000000000001},"min":{"1min":0.34899999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67300000000000004,"5min":48.631,"15min":48.631},"last":0.623},{"interface":"front","average":{"1min":0.56899999999999995,"5min":1.7070000000000001,"15min":2.8860000000000001},"min":{"1min":0.37,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.90200000000000002,"5min":56.18,"15min":56.18},"last":0.58799999999999997}]},{"osd":6,"last update":"Mon Apr 22 18:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":3.4769999999999999,"15min":1.516},"min":{"1min":0.39500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.86399999999999999,"5min":97.832999999999998,"15min":97.832999999999998},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.42799999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.80300000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Apr 22 18:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":1.577,"15min":0.90900000000000003},"min":{"1min":0.40799999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.84399999999999997,"5min":90.944999999999993,"15min":90.944999999999993},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":3.9249999999999998,"15min":1.698},"min":{"1min":0.40500000000000003,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.92200000000000004,"5min":97.861999999999995,"15min":97.861999999999995},"last":0.748}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59552,"kb_used_data":20248,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94312288,"statfs":{"total":96636764160,"available":96575782912,"internally_reserved":0,"allocated":20733952,"data_stored":21504200,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 18:42:44 2024","interfaces":[{"interface":"back","average":{"1min":1.879,"5min":4.6870000000000003,"15min":6.8010000000000002},"min":{"1min":0.215,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":27.411000000000001,"5min":88.171000000000006,"15min":88.171000000000006},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.885,"5min":5.4909999999999997,"15min":7.0720000000000001},"min":{"1min":0.26900000000000002,"5min":0.223,"15min":0.223},"max":{"1min":27.452000000000002,"5min":94.046000000000006,"15min":94.046000000000006},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Apr 22 18:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":11.509,"15min":9.5690000000000008},"min":{"1min":0.20999999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.1419999999999999,"5min":406.68799999999999,"15min":406.68799999999999},"last":0.313},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.626,"15min":0.41299999999999998},"min":{"1min":0.23699999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.90900000000000003,"5min":27.094999999999999,"15min":27.094999999999999},"last":0.33300000000000002}]},{"osd":3,"last update":"Mon Apr 22 18:42:45 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":3.4740000000000002,"15min":1.3799999999999999},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.86799999999999999,"5min":228.90100000000001,"15min":228.90100000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.34200000000000003,"5min":10.231999999999999,"15min":13.071999999999999},"min":{"1min":0.23899999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.69099999999999995,"5min":231.864,"15min":231.864},"last":0.36299999999999999}]},{"osd":4,"last update":"Mon Apr 22 18:42:46 2024","interfaces":[{"interface":"back","average":{"1min":31.378,"5min":20.890999999999998,"15min":15.619},"min":{"1min":0.36199999999999999,"5min":0.309,"15min":0.309},"max":{"1min":500.81900000000002,"5min":500.81900000000002,"15min":500.81900000000002},"last":0.55800000000000005},{"interface":"front","average":{"1min":26.916,"5min":10.682,"15min":3.8929999999999998},"min":{"1min":0.34599999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":501.77600000000001,"5min":501.77600000000001,"15min":501.77600000000001},"last":0.46600000000000003}]},{"osd":5,"last update":"Mon Apr 22 18:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":7.0389999999999997,"15min":11.727},"min":{"1min":0.34399999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.0089999999999999,"5min":138.089,"15min":138.089},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":16.216000000000001,"15min":18.789999999999999},"min":{"1min":0.33700000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.198,"5min":406.72899999999998,"15min":406.72899999999998},"last":0.59599999999999997}]},{"osd":6,"last update":"Mon Apr 22 18:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":3.3889999999999998,"15min":2.9809999999999999},"min":{"1min":0.35899999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.94999999999999996,"5min":94.302999999999997,"15min":94.302999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":16.515999999999998,"15min":16.526},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.2549999999999999,"5min":427.69299999999998,"15min":427.69299999999998},"last":0.45300000000000001}]},{"osd":7,"last update":"Mon Apr 22 18:42:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.83499999999999996,"15min":0.629},"min":{"1min":0.374,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":27.253,"15min":27.253},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54900000000000004,"5min":2.6309999999999998,"15min":2.903},"min":{"1min":0.41699999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.98199999999999998,"5min":94.260999999999996,"15min":94.260999999999996},"last":0.47899999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574903,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89396,"kb_used_data":54188,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94282444,"statfs":{"total":96636764160,"available":96545222656,"internally_reserved":0,"allocated":55488512,"data_stored":58197062,"data_compressed":3021454,"data_compressed_allocated":3239936,"data_compressed_original":6823936,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 18:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":8.2859999999999996,"15min":7.2679999999999998},"min":{"1min":0.20399999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":1.2649999999999999,"5min":133.09899999999999,"15min":133.09899999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":11.282999999999999,"15min":4.7649999999999997},"min":{"1min":0.214,"5min":0.188,"15min":0.188},"max":{"1min":0.78400000000000003,"5min":451.50799999999998,"15min":451.50799999999998},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon Apr 22 18:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":1.3129999999999999,"15min":1.4750000000000001},"min":{"1min":0.32400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":1.115,"5min":39.906999999999996,"15min":39.906999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":2.222,"15min":0.95199999999999996},"min":{"1min":0.23000000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.1819999999999999,"5min":94.820999999999998,"15min":94.820999999999998},"last":0.436}]},{"osd":3,"last update":"Mon Apr 22 18:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":3.5219999999999998,"15min":1.4830000000000001},"min":{"1min":0.27200000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.3069999999999999,"5min":99.332999999999998,"15min":99.332999999999998},"last":0.37},{"interface":"front","average":{"1min":0.42899999999999999,"5min":8.9890000000000008,"15min":4.0380000000000003},"min":{"1min":0.29799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.1479999999999999,"5min":297.82100000000003,"15min":297.82100000000003},"last":0.40699999999999997}]},{"osd":4,"last update":"Mon Apr 22 18:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":3.5009999999999999,"15min":1.5029999999999999},"min":{"1min":0.374,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0009999999999999,"5min":96.528999999999996,"15min":96.528999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.624,"5min":6.9420000000000002,"15min":8.9220000000000006},"min":{"1min":0.41199999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.2170000000000001,"5min":160.02199999999999,"15min":160.02199999999999},"last":0.81499999999999995}]},{"osd":5,"last update":"Mon Apr 22 18:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.956,"15min":1.8149999999999999},"min":{"1min":0.41199999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.93000000000000005,"5min":41.645000000000003,"15min":41.645000000000003},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":7.7789999999999999,"15min":11.896000000000001},"min":{"1min":0.39000000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.76000000000000001,"5min":282.197,"15min":282.197},"last":0.72799999999999998}]},{"osd":6,"last update":"Mon Apr 22 18:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":2.77,"15min":1.3819999999999999},"min":{"1min":0.40600000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.2430000000000001,"5min":99.355000000000004,"15min":99.355000000000004},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":2.6659999999999999,"15min":2.0539999999999998},"min":{"1min":0.42499999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.95599999999999996,"5min":52.762,"15min":52.762},"last":0.68400000000000005}]},{"osd":7,"last update":"Mon Apr 22 18:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":4.077,"15min":4.4770000000000003},"min":{"1min":0.41599999999999998,"5min":0.317,"15min":0.317},"max":{"1min":1.0489999999999999,"5min":94.662999999999997,"15min":94.662999999999997},"last":0.754},{"interface":"front","average":{"1min":0.68100000000000005,"5min":4.2889999999999997,"15min":1.899},"min":{"1min":0.42599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.286,"5min":157.089,"15min":157.089},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":30994,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29568,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19447808,"data_stored":20930426,"data_compressed":1339853,"data_compressed_allocated":1421312,"data_compressed_original":2990080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":54169600,"data_stored":57096091,"data_compressed":2990460,"data_compressed_allocated":3198976,"data_compressed_original":6299648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":27222016,"data_stored":28694786,"data_compressed":1537444,"data_compressed_allocated":1630208,"data_compressed_original":3198976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":27324416,"data_stored":28849251,"data_compressed":1581247,"data_compressed_allocated":1675264,"data_compressed_original":3284992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20492288,"data_stored":21621340,"data_compressed":1309252,"data_compressed_allocated":1392640,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T18:42:59.845 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-22T18:43:00.139 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-22T18:43:00.139 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-22T18:43:00.139 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-22T18:43:00.139 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T18:43:00.171 INFO:tasks.ceph.osd.0:Stopped 2024-04-22T18:43:00.172 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-22T18:43:00.172 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T18:43:00.244 INFO:tasks.ceph.osd.1:Stopped 2024-04-22T18:43:00.244 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-22T18:43:00.244 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T18:43:00.311 INFO:tasks.ceph.osd.2:Stopped 2024-04-22T18:43:00.312 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-22T18:43:00.312 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T18:43:00.385 INFO:tasks.ceph.osd.3:Stopped 2024-04-22T18:43:00.385 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-22T18:43:00.385 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T18:43:00.418 INFO:tasks.ceph.osd.4:Stopped 2024-04-22T18:43:00.418 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-22T18:43:00.418 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T18:43:00.484 INFO:tasks.ceph.osd.5:Stopped 2024-04-22T18:43:00.484 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-22T18:43:00.485 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T18:43:00.559 INFO:tasks.ceph.osd.6:Stopped 2024-04-22T18:43:00.559 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-22T18:43:00.559 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T18:43:00.630 INFO:tasks.ceph.osd.7:Stopped 2024-04-22T18:43:00.630 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-22T18:43:00.631 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-22T18:43:00.631 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T18:43:00.656 INFO:tasks.ceph.mgr.y:Stopped 2024-04-22T18:43:00.656 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-22T18:43:00.656 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T18:43:00.697 INFO:tasks.ceph.mgr.x:Stopped 2024-04-22T18:43:00.698 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-22T18:43:00.698 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-22T18:43:00.698 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T18:43:00.709 INFO:tasks.ceph.mon.a:Stopped 2024-04-22T18:43:00.709 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-22T18:43:00.709 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T18:43:00.759 INFO:tasks.ceph.mon.c:Stopped 2024-04-22T18:43:00.759 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-22T18:43:00.760 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T18:43:00.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-22T18:43:00.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-22T18:43:00.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-22T18:43:00.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-04-22T18:43:00.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-04-22T18:43:00.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-04-22T18:43:00.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-04-22T18:43:00.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-04-22T18:43:00.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-04-22T18:43:00.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-04-22T18:43:00.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-04-22T18:43:00.772 INFO:tasks.ceph.mon.b:Stopped 2024-04-22T18:43:00.773 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-22T18:43:00.773 DEBUG:teuthology.orchestra.run.smithi089:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-22T18:43:00.837 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-22T18:43:00.837 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-22T18:43:01.058 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-22T18:43:01.058 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-22T18:43:01.140 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-22T18:43:01.140 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-22T18:43:01.222 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi089.front.sepia.ceph.com 2024-04-22T18:43:01.222 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-22T18:43:01.279 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-22T18:43:01.280 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-22T18:43:01.400 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-22T18:43:01.400 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-22T18:43:01.505 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-22T18:43:01.505 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-22T18:43:01.563 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-22T18:43:01.563 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-22T18:43:01.612 INFO:tasks.ceph:Archiving mon data... 2024-04-22T18:43:01.613 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668454/data/mon.a.tgz 2024-04-22T18:43:01.613 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-04-22T18:43:01.628 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.FVizpcmL9b 2024-04-22T18:43:01.629 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.FVizpcmL9b 2024-04-22T18:43:01.795 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.FVizpcmL9b 2024-04-22T18:43:01.876 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.FVizpcmL9b is 469KB 2024-04-22T18:43:01.944 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.FVizpcmL9b 2024-04-22T18:43:01.958 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668454/data/mon.c.tgz 2024-04-22T18:43:01.958 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-04-22T18:43:02.011 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.B4JXWpZjL4 2024-04-22T18:43:02.012 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.B4JXWpZjL4 2024-04-22T18:43:02.157 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.B4JXWpZjL4 2024-04-22T18:43:02.237 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.B4JXWpZjL4 is 469KB 2024-04-22T18:43:02.333 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.B4JXWpZjL4 2024-04-22T18:43:02.349 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668454/data/mon.b.tgz 2024-04-22T18:43:02.350 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-04-22T18:43:02.367 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.WzXYPKqID1 2024-04-22T18:43:02.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.WzXYPKqID1 2024-04-22T18:43:02.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.WzXYPKqID1 2024-04-22T18:43:02.609 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.WzXYPKqID1 is 469KB 2024-04-22T18:43:02.707 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.WzXYPKqID1 2024-04-22T18:43:02.721 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-22T18:43:02.721 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-22T18:43:02.724 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-22T18:43:02.784 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-22T18:43:02.784 DEBUG:teuthology.orchestra.run.smithi089:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-22T18:43:02.788 DEBUG:teuthology.orchestra.run.smithi121:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-22T18:43:02.811 INFO:teuthology.orchestra.run.smithi089.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-22T18:43:02.849 INFO:teuthology.orchestra.run.smithi121.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-22T18:43:02.853 INFO:tasks.ceph:Archiving crash dumps... 2024-04-22T18:43:02.855 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668454/remote/smithi089/crash 2024-04-22T18:43:02.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-22T18:43:02.879 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668454/remote/smithi121/crash 2024-04-22T18:43:02.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-22T18:43:02.915 INFO:tasks.ceph:Compressing logs... 2024-04-22T18:43:02.915 DEBUG:teuthology.orchestra.run.smithi089:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T18:43:02.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T18:43:05.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-22T18:43:05.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-22T18:43:05.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-22T18:43:05.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-04-22T18:43:05.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-04-22T18:43:05.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-04-22T18:43:05.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-04-22T18:43:05.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-04-22T18:43:05.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-04-22T18:43:05.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-04-22T18:43:05.774 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-04-22T18:43:11.264 INFO:tasks.ceph:Archiving logs... 2024-04-22T18:43:11.264 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668454/remote/smithi089/log 2024-04-22T18:43:11.265 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T18:43:12.670 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668454/remote/smithi121/log 2024-04-22T18:43:12.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T18:43:13.429 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T18:43:13.439 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T18:43:13.439 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T18:43:13.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T18:43:13.529 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T18:43:13.538 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T18:43:13.539 DEBUG:teuthology.orchestra.run.smithi089:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T18:43:13.541 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T18:43:13.555 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-04-22T18:43:13.557 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-04-22T18:43:13.559 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T18:43:13.559 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-04-22T18:43:13.559 INFO:teuthology.orchestra.run.smithi089.stdout:^- hv01.front.sepia.ceph.com 2 6 377 25 +1469us[+1461us] +/- 572ms 2024-04-22T18:43:13.560 INFO:teuthology.orchestra.run.smithi089.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -20us[ -28us] +/- 54ms 2024-04-22T18:43:13.560 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 +26us[ +26us] +/- 91ms 2024-04-22T18:43:13.560 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T18:43:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T18:43:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-04-22T18:43:13.561 INFO:teuthology.orchestra.run.smithi121.stdout:^- hv01.front.sepia.ceph.com 2 6 377 26 +1485us[+1488us] +/- 572ms 2024-04-22T18:43:13.562 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -12us[-9927ns] +/- 54ms 2024-04-22T18:43:13.562 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +15us[ +18us] +/- 91ms 2024-04-22T18:43:13.562 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T18:43:13.563 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T18:43:13.572 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T18:43:13.573 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T18:43:13.581 DEBUG:teuthology.orchestra.run.smithi089:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T18:43:13.685 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T18:43:13.763 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T18:43:13.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T18:43:13.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T18:43:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T18:43:13.823 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T18:43:13.837 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T18:43:13.847 INFO:teuthology.task.internal:Duration was 780.898400 seconds 2024-04-22T18:43:13.847 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T18:43:13.856 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T18:43:13.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T18:43:13.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T18:43:13.894 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T18:43:13.900 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T18:43:14.188 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T18:43:14.188 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com 2024-04-22T18:43:14.189 DEBUG:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T18:43:14.224 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-04-22T18:43:14.224 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T18:43:14.279 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T18:43:14.279 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T18:43:14.297 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T18:43:14.362 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T18:43:14.362 DEBUG:teuthology.orchestra.run.smithi089:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T18:43:14.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T18:43:14.538 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T18:43:14.960 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T18:43:14.960 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T18:43:14.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T18:43:15.011 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T18:43:15.068 DEBUG:teuthology.orchestra.run.smithi089:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T18:43:15.072 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T18:43:15.092 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = core 2024-04-22T18:43:15.095 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-04-22T18:43:15.121 DEBUG:teuthology.orchestra.run.smithi089:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T18:43:15.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T18:43:15.172 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T18:43:15.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T18:43:15.187 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T18:43:15.202 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T18:43:15.203 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668454/remote/smithi089 2024-04-22T18:43:15.203 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T18:43:15.276 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668454/remote/smithi121 2024-04-22T18:43:15.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T18:43:15.310 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T18:43:15.311 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T18:43:15.313 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T18:43:15.360 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T18:43:15.376 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T18:43:15.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T18:43:15.386 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T18:43:15.387 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T18:43:15.389 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T18:43:15.401 INFO:teuthology.orchestra.run.smithi089.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 18:43 /home/ubuntu/cephtest 2024-04-22T18:43:15.414 INFO:teuthology.orchestra.run.smithi121.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 18:43 /home/ubuntu/cephtest 2024-04-22T18:43:15.416 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T18:43:15.429 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T18:43:15.453 INFO:teuthology.run:Summary data: description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/dedup-io-mixed} duration: 780.8983995914459 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-04-22T18:43:15.453 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T18:43:15.551 INFO:teuthology.run:pass